From vlad at lists.openfabrics.org Mon Jun 1 03:24:29 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 1 Jun 2009 03:24:29 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090601-0200 daily build status Message-ID: <20090601102429.5A595E615F1@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hnrose at comcast.net Mon Jun 1 04:46:22 2009 From: hnrose at comcast.net (hnrose at comcast.net) Date: Mon, 1 Jun 2009 07:46:22 -0400 Subject: [ofa-general] [PATCH] libibmad/resolve.c: Remove compile warning on x86_64 arch Message-ID: <20090601114622.GA31380@comcast.net> resolve.c:106: warning: pointer targets in passing argument 1 of 'mad_decode_field' differ in signedness Signed-off-by: Hal Rosenstock diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c index f17da11..abcab17 100644 --- a/libibmad/src/resolve.c +++ b/libibmad/src/resolve.c @@ -103,7 +103,7 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, buf)) < 0) return -1; - mad_decode_field(buf, IB_SA_PR_SL_F, &portid->sl); + mad_decode_field((uint8_t *)buf, IB_SA_PR_SL_F, &portid->sl); return 0; } From hal.rosenstock at gmail.com Mon Jun 1 04:50:24 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 1 Jun 2009 07:50:24 -0400 Subject: [ofa-general] [PATCH] infiniband-diags: Do not change logical state on SubnAdmSet In-Reply-To: <4A2297DE.3050707@gmail.com> References: <4A2297DE.3050707@gmail.com> Message-ID: On Sun, May 31, 2009 at 10:44 AM, Eli Dorfman (Voltaire) wrote: > Do not change logical state on SubnAdmSet The method is SubnSet r.t. SubnAdmSet. Subject might also indicate ibportstate.c and it would be clearer saying logical port state r.t. just logical state. -- Hal > When changing physical state do not change logical port state. > >From the IB spec When writing PortInfo:PortState, only legal transitions are > valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. > > This patch allows reset in a single MAD. > > Signed-off-by: Eli Dorfman > --- >  infiniband-diags/src/ibportstate.c |    5 ++++- >  1 files changed, 4 insertions(+), 1 deletions(-) > > diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c > index 65c9ca1..d19a2e5 100644 > --- a/infiniband-diags/src/ibportstate.c > +++ b/infiniband-diags/src/ibportstate.c > @@ -275,8 +275,10 @@ int main(int argc, char **argv) > >        /* Only if one of the "set" options is chosen */ >        if (port_op) { > -               if (port_op == 1)               /* Enable port */ > +               if (port_op == 1) {             /* Enable port */ >                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2);        /* Polling */ > +                       mad_set_field(data, 0, IB_PORT_STATE_F, 0);             /* No Change */ > +               } >                else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ >                        mad_set_field(data, 0, IB_PORT_STATE_F, 1);             /* Down */ >                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3);        /* Disabled */ > @@ -292,6 +294,7 @@ int main(int argc, char **argv) > >                if (port_op == 3) {     /* Reset port - so also enable */ >                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2);        /* Polling */ > +                       mad_set_field(data, 0, IB_PORT_STATE_F, 0);             /* No Change */ >                        err = set_port_info(&portid, data, portnum, port_op); >                        if (err < 0) >                                IBERROR("smp set portinfo failed"); > -- > 1.5.5 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From michael.heinz at qlogic.com Mon Jun 1 07:01:23 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Mon, 1 Jun 2009 09:01:23 -0500 Subject: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> One of our testers recently reported that he was getting different performance numbers from IPOIB depending on whether he was using interface ib0 or a child interface he had created with /sys/class/netib0/create_child. Investigating, I noticed that the two interfaces have different MTU sizes: ib0 Link encap:UNSPEC HWaddr 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:172.21.33.58 Bcast:172.21.255.255 Mask:255.255.0.0 inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:76 errors:0 dropped:0 overruns:0 frame:0 TX packets:23147668 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:5284 (5.1 Kb) TX bytes:17365044664 (16560.5 Mb) ib0.9001 Link encap:UNSPEC HWaddr 80-00-00-06-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:172.21.34.58 Bcast:172.21.255.255 Mask:255.255.0.0 inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:60413619 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:112572449188 (107357.4 Mb) TX bytes:0 (0.0 b) This leads me to believe that the child interface is in datagram mode even though the parent is in connected mode. Is this expected behavior? Is there a way to change this? -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Mon Jun 1 08:18:39 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 1 Jun 2009 18:18:39 +0300 Subject: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> Message-ID: <20090601151839.GA12578@mtls03> On Mon, Jun 01, 2009 at 09:01:23AM -0500, Mike Heinz wrote: > One of our testers recently reported that he was getting different performance numbers from IPOIB depending on whether he was using interface ib0 or a child interface he had created with /sys/class/netib0/create_child. > > Investigating, I noticed that the two interfaces have different MTU sizes: > > ib0 Link encap:UNSPEC HWaddr 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:172.21.33.58 Bcast:172.21.255.255 Mask:255.255.0.0 > inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 > RX packets:76 errors:0 dropped:0 overruns:0 frame:0 > TX packets:23147668 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:256 > RX bytes:5284 (5.1 Kb) TX bytes:17365044664 (16560.5 Mb) > > ib0.9001 Link encap:UNSPEC HWaddr 80-00-00-06-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:172.21.34.58 Bcast:172.21.255.255 Mask:255.255.0.0 > inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 > RX packets:60413619 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:256 > RX bytes:112572449188 (107357.4 Mb) TX bytes:0 (0.0 b) > > > This leads me to believe that the child interface is in datagram mode even though the parent is in connected mode. Probably it is, but you can always check the mode by reading the corresponding sysfs file: /sys/class/net/ib0.9001/mode > > Is this expected behavior? Yes, it is the expected behaviour. > Is there a way to change this? > Yes, it is possible > -- > Michael Heinz > Principal Engineer, Qlogic Corporation > King of Prussia, Pennsylvania > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From michael.heinz at qlogic.com Mon Jun 1 08:27:10 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Mon, 1 Jun 2009 10:27:10 -0500 Subject: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. In-Reply-To: <20090601151839.GA12578@mtls03> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> <20090601151839.GA12578@mtls03> Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BDD@MNEXMB1.qlogic.org> >> This leads me to believe that the child interface is in datagram mode even though the parent is in connected mode. >Probably it is, but you can always check the mode by reading the corresponding sysfs file: >/sys/class/net/ib0.9001/mode >> >> Is this expected behavior? >Yes, it is the expected behaviour. >> Is there a way to change this? >> >Yes, it is possible Is that a snarky way of telling me there are no plans to modify /etc/init/openibd to support connected mode for child interfaces? -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania > -- > Michael Heinz > Principal Engineer, Qlogic Corporation > King of Prussia, Pennsylvania > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From michael.heinz at qlogic.com Mon Jun 1 08:48:21 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Mon, 1 Jun 2009 10:48:21 -0500 Subject: [ofa-general] Ah - we got it. In-Reply-To: <20090601151839.GA12578@mtls03> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> <20090601151839.GA12578@mtls03> Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BE4@MNEXMB1.qlogic.org> The problem is that, apparently, ipoib is getting loaded even though it is turned off in openib.conf - so ib0 was coming up without the init script ever setting the options for it. -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania -----Original Message----- From: Eli Cohen [mailto:eli at dev.mellanox.co.il] Sent: Monday, June 01, 2009 11:19 AM To: Mike Heinz Cc: general at lists.openfabrics.org Subject: Re: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. On Mon, Jun 01, 2009 at 09:01:23AM -0500, Mike Heinz wrote: > One of our testers recently reported that he was getting different performance numbers from IPOIB depending on whether he was using interface ib0 or a child interface he had created with /sys/class/netib0/create_child. > > Investigating, I noticed that the two interfaces have different MTU sizes: > > ib0 Link encap:UNSPEC HWaddr 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:172.21.33.58 Bcast:172.21.255.255 Mask:255.255.0.0 > inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 > RX packets:76 errors:0 dropped:0 overruns:0 frame:0 > TX packets:23147668 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:256 > RX bytes:5284 (5.1 Kb) TX bytes:17365044664 (16560.5 Mb) > > ib0.9001 Link encap:UNSPEC HWaddr 80-00-00-06-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:172.21.34.58 Bcast:172.21.255.255 Mask:255.255.0.0 > inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 > RX packets:60413619 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:256 > RX bytes:112572449188 (107357.4 Mb) TX bytes:0 (0.0 b) > > > This leads me to believe that the child interface is in datagram mode even though the parent is in connected mode. Probably it is, but you can always check the mode by reading the corresponding sysfs file: /sys/class/net/ib0.9001/mode > > Is this expected behavior? Yes, it is the expected behaviour. > Is there a way to change this? > Yes, it is possible > -- > Michael Heinz > Principal Engineer, Qlogic Corporation > King of Prussia, Pennsylvania > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hal.rosenstock at gmail.com Mon Jun 1 08:53:36 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 1 Jun 2009 11:53:36 -0400 Subject: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> Message-ID: On Mon, Jun 1, 2009 at 10:01 AM, Mike Heinz wrote: > One of our testers recently reported that he was getting different > performance numbers from IPOIB depending on whether he was using interface > ib0 or  a child interface he had created with > /sys/class/netib0/create_child. > > > > Investigating, I noticed that the two interfaces have different MTU sizes: > > > > ib0       Link encap:UNSPEC  HWaddr > 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00 > >           inet addr:172.21.33.58  Bcast:172.21.255.255  Mask:255.255.0.0 > >           inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > >           UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1 > >           RX packets:76 errors:0 dropped:0 overruns:0 frame:0 > >           TX packets:23147668 errors:0 dropped:0 overruns:0 carrier:0 > >           collisions:0 txqueuelen:256 > >           RX bytes:5284 (5.1 Kb)  TX bytes:17365044664 (16560.5 Mb) > > > > ib0.9001  Link encap:UNSPEC  HWaddr > 80-00-00-06-FE-80-00-00-00-00-00-00-00-00-00-00 > >           inet addr:172.21.34.58  Bcast:172.21.255.255  Mask:255.255.0.0 > >           inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > >           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1 > >           RX packets:60413619 errors:0 dropped:0 overruns:0 frame:0 > >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >           collisions:0 txqueuelen:256 > >           RX bytes:112572449188 (107357.4 Mb)  TX bytes:0 (0.0 b) > > > > > > This leads me to believe that the child interface is in datagram mode even > though the parent is in connected mode. > > > > Is this expected behavior? Is there a way to change this? Can the child interface's mode (/sys/class/net/ib0.9001/mode) be changed (to connected) ? See Documentation/ipoib.txt -- Hal > -- > > Michael Heinz > > Principal Engineer, Qlogic Corporation > > King of Prussia, Pennsylvania > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From michael.heinz at qlogic.com Mon Jun 1 08:58:54 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Mon, 1 Jun 2009 10:58:54 -0500 Subject: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. In-Reply-To: References: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BBB@MNEXMB1.qlogic.org> Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E9069BE7@MNEXMB1.qlogic.org> Hal, We worked this out - once Eli poked me into digging deeper I noticed that the tester was bringing up the child interfaces by hand instead of at boot time - so the options weren't getting set correctly. -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania -----Original Message----- From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com] Sent: Monday, June 01, 2009 11:54 AM To: Mike Heinz Cc: general at lists.openfabrics.org Subject: Re: [ofa-general] Ipoib child interfaces are created in dgram mode, even if the config file (and main interface) specify connected mode. On Mon, Jun 1, 2009 at 10:01 AM, Mike Heinz wrote: > One of our testers recently reported that he was getting different > performance numbers from IPOIB depending on whether he was using interface > ib0 or  a child interface he had created with > /sys/class/netib0/create_child. > > > > Investigating, I noticed that the two interfaces have different MTU sizes: > > > > ib0       Link encap:UNSPEC  HWaddr > 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00 > >           inet addr:172.21.33.58  Bcast:172.21.255.255  Mask:255.255.0.0 > >           inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > >           UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1 > >           RX packets:76 errors:0 dropped:0 overruns:0 frame:0 > >           TX packets:23147668 errors:0 dropped:0 overruns:0 carrier:0 > >           collisions:0 txqueuelen:256 > >           RX bytes:5284 (5.1 Kb)  TX bytes:17365044664 (16560.5 Mb) > > > > ib0.9001  Link encap:UNSPEC  HWaddr > 80-00-00-06-FE-80-00-00-00-00-00-00-00-00-00-00 > >           inet addr:172.21.34.58  Bcast:172.21.255.255  Mask:255.255.0.0 > >           inet6 addr: fe80::211:7500:ff:8fa4/64 Scope:Link > >           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1 > >           RX packets:60413619 errors:0 dropped:0 overruns:0 frame:0 > >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >           collisions:0 txqueuelen:256 > >           RX bytes:112572449188 (107357.4 Mb)  TX bytes:0 (0.0 b) > > > > > > This leads me to believe that the child interface is in datagram mode even > though the parent is in connected mode. > > > > Is this expected behavior? Is there a way to change this? Can the child interface's mode (/sys/class/net/ib0.9001/mode) be changed (to connected) ? See Documentation/ipoib.txt -- Hal > -- > > Michael Heinz > > Principal Engineer, Qlogic Corporation > > King of Prussia, Pennsylvania > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From taylor at hpc.ufl.edu Mon Jun 1 08:58:27 2009 From: taylor at hpc.ufl.edu (Charles Taylor) Date: Mon, 1 Jun 2009 11:58:27 -0400 Subject: [ofa-general] Distributing ib_mthca interrupts Message-ID: <5DCFD342-C658-4EAA-8FBB-01450E61CF55@hpc.ufl.edu> We have servers that have both IB and ethernet-only clients. The ethernet-only clients access the servers via IPoIB through a gateway. For small MTUs, the IpoIB software interrupt load can get pretty unmanageable so we would like to be able to spread the load over multiple cores. Our current /proc/interrupts looks like... cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 snip... 67: 946 0 0 0 1364412993 0 0 0 PCI-MSI-X ib_mthca (comp) 75: 1785 0 0 0 6286882 0 0 0 PCI-MSI-X ib_mthca (async) snip... We'd like to be able to spread the irq 67 interrupts across several or all CPUs but echo f0 > /proc/irq/67/smp_affinity seems to have no effect. We can move the interrupts to any *single* cpu we like but we cannot distribute them among multiple cpus. The results is that under heavy load from the ethernet-only clients, the IPoIB traffic becomes interrupt bound. We found a site (http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux ) that suggests that disabling CONFIG_HOTPLUG_CPU will allow /proc/irq/ nn/smp_affinity to work as expected. My efforts so far with the 2.6.18-92 (RH 5.2) and the 2.6.18-128 kernels have resulted in kernels that don't boot (after simply configuring CONFIG_HOTPLUG_CPU=n). I've googled around a bit but have not come up with much. I thought I'd ask here since we are specifically interested in being able to do this for our IB HCAs. Is this a known issue and is there a way to make it work? Thanks, Charlie Taylor UF HPC Center -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahomike at us.ibm.com Mon Jun 1 10:12:55 2009 From: ahomike at us.ibm.com (Mike Aho) Date: Mon, 1 Jun 2009 12:12:55 -0500 Subject: [ofa-general] RDMA_CM--how to include an SRQ? Message-ID: Is there an example of how to incorporate an SRQ into using RDMA CM and IB verbs? Thank you for any assistance or suggestions. Best regards, Mike Aho -------------- next part -------------- An HTML attachment was scrubbed... URL: From yosefe at voltaire.com Mon Jun 1 10:50:06 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Mon, 01 Jun 2009 20:50:06 +0300 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <20090528175757.GA95655@nas.nasa.gov> References: <20090528175757.GA95655@nas.nasa.gov> Message-ID: <4A2414CE.7060300@voltaire.com> Bob Ciotti wrote: > Sorry to bounce this off the list - should it be too remedial. I promise > that I've been consuming a lot of the spec and OFA code. Maybe you consider > that a promise or a warning we will be more active :| > > Our configuration is >6000 CA in a mix of infinihostIII/connectx and > longbow extenders and >800 24 port switches on a single subnet. (SGI ICE > with lots of other stuff plugged in). Its DDR everywhere except across the > longbows. Hosts range from a few different generations of x86 xeon, x86 > opteron and itanium. We use lustre but have the srp traffic on a separate > subnet. > > A few weeks ago connection setup times were mentioned on this list along > with ARP and path record lookups not being scalable. We experience these > problems as well and need to address these scalability issues. I have a quite > a bit of test data and a few different ideas to bounce off the list RE path > records, once I am a little more versed in the spec. There has already been > some work done to limit ARP traffic. > > > Todays question has to do with SM errors. > We have been seeing lots of these - sometimes more than others. Digging > around some it appears that the 6777 represents the number of duplicates? > This value fluctuates around some, but not alot. Comments in the code > indicate that any valuse >1 is a problem. Question is, should or is this > OK to be happening and how does it occur? > > We will probably do an update to the 1.4 or 1.4.1 SM in the next few days. > We are currently running a pre 1.4 top of tree pull from back in dec. bob > > This may be the path record query triggered by ipoib (ib_sa_path_rec_get). It uses METHOD_GET and if there is more than one path record it will fail. using METHOD_GET_TABLE should solve it (the fix is in ib_sa module.) If you enable debugging in ipoib and see path record query failures, this is probably it. --Yossi. From sean.hefty at intel.com Mon Jun 1 10:57:16 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 1 Jun 2009 10:57:16 -0700 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <4A2414CE.7060300@voltaire.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> Message-ID: <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> >This may be the path record query triggered by ipoib (ib_sa_path_rec_get). >It uses METHOD_GET and if there is more than one path record it will fail. >using METHOD_GET_TABLE should solve it (the fix is in ib_sa module.) I believe this is a bug in the SA, not ipoib or ib_sa. The SA should simply select a single PR and return it. - Sean From yosefe at voltaire.com Mon Jun 1 11:11:25 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Mon, 01 Jun 2009 21:11:25 +0300 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> Message-ID: <4A2419CD.9060301@voltaire.com> Sean Hefty wrote: >> This may be the path record query triggered by ipoib (ib_sa_path_rec_get). >> It uses METHOD_GET and if there is more than one path record it will fail. >> using METHOD_GET_TABLE should solve it (the fix is in ib_sa module.) > > I believe this is a bug in the SA, not ipoib or ib_sa. The SA should simply > select a single PR and return it. > > - Sean I could not find anywhere in the spec how should the SA respond to SubnAdmGet() in case there is more than one record. What I did find is an example of path query mad, and it was with SubnAdmGetTable(). --Yossi From sean.hefty at intel.com Mon Jun 1 11:19:55 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 1 Jun 2009 11:19:55 -0700 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <4A2419CD.9060301@voltaire.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <4A2419CD.9060301@voltaire.com> Message-ID: <60CB3B7CB0824330A7298621A2CF5A62@amr.corp.intel.com> >I could not find anywhere in the spec how should the SA respond to >SubnAdmGet() in case there is more than one record. What I did find >is an example of path query mad, and it was with SubnAdmGetTable(). PR NumbPath - 'In a SubnAdmGet() query request, ignored; a value of 1 is used.' I'm not sure how else you can interpret this except to mean the same as for SubAdmGetTable: 'If more paths that satisfy the PathRecord query exist for a given SGID-DGID combination, only NumbPath paths shall be returned (implementation defined).' - Sean From hal.rosenstock at gmail.com Mon Jun 1 11:29:27 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 1 Jun 2009 14:29:27 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> Message-ID: On Mon, Jun 1, 2009 at 1:57 PM, Sean Hefty wrote: >>This may be the path record query triggered by ipoib (ib_sa_path_rec_get). >>It uses METHOD_GET and if there is more than one path record it will fail. >>using METHOD_GET_TABLE should solve it (the fix is in ib_sa module.) > > I believe this is a bug in the SA, not ipoib or ib_sa.  The SA should simply > select a single PR and return it. No, it is correct as is (returning an error of too many records for this case). See p.944: 15.4.6 SUBNADMGET() / SUBNADMGETRESP(): GET AN ATTRIBUTE C15-0.1.30: Ine response to a SubnAdmGet(), if a single attribute would be returned based on the access rules specified in 15.4.1 Restrictions on Access on page 938 and the matching of components specified by the ComponentMask, then SubAdmGetResp() shall return that attribute with a zero status value. C15-0.1.31: If SubnAdmGet() fails to satisfy C15-0.1.30:, SubnAdmGet- Resp() shall return with the status field providing the reason for failure (see Table 190 SA MAD Class-Specific Status Encodings on page 900). -- Hal > > - Sean > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sean.hefty at intel.com Mon Jun 1 11:41:51 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 1 Jun 2009 11:41:51 -0700 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> Message-ID: <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> >No, it is correct as is (returning an error of too many records for >this case). See p.944: > >15.4.6 SUBNADMGET() / SUBNADMGETRESP(): GET AN ATTRIBUTE > >C15-0.1.30: Ine response to a SubnAdmGet(), if a single attribute would >be returned based on the access rules specified in 15.4.1 Restrictions on >Access on page 938 and the matching of components specified by the >ComponentMask, then SubAdmGetResp() shall return that attribute with >a zero status value. > >C15-0.1.31: If SubnAdmGet() fails to satisfy C15-0.1.30:, SubnAdmGet- >Resp() shall return with the status field providing the reason for failure >(see Table 190 SA MAD Class-Specific Status Encodings on page 900). This ignores NumbPath = 1 (or defines NumbPath differently for PR SubAdmGet versus SubAdmGetTable). With NumbPath = 1, only a single attribute should be returned. - Sean From hal.rosenstock at gmail.com Mon Jun 1 12:00:57 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 1 Jun 2009 15:00:57 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> Message-ID: On Mon, Jun 1, 2009 at 2:41 PM, Sean Hefty wrote: >>No, it is correct as is (returning an error of too many records for >>this case). See p.944: >> >>15.4.6 SUBNADMGET() / SUBNADMGETRESP(): GET AN ATTRIBUTE >> >>C15-0.1.30: Ine response to a SubnAdmGet(), if a single attribute would >>be returned based on the access rules specified in 15.4.1 Restrictions on >>Access on page 938 and the matching of components specified by the >>ComponentMask, then SubAdmGetResp() shall return that attribute with >>a zero status value. >> >>C15-0.1.31: If SubnAdmGet() fails to satisfy C15-0.1.30:, SubnAdmGet- >>Resp() shall return with the status field providing the reason for failure >>(see Table 190 SA MAD Class-Specific Status Encodings on page 900). > > This ignores NumbPath = 1 (or defines NumbPath differently for PR SubAdmGet > versus SubAdmGetTable).  With NumbPath = 1, only a single attribute should be > returned. Yes, it is different from GetTable in that SA pares the responses down to that but Get doesn't (have that additional language to pare them down). The language about NumbPath for Get was originally added to indicate that the NumbPath was ignored on a Get even if it was included in the component mask. -- Hal > - Sean > > From sean.hefty at intel.com Mon Jun 1 12:19:47 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 1 Jun 2009 12:19:47 -0700 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> Message-ID: >Yes, it is different from GetTable in that SA pares the responses down >to that but Get doesn't (have that additional language to pare them >down). This seems like an implementation issue (aka bug) with the SA to me. >The language about NumbPath for Get was originally added to indicate >that the NumbPath was ignored on a Get even if it was included in the >component mask. It states that it's ignored and a value of 1 is used. What else would a NumbPath value of 1 mean if it's completely ignored? I consider this a spec bug. :) >From an implementation view, requiring users to use SubnAdmGetTable to get a single path record is less efficient than returning a single PR from SubnAdmGet. How have other SM implementations (not based on opensm) interpreted NumbPath for PR SubnAdmGet? From hal.rosenstock at gmail.com Mon Jun 1 13:12:24 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 1 Jun 2009 16:12:24 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> Message-ID: On 6/1/09, Sean Hefty wrote: >>Yes, it is different from GetTable in that SA pares the responses down >>to that but Get doesn't (have that additional language to pare them >>down). > > This seems like an implementation issue (aka bug) with the SA to me. No; it's what the spec says. >>The language about NumbPath for Get was originally added to indicate >>that the NumbPath was ignored on a Get even if it was included in the >>component mask. > > It states that it's ignored and a value of 1 is used. What else would a > NumbPath value of 1 mean if it's completely ignored? I consider this a spec > bug. :) It was also to deal with the broken case of Get PR with NumbPath > 1. > From an implementation view, requiring users to use SubnAdmGetTable to get a > single path record is less efficient than returning a single PR from > SubnAdmGet. Yes, RMPP is an overhead when the response is a single MAD but is this significant ? Anyhow, how can the spec be changed in a way that doesn't break existing implementations ? -- Hal > How have other SM implementations (not based on opensm) interpreted NumbPath > for PR SubnAdmGet? > > From sean.hefty at intel.com Mon Jun 1 13:27:53 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 1 Jun 2009 13:27:53 -0700 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> Message-ID: <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> >Yes, RMPP is an overhead when the response is a single MAD but is this >significant ? Anyhow, how can the spec be changed in a way that >doesn't break existing implementations ? But the implementations are assuming different things about SubnAdmGet. The SA is assuming that the query should fail if multiple records match. The client side software (ipoib and rdma_cm) assume that it will obtain a single record even if multiple paths are present. So, something needs to change. The spec indicates that value in the request is ignored and NumbPath is 1, not that NumbPath is completely ignored. Also see page 1242 in the SDP annex which reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet rather than SubnAdmGetTable)'. To me, this implies that SubnAdmGet should be treated equivalent as SubnAdmGetTable with NumbPath = 1. It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus PR SubnAdmGetTable and MPR SubAdmGetMulti. Basically, this makes PR SubnAdmGet useless. - Sean From sean.hefty at intel.com Mon Jun 1 13:51:30 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 1 Jun 2009 13:51:30 -0700 Subject: [ofa-general] RDMA_CM--how to include an SRQ? In-Reply-To: References: Message-ID: <6C90A3C768EC46919049CD2BBF3F05DF@amr.corp.intel.com> >Is there an example of how to incorporate an SRQ into using RDMA CM and IB >verbs? Thank you for any assistance or suggestions. I don't believe so. libibverbs has an srq_pingpong example program that uses an SRQ. Using an SRQ with the rdma_cm is basically trivial if the QP is created using rdma_create_qp. The rdma_cm reads the struct ibv_srq * field from the struct ibv_qp when establishing a connection. If the QP is created directly from libibverbs (ibv_create_qp), then the user should just indicate that an SRQ is in use when connecting. Note that I don't believe there's no real requirement to indicate to the remote side of a connection that an SRQ is in use. The remote QP doesn't use this information. - Sean From hal.rosenstock at gmail.com Mon Jun 1 14:36:23 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 1 Jun 2009 17:36:23 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> Message-ID: On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >>Yes, RMPP is an overhead when the response is a single MAD but is this >>significant ? Anyhow, how can the spec be changed in a way that >>doesn't break existing implementations ? > > But the implementations are assuming different things about SubnAdmGet.  The SA > is assuming that the query should fail if multiple records match.  The client > side software (ipoib and rdma_cm) assume that it will obtain a single record > even if multiple paths are present.  So, something needs to change. Seems so. > The spec indicates that value in the request is ignored and NumbPath is 1, not > that NumbPath is completely ignored. For Get, it doesn't say that the matches are paired down to this number as it does for GetTable. >  Also see page 1242 in the SDP annex which > reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet > rather than SubnAdmGetTable)'. SDP annex is not the primary source for this (chapter 15 is) and is inconsistent and no one caught this. > To me, this implies that SubnAdmGet should be > treated equivalent as SubnAdmGetTable with NumbPath = 1. > It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus > PR SubnAdmGetTable and MPR SubAdmGetMulti.  Basically, this makes PR SubnAdmGet > useless. when there's a subnet with multiple paths and the requests are not specific enough to use get. Seems like either the queries need to use RMPP, or the spec modified (if that's possible) and the SAs updated. -- Hal > - Sean From akepner at sgi.com Mon Jun 1 15:04:13 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Mon, 1 Jun 2009 15:04:13 -0700 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: <4A226B60.9070006@Voltaire.com> References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> Message-ID: <20090601220413.GC14335@sgi.com> On Sun, May 31, 2009 at 02:34:56PM +0300, Or Gerlitz wrote: > .... Looking on the code, I'm not sure why the non sucess/flush path > of ipoib_cm_handle_tx_wc() must access the neighbour while > ipoib_ib_handle_tx_wc can get a way with only a warning print... > do we agree that accessing the neigbour from the cm tx completion > flow is buggy? But the fundamental problem is that the ipoib_neigh structures aren't read and written using consitent locking (particularly in ipoib_neigh_cleanup()). What about introducing a set of locks for no other purpose than protecting ipoib_neigh structures? Something like this (which is only intended to convey the idea - it's obviously incomplete): NOT-Signed-off-by: Arthur Kepner --- diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 753a983..cc20074 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -403,6 +403,30 @@ static inline int ipoib_ud_need_sg(unsigned int ib_mtu) return IPOIB_UD_BUF_SIZE(ib_mtu) > PAGE_SIZE; } +struct ipoib_neigh_lock +{ + spinlock_t sl; +}__attribute__((__aligned__(SMP_CACHE_BYTES))); + +#define IPOIB_LOCK_SHIFT 6 +#define IPOIB_LOCK_SIZE (1 << IPOIB_LOCK_SHIFT) +#define IPOIB_LOCK_MASK (IPOIB_LOCK_SIZE -1) + +static struct ipoib_neigh_lock +ipoib_neigh_locks[IPOIB_LOCK_SIZE] __cacheline_aligned; + +static inline void lock_ipoib_neigh(unsigned int hval) +{ + spin_lock(&ipoib_neigh_locks[hval & IPOIB_LOCK_MASK].sl); +} + +static inline void unlock_ipoib_neigh(unsigned int hval) +{ + spin_unlock(&ipoib_neigh_locks[hval & IPOIB_LOCK_MASK].sl); +} + +unsigned int ipoib_neigh_hval(struct neighbour *n); + /* * We stash a pointer to our private neighbour information after our * hardware address in neigh->ha. The ALIGN() expression here makes diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index ab2c192..685b664 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -45,6 +45,7 @@ #include #include +#include #include @@ -844,6 +845,9 @@ static void ipoib_neigh_cleanup(struct neighbour *n) struct ipoib_dev_priv *priv = netdev_priv(n->dev); unsigned long flags; struct ipoib_ah *ah = NULL; + unsigned int hval = ipoib_neigh_hval(n); + + lock_ipoib_neigh(hval); neigh = *to_ipoib_neigh(n); if (neigh) @@ -864,6 +868,8 @@ static void ipoib_neigh_cleanup(struct neighbour *n) spin_unlock_irqrestore(&priv->lock, flags); + unlock_ipoib_neigh(hval); + if (ah) ipoib_put_ah(ah); } @@ -899,6 +905,13 @@ void ipoib_neigh_free(struct net_device *dev, struct ipoib_neigh *neigh) kfree(neigh); } +static unsigned int ipoib_neigh_rnd; + +unsigned int ipoib_neigh_hval(struct neighbour *n) +{ + return jhash(n->primary_key, n->tbl->key_len, ipoib_neigh_rnd); +} + static int ipoib_neigh_setup_dev(struct net_device *dev, struct neigh_parms *parms) { parms->neigh_cleanup = ipoib_neigh_cleanup; @@ -1408,6 +1421,8 @@ static int __init ipoib_init_module(void) ipoib_max_conn_qp = min(ipoib_max_conn_qp, IPOIB_CM_MAX_CONN_QP); #endif + get_random_bytes(&ipoib_neigh_rnd, sizeof(ipoib_neigh_rnd)); + /* * When copying small received packets, we only copy from the * linear data part of the SKB, so we rely on this condition. From ogerlitz at voltaire.com Tue Jun 2 01:49:09 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 02 Jun 2009 11:49:09 +0300 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: <20090601220413.GC14335@sgi.com> References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> Message-ID: <4A24E785.8070805@voltaire.com> akepner at sgi.com wrote: > On Sun, May 31, 2009 at 02:34:56PM +0300, Or Gerlitz wrote: >> do we agree that accessing the neigbour from the cm tx completion flow is buggy? > But the fundamental problem is that the ipoib_neigh structures aren't read and written using consitent locking (particularly in ipoib_neigh_cleanup()). I am still not with you as for the root cause of the problem... if by the time the neigh cleanup code is called, no locking would be required (since on one should have refs in that step) --> no consistency is needed. From what I understand the only code that can race with the neigh cleanup is the cm tx completion, do you agree to this assertion? do you see the race happening when working in datagram mode? Or. From alexander.supalov at intel.com Tue Jun 2 03:01:26 2009 From: alexander.supalov at intel.com (Supalov, Alexander) Date: Tue, 2 Jun 2009 11:01:26 +0100 Subject: [ofa-general] Memory registration redux In-Reply-To: References: <20090506214628.GM2590@obsidianresearch.com><20090506222638.GA16280@obsidianresearch.com><20090507000231.GB16280@obsidianresearch.com><20090507224806.GF16280@obsidianresearch.com> <5019F239-149F-49E1-8C23-436DE6094AB2@cisco.com> Message-ID: <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> Hi, Sorry, it's kind of difficult to deduce looking at this Q&A sequence what works how and when. Is it possible to create a brief and direct description of the proposed solution? Best regards. Alexander -----Original Message----- From: Jeff Squyres [mailto:jsquyres at cisco.com] Sent: Wednesday, May 27, 2009 9:03 PM To: Roland Dreier (rdreier) Cc: Pavel Shamis; Hans Westgaard Ry; Dontje; Lenny Verkhovsky; H??kon Bugge; Donald Kerr; OpenFabrics General; Supalov, Alexander Subject: Re: [ofa-general] Memory registration redux Other MPI implementors -- what do you think of this scheme? On May 27, 2009, at 1:49 PM, Roland Dreier (rdreier) wrote: > > > > /* > > > * If type field is INVAL, then user_cookie_counter holds the > > > * user_cookie for the region being reported; if the HINT flag > is set > > > * then hint_start/hint_end hold the start and end of the > mapping that > > > * was invalidated. (If HINT is not set, then multiple events > > > * invalidated parts of the registered range and hint_start/ > hint_end > > > * should be ignored) > > > I don't quite grok this. Is the intent that HINT will only be > set if > > an *entire* hint_start/hint_end range is invalidated by a single > > event? I.e., if only part of the hint_start/hint_end range is > > invalidated, you'll get the cookie back, but not what part of the > > range is invalid (because assumedly the entire IBV registration > is now > > invalid anyway)? > > Basically, I just keep one hint_start/hint_end. If multiple events > hit > the same registration then I just give up and don't give you a hint. > > > > * If type is LAST, then the read operation has emptied the > list of > > > * invalidated regions, and user_cookie_counter holds the value > of the > > > * kernel's generation counter when the empty list occurred. The > > > * other fields are not filled in for this event. > > > Just to be clear -- we're supposed to keep reading events until > we get > > a LAST event? > > Yes, that's probably the sanest use case. > > > 1. Will it increase by 1 each time a page (or set of pages?) is > > removed from a user process? > > As it stands it increases by 1 every time there is an MMU > notification, > even if that notification hits multiple registrations. It wouldn't be > hard to change that to count the number of events generated if that > works better. > > > 2. Does it change if pages are *added* to a user process? I.e., > does > > the counter indicate *removals* or *changes* to the user process > page > > table? > > No, additions don't trigger any MMU notification -- that's inherent in > the design of the MMU notifiers stuff. The idea is that you have a > "secondary MMU" and MMU notifications are the equivalent of TLB > shootdowns; the secondary MMU is responsible for populating itself on > faults etc. > > > Is the *unm_counter value guaranteed to have been changed by the > time > > munmap() returns? > > Yes. > > > Did you pick [2] here simply because you're only expecting an INVAL > > and a LAST event in this specific example? I'm assuming that we > > should normally loop over reading until we get LAST, correct? > > Right. > > > What happens if I register multiple regions with the same cookie > value? > > You get in trouble -- I need to fix things to reject duplicated > cookies > actually, because otherwise there's no way to unregister. > > > Is a process responsible for guaranteeing that it umn_unregister()s > > everything before exiting, or will all pending registrations be > > cleaned up/unregistered/whatever when a process exits? > > The kernel cleans up of course to handle crashes etc. > > - R. > -- Jeff Squyres Cisco Systems --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From vlad at lists.openfabrics.org Tue Jun 2 03:27:18 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 2 Jun 2009 03:27:18 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090602-0200 daily build status Message-ID: <20090602102718.F1D49E61713@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From devel-ofed at morey-chaisemartin.com Tue Jun 2 05:11:00 2009 From: devel-ofed at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Tue, 02 Jun 2009 14:11:00 +0200 Subject: [ofa-general] [PATCH] osm_ucast_ftree.c Allow horizontal links between switches of max rank In-Reply-To: <4A1D10ED.6040001@Sun.COM> References: <4A1D10ED.6040001@Sun.COM> Message-ID: <4A2516D4.1020502@morey-chaisemartin.com> Hi, I've just run a few tests with your patch using io_guid_files and reverse_hops and it seems your patch introduces a bug. We have a function to calculate the number per link and if I apply this patch it never returns which usually means there is a loop in the routing scheme. I have had a quick look at your code. One thing I'm not sure of is that when a reverse hop is done you increment both the reverse_hop counter and the current_hops one. Which mean when the hops count is calculate for sw_set_hops it is bigger than it is supposed to be. What should probably be done is increment both but only use current_hops to calculate the number of hops done. reverse_hop will only be used to check if there are still reverse_hop credits left or not (which was its original purpose). By changing the hops count in both call to set_hops_on_remote_sw to current_hops+1 and not current_hops+1 +2* reverse_hops, routing has no loops but results are different (some links are more loaded). I'll run some more tests to see if I can figure this out. Nicolas Le 27/05/2009 12:07, Line.Holen at Sun.COM a écrit : > This patch makes it legal to have cross links (horizontal links) between > switches at max rank. These switches do have same rank, so hop count cannot > be calculated based on rank anymore. > The horizontal links are treated as downlinks. Switch A has a downlink to B > while B has a downlink to A. Tests on lids and also number of hops makes sure > that we don't loop back and forth across the link. > > Signed-off-by: Frank Olaf Sem-Jacobsen > Signed-off-by: Line Holen > From hal.rosenstock at gmail.com Tue Jun 2 06:51:02 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 2 Jun 2009 09:51:02 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> Message-ID: On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: > On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >>>Yes, RMPP is an overhead when the response is a single MAD but is this >>>significant ? Anyhow, how can the spec be changed in a way that >>>doesn't break existing implementations ? >> >> But the implementations are assuming different things about SubnAdmGet.  The SA >> is assuming that the query should fail if multiple records match.  The client >> side software (ipoib and rdma_cm) assume that it will obtain a single record >> even if multiple paths are present.  So, something needs to change. > > Seems so. > >> The spec indicates that value in the request is ignored and NumbPath is 1, not >> that NumbPath is completely ignored. > > For Get, it doesn't say that the matches are paired down to this > number as it does for GetTable. > >>  Also see page 1242 in the SDP annex which >> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet >> rather than SubnAdmGetTable)'. > > SDP annex is not the primary source for this (chapter 15 is) and is > inconsistent and no one caught this. > >> To me, this implies that SubnAdmGet should be >> treated equivalent as SubnAdmGetTable with NumbPath = 1. > >> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus >> PR SubnAdmGetTable and MPR SubAdmGetMulti.  Basically, this makes PR SubnAdmGet >> useless. > > when there's a subnet with multiple paths and the requests are not > specific enough to use get. > > Seems like either the queries need to use RMPP, or the spec modified > (if that's possible) and the SAs updated. I sit corrected :-) Your interpretation of the spec is correct. Also, in looking at OpenSM, the intent is as you indicate: it does try to only return 1 attibute for get PR. If when returning the response, there is more than 1 attribute in the list, it returns the too many records error. There must be some code path I don't see right now which is doing this. It would be useful to know the details of the query (get request) causing this. -- Hal > -- Hal > >> - Sean > From rdreier at cisco.com Tue Jun 2 09:44:59 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 02 Jun 2009 09:44:59 -0700 Subject: [ofa-general] Memory registration redux In-Reply-To: <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> (Alexander Supalov's message of "Tue, 2 Jun 2009 11:01:26 +0100") References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> <5019F239-149F-49E1-8C23-436DE6094AB2@cisco.com> <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> Message-ID: > Sorry, it's kind of difficult to deduce looking at this Q&A sequence > what works how and when. Is it possible to create a brief and direct > description of the proposed solution? Did you see the original patch description I sent: As discussed in and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help. We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunot, that creates a /dev/ummunot node. A userspace process can open this node read-only and use the fd as follows: 1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunot_register_ioctl in ). 2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunot_event in ). select()/poll()/epoll() and SIGIO are handled for this IO. 3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call. From rdreier at cisco.com Tue Jun 2 09:46:17 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 02 Jun 2009 09:46:17 -0700 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: <20090601220413.GC14335@sgi.com> (akepner@sgi.com's message of "Mon, 1 Jun 2009 15:04:13 -0700") References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> Message-ID: Not sure I follow the proposal; where woul you call lock_ipoib_neigh() outside of ipoib_neigh_cleanup()? From akepner at sgi.com Tue Jun 2 09:49:35 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Tue, 2 Jun 2009 09:49:35 -0700 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> Message-ID: <20090602164935.GB26682@sgi.com> On Tue, Jun 02, 2009 at 09:46:17AM -0700, Roland Dreier wrote: > Not sure I follow the proposal; where woul you call lock_ipoib_neigh() > outside of ipoib_neigh_cleanup()? Anywhere to_ipoib_neigh() is used. I can send a complete patch (but I was sort of hoping that someone would have a better idea...) -- Arthur From rdreier at cisco.com Tue Jun 2 10:05:45 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 02 Jun 2009 10:05:45 -0700 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: <20090602164935.GB26682@sgi.com> (akepner@sgi.com's message of "Tue, 2 Jun 2009 09:49:35 -0700") References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> <20090602164935.GB26682@sgi.com> Message-ID: > I can send a complete patch (but I was sort of hoping that > someone would have a better idea...) I think it would be better to understand what ipoib_neigh_cleanup() is racing with. eg if CM TX completion handling is the problem, then maybe the right fix is to take a reference on the neigh when establishing a connection and drop the ref when closing the connection. From akepner at sgi.com Tue Jun 2 10:09:40 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Tue, 2 Jun 2009 10:09:40 -0700 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> <20090602164935.GB26682@sgi.com> Message-ID: <20090602170940.GC26682@sgi.com> On Tue, Jun 02, 2009 at 10:05:45AM -0700, Roland Dreier wrote: > .... > I think it would be better to understand what ipoib_neigh_cleanup() is > racing with. eg if CM TX completion handling is the problem, then maybe > the right fix is to take a reference on the neigh when establishing a > connection and drop the ref when closing the connection. OK, I'll have a look at that. Thanks. -- Arthur From tziporet at dev.mellanox.co.il Tue Jun 2 12:30:07 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 02 Jun 2009 22:30:07 +0300 Subject: [ofa-general] RFC: Do we wish to take MPI out of OFED? Message-ID: <4A257DBF.4000401@mellanox.co.il> Hello, In the EWG meeting yesterday there was a suggestion to take out all MPI packages including MPI tests out from OFED. Here are some of the reasons for keeping MPI in or out of OFED. Main reasons to take MPI out of OFED: - MPI is not developed under OFA - Need to synchronize between different projects - Some customers prefer to install a different MPI version then the one in OFED Main reasons to keep MPI in OFED: - All participants test with the same MPI versions, and when installing OFED it is ensured that MPI will work fine with this version. - Customers convenience in install (no need to go to more sites to get MPI) - MPI is an important RDMA ULP and although it is not developed in OFA it is widely used by OFED customers Before we decide to do such a change I would like to get the more opinions from the comunity. Thanks, Tziporet From rdreier at cisco.com Tue Jun 2 15:18:46 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 02 Jun 2009 15:18:46 -0700 Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined Message-ID: [I applied the patch below from Christoph, but I see now he sent it only to me and didn't cc the mailing list... so for completeness I'm including it below in case anyone else has concerns] From: Christoph Lameter The IPoIB drops multicast packets if more than 3 are queued while a multicast send is pending. The send queue can easily contain more than 3 packets and when the queue is processed only the first 3 packets will make it and the rest will be dropped. The IPoIB layer is unable to send a high speed stream of multicast traffic because of this. Since the packets are dropped the sender is never throttled and so continuous sending of data will lead to continual packet loss. To fix this, simply remove the code that drops packets, so the socket queue will build up and the sender will be throttled if it is continuously sending packets faster than the IPoIB layer can process them. This results in the IPoIB layer sending multicast traffic at the highest rate allowed by the underlying hardware and makes the IPoIB layer multicast behavior work like other ethernet NICs. (And it also means there's no risk of unbounded growth of pending packets, since they are accounted for as part of socket memory) We were able to send millions of UDP multicast messages at high rates after this patch. The tool for generating multicast traffic can be found on http://www.gentwo.org/ll. Signed-off-by: Christoph Lameter Signed-off-by: Roland Dreier --- drivers/infiniband/ulp/ipoib/ipoib.h | 1 - drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 7 +------ 2 files changed, 1 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 753a983..ddb5cd7 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -79,7 +79,6 @@ enum { IPOIB_NUM_WC = 4, IPOIB_MAX_PATH_REC_QUEUE = 3, - IPOIB_MAX_MCAST_QUEUE = 3, IPOIB_FLAG_OPER_UP = 0, IPOIB_FLAG_INITIALIZED = 1, diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c index 425e311..e6ea76a 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c @@ -685,12 +685,7 @@ void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb) } if (!mcast->ah) { - if (skb_queue_len(&mcast->pkt_queue) < IPOIB_MAX_MCAST_QUEUE) - skb_queue_tail(&mcast->pkt_queue, skb); - else { - ++dev->stats.tx_dropped; - dev_kfree_skb_any(skb); - } + skb_queue_tail(&mcast->pkt_queue, skb); if (test_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) ipoib_dbg_mcast(priv, "no address vector, " -- 1.6.0.4 From rdreier at cisco.com Tue Jun 2 15:33:53 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 02 Jun 2009 15:33:53 -0700 Subject: [ofa-general] InfiniBand/RDMA merge plans for 2.6.31 Message-ID: Since 2.6.30-rc7 has been out more than a week already, it's probably a good time to talk about 2.6.31 merge plans. All the pending things that I'm aware of are listed below. (For some reason this cycle seems to have not produced many patches -- we'll probably be very busy reviewing things for 2.6.32) Boilerplate: If something isn't already in my tree and it isn't listed below, I probably missed it or dropped it unintentionally. Please remind me. As usual, when submitting a patch: - Give a good changelog that explains what issue your patch addresses, how you address the issue, how serious the issue is, and any other information that would be useful to someone evaluating your patch or reading it years from now. - Please make sure that you include a "Signed-off-by:" line, and put any extra junk that should not go into the final kernel log *after* the "---" line so that git tools strip it off automatically. Make the subject line be appropriate for inclusion in the kernel log as well once the leading "[PATCH ...]" stuff is stripped off. I waste a lot of time fixing patches by hand that could otherwise be spent doing something productive like watching youtube. - Run your patch through checkpatch.pl so I don't have to nag you to fix trivial issues (or spend time fixing them myself). - Read your patch over so I don't see a memory leak or deadlock as soon as I look at it. - Build your patch with sparse checking ("C=2 CF=-D__CHECK_ENDIAN__") and make sure it doesn't introduce new warnings. (A big bonus in goodwill for sending patches that fix old warnings) - Test your patch on a kernel with things like slab debugging and lockdep turned on. And while you're waiting for me to get to your patch, I sure wouldn't mind if you read and commented on someone else's patch. None of this means you shouldn't remind me about pending patches, since I often lose track of things and drop them accidentally. Core: - Removal of unnecessary casts of void * to other pointer types. - Userspace MMU notifiers ("ummunot") -- the code looks great to me (since I wrote it), but I don't think this is suitable to merge through the InfiniBand/RDMA tree, and I think a prerequisite to even asking for broader review by the core kernel and VM community is to prove that this approach works by actually prototyping support for this in an MPI implementation. I'm working with Jeff Squyres on Open MPI support but I suspect we're already too close to the 2.6.31 window for this to be appropriate. However 2.6.32 should be easily doable, assuming the approach I took actually works and is palatable to the kernel community. ULPs: - One fix for IPoIB multicast packet drops. HW specific: - A few cxgb3 fixes. - A few optimizations to ehca QP creation. - Add a module parameter to mthca and mlx4 to be able to register more memory by increasing MTT segment size. - A fix to an almost-impossible-to-hit off-by-one in nes. Here are a few topics that I believe will not be ready in time for the 2.6.31 window and will need to wait for 2.6.32 at least: - Jack's XRC patch set. I still need time to work through and clean up the code. (I actually did make some progress on this during this cycle -- if you look closely you can see my xrc branch has changed -- but not enough to actually finish unfortunately) Here all the patches I already have in my for-next branch: Anton Blanchard (1): IB/ehca: Replace vmalloc() with kmalloc() for queue allocation Christoph Lameter (1): IPoIB: Don't drop multicast packets sent before group is joined Eli Cohen (2): IB/mthca: Add module parameter for number of MTTs per segment mlx4_core: Add module parameter for number of MTTs per segment Jack Stone (1): infiniband: Remove void casts Roel Kluin (1): RDMA/nes: Fix off-by-one bugs in reset_adapter_ne020() and init_serdes() Stefan Roscher (3): IB/ehca: Fall back to vmalloc() for big allocations IB/ehca: Remove unnecessary memory operations for userspace queue pairs IB/ehca: Increment version number Steve Wise (2): RDMA/cxgb3: Report correct port state and MTU RDMA/cxgb3: Limit fast register size based on T3 limitations From kliteyn at dev.mellanox.co.il Tue Jun 2 15:41:16 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 03 Jun 2009 01:41:16 +0300 Subject: [ofa-general] Re: osm_qos_parser_y question In-Reply-To: References: Message-ID: <4A25AA8C.4040607@dev.mellanox.co.il> Hi Hal, Hal Rosenstock wrote: > Hi Yevgeny, > > I was wondering about the following in osm_qos_parser_y: > > Both __parser_add_pkey_range_to_port_map and > __parser_add_partition_list_to_port_map seem to add the port map to > both the full and limited (partial) ports for that partition. Am I > reading this right ? Should it be that way ? They both do this: __parser_add_map_to_port_map(p_map, &p_prtn->part_guid_tbl); __parser_add_map_to_port_map(p_map, &p_prtn->full_guid_tbl); The function that they call is defined as follows: static void __parser_add_map_to_port_map( cl_qmap_t * p_dmap, cl_map_t * p_smap) Which means that they add all the guids from full and partial membership to the p_map. Looks OK to me. Perhaps you thought that they add p_map to p_prtn->full_guid_tbl and p_prtn->part_guid_tbl, and not vice versa? -- Yevgeny > Thanks. > > -- Hal > From sean.hefty at intel.com Tue Jun 2 17:10:11 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 2 Jun 2009 17:10:11 -0700 Subject: [ofa-general] [PATCH] dapl/windows cma provider: add support for network devices based on index Message-ID: <51EAC421351B42368735BDD2F53F8E7D@amr.corp.intel.com> The linux cma provider provides support for named network devices, such as 'ib0' or 'eth0'. This allows the same dapl configuration file to be used easily across a cluster. To allow similar support on Windows, allow users to specify the device name 'rdma_devN' in the dapl.conf file. The given index, N, is map to a corresponding IP address that is associated with an RDMA device. Signed-off-by: Sean Hefty --- diff -up -r -X \mshefty\scm\winof\trunk\docs\dontdiff.txt -I '\$Id:' trunk\ulp\dapl2/dapl/openib_cma/dapl_ib_util.c branches\winverbs\ulp\dapl2/dapl/openib_cma/dapl_ib_util.c --- trunk\ulp\dapl2/dapl/openib_cma/dapl_ib_util.c 2009-05-01 10:18:28.000000000 -0700 +++ branches\winverbs\ulp\dapl2/dapl/openib_cma/dapl_ib_util.c 2009-06-02 15:26:19.534649800 -0700 @@ -57,10 +57,50 @@ struct dapl_llist_entry *g_hca_list; #if defined(_WIN64) || defined(_WIN32) #include "..\..\..\..\..\etc\user\comp_channel.cpp" #include "..\..\..\..\..\etc\user\dlist.c" +#include -#define getipaddr_netdev(x,y,z) -1 struct ibvw_windata windata; +static int getipaddr_netdev(char *name, char *addr, int addr_len) +{ + IWVProvider *prov; + WV_DEVICE_ADDRESS devaddr; + struct addrinfo *res, *ai; + HRESULT hr; + int index; + + if (strncmp(name, "rdma_dev", 8)) { + return EINVAL; + } + + index = atoi(name + 8); + + hr = WvGetObject(&IID_IWVProvider, (LPVOID *) &prov); + if (FAILED(hr)) { + return hr; + } + + hr = getaddrinfo("..localmachine", NULL, NULL, &res); + if (hr) { + goto release; + } + + for (ai = res; ai; ai = ai->ai_next) { + hr = prov->lpVtbl->TranslateAddress(prov, ai->ai_addr, &devaddr); + if (SUCCEEDED(hr) && (ai->ai_addrlen <= addr_len) && (index-- == 0)) { + memcpy(addr, ai->ai_addr, ai->ai_addrlen); + goto free; + } + } + hr = ENODEV; + +free: + freeaddrinfo(res); +release: + prov->lpVtbl->Release(prov); + return hr; +} + static int dapls_os_init(void) { return ibvw_get_windata(&windata, IBVW_WINDATA_VERSION); diff -up -r -X \mshefty\scm\winof\trunk\docs\dontdiff.txt -I '\$Id:' trunk\ulp\dapl2/dapl/openib_cma/SOURCES branches\winverbs\ulp\dapl2/dapl/openib_cma/SOURCES --- trunk\ulp\dapl2/dapl/openib_cma/SOURCES 2009-05-27 07:25:19.000000000 -0700 +++ branches\winverbs\ulp\dapl2/dapl/openib_cma/SOURCES 2009-06-02 10:38:04.799012200 -0700 @@ -45,10 +45,12 @@ TARGETLIBS= \ $(SDK_LIB_PATH)\ws2_32.lib \ !if $(FREEBUILD) $(TARGETPATH)\*\dat2.lib \ + $(TARGETPATH)\*\winverbs.lib \ $(TARGETPATH)\*\libibverbs.lib \ $(TARGETPATH)\*\librdmacm.lib !else $(TARGETPATH)\*\dat2d.lib \ + $(TARGETPATH)\*\winverbsd.lib \ $(TARGETPATH)\*\libibverbsd.lib \ $(TARGETPATH)\*\librdmacmd.lib !endif From hilljj at ornl.gov Tue Jun 2 18:03:13 2009 From: hilljj at ornl.gov (Jason Hill) Date: Tue, 2 Jun 2009 21:03:13 -0400 Subject: [ofa-general] IB card manufacturer Message-ID: <20090603010313.GQ18018@cave.ccs.ornl.gov> All, Being somewhat cautious I'd like to see if anyone can point me at the correct manufacturer info for an Infiniband card that we bought through a cluster vendor (i.e. it was in the box when we got it). I need to upgrade the firmware to attempt to isolate an issue we're seeing with several nodes that have this combination: ==> /sys/class/infiniband/mthca0/board_id <== INT0020000001 ==> /sys/class/infiniband/mthca0/fw_ver <== 1.1.0 ==> /sys/class/infiniband/mthca0/hw_rev <== a0 ==> /sys/class/infiniband/mthca0/hca_type <== MT25204 ~$ lspci -vvv 07:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- On the card itself it has : Part Number: D83914 Serial Number: MEPT702000093 And the card has a tall bracket, but the vendor could have swapped that. Google has turned up nothing for the Board ID and part number/serial number. To me it looks kind of like a Cisco card, so I was thinking it might be a Topspin. We bought the cluster in 2007. Would the right thing be to install the MemFree MHGS18-XT firmware, or try a revision of the MHGS18-XTC? Thanks for the time and thoughts. -- -Jason ------------------------------------------------- // Jason J. Hill // // HPC Systems Administrator // // National Center for Computational Sciences // // Oak Ridge National Laboratory // // e-mail: hilljj at ornl.gov // // Phone: (865) 576-5867 // ------------------------------------------------- From panda at cse.ohio-state.edu Tue Jun 2 19:06:43 2009 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Tue, 2 Jun 2009 22:06:43 -0400 (EDT) Subject: [ofa-general] Announcing the Release of MVAPICH2 1.4RC1 Message-ID: The MVAPICH team is pleased to announce the release of MVAPICH2 1.4RC1 with the following NEW features: - MPI 2.1 standard compliant - Based on MPICH2 1.0.8p1 - Dynamic Process Management (DPM) Support with mpirun_rsh and MPD - Available for OpenFabrics (IB) interface - Support for eXtended Reliable Connection (XRC) - Available for OpenFabrics (IB) interface - Kernel-level single-copy intra-node communication support based on LiMIC2 - Delivers superior intra-node performance for medium and large messages - Available for all interfaces (IB, iWARP and uDAPL) - Enhancement to mpirun_rsh framework for faster job startup on large clusters - Hierarchical ssh to nodes to speedup job startup - Available for OpenFabrics (IB and iWARP), uDAPL interfaces (including Solaris) and the New QLogic-InfiniPath interface - Scalable checkpoint-restart with mpirun_rsh framework - Checkpoint-restart with intra-node shared memory (kernel-level with LiMIC2) support - Available for OpenFabrics (IB) Interface - K-nomial tree-based solution together with shared memory-based broadcast for scalable MPI_Bcast operation - Available for all interfaces (IB, iWARP and uDAPL) - Native support for QLogic InfiniPath - Provides support over PSM interface This release also contains multiple bug fixes since MVAPICH2-1.2p1. A summary of the major fixes are as follows: - Changed parameters for iWARP for increased scalability - Fix error with derived datatypes and Put and Accumulate operations - Unregister stale memory registrations earlier to prevent malloc failures - Fix for compilation issues with --enable-g=mem and --enable-g=all - Change dapl_prepost_noop_extra value from 5 to 8 to prevent credit flow issues - Re-enable RGET (RDMA Read) functionality - Fix SRQ Finalize error - Fix a multi-rail one-sided error when multiple QPs are used - PMI Lookup name failure with SLURM - Port auto-detection failure when the 1st HCA did not have an active failure - MPE support for shared memory collectives now available For downloading MVAPICH2 1.4RC1, associated user guide and accessing the SVN, please visit the following URL: http://mvapich.cse.ohio-state.edu All feedbacks, including bug reports and hints for performance tuning, patches and enhancements are welcome. Please post it to the mvapich-discuss mailing list. Thanks, The MVAPICH Team From ogerlitz at Voltaire.com Tue Jun 2 23:46:35 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Wed, 03 Jun 2009 09:46:35 +0300 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: References: <20090519215505.GN6837@sgi.com> <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> <20090602164935.GB26682@sgi.com> Message-ID: <4A261C4B.6090909@Voltaire.com> Roland Dreier wrote: > I think it would be better to understand what ipoib_neigh_cleanup() is > racing with. eg if CM TX completion handling is the problem, then maybe > the right fix is to take a reference on the neigh when establishing a > connection and drop the ref when closing the connection. or avoid accessing the neighbour in the cm tx completion... looking on the code I don't really see what neighbour access from the CQ handler buys the CM design, anyone has an idea? also, Arthur, again, does the race happen under datagram mode? Or. 797 if (wc->status != IB_WC_SUCCESS && 798 wc->status != IB_WC_WR_FLUSH_ERR) { 799 struct ipoib_neigh *neigh; 800 801 ipoib_dbg(priv, "failed cm send event " 802 "(status=%d, wrid=%d vend_err %x)\n", 803 wc->status, wr_id, wc->vendor_err); 804 805 spin_lock_irqsave(&priv->lock, flags); 806 neigh = tx->neigh; 807 808 if (neigh) { 809 neigh->cm = NULL; 810 list_del(&neigh->list); 811 if (neigh->ah) 812 ipoib_put_ah(neigh->ah); 813 ipoib_neigh_free(dev, neigh); 814 815 tx->neigh = NULL; 816 } 817 818 if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &tx->flags)) { 819 list_move(&tx->list, &priv->cm.reap_list); 820 queue_work(ipoib_workqueue, &priv->cm.reap_task); 821 } 822 823 clear_bit(IPOIB_FLAG_OPER_UP, &tx->flags); 824 825 spin_unlock_irqrestore(&priv->lock, flags); 826 } From sebastien.dugue at bull.net Wed Jun 3 00:01:22 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Wed, 3 Jun 2009 09:01:22 +0200 Subject: [ofa-general] IB card manufacturer In-Reply-To: <20090603010313.GQ18018@cave.ccs.ornl.gov> References: <20090603010313.GQ18018@cave.ccs.ornl.gov> Message-ID: <20090603090122.004e2b3c@frecb007965> Hi Jason, a few things you could try to determine who the manufacturer is: - 'mstflint -d 07:00.0 q' will give you some info, especially the HCA GUIDs. The 24 most significant bits of the board GUIDs gives you the manufacturer. For example: - 0002c9 Mellanox - 0008f1 Voltaire - try to run 'mstflint -d 07:00.0 dc'. The PS_INFO, ADAPTER and IB sections could give some more hints. Hope this helps. Sebastien. On Tue, 2 Jun 2009 21:03:13 -0400 Jason Hill wrote: > All, > > Being somewhat cautious I'd like to see if anyone can point me at the correct > manufacturer info for an Infiniband card that we bought through a cluster vendor > (i.e. it was in the box when we got it). I need to upgrade the firmware to > attempt to isolate an issue we're seeing with several nodes that have this > combination: > > ==> /sys/class/infiniband/mthca0/board_id <== > INT0020000001 > > ==> /sys/class/infiniband/mthca0/fw_ver <== > 1.1.0 > > ==> /sys/class/infiniband/mthca0/hw_rev <== > a0 > > ==> /sys/class/infiniband/mthca0/hca_type <== > MT25204 > > ~$ lspci -vvv > > > 07:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) > Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin A routed to IRQ 50 > Region 0: Memory at feb00000 (64-bit, non-prefetchable) [size=1M] > Region 2: Memory at fd800000 (64-bit, prefetchable) [size=8M] > Capabilities: > > On the card itself it has : > > Part Number: D83914 > Serial Number: MEPT702000093 > And the card has a tall bracket, but the vendor could have swapped that. > > Google has turned up nothing for the Board ID and part number/serial number. > > To me it looks kind of like a Cisco card, so I was thinking it might be a > Topspin. We bought the cluster in 2007. > > Would the right thing be to install the MemFree MHGS18-XT firmware, or try a > revision of the MHGS18-XTC? > > Thanks for the time and thoughts. > From sebastien.dugue at bull.net Wed Jun 3 00:11:38 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Wed, 3 Jun 2009 09:11:38 +0200 Subject: [ofa-general] IB card manufacturer In-Reply-To: <20090603090122.004e2b3c@frecb007965> References: <20090603010313.GQ18018@cave.ccs.ornl.gov> <20090603090122.004e2b3c@frecb007965> Message-ID: <20090603091138.05b14f56@frecb007965> On Wed, 3 Jun 2009 09:01:22 +0200 sebastien dugue wrote: > > Hi Jason, > > a few things you could try to determine who the manufacturer is: > > - 'mstflint -d 07:00.0 q' will give you some info, especially the HCA GUIDs. > The 24 most significant bits of the board GUIDs gives you the manufacturer. > For example: > - 0002c9 Mellanox > - 0008f1 Voltaire The complete list is here: http://standards.ieee.org/regauth/oui/oui.txt > > - try to run 'mstflint -d 07:00.0 dc'. The PS_INFO, ADAPTER and IB sections > could give some more hints. > > > > Hope this helps. > > Sebastien. > > > > On Tue, 2 Jun 2009 21:03:13 -0400 > Jason Hill wrote: > > > All, > > > > Being somewhat cautious I'd like to see if anyone can point me at the correct > > manufacturer info for an Infiniband card that we bought through a cluster vendor > > (i.e. it was in the box when we got it). I need to upgrade the firmware to > > attempt to isolate an issue we're seeing with several nodes that have this > > combination: > > > > ==> /sys/class/infiniband/mthca0/board_id <== > > INT0020000001 > > > > ==> /sys/class/infiniband/mthca0/fw_ver <== > > 1.1.0 > > > > ==> /sys/class/infiniband/mthca0/hw_rev <== > > a0 > > > > ==> /sys/class/infiniband/mthca0/hca_type <== > > MT25204 > > > > ~$ lspci -vvv > > > > > > 07:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) > > Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] > > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- > Latency: 0, Cache Line Size: 64 bytes > > Interrupt: pin A routed to IRQ 50 > > Region 0: Memory at feb00000 (64-bit, non-prefetchable) [size=1M] > > Region 2: Memory at fd800000 (64-bit, prefetchable) [size=8M] > > Capabilities: > > > > On the card itself it has : > > > > Part Number: D83914 > > Serial Number: MEPT702000093 > > And the card has a tall bracket, but the vendor could have swapped that. > > > > Google has turned up nothing for the Board ID and part number/serial number. > > > > To me it looks kind of like a Cisco card, so I was thinking it might be a > > Topspin. We bought the cluster in 2007. > > > > Would the right thing be to install the MemFree MHGS18-XT firmware, or try a > > revision of the MHGS18-XTC? > > > > Thanks for the time and thoughts. > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From dorfman.eli at gmail.com Wed Jun 3 00:28:32 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 10:28:32 +0300 Subject: [ofa-general] [PATCH] infiniband-diags: Do not change logical state on SubnAdmSet In-Reply-To: References: <4A2297DE.3050707@gmail.com> Message-ID: <4A262620.8020305@gmail.com> Hal Rosenstock wrote: > On Sun, May 31, 2009 at 10:44 AM, Eli Dorfman (Voltaire) > wrote: >> Do not change logical state on SubnAdmSet > > The method is SubnSet r.t. SubnAdmSet. Subject might also indicate > ibportstate.c and it would be clearer saying logical port state r.t. > just logical state. correct. my mistake. > > -- Hal > >> When changing physical state do not change logical port state. >> >From the IB spec When writing PortInfo:PortState, only legal transitions are >> valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. >> >> This patch allows reset in a single MAD. >> >> Signed-off-by: Eli Dorfman >> --- >> infiniband-diags/src/ibportstate.c | 5 ++++- >> 1 files changed, 4 insertions(+), 1 deletions(-) >> >> diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c >> index 65c9ca1..d19a2e5 100644 >> --- a/infiniband-diags/src/ibportstate.c >> +++ b/infiniband-diags/src/ibportstate.c >> @@ -275,8 +275,10 @@ int main(int argc, char **argv) >> >> /* Only if one of the "set" options is chosen */ >> if (port_op) { >> - if (port_op == 1) /* Enable port */ >> + if (port_op == 1) { /* Enable port */ >> mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ >> + mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ >> + } >> else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ >> mad_set_field(data, 0, IB_PORT_STATE_F, 1); /* Down */ >> mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3); /* Disabled */ >> @@ -292,6 +294,7 @@ int main(int argc, char **argv) >> >> if (port_op == 3) { /* Reset port - so also enable */ >> mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ >> + mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ >> err = set_port_info(&portid, data, portnum, port_op); >> if (err < 0) >> IBERROR("smp set portinfo failed"); >> -- >> 1.5.5 >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> From dzieko at wcss.pl Wed Jun 3 00:39:49 2009 From: dzieko at wcss.pl (Pawel Dziekonski) Date: Wed, 3 Jun 2009 09:39:49 +0200 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A257DBF.4000401@mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> Message-ID: <20090603073949.GC3261@cefeid.wcss.wroc.pl> On Tue, 02 Jun 2009 at 10:30:07PM +0300, Tziporet Koren wrote: > Main reasons to keep MPI in OFED: > - All participants test with the same MPI versions, and when installing > OFED it is ensured that MPI will work fine with this version. > - Customers convenience in install (no need to go to more sites to get MPI) > - MPI is an important RDMA ULP and although it is not developed in OFA it > is widely used by OFED customers As a customer I strongly support above mentioned pros. It's a guarantee for us that MPI is well tested with OFED release. I believe that this effort saves a lot of troubles that would be raised from separate releases of MPI and OFED distros. thanks, Pawel -- Pawel Dziekonski Wroclaw Centre for Networking & Supercomputing, HPC Department Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl From vlad at lists.openfabrics.org Wed Jun 3 03:14:59 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 3 Jun 2009 03:14:59 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090603-0200 daily build status Message-ID: <20090603101459.35700E305F2@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18-8.el5 Failed: Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.26 Build failed on x86_64 with linux-2.6.18 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.18/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.17/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -fomit-frame-pointer -g -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.19/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.21.1/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.20/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.22/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.25/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -m64 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: from /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.c:38: include/linux/workqueue.h:92:1: warning: this is the location of the previous definition /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.24/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: -I/home/vlad/kernel.org/x86_64/linux-2.6.26/arch/i386/include \ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -m64 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Iinclude/asm-x86/mach-default -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: -I/home/vlad/kernel.org/ia64/linux-2.6.25/arch/i386/include \ -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/.tmp_en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: -I/home/vlad/kernel.org/ia64/linux-2.6.26/arch/i386/include \ -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/.tmp_en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: -I/home/vlad/kernel.org/ia64/linux-2.6.24/arch/i386/include \ -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/.tmp_en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.c /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From alexander.supalov at intel.com Wed Jun 3 03:26:00 2009 From: alexander.supalov at intel.com (Supalov, Alexander) Date: Wed, 3 Jun 2009 11:26:00 +0100 Subject: [ofa-general] Memory registration redux In-Reply-To: References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> <5019F239-149F-49E1-8C23-436DE6094AB2@cisco.com> <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> Message-ID: <928CFBE8E7CB0040959E56B4EA41A77EBBBC67B1@irsmsx504.ger.corp.intel.com> Thanks. This is what I was looking for. Let me pass this by the key Intel MPI developers and get back to you. -----Original Message----- From: Roland Dreier [mailto:rdreier at cisco.com] Sent: Tuesday, June 02, 2009 6:45 PM To: Supalov, Alexander Cc: Jeff Squyres; Pavel Shamis; Hans Westgaard Ry; Dontje; Lenny Verkhovsky; H??kon Bugge; Donald Kerr; OpenFabrics General Subject: Re: [ofa-general] Memory registration redux > Sorry, it's kind of difficult to deduce looking at this Q&A sequence > what works how and when. Is it possible to create a brief and direct > description of the proposed solution? Did you see the original patch description I sent: As discussed in and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help. We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunot, that creates a /dev/ummunot node. A userspace process can open this node read-only and use the fd as follows: 1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunot_register_ioctl in ). 2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunot_event in ). select()/poll()/epoll() and SIGIO are handled for this IO. 3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call. --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From Line.Holen at Sun.COM Wed Jun 3 03:57:40 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Wed, 03 Jun 2009 12:57:40 +0200 Subject: [ofa-general] [PATCH] osm_ucast_ftree.c Allow horizontal links between switches of max rank In-Reply-To: <4A2516D4.1020502@morey-chaisemartin.com> References: <4A1D10ED.6040001@Sun.COM> <4A2516D4.1020502@morey-chaisemartin.com> Message-ID: <4A265724.7070308@Sun.COM> On 06/ 2/09 02:11 PM, Nicolas Morey-Chaisemartin wrote: > Hi, > > I've just run a few tests with your patch using io_guid_files and reverse_hops and it seems your patch introduces a bug. > We have a function to calculate the number per link and if I apply this patch it never returns which usually means there is a loop in the routing scheme. > > I have had a quick look at your code. > One thing I'm not sure of is that when a reverse hop is done you increment both the reverse_hop counter and the current_hops one. Which mean when the hops count is calculate for sw_set_hops it is bigger than it is supposed to be. > > > What should probably be done is increment both but only use current_hops to calculate the number of hops done. reverse_hop will only be used to check if there are still reverse_hop credits left or not (which was its original purpose). > I think you are right. current_hops is updated also for reverse hops and should be sufficient. > By changing the hops count in both call to set_hops_on_remote_sw to current_hops+1 and not current_hops+1 +2* reverse_hops, routing has no loops but results are different (some links are more loaded). > I'll run some more tests to see if I can figure this out. > OK. I'll run some tests myself. Please let me know if you find anything suspicious. Line > > Nicolas > > Le 27/05/2009 12:07, Line.Holen at Sun.COM a écrit : >> This patch makes it legal to have cross links (horizontal links) between >> switches at max rank. These switches do have same rank, so hop count cannot >> be calculated based on rank anymore. >> The horizontal links are treated as downlinks. Switch A has a downlink to B >> while B has a downlink to A. Tests on lids and also number of hops makes sure >> that we don't loop back and forth across the link. >> >> Signed-off-by: Frank Olaf Sem-Jacobsen >> Signed-off-by: Line Holen >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From dorfman.eli at gmail.com Wed Jun 3 04:01:12 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 14:01:12 +0300 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> Message-ID: <4A2657F8.3010807@gmail.com> Hal Rosenstock wrote: > On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: >> On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >>>> Yes, RMPP is an overhead when the response is a single MAD but is this >>>> significant ? Anyhow, how can the spec be changed in a way that >>>> doesn't break existing implementations ? >>> But the implementations are assuming different things about SubnAdmGet. The SA >>> is assuming that the query should fail if multiple records match. The client >>> side software (ipoib and rdma_cm) assume that it will obtain a single record >>> even if multiple paths are present. So, something needs to change. >> Seems so. >> >>> The spec indicates that value in the request is ignored and NumbPath is 1, not >>> that NumbPath is completely ignored. >> For Get, it doesn't say that the matches are paired down to this >> number as it does for GetTable. >> >>> Also see page 1242 in the SDP annex which >>> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet >>> rather than SubnAdmGetTable)'. >> SDP annex is not the primary source for this (chapter 15 is) and is >> inconsistent and no one caught this. >> >>> To me, this implies that SubnAdmGet should be >>> treated equivalent as SubnAdmGetTable with NumbPath = 1. >>> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus >>> PR SubnAdmGetTable and MPR SubAdmGetMulti. Basically, this makes PR SubnAdmGet >>> useless. >> when there's a subnet with multiple paths and the requests are not >> specific enough to use get. >> >> Seems like either the queries need to use RMPP, or the spec modified >> (if that's possible) and the SAs updated. > > I sit corrected :-) Your interpretation of the spec is correct. Also, > in looking at OpenSM, the intent is as you indicate: it does try to > only return 1 attibute for get PR. If when returning the response, > there is more than 1 attribute in the list, it returns the too many > records error. There must be some code path I don't see right now > which is doing this. It would be useful to know the details of the > query (get request) causing this. > This may happen when pr_rcv_get_port_pair_paths() is called several times. The only case i see is pr_rcv_process_world() that means the request is without or wrong src and dest port or component mask for SGID and DGID is 0. Eli From dorfman.eli at gmail.com Wed Jun 3 04:03:50 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 14:03:50 +0300 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <4A2657F8.3010807@gmail.com> References: <20090528175757.GA95655@nas.nasa.gov> <4A2414CE.7060300@voltaire.com> <8DB07B53FF294251ADBEC60C635CBE37@amr.corp.intel.com> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> <4A2657F8.3010807@gmail.com> Message-ID: <4A265896.2080801@gmail.com> Eli Dorfman (Voltaire) wrote: > Hal Rosenstock wrote: >> On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: >>> On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >>>>> Yes, RMPP is an overhead when the response is a single MAD but is this >>>>> significant ? Anyhow, how can the spec be changed in a way that >>>>> doesn't break existing implementations ? >>>> But the implementations are assuming different things about SubnAdmGet. The SA >>>> is assuming that the query should fail if multiple records match. The client >>>> side software (ipoib and rdma_cm) assume that it will obtain a single record >>>> even if multiple paths are present. So, something needs to change. >>> Seems so. >>> >>>> The spec indicates that value in the request is ignored and NumbPath is 1, not >>>> that NumbPath is completely ignored. >>> For Get, it doesn't say that the matches are paired down to this >>> number as it does for GetTable. >>> >>>> Also see page 1242 in the SDP annex which >>>> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet >>>> rather than SubnAdmGetTable)'. >>> SDP annex is not the primary source for this (chapter 15 is) and is >>> inconsistent and no one caught this. >>> >>>> To me, this implies that SubnAdmGet should be >>>> treated equivalent as SubnAdmGetTable with NumbPath = 1. >>>> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus >>>> PR SubnAdmGetTable and MPR SubAdmGetMulti. Basically, this makes PR SubnAdmGet >>>> useless. >>> when there's a subnet with multiple paths and the requests are not >>> specific enough to use get. >>> >>> Seems like either the queries need to use RMPP, or the spec modified >>> (if that's possible) and the SAs updated. >> I sit corrected :-) Your interpretation of the spec is correct. Also, >> in looking at OpenSM, the intent is as you indicate: it does try to >> only return 1 attibute for get PR. If when returning the response, >> there is more than 1 attribute in the list, it returns the too many >> records error. There must be some code path I don't see right now >> which is doing this. It would be useful to know the details of the >> query (get request) causing this. >> > > This may happen when pr_rcv_get_port_pair_paths() is called several times. > The only case i see is pr_rcv_process_world() that means the request is without or wrong > src and dest port or component mask for SGID and DGID is 0. correction - this may happen only when component mask for SGID and DGID is 0. Eli From hal.rosenstock at gmail.com Wed Jun 3 04:17:31 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 07:17:31 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <4A2657F8.3010807@gmail.com> References: <20090528175757.GA95655@nas.nasa.gov> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> <4A2657F8.3010807@gmail.com> Message-ID: On Wed, Jun 3, 2009 at 7:01 AM, Eli Dorfman (Voltaire) wrote: > Hal Rosenstock wrote: >> On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: >>> On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >>>>> Yes, RMPP is an overhead when the response is a single MAD but is this >>>>> significant ? Anyhow, how can the spec be changed in a way that >>>>> doesn't break existing implementations ? >>>> But the implementations are assuming different things about SubnAdmGet.  The SA >>>> is assuming that the query should fail if multiple records match.  The client >>>> side software (ipoib and rdma_cm) assume that it will obtain a single record >>>> even if multiple paths are present.  So, something needs to change. >>> Seems so. >>> >>>> The spec indicates that value in the request is ignored and NumbPath is 1, not >>>> that NumbPath is completely ignored. >>> For Get, it doesn't say that the matches are paired down to this >>> number as it does for GetTable. >>> >>>>  Also see page 1242 in the SDP annex which >>>> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet >>>> rather than SubnAdmGetTable)'. >>> SDP annex is not the primary source for this (chapter 15 is) and is >>> inconsistent and no one caught this. >>> >>>> To me, this implies that SubnAdmGet should be >>>> treated equivalent as SubnAdmGetTable with NumbPath = 1. >>>> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus >>>> PR SubnAdmGetTable and MPR SubAdmGetMulti.  Basically, this makes PR SubnAdmGet >>>> useless. >>> when there's a subnet with multiple paths and the requests are not >>> specific enough to use get. >>> >>> Seems like either the queries need to use RMPP, or the spec modified >>> (if that's possible) and the SAs updated. >> >> I sit corrected :-) Your interpretation of the spec is correct. Also, >> in looking at OpenSM, the intent is as you indicate: it does try to >> only return 1 attibute for get PR. If when returning the response, >> there is more than 1 attribute in the list, it returns the too many >> records error. There must be some code path I don't see right now >> which is doing this. It would be useful to know the details of the >> query (get request) causing this. >> > > This may happen when pr_rcv_get_port_pair_paths() is called several times. > The only case i see is pr_rcv_process_world() that means the request is without or wrong > src and dest port or component mask for SGID and DGID is 0. Should the non standard process world only return 1 attribute for get r.t. too many records status ? -- Hal > > Eli > From hal.rosenstock at gmail.com Wed Jun 3 04:28:47 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 07:28:47 -0400 Subject: [ofa-general] [PATCH] infiniband-diags: Do not change logical state on SubnAdmSet In-Reply-To: <4A262620.8020305@gmail.com> References: <4A2297DE.3050707@gmail.com> <4A262620.8020305@gmail.com> Message-ID: On Wed, Jun 3, 2009 at 3:28 AM, Eli Dorfman (Voltaire) wrote: > Hal Rosenstock wrote: >> On Sun, May 31, 2009 at 10:44 AM, Eli Dorfman (Voltaire) >> wrote: >>> Do not change logical state on SubnAdmSet >> >> The method is SubnSet r.t. SubnAdmSet. Subject might also indicate >> ibportstate.c and it would be clearer saying logical port state r.t. >> just logical state. > > correct. my mistake. Should an updated patch for this be reissued ? -- Hal > >> >> -- Hal >> >>> When changing physical state do not change logical port state. >>> >From the IB spec When writing PortInfo:PortState, only legal transitions are >>> valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. >>> >>> This patch allows reset in a single MAD. >>> >>> Signed-off-by: Eli Dorfman >>> --- >>>  infiniband-diags/src/ibportstate.c |    5 ++++- >>>  1 files changed, 4 insertions(+), 1 deletions(-) >>> >>> diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c >>> index 65c9ca1..d19a2e5 100644 >>> --- a/infiniband-diags/src/ibportstate.c >>> +++ b/infiniband-diags/src/ibportstate.c >>> @@ -275,8 +275,10 @@ int main(int argc, char **argv) >>> >>>        /* Only if one of the "set" options is chosen */ >>>        if (port_op) { >>> -               if (port_op == 1)               /* Enable port */ >>> +               if (port_op == 1) {             /* Enable port */ >>>                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2);        /* Polling */ >>> +                       mad_set_field(data, 0, IB_PORT_STATE_F, 0);             /* No Change */ >>> +               } >>>                else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ >>>                        mad_set_field(data, 0, IB_PORT_STATE_F, 1);             /* Down */ >>>                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3);        /* Disabled */ >>> @@ -292,6 +294,7 @@ int main(int argc, char **argv) >>> >>>                if (port_op == 3) {     /* Reset port - so also enable */ >>>                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2);        /* Polling */ >>> +                       mad_set_field(data, 0, IB_PORT_STATE_F, 0);             /* No Change */ >>>                        err = set_port_info(&portid, data, portnum, port_op); >>>                        if (err < 0) >>>                                IBERROR("smp set portinfo failed"); >>> -- >>> 1.5.5 >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>> > > From dorfman.eli at gmail.com Wed Jun 3 04:54:09 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 14:54:09 +0300 Subject: [ofa-general] [PATCH] Return single PR record for SubnAdmGet Message-ID: <4A266461.1000908@gmail.com> Return single PR record for SubnAdmGet Fix to return single PathRecord for SubnAdmGet when SGID and/or DGID are 0 in SA component mask (wildcard) Also avoid iterating beyond requested number of paths. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_sa_path_record.c | 29 ++++++++++++++++++++--------- 1 files changed, 20 insertions(+), 9 deletions(-) diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 6e7d5f6..531115c 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -83,6 +83,8 @@ typedef struct osm_path_parms { boolean_t reversible; } osm_path_parms_t; +#define OSM_SA_PR_MAX_NUM_PATH 127 + /********************************************************************** **********************************************************************/ static inline boolean_t sa_path_rec_is_tavor_port(IN const osm_port_t * p_port) @@ -891,7 +893,7 @@ Exit: /********************************************************************** **********************************************************************/ -static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, +static int pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, IN const osm_port_t * p_req_port, IN const osm_port_t * p_src_port, @@ -1020,14 +1022,14 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, Preferred paths come first in OpenSM */ preference = 0; - path_num = 0; + path_num = cl_qlist_count(p_list); /* If SubnAdmGet, assume NumbPaths 1 (1.2 erratum) */ if (p_sa_mad->method != IB_MAD_METHOD_GET) if (comp_mask & IB_PR_COMPMASK_NUMBPATH) iterations = ib_path_rec_num_path(p_pr); else - iterations = (uintn_t) (-1); + iterations = OSM_SA_PR_MAX_NUM_PATH; else iterations = 1; @@ -1042,6 +1044,8 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, comp_mask, preference); if (p_pr_item) { + if (p_sa_mad->method == IB_MAD_METHOD_GET && cl_qlist_count(p_list) == 1) + return -1; cl_qlist_insert_tail(p_list, &p_pr_item->list_item); ++path_num; } @@ -1105,6 +1109,8 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, comp_mask, preference); if (p_pr_item) { + if (p_sa_mad->method == IB_MAD_METHOD_GET && cl_qlist_count(p_list) == 1) + return -1; cl_qlist_insert_tail(p_list, &p_pr_item->list_item); ++path_num; } @@ -1112,6 +1118,7 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, Exit: OSM_LOG_EXIT(sa->p_log); + return 0; } /********************************************************************** @@ -1350,9 +1357,10 @@ static void pr_rcv_process_world(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, while (p_dest_port != (osm_port_t *) cl_qmap_end(p_tbl)) { p_src_port = (osm_port_t *) cl_qmap_head(p_tbl); while (p_src_port != (osm_port_t *) cl_qmap_end(p_tbl)) { - pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, + if (pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, p_src_port, p_dest_port, - p_dgid, comp_mask, p_list); + p_dgid, comp_mask, p_list) < 0) + return; p_src_port = (osm_port_t *) cl_qmap_next(&p_src_port->map_item); @@ -1393,9 +1401,10 @@ static void pr_rcv_process_half(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, */ p_port = (osm_port_t *) cl_qmap_head(p_tbl); while (p_port != (osm_port_t *) cl_qmap_end(p_tbl)) { - pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, + if (pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, p_src_port, p_port, p_dgid, - comp_mask, p_list); + comp_mask, p_list) < 0) + goto Exit; p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item); } } else { @@ -1404,13 +1413,15 @@ static void pr_rcv_process_half(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, */ p_port = (osm_port_t *) cl_qmap_head(p_tbl); while (p_port != (osm_port_t *) cl_qmap_end(p_tbl)) { - pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, + if (pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, p_port, p_dest_port, p_dgid, - comp_mask, p_list); + comp_mask, p_list) < 0) + goto Exit; p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item); } } +Exit: OSM_LOG_EXIT(sa->p_log); } -- 1.5.5 From dorfman.eli at gmail.com Wed Jun 3 04:55:56 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 14:55:56 +0300 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: References: <20090528175757.GA95655@nas.nasa.gov> <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> <4A2657F8.3010807@gmail.com> Message-ID: <4A2664CC.6020906@gmail.com> Hal Rosenstock wrote: > On Wed, Jun 3, 2009 at 7:01 AM, Eli Dorfman (Voltaire) > wrote: >> Hal Rosenstock wrote: >>> On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: >>>> On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >>>>>> Yes, RMPP is an overhead when the response is a single MAD but is this >>>>>> significant ? Anyhow, how can the spec be changed in a way that >>>>>> doesn't break existing implementations ? >>>>> But the implementations are assuming different things about SubnAdmGet. The SA >>>>> is assuming that the query should fail if multiple records match. The client >>>>> side software (ipoib and rdma_cm) assume that it will obtain a single record >>>>> even if multiple paths are present. So, something needs to change. >>>> Seems so. >>>> >>>>> The spec indicates that value in the request is ignored and NumbPath is 1, not >>>>> that NumbPath is completely ignored. >>>> For Get, it doesn't say that the matches are paired down to this >>>> number as it does for GetTable. >>>> >>>>> Also see page 1242 in the SDP annex which >>>>> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet >>>>> rather than SubnAdmGetTable)'. >>>> SDP annex is not the primary source for this (chapter 15 is) and is >>>> inconsistent and no one caught this. >>>> >>>>> To me, this implies that SubnAdmGet should be >>>>> treated equivalent as SubnAdmGetTable with NumbPath = 1. >>>>> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus >>>>> PR SubnAdmGetTable and MPR SubAdmGetMulti. Basically, this makes PR SubnAdmGet >>>>> useless. >>>> when there's a subnet with multiple paths and the requests are not >>>> specific enough to use get. >>>> >>>> Seems like either the queries need to use RMPP, or the spec modified >>>> (if that's possible) and the SAs updated. >>> I sit corrected :-) Your interpretation of the spec is correct. Also, >>> in looking at OpenSM, the intent is as you indicate: it does try to >>> only return 1 attibute for get PR. If when returning the response, >>> there is more than 1 attribute in the list, it returns the too many >>> records error. There must be some code path I don't see right now >>> which is doing this. It would be useful to know the details of the >>> query (get request) causing this. >>> >> This may happen when pr_rcv_get_port_pair_paths() is called several times. >> The only case i see is pr_rcv_process_world() that means the request is without or wrong >> src and dest port or component mask for SGID and DGID is 0. > > Should the non standard process world only return 1 attribute for get > r.t. too many records status ? SubnAdmGet should return only 1 record. This should be fixed for the case of SGID and/or DGID wildcarded. I submitted a patch with this fix to the list. Eli From dorfman.eli at gmail.com Wed Jun 3 04:57:57 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 14:57:57 +0300 Subject: [ofa-general] [PATCH] infiniband-diags: Do not change logical state on SubnAdmSet In-Reply-To: References: <4A2297DE.3050707@gmail.com> <4A262620.8020305@gmail.com> Message-ID: <4A266545.6080807@gmail.com> Hal Rosenstock wrote: > On Wed, Jun 3, 2009 at 3:28 AM, Eli Dorfman (Voltaire) > wrote: >> Hal Rosenstock wrote: >>> On Sun, May 31, 2009 at 10:44 AM, Eli Dorfman (Voltaire) >>> wrote: >>>> Do not change logical state on SubnAdmSet >>> The method is SubnSet r.t. SubnAdmSet. Subject might also indicate >>> ibportstate.c and it would be clearer saying logical port state r.t. >>> just logical state. >> correct. my mistake. > > Should an updated patch for this be reissued ? No, the patch is correct. The error was in the commit message. Eli > > -- Hal > >>> -- Hal >>> >>>> When changing physical state do not change logical port state. >>>> >From the IB spec When writing PortInfo:PortState, only legal transitions are >>>> valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. >>>> >>>> This patch allows reset in a single MAD. >>>> >>>> Signed-off-by: Eli Dorfman >>>> --- >>>> infiniband-diags/src/ibportstate.c | 5 ++++- >>>> 1 files changed, 4 insertions(+), 1 deletions(-) >>>> >>>> diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c >>>> index 65c9ca1..d19a2e5 100644 >>>> --- a/infiniband-diags/src/ibportstate.c >>>> +++ b/infiniband-diags/src/ibportstate.c >>>> @@ -275,8 +275,10 @@ int main(int argc, char **argv) >>>> >>>> /* Only if one of the "set" options is chosen */ >>>> if (port_op) { >>>> - if (port_op == 1) /* Enable port */ >>>> + if (port_op == 1) { /* Enable port */ >>>> mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ >>>> + mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ >>>> + } >>>> else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ >>>> mad_set_field(data, 0, IB_PORT_STATE_F, 1); /* Down */ >>>> mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3); /* Disabled */ >>>> @@ -292,6 +294,7 @@ int main(int argc, char **argv) >>>> >>>> if (port_op == 3) { /* Reset port - so also enable */ >>>> mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ >>>> + mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ >>>> err = set_port_info(&portid, data, portnum, port_op); >>>> if (err < 0) >>>> IBERROR("smp set portinfo failed"); >>>> -- >>>> 1.5.5 >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>>> >> From hal.rosenstock at gmail.com Wed Jun 3 04:59:40 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 07:59:40 -0400 Subject: [ofa-general] [PATCH] infiniband-diags: Do not change logical state on SubnAdmSet In-Reply-To: <4A266545.6080807@gmail.com> References: <4A2297DE.3050707@gmail.com> <4A262620.8020305@gmail.com> <4A266545.6080807@gmail.com> Message-ID: On Wed, Jun 3, 2009 at 7:57 AM, Eli Dorfman (Voltaire) wrote: > Hal Rosenstock wrote: >> On Wed, Jun 3, 2009 at 3:28 AM, Eli Dorfman (Voltaire) >> wrote: >>> Hal Rosenstock wrote: >>>> On Sun, May 31, 2009 at 10:44 AM, Eli Dorfman (Voltaire) >>>> wrote: >>>>> Do not change logical state on SubnAdmSet >>>> The method is SubnSet r.t. SubnAdmSet. Subject might also indicate >>>> ibportstate.c and it would be clearer saying logical port state r.t. >>>> just logical state. >>> correct. my mistake. >> >> Should an updated patch for this be reissued ? > > No, the patch is correct. The error was in the commit message. IMO it would be better to have the commit message accurate. -- Hal > Eli > >> >> -- Hal >> >>>> -- Hal >>>> >>>>> When changing physical state do not change logical port state. >>>>> >From the IB spec When writing PortInfo:PortState, only legal transitions are >>>>> valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. >>>>> >>>>> This patch allows reset in a single MAD. >>>>> >>>>> Signed-off-by: Eli Dorfman >>>>> --- >>>>>  infiniband-diags/src/ibportstate.c |    5 ++++- >>>>>  1 files changed, 4 insertions(+), 1 deletions(-) >>>>> >>>>> diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c >>>>> index 65c9ca1..d19a2e5 100644 >>>>> --- a/infiniband-diags/src/ibportstate.c >>>>> +++ b/infiniband-diags/src/ibportstate.c >>>>> @@ -275,8 +275,10 @@ int main(int argc, char **argv) >>>>> >>>>>        /* Only if one of the "set" options is chosen */ >>>>>        if (port_op) { >>>>> -               if (port_op == 1)               /* Enable port */ >>>>> +               if (port_op == 1) {             /* Enable port */ >>>>>                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2);        /* Polling */ >>>>> +                       mad_set_field(data, 0, IB_PORT_STATE_F, 0);             /* No Change */ >>>>> +               } >>>>>                else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ >>>>>                        mad_set_field(data, 0, IB_PORT_STATE_F, 1);             /* Down */ >>>>>                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3);        /* Disabled */ >>>>> @@ -292,6 +294,7 @@ int main(int argc, char **argv) >>>>> >>>>>                if (port_op == 3) {     /* Reset port - so also enable */ >>>>>                        mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2);        /* Polling */ >>>>> +                       mad_set_field(data, 0, IB_PORT_STATE_F, 0);             /* No Change */ >>>>>                        err = set_port_info(&portid, data, portnum, port_op); >>>>>                        if (err < 0) >>>>>                                IBERROR("smp set portinfo failed"); >>>>> -- >>>>> 1.5.5 >>>>> >>>>> _______________________________________________ >>>>> general mailing list >>>>> general at lists.openfabrics.org >>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>>> >>>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>>>> >>> > > From ogerlitz at Voltaire.com Wed Jun 3 05:21:15 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Wed, 03 Jun 2009 15:21:15 +0300 Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: References: Message-ID: <4A266ABB.6000603@Voltaire.com> Roland Dreier wrote: > The IPoIB drops multicast packets if more than 3 are queued while a > multicast send is pending. The send queue can easily contain more > than 3 packets and when the queue is processed only the first 3 > packets will make it and the rest will be dropped. If we go this way, I went and did the same thing for unicast, patch below. Alternatively we may have both mcast/unicast queues to remain and get their length set by the net.ipvY.neigh.ibX.unres_qlen? I tested my patch with TCP/UDP netperf/iperf over 2.6.29.1 and things seem to work fine. Or. This patch applies the same reasoning and method of "IPoIB: Don't drop multicast packets sent before group is joined" to unicast packets. Signed-off-by: Or Gerlitz Index: linus-linux-2.6/drivers/infiniband/ulp/ipoib/ipoib.h =================================================================== --- linus-linux-2.6.orig/drivers/infiniband/ulp/ipoib/ipoib.h +++ linus-linux-2.6/drivers/infiniband/ulp/ipoib/ipoib.h @@ -78,8 +78,6 @@ enum { IPOIB_NUM_WC = 4, - IPOIB_MAX_PATH_REC_QUEUE = 3, - IPOIB_FLAG_OPER_UP = 0, IPOIB_FLAG_INITIALIZED = 1, IPOIB_FLAG_ADMIN_UP = 2, Index: linus-linux-2.6/drivers/infiniband/ulp/ipoib/ipoib_main.c =================================================================== --- linus-linux-2.6.orig/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ linus-linux-2.6/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -597,13 +597,7 @@ static void neigh_add_path(struct sk_buf ipoib_neigh_free(dev, neigh); goto err_drop; } - if (skb_queue_len(&neigh->queue) < IPOIB_MAX_PATH_REC_QUEUE) - __skb_queue_tail(&neigh->queue, skb); - else { - ipoib_warn(priv, "queue length limit %d. Packet drop.\n", - skb_queue_len(&neigh->queue)); - goto err_drop; - } + __skb_queue_tail(&neigh->queue, skb); } else ipoib_send(dev, skb, path->ah, IPOIB_QPN(skb->dst->neighbour->ha)); } else { @@ -689,8 +683,7 @@ static void unicast_arp_send(struct sk_b be16_to_cpu(path->pathrec.dlid)); ipoib_send(dev, skb, path->ah, IPOIB_QPN(phdr->hwaddr)); - } else if ((path->query || !path_rec_start(dev, path)) && - skb_queue_len(&path->queue) < IPOIB_MAX_PATH_REC_QUEUE) { + } else if (path->query || !path_rec_start(dev, path)) { /* put pseudoheader back on for next time */ skb_push(skb, sizeof *phdr); __skb_queue_tail(&path->queue, skb); @@ -747,14 +740,9 @@ static int ipoib_start_xmit(struct sk_bu return NETDEV_TX_OK; } - if (skb_queue_len(&neigh->queue) < IPOIB_MAX_PATH_REC_QUEUE) { - spin_lock_irqsave(&priv->lock, flags); - __skb_queue_tail(&neigh->queue, skb); - spin_unlock_irqrestore(&priv->lock, flags); - } else { - ++dev->stats.tx_dropped; - dev_kfree_skb_any(skb); - } + spin_lock_irqsave(&priv->lock, flags); + __skb_queue_tail(&neigh->queue, skb); + spin_unlock_irqrestore(&priv->lock, flags); } else { struct ipoib_pseudoheader *phdr = (struct ipoib_pseudoheader *) skb->data; From jsquyres at cisco.com Wed Jun 3 05:24:23 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Wed, 3 Jun 2009 08:24:23 -0400 Subject: [ofa-general] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A257DBF.4000401@mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> Message-ID: <74D67503-94FD-4D02-B986-65ADC899740E@cisco.com> I'll state right up front that I'm in favor of removing MPI from OFED. :-) On Jun 2, 2009, at 3:30 PM, Tziporet Koren wrote: > Here are some of the reasons for keeping MPI in or out of OFED. > > Main reasons to take MPI out of OFED: > - MPI is not developed under OFA > - Need to synchronize between different projects > --> This point alone is hugely difficult. OFED has been held up multiple times because of MPI release delays. I would like to stress that the open source MPI implementations are entirely different software projects with minimal overlap of personnel between MPI and OFED. Also, OFED is but one of the stacks that the MPI's support. For example, many Open MPI members don't care about OpenFabrics at all. It has sometimes been difficult to rationalize MPI schedule adjustments because of OFED. > - Some customers prefer to install a different MPI version then the > one in OFED > - Also note that OFED only includes the open source MPI's; it's not a level playing field. - Several of Open MPI's features are disabled in OFED builds (e.g., scheduler / resource manager support), causing customers to re- download / re-build Open MPI anyway. *** This is a fairly important point to us. > Main reasons to keep MPI in OFED: > - All participants test with the same MPI versions, and when > installing OFED it is ensured that MPI will work fine with this > version. > --> I think we're all convinced that testing OFED with the various MPI implementations is a Good Thing -- nobody is suggesting that we remove that. As discussed on the call, we can certainly all test the same version of the open source MPI's during OFED releases and include this information in the release notes ("OFED x.y.z was tested with Open MPI a.b.c."). The Open MPI project would be happy to contribute the MPI Testing Tool (MTT; which itself is also open source) to OFED. The MTT was designed for almost exactly this purpose: a large, distributed set of organizations that all need to test the same versions of MPI and report the results to a central location. The MTT is not specific to Open MPI; it can be used with any MPI implementation. The MTT is essentially an engine to download/install MPI implementations, run a set of MPI tests (e.g., IMB, OSU and others), and then report the results to a central database. FWIW: several OFA members are using MTT internally for their own MPI testing. > - Customers convenience in install (no need to go to more sites to > get MPI) > --> My $0.02 is that this is a dubious point. In installing a production HPC cluster, you're getting 20 pieces of software and installing them together. So if you have to get OFED *and* MPI (i.e., 21 pieces of software), I think it makes little difference. Indeed, as noted above, MPI implementations move at a different speed than OFED, so many customers go install their own MPI's anyway (or, per an above point, need to re-install MPI's to enable features that the OFED packaging disables). I think the biggest issue related to this point is going to be customers asking "you *used* to bundle MPI, why don't do anymore?" > - MPI is an important RDMA ULP and although it is not developed in > OFA it is widely used by OFED customers > --> This point is actually used as a crutch by the OFA: why develop your own comprehensive performance testing / monitoring tools when you have MPI with all of its tools? Removing MPI may help motivate the OFA to have better networking tools than those provided by a ULP. (before you flame, I admit that this is a minor point, but it is still worth noting -- I've always thought it strange that people use MPI to monitor and test their OF-based networks...) -- Jeff Squyres Cisco Systems From jon at opengridcomputing.com Wed Jun 3 05:39:35 2009 From: jon at opengridcomputing.com (Jon Mason) Date: Wed, 3 Jun 2009 07:39:35 -0500 Subject: [ofa-general] ofa_1_4_kernel 20090603-0200 daily build status In-Reply-To: <20090603101459.35700E305F2@openfabrics.org> References: <20090603101459.35700E305F2@openfabrics.org> Message-ID: <20090603123934.GA2574@opengridcomputing.com> Is this still the 1.4 nightly build or is this now the 1.5 nightly? These errors look very similar to the ones I am seeing in 1.5, but the e-mail title says 1.4... Thanks, Jon On Wed, Jun 03, 2009 at 03:14:59AM -0700, Vladimir Sokolovsky (Mellanox) wrote: > This email was generated automatically, please do not reply > > > git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git > git_branch: ofed_kernel > > Common build parameters: > > Passed: > Passed on i686 with linux-2.6.16 > Passed on i686 with linux-2.6.27 > Passed on x86_64 with linux-2.6.16 > Passed on x86_64 with linux-2.6.16.21-0.8-smp > Passed on x86_64 with linux-2.6.16.43-0.3-smp > Passed on x86_64 with linux-2.6.16.60-0.21-smp > Passed on x86_64 with linux-2.6.18-1.2798.fc6 > Passed on x86_64 with linux-2.6.18-53.el5 > Passed on x86_64 with linux-2.6.18-128.el5 > Passed on x86_64 with linux-2.6.18-8.el5 > Passed on x86_64 with linux-2.6.18-93.el5 > Passed on x86_64 with linux-2.6.9-42.ELsmp > Passed on x86_64 with linux-2.6.27 > Passed on x86_64 with linux-2.6.9-78.ELsmp > Passed on x86_64 with linux-2.6.9-67.ELsmp > Passed on x86_64 with linux-2.6.9-55.ELsmp > Passed on ia64 with linux-2.6.16 > Passed on ia64 with linux-2.6.16.21-0.8-default > Passed on ppc64 with linux-2.6.16 > Passed on ppc64 with linux-2.6.18-8.el5 > > Failed: > Build failed on i686 with linux-2.6.18 > Build failed on i686 with linux-2.6.19 > Build failed on i686 with linux-2.6.17 > Build failed on i686 with linux-2.6.21.1 > Build failed on i686 with linux-2.6.22 > Build failed on i686 with linux-2.6.24 > Build failed on i686 with linux-2.6.26 > Build failed on x86_64 with linux-2.6.18 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.18/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.17 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.17/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -fomit-frame-pointer -g -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.19 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.19/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.21.1 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.21.1/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.20 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.20/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.20_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.22 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.22/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_rx)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.25 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.25/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -m64 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4/en_cq.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.22.5-31-default > Log: > from /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.c:38: > include/linux/workqueue.h:92:1: warning: this is the location of the previous definition > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.24 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.24/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4/en_cq.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.26 > Log: > -I/home/vlad/kernel.org/x86_64/linux-2.6.26/arch/i386/include \ > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -m64 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Iinclude/asm-x86/mach-default -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4/en_cq.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.18 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.17 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.19 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.21.1 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.21.1_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.23 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.23_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.22 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.22_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.25 > Log: > -I/home/vlad/kernel.org/ia64/linux-2.6.25/arch/i386/include \ > -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/.tmp_en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4/en_cq.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.25_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.26 > Log: > -I/home/vlad/kernel.org/ia64/linux-2.6.26/arch/i386/include \ > -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -fno-stack-protector -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/.tmp_en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4/en_cq.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.26_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.24 > Log: > -I/home/vlad/kernel.org/ia64/linux-2.6.24/arch/i386/include \ > -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(en_cq)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_en)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/.tmp_en_cq.o /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.c > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.c: In function 'mlx4_en_deactivate_cq': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.c:141: error: implicit declaration of function 'netif_napi_del' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4/en_cq.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.24_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.18 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.18_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.17 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.17_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.19 > Log: > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c:251: warning: format '%llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c: In function 'mlx4_en_fill_rx_buffers': > /home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.c:290: error: implicit declaration of function 'rounddown_pow_of_two' > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4/en_rx.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check/drivers/net/mlx4] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090603-0200_linux-2.6.19_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From acceptany at gmail.com Wed Jun 3 05:49:54 2009 From: acceptany at gmail.com (Jordan) Date: Wed, 3 Jun 2009 20:49:54 +0800 Subject: [ofa-general] An error during installation of the OFED Message-ID: <91fe68d50906030549y30f87daexabcb0cfbf8a355e9@mail.gmail.com> I try to install OFED on fedora 10 , but there is an error when build ofa_kernel RPM, and the log writes like this : make[1]: Entering directory `/lib/modules/2.6.27.5-117.fc10.i686/build' make[1]: *** No rule to make target `modules'. Stop. make[1]: Leaving directory `/lib/modules/2.6.27.5-117.fc10.i686/build' make: *** [kernel] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.tSfFAI (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.tSfFAI (%build) Can anybody tell me how to solve this problem ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at dev.mellanox.co.il Wed Jun 3 06:14:08 2009 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Wed, 03 Jun 2009 16:14:08 +0300 Subject: [ofa-general] ofa_1_4_kernel 20090603-0200 daily build status In-Reply-To: <20090603123934.GA2574@opengridcomputing.com> References: <20090603101459.35700E305F2@openfabrics.org> <20090603123934.GA2574@opengridcomputing.com> Message-ID: <4A267720.9090403@dev.mellanox.co.il> Jon Mason wrote: > Is this still the 1.4 nightly build or is this now the 1.5 nightly? > These errors look very similar to the ones I am seeing in 1.5, but the > e-mail title says 1.4... > > Thanks, > Jon > > On Wed, Jun 03, 2009 at 03:14:59AM -0700, Vladimir Sokolovsky (Mellanox) wrote: >> This email was generated automatically, please do not reply >> >> >> git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git >> git_branch: ofed_kernel >> >> Common build parameters: It was 1.4 nightly build. There were some patches committed today to ofed_1_4 git tree which cause these failures. I removed these patches from this git tree. Regards, Vladimir From tziporet at dev.mellanox.co.il Wed Jun 3 06:49:05 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Wed, 03 Jun 2009 16:49:05 +0300 Subject: [ofa-general] An error during installation of the OFED In-Reply-To: <91fe68d50906030549y30f87daexabcb0cfbf8a355e9@mail.gmail.com> References: <91fe68d50906030549y30f87daexabcb0cfbf8a355e9@mail.gmail.com> Message-ID: <4A267F51.6000309@mellanox.co.il> Jordan wrote: > I try to install OFED on fedora 10 , but there is an error when build > ofa_kernel RPM, > and the log writes like this : > > OFED 1.4 does not have backports for Fedora10 Since OFED does support kernel 2.6.27 you can try to add them yourself Tziporet From hal.rosenstock at gmail.com Wed Jun 3 07:04:20 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 10:04:20 -0400 Subject: [ofa-general] Re: osm_qos_parser_y question In-Reply-To: <4A25AA8C.4040607@dev.mellanox.co.il> References: <4A25AA8C.4040607@dev.mellanox.co.il> Message-ID: On Tue, Jun 2, 2009 at 6:41 PM, Yevgeny Kliteynik wrote: > Hi Hal, > > Hal Rosenstock wrote: >> >> Hi Yevgeny, >> >> I was wondering about the following in osm_qos_parser_y: >> >> Both __parser_add_pkey_range_to_port_map and >> __parser_add_partition_list_to_port_map seem to add the port map to >> both the full and limited (partial) ports for that partition. Am I >> reading this right ? Should it be that way ? > > They both do this: > >  __parser_add_map_to_port_map(p_map, &p_prtn->part_guid_tbl); >  __parser_add_map_to_port_map(p_map, &p_prtn->full_guid_tbl); > > The function that they call is defined as follows: > >  static void __parser_add_map_to_port_map( >                        cl_qmap_t * p_dmap, >                        cl_map_t  * p_smap) > > Which means that they add all the guids from full and partial > membership to the p_map. Looks OK to me. > > Perhaps you thought that they add p_map to p_prtn->full_guid_tbl > and p_prtn->part_guid_tbl, and not vice versa? Yes, I had the source and dest maps backwards. Thanks. -- Hal > -- Yevgeny > > >> Thanks. >> >> -- Hal >> > > From dorfman.eli at gmail.com Wed Jun 3 07:12:56 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 03 Jun 2009 17:12:56 +0300 Subject: [ofa-general] [PATCH v2] infiniband-diags: Do not change port state on SubnSet In-Reply-To: References: <4A2297DE.3050707@gmail.com> Message-ID: <4A2684E8.403@gmail.com> Do not change port state on SubnSet When changing physical state do not change logical port state to avoid invalid transitions in port state machine. >From the IB spec When writing PortInfo:PortState, only legal transitions are valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. This patch will allow reset in a single MAD by changing physical state to polling. Signed-off-by: Eli Dorfman --- infiniband-diags/src/ibportstate.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c index 65c9ca1..d19a2e5 100644 --- a/infiniband-diags/src/ibportstate.c +++ b/infiniband-diags/src/ibportstate.c @@ -275,8 +275,10 @@ int main(int argc, char **argv) /* Only if one of the "set" options is chosen */ if (port_op) { - if (port_op == 1) /* Enable port */ + if (port_op == 1) { /* Enable port */ mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ + mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ + } else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ mad_set_field(data, 0, IB_PORT_STATE_F, 1); /* Down */ mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3); /* Disabled */ @@ -292,6 +294,7 @@ int main(int argc, char **argv) if (port_op == 3) { /* Reset port - so also enable */ mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ + mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ err = set_port_info(&portid, data, portnum, port_op); if (err < 0) IBERROR("smp set portinfo failed"); -- 1.5.5 From dledford at redhat.com Wed Jun 3 07:14:07 2009 From: dledford at redhat.com (Doug Ledford) Date: Wed, 3 Jun 2009 10:14:07 -0400 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <20090603073949.GC3261@cefeid.wcss.wroc.pl> References: <4A257DBF.4000401@mellanox.co.il> <20090603073949.GC3261@cefeid.wcss.wroc.pl> Message-ID: On Jun 3, 2009, at 3:39 AM, Pawel Dziekonski wrote: > On Tue, 02 Jun 2009 at 10:30:07PM +0300, Tziporet Koren wrote: > >> Main reasons to keep MPI in OFED: >> - All participants test with the same MPI versions, and when >> installing >> OFED it is ensured that MPI will work fine with this version. >> - Customers convenience in install (no need to go to more sites to >> get MPI) >> - MPI is an important RDMA ULP and although it is not developed in >> OFA it >> is widely used by OFED customers > > As a customer I strongly support above mentioned pros. It's a > guarantee for us that MPI is well tested with OFED release. MPI makes an effective test bed for the RDMA stack whether it is shipped with it or not. Removing the MPIs from the distribution would not, in all likelihood, change the fact that MPIs would be used to test the RDMA stack prior to release. > I believe > that this effort saves a lot of troubles that would be raised from > separate releases of MPI and OFED distros. If you truly believe this, and you accept that shipping the MPI with the RDMA stack is an acceptable solution to the problem, then you are encouraging totally craptacular engineering as a customer. Since you have a non-US email address, and since craptacular is a word I use frequently, but which I also sort of just made up, let me define that. Sometimes, things are good. When they are really good, they are spectacular. Sometimes, things are crap. When they are *really* crap, they are craptacular. The RDMA stack provides an API. The MPI stacks are nothing more than a consumer of that API. This is no different than TCP sockets and MPI stacks: API provider/consumer relationship. If there is a problem between the MPIs and the RDMA stacks from version to version, it means that one of them isn't adhering to the API. The proper solution to that problem is *NOT* to put them together and throw the API out the window, it's to get the API provider and consumer to adhere to the API and to make sure that the API works. Otherwise, the only solution is to bundle *every* consumer of that API with the API provider because the API stability is, you guessed it, craptacular. If you aren't part of the solution, you're part of the problem. Don't encourage craptacular engineering that throws the API out the window. > > thanks, Pawel > -- > Pawel Dziekonski > Wroclaw Centre for Networking & Supercomputing, HPC Department > Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, > POLAND > phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford InfiniBand Specific RPMS http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part URL: From fenkes at de.ibm.com Wed Jun 3 07:28:17 2009 From: fenkes at de.ibm.com (Joachim Fenkes) Date: Wed, 3 Jun 2009 16:28:17 +0200 Subject: [ofa-general] [PATCH] IB/ehca: Remove superfluous bitmasks from QP control block In-Reply-To: <48499C11.7030504@gmail.com> References: <200806061835.43802.fenkes@de.ibm.com> <48499C11.7030504@gmail.com> Message-ID: <200906031628.19556.fenkes@de.ibm.com> All the fields in the control block are nicely right-aligned, so no masking is necessary. Signed-off-by: Joachim Fenkes --- drivers/infiniband/hw/ehca/ehca_classes_pSeries.h | 28 --------------------- drivers/infiniband/hw/ehca/ehca_qp.c | 18 +++--------- 2 files changed, 5 insertions(+), 41 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_classes_pSeries.h b/drivers/infiniband/hw/ehca/ehca_classes_pSeries.h index 1798e64..689c357 100644 --- a/drivers/infiniband/hw/ehca/ehca_classes_pSeries.h +++ b/drivers/infiniband/hw/ehca/ehca_classes_pSeries.h @@ -165,7 +165,6 @@ struct hcp_modify_qp_control_block { #define MQPCB_MASK_ALT_P_KEY_IDX EHCA_BMASK_IBM( 7, 7) #define MQPCB_MASK_RDMA_ATOMIC_CTRL EHCA_BMASK_IBM( 8, 8) #define MQPCB_MASK_QP_STATE EHCA_BMASK_IBM( 9, 9) -#define MQPCB_QP_STATE EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_RDMA_NR_ATOMIC_RESP_RES EHCA_BMASK_IBM(11, 11) #define MQPCB_MASK_PATH_MIGRATION_STATE EHCA_BMASK_IBM(12, 12) #define MQPCB_MASK_RDMA_ATOMIC_OUTST_DEST_QP EHCA_BMASK_IBM(13, 13) @@ -176,60 +175,33 @@ struct hcp_modify_qp_control_block { #define MQPCB_MASK_RETRY_COUNT EHCA_BMASK_IBM(18, 18) #define MQPCB_MASK_TIMEOUT EHCA_BMASK_IBM(19, 19) #define MQPCB_MASK_PATH_MTU EHCA_BMASK_IBM(20, 20) -#define MQPCB_PATH_MTU EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_MAX_STATIC_RATE EHCA_BMASK_IBM(21, 21) -#define MQPCB_MAX_STATIC_RATE EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_DLID EHCA_BMASK_IBM(22, 22) -#define MQPCB_DLID EHCA_BMASK_IBM(16, 31) #define MQPCB_MASK_RNR_RETRY_COUNT EHCA_BMASK_IBM(23, 23) -#define MQPCB_RNR_RETRY_COUNT EHCA_BMASK_IBM(29, 31) #define MQPCB_MASK_SOURCE_PATH_BITS EHCA_BMASK_IBM(24, 24) -#define MQPCB_SOURCE_PATH_BITS EHCA_BMASK_IBM(25, 31) #define MQPCB_MASK_TRAFFIC_CLASS EHCA_BMASK_IBM(25, 25) -#define MQPCB_TRAFFIC_CLASS EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_HOP_LIMIT EHCA_BMASK_IBM(26, 26) -#define MQPCB_HOP_LIMIT EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_SOURCE_GID_IDX EHCA_BMASK_IBM(27, 27) -#define MQPCB_SOURCE_GID_IDX EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_FLOW_LABEL EHCA_BMASK_IBM(28, 28) -#define MQPCB_FLOW_LABEL EHCA_BMASK_IBM(12, 31) #define MQPCB_MASK_DEST_GID EHCA_BMASK_IBM(30, 30) #define MQPCB_MASK_SERVICE_LEVEL_AL EHCA_BMASK_IBM(31, 31) -#define MQPCB_SERVICE_LEVEL_AL EHCA_BMASK_IBM(28, 31) #define MQPCB_MASK_SEND_GRH_FLAG_AL EHCA_BMASK_IBM(32, 32) -#define MQPCB_SEND_GRH_FLAG_AL EHCA_BMASK_IBM(31, 31) #define MQPCB_MASK_RETRY_COUNT_AL EHCA_BMASK_IBM(33, 33) -#define MQPCB_RETRY_COUNT_AL EHCA_BMASK_IBM(29, 31) #define MQPCB_MASK_TIMEOUT_AL EHCA_BMASK_IBM(34, 34) -#define MQPCB_TIMEOUT_AL EHCA_BMASK_IBM(27, 31) #define MQPCB_MASK_MAX_STATIC_RATE_AL EHCA_BMASK_IBM(35, 35) -#define MQPCB_MAX_STATIC_RATE_AL EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_DLID_AL EHCA_BMASK_IBM(36, 36) -#define MQPCB_DLID_AL EHCA_BMASK_IBM(16, 31) #define MQPCB_MASK_RNR_RETRY_COUNT_AL EHCA_BMASK_IBM(37, 37) -#define MQPCB_RNR_RETRY_COUNT_AL EHCA_BMASK_IBM(29, 31) #define MQPCB_MASK_SOURCE_PATH_BITS_AL EHCA_BMASK_IBM(38, 38) -#define MQPCB_SOURCE_PATH_BITS_AL EHCA_BMASK_IBM(25, 31) #define MQPCB_MASK_TRAFFIC_CLASS_AL EHCA_BMASK_IBM(39, 39) -#define MQPCB_TRAFFIC_CLASS_AL EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_HOP_LIMIT_AL EHCA_BMASK_IBM(40, 40) -#define MQPCB_HOP_LIMIT_AL EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_SOURCE_GID_IDX_AL EHCA_BMASK_IBM(41, 41) -#define MQPCB_SOURCE_GID_IDX_AL EHCA_BMASK_IBM(24, 31) #define MQPCB_MASK_FLOW_LABEL_AL EHCA_BMASK_IBM(42, 42) -#define MQPCB_FLOW_LABEL_AL EHCA_BMASK_IBM(12, 31) #define MQPCB_MASK_DEST_GID_AL EHCA_BMASK_IBM(44, 44) #define MQPCB_MASK_MAX_NR_OUTST_SEND_WR EHCA_BMASK_IBM(45, 45) -#define MQPCB_MAX_NR_OUTST_SEND_WR EHCA_BMASK_IBM(16, 31) #define MQPCB_MASK_MAX_NR_OUTST_RECV_WR EHCA_BMASK_IBM(46, 46) -#define MQPCB_MAX_NR_OUTST_RECV_WR EHCA_BMASK_IBM(16, 31) #define MQPCB_MASK_DISABLE_ETE_CREDIT_CHECK EHCA_BMASK_IBM(47, 47) -#define MQPCB_DISABLE_ETE_CREDIT_CHECK EHCA_BMASK_IBM(31, 31) -#define MQPCB_QP_NUMBER EHCA_BMASK_IBM( 8, 31) #define MQPCB_MASK_QP_ENABLE EHCA_BMASK_IBM(48, 48) -#define MQPCB_QP_ENABLE EHCA_BMASK_IBM(31, 31) #define MQPCB_MASK_CURR_SRQ_LIMIT EHCA_BMASK_IBM(49, 49) -#define MQPCB_CURR_SRQ_LIMIT EHCA_BMASK_IBM(16, 31) #define MQPCB_MASK_QP_AFF_ASYN_EV_LOG_REG EHCA_BMASK_IBM(50, 50) #define MQPCB_MASK_SHARED_RQ_HNDL EHCA_BMASK_IBM(51, 51) diff --git a/drivers/infiniband/hw/ehca/ehca_qp.c b/drivers/infiniband/hw/ehca/ehca_qp.c index ead4e71..0338f1f 100644 --- a/drivers/infiniband/hw/ehca/ehca_qp.c +++ b/drivers/infiniband/hw/ehca/ehca_qp.c @@ -1962,19 +1962,13 @@ int ehca_query_qp(struct ib_qp *qp, qp_attr->cap.max_inline_data = my_qp->sq_max_inline_data_size; qp_attr->dest_qp_num = qpcb->dest_qp_nr; - qp_attr->pkey_index = - EHCA_BMASK_GET(MQPCB_PRIM_P_KEY_IDX, qpcb->prim_p_key_idx); - - qp_attr->port_num = - EHCA_BMASK_GET(MQPCB_PRIM_PHYS_PORT, qpcb->prim_phys_port); - + qp_attr->pkey_index = qpcb->prim_p_key_idx; + qp_attr->port_num = qpcb->prim_phys_port; qp_attr->timeout = qpcb->timeout; qp_attr->retry_cnt = qpcb->retry_count; qp_attr->rnr_retry = qpcb->rnr_retry_count; - qp_attr->alt_pkey_index = - EHCA_BMASK_GET(MQPCB_PRIM_P_KEY_IDX, qpcb->alt_p_key_idx); - + qp_attr->alt_pkey_index = qpcb->alt_p_key_idx; qp_attr->alt_port_num = qpcb->alt_phys_port; qp_attr->alt_timeout = qpcb->timeout_al; @@ -2061,8 +2055,7 @@ int ehca_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr, update_mask |= EHCA_BMASK_SET(MQPCB_MASK_CURR_SRQ_LIMIT, 1) | EHCA_BMASK_SET(MQPCB_MASK_QP_AFF_ASYN_EV_LOG_REG, 1); - mqpcb->curr_srq_limit = - EHCA_BMASK_SET(MQPCB_CURR_SRQ_LIMIT, attr->srq_limit); + mqpcb->curr_srq_limit = attr->srq_limit; mqpcb->qp_aff_asyn_ev_log_reg = EHCA_BMASK_SET(QPX_AAELOG_RESET_SRQ_LIMIT, 1); } @@ -2125,8 +2118,7 @@ int ehca_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr) srq_attr->max_wr = qpcb->max_nr_outst_recv_wr - 1; srq_attr->max_sge = 3; - srq_attr->srq_limit = EHCA_BMASK_GET( - MQPCB_CURR_SRQ_LIMIT, qpcb->curr_srq_limit); + srq_attr->srq_limit = qpcb->curr_srq_limit; if (ehca_debug_level >= 2) ehca_dmp(qpcb, 4*70, "qp_num=%x", my_qp->real_qp_num); -- 1.5.5 From Bob.Ciotti at nasa.gov Wed Jun 3 10:11:20 2009 From: Bob.Ciotti at nasa.gov (Bob Ciotti) Date: Wed, 3 Jun 2009 10:11:20 -0700 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <4A265896.2080801@gmail.com> References: <9DD9848F225E4B57AC156EBC81B68520@amr.corp.intel.com> <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> <4A2657F8.3010807@gmail.com> <4A265896.2080801@gmail.com> Message-ID: <20090603171120.GB63375@nas.nasa.gov> On Wed, Jun 03, 2009 at 06:03:50AM -0500, Eli Dorfman (Voltaire) wrote: > Eli Dorfman (Voltaire) wrote: > > Hal Rosenstock wrote: > >> On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: > >>> On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: > >>>>> Yes, RMPP is an overhead when the response is a single MAD but is this > >>>>> significant ? Anyhow, how can the spec be changed in a way that > >>>>> doesn't break existing implementations ? > >>>> But the implementations are assuming different things about SubnAdmGet. The SA > >>>> is assuming that the query should fail if multiple records match. The client > >>>> side software (ipoib and rdma_cm) assume that it will obtain a single record > >>>> even if multiple paths are present. So, something needs to change. > >>> Seems so. > >>> > >>>> The spec indicates that value in the request is ignored and NumbPath is 1, not > >>>> that NumbPath is completely ignored. > >>> For Get, it doesn't say that the matches are paired down to this > >>> number as it does for GetTable. > >>> > >>>> Also see page 1242 in the SDP annex which > >>>> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet > >>>> rather than SubnAdmGetTable)'. > >>> SDP annex is not the primary source for this (chapter 15 is) and is > >>> inconsistent and no one caught this. > >>> > >>>> To me, this implies that SubnAdmGet should be > >>>> treated equivalent as SubnAdmGetTable with NumbPath = 1. > >>>> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus > >>>> PR SubnAdmGetTable and MPR SubAdmGetMulti. Basically, this makes PR SubnAdmGet > >>>> useless. > >>> when there's a subnet with multiple paths and the requests are not > >>> specific enough to use get. > >>> > >>> Seems like either the queries need to use RMPP, or the spec modified > >>> (if that's possible) and the SAs updated. > >> I sit corrected :-) Your interpretation of the spec is correct. Also, > >> in looking at OpenSM, the intent is as you indicate: it does try to > >> only return 1 attibute for get PR. If when returning the response, > >> there is more than 1 attribute in the list, it returns the too many > >> records error. There must be some code path I don't see right now > >> which is doing this. It would be useful to know the details of the > >> query (get request) causing this. > >> > > > > This may happen when pr_rcv_get_port_pair_paths() is called several times. > > The only case i see is pr_rcv_process_world() that means the request is without or wrong > > src and dest port or component mask for SGID and DGID is 0. > > correction - this may happen only when component mask for SGID and DGID is 0. Here is a mad dump of the offending sequence. Jun 02 12:43:01 355975 [3DD13940] 0x80 -> SUBNET UP Jun 02 12:43:03 484480 [5020B940] 0x20 -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x1 (SubnAdmGet) status..................0x0 resv....................0x0 trans_id................0x2b82ad0a0000 attr_id.................0x11 (NodeRecord) resv1...................0x0 attr_mod................0x0 rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000000001 Jun 02 12:43:03 490828 [19323940] 0x01 -> osm_sa_respond: ERR 4C05: Got more than one record for SubnAdmGet (6733) Jun 02 12:43:03 490891 [19323940] 0x20 -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x81 (SubnAdmGetResp) status..................0x400 resv....................0x0 trans_id................0x2b82ad0a0000 attr_id.................0x11 (NodeRecord) resv1...................0x0 attr_mod................0x0 rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000000001 bob From dzieko at wcss.pl Wed Jun 3 10:14:34 2009 From: dzieko at wcss.pl (Pawel Dziekonski) Date: Wed, 3 Jun 2009 19:14:34 +0200 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: <4A257DBF.4000401@mellanox.co.il> <20090603073949.GC3261@cefeid.wcss.wroc.pl> Message-ID: <20090603171434.GR3261@cefeid.wcss.wroc.pl> On Wed, 03 Jun 2009 at 10:14:07AM -0400, Doug Ledford wrote: > On Jun 3, 2009, at 3:39 AM, Pawel Dziekonski wrote: >> On Tue, 02 Jun 2009 at 10:30:07PM +0300, Tziporet Koren wrote: >> >>> Main reasons to keep MPI in OFED: >>> - All participants test with the same MPI versions, and when installing >>> OFED it is ensured that MPI will work fine with this version. >>> - Customers convenience in install (no need to go to more sites to get >>> MPI) >>> - MPI is an important RDMA ULP and although it is not developed in OFA it >>> is widely used by OFED customers >> >> As a customer I strongly support above mentioned pros. It's a >> guarantee for us that MPI is well tested with OFED release. > > MPI makes an effective test bed for the RDMA stack whether it is shipped > with it or not. Removing the MPIs from the distribution would not, in all > likelihood, change the fact that MPIs would be used to test the RDMA stack > prior to release. > >> I believe that this effort saves a lot of troubles that would be >> raised from separate releases of MPI and OFED distros. > > If you truly believe this, and you accept that shipping the MPI with > the RDMA stack is an acceptable solution to the problem, then you > are encouraging totally craptacular engineering as a customer. > Since you have a non-US email address, and since craptacular is a > word I use frequently, but which I also sort of just made up, let me > define that. Sometimes, things are good. When they are really good, > they are spectacular. Sometimes, things are crap. When they are > *really* crap, they are craptacular. > > The RDMA stack provides an API. The MPI stacks are nothing more than you look at the problem from vendor point of view (vendor-like mail domain? ;). you care about api. i care about mpi. from technical point of view it is enough for me if you say in ofed docs that THIS and THAT particular versions of MPI was tested and WORKS. just like suppoerted and tested list of linux kernels and linux distros. I definitely can download and compile mpi by myself. however since you already used some versions of MPI to test the RDMA stack prior to release why not simply attach it to release? it just makes the release more complete from CUSTOMER point of view. cheers, P -- Pawel Dziekonski Wroclaw Centre for Networking & Supercomputing, HPC Department Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl From hal.rosenstock at gmail.com Wed Jun 3 11:08:41 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 14:08:41 -0400 Subject: [ofa-general] SubnAdmGet (6777) In-Reply-To: <20090603171120.GB63375@nas.nasa.gov> References: <4787ACF898B245218D575069DE137A3E@amr.corp.intel.com> <4A2657F8.3010807@gmail.com> <4A265896.2080801@gmail.com> <20090603171120.GB63375@nas.nasa.gov> Message-ID: On Wed, Jun 3, 2009 at 1:11 PM, Bob Ciotti wrote: > On Wed, Jun 03, 2009 at 06:03:50AM -0500, Eli Dorfman (Voltaire) wrote: >> Eli Dorfman (Voltaire) wrote: >> > Hal Rosenstock wrote: >> >> On Mon, Jun 1, 2009 at 5:36 PM, Hal Rosenstock wrote: >> >>> On Mon, Jun 1, 2009 at 4:27 PM, Sean Hefty wrote: >> >>>>> Yes, RMPP is an overhead when the response is a single MAD but is this >> >>>>> significant ? Anyhow, how can the spec be changed in a way that >> >>>>> doesn't break existing implementations ? >> >>>> But the implementations are assuming different things about SubnAdmGet.  The SA >> >>>> is assuming that the query should fail if multiple records match.  The client >> >>>> side software (ipoib and rdma_cm) assume that it will obtain a single record >> >>>> even if multiple paths are present.  So, something needs to change. >> >>> Seems so. >> >>> >> >>>> The spec indicates that value in the request is ignored and NumbPath is 1, not >> >>>> that NumbPath is completely ignored. >> >>> For Get, it doesn't say that the matches are paired down to this >> >>> number as it does for GetTable. >> >>> >> >>>>  Also see page 1242 in the SDP annex which >> >>>> reads: 'NumbPath could be 1 (in which case the SA query may use SubnAdmGet >> >>>> rather than SubnAdmGetTable)'. >> >>> SDP annex is not the primary source for this (chapter 15 is) and is >> >>> inconsistent and no one caught this. >> >>> >> >>>> To me, this implies that SubnAdmGet should be >> >>>> treated equivalent as SubnAdmGetTable with NumbPath = 1. >> >>>> It just seems really odd to treat NumbPath differently for PR SubnAdmGet versus >> >>>> PR SubnAdmGetTable and MPR SubAdmGetMulti.  Basically, this makes PR SubnAdmGet >> >>>> useless. >> >>> when there's a subnet with multiple paths and the requests are not >> >>> specific enough to use get. >> >>> >> >>> Seems like either the queries need to use RMPP, or the spec modified >> >>> (if that's possible) and the SAs updated. >> >> I sit corrected :-) Your interpretation of the spec is correct. Also, >> >> in looking at OpenSM, the intent is as you indicate: it does try to >> >> only return 1 attibute for get PR. If when returning the response, >> >> there is more than 1 attribute in the list, it returns the too many >> >> records error. There must be some code path I don't see right now >> >> which is doing this. It would be useful to know the details of the >> >> query (get request) causing this. >> >> >> > >> > This may happen when pr_rcv_get_port_pair_paths() is called several times. >> > The only case i see is pr_rcv_process_world() that means the request is without or wrong >> > src and dest port or component mask for SGID and DGID is 0. >> >> correction - this may happen only when component mask for SGID and DGID is 0. > > Here is a mad dump of the offending sequence. > > Jun 02 12:43:01 355975 [3DD13940] 0x80 -> SUBNET UP > Jun 02 12:43:03 484480 [5020B940] 0x20 -> SA MAD dump: >                                base_ver................0x1 >                                mgmt_class..............0x3 >                                class_ver...............0x2 >                                method..................0x1 (SubnAdmGet) >                                status..................0x0 >                                resv....................0x0 >                                trans_id................0x2b82ad0a0000 >                                attr_id.................0x11 (NodeRecord) >                                resv1...................0x0 >                                attr_mod................0x0 >                                rmpp_version............0x0 >                                rmpp_type...............0x0 >                                rmpp_flags..............0x0 >                                rmpp_status.............0x0 >                                seg_num.................0x0 >                                payload_len/new_win.....0x0 >                                sm_key..................0x0000000000000000 >                                attr_offset.............0x0 >                                resv2...................0x0 >                                comp_mask...............0x0000000000000001 Looks like a LID match on NodeRecord is being requested but there's more than one. Returning an error is correct for this case. Any idea on what component is issuing the gets on NodeRecord ? By any chance are you using LMC > 0 ? -- Hal > [19323940] 0x20 -> SA MAD dump: >                                base_ver................0x1 >                                mgmt_class..............0x3 >                                class_ver...............0x2 >                                method..................0x81 (SubnAdmGetResp) >                                status..................0x400 >                                resv....................0x0 >                                trans_id................0x2b82ad0a0000 >                                attr_id.................0x11 (NodeRecord) >                                resv1...................0x0 >                                attr_mod................0x0 >                                rmpp_version............0x0 >                                rmpp_type...............0x0 >                                rmpp_flags..............0x0 >                                rmpp_status.............0x0 >                                seg_num.................0x0 >                                payload_len/new_win.....0x0 >                                sm_key..................0x0000000000000000 >                                attr_offset.............0x0 >                                resv2...................0x0 >                                comp_mask...............0x0000000000000001 > > > bob > From Frank.Leers at Sun.COM Wed Jun 3 13:10:44 2009 From: Frank.Leers at Sun.COM (Frank Leers) Date: Wed, 03 Jun 2009 13:10:44 -0700 Subject: [ofa-general] multi-rail subnet_prefix and MPI issue Message-ID: <3E42C826-5D0E-4A87-BBC3-5E2A2ACA539A@sun.com> Hi, I have a question regarding unique subnet prefixes for a 2-rail QDR configuration. I'm using connectX on CentOS 5.3 with the distro- supplied OFED 1.3.2, kernel 2.6.18-128.el5, opensm-3.2.2-3.el5. Any insight appreciated. -frank # ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.6.0 Hardware version: a0 Node GUID: 0x0003ba000100fc04 System image GUID: 0x0003ba000100fc07 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x0251086a Port GUID: 0x0003ba000100fc05 Port 2: State: Active Physical state: LinkUp Rate: 10 Base lid: 18 LMC: 0 SM lid: 18 Capability mask: 0x0251086a Port GUID: 0x0003ba000100fc06 ...and specify the unique subnet prefixes in each SM instance's config file like this: # grep prefix /etc/ofed/opensm-ib0.conf OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" # grep prefix /etc/ofed/opensm-ib1.conf OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000020" Which results in these instances running: # ps -ef|grep opensm root 1498 1 0 05:16 ? 00:00:00 /usr/sbin/opensm -B - maxsmps 0 -f /var/log/opensm-ib0.log -p 15 -g 0x0003ba000100fc05 subnet_prefix 0xfe80000000000010 root 2450 1 0 05:37 ? 00:00:00 /usr/sbin/opensm -B - maxsmps 0 -f /var/log/opensm-ib1.log -p 15 -g 0x0003ba000100fc06 subnet_prefix 0xfe80000000000020 Yet we are seeing complaints from openMPI as such(single rail works fine): $ mpirun -np 2 -machinefile ~/myduals ./IMB-MPI1 -------------------------------------------------------------------------- WARNING: There are more than one active ports on host 'n0051', but the default subnet GID prefix was detected on more than one of these ports. If these ports are connected to different physical IB networks, this configuration will fail in Open MPI. This version of Open MPI requires that every physically separate IB subnet that is used between connected MPI processes must have different subnet ID values. Please see this FAQ entry for more details: http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_default_gid_prefix to 0. -------------------------------------------------------------------------- [[1372,1],0][btl_openib_component.c:2827:handle_wc] from n0051 to: n0052 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 197710080 opcode 0 qp_idx 0 -------------------------------------------------------------------------- The InfiniBand retry count between two MPI processes has been exceeded. "Retry count" is defined in the InfiniBand spec 1.2 (section 12.7.38): The total number of times that the sender wishes the receiver to retry timeout, packet sequence, etc. errors before posting a completion error. This error typically means that there is something awry within the InfiniBand fabric itself. You should note the hosts on which this error has occurred; it has been observed that rebooting or removing a particular host from the job can sometimes resolve this issue. Two MCA parameters can be used to control Open MPI's behavior with respect to the retry count: * btl_openib_ib_retry_count - The number of times the sender will attempt to retry (defaulted to 7, the maximum value). * btl_openib_ib_timeout - The local ACK timeout parameter (defaulted to 10). The actual timeout value used is calculated as: 4.096 microseconds * (2^btl_openib_ib_timeout) See the InfiniBand spec 1.2 (section 12.7.34) for more details. Below is some information about the host that raised the error and the peer to which it was connected: Local host: n0051 Local device: mlx4_0 Peer host: n0052 You may need to consult with your system administrator to get this problem fixed. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 9242 on node n0051 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- [admin1:03239] 1 more process has sent help message help-mpi-btl- openib.txt / default subnet prefix [admin1:03239] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages From rdreier at cisco.com Wed Jun 3 13:24:46 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 03 Jun 2009 13:24:46 -0700 Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: <4A266ABB.6000603@Voltaire.com> (Or Gerlitz's message of "Wed, 03 Jun 2009 15:21:15 +0300") References: <4A266ABB.6000603@Voltaire.com> Message-ID: > If we go this way, I went and did the same thing for unicast, patch below. Alternatively we may have both mcast/unicast queues to remain and get their length set by the net.ipvY.neigh.ibX.unres_qlen? > I tested my patch with TCP/UDP netperf/iperf over 2.6.29.1 and things seem to work fine. Hmm... interesting point about the unres_qlen sysctl. It does seem the current net stack just drops packets during arp resolution, and IPoIB path resolution / multicast join is arguably the analogous thing. So now I begin to wonder about Christoph's patch again. With the old code we drop a lot of packets (potentially a lot more because of the unthrottled sender), but only during fabric events that cause multicast joins; but is delaying a lot of packets during a multicast join really better for actual apps? Or are we better off dropping those packets we can't deliver in a timely way? Christoph, is this making a real app work better, or just making a multicast flood test case report better numbers? - R. From rdreier at cisco.com Wed Jun 3 13:26:01 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 03 Jun 2009 13:26:01 -0700 Subject: [ofa-general] Re: [PATCH] IB/ehca: Remove superfluous bitmasks from QP control block In-Reply-To: <200906031628.19556.fenkes@de.ibm.com> (Joachim Fenkes's message of "Wed, 3 Jun 2009 16:28:17 +0200") References: <200806061835.43802.fenkes@de.ibm.com> <48499C11.7030504@gmail.com> <200906031628.19556.fenkes@de.ibm.com> Message-ID: looks fine, applied for 2.6.31 From cl at linux-foundation.org Wed Jun 3 13:33:57 2009 From: cl at linux-foundation.org (Christoph Lameter) Date: Wed, 3 Jun 2009 16:33:57 -0400 (EDT) Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: References: <4A266ABB.6000603@Voltaire.com> Message-ID: On Wed, 3 Jun 2009, Roland Dreier wrote: > current net stack just drops packets during arp resolution, and IPoIB > path resolution / multicast join is arguably the analogous thing. So > now I begin to wonder about Christoph's patch again. With the old code > we drop a lot of packets (potentially a lot more because of the > unthrottled sender), but only during fabric events that cause multicast > joins; but is delaying a lot of packets during a multicast join really > better for actual apps? Or are we better off dropping those packets we > can't deliver in a timely way? arps are a different thing from regular payloads. They are not handled using socket semantics. > Christoph, is this making a real app work better, or just making a > multicast flood test case report better numbers? It is necessary for a real app to work that was moved from 1G ethernet to IB. Startup fails because of significant loss of multicast packets during join. From akepner at sgi.com Wed Jun 3 13:40:41 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Wed, 3 Jun 2009 13:40:41 -0700 Subject: [ofa-general] ibdiagnet: invalid bareword "pkey_entry" Message-ID: <20090603204041.GF26682@sgi.com> (Yevgeny, looks like you're maintaining ibutils. Please let me know if it's otherwise.) We're getting this: # ibdiagnet ..... -I--------------------------------------------------- -I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list) -I--------------------------------------------------- invalid bareword "pkey_entry" in expression "-pkey_entry & 0x7fff"; should be "$pkey_entry" or "{pkey_entry}" or "pkey_entry(...)" or ... # when running ibdiagnet. Any idea why we're seeing this now, when we haven't before? The following fixes it, but maybe there's a better way (it's the first line of tcl I've ever written): --- /usr/lib64/ibdiagnet1.2/ibdebug.tcl 2009-06-03 13:11:09.000000000 -0700 +++ /usr/lib64/ibdiagnet1.2/ibdebug.tcl.orig 2009-06-03 11:01:06.000000000 -0700 @@ -3064,7 +3064,6 @@ proc GetPortPkeys {drPath portNum numPKe inform "-E-ibdiagnet:PKeys.getPkey" $drPath $portNum $block continue } - set pkeyTable [RemoveElementFromList $pkeyTable "-pkey_entry"] foreach pkey $pkeyTable { if {$pkey != 0} { lappend pkeys $pkey From akepner at sgi.com Wed Jun 3 13:46:59 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Wed, 3 Jun 2009 13:46:59 -0700 Subject: [ofa-general] Re: ibdiagnet: invalid bareword "pkey_entry" In-Reply-To: <20090603204041.GF26682@sgi.com> References: <20090603204041.GF26682@sgi.com> Message-ID: <20090603204659.GA20406@sgi.com> On Wed, Jun 03, 2009 at 01:40:41PM -0700, Arthur Kepner wrote: > .... > --- /usr/lib64/ibdiagnet1.2/ibdebug.tcl 2009-06-03 13:11:09.000000000 -0700 > +++ /usr/lib64/ibdiagnet1.2/ibdebug.tcl.orig 2009-06-03 11:01:06.000000000 -0700 > @@ -3064,7 +3064,6 @@ proc GetPortPkeys {drPath portNum numPKe > inform "-E-ibdiagnet:PKeys.getPkey" $drPath $portNum $block > continue > } > - set pkeyTable [RemoveElementFromList $pkeyTable "-pkey_entry"] > foreach pkey $pkeyTable { > if {$pkey != 0} { > lappend pkeys $pkey Well, obviously that patch is ass-backward. But you get the idea. -- Arthur From hal.rosenstock at gmail.com Wed Jun 3 13:52:18 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 16:52:18 -0400 Subject: [ofa-general] multi-rail subnet_prefix and MPI issue In-Reply-To: <3E42C826-5D0E-4A87-BBC3-5E2A2ACA539A@sun.com> References: <3E42C826-5D0E-4A87-BBC3-5E2A2ACA539A@sun.com> Message-ID: On Wed, Jun 3, 2009 at 4:10 PM, Frank Leers wrote: > Hi, > > I have a question regarding unique subnet prefixes for a 2-rail QDR > configuration.  I'm using connectX on CentOS 5.3 with the distro-supplied > OFED 1.3.2, kernel 2.6.18-128.el5, opensm-3.2.2-3.el5. > > Any insight appreciated. > > -frank > > > > > # ibstat > CA 'mlx4_0' >        CA type: MT26428 >        Number of ports: 2 >        Firmware version: 2.6.0 >        Hardware version: a0 >        Node GUID: 0x0003ba000100fc04 >        System image GUID: 0x0003ba000100fc07 >        Port 1: >                State: Active >                Physical state: LinkUp >                Rate: 40 >                Base lid: 1 >                LMC: 0 >                SM lid: 1 >                Capability mask: 0x0251086a >                Port GUID: 0x0003ba000100fc05 >        Port 2: >                State: Active >                Physical state: LinkUp >                Rate: 10 >                Base lid: 18 >                LMC: 0 >                SM lid: 18 >                Capability mask: 0x0251086a >                Port GUID: 0x0003ba000100fc06 > > ...and specify the unique subnet prefixes in each SM instance's config file > like this: > > # grep prefix /etc/ofed/opensm-ib0.conf > OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" > > # grep prefix /etc/ofed/opensm-ib1.conf > OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000020" > > > Which results in these instances running: > > # ps -ef|grep opensm > root      1498     1  0 05:16 ?        00:00:00 /usr/sbin/opensm -B -maxsmps > 0 -f /var/log/opensm-ib0.log -p 15 -g 0x0003ba000100fc05 subnet_prefix > 0xfe80000000000010 > root      2450     1  0 05:37 ?        00:00:00 /usr/sbin/opensm -B -maxsmps > 0 -f /var/log/opensm-ib1.log -p 15 -g 0x0003ba000100fc06 subnet_prefix > 0xfe80000000000020 I'm unaware of being able to set the subnet prefix as you indicate. It needs to be set in the opensm config file. You can double check the subnet prefix on the two ports by using smpquery portinfo if you have infiniband diags installed. -- Hal > Yet we are seeing complaints from openMPI as such(single rail works fine): > > $ mpirun -np 2 -machinefile ~/myduals ./IMB-MPI1 > -------------------------------------------------------------------------- > WARNING: There are more than one active ports on host 'n0051', but the > default subnet GID prefix was detected on more than one of these > ports.  If these ports are connected to different physical IB > networks, this configuration will fail in Open MPI.  This version of > Open MPI requires that every physically separate IB subnet that is > used between connected MPI processes must have different subnet ID > values. > > Please see this FAQ entry for more details: > > http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid > > NOTE: You can turn off this warning by setting the MCA parameter >    btl_openib_warn_default_gid_prefix to 0. > -------------------------------------------------------------------------- > [[1372,1],0][btl_openib_component.c:2827:handle_wc] from n0051 to: n0052 > error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for > wr_id 197710080 opcode 0 qp_idx 0 > -------------------------------------------------------------------------- > The InfiniBand retry count between two MPI processes has been > exceeded.  "Retry count" is defined in the InfiniBand spec 1.2 > (section 12.7.38): > >  The total number of times that the sender wishes the receiver to >  retry timeout, packet sequence, etc. errors before posting a >  completion error. > > This error typically means that there is something awry within the > InfiniBand fabric itself.  You should note the hosts on which this > error has occurred; it has been observed that rebooting or removing a > particular host from the job can sometimes resolve this issue. > Two MCA parameters can be used to control Open MPI's behavior with > respect to the retry count: > > * btl_openib_ib_retry_count - The number of times the sender will > attempt to retry (defaulted to 7, the maximum value). > * btl_openib_ib_timeout - The local ACK timeout parameter (defaulted > to 10).  The actual timeout value used is calculated as: > >   4.096 microseconds * (2^btl_openib_ib_timeout) > > See the InfiniBand spec 1.2 (section 12.7.34) for more details. > > Below is some information about the host that raised the error and the > peer to which it was connected: > > Local host:   n0051 > Local device: mlx4_0 > Peer host:    n0052 > > You may need to consult with your system administrator to get this > problem fixed. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > mpirun has exited due to process rank 0 with PID 9242 on > node n0051 exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > [admin1:03239] 1 more process has sent help message help-mpi-btl-openib.txt > / default subnet prefix > [admin1:03239] Set MCA parameter "orte_base_help_aggregate" to 0 to see all > help / error messages > > > > > > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From rdreier at cisco.com Wed Jun 3 14:13:26 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 03 Jun 2009 14:13:26 -0700 Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: (Christoph Lameter's message of "Wed, 3 Jun 2009 16:33:57 -0400 (EDT)") References: <4A266ABB.6000603@Voltaire.com> Message-ID: > arps are a different thing from regular payloads. They are not handled > using socket semantics. Oh right, I misunderstood the way the net core is using queue_len. OK, I think your patch makes sense and I'll add Or's too. - R. From akepner at sgi.com Wed Jun 3 14:22:39 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Wed, 3 Jun 2009 14:22:39 -0700 Subject: [ofa-general] [RFC] ipoib: avoid using stale ipoib_neigh* in ipoib_neigh_cleanup() In-Reply-To: <4A261C4B.6090909@Voltaire.com> References: <4A13ADDA.5040908@Voltaire.com> <20090521210049.GY6837@sgi.com> <4A2226B2.5070604@Voltaire.com> <20090531072115.GA9211@mtls03> <4A226B60.9070006@Voltaire.com> <20090601220413.GC14335@sgi.com> <20090602164935.GB26682@sgi.com> <4A261C4B.6090909@Voltaire.com> Message-ID: <20090603212239.GB20406@sgi.com> On Wed, Jun 03, 2009 at 09:46:35AM +0300, Or Gerlitz wrote: > .... > Arthur, again, does the race happen under datagram mode? > It's very likely that we've only seen this with connected mode. But we've seen the bug only a very few times, and not recently (since adding our 'hack-around'), so I don't have the data available to answer with absolute certainty. -- Arthur From Frank.Leers at Sun.COM Wed Jun 3 14:35:54 2009 From: Frank.Leers at Sun.COM (Frank Leers) Date: Wed, 03 Jun 2009 14:35:54 -0700 Subject: [ofa-general] multi-rail subnet_prefix and MPI issue In-Reply-To: References: <3E42C826-5D0E-4A87-BBC3-5E2A2ACA539A@sun.com> Message-ID: Thanks for the reply Hal, On Jun 3, 2009, at 1:52 PM, Hal Rosenstock wrote: > On Wed, Jun 3, 2009 at 4:10 PM, Frank Leers > wrote: >> Hi, >> >> I have a question regarding unique subnet prefixes for a 2-rail QDR >> configuration. I'm using connectX on CentOS 5.3 with the distro- >> supplied >> OFED 1.3.2, kernel 2.6.18-128.el5, opensm-3.2.2-3.el5. >> >> Any insight appreciated. >> >> -frank >> >> >> >> >> # ibstat >> CA 'mlx4_0' >> CA type: MT26428 >> Number of ports: 2 >> Firmware version: 2.6.0 >> Hardware version: a0 >> Node GUID: 0x0003ba000100fc04 >> System image GUID: 0x0003ba000100fc07 >> Port 1: >> State: Active >> Physical state: LinkUp >> Rate: 40 >> Base lid: 1 >> LMC: 0 >> SM lid: 1 >> Capability mask: 0x0251086a >> Port GUID: 0x0003ba000100fc05 >> Port 2: >> State: Active >> Physical state: LinkUp >> Rate: 10 >> Base lid: 18 >> LMC: 0 >> SM lid: 18 >> Capability mask: 0x0251086a >> Port GUID: 0x0003ba000100fc06 >> >> ...and specify the unique subnet prefixes in each SM instance's >> config file >> like this: >> >> # grep prefix /etc/ofed/opensm-ib0.conf >> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" >> >> # grep prefix /etc/ofed/opensm-ib1.conf >> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000020" >> >> >> Which results in these instances running: >> >> # ps -ef|grep opensm >> root 1498 1 0 05:16 ? 00:00:00 /usr/sbin/opensm -B >> -maxsmps >> 0 -f /var/log/opensm-ib0.log -p 15 -g 0x0003ba000100fc05 >> subnet_prefix >> 0xfe80000000000010 >> root 2450 1 0 05:37 ? 00:00:00 /usr/sbin/opensm -B >> -maxsmps >> 0 -f /var/log/opensm-ib1.log -p 15 -g 0x0003ba000100fc06 >> subnet_prefix >> 0xfe80000000000020 > > I'm unaware of being able to set the subnet prefix as you indicate. It > needs to be set in the opensm config file. That's what I thought I was doing. The only reference that I find for this option is on the openMPI FAQ, there is no default or uncommented value in opensm.conf Do you happen to know what the correct value is, I have : OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" ..and there isn't an uncommented option already there to learn from. > > > You can double check the subnet prefix on the two ports by using > smpquery portinfo if you have infiniband diags installed. Thanks for that clue...it reveals that what I am doing is not affecting anything, yet opensm hapily runs with this as an option. # smpquery portinfo 1 1 # Port info: Lid 1 port 1 Mkey:............................0x0000000000000000 GidPrefix:.......................0xfe80000000000000 > > > -- Hal > >> Yet we are seeing complaints from openMPI as such(single rail works >> fine): >> >> $ mpirun -np 2 -machinefile ~/myduals ./IMB-MPI1 >> -------------------------------------------------------------------------- >> WARNING: There are more than one active ports on host 'n0051', but >> the >> default subnet GID prefix was detected on more than one of these >> ports. If these ports are connected to different physical IB >> networks, this configuration will fail in Open MPI. This version of >> Open MPI requires that every physically separate IB subnet that is >> used between connected MPI processes must have different subnet ID >> values. >> >> Please see this FAQ entry for more details: >> >> http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid >> >> NOTE: You can turn off this warning by setting the MCA parameter >> btl_openib_warn_default_gid_prefix to 0. >> -------------------------------------------------------------------------- >> [[1372,1],0][btl_openib_component.c:2827:handle_wc] from n0051 to: >> n0052 >> error polling LP CQ with status RETRY EXCEEDED ERROR status number >> 12 for >> wr_id 197710080 opcode 0 qp_idx 0 >> -------------------------------------------------------------------------- >> The InfiniBand retry count between two MPI processes has been >> exceeded. "Retry count" is defined in the InfiniBand spec 1.2 >> (section 12.7.38): >> >> The total number of times that the sender wishes the receiver to >> retry timeout, packet sequence, etc. errors before posting a >> completion error. >> >> This error typically means that there is something awry within the >> InfiniBand fabric itself. You should note the hosts on which this >> error has occurred; it has been observed that rebooting or removing a >> particular host from the job can sometimes resolve this issue. >> Two MCA parameters can be used to control Open MPI's behavior with >> respect to the retry count: >> >> * btl_openib_ib_retry_count - The number of times the sender will >> attempt to retry (defaulted to 7, the maximum value). >> * btl_openib_ib_timeout - The local ACK timeout parameter (defaulted >> to 10). The actual timeout value used is calculated as: >> >> 4.096 microseconds * (2^btl_openib_ib_timeout) >> >> See the InfiniBand spec 1.2 (section 12.7.34) for more details. >> >> Below is some information about the host that raised the error and >> the >> peer to which it was connected: >> >> Local host: n0051 >> Local device: mlx4_0 >> Peer host: n0052 >> >> You may need to consult with your system administrator to get this >> problem fixed. >> -------------------------------------------------------------------------- >> -------------------------------------------------------------------------- >> mpirun has exited due to process rank 0 with PID 9242 on >> node n0051 exiting without calling "finalize". This may >> have caused other processes in the application to be >> terminated by signals sent by mpirun (as reported here). >> -------------------------------------------------------------------------- >> [admin1:03239] 1 more process has sent help message help-mpi-btl- >> openib.txt >> / default subnet prefix >> [admin1:03239] Set MCA parameter "orte_base_help_aggregate" to 0 to >> see all >> help / error messages >> >> >> >> >> >> >> >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> From hal.rosenstock at gmail.com Wed Jun 3 14:42:40 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 3 Jun 2009 17:42:40 -0400 Subject: [ofa-general] multi-rail subnet_prefix and MPI issue In-Reply-To: References: <3E42C826-5D0E-4A87-BBC3-5E2A2ACA539A@sun.com> Message-ID: On Wed, Jun 3, 2009 at 5:35 PM, Frank Leers wrote: > Thanks for the reply Hal, > > > On Jun 3, 2009, at 1:52 PM, Hal Rosenstock wrote: > >> On Wed, Jun 3, 2009 at 4:10 PM, Frank Leers wrote: >>> >>> Hi, >>> >>> I have a question regarding unique subnet prefixes for a 2-rail QDR >>> configuration.  I'm using connectX on CentOS 5.3 with the distro-supplied >>> OFED 1.3.2, kernel 2.6.18-128.el5, opensm-3.2.2-3.el5. >>> >>> Any insight appreciated. >>> >>> -frank >>> >>> >>> >>> >>> # ibstat >>> CA 'mlx4_0' >>>       CA type: MT26428 >>>       Number of ports: 2 >>>       Firmware version: 2.6.0 >>>       Hardware version: a0 >>>       Node GUID: 0x0003ba000100fc04 >>>       System image GUID: 0x0003ba000100fc07 >>>       Port 1: >>>               State: Active >>>               Physical state: LinkUp >>>               Rate: 40 >>>               Base lid: 1 >>>               LMC: 0 >>>               SM lid: 1 >>>               Capability mask: 0x0251086a >>>               Port GUID: 0x0003ba000100fc05 >>>       Port 2: >>>               State: Active >>>               Physical state: LinkUp >>>               Rate: 10 >>>               Base lid: 18 >>>               LMC: 0 >>>               SM lid: 18 >>>               Capability mask: 0x0251086a >>>               Port GUID: 0x0003ba000100fc06 >>> >>> ...and specify the unique subnet prefixes in each SM instance's config >>> file >>> like this: >>> >>> # grep prefix /etc/ofed/opensm-ib0.conf >>> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" >>> >>> # grep prefix /etc/ofed/opensm-ib1.conf >>> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000020" >>> >>> >>> Which results in these instances running: >>> >>> # ps -ef|grep opensm >>> root      1498     1  0 05:16 ?        00:00:00 /usr/sbin/opensm -B >>> -maxsmps >>> 0 -f /var/log/opensm-ib0.log -p 15 -g 0x0003ba000100fc05 subnet_prefix >>> 0xfe80000000000010 >>> root      2450     1  0 05:37 ?        00:00:00 /usr/sbin/opensm -B >>> -maxsmps >>> 0 -f /var/log/opensm-ib1.log -p 15 -g 0x0003ba000100fc06 subnet_prefix >>> 0xfe80000000000020 >> >> I'm unaware of being able to set the subnet prefix as you indicate. It >> needs to be set in the opensm config file. > > That's what I thought I was doing.  The only reference that I find for this > option is on the openMPI FAQ, there is no default or uncommented value in > opensm.conf > > Do you happen to know what the correct value is, I have : > > OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" > > ..and there isn't an uncommented option already there to learn from. Remove those subnet_prefix es there. Look at opensm man page. Generate a config file using opensm -c and then copy this to two different config files opensm.config.ib0 and ib1. Edit those files for the appropriate subnet prefixes and then add --config /opensm.config.ib0/1 depending on the port. If it's done correctly, you should see the results in smpquery portinfo and not see that error message from MPI. -- Hal > >> >> >> You can double check the subnet prefix on the two ports by using >> smpquery portinfo if you have infiniband diags installed. > > Thanks for that clue...it reveals that what I am doing is not affecting > anything, yet opensm hapily runs with this as an option. > > > # smpquery portinfo 1 1 > # Port info: Lid 1 port 1 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0xfe80000000000000 > > > > > >> >> >> -- Hal >> >>> Yet we are seeing complaints from openMPI as such(single rail works >>> fine): >>> >>> $ mpirun -np 2 -machinefile ~/myduals ./IMB-MPI1 >>> >>> -------------------------------------------------------------------------- >>> WARNING: There are more than one active ports on host 'n0051', but the >>> default subnet GID prefix was detected on more than one of these >>> ports.  If these ports are connected to different physical IB >>> networks, this configuration will fail in Open MPI.  This version of >>> Open MPI requires that every physically separate IB subnet that is >>> used between connected MPI processes must have different subnet ID >>> values. >>> >>> Please see this FAQ entry for more details: >>> >>> http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid >>> >>> NOTE: You can turn off this warning by setting the MCA parameter >>>   btl_openib_warn_default_gid_prefix to 0. >>> >>> -------------------------------------------------------------------------- >>> [[1372,1],0][btl_openib_component.c:2827:handle_wc] from n0051 to: n0052 >>> error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for >>> wr_id 197710080 opcode 0 qp_idx 0 >>> >>> -------------------------------------------------------------------------- >>> The InfiniBand retry count between two MPI processes has been >>> exceeded.  "Retry count" is defined in the InfiniBand spec 1.2 >>> (section 12.7.38): >>> >>>  The total number of times that the sender wishes the receiver to >>>  retry timeout, packet sequence, etc. errors before posting a >>>  completion error. >>> >>> This error typically means that there is something awry within the >>> InfiniBand fabric itself.  You should note the hosts on which this >>> error has occurred; it has been observed that rebooting or removing a >>> particular host from the job can sometimes resolve this issue. >>> Two MCA parameters can be used to control Open MPI's behavior with >>> respect to the retry count: >>> >>> * btl_openib_ib_retry_count - The number of times the sender will >>> attempt to retry (defaulted to 7, the maximum value). >>> * btl_openib_ib_timeout - The local ACK timeout parameter (defaulted >>> to 10).  The actual timeout value used is calculated as: >>> >>>  4.096 microseconds * (2^btl_openib_ib_timeout) >>> >>> See the InfiniBand spec 1.2 (section 12.7.34) for more details. >>> >>> Below is some information about the host that raised the error and the >>> peer to which it was connected: >>> >>> Local host:   n0051 >>> Local device: mlx4_0 >>> Peer host:    n0052 >>> >>> You may need to consult with your system administrator to get this >>> problem fixed. >>> >>> -------------------------------------------------------------------------- >>> >>> -------------------------------------------------------------------------- >>> mpirun has exited due to process rank 0 with PID 9242 on >>> node n0051 exiting without calling "finalize". This may >>> have caused other processes in the application to be >>> terminated by signals sent by mpirun (as reported here). >>> >>> -------------------------------------------------------------------------- >>> [admin1:03239] 1 more process has sent help message >>> help-mpi-btl-openib.txt >>> / default subnet prefix >>> [admin1:03239] Set MCA parameter "orte_base_help_aggregate" to 0 to see >>> all >>> help / error messages >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general >>> > > From Frank.Leers at Sun.COM Wed Jun 3 17:59:40 2009 From: Frank.Leers at Sun.COM (Frank Leers) Date: Wed, 03 Jun 2009 17:59:40 -0700 Subject: [ofa-general] multi-rail subnet_prefix and MPI issue In-Reply-To: References: <3E42C826-5D0E-4A87-BBC3-5E2A2ACA539A@sun.com> Message-ID: <5D5C1B14-A542-47E1-AAB5-9C53DC98797E@Sun.COM> On Jun 3, 2009, at 2:42 PM, Hal Rosenstock wrote: > On Wed, Jun 3, 2009 at 5:35 PM, Frank Leers > wrote: >> Thanks for the reply Hal, >> >> >> On Jun 3, 2009, at 1:52 PM, Hal Rosenstock wrote: >> >>> On Wed, Jun 3, 2009 at 4:10 PM, Frank Leers >>> wrote: >>>> >>>> Hi, >>>> >>>> I have a question regarding unique subnet prefixes for a 2-rail QDR >>>> configuration. I'm using connectX on CentOS 5.3 with the distro- >>>> supplied >>>> OFED 1.3.2, kernel 2.6.18-128.el5, opensm-3.2.2-3.el5. >>>> >>>> Any insight appreciated. >>>> >>>> -frank >>>> >>>> >>>> >>>> >>>> # ibstat >>>> CA 'mlx4_0' >>>> CA type: MT26428 >>>> Number of ports: 2 >>>> Firmware version: 2.6.0 >>>> Hardware version: a0 >>>> Node GUID: 0x0003ba000100fc04 >>>> System image GUID: 0x0003ba000100fc07 >>>> Port 1: >>>> State: Active >>>> Physical state: LinkUp >>>> Rate: 40 >>>> Base lid: 1 >>>> LMC: 0 >>>> SM lid: 1 >>>> Capability mask: 0x0251086a >>>> Port GUID: 0x0003ba000100fc05 >>>> Port 2: >>>> State: Active >>>> Physical state: LinkUp >>>> Rate: 10 >>>> Base lid: 18 >>>> LMC: 0 >>>> SM lid: 18 >>>> Capability mask: 0x0251086a >>>> Port GUID: 0x0003ba000100fc06 >>>> >>>> ...and specify the unique subnet prefixes in each SM instance's >>>> config >>>> file >>>> like this: >>>> >>>> # grep prefix /etc/ofed/opensm-ib0.conf >>>> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" >>>> >>>> # grep prefix /etc/ofed/opensm-ib1.conf >>>> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000020" >>>> >>>> >>>> Which results in these instances running: >>>> >>>> # ps -ef|grep opensm >>>> root 1498 1 0 05:16 ? 00:00:00 /usr/sbin/opensm -B >>>> -maxsmps >>>> 0 -f /var/log/opensm-ib0.log -p 15 -g 0x0003ba000100fc05 >>>> subnet_prefix >>>> 0xfe80000000000010 >>>> root 2450 1 0 05:37 ? 00:00:00 /usr/sbin/opensm -B >>>> -maxsmps >>>> 0 -f /var/log/opensm-ib1.log -p 15 -g 0x0003ba000100fc06 >>>> subnet_prefix >>>> 0xfe80000000000020 >>> >>> I'm unaware of being able to set the subnet prefix as you >>> indicate. It >>> needs to be set in the opensm config file. >> >> That's what I thought I was doing. The only reference that I find >> for this >> option is on the openMPI FAQ, there is no default or uncommented >> value in >> opensm.conf >> >> Do you happen to know what the correct value is, I have : >> >> OPTIONS="$OPTIONS subnet_prefix 0xfe80000000000010" >> >> ..and there isn't an uncommented option already there to learn from. > > Remove those subnet_prefix es there. > > Look at opensm man page. Generate a config file using opensm -c > and then copy this to two different config files opensm.config.ib0 > and ib1. > Edit those files for the appropriate subnet prefixes and then add > --config /opensm.config.ib0/1 depending > on the port. > > If it's done correctly, you should see the results in smpquery > portinfo and not see that error message from MPI. This did get me past the MPI issue, thanks! > > > -- Hal >> >>> >>> >>> You can double check the subnet prefix on the two ports by using >>> smpquery portinfo if you have infiniband diags installed. >> >> Thanks for that clue...it reveals that what I am doing is not >> affecting >> anything, yet opensm hapily runs with this as an option. >> >> >> # smpquery portinfo 1 1 >> # Port info: Lid 1 port 1 >> Mkey:............................0x0000000000000000 >> GidPrefix:.......................0xfe80000000000000 >> >> >> >> >> >>> >>> >>> -- Hal >>> >>>> Yet we are seeing complaints from openMPI as such(single rail works >>>> fine): >>>> >>>> $ mpirun -np 2 -machinefile ~/myduals ./IMB-MPI1 >>>> >>>> -------------------------------------------------------------------------- >>>> WARNING: There are more than one active ports on host 'n0051', >>>> but the >>>> default subnet GID prefix was detected on more than one of these >>>> ports. If these ports are connected to different physical IB >>>> networks, this configuration will fail in Open MPI. This version >>>> of >>>> Open MPI requires that every physically separate IB subnet that is >>>> used between connected MPI processes must have different subnet ID >>>> values. >>>> >>>> Please see this FAQ entry for more details: >>>> >>>> http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid >>>> >>>> NOTE: You can turn off this warning by setting the MCA parameter >>>> btl_openib_warn_default_gid_prefix to 0. >>>> >>>> -------------------------------------------------------------------------- >>>> [[1372,1],0][btl_openib_component.c:2827:handle_wc] from n0051 >>>> to: n0052 >>>> error polling LP CQ with status RETRY EXCEEDED ERROR status >>>> number 12 for >>>> wr_id 197710080 opcode 0 qp_idx 0 >>>> >>>> -------------------------------------------------------------------------- >>>> The InfiniBand retry count between two MPI processes has been >>>> exceeded. "Retry count" is defined in the InfiniBand spec 1.2 >>>> (section 12.7.38): >>>> >>>> The total number of times that the sender wishes the receiver to >>>> retry timeout, packet sequence, etc. errors before posting a >>>> completion error. >>>> >>>> This error typically means that there is something awry within the >>>> InfiniBand fabric itself. You should note the hosts on which this >>>> error has occurred; it has been observed that rebooting or >>>> removing a >>>> particular host from the job can sometimes resolve this issue. >>>> Two MCA parameters can be used to control Open MPI's behavior with >>>> respect to the retry count: >>>> >>>> * btl_openib_ib_retry_count - The number of times the sender will >>>> attempt to retry (defaulted to 7, the maximum value). >>>> * btl_openib_ib_timeout - The local ACK timeout parameter >>>> (defaulted >>>> to 10). The actual timeout value used is calculated as: >>>> >>>> 4.096 microseconds * (2^btl_openib_ib_timeout) >>>> >>>> See the InfiniBand spec 1.2 (section 12.7.34) for more details. >>>> >>>> Below is some information about the host that raised the error >>>> and the >>>> peer to which it was connected: >>>> >>>> Local host: n0051 >>>> Local device: mlx4_0 >>>> Peer host: n0052 >>>> >>>> You may need to consult with your system administrator to get this >>>> problem fixed. >>>> >>>> -------------------------------------------------------------------------- >>>> >>>> -------------------------------------------------------------------------- >>>> mpirun has exited due to process rank 0 with PID 9242 on >>>> node n0051 exiting without calling "finalize". This may >>>> have caused other processes in the application to be >>>> terminated by signals sent by mpirun (as reported here). >>>> >>>> -------------------------------------------------------------------------- >>>> [admin1:03239] 1 more process has sent help message >>>> help-mpi-btl-openib.txt >>>> / default subnet prefix >>>> [admin1:03239] Set MCA parameter "orte_base_help_aggregate" to 0 >>>> to see >>>> all >>>> help / error messages >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >> >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From anatolyg at voltaire.com Wed Jun 3 22:40:08 2009 From: anatolyg at voltaire.com (Anatoly Greenblatt) Date: Thu, 4 Jun 2009 08:40:08 +0300 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A257DBF.4000401@mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> Message-ID: <39C75744D164D948A170E9792AF8E7CA017F078A@exil.voltaire.com> Hi, I vote to remove. This is our Open MPI team leader answer: I think it should be taken out from the following reasons: ompi related build errors or bugs can affect OFED release dates many customers prefer their own mpi packages almost all linux distros contain pre-compiled mpi rpms which can be installed with yum/apt-get/... having MPI as part of OFED comlicates its release process and scripts. Regards, Anatoly. -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Tziporet Koren Sent: Tuesday, June 02, 2009 22:30 To: EWG; OpenFabrics General Subject: [ewg] RFC: Do we wish to take MPI out of OFED? Hello, In the EWG meeting yesterday there was a suggestion to take out all MPI packages including MPI tests out from OFED. Here are some of the reasons for keeping MPI in or out of OFED. Main reasons to take MPI out of OFED: - MPI is not developed under OFA - Need to synchronize between different projects - Some customers prefer to install a different MPI version then the one in OFED Main reasons to keep MPI in OFED: - All participants test with the same MPI versions, and when installing OFED it is ensured that MPI will work fine with this version. - Customers convenience in install (no need to go to more sites to get MPI) - MPI is an important RDMA ULP and although it is not developed in OFA it is widely used by OFED customers Before we decide to do such a change I would like to get the more opinions from the comunity. Thanks, Tziporet _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From Bob.Ciotti at nasa.gov Wed Jun 3 23:23:14 2009 From: Bob.Ciotti at nasa.gov (Bob Ciotti) Date: Wed, 3 Jun 2009 23:23:14 -0700 Subject: [ofa-general] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <74D67503-94FD-4D02-B986-65ADC899740E@cisco.com> References: <4A257DBF.4000401@mellanox.co.il> <74D67503-94FD-4D02-B986-65ADC899740E@cisco.com> Message-ID: <20090604062314.GB54889@nas.nasa.gov> On Wed, Jun 03, 2009 at 07:24:23AM -0500, Jeff Squyres wrote: > I'll state right up front that I'm in favor of removing MPI from > OFED. :-) > > > On Jun 2, 2009, at 3:30 PM, Tziporet Koren wrote: > > > Here are some of the reasons for keeping MPI in or out of OFED. > > > > Main reasons to take MPI out of OFED: > > - MPI is not developed under OFA > > - Need to synchronize between different projects > > > --> This point alone is hugely difficult. OFED has been held up > multiple times because of MPI release delays. I would like to stress > that the open source MPI implementations are entirely different > software projects with minimal overlap of personnel between MPI and > OFED. Also, OFED is but one of the stacks that the MPI's support. On our systems we currently have 21 (just counted them) different versions of of MPI installed. And we are thinking about adding OpenMPI into the mix. In addition, there are 24 different compilers, 29 scientific libraries, 8 tool sets, 5 debuggers and 4 vis tools. This, all in addition to what comes in the SLES release. Point is - adding another MPI module is orthogonal to updating OFED. I don't see a good reason to have an artificial release dependency. Any serious regression test for OFED needs to have been tested against much more than OpenMPI. Its far more important to improve the OFED release and not break existing code/ULP than to get another MPI released at the same time. The only possible counter I can see is if OpenMPI is better able to drive feature enhancement/development in OFED than say mvapich. DK? In my opinion, anything that slows the OFED release is a bad thing. Of course, that does not mean we are willing to do more ;) debugging in the field. bob ------------------------------------------------------------------------- Robert B. Ciotti Supercomputing Systems Lead NASA Advanced Supercomputing (NAS) Division TEL (650) 604-4408 NASA Ames Research Center FAX (650) 604-4377 Moffett Field, CA 94035-1000 Bob.Ciotti at NASA.gov ------------------------------------------------------------------------- From ogerlitz at Voltaire.com Wed Jun 3 23:54:21 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Thu, 04 Jun 2009 09:54:21 +0300 Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: References: <4A266ABB.6000603@Voltaire.com> Message-ID: <4A276F9D.6080504@Voltaire.com> Roland Dreier wrote: >> arps are a different thing from regular payloads. They are not handled >> using socket semantics. > Oh right, I misunderstood the way the net core is using queue_len. I assume we are talking on this net/core/neighbour.c :: _neigh_event_send() code piece 923 if (neigh->nud_state == NUD_INCOMPLETE) { 924 if (skb) { 925 if (skb_queue_len(&neigh->arp_queue) >= 926 neigh->parms->queue_len) { 927 struct sk_buff *buff; 928 buff = __skb_dequeue(&neigh->arp_queue); 929 kfree_skb(buff); 930 NEIGH_CACHE_STAT_INC(neigh->tbl, unres_discards); 931 } 932 __skb_queue_tail(&neigh->arp_queue, skb); 933 } 934 rc = 1; 935 } my understanding is that the packet to be queued is not the ARP but rather what the socket / user attempt to send which can't be served by the kernel before the resolution is known. So in that respect, since with IPoIB two lookups are needed - 1st, resolve from L3 IP address to L3 IPoIB address and 2nd, resolve from L3 to L2 IB address (GID --> LID, MGID --> MLID). Following that, maybe we could integrate somehow with that sysctl param, such that the user would be able to control how much packets would be queued before the IB L2 resolution is done. As was stated here, with the two patches, the user still has control through the socket send buffer, which sounds fair-enough to me. > OK, I think your patch makes sense and I'll add Or's too. thanks Or. From vlad at lists.openfabrics.org Thu Jun 4 03:22:14 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 4 Jun 2009 03:22:14 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090604-0200 daily build status Message-ID: <20090604102215.1AD0BE61573@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From dorfman.eli at gmail.com Thu Jun 4 05:30:34 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 04 Jun 2009 15:30:34 +0300 Subject: [ofa-general] [PATCH] opensm/osm_sa.c add attribute and component mask to error message Message-ID: <4A27BE6A.9090101@gmail.com> add attribute and component mask to error message about too many records for SubnAdmGet query Signed-off-by: Eli Dorfman --- opensm/opensm/osm_sa.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index 3521132..20e70cd 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -422,8 +422,9 @@ void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, */ if (sa_mad->method == IB_MAD_METHOD_GET && num_rec > 1) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4C05: " - "Got more than one record for SubnAdmGet (%u)\n", - num_rec); + "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" PRIx64 "\n", + num_rec, ib_get_sa_attr_str(sa_mad->attr_id), + cl_ntoh64(sa_mad->comp_mask)); osm_sa_send_error(sa, madw, IB_SA_MAD_STATUS_TOO_MANY_RECORDS); goto Exit; } -- 1.5.5 From jackm at dev.mellanox.co.il Thu Jun 4 07:40:10 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Thu, 4 Jun 2009 17:40:10 +0300 Subject: [ofa-general] [PATCH] mlx4: fix post send of local invalidate and fast registration packets. Message-ID: <200906041740.10409.jackm@dev.mellanox.co.il> The ConnectX PRM states that the "SO" bit must be set when posting Fast Reg and Local Invalidate send packets. When this bit is set, the current WQE will be executed only after all previous WQEs on this QP have been executed. (If the bit is not set, Fast Reg and Local Invalidate WQEs may begin execution too early). Signed-off-by: Jack Morgenstein --- Roland, Please try to get this patch into 2.6.30 -- it is an important fix for nfsrdma. Thanks! diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 20724ae..c4a0264 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -1585,12 +1585,16 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case IB_WR_LOCAL_INV: + ctrl->srcrb_flags |= + cpu_to_be32(MLX4_WQE_CTRL_STRONG_ORDER); set_local_inv_seg(wqe, wr->ex.invalidate_rkey); wqe += sizeof (struct mlx4_wqe_local_inval_seg); size += sizeof (struct mlx4_wqe_local_inval_seg) / 16; break; case IB_WR_FAST_REG_MR: + ctrl->srcrb_flags |= + cpu_to_be32(MLX4_WQE_CTRL_STRONG_ORDER); set_fmr_seg(wqe, wr); wqe += sizeof (struct mlx4_wqe_fmr_seg); size += sizeof (struct mlx4_wqe_fmr_seg) / 16; diff --git a/include/linux/mlx4/qp.h b/include/linux/mlx4/qp.h index bf8f119..9f29d86 100644 --- a/include/linux/mlx4/qp.h +++ b/include/linux/mlx4/qp.h @@ -165,6 +165,7 @@ enum { MLX4_WQE_CTRL_IP_CSUM = 1 << 4, MLX4_WQE_CTRL_TCP_UDP_CSUM = 1 << 5, MLX4_WQE_CTRL_INS_VLAN = 1 << 6, + MLX4_WQE_CTRL_STRONG_ORDER = 1 << 7, }; struct mlx4_wqe_ctrl_seg { From dledford at redhat.com Thu Jun 4 07:43:05 2009 From: dledford at redhat.com (Doug Ledford) Date: Thu, 04 Jun 2009 10:43:05 -0400 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <20090603171434.GR3261@cefeid.wcss.wroc.pl> References: <4A257DBF.4000401@mellanox.co.il> <20090603073949.GC3261@cefeid.wcss.wroc.pl> <20090603171434.GR3261@cefeid.wcss.wroc.pl> Message-ID: <1244126585.19973.9.camel@firewall.xsintricity.com> On Wed, 2009-06-03 at 19:14 +0200, Pawel Dziekonski wrote: > On Wed, 03 Jun 2009 at 10:14:07AM -0400, Doug Ledford wrote: > > On Jun 3, 2009, at 3:39 AM, Pawel Dziekonski wrote: > >> On Tue, 02 Jun 2009 at 10:30:07PM +0300, Tziporet Koren wrote: > >> > >>> Main reasons to keep MPI in OFED: > >>> - All participants test with the same MPI versions, and when installing > >>> OFED it is ensured that MPI will work fine with this version. > >>> - Customers convenience in install (no need to go to more sites to get > >>> MPI) > >>> - MPI is an important RDMA ULP and although it is not developed in OFA it > >>> is widely used by OFED customers > >> > >> As a customer I strongly support above mentioned pros. It's a > >> guarantee for us that MPI is well tested with OFED release. > > > > MPI makes an effective test bed for the RDMA stack whether it is shipped > > with it or not. Removing the MPIs from the distribution would not, in all > > likelihood, change the fact that MPIs would be used to test the RDMA stack > > prior to release. > > > >> I believe that this effort saves a lot of troubles that would be > >> raised from separate releases of MPI and OFED distros. > > > > If you truly believe this, and you accept that shipping the MPI with > > the RDMA stack is an acceptable solution to the problem, then you > > are encouraging totally craptacular engineering as a customer. > > Since you have a non-US email address, and since craptacular is a > > word I use frequently, but which I also sort of just made up, let me > > define that. Sometimes, things are good. When they are really good, > > they are spectacular. Sometimes, things are crap. When they are > > *really* crap, they are craptacular. > > > > The RDMA stack provides an API. The MPI stacks are nothing more than > > you look at the problem from vendor point of view (vendor-like mail > domain? ;). you care about api. i care about mpi. I care about both. I care about the fact that a solid, well adhered to API makes for lots of happy MPI campers, not just one happy MPI camper. And the API road is the path to long term interoperability, not just short term. So if you really care about MPI, I would recommend you look to the long term, and you may find you agree with me then. > from technical point of view it is enough for me if you say in ofed > docs that THIS and THAT particular versions of MPI was tested and > WORKS. just like suppoerted and tested list of linux kernels and linux > distros. I definitely can download and compile mpi by myself. however > since you already used some versions of MPI to test the RDMA stack > prior to release why not simply attach it to release? it just makes > the release more complete from CUSTOMER point of view. For the cons that you stripped from the original email: it throws off one or the other's release schedule, it delays things, etc. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From panda at cse.ohio-state.edu Thu Jun 4 11:02:11 2009 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Thu, 4 Jun 2009 14:02:11 -0400 (EDT) Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A257DBF.4000401@mellanox.co.il> Message-ID: Tziporet, >Main reasons to keep MPI in OFED: >- All participants test with the same MPI versions, and when installing >OFED it is ensured that MPI will work fine with this version. >- Customers convenience in install (no need to go to more sites to get MPI) >- MPI is an important RDMA ULP and although it is not developed in OFA >it is widely used by OFED customers I support keeping MPI packages in the OFED because of the above positive points you have mentioned. I would also like to mention that keeping MPI packages in OFED helps to test out various new features and functionalities (such as APM and XRC in the past and the new memory registration scheme being discussed now) as they get introduced. Such an integrated approach helps to test out these features at the lower layers as well as at the MPI layer. This process helps to resolve out any bugs with the new features during the testing process itself. It also accelerates the deployment and use of these new features in the community. However, to make the complete OFED release process work smoothly for everybody (vendors, distros, users, etc.) without affecting the release schedule, it is essential that stable MPI packages are added to OFED. This is what we have been doing wrt MVAPICH and MVAPICH2 for the last several years. If the developers of any MPI package do not want it to be a part of the OFED due to any constraints, it should be allowed. However, such an action should not force to remove all MPI packages. >From the point of view of MVAPICH and MVAPICH2 packages in OFED, we have been providing stable packages to OFED for the last several years helping the OFED community and would like to continue with this process. Thanks, DK From weiny2 at llnl.gov Thu Jun 4 11:09:51 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 4 Jun 2009 11:09:51 -0700 Subject: [ofa-general] [PATCH management] Move _with_node_name_map rpm spec option from infiniband-diags to opensm. Message-ID: <20090604110951.3fe59f93.weiny2@llnl.gov> From: Ira Weiny Date: Thu, 4 Jun 2009 11:08:27 -0700 Subject: [PATCH] Move _with_node_name_map rpm spec option from infiniband-diags to opensm. The node name map is now part of libosmcomp. Signed-off-by: Ira Weiny --- infiniband-diags/infiniband-diags.spec.in | 6 +----- opensm/opensm.spec.in | 7 ++++++- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/infiniband-diags/infiniband-diags.spec.in b/infiniband-diags/infiniband-diags.spec.in index 07c46c9..c50ce09 100644 --- a/infiniband-diags/infiniband-diags.spec.in +++ b/infiniband-diags/infiniband-diags.spec.in @@ -22,12 +22,8 @@ diagnose an IB subnet. %prep %setup -q -%if %{?_with_node_name_map:1}%{!?_with_node_name_map:0} -%define _enable_node_name_map --with-node-name-map%{?_with_node_name_map} -%endif - %build -%configure %{?_enable_node_name_map} +%configure make %install diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in index 10e0147..c541804 100644 --- a/opensm/opensm.spec.in +++ b/opensm/opensm.spec.in @@ -21,6 +21,10 @@ %define _disable_event_plugin --disable-default-event-plugin %endif +%if %{?_with_node_name_map:1}%{!?_with_node_name_map:0} +%define _enable_node_name_map --with-node-name-map%{?_with_node_name_map} +%endif + Summary: InfiniBand subnet manager and administration Name: opensm Version: @VERSION@ @@ -80,7 +84,8 @@ Static version of the opensm libraries %{?_enable_perf_mgr} \ %{?_disable_perf_mgr} \ %{?_enable_event_plugin} \ - %{?_disable_event_plugin} + %{?_disable_event_plugin} \ + %{?_enable_node_name_map} make %{?_smp_mflags} %install -- 1.5.4.5 From arlin.r.davis at intel.com Thu Jun 4 11:22:44 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Thu, 4 Jun 2009 11:22:44 -0700 Subject: [ofa-general] RE: [PATCH] dapl/windows cma provider: add support for network devices based on index In-Reply-To: <51EAC421351B42368735BDD2F53F8E7D@amr.corp.intel.com> References: <51EAC421351B42368735BDD2F53F8E7D@amr.corp.intel.com> Message-ID: thanks, applied. >Subject: [PATCH] dapl/windows cma provider: add support for >network devices based on index > >The linux cma provider provides support for named network devices, such >as 'ib0' or 'eth0'. This allows the same dapl configuration file to >be used easily across a cluster. > >To allow similar support on Windows, allow users to specify the device >name 'rdma_devN' in the dapl.conf file. The given index, N, >is map to a >corresponding IP address that is associated with an RDMA device. > >Signed-off-by: Sean Hefty From sean.hefty at intel.com Thu Jun 4 16:14:02 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 4 Jun 2009 16:14:02 -0700 Subject: [ofa-general] skipping QP states during transitions Message-ID: Does anyone know if the HCAs are capable of transitioning directly from reset to RTS using a single command? From rdreier at cisco.com Thu Jun 4 16:47:24 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 04 Jun 2009 16:47:24 -0700 Subject: [ofa-general] Re: [PATCH] mlx4: fix post send of local invalidate and fast registration packets. In-Reply-To: <200906041740.10409.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Thu, 4 Jun 2009 17:40:10 +0300") References: <200906041740.10409.jackm@dev.mellanox.co.il> Message-ID: > Please try to get this patch into 2.6.30 -- it is an important fix for nfsrdma. Would be easier to get it in if you had a pointer to the NFS/RDMA bug report. Not sure why you think this info isn't worth including in the changelog.... - R. From jackm at dev.mellanox.co.il Fri Jun 5 00:34:30 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Fri, 5 Jun 2009 10:34:30 +0300 Subject: [ofa-general] Re: [PATCH] mlx4: fix post send of local invalidate and fast registration packets. In-Reply-To: References: <200906041740.10409.jackm@dev.mellanox.co.il> Message-ID: <200906051034.30357.jackm@dev.mellanox.co.il> On Friday 05 June 2009 02:47, Roland Dreier wrote: > > Please try to get this patch into 2.6.30 -- it is an important fix for nfsrdma. > > Would be easier to get it in if you had a pointer to the NFS/RDMA bug > report. Not sure why you think this info isn't worth including in the > changelog.... > > - R. > I don't think there is an NFS/RDMA bug open yet. However, there is a thread in the OFA general list archive regarding this problem (thread called "Can't mount NFS/RDMA partition"). Link to message towards end of thread (with very specific problem description): http://lists.openfabrics.org/pipermail/general/2009-April/059253.html This patch fixes the problem described in the thread. -Jack From titoff.a at gmail.com Fri Jun 5 01:06:53 2009 From: titoff.a at gmail.com (Titov Alexander) Date: Fri, 5 Jun 2009 12:06:53 +0400 Subject: [ofa-general] IPoIB in Xen guest machine Message-ID: Hello, I work in VDS hosting provider and we want increase our bandwidth with Infiniband. I make Infiniband test stand with two machines with Mellanox MT26428, switch MTS3600 and SLES11 with Xen. How can I connect ip network over IB to Xen guest machine? I see two ways: 1) I use Qlogic VNIC and bridge virtual xen interface with VNIC ethernet. It isn't true way, because data transfers with great overhead ( Maybe I mistake?). I try this way and I receve error bad MAD status (110) from lid 0 from command ib_qlgc_vnic_query. I try it with OFED and SLES11, Qlogic OFED+ and SLES10SP2, Mellanox OFED and SLES10SP2. I get this error for all configurations. 2) I use IPoIB with Ethernet IB bridge (gateway). But I found about it only few phrases on openfabrics site. (for example http://www.openfabrics.org/OFAReport/article4.htm ). Where can I find this code? Can you suggest me best solution? -- Best regards, Titov Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at lists.openfabrics.org Fri Jun 5 03:22:08 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 5 Jun 2009 03:22:08 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090605-0200 daily build status Message-ID: <20090605102208.F10D5E611CF@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Fri Jun 5 04:11:11 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 5 Jun 2009 07:11:11 -0400 Subject: [ofa-general] IPoIB in Xen guest machine In-Reply-To: References: Message-ID: On Fri, Jun 5, 2009 at 4:06 AM, Titov Alexander wrote: > Hello, > I work in VDS hosting provider and we want increase our bandwidth with > Infiniband. I make Infiniband test stand with two machines with > Mellanox MT26428, switch MTS3600 and SLES11 with Xen. > How can I connect ip network over IB to Xen guest machine? > I see two ways: > 1) I use Qlogic VNIC and bridge virtual xen interface with VNIC ethernet. It > isn't true way, because data transfers with great overhead ( Maybe I > mistake?). > I try this way and I receve error  bad MAD status (110) from lid 0 from > command  ib_qlgc_vnic_query. I try it with OFED and SLES11, Qlogic OFED+ and > SLES10SP2, Mellanox OFED and SLES10SP2. I get this error for all > configurations. Is an SM running in the IB subnet ? > 2) I use IPoIB with Ethernet IB bridge (gateway). But I found about it only > few phrases on openfabrics site. (for > example http://www.openfabrics.org/OFAReport/article4.htm ). Where can I > find this code? > Can you suggest me best solution? I think there are a few general approaches for IP connectivity between IB and ethernet: 1. Gateways (Voltaire has one, QLogic may (I forget)) 2. VNIC approaches (QLogic, Xsigo, maybe others) 3. Use linux box as IP router with ethernet and IPoIB interfaces What's best depends on your specific requirements. -- Hal > -- > Best regards, > Titov Alexander > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From john.russo at qlogic.com Fri Jun 5 05:37:46 2009 From: john.russo at qlogic.com (John Russo) Date: Fri, 5 Jun 2009 07:37:46 -0500 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: <4A257DBF.4000401@mellanox.co.il> Message-ID: I agree with DK from OSU. There are clear advantages to having MPI included with OFED. Not only will it make testing of a complete solution easier by both OFED and MPI suppliers, but it will also improve ease of use for end users and ease of inclusion for linux distro suppliers. As DK points out there are continual improvements in MPIs which may depend on bug fixes and/or new features in newer versions of OFED. Identifying a known good combination will be important to most end users, etc. -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Dhabaleswar Panda Sent: Thursday, June 04, 2009 2:02 PM To: Tziporet Koren Cc: EWG; OpenFabrics General Subject: Re: [ewg] RFC: Do we wish to take MPI out of OFED? Tziporet, >Main reasons to keep MPI in OFED: >- All participants test with the same MPI versions, and when installing >OFED it is ensured that MPI will work fine with this version. >- Customers convenience in install (no need to go to more sites to get MPI) >- MPI is an important RDMA ULP and although it is not developed in OFA >it is widely used by OFED customers I support keeping MPI packages in the OFED because of the above positive points you have mentioned. I would also like to mention that keeping MPI packages in OFED helps to test out various new features and functionalities (such as APM and XRC in the past and the new memory registration scheme being discussed now) as they get introduced. Such an integrated approach helps to test out these features at the lower layers as well as at the MPI layer. This process helps to resolve out any bugs with the new features during the testing process itself. It also accelerates the deployment and use of these new features in the community. However, to make the complete OFED release process work smoothly for everybody (vendors, distros, users, etc.) without affecting the release schedule, it is essential that stable MPI packages are added to OFED. This is what we have been doing wrt MVAPICH and MVAPICH2 for the last several years. If the developers of any MPI package do not want it to be a part of the OFED due to any constraints, it should be allowed. However, such an action should not force to remove all MPI packages. >From the point of view of MVAPICH and MVAPICH2 packages in OFED, we have been providing stable packages to OFED for the last several years helping the OFED community and would like to continue with this process. Thanks, DK _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From dzieko at wcss.pl Fri Jun 5 05:40:33 2009 From: dzieko at wcss.pl (Pawel Dziekonski) Date: Fri, 5 Jun 2009 14:40:33 +0200 Subject: [ofa-general] ib_mthca Catastrophic errors Message-ID: <20090605124033.GK18273@cefeid.wcss.wroc.pl> Hi, from time to time I get Catastrophic errors like below. software stack is kernel 2.6.18-92.1.10.el5 with Lustre client. device and OFED info is also below. any hints? thanks in advance, Pawel 06:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) # ibv_devices device node GUID ------ ---------------- mthca0 0030487e07700000 # ibv_devinfo hca_id: mthca0 fw_ver: 1.2.0 node_guid: 0030:487e:0770:0000 sys_image_guid: 0030:487e:0770:0003 vendor_id: 0x02c9 vendor_part_id: 25204 hw_ver: 0xA0 board_id: SM_0000000003 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 1 port_lid: 441 port_lmc: 0x00 # ofed_info OFED-1.3.1 libibverbs: git://git.openfabrics.org/ofed_1_3/libibverbs.git ofed_1_3 commit 40b771aa6a9c0ad092b2e20775b4723d3b173792 libmthca: git://git.openfabrics.org/ofed_1_3/libmthca.git ofed_1_3 commit 9501e698d257949acfab2edc90812602966dbcc9 libmlx4: git://git.openfabrics.org/ofed_1_3/libmlx4.git ofed_1_3 commit 3869d6dab7e12fe452270ca641f7dd7082b42482 libehca: git://git.openfabrics.org/ofed_1_3/libehca.git ofed_1_3 commit fd898180cfa3b737f893f432a80b91bac3396325 libipathverbs: git://git.openfabrics.org/ofed_1_3/libipathverbs.git ofed_1_3 commit 82be4d81859d1fd2edf830220fe65a9923b80a46 libcxgb3: git://git.openfabrics.org/ofed_1_3/libcxgb3.git ofed_1_3 commit 6f7485feb244d8571fcab2292ef92c97bea48df0 libnes: git://git.openfabrics.org/ofed_1_3/libnes.git ofed_1_3 commit 471fa2e5a7bb2f8946119396358c31adcc6c2fb3 libibcm: git://git.openfabrics.org/ofed_1_3/libibcm.git ofed_1_3 commit 53ec35f544bbc1838bbadc2210909c25a954a5e2 librdmacm: git://git.openfabrics.org/ofed_1_3/librdmacm.git ofed_1_3 commit a0ef80a1e0d5debdae48a844fbc8d09aec5b24b1 dapl1: git://git.openfabrics.org/ofed_1_3/dapl1.git ofed_1_3 commit 7a9b58d6c50fc0a357de540ec3eb2ab2e07f8779 dapl2: git://git.openfabrics.org/ofed_1_3/dapl2.git ofed_1_3 commit 2583f07d9d0f55eee14e0b0e6074bc6fd0712177 libsdp: git://git.openfabrics.org/ofed_1_3/libsdp.git ofed_1_3 commit c8102dccc502930442b23de658674d386456b350 sdpnetstat: git://git.openfabrics.org/ofed_1_3/sdpnetstat.git ofed_1_3 commit 3341620a7259c4f7bdd4180864b98e260c3dc223 srptools: git://git.openfabrics.org/ofed_1_3/srptools.git ofed_1_3 commit e0ce2d42eeb25f8e89b8f6daaa32a630c9b64f0d perftest: git://git.openfabrics.org/ofed_1_3/perftest.git ofed_1_3 commit 6321b5468f7293088cc003809049c02b176130d8 qlvnictools: git://git.openfabrics.org/ofed_1_3/qlvnictools.git ofed_1_3 commit 086f9cb80ee790d61bddaf201ecbae32a2ff21dd tvflash: git://git.openfabrics.org/ofed_1_3/tvflash.git ofed_1_3 commit f5e7407a7f2058448df5e5320d9843f944427429 mstflint: git://git.openfabrics.org/ofed_1_3/mstflint.git ofed_1_3 commit 78bbd3d521a9078553a991111ffb6f76665b9ee9 qperf: git://git.openfabrics.org/ofed_1_3/qperf.git ofed_1_3 commit 6221aabd038df0b7033e035378ca190641ed2295 management: git://git.openfabrics.org/ofed_1_3/management.git ofed_1_3 commit d9c852406dae14e8284f9cfb1c7f495bbb55fddf ibutils: git://git.openfabrics.org/ofed_1_3/ibutils.git ofed_1_3 commit 7daf94fab6eaf307316326f3f49704e6080a1508 ibsim: git://git.openfabrics.org/ofed_1_3/ibsim.git ofed_1_3 commit 55113d9f919709c7c97ea41d29991941b9c8be70 ofa_kernel-1.3.1: Git: git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel commit 39e1dc833f98e5134f91fcf7f33df402adf4bc0c # MPI mvapich-1.0.1-2533.src.rpm mvapich2-1.0.3-1.src.rpm openmpi-1.2.6-1.src.rpm mpitests-3.0-773.src.rpm kernel: ib_mthca 0000:06:00.0: Catastrophic error detected: unknown error kernel: ib_mthca 0000:06:00.0: buf[00]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[01]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[02]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[03]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[04]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[05]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[06]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[07]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[08]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[09]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[0a]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[0b]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[0c]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[0d]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[0e]: ffffffff kernel: ib_mthca 0000:06:00.0: buf[0f]: ffffffff kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) kernel: ib0: ib_detach_mcast failed (result = -11) kernel: ib0: ipoib_mcast_detach failed (result = -11) kernel: ib0: ib_detach_mcast failed (result = -11) kernel: ib0: ipoib_mcast_detach failed (result = -11) kernel: ib0: Failed to modify QP to ERROR state kernel: ib0: timing out; 0 sends 128 receives not completed kernel: ib0: Failed to modify QP to RESET state kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) kernel: ib_mthca 0000:06:00.0: HW2SW_CQ failed (-11) kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) kernel: ib_mthca 0000:06:00.0: HW2SW_SRQ failed (-11) kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) kernel: ib_mthca 0000:01:00.0: Catastrophic error detected: internal parity error kernel: ib_mthca 0000:01:00.0: buf[00]: 05000000 kernel: ib_mthca 0000:01:00.0: buf[01]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[02]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[03]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[04]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[05]: 00127f2c kernel: ib_mthca 0000:01:00.0: buf[06]: 000a0056 kernel: ib_mthca 0000:01:00.0: buf[07]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[08]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[09]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[0a]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[0b]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[0c]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[0d]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[0e]: 00000000 kernel: ib_mthca 0000:01:00.0: buf[0f]: 00000000 kernel: ib0: ib_query_port failed -- Pawel Dziekonski Wroclaw Centre for Networking & Supercomputing, HPC Department Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl From tmtalpey at gmail.com Fri Jun 5 05:44:14 2009 From: tmtalpey at gmail.com (Tom Talpey) Date: Fri, 05 Jun 2009 08:44:14 -0400 Subject: [ofa-general] [PATCH] mlx4: fix post send of local invalidate and fast registration packets. In-Reply-To: <200906041740.10409.jackm@dev.mellanox.co.il> References: <200906041740.10409.jackm@dev.mellanox.co.il> Message-ID: <4A29131E.2030806@gmail.com> On 6/4/2009 10:40 AM, Jack Morgenstein wrote: > The ConnectX PRM states that the "SO" bit must be set when posting Fast Reg > and Local Invalidate send packets. When this bit is set, the current WQE will > be executed only after all previous WQEs on this QP have been executed. > (If the bit is not set, Fast Reg and Local Invalidate WQEs may begin execution > too early). > > Signed-off-by: Jack Morgenstein > > --- > Roland, > Please try to get this patch into 2.6.30 -- it is an important fix for nfsrdma. I wasn't aware of this - tell me more what is happening in NFS/RDMA? There is a problem in the NFS client where it isn't setting the LOCAL_WRITE attribute in the FRMR code path, which causes a problem on ConnectX if you don't use the workaround to modify the client's memreg sysctl. That bug is fixe and currently queued in Linus' tree for 2.6.30. If you want, I'll dig up the git change. Tom. > Thanks! > > diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c > index 20724ae..c4a0264 100644 > --- a/drivers/infiniband/hw/mlx4/qp.c > +++ b/drivers/infiniband/hw/mlx4/qp.c > @@ -1585,12 +1585,16 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, > break; > > case IB_WR_LOCAL_INV: > + ctrl->srcrb_flags |= > + cpu_to_be32(MLX4_WQE_CTRL_STRONG_ORDER); > set_local_inv_seg(wqe, wr->ex.invalidate_rkey); > wqe += sizeof (struct mlx4_wqe_local_inval_seg); > size += sizeof (struct mlx4_wqe_local_inval_seg) / 16; > break; > > case IB_WR_FAST_REG_MR: > + ctrl->srcrb_flags |= > + cpu_to_be32(MLX4_WQE_CTRL_STRONG_ORDER); > set_fmr_seg(wqe, wr); > wqe += sizeof (struct mlx4_wqe_fmr_seg); > size += sizeof (struct mlx4_wqe_fmr_seg) / 16; > diff --git a/include/linux/mlx4/qp.h b/include/linux/mlx4/qp.h > index bf8f119..9f29d86 100644 > --- a/include/linux/mlx4/qp.h > +++ b/include/linux/mlx4/qp.h > @@ -165,6 +165,7 @@ enum { > MLX4_WQE_CTRL_IP_CSUM = 1<< 4, > MLX4_WQE_CTRL_TCP_UDP_CSUM = 1<< 5, > MLX4_WQE_CTRL_INS_VLAN = 1<< 6, > + MLX4_WQE_CTRL_STRONG_ORDER = 1<< 7, > }; > > struct mlx4_wqe_ctrl_seg { > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From titoff.a at gmail.com Fri Jun 5 06:00:32 2009 From: titoff.a at gmail.com (Titov Alexander) Date: Fri, 5 Jun 2009 17:00:32 +0400 Subject: [ofa-general] IPoIB in Xen guest machine In-Reply-To: References: Message-ID: 2009/6/5 Hal Rosenstock > On Fri, Jun 5, 2009 at 4:06 AM, Titov Alexander > wrote: > > Hello, > > I work in VDS hosting provider and we want increase our bandwidth with > > Infiniband. I make Infiniband test stand with two machines with > > Mellanox MT26428, switch MTS3600 and SLES11 with Xen. > > How can I connect ip network over IB to Xen guest machine? > > I see two ways: > > 1) I use Qlogic VNIC and bridge virtual xen interface with VNIC ethernet. > It > > isn't true way, because data transfers with great overhead ( Maybe I > > mistake?). > > I try this way and I receve error bad MAD status (110) from lid 0 from > > command ib_qlgc_vnic_query. I try it with OFED and SLES11, Qlogic OFED+ > and > > SLES10SP2, Mellanox OFED and SLES10SP2. I get this error for all > > configurations. > > Is an SM running in the IB subnet ? Yes, OpenSM is running. > > 2) I use IPoIB with Ethernet IB bridge (gateway). But I found about it > only > > few phrases on openfabrics site. (for > > example http://www.openfabrics.org/OFAReport/article4.htm ). Where can I > > find this code? > > Can you suggest me best solution? > > I think there are a few general approaches for IP connectivity between > IB and ethernet: > > 1. Gateways (Voltaire has one, QLogic may (I forget)) > 2. VNIC approaches (QLogic, Xsigo, maybe others) > 3. Use linux box as IP router with ethernet and IPoIB interfaces > > What's best depends on your specific requirements. Thank You. But you don't understand me enough. I wrote about Xen host machine (Dom0) and Xen guest machine (DomU). I need connect infiniband interface in Dom0 and virtual ethernet interface in Dom0. Usually its makes with ethernet bridge, but this bridge cannot work with infiniband logical. I thought that VNIC is best solution for us. But I want hear about alternative solution. > > -- Hal > > > -- > > Best regards, > > Titov Alexander > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > -- С уважением, Титов Александр -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at cisco.com Fri Jun 5 06:44:09 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Fri, 5 Jun 2009 09:44:09 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: <4A257DBF.4000401@mellanox.co.il> Message-ID: <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> A few clarifications are probably worthwhile at this point. 1. No one is suggesting divorcing MPI from the OFED testing/release cycle. It's obviously a Good Thing (and has been stated multiple times on this thread). It is easy to indicate "good combos" of MPI and OFED in documentation and web pages. However, most users expect the latest MPI implementation versions to work with the latest version of OFED (which is not unreasonable). Looking for compatibility between older MPI/OFED versions is something that is almost always relegated to googling. 2. The distinction between development and release phases seems to have been blurred in this discussion. The development phase is where MPI's collaborate with verbs developers to test new APIs, functionality, etc. OFED is the final product, not the development phase. Hence, I'm not sure that the point of "continual improvements in MPI depend on bug fixes/new features in OFED" is relevant. 3. As Doug described, packaging MPI and OFED together actually makes it *harder* for distros. Remember that RHEL and SUSE don't end up using any of the OFED packaging; they essentially use the individual SRPMs. On Jun 5, 2009, at 8:37 AM, John Russo wrote: > I agree with DK from OSU. There are clear advantages to having MPI > included with OFED. Not only will it make testing of a complete > solution easier by both OFED and MPI suppliers, but it will also > improve ease of use for end users and ease of inclusion for linux > distro suppliers. As DK points out there are continual improvements > in MPIs which may depend on bug fixes and/or new features in newer > versions of OFED. Identifying a known good combination will be > important to most end users, etc. > > > > -----Original Message----- > From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org > ] On Behalf Of Dhabaleswar Panda > Sent: Thursday, June 04, 2009 2:02 PM > To: Tziporet Koren > Cc: EWG; OpenFabrics General > Subject: Re: [ewg] RFC: Do we wish to take MPI out of OFED? > > Tziporet, > > >Main reasons to keep MPI in OFED: > >- All participants test with the same MPI versions, and when > installing > >OFED it is ensured that MPI will work fine with this version. > >- Customers convenience in install (no need to go to more sites to > get > MPI) > >- MPI is an important RDMA ULP and although it is not developed in > OFA > >it is widely used by OFED customers > > I support keeping MPI packages in the OFED because of the above > positive > points you have mentioned. > > I would also like to mention that keeping MPI packages in OFED helps > to > test out various new features and functionalities (such as APM and > XRC in > the past and the new memory registration scheme being discussed now) > as > they get introduced. Such an integrated approach helps to test out > these > features at the lower layers as well as at the MPI layer. This process > helps to resolve out any bugs with the new features during the testing > process itself. It also accelerates the deployment and use of these > new > features in the community. > > However, to make the complete OFED release process work smoothly for > everybody (vendors, distros, users, etc.) without affecting the > release > schedule, it is essential that stable MPI packages are added to > OFED. This > is what we have been doing wrt MVAPICH and MVAPICH2 for the last > several > years. > > If the developers of any MPI package do not want it to be a part of > the > OFED due to any constraints, it should be allowed. However, such an > action > should not force to remove all MPI packages. > > >From the point of view of MVAPICH and MVAPICH2 packages in OFED, we > have > been providing stable packages to OFED for the last several years > helping > the OFED community and would like to continue with this process. > > Thanks, > > DK > > > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > -- Jeff Squyres Cisco Systems From rdreier at cisco.com Fri Jun 5 07:01:00 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 05 Jun 2009 07:01:00 -0700 Subject: [ofa-general] ib_mthca Catastrophic errors In-Reply-To: <20090605124033.GK18273@cefeid.wcss.wroc.pl> (Pawel Dziekonski's message of "Fri, 5 Jun 2009 14:40:33 +0200") References: <20090605124033.GK18273@cefeid.wcss.wroc.pl> Message-ID: > kernel: ib_mthca 0000:06:00.0: Catastrophic error detected: unknown error > kernel: ib_mthca 0000:06:00.0: buf[00]: ffffffff Looks like an error on the PCI bus. > kernel: ib_mthca 0000:01:00.0: Catastrophic error detected: internal parity error > kernel: ib_mthca 0000:01:00.0: buf[00]: 05000000 probably what it says it is -- a parity error inside the HCA. Both point to a physical problem to me -- HCA not perfectly seated in PCI slot, power supply flaky, thermal issue, something like that. - R. From hal.rosenstock at gmail.com Fri Jun 5 08:29:58 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 5 Jun 2009 11:29:58 -0400 Subject: [ofa-general] IPoIB CM mode and packet drops Message-ID: I'm seeing the following issue and was hoping for some additional insight: The local IB interface on a "server" node is in CM (connected) mode with an MTU of 64K. dmesg shows numerous errors like: ib0: packet len 8380 (> 2048) too long to send, dropping Client machines are attempting to access this node with IPoIB UD. When the server ib0 interface is switched back to UD (datagram) mode, this problem goes away but that's only a workaround. Has anyone seen this problem ? Any ideas on what causes this and/or how to resolve it ? I've seen some comments about this occuring prior to path MTU discovery. Is the issue is that some IP services don't utilize path MTU discovery ? -- Hal From rdreier at cisco.com Fri Jun 5 10:31:31 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 05 Jun 2009 10:31:31 -0700 Subject: [ofa-general] Re: [PATCH] mlx4: fix post send of local invalidate and fast registration packets. In-Reply-To: <200906051034.30357.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Fri, 5 Jun 2009 10:34:30 +0300") References: <200906041740.10409.jackm@dev.mellanox.co.il> <200906051034.30357.jackm@dev.mellanox.co.il> Message-ID: > Link to message towards end of thread (with very specific problem description): > http://lists.openfabrics.org/pipermail/general/2009-April/059253.html > This patch fixes the problem described in the thread. That is very useful information. Please include that sort of thing in the changelog in the future. I think given that this is not a regression from 2.6.29 and that we are most likely only a day or two before the final 2.6.30 release, I will just queue this patch for 2.6.31 with a cc: stable so that it gets into 2.6.30.1. - R. From sean.hefty at intel.com Fri Jun 5 10:55:43 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 5 Jun 2009 10:55:43 -0700 Subject: [ofa-general] RE: [ofw] skipping QP states during transitions In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD02E72E65@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD02E72E65@mtlexch01.mtl.com> Message-ID: >No, you need to move from reset to init to RTR and only than to RTS. Ok - thanks. >Look at the IB spec on section 10.3 I was just exploring whether any hardware, separate from the existing software stacks, supported 'skipping' QP states -- assuming necessary values for the other states were also given. In theory, hardware could walk through the states internally. The motivation is to decrease the time to connect QPs by reducing the number of commands that need to be issued to the hardware. And to be clear, I'm not suggesting that such a feature is all that important. I'm just exploring ideas. - Sean From Jeffrey.C.Becker at nasa.gov Fri Jun 5 11:04:04 2009 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Fri, 05 Jun 2009 11:04:04 -0700 Subject: [ofa-general] RE: [mwg] Re: RDMA tutorial and OFA In-Reply-To: <35AAF1E4A771E142979F27B51793A4885D9B5E36A0@AVEXMB1.qlogic.org> References: <49F8A59C.3070001@oracle.com> <49F8B149.2050904@oracle.com> <11F9001990984F12B589A718D8976045@BillGWAYLAPTOP> <517c62fb0904301126n277faae7ye816eb496d985945@mail.gmail.com> <35AAF1E4A771E142979F27B51793A4885D9B5E36A0@AVEXMB1.qlogic.org> Message-ID: <4A295E14.9040509@nasa.gov> Hi all. What is the status of this tutorial? The last e-mail I saw about it was at the beginning of May. I'd like to help out. In particular I'm interested in developing a good working set of example codes (starting with those already provided with OFED). Please let me know. Thanks. -jeff Lloyd Dickman wrote: > > I support the idea of the RDMA tutorial. Beyond the “meat” as > described below, I would encourage the tutorial to include a “how to > program RDMA” section. While OFA Verbs provides a rich set of > mechanisms, it is difficult for the average programmer to get a solid > handle on how to use the capabilities, register memory, … Some > cookbook examples, or perhaps development of several programming > “patterns” can go a long way to having RDMA become a much more > mainstream application programming paradigm. > > > > Lloyd > > > > *From:* mwg-bounces at lists.openfabrics.org > [mailto:mwg-bounces at lists.openfabrics.org] *On Behalf Of *arkady kanevsky > *Sent:* Thursday, April 30, 2009 11:27 AM > *To:* bill.boas at openfabrics.org > *Cc:* iwg at lists.openfabrics.org; Paul Grun; Paul Gray; OFA Marketing > Working Group; Wayne Augsburger; Andy Grover; Richard Frank; > asafs`@voltaire.com; Jeff Squyres; Mikkel Hagen; > general at lists.openfabrics.org; Scott Friedman; bobs at voltaire.com; > Sumanta Chatterjee; Roland Dreier > *Subject:* [mwg] Re: RDMA tutorial and OFA > > > > Keep me in the loop. > > I am interested to do it also. > > Thanks, > > Arkady > > On Thu, Apr 30, 2009 at 1:39 PM, Bill Boas > wrote: > > Richard, Andy, > > Thanks for copying me Richard. I had not seen Andy's email on the general > list. > > Figuring out how to get tutorial and other documentation created and > published in the list of things to get done in 2009 for me in my part-time > role as Exec. Dir. > > There is no funding set up for this at the moment but I believe there will > be in about 30 days. > > That's because I'm thinking that we can get funding for this by making it > part of the funding for a new marketing plan for OFA that, with Wayne > Augsburger and Jim Ryan, we are preparing for the OFA Board to vote on at > the next con-call meeting which is on May 20 at 9.00AM PDT. > > Would you be willing to work with me and create a small team from others > within OFA who have the same interest to prepare a description by May > 20 of > what the tutorial would look like, who would contribute to it, how to > get it > "polished up" for web and/or book style publication, what the overall > costs > would be, etc. > > My thoughts, that could be a starting point for the team's work, are > that we > would make the creation a collective effort. > > The tutorial would have several sections for example general intro, > benefits > of RDMA, applicability in HPC and Enterprise, networking background etc. > Members of the Marketing Working Group would be responsible for this. > > The "meat" would be sections for kernel level things (verbs etc.), > then user > space things (verbs etc.), then APIs like MPI, SDP, EDS etc. - each > section > overseen by the technical leaders/maintainers of the code within OFA for > that section (for Example Tom Talpey for NFSoRDMA, or you Richard for RDS) > > Finally the tutorial would have sections about Interoperability > Testing that > OFA/IOL does but also what customers can do on there own systems - Arkady > and Rupert and IOL have put in an SC09 tutorial proposal that we could > leverage in this section. > > To all readers of this email:- > If you have read this far, please give us all some feedback. If you have > material you'd like to contribute please say so. If there's a better way, > tell us what you think it is! > > Thanks, > > Bill. > > Bill Boas > Executive Director and Vice Chair > OpenFabrics Alliance > 510-375-8840 > Bill.Boas at openfabrics.org > www.openfabrics.org > > -----Original Message----- > From: Richard Frank [mailto:richard.frank at oracle.com > ] > Sent: Wednesday, April 29, 2009 12:58 PM > To: Andy Grover > Cc: Bill Boas; Sumanta Chatterjee > Subject: Re: RDMA tutorial and OFA > > Andy, I saw your postings to ofa-general on this and I agree it would be > great to have this documentation. > > As OpenFabrics is really about RDMA... we need to make it simpler > for folks to pick up and run with RDMA concepts ...vs.. digging thru > the IB > specs and code examples, etc. > > Let's see what Bill Boas thinks...perhaps OFA has a writer on board that > can help us do this..? > > I can also help provide input for a new OFA RDMA tutorial doc.. > > Rick > > Andy Grover wrote: > > Hi Rick, > > > > Are you around for a brief chat this afternoon? I have a crazy idea that > > involves OFA doing something (or putting up $$) and I wanted to see what > > you thought, since you're Oracle's OFA rep, right? > > > > -- Andy > > > > > > > > > -- > Cheers, > Arkady Kanevsky > From ftillier at windows.microsoft.com Fri Jun 5 11:03:36 2009 From: ftillier at windows.microsoft.com (Fab Tillier) Date: Fri, 5 Jun 2009 18:03:36 +0000 Subject: [ofa-general] RE: [ofw] skipping QP states during transitions In-Reply-To: References: <5D49E7A8952DC44FB38C38FA0D758EAD02E72E65@mtlexch01.mtl.com> Message-ID: <38877AA1B953874AAE8FBAEE42569B100146A1@TK5EX14MBXW653.wingroup.windeploy.ntdev.microsoft.com> >> Look at the IB spec on section 10.3 > > I was just exploring whether any hardware, separate from the existing > software stacks, supported 'skipping' QP states -- assuming necessary > values for the other states were also given. In theory, hardware could > walk through the states internally. The motivation is to decrease the > time to connect QPs by reducing the number of commands that need to be > issued to the hardware. Assuming the hardware processes commands in the order they're submitted (at least per serializes commands by resource), the HCA driver should be able to queue up the various modify commands so that they are executed by the HCA driver back-to-back. A failure anywhere in this process should cause all subsequent commands to fail due to the QP being in the wrong state. A single IOCTL could generate multiple back-to-back asynchronous QP modify requests. This would eliminate the command completion processing time from the QP modify. However, you need to have async QP modify support in the HCA driver, and ideally the HW needs to be able to process multiple commands issued against a single QP properly (Tzachi, is this something the Mellanox HCAs support?) Note that if the HW driver cannot process multiple commands outstanding for a single resource, the driver could queue up the commands and issue the next from the DPC as soon as the first is completed. In the current model, the DPC has to wake up the blocked thread which does whatever post processing of the command and unwinds to the caller, who then has to make the next blocking QP modify call. -Fab From jgunthorpe at obsidianresearch.com Fri Jun 5 13:41:21 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 5 Jun 2009 14:41:21 -0600 Subject: [ofa-general] IPoIB CM mode and packet drops In-Reply-To: References: Message-ID: <20090605204121.GW29521@obsidianresearch.com> On Fri, Jun 05, 2009 at 11:29:58AM -0400, Hal Rosenstock wrote: > I've seen some comments about this occuring prior to path MTU > discovery. Is the issue is that some IP services don't utilize path > MTU discovery ? This is the standard problem with mixed MTU segments, it happens on ethernet too. path mtu only helps if everything is done properly, and ICMP Can't Fragment messages are being produced. In your case, the IPoIB stack on your host is dropping because it cannot open an RC connection to the target. (The target is UD only) So in this case the kernel on the server should be generating ICMP Can't Fragment errors back to its own networking stack at the same time it prints that dmesg. No idea if there is code to do this or if it is happening for you. It is a bug if that code is missing.. tcpdump can probably tell you. When the server's kernel receives those icmps from itself it can reduce the MTU to that host. Jason From jgunthorpe at obsidianresearch.com Fri Jun 5 15:46:40 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 5 Jun 2009 16:46:40 -0600 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> References: <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> Message-ID: <20090605224640.GA29521@obsidianresearch.com> On Fri, Jun 05, 2009 at 09:44:09AM -0400, Jeff Squyres wrote: > 3. As Doug described, packaging MPI and OFED together actually makes it > *harder* for distros. Remember that RHEL and SUSE don't end up using any > of the OFED packaging; they essentially use the individual SRPMs. I would almost say the entire OFED process is making it harder for the distros, and I mean that in a very general sense. A working IB stack has been in the kernel for a long time now. Can I install Fedora Core, Ubuntu, Debian, etc and have all the necessary IB tools and diagostics available? Nope. IMHO, the entire process would be better served if OFA focused on producing a bleeding edge environment within the community distributions (Fedora, Debian, OpenSuse, etc) and testing that entire stack, including MPIs included in the distro. If this had been done from the start I would have a complete IB stack available in every single distribution TODAY. The truth is, within the Linux world, if you want bleeding edge stuff you use something like FC or Debian Unstable. Otherwise you settle for older, hopefully more tested stuff. The OFED distribution ontop of a distribution is just weird and painfull... For selectivly moving an old distro forward I would *much* rather have true backport packages that exactly match in form and function the native packages in my distro with new versions. That is a far safer path if I actually care about not disrupting my core distribution. Jason From worleys at gmail.com Fri Jun 5 16:15:55 2009 From: worleys at gmail.com (Chris Worley) Date: Fri, 5 Jun 2009 17:15:55 -0600 Subject: [ofa-general] How does ib_srpt decide which ports to use? Message-ID: Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports direct connected, opensm running on all port GUIDs from one host, all links active. Problem: ibsrpdm only advertises the first port of the first HCA of the target. Next problem: I can add targets via /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only when naming the two port guids of the first HCA on the target. In testing, both ports are used. Can somebody aim me in the right direction of what/who's stopping after the first HCA? Thanks, Chris From bart.vanassche at gmail.com Sat Jun 6 00:36:25 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Sat, 6 Jun 2009 09:36:25 +0200 Subject: [ofa-general] How does ib_srpt decide which ports to use? In-Reply-To: References: Message-ID: On Sat, Jun 6, 2009 at 1:15 AM, Chris Worley wrote: > Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports > direct connected, opensm running on all port GUIDs from one host, all > links active. > > Problem: ibsrpdm only advertises the first port of the first HCA of the target. > Next problem: I can add targets via > /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only > when naming the two port guids of the first HCA on the target.  In > testing, both ports are used. > > Can somebody aim me in the right direction of what/who's stopping > after the first HCA? Please have a look at the /sys/class/infiniband_srpt/srpt-*/login_info information on the target. The following information should be present: * One /sys/class/infiniband_srpt/srpt-* entry per HCA. * For each HCA, /sys/class/infiniband_srpt/srpt-${HCA}/login_info should contain one line for each port of that HCA. On the initiator you can use the information obtained from "login_info" (after having replaced tid_ext by id_ext) to log in to the target: echo ... > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target Bart. From vlad at lists.openfabrics.org Sat Jun 6 03:21:54 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 6 Jun 2009 03:21:54 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090606-0200 daily build status Message-ID: <20090606102155.0EAAEE61556@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From worleys at gmail.com Sat Jun 6 10:27:34 2009 From: worleys at gmail.com (Chris Worley) Date: Sat, 6 Jun 2009 11:27:34 -0600 Subject: [ofa-general] How does ib_srpt decide which ports to use? In-Reply-To: References: Message-ID: On Sat, Jun 6, 2009 at 1:36 AM, Bart Van Assche wrote: > On Sat, Jun 6, 2009 at 1:15 AM, Chris Worley wrote: >> Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports >> direct connected, opensm running on all port GUIDs from one host, all >> links active. >> >> Problem: ibsrpdm only advertises the first port of the first HCA of the target. >> Next problem: I can add targets via >> /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only >> when naming the two port guids of the first HCA on the target.  In >> testing, both ports are used. >> >> Can somebody aim me in the right direction of what/who's stopping >> after the first HCA? > > Please have a look at the /sys/class/infiniband_srpt/srpt-*/login_info > information on the target. The following information should be > present: > * One /sys/class/infiniband_srpt/srpt-* entry per HCA. > * For each HCA, /sys/class/infiniband_srpt/srpt-${HCA}/login_info > should contain one line for each port of that HCA. # cat /sys/class/infiniband_srpt/srpt-*/login_info tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000041,service_id=0024710000000040 tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000042,service_id=0024710000000040 tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000045,service_id=0024710000000040 tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000046,service_id=0024710000000040 tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000002c903000292af,service_id=0024710000000040 tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000002c903000292b0,service_id=0024710000000040 Each port has an entry, and the port GUIDs are correct (dgid's), but the rest of the GUIDs refer to the node GUID of the first IB HCA: 0024710000000040. Is that expected? > > On the initiator you can use the information obtained from > "login_info" (after having replaced tid_ext by id_ext) to log in to > the target: > echo ... > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target Using the first HCA's node GUIDs from my target adds on the initiator seems to work, but soon after (and not doing anything w/ the devices) the system panic'd (and remote power cycling is not working). It doesn't look like the panic was anywhere in IB or SRP modules: ... SCSI device sdbo: 314287168 512-byte hdwr sectors (160915 MB) sdbo: Write Protect is off sdbo: Mode Sense: 83 00 10 08 SCSI device sdbo: drive cache: write back w/ FUA SCSI device sdbo: 314287168 512-byte hdwr sectors (160915 MB) sdbo: Write Protect is off sdbo: Mode Sense: 83 00 10 08 SCSI device sdbo: drive cache: write back w/ FUA sdbo: unknown partition table sd 42:0:0:5: Attached scsi disk sdbo Vendor: SCST_BIO Model: vdisk6 Rev: 102 Type: Direct-Access ANSI SCSI revision: 04 SCSI device sdbp: 314287168 512-byte hdwr sectors (160915 MB) sdbp: Write Protect is off sdbp: Mode Sense: 83 00 10 08 SCSI device sdbp: drive cache: write back w/ FUA host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer SCSI device sdbp: 314287168 512-byte hdwr sectors (160915 MB) sdbp: Write Protect is off sdbp: Mode Sense: 83 00 10 08 SCSI device sdbp: drive cache: write back w/ FUA sdbp: unknown partition table sd 42:0:0:6: Attached scsi disk sdbp Vendor: SCST_BIO Model: vdisk7 Rev: 102 Type: Direct-Access ANSI SCSI revision: 04 SCSI device sdbq: 314287168 512-byte hdwr sectors (160915 MB) sdbq: Write Protect is off sdbq: Mode Sense: 83 00 10 08 SCSI device sdbq: drive cache: write back w/ FUA SCSI device sdbq: 314287168 512-byte hdwr sectors (160915 MB) sdbq: Write Protect is off sdbq: Mode Sense: 83 00 10 08 SCSI device sdbq: drive cache: write back w/ FUA sdbq: unknown partition table sd 42:0:0:7: Attached scsi disk sdbq host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: srp_qp_in_err_timer called host31: ib_srp: srp_qp_in_err_timer flushed reset - done host31: ib_srp: Sending CM DREQ failed host37: ib_srp: DREQ received - connection closed host32: ib_srp: srp_qp_in_err_timer called host32: ib_srp: srp_qp_in_err_timer flushed reset - done host32: ib_srp: Sending CM DREQ failed host38: ib_srp: DREQ received - connection closed host37: ib_srp: connection closed ib_srp: host37: add qp_in_err timer host38: ib_srp: connection closed ib_srp: host38: add qp_in_err timer host37: ib_srp: srp_qp_in_err_timer called host37: ib_srp: srp_qp_in_err_timer flushed reset - done host37: ib_srp: Sending CM DREQ failed host31: ib_srp: DREQ received - connection closed host38: ib_srp: srp_qp_in_err_timer called host38: ib_srp: srp_qp_in_err_timer flushed reset - done host38: ib_srp: Sending CM DREQ failed host32: ib_srp: DREQ received - connection closed host31: ib_srp: connection closed ib_srp: host31: add qp_in_err timer host32: ib_srp: connection closed ib_srp: host32: add qp_in_err timer host31: ib_srp: Sending CM DREQ failed host32: ib_srp: Sending CM DREQ failed Unable to handle kernel paging request at ffffffff882539ee RIP: [] PGD 203027 PUD 205027 PMD 407f4f067 PTE 0 Oops: 0010 [1] PREEMPT SMP CPU 0 Modules linked in: mlx4_ib mlx4_core ib_uverbs ib_umad ib_mad ib_core ppdev parport_pc lp parport button ac battery tsdev dm_snapshot dm_mirror dm_mod loop i2c_i801 psmouse i2c_core floppy serio_raw pcspkr shpchp pci_hotplug evdev ext2 mbcache ide_cd cdrom piix ata_piix libata sd_mod generic ehci_hcd ide_core uhci_hcd e1000 qla2xxx firmware_class scsi_transport_fc scsi_mod thermal processor fan Pid: 0, comm: swapper Not tainted 2.6.18-6-clim-amd64 #1 RIP: 0010:[] [] RSP: 0018:ffffffff80597ef8 EFLAGS: 00010246 RAX: ffffffff80625fd8 RBX: ffff8103f12584f0 RCX: ffff8103f125a840 RDX: ffffffff80597f00 RSI: 1144ab87d59a6f6a RDI: ffff8103f12584f0 RBP: ffffffff805cc400 R08: 0000000000000000 R09: ffffffff80597ed8 R10: 00004131a65e699e R11: 0000000000000000 R12: 0000000000000102 R13: ffffffff882539ee R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffffffff80616000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: ffffffff882539ee CR3: 000000041ac7e000 CR4: 00000000000006e0 Process swapper (pid: 0, threadinfo ffffffff80624000, task ffffffff805144c0) Stack: ffffffff8028de7c ffffffff80597f00 ffffffff80597f00 ffff810001035400 0000000000000001 ffffffff80619110 000000000000000a 0000000000000000 ffffffff8020ffbc ffffffff805144c0 0000000000000046 ffffffff80597f78 Call Trace: [] run_timer_softirq+0x13b/0x1be [] __do_softirq+0x52/0xcb [] call_softirq+0x1c/0x28 [] do_softirq+0x2c/0x7d [] irq_exit+0x3f/0x4c [] smp_apic_timer_interrupt+0x3d/0x3f [] mwait_idle+0x0/0x4a [] apic_timer_interrupt+0x66/0x6c [] mwait_idle+0x36/0x4a [] cpu_idle+0x92/0xc9 [] rest_init+0x3f/0x41 [] start_kernel+0x241/0x246 [] _sinittext+0x288/0x28c Code: Bad RIP value. RIP [] RSP CR2: ffffffff882539ee <0>Kernel panic - not syncing: Aiee, killing interrupt handler! I'll go in and power-cycle this in a few hours and try again. Chris > > Bart. > From bart.vanassche at gmail.com Sat Jun 6 10:40:05 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Sat, 6 Jun 2009 19:40:05 +0200 Subject: [ofa-general] How does ib_srpt decide which ports to use? In-Reply-To: References: Message-ID: On Sat, Jun 6, 2009 at 7:27 PM, Chris Worley wrote: > > On Sat, Jun 6, 2009 at 1:36 AM, Bart Van Assche > wrote: > > On Sat, Jun 6, 2009 at 1:15 AM, Chris Worley wrote: > >> Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports > >> direct connected, opensm running on all port GUIDs from one host, all > >> links active. > >> > >> Problem: ibsrpdm only advertises the first port of the first HCA of the target. > >> Next problem: I can add targets via > >> /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only > >> when naming the two port guids of the first HCA on the target.  In > >> testing, both ports are used. > >> > >> Can somebody aim me in the right direction of what/who's stopping > >> after the first HCA? > > > > Please have a look at the /sys/class/infiniband_srpt/srpt-*/login_info > > information on the target. The following information should be > > present: > > * One /sys/class/infiniband_srpt/srpt-* entry per HCA. > > * For each HCA, /sys/class/infiniband_srpt/srpt-${HCA}/login_info > > should contain one line for each port of that HCA. > > # cat /sys/class/infiniband_srpt/srpt-*/login_info > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000041,service_id=0024710000000040 > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000042,service_id=0024710000000040 > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000045,service_id=0024710000000040 > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000046,service_id=0024710000000040 > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000002c903000292af,service_id=0024710000000040 > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000002c903000292b0,service_id=0024710000000040 > > Each port has an entry, and the port GUIDs are correct (dgid's), but > the rest of the GUIDs refer to the node GUID of the first IB HCA: > 0024710000000040. > > Is that expected? Yes. The ioc_guid in the above output is a GUID that identifies the SRP target. A quote from the ib_srpt source code: /* * We do not have a consistent service_id (ie. also id_ext of target_id) * to identify this target. We currently use the guid of the first HCA * in the system as service_id; therefore, the target_id will change * if this HCA is gone bad and replaced by different HCA. */ I'm not sure however what ibsrpdm should display -- I don't know whether it should display one single set of login parameters or all possible login parameters. > > On the initiator you can use the information obtained from > > "login_info" (after having replaced tid_ext by id_ext) to log in to > > the target: > > echo ... > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target > > Using the first HCA's node GUIDs from my target adds on the initiator > seems to work, but soon after (and not doing anything w/ the devices) > the system panic'd (and remote power cycling is not working).  It > doesn't look like the panic was anywhere in IB or SRP modules: > [ ... ] That's bad news. Anyway, if the kernel on the initiator system crashes, that's a bug in the kernel of the initiator system. I hope that this can be resolved through a support contract. If not, I'm afraid that you will have to experiment with kernel versions and OFED versions in order to find a combination that works. Bart. From dledford at redhat.com Sat Jun 6 17:33:52 2009 From: dledford at redhat.com (Doug Ledford) Date: Sat, 06 Jun 2009 20:33:52 -0400 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: Message-ID: <1244334832.21347.62.camel@firewall.xsintricity.com> On Thu, 2009-06-04 at 14:02 -0400, Dhabaleswar Panda wrote: > Tziporet, > > >Main reasons to keep MPI in OFED: > >- All participants test with the same MPI versions, and when installing > >OFED it is ensured that MPI will work fine with this version. > >- Customers convenience in install (no need to go to more sites to get > MPI) > >- MPI is an important RDMA ULP and although it is not developed in OFA > >it is widely used by OFED customers > > I support keeping MPI packages in the OFED because of the above positive > points you have mentioned. > > I would also like to mention that keeping MPI packages in OFED helps to > test out various new features and functionalities (such as APM and XRC in > the past and the new memory registration scheme being discussed now) It's interesting that you mention XRC support as something from the past. As far as I'm concerned, it's still a "hasn't landed yet" feature as it still hasn't landed upstream. > as > they get introduced. Such an integrated approach helps to test out these > features at the lower layers as well as at the MPI layer. Such an "integrated" approach is great for a test lab, but has no place in a production environment. This "integrated" approach is what you do when you are in proof of concept phase. After that, you move into development phase where you engineer it properly, define the API, clean up the code, fix bugs, etc. (this is generally what happens during the linux kernel review process, or at least partially). Finally, you reach a stable phase, where most of the bugs are fixed, the API is fixed, and you can generally rely upon the software to work as expected and to handle the majority of situations it is likely to encounter. This "integrated" approach you mention is really only useful when you are trying to leap frog directly from proof of concept to production use without ever going through the other phases and without ever bringing your code quality up to snuff. > However, to make the complete OFED release process work smoothly for > everybody (vendors, distros, users, etc.) without affecting the release > schedule, it is essential that stable MPI packages are added to OFED. This > is what we have been doing wrt MVAPICH and MVAPICH2 for the last several > years. If you're just throwing in the latest stable release, then it serves no purpose. Whether it's in OFED or on your site makes absolutely 0 difference except to the size of the OFED tarball. > If the developers of any MPI package do not want it to be a part of the > OFED due to any constraints, it should be allowed. However, such an action > should not force to remove all MPI packages. > > >From the point of view of MVAPICH and MVAPICH2 packages in OFED, we have > been providing stable packages to OFED for the last several years helping > the OFED community and would like to continue with this process. > > Thanks, > > DK > > > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jackm at dev.mellanox.co.il Sun Jun 7 00:18:27 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 7 Jun 2009 10:18:27 +0300 Subject: [ofa-general] Re: [PATCH] mlx4: fix post send of local invalidate and fast registration packets. In-Reply-To: References: <200906041740.10409.jackm@dev.mellanox.co.il> <200906051034.30357.jackm@dev.mellanox.co.il> Message-ID: <200906071018.27280.jackm@dev.mellanox.co.il> On Friday 05 June 2009 20:31, Roland Dreier wrote: > > > Link to message towards end of thread (with very specific problem description): > > http://lists.openfabrics.org/pipermail/general/2009-April/059253.html > > > This patch fixes the problem described in the thread. > > That is very useful information. Please include that sort of thing in > the changelog in the future. > I'll do that. > I think given that this is not a regression from 2.6.29 and that we are > most likely only a day or two before the final 2.6.30 release, I will > just queue this patch for 2.6.31 with a cc: stable so that it gets into > 2.6.30.1. > Thanks! -Jack From pashash at gmail.com Sun Jun 7 01:09:56 2009 From: pashash at gmail.com (Pavel Shamis (Pasha)) Date: Sun, 07 Jun 2009 11:09:56 +0300 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A257DBF.4000401@mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> Message-ID: <4A2B75D4.7080600@dev.mellanox.co.il> > - Need to synchronize between different projects The synchronization between OFED and MPIs are minimal. In most cases OFED just take last available MPI version. It happens that OFED discovers some really critical bug and we ask MPI to provide new version. But for critical bugs, MPI team anyway releases new bugfix version ASAP. > > - MPI is an important RDMA ULP and although it is not developed in OFA > it is widely used by OFED customers I guess that MPI users are the widest OFED user community and including MPI as part of OFED definitely simplify end-user live. I'm personally against excluding MPI from OFED package. Pasha. From pashash at gmail.com Sun Jun 7 01:36:45 2009 From: pashash at gmail.com (Pavel Shamis (Pasha)) Date: Sun, 07 Jun 2009 11:36:45 +0300 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <1244126585.19973.9.camel@firewall.xsintricity.com> References: <4A257DBF.4000401@mellanox.co.il> <20090603073949.GC3261@cefeid.wcss.wroc.pl> <20090603171434.GR3261@cefeid.wcss.wroc.pl> <1244126585.19973.9.camel@firewall.xsintricity.com> Message-ID: <4A2B7C1D.50803@dev.mellanox.co.il> > I care about both. I care about the fact that a solid, well adhered to > API makes for lots of happy MPI campers, not just one happy MPI camper. > And the API road is the path to long term interoperability, not just > short term. So if you really care about MPI, I would recommend you look > to the long term, and you may find you agree with me then. > > I agree here with Doug, API road is way to go. But It is not reason to exclude MPIs from OFED. OFED API issues maybe resolved even when MPIs are part of package. Pasha From vlad at lists.openfabrics.org Sun Jun 7 03:23:10 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 7 Jun 2009 03:23:10 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090607-0200 daily build status Message-ID: <20090607102310.E8531E6159B@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From jackm at dev.mellanox.co.il Sun Jun 7 03:47:36 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 7 Jun 2009 13:47:36 +0300 Subject: [ofa-general] [PATCH] mlx4: fix post send of local invalidate =?iso-8859-1?q?and=09fast_registration?= packets. In-Reply-To: <4A29131E.2030806@gmail.com> References: <200906041740.10409.jackm@dev.mellanox.co.il> <4A29131E.2030806@gmail.com> Message-ID: <200906071347.36433.jackm@dev.mellanox.co.il> On Friday 05 June 2009 15:44, Tom Talpey wrote: > If you want, I'll dig up the git change. > Thanks, but no need. I know about that one. This is a different bug. -Jack From tziporet at dev.mellanox.co.il Sun Jun 7 03:50:18 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 07 Jun 2009 13:50:18 +0300 Subject: [ofa-general] ib_mthca Catastrophic errors In-Reply-To: <20090605124033.GK18273@cefeid.wcss.wroc.pl> References: <20090605124033.GK18273@cefeid.wcss.wroc.pl> Message-ID: <4A2B9B6A.40500@mellanox.co.il> Pawel Dziekonski wrote: > Hi, > > from time to time I get Catastrophic errors like below. software stack is > kernel 2.6.18-92.1.10.el5 with Lustre client. device and OFED info is also > below. > > any hints? > > thanks in advance, Pawel > > > > 06:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) > > # ibv_devices > device node GUID > ------ ---------------- > mthca0 0030487e07700000 > # ibv_devinfo > hca_id: mthca0 > fw_ver: 1.2.0 > node_guid: 0030:487e:0770:0000 > sys_image_guid: 0030:487e:0770:0003 > vendor_id: 0x02c9 > vendor_part_id: 25204 > hw_ver: 0xA0 > board_id: SM_0000000003 > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 2048 (4) > active_mtu: 2048 (4) > sm_lid: 1 > port_lid: 441 > port_lmc: 0x00 > > > > > > kernel: ib_mthca 0000:06:00.0: Catastrophic error detected: unknown error > kernel: ib_mthca 0000:06:00.0: buf[00]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[01]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[02]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[03]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[04]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[05]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[06]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[07]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[08]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[09]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[0a]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[0b]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[0c]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[0d]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[0e]: ffffffff > kernel: ib_mthca 0000:06:00.0: buf[0f]: ffffffff > kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) > kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) > kernel: ib0: ib_detach_mcast failed (result = -11) > kernel: ib0: ipoib_mcast_detach failed (result = -11) > kernel: ib0: ib_detach_mcast failed (result = -11) > kernel: ib0: ipoib_mcast_detach failed (result = -11) > kernel: ib0: Failed to modify QP to ERROR state > kernel: ib0: timing out; 0 sends 128 receives not completed > kernel: ib0: Failed to modify QP to RESET state > kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) > kernel: ib_mthca 0000:06:00.0: HW2SW_CQ failed (-11) > kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) > kernel: ib_mthca 0000:06:00.0: HW2SW_SRQ failed (-11) > kernel: ib_mthca 0000:06:00.0: HW2SW_MPT failed (-11) > > > kernel: ib_mthca 0000:01:00.0: Catastrophic error detected: internal parity error > kernel: ib_mthca 0000:01:00.0: buf[00]: 05000000 > kernel: ib_mthca 0000:01:00.0: buf[01]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[02]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[03]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[04]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[05]: 00127f2c > kernel: ib_mthca 0000:01:00.0: buf[06]: 000a0056 > kernel: ib_mthca 0000:01:00.0: buf[07]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[08]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[09]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[0a]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[0b]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[0c]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[0d]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[0e]: 00000000 > kernel: ib_mthca 0000:01:00.0: buf[0f]: 00000000 > kernel: ib0: ib_query_port failed > > This is a known issue with Infinihost III HCA FW 1.2.0 Please contact Mellanox support to get an updated version for the FW Tziporet From yevgenyp at mellanox.co.il Sun Jun 7 08:16:09 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Sun, 07 Jun 2009 18:16:09 +0300 Subject: [ofa-general][PATCH] mlx4: FIX error flow when initializing EQ table Message-ID: <4A2BD9B9.6000802@mellanox.co.il> If mlx4_create_eq() would fail for one of EQ's assigned for completion handling, the code would try to free the same EQ we failed to create. The crash was found by Christoph Lameter Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/eq.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/drivers/net/mlx4/eq.c b/drivers/net/mlx4/eq.c index 8830dcb..dee1887 100644 --- a/drivers/net/mlx4/eq.c +++ b/drivers/net/mlx4/eq.c @@ -623,8 +623,10 @@ int mlx4_init_eq_table(struct mlx4_dev *dev) err = mlx4_create_eq(dev, dev->caps.num_cqs + MLX4_NUM_SPARE_EQE, (dev->flags & MLX4_FLAG_MSI_X) ? i : 0, &priv->eq_table.eq[i]); - if (err) + if (err) { + --i; goto err_out_unmap; + } } err = mlx4_create_eq(dev, MLX4_NUM_ASYNC_EQE + MLX4_NUM_SPARE_EQE, -- 1.6.1.3 From tziporet at mellanox.co.il Sun Jun 7 07:52:48 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Sun, 7 Jun 2009 17:52:48 +0300 Subject: [ofa-general] OFED 1.5 schedule and OS support Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD02EE7404@mtlexch01.mtl.com> Hi All OFED 1.5 schedule: * Feature freeze June 22 * Alpha June 25 * Beta July 22 * RC1 Aug 12 * RC2-RCx: About every 2 weeks as needed We usually have ~6 RCs * GA Oct 30 OSes: * RHEL4: up6, up7, up8 (when released) * RHEL5: up2 and up3 * SLES10: SP2, SP3 (when released) * SLES 11 * Fedora Core 10* * OpenSuSE 11* * Kernel.org: 2.6.18-2.6.30 * OSes with partial QA I will update OFED roadmap with this info Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From dorfman.eli at gmail.com Sun Jun 7 08:06:00 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Sun, 07 Jun 2009 18:06:00 +0300 Subject: [ofa-general] [PATCH] opensm/osm_sa_node_record.c use comp mask to match by LID or GUID Message-ID: <4A2BD758.9080204@gmail.com> use comp mask to match by LID or GUID when matching node records use comp mask to filter queries that use 0 lid or guid while mask is set Signed-off-by: Eli Dorfman --- opensm/opensm/osm_sa_node_record.c | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/opensm/opensm/osm_sa_node_record.c b/opensm/opensm/osm_sa_node_record.c index d42e6ac..d04ce63 100644 --- a/opensm/opensm/osm_sa_node_record.c +++ b/opensm/opensm/osm_sa_node_record.c @@ -114,7 +114,8 @@ static void nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * p_node, IN cl_qlist_t * p_list, IN ib_net64_t const match_port_guid, IN ib_net16_t const match_lid, - IN const osm_physp_t * p_req_physp) + IN const osm_physp_t * p_req_physp, + IN const ib_net64_t comp_mask) { const osm_physp_t *p_physp; uint8_t port_num; @@ -153,7 +154,7 @@ static void nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * p_node, port_guid = osm_physp_get_port_guid(p_physp); - if (match_port_guid && (port_guid != match_port_guid)) + if ((comp_mask & IB_NR_COMPMASK_PORTGUID) && (port_guid != match_port_guid)) continue; base_lid = osm_physp_get_base_lid(p_physp); @@ -162,7 +163,7 @@ static void nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * p_node, max_lid_ho = (uint16_t) (base_lid_ho + (1 << lmc) - 1); match_lid_ho = cl_ntoh16(match_lid); - if (match_lid_ho) { + if (comp_mask & IB_NR_COMPMASK_LID) { /* We validate that the lid belongs to this node. */ @@ -190,7 +191,7 @@ static void nr_rcv_by_comp_mask(IN cl_map_item_t * p_map_item, IN void *context) const ib_node_record_t *const p_rcvd_rec = p_ctxt->p_rcvd_rec; const osm_physp_t *const p_req_physp = p_ctxt->p_req_physp; osm_sa_t *sa = p_ctxt->sa; - ib_net64_t const comp_mask = p_ctxt->comp_mask; + ib_net64_t comp_mask = p_ctxt->comp_mask; ib_net64_t match_port_guid = 0; ib_net16_t match_lid = 0; @@ -274,7 +275,7 @@ static void nr_rcv_by_comp_mask(IN cl_map_item_t * p_map_item, IN void *context) } nr_rcv_create_nr(sa, p_node, p_ctxt->p_list, match_port_guid, - match_lid, p_req_physp); + match_lid, p_req_physp, comp_mask); Exit: OSM_LOG_EXIT(p_ctxt->sa->p_log); -- 1.5.5 From dledford at redhat.com Sun Jun 7 09:18:48 2009 From: dledford at redhat.com (Doug Ledford) Date: Sun, 07 Jun 2009 12:18:48 -0400 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A2B7C1D.50803@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> <20090603073949.GC3261@cefeid.wcss.wroc.pl> <20090603171434.GR3261@cefeid.wcss.wroc.pl> <1244126585.19973.9.camel@firewall.xsintricity.com> <4A2B7C1D.50803@dev.mellanox.co.il> Message-ID: <1244391528.21347.96.camel@firewall.xsintricity.com> On Sun, 2009-06-07 at 11:36 +0300, Pavel Shamis (Pasha) wrote: > > I care about both. I care about the fact that a solid, well adhered to > > API makes for lots of happy MPI campers, not just one happy MPI camper. > > And the API road is the path to long term interoperability, not just > > short term. So if you really care about MPI, I would recommend you look > > to the long term, and you may find you agree with me then. > > > > > I agree here with Doug, API road is way to go. > But It is not reason to exclude MPIs from OFED. > OFED API issues maybe resolved even when > MPIs are part of package. You're right, API issues *can* be resolved even if the latest MPIs are shipped with OFED. It's just that this hasn't been the case in the past, and throwing everything in the same bucket is a strong incentive not to worry about it in the future. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From dledford at redhat.com Sun Jun 7 09:58:55 2009 From: dledford at redhat.com (Doug Ledford) Date: Sun, 07 Jun 2009 12:58:55 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> References: <4A257DBF.4000401@mellanox.co.il> <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> Message-ID: <1244393935.21347.110.camel@firewall.xsintricity.com> On Fri, 2009-06-05 at 09:44 -0400, Jeff Squyres wrote: > 3. As Doug described, packaging MPI and OFED together actually makes > it *harder* for distros. Remember that RHEL and SUSE don't end up > using any of the OFED packaging; they essentially use the individual > SRPMs. One minor clarification, it's not so much the RPM packaging that makes things difficult, it's the compatibility matrix. Since things aren't designed to cleanly inter-operate with each other in anything other than very specific combinations, it means that updates are an all or nothing affair. This is in direct contrast to the rest of our entire operating system where we isolate and target things that need fixed and only things that need fixed. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From robert.j.woodruff at intel.com Sun Jun 7 13:51:03 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Sun, 7 Jun 2009 13:51:03 -0700 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <1244393935.21347.110.camel@firewall.xsintricity.com> References: <4A257DBF.4000401@mellanox.co.il> <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> <1244393935.21347.110.camel@firewall.xsintricity.com> Message-ID: <382A478CAD40FA4FB46605CF81FE39F4351543A2@orsmsx507.amr.corp.intel.com> Doug wrote, >One minor clarification, it's not so much the RPM packaging that makes >things difficult, it's the compatibility matrix. Since things aren't >designed to cleanly inter-operate with each other in anything other than >very specific combinations, it means that updates are an all or nothing >affair. This is in direct contrast to the rest of our entire operating >system where we isolate and target things that need fixed and only >things that need fixed. I think this was true early on with the OFA and OFED releases, but I do think things are stabilizing in this area as the code has matured and thus I think that having various components decoupled should be easier going forward. So from a technical standpoint, I do not see a problem with removing MPIs from OFED. I can however see the benefit of that packaging from a customer point of view in making it easier to install without having to go get packages from multiple places. I personally do not really care either way as I use Intel MPI for my testing, and BTW, Intel MPI has always been decoupled and we have not seen this to be a problem. We have recommended people install newer versions of OFED from time to time as MPI found bugs that were fixed in the newer OFED, but it was not the API that was not stable, it was just bugs that were found that required a newer OFED version. woody From dledford at redhat.com Sun Jun 7 15:31:11 2009 From: dledford at redhat.com (Doug Ledford) Date: Sun, 07 Jun 2009 18:31:11 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <382A478CAD40FA4FB46605CF81FE39F4351543A2@orsmsx507.amr.corp.intel.com> References: <4A257DBF.4000401@mellanox.co.il> <7AB34518-0C5F-47E3-9714-D1B339A3320E@cisco.com> <1244393935.21347.110.camel@firewall.xsintricity.com> <382A478CAD40FA4FB46605CF81FE39F4351543A2@orsmsx507.amr.corp.intel.com> Message-ID: <1244413872.21347.139.camel@firewall.xsintricity.com> On Sun, 2009-06-07 at 13:51 -0700, Woodruff, Robert J wrote: > Doug wrote, > > >One minor clarification, it's not so much the RPM packaging that makes > >things difficult, it's the compatibility matrix. Since things aren't > >designed to cleanly inter-operate with each other in anything other than > >very specific combinations, it means that updates are an all or nothing > >affair. This is in direct contrast to the rest of our entire operating > >system where we isolate and target things that need fixed and only > >things that need fixed. > > I think this was true early on with the OFA and OFED releases, but > I do think things are stabilizing in this area as the code has > matured and thus I think that having various components decoupled > should be easier going forward. For most things this is true, but not for all. > BTW, Intel MPI has always > been decoupled and we have not seen this to be a problem. We have > recommended people install newer versions of OFED from time to time > as MPI found bugs that were fixed in the newer OFED, but it was > not the API that was not stable, it was just bugs that were found > that required a newer OFED version. I don't know the particulars of how Intel MPI is distributed, built, etc. But, when you tell me that you have from time to time recommended customers install a later version of OFED to get a bug fix, I get the impression that you start off by telling customers that OFED is a prerequisite to running Intel MPI in the first place. If that's the case, then I would question whether that's decoupled from OFED, or just not in the OFED tarball. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From karun.sharma at qlogic.com Sun Jun 7 21:59:48 2009 From: karun.sharma at qlogic.com (Karun Sharma (Contractor - GS Labs)) Date: Sun, 7 Jun 2009 23:59:48 -0500 Subject: [ofa-general] How does ib_srpt decide which ports to use? In-Reply-To: References: Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E990A6D9@MNEXMB1.qlogic.org> By default, ibsrpdm will use umad0. To discover the target using umad1, use "ibsrpdm -d /dev/umad1" Regards Karun -----Original Message----- From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Chris Worley Sent: Saturday, June 06, 2009 4:46 AM To: OpenIB; scst-devel Subject: [ofa-general] How does ib_srpt decide which ports to use? Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports direct connected, opensm running on all port GUIDs from one host, all links active. Problem: ibsrpdm only advertises the first port of the first HCA of the target. Next problem: I can add targets via /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only when naming the two port guids of the first HCA on the target. In testing, both ports are used. Can somebody aim me in the right direction of what/who's stopping after the first HCA? Thanks, Chris _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From davem at davemloft.net Mon Jun 8 00:40:07 2009 From: davem at davemloft.net (David Miller) Date: Mon, 08 Jun 2009 00:40:07 -0700 (PDT) Subject: [ofa-general][PATCH] mlx4: FIX error flow when initializing EQ table In-Reply-To: <4A2BD9B9.6000802@mellanox.co.il> References: <4A2BD9B9.6000802@mellanox.co.il> Message-ID: <20090608.004007.186996388.davem@davemloft.net> From: Yevgeny Petrilin Date: Sun, 07 Jun 2009 18:16:09 +0300 > If mlx4_create_eq() would fail for one of EQ's assigned for > completion handling, the code would try to free the same EQ > we failed to create. > The crash was found by Christoph Lameter > > Signed-off-by: Yevgeny Petrilin Applied, thanks. From nicolas.morey-chaisemartin at ext.bull.net Mon Jun 8 01:52:19 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey-Chaisemartin) Date: Mon, 08 Jun 2009 10:52:19 +0200 Subject: [ofa-general] [PATCH] Fixed memory leak in drivers/net/mlx4/main.c Message-ID: <4A2CD143.6050506@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- This patch made it into linux mainstream but seems to have bit lost in the ofed 1.5 branch. drivers/net/mlx4/main.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 102bac9..30bea96 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -976,7 +976,7 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev) nreq = err; goto retry; } - + kfree(entries); goto no_msi; } -- 1.6.3.1 From dorfman.eli at gmail.com Mon Jun 8 03:03:01 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 08 Jun 2009 13:03:01 +0300 Subject: [ofa-general] [PATCH] infiniband_diags/ibportstate.c perform reset in a single mad In-Reply-To: <4A2684E8.403@gmail.com> References: <4A2297DE.3050707@gmail.com> <4A2684E8.403@gmail.com> Message-ID: <4A2CE1D5.2050503@gmail.com> perform reset in a single mad Perform reset by changing physical state to polling instead of disable and enable. This ensures that reset a port will not disconnect it (assuming negotiation succeeds). Only disable operation is not allowed on HCA ports since it will irreversibly disconnect it though this may also happen on a switch ingress port. Signed-off-by: Eli Dorfman --- infiniband-diags/src/ibportstate.c | 24 ++++++++---------------- 1 files changed, 8 insertions(+), 16 deletions(-) diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c index d19a2e5..93738c0 100644 --- a/infiniband-diags/src/ibportstate.c +++ b/infiniband-diags/src/ibportstate.c @@ -233,6 +233,9 @@ int main(int argc, char **argv) ibd_sm_id, srcport) < 0) IBERROR("can't resolve destination port %s", argv[0]); + if (argc > 1) + portnum = strtol(argv[1], 0, 0); + /* First, make sure it is a switch port if it is a "set" */ if (argc >= 3) { if (!strcmp(argv[2], "enable")) @@ -258,13 +261,10 @@ int main(int argc, char **argv) if (err) { /* not switch */ if (port_op == 0) /* query op */ is_switch = 0; - else if (port_op != 4) /* other than speed op */ - IBERROR("smp query nodeinfo: Node type not switch"); + else if (port_op == 2) /* disable */ + IBERROR("Node type not switch - disable not allowed"); } - if (argc-1 > 0) - portnum = strtol(argv[1], 0, 0); - if (port_op) printf("Initial PortInfo:\n"); else @@ -275,11 +275,11 @@ int main(int argc, char **argv) /* Only if one of the "set" options is chosen */ if (port_op) { - if (port_op == 1) { /* Enable port */ + if ((port_op == 1) || (port_op == 3)) { /* Enable or Reset port */ mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ - } - else if ((port_op == 2) || (port_op == 3)) { /* Disable port */ + } else if (port_op == 2) { /* Disable port */ + printf("Disable may be irreversible\n"); mad_set_field(data, 0, IB_PORT_STATE_F, 1); /* Down */ mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 3); /* Disabled */ } else if (port_op == 4) { /* Set speed */ @@ -291,14 +291,6 @@ int main(int argc, char **argv) err = set_port_info(&portid, data, portnum, port_op); if (err < 0) IBERROR("smp set portinfo failed"); - - if (port_op == 3) { /* Reset port - so also enable */ - mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ - mad_set_field(data, 0, IB_PORT_STATE_F, 0); /* No Change */ - err = set_port_info(&portid, data, portnum, port_op); - if (err < 0) - IBERROR("smp set portinfo failed"); - } } else { /* query op */ /* only compare peer port if switch port */ if (is_switch) { -- 1.5.5 From jsquyres at cisco.com Mon Jun 8 03:17:30 2009 From: jsquyres at cisco.com (Jeff Squyres (jsquyres)) Date: Mon, 8 Jun 2009 06:17:30 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out ofOFED? Message-ID: Agreed. Ofed and mpis don't ship often enough to create incompatible releases (ie we should be able to handle them separately through qa in both sets of release procedures). Thanks for the clarification. -jms Sent from my PDA. No type good. ----- Original Message ----- From: Doug Ledford To: Jeff Squyres (jsquyres) Cc: John Russo ; EWG ; OpenFabrics General Sent: Sun Jun 07 12:58:55 2009 Subject: Re: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out ofOFED? On Fri, 2009-06-05 at 09:44 -0400, Jeff Squyres wrote: > 3. As Doug described, packaging MPI and OFED together actually makes > it *harder* for distros. Remember that RHEL and SUSE don't end up > using any of the OFED packaging; they essentially use the individual > SRPMs. One minor clarification, it's not so much the RPM packaging that makes things difficult, it's the compatibility matrix. Since things aren't designed to cleanly inter-operate with each other in anything other than very specific combinations, it means that updates are an all or nothing affair. This is in direct contrast to the rest of our entire operating system where we isolate and target things that need fixed and only things that need fixed. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at lists.openfabrics.org Mon Jun 8 03:23:56 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 8 Jun 2009 03:23:56 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090608-0200 daily build status Message-ID: <20090608102356.99E54E6167F@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From alexander.supalov at intel.com Mon Jun 8 05:42:25 2009 From: alexander.supalov at intel.com (Supalov, Alexander) Date: Mon, 8 Jun 2009 13:42:25 +0100 Subject: [ofa-general] Memory registration redux References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> <5019F239-149F-49E1-8C23-436DE6094AB2@cisco.com> <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> Message-ID: <928CFBE8E7CB0040959E56B4EA41A77EBBC092A7@irsmsx504.ger.corp.intel.com> Hi, Intel MPI developers are in principle OK with this proposal. What way of delivery is envisioned? Will this become a part of OFED or of the mainstream kernel? How fast will it spread? Are there any comparable Windows plans? Best regards. Alexander -----Original Message----- From: Supalov, Alexander Sent: Wednesday, June 03, 2009 12:26 PM To: 'Roland Dreier' Cc: Jeff Squyres; Pavel Shamis; Hans Westgaard Ry; Dontje; Lenny Verkhovsky; H??kon Bugge; Donald Kerr; OpenFabrics General Subject: RE: [ofa-general] Memory registration redux Thanks. This is what I was looking for. Let me pass this by the key Intel MPI developers and get back to you. -----Original Message----- From: Roland Dreier [mailto:rdreier at cisco.com] Sent: Tuesday, June 02, 2009 6:45 PM To: Supalov, Alexander Cc: Jeff Squyres; Pavel Shamis; Hans Westgaard Ry; Dontje; Lenny Verkhovsky; H??kon Bugge; Donald Kerr; OpenFabrics General Subject: Re: [ofa-general] Memory registration redux > Sorry, it's kind of difficult to deduce looking at this Q&A sequence > what works how and when. Is it possible to create a brief and direct > description of the proposed solution? Did you see the original patch description I sent: As discussed in and follow-up messages, libraries using RDMA would like to track precisely when application code changes memory mapping via free(), munmap(), etc. Current pure-userspace solutions using malloc hooks and other tricks are not robust, and the feeling among experts is that the issue is unfixable without kernel help. We solve this not by implementing the full API proposed in the email linked above but rather with a simpler and more generic interface, which may be useful in other contexts. Specifically, we implement a new character device driver, ummunot, that creates a /dev/ummunot node. A userspace process can open this node read-only and use the fd as follows: 1. ioctl() to register/unregister an address range to watch in the kernel (cf struct ummunot_register_ioctl in ). 2. read() to retrieve events generated when a mapping in a watched address range is invalidated (cf struct ummunot_event in ). select()/poll()/epoll() and SIGIO are handled for this IO. 3. mmap() one page at offset 0 to map a kernel page that contains a generation counter that is incremented each time an event is generated. This allows userspace to have a fast path that checks that no events have occurred without a system call. --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From tziporet at dev.mellanox.co.il Mon Jun 8 06:27:32 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Mon, 08 Jun 2009 16:27:32 +0300 Subject: [ofa-general] Memory registration redux In-Reply-To: <928CFBE8E7CB0040959E56B4EA41A77EBBC092A7@irsmsx504.ger.corp.intel.com> References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> <5019F239-149F-49E1-8C23-436DE6094AB2@cisco.com> <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> <928CFBE8E7CB0040959E56B4EA41A77EBBC092A7@irsmsx504.ger.corp.intel.com> Message-ID: <4A2D11C4.5070306@mellanox.co.il> Supalov, Alexander wrote: > Hi, > > Intel MPI developers are in principle OK with this proposal. What way of delivery is envisioned? Will this become a part of OFED or of the mainstream kernel? Roland is planing to push it to kernel 2.6.31 And OFED will take it from the kernel. We will check if we can do backports for distros. I assume it will be available only for distros that have the MMU notifiers in the kernel. > How fast will it spread? Are there any comparable Windows plans? > > I cannot answer on Windows Tziporet From todd.rimmer at qlogic.com Mon Jun 8 06:59:52 2009 From: todd.rimmer at qlogic.com (Todd Rimmer) Date: Mon, 8 Jun 2009 08:59:52 -0500 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: <4A257DBF.4000401@mellanox.co.il> Message-ID: <5AEC2602AE03EB46BFC16C6B9B200DA81455DD0350@MNEXMB2.qlogic.org> I agree with DK from OSU. There are clear advantages to having MPI included with OFED. Not only will it make testing of a complete solution easier by both OFED and MPI suppliers, but it will also improve ease of use for end users. As DK points out there are continual improvements in MPIs which may depend on bug fixes and/or new features in newer versions of OFED. Identifying a known good combination will be important to most end users, etc. I would point out that the OFED API seems to be holding up quite well. Applications which we built against OFED 1.3 have worked in binary form against OFED 1.4 and 1.4.1 without issue, in fact there are many cases where we build binaries for applications against OFED 1.3 and ship the same binary for use on both old and new OFED versions. However MPI has some differences in this area. In the constant search for better latency and bandwidth, there will be ongoing development and tuning. So in many cases the best performance for MPI will require the latest MPI and the latest OFED. This does not necessarily mean the API is broken, but rather it means a combination has been well tuned and tested. In most if not all cases, other combinations will work well, but may not achieve the same performance level. This is simply the nature of the HPC marketplace. Todd Rimmer Chief Architect QLogic Network Systems Group Voice: 610-233-4852 Fax: 610-233-4777 Todd.Rimmer at QLogic.com www.QLogic.com > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general- > bounces at lists.openfabrics.org] On Behalf Of Dhabaleswar Panda > Sent: Thursday, June 04, 2009 2:02 PM > To: Tziporet Koren > Cc: EWG; OpenFabrics General > Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of > OFED? > > Tziporet, > > >Main reasons to keep MPI in OFED: > >- All participants test with the same MPI versions, and when > installing > >OFED it is ensured that MPI will work fine with this version. > >- Customers convenience in install (no need to go to more sites to get > MPI) > >- MPI is an important RDMA ULP and although it is not developed in OFA > >it is widely used by OFED customers > > I support keeping MPI packages in the OFED because of the above > positive > points you have mentioned. > > I would also like to mention that keeping MPI packages in OFED helps to > test out various new features and functionalities (such as APM and XRC > in > the past and the new memory registration scheme being discussed now) as > they get introduced. Such an integrated approach helps to test out > these > features at the lower layers as well as at the MPI layer. This process > helps to resolve out any bugs with the new features during the testing > process itself. It also accelerates the deployment and use of these new > features in the community. > > However, to make the complete OFED release process work smoothly for > everybody (vendors, distros, users, etc.) without affecting the release > schedule, it is essential that stable MPI packages are added to OFED. > This > is what we have been doing wrt MVAPICH and MVAPICH2 for the last > several > years. > > If the developers of any MPI package do not want it to be a part of the > OFED due to any constraints, it should be allowed. However, such an > action > should not force to remove all MPI packages. > > >From the point of view of MVAPICH and MVAPICH2 packages in OFED, we > have > been providing stable packages to OFED for the last several years > helping > the OFED community and would like to continue with this process. > > Thanks, > > DK > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general From jackm at dev.mellanox.co.il Mon Jun 8 07:53:44 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 8 Jun 2009 17:53:44 +0300 Subject: [ofa-general] Re: [PATCH] Fixed memory leak in drivers/net/mlx4/main.c In-Reply-To: <4A2CD143.6050506@ext.bull.net> References: <4A2CD143.6050506@ext.bull.net> Message-ID: <200906081753.44464.jackm@dev.mellanox.co.il> OFED 1.5 is still based on 2.6.30-rc2. If this patch in is 2.6.30-rc8, we will grab it from the mainstream within the next couple of days (when we rebase to that RC). (For that reason, I'm not checking this in as a patch right now). -Jack On Monday 08 June 2009 11:52, Nicolas Morey-Chaisemartin wrote: > > Signed-off-by: Nicolas Morey-Chaisemartin > --- > This patch made it into linux mainstream but seems to have bit lost in the ofed 1.5 branch. > drivers/net/mlx4/main.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c > index 102bac9..30bea96 100644 > --- a/drivers/net/mlx4/main.c > +++ b/drivers/net/mlx4/main.c > @@ -976,7 +976,7 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev) > nreq = err; > goto retry; > } > - > + kfree(entries); > goto no_msi; > } > From cap at nsc.liu.se Mon Jun 8 08:07:39 2009 From: cap at nsc.liu.se (Peter Kjellstrom) Date: Mon, 8 Jun 2009 17:07:39 +0200 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A2B75D4.7080600@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il> Message-ID: <200906081707.43472.cap@nsc.liu.se> On Sunday 07 June 2009, Pavel Shamis (Pasha) wrote: ... > > - MPI is an important RDMA ULP and although it is not developed in OFA > > it is widely used by OFED customers > > I guess that MPI users are the widest OFED user community and including > MPI as part of OFED definitely simplify > end-user live. I've heard this argument more than once now and I don't agree. MPI installation is at our site de-coupled from the OFED installation (for quite a few reasons such as, different compilers, multiple versions, RM-integration, ...). I'm sure there are quite a few that do use the OFED shipped MPIs but I'm equally certain I'm far from alone in building MPI on the side. One vote for keeping MPI out of OFED, Petere > I'm personally against excluding MPI from OFED package. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From nicolas.morey-chaisemartin at ext.bull.net Mon Jun 8 08:08:38 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey-Chaisemartin) Date: Mon, 08 Jun 2009 17:08:38 +0200 Subject: [ofa-general] Re: [PATCH] Fixed memory leak in drivers/net/mlx4/main.c In-Reply-To: <200906081753.44464.jackm@dev.mellanox.co.il> References: <4A2CD143.6050506@ext.bull.net> <200906081753.44464.jackm@dev.mellanox.co.il> Message-ID: <4A2D2976.9080302@ext.bull.net> Le 08/06/2009 16:53, Jack Morgenstein a écrit : > OFED 1.5 is still based on 2.6.30-rc2. If this patch in is 2.6.30-rc8, > we will grab it from the mainstream within the next couple of days (when we rebase to that RC). > (For that reason, I'm not checking this in as a patch right now). > > -Jack > Ok no problem. I'm still having difficulties to understand how mainstream code and ofed code interacts. And FYI, branch builds fine on FC11. Only problem left is NFS RDMA not building on 2.6.29 due to patches related to fscache.c. Nicolas From worleys at gmail.com Mon Jun 8 08:39:23 2009 From: worleys at gmail.com (Chris Worley) Date: Mon, 8 Jun 2009 09:39:23 -0600 Subject: [ofa-general] How does ib_srpt decide which ports to use? In-Reply-To: References: Message-ID: On Sat, Jun 6, 2009 at 11:40 AM, Bart Van Assche wrote: > On Sat, Jun 6, 2009 at 7:27 PM, Chris Worley wrote: > > > > On Sat, Jun 6, 2009 at 1:36 AM, Bart Van Assche > > wrote: > > > On Sat, Jun 6, 2009 at 1:15 AM, Chris Worley wrote: > > >> Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports > > >> direct connected, opensm running on all port GUIDs from one host, all > > >> links active. > > >> > > >> Problem: ibsrpdm only advertises the first port of the first HCA of > the target. > > >> Next problem: I can add targets via > > >> /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only > > >> when naming the two port guids of the first HCA on the target. In > > >> testing, both ports are used. > > >> > > >> Can somebody aim me in the right direction of what/who's stopping > > >> after the first HCA? > > > > > > Please have a look at the /sys/class/infiniband_srpt/srpt-*/login_info > > > information on the target. The following information should be > > > present: > > > * One /sys/class/infiniband_srpt/srpt-* entry per HCA. > > > * For each HCA, /sys/class/infiniband_srpt/srpt-${HCA}/login_info > > > should contain one line for each port of that HCA. > > > > # cat /sys/class/infiniband_srpt/srpt-*/login_info > > > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000041,service_id=0024710000000040 > > > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000042,service_id=0024710000000040 > > > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000045,service_id=0024710000000040 > > > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000024710000000046,service_id=0024710000000040 > > > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000002c903000292af,service_id=0024710000000040 > > > tid_ext=0024710000000040,ioc_guid=0024710000000040,pkey=ffff,dgid=fe800000000000000002c903000292b0,service_id=0024710000000040 > > > > Each port has an entry, and the port GUIDs are correct (dgid's), but > > the rest of the GUIDs refer to the node GUID of the first IB HCA: > > 0024710000000040. > > > > Is that expected? > > Yes. The ioc_guid in the above output is a GUID that identifies the > SRP target. A quote from the ib_srpt source code: > > /* > * We do not have a consistent service_id (ie. also id_ext of target_id) > * to identify this target. We currently use the guid of the first HCA > * in the system as service_id; therefore, the target_id will change > * if this HCA is gone bad and replaced by different HCA. > */ > > I'm not sure however what ibsrpdm should display -- I don't know > whether it should display one single set of login parameters or all > possible login parameters. > That works! Thanks! I've verified that I can control which ports are being used (and get multiple ports in use simultaneously) using the above approach. This gives SRP a great performance edge over iSer... as I've found no way to effect which ports are used w/ iSer. Thanks, Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From worleys at gmail.com Mon Jun 8 08:41:58 2009 From: worleys at gmail.com (Chris Worley) Date: Mon, 8 Jun 2009 09:41:58 -0600 Subject: [ofa-general] How does ib_srpt decide which ports to use? In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB43E990A6D9@MNEXMB1.qlogic.org> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E990A6D9@MNEXMB1.qlogic.org> Message-ID: On Sun, Jun 7, 2009 at 10:59 PM, Karun Sharma (Contractor - GS Labs) < karun.sharma at qlogic.com> wrote: > By default, ibsrpdm will use umad0. To discover the target using umad1, use > "ibsrpdm -d /dev/umad1" Good tip. Thanks! Chris > > > Regards > Karun > > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto: > general-bounces at lists.openfabrics.org] On Behalf Of Chris Worley > Sent: Saturday, June 06, 2009 4:46 AM > To: OpenIB; scst-devel > Subject: [ofa-general] How does ib_srpt decide which ports to use? > > Setup: 1.4.1 w/ 3 dual-port QDR cards in each of two hosts, all ports > direct connected, opensm running on all port GUIDs from one host, all > links active. > > Problem: ibsrpdm only advertises the first port of the first HCA of the > target. > Next problem: I can add targets via > /sys/class/infiniband_srp/srp-*/add_target on the initiator, but only > when naming the two port guids of the first HCA on the target. In > testing, both ports are used. > > Can somebody aim me in the right direction of what/who's stopping > after the first HCA? > > Thanks, > > Chris > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Mon Jun 8 09:28:59 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 8 Jun 2009 09:28:59 -0700 Subject: [ofa-general] Memory registration redux In-Reply-To: <928CFBE8E7CB0040959E56B4EA41A77EBBC092A7@irsmsx504.ger.corp.intel.com> References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> <5019F239-149F-49E1-8C23-436DE6094AB2@cisco.com> <928CFBE8E7CB0040959E56B4EA41A77EBBB752D6@irsmsx504.ger.corp.intel.com> <928CFBE8E7CB0040959E56B4EA41A77EBBC092A7@irsmsx504.ger.corp.intel.com> Message-ID: >Are there any comparable Windows plans? I believe that Windows already provides an equivalent functionality as part of the OS (Windows 2008 / Vista). See SecureMemoryCacheCallback. There are no plans for WinOF to provide anything separately from this. (It's likely impossible without OS support anyway.) - Sean From worleys at gmail.com Mon Jun 8 12:38:17 2009 From: worleys at gmail.com (Chris Worley) Date: Mon, 8 Jun 2009 13:38:17 -0600 Subject: [ofa-general] srp_daemon and partitions. In-Reply-To: <4925B52F.3030106@mellanox.com> References: <774A4005-446E-40D1-A70E-DBCBF12219F0@catbus.co.uk> <4925B52F.3030106@mellanox.com> Message-ID: On Thu, Nov 20, 2008 at 1:06 PM, Vu Pham wrote: > > Hi James, > > it's srp_daemon and ibsrpdm bug. We'll try to fix it to provide zoning thru pkey. I don't think pkeys are the answer to zoning, as I don't see a way to tie a specific disk or partition to a pkey.   At one point I tried to tie IB ports to pkeys, but found that all levels of the SRP stack needed to be pkey-aware.  It looks like a daunting task, and probably not what pkeys were intended to do. I think SCST "security groups" are the intended way to zone.  The scst/README says: ...   2. Initiator-oriented. In this mode you define which devices and their LUNs are accessible for each initiator. In this mode you should createfor each set of one or more initiators, which should access to the same set of devices with the same LUNs, a separate security group, then add to it available devices and names of allowed initiator(s). ... # echo "add_group Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz" >/proc/scsi_tgt/scsi_tgt # echo "add dev1 0" >/proc/scsi_tgt/groups/Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz/devices # echo "add dev2 1" >/proc/scsi_tgt/groups/Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz/devices # echo "add_group spec_ini" >/proc/scsi_tgt/scsi_tgt # echo "add iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" >/proc/scsi_tgt/groups/spec_ini/names # echo "add dev2 0" >/proc/scsi_tgt/groups/spec_ini/devices ...  But, I don't  understand how the zoning example selects the initiator visibility... I'd hope w/ IB this could be done w/ the port GUID of the initiator.  The example uses the name of "iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" to specify the initiator.  I'm guessing "iqn.2007-05.com.example" specifies the host name of the initiator, but not a clue what "storage.disk1.spec_ini.xyz" means. Chris From vlad at lists.openfabrics.org Tue Jun 9 03:29:33 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 9 Jun 2009 03:29:33 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090609-0200 daily build status Message-ID: <20090609102933.D4BFDE615DE@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From jsquyres at cisco.com Tue Jun 9 04:38:05 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 9 Jun 2009 04:38:05 -0700 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <5AEC2602AE03EB46BFC16C6B9B200DA81455DD0350@MNEXMB2.qlogic.org> References: <4A257DBF.4000401@mellanox.co.il> <5AEC2602AE03EB46BFC16C6B9B200DA81455DD0350@MNEXMB2.qlogic.org> Message-ID: On Jun 8, 2009, at 6:59 AM, Todd Rimmer wrote: > I agree with DK from OSU. There are clear advantages to having MPI > included with OFED. Not only will it make testing of a complete > solution easier by both OFED and MPI suppliers, > Can you specify how, specifically? Remember that all that Open MPI and MVAPICH do is provide SRPMs. There is no co-mingling of development / source trees, for example. You seem to be blurring the distinction between co-development of MPI +OpenFabrics and shipping OFED. Developing the two together is a Good Thing -- and that happens. But that is unrelated to shipping the MPI's in OFED. As has been specified multiple times on this thread, using MPI to test verbs is a Good Thing and it can easily be maintained without distributing MPI in OFED. > but it will also improve ease of use for end users. > Can you specify how, specifically? Recall that: - Open MPI users get a stripped-down version with several important features disabled - At least one user has chimed in that they install MPI separately from OFED for a variety of reasons (I have seen this at customer sites as well) > As DK points out there are continual improvements in MPIs which may > depend on bug fixes and/or new features in newer versions of OFED. > Identifying a known good combination will be important to most end > users, etc. > Easy to do in documentation and/or in the technology of the MPI implementations themselves. The verbs API should allow this kind of run-time checking as a matter of course (and it seems to allow it well enough). Additionally -- and your later comments seemed to support it -- possibly the most important combination that needs to work is that of + , which will continue to work because OFED would be insane to remove MPI from its testing/QA/release process. -- Jeff Squyres Cisco Systems From pashash at gmail.com Tue Jun 9 06:49:44 2009 From: pashash at gmail.com (Pavel Shamis (Pasha)) Date: Tue, 09 Jun 2009 16:49:44 +0300 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <200906081707.43472.cap@nsc.liu.se> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il> <200906081707.43472.cap@nsc.liu.se> Message-ID: <4A2E6878.6080200@dev.mellanox.co.il> >>> - MPI is an important RDMA ULP and although it is not developed in OFA >>> it is widely used by OFED customers >>> >> I guess that MPI users are the widest OFED user community and including >> MPI as part of OFED definitely simplify >> end-user live. >> > > I've heard this argument more than once now and I don't agree. MPI > installation is at our site de-coupled from the OFED installation (for quite > a few reasons such as, different compilers, multiple versions, > RM-integration, ...). > > I'm sure there are quite a few that do use the OFED shipped MPIs but I'm > equally certain I'm far from alone in building MPI on the side. > We do not force user to use the only MPI that is shipped with OFED. You may use your own customization build and it is ok. But I do not see how including MPI as part of OFED make the end user installation more complicated or problematic. The excluding of MPI from OFED, will only push users to use vendor specific OFED builds (that provides MPI out of box) and I'm not sure that it is good for OFED community. Pasha. From truelove at array.ca Tue Jun 9 07:37:24 2009 From: truelove at array.ca (Steven Truelove) Date: Tue, 09 Jun 2009 10:37:24 -0400 Subject: [ofa-general] Set ports to SDR speed Message-ID: <4A2E73A4.1020500@array.ca> Hi, I would like to force all my nodes to run at 4x SDR (10Gbps) speed. Using OpenSM, I understand I should get the option force_link_speed to 1. I'm just not clear where I do this. I have tried to find some documentation mentioning this option with no luck -- I only know about it from an e-mail on this list. Thanks, Steven Truelove -- Steven Truelove Array Systems Computing, Inc. 1120 Finch Avenue West, 7th Floor Toronto, Ontario M3J 3H7 CANADA http://www.array.ca truelove at array.ca Phone: (416) 736-0900 x307 Fax: (416) 736-4715 From hal.rosenstock at gmail.com Tue Jun 9 07:45:25 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 9 Jun 2009 10:45:25 -0400 Subject: [ofa-general] Set ports to SDR speed In-Reply-To: <4A2E73A4.1020500@array.ca> References: <4A2E73A4.1020500@array.ca> Message-ID: On Tue, Jun 9, 2009 at 10:37 AM, Steven Truelove wrote: > Hi, > >   I would like to force all my nodes to run at 4x SDR (10Gbps) speed.  Using > OpenSM, I understand I should get the option force_link_speed to 1.  I'm > just not clear where I do this.  I have tried to find some documentation > mentioning this option with no luck -- I only know about it from an e-mail > on this list. It's found in the OpenSM options file. -- Hal > > Thanks, > > Steven Truelove > > -- > Steven Truelove > Array Systems Computing, Inc. > 1120 Finch Avenue West, 7th Floor > Toronto, Ontario > M3J 3H7 > CANADA > http://www.array.ca > truelove at array.ca > Phone: (416) 736-0900 x307 > Fax: (416) 736-4715 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From gus at ldeo.columbia.edu Tue Jun 9 09:01:00 2009 From: gus at ldeo.columbia.edu (Gus Correa) Date: Tue, 09 Jun 2009 12:01:00 -0400 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: <4A257DBF.4000401@mellanox.co.il> <5AEC2602AE03EB46BFC16C6B9B200DA81455DD0350@MNEXMB2.qlogic.org> Message-ID: <4A2E873C.5090109@ldeo.columbia.edu> Hi OFA list I never post to this list, but I read it. I am just reporting how we use MPI here, in case this matters. I am not trying to flare any tempers. I administer (and use) a cluster with Infiniband and GigE networks. We use OFED, OpenMPI and MVAPICH2 (and even MPICH2 on our GigE network). We don't use the MPI versions that come bundled with OFED, not even for tests. We don't use other MPIs (TCP/IP based) that come with Linux distributions or compiler packages, either. The reasons are the same in both cases. We prefer to stay up to date with the latest stable OpenMPI and MVAPICH2 releases (and MPICH2 also). Moreover, due to compiler incompatibilities and specific application requirements (some only compile and run with certain compilers, or run faster with certain compilers), we tend to build all MPIs with a few compilers and compiler combinations. We install these builds on specific (non standard) directories on our system, managing the user environment with the environment modules package. That is what is most convenient and rational for us. Correct me if I am wrong, please, but my perception is that much larger Linux clusters (e.g. TACC Ranger, etc), have a similar setup. My $0.02. Thank you, Gus Correa --------------------------------------------------------------------- Gustavo Correa Lamont-Doherty Earth Observatory - Columbia University Palisades, NY, 10964-8000 - USA --------------------------------------------------------------------- Jeff Squyres wrote: > On Jun 8, 2009, at 6:59 AM, Todd Rimmer wrote: > >> I agree with DK from OSU. There are clear advantages to having MPI >> included with OFED. Not only will it make testing of a complete >> solution easier by both OFED and MPI suppliers, >> > > Can you specify how, specifically? > > Remember that all that Open MPI and MVAPICH do is provide SRPMs. There > is no co-mingling of development / source trees, for example. You seem > to be blurring the distinction between co-development of MPI+OpenFabrics > and shipping OFED. Developing the two together is a Good Thing -- and > that happens. But that is unrelated to shipping the MPI's in OFED. > > As has been specified multiple times on this thread, using MPI to test > verbs is a Good Thing and it can easily be maintained without > distributing MPI in OFED. > >> but it will also improve ease of use for end users. >> > > Can you specify how, specifically? > > Recall that: > > - Open MPI users get a stripped-down version with several important > features disabled > - At least one user has chimed in that they install MPI separately from > OFED for a variety of reasons (I have seen this at customer sites as well) > >> As DK points out there are continual improvements in MPIs which may >> depend on bug fixes and/or new features in newer versions of OFED. >> Identifying a known good combination will be important to most end >> users, etc. >> > > Easy to do in documentation and/or in the technology of the MPI > implementations themselves. The verbs API should allow this kind of > run-time checking as a matter of course (and it seems to allow it well > enough). > > Additionally -- and your later comments seemed to support it -- possibly > the most important combination that needs to work is that of OFED> + , which will continue to work because OFED would be > insane to remove MPI from its testing/QA/release process. > From rdreier at cisco.com Tue Jun 9 13:46:23 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 09 Jun 2009 13:46:23 -0700 Subject: [ofa-general][PATCH] mlx4: FIX error flow when initializing EQ table In-Reply-To: <4A2BD9B9.6000802@mellanox.co.il> (Yevgeny Petrilin's message of "Sun, 07 Jun 2009 18:16:09 +0300") References: <4A2BD9B9.6000802@mellanox.co.il> Message-ID: > If mlx4_create_eq() would fail for one of EQ's assigned for > completion handling, the code would try to free the same EQ > we failed to create. > The crash was found by Christoph Lameter Thanks, good catch. However looking at the code it seems that a simpler way to fix this would be as below. Can you confirm that this works too? --- drivers/net/mlx4/eq.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/net/mlx4/eq.c b/drivers/net/mlx4/eq.c index 8830dcb..033817e 100644 --- a/drivers/net/mlx4/eq.c +++ b/drivers/net/mlx4/eq.c @@ -677,12 +677,12 @@ err_out_async: mlx4_free_eq(dev, &priv->eq_table.eq[dev->caps.num_comp_vectors]); err_out_comp: - i = dev->caps.num_comp_vectors - 1; + i = dev->caps.num_comp_vectors; err_out_unmap: - while (i >= 0) { - mlx4_free_eq(dev, &priv->eq_table.eq[i]); + while (i > 0) { --i; + mlx4_free_eq(dev, &priv->eq_table.eq[i]); } mlx4_unmap_clr_int(dev); mlx4_free_irqs(dev); From worleys at gmail.com Tue Jun 9 14:05:40 2009 From: worleys at gmail.com (Chris Worley) Date: Tue, 9 Jun 2009 15:05:40 -0600 Subject: [ofa-general] Coalescing interrupts in SRP Message-ID: I'm not sure if the SRP stack (initiator side) polls or interrupts, but I seem to be hitting a wall at ~120K transactions/second. If there's interrupt coalescing available, this might help. I realize this is antithetical to most IB use cases. Is there any way to coalesce interrupts w/ SRP? Thanks, Chris From Zhenyi.Huang at hp.com Tue Jun 9 15:24:17 2009 From: Zhenyi.Huang at hp.com (Huang, Zhenyi) Date: Tue, 9 Jun 2009 22:24:17 +0000 Subject: [ofa-general] Howto build static libraries Message-ID: <8FD93B8358C0EB43AAB68265DA71DC7A6EB1CA8796@GVW0433EXB.americas.hpqcorp.net> Hi folks, I installed OFED-1.4.1-20090527-0600 via install.pl. Things were fine with my Chelsio 10Gb PCI adaptor. So, I wanted to test build and install packages such as libibverbs. I followed instructions in OFED_tips.txt: Example for libibverbs: 1. tar xzf OFED-1.4.tgz 2. rpm -ihv SRPMS/libibverbs-1.1.2-1.ofed1.4.src.rpm 3. cd /usr/src/redhat/SOURCES (for RedHat) or cd /usr/src/packages/SOURCES (for SuSE) 4. tar xzf libibverbs-1.1.2.tgz 5. cd libibverbs-1.1.2 6. ./configure (specify parameters, if required) 7. make 8. make install Now, I get the following error: ibv_devices libibverbs: Warning: couldn't open config directory '/usr/local/etc/libibverbs.d'. libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1 device node GUID ------ ---------------- I'm sure I'm making a newbie mistake here.... Hopefuly someone can point me in the right direction. Thanks, Zhen ------------------------------------------------------------------ Zhenyi Huang Hewlett-Packard Company, NonStop Enterprise Division Cupertino, CA From rdreier at cisco.com Tue Jun 9 15:55:20 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 09 Jun 2009 15:55:20 -0700 Subject: [ofa-general] Coalescing interrupts in SRP In-Reply-To: (Chris Worley's message of "Tue, 9 Jun 2009 15:05:40 -0600") References: Message-ID: > I'm not sure if the SRP stack (initiator side) polls or interrupts, > but I seem to be hitting a wall at ~120K transactions/second. Interesting. The SRP initiator is using interrupts to call srp_completion(), which then services all pending completions and rearms the CQ. So it is essentially interrupt driven -- there is no polling mechanism a la NAPI for network devices. > Is there any way to coalesce interrupts w/ SRP? Not at the moment, although one could do a hack with ib_modify_cq() a la IPoIB (assuming you're using ConnectX HCA). A bigger piece of work would be some sort of NAPI framework -- I think people have alreay looked at "NAPI for blockdevices" as part of work to optimize for high-IOPS SSDs, but I don't have pointers at the moment. - R. From worleys at gmail.com Tue Jun 9 16:36:19 2009 From: worleys at gmail.com (Chris Worley) Date: Tue, 9 Jun 2009 17:36:19 -0600 Subject: [ofa-general] Coalescing interrupts in SRP In-Reply-To: References: Message-ID: On Tue, Jun 9, 2009 at 4:55 PM, Roland Dreier wrote: > >  > I'm not sure if the SRP stack (initiator side) polls or interrupts, >  > but I seem to be hitting a wall at ~120K transactions/second. > > Interesting.  The SRP initiator is using interrupts to call > srp_completion(), which then services all pending completions and rearms > the CQ. Does the "ib_req_notify_cq" call do the rearm? If so, does that mean that until it rearms, the interrupt is disabled? If so, shouldn't a delay before the rearm (after clearing the current queue) in effect coalesce? Chris From davem at systemfabricworks.com Tue Jun 9 20:07:01 2009 From: davem at systemfabricworks.com (David McMillen) Date: Tue, 09 Jun 2009 22:07:01 -0500 Subject: [ofa-general] srp_daemon and partitions. In-Reply-To: <4A2F1E2D.1050205@systemfabricworks.com> References: <4A2F1E2D.1050205@systemfabricworks.com> Message-ID: <4A2F2355.50806@systemfabricworks.com> Chris Worley wrote: >On Thu, Nov 20, 2008 at 1:06 PM, Vu Pham > wrote: >>/ >/>/ Hi James, >/>/ >/>/ it's srp_daemon and ibsrpdm bug. We'll try to fix it to provide zoning thru pkey. >/ >I don't think pkeys are the answer to zoning, as I don't see a way to >tie a specific disk or partition to a pkey. At one point I tried to >tie IB ports to pkeys, but found that all levels of the SRP stack >needed to be pkey-aware. It looks like a daunting task, and probably >not what pkeys were intended to do. > >I think SCST "security groups" are the intended way to zone. The >scst/README says: > >... > 2. Initiator-oriented. In this mode you define which devices and >their LUNs are accessible for each initiator. In this mode you should >createfor each set of one or more initiators, which should access to >the same set of devices with the same LUNs, a separate security group, >then add to it available devices and names of allowed initiator(s). >... ># echo "add_group >Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz" >>//proc/scsi_tgt/scsi_tgt >/# echo "add dev1 0" >>//proc/scsi_tgt/groups/Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz/devices >/# echo "add dev2 1" >>//proc/scsi_tgt/groups/Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz/devices >/ ># echo "add_group spec_ini" >/proc/scsi_tgt/scsi_tgt ># echo "add iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" >>//proc/scsi_tgt/groups/spec_ini/names >/# echo "add dev2 0" >/proc/scsi_tgt/groups/spec_ini/devices >... > > But, I don't understand how the zoning example selects the initiator >visibility... I'd hope w/ IB this could be done w/ the port GUID of >the initiator. The example uses the name of >"iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" to specify the >initiator. I'm guessing "iqn.2007-05.com.example" specifies the host >name of the initiator, but not a clue what >"storage.disk1.spec_ini.xyz" means. > >Chris > You got pretty close. I think there is at least one typographical error in the explanation above, but let me try to explain it differently. The groups are used to create different sets of LUNs that are presented to the initiators that match a group. When a login request comes in, scst is asked to find the group given a name. You can get scst to output a message like Using security group "Default" for initiator "xxx" on the console. The value in "xxx" (not really xxx, but dependent upon the target code using scst) is matched against all of the entries in the "names" pseudo file under each group's directory. The first group that matches is the winner. The LUNs defined by the entries in the "devices" pseudo file under that same group will be the only LUNs seen for that session. If no group has a line in "names" that matches, then the group "Default" is used. There is a "names" pseudo file here but it's contents are ignored -- everybody is a winner matching the Default group. The name of the group is completely arbitrary and meaningless to scst other than to differentiate groups. If you intend to have a unique group for every initiator, then you could use the same value you will write to "names" as the name of the group, but you could just as easily just generate random names that are unique. If you want a particular LUN to show up in more than one group, you have to add it to "devices" for every group that will use it. Also, if you want that LUN to have the same LUN number, you have to arrange that as well. It is probably worth noting that there can be problems if a group does not have a device defined for LUN 0. It works, but there are initiators and user-level programs that issue SCSI commands and expect to get something when LUN 0 is addressed. All real devices have some kind of response at LUN 0, even if it is an odd SCSI device or a zero byte / offline disk. Back to the "xxx" mentioned above, I don't know what the code you have uses as the key for scst to find a group. That is why I suggested you check your dmesg output, and you might have to enable some verbose/debug output. I believe this changed at least once, and in our implementation we are reworking it so we can have highly selective LUN masking against the initiator/target pair. Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From yevgenyp at mellanox.co.il Tue Jun 9 23:52:26 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Wed, 10 Jun 2009 09:52:26 +0300 Subject: [ofa-general][PATCH] mlx4: FIX error flow when initializing EQ table In-Reply-To: References: <4A2BD9B9.6000802@mellanox.co.il> Message-ID: <4A2F582A.7050106@mellanox.co.il> Roland Dreier wrote: > > If mlx4_create_eq() would fail for one of EQ's assigned for > > completion handling, the code would try to free the same EQ > > we failed to create. > > The crash was found by Christoph Lameter > > Thanks, good catch. However looking at the code it seems that a simpler > way to fix this would be as below. Can you confirm that this works too? > --- > drivers/net/mlx4/eq.c | 6 +++--- > 1 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/drivers/net/mlx4/eq.c b/drivers/net/mlx4/eq.c > index 8830dcb..033817e 100644 > --- a/drivers/net/mlx4/eq.c > +++ b/drivers/net/mlx4/eq.c > @@ -677,12 +677,12 @@ err_out_async: > mlx4_free_eq(dev, &priv->eq_table.eq[dev->caps.num_comp_vectors]); > > err_out_comp: > - i = dev->caps.num_comp_vectors - 1; > + i = dev->caps.num_comp_vectors; > > err_out_unmap: > - while (i >= 0) { > - mlx4_free_eq(dev, &priv->eq_table.eq[i]); > + while (i > 0) { > --i; > + mlx4_free_eq(dev, &priv->eq_table.eq[i]); > } > mlx4_unmap_clr_int(dev); > mlx4_free_irqs(dev); > Yes, it also works Yevgeny From ogerlitz at Voltaire.com Tue Jun 9 23:56:48 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Wed, 10 Jun 2009 09:56:48 +0300 Subject: [ofa-general] Coalescing interrupts in SRP In-Reply-To: References: Message-ID: <4A2F5930.30802@Voltaire.com> Roland Dreier wrote: >> Is there any way to coalesce interrupts w/ SRP? > one could do a hack with ib_modify_cq() a la IPoIB (assuming you're using ConnectX HCA). Hi Roland, I wonder why you consider this a hack... is programming interrupt coalescing uncommon with HBAs? with networking I understand that the common practice is to apply both HW mitigation and "SW mitigation" (NAPI) so with SSD IOPS rates peaking to hundreds of thousands I assume such an approach is valid for disks as well. Or. From jackm at dev.mellanox.co.il Wed Jun 10 00:14:00 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Wed, 10 Jun 2009 10:14:00 +0300 Subject: [ofa-general] Re: [PATCH] Fixed memory leak =?iso-8859-1?q?in=09drivers/net/mlx4/main=2Ec?= In-Reply-To: <4A2D2976.9080302@ext.bull.net> References: <4A2CD143.6050506@ext.bull.net> <200906081753.44464.jackm@dev.mellanox.co.il> <4A2D2976.9080302@ext.bull.net> Message-ID: <200906101014.00741.jackm@dev.mellanox.co.il> On Monday 08 June 2009 18:08, Nicolas Morey-Chaisemartin wrote: > I'm still having difficulties to understand how mainstream code and ofed code interacts. > The base kernel files (in this case, 2.6.30) are taken unmodified into OFED. Adjustments (patches) to the base kernel are placed in directory kernel_patches/fixes. Further adjustments, for earlier kernels, are placed in kernel_patches/backports/. Each kernel_patches/backports/ is independent. When building for a given kernel, first the "fixes" patches are applied, then the "backport" patches for that kernel. In addition, there is also the "kernel_addons" directory, which contains, per backport, adjustments to include files so that we can perform fewer changes to the code when backporting. The include files under "kernel_addons" are taken first when compiling. -Jack From vlad at lists.openfabrics.org Wed Jun 10 03:26:26 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 10 Jun 2009 03:26:26 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090610-0200 daily build status Message-ID: <20090610102626.AB8A1E6167D@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From zafargilani at gmail.com Wed Jun 10 07:07:57 2009 From: zafargilani at gmail.com (Zafar Gilani) Date: Wed, 10 Jun 2009 19:07:57 +0500 Subject: [ofa-general] SDP - How to? Message-ID: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> Hi, I read a paper on "Zero Copy Sockets Direct Protocol over InfiniBand". I had a few questions, if anybody could answer I will be thankful. 1. SDP BCopy approach can be used without the change of the code via the use of AF_INET_SDP parameter when calling socket() system call. But how can I implement SDP ZCopy approach if I want to implement ZCopy_Read and ZCopy_Write methods? 2. Is there any documentation regarding how to implement SDP ZCopy? Any Hello World sort of example code that I can use to learn how to work with this? 3. SDP BCopy is better for smaller messages, but how small? Is 64K a good threshold to change from BCopy to ZCopy at 128K? Looking forward for your reply. Best Regards, -- Syed Zafar ul Hussan Gilani | BIT-7 Research Student | CHPSC MSP 2008-09 NUST SEECS | http://hpc.niit.edu.pk/~zafar -------------- next part -------------- An HTML attachment was scrubbed... URL: From Line.Holen at Sun.COM Wed Jun 10 07:14:33 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Wed, 10 Jun 2009 16:14:33 +0200 Subject: [ofa-general] [PATCH] osm_sa_patch_record.c Remove unneeded break statement Message-ID: <4A2FBFC9.7010003@Sun.COM> Signed-off-by: Line Holen --- diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 6e7d5f6..9b50deb 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -140,7 +140,6 @@ sa_path_rec_apply_tavor_mtu_limit(IN const ib_path_rec_t * p_pr, /* the ULP intentionally requested */ /* the largest MTU possible */ return (FALSE); - break; default: /* if we're here, there's a bug in ib_path_rec_mtu_sel() */ From hnrose at comcast.net Wed Jun 10 07:15:10 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 10 Jun 2009 10:15:10 -0400 Subject: [ofa-general] [PATCH] perftest: Add command line SL support Message-ID: <20090610141509.GA23673@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/rdma_bw.c b/rdma_bw.c index 0a6c6ac..2628ac4 100755 --- a/rdma_bw.c +++ b/rdma_bw.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -59,6 +60,7 @@ #define PINGPONG_RDMA_WRID 3 +static int sl = 0; static int page_size; static pid_t pid; @@ -651,7 +653,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, struct pp_data data) attr.min_rnr_timer = 12; attr.ah_attr.is_global = 0; attr.ah_attr.dlid = data.rem_dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = data.ib_port; if (ibv_modify_qp(ctx->qp, &attr, @@ -858,6 +860,7 @@ static void usage(const char *argv0) printf(" -s, --size= size of message to exchange (default 65536)\n"); printf(" -t, --tx-depth= size of tx queue (default 100)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); printf(" -c, --cma use RDMA CM\n"); } @@ -943,12 +946,13 @@ int main(int argc, char *argv[]) { .name = "size", .has_arg = 1, .val = 's' }, { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, { .name = "cma", .has_arg = 0, .val = 'c' }, { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:n:t:bc", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:n:t:S:bc", long_options, NULL); if (c == -1) break; @@ -995,6 +999,11 @@ int main(int argc, char *argv[]) break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 1; } + break; + case 'b': duplex = 1; break; @@ -1021,9 +1030,9 @@ int main(int argc, char *argv[]) */ pid = getpid(); - printf("%d: | port=%d | ib_port=%d | size=%d | tx_depth=%d | iters=%d | duplex=%d | cma=%d |\n", + printf("%d: | port=%d | ib_port=%d | size=%d | tx_depth=%d | sl=%d | iters=%d | duplex=%d | cma=%d |\n", pid, data.port, data.ib_port, data.size, data.tx_depth, - iters, duplex, data.use_cma); + sl, iters, duplex, data.use_cma); /* Done with parameter parsing. Perform setup. */ diff --git a/rdma_lat.c b/rdma_lat.c index 09a5f78..3681b35 100755 --- a/rdma_lat.c +++ b/rdma_lat.c @@ -2,6 +2,7 @@ * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * Copyright (c) 2005 Hewlett Packard, Inc (Grant Grundler) + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -61,6 +62,7 @@ #define MAX_INLINE 400 static int inline_size = MAX_INLINE; +static int sl = 0; static int page_size; static pid_t pid; @@ -656,7 +658,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, struct pp_data *data) .min_rnr_timer = 12, .ah_attr.is_global = 0, .ah_attr.dlid = data->rem_dest->lid, - .ah_attr.sl = 0, + .ah_attr.sl = sl, .ah_attr.src_path_bits = 0, .ah_attr.port_num = data->ib_port }; @@ -921,6 +923,7 @@ static void usage(const char *argv0) printf(" -s, --size= size of message to exchange (default 1)\n"); printf(" -t, --tx-depth= size of tx queue (default 50)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -I, --inline_size= max size of message to be sent in inline mode (default 400)\n"); printf(" -C, --report-cycles report times in cpu cycle units (default microseconds)\n"); printf(" -H, --report-histogram print out all results (default print summary only)\n"); @@ -1043,6 +1046,7 @@ int main(int argc, char *argv[]) { .name = "size", .has_arg = 1, .val = 's' }, { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "inline_size", .has_arg = 1, .val = 'I' }, { .name = "report-cycles", .has_arg = 0, .val = 'C' }, { .name = "report-histogram",.has_arg = 0, .val = 'H' }, @@ -1051,7 +1055,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:n:t:I:CHUc", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:n:t:S:I:CHUc", long_options, NULL); if (c == -1) break; @@ -1094,6 +1098,11 @@ int main(int argc, char *argv[]) } break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 6; } + break; + case 'I': inline_size = strtol(optarg, NULL, 0); break; @@ -1116,7 +1125,7 @@ int main(int argc, char *argv[]) default: usage(argv[0]); - return 5; + return 7; } } diff --git a/read_bw.c b/read_bw.c index 8c3d553..6e0fa1d 100755 --- a/read_bw.c +++ b/read_bw.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2006 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -71,6 +72,7 @@ struct user_parameters { int use_event; int qp_timeout; }; +static int sl = 0; static int page_size; cycles_t *tposted; cycles_t *tcompleted; @@ -421,7 +423,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, attr.min_rnr_timer = 12; attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; if (ibv_modify_qp(ctx->qp, &attr, @@ -471,6 +473,7 @@ static void usage(const char *argv0) printf(" -t, --tx-depth= size of tx queue (default 100)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); printf(" -V, --version display version number\n"); printf(" -e, --events sleep on CQ events (default poll)\n"); @@ -630,6 +633,7 @@ int main(int argc, char *argv[]) { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, { .name = "version", .has_arg = 0, .val = 'V' }, @@ -638,7 +642,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:d:i:m:o:s:n:t:u:abVeF", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:m:o:s:n:t:u:S:abVeF", long_options, NULL); if (c == -1) break; @@ -712,6 +716,11 @@ int main(int argc, char *argv[]) user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 1; } + break; + default: usage(argv[0]); return 1; diff --git a/read_lat.c b/read_lat.c index 49f28cf..4615179 100755 --- a/read_lat.c +++ b/read_lat.c @@ -2,6 +2,7 @@ * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * Copyright (c) 2005 Hewlett Packard, Inc (Grant Grundler) + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -59,6 +60,7 @@ #define PINGPONG_READ_WRID 1 #define VERSION 1.1 #define ALL 1 +static int sl = 0; static int page_size; cycles_t *tstamp; struct pingpong_dest my_dest; @@ -444,7 +446,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, attr.min_rnr_timer = 12; attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; if (ibv_modify_qp(ctx->qp, &attr, @@ -563,6 +565,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -o, --outs= num of outstanding read/atom(default 4)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -a, --all Run sizes from 2 till 2^23\n"); printf(" -C, --report-cycles report times in cpu cycle units (default microseconds)\n"); printf(" -H, --report-histogram print out all results (default print summary only)\n"); @@ -767,6 +770,7 @@ int main(int argc, char *argv[]) { .name = "outs", .has_arg = 1, .val = 'o' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "report-cycles", .has_arg = 0, .val = 'C' }, { .name = "report-histogram",.has_arg = 0, .val = 'H' }, @@ -777,7 +781,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:c:m:d:i:s:o:n:t:u:aeHUVF", long_options, NULL); + c = getopt_long(argc, argv, "p:c:m:d:i:s:o:n:t:u:S:aeHUVF", long_options, NULL); if (c == -1) break; @@ -866,9 +870,14 @@ int main(int argc, char *argv[]) user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 5; } + break; + default: usage(argv[0]); - return 5; + return 6; } } diff --git a/send_bw.c b/send_bw.c index a04703c..98b17f0 100755 --- a/send_bw.c +++ b/send_bw.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -84,6 +85,7 @@ struct user_parameters { int inline_size; int qp_timeout; }; +static int sl = 0; static int page_size; cycles_t *tposted; cycles_t *tcompleted; @@ -504,7 +506,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, } attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; if ((user_parm->connection_type==UD) && (user_parm->use_mcg)) { @@ -629,6 +631,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -I, --inline_size= max size of message to be sent in inline mode (default 400)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); printf(" -V, --version display version number\n"); printf(" -e, --events sleep on CQ events (default poll)\n"); @@ -970,6 +973,7 @@ int main(int argc, char *argv[]) { .name = "inline_size", .has_arg = 1, .val = 'I' }, { .name = "rx-depth", .has_arg = 1, .val = 'r' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, { .name = "version", .has_arg = 0, .val = 'V' }, @@ -980,7 +984,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:d:i:m:c:s:n:t:I:r:u:ebaVgNF", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:m:c:s:n:t:I:r:u:S:ebaVgNF", long_options, NULL); if (c == -1) break; @@ -1078,6 +1082,11 @@ int main(int argc, char *argv[]) user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 1; } + break; + default: usage(argv[0]); return 1; diff --git a/send_lat.c b/send_lat.c index d3b884a..8d2598d 100755 --- a/send_lat.c +++ b/send_lat.c @@ -2,6 +2,7 @@ * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * Copyright (c) 2005 Hewlett Packard, Inc (Grant Grundler) + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -66,6 +67,7 @@ #define MAX_INLINE 400 #define MCG_LID 0xc001 #define MCG_GID {255,1,0,0,0,2,201,133,0,0,0,0,0,0,0,0} +static int sl = 0; static int page_size; cycles_t *tstamp; struct user_parameters { @@ -522,7 +524,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; if ((user_parm->connection_type==UD) && (user_parm->use_mcg)) { @@ -711,6 +713,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -I, --inline_size= max size of message to be sent in inline mode (default 400)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -C, --report-cycles report times in cpu cycle units (default microseconds)\n"); printf(" -H, --report-histogram print out all results (default print summary only)\n"); printf(" -U, --report-unsorted (implies -H) print out unsorted results (default sorted)\n"); @@ -1023,6 +1026,7 @@ int main(int argc, char *argv[]) { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "inline_size", .has_arg = 1, .val = 'I' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "signal", .has_arg = 0, .val = 'l' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "report-cycles", .has_arg = 0, .val = 'C' }, @@ -1034,7 +1038,7 @@ int main(int argc, char *argv[]) { .name = "CPU-freq", .has_arg = 0, .val = 'F' }, { 0 } }; - c = getopt_long(argc, argv, "p:c:m:d:i:s:n:t:I:u:laeCHUVgF", long_options, NULL); + c = getopt_long(argc, argv, "p:c:m:d:i:s:n:t:I:u:S:laeCHUVgF", long_options, NULL); if (c == -1) break; @@ -1135,9 +1139,14 @@ int main(int argc, char *argv[]) user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 6; } + break; + default: usage(argv[0]); - return 5; + return 7; } } diff --git a/write_bw.c b/write_bw.c index 683fb80..21f994b 100755 --- a/write_bw.c +++ b/write_bw.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -79,6 +80,7 @@ struct extended_qp { struct ibv_qp *qp; int scnt, ccnt ; }; +static int sl = 0; static int page_size; cycles_t *tposted; @@ -444,7 +446,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, } attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; if (user_parm->connection_type == RC) { @@ -521,6 +523,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (at least 2, default 5000)\n"); printf(" -I, --inline_size= max size of message to be sent in inline mode (default 400)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); printf(" -V, --version display version number\n"); printf(" -N, --no peak-bw cancel peak-bw calculation (default with peak-bw)\n"); @@ -712,6 +715,7 @@ int main(int argc, char *argv[]) { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "inline_size", .has_arg = 1, .val = 'I' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, { .name = "version", .has_arg = 0, .val = 'V' }, @@ -720,7 +724,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:d:i:m:q:g:c:s:n:t:I:u:baVNF", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:m:q:g:c:s:n:t:I:u:S:baVNF", long_options, NULL); if (c == -1) break; @@ -812,6 +816,11 @@ int main(int argc, char *argv[]) user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 1; } + break; + default: usage(argv[0]); return 1; diff --git a/write_bw_postlist.c b/write_bw_postlist.c index d9e8dc2..d7b369a 100755 --- a/write_bw_postlist.c +++ b/write_bw_postlist.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -79,6 +80,7 @@ struct extended_qp { struct ibv_qp *qp; int scnt, ccnt ; }; +static int sl = 0; static int page_size; cycles_t *tposted; @@ -444,7 +446,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, } attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; if (user_parm->connection_type == RC) { @@ -521,6 +523,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (at least 2, default 5000)\n"); printf(" -I, --inline_size= max size of message to be sent in inline mode (default 400)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); printf(" -V, --version display version number\n"); printf(" -F, --CPU-freq do not fail even if cpufreq_ondemand module is loaded\n"); @@ -710,6 +713,7 @@ int main(int argc, char *argv[]) { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "inline_size", .has_arg = 1, .val = 'I' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, { .name = "version", .has_arg = 0, .val = 'V' }, @@ -717,7 +721,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:d:i:m:q:g:c:s:n:t:I:u:baVF", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:m:q:g:c:s:n:t:I:u:S:baVF", long_options, NULL); if (c == -1) break; @@ -806,6 +810,11 @@ int main(int argc, char *argv[]) user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 1; } + break; + default: usage(argv[0]); return 1; diff --git a/write_lat.c b/write_lat.c index 819cf33..89aefbe 100755 --- a/write_lat.c +++ b/write_lat.c @@ -2,6 +2,7 @@ * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * Copyright (c) 2005 Hewlett Packard, Inc (Grant Grundler) + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -60,6 +61,7 @@ #define VERSION 1.0 #define ALL 1 #define MAX_INLINE 400 +static int sl = 0; static int page_size; cycles_t *tstamp; struct user_parameters { @@ -439,7 +441,7 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, attr.ah_attr.is_global = 0; attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; + attr.ah_attr.sl = sl; attr.ah_attr.src_path_bits = 0; attr.ah_attr.port_num = port; @@ -579,6 +581,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -I, --inline_size= max size of message to be sent in inline mode (default 400)\n"); printf(" -u, --qp-timeout= QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14\n"); + printf(" -S, --sl= SL (default 0)\n"); printf(" -C, --report-cycles report times in cpu cycle units (default microseconds)\n"); printf(" -H, --report-histogram print out all results (default print summary only)\n"); printf(" -U, --report-unsorted (implies -H) print out unsorted results (default sorted)\n"); @@ -789,6 +792,7 @@ int main(int argc, char *argv[]) { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "inline_size", .has_arg = 1, .val = 'I' }, { .name = "qp-timeout", .has_arg = 1, .val = 'u' }, + { .name = "sl", .has_arg = 1, .val = 'S' }, { .name = "all", .has_arg = 0, .val = 'a' }, { .name = "report-cycles", .has_arg = 0, .val = 'C' }, { .name = "report-histogram",.has_arg = 0, .val = 'H' }, @@ -798,7 +802,7 @@ int main(int argc, char *argv[]) { 0 } }; - c = getopt_long(argc, argv, "p:c:m:d:i:s:n:t:I:u:aCHUVF", long_options, NULL);///cpufreq + c = getopt_long(argc, argv, "p:c:m:d:i:s:n:t:I:u:S:aCHUVF", long_options, NULL);///cpufreq if (c == -1) break; @@ -887,10 +891,14 @@ int main(int argc, char *argv[]) case 'u': user_param.qp_timeout = strtol(optarg, NULL, 0); break; + case 'S': + sl = strtol(optarg, NULL, 0); + if (sl > 15) { usage(argv[0]); return 6; } + break; default: usage(argv[0]); - return 5; + return 7; } } From bart.vanassche at gmail.com Wed Jun 10 07:26:43 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Wed, 10 Jun 2009 16:26:43 +0200 Subject: [ofa-general] SDP - How to? In-Reply-To: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> Message-ID: On Wed, Jun 10, 2009 at 4:07 PM, Zafar Gilani wrote: > I read a paper on "Zero Copy Sockets Direct Protocol over InfiniBand". I had > a few questions, if anybody could answer I will be thankful. > > 1. SDP BCopy approach can be used without the change of the code via the use > of AF_INET_SDP parameter when calling socket() system call. But how can I > implement SDP ZCopy approach if I want to implement ZCopy_Read and > ZCopy_Write methods? > > 2. Is there any documentation regarding how to implement SDP ZCopy? Any > Hello World sort of example code that I can use to learn how to work with > this? > > 3. SDP BCopy is better for smaller messages, but how small? Is 64K a good > threshold to change from BCopy to ZCopy at 128K? May I ask you why you are looking at SDP ? The performance of IPoIB in connected mode and with large MTU is close to that of SDP, while the former is a lot easier to use than the latter. Bart. From zafargilani at gmail.com Wed Jun 10 07:47:12 2009 From: zafargilani at gmail.com (Zafar Gilani) Date: Wed, 10 Jun 2009 19:47:12 +0500 Subject: [ofa-general] SDP - How to? In-Reply-To: References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> Message-ID: <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> Yes of course good question. Well the reason is that SDP allows two approaches: BCopy (Buffered Copy) and ZCopy (Zero Copy or RDMA). BCopy as I described is simple to use (change AF_INET to AF_INET_SDP in socket() call and you are done, no code changes). Whereas ZCopy exploits the RDMA capability of InfiniBand. In contrast IPoIB just provides IP protocol encapsulation over InfiniBand. You can look at the benchmark graphs I generated at [ http://hpc.niit.edu.pk/~zafar/work/results_rocks/index.html] by executing MPJ Express over IPoIB and SDP/BCopy. Now I want to implement SDP/ZCopy for MPJ Express and I do not know how to do this. This is the primary reason for my questions :). You can say this is sort of enabling MPJ Express to work over wide range of IB protocols. More information on MPJ Express is available at: http://mpj-express.org/ Thank you, Zafar On Wed, Jun 10, 2009 at 4:07 PM, Zafar Gilani wrote: > I read a paper on "Zero Copy Sockets Direct Protocol over InfiniBand". I had > a few questions, if anybody could answer I will be thankful. > > 1. SDP BCopy approach can be used without the change of the code via the use > of AF_INET_SDP parameter when calling socket() system call. But how can I > implement SDP ZCopy approach if I want to implement ZCopy_Read and > ZCopy_Write methods? > > 2. Is there any documentation regarding how to implement SDP ZCopy? Any > Hello World sort of example code that I can use to learn how to work with > this? > > 3. SDP BCopy is better for smaller messages, but how small? Is 64K a good > threshold to change from BCopy to ZCopy at 128K? May I ask you why you are looking at SDP ? The performance of IPoIB in connected mode and with large MTU is close to that of SDP, while the former is a lot easier to use than the latter. Bart. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Wed Jun 10 08:10:29 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 10 Jun 2009 08:10:29 -0700 Subject: [ofa-general] [ofw] [PATCH-resend] ib-mgmt/libibnetdisc: fix typecast warning Message-ID: <25C15E8A90F247C68A27939B5D3B0164@amr.corp.intel.com> Signed-off-by: Sean Hefty --- I tried converting ib_portid_t lid to a uint16_t, but that resulted in a cascade of changes, so I kept the simple approach. :) Resending - I didn't see a response to this. infiniband-diags/libibnetdisc/src/ibnetdisc.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c index 1e93ff8..baea98e 100644 --- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c +++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c @@ -188,7 +188,7 @@ extend_dpath(struct ibnd_fabric *f, ib_portid_t *portid, int nextport) f->fabric.ibmad_port) < 0) return -1; - portid->drpath.drslid = f->selfportid.lid; + portid->drpath.drslid = (uint16_t) f->selfportid.lid; portid->drpath.drdlid = 0xFFFF; } _______________________________________________ ofw mailing list ofw at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw From ahomike at us.ibm.com Wed Jun 10 08:32:49 2009 From: ahomike at us.ibm.com (Mike Aho) Date: Wed, 10 Jun 2009 10:32:49 -0500 Subject: [ofa-general] Setting mtu on an IB port Message-ID: Is there a way to set the mtu on an IB port? Thanks. --Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Wed Jun 10 09:14:03 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 10 Jun 2009 09:14:03 -0700 Subject: [ofa-general] Coalescing interrupts in SRP In-Reply-To: <4A2F5930.30802@Voltaire.com> (Or Gerlitz's message of "Wed, 10 Jun 2009 09:56:48 +0300") References: <4A2F5930.30802@Voltaire.com> Message-ID: > > one could do a hack with ib_modify_cq() a la IPoIB (assuming you're using ConnectX HCA). > I wonder why you consider this a hack... is programming interrupt > coalescing uncommon with HBAs? with networking I understand that the > common practice is to apply both HW mitigation and "SW mitigation" > (NAPI) so with SSD IOPS rates peaking to hundreds of thousands I > assume such an approach is valid for disks as well. Just because it is ConnectX only, and SW mitigation is likely to have even more benefit. From rdreier at cisco.com Wed Jun 10 09:18:40 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 10 Jun 2009 09:18:40 -0700 Subject: [ofa-general] Coalescing interrupts in SRP In-Reply-To: (Chris Worley's message of "Tue, 9 Jun 2009 17:36:19 -0600") References: Message-ID: > Does the "ib_req_notify_cq" call do the rearm? Yes. > If so, does that mean that until it rearms, the interrupt is disabled? Yes. > If so, shouldn't a delay before the rearm (after clearing the current > queue) in effect coalesce? One could do that... probably in conjunction with IB_CQ_REPORT_MISSED_EVENTS to avoid races on some hardware. However, if one is going to add a timer and delay before rearming, it seems you might as well use that timer to poll the CQ again and see if there's more work, and go to a NAPI/polling model instead. From suri at baymicrosystems.com Wed Jun 10 09:24:55 2009 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Wed, 10 Jun 2009 12:24:55 -0400 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: References: <4A2BD9B9.6000802@mellanox.co.il> Message-ID: Folks: mad.c:ib_send_mad() calls ib_post_send() after taking spin_lock_irqsave(). Is it really necessary to take the spinlock during the entire time of ib_post_send()? It appears like it is only necessary for list manipulation! Because in many other places where ib_post_send is called, this lock is not taken. Thanks for your input, Suri From demption at hotmail.com Wed Jun 10 09:41:18 2009 From: demption at hotmail.com (chris kinney) Date: Wed, 10 Jun 2009 09:41:18 -0700 Subject: [ofa-general] after "SUBNET UP" opensm hangs Message-ID: after i start opensm, i get "SUBNET UP" and then it hangs. im using centos 5 and even when i do a /etc/init.d/opensmd start it acts like its starting but never says started just hangs. but it seems to work if i want to ping to other machines.. why would this be? is there a fix? _________________________________________________________________ Lauren found her dream laptop. Find the PC that’s right for you. http://www.microsoft.com/windows/choosepc/?ocid=ftp_val_wl_290 -------------- next part -------------- An HTML attachment was scrubbed... URL: From demption at hotmail.com Wed Jun 10 09:48:21 2009 From: demption at hotmail.com (chris kinney) Date: Wed, 10 Jun 2009 09:48:21 -0700 Subject: [ofa-general] after "SUBNET UP" opensm hangs In-Reply-To: References: Message-ID: here is what my log file says: Jun 10 02:47:44 507274 [5F822670] 0x03 -> OpenSM 3.2.2 Jun 10 02:47:44 507323 [5F822670] 0x80 -> OpenSM 3.2.2 Jun 10 02:47:44 508176 [5F822670] 0x02 -> osm_vendor_init: 1000 pending umads specified Jun 10 02:47:44 508471 [5F822670] 0x80 -> Entering DISCOVERING state Jun 10 02:47:44 521533 [5F822670] 0x02 -> osm_vendor_bind: Binding to port 0x2c901079908d1 Jun 10 02:47:44 551979 [5F822670] 0x02 -> osm_vendor_bind: Binding to port 0x2c901079908d1 Jun 10 02:47:44 816721 [457CF940] 0x80 -> Entering MASTER state Jun 10 02:47:44 817394 [457CF940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0 GID:0xfe80000000000000,0x0002c901079908d1 Jun 10 02:47:44 817422 [457CF940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0 GID:0xfe80000000000000,0x0002c901079908d1 Jun 10 02:47:45 023159 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop tables configured on all switches Jun 10 02:47:45 042520 [457CF940] 0x80 -> SUBNET UP Jun 10 02:47:45 523365 [443CD940] 0x01 -> __osm_trap_rcv_process_request: Received Generic Notice type:0x04 num:144 Producer:1 (Channel Adapter) from LID:1 TID:0x000000000000003a Jun 10 02:47:45 523465 [443CD940] 0x02 -> osm_report_notice: Reporting Generic Notice type:4 num:144 from LID:1 GID:0xfe80000000000000,0x0002c901079908d1 Jun 10 02:47:45 728749 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop tables configured on all switches Jun 10 02:47:45 731570 [457CF940] 0x02 -> SUBNET UP Jun 10 02:47:46 821707 [439CC940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:1 GID:0xfe80000000000000,0x0002c901079908d1 Jun 10 02:47:46 822492 [443CD940] 0x02 -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:1 GID:0xfe80000000000000,0x0002c901079908d1 Jun 10 02:47:57 250668 [5F822670] 0x80 -> Exiting SM its getting an exiting because i do a ctl-c because it just hangs From: demption at hotmail.com To: general at lists.openfabrics.org Date: Wed, 10 Jun 2009 09:41:18 -0700 Subject: [ofa-general] after "SUBNET UP" opensm hangs after i start opensm, i get "SUBNET UP" and then it hangs. im using centos 5 and even when i do a /etc/init.d/opensmd start it acts like its starting but never says started just hangs. but it seems to work if i want to ping to other machines.. why would this be? is there a fix? Lauren found her dream laptop. Find the PC that’s right for you. _________________________________________________________________ Windows Live™: Keep your life in sync. http://windowslive.com/explore?ocid=TXT_TAGLM_WL_BR_life_in_synch_062009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Wed Jun 10 10:01:21 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 10 Jun 2009 10:01:21 -0700 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: References: <4A2BD9B9.6000802@mellanox.co.il> Message-ID: <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> >mad.c:ib_send_mad() calls ib_post_send() after taking spin_lock_irqsave(). > >Is it really necessary to take the spinlock during the entire time of >ib_post_send()? It appears like it is only necessary for list manipulation! It protects the list and the counters. It's technically not needed around ib_post_send. - Sean From hal.rosenstock at gmail.com Wed Jun 10 10:33:45 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 10 Jun 2009 13:33:45 -0400 Subject: [ofa-general] after "SUBNET UP" opensm hangs In-Reply-To: References: Message-ID: On Wed, Jun 10, 2009 at 12:48 PM, chris kinney wrote: > here is what my log file says: > > Jun 10 02:47:44 507274 [5F822670] 0x03 -> OpenSM 3.2.2 > Jun 10 02:47:44 507323 [5F822670] 0x80 -> OpenSM 3.2.2 > Jun 10 02:47:44 508176 [5F822670] 0x02 -> osm_vendor_init: 1000 pending > umads specified > Jun 10 02:47:44 508471 [5F822670] 0x80 -> Entering DISCOVERING state > Jun 10 02:47:44 521533 [5F822670] 0x02 -> osm_vendor_bind: Binding to port > 0x2c901079908d1 > Jun 10 02:47:44 551979 [5F822670] 0x02 -> osm_vendor_bind: Binding to port > 0x2c901079908d1 > Jun 10 02:47:44 816721 [457CF940] 0x80 -> Entering MASTER state > Jun 10 02:47:44 817394 [457CF940] 0x02 -> osm_report_notice: Reporting > Generic Notice type:3 num:66 from LID:0 > GID:0xfe80000000000000,0x0002c901079908d1 > Jun 10 02:47:44 817422 [457CF940] 0x02 -> osm_report_notice: Reporting > Generic Notice type:3 num:66 from LID:0 > GID:0xfe80000000000000,0x0002c901079908d1 > Jun 10 02:47:45 023159 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop > tables configured on all switches > Jun 10 02:47:45 042520 [457CF940] 0x80 -> SUBNET UP > Jun 10 02:47:45 523365 [443CD940] 0x01 -> __osm_trap_rcv_process_request: > Received Generic Notice type:0x04 num:144 Producer:1 (Channel Adapter) from > LID:1 TID:0x000000000000003a > Jun 10 02:47:45 523465 [443CD940] 0x02 -> osm_report_notice: Reporting > Generic Notice type:4 num:144 from LID:1 > GID:0xfe80000000000000,0x0002c901079908d1 > Jun 10 02:47:45 728749 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop > tables configured on all switches > Jun 10 02:47:45 731570 [457CF940] 0x02 -> SUBNET UP > Jun 10 02:47:46 821707 [439CC940] 0x02 -> osm_report_notice: Reporting > Generic Notice type:3 num:66 from LID:1 > GID:0xfe80000000000000,0x0002c901079908d1 > Jun 10 02:47:46 822492 [443CD940] 0x02 -> osm_report_notice: Reporting > Generic Notice type:3 num:66 from LID:1 > GID:0xfe80000000000000,0x0002c901079908d1 > Jun 10 02:47:57 250668 [5F822670] 0x80 -> Exiting SM > > its getting an exiting because i do a ctl-c because it just hangs > > ________________________________ > From: demption at hotmail.com > To: general at lists.openfabrics.org > Date: Wed, 10 Jun 2009 09:41:18 -0700 > Subject: [ofa-general] after "SUBNET UP" opensm hangs > > after i start opensm, i get "SUBNET UP" and then it hangs. im using > centos 5 and even when i do a /etc/init.d/opensmd start it acts like its > starting but never says started just hangs. What makes you say it is hanging ? No additional output in the opensm log file or something else ? -- Hal > but it seems to work if i > want to ping to other machines.. why would this be? is there a fix? > > ________________________________ > Lauren found her dream laptop. Find the PC that’s right for you. > ________________________________ > Windows Live™: Keep your life in sync. Check it out. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From demption at hotmail.com Wed Jun 10 10:35:32 2009 From: demption at hotmail.com (chris kinney) Date: Wed, 10 Jun 2009 10:35:32 -0700 Subject: [ofa-general] after "SUBNET UP" opensm hangs In-Reply-To: References: Message-ID: because when i run the /etc/init.d/opensmd start command, it never put me back into a shell i keep pressing enter but never get a command prompt back. i have to ctl-c to get back there > Date: Wed, 10 Jun 2009 13:33:45 -0400 > Subject: Re: [ofa-general] after "SUBNET UP" opensm hangs > From: hal.rosenstock at gmail.com > To: demption at hotmail.com > CC: general at lists.openfabrics.org > > On Wed, Jun 10, 2009 at 12:48 PM, chris kinney wrote: > > here is what my log file says: > > > > Jun 10 02:47:44 507274 [5F822670] 0x03 -> OpenSM 3.2.2 > > Jun 10 02:47:44 507323 [5F822670] 0x80 -> OpenSM 3.2.2 > > Jun 10 02:47:44 508176 [5F822670] 0x02 -> osm_vendor_init: 1000 pending > > umads specified > > Jun 10 02:47:44 508471 [5F822670] 0x80 -> Entering DISCOVERING state > > Jun 10 02:47:44 521533 [5F822670] 0x02 -> osm_vendor_bind: Binding to port > > 0x2c901079908d1 > > Jun 10 02:47:44 551979 [5F822670] 0x02 -> osm_vendor_bind: Binding to port > > 0x2c901079908d1 > > Jun 10 02:47:44 816721 [457CF940] 0x80 -> Entering MASTER state > > Jun 10 02:47:44 817394 [457CF940] 0x02 -> osm_report_notice: Reporting > > Generic Notice type:3 num:66 from LID:0 > > GID:0xfe80000000000000,0x0002c901079908d1 > > Jun 10 02:47:44 817422 [457CF940] 0x02 -> osm_report_notice: Reporting > > Generic Notice type:3 num:66 from LID:0 > > GID:0xfe80000000000000,0x0002c901079908d1 > > Jun 10 02:47:45 023159 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop > > tables configured on all switches > > Jun 10 02:47:45 042520 [457CF940] 0x80 -> SUBNET UP > > Jun 10 02:47:45 523365 [443CD940] 0x01 -> __osm_trap_rcv_process_request: > > Received Generic Notice type:0x04 num:144 Producer:1 (Channel Adapter) from > > LID:1 TID:0x000000000000003a > > Jun 10 02:47:45 523465 [443CD940] 0x02 -> osm_report_notice: Reporting > > Generic Notice type:4 num:144 from LID:1 > > GID:0xfe80000000000000,0x0002c901079908d1 > > Jun 10 02:47:45 728749 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop > > tables configured on all switches > > Jun 10 02:47:45 731570 [457CF940] 0x02 -> SUBNET UP > > Jun 10 02:47:46 821707 [439CC940] 0x02 -> osm_report_notice: Reporting > > Generic Notice type:3 num:66 from LID:1 > > GID:0xfe80000000000000,0x0002c901079908d1 > > Jun 10 02:47:46 822492 [443CD940] 0x02 -> osm_report_notice: Reporting > > Generic Notice type:3 num:66 from LID:1 > > GID:0xfe80000000000000,0x0002c901079908d1 > > Jun 10 02:47:57 250668 [5F822670] 0x80 -> Exiting SM > > > > its getting an exiting because i do a ctl-c because it just hangs > > > > ________________________________ > > From: demption at hotmail.com > > To: general at lists.openfabrics.org > > Date: Wed, 10 Jun 2009 09:41:18 -0700 > > Subject: [ofa-general] after "SUBNET UP" opensm hangs > > > > after i start opensm, i get "SUBNET UP" and then it hangs. im using > > centos 5 and even when i do a /etc/init.d/opensmd start it acts like its > > starting but never says started just hangs. > > What makes you say it is hanging ? No additional output in the opensm > log file or something else ? > > -- Hal > > > but it seems to work if i > > want to ping to other machines.. why would this be? is there a fix? > > > > ________________________________ > > Lauren found her dream laptop. Find the PC that’s right for you. > > ________________________________ > > Windows Live™: Keep your life in sync. Check it out. > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > _________________________________________________________________ Insert movie times and more without leaving Hotmail®. http://windowslive.com/Tutorial/Hotmail/QuickAdd?ocid=TXT_TAGLM_WL_HM_Tutorial_QuickAdd_062009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From zafargilani at gmail.com Wed Jun 10 10:39:36 2009 From: zafargilani at gmail.com (Zafar Gilani) Date: Wed, 10 Jun 2009 22:39:36 +0500 Subject: [ofa-general] SDP - How to? In-Reply-To: <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> Message-ID: <7d4423d30906101039s3d44ae75uf7ec73bb2c3fdc03@mail.gmail.com> I asked Pavan Balaji, who is working at Argonne National Laboratories and he said that SDP ZCopy should be enabled by default for large messages (larger than 64K) if OFED SDP stack is loaded and used. This pretty much answers my question. However since I am working on Java and did not actually use C for MPI purposes, is there any resource I could use to learn how to program SOCK_STREAM in C for writing an MPI based communication device? On Wed, Jun 10, 2009 at 7:47 PM, Zafar Gilani wrote: > Yes of course good question. Well the reason is that SDP allows two > approaches: BCopy (Buffered Copy) and ZCopy (Zero Copy or RDMA). BCopy as I > described is simple to use (change AF_INET to AF_INET_SDP in socket() call > and you are done, no code changes). Whereas ZCopy exploits the RDMA > capability of InfiniBand. In contrast IPoIB just provides IP protocol > encapsulation over InfiniBand. > > You can look at the benchmark graphs I generated at [ > http://hpc.niit.edu.pk/~zafar/work/results_rocks/index.html] > by executing MPJ Express over IPoIB and SDP/BCopy. Now I want to implement > SDP/ZCopy for MPJ Express and I do not know how to do this. This is the > primary reason for my questions :). You can say this is sort of enabling MPJ > Express to work over wide range of IB protocols. > > More information on MPJ Express is available at: http://mpj-express.org/ > > Thank you, > Zafar > > > On Wed, Jun 10, 2009 at 4:07 PM, Zafar Gilani > wrote: > > I read a paper on "Zero Copy Sockets Direct Protocol over InfiniBand". I > had > > a few questions, if anybody could answer I will be thankful. > > > > 1. SDP BCopy approach can be used without the change of the code via the > use > > of AF_INET_SDP parameter when calling socket() system call. But how can I > > implement SDP ZCopy approach if I want to implement ZCopy_Read and > > ZCopy_Write methods? > > > > 2. Is there any documentation regarding how to implement SDP ZCopy? Any > > Hello World sort of example code that I can use to learn how to work with > > this? > > > > 3. SDP BCopy is better for smaller messages, but how small? Is 64K a good > > threshold to change from BCopy to ZCopy at 128K? > > May I ask you why you are looking at SDP ? The performance of IPoIB in > connected mode and with large MTU is close to that of SDP, while the > former is a lot easier to use than the latter. > > Bart. > -- Zafar -------------- next part -------------- An HTML attachment was scrubbed... URL: From hal.rosenstock at gmail.com Wed Jun 10 10:49:43 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 10 Jun 2009 13:49:43 -0400 Subject: [ofa-general] after "SUBNET UP" opensm hangs In-Reply-To: References: Message-ID: On Wed, Jun 10, 2009 at 1:35 PM, chris kinney wrote: > because when i run the /etc/init.d/opensmd start command, What happens when you run: service opensmd start instead ? > it never put me > back into a shell i keep pressing enter but never get a command prompt back. > i have to ctl-c to get back there > >> Date: Wed, 10 Jun 2009 13:33:45 -0400 >> Subject: Re: [ofa-general] after "SUBNET UP" opensm hangs >> From: hal.rosenstock at gmail.com >> To: demption at hotmail.com >> CC: general at lists.openfabrics.org >> >> On Wed, Jun 10, 2009 at 12:48 PM, chris kinney >> wrote: >> > here is what my log file says: >> > >> > Jun 10 02:47:44 507274 [5F822670] 0x03 -> OpenSM 3.2.2 >> > Jun 10 02:47:44 507323 [5F822670] 0x80 -> OpenSM 3.2.2 >> > Jun 10 02:47:44 508176 [5F822670] 0x02 -> osm_vendor_init: 1000 pending >> > umads specified >> > Jun 10 02:47:44 508471 [5F822670] 0x80 -> Entering DISCOVERING state >> > Jun 10 02:47:44 521533 [5F822670] 0x02 -> osm_vendor_bind: Binding to >> > port >> > 0x2c901079908d1 >> > Jun 10 02:47:44 551979 [5F822670] 0x02 -> osm_vendor_bind: Binding to >> > port >> > 0x2c901079908d1 >> > Jun 10 02:47:44 816721 [457CF940] 0x80 -> Entering MASTER state >> > Jun 10 02:47:44 817394 [457CF940] 0x02 -> osm_report_notice: Reporting >> > Generic Notice type:3 num:66 from LID:0 >> > GID:0xfe80000000000000,0x0002c901079908d1 >> > Jun 10 02:47:44 817422 [457CF940] 0x02 -> osm_report_notice: Reporting >> > Generic Notice type:3 num:66 from LID:0 >> > GID:0xfe80000000000000,0x0002c901079908d1 >> > Jun 10 02:47:45 023159 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop >> > tables configured on all switches >> > Jun 10 02:47:45 042520 [457CF940] 0x80 -> SUBNET UP >> > Jun 10 02:47:45 523365 [443CD940] 0x01 -> >> > __osm_trap_rcv_process_request: >> > Received Generic Notice type:0x04 num:144 Producer:1 (Channel Adapter) >> > from >> > LID:1 TID:0x000000000000003a >> > Jun 10 02:47:45 523465 [443CD940] 0x02 -> osm_report_notice: Reporting >> > Generic Notice type:4 num:144 from LID:1 >> > GID:0xfe80000000000000,0x0002c901079908d1 >> > Jun 10 02:47:45 728749 [457CF940] 0x02 -> osm_ucast_mgr_process: minhop >> > tables configured on all switches >> > Jun 10 02:47:45 731570 [457CF940] 0x02 -> SUBNET UP >> > Jun 10 02:47:46 821707 [439CC940] 0x02 -> osm_report_notice: Reporting >> > Generic Notice type:3 num:66 from LID:1 >> > GID:0xfe80000000000000,0x0002c901079908d1 >> > Jun 10 02:47:46 822492 [443CD940] 0x02 -> osm_report_notice: Reporting >> > Generic Notice type:3 num:66 from LID:1 >> > GID:0xfe80000000000000,0x0002c901079908d1 >> > Jun 10 02:47:57 250668 [5F822670] 0x80 -> Exiting SM >> > >> > its getting an exiting because i do a ctl-c because it just hangs >> > >> > ________________________________ >> > From: demption at hotmail.com >> > To: general at lists.openfabrics.org >> > Date: Wed, 10 Jun 2009 09:41:18 -0700 >> > Subject: [ofa-general] after "SUBNET UP" opensm hangs >> > >> > after i start opensm, i get "SUBNET UP" and then it hangs. im using >> > centos 5 and even when i do a /etc/init.d/opensmd start it acts like its >> > starting but never says started just hangs. >> >> What makes you say it is hanging ? No additional output in the opensm >> log file or something else ? >> >> -- Hal >> >> > but it seems to work if i >> > want to ping to other machines.. why would this be? is there a fix? >> > >> > ________________________________ >> > Lauren found her dream laptop. Find the PC that’s right for you. >> > ________________________________ >> > Windows Live™: Keep your life in sync. Check it out. >> > _______________________________________________ >> > general mailing list >> > general at lists.openfabrics.org >> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> > >> > To unsubscribe, please visit >> > http://openib.org/mailman/listinfo/openib-general >> > > > ________________________________ > Insert movie times and more without leaving Hotmail®. See how. From vst at vlnb.net Wed Jun 10 11:41:34 2009 From: vst at vlnb.net (Vladislav Bolkhovitin) Date: Wed, 10 Jun 2009 22:41:34 +0400 Subject: [ofa-general] srp_daemon and partitions. In-Reply-To: <4A2F2355.50806@systemfabricworks.com> References: <4A2F1E2D.1050205@systemfabricworks.com> <4A2F2355.50806@systemfabricworks.com> Message-ID: <4A2FFE5E.6040600@vlnb.net> David McMillen, on 06/10/2009 07:07 AM wrote: > Chris Worley wrote: >> On Thu, Nov 20, 2008 at 1:06 PM, Vu Pham > wrote: >> >/ >> />/ Hi James, >> />/ >> />/ it's srp_daemon and ibsrpdm bug. We'll try to fix it to provide zoning thru pkey. >> / >> I don't think pkeys are the answer to zoning, as I don't see a way to >> tie a specific disk or partition to a pkey. At one point I tried to >> tie IB ports to pkeys, but found that all levels of the SRP stack >> needed to be pkey-aware. It looks like a daunting task, and probably >> not what pkeys were intended to do. >> >> I think SCST "security groups" are the intended way to zone. The >> scst/README says: >> >> ... >> 2. Initiator-oriented. In this mode you define which devices and >> their LUNs are accessible for each initiator. In this mode you should >> createfor each set of one or more initiators, which should access to >> the same set of devices with the same LUNs, a separate security group, >> then add to it available devices and names of allowed initiator(s). >> ... >> # echo "add_group >> Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz" >> >//proc/scsi_tgt/scsi_tgt >> /# echo "add dev1 0" >> >//proc/scsi_tgt/groups/Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz/devices >> /# echo "add dev2 1" >> >//proc/scsi_tgt/groups/Default_iqn.2007-05.com.example:storage.disk1.sys1.xyz/devices >> / >> # echo "add_group spec_ini" >/proc/scsi_tgt/scsi_tgt >> # echo "add iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" >> >//proc/scsi_tgt/groups/spec_ini/names >> /# echo "add dev2 0" >/proc/scsi_tgt/groups/spec_ini/devices >> ... >> >> But, I don't understand how the zoning example selects the initiator >> visibility... I'd hope w/ IB this could be done w/ the port GUID of >> the initiator. The example uses the name of >> "iqn.2007-05.com.example:storage.disk1.spec_ini.xyz" to specify the >> initiator. I'm guessing "iqn.2007-05.com.example" specifies the host >> name of the initiator, but not a clue what >> "storage.disk1.spec_ini.xyz" means. >> >> Chris > You got pretty close. I think there is at least one typographical error > in the explanation above, but let me try to explain it differently. > > The groups are used to create different sets of LUNs that are presented > to the initiators that match a group. When a login request comes in, > scst is asked to find the group given a name. You can get scst to > output a message like > > Using security group "Default" for initiator "xxx" > > on the console. The value in "xxx" (not really xxx, but dependent upon > the target code using scst) is matched against all of the entries in the > "names" pseudo file under each group's directory. The first group that > matches is the winner. The LUNs defined by the entries in the "devices" > pseudo file under that same group will be the only LUNs seen for that > session. If no group has a line in "names" that matches, then the group > "Default" is used. There is a "names" pseudo file here but it's > contents are ignored -- everybody is a winner matching the Default group. > > The name of the group is completely arbitrary and meaningless to scst > other than to differentiate groups. If you intend to have a unique > group for every initiator, then you could use the same value you will > write to "names" as the name of the group, but you could just as easily > just generate random names that are unique. > > If you want a particular LUN to show up in more than one group, you have > to add it to "devices" for every group that will use it. Also, if you > want that LUN to have the same LUN number, you have to arrange that as well. > > It is probably worth noting that there can be problems if a group does > not have a device defined for LUN 0. It works, but there are initiators > and user-level programs that issue SCSI commands and expect to get > something when LUN 0 is addressed. All real devices have some kind of > response at LUN 0, even if it is an odd SCSI device or a zero byte / > offline disk. > > Back to the "xxx" mentioned above, I don't know what the code you have > uses as the key for scst to find a group. That is why I suggested you > check your dmesg output, and you might have to enable some verbose/debug > output. I believe this changed at least once, and in our implementation > we are reworking it so we can have highly selective LUN masking against > the initiator/target pair. This is an excellent explanation! I can only add to it that "xxx" is a transport specific name. For iSCSI it is an iSCSI name, like iqn.2007-05.com.example:storage.disk1.spec_ini.xyz, for SRP it is a port name. > Dave > > > > ------------------------------------------------------------------------ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From suri at baymicrosystems.com Wed Jun 10 11:58:52 2009 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Wed, 10 Jun 2009 14:58:52 -0400 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> Message-ID: Sean: Do you see a problem if the piece of code in ib_send_mad is re-written as follows: ---------------------------------------------------- if (qp_info->send_queue.count < qp_info->send_queue.max_active) { ret = ib_post_send(mad_agent->qp, &mad_send_wr->send_wr, &bad_send_wr); spin_lock_irqsave(&qp_info->send_queue.lock, flags); list = &qp_info->send_queue.list; } else { ret = 0; spin_lock_irqsave(&qp_info->send_queue.lock, flags); list = &qp_info->overflow_list; } -------------------------------------------------------- I tested it in my environment and it seemed to work OK. Thanks, Suri > -----Original Message----- > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Wednesday, June 10, 2009 1:01 PM > To: 'Suresh Shelvapille'; general at lists.openfabrics.org > Cc: 'Roland Dreier' > Subject: RE: [ofa-general] spin_lock_irqsave in ib_send_mad > > >mad.c:ib_send_mad() calls ib_post_send() after taking spin_lock_irqsave(). > > > >Is it really necessary to take the spinlock during the entire time of > >ib_post_send()? It appears like it is only necessary for list manipulation! > > It protects the list and the counters. It's technically not needed around > ib_post_send. > > - Sean From sean.hefty at intel.com Wed Jun 10 12:07:21 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 10 Jun 2009 12:07:21 -0700 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> Message-ID: >---------------------------------------------------- >if (qp_info->send_queue.count < qp_info->send_queue.max_active) { This loses the locking around send_queue.count, which can allow trying to post more sends than the QP supports. > ret = ib_post_send(mad_agent->qp, &mad_send_wr->send_wr, > &bad_send_wr); > spin_lock_irqsave(&qp_info->send_queue.lock, flags); > list = &qp_info->send_queue.list; >} else { > ret = 0; > spin_lock_irqsave(&qp_info->send_queue.lock, flags); > list = &qp_info->overflow_list; >} I think the code will end up quite a bit more complex not locking around ib_post_send, but maybe there's a clever way to handle this. - Sean From devel-ofed at morey-chaisemartin.com Wed Jun 10 12:10:27 2009 From: devel-ofed at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Wed, 10 Jun 2009 21:10:27 +0200 Subject: [ofa-general] after "SUBNET UP" opensm hangs In-Reply-To: References: Message-ID: <4A300523.5080703@morey-chaisemartin.com> Le 10/06/2009 18:41, chris kinney a écrit : > after i start opensm, i get "SUBNET UP" and then it hangs. im using > centos 5 and even when i do a /etc/init.d/opensmd start it acts like its > starting but never says started just hangs. but it seems to work if i > want to ping to other machines.. why would this be? is there a fix? > > Which OS are you running? I've seen on a RC of RHEL5.3 that opensm start script (/etc/iniut.d/opensm) did not use the --daemon option so it never returns.. Nicolas From akepner at sgi.com Wed Jun 10 12:07:28 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Wed, 10 Jun 2009 12:07:28 -0700 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM Message-ID: <20090610190728.GR13781@sgi.com> There is a race between the neighbour cleanup code, and some uses of the closely related ipoib_neigh structure when using IPoIB-CM. To prevent the race, take a reference to the neighbour before the connection is established, and release the reference when the connection is destroyed. Also, defer the cleanup and release of the ipoib_neigh structure until the connection is destroyed. Signed-off-by: Arthur Kepner --- ipoib_cm.c | 40 ++++++++++++++++++++++++++++++++-------- ipoib_main.c | 7 +++++-- 2 files changed, 37 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 47d588b..ed22a37 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -810,9 +810,6 @@ void ipoib_cm_handle_tx_wc(struct net_device *dev, struct ib_wc *wc) list_del(&neigh->list); if (neigh->ah) ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); - - tx->neigh = NULL; } if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &tx->flags)) { @@ -1230,9 +1227,6 @@ static int ipoib_cm_tx_handler(struct ib_cm_id *cm_id, list_del(&neigh->list); if (neigh->ah) ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); - - tx->neigh = NULL; } if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &tx->flags)) { @@ -1278,7 +1272,6 @@ void ipoib_cm_destroy_tx(struct ipoib_cm_tx *tx) queue_work(ipoib_workqueue, &priv->cm.reap_task); ipoib_dbg(priv, "Reap connection for gid %pI6\n", tx->neigh->dgid.raw); - tx->neigh = NULL; } } @@ -1302,6 +1295,10 @@ static void ipoib_cm_tx_start(struct work_struct *work) p = list_entry(priv->cm.start_list.next, typeof(*p), list); list_del_init(&p->list); neigh = p->neigh; + ipoib_dbg(priv, "%s: holding ref to %pI6 (refcnt: %d)\n", + __func__, neigh->dgid.raw, + atomic_read(&neigh->neighbour->refcnt)); + neigh_hold(neigh->neighbour); qpn = IPOIB_QPN(neigh->neighbour->ha); memcpy(&pathrec, &p->path->pathrec, sizeof pathrec); @@ -1316,11 +1313,22 @@ static void ipoib_cm_tx_start(struct work_struct *work) if (ret) { neigh = p->neigh; if (neigh) { + struct sk_buff *skb; neigh->cm = NULL; list_del(&neigh->list); if (neigh->ah) ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); + *to_ipoib_neigh(neigh->neighbour) = NULL; + ipoib_dbg(priv, "%s: releasing ref to %pI6 " + "(refcnt: %d)\n", __func__, + neigh->dgid.raw, + atomic_read(&neigh->neighbour->refcnt)); + neigh_release(neigh->neighbour); + while ((skb = __skb_dequeue(&neigh->queue))) { + ++dev->stats.tx_dropped; + dev_kfree_skb_any(skb); + } + kfree(neigh); } list_del(&p->list); kfree(p); @@ -1343,7 +1351,23 @@ static void ipoib_cm_tx_reap(struct work_struct *work) spin_lock_irqsave(&priv->lock, flags); while (!list_empty(&priv->cm.reap_list)) { + struct ipoib_neigh *neigh; + struct sk_buff *skb; p = list_entry(priv->cm.reap_list.next, typeof(*p), list); + neigh = p->neigh; + if (neigh) { + *to_ipoib_neigh(neigh->neighbour) = NULL; + ipoib_dbg(priv, "%s: releasing ref to %pI6 " + "(refcnt: %d)\n", __func__, neigh->dgid.raw, + atomic_read(&neigh->neighbour->refcnt)); + neigh_release(neigh->neighbour); + while ((skb = __skb_dequeue(&neigh->queue))) { + ++dev->stats.tx_dropped; + dev_kfree_skb_any(skb); + } + kfree(neigh); + } + p->neigh = NULL; list_del(&p->list); spin_unlock_irqrestore(&priv->lock, flags); netif_tx_unlock_bh(dev); diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index ab2c192..8841160 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -889,13 +889,16 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, void ipoib_neigh_free(struct net_device *dev, struct ipoib_neigh *neigh) { struct sk_buff *skb; + if (ipoib_cm_get(neigh)) { + ipoib_cm_destroy_tx(ipoib_cm_get(neigh)); + return; + } + *to_ipoib_neigh(neigh->neighbour) = NULL; while ((skb = __skb_dequeue(&neigh->queue))) { ++dev->stats.tx_dropped; dev_kfree_skb_any(skb); } - if (ipoib_cm_get(neigh)) - ipoib_cm_destroy_tx(ipoib_cm_get(neigh)); kfree(neigh); } From pradeeps at linux.vnet.ibm.com Wed Jun 10 12:20:40 2009 From: pradeeps at linux.vnet.ibm.com (Pradeep Satyanarayana) Date: Wed, 10 Jun 2009 12:20:40 -0700 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: References: Message-ID: <4A300788.1000001@linux.vnet.ibm.com> Mike Aho wrote: > > Is there a way to set the mtu on an IB port? Thanks. --Mike > > Mike, you can use "ip link set ib0 mtu " for ib0 Pradeep From demption at hotmail.com Wed Jun 10 12:37:16 2009 From: demption at hotmail.com (chris kinney) Date: Wed, 10 Jun 2009 12:37:16 -0700 Subject: [ofa-general] after "SUBNET UP" opensm hangs In-Reply-To: <4A300523.5080703@morey-chaisemartin.com> References: <4A300523.5080703@morey-chaisemartin.com> Message-ID: i wiped everything clean and redid it and it worked. not sure why but it seems to be returning now. thanks for the input guys! > Date: Wed, 10 Jun 2009 21:10:27 +0200 > From: devel-ofed at morey-chaisemartin.com > To: demption at hotmail.com > CC: general at lists.openfabrics.org > Subject: Re: [ofa-general] after "SUBNET UP" opensm hangs > > Le 10/06/2009 18:41, chris kinney a écrit : > > after i start opensm, i get "SUBNET UP" and then it hangs. im using > > centos 5 and even when i do a /etc/init.d/opensmd start it acts like its > > starting but never says started just hangs. but it seems to work if i > > want to ping to other machines.. why would this be? is there a fix? > > > > > > Which OS are you running? > I've seen on a RC of RHEL5.3 that opensm start script (/etc/iniut.d/opensm) did not use the --daemon option so it never returns.. > > Nicolas _________________________________________________________________ Lauren found her dream laptop. Find the PC that’s right for you. http://www.microsoft.com/windows/choosepc/?ocid=ftp_val_wl_290 -------------- next part -------------- An HTML attachment was scrubbed... URL: From yosefe at voltaire.com Wed Jun 10 12:52:29 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Wed, 10 Jun 2009 22:52:29 +0300 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM In-Reply-To: <20090610190728.GR13781@sgi.com> References: <20090610190728.GR13781@sgi.com> Message-ID: <4A300EFD.3050303@voltaire.com> akepner at sgi.com wrote: > @@ -810,9 +810,6 @@ void ipoib_cm_handle_tx_wc(struct net_device *dev, struct ib_wc *wc) > list_del(&neigh->list); > if (neigh->ah) > ipoib_put_ah(neigh->ah); > - ipoib_neigh_free(dev, neigh); > - > - tx->neigh = NULL; > } > > if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &tx->flags)) { Can't calling ipoib_put_ah() and keeping the neighbour alive cause neigh->ah pointer become invalid? What about doing list_del(&neigh->list) twice? > @@ -1316,11 +1313,22 @@ static void ipoib_cm_tx_start(struct work_struct *work) > if (ret) { > neigh = p->neigh; > if (neigh) { > + struct sk_buff *skb; > neigh->cm = NULL; > list_del(&neigh->list); > if (neigh->ah) > ipoib_put_ah(neigh->ah); > - ipoib_neigh_free(dev, neigh); > + *to_ipoib_neigh(neigh->neighbour) = NULL; > + ipoib_dbg(priv, "%s: releasing ref to %pI6 " > + "(refcnt: %d)\n", __func__, > + neigh->dgid.raw, > + atomic_read(&neigh->neighbour->refcnt)); > + neigh_release(neigh->neighbour); > + while ((skb = __skb_dequeue(&neigh->queue))) { > + ++dev->stats.tx_dropped; > + dev_kfree_skb_any(skb); > + } > + kfree(neigh); > } > list_del(&p->list); > kfree(p); The piece of code that does '*to_ipoib_neigh(neigh->neighbour) = NULL' and releases skb's will appear 3 times in ipoib. I think it can be put in a separate function. > @@ -1343,7 +1351,23 @@ static void ipoib_cm_tx_reap(struct work_struct *work) > spin_lock_irqsave(&priv->lock, flags); > > while (!list_empty(&priv->cm.reap_list)) { > + struct ipoib_neigh *neigh; > + struct sk_buff *skb; > p = list_entry(priv->cm.reap_list.next, typeof(*p), list); > + neigh = p->neigh; > + if (neigh) { > + *to_ipoib_neigh(neigh->neighbour) = NULL; > + ipoib_dbg(priv, "%s: releasing ref to %pI6 " > + "(refcnt: %d)\n", __func__, neigh->dgid.raw, > + atomic_read(&neigh->neighbour->refcnt)); > + neigh_release(neigh->neighbour); > + while ((skb = __skb_dequeue(&neigh->queue))) { > + ++dev->stats.tx_dropped; > + dev_kfree_skb_any(skb); > + } > + kfree(neigh); > + } > + p->neigh = NULL; > list_del(&p->list); > spin_unlock_irqrestore(&priv->lock, flags); > netif_tx_unlock_bh(dev); Couldn't this race with ipoib_neigh_cleanup()? --Yossi From ahomike at us.ibm.com Wed Jun 10 13:00:41 2009 From: ahomike at us.ibm.com (Mike Aho) Date: Wed, 10 Jun 2009 15:00:41 -0500 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: <4A300788.1000001@linux.vnet.ibm.com> References: <4A300788.1000001@linux.vnet.ibm.com> Message-ID: Pradeep, Thank you, that works well on one machine. I was able to set the mtu on interface ib1 to 4096 instead of 2044. The max_mtu from ibv_devinfo is still 2048 on port 2. On the other machine, setting the ip mtu fails if I try a number higher than 2044. So which one is the correct behavior? By the way, opensmd is running on the other machine. I appreciate having the capability to do ip mtu and now have another command in my arsenal. I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the ibv_devinfo. Any ideas on that? Best Regards, Mike Aho |------------> | From: | |------------> >--------------------------------------------------------------------------------------------------------------------------------------------------| |Pradeep Satyanarayana | >--------------------------------------------------------------------------------------------------------------------------------------------------| |------------> | To: | |------------> >--------------------------------------------------------------------------------------------------------------------------------------------------| |Mike Aho/Rochester/IBM at IBMUS | >--------------------------------------------------------------------------------------------------------------------------------------------------| |------------> | Cc: | |------------> >--------------------------------------------------------------------------------------------------------------------------------------------------| |OpenFabrics General | >--------------------------------------------------------------------------------------------------------------------------------------------------| |------------> | Date: | |------------> >--------------------------------------------------------------------------------------------------------------------------------------------------| |06/10/2009 02:20 PM | >--------------------------------------------------------------------------------------------------------------------------------------------------| |------------> | Subject: | |------------> >--------------------------------------------------------------------------------------------------------------------------------------------------| |Re: [ofa-general] Setting mtu on an IB port | >--------------------------------------------------------------------------------------------------------------------------------------------------| Mike Aho wrote: > > Is there a way to set the mtu on an IB port? Thanks. --Mike > > Mike, you can use "ip link set ib0 mtu " for ib0 Pradeep From pradeeps at linux.vnet.ibm.com Wed Jun 10 14:06:06 2009 From: pradeeps at linux.vnet.ibm.com (Pradeep Satyanarayana) Date: Wed, 10 Jun 2009 14:06:06 -0700 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: References: <4A300788.1000001@linux.vnet.ibm.com> Message-ID: <4A30203E.9050309@linux.vnet.ibm.com> Mike Aho wrote: > Pradeep, > > Thank you, that works well on one machine. I was able to set the mtu on > interface ib1 to 4096 instead of 2044. The max_mtu from ibv_devinfo is > still 2048 on port 2. > On the other machine, setting the ip mtu fails if I try a number higher > than 2044. So which one is the correct behavior? By the way, opensmd is > running on the other machine. > > I appreciate having the capability to do ip mtu and now have another > command in my arsenal. > > I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the > ibv_devinfo. Any ideas on that? > Mike, The issue here is likely that one of the nodes/ports may be in datagram mode and the other one in connected mode. cat /sys/class/net/ib0/mode will reveal the mode for ib0. The datagram mode will limit the mtu to 2044 and one can typically go up to 65520 with connected mode. If you would like to switch to connected mode, you could (see caveats below) do a echo connected > /sys/class/net/ib0/mode followed by the ip link command. Caveats ------- The above (echo) command should be okay if you are trying with a mainline kernel, ODED-1.4 or OFED-1.4.1 installation, or even Sles varaints However, this will not work Rhel5.3 (if I remember correctly, should work for Rhel5.2). For Rhel5.3, you need to edit the /etc/sysconfig/network-scripts/ifcfg-ib* and insert an entry CONNECTED_MODE=yes, followed by an /etc/init.d/openibd restart. I think that should achieve what you want. Let me know if that doesn't work. Pradeep From sean.hefty at intel.com Wed Jun 10 14:12:28 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 10 Jun 2009 14:12:28 -0700 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: <4A30203E.9050309@linux.vnet.ibm.com> References: <4A300788.1000001@linux.vnet.ibm.com> <4A30203E.9050309@linux.vnet.ibm.com> Message-ID: <3322AD31CA7140D3A5A14E510E978A0E@amr.corp.intel.com> >> I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the >> ibv_devinfo. Any ideas on that? >> > >Mike, The issue here is likely that one of the nodes/ports may be in datagram >mode and the other one in connected mode. I believe he's trying to control the IB MTU, not the IP MTU. The IB MTU can be restricted based on the HCA capabilities or the switch capabilities that the HCA may be connected to. Do you know if these support a larger MTU size? - Sean From pradeeps at linux.vnet.ibm.com Wed Jun 10 17:30:36 2009 From: pradeeps at linux.vnet.ibm.com (Pradeep Satyanarayana) Date: Wed, 10 Jun 2009 17:30:36 -0700 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: <3322AD31CA7140D3A5A14E510E978A0E@amr.corp.intel.com> References: <4A300788.1000001@linux.vnet.ibm.com> <4A30203E.9050309@linux.vnet.ibm.com> <3322AD31CA7140D3A5A14E510E978A0E@amr.corp.intel.com> Message-ID: <4A30502C.5090405@linux.vnet.ibm.com> Sean Hefty wrote: >>> I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the >>> ibv_devinfo. Any ideas on that? >>> >> Mike, The issue here is likely that one of the nodes/ports may be in datagram >> mode and the other one in connected mode. > > I believe he's trying to control the IB MTU, not the IP MTU. > > The IB MTU can be restricted based on the HCA capabilities or the switch > capabilities that the HCA may be connected to. Do you know if these support a > larger MTU size? Sean, You are right about the IB MTU. My guess was that he was trying to set both (based on the 2044 in his mail). I use the embedded SM rather than openSM. Hence guidance on that will be helpful. Pradeep From ogerlitz at voltaire.com Wed Jun 10 23:12:56 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 11 Jun 2009 09:12:56 +0300 Subject: [ofa-general] Coalescing interrupts in SRP In-Reply-To: References: <4A2F5930.30802@Voltaire.com> Message-ID: <4A30A068.3050701@voltaire.com> Roland Dreier wrote: > Just because it is ConnectX only [...] yep, but this argument also apply to all the stateless offloads integrated into ipoib, anyway > [...] and SW mitigation is likely to have even more benefit. okay, thanks for the heads up on sw mitigation, let me do some reading... Or. From amirv at mellanox.co.il Wed Jun 10 23:22:14 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 11 Jun 2009 09:22:14 +0300 Subject: [ofa-general] SDP - How to? In-Reply-To: <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> Message-ID: <4A30A296.60109@mellanox.co.il> Hi Zafar, I am the maintainer of SDP in OFED. The current SDP implementation does not have ZCopy. What we have is BCopy and something that we call BZcopy. Small packets (<=32k) are sent using BCopy - meaning the SEND verb is in use and data is not zero copied. Packets bigger than 32K are sent using BZcopy - they are also sent using the SEND verb but they are zero copied on the sender side. on the RX side packets are memcpy'ed. When we implement ZCopy it will be automatic too - when sending packets bigger than a threshold the driver will automatically use RDMA verb instead of the SEND. BTW, your numbers are different that what we measured. See attached. Was measured on a high end host using QDR IB PCI gen 2 and Mellanox ofed 1.4. - Amir On 06/10/2009 05:47 PM, Zafar Gilani wrote: > Yes of course good question. Well the reason is that SDP allows two > approaches: BCopy (Buffered Copy) and ZCopy (Zero Copy or RDMA). BCopy > as I described is simple to use (change AF_INET to AF_INET_SDP in > socket() call and you are done, no code changes). Whereas ZCopy exploits > the RDMA capability of InfiniBand. In contrast IPoIB just provides IP > protocol encapsulation over InfiniBand. > > You can look at the benchmark graphs I generated at > [http://hpc.niit.edu.pk/~zafar/work/results_rocks/index.html] by > executing MPJ Express over IPoIB and SDP/BCopy. Now I want to implement > SDP/ZCopy for MPJ Express and I do not know how to do this. This is the > primary reason for my questions :). You can say this is sort of enabling > MPJ Express to work over wide range of IB protocols. > > More information on MPJ Express is available at: http://mpj-express.org/ > > Thank you, > Zafar > > On Wed, Jun 10, 2009 at 4:07 PM, Zafar Gilani > wrote: > > I read a paper on "Zero Copy Sockets Direct Protocol over > InfiniBand". I had > > a few questions, if anybody could answer I will be thankful. > > > > 1. SDP BCopy approach can be used without the change of the code via > the use > > of AF_INET_SDP parameter when calling socket() system call. But how can I > > implement SDP ZCopy approach if I want to implement ZCopy_Read and > > ZCopy_Write methods? > > > > 2. Is there any documentation regarding how to implement SDP ZCopy? Any > > Hello World sort of example code that I can use to learn how to work with > > this? > > > > 3. SDP BCopy is better for smaller messages, but how small? Is 64K a good > > threshold to change from BCopy to ZCopy at 128K? > > May I ask you why you are looking at SDP ? The performance of IPoIB in > connected mode and with large MTU is close to that of SDP, while the > former is a lot easier to use than the latter. > > Bart. > > > ------------------------------------------------------------------------ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Amir Vadai Software Eng. Mellanox Technologies mailto: amirv at mellanox.co.il Tel +972-3-6259539 -------------- next part -------------- A non-text attachment was scrubbed... Name: sdp_vs_ip_vs_10ge.png Type: image/png Size: 29636 bytes Desc: not available URL: From dotanba at gmail.com Wed Jun 10 23:49:57 2009 From: dotanba at gmail.com (Dotan Barak) Date: Thu, 11 Jun 2009 09:49:57 +0300 Subject: [ofa-general] There is a dead link in the page: http://www.openfabrics.org/downloads.htm Message-ID: <2f3bf9a60906102349i413a3e98kfe93c134a7ad718@mail.gmail.com> Hi. When i tried to press the link to "The previous version of the Downloads page can be found here", I got the following error page: Not Found The requested URL /OldDownloads.htm was not found on this server. Thanks Dotan From jean-vincent.ficet at bull.net Thu Jun 11 00:35:54 2009 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Thu, 11 Jun 2009 09:35:54 +0200 Subject: [ofa-general] [PATCH 0/2] Opensm support for external routing engines Message-ID: <4A30B3DA.30601@bull.net> Hi Sasha, We worked on a scheme that allows to dynamically load external routing engines. The plugin system is very similar to the perfmgr plugin implementation. It allows to provide customised vendor routing engines and algorithms whithout having to recompile opensm: Cheers, Vincent Nicolas Morey-Chaisemartin (2): opensm/osm_opensm.c: Added routing plugin interface. opensm: Added documentation for external routing engines. opensm/doc/current-routing.txt | 26 +++++++++- opensm/include/opensm/osm_opensm.h | 38 +++++++++++++- opensm/man/opensm.8.in | 2 + opensm/opensm/osm_event_plugin.c | 8 --- opensm/opensm/osm_opensm.c | 98 ++++++++++++++++++++++++++++++++++-- 5 files changed, 158 insertions(+), 14 deletions(-) From jean-vincent.ficet at bull.net Thu Jun 11 00:38:58 2009 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Thu, 11 Jun 2009 09:38:58 +0200 Subject: [ofa-general] [PATCH 1/2] opensm/osm_opensm.c: Added routing plugin interface. In-Reply-To: References: Message-ID: <4A30B492.1000001@bull.net> * Added a plugin interface for routing algorithms similar to event plugins * When a routing algorithm name cannot be found in the routing_engine_module, we try to open the file and load a struct using dlopen. * Plugin have to: - be named libPLUGIN_NAME.so - export a osm_routing_plugin_t struct named OSM_ROUTING_PLUGIN_IMPL_NAME that contains - osm_version equal to OSM_VERSION - a setup function which fills the routing_engine methods Signed-off-by: Jean-Vincent Ficet --- opensm/include/opensm/osm_opensm.h | 38 +++++++++++++- opensm/opensm/osm_event_plugin.c | 8 --- opensm/opensm/osm_opensm.c | 98 ++++++++++++++++++++++++++++++++++-- 3 files changed, 131 insertions(+), 13 deletions(-) diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index c121be4..b540fe0 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -70,6 +70,14 @@ #endif /* __cplusplus */ BEGIN_C_DECLS +/* Defines for routing and event plugin names */ +#if defined(PATH_MAX) +#define OSM_PATH_MAX (PATH_MAX + 1) +#elif defined (_POSIX_PATH_MAX) +#define OSM_PATH_MAX (_POSIX_PATH_MAX + 1) +#else +#define OSM_PATH_MAX 256 +#endif /****h* OpenSM/OpenSM * NAME * OpenSM @@ -105,7 +113,7 @@ typedef enum _osm_routing_engine_type { OSM_ROUTING_ENGINE_TYPE_FTREE, OSM_ROUTING_ENGINE_TYPE_LASH, OSM_ROUTING_ENGINE_TYPE_DOR, - OSM_ROUTING_ENGINE_TYPE_UNKNOWN + OSM_ROUTING_ENGINE_TYPE_EXTERNAL } osm_routing_engine_type_t; /***********/ @@ -126,6 +134,7 @@ struct osm_routing_engine { int (*ucast_build_fwd_tables) (void *context); void (*ucast_dump_tables) (void *context); void (*delete) (void *context); + void *handle; struct osm_routing_engine *next; }; /* @@ -150,6 +159,9 @@ struct osm_routing_engine { * The delete method, may be used for routing engine * internals cleanup. * +* handle +* Pointer to the handle (if it exists) for plugin routing algorithms. +* * next * Pointer to next routing engine in the list. */ @@ -237,6 +249,30 @@ typedef struct osm_opensm { * SEE ALSO *********/ +/****s* OpenSM: OpenSM/osm_routing_plugin_t +* NAME +* struct osm_routing_plugin_t +* +* DESCRIPTION +* OpenSM routing plugin definition. +* NOTES +* Routing plugin should export a osm_routing_plugin_t named OSM_EVENT_PLUGIN_IMPL_NAME +*/ +#define OSM_ROUTING_PLUGIN_IMPL_NAME "osm_routing_plugin" +typedef struct _osm_routing_plugin_t { + const char *osm_version; + int (*setup) (struct osm_routing_engine *, osm_opensm_t *); +} osm_routing_plugin_t; +/* +* FIELDS +* osm_version +* osm_version the plugin was built on. +* +* setup +* Plugin setup function which fills the osm_routing_engine with routing methods. +* +*/ + /****f* OpenSM: OpenSM/osm_opensm_construct * NAME * osm_opensm_construct diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c index c77494e..6c5ef05 100644 --- a/opensm/opensm/osm_event_plugin.c +++ b/opensm/opensm/osm_event_plugin.c @@ -52,14 +52,6 @@ #include #include -#if defined(PATH_MAX) -#define OSM_PATH_MAX (PATH_MAX + 1) -#elif defined (_POSIX_PATH_MAX) -#define OSM_PATH_MAX (_POSIX_PATH_MAX + 1) -#else -#define OSM_PATH_MAX 256 -#endif - /** * functions */ diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 50d1349..61d57bf 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -47,6 +47,7 @@ #include #include #include +#include #include #include #include @@ -103,7 +104,7 @@ const char *osm_routing_engine_type_str(IN osm_routing_engine_type_t type) default: break; } - return "unknown"; + return "external"; } /********************************************************************** @@ -129,7 +130,7 @@ osm_routing_engine_type_t osm_routing_engine_type(IN const char *str) else if (!strcasecmp(str, "dor")) return OSM_ROUTING_ENGINE_TYPE_DOR; else - return OSM_ROUTING_ENGINE_TYPE_UNKNOWN; + return OSM_ROUTING_ENGINE_TYPE_EXTERNAL; } /********************************************************************** @@ -153,7 +154,88 @@ static void append_routing_engine(osm_opensm_t *osm, r->next = routing_engine; } -static void setup_routing_engine(osm_opensm_t *osm, const char *name) +static void setup_plugin_routing_engine(osm_opensm_t * osm, const char *name) +{ + struct osm_routing_engine *re; + void *handle; + char lib_name[OSM_PATH_MAX]; + osm_routing_plugin_t *routing_plugin; + + snprintf(lib_name, sizeof(lib_name), "lib%s.so", name); + handle = dlopen(lib_name, RTLD_LAZY); + if (!handle) { + OSM_LOG(&osm->log, OSM_LOG_ERROR, + "cannot find or setup routing engine \'%s\'\n", name); + return; + } + + /* Allocate and initialize the routing engine struct */ + re = malloc(sizeof(struct osm_routing_engine)); + if (!re) { + OSM_LOG(&osm->log, OSM_LOG_VERBOSE, + "memory allocation failed\n"); + goto Exit_handle; + } + memset(re, 0, sizeof(struct osm_routing_engine)); + + /* Loading setup function from plugin */ + routing_plugin = + (osm_routing_plugin_t *) dlsym(handle, + OSM_ROUTING_PLUGIN_IMPL_NAME); + if (!routing_plugin) { + OSM_LOG(&osm->log, OSM_LOG_ERROR, + "Failed to find \"%s\" symbol in \"%s\" : \"%s\"\n", + OSM_ROUTING_PLUGIN_IMPL_NAME, lib_name, dlerror()); + goto Exit; + } + + /* Check the version to make sure this module will work with us */ + if (strcmp(routing_plugin->osm_version, osm->osm_version)) { + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Error loading routing plugin" + " \'%s\': OpenSM version mismatch - plugin was built" + " against %s version of OpenSM. Skip loading.\n", + name, routing_plugin->osm_version); + goto Exit; + } + + /* Check for the setup method */ + if (!routing_plugin->setup) { + OSM_LOG(&osm->log, OSM_LOG_ERROR, + "Error loading routing_plugin \'%s\': no setup() method.\n", + name); + goto Exit; + } + + /* Configure routing_engine struct */ + re->name = strdup(name); + if (!re->name) { + OSM_LOG(&osm->log, OSM_LOG_VERBOSE, + "memory allocation failed\n"); + goto Exit; + } + if (routing_plugin->setup(re, osm)) { + OSM_LOG(&osm->log, OSM_LOG_VERBOSE, + "setup of routing" " engine \'%s\' failed\n", name); + goto Exit_name; + } + re->handle = handle; + + OSM_LOG(&osm->log, OSM_LOG_DEBUG, + "\'%s\' routing engine set up\n", re->name); + append_routing_engine(osm, re); + + return; + +Exit_name: + free((char *)re->name); +Exit: + free(re); +Exit_handle: + dlclose(handle); + return; +} + +static void setup_routing_engine(osm_opensm_t * osm, const char *name) { struct osm_routing_engine *re; const struct routing_engine_module *m; @@ -182,8 +264,12 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) } } + /* As the algorithm is not in OpenSM, we try to load it using dlopen */ OSM_LOG(&osm->log, OSM_LOG_ERROR, - "cannot find or setup routing engine \'%s\'\n", name); + " \'%s\' is a not a standard OpenSM algorithm. Trying to open it as a routing plugin\n", + name); + setup_plugin_routing_engine(osm, name); + } static void setup_routing_engines(osm_opensm_t *osm, const char *engine_names) @@ -234,6 +320,10 @@ static void destroy_routing_engines(osm_opensm_t *osm) next = r->next; if (r->delete) r->delete(r->context); + if (r->handle) { + free((char *)r->name); + dlclose(r->handle); + } free(r); } } -- 1.6.2.rc1.30.gd43c From jean-vincent.ficet at bull.net Thu Jun 11 00:39:10 2009 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Thu, 11 Jun 2009 09:39:10 +0200 Subject: [ofa-general] [PATCH 2/2] opensm: Added documentation for external routing engines. In-Reply-To: References: Message-ID: <4A30B49E.3030800@bull.net> Signed-off-by: Jean-Vincent Ficet --- opensm/doc/current-routing.txt | 26 +++++++++++++++++++++++++- opensm/man/opensm.8.in | 2 ++ 2 files changed, 27 insertions(+), 1 deletions(-) diff --git a/opensm/doc/current-routing.txt b/opensm/doc/current-routing.txt index 1302860..b0dc5bc 100644 --- a/opensm/doc/current-routing.txt +++ b/opensm/doc/current-routing.txt @@ -1,7 +1,7 @@ Current OpenSM Routing 7/9/07 -OpenSM offers five routing engines: +OpenSM offers six routing engines: 1. Min Hop Algorithm - based on the minimum hops to each node where the path length is optimized. @@ -28,6 +28,10 @@ two switches. This provides deadlock free routes for hypercubes when the fabric is cabled as a hypercube and for meshes when cabled as a mesh (see details below). +6. External routing engine - OpenSM provides the ability to dynamically load +an external routing engine. + + OpenSM provides an optional unicast routing cache (enabled by -A or --ucast_cache options). When enabled, unicast routing cache prevents routing recalculation (which is a heavy task in a large cluster) when @@ -388,3 +392,23 @@ ports, one port on one end of the cable, and the other port on the other end, continuing along the mesh dimension. Use '-R dor' option to activate the DOR algorithm. + +External routing engines +------------------------ + +OpenSM also offer the ability to dynamically load routing plugins. +When OpenSM is provided with a routing engine name which is not in the +standard list, OpenSM will try to open a library called lib'engine_name'.so. +There are three things required to build a fonctional plugin: + - should be included first in the plugin to + ensure the right defines and struct are used. + - a setup function that fills a routing_engine struct with the + appropriate methods + - a osm_routing_plugin_t struct named OSM_ROUTING_PLUGIN_IMPL_NAME + which contains: + * osm_version = OSM_VERSION to make sure the plugin was build + with the same OpenSM Version + * setup pointing to the setup function described above. +If these requirements are filled, OpenSM will load the external plugin and +setup the routing engine. It can then be used as any other routing engine. + diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index 66d2fe6..6a6fb30 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -141,6 +141,8 @@ algorithm (default). Multiple routing engines can be specified separated by commas so that specific ordering of routing algorithms will be tried if earlier routing engines fail. Supported engines: minhop, updn, file, ftree, lash, dor +If the algorithm name cannot be found within the standard list, +OpenSM will try do open an external library named lib'name'.so. .TP \fB\-\-do_mesh_analysis\fR This option enables additional analysis for the lash routing engine to -- 1.6.2.rc1.30.gd43c From tziporet at dev.mellanox.co.il Thu Jun 11 01:58:40 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 11 Jun 2009 11:58:40 +0300 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: <4A30502C.5090405@linux.vnet.ibm.com> References: <4A300788.1000001@linux.vnet.ibm.com> <4A30203E.9050309@linux.vnet.ibm.com> <3322AD31CA7140D3A5A14E510E978A0E@amr.corp.intel.com> <4A30502C.5090405@linux.vnet.ibm.com> Message-ID: <4A30C740.3010000@mellanox.co.il> Pradeep Satyanarayana wrote: > > Sean, You are right about the IB MTU. My guess was that he was trying to set both > (based on the 2044 in his mail). > > I use the embedded SM rather than openSM. Hence guidance on that will be helpful. > > If you are using ConnectX and OFED 1.4.1 we added a module parameter to work with 4K mtu: set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (default 0) All the InfiniHost devices support only 2K MTU Regarding the switches: Infiniscale IV supports 4K MTU by defualt You also need to configure OpenSM to support 4K MTU - but someone from management people should answer this Tziporet From vlad at lists.openfabrics.org Thu Jun 11 02:20:35 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 11 Jun 2009 02:20:35 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090611-0200 daily build status Message-ID: <20090611092035.BEE1C102027E@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: Build failed on x86_64 with linux-2.6.18-8.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:37: include/linux/slub_def.h: In function 'kmalloc_index': include/linux/slub_def.h:90: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/.tmp_umem.o /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090611-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From zafargilani at gmail.com Thu Jun 11 02:55:02 2009 From: zafargilani at gmail.com (Zafar Gilani) Date: Thu, 11 Jun 2009 14:55:02 +0500 Subject: [ofa-general] SDP - How to? In-Reply-To: <4A30A296.60109@mellanox.co.il> References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> <4A30A296.60109@mellanox.co.il> Message-ID: <7d4423d30906110255s4f03f764h31171b141514643f@mail.gmail.com> Hi, Thank you for reply! My numbers are different may be because I am executing MPJ Express (pure Java MPJ) over SDP. However now I need to implement code in C now (natively). Any documentation that I may be able to use for programming MPI using C Sockets and SDP? Thanks On Thu, Jun 11, 2009 at 11:22 AM, Amir Vadai wrote: > Hi Zafar, > > I am the maintainer of SDP in OFED. > > The current SDP implementation does not have ZCopy. What we have is BCopy > and something that we call BZcopy. > Small packets (<=32k) are sent using BCopy - meaning the SEND verb is in > use and data is not zero copied. > Packets bigger than 32K are sent using BZcopy - they are also sent using > the SEND verb but they are zero copied on the sender side. on the RX side > packets are memcpy'ed. > When we implement ZCopy it will be automatic too - when sending packets > bigger than a threshold the driver will automatically use RDMA verb instead > of the SEND. > > BTW, your numbers are different that what we measured. See attached. Was > measured on a high end host using QDR IB PCI gen 2 and Mellanox ofed 1.4. > > - Amir > Zafar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at voltaire.com Thu Jun 11 03:15:41 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 11 Jun 2009 13:15:41 +0300 (IDT) Subject: [ofa-general] high latency results with qperf Message-ID: Hi Johann, Using qperf I see unexplained high latancy of around 8-10us, where if I use a different utility on the same setup (e.g ib_rdma_lat, mpi) I get latency of about 1-2us, any idea? The setup is DDR based, connectx, this test was done with qperf-0.4.0-1.ofed1.3.1 but it happens also with later and probably also with the latest bits. Or. # qperf h2 -m 4 -t 20 rc_lat rc_lat: latency = 10.6 us # qperf h2 -m 4 -t 20 ud_lat ud_lat: latency = 8.64 us # ib_rdma_lat h2 -s 4 Latency typical: 1.49403 usec Latency best : 1.47152 usec Latency worst : 5.8651 usec From ogerlitz at voltaire.com Thu Jun 11 03:18:42 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 11 Jun 2009 13:18:42 +0300 (IDT) Subject: [ofa-general] Re: high latency results with qperf In-Reply-To: References: Message-ID: to make it more a-2-a I also tested with ib_send_lat and qperf's rc_rdma_read_lat, same story. # ib_send_lat h2 -s 4 ------------------------------------------------------------------ Send Latency Test Inline data is used up to 400 bytes message Connection type : RC local address: LID 0x04 QPN 0x18004d PSN 0x42dd33 remote address: LID 0x01 QPN 0x7a004b PSN 0x1aa1ed Mtu : 2048 ------------------------------------------------------------------ #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 4 1000 1.62 25.29 1.67 ------------------------------------------------------------------ # qperf h2 -m 4 -t 20 rc_rdma_write_lat rc_rdma_write_lat: latency = 10.2 us From hal.rosenstock at gmail.com Thu Jun 11 03:36:40 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 11 Jun 2009 06:36:40 -0400 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: <4A30C740.3010000@mellanox.co.il> References: <4A300788.1000001@linux.vnet.ibm.com> <4A30203E.9050309@linux.vnet.ibm.com> <3322AD31CA7140D3A5A14E510E978A0E@amr.corp.intel.com> <4A30502C.5090405@linux.vnet.ibm.com> <4A30C740.3010000@mellanox.co.il> Message-ID: On Thu, Jun 11, 2009 at 4:58 AM, Tziporet Koren wrote: > Pradeep Satyanarayana wrote: >> >> Sean, You are right about the IB MTU. My guess was that he was trying to >> set both >> (based on the 2044 in his mail). >> >> I use the embedded SM rather than openSM. Hence guidance on that will be >> helpful. >> >> > > If you are using ConnectX and OFED 1.4.1 we added a module parameter to work > with 4K mtu: > >  set_4k_mtu:          attempt to set 4K MTU to all ConnectX ports (default > 0) > > All the InfiniHost devices support only 2K MTU > > Regarding the switches: Infiniscale IV supports 4K MTU by defualt > > You also need to configure OpenSM to support 4K MTU - but someone from > management people should answer this Nothing needs to be done to OpenSM. OpenSM uses the largest MTU common to the peers at the end of the links so this depends on the MTUCap advertised by the devices. -- Hal > Tziporet > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From jsquyres at cisco.com Thu Jun 11 04:48:42 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Thu, 11 Jun 2009 04:48:42 -0700 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A2E6878.6080200@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il><200906081707.43472.cap@nsc.liu.se> <4A2E6878.6080200@dev.mellanox.co.il> Message-ID: <10CDE7FD-3D89-43AF-9E25-F9B76630FBE5@cisco.com> On Jun 9, 2009, at 6:49 AM, Pavel Shamis (Pasha) wrote: > The excluding of MPI from OFED, will only push users to use vendor > specific OFED builds (that provides MPI out of box) and I'm not sure > that it is good for OFED community. > Pasha and I discussed exactly this point in IM and we agreed to disagree. :-) FWIW, I think this point touches on an issue that is deeper than just MPI in OFED. Different vendors having their own [potentially incompatible] versions of OFED -- each with different value-add -- is both good and bad. GOOD: vendors can innovate and differentiate BAD: it seems like the bad old days of different/incompatible vendor versions of mVAPI: OFED != OFED != OFED I've heard similar stories from many users, "I have OFED -- is that the same/compatible as OFED?" And sometimes the answer is "not entirely". Yow! We need a technology/system/process/whatever for allowing vendors to distribute what they need without effectively forking OFED to make their own " OFED". Even if OFED *should* be the same as community OFED, it sometimes (?usually?) is different in very subtle/small-but-important ways (e.g., vendorX compiling/ installing vendorY's drivers, but not QA'ing them). To be clear: with each vendor putting out their own different versions of OFED, it makes for big user confusion about compatibility and ecosystem. Note that Pasha's answer at least somewhat implies that he feels the same way. :-) -- Jeff Squyres Cisco Systems From tziporet at mellanox.co.il Thu Jun 11 08:14:21 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Thu, 11 Jun 2009 18:14:21 +0300 Subject: [ofa-general] OFED 1.5 daily build have started Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD0300080A@mtlexch01.mtl.com> Hi All, We started OFED 1.5 daily build (see http://lists.openfabrics.org/pipermail/general/2009-June/060016.html) As expected several kernel modules still fail - each owner please fix the failures We also have daily builds: http://www.openfabrics.org/downloads/OFED/ofed-1.5-daily/ So now is the time for all maintainers to work on their apps/modules and make sure we have a working package next week Thanks Tziporet & Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From cl at linux-foundation.org Thu Jun 11 08:13:47 2009 From: cl at linux-foundation.org (Christoph Lameter) Date: Thu, 11 Jun 2009 11:13:47 -0400 (EDT) Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: <4A276F9D.6080504@Voltaire.com> References: <4A266ABB.6000603@Voltaire.com> <4A276F9D.6080504@Voltaire.com> Message-ID: Could you reapply the patch? The semantics are exaclty as demanded by David Miller. It blocks the process that is doing the join and not the others. From leonid at mellanox.co.il Thu Jun 11 08:28:07 2009 From: leonid at mellanox.co.il (Leonid Keller) Date: Thu, 11 Jun 2009 18:28:07 +0300 Subject: [ofa-general] RE: [ofw] skipping QP states during transitions In-Reply-To: <38877AA1B953874AAE8FBAEE42569B100146A1@TK5EX14MBXW653.wingroup.windeploy.ntdev.microsoft.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD02E72E65@mtlexch01.mtl.com> <38877AA1B953874AAE8FBAEE42569B100146A1@TK5EX14MBXW653.wingroup.windeploy.ntdev.microsoft.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD03000827@mtlexch01.mtl.com> Mellanox HCA doesn't support multiple commands to the same QP. Driver prevents this case with a mutex, taken in the beginning of modify-qp and query_qp. > -----Original Message----- > From: ofw-bounces at lists.openfabrics.org > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Fab Tillier > Sent: Friday, June 05, 2009 9:04 PM > To: Sean Hefty; Tzachi Dar; general at lists.openfabrics.org; > ofw at lists.openfabrics.org > Subject: RE: [ofw] skipping QP states during transitions > > >> Look at the IB spec on section 10.3 > > > > I was just exploring whether any hardware, separate from > the existing > > software stacks, supported 'skipping' QP states -- assuming > necessary > > values for the other states were also given. In theory, hardware > > could walk through the states internally. The motivation is to > > decrease the time to connect QPs by reducing the number of commands > > that need to be issued to the hardware. > > Assuming the hardware processes commands in the order they're > submitted (at least per serializes commands by resource), the > HCA driver should be able to queue up the various modify > commands so that they are executed by the HCA driver > back-to-back. A failure anywhere in this process should > cause all subsequent commands to fail due to the QP being in > the wrong state. > > A single IOCTL could generate multiple back-to-back > asynchronous QP modify requests. > > This would eliminate the command completion processing time > from the QP modify. > > However, you need to have async QP modify support in the HCA > driver, and ideally the HW needs to be able to process > multiple commands issued against a single QP properly > (Tzachi, is this something the Mellanox HCAs support?) Note > that if the HW driver cannot process multiple commands > outstanding for a single resource, the driver could queue up > the commands and issue the next from the DPC as soon as the > first is completed. In the current model, the DPC has to > wake up the blocked thread which does whatever post > processing of the command and unwinds to the caller, who then > has to make the next blocking QP modify call. > > -Fab > _______________________________________________ > ofw mailing list > ofw at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > From suri at baymicrosystems.com Thu Jun 11 09:07:26 2009 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Thu, 11 Jun 2009 12:07:26 -0400 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> Message-ID: <1A4894EC8A5E48F2B9D99D8121D696BE@md.baymicrosystems.com> Sean: I mis-read the 'If' condition, see how the below looks: I have a related question: Why is send_queue.count incremented even when the wr goes on the overflow list? BTW, I am trying to do all this because in the same thread (ib_post_send calls Our driver post_send etc...) I have to call dev_queue_xmit which complains when Irqs are disabled. I can get around that different ways in my driver (in a not so elegant manner), hence all this.... ---------------------------------------------------------------- spin_lock_irqsave(&qp_info->send_queue.lock, flags); if (qp_info->send_queue.count < qp_info->send_queue.max_active) { + qp_info->send_queue.count++; + spin_unlock_irqrestore(&qp_info->send_queue.lock, flags); ret = ib_post_send(mad_agent->qp, &mad_send_wr->send_wr, &bad_send_wr); + spin_lock_irqsave(&qp_info->send_queue.lock, flags); list = &qp_info->send_queue.list; } else { ret = 0; + qp_info->send_queue.count++; list = &qp_info->overflow_list; } if (!ret) list_add_tail(&mad_send_wr->mad_list.list, list); + else + qp_info->send_queue.count--; --------------------------------------------------------------------- Many thanks, Suri > -----Original Message----- > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Wednesday, June 10, 2009 3:07 PM > To: 'Suresh Shelvapille'; general at lists.openfabrics.org > Cc: 'Roland Dreier' > Subject: RE: [ofa-general] spin_lock_irqsave in ib_send_mad > > >---------------------------------------------------- > >if (qp_info->send_queue.count < qp_info->send_queue.max_active) { > > This loses the locking around send_queue.count, which can allow trying to post > more sends than the QP supports. > > > ret = ib_post_send(mad_agent->qp, &mad_send_wr->send_wr, > > &bad_send_wr); > > spin_lock_irqsave(&qp_info->send_queue.lock, flags); > > list = &qp_info->send_queue.list; > >} else { > > ret = 0; > > spin_lock_irqsave(&qp_info->send_queue.lock, flags); > > list = &qp_info->overflow_list; > >} > > I think the code will end up quite a bit more complex not locking around > ib_post_send, but maybe there's a clever way to handle this. > > - Sean From akepner at sgi.com Thu Jun 11 09:33:22 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Thu, 11 Jun 2009 09:33:22 -0700 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM In-Reply-To: <4A300EFD.3050303@voltaire.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> Message-ID: <20090611163322.GB32601@sgi.com> On Wed, Jun 10, 2009 at 10:52:29PM +0300, Yossi Etigin wrote: > akepner at sgi.com wrote: > > @@ -810,9 +810,6 @@ void ipoib_cm_handle_tx_wc(struct net_device *dev, struct ib_wc *wc) > > list_del(&neigh->list); > > if (neigh->ah) > > ipoib_put_ah(neigh->ah); > > - ipoib_neigh_free(dev, neigh); > > - > > - tx->neigh = NULL; > > } > > > > if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &tx->flags)) { > > Can't calling ipoib_put_ah() and keeping the neighbour alive cause neigh->ah > pointer become invalid? What about doing list_del(&neigh->list) twice? > Yeah, looks like there's a hole there. By changing the test from: if (neigh) { neigh->cm = NULL; to: if (neigh && neigh->cm) { neigh->cm = NULL; it can be closed. > ...... > > The piece of code that does '*to_ipoib_neigh(neigh->neighbour) = NULL' and > releases skb's will appear 3 times in ipoib. I think it can be put in a > separate function. OK. > > > > @@ -1343,7 +1351,23 @@ static void ipoib_cm_tx_reap(struct work_struct *work) > > spin_lock_irqsave(&priv->lock, flags); > > > > while (!list_empty(&priv->cm.reap_list)) { > > + struct ipoib_neigh *neigh; > > + struct sk_buff *skb; > > p = list_entry(priv->cm.reap_list.next, typeof(*p), list); > > + neigh = p->neigh; > > + if (neigh) { > > + *to_ipoib_neigh(neigh->neighbour) = NULL; > > + ipoib_dbg(priv, "%s: releasing ref to %pI6 " > > + "(refcnt: %d)\n", __func__, neigh->dgid.raw, > > + atomic_read(&neigh->neighbour->refcnt)); > > + neigh_release(neigh->neighbour); > > + while ((skb = __skb_dequeue(&neigh->queue))) { > > + ++dev->stats.tx_dropped; > > + dev_kfree_skb_any(skb); > > + } > > + kfree(neigh); > > + } > > + p->neigh = NULL; > > list_del(&p->list); > > spin_unlock_irqrestore(&priv->lock, flags); > > netif_tx_unlock_bh(dev); > > Couldn't this race with ipoib_neigh_cleanup()? > Generally the fact that we hold a reference should prevent ipoib_neigh_cleanup() from being invoked. But, alas, there are a couple of cases where the neighbour can be have the neigh_cleanup() method invoked even when there are references held (e.g., when the interface goes down). So, yeah, there seems to be a hole there. A smaller hole, but still.... -- Arthur From charlesz at opengridcomputing.com Thu Jun 11 09:38:21 2009 From: charlesz at opengridcomputing.com (charles zhuang) Date: Thu, 11 Jun 2009 11:38:21 -0500 Subject: [ofa-general] crc32c module problem on iscsi connection Message-ID: <1F7361B08914417B997A09CF43502392@czdesktop> Hi, I have 2 questions. 1. I am using centos 5.3 (kernel 2.6.18) + ofed 1.4 as my initiator, when I try to login in my target (also install ofed 1.4) using iscsi (tcp or IPoIB), there's following errors in the initiator side: iscsi: registered transport (tcp) scsi5 : iSCSI Initiator over TCP/IP connection4:0: Could not create connection due to crc32c loading error. Make sure the crc32c module is built as a module or into the kernel session4: couldn't create a new connection.<6>scsi6 : iSCSI Initiator over TCP/IP There's no crc32c.ko and libcrc32c.ko under /lib/modules. How can I solve this problem? 2. Using the same target machine but with different initiator which install open-iscsi package, that will allow me to correctly login to the target. So what's the relationship between open-iscsi package and ofed package? Can they be installed on the same machine and use at the same time? Thanks for any help. Thanks, Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Thu Jun 11 09:54:21 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 11 Jun 2009 09:54:21 -0700 Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: (Christoph Lameter's message of "Thu, 11 Jun 2009 11:13:47 -0400 (EDT)") References: <4A266ABB.6000603@Voltaire.com> <4A276F9D.6080504@Voltaire.com> Message-ID: > Could you reapply the patch? The semantics are exaclty as demanded by > David Miller. It blocks the process that is doing the join and not the > others. No, the semantics are not the same. It doesn't block until socket space is exhausted, and it will no longer work if the net stack does skb_orphan before giving the packet to the network driver. From cl at linux-foundation.org Thu Jun 11 10:08:13 2009 From: cl at linux-foundation.org (Christoph Lameter) Date: Thu, 11 Jun 2009 13:08:13 -0400 (EDT) Subject: [ofa-general] [PATCH/RFC] IPoIB: Don't drop multicast packets sent before group is joined In-Reply-To: References: <4A266ABB.6000603@Voltaire.com> <4A276F9D.6080504@Voltaire.com> Message-ID: On Thu, 11 Jun 2009, Roland Dreier wrote: > > Could you reapply the patch? The semantics are exaclty as demanded by > > David Miller. It blocks the process that is doing the join and not the > > others. > > No, the semantics are not the same. It doesn't block until socket space > is exhausted, and it will no longer work if the net stack does > skb_orphan before giving the packet to the network driver. So it does block (the socket space for one process is exhausted!) and it will work until the skb_orphan goes in. At that point different methods will be available for the regular NICs as well. From yosefe at voltaire.com Thu Jun 11 10:11:10 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Thu, 11 Jun 2009 20:11:10 +0300 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM In-Reply-To: <20090611163322.GB32601@sgi.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> Message-ID: <4A313AAE.2070108@voltaire.com> akepner at sgi.com wrote: > Yeah, looks like there's a hole there. By changing the test from: > > if (neigh) { > neigh->cm = NULL; > > to: > > if (neigh && neigh->cm) { > neigh->cm = NULL; > > it can be closed. It does not close it completely. What if ipoib_neigh_cleanup() is called after you do ipoib_put_ah(neigh->ah) in the CM code? > > Generally the fact that we hold a reference should prevent > ipoib_neigh_cleanup() from being invoked. > > But, alas, there are a couple of cases where the neighbour can > be have the neigh_cleanup() method invoked even when there are > references held (e.g., when the interface goes down). > > So, yeah, there seems to be a hole there. A smaller hole, but > still.... > How about checking the refcnt in the neigh_cleanup function, and if it's not 1 give up the ipoib cleanup? From sean.hefty at intel.com Thu Jun 11 10:57:38 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 11 Jun 2009 10:57:38 -0700 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: <1A4894EC8A5E48F2B9D99D8121D696BE@md.baymicrosystems.com> References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> <1A4894EC8A5E48F2B9D99D8121D696BE@md.baymicrosystems.com> Message-ID: <34BACFADF9C247DD9FD090C7FFB31FB0@amr.corp.intel.com> > spin_lock_irqsave(&qp_info->send_queue.lock, flags); > if (qp_info->send_queue.count < qp_info->send_queue.max_active) { >+ qp_info->send_queue.count++; > >+ spin_unlock_irqrestore(&qp_info->send_queue.lock, flags); > > ret = ib_post_send(mad_agent->qp, &mad_send_wr->send_wr, > &bad_send_wr); > >+ spin_lock_irqsave(&qp_info->send_queue.lock, flags); > list = &qp_info->send_queue.list; > } else { > ret = 0; >+ qp_info->send_queue.count++; > list = &qp_info->overflow_list; > } > > if (!ret) > list_add_tail(&mad_send_wr->mad_list.list, list); >+ else >+ qp_info->send_queue.count--; It's not quite this simple. Once the lock is released before calling ib_post_send, another thread could come down and queue a MAD to the overflow list. If ib_post_send fails, the overflow list must be checked to see if a queued mad should now be sent. As for being able to hold a lock when calling ib_post_send, that's something that should be allowed. - Sean From suri at baymicrosystems.com Thu Jun 11 11:52:24 2009 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Thu, 11 Jun 2009 14:52:24 -0400 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: <34BACFADF9C247DD9FD090C7FFB31FB0@amr.corp.intel.com> References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> <1A4894EC8A5E48F2B9D99D8121D696BE@md.baymicrosystems.com> <34BACFADF9C247DD9FD090C7FFB31FB0@amr.corp.intel.com> Message-ID: <7BC0833A80E7481DBE26C697101FF7FD@md.baymicrosystems.com> > It's not quite this simple. Once the lock is released before calling > ib_post_send, another thread could come down and queue a MAD to the overflow > list. If ib_post_send fails, the overflow list must be checked to see if a > queued mad should now be sent. > Why check the overflow list only when the ib_post_send fails? Don't you need to do this regardless? It looks like you could get stuff into the overflow list even with the existing code... From sean.hefty at intel.com Thu Jun 11 11:56:22 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 11 Jun 2009 11:56:22 -0700 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: <7BC0833A80E7481DBE26C697101FF7FD@md.baymicrosystems.com> References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> <1A4894EC8A5E48F2B9D99D8121D696BE@md.baymicrosystems.com> <34BACFADF9C247DD9FD090C7FFB31FB0@amr.corp.intel.com> <7BC0833A80E7481DBE26C697101FF7FD@md.baymicrosystems.com> Message-ID: >Why check the overflow list only when the ib_post_send fails? Don't you >need to do this regardless? It looks like you could get stuff into the overflow >list even with the existing code... You only need to check it when decrementing send_queue.count, which is currently only after a send completes. From suri at baymicrosystems.com Thu Jun 11 13:15:12 2009 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Thu, 11 Jun 2009 16:15:12 -0400 Subject: [ofa-general] spin_lock_irqsave in ib_send_mad In-Reply-To: References: <4A2BD9B9.6000802@mellanox.co.il> <71B4C97204FE4F81B6D00E60B960978D@amr.corp.intel.com> <1A4894EC8A5E48F2B9D99D8121D696BE@md.baymicrosystems.com> <34BACFADF9C247DD9FD090C7FFB31FB0@amr.corp.intel.com> <7BC0833A80E7481DBE26C697101FF7FD@md.baymicrosystems.com> Message-ID: <9A94A6088CC74109AD4D2B7AD25E1945@md.baymicrosystems.com> Sean: Ah...now I get it, it does appear like more code would have to be added to handle the overflow list when post_send fails(as in mad_send_done_handler). I think I will fix my driver for now. Many Thanks for your help, Suri > -----Original Message----- > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Thursday, June 11, 2009 2:56 PM > To: 'Suresh Shelvapille'; general at lists.openfabrics.org > Cc: 'Roland Dreier' > Subject: RE: [ofa-general] spin_lock_irqsave in ib_send_mad > > >Why check the overflow list only when the ib_post_send fails? Don't you > >need to do this regardless? It looks like you could get stuff into the overflow > >list even with the existing code... > > You only need to check it when decrementing send_queue.count, which is currently > only after a send completes. From ahomike at us.ibm.com Thu Jun 11 14:27:25 2009 From: ahomike at us.ibm.com (Mike Aho) Date: Thu, 11 Jun 2009 16:27:25 -0500 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: <4A30203E.9050309@linux.vnet.ibm.com> References: <4A300788.1000001@linux.vnet.ibm.com> <4A30203E.9050309@linux.vnet.ibm.com> Message-ID: Pradeep, The configuration was the "datagram". Wow, that was great to know how to find and fix it. Thank you very much. So how does that value get set to "connected" or "datagram"? The opensm daemon was running on the side which had datagram setting. From: Pradeep Satyanarayana To: Mike Aho/Rochester/IBM at IBMUS Cc: OpenFabrics General Date: 06/10/2009 04:06 PM Subject: Re: [ofa-general] Setting mtu on an IB port Mike Aho wrote: > Pradeep, > > Thank you, that works well on one machine. I was able to set the mtu on > interface ib1 to 4096 instead of 2044. The max_mtu from ibv_devinfo is > still 2048 on port 2. > On the other machine, setting the ip mtu fails if I try a number higher > than 2044. So which one is the correct behavior? By the way, opensmd is > running on the other machine. > > I appreciate having the capability to do ip mtu and now have another > command in my arsenal. > > I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the > ibv_devinfo. Any ideas on that? > Mike, The issue here is likely that one of the nodes/ports may be in datagram mode and the other one in connected mode. cat /sys/class/net/ib0/mode will reveal the mode for ib0. The datagram mode will limit the mtu to 2044 and one can typically go up to 65520 with connected mode. If you would like to switch to connected mode, you could (see caveats below) do a echo connected > /sys/class/net/ib0/mode followed by the ip link command. Caveats ------- The above (echo) command should be okay if you are trying with a mainline kernel, ODED-1.4 or OFED-1.4.1 installation, or even Sles varaints However, this will not work Rhel5.3 (if I remember correctly, should work for Rhel5.2). For Rhel5.3, you need to edit the /etc/sysconfig/network-scripts/ifcfg-ib* and insert an entry CONNECTED_MODE=yes, followed by an /etc/init.d/openibd restart. I think that should achieve what you want. Let me know if that doesn't work. Pradeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From pradeep at us.ibm.com Thu Jun 11 15:01:46 2009 From: pradeep at us.ibm.com (Pradeep Satyanarayana) Date: Thu, 11 Jun 2009 15:01:46 -0700 Subject: [ofa-general] Setting mtu on an IB port In-Reply-To: Message-ID: general-bounces at lists.openfabrics.org wrote on 06/11/2009 02:27:25 PM: > > Pradeep, The configuration was the "datagram". Wow, that was > great to know how to find and fix it. Thank you very much. > > So how does that value get set to "connected" or "datagram"? The > opensm daemon was running on the side which had datagram setting. Mike, That depends on what distro you are running, or if you are running> an OFED build directly. Connected vs datagram is an IPoIB setting and has nothing to do with opensm. Pradeep pradeep at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From weiny2 at llnl.gov Thu Jun 11 15:18:14 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 11 Jun 2009 15:18:14 -0700 Subject: [ofa-general] When is the next planned release of libmlx4? Message-ID: <20090611151814.95f7ae64.weiny2@llnl.gov> Our MPI folks detected a hang while using Valgrind with our ConnectX cards. After trying the current master branch in git we solved the problem by applying this patch from the git tree to v1.0. Don't use memcpy() to write blueflame sends Some memcpy() implementations may use move-string-buffer assembly instructions, which do not guarantee copy order into the blueflame buffer. This causes problems when writing into a blueflame buffer, so use our own copy function instead. Signed-off-by: Jack Morgenstein Signed-off-by: Roland Dreier While doing this I noticed that the last release tag in the git tree is way back in Dec 2007. Is there a planned release of libmlx4 soon? What version is OFED pulling? RedHat is still using 1.0. Thanks, Ira -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab weiny2 at llnl.gov From rdreier at cisco.com Thu Jun 11 15:51:12 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 11 Jun 2009 15:51:12 -0700 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: <20090611151814.95f7ae64.weiny2@llnl.gov> (Ira Weiny's message of "Thu, 11 Jun 2009 15:18:14 -0700") References: <20090611151814.95f7ae64.weiny2@llnl.gov> Message-ID: > Our MPI folks detected a hang while using Valgrind with our ConnectX cards. > After trying the current master branch in git we solved the problem by applying > this patch from the git tree to v1.0. > > Don't use memcpy() to write blueflame sends Didn't realize this had that implication (I thought it just made blueflame not give latency benefit). Anyway yes it has been a while since a libmlx4 release. I'll make one soon, probably next week. From vlad at lists.openfabrics.org Fri Jun 12 02:18:59 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 12 Jun 2009 02:18:59 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090612-0200 daily build status Message-ID: <20090612091859.5E48D102061B@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:37: include/linux/slub_def.h: In function 'kmalloc_index': include/linux/slub_def.h:90: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/.tmp_umem.o /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090612-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From yhlu.kernel at gmail.com Thu Jun 11 19:45:21 2009 From: yhlu.kernel at gmail.com (Yinghai Lu) Date: Thu, 11 Jun 2009 19:45:21 -0700 Subject: [ofa-general] mthca double free irqs Message-ID: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> [ 604.800358] ib_mthca: Initializing 0000:c4:00.0 [ 604.803593] ib_mthca 0000:c4:00.0: PCI INT A -> GSI 98 (level, low) -> IRQ 98 [ 604.820587] ib_mthca 0000:c4:00.0: setting latency timer to 64 [ 604.824369] ib_mthca 0000:c4:00.0: using 64bit DMA mask [ 604.840003] ib_mthca 0000:c4:00.0: using 64bit consistent DMA mask [ 604.845961] reserve_memtype added 0xfbef0000-0xfbef1000, track uncached-minus, req uncached-minus, ret uncached-minus [ 604.863794] free_memtype request 0xfbef0000-0xfbef1000 [ 605.882922] reserve_memtype added 0xfbe80000-0xfbe81000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.179925] reserve_memtype added 0xfa800000-0xfa801000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.222053] ib_mthca 0000:c4:00.0: HCA FW version 4.6.002 is old (4.8.200 is current). [ 606.229839] ib_mthca 0000:c4:00.0: If you have problems, try updating your HCA FW. [ 606.239838] reserve_memtype added 0xfbe82000-0xfbe83000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.259063] alloc irq_desc for 232 on node 0 [ 606.263133] alloc kstat_irqs on node 0 [ 606.277766] ib_mthca 0000:c4:00.0: irq 232 for MSI/MSI-X [ 606.282177] alloc irq_desc for 233 on node 0 [ 606.286160] alloc kstat_irqs on node 0 [ 606.299309] ib_mthca 0000:c4:00.0: irq 233 for MSI/MSI-X [ 606.303756] alloc irq_desc for 234 on node 0 [ 606.319225] alloc kstat_irqs on node 0 [ 606.322687] ib_mthca 0000:c4:00.0: irq 234 for MSI/MSI-X [ 606.337147] reserve_memtype added 0xfa802000-0xfa803000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.357274] reserve_memtype added 0xe5800000-0xe6000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.366082] reserve_memtype added 0xe0000000-0xe4000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.389823] reserve_memtype added 0xfbef0000-0xfbef1000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.401964] Overlap at 0xfbe80000-0xfbe81000 [ 606.418913] reserve_memtype added 0xfbe80000-0xfbe81000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.538429] ib_mthca 0000:c4:00.0: NOP command failed to generate interrupt (IRQ 234). [ 606.543348] ib_mthca 0000:c4:00.0: Trying again with MSI-X disabled. [ 606.560852] free_memtype request 0xfbe80000-0xfbe81000 [ 606.564779] free_memtype request 0xfbef0000-0xfbef1000 [ 606.578039] free_memtype request 0xe0000000-0xe4000000 [ 606.581770] free_memtype request 0xe5800000-0xe6000000 [ 606.597795] free_memtype request 0xfa802000-0xfa803000 [ 606.600688] free_memtype request 0xfbe82000-0xfbe83000 [ 606.616891] reserve_memtype added 0xfa802000-0xfa803000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.639208] reserve_memtype added 0xe5800000-0xe6000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.647364] reserve_memtype added 0xe0000000-0xe4000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.671960] reserve_memtype added 0xfbef0000-0xfbef1000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.686785] Overlap at 0xfbe80000-0xfbe81000 [ 606.696978] reserve_memtype added 0xfbe80000-0xfbe81000, track uncached-minus, req uncached-minus, ret uncached-minus [ 606.818317] ib_mthca 0000:c4:00.0: NOP command failed to generate interrupt (IRQ 98), aborting. [ 606.826709] ib_mthca 0000:c4:00.0: BIOS or ACPI interrupt routing problem? [ 606.839353] ------------[ cut here ]------------ [ 606.841560] WARNING: at kernel/irq/manage.c:740 __free_irq+0xad/0x17f() [ 606.859197] Hardware name: System Name [ 606.861049] Trying to free already-free IRQ 234 [ 606.877055] Modules linked in: [ 606.881444] Pid: 10719, comm: work_for_cpu Not tainted 2.6.30-tip-03093-gff58544-dirty #380 [ 606.897083] Call Trace: [ 606.900366] [] ? __free_irq+0xad/0x17f [ 606.904024] [] warn_slowpath_common+0x8d/0xd0 [ 606.918689] [] warn_slowpath_fmt+0x50/0x66 [ 606.924606] [] ? __free_irq+0x80/0x17f [ 606.939724] [] __free_irq+0xad/0x17f [ 606.943368] [] ? trace_hardirqs_on+0x20/0x36 [ 606.960565] [] free_irq+0x21/0x3f [ 606.962793] [] mthca_free_irqs+0x5a/0xad [ 606.979683] [] mthca_cleanup_eq_table+0x2b/0xda [ 606.983649] [] mthca_setup_hca+0x4f7/0x558 [ 606.998302] [] __mthca_init_one+0x5f6/0x708 [ 607.003948] [] mthca_init_one+0x94/0xbf [ 607.019447] [] ? do_work_for_cpu+0x0/0x50 [ 607.023666] [] local_pci_probe+0x2a/0x42 [ 607.039153] [] do_work_for_cpu+0x27/0x50 [ 607.041672] [] ? do_work_for_cpu+0x0/0x50 [ 607.059565] [] kthread+0x6e/0xb1 [ 607.063474] [] child_rip+0xa/0x20 [ 607.079096] [] ? restore_args+0x0/0x30 [ 607.083027] [] ? kthread+0x0/0xb1 [ 607.098768] [] ? child_rip+0x0/0x20 [ 607.101323] ---[ end trace e6652b295472e0d7 ]--- [ 607.117066] ------------[ cut here ]------------ [ 607.119830] WARNING: at kernel/irq/manage.c:740 __free_irq+0xad/0x17f() [ 607.137557] Hardware name: Libra [ 607.140825] Trying to free already-free IRQ 233 [ 607.145850] Modules linked in: [ 607.157651] Pid: 10719, comm: work_for_cpu Tainted: G W 2.6.30-tip-03093-gff58544-dirty #380 [ 607.176018] Call Trace: [ 607.178982] [] ? __free_irq+0xad/0x17f [ 607.182636] [] warn_slowpath_common+0x8d/0xd0 [ 607.198688] [] warn_slowpath_fmt+0x50/0x66 [ 607.203487] [] ? __free_irq+0x80/0x17f [ 607.219523] [] __free_irq+0xad/0x17f [ 607.222882] [] ? trace_hardirqs_on+0x20/0x36 [ 607.238103] [] free_irq+0x21/0x3f [ 607.241173] [] mthca_free_irqs+0x76/0xad [ 607.257516] [] mthca_cleanup_eq_table+0x2b/0xda [ 607.264306] [] mthca_setup_hca+0x4f7/0x558 [ 607.277819] [] __mthca_init_one+0x5f6/0x708 [ 607.283174] [] mthca_init_one+0x94/0xbf [ 607.297829] [] ? do_work_for_cpu+0x0/0x50 [ 607.301208] [] local_pci_probe+0x2a/0x42 [ 607.318688] [] do_work_for_cpu+0x27/0x50 [ 607.322902] [] ? do_work_for_cpu+0x0/0x50 [ 607.339817] [] kthread+0x6e/0xb1 [ 607.344286] [] child_rip+0xa/0x20 [ 607.357248] [] ? restore_args+0x0/0x30 [ 607.360923] [] ? kthread+0x0/0xb1 [ 607.376587] [] ? child_rip+0x0/0x20 [ 607.383884] ---[ end trace e6652b295472e0d8 ]--- [ 607.396025] ------------[ cut here ]------------ [ 607.400016] WARNING: at kernel/irq/manage.c:740 __free_irq+0xad/0x17f() [ 607.416804] Hardware name: Libra [ 607.419503] Trying to free already-free IRQ 232 [ 607.424234] Modules linked in: [ 607.436788] Pid: 10719, comm: work_for_cpu Tainted: G W 2.6.30-tip-03093-gff58544-dirty #380 [ 607.442583] Call Trace: [ 607.456233] [] ? __free_irq+0xad/0x17f [ 607.459883] [] warn_slowpath_common+0x8d/0xd0 [ 607.477933] [] warn_slowpath_fmt+0x50/0x66 [ 607.481881] [] ? __free_irq+0x80/0x17f [ 607.497082] [] __free_irq+0xad/0x17f [ 607.500157] [] ? trace_hardirqs_on+0x20/0x36 [ 607.516507] [] free_irq+0x21/0x3f [ 607.520978] [] mthca_free_irqs+0x92/0xad [ 607.536204] [] mthca_cleanup_eq_table+0x2b/0xda [ 607.542430] [] mthca_setup_hca+0x4f7/0x558 [ 607.557645] [] __mthca_init_one+0x5f6/0x708 [ 607.561602] [] mthca_init_one+0x94/0xbf [ 607.578214] [] ? do_work_for_cpu+0x0/0x50 [ 607.581882] [] local_pci_probe+0x2a/0x42 [ 607.597365] [] do_work_for_cpu+0x27/0x50 [ 607.604409] [] ? do_work_for_cpu+0x0/0x50 [ 607.617720] [] kthread+0x6e/0xb1 [ 607.620780] [] child_rip+0xa/0x20 [ 607.637011] [] ? restore_args+0x0/0x30 [ 607.642633] [] ? kthread+0x0/0xb1 [ 607.656116] [] ? child_rip+0x0/0x20 [ 607.659478] ---[ end trace e6652b295472e0d9 ]--- [ 607.662779] free_memtype request 0xfbe80000-0xfbe81000 [ 607.678284] free_memtype request 0xfbef0000-0xfbef1000 [ 607.682522] free_memtype request 0xe0000000-0xe4000000 [ 607.697438] free_memtype request 0xe5800000-0xe6000000 [ 607.701085] free_memtype request 0xfa802000-0xfa803000 [ 607.718191] free_memtype request 0xfbe80000-0xfbe81000 [ 607.721542] free_memtype request 0xfa800000-0xfa801000 [ 607.737028] ib_mthca 0000:c4:00.0: PCI INT A disabled [ 607.741860] ib_mthca: probe of 0000:c4:00.0 failed with error -16 From rdreier at cisco.com Fri Jun 12 07:31:42 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 07:31:42 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> (Yinghai Lu's message of "Thu, 11 Jun 2009 19:45:21 -0700") References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> Message-ID: Thanks, will take a look. It does seem this is on an error path: > [ 606.818317] ib_mthca 0000:c4:00.0: NOP command failed to generate > interrupt (IRQ 98), aborting. > [ 606.826709] ib_mthca 0000:c4:00.0: BIOS or ACPI interrupt routing problem? So your system already had problems -- the driver didn't get an expected interrupt from the device. - R. From rdreier at cisco.com Fri Jun 12 07:33:30 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 07:33:30 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> (Yinghai Lu's message of "Thu, 11 Jun 2009 19:45:21 -0700") References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> Message-ID: By the way, can you try this with the ib_mthca module option "msi_x=0" -- because I see > [ 606.538429] ib_mthca 0000:c4:00.0: NOP command failed to generate > interrupt (IRQ 234). > [ 606.543348] ib_mthca 0000:c4:00.0: Trying again with MSI-X disabled. But perhaps the fallback to non-MSI-X in the driver is messing things up. Also you have: > [ 606.222053] ib_mthca 0000:c4:00.0: HCA FW version 4.6.002 is old > (4.8.200 is current). So updating the HCA firmware likely wouldn't hurt. - R. From jesse.barnes at intel.com Fri Jun 12 08:50:53 2009 From: jesse.barnes at intel.com (Jesse Barnes) Date: Fri, 12 Jun 2009 08:50:53 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> Message-ID: <20090612085053.0fbe9731@jbarnes-g45> On Fri, 12 Jun 2009 07:33:30 -0700 Roland Dreier wrote: > By the way, can you try this with the ib_mthca module option > "msi_x=0" -- because I see > > > [ 606.538429] ib_mthca 0000:c4:00.0: NOP command failed to > > generate interrupt (IRQ 234). > > [ 606.543348] ib_mthca 0000:c4:00.0: Trying again with MSI-X > > disabled. > > But perhaps the fallback to non-MSI-X in the driver is messing things > up. Also you have: My linux-next branch has a change to the return value from the MSI-X allocation function: pci_enable_msix now returns the number of available entries rather than -EINVAL if the allocation failed. Would that cause problems in the mthca driver? -- Jesse Barnes, Intel Open Source Technology Center From rdreier at cisco.com Fri Jun 12 09:25:36 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 09:25:36 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: <20090612085053.0fbe9731@jbarnes-g45> (Jesse Barnes's message of "Fri, 12 Jun 2009 08:50:53 -0700") References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> <20090612085053.0fbe9731@jbarnes-g45> Message-ID: > My linux-next branch has a change to the return value from the MSI-X > allocation function: pci_enable_msix now returns the number of > available entries rather than -EINVAL if the allocation failed. Would > that cause problems in the mthca driver? I don't think so in this case ... the driver is getting past enabling MSI-X and actually testing the interrupt before it fails. I'll audit the code just to make sure we handle that case though. But I thought pci_enable_msix() always returned the number of entries actually available if the allocation failed? - R. From rdreier at cisco.com Fri Jun 12 09:26:19 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 09:26:19 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: <20090612085053.0fbe9731@jbarnes-g45> (Jesse Barnes's message of "Fri, 12 Jun 2009 08:50:53 -0700") References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> <20090612085053.0fbe9731@jbarnes-g45> Message-ID: Yeah, the code as it is should be OK: err = pci_enable_msix(mdev->pdev, entries, ARRAY_SIZE(entries)); if (err) { if (err > 0) mthca_info(mdev, "Only %d MSI-X vectors available, " "not using MSI-X\n", err); return err; } - R. From arlin.r.davis at intel.com Fri Jun 12 09:39:33 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 12 Jun 2009 09:39:33 -0700 Subject: [ofa-general] [PATCH] uDAPL v2: dapltest: windows scripts updated Message-ID: Support added for provider specification and general simplification of internal workings. Signed-off-by: Stan Smith --- test/dapltest/scripts/dt-cli.bat | 135 ++++++++++++++++--------------------- test/dapltest/scripts/dt-svr.bat | 9 +-- 2 files changed, 61 insertions(+), 83 deletions(-) diff --git a/test/dapltest/scripts/dt-cli.bat b/test/dapltest/scripts/dt-cli.bat index 9f2284a..9cbb202 100644 --- a/test/dapltest/scripts/dt-cli.bat +++ b/test/dapltest/scripts/dt-cli.bat @@ -30,15 +30,16 @@ if NOT EXIST %PF%\dapl2test.exe ( set DT=dapl2test.exe rem To debug dapl2test - use dapl2testd.exe with ibnic0v2d +rem setup DAPL provider name: translate shorthand name or use name from dat.conf. +rem if DAPL provider name is incorrect, DAPL will fail correctly. if "%1" == "" goto usage if "%1" == "ibal" set D=ibnic0v2 +if "%1" == "ibal0" set D=ibnic0v2 +if "%1" == "ibal1" set D=ibnic1v2 if "%1" == "scm" set D=ibnic0v2-scm if "%1" == "cma" set D=ibnic0v2-cma -if "%D%" == "" ( - echo Invalid DAPL Provider name '%1' ? - goto usage -) +if "%D%" == "" set D=%1 rem DaplTest server hostname if "%2" == "" goto usage @@ -48,7 +49,7 @@ rem Which test ? if "%3" == "" goto usage set T=%3 -set LPS=3 +set LPS=5 rem Enable DEBUG? @@ -64,6 +65,8 @@ rem set DAT_DBG_TYPE=0x1 rem set DAT_DBG_LEVEL=1 ) +if "%4" == "-Q" ( set QUIET=1 ) else ( set QUIET=0 ) + rem %DT% -T T -V -t 2 -w 2 -i 1000111 -s %S% -D %D% rem client RW 4096 1 server RW 2048 4 rem client RR 1024 2 server RR 2048 2 @@ -83,61 +86,47 @@ if "%T%" == "bench" ( ) if "%T%" == "conn" ( -rem Connectivity test - client sends one buffer with one 4KB segments, one time. -rem add '-d' for debug output. + rem Connectivity test - client sends one buffer with one 4KB segments, one time. + rem add '-d' for debug output. echo Simple Connectivity test - set STIME=!DATE! !TIME! - %DT% -T T -s %S% -D %D% -i 1 -t 1 -w 1 client SR 4096 server SR 4096 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -s %S% -D %D% -i 1 -t 1 -w 1 client SR 4096 server SR 4096 + goto xcmd ) if "%T%" == "trans" ( echo %T%: Transaction test - 8192 iterations, 1 thread, SR 4KB buffers - set STIME=!DATE! !TIME! - %DT% -T T -s %S% -D %D% -i 8192 -t 1 -w 1 client SR 4096 server SR 4096 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -s %S% -D %D% -i 8192 -t 1 -w 1 client SR 4096 server SR 4096 + goto xcmd ) if "%T%" == "transm" ( echo %T%: Multiple RW, RR, SR transactions, 4096 iterations - set STIME=!DATE! !TIME! - %DT% -T T -P -t 1 -w 1 -i 4096 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -P -t 1 -w 1 -i 4096 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f + goto xcmd ) if "%T%" == "transt" ( echo %T%: Threads[4] Transaction test - 4096 iterations, 1 thread, SR 4KB buffers - set STIME=!DATE! !TIME! - %DT% -T T -s %S% -D %D% -i 4096 -t 4 -w 1 client SR 8192 3 server SR 8192 3 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -s %S% -D %D% -i 4096 -t 4 -w 1 client SR 8192 3 server SR 8192 3 + goto xcmd ) if "%T%" == "transme" ( echo %T%: 1 Thread Endpoints[4] transactions [RW, RR, SR], 4096 iterations - set STIME=!DATE! !TIME! - %DT% -T T -P -t 1 -w 4 -i 4096 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -P -t 1 -w 4 -i 4096 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f + goto xcmd ) if "%T%" == "transmet" ( echo %T%: Threads[2] Endpoints[4] transactions[RW, RR, SR], 4096 iterations - set STIME=!DATE! !TIME! - %DT% -T T -P -t 2 -w 4 -i 4096 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -P -t 2 -w 4 -i 4096 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f + goto xcmd ) if "%T%" == "transmete" ( echo %T%: Threads[4] Endpoints[4] transactions[RW, RR, SR], 8192 iterations - set STIME=!DATE! !TIME! - %DT% -T T -P -t 2 -w 4 -i 8192 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -P -t 2 -w 4 -i 8192 -s %S% -D %D% client RW 4096 1 server RW 2048 4 server RR 1024 2 client RR 2048 2 client SR 1024 3 -f server SR 256 3 -f + goto xcmd ) if "%T%" == "EPA" ( @@ -159,76 +148,58 @@ if "%T%" == "EP" ( set TH=4 set EP=3 echo %T%: Multi: Threads[!TH!] endpoints[!EP!] Send/Recv test - 4096 iterations, 3 8K segs - set STIME=!DATE! !TIME! - %DT% -T T -s %S% -D %D% -i 4096 -t !TH! -w !EP! client SR 8192 3 server SR 8192 3 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -s %S% -D %D% -i 4096 -t !TH! -w !EP! client SR 8192 3 server SR 8192 3 + goto xcmd ) if "%T%" == "threads" ( echo %T%: Multi Threaded[6] Send/Recv test - 4096 iterations, 3 8K segs - set STIME=!DATE! !TIME! - %DT% -T T -s %S% -D %D% -i 4096 -t 6 -w 1 client SR 8192 3 server SR 8192 3 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -s %S% -D %D% -i 4096 -t 6 -w 1 client SR 8192 3 server SR 8192 3 + goto xcmd ) if "%T%" == "threadsm" ( set TH=5 set EP=3 echo %T%: Multi: Threads[!TH!] endpoints[!EP!] Send/Recv test - 4096 iterations, 3 8K segs - set STIME=!DATE! !TIME! - %DT% -T T -s %S% -D %D% -i 4096 -t !TH! -w !EP! client SR 8192 3 server SR 8192 3 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T T -s %S% -D %D% -i 4096 -t !TH! -w !EP! client SR 8192 3 server SR 8192 3 + goto xcmd ) if "%T%" == "perf" ( rem echo Performance test - set STIME=!DATE! !TIME! - %DT% -T P %DBG% -s %S% -D %D% -i 2048 RW 4096 2 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T P %DBG% -s %S% -D %D% -i 2048 RW 4096 2 + goto xcmd ) if "%T%" == "rdma-read" ( echo %T% 4 32K segs - set STIME=!DATE! !TIME! - %DT% -T P -s %S% -D %D% -i 4096 RR 32768 4 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T P -s %S% -D %D% -i 4096 RR 32768 4 + goto xcmd ) if "%T%" == "rdma-write" ( echo %T% 4 32K segs - set STIME=!DATE! !TIME! - %DT% -T P -s %S% -D %D% -i 4096 RW 32768 4 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T P -s %S% -D %D% -i 4096 RW 32768 4 + goto xcmd ) if "%T%" == "bw" ( echo bandwidth 4096 iterations of 2 65K mesgs - set STIME=!DATE! !TIME! - %DT% -T P -s %S% -D %D% -i 4096 -p 16 -m p RW 65536 2 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T P -s %S% -D %D% -i 4096 -p 16 -m p RW 65536 2 + goto xcmd ) if "%T%" == "latb" ( echo latency test - block for completion events - set STIME=!DATE! !TIME! - %DT% -T P -s %S% -D %D% -i 8192 -p 1 -m b RW 4 1 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T P -s %S% -D %D% -i 8192 -p 1 -m b RW 4 1 + goto xcmd ) if "%T%" == "latp" ( echo latency test - poll completion events - set STIME=!DATE! !TIME! - %DT% -T P -s %S% -D %D% -i 8192 -p 1 -m p RW 4 1 - set ETIME=!DATE! !TIME! - goto xit + set CMD=%DT% -T P -s %S% -D %D% -i 8192 -p 1 -m p RW 4 1 + goto xcmd ) if "%T%" == "lim" ( @@ -259,7 +230,7 @@ if "%T%" == "regression" ( FOR /L %%i IN (1,1,!LPS!) DO ( for %%r in ( !RT! ) do ( echo loop %%i - start test %%r - call %0 %1 %%r + call %0 %1 %2 %%r if %ERRORLEVEL% GTR 1 ( echo Error %ERRORLEVEL% in regression test %%r exit /B %ERRORLEVEL% @@ -337,10 +308,8 @@ if "%T%" == "interop" ( ) if "%T%" == "stop" ( - set STIME=!DATE! !TIME! - %DT% -T Q -s %S% -D %D% - set ETIME=!DATE! !TIME! - goto xit + %DT% -T Q -s %S% -D %D% > nul + goto rxt ) :usage @@ -350,7 +319,7 @@ echo usage: dt-cli dapl-provider dt-svr-hostname [testname [-D]] echo. echo where: echo. -echo dapl-provider - ibal scm cma +echo dapl-provider: ibal, scm, cma or %SystemDrive%\DAT\dat.conf DAPL-provider name. echo. echo dt-svr-hostname - IPv4 hostanme where the DaplTest server is running echo. @@ -378,6 +347,18 @@ echo regression {loopCnt,default=%LPS%} - regression + stress. echo interop {loopCnt,default=%LPS%} - 2007 OFA interoperability event tests. goto rxt +rem Execute the single daplest Command (CMD), observe -Q switch +:xcmd + set STIME=!DATE! !TIME! + if %QUIET% EQU 1 ( + %CMD% > nul + ) else ( + %CMD% + ) + set ETIME=!DATE! !TIME! + + rem fall thru... + :xit if %ERRORLEVEL% EQU 0 ( diff --git a/test/dapltest/scripts/dt-svr.bat b/test/dapltest/scripts/dt-svr.bat index a774aac..1d3760d 100644 --- a/test/dapltest/scripts/dt-svr.bat +++ b/test/dapltest/scripts/dt-svr.bat @@ -25,19 +25,16 @@ rem which Dapl provider? if "%1" == "" ( echo usage: dt-svr dapl-provider [ -D [dbg-bit-mask] ] echo. - echo Where: dapl-provider can be [ ibal, scm, cma ] + echo Where: dapl-provider can be [ ibal, scm, cma or %SystemDrive%\DAT\dat.conf provider name ] exit /B 1 ) if "%1" == "ibal" set DEV=ibnic0v2 if "%1" == "scm" set DEV=ibnic0v2-scm if "%1" == "cma" set DEV=ibnic0v2-cma -if "%DEV%" == "" ( - echo Invalid DAPL Provider name '%1', see %SystemDrive%\DAT\dat.conf ? - goto usage -) +if "%DEV%" == "" set DEV=%1 -rem '-D' enables full debug output +rem '-D' enables full debug output? rem '-D hex-bit-mask' enables selective debug output - see manual.htm for details. if "%2" == "-D" ( if "%2" == "" ( -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 12 09:39:44 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Fri, 12 Jun 2009 09:39:44 -0700 Subject: [ofa-general] [PATCH] uDAPL v2: scm: update CM code to shutdown before closing socket Message-ID: connect req/reply data could be lost without calling shutdown on the socket before closing. Update provider to shutdown socket before closing. Add definition for SHUT_RW for windows. Signed-off-by: Arlin Davis --- --- dapl/openib_scm/dapl_ib_cm.c | 8 +++++++- dapl/openib_scm/windows/openib_osd.h | 1 + 2 files changed, 8 insertions(+), 1 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index 9cad5be..db0a25b 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -261,8 +261,10 @@ static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) /* cleanup, never made it to work queue */ if (cm_ptr->state == SCM_INIT) { - if (cm_ptr->socket != DAPL_INVALID_SOCKET) + if (cm_ptr->socket != DAPL_INVALID_SOCKET) { + shutdown(cm_ptr->socket, SHUT_RDWR); closesocket(cm_ptr->socket); + } dapl_os_free(cm_ptr, sizeof(*cm_ptr)); return; } @@ -274,6 +276,7 @@ static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) /* close socket if still active */ if (cm_ptr->socket != DAPL_INVALID_SOCKET) { + shutdown(cm_ptr->socket, SHUT_RDWR); closesocket(cm_ptr->socket); cm_ptr->socket = DAPL_INVALID_SOCKET; } @@ -328,6 +331,7 @@ static DAT_RETURN dapli_socket_disconnect(dp_ib_cm_handle_t cm_ptr) dapl_log(DAPL_DBG_TYPE_WARN, " cm_disc: write error = %s\n", strerror(errno)); + shutdown(cm_ptr->socket, SHUT_RDWR); closesocket(cm_ptr->socket); cm_ptr->socket = DAPL_INVALID_SOCKET; } @@ -1202,6 +1206,7 @@ dapls_ib_remove_conn_listener(IN DAPL_IA * ia_ptr, IN DAPL_SP * sp_ptr) /* close accepted socket, free cm_srvc_handle and return */ if (cm_ptr != NULL) { if (cm_ptr->socket != DAPL_INVALID_SOCKET) { + shutdown(cm_ptr->socket, SHUT_RDWR); closesocket(cm_ptr->socket); cm_ptr->socket = DAPL_INVALID_SOCKET; } @@ -1305,6 +1310,7 @@ dapls_ib_reject_connection(IN dp_ib_cm_handle_t cm_ptr, writev(cm_ptr->socket, iov, 1); } + shutdown(cm_ptr->socket, SHUT_RDWR); closesocket(cm_ptr->socket); cm_ptr->socket = DAPL_INVALID_SOCKET; } diff --git a/dapl/openib_scm/windows/openib_osd.h b/dapl/openib_scm/windows/openib_osd.h index 7eb3df3..9b0f089 100644 --- a/dapl/openib_scm/windows/openib_osd.h +++ b/dapl/openib_scm/windows/openib_osd.h @@ -16,6 +16,7 @@ #define DAPL_SOCKET SOCKET #define DAPL_INVALID_SOCKET INVALID_SOCKET +#define SHUT_RDWR SD_BOTH /* allow casting to WSABUF */ struct iovec -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 12 09:39:48 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 12 Jun 2009 09:39:48 -0700 Subject: [ofa-general] [PATCH] uDAPL v2:scm: provider specific query for default UD MTU is wrong. Message-ID: Change the provider specific query DAT_IB_TRANSPORT_MTU to report 2048 for new default MTU size. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_util.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index 30c71fa..ad30f73 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -704,7 +704,7 @@ DAT_RETURN dapls_ib_setup_async_callback(IN DAPL_IA * ia_ptr, */ DAT_NAMED_ATTR ib_attrs[] = { { - "DAT_IB_TRANSPORT_MTU", "1024"} + "DAT_IB_TRANSPORT_MTU", "2048"} , #ifdef DAT_EXTENSIONS { -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 12 09:40:09 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 12 Jun 2009 09:40:09 -0700 Subject: [ofa-general] [PATCH] uDAPL v2 scm: private data is not handled properly via CR rejects. Message-ID: For both RC and UD connect requests, the private data is not being received on socket and passed back via the active side REJECT event. UD requires new extended reject event type of DAT_IB_UD_CONNECTION_REJECT_EVENT to distiquish between RC and UD type rejects. cr_thread exit/cleanup processing fixed to insure all items are off the list before exiting. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_cm.c | 96 +++++++++++++++++++++++---------- dat/include/dat2/dat_ib_extensions.h | 7 ++- 2 files changed, 72 insertions(+), 31 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index db0a25b..d116c62 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -539,17 +539,6 @@ static void dapli_socket_connect_rtu(dp_ib_cm_handle_t cm_ptr) sin_addr)); goto bail; } - /* check for consumer reject */ - if (cm_ptr->dst.rej) { - dapl_log(DAPL_DBG_TYPE_CM, - " CONN_RTU read: PEER REJ reason=0x%x -> %s\n", - ntohs(cm_ptr->dst.rej), - inet_ntoa(((struct sockaddr_in *) - ep_ptr->param.remote_ia_address_ptr)-> - sin_addr)); - event = IB_CME_DESTINATION_REJECT_PRIVATE_DATA; - goto bail; - } /* convert peer response values to host order */ cm_ptr->dst.port = ntohs(cm_ptr->dst.port); @@ -601,6 +590,23 @@ static void dapli_socket_connect_rtu(dp_ib_cm_handle_t cm_ptr) } } + /* check for consumer reject */ + if (cm_ptr->dst.rej) { + dapl_log(DAPL_DBG_TYPE_CM, + " CONN_RTU read: PEER REJ reason=0x%x -> %s\n", + ntohs(cm_ptr->dst.rej), + inet_ntoa(((struct sockaddr_in *) + ep_ptr->param.remote_ia_address_ptr)-> + sin_addr)); + event = IB_CME_DESTINATION_REJECT_PRIVATE_DATA; +#ifdef DAT_EXTENSIONS + if (cm_ptr->dst.qp_type == IBV_QPT_UD) + goto ud_bail; + else +#endif + goto bail; + } + /* modify QP to RTR and then to RTS with remote info */ dapl_os_lock(&ep_ptr->header.lock); if (dapls_modify_qp_state(ep_ptr->qp_handle, @@ -637,9 +643,11 @@ static void dapli_socket_connect_rtu(dp_ib_cm_handle_t cm_ptr) /* init cm_handle and post the event with private data */ ep_ptr->cm_handle = cm_ptr; cm_ptr->state = SCM_CONNECTED; + event = IB_CME_CONNECTED; dapl_dbg_log(DAPL_DBG_TYPE_EP, " ACTIVE: connected!\n"); #ifdef DAT_EXTENSIONS +ud_bail: if (cm_ptr->dst.qp_type == IBV_QPT_UD) { DAT_IB_EXTENSION_EVENT_DATA xevent; @@ -652,26 +660,35 @@ static void dapli_socket_connect_rtu(dp_ib_cm_handle_t cm_ptr) &cm_ptr->dst.ia_address, sizeof(cm_ptr->dst.ia_address)); + if (event == IB_CME_CONNECTED) + event = DAT_IB_UD_CONNECTION_EVENT_ESTABLISHED; + else + event = DAT_IB_UD_CONNECTION_REJECT_EVENT; + dapls_evd_post_connection_event_ext((DAPL_EVD *) ep_ptr->param. connect_evd_handle, - DAT_IB_UD_CONNECTION_EVENT_ESTABLISHED, + event, (DAT_EP_HANDLE) ep_ptr, - (DAT_COUNT) cm_ptr->dst. - p_size, - (DAT_PVOID *) cm_ptr-> - p_data, - (DAT_PVOID *) & xevent); + (DAT_COUNT) cm_ptr->dst.p_size, + (DAT_PVOID *) cm_ptr->p_data, + (DAT_PVOID *) &xevent); + + /* done with socket, don't destroy cm_ptr, need pdata */ + closesocket(cm_ptr->socket); + cm_ptr->socket = DAPL_INVALID_SOCKET; } else #endif dapl_evd_connection_callback(cm_ptr, IB_CME_CONNECTED, cm_ptr->p_data, ep_ptr); return; - bail: - /* close socket, free cm structure and post error event */ - dapli_cm_destroy(cm_ptr); + +bail: + /* close socket, and post error event */ dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ - dapl_evd_connection_callback(NULL, event, NULL, ep_ptr); + closesocket(cm_ptr->socket); + cm_ptr->socket = DAPL_INVALID_SOCKET; + dapl_evd_connection_callback(NULL, event, cm_ptr->p_data, ep_ptr); } /* @@ -1034,11 +1051,13 @@ void dapli_socket_accept_rtu(dp_ib_cm_handle_t cm_ptr) param.connect_evd_handle, DAT_IB_UD_CONNECTION_EVENT_ESTABLISHED, (DAT_EP_HANDLE) cm_ptr->ep, - (DAT_COUNT) cm_ptr->dst. - p_size, - (DAT_PVOID *) cm_ptr-> - p_data, - (DAT_PVOID *) & xevent); + (DAT_COUNT) cm_ptr->dst.p_size, + (DAT_PVOID *) cm_ptr->p_data, + (DAT_PVOID *) &xevent); + + /* done with socket, don't destroy cm_ptr, need pdata */ + closesocket(cm_ptr->socket); + cm_ptr->socket = DAPL_INVALID_SOCKET; } else #endif dapls_cr_callback(cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp); @@ -1295,10 +1314,16 @@ dapls_ib_reject_connection(IN dp_ib_cm_handle_t cm_ptr, " reject(cm %p reason %x, pdata %p, psize %d)\n", cm_ptr, reason, pdata, psize); + if (psize > IB_MAX_REJ_PDATA_SIZE) + return DAT_LENGTH_ERROR; + /* write reject data to indicate reject */ if (cm_ptr->socket != DAPL_INVALID_SOCKET) { cm_ptr->dst.rej = (uint16_t) reason; cm_ptr->dst.rej = htons(cm_ptr->dst.rej); + cm_ptr->dst.p_size = htonl(psize); + /* get qp_type from request */ + cm_ptr->dst.qp_type = ntohs(cm_ptr->dst.qp_type); iov[0].iov_base = (void *)&cm_ptr->dst; iov[0].iov_len = sizeof(ib_qp_cm_t); @@ -1316,7 +1341,7 @@ dapls_ib_reject_connection(IN dp_ib_cm_handle_t cm_ptr, } /* cr_thread will destroy CR */ - cm_ptr->state = SCM_REJECTED; + cm_ptr->state = SCM_DESTROY; if (send(cm_ptr->hca->ib_trans.scm[1], "w", sizeof "w", 0) == -1) dapl_log(DAPL_DBG_TYPE_CM, " cm_destroy: thread wakeup error = %s\n", @@ -1555,7 +1580,7 @@ void cr_thread(void *arg) dapl_os_lock(&hca_ptr->ib_trans.lock); hca_ptr->ib_trans.cr_state = IB_THREAD_RUN; - while (hca_ptr->ib_trans.cr_state == IB_THREAD_RUN) { + while (1) { dapl_fd_zero(set); dapl_fd_set(hca_ptr->ib_trans.scm[0], set, DAPL_FD_READ); @@ -1574,10 +1599,13 @@ void cr_thread(void *arg) dapl_llist_remove_entry(&hca_ptr->ib_trans.list, (DAPL_LLIST_ENTRY *) & cr->entry); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " CR FREE: %p ep=%p st=%d sock=%d\n", + cr, cr->ep, cr->state, cr->socket); dapl_os_free(cr, sizeof(*cr)); continue; } - if (cr->socket == DAPL_INVALID_SOCKET) + if (cr->socket == DAPL_INVALID_SOCKET) continue; event = (cr->state == SCM_CONN_PENDING) ? @@ -1664,6 +1692,11 @@ void cr_thread(void *arg) dapl_os_lock(&hca_ptr->ib_trans.lock); } + /* set to exit and all resources destroyed */ + if ((hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) && + (dapl_llist_is_empty(&hca_ptr->ib_trans.list))) + break; + dapl_os_unlock(&hca_ptr->ib_trans.lock); dapl_select(set); @@ -1676,6 +1709,11 @@ void cr_thread(void *arg) strerror(errno)); } dapl_os_lock(&hca_ptr->ib_trans.lock); + + /* set to exit and all resources destroyed */ + if ((hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) && + (dapl_llist_is_empty(&hca_ptr->ib_trans.list))) + break; } dapl_os_unlock(&hca_ptr->ib_trans.lock); diff --git a/dat/include/dat2/dat_ib_extensions.h b/dat/include/dat2/dat_ib_extensions.h index 2c082b1..59df1de 100755 --- a/dat/include/dat2/dat_ib_extensions.h +++ b/dat/include/dat2/dat_ib_extensions.h @@ -70,8 +70,10 @@ * 2.0.3 - Add query/print counter support for IA, EP, and EVD's * dat_query_counters(), dat_print_counters() * + * 2.0.4 - Add DAT_IB_UD_CONNECTION_REJECT_EVENT extended UD event + * */ -#define DAT_IB_EXTENSION_VERSION 203 /* 2.0.3 */ +#define DAT_IB_EXTENSION_VERSION 204 /* 2.0.4 */ #define DAT_ATTR_COUNTERS "DAT_COUNTERS" #define DAT_IB_ATTR_FETCH_AND_ADD "DAT_IB_FETCH_AND_ADD" #define DAT_IB_ATTR_CMP_AND_SWAP "DAT_IB_CMP_AND_SWAP" @@ -89,7 +91,8 @@ typedef enum dat_ib_event_number { DAT_IB_DTO_EVENT = DAT_IB_EXTENSION_RANGE_BASE, DAT_IB_UD_CONNECTION_REQUEST_EVENT, - DAT_IB_UD_CONNECTION_EVENT_ESTABLISHED + DAT_IB_UD_CONNECTION_EVENT_ESTABLISHED, + DAT_IB_UD_CONNECTION_REJECT_EVENT } DAT_IB_EVENT_NUMBER; -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 12 09:40:24 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 12 Jun 2009 09:40:24 -0700 Subject: [ofa-general] [PATCH] uDAPL v2:scm: cleanup orphaned UD CR's when destroying the EP Message-ID: UD CR objects are kept active because of direct private data references from CONN events. The cr->socket is closed and marked inactive but the object remains allocated and queued on the CR resource list. There can be multiple CR's associated with a given EP and there is no way to determine when consumer is finished with event until the dat_ep_free. Schedule destruction for all CR's associated with this EP during free call. cr_thread will complete cleanup with state of SCM_DESTROY. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_qp.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 38 insertions(+), 0 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index e14d4d9..f30da11 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -189,6 +189,44 @@ DAT_RETURN dapls_ib_qp_free(IN DAPL_IA * ia_ptr, IN DAPL_EP * ep_ptr) ep_ptr->qp_handle = IB_INVALID_HANDLE; } +#ifdef DAT_EXTENSIONS +{ + dp_ib_cm_handle_t cr, next_cr; + + /* + * UD CR objects are kept active because of direct private data references + * from CONN events. The cr->socket is closed and marked inactive but the + * object remains allocated and queued on the CR resource list. There can + * be multiple CR's associated with a given EP. There is no way to determine + * when consumer is finished with event until the dat_ep_free. + * + * Schedule destruction for all CR's associated with this EP, cr_thread will + * complete the cleanup with state == SCM_DESTROY. + */ + dapl_os_lock(&ia_ptr->hca_ptr->ib_trans.lock); + if (!dapl_llist_is_empty((DAPL_LLIST_HEAD*) + &ia_ptr->hca_ptr->ib_trans.list)) + next_cr = dapl_llist_peek_head((DAPL_LLIST_HEAD*) + &ia_ptr->hca_ptr->ib_trans.list); + else + next_cr = NULL; + + while (next_cr) { + cr = next_cr; + next_cr = dapl_llist_next_entry((DAPL_LLIST_HEAD*) + &ia_ptr->hca_ptr->ib_trans.list, + (DAPL_LLIST_ENTRY*)&cr->entry); + if (cr->ep == ep_ptr) { + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " qp_free CR: ep %p cr %p\n", ep_ptr, cr); + cr->state = SCM_DESTROY; + } + } + dapl_os_unlock(&ia_ptr->hca_ptr->ib_trans.lock); + send(ia_ptr->hca_ptr->ib_trans.scm[1], "w", sizeof "w", 0); +} +#endif + return DAT_SUCCESS; } -- 1.5.2.5 From yhlu.kernel at gmail.com Fri Jun 12 10:22:49 2009 From: yhlu.kernel at gmail.com (Yinghai Lu) Date: Fri, 12 Jun 2009 10:22:49 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> Message-ID: <86802c440906121022s532ee0f7tc556251d22ffd139@mail.gmail.com> On Fri, Jun 12, 2009 at 7:31 AM, Roland Dreier wrote: > Thanks, will take a look.  It does seem this is on an error path: > >  > [  606.818317] ib_mthca 0000:c4:00.0: NOP command failed to generate >  > interrupt (IRQ 98), aborting. >  > [  606.826709] ib_mthca 0000:c4:00.0: BIOS or ACPI interrupt routing problem? > > So your system already had problems -- the driver didn't get an > expected interrupt from the device. I knew that. but we should not get that double freeing. BIOS doesn't allocate proper MMIO range for that card. YH From gyllen at llnl.gov Fri Jun 12 09:52:15 2009 From: gyllen at llnl.gov (John Gyllenhaal) Date: Fri, 12 Jun 2009 09:52:15 -0700 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: References: <20090611151814.95f7ae64.weiny2@llnl.gov> Message-ID: <200906121652.n5CGqFru004806@mail-1.llnl.gov> Howdy Roland; Here is more details on why not using memcpy appears to be a good idea under valgrind. Valgrind replaces the libc memcpy call with a simple version that copies a byte at a time (in order). If libmlx4 is not built with --with-valgrind, valgrind considers each write an invalid write and spends a very long time after each write updating its error database. We experimented with replacing the Valgrind error database update with a configurable spin loop and found that if we put a delay of around 100,000 cycles between writes in the 'byte memcpy' when writing to the blueflame page, that a sent message gets lost/misplaced in a simple testcase with two MPI_barriers back to back (resulting in a hang because not all processes exit the first barrier). Our theory is the card sees 'byte' writes to the blueflame page and due to the long delay, uses the information before it is all written out (and thus getting wrong info). With the patched version, longs are written to the blueflame page and it now happens to work under valgrind. Of course, it may be luck. I could be that writing longs are 4-8 times more efficient, so the delay is not longer big enough to matter. It could be that it simply fixes our testcase in that the card is still reading early but happening to get the correct data in this case. Or it could be that writing longs actually fixes things and that writing bytes is a bad idea (since you could get a context switch at any time since this is user code and that could give the same effect). It any case, it fixes our testcases and seems like having control how data is written to the blueflame page is a good idea in any case. When users use our valgrind wrapper scripts (they don't always) , we LD_PRELOAD a patched version of this library compiled with --with-valgrind, which prevents the delay to begin with (and runs much faster under Valgrind as a result). I hope this clarifies things a little. -John G. P.S. If a context switch happens during a write to the blueflame page or some other memory mapped NIC addresses, could bad things happen? This is why I continued the detailed hunt after figuring out compiling with --with-valgrind resolved our problems, since similar delays could happen during context switches. At 03:51 PM 6/11/2009, Roland Dreier wrote: > > Our MPI folks detected a hang while using Valgrind with our ConnectX cards. > > After trying the current master branch in git we solved the problem by applying > > this patch from the git tree to v1.0. > > > > Don't use memcpy() to write blueflame sends > >Didn't realize this had that implication (I thought it just made >blueflame not give latency benefit). Anyway yes it has been a while >since a libmlx4 release. I'll make one soon, probably next week. --------------------------------------------------------------------- John C. Gyllenhaal Bldg: 453 Rm: 4151 Computation Department Email: gyllen at llnl.gov Lawrence Livermore National Lab Voice: (925) 424-5485 7000 East Ave, L-557 Fax: (925) 423-6961 Livermore, CA. 94551-0808 URL: http://www.llnl.gov/icc/lc/DEG --------------------------------------------------------------------- From jgunthorpe at obsidianresearch.com Fri Jun 12 12:56:32 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 12 Jun 2009 13:56:32 -0600 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: <200906121652.n5CGqFru004806@mail-1.llnl.gov> References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> Message-ID: <20090612195632.GO29521@obsidianresearch.com> On Fri, Jun 12, 2009 at 09:52:15AM -0700, John Gyllenhaal wrote: > Valgrind replaces the libc memcpy call with a simple version that > copies a byte at a time (in order). If libmlx4 is not built with > --with-valgrind, valgrind considers each write an invalid write and > spends a very long time after each write updating its error > database. We experimented with replacing the Valgrind error > database update with a configurable spin loop and found that if we > put a delay of around 100,000 cycles between writes in the 'byte > memcpy' when writing to the blueflame page, that a sent message gets > lost/misplaced in a simple testcase with two MPI_barriers back to > back (resulting in a hang because not all processes exit the first > barrier). Our theory is the card sees 'byte' writes to the > blueflame page and due to the long delay, uses the information > before it is all written out (and thus getting wrong info). There are lots of ways adding a timing delay here can cause problems. x86 CPUs have write combining buffers that can be enabled and will aggregate byte writes into larger transfers, they do flush based on a timer in some cases. Some devices that do this also have internal aggregation buffers that will flush in certain cases, often non-sequential writes or again timers.. I'm not sure what the chip's expectation is for the actual bus transfers in this area, but I think you are right to be concerned about atomicity, even when transfering based on longs. For instance, you do not want to rely upon write combining to create a single PCI-E transaction out of the message to ensure atomicity in a multi-process environment. This will not work reliably 100% of the time. It is worth looking at using SSE instructions to burst transfer the entire message in one atomic go. Jason From andy.grover at oracle.com Fri Jun 12 17:03:22 2009 From: andy.grover at oracle.com (Andy Grover) Date: Fri, 12 Jun 2009 17:03:22 -0700 Subject: [ofa-general] proper use of dev_attr->page_size_cap Message-ID: <4A32ECCA.20003@oracle.com> Hey all, So RDS (and srp similarly) are doing this: rds_ibdev->fmr_page_shift = max(9, ffs(dev_attr->page_size_cap) - 1); On my mlx4's, page_size_cap is fffffe00, which will always result in using a 512 byte fmr page size, not really so great. So ideally we want fmr page shift to match arch's PAGE_SHIFT, but we have to handle if it must be smaller? And do this: rds_ibdev->fmr_page_shift = min(PAGE_SHIFT, fls(PAGE_MASK & dev_attr->page_size_cap) - 1); It would also be nice if we could instead do this: rds_ibdev->fmr_page_shift = PAGE_SHIFT; ? :-) Regards -- Andy From rdreier at cisco.com Fri Jun 12 21:06:43 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 21:06:43 -0700 Subject: [ofa-general] proper use of dev_attr->page_size_cap In-Reply-To: <4A32ECCA.20003@oracle.com> (Andy Grover's message of "Fri, 12 Jun 2009 17:03:22 -0700") References: <4A32ECCA.20003@oracle.com> Message-ID: > It would also be nice if we could instead do this: > > rds_ibdev->fmr_page_shift = PAGE_SHIFT; Yes, I think it is safe to assume that a device can handle a page size matching the native system's page size. The other sizes are there if they help (eg 512 byte pages are good for doing block storage traffic with 512-byte sectors) - R. From rdreier at cisco.com Fri Jun 12 21:22:06 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 21:22:06 -0700 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: <200906121652.n5CGqFru004806@mail-1.llnl.gov> (John Gyllenhaal's message of "Fri, 12 Jun 2009 09:52:15 -0700") References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> Message-ID: > Valgrind replaces the libc memcpy call with a simple version that > copies a byte at a time (in order). If libmlx4 is not built with > --with-valgrind, valgrind considers each write an invalid write and > spends a very long time after each write updating its error database. > We experimented with replacing the Valgrind error database update > with a configurable spin loop and found that if we put a delay of > around 100,000 cycles between writes in the 'byte memcpy' when > writing to the blueflame page, that a sent message gets > lost/misplaced in a simple testcase with two MPI_barriers back to > back (resulting in a hang because not all processes exit the first > barrier). Our theory is the card sees 'byte' writes to the blueflame > page and due to the long delay, uses the information before it is all > written out (and thus getting wrong info). That makes sense. The HW documentation says that blueflame writes must be done in aligned chunks of at least 4 bytes, so it's not surprising that byte writes confuse the HW in some cases. - R. From rdreier at cisco.com Fri Jun 12 21:24:42 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 21:24:42 -0700 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: <20090612195632.GO29521@obsidianresearch.com> (Jason Gunthorpe's message of "Fri, 12 Jun 2009 13:56:32 -0600") References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> <20090612195632.GO29521@obsidianresearch.com> Message-ID: > I'm not sure what the chip's expectation is for the actual bus > transfers in this area, but I think you are right to be concerned > about atomicity, even when transfering based on longs. The chip docs seem to suggest that we're OK as long as we do 4-byte writes aligned to 4 bytes. > It is worth looking at using SSE instructions to burst transfer the > entire message in one atomic go. I'm not aware of any SSE instructions that work on chunks bigger than 16 bytes at a time. In fact the latest mlx4 kernel driver maps the blueflame page to userspace with write-combining enabled, and this improves performance quite a bit. The HCA doesn't care what order that the CPU drains the WC buffer in (according to docs at least) - R. From rdreier at cisco.com Fri Jun 12 21:30:34 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 12 Jun 2009 21:30:34 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> (Yinghai Lu's message of "Thu, 11 Jun 2009 19:45:21 -0700") References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> Message-ID: Something like this should help -- can you verify this fixes the double free of irq? drivers/infiniband/hw/mthca/mthca_eq.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/mthca/mthca_eq.c b/drivers/infiniband/hw/mthca/mthca_eq.c index 28f0e0c..90e4e45 100644 --- a/drivers/infiniband/hw/mthca/mthca_eq.c +++ b/drivers/infiniband/hw/mthca/mthca_eq.c @@ -641,9 +641,11 @@ static void mthca_free_irqs(struct mthca_dev *dev) if (dev->eq_table.have_irq) free_irq(dev->pdev->irq, dev); for (i = 0; i < MTHCA_NUM_EQ; ++i) - if (dev->eq_table.eq[i].have_irq) + if (dev->eq_table.eq[i].have_irq) { free_irq(dev->eq_table.eq[i].msi_x_vector, dev->eq_table.eq + i); + dev->eq_table.eq[i].have_irq = 0; + } } static int mthca_map_reg(struct mthca_dev *dev, From jgunthorpe at obsidianresearch.com Fri Jun 12 23:00:42 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Sat, 13 Jun 2009 00:00:42 -0600 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> <20090612195632.GO29521@obsidianresearch.com> Message-ID: <20090613060042.GD12430@obsidianresearch.com> On Fri, Jun 12, 2009 at 09:24:42PM -0700, Roland Dreier wrote: > > > I'm not sure what the chip's expectation is for the actual bus > > transfers in this area, but I think you are right to be concerned > > about atomicity, even when transfering based on longs. > > The chip docs seem to suggest that we're OK as long as we do 4-byte > writes aligned to 4 bytes. Ok, that would certainly explain why there were problems with non long writes when using valgrind, the write combining will hide it, until you take too long. > > It is worth looking at using SSE instructions to burst transfer the > > entire message in one atomic go. > > I'm not aware of any SSE instructions that work on chunks bigger than 16 > bytes at a time. Right, I didn't notice it was larger :) Using 4 16 byte stores and a SFENCE would narrow the window considerably, though alignment of the source wqe becomes important to get good speed on the load into the xmm registers. > In fact the latest mlx4 kernel driver maps the blueflame page to > userspace with write-combining enabled, and this improves performance Yes, I bet, WC should get you 64 byte write transactions at the PCI-E level which surely makes everything better. > quite a bit. The HCA doesn't care what order that the CPU drains the WC > buffer in (according to docs at least) I would see the risk not so much as order for a single process, but what happens when there are alot of processes/cores doing the write. The chip must have a limit on the number of parallel writes it can re-assemble, I guess the only question is if the mlx4 limit is less than the number of pages it provides address space for? If so then it is worth minimizing the CPU instructions to do this transfer. Jason From vlad at lists.openfabrics.org Sat Jun 13 02:18:26 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 13 Jun 2009 02:18:26 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090613-0200 daily build status Message-ID: <20090613091827.1C18110208CE@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:37: include/linux/slub_def.h: In function 'kmalloc_index': include/linux/slub_def.h:90: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/.tmp_umem.o /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: Applying patch addr_1_netevents_revert_to_2_6_17.patch patching file drivers/infiniband/core/addr.c Hunk #1 FAILED at 40. Hunk #2 succeeded at 496 (offset 139 lines). Hunk #3 succeeded at 523 (offset 139 lines). 1 out of 3 hunks FAILED -- rejects in file drivers/infiniband/core/addr.c Patch addr_1_netevents_revert_to_2_6_17.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090613-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From sashak at voltaire.com Sat Jun 13 02:27:30 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 12:27:30 +0300 Subject: [ofa-general] Re: [PATCH] osm_dump.c dump port if lft is set up In-Reply-To: <4A1D16B7.7070300@Sun.COM> References: <4A1D16B7.7070300@Sun.COM> Message-ID: <20090613092730.GD10489@me> On 12:32 Wed 27 May , Line.Holen at Sun.COM wrote: > dump_ucast_routes() claims that a node is unreachable if the number of > hops to it is unknown. This is changed to print actual port and give > proper warning about hops. > > Signed-off-by: Line Holen Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 13 02:28:26 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 12:28:26 +0300 Subject: [ofa-general] Re: [PATCH] osm_sa_patch_record.c Remove unneeded break statement In-Reply-To: <4A2FBFC9.7010003@Sun.COM> References: <4A2FBFC9.7010003@Sun.COM> Message-ID: <20090613092826.GF10489@me> On 16:14 Wed 10 Jun , Line.Holen at Sun.COM wrote: > Signed-off-by: Line Holen Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 13 02:30:11 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 12:30:11 +0300 Subject: [ofa-general] Re: [ofw] [PATCH-resend] ib-mgmt/libibnetdisc: fix typecast warning In-Reply-To: <25C15E8A90F247C68A27939B5D3B0164@amr.corp.intel.com> References: <25C15E8A90F247C68A27939B5D3B0164@amr.corp.intel.com> Message-ID: <20090613093011.GG10489@me> On 08:10 Wed 10 Jun , Sean Hefty wrote: > Signed-off-by: Sean Hefty This patch is malformed (see below), so I applied the original. Thanks. Sasha > --- > I tried converting ib_portid_t lid to a uint16_t, but that resulted in a cascade > of changes, so I kept the simple approach. :) > > Resending - I didn't see a response to this. > > infiniband-diags/libibnetdisc/src/ibnetdisc.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c > b/infiniband-diags/libibnetdisc/src/ibnetdisc.c > index 1e93ff8..baea98e 100644 > --- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c > +++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c > @@ -188,7 +188,7 @@ extend_dpath(struct ibnd_fabric *f, ib_portid_t *portid, int > nextport) ^^^ line break > f->fabric.ibmad_port) < 0) > return -1; > > - portid->drpath.drslid = f->selfportid.lid; > + portid->drpath.drslid = (uint16_t) f->selfportid.lid; > portid->drpath.drdlid = 0xFFFF; > } > > > > > _______________________________________________ > ofw mailing list > ofw at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > From sashak at voltaire.com Sat Jun 13 03:18:36 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 13:18:36 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/ibdiag_common.c: Eliminate compile warning on x86_64 archs In-Reply-To: <20090529193112.GA14170@comcast.net> References: <20090529193112.GA14170@comcast.net> Message-ID: <20090613101836.GI10489@me> On 15:31 Fri 29 May , Hal Rosenstock wrote: > > src/ibdiag_common.c: In function pretty_print > src/ibdiag_common.c:95: warning: field precision should have type int, > but argument 3 has type long int > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 13 03:34:00 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 13:34:00 +0300 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Determine SL properly In-Reply-To: <20090529153515.GA10301@comcast.net> References: <20090529153515.GA10301@comcast.net> Message-ID: <20090613103400.GJ10489@me> Hi Hal, On 11:35 Fri 29 May , Hal Rosenstock wrote: > > rather than assuming SL 0 > > Signed-off-by: Hal Rosenstock > --- > diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c > index 691bdc3..f17da11 100644 > --- a/libibmad/src/resolve.c > +++ b/libibmad/src/resolve.c > @@ -59,6 +59,7 @@ int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, > return -1; > > mad_decode_field(portinfo, IB_PORT_SMLID_F, &lid); > + mad_decode_field(portinfo, IB_PORT_SMSL_F, &sm_id->sl); > > return ib_portid_set(sm_id, lid, 0, 0); > } > @@ -74,12 +75,23 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > { > ib_portid_t sm_portid; > char buf[IB_SA_DATA_SIZE] = { 0 }; > + ib_portid_t self = { 0 }; > + uint64_t selfguid; > + ibmad_gid_t selfgid; > + uint8_t nodeinfo[64]; > > if (!sm_id) { > sm_id = &sm_portid; > if (ib_resolve_smlid_via(sm_id, timeout, srcport) < 0) > return -1; > } > + > + if (!smp_query_via(nodeinfo, &self, IB_ATTR_NODE_INFO, 0, 0, srcport)) > + return -1; > + mad_decode_field(nodeinfo, IB_NODE_PORT_GUID_F, &selfguid); > + mad_set_field64(selfgid, 0, IB_GID_PREFIX_F, IB_DEFAULT_SUBN_PREFIX); > + mad_set_field64(selfgid, 0, IB_GID_GUID_F, selfguid); > + > if (*(uint64_t *) & portid->gid == 0) > mad_set_field64(portid->gid, 0, IB_GID_PREFIX_F, > IB_DEFAULT_SUBN_PREFIX); > @@ -87,10 +99,11 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > mad_set_field64(portid->gid, 0, IB_GID_GUID_F, *guid); > > if ((portid->lid = > - ib_path_query_via(srcport, portid->gid, portid->gid, sm_id, > + ib_path_query_via(srcport, selfgid, portid->gid, sm_id, > buf)) < 0) > return -1; > > + mad_decode_field(buf, IB_SA_PR_SL_F, &portid->sl); Just to make sure what did you mean here (please more informative commit message would be helpful)... Instead of just resolving destination port's lid by guid using its port to itself SA PR you are trying also to resolve SL which should be used from "self" to destination port. Right? Assuming so. Would in this case "self" and srcport be same? Sasha > return 0; > } > > @@ -167,6 +180,7 @@ int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, > return -1; > > mad_decode_field(portinfo, IB_PORT_LID_F, &portid->lid); > + mad_decode_field(portinfo, IB_PORT_SMSL_F, &portid->sl); > mad_decode_field(portinfo, IB_PORT_GID_PREFIX_F, &prefix); > mad_decode_field(nodeinfo, IB_NODE_PORT_GUID_F, &guid); > From sashak at voltaire.com Sat Jun 13 03:36:45 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 13:36:45 +0300 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Remove compile warning on x86_64 arch In-Reply-To: <20090601114622.GA31380@comcast.net> References: <20090601114622.GA31380@comcast.net> Message-ID: <20090613103645.GK10489@me> On 07:46 Mon 01 Jun , hnrose at comcast.net wrote: > > resolve.c:106: warning: pointer targets in passing argument 1 of 'mad_decode_field' differ in signedness > > Signed-off-by: Hal Rosenstock > > diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c > index f17da11..abcab17 100644 > --- a/libibmad/src/resolve.c > +++ b/libibmad/src/resolve.c > @@ -103,7 +103,7 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > buf)) < 0) > return -1; > > - mad_decode_field(buf, IB_SA_PR_SL_F, &portid->sl); > + mad_decode_field((uint8_t *)buf, IB_SA_PR_SL_F, &portid->sl); Why to not change type of buf and eliminate casting (and warning)? Sasha > return 0; > } > > From sashak at voltaire.com Sat Jun 13 04:12:00 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 14:12:00 +0300 Subject: [ofa-general] [PATCH] infiniband-diags: Do not change logical state on SubnAdmSet In-Reply-To: <4A266545.6080807@gmail.com> References: <4A2297DE.3050707@gmail.com> <4A262620.8020305@gmail.com> <4A266545.6080807@gmail.com> Message-ID: <20090613111200.GL10489@me> On 14:57 Wed 03 Jun , Eli Dorfman (Voltaire) wrote: > > > > Should an updated patch for this be reissued ? > > No, the patch is correct. The error was in the commit message. The commit message is part of the patch. Sasha From sashak at voltaire.com Sat Jun 13 04:20:40 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 14:20:40 +0300 Subject: [ofa-general] [PATCH v2] infiniband-diags: Do not change port state on SubnSet In-Reply-To: <4A2684E8.403@gmail.com> References: <4A2297DE.3050707@gmail.com> <4A2684E8.403@gmail.com> Message-ID: <20090613112040.GM10489@me> On 17:12 Wed 03 Jun , Eli Dorfman (Voltaire) wrote: > Do not change port state on SubnSet > > When changing physical state do not change logical port state to avoid invalid > transitions in port state machine. > From the IB spec When writing PortInfo:PortState, only legal transitions are > valid. So if PortState is ACTIVE and we try to set it to ACTIVE this will fail. > > This patch will allow reset in a single MAD by changing physical state to polling. > > Signed-off-by: Eli Dorfman Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 13 04:21:08 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 13 Jun 2009 14:21:08 +0300 Subject: [ofa-general] Re: [PATCH] infiniband_diags/ibportstate.c perform reset in a single mad In-Reply-To: <4A2CE1D5.2050503@gmail.com> References: <4A2297DE.3050707@gmail.com> <4A2684E8.403@gmail.com> <4A2CE1D5.2050503@gmail.com> Message-ID: <20090613112108.GN10489@me> On 13:03 Mon 08 Jun , Eli Dorfman (Voltaire) wrote: > perform reset in a single mad > > Perform reset by changing physical state to polling instead of > disable and enable. > This ensures that reset a port will not disconnect it (assuming negotiation succeeds). > Only disable operation is not allowed on HCA ports since it will irreversibly > disconnect it though this may also happen on a switch ingress port. > > Signed-off-by: Eli Dorfman Applied. Thanks. Sasha From devel-ofed at morey-chaisemartin.com Sat Jun 13 04:51:32 2009 From: devel-ofed at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Sat, 13 Jun 2009 13:51:32 +0200 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: Enhanced Fat-Tree algorithm Message-ID: <4A3392C4.1090608@morey-chaisemartin.com> This patch introduce many updates in the Fat-tree discussed here: http://www.mail-archive.com/general at lists.openfabrics.org/msg22416.html Major changes are: - Ensure minimum hop path is used - Balance secondary routes (necessary as a secondary route may become primary if shorter) - Optimizing balancing (quality and computation time) Squashed commits: Added balanced minhop within Fat-Tree Added qsort for choosing secondary routes order instead of simple index Enhanced qsort compareason function so it looks at the next switch level Enhanced qsort performances for 2nd level access Removed copy of port group arrays to save some time Replaced qsort by bubble sort Inlining compare and sort functions Updated bubble sort to allow inline compareason functions Enhanced sorting and reduced necessary compareasons Fixed increment to earn one useless loop when we are on the main path Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 413 ++++++++++++++++++++++++++++----------- 1 files changed, 296 insertions(+), 117 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 8ed2f74..0c89892 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -82,7 +82,6 @@ typedef enum { ** Forward references ** ***************************************************/ - struct ftree_sw_t_; struct ftree_hca_t_; struct ftree_port_t_; @@ -153,6 +152,7 @@ typedef struct ftree_port_group_t_ { boolean_t is_cn; /* whether this port is a compute node */ boolean_t is_io; /* whether this port is an I/O node */ uint32_t counter_down; /* number of allocated routs downwards */ + uint32_t counter_up; /* number of allocated routs downwards */ } ftree_port_group_t; /*************************************************** @@ -174,6 +174,8 @@ typedef struct ftree_sw_t_ { boolean_t is_leaf; unsigned down_port_groups_idx; uint8_t *hops; + uint32_t min_counter_down; + boolean_t counter_up_changed; } ftree_sw_t; /*************************************************** @@ -1886,6 +1888,162 @@ static void set_sw_fwd_table(IN cl_map_item_t * const p_map_item, ***************************************************/ /* + * Function: Finds the least loaded port group and stores its counter + * Given : A switch + */ +static inline void +recalculate_min_counter_down(ftree_sw_t *p_sw){ + uint32_t min= (1<<30); + uint32_t i; + for(i=0;i < p_sw->down_port_groups_num; i++) { + if(p_sw->down_port_groups[i]->counter_down < min){ + min = p_sw->down_port_groups[i]->counter_down; + } + } + p_sw->min_counter_down = min; + return; +} + +/* + * Function: Return the counter value of the least loaded down port group + * Given : A switch + */ +static inline uint32_t +find_lowest_loaded_group_on_sw(ftree_sw_t *p_sw){ + return p_sw->min_counter_down; +} + +/* + * Function: Compare the load of two port groups and return which is the least loaded + * Given : Two port groups with remote switch + * When both port groups are equally loaded, it picks the one whom + * remote switch down ports are least loaded. + * This way, it prefers the switch from where it will be easier to go down (creating upward routes). + * If both are equal, it picks the bigger GUID to be deterministic. + */ +static inline int +port_group_compare_load_down(const ftree_port_group_t *p1,const ftree_port_group_t *p2){ + int temp = p1->counter_down -p2->counter_down ; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + + /* Find the less loaded remote sw and choose this one */ + do{ + uint32_t load1= find_lowest_loaded_group_on_sw(p1->remote_hca_or_sw.p_sw); + uint32_t load2= find_lowest_loaded_group_on_sw(p2->remote_hca_or_sw.p_sw); + temp = load1-load2; + if(temp > 0) + return 1; + }while(0); + /* If they are both equal, choose the biggest GUID */ + if(p1->remote_port_guid > p2->remote_port_guid) + return 1; + + return -1; + +} + +/* + * Function: Sorts an array of port group by up load order + * Given : A port group array and its length + * As the list is mostly sorted, we used a bubble sort instead of qsort + * as it is much faster. + * + * Important note: + * This function and __osm_ftree_bubble_sort_down must NOT be factorized. + * Although most of the code is the same and a function pointer could be used + * for the compareason function, it would prevent the compareason function to be inlined + * and cost a great deal to performances. + */ +static inline void +bubble_sort_up(ftree_port_group_t **p_group_array, uint32_t nmemb) +{ + uint32_t i = 0; + uint32_t j = 0; + ftree_port_group_t *tmp = p_group_array[0]; + + /* As this function is a great number of times, we only go into the loop + * if one of the port counters has changed, thus saving some tests */ + if(tmp->hca_or_sw.p_sw->counter_up_changed == FALSE){ + return; + } + /* While we did modifications on the array order*/ + /* i may grew above array length but next loop will fail and tmp will be null for the next time + * this way we save a test i < nmemb for each pass through the loop */ + for(i = 0 ; tmp ; i++) + { + /* Assume the array is orderd */ + tmp = NULL; + /* Comparing elements j and j-1 */ + for(j = 1 ; j < (nmemb - i) ; j++) + { + /* If they are the wrong way around */ + if( p_group_array[j]->counter_up < p_group_array[j-1]->counter_up) + { + /* We invert them */ + tmp = p_group_array[j-1]; + p_group_array[j-1] = p_group_array[j]; + p_group_array[j] = tmp; + /* This sets tmp != NULL so the main loop will make another pass */ + } + } + } + + /* We have reordered the array so as long noone changes the counter + * it's not necessary to do it again */ + p_group_array[0]->hca_or_sw.p_sw->counter_up_changed = FALSE; +} + +/* + * Function: Sorts an array of port group. Order is decide through + * __osm_ftree_port_group_compare_load_down ( up counters, least load remote switch, biggest GUID) + * Given : A port group array and its length. Each port group points to a remote switch (not a HCA) + * As the list is mostly sorted, we used a bubble sort instead of qsort + * as it is much faster. + * + * Important note: + * This function and __osm_ftree_bubble_sort_up must NOT be factorized. + * Although most of the code is the same and a function pointer could be used + * for the compareason function, it would prevent the compareason function to be inlined + * and cost a great deal to performances. + */ +static inline void +bubble_sort_down(ftree_port_group_t **p_group_array, uint32_t nmemb) +{ + uint32_t i = 0; + uint32_t j = 0; + ftree_port_group_t *tmp = p_group_array[0]; + + /* While we did modifications on the array order*/ + /* i may grew above array length but next loop will fail and tmp will be null for the next time + * this way we save a test i < nmemb for each pass through the loop */ + for(i = 0 ; tmp ; i++) + { + /* Assume the array is orderd */ + tmp = NULL; + /* Comparing elements j and j-1 */ + for(j = 1 ; j < (nmemb - i) ; j++) + { + /* If they are the wrong way around */ + if( port_group_compare_load_down(p_group_array[j],p_group_array[j-1]) < 0 ) + { + /* We invert them */ + tmp = p_group_array[j-1]; + p_group_array[j-1] = p_group_array[j]; + p_group_array[j] = tmp; + + } + } + } +} + + +/*************************************************** + ***************************************************/ + +/* * Function: assign-up-going-port-by-descending-down * Given : a switch and a LID * Pseudo code: @@ -1915,10 +2073,10 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, ftree_port_group_t *p_group; ftree_port_t *p_port; ftree_port_t *p_min_port; - uint16_t i; uint16_t j; uint16_t k; boolean_t created_route = FALSE; + boolean_t routed=0; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); @@ -1928,11 +2086,10 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return FALSE; /* foreach down-going port group (in indexing order) */ - i = p_sw->down_port_groups_idx; + bubble_sort_up(p_sw->down_port_groups,p_sw->down_port_groups_num); for (k = 0; k < p_sw->down_port_groups_num; k++) { - p_group = p_sw->down_port_groups[i]; - i = (i + 1) % p_sw->down_port_groups_num; + p_group = p_sw->down_port_groups[k]; /* If this port group doesn't point to a switch, mark that the route was created and skip to the next group */ @@ -1980,6 +2137,10 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, p_group->base_lid, tuple_to_str(p_sw->tuple), p_group->remote_base_lid); + /* We skip only if we have come through a longer path */ + if(((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + 2*reverse_hops) >= + sw_get_least_hops(p_remote_sw, target_lid)) continue; } @@ -2010,12 +2171,6 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, * - illegal state - we shouldn't get here */ - /* second case: skip the port group if the remote (lower) - switch has been already configured for this target LID */ - if (is_real_lid && !is_main_path && - p_remote_sw->p_osm_sw->new_lft[target_lid] != - OSM_NO_PATH) - continue; /* setting fwd tbl port only if this is real LID */ if (is_real_lid) { @@ -2044,19 +2199,26 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, the upper side of the link (on switch with lower rank). Counter is promoted only if we're routing LID on the main path (whether it's a real LID or a dummy one). */ - if (is_main_path) - p_min_port->counter_up++; + /* if (is_main_path) + p_min_port->counter_up++;*/ /* Recursion step: Assign upgoing ports by stepping down, starting on REMOTE switch */ - created_route |= fabric_route_upgoing_by_going_down(p_ftree, p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ - NULL, /* prev. position - NULL to mark that we went down and not up */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether the target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ + routed = fabric_route_upgoing_by_going_down(p_ftree, + p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ + NULL, /* prev. position - NULL to mark that we went down and not up */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether the target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ + created_route|=routed; + if(routed){ + p_min_port->counter_up++; + p_group->counter_up++; + p_group->hca_or_sw.p_sw->counter_up_changed = TRUE; + } } /* done scanning all the down-going port groups */ @@ -2070,6 +2232,8 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return created_route; } /* fabric_route_upgoing_by_going_down() */ + + /***************************************************/ /* @@ -2083,7 +2247,8 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, * assign-down-going-port-by-ascending-up on REMOTE switch (recursion) */ -static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, +static boolean_t +fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, IN ftree_sw_t * p_sw, IN ftree_sw_t * p_prev_sw, IN uint16_t target_lid, @@ -2102,70 +2267,62 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, ftree_port_t *p_min_port; uint16_t i; uint16_t j; + boolean_t created_route = FALSE; + boolean_t routed = FALSE; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); /* Assign upgoing ports by stepping down, starting on THIS switch */ - fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ - p_prev_sw, /* switch that we went up from (NULL means that we went down) */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this path to HCA should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - p_sw->rank, /* the highest visited point in the tree before going down */ - reverse_hops); /* Number of reverse_hops done up to this point */ + created_route = fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ + p_prev_sw, /* switch that we went up from (NULL means that we went down) */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this path to HCA should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + p_sw->rank, /* the highest visited point in the tree before going down */ + reverse_hops); /* Number of reverse_hops done up to this point */ /* recursion stop condition - if it's a root switch, */ - if (p_sw->rank == 0) { - if (reverse_hop_credit > 0) { - /* We go up by going down as we have some reverse_hop_credit left */ - /* We use the index to scatter a bit the reverse up routes */ - p_sw->down_port_groups_idx = - (p_sw->down_port_groups_idx + - 1) % p_sw->down_port_groups_num; - i = p_sw->down_port_groups_idx; - for (j = 0; j < p_sw->down_port_groups_num; j++) { - - p_group = p_sw->down_port_groups[i]; - i = (i + 1) % p_sw->down_port_groups_num; - - /* Skip this port group unless it points to a switch */ - if (p_group->remote_node_type != - IB_NODE_TYPE_SWITCH) - continue; - p_remote_sw = p_group->remote_hca_or_sw.p_sw; - - fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ - } - - } - return; - } - - /* Find the least loaded upgoing port group */ - p_min_group = NULL; - for (i = 0; i < p_sw->up_port_groups_num; i++) { - p_group = p_sw->up_port_groups[i]; - if (!p_min_group) { - /* first group that we're checking - use - it as a group with the lowest load */ - p_min_group = p_group; - } else if (p_group->counter_down < p_min_group->counter_down) { - /* this group is less loaded - use it as min */ - p_min_group = p_group; - } - } - + if (p_sw->rank == 0){ + if(reverse_hop_credit>0){ + /* We go up by going down as we have some reverse_hop_credit left*/ + /* We use the index to scatter a bit the reverse up routes */ + p_sw->down_port_groups_idx = + (p_sw->down_port_groups_idx + 1) % p_sw->down_port_groups_num; + i=p_sw->down_port_groups_idx; + for (j = 0; j < p_sw->down_port_groups_num; j++) { + + p_group = p_sw->down_port_groups[i]; + i = (i + 1) % p_sw->down_port_groups_num; + + /* Skip this port group unless it points to a switch */ + if (p_group->remote_node_type != IB_NODE_TYPE_SWITCH) + continue; + p_remote_sw = p_group->remote_hca_or_sw.p_sw; + + created_route |= fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ + reverse_hops + 1); /* Number of reverse_hops done up to this point */ + } + + } + return created_route; + } + + /* We should generate a list of port sorted by load so we can find easily the least + * going port and explore the other pots on secondary routes more easily (and quickly) */ + bubble_sort_down(p_sw->up_port_groups,p_sw->up_port_groups_num); + + + p_min_group = p_sw->up_port_groups[0]; /* Find the least loaded upgoing port in the selected group */ p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_min_group->ports); @@ -2235,6 +2392,10 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, (on switch with higher rank) */ p_min_group->counter_down++; p_min_port->counter_down++; + if(p_min_group->counter_down == (p_min_group->remote_hca_or_sw.p_sw->min_counter_down+1)){ + recalculate_min_counter_down(p_min_group->remote_hca_or_sw.p_sw); + } + if (is_real_lid) { /* This LID may already be in the LFT in the reverse_hop feature is used */ /* We update the LFT only if this LID isn't already present. */ @@ -2259,20 +2420,20 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ + created_route|= fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + reverse_hop_credit, /* Remaining reverse_hops allowed */ + reverse_hops); /* Number of reverse_hops done up to this point */ } /* we're done for the third case */ if (!is_real_lid) - return; + return created_route; /* What's left to do at this point: * @@ -2306,14 +2467,15 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * - go UP(TRUE,FALSE) to the remote switch */ - for (i = 0; i < p_sw->up_port_groups_num; i++) { + for (i = is_main_path? 1 : 0 ; i < p_sw->up_port_groups_num; i++) { p_group = p_sw->up_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; - /* skip if target lid has been already set on remote switch fwd tbl */ - if (p_remote_sw->p_osm_sw->new_lft[target_lid] != - OSM_NO_PATH) - continue; + /* skip if target lid has been already set on remote switch fwd tbl (with a bigger hop count)*/ + if (p_remote_sw->p_osm_sw->new_lft[target_lid] != OSM_NO_PATH) + if((target_rank -p_remote_sw->rank + 2*reverse_hops) >= + sw_get_least_hops(p_remote_sw, target_lid)) + continue; if (p_sw->is_leaf) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, @@ -2328,8 +2490,23 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, We can safely assume that switch will initiate very few traffic, so there's no point waisting runtime on trying to balance these routes - always pick port 0. */ + /* GET MIN PORT HERE */ + p_min_port = NULL; + ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); + for (j = 0; j < ports_num; j++) { + cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); + if (!p_min_port) { + /* first port that we're checking - use + it as a port with the lowest load */ + p_min_port = p_port; + } else if (p_port->counter_down < p_min_port->counter_down) { + /* this port is less loaded - use it as min */ + p_min_port = p_port; + } + } - cl_ptr_vector_at(&p_group->ports, 0, (void *)&p_port); + p_port = p_min_port; + //cl_ptr_vector_at(&p_group->ports, 0, (void *)&p_port); p_remote_sw->p_osm_sw->new_lft[target_lid] = p_port->remote_port_num; @@ -2342,25 +2519,26 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - TRUE, /* whether the target LID is real or dummy */ - FALSE, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ - } - - /* If we don't have any reverse hop credits, we are done */ - if (reverse_hop_credit == 0) - return; - - /* We explore all the down group ports */ - /* We try to reverse jump for each of them */ - /* They already have a route to us from the upgoing_by_going_down started earlier */ - /* This is only so it'll continue exploring up, after this step backwards */ + routed = fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + TRUE, /* whether the target LID is real or dummy */ + FALSE, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + reverse_hop_credit, /* Remaining reverse_hops allowed */ + reverse_hops); /* Number of reverse_hops done up to this point */ + created_route|=routed; + } + + /* If we don't have any reverse hop credits, we are done */ + if(reverse_hop_credit==0) + return created_route; + + /* We explore all the down group ports */ + /* We try to reverse jump for each of them */ + /* They already have a route to us from the upgoing_by_going_down started earlier */ + /* This is only so it'll continue exploring up, after this step backwards*/ for (i = 0; i < p_sw->down_port_groups_num; i++) { p_group = p_sw->down_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; @@ -2371,16 +2549,17 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, fter doing one step down starting on REMOTE switch. */ - fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - TRUE, /* whether the target LID is real or dummy */ - TRUE, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + created_route |= fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + TRUE, /* whether the target LID is real or dummy */ + TRUE, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ + reverse_hops + 1); /* Number of reverse_hops done up to this point */ } + return created_route; } /* ftree_fabric_route_downgoing_by_going_up() */ From hal.rosenstock at gmail.com Sat Jun 13 07:58:20 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sat, 13 Jun 2009 10:58:20 -0400 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Determine SL properly In-Reply-To: <20090613103400.GJ10489@me> References: <20090529153515.GA10301@comcast.net> <20090613103400.GJ10489@me> Message-ID: Hi Sasha, On Sat, Jun 13, 2009 at 6:34 AM, Sasha Khapyorsky wrote: > Hi Hal, > > On 11:35 Fri 29 May     , Hal Rosenstock wrote: >> >> rather than assuming SL 0 >> >> Signed-off-by: Hal Rosenstock >> --- >> diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c >> index 691bdc3..f17da11 100644 >> --- a/libibmad/src/resolve.c >> +++ b/libibmad/src/resolve.c >> @@ -59,6 +59,7 @@ int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, >>               return -1; >> >>       mad_decode_field(portinfo, IB_PORT_SMLID_F, &lid); >> +     mad_decode_field(portinfo, IB_PORT_SMSL_F, &sm_id->sl); >> >>       return ib_portid_set(sm_id, lid, 0, 0); >>  } >> @@ -74,12 +75,23 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, >>  { >>       ib_portid_t sm_portid; >>       char buf[IB_SA_DATA_SIZE] = { 0 }; >> +     ib_portid_t self = { 0 }; >> +     uint64_t selfguid; >> +     ibmad_gid_t selfgid; >> +     uint8_t nodeinfo[64]; >> >>       if (!sm_id) { >>               sm_id = &sm_portid; >>               if (ib_resolve_smlid_via(sm_id, timeout, srcport) < 0) >>                       return -1; >>       } >> + >> +     if (!smp_query_via(nodeinfo, &self, IB_ATTR_NODE_INFO, 0, 0, srcport)) >> +             return -1; >> +     mad_decode_field(nodeinfo, IB_NODE_PORT_GUID_F, &selfguid); >> +     mad_set_field64(selfgid, 0, IB_GID_PREFIX_F, IB_DEFAULT_SUBN_PREFIX); >> +     mad_set_field64(selfgid, 0, IB_GID_GUID_F, selfguid); >> + >>       if (*(uint64_t *) & portid->gid == 0) >>               mad_set_field64(portid->gid, 0, IB_GID_PREFIX_F, >>                               IB_DEFAULT_SUBN_PREFIX); >> @@ -87,10 +99,11 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, >>               mad_set_field64(portid->gid, 0, IB_GID_GUID_F, *guid); >> >>       if ((portid->lid = >> -          ib_path_query_via(srcport, portid->gid, portid->gid, sm_id, >> +          ib_path_query_via(srcport, selfgid, portid->gid, sm_id, >>                              buf)) < 0) >>               return -1; >> >> +     mad_decode_field(buf, IB_SA_PR_SL_F, &portid->sl); > > Just to make sure what did you mean here (please more informative commit > message would be helpful)... I thought it was a minor change and obvious from the code diff. > Instead of just resolving destination port's lid by guid using its port gid (and gid is derived from guid) > to itself SA PR you are trying also to resolve SL which should be used > from "self" to destination port. Right? Yes. > Assuming so. Would in this case "self" and srcport be same? Not sure what you mean by "same". They are different as to their underlying type. -- Hal > Sasha > >>       return 0; >>  } >> >> @@ -167,6 +180,7 @@ int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, >>               return -1; >> >>       mad_decode_field(portinfo, IB_PORT_LID_F, &portid->lid); >> +     mad_decode_field(portinfo, IB_PORT_SMSL_F, &portid->sl); >>       mad_decode_field(portinfo, IB_PORT_GID_PREFIX_F, &prefix); >>       mad_decode_field(nodeinfo, IB_NODE_PORT_GUID_F, &guid); >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Sat Jun 13 07:58:51 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sat, 13 Jun 2009 10:58:51 -0400 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Remove compile warning on x86_64 arch In-Reply-To: <20090613103645.GK10489@me> References: <20090601114622.GA31380@comcast.net> <20090613103645.GK10489@me> Message-ID: On Sat, Jun 13, 2009 at 6:36 AM, Sasha Khapyorsky wrote: > On 07:46 Mon 01 Jun     , hnrose at comcast.net wrote: >> >> resolve.c:106: warning: pointer targets in passing argument 1 of 'mad_decode_field' differ in signedness >> >> Signed-off-by: Hal Rosenstock >> >> diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c >> index f17da11..abcab17 100644 >> --- a/libibmad/src/resolve.c >> +++ b/libibmad/src/resolve.c >> @@ -103,7 +103,7 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, >>                              buf)) < 0) >>               return -1; >> >> -     mad_decode_field(buf, IB_SA_PR_SL_F, &portid->sl); >> +     mad_decode_field((uint8_t *)buf, IB_SA_PR_SL_F, &portid->sl); > > Why to not change type of buf and eliminate casting (and warning)? I'll do it that way (in the next patch version). -- Hal > > Sasha > >>       return 0; >>  } >> >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From yhlu.kernel at gmail.com Sat Jun 13 12:28:29 2009 From: yhlu.kernel at gmail.com (Yinghai Lu) Date: Sat, 13 Jun 2009 12:28:29 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> Message-ID: <86802c440906131228y6365f23fi59cd04f381fe92fc@mail.gmail.com> On Fri, Jun 12, 2009 at 9:30 PM, Roland Dreier wrote: > Something like this should help -- can you verify this fixes the double > free of irq? > >  drivers/infiniband/hw/mthca/mthca_eq.c |    4 +++- >  1 files changed, 3 insertions(+), 1 deletions(-) > > diff --git a/drivers/infiniband/hw/mthca/mthca_eq.c b/drivers/infiniband/hw/mthca/mthca_eq.c > index 28f0e0c..90e4e45 100644 > --- a/drivers/infiniband/hw/mthca/mthca_eq.c > +++ b/drivers/infiniband/hw/mthca/mthca_eq.c > @@ -641,9 +641,11 @@ static void mthca_free_irqs(struct mthca_dev *dev) >        if (dev->eq_table.have_irq) >                free_irq(dev->pdev->irq, dev); >        for (i = 0; i < MTHCA_NUM_EQ; ++i) > -               if (dev->eq_table.eq[i].have_irq) > +               if (dev->eq_table.eq[i].have_irq) { >                        free_irq(dev->eq_table.eq[i].msi_x_vector, >                                 dev->eq_table.eq + i); > +                       dev->eq_table.eq[i].have_irq = 0; > +               } >  } > >  static int mthca_map_reg(struct mthca_dev *dev, > thanks, that fixes the problem. got [ 688.153102] ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 2008) [ 688.164128] ib_mthca: Initializing 0000:c4:00.0 [ 688.170696] ib_mthca 0000:c4:00.0: PCI INT A -> GSI 98 (level, low) -> IRQ 98 [ 688.184033] ib_mthca 0000:c4:00.0: setting latency timer to 64 [ 688.188594] ib_mthca 0000:c4:00.0: using 64bit DMA mask [ 688.208718] ib_mthca 0000:c4:00.0: using 64bit consistent DMA mask [ 688.217191] reserve_memtype added 0xfbef0000-0xfbef1000, track uncached-minus, req uncached-minus, ret uncached-minus [ 688.232259] free_memtype request 0xfbef0000-0xfbef1000 [ 689.243254] reserve_memtype added 0xfbe80000-0xfbe81000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.537336] reserve_memtype added 0xfa800000-0xfa801000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.587286] ib_mthca 0000:c4:00.0: HCA FW version 4.7.400 is old (4.8.200 is current). [ 689.592067] ib_mthca 0000:c4:00.0: If you have problems, try updating your HCA FW. [ 689.603264] reserve_memtype added 0xfbe82000-0xfbe83000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.622765] alloc irq_desc for 230 on node 0 [ 689.629521] alloc kstat_irqs on node 0 [ 689.641401] ib_mthca 0000:c4:00.0: irq 230 for MSI/MSI-X [ 689.645155] alloc irq_desc for 231 on node 0 [ 689.647072] alloc kstat_irqs on node 0 [ 689.663348] ib_mthca 0000:c4:00.0: irq 231 for MSI/MSI-X [ 689.666000] alloc irq_desc for 232 on node 0 [ 689.682655] alloc kstat_irqs on node 0 [ 689.685768] ib_mthca 0000:c4:00.0: irq 232 for MSI/MSI-X [ 689.701511] reserve_memtype added 0xfa802000-0xfa803000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.721752] reserve_memtype added 0xe5800000-0xe6000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.731141] reserve_memtype added 0xe0000000-0xe4000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.753919] reserve_memtype added 0xfbef0000-0xfbef1000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.770375] Overlap at 0xfbe80000-0xfbe81000 [ 689.782317] reserve_memtype added 0xfbe80000-0xfbe81000, track uncached-minus, req uncached-minus, ret uncached-minus [ 689.902837] ib_mthca 0000:c4:00.0: NOP command failed to generate interrupt (IRQ 232). [ 689.907900] ib_mthca 0000:c4:00.0: Trying again with MSI-X disabled. [ 689.923292] free_memtype request 0xfbe80000-0xfbe81000 [ 689.926430] free_memtype request 0xfbef0000-0xfbef1000 [ 689.942964] free_memtype request 0xe0000000-0xe4000000 [ 689.947059] free_memtype request 0xe5800000-0xe6000000 [ 689.962869] free_memtype request 0xfa802000-0xfa803000 [ 689.967293] free_memtype request 0xfbe82000-0xfbe83000 [ 689.980961] reserve_memtype added 0xfa802000-0xfa803000, track uncached-minus, req uncached-minus, ret uncached-minus [ 690.003659] reserve_memtype added 0xe5800000-0xe6000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 690.015144] reserve_memtype added 0xe0000000-0xe4000000, track uncached-minus, req uncached-minus, ret uncached-minus [ 690.033814] reserve_memtype added 0xfbef0000-0xfbef1000, track uncached-minus, req uncached-minus, ret uncached-minus [ 690.046945] Overlap at 0xfbe80000-0xfbe81000 [ 690.062803] reserve_memtype added 0xfbe80000-0xfbe81000, track uncached-minus, req uncached-minus, ret uncached-minus [ 690.182718] ib_mthca 0000:c4:00.0: NOP command failed to generate interrupt (IRQ 98), aborting. [ 690.188676] ib_mthca 0000:c4:00.0: BIOS or ACPI interrupt routing problem? [ 690.203724] free_memtype request 0xfbe80000-0xfbe81000 [ 690.205768] free_memtype request 0xfbef0000-0xfbef1000 [ 690.224080] free_memtype request 0xe0000000-0xe4000000 [ 690.228558] free_memtype request 0xe5800000-0xe6000000 [ 690.241523] free_memtype request 0xfa802000-0xfa803000 [ 690.246196] free_memtype request 0xfbe80000-0xfbe81000 [ 690.263133] free_memtype request 0xfa800000-0xfa801000 [ 690.268628] ib_mthca 0000:c4:00.0: PCI INT A disabled [ 690.281553] ib_mthca: probe of 0000:c4:00.0 failed with error -16 as expected. YH From rdreier at cisco.com Sat Jun 13 15:04:28 2009 From: rdreier at cisco.com (Roland Dreier) Date: Sat, 13 Jun 2009 15:04:28 -0700 Subject: [ofa-general] Re: mthca double free irqs In-Reply-To: <86802c440906131228y6365f23fi59cd04f381fe92fc@mail.gmail.com> (Yinghai Lu's message of "Sat, 13 Jun 2009 12:28:29 -0700") References: <86802c440906111945n2a42dc94mf08bd2b1a252842f@mail.gmail.com> <86802c440906131228y6365f23fi59cd04f381fe92fc@mail.gmail.com> Message-ID: > thanks, that fixes the problem. Thanks, will merge with your reported-by/tested-by included. - R. From dorfman.eli at gmail.com Sat Jun 13 23:47:38 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Sun, 14 Jun 2009 09:47:38 +0300 Subject: [ofa-general] [Fwd: Re: [PATCH ] opensm: MFT tables are not set after non full member re-join] Message-ID: <4A349D0A.1070004@gmail.com> Hi Sasha, Resending. I haven't seen a response to this patch. Eli -------- Original Message -------- Subject: Re: [PATCH ] opensm: MFT tables are not set after non full member re-join Date: Tue, 19 May 2009 11:56:36 +0300 From: Eli Dorfman (Voltaire) To: Sasha Khapyorsky CC: OpenIB References: <4A1019F6.5060900 at gmail.com> Eli Dorfman (Voltaire) wrote: > MFT tables are not set after non full member re-join > > In case of non full member re-join MFT tables are not set. > No need to set or check non full member reference to mlid (port->mcm_list). > This list should be used only for full members for cleanup when port goes down. > > A simple scenarion to reproduce this: > 1. Full member creates group > 2. Non-member join - MFT sent > 3. Full member leave > a. group is deleted but non member port has still reference to the MLID > 4. Full member re-creates the group > 5. Non member re-joins - MFT *NOT* sent to switches > > Signed-off-by: Eli Dorfman > --- > opensm/include/opensm/osm_sm.h | 3 ++- > opensm/opensm/osm_sa_mcmember_record.c | 6 +++--- > opensm/opensm/osm_sm.c | 22 +++++++++++++++++++++- > 3 files changed, 26 insertions(+), 5 deletions(-) > > diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h > index cc8321d..1a8a577 100644 > --- a/opensm/include/opensm/osm_sm.h > +++ b/opensm/include/opensm/osm_sm.h > @@ -539,7 +539,8 @@ osm_resp_send(IN osm_sm_t * sm, > ib_api_status_t > osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > IN const ib_net16_t mlid, > - IN const ib_net64_t port_guid); > + IN const ib_net64_t port_guid, > + IN uint8_t scope_state); > /* > * PARAMETERS > * p_sm > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > index 5543221..fe29dd6 100644 > --- a/opensm/opensm/osm_sa_mcmember_record.c > +++ b/opensm/opensm/osm_sa_mcmember_record.c > @@ -1039,7 +1039,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > if (!p_mgrp) { > char gid_str[INET6_ADDRSTRLEN]; > CL_PLOCK_RELEASE(sa->p_lock); > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > + OSM_LOG(sa->p_log, OSM_LOG_INFO, > "Failed since multicast group %s not present\n", > inet_ntop(AF_INET6, p_recvd_mcmember_rec->mgid.raw, > gid_str, sizeof gid_str)); > @@ -1309,8 +1309,8 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > > /* do the actual routing (actually schedule the update) */ > status = osm_sm_mcgrp_join(sa->sm, mlid, > - p_recvd_mcmember_rec->port_gid.unicast. > - interface_id); > + p_recvd_mcmember_rec->port_gid.unicast.interface_id, > + p_recvd_mcmember_rec->scope_state); > > if (status != IB_SUCCESS) { > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " > diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c > index daa60ff..b334d39 100644 > --- a/opensm/opensm/osm_sm.c > +++ b/opensm/opensm/osm_sm.c > @@ -468,7 +468,7 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, > /********************************************************************** > **********************************************************************/ > ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > - IN const ib_net64_t port_guid) > + IN const ib_net64_t port_guid, IN uint8_t scope_state) > { > osm_mgrp_t *p_mgrp; > osm_port_t *p_port; > @@ -515,6 +515,25 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > goto Exit; > } > > + /* if there was no change from the last time > + * we processed the group we can skip doing anything > + */ > + if (p_mgrp->last_change_id == p_mgrp->last_tree_id) { > + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, > + "Skip processing mgrp with lid:0x%X last change id:%u\n", > + cl_ntoh16(mlid), p_mgrp->last_change_id); > + goto Exit; > + } else { > + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, > + "processing mgrp with lid:0x%X port: 0x%016" PRIx64 " last change id:%u tree id:%u\n", > + cl_ntoh16(mlid), cl_ntoh64(port_guid), > + p_mgrp->last_change_id, p_mgrp->last_tree_id); > + } > + > + /* add mgrp only to FULL member port. used for cleanup when port goes down */ > + if (!(scope_state & IB_JOIN_STATE_FULL)) > + goto MgrpProcess; > + > /* > * Check if the object (according to mlid) already exists on this port. > * If it does - then no need to update it again, and no need to > @@ -543,6 +562,7 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > goto Exit; > } > > +MgrpProcess: > status = sm_mgrp_process(p_sm, p_mgrp); > CL_PLOCK_RELEASE(p_sm->p_lock); > The following fixes a bug in the above PATCH that will lock the opensm when multicast group was not changed. diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index b334d39..28cd76f 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -519,6 +519,7 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, * we processed the group we can skip doing anything */ if (p_mgrp->last_change_id == p_mgrp->last_tree_id) { + CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, "Skip processing mgrp with lid:0x%X last change id:%u\n", cl_ntoh16(mlid), p_mgrp->last_change_id); From dorfman.eli at gmail.com Sat Jun 13 23:48:37 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Sun, 14 Jun 2009 09:48:37 +0300 Subject: [ofa-general] [Fwd: [PATCH] opensm/osm_sa.c add attribute and component mask to error message] Message-ID: <4A349D45.3010301@gmail.com> Resending. -------- Original Message -------- Subject: [PATCH] opensm/osm_sa.c add attribute and component mask to error message Date: Thu, 04 Jun 2009 15:30:34 +0300 From: Eli Dorfman (Voltaire) To: Sasha Khapyorsky CC: OpenIB add attribute and component mask to error message about too many records for SubnAdmGet query Signed-off-by: Eli Dorfman --- opensm/opensm/osm_sa.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index 3521132..20e70cd 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -422,8 +422,9 @@ void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, */ if (sa_mad->method == IB_MAD_METHOD_GET && num_rec > 1) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4C05: " - "Got more than one record for SubnAdmGet (%u)\n", - num_rec); + "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" PRIx64 "\n", + num_rec, ib_get_sa_attr_str(sa_mad->attr_id), + cl_ntoh64(sa_mad->comp_mask)); osm_sa_send_error(sa, madw, IB_SA_MAD_STATUS_TOO_MANY_RECORDS); goto Exit; } -- 1.5.5 From dorfman.eli at gmail.com Sat Jun 13 23:49:10 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Sun, 14 Jun 2009 09:49:10 +0300 Subject: [ofa-general] [Fwd: [PATCH] opensm/osm_sa_node_record.c use comp mask to match by LID or GUID] Message-ID: <4A349D66.30205@gmail.com> Resending. -------- Original Message -------- Subject: [PATCH] opensm/osm_sa_node_record.c use comp mask to match by LID or GUID Date: Sun, 07 Jun 2009 18:06:00 +0300 From: Eli Dorfman (Voltaire) To: Sasha Khapyorsky CC: OpenIB , Hal Rosenstock use comp mask to match by LID or GUID when matching node records use comp mask to filter queries that use 0 lid or guid while mask is set Signed-off-by: Eli Dorfman --- opensm/opensm/osm_sa_node_record.c | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/opensm/opensm/osm_sa_node_record.c b/opensm/opensm/osm_sa_node_record.c index d42e6ac..d04ce63 100644 --- a/opensm/opensm/osm_sa_node_record.c +++ b/opensm/opensm/osm_sa_node_record.c @@ -114,7 +114,8 @@ static void nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * p_node, IN cl_qlist_t * p_list, IN ib_net64_t const match_port_guid, IN ib_net16_t const match_lid, - IN const osm_physp_t * p_req_physp) + IN const osm_physp_t * p_req_physp, + IN const ib_net64_t comp_mask) { const osm_physp_t *p_physp; uint8_t port_num; @@ -153,7 +154,7 @@ static void nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * p_node, port_guid = osm_physp_get_port_guid(p_physp); - if (match_port_guid && (port_guid != match_port_guid)) + if ((comp_mask & IB_NR_COMPMASK_PORTGUID) && (port_guid != match_port_guid)) continue; base_lid = osm_physp_get_base_lid(p_physp); @@ -162,7 +163,7 @@ static void nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * p_node, max_lid_ho = (uint16_t) (base_lid_ho + (1 << lmc) - 1); match_lid_ho = cl_ntoh16(match_lid); - if (match_lid_ho) { + if (comp_mask & IB_NR_COMPMASK_LID) { /* We validate that the lid belongs to this node. */ @@ -190,7 +191,7 @@ static void nr_rcv_by_comp_mask(IN cl_map_item_t * p_map_item, IN void *context) const ib_node_record_t *const p_rcvd_rec = p_ctxt->p_rcvd_rec; const osm_physp_t *const p_req_physp = p_ctxt->p_req_physp; osm_sa_t *sa = p_ctxt->sa; - ib_net64_t const comp_mask = p_ctxt->comp_mask; + ib_net64_t comp_mask = p_ctxt->comp_mask; ib_net64_t match_port_guid = 0; ib_net16_t match_lid = 0; @@ -274,7 +275,7 @@ static void nr_rcv_by_comp_mask(IN cl_map_item_t * p_map_item, IN void *context) } nr_rcv_create_nr(sa, p_node, p_ctxt->p_list, match_port_guid, - match_lid, p_req_physp); + match_lid, p_req_physp, comp_mask); Exit: OSM_LOG_EXIT(p_ctxt->sa->p_log); -- 1.5.5 From vlad at lists.openfabrics.org Sun Jun 14 02:19:25 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 14 Jun 2009 02:19:25 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090614-0200 daily build status Message-ID: <20090614091925.E82941020B7E@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:405: error: implicit declaration of function 'ipv6_addr_any' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:405: error: implicit declaration of function 'ipv6_addr_any' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'ia64_mv.dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'ia64_mv.dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'ia64_mv.dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'ia64_mv.dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:37: include/linux/slub_def.h: In function 'kmalloc_index': include/linux/slub_def.h:90: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/.tmp_umem.o /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090614-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From rdreier at cisco.com Sun Jun 14 13:47:59 2009 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 14 Jun 2009 13:47:59 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This has the main batch of InfiniBand/RDMA changes for 2.6.31; sorry for being so late in the merge cycle but the whole family was sick with the stomach flu for the past week and, well, merge requests weren't a top priority. Anyway, this has been a slow cycle for my tree for whatever reason, so there's not much here; everything has been in -next for a while except for my two IRQ double-free patches, which fix a just-reported bug (and are tiny and have been tested). Anton Blanchard (1): IB/ehca: Replace vmalloc() with kmalloc() for queue allocation Eli Cohen (2): IB/mthca: Add module parameter for number of MTTs per segment mlx4_core: Add module parameter for number of MTTs per segment Jack Morgenstein (1): IB/mlx4: Add strong ordering to local inval and fast reg work requests Jack Stone (1): infiniband: Remove void casts Joachim Fenkes (1): IB/ehca: Remove superfluous bitmasks from QP control block Roel Kluin (1): RDMA/nes: Fix off-by-one bugs in reset_adapter_ne020() and init_serdes() Roland Dreier (3): IB/mthca: Don't double-free IRQs when falling back from MSI-X to INTx mlx4_core: Don't double-free IRQs when falling back from MSI-X to INTx Merge branches 'cxgb3', 'ehca', 'misc', 'mlx4', 'mthca' and 'nes' into for-linus Stefan Roscher (3): IB/ehca: Fall back to vmalloc() for big allocations IB/ehca: Remove unnecessary memory operations for userspace queue pairs IB/ehca: Increment version number Steve Wise (2): RDMA/cxgb3: Report correct port state and MTU RDMA/cxgb3: Limit fast register size based on T3 limitations drivers/infiniband/hw/amso1100/c2_cq.c | 4 +- drivers/infiniband/hw/cxgb3/cxio_wr.h | 2 +- drivers/infiniband/hw/cxgb3/iwch_provider.c | 32 ++++++- drivers/infiniband/hw/ehca/ehca_classes_pSeries.h | 28 ----- drivers/infiniband/hw/ehca/ehca_irq.c | 9 +- drivers/infiniband/hw/ehca/ehca_main.c | 2 +- drivers/infiniband/hw/ehca/ehca_qp.c | 112 +++++++++++---------- drivers/infiniband/hw/ehca/hcp_if.c | 6 +- drivers/infiniband/hw/ehca/hcp_if.h | 2 +- drivers/infiniband/hw/ehca/hcp_phyp.c | 11 ++- drivers/infiniband/hw/ehca/hcp_phyp.h | 2 +- drivers/infiniband/hw/ehca/ipz_pt_fn.c | 19 +++- drivers/infiniband/hw/mlx4/qp.c | 4 + drivers/infiniband/hw/mthca/mthca_cmd.c | 2 +- drivers/infiniband/hw/mthca/mthca_dev.h | 1 + drivers/infiniband/hw/mthca/mthca_eq.c | 4 +- drivers/infiniband/hw/mthca/mthca_main.c | 17 +++- drivers/infiniband/hw/mthca/mthca_mr.c | 16 ++-- drivers/infiniband/hw/mthca/mthca_profile.c | 4 +- drivers/infiniband/hw/nes/nes_hw.c | 14 ++-- drivers/net/mlx4/eq.c | 4 +- drivers/net/mlx4/main.c | 14 ++- drivers/net/mlx4/mr.c | 6 +- drivers/net/mlx4/profile.c | 2 +- include/linux/mlx4/device.h | 1 + include/linux/mlx4/qp.h | 1 + 26 files changed, 183 insertions(+), 136 deletions(-) From acceptany at gmail.com Sun Jun 14 19:19:30 2009 From: acceptany at gmail.com (Jordan) Date: Mon, 15 Jun 2009 10:19:30 +0800 Subject: [ofa-general] Can I assign custom LIDs to a port ? Message-ID: <91fe68d50906141919r352d5f33x3b08056dae1b0668@mail.gmail.com> Now I need to assign a port with custom LIDs, for example , I want to assign the number "1, 2, 3" to a port . Is there some methods to do so ? It seems that opensm doesn't have this function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dotanba at gmail.com Sun Jun 14 22:18:10 2009 From: dotanba at gmail.com (Dotan Barak) Date: Mon, 15 Jun 2009 08:18:10 +0300 Subject: [ofa-general] Can I assign custom LIDs to a port ? In-Reply-To: <91fe68d50906141919r352d5f33x3b08056dae1b0668@mail.gmail.com> References: <91fe68d50906141919r352d5f33x3b08056dae1b0668@mail.gmail.com> Message-ID: <2f3bf9a60906142218v4ab1b3dekd50af4b2a9f2726a@mail.gmail.com> Hi. LMC is the most similar thing to this question. Dotan On Mon, Jun 15, 2009 at 5:19 AM, Jordan wrote: > Now I need to assign a port with custom LIDs, for example , I want to assign > the number "1, 2, 3" to a port . Is there some methods to do so ? It seems > that opensm doesn't have this  function. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From dorfman.eli at gmail.com Sun Jun 14 23:07:38 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 15 Jun 2009 09:07:38 +0300 Subject: [ofa-general] [PATCH v2] opensm: MFT tables are not set after non full member re-join In-Reply-To: <4A127444.8080707@gmail.com> References: <4A1019F6.5060900@gmail.com> <4A127444.8080707@gmail.com> Message-ID: <4A35E52A.3020502@gmail.com> MFT tables are not set after non full member re-join In case of non full member re-join MFT tables are not set. No need to set or check non full member reference to mlid (port->mcm_list). This list should be used only for full members for cleanup when port goes down. A simple scenarion to reproduce this: 1. Full member creates group 2. Non-member join - MFT sent 3. Full member leave a. group is deleted but non member port has still reference to the MLID 4. Full member re-creates the group 5. Non member re-joins - MFT not sent to switches Signed-off-by: Eli Dorfman --- opensm/include/opensm/osm_sm.h | 3 ++- opensm/opensm/osm_sa_mcmember_record.c | 6 +++--- opensm/opensm/osm_sm.c | 22 +++++++++++++++++++++- 3 files changed, 26 insertions(+), 5 deletions(-) diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h index cc8321d..1a8a577 100644 --- a/opensm/include/opensm/osm_sm.h +++ b/opensm/include/opensm/osm_sm.h @@ -539,7 +539,8 @@ osm_resp_send(IN osm_sm_t * sm, ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid); + IN const ib_net64_t port_guid, + IN uint8_t scope_state); /* * PARAMETERS * p_sm diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 5543221..fe29dd6 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1039,7 +1039,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) if (!p_mgrp) { char gid_str[INET6_ADDRSTRLEN]; CL_PLOCK_RELEASE(sa->p_lock); - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + OSM_LOG(sa->p_log, OSM_LOG_INFO, "Failed since multicast group %s not present\n", inet_ntop(AF_INET6, p_recvd_mcmember_rec->mgid.raw, gid_str, sizeof gid_str)); @@ -1309,8 +1309,8 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* do the actual routing (actually schedule the update) */ status = osm_sm_mcgrp_join(sa->sm, mlid, - p_recvd_mcmember_rec->port_gid.unicast. - interface_id); + p_recvd_mcmember_rec->port_gid.unicast.interface_id, + p_recvd_mcmember_rec->scope_state); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index daa60ff..b334d39 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -468,7 +468,7 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, /********************************************************************** **********************************************************************/ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid) + IN const ib_net64_t port_guid, IN uint8_t scope_state) { osm_mgrp_t *p_mgrp; osm_port_t *p_port; @@ -515,6 +515,26 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } + /* if there was no change from the last time + * we processed the group we can skip doing anything + */ + if (p_mgrp->last_change_id == p_mgrp->last_tree_id) { + CL_PLOCK_RELEASE(p_sm->p_lock); + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, + "Skip processing mgrp with lid:0x%X last change id:%u\n", + cl_ntoh16(mlid), p_mgrp->last_change_id); + goto Exit; + } else { + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "processing mgrp with lid:0x%X port: 0x%016" PRIx64 " last change id:%u tree id:%u\n", + cl_ntoh16(mlid), cl_ntoh64(port_guid), + p_mgrp->last_change_id, p_mgrp->last_tree_id); + } + + /* add mgrp only to FULL member port. used for cleanup when port goes down */ + if (!(scope_state & IB_JOIN_STATE_FULL)) + goto MgrpProcess; + /* * Check if the object (according to mlid) already exists on this port. * If it does - then no need to update it again, and no need to @@ -543,6 +563,7 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } +MgrpProcess: status = sm_mgrp_process(p_sm, p_mgrp); CL_PLOCK_RELEASE(p_sm->p_lock); -- 1.5.3.6 From yiftahs at voltaire.com Sun Jun 14 23:32:02 2009 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Mon, 15 Jun 2009 09:32:02 +0300 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A2B75D4.7080600@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il> Message-ID: <39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> Pasha, All, I wrote a long detailed mail addressing many of the issues but most, if not all, of the points were already addressed by others so I will try to be short. I can see from the mails and from my personal experience that most of the "end users" do not need/use the MPI coming as part of OFED (they have many different MPI installed in their clusters), as we can see distro are not using it and also some (if not all) of OFED binary package providers (i.e. companies) do not use it. We think that the simple & clear answer is: "take the MPI packages out of OFED" This will serve us all good and it will be step in the right direction. Regards, Yiftah > -----Original Message----- > From: ewg-bounces at lists.openfabrics.org [mailto:ewg- > bounces at lists.openfabrics.org] On Behalf Of Pavel Shamis (Pasha) > Sent: Sunday, June 07, 2009 11:10 > To: Tziporet Koren > Cc: EWG; OpenFabrics General > Subject: Re: [ewg] RFC: Do we wish to take MPI out of OFED? > > > > - Need to synchronize between different projects > The synchronization between OFED and MPIs are minimal. > In most cases OFED just take last available MPI version. > > It happens that OFED discovers some really critical bug > and we ask MPI to provide new version. But for critical > bugs, MPI team anyway releases new bugfix version ASAP. > > > > > - MPI is an important RDMA ULP and although it is not developed in OFA > > it is widely used by OFED customers > I guess that MPI users are the widest OFED user community and including > MPI as part of OFED definitely simplify > end-user live. > > I'm personally against excluding MPI from OFED package. > > Pasha. > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From eli at dev.mellanox.co.il Mon Jun 15 00:34:08 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:34:08 +0300 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet Message-ID: <20090615073408.GA27543@mtls03> RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over Ethernet, providing IB capabilities for Ethernet fabrics. The packets are standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport headers and payload. HCA RDMAoE ports are no different than regular IB ports from the RDMA stack perspective. IB subnet management and SA services are not required for RDMAoE operation; Ethernet management practices are used instead. In Ethernet, nodes are commonly referred to by applications by means of an IP address. RDMAoE treats IP addresses that were assigned to the corresponding Ethernet port as GIDs, and makes use of the IP stack to bind a destination address to the corresponding netdevice (just as the CMA does today for IB and iWARP) and to obtain its L2 MAC addresses. The RDMA Verbs API is syntactically unmodified. When referring to RDMAoE ports, Address handles are required to contain GIDs and the L2 address fields in the API are ignored. The Ethernet L2 information is then obtained by the vendor-specific driver (both in kernel- and user-space) while modifying QPs to RTR and creating address handles. In order to maintain application compatibility, RDMAoE implements a SA_Query API that locally returns path records with the corresponding GIDs and the other relevant parameters . Consequently, any CMA or native Verbs application, in kernel or user-space, that uses path queries to obtain its address information, will run transparently over RDMAoE with no changes. We have successfully tested MPI, SDP, RDS, and native Verbs applications over RDMAoE without *any* changes. In the mlx4 driver stack, mlx4_en must be loaded and the corresponding eth Ethernet (with support for RDMAoE) or IB, as it was already the case. Following is a series of 9 patches based on version 2.6.30 of the Linux kernel. Signed-off-by: Eli Cohen drivers/infiniband/core/addr.c | 20 ++- drivers/infiniband/core/agent.c | 16 +- drivers/infiniband/core/cma.c | 39 ++++- drivers/infiniband/core/mad.c | 48 ++++-- drivers/infiniband/core/multicast.c | 153 ++++++++++++++-- drivers/infiniband/core/sa_query.c | 167 ++++++++++++++---- drivers/infiniband/core/ud_header.c | 111 ++++++++++++ drivers/infiniband/core/uverbs.h | 1 + drivers/infiniband/core/uverbs_cmd.c | 34 ++++ drivers/infiniband/core/uverbs_main.c | 1 + drivers/infiniband/core/verbs.c | 18 ++ drivers/infiniband/hw/mlx4/ah.c | 228 ++++++++++++++++++++---- drivers/infiniband/hw/mlx4/main.c | 276 ++++++++++++++++++++++++++--- drivers/infiniband/hw/mlx4/mlx4_ib.h | 30 +++- drivers/infiniband/hw/mlx4/qp.c | 253 +++++++++++++++++++++----- drivers/infiniband/ulp/ipoib/ipoib_main.c | 3 + drivers/net/mlx4/cmd.c | 6 + drivers/net/mlx4/en_main.c | 15 ++- drivers/net/mlx4/en_port.c | 4 +- drivers/net/mlx4/en_port.h | 3 +- drivers/net/mlx4/intf.c | 20 ++ drivers/net/mlx4/main.c | 6 + drivers/net/mlx4/mlx4.h | 1 + include/linux/mlx4/cmd.h | 1 + include/linux/mlx4/device.h | 31 +++- include/linux/mlx4/driver.h | 16 ++- include/linux/mlx4/qp.h | 8 +- include/rdma/ib_addr.h | 51 ++++++ include/rdma/ib_pack.h | 26 +++ include/rdma/ib_user_verbs.h | 21 ++- include/rdma/ib_verbs.h | 22 +++ 31 files changed, 1419 insertions(+), 210 deletions(-) From eli at mellanox.co.il Mon Jun 15 00:34:19 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:34:19 +0300 Subject: [ofa-general] [PATCH 1/9] ib_core: Add API to query port link type Message-ID: <20090615073419.GB27543@mtls03> This allows to get the type of a port to be either Ethernet or IB which is required by following patches for implementing RDMA over Ethernet - RDMAoE. Signed-off-by: Eli Cohen --- drivers/infiniband/core/verbs.c | 7 +++++++ include/rdma/ib_verbs.h | 11 +++++++++++ 2 files changed, 18 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index a7da9be..18631b1 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -94,6 +94,13 @@ rdma_node_get_transport(enum rdma_node_type node_type) } EXPORT_SYMBOL(rdma_node_get_transport); +enum ib_port_link_type ib_get_port_link_type(struct ib_device *device, u8 port_num) +{ + return device->get_port_link_type ? + device->get_port_link_type(device, port_num) : PORT_LINK_IB; +} +EXPORT_SYMBOL(ib_get_port_link_type); + /* Protection domains */ struct ib_pd *ib_alloc_pd(struct ib_device *device) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index c179318..8f98c8f 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -72,6 +72,11 @@ enum rdma_transport_type { RDMA_TRANSPORT_IWARP }; +enum ib_port_link_type { + PORT_LINK_IB, + PORT_LINK_ETH +}; + enum rdma_transport_type rdma_node_get_transport(enum rdma_node_type node_type) __attribute_const__; @@ -298,6 +303,7 @@ struct ib_port_attr { u8 active_width; u8 active_speed; u8 phys_state; + enum ib_port_link_type link_type; }; enum ib_device_modify_flags { @@ -1003,6 +1009,8 @@ struct ib_device { int (*query_port)(struct ib_device *device, u8 port_num, struct ib_port_attr *port_attr); + enum ib_port_link_type (*get_port_link_type)(struct ib_device *device, + u8 port_num); int (*query_gid)(struct ib_device *device, u8 port_num, int index, union ib_gid *gid); @@ -1213,6 +1221,9 @@ int ib_query_device(struct ib_device *device, int ib_query_port(struct ib_device *device, u8 port_num, struct ib_port_attr *port_attr); +enum ib_port_link_type ib_get_port_link_type(struct ib_device *device, + u8 port_num); + int ib_query_gid(struct ib_device *device, u8 port_num, int index, union ib_gid *gid); -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:34:31 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:34:31 +0300 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID --> MAC translations Message-ID: <20090615073431.GC27543@mtls03> A few support functions are added to allow the translation from GID to MAC which is required by hw drivers supporting RDMAoE. Signed-off-by: Eli Cohen --- drivers/infiniband/core/uverbs.h | 1 + drivers/infiniband/core/uverbs_cmd.c | 34 +++++++++++++++++++++++++++++++++ drivers/infiniband/core/uverbs_main.c | 1 + drivers/infiniband/core/verbs.c | 11 ++++++++++ include/rdma/ib_user_verbs.h | 21 +++++++++++++++++-- include/rdma/ib_verbs.h | 11 ++++++++++ 6 files changed, 76 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h index b3ea958..e69b04c 100644 --- a/drivers/infiniband/core/uverbs.h +++ b/drivers/infiniband/core/uverbs.h @@ -194,5 +194,6 @@ IB_UVERBS_DECLARE_CMD(create_srq); IB_UVERBS_DECLARE_CMD(modify_srq); IB_UVERBS_DECLARE_CMD(query_srq); IB_UVERBS_DECLARE_CMD(destroy_srq); +IB_UVERBS_DECLARE_CMD(get_mac); #endif /* UVERBS_H */ diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c index 56feab6..0f1821d 100644 --- a/drivers/infiniband/core/uverbs_cmd.c +++ b/drivers/infiniband/core/uverbs_cmd.c @@ -452,6 +452,7 @@ ssize_t ib_uverbs_query_port(struct ib_uverbs_file *file, resp.active_width = attr.active_width; resp.active_speed = attr.active_speed; resp.phys_state = attr.phys_state; + resp.link_type = attr.link_type; if (copy_to_user((void __user *) (unsigned long) cmd.response, &resp, sizeof resp)) @@ -1824,6 +1825,39 @@ err: return ret; } +ssize_t ib_uverbs_get_mac(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_get_mac cmd; + struct ib_uverbs_get_mac_resp resp; + int ret; + struct ib_pd *pd; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + pd = idr_read_pd(cmd.pd_handle, file->ucontext); + if (!pd) + return -EINVAL; + + ret = ib_get_mac(pd->device, cmd.port, cmd.gid, resp.mac); + put_pd_read(pd); + if (!ret) { + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + return -EFAULT; + } + return in_len; + } + + + return ret; +} + ssize_t ib_uverbs_destroy_ah(struct ib_uverbs_file *file, const char __user *buf, int in_len, int out_len) { diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c index eb36a81..b2f148f 100644 --- a/drivers/infiniband/core/uverbs_main.c +++ b/drivers/infiniband/core/uverbs_main.c @@ -108,6 +108,7 @@ static ssize_t (*uverbs_cmd_table[])(struct ib_uverbs_file *file, [IB_USER_VERBS_CMD_MODIFY_SRQ] = ib_uverbs_modify_srq, [IB_USER_VERBS_CMD_QUERY_SRQ] = ib_uverbs_query_srq, [IB_USER_VERBS_CMD_DESTROY_SRQ] = ib_uverbs_destroy_srq, + [IB_USER_VERBS_CMD_GET_MAC] = ib_uverbs_get_mac }; static struct vfsmount *uverbs_event_mnt; diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 18631b1..e3600c7 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -911,3 +911,14 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid) return qp->device->detach_mcast(qp, gid, lid); } EXPORT_SYMBOL(ib_detach_mcast); + + +int ib_get_mac(struct ib_device *device, u8 port, u8 *gid, u8 *mac) +{ + if (!device->get_mac) + return -ENOSYS; + + return device->get_mac(device, port, gid, mac); +} +EXPORT_SYMBOL(ib_get_mac); + diff --git a/include/rdma/ib_user_verbs.h b/include/rdma/ib_user_verbs.h index a17f771..5bfa2e6 100644 --- a/include/rdma/ib_user_verbs.h +++ b/include/rdma/ib_user_verbs.h @@ -81,7 +81,8 @@ enum { IB_USER_VERBS_CMD_MODIFY_SRQ, IB_USER_VERBS_CMD_QUERY_SRQ, IB_USER_VERBS_CMD_DESTROY_SRQ, - IB_USER_VERBS_CMD_POST_SRQ_RECV + IB_USER_VERBS_CMD_POST_SRQ_RECV, + IB_USER_VERBS_CMD_GET_MAC }; /* @@ -204,8 +205,9 @@ struct ib_uverbs_query_port_resp { __u8 init_type_reply; __u8 active_width; __u8 active_speed; - __u8 phys_state; - __u8 reserved[3]; + __u8 phys_state ; + __u8 link_type; + __u8 reserved[2]; }; struct ib_uverbs_alloc_pd { @@ -621,6 +623,19 @@ struct ib_uverbs_destroy_ah { __u32 ah_handle; }; +struct ib_uverbs_get_mac { + __u64 response; + __u32 pd_handle; + __u8 port; + __u8 reserved[3]; + __u8 gid[16]; +}; + +struct ib_uverbs_get_mac_resp { + __u8 mac[6]; + __u16 reserved; +}; + struct ib_uverbs_attach_mcast { __u8 gid[16]; __u32 qp_handle; diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 8f98c8f..b6d8aea 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1138,6 +1138,8 @@ struct ib_device { struct ib_grh *in_grh, struct ib_mad *in_mad, struct ib_mad *out_mad); + int (*get_mac)(struct ib_device *device, u8 port, + u8 *gid, u8 *mac); struct ib_dma_mapping_ops *dma_ops; @@ -2042,4 +2044,13 @@ int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid); */ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid); + +/** + * ib_get_mac - get the mac address for the specified gid + * @device: device used for traffic + * @gid: port number used. + * @lid: mac of the port bearing this gid + */ +int ib_get_mac(struct ib_device *device, u8 port, u8 *gid, u8 *mac); + #endif /* IB_VERBS_H */ -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:34:42 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:34:42 +0300 Subject: [ofa-general] [PATCH 3/9] ib_core: RDMAoE support only QP1 Message-ID: <20090615073442.GD27543@mtls03> Since RDMAoE is using Ethernet there is no need for QP0. This patch will create only QP1 for RDMAoE ports. Signed-off-by: Eli Cohen --- drivers/infiniband/core/agent.c | 16 ++++++++----- drivers/infiniband/core/mad.c | 48 ++++++++++++++++++++++++++++++--------- 2 files changed, 47 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/core/agent.c b/drivers/infiniband/core/agent.c index ae7c288..658a278 100644 --- a/drivers/infiniband/core/agent.c +++ b/drivers/infiniband/core/agent.c @@ -46,8 +46,10 @@ #define SPFX "ib_agent: " struct ib_agent_port_private { - struct list_head port_list; - struct ib_mad_agent *agent[2]; + struct list_head port_list; + struct ib_mad_agent *agent[2]; + struct ib_device *device; + u8 port_num; }; static DEFINE_SPINLOCK(ib_agent_port_list_lock); @@ -58,11 +60,10 @@ __ib_get_agent_port(struct ib_device *device, int port_num) { struct ib_agent_port_private *entry; - list_for_each_entry(entry, &ib_agent_port_list, port_list) { - if (entry->agent[0]->device == device && - entry->agent[0]->port_num == port_num) + list_for_each_entry(entry, &ib_agent_port_list, port_list) + if (entry->device == device && entry->port_num == port_num) return entry; - } + return NULL; } @@ -175,6 +176,9 @@ int ib_agent_port_open(struct ib_device *device, int port_num) goto error3; } + port_priv->device = device; + port_priv->port_num = port_num; + spin_lock_irqsave(&ib_agent_port_list_lock, flags); list_add_tail(&port_priv->port_list, &ib_agent_port_list); spin_unlock_irqrestore(&ib_agent_port_list_lock, flags); diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c index de922a0..aadf396 100644 --- a/drivers/infiniband/core/mad.c +++ b/drivers/infiniband/core/mad.c @@ -199,6 +199,16 @@ struct ib_mad_agent *ib_register_mad_agent(struct ib_device *device, unsigned long flags; u8 mgmt_class, vclass; + /* Validate device and port */ + port_priv = ib_get_mad_port(device, port_num); + if (!port_priv) { + ret = ERR_PTR(-ENODEV); + goto error1; + } + + if (!port_priv->qp_info[qp_type].qp) + return NULL; + /* Validate parameters */ qpn = get_spl_qp_index(qp_type); if (qpn == -1) @@ -260,13 +270,6 @@ struct ib_mad_agent *ib_register_mad_agent(struct ib_device *device, goto error1; } - /* Validate device and port */ - port_priv = ib_get_mad_port(device, port_num); - if (!port_priv) { - ret = ERR_PTR(-ENODEV); - goto error1; - } - /* Allocate structures */ mad_agent_priv = kzalloc(sizeof *mad_agent_priv, GFP_KERNEL); if (!mad_agent_priv) { @@ -556,6 +559,9 @@ int ib_unregister_mad_agent(struct ib_mad_agent *mad_agent) struct ib_mad_agent_private *mad_agent_priv; struct ib_mad_snoop_private *mad_snoop_priv; + if (!mad_agent) + return 0; + /* If the TID is zero, the agent can only snoop. */ if (mad_agent->hi_tid) { mad_agent_priv = container_of(mad_agent, @@ -2602,6 +2608,9 @@ static void cleanup_recv_queue(struct ib_mad_qp_info *qp_info) struct ib_mad_private *recv; struct ib_mad_list_head *mad_list; + if (!qp_info->qp) + return; + while (!list_empty(&qp_info->recv_queue.list)) { mad_list = list_entry(qp_info->recv_queue.list.next, @@ -2643,6 +2652,9 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) for (i = 0; i < IB_MAD_QPS_CORE; i++) { qp = port_priv->qp_info[i].qp; + if (!qp) + continue; + /* * PKey index for QP1 is irrelevant but * one is needed for the Reset to Init transition @@ -2684,6 +2696,9 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) } for (i = 0; i < IB_MAD_QPS_CORE; i++) { + if (!port_priv->qp_info[i].qp) + continue; + ret = ib_mad_post_receive_mads(&port_priv->qp_info[i], NULL); if (ret) { printk(KERN_ERR PFX "Couldn't post receive WRs\n"); @@ -2762,6 +2777,9 @@ error: static void destroy_mad_qp(struct ib_mad_qp_info *qp_info) { + if (!qp_info->qp) + return; + ib_destroy_qp(qp_info->qp); kfree(qp_info->snoop_table); } @@ -2777,6 +2795,7 @@ static int ib_mad_port_open(struct ib_device *device, struct ib_mad_port_private *port_priv; unsigned long flags; char name[sizeof "ib_mad123"]; + int has_smi; /* Create new device info */ port_priv = kzalloc(sizeof *port_priv, GFP_KERNEL); @@ -2792,7 +2811,11 @@ static int ib_mad_port_open(struct ib_device *device, init_mad_qp(port_priv, &port_priv->qp_info[0]); init_mad_qp(port_priv, &port_priv->qp_info[1]); + has_smi = ib_get_port_link_type(device, port_num) == PORT_LINK_IB; cq_size = (IB_MAD_QP_SEND_SIZE + IB_MAD_QP_RECV_SIZE) * 2; + if (has_smi) + cq_size *= 2; + port_priv->cq = ib_create_cq(port_priv->device, ib_mad_thread_completion_handler, NULL, port_priv, cq_size, 0); @@ -2816,9 +2839,11 @@ static int ib_mad_port_open(struct ib_device *device, goto error5; } - ret = create_mad_qp(&port_priv->qp_info[0], IB_QPT_SMI); - if (ret) - goto error6; + if (has_smi) { + ret = create_mad_qp(&port_priv->qp_info[0], IB_QPT_SMI); + if (ret) + goto error6; + } ret = create_mad_qp(&port_priv->qp_info[1], IB_QPT_GSI); if (ret) goto error7; @@ -2852,7 +2877,8 @@ error9: error8: destroy_mad_qp(&port_priv->qp_info[1]); error7: - destroy_mad_qp(&port_priv->qp_info[0]); + if (has_smi) + destroy_mad_qp(&port_priv->qp_info[0]); error6: ib_dereg_mr(port_priv->mr); error5: -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:34:50 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:34:50 +0300 Subject: [ofa-general] [PATCH 4/9] ib_core: Add RDMAoE SA support Message-ID: <20090615073450.GE27543@mtls03> Add support for resolving paths and joining multicast group for RDMAoE ports. The Ethernet specific code will complete immediately but will call the callback from a workqueue context to avoid deadloks. Signed-off-by: Eli Cohen --- drivers/infiniband/core/multicast.c | 153 ++++++++++++++++++++++++++++---- drivers/infiniband/core/sa_query.c | 167 +++++++++++++++++++++++++++------- 2 files changed, 269 insertions(+), 51 deletions(-) diff --git a/drivers/infiniband/core/multicast.c b/drivers/infiniband/core/multicast.c index 107f170..2417f6b 100644 --- a/drivers/infiniband/core/multicast.c +++ b/drivers/infiniband/core/multicast.c @@ -488,6 +488,36 @@ retest: } } +struct eth_work { + struct work_struct work; + struct mcast_member *member; + struct ib_device *device; + u8 port_num; +}; + +static void eth_mcast_work_handler(struct work_struct *work) +{ + struct eth_work *w = container_of(work, struct eth_work, work); + int err; + struct ib_port_attr port_attr; + int status = 0; + + err = ib_query_port(w->device, w->port_num, &port_attr); + if (err) + status = err; + else if (port_attr.state != IB_PORT_ACTIVE) + status = -EAGAIN; + + w->member->multicast.rec.qkey = cpu_to_be32(0xc2c); + atomic_inc(&w->member->refcount); + err = w->member->multicast.callback(status, &w->member->multicast); + deref_member(w->member); + if (err) + ib_sa_free_multicast(&w->member->multicast); + + kfree(w); +} + /* * Fail a join request if it is still active - at the head of the pending queue. */ @@ -586,21 +616,14 @@ found: return group; } -/* - * We serialize all join requests to a single group to make our lives much - * easier. Otherwise, two users could try to join the same group - * simultaneously, with different configurations, one could leave while the - * join is in progress, etc., which makes locking around error recovery - * difficult. - */ -struct ib_sa_multicast * -ib_sa_join_multicast(struct ib_sa_client *client, - struct ib_device *device, u8 port_num, - struct ib_sa_mcmember_rec *rec, - ib_sa_comp_mask comp_mask, gfp_t gfp_mask, - int (*callback)(int status, - struct ib_sa_multicast *multicast), - void *context) +static struct ib_sa_multicast * +ib_join_multicast(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_mcmember_rec *rec, + ib_sa_comp_mask comp_mask, gfp_t gfp_mask, + int (*callback)(int status, + struct ib_sa_multicast *multicast), + void *context) { struct mcast_device *dev; struct mcast_member *member; @@ -647,9 +670,81 @@ err: kfree(member); return ERR_PTR(ret); } + +static struct ib_sa_multicast * +eth_join_multicast(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_mcmember_rec *rec, + ib_sa_comp_mask comp_mask, gfp_t gfp_mask, + int (*callback)(int status, + struct ib_sa_multicast *multicast), + void *context) +{ + struct mcast_device *dev; + struct eth_work *w; + struct mcast_member *member; + int err; + + dev = ib_get_client_data(device, &mcast_client); + if (!dev) + return ERR_PTR(-ENODEV); + + member = kzalloc(sizeof *member, gfp_mask); + if (!member) + return ERR_PTR(-ENOMEM); + + w = kzalloc(sizeof *w, gfp_mask); + if (!w) { + err = -ENOMEM; + goto out1; + } + w->member = member; + w->device = device; + w->port_num = port_num; + + member->multicast.context = context; + member->multicast.callback = callback; + member->client = client; + member->multicast.rec.mgid = rec->mgid; + init_completion(&member->comp); + atomic_set(&member->refcount, 1); + + ib_sa_client_get(client); + INIT_WORK(&w->work, eth_mcast_work_handler); + queue_work(mcast_wq, &w->work); + + return &member->multicast; + +out1: + kfree(member); + return ERR_PTR(err); +} + +/* + * We serialize all join requests to a single group to make our lives much + * easier. Otherwise, two users could try to join the same group + * simultaneously, with different configurations, one could leave while the + * join is in progress, etc., which makes locking around error recovery + * difficult. + */ +struct ib_sa_multicast * +ib_sa_join_multicast(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_mcmember_rec *rec, + ib_sa_comp_mask comp_mask, gfp_t gfp_mask, + int (*callback)(int status, + struct ib_sa_multicast *multicast), + void *context) +{ + return ib_get_port_link_type(device, port_num) == PORT_LINK_IB ? + ib_join_multicast(client, device, port_num, rec, comp_mask, + gfp_mask, callback, context) : + eth_join_multicast(client, device, port_num, rec, comp_mask, + gfp_mask, callback, context); +} EXPORT_SYMBOL(ib_sa_join_multicast); -void ib_sa_free_multicast(struct ib_sa_multicast *multicast) +static void ib_free_multicast(struct ib_sa_multicast *multicast) { struct mcast_member *member; struct mcast_group *group; @@ -678,6 +773,32 @@ void ib_sa_free_multicast(struct ib_sa_multicast *multicast) ib_sa_client_put(member->client); kfree(member); } + +static void eth_free_multicast(struct ib_sa_multicast *multicast) +{ + struct mcast_member *member; + + member = container_of(multicast, struct mcast_member, multicast); + + deref_member(member); + wait_for_completion(&member->comp); + ib_sa_client_put(member->client); + kfree(member); +} + +void ib_sa_free_multicast(struct ib_sa_multicast *multicast) +{ + struct mcast_member *member; + struct mcast_group *group; + + member = container_of(multicast, struct mcast_member, multicast); + group = member->group; + + if (!group) + eth_free_multicast(multicast); + else + ib_free_multicast(multicast); +} EXPORT_SYMBOL(ib_sa_free_multicast); int ib_sa_get_mcmember_rec(struct ib_device *device, u8 port_num, diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c index 1865049..7bf9b5c 100644 --- a/drivers/infiniband/core/sa_query.c +++ b/drivers/infiniband/core/sa_query.c @@ -45,6 +45,7 @@ #include #include +#include #include "sa.h" MODULE_AUTHOR("Roland Dreier"); @@ -92,6 +93,8 @@ struct ib_sa_path_query { void (*callback)(int, struct ib_sa_path_rec *, void *); void *context; struct ib_sa_query sa_query; + union ib_gid dgid; + union ib_gid sgid; }; struct ib_sa_mcmember_query { @@ -304,6 +307,9 @@ static const struct ib_field mcmember_rec_table[] = { .struct_size_bytes = sizeof ((struct ib_sa_service_rec *) 0)->field, \ .field_name = "sa_service_rec:" #field + +static struct rdma_addr_client addr_client; + static const struct ib_field service_rec_table[] = { { SERVICE_REC_FIELD(id), .offset_words = 0, @@ -363,6 +369,11 @@ static void update_sm_ah(struct work_struct *work) struct ib_port_attr port_attr; struct ib_ah_attr ah_attr; + + if (ib_get_port_link_type(port->agent->device, port->port_num) != + PORT_LINK_IB) + return; + if (ib_query_port(port->agent->device, port->port_num, &port_attr)) { printk(KERN_WARNING "Couldn't query port\n"); return; @@ -626,41 +637,16 @@ static void ib_sa_path_rec_release(struct ib_sa_query *sa_query) kfree(container_of(sa_query, struct ib_sa_path_query, sa_query)); } -/** - * ib_sa_path_rec_get - Start a Path get query - * @client:SA client - * @device:device to send query on - * @port_num: port number to send query on - * @rec:Path Record to send in query - * @comp_mask:component mask to send in query - * @timeout_ms:time to wait for response - * @gfp_mask:GFP mask to use for internal allocations - * @callback:function called when query completes, times out or is - * canceled - * @context:opaque user context passed to callback - * @sa_query:query context, used to cancel query - * - * Send a Path Record Get query to the SA to look up a path. The - * callback function will be called when the query completes (or - * fails); status is 0 for a successful response, -EINTR if the query - * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error - * occurred sending the query. The resp parameter of the callback is - * only valid if status is 0. - * - * If the return value of ib_sa_path_rec_get() is negative, it is an - * error code. Otherwise it is a query ID that can be used to cancel - * the query. - */ -int ib_sa_path_rec_get(struct ib_sa_client *client, - struct ib_device *device, u8 port_num, - struct ib_sa_path_rec *rec, - ib_sa_comp_mask comp_mask, - int timeout_ms, gfp_t gfp_mask, - void (*callback)(int status, - struct ib_sa_path_rec *resp, - void *context), - void *context, - struct ib_sa_query **sa_query) +static int ib_path_rec_get(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_path_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, gfp_t gfp_mask, + void (*callback)(int status, + struct ib_sa_path_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query) { struct ib_sa_path_query *query; struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); @@ -717,6 +703,114 @@ err1: kfree(query); return ret; } + +struct eth_work_container { + struct work_struct work; + struct ib_sa_path_query *query; +}; + +static void resolve_callback(struct work_struct *work) +{ + struct eth_work_container *eth = + container_of(work, struct eth_work_container, work); + struct ib_sa_path_query *query = eth->query; + struct ib_sa_path_rec res = {}; + + res.dgid = query->dgid; + res.sgid = query->sgid; + res.hop_limit = 2; /* TBD fix this */ + res.mtu = IB_MTU_1024; /* TBD fix me */ + query->callback(0, &res, query->context); + + ib_sa_client_put(query->sa_query.client); +} + +static int eth_path_rec_get(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_path_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, gfp_t gfp_mask, + void (*callback)(int status, + struct ib_sa_path_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query) +{ + struct ib_sa_path_query *query; + struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); + struct eth_work_container *eth; + + if (!sa_dev) + return -ENODEV; + + query = kzalloc(sizeof *query, gfp_mask); + if (!query) + return -ENOMEM; + + eth = kzalloc(sizeof *eth, gfp_mask); + if (!eth) { + kfree(query); + return -ENOMEM; + } + + ib_sa_client_get(client); + query->sa_query.client = client; + query->callback = callback; + query->context = context; + query->dgid = rec->dgid; + query->sgid = rec->sgid; + + *sa_query = &query->sa_query; + + eth->query = query; + INIT_WORK(ð->work, resolve_callback); + schedule_work(ð->work); + + return 0; +} + +/** + * ib_sa_path_rec_get - Start a Path get query + * @client:SA client + * @device:device to send query on + * @port_num: port number to send query on + * @rec:Path Record to send in query + * @comp_mask:component mask to send in query + * @timeout_ms:time to wait for response + * @gfp_mask:GFP mask to use for internal allocations + * @callback:function called when query completes, times out or is + * canceled + * @context:opaque user context passed to callback + * @sa_query:query context, used to cancel query + * + * Send a Path Record Get query to the SA to look up a path. The + * callback function will be called when the query completes (or + * fails); status is 0 for a successful response, -EINTR if the query + * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error + * occurred sending the query. The resp parameter of the callback is + * only valid if status is 0. + * + * If the return value of ib_sa_path_rec_get() is negative, it is an + * error code. Otherwise it is a query ID that can be used to cancel + * the query. + */ +int ib_sa_path_rec_get(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_path_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, gfp_t gfp_mask, + void (*callback)(int status, + struct ib_sa_path_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query) +{ + return ib_get_port_link_type(device, port_num) == PORT_LINK_IB ? + ib_path_rec_get(client, device, port_num, rec, comp_mask, + timeout_ms, gfp_mask, callback, context, sa_query) : + eth_path_rec_get(client, device, port_num, rec, comp_mask, + timeout_ms, gfp_mask, callback, context, sa_query); +} EXPORT_SYMBOL(ib_sa_path_rec_get); static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query, @@ -1094,6 +1188,8 @@ static int __init ib_sa_init(void) goto err2; } + rdma_addr_register_client(&addr_client); + return 0; err2: ib_unregister_client(&sa_client); @@ -1103,6 +1199,7 @@ err1: static void __exit ib_sa_cleanup(void) { + rdma_addr_unregister_client(&addr_client); mcast_cleanup(); ib_unregister_client(&sa_client); idr_destroy(&query_idr); -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:34:59 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:34:59 +0300 Subject: [ofa-general] [PATCH 5/9] ib_core: CMA device binding Message-ID: <20090615073459.GF27543@mtls03> Add support for RDMAoE device binding and IP --> GID resolution. Signed-off-by: Eli Cohen --- drivers/infiniband/core/addr.c | 20 ++++++++------- drivers/infiniband/core/cma.c | 39 ++++++++++++++++++++++++++---- include/rdma/ib_addr.h | 51 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c index ce511d8..440e613 100644 --- a/drivers/infiniband/core/addr.c +++ b/drivers/infiniband/core/addr.c @@ -64,7 +64,7 @@ struct addr_req { static void process_req(struct work_struct *work); -static DEFINE_MUTEX(lock); +static DEFINE_SPINLOCK(lock); static LIST_HEAD(req_list); static DECLARE_DELAYED_WORK(work, process_req); static struct workqueue_struct *addr_wq; @@ -163,7 +163,7 @@ static void queue_req(struct addr_req *req) { struct addr_req *temp_req; - mutex_lock(&lock); + spin_lock(&lock); list_for_each_entry_reverse(temp_req, &req_list, list) { if (time_after_eq(req->timeout, temp_req->timeout)) break; @@ -173,7 +173,7 @@ static void queue_req(struct addr_req *req) if (req_list.next == &req->list) set_timeout(req->timeout); - mutex_unlock(&lock); + spin_unlock(&lock); } static void addr_send_arp(struct sockaddr *dst_in) @@ -207,7 +207,9 @@ static void addr_send_arp(struct sockaddr *dst_in) if (!dst) return; - neigh_event_send(dst->neighbour, NULL); + if (dst->neighbour) + neigh_event_send(dst->neighbour, NULL); + dst_release(dst); break; } @@ -322,7 +324,7 @@ static void process_req(struct work_struct *work) INIT_LIST_HEAD(&done_list); - mutex_lock(&lock); + spin_lock(&lock); list_for_each_entry_safe(req, temp_req, &req_list, list) { if (req->status == -ENODATA) { src_in = (struct sockaddr *) &req->src_addr; @@ -341,7 +343,7 @@ static void process_req(struct work_struct *work) req = list_entry(req_list.next, struct addr_req, list); set_timeout(req->timeout); } - mutex_unlock(&lock); + spin_unlock(&lock); list_for_each_entry_safe(req, temp_req, &done_list, list) { list_del(&req->list); @@ -439,7 +441,7 @@ int rdma_resolve_ip(struct rdma_addr_client *client, struct addr_req *req; int ret = 0; - req = kzalloc(sizeof *req, GFP_KERNEL); + req = kzalloc(sizeof *req, GFP_ATOMIC); if (!req) return -ENOMEM; @@ -483,7 +485,7 @@ void rdma_addr_cancel(struct rdma_dev_addr *addr) { struct addr_req *req, *temp_req; - mutex_lock(&lock); + spin_lock(&lock); list_for_each_entry_safe(req, temp_req, &req_list, list) { if (req->addr == addr) { req->status = -ECANCELED; @@ -493,7 +495,7 @@ void rdma_addr_cancel(struct rdma_dev_addr *addr) break; } } - mutex_unlock(&lock); + spin_unlock(&lock); } EXPORT_SYMBOL(rdma_addr_cancel); diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 851de83..0f15d9d 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -327,22 +327,30 @@ static int cma_acquire_dev(struct rdma_id_private *id_priv) { struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr; struct cma_device *cma_dev; - union ib_gid gid; + union ib_gid gid, llgid, *tmp; int ret = -ENODEV; switch (rdma_node_get_transport(dev_addr->dev_type)) { case RDMA_TRANSPORT_IB: ib_addr_get_sgid(dev_addr, &gid); + tmp = &gid; break; case RDMA_TRANSPORT_IWARP: iw_addr_get_sgid(dev_addr, &gid); + rdma_mac_to_ll_addr(dev_addr->src_dev_addr, &llgid); break; default: return -ENODEV; } list_for_each_entry(cma_dev, &dev_list, list) { - ret = ib_find_cached_gid(cma_dev->device, &gid, + if (ib_get_port_link_type(cma_dev->device, id_priv->id.port_num) + == PORT_LINK_ETH && + rdma_node_get_transport(dev_addr->dev_type) == + RDMA_TRANSPORT_IWARP) + tmp = &llgid; + + ret = ib_find_cached_gid(cma_dev->device, tmp, &id_priv->id.port_num, NULL); if (!ret) { cma_attach_to_dev(id_priv, cma_dev); @@ -1032,7 +1040,19 @@ static struct rdma_id_private *cma_new_conn_id(struct rdma_cm_id *listen_id, if (rt->num_paths == 2) rt->path_rec[1] = *ib_event->param.req_rcvd.alternate_path; - ib_addr_set_dgid(&rt->addr.dev_addr, &rt->path_rec[0].dgid); + + switch (ib_get_port_link_type(listen_id->device, + ib_event->param.req_rcvd.port)) { + case PORT_LINK_IB: + ib_addr_set_dgid(&rt->addr.dev_addr, &rt->path_rec[0].dgid); + break; + case PORT_LINK_ETH: + iw_addr_set_dgid(&rt->addr.dev_addr, &rt->path_rec[0].dgid); + break; + default: + printk(KERN_ERR "RDMA CMA: unknown rdma port link type\n"); + goto destroy_id; + } ret = rdma_translate_ip((struct sockaddr *) &id->route.addr.src_addr, &id->route.addr.dev_addr); if (ret) @@ -1563,10 +1583,19 @@ static int cma_query_ib_route(struct rdma_id_private *id_priv, int timeout_ms, struct ib_sa_path_rec path_rec; ib_sa_comp_mask comp_mask; struct sockaddr_in6 *sin6; + enum ib_port_link_type lt; + lt = ib_get_port_link_type(id_priv->id.device, id_priv->id.port_num); memset(&path_rec, 0, sizeof path_rec); - ib_addr_get_sgid(&addr->dev_addr, &path_rec.sgid); - ib_addr_get_dgid(&addr->dev_addr, &path_rec.dgid); + if (lt == PORT_LINK_IB || + (ib_addr_hw_addr_is_gid(addr->dev_addr.src_dev_addr) && + ib_addr_hw_addr_is_gid(addr->dev_addr.dst_dev_addr))) { + ib_addr_get_sgid(&addr->dev_addr, &path_rec.sgid); + ib_addr_get_dgid(&addr->dev_addr, &path_rec.dgid); + } else { + rdma_mac_to_ll_addr(addr->dev_addr.src_dev_addr, &path_rec.sgid); + rdma_mac_to_ll_addr(addr->dev_addr.dst_dev_addr, &path_rec.dgid); + } path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(&addr->dev_addr)); path_rec.numb_path = 1; path_rec.reversible = 1; diff --git a/include/rdma/ib_addr.h b/include/rdma/ib_addr.h index 483057b..9636a97 100644 --- a/include/rdma/ib_addr.h +++ b/include/rdma/ib_addr.h @@ -151,10 +151,61 @@ static inline void iw_addr_get_sgid(struct rdma_dev_addr *dev_addr, memcpy(gid, dev_addr->src_dev_addr, sizeof *gid); } +static inline void iw_addr_set_dgid(struct rdma_dev_addr *dev_addr, + union ib_gid *gid) +{ + memcpy(dev_addr->dst_dev_addr, gid, sizeof *gid); +} + static inline void iw_addr_get_dgid(struct rdma_dev_addr *dev_addr, union ib_gid *gid) { memcpy(gid, dev_addr->dst_dev_addr, sizeof *gid); } +static inline int rdma_link_local_addr(struct in6_addr *addr) +{ + if (addr->s6_addr32[0] == cpu_to_be32(0xfe800000) && + addr->s6_addr32[1] == 0) + return 1; + else + return 0; +} + +static inline void rdma_mac_to_ll_addr(u8 *mac, union ib_gid *gid) +{ + memset(gid->raw, 0, 16); + *((u32 *)gid->raw) = cpu_to_be32(0xfe800000); + gid->raw[12] = 0xfe; + gid->raw[11] = 0xff; + memcpy(gid->raw + 13, mac + 3, 3); + memcpy(gid->raw + 8, mac, 3); + gid->raw[8] ^= 2; +} + +static inline int rdma_is_multicast_addr(struct in6_addr *addr) +{ + return addr->s6_addr[0] = 0xff ? 1 : 0; +} + +static inline void rdma_get_ll_mac(struct in6_addr *addr, u8 *mac) +{ + memcpy(mac, &addr->s6_addr[8], 3); + memcpy(mac + 3, &addr->s6_addr[13], 3); + mac[0] ^= 2; +} + +static inline void rdma_get_mcast_mac(struct in6_addr *addr, u8 *mac) +{ + int i; + + for (i = 0; i < 6; ++i) + mac[i] = 0xff; +} + + static inline int ib_addr_hw_addr_is_gid(u8 *addr) + { + return be16_to_cpu(*(u16 *)(addr + 4)) == 0xfe80; + } + #endif /* IB_ADDR_H */ -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:35:08 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:35:08 +0300 Subject: [ofa-general] [PATCH 6/9] ib_core: RDMAoE UD packet packing support Message-ID: <20090615073508.GG27543@mtls03> Add support functions to aid in packing RDMAoE packets. Signed-off-by: Eli Cohen --- drivers/infiniband/core/ud_header.c | 111 +++++++++++++++++++++++++++++++++++ include/rdma/ib_pack.h | 26 ++++++++ 2 files changed, 137 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/core/ud_header.c b/drivers/infiniband/core/ud_header.c index 8ec7876..d04b6f2 100644 --- a/drivers/infiniband/core/ud_header.c +++ b/drivers/infiniband/core/ud_header.c @@ -80,6 +80,29 @@ static const struct ib_field lrh_table[] = { .size_bits = 16 } }; +static const struct ib_field eth_table[] = { + { STRUCT_FIELD(eth, dmac_h), + .offset_words = 0, + .offset_bits = 0, + .size_bits = 32 }, + { STRUCT_FIELD(eth, dmac_l), + .offset_words = 1, + .offset_bits = 0, + .size_bits = 16 }, + { STRUCT_FIELD(eth, smac_h), + .offset_words = 1, + .offset_bits = 16, + .size_bits = 16 }, + { STRUCT_FIELD(eth, smac_l), + .offset_words = 2, + .offset_bits = 0, + .size_bits = 32 }, + { STRUCT_FIELD(eth, type), + .offset_words = 3, + .offset_bits = 0, + .size_bits = 16 } +}; + static const struct ib_field grh_table[] = { { STRUCT_FIELD(grh, ip_version), .offset_words = 0, @@ -241,6 +264,53 @@ void ib_ud_header_init(int payload_bytes, EXPORT_SYMBOL(ib_ud_header_init); /** + * ib_rdmaoe_ud_header_init - Initialize UD header structure + * @payload_bytes:Length of packet payload + * @grh_present:GRH flag (if non-zero, GRH will be included) + * @header:Structure to initialize + * + * ib_rdmaoe_ud_header_init() initializes the grh.ip_version, grh.payload_length, + * grh.next_header, bth.opcode, bth.pad_count and + * bth.transport_header_version fields of a &struct eth_ud_header given + * the payload length and whether a GRH will be included. + */ +void ib_rdmaoe_ud_header_init(int payload_bytes, + int grh_present, + struct eth_ud_header *header) +{ + int header_len; + + memset(header, 0, sizeof *header); + + header_len = + sizeof header->eth + + IB_BTH_BYTES + + IB_DETH_BYTES; + if (grh_present) + header_len += IB_GRH_BYTES; + + header->grh_present = grh_present; + if (grh_present) { + header->grh.ip_version = 6; + header->grh.payload_length = + cpu_to_be16((IB_BTH_BYTES + + IB_DETH_BYTES + + payload_bytes + + 4 + /* ICRC */ + 3) & ~3); /* round up */ + header->grh.next_header = 0x1b; + } + + if (header->immediate_present) + header->bth.opcode = IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE; + else + header->bth.opcode = IB_OPCODE_UD_SEND_ONLY; + header->bth.pad_count = (4 - payload_bytes) & 3; + header->bth.transport_header_version = 0; +} +EXPORT_SYMBOL(ib_rdmaoe_ud_header_init); + +/** * ib_ud_header_pack - Pack UD header struct into wire format * @header:UD header struct * @buf:Buffer to pack into @@ -281,6 +351,47 @@ int ib_ud_header_pack(struct ib_ud_header *header, EXPORT_SYMBOL(ib_ud_header_pack); /** + * rdmaoe_ud_header_pack - Pack UD header struct into eth wire format + * @header:UD header struct + * @buf:Buffer to pack into + * + * ib_ud_header_pack() packs the UD header structure @header into wire + * format in the buffer @buf. + */ +int rdmaoe_ud_header_pack(struct eth_ud_header *header, + void *buf) +{ + int len = 0; + + ib_pack(eth_table, ARRAY_SIZE(eth_table), + &header->eth, buf); + len += IB_ETH_BYTES; + + if (header->grh_present) { + ib_pack(grh_table, ARRAY_SIZE(grh_table), + &header->grh, buf + len); + len += IB_GRH_BYTES; + } + + ib_pack(bth_table, ARRAY_SIZE(bth_table), + &header->bth, buf + len); + len += IB_BTH_BYTES; + + ib_pack(deth_table, ARRAY_SIZE(deth_table), + &header->deth, buf + len); + len += IB_DETH_BYTES; + + if (header->immediate_present) { + memcpy(buf + len, &header->immediate_data, + sizeof header->immediate_data); + len += sizeof header->immediate_data; + } + + return len; +} +EXPORT_SYMBOL(rdmaoe_ud_header_pack); + +/** * ib_ud_header_unpack - Unpack UD header struct from wire format * @header:UD header struct * @buf:Buffer to pack into diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h index d7fc45c..bf199eb 100644 --- a/include/rdma/ib_pack.h +++ b/include/rdma/ib_pack.h @@ -37,6 +37,7 @@ enum { IB_LRH_BYTES = 8, + IB_ETH_BYTES = 14, IB_GRH_BYTES = 40, IB_BTH_BYTES = 12, IB_DETH_BYTES = 8 @@ -210,6 +211,14 @@ struct ib_unpacked_deth { __be32 source_qpn; }; +struct ib_unpacked_eth { + u8 dmac_h[4]; + u8 dmac_l[2]; + u8 smac_h[2]; + u8 smac_l[4]; + __be16 type; +}; + struct ib_ud_header { struct ib_unpacked_lrh lrh; int grh_present; @@ -220,6 +229,16 @@ struct ib_ud_header { __be32 immediate_data; }; +struct eth_ud_header { + struct ib_unpacked_eth eth; + int grh_present; + struct ib_unpacked_grh grh; + struct ib_unpacked_bth bth; + struct ib_unpacked_deth deth; + int immediate_present; + __be32 immediate_data; +}; + void ib_pack(const struct ib_field *desc, int desc_len, void *structure, @@ -234,10 +253,17 @@ void ib_ud_header_init(int payload_bytes, int grh_present, struct ib_ud_header *header); +void ib_rdmaoe_ud_header_init(int payload_bytes, + int grh_present, + struct eth_ud_header *header); + int ib_ud_header_pack(struct ib_ud_header *header, void *buf); int ib_ud_header_unpack(void *buf, struct ib_ud_header *header); +int rdmaoe_ud_header_pack(struct eth_ud_header *header, + void *buf); + #endif /* IB_PACK_H */ -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:35:28 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:35:28 +0300 Subject: [ofa-general] [PATCH 8/9] mlx4: Add support for RDMAoE - address resolution Message-ID: <20090615073528.GI27543@mtls03> The following path handles address vectors creation for RDMAoE ports. mlx4 needs the MAC address of the remote node to include it in the WQE of a UD QP or in the QP context of connected QPs. Address resolution is done atomically in the case of a link local address or using service from core/addr.c to do that. mlx4 transport packets were changed too to accomodate for RDMAoE. Signed-off-by: Eli Cohen --- drivers/infiniband/hw/mlx4/ah.c | 228 ++++++++++++++++++++++++++----- drivers/infiniband/hw/mlx4/mlx4_ib.h | 30 ++++- drivers/infiniband/hw/mlx4/qp.c | 253 +++++++++++++++++++++++++++------- include/linux/mlx4/device.h | 31 ++++- include/linux/mlx4/qp.h | 8 +- 5 files changed, 463 insertions(+), 87 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/ah.c b/drivers/infiniband/hw/mlx4/ah.c index c75ac94..c994e1f 100644 --- a/drivers/infiniband/hw/mlx4/ah.c +++ b/drivers/infiniband/hw/mlx4/ah.c @@ -31,63 +31,200 @@ */ #include "mlx4_ib.h" +#include +#include +#include -struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr) +static struct rdma_addr_client addr_client; + +struct resolve_ctx { + struct completion done; + int status; +}; + + +static int status2err(int status) { - struct mlx4_dev *dev = to_mdev(pd->device)->dev; - struct mlx4_ib_ah *ah; + return status; /* TBD */ +} - ah = kmalloc(sizeof *ah, GFP_ATOMIC); - if (!ah) - return ERR_PTR(-ENOMEM); +static void resolve_callback(int status, struct sockaddr *src_addr, + struct rdma_dev_addr *addr, void *context) +{ + struct resolve_ctx *ctx = context; - memset(&ah->av, 0, sizeof ah->av); + ctx->status = status; + complete(&ctx->done); +} - ah->av.port_pd = cpu_to_be32(to_mpd(pd)->pdn | (ah_attr->port_num << 24)); - ah->av.g_slid = ah_attr->src_path_bits; - ah->av.dlid = cpu_to_be16(ah_attr->dlid); - if (ah_attr->static_rate) { - ah->av.stat_rate = ah_attr->static_rate + MLX4_STAT_RATE_OFFSET; - while (ah->av.stat_rate > IB_RATE_2_5_GBPS + MLX4_STAT_RATE_OFFSET && - !(1 << ah->av.stat_rate & dev->caps.stat_rate_support)) - --ah->av.stat_rate; +int mlx4_ib_resolve_grh(struct mlx4_ib_dev *dev, const struct ib_ah_attr *ah_attr, + u8 *mac, int *is_mcast) +{ + struct mlx4_ib_rdmaoe *rdmaoe = &dev->rdmaoe; + struct sockaddr_in6 dst = {0}; + struct rdma_dev_addr addr; + struct resolve_ctx ctx; + struct net_device *netdev; + int err = 0; + int ifidx; + + *is_mcast = 0; + spin_lock(&rdmaoe->lock); + netdev = rdmaoe->netdevs[ah_attr->port_num - 1]; + if (!netdev) { + spin_unlock(&rdmaoe->lock); + return -EINVAL; + } + ifidx = netdev->ifindex; + spin_unlock(&rdmaoe->lock); + + init_completion(&ctx.done); + memcpy(dst.sin6_addr.s6_addr, ah_attr->grh.dgid.raw, sizeof ah_attr->grh); + dst.sin6_family = AF_INET6; + dst.sin6_scope_id = ifidx; + if (rdma_link_local_addr(&dst.sin6_addr)) + rdma_get_ll_mac(&dst.sin6_addr, mac); + else if (rdma_is_multicast_addr(&dst.sin6_addr)) { + rdma_get_mcast_mac(&dst.sin6_addr, mac); + *is_mcast = 1; + } else { + err = rdma_resolve_ip(&addr_client, NULL, (struct sockaddr *)&dst, &addr, + 2000, resolve_callback, &ctx); + if (!err) + wait_for_completion(&ctx.done); + else + ctx.status = err; + + err = status2err(ctx.status); } - ah->av.sl_tclass_flowlabel = cpu_to_be32(ah_attr->sl << 28); + + return err; +} + +static struct ib_ah *create_ib_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr, + struct mlx4_ib_ah *ah) +{ + struct mlx4_dev *dev = to_mdev(pd->device)->dev; + + ah->av.ib.port_pd = cpu_to_be32(to_mpd(pd)->pdn | (ah_attr->port_num << 24)); + ah->av.ib.g_slid = ah_attr->src_path_bits; if (ah_attr->ah_flags & IB_AH_GRH) { - ah->av.g_slid |= 0x80; - ah->av.gid_index = ah_attr->grh.sgid_index; - ah->av.hop_limit = ah_attr->grh.hop_limit; - ah->av.sl_tclass_flowlabel |= + ah->av.ib.g_slid |= 0x80; + ah->av.ib.gid_index = ah_attr->grh.sgid_index; + ah->av.ib.hop_limit = ah_attr->grh.hop_limit; + ah->av.ib.sl_tclass_flowlabel |= cpu_to_be32((ah_attr->grh.traffic_class << 20) | ah_attr->grh.flow_label); - memcpy(ah->av.dgid, ah_attr->grh.dgid.raw, 16); + memcpy(ah->av.ib.dgid, ah_attr->grh.dgid.raw, 16); } + ah->av.ib.dlid = cpu_to_be16(ah_attr->dlid); + if (ah_attr->static_rate) { + ah->av.ib.stat_rate = ah_attr->static_rate + MLX4_STAT_RATE_OFFSET; + while (ah->av.ib.stat_rate > IB_RATE_2_5_GBPS + MLX4_STAT_RATE_OFFSET && + !(1 << ah->av.ib.stat_rate & dev->caps.stat_rate_support)) + --ah->av.ib.stat_rate; + } + ah->av.ib.sl_tclass_flowlabel = cpu_to_be32(ah_attr->sl << 28); + return &ah->ibah; } +static struct ib_ah *create_rdmaoe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr, + struct mlx4_ib_ah *ah) +{ + struct mlx4_ib_dev *ibdev = to_mdev(pd->device); + struct mlx4_dev *dev = ibdev->dev; + u8 mac[6]; + int err; + int is_mcast; + + err = mlx4_ib_resolve_grh(ibdev, ah_attr, mac, &is_mcast); + if (err) + return ERR_PTR(err); + + memcpy(ah->av.eth.mac_0_1, mac, 2); + memcpy(ah->av.eth.mac_2_5, mac + 2, 4); + ah->av.ib.port_pd = cpu_to_be32(to_mpd(pd)->pdn | (ah_attr->port_num << 24)); + ah->av.ib.g_slid = 0x80; + if (ah_attr->static_rate) { + ah->av.ib.stat_rate = ah_attr->static_rate + MLX4_STAT_RATE_OFFSET; + while (ah->av.ib.stat_rate > IB_RATE_2_5_GBPS + MLX4_STAT_RATE_OFFSET && + !(1 << ah->av.ib.stat_rate & dev->caps.stat_rate_support)) + --ah->av.ib.stat_rate; + } + + /* + * HW requires multicast LID so we just choose one. + */ + if (is_mcast) + ah->av.ib.dlid = cpu_to_be16(0xc000); + + memcpy(ah->av.ib.dgid, ah_attr->grh.dgid.raw, 16); + ah->av.ib.sl_tclass_flowlabel = cpu_to_be32(ah_attr->sl << 28); + + return &ah->ibah; +} + +struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr) +{ + struct mlx4_ib_ah *ah; + enum ib_port_link_type link_type; + struct ib_ah *ret; + + ah = kzalloc(sizeof *ah, GFP_ATOMIC); + if (!ah) + return ERR_PTR(-ENOMEM); + + link_type = ib_get_port_link_type(pd->device, ah_attr->port_num); + if (link_type == PORT_LINK_ETH) { + if (!(ah_attr->ah_flags & IB_AH_GRH)) { + ret = ERR_PTR(-EINVAL); + goto out; + } else { + /* TBD: need to handle the case when we get called + in an atomic context and there we might sleep. We + don't expect this currently since we're working with + link local addresses which we can translate without + going to sleep */ + ret = create_rdmaoe_ah(pd, ah_attr, ah); + if (IS_ERR(ret)) + goto out; + else + return ret; + } + } else + return create_ib_ah(pd, ah_attr, ah); /* never fails */ + +out: + kfree(ah); + return ret; +} + int mlx4_ib_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr) { struct mlx4_ib_ah *ah = to_mah(ibah); + enum ib_port_link_type lt; + lt = ib_get_port_link_type(ibah->device, ah_attr->port_num); memset(ah_attr, 0, sizeof *ah_attr); - ah_attr->dlid = be16_to_cpu(ah->av.dlid); - ah_attr->sl = be32_to_cpu(ah->av.sl_tclass_flowlabel) >> 28; - ah_attr->port_num = be32_to_cpu(ah->av.port_pd) >> 24; - if (ah->av.stat_rate) - ah_attr->static_rate = ah->av.stat_rate - MLX4_STAT_RATE_OFFSET; - ah_attr->src_path_bits = ah->av.g_slid & 0x7F; + ah_attr->dlid = lt == PORT_LINK_IB ? be16_to_cpu(ah->av.ib.dlid) : 0; + ah_attr->sl = be32_to_cpu(ah->av.ib.sl_tclass_flowlabel) >> 28; + ah_attr->port_num = be32_to_cpu(ah->av.ib.port_pd) >> 24; + if (ah->av.ib.stat_rate) + ah_attr->static_rate = ah->av.ib.stat_rate - MLX4_STAT_RATE_OFFSET; + ah_attr->src_path_bits = ah->av.ib.g_slid & 0x7F; if (mlx4_ib_ah_grh_present(ah)) { ah_attr->ah_flags = IB_AH_GRH; ah_attr->grh.traffic_class = - be32_to_cpu(ah->av.sl_tclass_flowlabel) >> 20; + be32_to_cpu(ah->av.ib.sl_tclass_flowlabel) >> 20; ah_attr->grh.flow_label = - be32_to_cpu(ah->av.sl_tclass_flowlabel) & 0xfffff; - ah_attr->grh.hop_limit = ah->av.hop_limit; - ah_attr->grh.sgid_index = ah->av.gid_index; - memcpy(ah_attr->grh.dgid.raw, ah->av.dgid, 16); + be32_to_cpu(ah->av.ib.sl_tclass_flowlabel) & 0xfffff; + ah_attr->grh.hop_limit = ah->av.ib.hop_limit; + ah_attr->grh.sgid_index = ah->av.ib.gid_index; + memcpy(ah_attr->grh.dgid.raw, ah->av.ib.dgid, 16); } return 0; @@ -98,3 +235,30 @@ int mlx4_ib_destroy_ah(struct ib_ah *ah) kfree(to_mah(ah)); return 0; } + +int mlx4_ib_get_mac(struct ib_device *device, u8 port, u8 *gid, u8 *mac) +{ + int err; + struct mlx4_ib_dev *ibdev = to_mdev(device); + struct ib_ah_attr ah_attr = { + .port_num = port, + }; + int is_mcast; + + memcpy(ah_attr.grh.dgid.raw, gid, 16); + err = mlx4_ib_resolve_grh(ibdev, &ah_attr, mac, &is_mcast); + if (err) + ERR_PTR(err); + + return 0; +} + +void mlx4_ib_addr_init(void) +{ + rdma_addr_register_client(&addr_client); +} + +void mlx4_ib_addr_cleanup(void) +{ + rdma_addr_unregister_client(&addr_client); +} diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index 8a7dd67..e2d6e62 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -138,6 +138,7 @@ struct mlx4_ib_qp { u8 resp_depth; u8 sq_no_prefetch; u8 state; + int mlx_type; }; struct mlx4_ib_srq { @@ -157,7 +158,15 @@ struct mlx4_ib_srq { struct mlx4_ib_ah { struct ib_ah ibah; - struct mlx4_av av; + union mlx4_ext_av av; +}; + +struct mlx4_ib_rdmaoe { + spinlock_t lock; + struct net_device *netdevs[MLX4_MAX_PORTS]; + int enstate[MLX4_MAX_PORTS]; + int mtu[MLX4_MAX_PORTS]; + struct notifier_block nb; }; struct mlx4_ib_dev { @@ -175,6 +184,8 @@ struct mlx4_ib_dev { spinlock_t sm_lock; struct mutex cap_mask_mutex; + + struct mlx4_ib_rdmaoe rdmaoe; }; static inline struct mlx4_ib_dev *to_mdev(struct ib_device *ibdev) @@ -312,10 +323,25 @@ int mlx4_ib_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list, int npages, u64 iova); int mlx4_ib_unmap_fmr(struct list_head *fmr_list); int mlx4_ib_fmr_dealloc(struct ib_fmr *fmr); +void mlx4_ib_addr_init(void); +void mlx4_ib_addr_cleanup(void); + +int mlx4_ib_resolve_grh(struct mlx4_ib_dev *dev, const struct ib_ah_attr *ah_attr, + u8 *mac, int *is_mcast); + +int mlx4_ib_get_mac(struct ib_device *device, u8 port, u8 *gid, u8 *mac); static inline int mlx4_ib_ah_grh_present(struct mlx4_ib_ah *ah) { - return !!(ah->av.g_slid & 0x80); + /* + * port number is located at the same place for both IB and Eth + */ + u8 port = (ah->av.ib.port_pd >> 24) & 3; + + if (ib_get_port_link_type(ah->ibah.device, port) == PORT_LINK_ETH) + return 1; + else + return !!(ah->av.ib.g_slid & 0x80); } #endif /* MLX4_IB_H */ diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 20724ae..7a6b765 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -32,6 +32,7 @@ */ #include +#include #include #include @@ -47,14 +48,21 @@ enum { enum { MLX4_IB_DEFAULT_SCHED_QUEUE = 0x83, - MLX4_IB_DEFAULT_QP0_SCHED_QUEUE = 0x3f + MLX4_IB_DEFAULT_QP0_SCHED_QUEUE = 0x3f, + MLX4_IB_LINK_TYPE_IB = 0, + MLX4_IB_LINK_TYPE_ETH = 1 }; enum { /* * Largest possible UD header: send with GRH and immediate data. + * 4 bytes added to accommodate for eth header instead of lrh */ - MLX4_IB_UD_HEADER_SIZE = 72 + MLX4_IB_UD_HEADER_SIZE = 76 +}; + +enum { + MLX4_RDMAOE_ETHERTYPE = 0x8915 }; struct mlx4_ib_sqp { @@ -62,7 +70,10 @@ struct mlx4_ib_sqp { int pkey_index; u32 qkey; u32 send_psn; - struct ib_ud_header ud_header; + union { + struct ib_ud_header ib; + struct eth_ud_header eth; + } hdr; u8 header_buf[MLX4_IB_UD_HEADER_SIZE]; }; @@ -782,18 +793,6 @@ int mlx4_ib_destroy_qp(struct ib_qp *qp) return 0; } -static int to_mlx4_st(enum ib_qp_type type) -{ - switch (type) { - case IB_QPT_RC: return MLX4_QP_ST_RC; - case IB_QPT_UC: return MLX4_QP_ST_UC; - case IB_QPT_UD: return MLX4_QP_ST_UD; - case IB_QPT_SMI: - case IB_QPT_GSI: return MLX4_QP_ST_MLX; - default: return -1; - } -} - static __be32 to_mlx4_access_flags(struct mlx4_ib_qp *qp, const struct ib_qp_attr *attr, int attr_mask) { @@ -843,6 +842,12 @@ static void mlx4_set_sched(struct mlx4_qp_path *path, u8 port) static int mlx4_set_path(struct mlx4_ib_dev *dev, const struct ib_ah_attr *ah, struct mlx4_qp_path *path, u8 port) { + int err; + int is_eth = ib_get_port_link_type(&dev->ib_dev, port) == + PORT_LINK_ETH ? 1 : 0; + u8 mac[6]; + int is_mcast; + path->grh_mylmc = ah->src_path_bits & 0x7f; path->rlid = cpu_to_be16(ah->dlid); if (ah->static_rate) { @@ -873,9 +878,36 @@ static int mlx4_set_path(struct mlx4_ib_dev *dev, const struct ib_ah_attr *ah, path->sched_queue = MLX4_IB_DEFAULT_SCHED_QUEUE | ((port - 1) << 6) | ((ah->sl & 0xf) << 2); + if (is_eth) { + if (!(ah->ah_flags & IB_AH_GRH)) + return -1; + + err = mlx4_ib_resolve_grh(dev, ah, mac, &is_mcast); + if (err) + return err; + + memcpy(path->dmac_h, mac, 2); + memcpy(path->dmac_l, mac + 2, 4); + path->ackto = MLX4_IB_LINK_TYPE_ETH; + /* use index 0 into MAC table for RDMAoE */ + path->grh_mylmc &= 0x80; + } + return 0; } +static int to_mlx4_st(enum ib_qp_type type) +{ + switch (type) { + case IB_QPT_RC: return MLX4_QP_ST_RC; + case IB_QPT_UC: return MLX4_QP_ST_UC; + case IB_QPT_UD: return MLX4_QP_ST_UD; + case IB_QPT_SMI: + case IB_QPT_GSI: return MLX4_QP_ST_MLX; + default: return -1; + } +} + static int __mlx4_ib_modify_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr, int attr_mask, enum ib_qp_state cur_state, enum ib_qp_state new_state) @@ -972,7 +1004,7 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp, } if (attr_mask & IB_QP_TIMEOUT) { - context->pri_path.ackto = attr->timeout << 3; + context->pri_path.ackto |= (attr->timeout << 3); optpar |= MLX4_QP_OPTPAR_ACK_TIMEOUT; } @@ -1206,8 +1238,8 @@ out: return err; } -static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, - void *wqe, unsigned *mlx_seg_len) +static int build_ib_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, + void *wqe, unsigned *mlx_seg_len) { struct ib_device *ib_dev = &to_mdev(sqp->qp.ibqp.device)->ib_dev; struct mlx4_wqe_mlx_seg *mlx = wqe; @@ -1223,61 +1255,171 @@ static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, for (i = 0; i < wr->num_sge; ++i) send_size += wr->sg_list[i].length; - ib_ud_header_init(send_size, mlx4_ib_ah_grh_present(ah), &sqp->ud_header); + ib_ud_header_init(send_size, mlx4_ib_ah_grh_present(ah), &sqp->hdr.ib); - sqp->ud_header.lrh.service_level = - be32_to_cpu(ah->av.sl_tclass_flowlabel) >> 28; - sqp->ud_header.lrh.destination_lid = ah->av.dlid; - sqp->ud_header.lrh.source_lid = cpu_to_be16(ah->av.g_slid & 0x7f); + sqp->hdr.ib.lrh.service_level = + be32_to_cpu(ah->av.ib.sl_tclass_flowlabel) >> 28; + sqp->hdr.ib.lrh.destination_lid = ah->av.ib.dlid; + sqp->hdr.ib.lrh.source_lid = cpu_to_be16(ah->av.ib.g_slid & 0x7f); if (mlx4_ib_ah_grh_present(ah)) { - sqp->ud_header.grh.traffic_class = - (be32_to_cpu(ah->av.sl_tclass_flowlabel) >> 20) & 0xff; - sqp->ud_header.grh.flow_label = - ah->av.sl_tclass_flowlabel & cpu_to_be32(0xfffff); - sqp->ud_header.grh.hop_limit = ah->av.hop_limit; - ib_get_cached_gid(ib_dev, be32_to_cpu(ah->av.port_pd) >> 24, - ah->av.gid_index, &sqp->ud_header.grh.source_gid); - memcpy(sqp->ud_header.grh.destination_gid.raw, - ah->av.dgid, 16); + sqp->hdr.ib.grh.traffic_class = + (be32_to_cpu(ah->av.ib.sl_tclass_flowlabel) >> 20) & 0xff; + sqp->hdr.ib.grh.flow_label = + ah->av.ib.sl_tclass_flowlabel & cpu_to_be32(0xfffff); + sqp->hdr.ib.grh.hop_limit = ah->av.ib.hop_limit; + ib_get_cached_gid(ib_dev, be32_to_cpu(ah->av.ib.port_pd) >> 24, + ah->av.ib.gid_index, &sqp->hdr.ib.grh.source_gid); + memcpy(sqp->hdr.ib.grh.destination_gid.raw, + ah->av.ib.dgid, 16); } mlx->flags &= cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE); mlx->flags |= cpu_to_be32((!sqp->qp.ibqp.qp_num ? MLX4_WQE_MLX_VL15 : 0) | - (sqp->ud_header.lrh.destination_lid == + (sqp->hdr.ib.lrh.destination_lid == IB_LID_PERMISSIVE ? MLX4_WQE_MLX_SLR : 0) | - (sqp->ud_header.lrh.service_level << 8)); - mlx->rlid = sqp->ud_header.lrh.destination_lid; + (sqp->hdr.ib.lrh.service_level << 8)); + mlx->rlid = sqp->hdr.ib.lrh.destination_lid; + + switch (wr->opcode) { + case IB_WR_SEND: + sqp->hdr.ib.bth.opcode = IB_OPCODE_UD_SEND_ONLY; + sqp->hdr.ib.immediate_present = 0; + break; + case IB_WR_SEND_WITH_IMM: + sqp->hdr.ib.bth.opcode = IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE; + sqp->hdr.ib.immediate_present = 1; + sqp->hdr.ib.immediate_data = wr->ex.imm_data; + break; + default: + return -EINVAL; + } + + sqp->hdr.ib.lrh.virtual_lane = !sqp->qp.ibqp.qp_num ? 15 : 0; + if (sqp->hdr.ib.lrh.destination_lid == IB_LID_PERMISSIVE) + sqp->hdr.ib.lrh.source_lid = IB_LID_PERMISSIVE; + sqp->hdr.ib.bth.solicited_event = !!(wr->send_flags & IB_SEND_SOLICITED); + if (!sqp->qp.ibqp.qp_num) + ib_get_cached_pkey(ib_dev, sqp->qp.port, sqp->pkey_index, &pkey); + else + ib_get_cached_pkey(ib_dev, sqp->qp.port, wr->wr.ud.pkey_index, &pkey); + sqp->hdr.ib.bth.pkey = cpu_to_be16(pkey); + sqp->hdr.ib.bth.destination_qpn = cpu_to_be32(wr->wr.ud.remote_qpn); + sqp->hdr.ib.bth.psn = cpu_to_be32((sqp->send_psn++) & ((1 << 24) - 1)); + sqp->hdr.ib.deth.qkey = cpu_to_be32(wr->wr.ud.remote_qkey & 0x80000000 ? + sqp->qkey : wr->wr.ud.remote_qkey); + sqp->hdr.ib.deth.source_qpn = cpu_to_be32(sqp->qp.ibqp.qp_num); + + header_size = ib_ud_header_pack(&sqp->hdr.ib, sqp->header_buf); + + /* + * Inline data segments may not cross a 64 byte boundary. If + * our UD header is bigger than the space available up to the + * next 64 byte boundary in the WQE, use two inline data + * segments to hold the UD header. + */ + spc = MLX4_INLINE_ALIGN - + ((unsigned long) (inl + 1) & (MLX4_INLINE_ALIGN - 1)); + if (header_size <= spc) { + inl->byte_count = cpu_to_be32(1 << 31 | header_size); + memcpy(inl + 1, sqp->header_buf, header_size); + i = 1; + } else { + inl->byte_count = cpu_to_be32(1 << 31 | spc); + memcpy(inl + 1, sqp->header_buf, spc); + + inl = (void *) (inl + 1) + spc; + memcpy(inl + 1, sqp->header_buf + spc, header_size - spc); + /* + * Need a barrier here to make sure all the data is + * visible before the byte_count field is set. + * Otherwise the HCA prefetcher could grab the 64-byte + * chunk with this inline segment and get a valid (!= + * 0xffffffff) byte count but stale data, and end up + * generating a packet with bad headers. + * + * The first inline segment's byte_count field doesn't + * need a barrier, because it comes after a + * control/MLX segment and therefore is at an offset + * of 16 mod 64. + */ + wmb(); + inl->byte_count = cpu_to_be32(1 << 31 | (header_size - spc)); + i = 2; + } + + *mlx_seg_len = + ALIGN(i * sizeof (struct mlx4_wqe_inline_seg) + header_size, 16); + return 0; +} + +static int build_eth_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, + void *wqe, unsigned *mlx_seg_len) +{ + struct ib_device *ib_dev = &to_mdev(sqp->qp.ibqp.device)->ib_dev; + struct mlx4_wqe_mlx_seg *mlx = wqe; + struct mlx4_wqe_inline_seg *inl = wqe + sizeof *mlx; + struct mlx4_ib_ah *ah = to_mah(wr->wr.ud.ah); + u16 pkey; + int send_size; + int header_size; + int spc; + int i; + void *tmp; + + send_size = 0; + for (i = 0; i < wr->num_sge; ++i) + send_size += wr->sg_list[i].length; + + ib_rdmaoe_ud_header_init(send_size, mlx4_ib_ah_grh_present(ah), &sqp->hdr.eth); + + if (mlx4_ib_ah_grh_present(ah)) { + sqp->hdr.eth.grh.traffic_class = + (be32_to_cpu(ah->av.ib.sl_tclass_flowlabel) >> 20) & 0xff; + sqp->hdr.eth.grh.flow_label = + ah->av.ib.sl_tclass_flowlabel & cpu_to_be32(0xfffff); + sqp->hdr.eth.grh.hop_limit = ah->av.ib.hop_limit; + ib_get_cached_gid(ib_dev, be32_to_cpu(ah->av.ib.port_pd) >> 24, + ah->av.ib.gid_index, &sqp->hdr.eth.grh.source_gid); + memcpy(sqp->hdr.eth.grh.destination_gid.raw, + ah->av.ib.dgid, 16); + } + + mlx->flags &= cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE); switch (wr->opcode) { case IB_WR_SEND: - sqp->ud_header.bth.opcode = IB_OPCODE_UD_SEND_ONLY; - sqp->ud_header.immediate_present = 0; + sqp->hdr.eth.bth.opcode = IB_OPCODE_UD_SEND_ONLY; + sqp->hdr.eth.immediate_present = 0; break; case IB_WR_SEND_WITH_IMM: - sqp->ud_header.bth.opcode = IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE; - sqp->ud_header.immediate_present = 1; - sqp->ud_header.immediate_data = wr->ex.imm_data; + sqp->hdr.eth.bth.opcode = IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE; + sqp->hdr.eth.immediate_present = 1; + sqp->hdr.eth.immediate_data = wr->ex.imm_data; break; default: return -EINVAL; } - sqp->ud_header.lrh.virtual_lane = !sqp->qp.ibqp.qp_num ? 15 : 0; - if (sqp->ud_header.lrh.destination_lid == IB_LID_PERMISSIVE) - sqp->ud_header.lrh.source_lid = IB_LID_PERMISSIVE; - sqp->ud_header.bth.solicited_event = !!(wr->send_flags & IB_SEND_SOLICITED); + memcpy(sqp->hdr.eth.eth.dmac_h, ah->av.eth.mac_0_1, 2); + memcpy(sqp->hdr.eth.eth.dmac_h + 2, ah->av.eth.mac_2_5, 2); + memcpy(sqp->hdr.eth.eth.dmac_l, ah->av.eth.mac_2_5 + 2, 2); + tmp = to_mdev(sqp->qp.ibqp.device)->rdmaoe.netdevs[sqp->qp.port - 1]->dev_addr; + memcpy(sqp->hdr.eth.eth.smac_h, tmp, 2); + memcpy(sqp->hdr.eth.eth.smac_l, tmp + 2, 4); + sqp->hdr.eth.eth.type = cpu_to_be16(MLX4_RDMAOE_ETHERTYPE); + sqp->hdr.eth.bth.solicited_event = !!(wr->send_flags & IB_SEND_SOLICITED); if (!sqp->qp.ibqp.qp_num) ib_get_cached_pkey(ib_dev, sqp->qp.port, sqp->pkey_index, &pkey); else ib_get_cached_pkey(ib_dev, sqp->qp.port, wr->wr.ud.pkey_index, &pkey); - sqp->ud_header.bth.pkey = cpu_to_be16(pkey); - sqp->ud_header.bth.destination_qpn = cpu_to_be32(wr->wr.ud.remote_qpn); - sqp->ud_header.bth.psn = cpu_to_be32((sqp->send_psn++) & ((1 << 24) - 1)); - sqp->ud_header.deth.qkey = cpu_to_be32(wr->wr.ud.remote_qkey & 0x80000000 ? + sqp->hdr.eth.bth.pkey = cpu_to_be16(pkey); + sqp->hdr.eth.bth.destination_qpn = cpu_to_be32(wr->wr.ud.remote_qpn); + sqp->hdr.eth.bth.psn = cpu_to_be32((sqp->send_psn++) & ((1 << 24) - 1)); + sqp->hdr.eth.deth.qkey = cpu_to_be32(wr->wr.ud.remote_qkey & 0x80000000 ? sqp->qkey : wr->wr.ud.remote_qkey); - sqp->ud_header.deth.source_qpn = cpu_to_be32(sqp->qp.ibqp.qp_num); + sqp->hdr.eth.deth.source_qpn = cpu_to_be32(sqp->qp.ibqp.qp_num); - header_size = ib_ud_header_pack(&sqp->ud_header, sqp->header_buf); + header_size = rdmaoe_ud_header_pack(&sqp->hdr.eth, sqp->header_buf); if (0) { printk(KERN_ERR "built UD header of size %d:\n", header_size); @@ -1333,6 +1475,15 @@ static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, return 0; } +static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, + void *wqe, unsigned *mlx_seg_len) +{ + if (ib_get_port_link_type(sqp->qp.ibqp.device, sqp->qp.port) == PORT_LINK_IB) + return build_ib_mlx_header(sqp, wr, wqe, mlx_seg_len); + else + return build_eth_mlx_header(sqp, wr, wqe, mlx_seg_len); +} + static int mlx4_wq_overflow(struct mlx4_ib_wq *wq, int nreq, struct ib_cq *ib_cq) { unsigned cur; @@ -1414,6 +1565,8 @@ static void set_datagram_seg(struct mlx4_wqe_datagram_seg *dseg, memcpy(dseg->av, &to_mah(wr->wr.ud.ah)->av, sizeof (struct mlx4_av)); dseg->dqpn = cpu_to_be32(wr->wr.ud.remote_qpn); dseg->qkey = cpu_to_be32(wr->wr.ud.remote_qkey); + dseg->vlan = to_mah(wr->wr.ud.ah)->av.eth.vlan; + memcpy(dseg->mac_0_1, to_mah(wr->wr.ud.ah)->av.eth.mac_0_1, 6); } static void set_mlx_icrc_seg(void *dseg) diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 3aff8a6..b73b5f0 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -66,7 +66,8 @@ enum { MLX4_DEV_CAP_FLAG_ATOMIC = 1 << 18, MLX4_DEV_CAP_FLAG_RAW_MCAST = 1 << 19, MLX4_DEV_CAP_FLAG_UD_AV_PORT = 1 << 20, - MLX4_DEV_CAP_FLAG_UD_MCAST = 1 << 21 + MLX4_DEV_CAP_FLAG_UD_MCAST = 1 << 21, + MLX4_DEV_CAP_FLAG_RDMAOE = 1 << 30 }; enum { @@ -371,6 +372,28 @@ struct mlx4_av { u8 dgid[16]; }; +struct mlx4_eth_av { + __be32 port_pd; + u8 reserved1; + u8 smac_idx; + u16 reserved2; + u8 reserved3; + u8 gid_index; + u8 stat_rate; + u8 hop_limit; + __be32 sl_tclass_flowlabel; + u8 dgid[16]; + u32 reserved4[2]; + __be16 vlan; + u8 mac_0_1[2]; + u8 mac_2_5[4]; +}; + +union mlx4_ext_av { + struct mlx4_av ib; + struct mlx4_eth_av eth; +}; + struct mlx4_dev { struct pci_dev *pdev; unsigned long flags; @@ -399,6 +422,12 @@ struct mlx4_init_port_param { if (((type) == MLX4_PORT_TYPE_IB ? (dev)->caps.port_mask : \ ~(dev)->caps.port_mask) & 1 << ((port) - 1)) +#define mlx4_foreach_ib_transport_port(port, dev) \ + for ((port) = 1; (port) <= (dev)->caps.num_ports; (port)++) \ + if (((dev)->caps.port_mask & 1 << ((port) - 1)) || \ + ((dev)->caps.flags & MLX4_DEV_CAP_FLAG_RDMAOE)) + + int mlx4_buf_alloc(struct mlx4_dev *dev, int size, int max_direct, struct mlx4_buf *buf); void mlx4_buf_free(struct mlx4_dev *dev, int size, struct mlx4_buf *buf); diff --git a/include/linux/mlx4/qp.h b/include/linux/mlx4/qp.h index bf8f119..d73534f 100644 --- a/include/linux/mlx4/qp.h +++ b/include/linux/mlx4/qp.h @@ -112,7 +112,9 @@ struct mlx4_qp_path { u8 snooper_flags; u8 reserved3[2]; u8 counter_index; - u8 reserved4[7]; + u8 reserved4; + u8 dmac_h[2]; + u8 dmac_l[4]; }; struct mlx4_qp_context { @@ -218,7 +220,9 @@ struct mlx4_wqe_datagram_seg { __be32 av[8]; __be32 dqpn; __be32 qkey; - __be32 reservd[2]; + __be16 vlan; + u8 mac_0_1[2]; + u8 mac_2_5[4]; }; struct mlx4_wqe_lso_seg { -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 00:35:37 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 10:35:37 +0300 Subject: [ofa-general] [PATCH 9/9] mlx4: Add RDMAoE support - allow interfaces to correspond to each other Message-ID: <20090615073537.GJ27543@mtls03> This patch add support RDMAoE for mlx4. Since mlx4_ib now needs to reference mlx4_en netdevices, a new mechanism was added. Two new fields were added to struct mlx4_interface to define a protocol and a get_prot_dev method to retrieve the corresponding protocol's net device. An implementation of the new verb ib_get_port_link_type() - mlx4_ib_get_port_link_type - was added. mlx4_ib_query_port() has been modified to support eth link types. An interface is considered to be active if its corresponding eth interface is active. Code for setting the GID table of a port has been added. Currently, each IB port has a single GID entry in its table and that GID entery equals the link local IPv6 address. Signed-off-by: Eli Cohen --- drivers/infiniband/hw/mlx4/main.c | 276 +++++++++++++++++++++++++++++++++---- drivers/net/mlx4/cmd.c | 6 + drivers/net/mlx4/en_main.c | 15 ++- drivers/net/mlx4/en_port.c | 4 +- drivers/net/mlx4/en_port.h | 3 +- drivers/net/mlx4/intf.c | 20 +++ drivers/net/mlx4/main.c | 6 + drivers/net/mlx4/mlx4.h | 1 + include/linux/mlx4/cmd.h | 1 + include/linux/mlx4/driver.h | 16 ++- 10 files changed, 310 insertions(+), 38 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index ae3d759..40442b8 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include @@ -152,28 +153,19 @@ out: return err; } -static int mlx4_ib_query_port(struct ib_device *ibdev, u8 port, - struct ib_port_attr *props) +static enum ib_port_link_type mlx4_ib_get_port_link_type(struct ib_device *device, + u8 port_num) { - struct ib_smp *in_mad = NULL; - struct ib_smp *out_mad = NULL; - int err = -ENOMEM; - - in_mad = kzalloc(sizeof *in_mad, GFP_KERNEL); - out_mad = kmalloc(sizeof *out_mad, GFP_KERNEL); - if (!in_mad || !out_mad) - goto out; - - memset(props, 0, sizeof *props); - - init_query_mad(in_mad); - in_mad->attr_id = IB_SMP_ATTR_PORT_INFO; - in_mad->attr_mod = cpu_to_be32(port); + struct mlx4_dev *dev = to_mdev(device)->dev; - err = mlx4_MAD_IFC(to_mdev(ibdev), 1, 1, port, NULL, NULL, in_mad, out_mad); - if (err) - goto out; + return dev->caps.port_mask & (1 << (port_num - 1)) ? + PORT_LINK_IB : PORT_LINK_ETH; +} +static void ib_link_query_port(struct ib_device *ibdev, u8 port, + struct ib_port_attr *props, + struct ib_smp *out_mad) +{ props->lid = be16_to_cpup((__be16 *) (out_mad->data + 16)); props->lmc = out_mad->data[34] & 0x7; props->sm_lid = be16_to_cpup((__be16 *) (out_mad->data + 18)); @@ -193,6 +185,67 @@ static int mlx4_ib_query_port(struct ib_device *ibdev, u8 port, props->subnet_timeout = out_mad->data[51] & 0x1f; props->max_vl_num = out_mad->data[37] >> 4; props->init_type_reply = out_mad->data[41] >> 4; + props->link_type = PORT_LINK_IB; +} + +static void eth_link_query_port(struct ib_device *ibdev, u8 port, + struct ib_port_attr *props, + struct ib_smp *out_mad) +{ + struct mlx4_ib_rdmaoe *rdmaoe = &to_mdev(ibdev)->rdmaoe; + struct net_device *ndev; + + props->port_cap_flags = be32_to_cpup((__be32 *) (out_mad->data + 20)); + props->gid_tbl_len = to_mdev(ibdev)->dev->caps.gid_table_len[port]; + props->max_msg_sz = to_mdev(ibdev)->dev->caps.max_msg_sz; + props->pkey_tbl_len = to_mdev(ibdev)->dev->caps.pkey_table_len[port]; + props->bad_pkey_cntr = be16_to_cpup((__be16 *) (out_mad->data + 46)); + props->qkey_viol_cntr = be16_to_cpup((__be16 *) (out_mad->data + 48)); + props->active_width = out_mad->data[31] & 0xf; + props->active_speed = out_mad->data[35] >> 4; + props->max_mtu = out_mad->data[41] & 0xf; + props->active_mtu = rdmaoe->mtu[port - 1]; + props->subnet_timeout = out_mad->data[51] & 0x1f; + props->max_vl_num = out_mad->data[37] >> 4; + props->init_type_reply = out_mad->data[41] >> 4; + props->link_type = PORT_LINK_ETH; + spin_lock(&rdmaoe->lock); + ndev = rdmaoe->netdevs[port - 1]; + if (!ndev) + goto out; + + props->state = netif_running(ndev) && netif_oper_up(ndev) ? + IB_PORT_ACTIVE : IB_PORT_DOWN; + props->phys_state = props->state; +out: + spin_unlock(&rdmaoe->lock); +} + +static int mlx4_ib_query_port(struct ib_device *ibdev, u8 port, + struct ib_port_attr *props) +{ + struct ib_smp *in_mad = NULL; + struct ib_smp *out_mad = NULL; + int err = -ENOMEM; + + in_mad = kzalloc(sizeof *in_mad, GFP_KERNEL); + out_mad = kmalloc(sizeof *out_mad, GFP_KERNEL); + if (!in_mad || !out_mad) + goto out; + + memset(props, 0, sizeof *props); + + init_query_mad(in_mad); + in_mad->attr_id = IB_SMP_ATTR_PORT_INFO; + in_mad->attr_mod = cpu_to_be32(port); + + err = mlx4_MAD_IFC(to_mdev(ibdev), 1, 1, port, NULL, NULL, in_mad, out_mad); + if (err) + goto out; + + mlx4_ib_get_port_link_type(ibdev, port) == PORT_LINK_IB ? + ib_link_query_port(ibdev, port, props, out_mad) : + eth_link_query_port(ibdev, port, props, out_mad); out: kfree(in_mad); @@ -287,6 +340,7 @@ static int mlx4_SET_PORT(struct mlx4_ib_dev *dev, u8 port, int reset_qkey_viols, { struct mlx4_cmd_mailbox *mailbox; int err; + u8 is_eth = dev->dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH; mailbox = mlx4_alloc_cmd_mailbox(dev->dev); if (IS_ERR(mailbox)) @@ -302,7 +356,7 @@ static int mlx4_SET_PORT(struct mlx4_ib_dev *dev, u8 port, int reset_qkey_viols, ((__be32 *) mailbox->buf)[1] = cpu_to_be32(cap_mask); } - err = mlx4_cmd(dev->dev, mailbox->dma, port, 0, MLX4_CMD_SET_PORT, + err = mlx4_cmd(dev->dev, mailbox->dma, port, is_eth, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B); mlx4_free_cmd_mailbox(dev->dev, mailbox); @@ -538,19 +592,153 @@ static struct device_attribute *mlx4_class_attributes[] = { &dev_attr_board_id }; +static void mlx4_addrconf_ifid_eui48(u8 *eui, struct net_device *dev) +{ + memcpy(eui, dev->dev_addr, 3); + memcpy(eui + 5, dev->dev_addr + 3, 3); + eui[3] = 0xFF; + eui[4] = 0xFE; + eui[0] ^= 2; +} + +static int update_ipv6_gids(struct mlx4_ib_dev *dev, int port, int clear) +{ + struct net_device *ndev = dev->rdmaoe.netdevs[port - 1]; + struct mlx4_cmd_mailbox *mailbox; + union ib_gid *gids, *tmpgids; + int err; + + tmpgids = kzalloc(128 * sizeof *gids, GFP_ATOMIC); + if (!tmpgids) + return -ENOMEM; + + if (!clear) { + mlx4_addrconf_ifid_eui48(&tmpgids[0].raw[8], ndev); + tmpgids[0].global.subnet_prefix = cpu_to_be64(0xfe80000000000000LL); + } + spin_unlock(&dev->rdmaoe.lock); + + mailbox = mlx4_alloc_cmd_mailbox(dev->dev); + if (IS_ERR(mailbox)) { + err = PTR_ERR(mailbox); + goto out; + } + + gids = mailbox->buf; + memcpy(gids, tmpgids, 128 * sizeof *gids); + + err = mlx4_cmd(dev->dev, mailbox->dma, MLX4_SET_PORT_GID_TABLE << 8 | port, + 1, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B); + + mlx4_free_cmd_mailbox(dev->dev, mailbox); + +out: + kfree(tmpgids); + spin_lock(&dev->rdmaoe.lock); + return err; +} + +static void update_mtu(struct mlx4_ib_dev *dev, int port) +{ + struct net_device *ndev = dev->rdmaoe.netdevs[port - 1]; + + if (ndev->mtu >= 4096) + dev->rdmaoe.mtu[port - 1] = IB_MTU_4096; + else if (ndev->mtu >= 2048) + dev->rdmaoe.mtu[port - 1] = IB_MTU_2048; + else if (ndev->mtu >= 1024) + dev->rdmaoe.mtu[port - 1] = IB_MTU_1024; + else if (ndev->mtu >= 512) + dev->rdmaoe.mtu[port - 1] = IB_MTU_512; + else if (ndev->mtu >= 256) + dev->rdmaoe.mtu[port - 1] = IB_MTU_256; + else { + printk(KERN_WARNING "Eth MTU %d too small - IB may not work properly\n", + ndev->mtu); + dev->rdmaoe.mtu[port - 1] = IB_MTU_256; + } +} + +static void handle_en_event(struct mlx4_ib_dev *dev, int port, unsigned long event) +{ + switch (event) { + case NETDEV_UP: + update_ipv6_gids(dev, port, 0); + break; + + case NETDEV_DOWN: + update_ipv6_gids(dev, port, 1); + + case NETDEV_CHANGEMTU: + update_mtu(dev, port); + break; + } +} + +static void netdev_added(struct mlx4_ib_dev *dev, int port) +{ + update_ipv6_gids(dev, port, 0); + update_mtu(dev, port); +} + +static void netdev_removed(struct mlx4_ib_dev *dev, int port) +{ + update_ipv6_gids(dev, port, 1); +} + +static int mlx4_ib_netdev_event(struct notifier_block *this, unsigned long event, + void *ptr) +{ + struct net_device *dev = ptr; + struct mlx4_ib_dev *ibdev; + struct net_device *oldnd; + struct mlx4_ib_rdmaoe *rdmaoe; + int port; + + if (!net_eq(dev_net(dev), &init_net)) + return NOTIFY_DONE; + + ibdev = container_of(this, struct mlx4_ib_dev, rdmaoe.nb); + rdmaoe = &ibdev->rdmaoe; + + spin_lock(&rdmaoe->lock); + mlx4_foreach_ib_transport_port(port, ibdev->dev) { + oldnd = rdmaoe->netdevs[port - 1]; + rdmaoe->netdevs[port - 1] = mlx4_get_prot_dev(ibdev->dev, MLX4_PROT_EN, port); + if (oldnd != rdmaoe->netdevs[port - 1]) { + if (rdmaoe->netdevs[port - 1]) + netdev_added(ibdev, port); + else + netdev_removed(ibdev, port); + } + } + + if (dev == rdmaoe->netdevs[0]) + handle_en_event(ibdev, 1, event); + else if (dev == rdmaoe->netdevs[1]) + handle_en_event(ibdev, 2, event); + + spin_unlock(&rdmaoe->lock); + + return NOTIFY_DONE; +} + static void *mlx4_ib_add(struct mlx4_dev *dev) { static int mlx4_ib_version_printed; struct mlx4_ib_dev *ibdev; int num_ports = 0; int i; + int err; + int port; + struct mlx4_ib_rdmaoe *rdmaoe; if (!mlx4_ib_version_printed) { printk(KERN_INFO "%s", mlx4_ib_version); ++mlx4_ib_version_printed; } - mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_IB) + mlx4_foreach_ib_transport_port(i, dev) num_ports++; /* No point in registering a device with no ports... */ @@ -563,6 +751,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) return NULL; } + rdmaoe = &ibdev->rdmaoe; + if (mlx4_pd_alloc(dev, &ibdev->priv_pdn)) goto err_dealloc; @@ -607,10 +797,12 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) (1ull << IB_USER_VERBS_CMD_CREATE_SRQ) | (1ull << IB_USER_VERBS_CMD_MODIFY_SRQ) | (1ull << IB_USER_VERBS_CMD_QUERY_SRQ) | - (1ull << IB_USER_VERBS_CMD_DESTROY_SRQ); + (1ull << IB_USER_VERBS_CMD_DESTROY_SRQ) | + (1ull << IB_USER_VERBS_CMD_GET_MAC); ibdev->ib_dev.query_device = mlx4_ib_query_device; ibdev->ib_dev.query_port = mlx4_ib_query_port; + ibdev->ib_dev.get_port_link_type = mlx4_ib_get_port_link_type; ibdev->ib_dev.query_gid = mlx4_ib_query_gid; ibdev->ib_dev.query_pkey = mlx4_ib_query_pkey; ibdev->ib_dev.modify_device = mlx4_ib_modify_device; @@ -654,15 +846,26 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) ibdev->ib_dev.map_phys_fmr = mlx4_ib_map_phys_fmr; ibdev->ib_dev.unmap_fmr = mlx4_ib_unmap_fmr; ibdev->ib_dev.dealloc_fmr = mlx4_ib_fmr_dealloc; + ibdev->ib_dev.get_mac = mlx4_ib_get_mac; + + mlx4_foreach_ib_transport_port(port, dev) + rdmaoe->netdevs[port - 1] = mlx4_get_prot_dev(dev, MLX4_PROT_EN, port); + spin_lock_init(&rdmaoe->lock); + if (dev->caps.flags & MLX4_DEV_CAP_FLAG_RDMAOE && !rdmaoe->nb.notifier_call) { + rdmaoe->nb.notifier_call = mlx4_ib_netdev_event; + err = register_netdevice_notifier(&rdmaoe->nb); + if (err) + goto err_map; + } if (init_node_data(ibdev)) - goto err_map; + goto err_notif; spin_lock_init(&ibdev->sm_lock); mutex_init(&ibdev->cap_mask_mutex); if (ib_register_device(&ibdev->ib_dev)) - goto err_map; + goto err_notif; if (mlx4_ib_mad_init(ibdev)) goto err_reg; @@ -678,6 +881,9 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) err_reg: ib_unregister_device(&ibdev->ib_dev); +err_notif: + unregister_netdevice_notifier(&rdmaoe->nb); + err_map: iounmap(ibdev->uar_map); @@ -700,11 +906,15 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) mlx4_ib_mad_cleanup(ibdev); ib_unregister_device(&ibdev->ib_dev); + if (ibdev->rdmaoe.nb.notifier_call) { + unregister_netdevice_notifier(&ibdev->rdmaoe.nb); + ibdev->rdmaoe.nb.notifier_call = NULL; + } + iounmap(ibdev->uar_map); - for (p = 1; p <= ibdev->num_ports; ++p) + mlx4_foreach_port(p, dev, MLX4_PORT_TYPE_IB) mlx4_CLOSE_PORT(dev, p); - iounmap(ibdev->uar_map); mlx4_uar_free(dev, &ibdev->priv_uar); mlx4_pd_free(dev, ibdev->priv_pdn); ib_dealloc_device(&ibdev->ib_dev); @@ -745,17 +955,27 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, static struct mlx4_interface mlx4_ib_interface = { .add = mlx4_ib_add, .remove = mlx4_ib_remove, - .event = mlx4_ib_event + .event = mlx4_ib_event, + .protocol = MLX4_PROT_IB }; static int __init mlx4_ib_init(void) { - return mlx4_register_interface(&mlx4_ib_interface); + int err; + + mlx4_ib_addr_init(); + + err = mlx4_register_interface(&mlx4_ib_interface); + if (err) + mlx4_ib_addr_cleanup(); + + return err; } static void __exit mlx4_ib_cleanup(void) { mlx4_unregister_interface(&mlx4_ib_interface); + mlx4_ib_addr_cleanup(); } module_init(mlx4_ib_init); diff --git a/drivers/net/mlx4/cmd.c b/drivers/net/mlx4/cmd.c index 2845a05..07a646b 100644 --- a/drivers/net/mlx4/cmd.c +++ b/drivers/net/mlx4/cmd.c @@ -103,6 +103,7 @@ enum { struct mlx4_cmd_context { struct completion done; int result; + u16 command; int next; u64 out_param; u16 token; @@ -255,6 +256,10 @@ void mlx4_cmd_event(struct mlx4_dev *dev, u16 token, u8 status, u64 out_param) return; context->result = mlx4_status_to_errno(status); + if (context->result) + mlx4_warn(dev, "FW command 0x%x failed with FW status = %d\n", + context->command, status); + context->out_param = out_param; complete(&context->done); @@ -274,6 +279,7 @@ static int mlx4_cmd_wait(struct mlx4_dev *dev, u64 in_param, u64 *out_param, BUG_ON(cmd->free_head < 0); context = &cmd->context[cmd->free_head]; context->token += cmd->token_mask + 1; + context->command = op; cmd->free_head = context->next; spin_unlock(&cmd->context_lock); diff --git a/drivers/net/mlx4/en_main.c b/drivers/net/mlx4/en_main.c index 510633f..6f30eca 100644 --- a/drivers/net/mlx4/en_main.c +++ b/drivers/net/mlx4/en_main.c @@ -51,6 +51,13 @@ static const char mlx4_en_version[] = DRV_NAME ": Mellanox ConnectX HCA Ethernet driver v" DRV_VERSION " (" DRV_RELDATE ")\n"; +static void *get_netdev(struct mlx4_dev *dev, void *ctx, u8 port) +{ + struct mlx4_en_dev *endev = ctx; + + return endev->pndev[port]; +} + static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, enum mlx4_dev_event event, int port) { @@ -229,9 +236,11 @@ err_free_res: } static struct mlx4_interface mlx4_en_interface = { - .add = mlx4_en_add, - .remove = mlx4_en_remove, - .event = mlx4_en_event, + .add = mlx4_en_add, + .remove = mlx4_en_remove, + .event = mlx4_en_event, + .get_prot_dev = get_netdev, + .protocol = MLX4_PROT_EN, }; static int __init mlx4_en_init(void) diff --git a/drivers/net/mlx4/en_port.c b/drivers/net/mlx4/en_port.c index a29abe8..a249887 100644 --- a/drivers/net/mlx4/en_port.c +++ b/drivers/net/mlx4/en_port.c @@ -127,8 +127,8 @@ int mlx4_SET_PORT_qpn_calc(struct mlx4_dev *dev, u8 port, u32 base_qpn, memset(context, 0, sizeof *context); context->base_qpn = cpu_to_be32(base_qpn); - context->promisc = cpu_to_be32(promisc << SET_PORT_PROMISC_SHIFT | base_qpn); - context->mcast = cpu_to_be32(1 << SET_PORT_PROMISC_SHIFT | base_qpn); + context->promisc = cpu_to_be32(promisc << SET_PORT_PROMISC_EN_SHIFT | base_qpn); + context->mcast = cpu_to_be32(1 << SET_PORT_PROMISC_MODE_SHIFT | base_qpn); context->intra_no_vlan = 0; context->no_vlan = MLX4_NO_VLAN_IDX; context->intra_vlan_miss = 0; diff --git a/drivers/net/mlx4/en_port.h b/drivers/net/mlx4/en_port.h index e6477f1..9354891 100644 --- a/drivers/net/mlx4/en_port.h +++ b/drivers/net/mlx4/en_port.h @@ -36,7 +36,8 @@ #define SET_PORT_GEN_ALL_VALID 0x7 -#define SET_PORT_PROMISC_SHIFT 31 +#define SET_PORT_PROMISC_EN_SHIFT 31 +#define SET_PORT_PROMISC_MODE_SHIFT 30 enum { MLX4_CMD_SET_VLAN_FLTR = 0x47, diff --git a/drivers/net/mlx4/intf.c b/drivers/net/mlx4/intf.c index 0e7eb10..d64530e 100644 --- a/drivers/net/mlx4/intf.c +++ b/drivers/net/mlx4/intf.c @@ -159,3 +159,23 @@ void mlx4_unregister_device(struct mlx4_dev *dev) mutex_unlock(&intf_mutex); } + +void *mlx4_find_get_prot_dev(struct mlx4_dev *dev, enum mlx4_prot proto, int port) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + struct mlx4_device_context *dev_ctx; + unsigned long flags; + void *result = NULL; + + spin_lock_irqsave(&priv->ctx_lock, flags); + + list_for_each_entry(dev_ctx, &priv->ctx_list, list) + if (dev_ctx->intf->protocol == proto && dev_ctx->intf->get_prot_dev) { + result = dev_ctx->intf->get_prot_dev(dev, dev_ctx->context, port); + break; + } + + spin_unlock_irqrestore(&priv->ctx_lock, flags); + + return result; +} diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 30bea96..c72af51 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -100,6 +100,12 @@ module_param_named(use_prio, use_prio, bool, 0444); MODULE_PARM_DESC(use_prio, "Enable steering by VLAN priority on ETH ports " "(0/1, default 0)"); +void *mlx4_get_prot_dev(struct mlx4_dev *dev, enum mlx4_prot proto, int port) +{ + return mlx4_find_get_prot_dev(dev, proto, port); +} +EXPORT_SYMBOL(mlx4_get_prot_dev); + int mlx4_check_port_params(struct mlx4_dev *dev, enum mlx4_port_type *port_type) { diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index 5bd79c2..db068c9 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h @@ -364,6 +364,7 @@ int mlx4_restart_one(struct pci_dev *pdev); int mlx4_register_device(struct mlx4_dev *dev); void mlx4_unregister_device(struct mlx4_dev *dev); void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, int port); +void *mlx4_find_get_prot_dev(struct mlx4_dev *dev, enum mlx4_prot proto, int port); struct mlx4_dev_cap; struct mlx4_init_hca_param; diff --git a/include/linux/mlx4/cmd.h b/include/linux/mlx4/cmd.h index 0f82293..22bd8d3 100644 --- a/include/linux/mlx4/cmd.h +++ b/include/linux/mlx4/cmd.h @@ -140,6 +140,7 @@ enum { MLX4_SET_PORT_MAC_TABLE = 0x2, MLX4_SET_PORT_VLAN_TABLE = 0x3, MLX4_SET_PORT_PRIO_MAP = 0x4, + MLX4_SET_PORT_GID_TABLE = 0x5, }; struct mlx4_dev; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 53c5fdb..0083256 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -44,15 +44,23 @@ enum mlx4_dev_event { MLX4_DEV_EVENT_PORT_REINIT, }; +enum mlx4_prot { + MLX4_PROT_IB, + MLX4_PROT_EN, +}; + struct mlx4_interface { - void * (*add) (struct mlx4_dev *dev); - void (*remove)(struct mlx4_dev *dev, void *context); - void (*event) (struct mlx4_dev *dev, void *context, - enum mlx4_dev_event event, int port); + void * (*add) (struct mlx4_dev *dev); + void (*remove)(struct mlx4_dev *dev, void *context); + void (*event) (struct mlx4_dev *dev, void *context, + enum mlx4_dev_event event, int port); + void * (*get_prot_dev) (struct mlx4_dev *dev, void *context, u8 port); + enum mlx4_prot protocol; struct list_head list; }; int mlx4_register_interface(struct mlx4_interface *intf); void mlx4_unregister_interface(struct mlx4_interface *intf); +void *mlx4_get_prot_dev(struct mlx4_dev *dev, enum mlx4_prot proto, int port); #endif /* MLX4_DRIVER_H */ -- 1.6.3.1 From vlad at lists.openfabrics.org Mon Jun 15 02:19:06 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 15 Jun 2009 02:19:06 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090615-0200 daily build status Message-ID: <20090615091906.5073B1020EF4@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: Hunk #1 FAILED at 1016. Hunk #2 FAILED at 1035. 2 out of 2 hunks FAILED -- rejects in file drivers/infiniband/core/user_mad.c patching file drivers/infiniband/core/uverbs_main.c Hunk #1 FAILED at 802. 1 out of 1 hunk FAILED -- rejects in file drivers/infiniband/core/uverbs_main.c Patch core_0_sysfs_to_2_6_25.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:405: error: implicit declaration of function 'ipv6_addr_any' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:405: error: implicit declaration of function 'ipv6_addr_any' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'ia64_mv.dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'ia64_mv.dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'ia64_mv.dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'ia64_mv.dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:37: include/linux/slub_def.h: In function 'kmalloc_index': include/linux/slub_def.h:90: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: -DHAVE_WORKING_TEXT_ALIGN -DHAVE_MODEL_SMALL_ATTRIBUTE -DHAVE_SERIALIZE_DIRECTIVE -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -pipe -ffixed-r13 -mfixed-range=f12-f15,f32-f127 -falign-functions=32 -frename-registers -fno-optimize-sibling-calls -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(umem)" -D"KBUILD_MODNAME=KBUILD_STR(ib_core)" -c -o /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/.tmp_umem.o /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c: In function 'ib_umem_page_count': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.c:334: error: implicit declaration of function 'ilog2' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core/umem.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check/kernel_addons/backport/2.6.26/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:37: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h: In function 'skb_record_rx_queue': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/skbuff.h:7: error: 'struct sk_buff' has no member named 'queue_mapping' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check/include/rdma/ib_verbs.h: In function 'ib_dma_mapping_error': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check/include/rdma/ib_verbs.h:1677: warning: passing argument 1 of 'dma_mapping_error' makes integer from pointer without a cast /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check/include/rdma/ib_verbs.h:1677: error: too many arguments to function 'dma_mapping_error' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:394: error: 'init_net' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090615-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From eli at mellanox.co.il Mon Jun 15 06:42:36 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 16:42:36 +0300 Subject: [ofa-general] [PATCH] libibverbs: Add RDMAoE support Message-ID: <20090615134236.GA25353@mtls03> Extend the ibv_query_port() verb to return enum ibv_port_link_type which reports the link type to be either IB or Ethernet. This can be used by applications to know if they must use GRH as is the case in RDMAoE. Add a new system call to get the MAC address of the remote port that a UD address vector refers to. Update ibv_rc_pingpong and ibv_ud_pingpong to accept a remote GID so that they can be used with an RDMAoE port. Signed-off-by: Eli Cohen --- examples/devinfo.c | 13 ++++++++++ examples/pingpong.c | 31 +++++++++++++++++++++++++ examples/pingpong.h | 3 ++ examples/rc_pingpong.c | 50 ++++++++++++++++++++++++++++++++-------- examples/ud_pingpong.c | 37 +++++++++++++++++++++++++++--- include/infiniband/driver.h | 1 + include/infiniband/kern-abi.h | 23 +++++++++++++++++- include/infiniband/verbs.h | 7 +++++ src/cmd.c | 20 ++++++++++++++++ src/libibverbs.map | 1 + 10 files changed, 170 insertions(+), 16 deletions(-) diff --git a/examples/devinfo.c b/examples/devinfo.c index caa5d5f..fc9dbb9 100644 --- a/examples/devinfo.c +++ b/examples/devinfo.c @@ -175,6 +175,18 @@ static int print_all_port_gids(struct ibv_context *ctx, uint8_t port_num, int tb return rc; } +static const char *link_type_str(enum ibv_port_link_type type) +{ + switch (type) { + case PORT_LINK_IB: + return "PORT_LINK_IB"; + case PORT_LINK_ETH: + return "PORT_LINK_ETH"; + default: + return "Unknown"; + } +} + static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port) { struct ibv_context *ctx; @@ -273,6 +285,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port) printf("\t\t\tsm_lid:\t\t\t%d\n", port_attr.sm_lid); printf("\t\t\tport_lid:\t\t%d\n", port_attr.lid); printf("\t\t\tport_lmc:\t\t0x%02x\n", port_attr.lmc); + printf("\t\t\tlink_type:\t\t%s\n", link_type_str(port_attr.link_type)); if (verbose) { printf("\t\t\tmax_msg_sz:\t\t0x%x\n", port_attr.max_msg_sz); diff --git a/examples/pingpong.c b/examples/pingpong.c index b916f59..e53e2fa 100644 --- a/examples/pingpong.c +++ b/examples/pingpong.c @@ -31,6 +31,8 @@ */ #include "pingpong.h" +#include +#include enum ibv_mtu pp_mtu_to_enum(int mtu) { @@ -53,3 +55,32 @@ uint16_t pp_get_local_lid(struct ibv_context *context, int port) return attr.lid; } + +int pp_get_port_info(struct ibv_context *context, int port, + struct ibv_port_attr *attr) +{ + return ibv_query_port(context, port, attr); +} + +void str2gid(char *grh, union ibv_gid *gid) +{ + char tmp; + + tmp = grh[8]; + grh[8] = 0; + gid->dwords[0] = htonl(strtoul(grh, NULL, 16)); + grh[8] = tmp; + + tmp = grh[16]; + grh[16] = 0; + gid->dwords[1] = htonl(strtoul(grh + 8, NULL, 16)); + grh[16] = tmp; + + tmp = grh[24]; + grh[24] = 0; + gid->dwords[2] = htonl(strtoul(grh + 16, NULL, 16)); + grh[24] = tmp; + + gid->dwords[3] = htonl(strtoul(grh + 24, NULL, 16)); +} + diff --git a/examples/pingpong.h b/examples/pingpong.h index 71d7c3f..8c82b32 100644 --- a/examples/pingpong.h +++ b/examples/pingpong.h @@ -37,5 +37,8 @@ enum ibv_mtu pp_mtu_to_enum(int mtu); uint16_t pp_get_local_lid(struct ibv_context *context, int port); +int pp_get_port_info(struct ibv_context *context, int port, + struct ibv_port_attr *attr); +void str2gid(char *grh, union ibv_gid *gid); #endif /* IBV_PINGPONG_H */ diff --git a/examples/rc_pingpong.c b/examples/rc_pingpong.c index 26fa45c..23cad7a 100644 --- a/examples/rc_pingpong.c +++ b/examples/rc_pingpong.c @@ -67,6 +67,8 @@ struct pingpong_context { int size; int rx_depth; int pending; + struct ibv_port_attr portinfo; + union ibv_gid dgid; }; struct pingpong_dest { @@ -94,6 +96,12 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, .port_num = port } }; + + if (ctx->dgid.global.interface_id) { + attr.ah_attr.is_global = 1; + attr.ah_attr.grh.hop_limit = 1; + attr.ah_attr.grh.dgid = ctx->dgid; + } if (ibv_modify_qp(ctx->qp, &attr, IBV_QP_STATE | IBV_QP_AV | @@ -289,11 +297,11 @@ out: static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size, int rx_depth, int port, - int use_event) + int use_event, int is_server) { struct pingpong_context *ctx; - ctx = malloc(sizeof *ctx); + ctx = calloc(1, sizeof *ctx); if (!ctx) return NULL; @@ -306,7 +314,7 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size, return NULL; } - memset(ctx->buf, 0, size); + memset(ctx->buf, 0x7b + is_server, size); ctx->context = ibv_open_device(ib_dev); if (!ctx->context) { @@ -481,6 +489,7 @@ static void usage(const char *argv0) printf(" -n, --iters= number of exchanges (default 1000)\n"); printf(" -l, --sl= service level value\n"); printf(" -e, --events sleep on CQ events (default poll)\n"); + printf(" -g, --gid= gid of the other port\n"); } int main(int argc, char *argv[]) @@ -504,6 +513,7 @@ int main(int argc, char *argv[]) int rcnt, scnt; int num_cq_events = 0; int sl = 0; + char *grh = NULL; srand48(getpid() * time(NULL)); @@ -520,10 +530,11 @@ int main(int argc, char *argv[]) { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "sl", .has_arg = 1, .val = 'l' }, { .name = "events", .has_arg = 0, .val = 'e' }, + { .name = "gid", .has_arg = 1, .val = 'g' }, { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:e", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:e:g", long_options, NULL); if (c == -1) break; @@ -575,6 +586,10 @@ int main(int argc, char *argv[]) ++use_event; break; + case 'g': + grh = strdupa(optarg); + break; + default: usage(argv[0]); return 1; @@ -614,7 +629,7 @@ int main(int argc, char *argv[]) } } - ctx = pp_init_ctx(ib_dev, size, rx_depth, ib_port, use_event); + ctx = pp_init_ctx(ib_dev, size, rx_depth, ib_port, use_event, !servername); if (!ctx) return 1; @@ -630,17 +645,31 @@ int main(int argc, char *argv[]) return 1; } - my_dest.lid = pp_get_local_lid(ctx->context, ib_port); - my_dest.qpn = ctx->qp->qp_num; - my_dest.psn = lrand48() & 0xffffff; - if (!my_dest.lid) { - fprintf(stderr, "Couldn't get local LID\n"); + + if (pp_get_port_info(ctx->context, ib_port, &ctx->portinfo)) { + fprintf(stderr, "Couldn't get port info\n"); return 1; } + my_dest.lid = ctx->portinfo.lid; + if (ctx->portinfo.link_type == PORT_LINK_ETH) { + if (!grh) { + fprintf(stderr, "Couldn't get local LID\n"); + return 1; + } + str2gid(grh, &ctx->dgid); + } else { + if (!my_dest.lid) { + fprintf(stderr, "Couldn't get local LID\n"); + return 1; + } + } + my_dest.qpn = ctx->qp->qp_num; + my_dest.psn = lrand48() & 0xffffff; printf(" local address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n", my_dest.lid, my_dest.qpn, my_dest.psn); + if (servername) rem_dest = pp_client_exch_dest(servername, port, &my_dest); else @@ -705,6 +734,7 @@ int main(int argc, char *argv[]) fprintf(stderr, "poll CQ failed %d\n", ne); return 1; } + } while (!use_event && ne < 1); for (i = 0; i < ne; ++i) { diff --git a/examples/ud_pingpong.c b/examples/ud_pingpong.c index 8f3d50b..3709b16 100644 --- a/examples/ud_pingpong.c +++ b/examples/ud_pingpong.c @@ -68,6 +68,8 @@ struct pingpong_context { int size; int rx_depth; int pending; + struct ibv_port_attr portinfo; + union ibv_gid dgid; }; struct pingpong_dest { @@ -105,6 +107,12 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, return 1; } + if (ctx->dgid.global.interface_id) { + ah_attr.is_global = 1; + ah_attr.grh.hop_limit = 1; + ah_attr.grh.dgid = ctx->dgid; + } + ctx->ah = ibv_create_ah(ctx->pd, &ah_attr); if (!ctx->ah) { fprintf(stderr, "Failed to create AH\n"); @@ -478,6 +486,7 @@ static void usage(const char *argv0) printf(" -r, --rx-depth= number of receives to post at a time (default 500)\n"); printf(" -n, --iters= number of exchanges (default 1000)\n"); printf(" -e, --events sleep on CQ events (default poll)\n"); + printf(" -g, --gid specify remote gid\n"); } int main(int argc, char *argv[]) @@ -500,6 +509,7 @@ int main(int argc, char *argv[]) int rcnt, scnt; int num_cq_events = 0; int sl = 0; + char *gid; srand48(getpid() * time(NULL)); @@ -515,6 +525,7 @@ int main(int argc, char *argv[]) { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "sl", .has_arg = 1, .val = 'l' }, { .name = "events", .has_arg = 0, .val = 'e' }, + { .name = "gid", .has_arg = 1, .val = 'g' }, { 0 } }; @@ -563,6 +574,11 @@ int main(int argc, char *argv[]) ++use_event; break; + case 'g': + gid = strdupa(optarg); + printf("gid = %s\n", gid); + break; + default: usage(argv[0]); return 1; @@ -618,12 +634,25 @@ int main(int argc, char *argv[]) return 1; } - my_dest.lid = pp_get_local_lid(ctx->context, ib_port); + if (pp_get_port_info(ctx->context, ib_port, &ctx->portinfo)) { + fprintf(stderr, "Couldn't get port info\n"); + return 1; + } + my_dest.lid = ctx->portinfo.lid; + my_dest.qpn = ctx->qp->qp_num; my_dest.psn = lrand48() & 0xffffff; - if (!my_dest.lid) { - fprintf(stderr, "Couldn't get local LID\n"); - return 1; + if (ctx->portinfo.link_type == PORT_LINK_IB) { + if (!my_dest.lid) { + fprintf(stderr, "Couldn't get local LID\n"); + return 1; + } + } else { + if (!gid) { + fprintf(stderr, "must specify remote GID\n"); + return 1; + } + str2gid(gid, &ctx->dgid); } printf(" local address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n", diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h index 67a3bf8..cbd261f 100644 --- a/include/infiniband/driver.h +++ b/include/infiniband/driver.h @@ -131,6 +131,7 @@ int ibv_cmd_create_ah(struct ibv_pd *pd, struct ibv_ah *ah, int ibv_cmd_destroy_ah(struct ibv_ah *ah); int ibv_cmd_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); int ibv_cmd_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); +int ibv_cmd_get_mac(struct ibv_pd *pd, uint8_t port, uint8_t *gid, uint8_t *mac); int ibv_dontfork_range(void *base, size_t size); int ibv_dofork_range(void *base, size_t size); diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h index 0db083a..1dc5f0c 100644 --- a/include/infiniband/kern-abi.h +++ b/include/infiniband/kern-abi.h @@ -85,7 +85,8 @@ enum { IB_USER_VERBS_CMD_MODIFY_SRQ, IB_USER_VERBS_CMD_QUERY_SRQ, IB_USER_VERBS_CMD_DESTROY_SRQ, - IB_USER_VERBS_CMD_POST_SRQ_RECV + IB_USER_VERBS_CMD_POST_SRQ_RECV, + IB_USER_VERBS_CMD_GET_MAC }; /* @@ -223,7 +224,8 @@ struct ibv_query_port_resp { __u8 active_width; __u8 active_speed; __u8 phys_state; - __u8 reserved[3]; + __u8 link_type; + __u8 reserved[2]; }; struct ibv_alloc_pd { @@ -803,6 +805,7 @@ enum { * trick opcodes in IBV_INIT_CMD() doesn't break. */ IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL_V2 = -1, + IB_USER_VERBS_CMD_GET_MAC_V2 = -1, }; struct ibv_destroy_cq_v1 { @@ -878,4 +881,20 @@ struct ibv_create_srq_resp_v5 { __u32 srq_handle; }; +struct ibv_get_mac { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u32 pd_handle; + __u8 port; + __u8 reserved[3]; + __u8 dgid[16]; +}; + +struct ibv_get_mac_resp { + __u8 mac[6]; + __u16 reserved; +}; + #endif /* KERN_ABI_H */ diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index a04cc62..badbb02 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -61,6 +61,7 @@ union ibv_gid { uint64_t subnet_prefix; uint64_t interface_id; } global; + uint32_t dwords[4]; }; enum ibv_node_type { @@ -161,6 +162,11 @@ enum ibv_port_state { IBV_PORT_ACTIVE_DEFER = 5 }; +enum ibv_port_link_type { + PORT_LINK_IB, + PORT_LINK_ETH +}; + struct ibv_port_attr { enum ibv_port_state state; enum ibv_mtu max_mtu; @@ -181,6 +187,7 @@ struct ibv_port_attr { uint8_t active_width; uint8_t active_speed; uint8_t phys_state; + enum ibv_port_link_type link_type; }; enum ibv_event_type { diff --git a/src/cmd.c b/src/cmd.c index 66d7134..9007bd3 100644 --- a/src/cmd.c +++ b/src/cmd.c @@ -162,6 +162,7 @@ int ibv_cmd_query_device(struct ibv_context *context, return 0; } +#include int ibv_cmd_query_port(struct ibv_context *context, uint8_t port_num, struct ibv_port_attr *port_attr, struct ibv_query_port *cmd, size_t cmd_size) @@ -196,6 +197,7 @@ int ibv_cmd_query_port(struct ibv_context *context, uint8_t port_num, port_attr->active_width = resp.active_width; port_attr->active_speed = resp.active_speed; port_attr->phys_state = resp.phys_state; + port_attr->link_type = resp.link_type; return 0; } @@ -1122,3 +1124,21 @@ int ibv_cmd_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid) return 0; } + +int ibv_cmd_get_mac(struct ibv_pd *pd, uint8_t port, uint8_t *gid, uint8_t *mac) +{ + struct ibv_get_mac cmd; + struct ibv_get_mac_resp resp; + + IBV_INIT_CMD_RESP(&cmd, sizeof cmd, GET_MAC, &resp, sizeof resp); + memcpy(cmd.dgid, gid, sizeof cmd.dgid); + cmd.pd_handle = pd->handle; + cmd.port = port; + + if (write(pd->context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + + memcpy(mac, resp.mac, 6); + + return 0; +} diff --git a/src/libibverbs.map b/src/libibverbs.map index 1827da0..1688e73 100644 --- a/src/libibverbs.map +++ b/src/libibverbs.map @@ -64,6 +64,7 @@ IBVERBS_1.0 { ibv_cmd_destroy_ah; ibv_cmd_attach_mcast; ibv_cmd_detach_mcast; + ibv_cmd_get_mac; ibv_copy_qp_attr_from_kern; ibv_copy_path_rec_from_kern; ibv_copy_path_rec_to_kern; -- 1.6.3.1 From eli at mellanox.co.il Mon Jun 15 06:43:32 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 15 Jun 2009 16:43:32 +0300 Subject: [ofa-general] [PATCH] libmlx4: Add RDMAoE support Message-ID: <20090615134332.GA25394@mtls03> Modify mlx4_create_ah() to check the port link type and for Ethernet ports, do a system call to retrieve the remote port's MAC address. Add ConnectX EN 10GigE PCIe gen2 to the list of supported device. Make modifications to address vector data structs and code to accomodate for RDMAoE. Signed-off-by: Eli Cohen --- src/mlx4.c | 1 + src/mlx4.h | 3 +++ src/qp.c | 2 ++ src/verbs.c | 14 ++++++++++++++ src/wqe.h | 3 ++- 5 files changed, 22 insertions(+), 1 deletions(-) diff --git a/src/mlx4.c b/src/mlx4.c index 34ece39..d2e32fd 100644 --- a/src/mlx4.c +++ b/src/mlx4.c @@ -66,6 +66,7 @@ struct { HCA(MELLANOX, 0x6354), /* MT25408 "Hermon" QDR */ HCA(MELLANOX, 0x6732), /* MT25408 "Hermon" DDR PCIe gen2 */ HCA(MELLANOX, 0x673c), /* MT25408 "Hermon" QDR PCIe gen2 */ + HCA(MELLANOX, 0x6750), /* MT25408 "Hermon" EN 10GigE PCIe gen2 */ }; static struct ibv_context_ops mlx4_ctx_ops = { diff --git a/src/mlx4.h b/src/mlx4.h index 827a201..20d3fdd 100644 --- a/src/mlx4.h +++ b/src/mlx4.h @@ -236,11 +236,14 @@ struct mlx4_av { uint8_t hop_limit; uint32_t sl_tclass_flowlabel; uint8_t dgid[16]; + uint8_t mac[8]; }; struct mlx4_ah { struct ibv_ah ibv_ah; struct mlx4_av av; + uint16_t vlan; + uint8_t mac[6]; }; static inline unsigned long align(unsigned long val, unsigned long align) diff --git a/src/qp.c b/src/qp.c index d194ae3..cd8fab0 100644 --- a/src/qp.c +++ b/src/qp.c @@ -143,6 +143,8 @@ static void set_datagram_seg(struct mlx4_wqe_datagram_seg *dseg, memcpy(dseg->av, &to_mah(wr->wr.ud.ah)->av, sizeof (struct mlx4_av)); dseg->dqpn = htonl(wr->wr.ud.remote_qpn); dseg->qkey = htonl(wr->wr.ud.remote_qkey); + dseg->vlan = htons(to_mah(wr->wr.ud.ah)->vlan); + memcpy(dseg->mac, to_mah(wr->wr.ud.ah)->mac, 6); } static void __set_data_seg(struct mlx4_wqe_data_seg *dseg, struct ibv_sge *sg) diff --git a/src/verbs.c b/src/verbs.c index cc179a0..ff59a93 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -617,6 +617,7 @@ int mlx4_destroy_qp(struct ibv_qp *ibqp) struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr) { struct mlx4_ah *ah; + struct ibv_port_attr port_attr; ah = malloc(sizeof *ah); if (!ah) @@ -642,7 +643,20 @@ struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr) memcpy(ah->av.dgid, attr->grh.dgid.raw, 16); } + if (ibv_query_port(pd->context, attr->port_num, &port_attr)) + goto err; + + if (port_attr.link_type == PORT_LINK_ETH) { + if (ibv_cmd_get_mac(pd, attr->port_num, ah->av.dgid, ah->mac)) + goto err; + ah->vlan = 0; + } + + return &ah->ibv_ah; +err: + free(ah); + return NULL; } int mlx4_destroy_ah(struct ibv_ah *ah) diff --git a/src/wqe.h b/src/wqe.h index 6f7f309..ea6f27f 100644 --- a/src/wqe.h +++ b/src/wqe.h @@ -78,7 +78,8 @@ struct mlx4_wqe_datagram_seg { uint32_t av[8]; uint32_t dqpn; uint32_t qkey; - uint32_t reserved[2]; + __be16 vlan; + uint8_t mac[6]; }; struct mlx4_wqe_data_seg { -- 1.6.3.1 From pashash at gmail.com Mon Jun 15 06:52:40 2009 From: pashash at gmail.com (Pavel Shamis (Pasha)) Date: Mon, 15 Jun 2009 16:52:40 +0300 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il> <39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> Message-ID: <4A365228.4080702@dev.mellanox.co.il> > > I can see from the mails and from my personal experience that most of > the "end users" do not need/use the MPI coming as part of OFED (they > have many different MPI installed in their clusters), as we can see > distro are not using it and also some (if not all) of OFED binary > package providers (i.e. companies) do not use it. > > We think that the simple & clear answer is: "take the MPI packages out > of OFED" > It is not so simple and clear for me after all this discussion on the thread. Some OFED member want to remove MPIs and some strongly against it (the same correct for OFED user community too). As I mentioned previously I sure that this step will push users from OFED distributions towards vendors packages, like it was before first OFED release. What will you answer on user's question: "How can I install MPI with OFED support ?" Will you send user to read 10 pages of MPI Installation FAQ ? :-) My 0.02 $ Pasha From jsquyres at cisco.com Mon Jun 15 08:25:34 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 15 Jun 2009 11:25:34 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A365228.4080702@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il><39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> <4A365228.4080702@dev.mellanox.co.il> Message-ID: <4AC3CA38-7083-44F9-A96D-F0CEDAAA559B@cisco.com> On Jun 15, 2009, at 9:52 AM, Pavel Shamis (Pasha) wrote: > > We think that the simple & clear answer is: "take the MPI packages > out > > of OFED" > > It is not so simple and clear for me after all this discussion on > the thread. Some OFED member want to remove MPIs and some strongly > against it (the same correct for OFED user community too). > It speaks volumes to me that Red Hat has told us -- repeatedly -- that this is bad software engineering, not what the rest of the Linux community does, and has specifically asked us not to do it. But yet we still do. The main objection so far seems to have been "but we want to include MPI" (the co-development arguments do not make sense for OFED *distribution*). > As I mentioned previously I sure that this step will push users from > OFED distributions towards vendors packages, like it was before > first OFED release. > Perhaps it's time that we start discussion of how the every vendor has incompatible/different OFED distributions is far worse than including MPI. As I indicated earlier, we're back to the bad old days of mVAPI. This compatibility stuff is confusing enough for those of us who are in the industry and working with this stuff every day -- can you imagine being a customer just trying to get a bug fix? > What will you answer on user's question: "How can I install MPI with > OFED support ?" Will you send user to read 10 pages of MPI > Installation FAQ ? :-) > Many customers are in the habit of downloading/installing MPI manually, anyway, since they want to be on the bleeding edge of releases (as has been stated on this list). For customers who just want an MPI that works, a simple installation guide, and/or getting the MPI's to distribute binaries, and/or having customers download MPI's from the distros are all viable solutions. In short: these are all solvable problems. More specifically: wasn't OFED created to solve the problem of mVAPI ! = mVAPI != mVAPI? If so, it has failed. There are two obvious solutions (and probably others): 1. Test together and only use the distros to distribute new versions, or 2. Use better packaging such that vendors can *contribute* their own technology without effectively making (potentially incompatible) forks of OFED. -- Jeff Squyres Cisco Systems From roel.kluin at gmail.com Mon Jun 15 11:04:08 2009 From: roel.kluin at gmail.com (Roel Kluin) Date: Mon, 15 Jun 2009 20:04:08 +0200 Subject: [ofa-general] [PATCH] EHCA: Enforce a positive guid_entry index Message-ID: <4A368D18.108@gmail.com> This prevents the memcpy of a guid_entries element using a negative index. Signed-off-by: Roel Kluin --- drivers/infiniband/hw/ehca/ehca_hca.c | 2 +- drivers/infiniband/hw/ehca/ehca_iverbs.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_hca.c b/drivers/infiniband/hw/ehca/ehca_hca.c index 9209c53..1175ee1 100644 --- a/drivers/infiniband/hw/ehca/ehca_hca.c +++ b/drivers/infiniband/hw/ehca/ehca_hca.c @@ -311,7 +311,7 @@ query_pkey1: } int ehca_query_gid(struct ib_device *ibdev, u8 port, - int index, union ib_gid *gid) + unsigned index, union ib_gid *gid) { int ret = 0; u64 h_ret; diff --git a/drivers/infiniband/hw/ehca/ehca_iverbs.h b/drivers/infiniband/hw/ehca/ehca_iverbs.h index 8f7f282..05a83bb 100644 --- a/drivers/infiniband/hw/ehca/ehca_iverbs.h +++ b/drivers/infiniband/hw/ehca/ehca_iverbs.h @@ -54,7 +54,7 @@ int ehca_query_sma_attr(struct ehca_shca *shca, u8 port, int ehca_query_pkey(struct ib_device *ibdev, u8 port, u16 index, u16 * pkey); -int ehca_query_gid(struct ib_device *ibdev, u8 port, int index, +int ehca_query_gid(struct ib_device *ibdev, u8 port, unsigned index, union ib_gid *gid); int ehca_modify_port(struct ib_device *ibdev, u8 port, int port_modify_mask, From jgunthorpe at obsidianresearch.com Mon Jun 15 09:32:03 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Mon, 15 Jun 2009 10:32:03 -0600 Subject: [ofa-general] [PATCH] libibverbs: Add RDMAoE support In-Reply-To: <20090615134236.GA25353@mtls03> References: <20090615134236.GA25353@mtls03> Message-ID: <20090615163203.GF12430@obsidianresearch.com> On Mon, Jun 15, 2009 at 04:42:36PM +0300, Eli Cohen wrote: > +void str2gid(char *grh, union ibv_gid *gid) > +{ > + char tmp; > + > + tmp = grh[8]; > + grh[8] = 0; > + gid->dwords[0] = htonl(strtoul(grh, NULL, 16)); > + grh[8] = tmp; > + > + tmp = grh[16]; > + grh[16] = 0; > + gid->dwords[1] = htonl(strtoul(grh + 8, NULL, 16)); > + grh[16] = tmp; > + > + tmp = grh[24]; > + grh[24] = 0; > + gid->dwords[2] = htonl(strtoul(grh + 16, NULL, 16)); > + grh[24] = tmp; > + > + gid->dwords[3] = htonl(strtoul(grh + 24, NULL, 16)); > +} Please use inet_pton(AF_INET6,grh,gid) for this? > + if (ctx->dgid.global.interface_id) { > + attr.ah_attr.is_global = 1; > + attr.ah_attr.grh.hop_limit = 1; > + attr.ah_attr.grh.dgid = ctx->dgid; > + } We've been working to get rid of stuff like this for the IB router work. Can you call into the SA path resolution stuff instead of doing this? > + if (ctx->dgid.global.interface_id) { > + ah_attr.is_global = 1; > + ah_attr.grh.hop_limit = 1; > + ah_attr.grh.dgid = ctx->dgid; > + } > + Same Jason From robert.j.woodruff at intel.com Mon Jun 15 10:01:07 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Mon, 15 Jun 2009 10:01:07 -0700 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <10CDE7FD-3D89-43AF-9E25-F9B76630FBE5@cisco.com> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il><200906081707.43472.cap@nsc.liu.se> <4A2E6878.6080200@dev.mellanox.co.il> <10CDE7FD-3D89-43AF-9E25-F9B76630FBE5@cisco.com> Message-ID: <382A478CAD40FA4FB46605CF81FE39F4352A7D47@orsmsx507.amr.corp.intel.com> Jeff wrote, >Yow! We need a technology/system/process/whatever for allowing >vendors to distribute what they need without effectively forking OFED >to make their own " OFED". Even if OFED *should* >be the same as community OFED, it sometimes (?usually?) is different >in very subtle/small-but-important ways (e.g., vendorX compiling/ >installing vendorY's drivers, but not QA'ing them). To be clear: with >each vendor putting out their own different versions of OFED, it makes >for big user confusion about compatibility and ecosystem. I see this as a probalem as well. I think that some cases, the forked OFED stacks are a superset of OFED plus other vendor supplied stuff, like firmware for their card. If they are however removing support for the other IHV's drivers, I see this as a problem. I think that if we split all of the common code from the kernel RPM into it's own RPM and each IHV provide RPMs for their drivers (as I suggested in the last EWG meeting), that should help this problem as then a vendor supplied release could contain exactly the RPMs from OFED and not a derivative that removes support for another vendor's hardware. my 2 cents. woody -----Original Message----- From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Jeff Squyres Sent: Thursday, June 11, 2009 4:49 AM To: Pavel Shamis (Pasha) Cc: general at lists.openfabrics.org Subject: Re: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? On Jun 9, 2009, at 6:49 AM, Pavel Shamis (Pasha) wrote: > The excluding of MPI from OFED, will only push users to use vendor > specific OFED builds (that provides MPI out of box) and I'm not sure > that it is good for OFED community. > Pasha and I discussed exactly this point in IM and we agreed to disagree. :-) FWIW, I think this point touches on an issue that is deeper than just MPI in OFED. Different vendors having their own [potentially incompatible] versions of OFED -- each with different value-add -- is both good and bad. GOOD: vendors can innovate and differentiate BAD: it seems like the bad old days of different/incompatible vendor versions of mVAPI: OFED != OFED != OFED I've heard similar stories from many users, "I have OFED -- is that the same/compatible as OFED?" And sometimes the answer is "not entirely". Yow! We need a technology/system/process/whatever for allowing vendors to distribute what they need without effectively forking OFED to make their own " OFED". Even if OFED *should* be the same as community OFED, it sometimes (?usually?) is different in very subtle/small-but-important ways (e.g., vendorX compiling/ installing vendorY's drivers, but not QA'ing them). To be clear: with each vendor putting out their own different versions of OFED, it makes for big user confusion about compatibility and ecosystem. Note that Pasha's answer at least somewhat implies that he feels the same way. :-) -- Jeff Squyres Cisco Systems _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From robert.j.woodruff at intel.com Mon Jun 15 10:13:17 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Mon, 15 Jun 2009 10:13:17 -0700 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A365228.4080702@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il> <39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> <4A365228.4080702@dev.mellanox.co.il> Message-ID: <382A478CAD40FA4FB46605CF81FE39F4352A7D88@orsmsx507.amr.corp.intel.com> Pasha wrote, >It is not so simple and clear for me after all this discussion on the >thread. >Some OFED member want to remove MPIs and some strongly against it (the >same correct for OFED user community >too). I would agree, there is clearly no consensus on this one and I do not thing there is going to be. I think there has probably been enough discussion on this one. We can re-discuss it at the next EWG meeting, but I would say that given that lack of consensus, we should probably leave OFED as is, with the MPIs included. woody From sean.hefty at intel.com Mon Jun 15 10:24:40 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 15 Jun 2009 10:24:40 -0700 Subject: [ofa-general] [PATCH 1/9] ib_core: Add API to query port link type In-Reply-To: <20090615073419.GB27543@mtls03> References: <20090615073419.GB27543@mtls03> Message-ID: <8E9B862093B44C0F93D38C38B8A4655A@amr.corp.intel.com> >This allows to get the type of a port to be either Ethernet or IB which is >required by following patches for implementing RDMA over Ethernet - RDMAoE. I don't know if this makes more sense without studying the changes in more detail, but was there a reason why node_type just wasn't extended instead? From yiftahs at voltaire.com Mon Jun 15 10:26:37 2009 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Mon, 15 Jun 2009 20:26:37 +0300 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out ofOFED? In-Reply-To: <4A365228.4080702@dev.mellanox.co.il> References: <4A257DBF.4000401@mellanox.co.il><4A2B75D4.7080600@dev.mellanox.co.il><39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> <4A365228.4080702@dev.mellanox.co.il> Message-ID: <39C75744D164D948A170E9792AF8E7CA024DD222@exil.voltaire.com> Hi Pasha, > As I mentioned previously I sure that this step will push users from > OFED distributions towards vendors packages, like it was before > first OFED release. First I want it will be clear - I do not want to push customers to "vendor packages". Our "end in mind" is that our customers will get what they need from distro, not from any "OFED package". Both RH and Novell (and others) are doing excellent work taking what we all developed (to upstream kernel) and they are now almost up to speed with OFED release cycle. HPC customer that taking OFED src from OFA, compiling it to their specific OS/kernel can easily handle taking MPI package from the MPI .org www. Other customers that want to work with "vendor OFED" stack do not "care" what is the MPI in OFED, they get what vendor provide. For latest OS we recommend to our customers - use IB stack that is coming from OS, it is mature & stable, if you have any issue (or need new feature/bug fix) we will work directly with distro to fix it or can provide a fast fix. > What will you answer on user's question: "How can I install MPI with > OFED support ?" > Will you send user to read 10 pages of MPI Installation FAQ ? :-) How does Ethernet customers have MPI, do they get it as part of the TCP stack ? (actually they do not get the stack outside the kernel or distro but this is different discussion :-) ) Our customers (especially the HPC ones) can go to open src project download and install it, if the MPI is too complicated to be install with "RDMA" support then MPI maintainers need to help their customers and make it simpler (simple installation and better readme). Regards, Yiftah > -----Original Message----- > From: Pavel Shamis (Pasha) [mailto:pashash at gmail.com] > Sent: Monday, June 15, 2009 16:53 > To: Yiftah Shahar > Cc: Pavel Shamis (Pasha); Tziporet Koren; EWG; OpenFabrics General > Subject: Re: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out > ofOFED? > > > > > > I can see from the mails and from my personal experience that most of > > the "end users" do not need/use the MPI coming as part of OFED (they > > have many different MPI installed in their clusters), as we can see > > distro are not using it and also some (if not all) of OFED binary > > package providers (i.e. companies) do not use it. > > > > We think that the simple & clear answer is: "take the MPI packages out > > of OFED" > > > > It is not so simple and clear for me after all this discussion on the > thread. > Some OFED member want to remove MPIs and some strongly against it (the > same correct for OFED user community > too). > > As I mentioned previously I sure that this step will push users from > OFED distributions > towards vendors packages, like it was before first OFED release. > > What will you answer on user's question: "How can I install MPI with > OFED support ?" > Will you send user to read 10 pages of MPI Installation FAQ ? :-) > > My 0.02 $ > > Pasha > > > From hal.rosenstock at gmail.com Mon Jun 15 11:26:42 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 15 Jun 2009 14:26:42 -0400 Subject: [ofa-general] Re: [ewg] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: <20090615073442.GD27543@mtls03> References: <20090615073442.GD27543@mtls03> Message-ID: On Mon, Jun 15, 2009 at 3:34 AM, Eli Cohen wrote: > Since RDMAoE is using Ethernet there is no need for QP0. This patch will create > only QP1 for RDMAoE ports. Should ib_post_send_mad return some error on QP0 sends on RDMAoE ports ? What QP1 sends are allowed ? Is it only SA sends which are faked ? How are others handled ? These questions are important to what happens to the IB management/diag tools when invoked on an RDMAoE port. We need to be able to handle that scenario cleanly. Also, one minor comment below. > Signed-off-by: Eli Cohen > --- >  drivers/infiniband/core/agent.c |   16 ++++++++----- >  drivers/infiniband/core/mad.c   |   48 ++++++++++++++++++++++++++++++--------- >  2 files changed, 47 insertions(+), 17 deletions(-) > > diff --git a/drivers/infiniband/core/agent.c b/drivers/infiniband/core/agent.c > index ae7c288..658a278 100644 > --- a/drivers/infiniband/core/agent.c > +++ b/drivers/infiniband/core/agent.c > @@ -46,8 +46,10 @@ >  #define SPFX "ib_agent: " > >  struct ib_agent_port_private { > -       struct list_head port_list; > -       struct ib_mad_agent *agent[2]; > +       struct list_head        port_list; > +       struct ib_mad_agent    *agent[2]; > +       struct ib_device       *device; > +       u8                      port_num; >  }; > >  static DEFINE_SPINLOCK(ib_agent_port_list_lock); > @@ -58,11 +60,10 @@ __ib_get_agent_port(struct ib_device *device, int port_num) >  { >        struct ib_agent_port_private *entry; > > -       list_for_each_entry(entry, &ib_agent_port_list, port_list) { > -               if (entry->agent[0]->device == device && > -                   entry->agent[0]->port_num == port_num) > +       list_for_each_entry(entry, &ib_agent_port_list, port_list) > +               if (entry->device == device && entry->port_num == port_num) >                        return entry; > -       } > + >        return NULL; >  } > > @@ -175,6 +176,9 @@ int ib_agent_port_open(struct ib_device *device, int port_num) >                goto error3; >        } > > +       port_priv->device = device; > +       port_priv->port_num = port_num; > + >        spin_lock_irqsave(&ib_agent_port_list_lock, flags); >        list_add_tail(&port_priv->port_list, &ib_agent_port_list); >        spin_unlock_irqrestore(&ib_agent_port_list_lock, flags); > diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c > index de922a0..aadf396 100644 > --- a/drivers/infiniband/core/mad.c > +++ b/drivers/infiniband/core/mad.c > @@ -199,6 +199,16 @@ struct ib_mad_agent *ib_register_mad_agent(struct ib_device *device, >        unsigned long flags; >        u8 mgmt_class, vclass; > > +       /* Validate device and port */ > +       port_priv = ib_get_mad_port(device, port_num); > +       if (!port_priv) { > +               ret = ERR_PTR(-ENODEV); > +               goto error1; > +       } > + > +       if (!port_priv->qp_info[qp_type].qp) > +               return NULL; > + >        /* Validate parameters */ >        qpn = get_spl_qp_index(qp_type); >        if (qpn == -1) > @@ -260,13 +270,6 @@ struct ib_mad_agent *ib_register_mad_agent(struct ib_device *device, >                        goto error1; >        } > > -       /* Validate device and port */ > -       port_priv = ib_get_mad_port(device, port_num); > -       if (!port_priv) { > -               ret = ERR_PTR(-ENODEV); > -               goto error1; > -       } > - >        /* Allocate structures */ >        mad_agent_priv = kzalloc(sizeof *mad_agent_priv, GFP_KERNEL); >        if (!mad_agent_priv) { > @@ -556,6 +559,9 @@ int ib_unregister_mad_agent(struct ib_mad_agent *mad_agent) >        struct ib_mad_agent_private *mad_agent_priv; >        struct ib_mad_snoop_private *mad_snoop_priv; > > +       if (!mad_agent) > +               return 0; > + >        /* If the TID is zero, the agent can only snoop. */ >        if (mad_agent->hi_tid) { >                mad_agent_priv = container_of(mad_agent, > @@ -2602,6 +2608,9 @@ static void cleanup_recv_queue(struct ib_mad_qp_info *qp_info) >        struct ib_mad_private *recv; >        struct ib_mad_list_head *mad_list; > > +       if (!qp_info->qp) > +               return; > + >        while (!list_empty(&qp_info->recv_queue.list)) { > >                mad_list = list_entry(qp_info->recv_queue.list.next, > @@ -2643,6 +2652,9 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) > >        for (i = 0; i < IB_MAD_QPS_CORE; i++) { >                qp = port_priv->qp_info[i].qp; > +               if (!qp) > +                       continue; > + >                /* >                 * PKey index for QP1 is irrelevant but >                 * one is needed for the Reset to Init transition > @@ -2684,6 +2696,9 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) >        } > >        for (i = 0; i < IB_MAD_QPS_CORE; i++) { > +               if (!port_priv->qp_info[i].qp) > +                       continue; > + >                ret = ib_mad_post_receive_mads(&port_priv->qp_info[i], NULL); >                if (ret) { >                        printk(KERN_ERR PFX "Couldn't post receive WRs\n"); > @@ -2762,6 +2777,9 @@ error: > >  static void destroy_mad_qp(struct ib_mad_qp_info *qp_info) >  { > +       if (!qp_info->qp) > +               return; > + >        ib_destroy_qp(qp_info->qp); >        kfree(qp_info->snoop_table); >  } > @@ -2777,6 +2795,7 @@ static int ib_mad_port_open(struct ib_device *device, >        struct ib_mad_port_private *port_priv; >        unsigned long flags; >        char name[sizeof "ib_mad123"]; > +       int has_smi; > >        /* Create new device info */ >        port_priv = kzalloc(sizeof *port_priv, GFP_KERNEL); > @@ -2792,7 +2811,11 @@ static int ib_mad_port_open(struct ib_device *device, >        init_mad_qp(port_priv, &port_priv->qp_info[0]); >        init_mad_qp(port_priv, &port_priv->qp_info[1]); > > +       has_smi = ib_get_port_link_type(device, port_num) == PORT_LINK_IB; >        cq_size = (IB_MAD_QP_SEND_SIZE + IB_MAD_QP_RECV_SIZE) * 2; > +       if (has_smi) > +               cq_size *= 2; > + >        port_priv->cq = ib_create_cq(port_priv->device, >                                     ib_mad_thread_completion_handler, >                                     NULL, port_priv, cq_size, 0); > @@ -2816,9 +2839,11 @@ static int ib_mad_port_open(struct ib_device *device, >                goto error5; >        } > > -       ret = create_mad_qp(&port_priv->qp_info[0], IB_QPT_SMI); > -       if (ret) > -               goto error6; > +       if (has_smi) { > +               ret = create_mad_qp(&port_priv->qp_info[0], IB_QPT_SMI); > +               if (ret) > +                       goto error6; > +       } >        ret = create_mad_qp(&port_priv->qp_info[1], IB_QPT_GSI); >        if (ret) >                goto error7; > @@ -2852,7 +2877,8 @@ error9: >  error8: >        destroy_mad_qp(&port_priv->qp_info[1]); >  error7: > -       destroy_mad_qp(&port_priv->qp_info[0]); > +       if (has_smi) > +               destroy_mad_qp(&port_priv->qp_info[0]); Is something similar needed in ib_mad_port_close for handling no QP0 on RDMAoE ports in terms of destroy_mad_qp/cleanup_recv_queue calls ? -- Hal >  error6: >        ib_dereg_mr(port_priv->mr); >  error5: > -- > 1.6.3.1 > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > From hal.rosenstock at gmail.com Mon Jun 15 11:53:25 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 15 Jun 2009 14:53:25 -0400 Subject: [ofa-general] Can I assign custom LIDs to a port ? In-Reply-To: <91fe68d50906141919r352d5f33x3b08056dae1b0668@mail.gmail.com> References: <91fe68d50906141919r352d5f33x3b08056dae1b0668@mail.gmail.com> Message-ID: On Sun, Jun 14, 2009 at 10:19 PM, Jordan wrote: > Now I need to assign a port with custom LIDs, for example , I want to assign > the number "1, 2, 3" to a port . Is there some methods to do so ? It seems > that opensm doesn't have this  function. Are you asking about multiple LIDs for a port, or custom LID assignment to ports (or both) ? Dotan answered the multiple LIDs for a port. -- Hal > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From jim.ryan at intel.com Mon Jun 15 18:54:49 2009 From: jim.ryan at intel.com (Ryan, Jim) Date: Mon, 15 Jun 2009 18:54:49 -0700 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet -- some procedural questions In-Reply-To: <20090615073408.GA27543@mtls03> Message-ID: <3F6F638B8D880340AB536D29CD4C1E194DAFEEFB@orsmsx501.amr.corp.intel.com> I'm responding to this posting because it offered a clear summary of the RDMAoE proposal from Mellanox. I want to raise some possible procedural questions and ask for comment. I don't have detailed notes on this, so this is to the best of my recollection. I recall RDMAoE was introduced, I think for the first time, at an EWG meeting. There were some concerns raised then and the Mellanox proponents withdrew the proposal. The concerns raised included: - There was no documentation on what Mellanox was proposing - There was no source code - There were concerns about potential IP issues - There was no standards body associated with the proposal. Recall the bylaws of OFA requires that any ULPs that OFA supports will be produced by some recognized standards organization. No such organization was known to be associated with the Mellanox proposal. So, the simple question is do we have an issue or is the Mellanox proposal somehow exempt from our bylaws requirements? Thanks, Jim -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Eli Cohen Sent: Monday, June 15, 2009 12:34 AM To: Roland Dreier Cc: ewg; general-list Subject: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over Ethernet, providing IB capabilities for Ethernet fabrics. The packets are standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport headers and payload. HCA RDMAoE ports are no different than regular IB ports from the RDMA stack perspective. IB subnet management and SA services are not required for RDMAoE operation; Ethernet management practices are used instead. In Ethernet, nodes are commonly referred to by applications by means of an IP address. RDMAoE treats IP addresses that were assigned to the corresponding Ethernet port as GIDs, and makes use of the IP stack to bind a destination address to the corresponding netdevice (just as the CMA does today for IB and iWARP) and to obtain its L2 MAC addresses. The RDMA Verbs API is syntactically unmodified. When referring to RDMAoE ports, Address handles are required to contain GIDs and the L2 address fields in the API are ignored. The Ethernet L2 information is then obtained by the vendor-specific driver (both in kernel- and user-space) while modifying QPs to RTR and creating address handles. In order to maintain application compatibility, RDMAoE implements a SA_Query API that locally returns path records with the corresponding GIDs and the other relevant parameters . Consequently, any CMA or native Verbs application, in kernel or user-space, that uses path queries to obtain its address information, will run transparently over RDMAoE with no changes. We have successfully tested MPI, SDP, RDS, and native Verbs applications over RDMAoE without *any* changes. In the mlx4 driver stack, mlx4_en must be loaded and the corresponding eth Ethernet (with support for RDMAoE) or IB, as it was already the case. Following is a series of 9 patches based on version 2.6.30 of the Linux kernel. Signed-off-by: Eli Cohen drivers/infiniband/core/addr.c | 20 ++- drivers/infiniband/core/agent.c | 16 +- drivers/infiniband/core/cma.c | 39 ++++- drivers/infiniband/core/mad.c | 48 ++++-- drivers/infiniband/core/multicast.c | 153 ++++++++++++++-- drivers/infiniband/core/sa_query.c | 167 ++++++++++++++---- drivers/infiniband/core/ud_header.c | 111 ++++++++++++ drivers/infiniband/core/uverbs.h | 1 + drivers/infiniband/core/uverbs_cmd.c | 34 ++++ drivers/infiniband/core/uverbs_main.c | 1 + drivers/infiniband/core/verbs.c | 18 ++ drivers/infiniband/hw/mlx4/ah.c | 228 ++++++++++++++++++++---- drivers/infiniband/hw/mlx4/main.c | 276 ++++++++++++++++++++++++++--- drivers/infiniband/hw/mlx4/mlx4_ib.h | 30 +++- drivers/infiniband/hw/mlx4/qp.c | 253 +++++++++++++++++++++----- drivers/infiniband/ulp/ipoib/ipoib_main.c | 3 + drivers/net/mlx4/cmd.c | 6 + drivers/net/mlx4/en_main.c | 15 ++- drivers/net/mlx4/en_port.c | 4 +- drivers/net/mlx4/en_port.h | 3 +- drivers/net/mlx4/intf.c | 20 ++ drivers/net/mlx4/main.c | 6 + drivers/net/mlx4/mlx4.h | 1 + include/linux/mlx4/cmd.h | 1 + include/linux/mlx4/device.h | 31 +++- include/linux/mlx4/driver.h | 16 ++- include/linux/mlx4/qp.h | 8 +- include/rdma/ib_addr.h | 51 ++++++ include/rdma/ib_pack.h | 26 +++ include/rdma/ib_user_verbs.h | 21 ++- include/rdma/ib_verbs.h | 22 +++ 31 files changed, 1419 insertions(+), 210 deletions(-) _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From jgunthorpe at obsidianresearch.com Mon Jun 15 21:13:12 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Mon, 15 Jun 2009 22:13:12 -0600 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet -- some procedural questions In-Reply-To: <3F6F638B8D880340AB536D29CD4C1E194DAFEEFB@orsmsx501.amr.corp.intel.com> References: <20090615073408.GA27543@mtls03> <3F6F638B8D880340AB536D29CD4C1E194DAFEEFB@orsmsx501.amr.corp.intel.com> Message-ID: <20090616041312.GH12430@obsidianresearch.com> On Mon, Jun 15, 2009 at 06:54:49PM -0700, Ryan, Jim wrote: > Recall the bylaws of OFA requires that any ULPs that OFA supports > will be produced by some recognized standards organization. No such > organization was known to be associated with the Mellanox proposal. Is this being submitted as something to be included with OFA/Linux or is this just a technology demonstration? Personally, I don't think the proliferation of new networking protocols is something to be encouraged. If this is really what people want it should be IBoE and just like FCoE all the issues regarding interworking with actual IB networks must be delt with. Jason From gregkh at suse.de Mon Jun 15 22:46:30 2009 From: gregkh at suse.de (Greg Kroah-Hartman) Date: Mon, 15 Jun 2009 22:46:30 -0700 Subject: [ofa-general] [PATCH 41/64] infiniband: remove driver_data direct access of struct device In-Reply-To: <20090616051351.GA23627@kroah.com> References: <20090616051351.GA23627@kroah.com> Message-ID: <1245131213-24168-41-git-send-email-gregkh@suse.de> In the near future, the driver core is going to not allow direct access to the driver_data pointer in struct device. Instead, the functions dev_get_drvdata() and dev_set_drvdata() should be used. These functions have been around since the beginning, so are backwards compatible with all older kernel versions. Cc: general at lists.openfabrics.org Cc: Roland Dreier Cc: Hal Rosenstock Cc: Sean Hefty Signed-off-by: Greg Kroah-Hartman --- drivers/infiniband/core/sysfs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 5c04cfb..158a214 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -760,9 +760,9 @@ int ib_device_register_sysfs(struct ib_device *device) int i; class_dev->class = &ib_class; - class_dev->driver_data = device; class_dev->parent = device->dma_device; dev_set_name(class_dev, device->name); + dev_set_drvdata(class_dev, device); INIT_LIST_HEAD(&device->port_list); -- 1.6.3.2 From gregkh at suse.de Mon Jun 15 22:46:41 2009 From: gregkh at suse.de (Greg Kroah-Hartman) Date: Mon, 15 Jun 2009 22:46:41 -0700 Subject: [ofa-general] [PATCH 52/64] infiniband: ehca: remove driver_data direct access of struct device In-Reply-To: <20090616051351.GA23627@kroah.com> References: <20090616051351.GA23627@kroah.com> Message-ID: <1245131213-24168-52-git-send-email-gregkh@suse.de> In the near future, the driver core is going to not allow direct access to the driver_data pointer in struct device. Instead, the functions dev_get_drvdata() and dev_set_drvdata() should be used. These functions have been around since the beginning, so are backwards compatible with all older kernel versions. Cc: Sean Hefty Cc: Roland Dreier Cc: Hal Rosenstock Cc: general at lists.openfabrics.org Cc: Christoph Raisch Acked-by: Hoang-Nam Nguyen Signed-off-by: Greg Kroah-Hartman --- drivers/infiniband/hw/ehca/ehca_main.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_main.c b/drivers/infiniband/hw/ehca/ehca_main.c index 85905ab..ce4e6ef 100644 --- a/drivers/infiniband/hw/ehca/ehca_main.c +++ b/drivers/infiniband/hw/ehca/ehca_main.c @@ -636,7 +636,7 @@ static ssize_t ehca_show_##name(struct device *dev, \ struct hipz_query_hca *rblock; \ int data; \ \ - shca = dev->driver_data; \ + shca = dev_get_drvdata(dev); \ \ rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL); \ if (!rblock) { \ @@ -680,7 +680,7 @@ static ssize_t ehca_show_adapter_handle(struct device *dev, struct device_attribute *attr, char *buf) { - struct ehca_shca *shca = dev->driver_data; + struct ehca_shca *shca = dev_get_drvdata(dev); return sprintf(buf, "%llx\n", shca->ipz_hca_handle.handle); @@ -749,7 +749,7 @@ static int __devinit ehca_probe(struct of_device *dev, shca->ofdev = dev; shca->ipz_hca_handle.handle = *handle; - dev->dev.driver_data = shca; + dev_set_drvdata(&dev->dev, shca); ret = ehca_sense_attributes(shca); if (ret < 0) { @@ -878,7 +878,7 @@ probe1: static int __devexit ehca_remove(struct of_device *dev) { - struct ehca_shca *shca = dev->dev.driver_data; + struct ehca_shca *shca = dev_get_drvdata(&dev->dev); unsigned long flags; int ret; -- 1.6.3.2 From alexs at linux.vnet.ibm.com Tue Jun 16 00:35:57 2009 From: alexs at linux.vnet.ibm.com (Alexander Schmidt) Date: Tue, 16 Jun 2009 09:35:57 +0200 Subject: [ofa-general] [PATCH] EHCA: Enforce a positive guid_entry index In-Reply-To: <4A368D18.108@gmail.com> References: <4A368D18.108@gmail.com> Message-ID: <20090616093557.36113ee1@BL3D1974.boeblingen.de.ibm.com> Hi Roel, On Mon, 15 Jun 2009 20:04:08 +0200 Roel Kluin wrote: > This prevents the memcpy of a guid_entries element using a negative index. > > Signed-off-by: Roel Kluin > --- > drivers/infiniband/hw/ehca/ehca_hca.c | 2 +- > drivers/infiniband/hw/ehca/ehca_iverbs.h | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/drivers/infiniband/hw/ehca/ehca_hca.c b/drivers/infiniband/hw/ehca/ehca_hca.c > index 9209c53..1175ee1 100644 > --- a/drivers/infiniband/hw/ehca/ehca_hca.c > +++ b/drivers/infiniband/hw/ehca/ehca_hca.c > @@ -311,7 +311,7 @@ query_pkey1: > } > > int ehca_query_gid(struct ib_device *ibdev, u8 port, > - int index, union ib_gid *gid) > + unsigned index, union ib_gid *gid) > { > int ret = 0; > u64 h_ret; > diff --git a/drivers/infiniband/hw/ehca/ehca_iverbs.h b/drivers/infiniband/hw/ehca/ehca_iverbs.h > index 8f7f282..05a83bb 100644 > --- a/drivers/infiniband/hw/ehca/ehca_iverbs.h > +++ b/drivers/infiniband/hw/ehca/ehca_iverbs.h > @@ -54,7 +54,7 @@ int ehca_query_sma_attr(struct ehca_shca *shca, u8 port, > > int ehca_query_pkey(struct ib_device *ibdev, u8 port, u16 index, u16 * pkey); > > -int ehca_query_gid(struct ib_device *ibdev, u8 port, int index, > +int ehca_query_gid(struct ib_device *ibdev, u8 port, unsigned index, > union ib_gid *gid); > > int ehca_modify_port(struct ib_device *ibdev, u8 port, int port_modify_mask, Thank you for checking this code. I think a better solution for this would be to change the index parameter of ehca_query_gid() from int to u8. Both functions that call ehca_query_gid() pass an u8 and this way you could also remove the if (index > 255) { ehca_err(&shca->ib_device, "Invalid index: %x.", index); return -EINVAL; } check from ehca_query_gid(). If you could send an updated patch, I would be happy to take a look again and ack it. Regards, Alex From kliteyn at dev.mellanox.co.il Tue Jun 16 01:35:20 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 16 Jun 2009 11:35:20 +0300 Subject: [ofa-general] Re: ibdiagnet: invalid bareword "pkey_entry" In-Reply-To: <20090603204041.GF26682@sgi.com> References: <20090603204041.GF26682@sgi.com> Message-ID: <4A375948.7090707@dev.mellanox.co.il> Hi Arthur, Thanks for reporting this. akepner at sgi.com wrote: > (Yevgeny, looks like you're maintaining ibutils. Please let me > know if it's otherwise.) > > We're getting this: > > # ibdiagnet > ..... > -I--------------------------------------------------- > -I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list) > -I--------------------------------------------------- > invalid bareword "pkey_entry" > in expression "-pkey_entry & 0x7fff"; > should be "$pkey_entry" or "{pkey_entry}" or "pkey_entry(...)" or ... > # This was fixed in OFED 1.4.1 and main trunk: http://www.openfabrics.org/git/?p=~kliteyn/ibutils.git;a=commit;h=2d0eaa74cc5d1b62cdfa8e2b310e28431d33b951 Are you using OFED 1.4? -- Yevgeny > when running ibdiagnet. > > Any idea why we're seeing this now, when we haven't before? > > The following fixes it, but maybe there's a better way (it's > the first line of tcl I've ever written): > > > --- /usr/lib64/ibdiagnet1.2/ibdebug.tcl 2009-06-03 13:11:09.000000000 -0700 > +++ /usr/lib64/ibdiagnet1.2/ibdebug.tcl.orig 2009-06-03 11:01:06.000000000 -0700 > @@ -3064,7 +3064,6 @@ proc GetPortPkeys {drPath portNum numPKe > inform "-E-ibdiagnet:PKeys.getPkey" $drPath $portNum $block > continue > } > - set pkeyTable [RemoveElementFromList $pkeyTable "-pkey_entry"] > foreach pkey $pkeyTable { > if {$pkey != 0} { > lappend pkeys $pkey > 2d0eaa74cc5d1b62cdfa8e2b310e28431d33b951 ibdiag/src/ibdebug.tcl | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/ibdiag/src/ibdebug.tcl b/ibdiag/src/ibdebug.tcl index a36774b..28ce3a7 100644 --- a/ibdiag/src/ibdebug.tcl +++ b/ibdiag/src/ibdebug.tcl @@ -3048,7 +3048,7 @@ proc GetPortPkeys {drPath portNum numPKeys} { continue } foreach pkey $pkeyTable { - if {$pkey != 0} { + if {[regexp {^0x[0-9a-fA-F]$} $pkey]} { lappend pkeys $pkey } } From vlad at lists.openfabrics.org Tue Jun 16 02:22:21 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 16 Jun 2009 02:22:21 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090616-0200 daily build status Message-ID: <20090616092221.61E111020312@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: patching file net/sunrpc/svcsock.c patching file net/sunrpc/xprtrdma/svc_rdma_transport.c Hunk #1 succeeded at 1049 (offset 1 line). patching file net/sunrpc/xprtrdma/transport.c patching file net/sunrpc/xprtrdma/verbs.c patching file net/sunrpc/xprtsock.c Patch rnfs_fs.patch does not apply (enforce with -f) Failed executing /usr/bin/quilt ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From dotanba at gmail.com Tue Jun 16 02:33:52 2009 From: dotanba at gmail.com (Dotan Barak) Date: Tue, 16 Jun 2009 12:33:52 +0300 Subject: [ofa-general] ofa_1_5_kernel 20090616-0200 daily build status In-Reply-To: <20090616092221.61E111020312@openfabrics.org> References: <20090616092221.61E111020312@openfabrics.org> Message-ID: <2f3bf9a60906160233m9bbbd79lb216239d55889bf8@mail.gmail.com> Hi Vlad. I think that it will be useful if the email subject will indicate if there was a failure or everything passed. thanks Dotan On Tue, Jun 16, 2009 at 12:22 PM, Vladimir Sokolovsky (Mellanox) wrote: > This email was generated automatically, please do not reply > > > git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git > git_branch: ofed_kernel_1_5 > > Common build parameters: > > Passed: > > Failed: > Build failed on i686 with linux-2.6.16 > Build failed on i686 with linux-2.6.17 > Build failed on i686 with linux-2.6.21.1 > Build failed on i686 with linux-2.6.18 > Build failed on i686 with linux-2.6.19 > Build failed on i686 with linux-2.6.22 > Build failed on i686 with linux-2.6.26 > Build failed on i686 with linux-2.6.24 > Build failed on i686 with linux-2.6.27 > Build failed on x86_64 with linux-2.6.16 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.16.21-0.8-smp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.17 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.16.43-0.3-smp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.16.60-0.21-smp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.18-1.2798.fc6 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.18 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.18-128.el5 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.18-8.el5 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.18-53.el5 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.18-93.el5 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.21.1 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.20 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.20_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.19 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.22 > Log: > patching file net/sunrpc/svcsock.c > patching file net/sunrpc/xprtrdma/svc_rdma_transport.c > Hunk #1 succeeded at 1049 (offset 1 line). > patching file net/sunrpc/xprtrdma/transport.c > patching file net/sunrpc/xprtrdma/verbs.c > patching file net/sunrpc/xprtsock.c > Patch rnfs_fs.patch does not apply (enforce with -f) > > Failed executing /usr/bin/quilt > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.22.5-31-default > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.24 > Log: > In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.25 > Log: > patching file net/sunrpc/svcsock.c > patching file net/sunrpc/xprtrdma/svc_rdma_transport.c > Hunk #1 succeeded at 1049 (offset 1 line). > patching file net/sunrpc/xprtrdma/transport.c > patching file net/sunrpc/xprtrdma/verbs.c > patching file net/sunrpc/xprtsock.c > Patch rnfs_fs.patch does not apply (enforce with -f) > > Failed executing /usr/bin/quilt > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.26 > Log: > In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.27 > Log: > In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.27_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.9-55.ELsmp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.9-42.ELsmp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.9-67.ELsmp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.16 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.16.21-0.8-default > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on x86_64 with linux-2.6.9-78.ELsmp > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.17 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.19 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.18 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.22 > Log: > patching file net/sunrpc/svcsock.c > patching file net/sunrpc/xprtrdma/svc_rdma_transport.c > Hunk #1 succeeded at 1049 (offset 1 line). > patching file net/sunrpc/xprtrdma/transport.c > patching file net/sunrpc/xprtrdma/verbs.c > patching file net/sunrpc/xprtsock.c > Patch rnfs_fs.patch does not apply (enforce with -f) > > Failed executing /usr/bin/quilt > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.23 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.23_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.21.1 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.21.1_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.25 > Log: > patching file net/sunrpc/svcsock.c > patching file net/sunrpc/xprtrdma/svc_rdma_transport.c > Hunk #1 succeeded at 1049 (offset 1 line). > patching file net/sunrpc/xprtrdma/transport.c > patching file net/sunrpc/xprtrdma/verbs.c > patching file net/sunrpc/xprtsock.c > Patch rnfs_fs.patch does not apply (enforce with -f) > > Failed executing /usr/bin/quilt > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.26 > Log: > In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.26_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ia64 with linux-2.6.24 > Log: > In file included from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.24_ia64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.16 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.16_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.17 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.17_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.18 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.18-8.el5 > Log: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > Build failed on ppc64 with linux-2.6.19 > Log: >                 from /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type > /home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type > make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 > make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 > make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090616-0200_linux-2.6.19_ppc64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From alexs at linux.vnet.ibm.com Tue Jun 16 04:45:31 2009 From: alexs at linux.vnet.ibm.com (Alexander Schmidt) Date: Tue, 16 Jun 2009 13:45:31 +0200 Subject: [ofa-general] [PATCH] EHCA: Enforce a positive guid_entry index In-Reply-To: <20090616093557.36113ee1@BL3D1974.boeblingen.de.ibm.com> References: <4A368D18.108@gmail.com> <20090616093557.36113ee1@BL3D1974.boeblingen.de.ibm.com> Message-ID: <20090616134531.01f556b9@BL3D1974.boeblingen.de.ibm.com> On Tue, 16 Jun 2009 09:35:57 +0200 Alexander Schmidt wrote: > Hi Roel, > > On Mon, 15 Jun 2009 20:04:08 +0200 > Roel Kluin wrote: > > > This prevents the memcpy of a guid_entries element using a negative index. > > > > Signed-off-by: Roel Kluin > > --- > > drivers/infiniband/hw/ehca/ehca_hca.c | 2 +- > > drivers/infiniband/hw/ehca/ehca_iverbs.h | 2 +- > > 2 files changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/infiniband/hw/ehca/ehca_hca.c b/drivers/infiniband/hw/ehca/ehca_hca.c > > index 9209c53..1175ee1 100644 > > --- a/drivers/infiniband/hw/ehca/ehca_hca.c > > +++ b/drivers/infiniband/hw/ehca/ehca_hca.c > > @@ -311,7 +311,7 @@ query_pkey1: > > } > > > > int ehca_query_gid(struct ib_device *ibdev, u8 port, > > - int index, union ib_gid *gid) > > + unsigned index, union ib_gid *gid) > > { > > int ret = 0; > > u64 h_ret; > > diff --git a/drivers/infiniband/hw/ehca/ehca_iverbs.h b/drivers/infiniband/hw/ehca/ehca_iverbs.h > > index 8f7f282..05a83bb 100644 > > --- a/drivers/infiniband/hw/ehca/ehca_iverbs.h > > +++ b/drivers/infiniband/hw/ehca/ehca_iverbs.h > > @@ -54,7 +54,7 @@ int ehca_query_sma_attr(struct ehca_shca *shca, u8 port, > > > > int ehca_query_pkey(struct ib_device *ibdev, u8 port, u16 index, u16 * pkey); > > > > -int ehca_query_gid(struct ib_device *ibdev, u8 port, int index, > > +int ehca_query_gid(struct ib_device *ibdev, u8 port, unsigned index, > > union ib_gid *gid); > > > > int ehca_modify_port(struct ib_device *ibdev, u8 port, int port_modify_mask, > > Thank you for checking this code. I think a better solution for this would be to > change the index parameter of ehca_query_gid() from int to u8. Both functions > that call ehca_query_gid() pass an u8 and this way you could also remove the > > if (index > 255) { > ehca_err(&shca->ib_device, "Invalid index: %x.", index); > return -EINVAL; > } > > check from ehca_query_gid(). > > If you could send an updated patch, I would be happy to take a look again > and ack it. I just noticed that ehca_query_gid() is an exported interface and there is a generic ib_query_gid() function which also gets an int, so we cannot accept this change. Regards, Alex From tziporet at dev.mellanox.co.il Tue Jun 16 05:34:02 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 16 Jun 2009 15:34:02 +0300 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <382A478CAD40FA4FB46605CF81FE39F4352A7D88@orsmsx507.amr.corp.intel.com> References: <4A257DBF.4000401@mellanox.co.il> <4A2B75D4.7080600@dev.mellanox.co.il> <39C75744D164D948A170E9792AF8E7CA024DCF89@exil.voltaire.com> <4A365228.4080702@dev.mellanox.co.il> <382A478CAD40FA4FB46605CF81FE39F4352A7D88@orsmsx507.amr.corp.intel.com> Message-ID: <4A37913A.7070603@mellanox.co.il> Woodruff, Robert J wrote: > I would agree, there is clearly no consensus on this one and > I do not thing there is going to be. > I think there has probably been enough discussion on this one. We can > re-discuss it at the next EWG meeting, but I would say that given > that lack of consensus, we should probably leave OFED as is, > with the MPIs included. > > > > I will bring this subject again in our next EWG meeting. I have a procedural question: Till now we got all our decisions by consensus. Since there is no consensus on this subject - should we keep things as they are today or vote on the two options? If we vote - what should be the voting procedure? Tziporet From Line.Holen at Sun.COM Tue Jun 16 06:09:18 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Tue, 16 Jun 2009 15:09:18 +0200 Subject: [ofa-general] [PATCH v2] osm_ucast_ftree.c Allow horizontal links between switches of max rank Message-ID: <4A37997E.8080708@Sun.COM> This patch makes it legal to have cross links (horizontal links) between switches at max rank. These switches do have same rank, so hop count cannot be calculated based on rank anymore. The horizontal links are treated as downlinks. Switch A has a downlink to B while B has a downlink to A. Tests on lids and also number of hops makes sure that we don't loop back and forth across the link. Signed-off-by: Frank Olaf Sem-Jacobsen Signed-off-by: Line Holen --- Changes compared to v1 includes the hop value sent as argument to set_hops_on_remote_sw(). In addition the counter_up is decremented if lft is changed due to shorter path. diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 8ed2f74..9ab4aef 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -1,4 +1,5 @@ /* + * Copyright (c) 2009 Simula Research Laboratory. All rights reserved. * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2007 Mellanox Technologies LTD. All rights reserved. @@ -1495,7 +1496,8 @@ static void fabric_make_indexing(IN ftree_fabric_t * p_ftree) p_remote_sw = p_sw->down_port_groups[i]->remote_hca_or_sw. p_sw; - if (tuple_assigned(p_remote_sw->tuple)) { + if (tuple_assigned(p_remote_sw->tuple) || + (p_sw->rank == p_remote_sw->rank)) { /* this switch has been already indexed */ continue; } @@ -1884,6 +1886,53 @@ static void set_sw_fwd_table(IN cl_map_item_t * const p_map_item, /*************************************************** ***************************************************/ +/* + * This function will count down counter_up on the remote side of + * the indicated switch and port */ +static void decrement_counter_up(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw, IN uint8_t port) +{ + ftree_port_t *p_port; + ftree_port_group_t *p_group; + ftree_sw_t *neighbor_sw; + uint8_t neighbor_port_num; + osm_node_t *neighbor_node; + int i, j; + + neighbor_node = osm_node_get_remote_node(p_sw->p_osm_sw->p_node, port, &neighbor_port_num); + if (!neighbor_node) + return; + if (osm_node_get_type(neighbor_node) != IB_NODE_TYPE_SWITCH) + return; + + neighbor_sw = fabric_get_sw_by_guid(p_ftree, + osm_node_get_node_guid(neighbor_node)); + if (!neighbor_sw) + return; + + /* Find the port on the neighbor_sw that has port_num = neighbor_port_num + This is the port that needs to decrement counter_up + This port should be part of the downgroups */ + for (i = 0; i < neighbor_sw->down_port_groups_num; i++) { + p_group = neighbor_sw->down_port_groups[i]; + for (j = 0; + j < (uint16_t) cl_ptr_vector_get_size(&p_group->ports); + j++) { + cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); + if (p_port->port_num == neighbor_port_num) { + p_port->counter_up--; + OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, + "Decremented counter_up on %s port %d\n", + neighbor_sw->p_osm_sw->p_node->print_desc, + neighbor_port_num); + return; + } + } + } +} + +/*************************************************** + ***************************************************/ /* * Function: assign-up-going-port-by-descending-down @@ -1903,12 +1952,11 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, IN ftree_sw_t * p_sw, IN ftree_sw_t * p_prev_sw, IN uint16_t target_lid, - IN uint8_t target_rank, IN boolean_t is_real_lid, IN boolean_t is_main_path, IN boolean_t is_target_a_sw, - IN uint8_t highest_rank_in_route, - IN uint16_t reverse_hops) + IN uint16_t reverse_hops, + IN uint8_t current_hops) { ftree_sw_t *p_remote_sw; uint16_t ports_num; @@ -1919,6 +1967,8 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, uint16_t j; uint16_t k; boolean_t created_route = FALSE; + uint8_t least_hops; + uint8_t old_lft_val = OSM_NO_PATH; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); @@ -1968,14 +2018,15 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, Set on the remote switch how to get to the target_lid - set LFT(target_lid) on the remote switch to the remote port */ p_remote_sw = p_group->remote_hca_or_sw.p_sw; + least_hops = sw_get_least_hops(p_remote_sw, target_lid); - if (sw_get_least_hops(p_remote_sw, target_lid) != OSM_NO_PATH) { + if ((least_hops != OSM_NO_PATH) && (least_hops <= current_hops)) { /* Loop in the fabric - we already routed the remote switch on our way UP, and now we see it again on our way DOWN */ OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Loop of lenght %d in the fabric:\n " "Switch %s (LID %u) closes loop through switch %s (LID %u)\n", - (p_remote_sw->rank - highest_rank_in_route) * 2, + current_hops, tuple_to_str(p_remote_sw->tuple), p_group->base_lid, tuple_to_str(p_sw->tuple), @@ -2019,24 +2070,22 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* setting fwd tbl port only if this is real LID */ if (is_real_lid) { + old_lft_val = p_remote_sw->p_osm_sw->new_lft[target_lid]; p_remote_sw->p_osm_sw->new_lft[target_lid] = p_min_port->remote_port_num; + OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to CA LID %u through port %u\n", + "Switch %s: set path to CA LID %u through port %u, hops %u\n", tuple_to_str(p_remote_sw->tuple), target_lid, - p_min_port->remote_port_num); + p_min_port->remote_port_num, + current_hops + 1); /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ set_hops_on_remote_sw(p_group, target_lid, - ((target_rank - - highest_rank_in_route) + - (p_remote_sw->rank - - highest_rank_in_route) + - reverse_hops * 2), - is_target_a_sw); + current_hops + 1, is_target_a_sw); } /* The number of upgoing routes is tracked in the @@ -2044,19 +2093,30 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, the upper side of the link (on switch with lower rank). Counter is promoted only if we're routing LID on the main path (whether it's a real LID or a dummy one). */ - if (is_main_path) + if (is_main_path) { p_min_port->counter_up++; + /* If we have changed a previously set lft, then we need + to reduce counter_up on the old link */ + if (old_lft_val != OSM_NO_PATH) { + OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, + "lft on %s already set, need to decrement counter " + "for old lft setup\n", + p_remote_sw->p_osm_sw->p_node->print_desc); + decrement_counter_up(p_ftree, p_remote_sw, old_lft_val); + } + } + /* Recursion step: Assign upgoing ports by stepping down, starting on REMOTE switch */ created_route |= fabric_route_upgoing_by_going_down(p_ftree, p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ - NULL, /* prev. position - NULL to mark that we went down and not up */ + p_sw, /* prev. position */ target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether the target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ is_target_a_sw, /* Wheter target lid is a switch or not */ - highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ + reverse_hops, + current_hops + 1); } /* done scanning all the down-going port groups */ @@ -2087,12 +2147,12 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, IN ftree_sw_t * p_sw, IN ftree_sw_t * p_prev_sw, IN uint16_t target_lid, - IN uint8_t target_rank, IN boolean_t is_real_lid, IN boolean_t is_main_path, IN boolean_t is_target_a_sw, IN uint16_t reverse_hop_credit, - IN uint16_t reverse_hops) + IN uint16_t reverse_hops, + IN uint8_t current_hops) { ftree_sw_t *p_remote_sw; uint16_t ports_num; @@ -2110,12 +2170,11 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ p_prev_sw, /* switch that we went up from (NULL means that we went down) */ target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this path to HCA should by tracked by counters */ is_target_a_sw, /* Wheter target lid is a switch or not */ - p_sw->rank, /* the highest visited point in the tree before going down */ - reverse_hops); /* Number of reverse_hops done up to this point */ + reverse_hops, /* Number of reverse_hops done up to this point */ + current_hops); /* recursion stop condition - if it's a root switch, */ if (p_sw->rank == 0) { @@ -2140,12 +2199,12 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + reverse_hops + 1, /* Number of reverse_hops done up to this point */ + current_hops + 1); } } @@ -2244,17 +2303,16 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, new_lft[target_lid] = p_min_port->remote_port_num; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to CA LID %u through port %u\n", + "Switch %s: set path to CA LID %u through port %u, hops %u\n", tuple_to_str(p_remote_sw->tuple), target_lid, - p_min_port->remote_port_num); + p_min_port->remote_port_num, current_hops + 1); } /* On the remote switch that is pointed by the min_group, set hops for ALL the ports in the remote group. */ set_hops_on_remote_sw(p_min_group, target_lid, - target_rank - p_remote_sw->rank + - 2 * reverse_hops, is_target_a_sw); + current_hops + 1, is_target_a_sw); } /* Recursion step: @@ -2262,12 +2320,12 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ + reverse_hops, /* Number of reverse_hops done up to this point */ + current_hops + 1); } /* we're done for the third case */ @@ -2335,22 +2393,20 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - set_hops_on_remote_sw(p_group, target_lid, - target_rank - p_remote_sw->rank + - 2 * reverse_hops, is_target_a_sw); + current_hops + 1, is_target_a_sw); /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ TRUE, /* whether the target LID is real or dummy */ FALSE, /* whether this is path to HCA that should by tracked by counters */ is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ + reverse_hops, /* Number of reverse_hops done up to this point */ + current_hops + 1); } /* If we don't have any reverse hop credits, we are done */ @@ -2374,12 +2430,12 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ TRUE, /* whether the target LID is real or dummy */ TRUE, /* whether this is path to HCA that should by tracked by counters */ is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + reverse_hops + 1, /* Number of reverse_hops done up to this point */ + current_hops + 1); } } /* ftree_fabric_route_downgoing_by_going_up() */ @@ -2451,7 +2507,7 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) p_port->port_num; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to CN LID %u through port %u\n", + "Switch %s: set path to CN LID %u through port %u, hop 1\n", tuple_to_str(p_sw->tuple), hca_lid, p_port->port_num); @@ -2464,12 +2520,12 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ hca_lid, /* LID that we're routing to */ - p_sw->rank + 1, /* rank of the LID that we're routing to */ TRUE, /* whether this HCA LID is real or dummy */ TRUE, /* whether this path to HCA should by tracked by counters */ FALSE, /* wheter target lid is a switch or not */ 0, /* Number of reverse hops allowed */ - 0); /* Number of reverse hops done yet */ + 0, /* Number of reverse hops done yet */ + 1); /* count how many real targets have been routed from this leaf switch */ routed_targets_on_leaf++; @@ -2492,12 +2548,12 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ 0, /* LID that we're routing to - ignored for dummy HCA */ - 0, /* rank of the LID that we're routing to - ignored for dummy HCA */ FALSE, /* whether this HCA LID is real or dummy */ TRUE, /* whether this path to HCA should by tracked by counters */ FALSE, /* Wheter the target LID is a switch or not */ 0, /* Number of reverse hops allowed */ - 0); /* Number of reverse hops done yet */ + 0, /* Number of reverse hops done yet */ + 1); } } } @@ -2579,12 +2635,12 @@ static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ hca_lid, /* LID that we're routing to */ - p_sw->rank + 1, /* rank of the LID that we're routing to */ TRUE, /* whether this HCA LID is real or dummy */ TRUE, /* whether this path to HCA should by tracked by counters */ FALSE, /* Wheter the target LID is a switch or not */ p_hca_port_group->is_io ? p_ftree->p_osm->subn.opt.max_reverse_hops : 0, /* Number or reverse hops allowed */ - 0); /* Number or reverse hops done yet */ + 0, /* Number or reverse hops done yet */ + 1); } /* done with all the port groups of this HCA - go to next HCA */ } @@ -2632,12 +2688,12 @@ static void fabric_route_to_switches(IN ftree_fabric_t * p_ftree) fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ p_sw->base_lid, /* LID that we're routing to */ - p_sw->rank, /* rank of the LID that we're routing to */ TRUE, /* whether the target LID is a real or dummy */ FALSE, /* whether this path to HCA should by tracked by counters */ TRUE, /* Wheter the target LID is a switch or not */ 0, /* Number of reverse hops allowed */ - 0); /* Number of reverse hops done yet */ + 0, /* Number of reverse hops done yet */ + 0); } OSM_LOG_EXIT(&p_ftree->p_osm->log); @@ -3058,7 +3114,8 @@ static int fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, p_remote_hca_or_sw = (void *)p_remote_sw; - if (abs(p_sw->rank - p_remote_sw->rank) != 1) { + if ((abs(p_sw->rank - p_remote_sw->rank) != 1) && + (p_sw->rank != p_ftree->max_switch_rank)) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB16: " "Illegal link between switches with ranks %u and %u:\n" From dledford at redhat.com Tue Jun 16 06:25:43 2009 From: dledford at redhat.com (Doug Ledford) Date: Tue, 16 Jun 2009 09:25:43 -0400 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> Message-ID: <1245158743.12931.40.camel@firewall.xsintricity.com> On Fri, 2009-06-12 at 21:22 -0700, Roland Dreier wrote: > > Valgrind replaces the libc memcpy call with a simple version that > > copies a byte at a time (in order). If libmlx4 is not built with > > --with-valgrind, valgrind considers each write an invalid write and > > spends a very long time after each write updating its error database. > > We experimented with replacing the Valgrind error database update > > with a configurable spin loop and found that if we put a delay of > > around 100,000 cycles between writes in the 'byte memcpy' when > > writing to the blueflame page, that a sent message gets > > lost/misplaced in a simple testcase with two MPI_barriers back to > > back (resulting in a hang because not all processes exit the first > > barrier). Our theory is the card sees 'byte' writes to the blueflame > > page and due to the long delay, uses the information before it is all > > written out (and thus getting wrong info). > > That makes sense. The HW documentation says that blueflame writes must > be done in aligned chunks of at least 4 bytes, so it's not surprising > that byte writes confuse the HW in some cases. Hi Roland. I'm removing the XRC patch from our kernel like we discussed (for the other people on the list, it's because the XRC API is likely to change by the time integration into the official packages are done, and we don't want to ship one API for the support now and then a different API later). This also means that I need to respin libibverbs and all driver packages. However, my deadline is *very* tight for getting this done. Any chance I could talk you into rushing the release? Today would be best, but tomorrow would work too. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jim.ryan at intel.com Tue Jun 16 07:22:25 2009 From: jim.ryan at intel.com (Ryan, Jim) Date: Tue, 16 Jun 2009 07:22:25 -0700 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <4A37913A.7070603@mellanox.co.il> Message-ID: <3F6F638B8D880340AB536D29CD4C1E194DAFEFE4@orsmsx501.amr.corp.intel.com> Tziporet, there is some guidance on this question from the Bylaws: "...ARTICLE 14. MAINTENANCE OF AND MODIFICATION TO OPENFABRICS SOFTWARE STACKS 14.1 Updates. By a majority vote, the Board or its designated Working Group may at any time update an OpenFabrics software stack for the sole purpose of making error corrections or updates (e.g. a subnet manager change, additional driver, or operating system release synchronization) that do not substantially alter or augment the functionality or capabilities of an OpenFabrics software stack. The Board or Working Group may publish such updates subject to the terms of Article 13. 14.2 Modification. Once an OpenFabrics software stack has been approved, published and released, any additions or alterations (but not updates) that substantially augment (e.g. a new transport, or operating system) the functionality and capability of such OpenFabrics software stack shall follow the procedures for approval of a Proposal in Article 12..." However, as you may infer from the text, it also sort of bucks the question back to you. If the changes are characterized as "updates", it takes a simple majority of the EWG. If the changes are "modifications", the reference to section 12 requires a 2/3 majority. We have an XWG meeting tomorrow. Would you like to discuss the question then? Thanks, Jim -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Tziporet Koren Sent: Tuesday, June 16, 2009 5:34 AM To: Woodruff, Robert J Cc: OpenFabrics General; EWG; Pavel Shamis (Pasha) Subject: Re: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? Woodruff, Robert J wrote: > I would agree, there is clearly no consensus on this one and > I do not thing there is going to be. > I think there has probably been enough discussion on this one. We can > re-discuss it at the next EWG meeting, but I would say that given > that lack of consensus, we should probably leave OFED as is, > with the MPIs included. > > > > I will bring this subject again in our next EWG meeting. I have a procedural question: Till now we got all our decisions by consensus. Since there is no consensus on this subject - should we keep things as they are today or vote on the two options? If we vote - what should be the voting procedure? Tziporet _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From jsquyres at cisco.com Tue Jun 16 07:39:01 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 16 Jun 2009 10:39:01 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <3F6F638B8D880340AB536D29CD4C1E194DAFEFE4@orsmsx501.amr.corp.intel.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFEFE4@orsmsx501.amr.corp.intel.com> Message-ID: On Jun 16, 2009, at 10:22 AM, Ryan, Jim wrote: > Tziporet, there is some guidance on this question from the Bylaws: > > "...ARTICLE 14. MAINTENANCE OF AND MODIFICATION TO OPENFABRICS > SOFTWARE STACKS > > 14.1 Updates. By a majority vote, the Board or its designated > Working Group may at any time update an OpenFabrics software stack > for the sole purpose of making error corrections or updates (e.g. a > subnet manager change, additional driver, or operating system > release synchronization) that do not substantially alter or augment > the functionality or capabilities of an OpenFabrics software stack. > The Board or Working Group may publish such updates subject to the > terms of Article 13. > Has the XWG (or some other group) been voting on all the "updates" that have gone into OFED over the years? > 14.2 Modification. Once an OpenFabrics software stack has been > approved, published and released, any additions or alterations (but > not updates) that substantially augment (e.g. a new transport, or > operating system) the functionality and capability of such > OpenFabrics software stack shall follow the procedures for approval > of a Proposal in Article 12..." > What does "an OpenFabrics software stack" mean -- is that specific to a release series such as 1.4.x? Or does it effectively mean the entire OF stack, regardless of version? I ask because the MPI change is proposed for v1.5. Since 1.5 doesn't exist yet, is it covered by 14.1 or 14.2? -- Jeff Squyres Cisco Systems From robert.j.woodruff at intel.com Tue Jun 16 07:57:20 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 16 Jun 2009 07:57:20 -0700 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: References: <3F6F638B8D880340AB536D29CD4C1E194DAFEFE4@orsmsx501.amr.corp.intel.com> Message-ID: <382A478CAD40FA4FB46605CF81FE39F43534A812@orsmsx507.amr.corp.intel.com> I think that up till now, there has not been any updates that have been so contentious that we needed a formal vote. I think that with most of the updates so far, there has been a general consensus to add things. -----Original Message----- From: Jeff Squyres [mailto:jsquyres at cisco.com] Sent: Tuesday, June 16, 2009 7:39 AM To: Ryan, Jim Cc: Tziporet Koren; Woodruff, Robert J; EWG; OpenFabrics General; Pavel Shamis (Pasha) Subject: Re: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? On Jun 16, 2009, at 10:22 AM, Ryan, Jim wrote: > Tziporet, there is some guidance on this question from the Bylaws: > > "...ARTICLE 14. MAINTENANCE OF AND MODIFICATION TO OPENFABRICS > SOFTWARE STACKS > > 14.1 Updates. By a majority vote, the Board or its designated > Working Group may at any time update an OpenFabrics software stack > for the sole purpose of making error corrections or updates (e.g. a > subnet manager change, additional driver, or operating system > release synchronization) that do not substantially alter or augment > the functionality or capabilities of an OpenFabrics software stack. > The Board or Working Group may publish such updates subject to the > terms of Article 13. > Has the XWG (or some other group) been voting on all the "updates" that have gone into OFED over the years? > 14.2 Modification. Once an OpenFabrics software stack has been > approved, published and released, any additions or alterations (but > not updates) that substantially augment (e.g. a new transport, or > operating system) the functionality and capability of such > OpenFabrics software stack shall follow the procedures for approval > of a Proposal in Article 12..." > What does "an OpenFabrics software stack" mean -- is that specific to a release series such as 1.4.x? Or does it effectively mean the entire OF stack, regardless of version? I ask because the MPI change is proposed for v1.5. Since 1.5 doesn't exist yet, is it covered by 14.1 or 14.2? -- Jeff Squyres Cisco Systems From tziporet at dev.mellanox.co.il Tue Jun 16 08:00:45 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 16 Jun 2009 18:00:45 +0300 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: <1245158743.12931.40.camel@firewall.xsintricity.com> References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> <1245158743.12931.40.camel@firewall.xsintricity.com> Message-ID: <4A37B39D.3020604@mellanox.co.il> Doug Ledford wrote: > On Fri, 2009-06-12 at 21:22 -0700, Roland Dreier wrote: > >> >> That makes sense. The HW documentation says that blueflame writes must >> be done in aligned chunks of at least 4 bytes, so it's not surprising >> that byte writes confuse the HW in some cases. >> > > Hi Roland. I'm removing the XRC patch from our kernel like we discussed > (for the other people on the list, it's because the XRC API is likely to > change by the time integration into the official packages are done, and > we don't want to ship one API for the support now and then a different > API later). This also means that I need to respin libibverbs and all > driver packages. However, my deadline is *very* tight for getting this > done. Any chance I could talk you into rushing the release? Today > would be best, but tomorrow would work too. > > Doug Have you taken the new write-combining patch in the kernel that will enable the fast pass (blueflame) in the driver to work? If not then it is not that important to have the new library Tziporet From jsquyres at cisco.com Tue Jun 16 08:03:56 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 16 Jun 2009 11:03:56 -0400 Subject: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out of OFED? In-Reply-To: <382A478CAD40FA4FB46605CF81FE39F43534A812@orsmsx507.amr.corp.intel.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFEFE4@orsmsx501.amr.corp.intel.com> <382A478CAD40FA4FB46605CF81FE39F43534A812@orsmsx507.amr.corp.intel.com> Message-ID: On Jun 16, 2009, at 10:57 AM, Woodruff, Robert J wrote: > I think that up till now, there has not been any updates that have > been so contentious that we needed a formal vote. > I think that with most of the updates so far, there has > been a general consensus to add things. > Understood, but the quoted wording of the bylaws doesn't seem to have any leeway for "consensus" vs. voting. If this is the case, then the EWG has been acting outside of its charter. If the bylaws should be changed to allow for consensus, then I assume there's a process for that and that process should be initiated. My $0.02: self-created rules are only useful if they're followed. If they're not followed, they should be changed or thrown out. :-) > > > -----Original Message----- > From: Jeff Squyres [mailto:jsquyres at cisco.com] > Sent: Tuesday, June 16, 2009 7:39 AM > To: Ryan, Jim > Cc: Tziporet Koren; Woodruff, Robert J; EWG; OpenFabrics General; > Pavel Shamis (Pasha) > Subject: Re: [ofa-general] RE: [ewg] RFC: Do we wish to take MPI out > of OFED? > > On Jun 16, 2009, at 10:22 AM, Ryan, Jim wrote: > > > Tziporet, there is some guidance on this question from the Bylaws: > > > > "...ARTICLE 14. MAINTENANCE OF AND MODIFICATION TO OPENFABRICS > > SOFTWARE STACKS > > > > 14.1 Updates. By a majority vote, the Board or its designated > > Working Group may at any time update an OpenFabrics software stack > > for the sole purpose of making error corrections or updates (e.g. a > > subnet manager change, additional driver, or operating system > > release synchronization) that do not substantially alter or augment > > the functionality or capabilities of an OpenFabrics software stack. > > The Board or Working Group may publish such updates subject to the > > terms of Article 13. > > > > Has the XWG (or some other group) been voting on all the "updates" > that have gone into OFED over the years? > > > 14.2 Modification. Once an OpenFabrics software stack has been > > approved, published and released, any additions or alterations (but > > not updates) that substantially augment (e.g. a new transport, or > > operating system) the functionality and capability of such > > OpenFabrics software stack shall follow the procedures for approval > > of a Proposal in Article 12..." > > > > What does "an OpenFabrics software stack" mean -- is that specific to > a release series such as 1.4.x? Or does it effectively mean the > entire OF stack, regardless of version? I ask because the MPI change > is proposed for v1.5. Since 1.5 doesn't exist yet, is it covered by > 14.1 or 14.2? > > -- > Jeff Squyres > Cisco Systems > > -- Jeff Squyres Cisco Systems From rdreier at cisco.com Tue Jun 16 09:12:08 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 16 Jun 2009 09:12:08 -0700 Subject: [ofa-general] Re: When is the next planned release of libmlx4? In-Reply-To: <1245158743.12931.40.camel@firewall.xsintricity.com> (Doug Ledford's message of "Tue, 16 Jun 2009 09:25:43 -0400") References: <20090611151814.95f7ae64.weiny2@llnl.gov> <200906121652.n5CGqFru004806@mail-1.llnl.gov> <1245158743.12931.40.camel@firewall.xsintricity.com> Message-ID: > Hi Roland. I'm removing the XRC patch from our kernel like we discussed > (for the other people on the list, it's because the XRC API is likely to > change by the time integration into the official packages are done, and > we don't want to ship one API for the support now and then a different > API later). This also means that I need to respin libibverbs and all > driver packages. However, my deadline is *very* tight for getting this > done. Any chance I could talk you into rushing the release? Today > would be best, but tomorrow would work too. OK, I'll try for today but tomorrow is more likely, sorry. - R. From swise at opengridcomputing.com Tue Jun 16 09:16:28 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 16 Jun 2009 11:16:28 -0500 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <20090615073408.GA27543@mtls03> References: <20090615073408.GA27543@mtls03> Message-ID: <4A37C55C.6030209@opengridcomputing.com> Hey Eli, Does this series implement UD/multicast support? I didn't see it with a quick perusal of the patches. Steve. Eli Cohen wrote: > RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over > Ethernet, providing IB capabilities for Ethernet fabrics. The packets are > standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport > headers and payload. HCA RDMAoE ports are no different than regular IB ports > from the RDMA stack perspective. > > IB subnet management and SA services are not required for RDMAoE operation; > Ethernet management practices are used instead. In Ethernet, nodes are commonly > referred to by applications by means of an IP address. RDMAoE treats IP > addresses that were assigned to the corresponding Ethernet port as GIDs, and > makes use of the IP stack to bind a destination address to the corresponding > netdevice (just as the CMA does today for IB and iWARP) and to obtain its L2 > MAC addresses. > > The RDMA Verbs API is syntactically unmodified. When referring to RDMAoE ports, > Address handles are required to contain GIDs and the L2 address fields in the > API are ignored. The Ethernet L2 information is then obtained by the > vendor-specific driver (both in kernel- and user-space) while modifying QPs to > RTR and creating address handles. > > In order to maintain application compatibility, RDMAoE implements a SA_Query > API that locally returns path records with the corresponding GIDs and the other > relevant parameters . Consequently, any CMA or native Verbs application, in > kernel or user-space, that uses path queries to obtain its address information, > will run transparently over RDMAoE with no changes. We have successfully tested > MPI, SDP, RDS, and native Verbs applications over RDMAoE without *any* changes. > > In the mlx4 driver stack, mlx4_en must be loaded and the corresponding eth > Ethernet (with support for RDMAoE) or IB, as it was already the case. > > Following is a series of 9 patches based on version 2.6.30 of the > Linux kernel. > > Signed-off-by: Eli Cohen > > > drivers/infiniband/core/addr.c | 20 ++- > drivers/infiniband/core/agent.c | 16 +- > drivers/infiniband/core/cma.c | 39 ++++- > drivers/infiniband/core/mad.c | 48 ++++-- > drivers/infiniband/core/multicast.c | 153 ++++++++++++++-- > drivers/infiniband/core/sa_query.c | 167 ++++++++++++++---- > drivers/infiniband/core/ud_header.c | 111 ++++++++++++ > drivers/infiniband/core/uverbs.h | 1 + > drivers/infiniband/core/uverbs_cmd.c | 34 ++++ > drivers/infiniband/core/uverbs_main.c | 1 + > drivers/infiniband/core/verbs.c | 18 ++ > drivers/infiniband/hw/mlx4/ah.c | 228 ++++++++++++++++++++---- > drivers/infiniband/hw/mlx4/main.c | 276 ++++++++++++++++++++++++++--- > drivers/infiniband/hw/mlx4/mlx4_ib.h | 30 +++- > drivers/infiniband/hw/mlx4/qp.c | 253 +++++++++++++++++++++----- > drivers/infiniband/ulp/ipoib/ipoib_main.c | 3 + > drivers/net/mlx4/cmd.c | 6 + > drivers/net/mlx4/en_main.c | 15 ++- > drivers/net/mlx4/en_port.c | 4 +- > drivers/net/mlx4/en_port.h | 3 +- > drivers/net/mlx4/intf.c | 20 ++ > drivers/net/mlx4/main.c | 6 + > drivers/net/mlx4/mlx4.h | 1 + > include/linux/mlx4/cmd.h | 1 + > include/linux/mlx4/device.h | 31 +++- > include/linux/mlx4/driver.h | 16 ++- > include/linux/mlx4/qp.h | 8 +- > include/rdma/ib_addr.h | 51 ++++++ > include/rdma/ib_pack.h | 26 +++ > include/rdma/ib_user_verbs.h | 21 ++- > include/rdma/ib_verbs.h | 22 +++ > 31 files changed, 1419 insertions(+), 210 deletions(-) > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From akepner at sgi.com Tue Jun 16 09:09:46 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Tue, 16 Jun 2009 09:09:46 -0700 Subject: [ofa-general] Re: ibdiagnet: invalid bareword "pkey_entry" In-Reply-To: <4A375948.7090707@dev.mellanox.co.il> References: <20090603204041.GF26682@sgi.com> <4A375948.7090707@dev.mellanox.co.il> Message-ID: <20090616160946.GL9702@sgi.com> On Tue, Jun 16, 2009 at 11:35:20AM +0300, Yevgeny Kliteynik wrote: > > This was fixed in OFED 1.4.1 and main trunk: > > http://www.openfabrics.org/git/?p=~kliteyn/ibutils.git;a=commit;h=2d0eaa74cc5d1b62cdfa8e2b310e28431d33b951 OK, thanks for the pointer. > > Are you using OFED 1.4? Yes. -- Arthur From jsquyres at cisco.com Tue Jun 16 09:24:39 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 16 Jun 2009 12:24:39 -0400 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet -- some procedural questions In-Reply-To: <3F6F638B8D880340AB536D29CD4C1E194DAFEEFB@orsmsx501.amr.corp.intel.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFEEFB@orsmsx501.amr.corp.intel.com> Message-ID: <65C4A918-E9E1-4E60-B0D1-BFB85E9622C4@cisco.com> On Jun 15, 2009, at 9:54 PM, Ryan, Jim wrote: > - There was no standards body associated with the proposal. > Clarification question on this -- what exactly does this statement mean? I ask because if you take that statement to the extreme, many things in OFED may not qualify. For example, the uDAPL software API is standardized, so I assume it's fine. The iWARP and IB hardware layers are standardized, but are there standardized software APIs? Specifically: the verbs software stack is not standardized by any standards body, is it? (what is the exact definition of "standards body", anyway -- does consensus of OFED members count?) These may be naive questions (I don't follow these APIs at all), but is the RDS API standardized? Is SRP? Are the various tools that are shipped in OFED standardized? I'm sure we don't want to take all these things out of OFED, but some clarification of the rules might be useful. -- Jeff Squyres Cisco Systems From sean.hefty at intel.com Tue Jun 16 09:32:25 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 16 Jun 2009 09:32:25 -0700 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <20090615073408.GA27543@mtls03> References: <20090615073408.GA27543@mtls03> Message-ID: <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> >RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over >Ethernet, providing IB capabilities for Ethernet fabrics. The packets are >standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport >headers and payload. HCA RDMAoE ports are no different than regular IB ports >from the RDMA stack perspective. I would refer to this as IBoE, not RDMAoE. The RDMA stack should see these ports different than regular IB HCA ports. There are a lot of differences that should not simply be hidden or incorrectly assumed: QP0, QoS, multiple paths, routing(?), no SA, etc. >IB subnet management and SA services are not required for RDMAoE operation; Then I would not try to emulate it at all. As Hal mentioned in a separate post, there are too many ways to interact with the SA that an emulation won't cover. >Ethernet management practices are used instead. In Ethernet, nodes are commonly >referred to by applications by means of an IP address. RDMAoE treats IP >addresses that were assigned to the corresponding Ethernet port as GIDs, and >makes use of the IP stack to bind a destination address to the corresponding >netdevice (just as the CMA does today for IB and iWARP) and to obtain its L2 >MAC addresses. Is the actual L3 address an IP address, or just an encoded IP address in an IBoE L3 address? What L3 protocol is being used and will it interoperate with some peer L3 protocol (IP or IB)? - Sean From sean.hefty at intel.com Tue Jun 16 09:35:21 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 16 Jun 2009 09:35:21 -0700 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID --> MAC translations In-Reply-To: <20090615073431.GC27543@mtls03> References: <20090615073431.GC27543@mtls03> Message-ID: >A few support functions are added to allow the translation from GID to MAC >which is required by hw drivers supporting RDMAoE. Why not just use IP to MAC calls? Or use the MAC as the GUID? Do the GIDs follow the IB GID format? From jim.ryan at intel.com Tue Jun 16 09:37:09 2009 From: jim.ryan at intel.com (Ryan, Jim) Date: Tue, 16 Jun 2009 09:37:09 -0700 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet -- some procedural questions In-Reply-To: <65C4A918-E9E1-4E60-B0D1-BFB85E9622C4@cisco.com> Message-ID: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> Jeff, thanks, I'll try to keep this as brief as possible and offer more offline if anyone wants it: - The bylaws state the requirement for a standards body to be associated with a proposal. It contains some examples, including the IETF and the IBTA (and others) - I think it's fair to say there's some ambiguity around what a "proposal" would consist of. That's part of the discussion I've tried to get started in the past, unsuccessfully - The bylaws also say the Board can ID other such bodies as the need is perceived - My best understanding is the IBTA is developing a spec that addresses this area. Work has started, but I don't have a timeline for delivery. An obvious concern to me is having code submitted prior to a spec being agreed Jim -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Jeff Squyres Sent: Tuesday, June 16, 2009 9:25 AM To: OpenFabrics EWG; OpenFabrics General Subject: Re: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet -- some procedural questions On Jun 15, 2009, at 9:54 PM, Ryan, Jim wrote: > - There was no standards body associated with the proposal. > Clarification question on this -- what exactly does this statement mean? I ask because if you take that statement to the extreme, many things in OFED may not qualify. For example, the uDAPL software API is standardized, so I assume it's fine. The iWARP and IB hardware layers are standardized, but are there standardized software APIs? Specifically: the verbs software stack is not standardized by any standards body, is it? (what is the exact definition of "standards body", anyway -- does consensus of OFED members count?) These may be naive questions (I don't follow these APIs at all), but is the RDS API standardized? Is SRP? Are the various tools that are shipped in OFED standardized? I'm sure we don't want to take all these things out of OFED, but some clarification of the rules might be useful. -- Jeff Squyres Cisco Systems _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From sean.hefty at intel.com Tue Jun 16 09:54:34 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 16 Jun 2009 09:54:34 -0700 Subject: [ofa-general] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: <20090615073442.GD27543@mtls03> References: <20090615073442.GD27543@mtls03> Message-ID: <106107653A20408FB313639BF9294DD8@amr.corp.intel.com> >Since RDMAoE is using Ethernet there is no need for QP0. This patch will create >only QP1 for RDMAoE ports. Which modules will use QP1 and for what purpose? I see sa_query/multicast, but there's not an actual SA. I'm guessing that the ib_cm works without changes. To clarify, do all IBoE packets carry a GRH? From sean.hefty at intel.com Tue Jun 16 10:27:33 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 16 Jun 2009 10:27:33 -0700 Subject: [ofa-general] [PATCH 4/9] ib_core: Add RDMAoE SA support In-Reply-To: <20090615073450.GE27543@mtls03> References: <20090615073450.GE27543@mtls03> Message-ID: >diff --git a/drivers/infiniband/core/multicast.c >b/drivers/infiniband/core/multicast.c >index 107f170..2417f6b 100644 >--- a/drivers/infiniband/core/multicast.c >+++ b/drivers/infiniband/core/multicast.c >@@ -488,6 +488,36 @@ retest: > } > } > >+struct eth_work { >+ struct work_struct work; >+ struct mcast_member *member; >+ struct ib_device *device; >+ u8 port_num; >+}; >+ >+static void eth_mcast_work_handler(struct work_struct *work) >+{ >+ struct eth_work *w = container_of(work, struct eth_work, work); >+ int err; >+ struct ib_port_attr port_attr; >+ int status = 0; >+ >+ err = ib_query_port(w->device, w->port_num, &port_attr); >+ if (err) >+ status = err; >+ else if (port_attr.state != IB_PORT_ACTIVE) >+ status = -EAGAIN; >+ >+ w->member->multicast.rec.qkey = cpu_to_be32(0xc2c); How can a user control this? An app needs the same qkey for unicast traffic. >+ atomic_inc(&w->member->refcount); This needs to be moved below... >+ err = w->member->multicast.callback(status, &w->member->multicast); >+ deref_member(w->member); >+ if (err) >+ ib_sa_free_multicast(&w->member->multicast); >+ >+ kfree(w); >+} >+ > /* > * Fail a join request if it is still active - at the head of the pending >queue. > */ >@@ -586,21 +616,14 @@ found: > return group; > } > >-/* >- * We serialize all join requests to a single group to make our lives much >- * easier. Otherwise, two users could try to join the same group >- * simultaneously, with different configurations, one could leave while the >- * join is in progress, etc., which makes locking around error recovery >- * difficult. >- */ >-struct ib_sa_multicast * >-ib_sa_join_multicast(struct ib_sa_client *client, >- struct ib_device *device, u8 port_num, >- struct ib_sa_mcmember_rec *rec, >- ib_sa_comp_mask comp_mask, gfp_t gfp_mask, >- int (*callback)(int status, >- struct ib_sa_multicast *multicast), >- void *context) >+static struct ib_sa_multicast * >+ib_join_multicast(struct ib_sa_client *client, >+ struct ib_device *device, u8 port_num, >+ struct ib_sa_mcmember_rec *rec, >+ ib_sa_comp_mask comp_mask, gfp_t gfp_mask, >+ int (*callback)(int status, >+ struct ib_sa_multicast *multicast), >+ void *context) > { > struct mcast_device *dev; > struct mcast_member *member; >@@ -647,9 +670,81 @@ err: > kfree(member); > return ERR_PTR(ret); > } >+ >+static struct ib_sa_multicast * >+eth_join_multicast(struct ib_sa_client *client, >+ struct ib_device *device, u8 port_num, >+ struct ib_sa_mcmember_rec *rec, >+ ib_sa_comp_mask comp_mask, gfp_t gfp_mask, >+ int (*callback)(int status, >+ struct ib_sa_multicast *multicast), >+ void *context) >+{ >+ struct mcast_device *dev; >+ struct eth_work *w; >+ struct mcast_member *member; >+ int err; >+ >+ dev = ib_get_client_data(device, &mcast_client); >+ if (!dev) >+ return ERR_PTR(-ENODEV); >+ >+ member = kzalloc(sizeof *member, gfp_mask); >+ if (!member) >+ return ERR_PTR(-ENOMEM); >+ >+ w = kzalloc(sizeof *w, gfp_mask); >+ if (!w) { >+ err = -ENOMEM; >+ goto out1; >+ } >+ w->member = member; >+ w->device = device; >+ w->port_num = port_num; >+ >+ member->multicast.context = context; >+ member->multicast.callback = callback; >+ member->client = client; >+ member->multicast.rec.mgid = rec->mgid; >+ init_completion(&member->comp); >+ atomic_set(&member->refcount, 1); >+ >+ ib_sa_client_get(client); >+ INIT_WORK(&w->work, eth_mcast_work_handler); >+ queue_work(mcast_wq, &w->work); >+ >+ return &member->multicast; The user could leave/destroy the multicast join request before the queued work item runs. We need to hold an additional reference on the member until the work item completes. >+ >+out1: >+ kfree(member); >+ return ERR_PTR(err); >+} >+ >+/* >+ * We serialize all join requests to a single group to make our lives much >+ * easier. Otherwise, two users could try to join the same group >+ * simultaneously, with different configurations, one could leave while the >+ * join is in progress, etc., which makes locking around error recovery >+ * difficult. >+ */ >+struct ib_sa_multicast * >+ib_sa_join_multicast(struct ib_sa_client *client, >+ struct ib_device *device, u8 port_num, >+ struct ib_sa_mcmember_rec *rec, >+ ib_sa_comp_mask comp_mask, gfp_t gfp_mask, >+ int (*callback)(int status, >+ struct ib_sa_multicast *multicast), >+ void *context) >+{ >+ return ib_get_port_link_type(device, port_num) == PORT_LINK_IB ? >+ ib_join_multicast(client, device, port_num, rec, comp_mask, >+ gfp_mask, callback, context) : >+ eth_join_multicast(client, device, port_num, rec, comp_mask, >+ gfp_mask, callback, context); >+} > EXPORT_SYMBOL(ib_sa_join_multicast); There's substantial differences in functionality between an IB multicast group and the Ethernet group. I would rather see the differences hidden by the rdma_cm, than the IB SA module. >diff --git a/drivers/infiniband/core/sa_query.c >b/drivers/infiniband/core/sa_query.c >index 1865049..7bf9b5c 100644 >--- a/drivers/infiniband/core/sa_query.c >+++ b/drivers/infiniband/core/sa_query.c >@@ -45,6 +45,7 @@ There's not a userspace interface into the sa_query module. How will those apps work, or apps that send MADs on QPs other than QP1? - Sean From jsquyres at cisco.com Tue Jun 16 11:24:26 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 16 Jun 2009 14:24:26 -0400 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions In-Reply-To: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> Message-ID: <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> Understood -- and I'm not debating these questions in light of RDMAoE/ IBoE/whateveroE. I'm asking how tightly these questions have been applied to all the other stacks in OFED. For example, RDS, SRP, and verbs itself. Are any of these published/standardized APIs other than the fact that they're in OFED? What does it mean to have a standard body "associated" with a proposal? Does "associated" mean that the proposal/API is a published standard in that standards body? Or some weaker definition? On Jun 16, 2009, at 12:37 PM, Ryan, Jim wrote: > Jeff, thanks, I'll try to keep this as brief as possible and offer > more offline if anyone wants it: > - The bylaws state the requirement for a standards body to be > associated with a proposal. It contains some examples, including the > IETF and the IBTA (and others) > > - I think it's fair to say there's some ambiguity around what a > "proposal" would consist of. That's part of the discussion I've > tried to get started in the past, unsuccessfully > > - The bylaws also say the Board can ID other such bodies as the need > is perceived > > - My best understanding is the IBTA is developing a spec that > addresses this area. Work has started, but I don't have a timeline > for delivery. > > An obvious concern to me is having code submitted prior to a spec > being agreed > > Jim > > -----Original Message----- > From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org > ] On Behalf Of Jeff Squyres > Sent: Tuesday, June 16, 2009 9:25 AM > To: OpenFabrics EWG; OpenFabrics General > Subject: Re: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over > Ethernet -- some procedural questions > > On Jun 15, 2009, at 9:54 PM, Ryan, Jim wrote: > > > - There was no standards body associated with the proposal. > > > > > Clarification question on this -- what exactly does this statement > mean? I ask because if you take that statement to the extreme, many > things in OFED may not qualify. > > For example, the uDAPL software API is standardized, so I assume it's > fine. The iWARP and IB hardware layers are standardized, but are > there standardized software APIs? Specifically: the verbs software > stack is not standardized by any standards body, is it? (what is the > exact definition of "standards body", anyway -- does consensus of OFED > members count?) > > These may be naive questions (I don't follow these APIs at all), but > is the RDS API standardized? Is SRP? Are the various tools that are > shipped in OFED standardized? > > I'm sure we don't want to take all these things out of OFED, but some > clarification of the rules might be useful. > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > -- Jeff Squyres Cisco Systems From sean.hefty at intel.com Tue Jun 16 11:38:03 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 16 Jun 2009 11:38:03 -0700 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions In-Reply-To: <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> Message-ID: <6D82D33BD9DF4E6AA2F29FB0C4BF6417@amr.corp.intel.com> >What does it mean to have a standard body "associated" with a >proposal? Does "associated" mean that the proposal/API is a published >standard in that standards body? Or some weaker definition? There's a significant difference between an API and a wire protocol, and even most wire protocols shouldn't be subject to any standards body. We need to be realistic here. From robert.j.woodruff at intel.com Tue Jun 16 11:42:03 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 16 Jun 2009 11:42:03 -0700 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions In-Reply-To: <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> Message-ID: <382A478CAD40FA4FB46605CF81FE39F43534AC10@orsmsx507.amr.corp.intel.com> Verbs is a software implementation of the IBTA and iWarp verbs for Linux, although there may be some features that are a superset of the IBTA and iWarp specifications, and/or were implemented first in OFA and then submitted to IBTA for standardization. SRP is an ANSI T10 standard. IPoIB is an IETF standard. There is no standard for RDS that I know of, but I think it was OFA that "asked" Oracle to submit it. The real question is does OFA want to limit itself to accepting only code that is part of some standard ? As for kernel.org, they typically do not follow such a rule, they do what they think is right for Linux, sometimes the code is associated with a standard, sometimes it is not. I think that the ofa-general list pretty much has been following the kernel.org model. I think that in any case, OFA needs to have a consistent policy, and if we allow something that is not a standard for one member, it should be allowed for all members. my 2 cents. woody -----Original Message----- From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Jeff Squyres Sent: Tuesday, June 16, 2009 11:24 AM To: Ryan, Jim Cc: OpenFabrics EWG; OpenFabrics General Subject: Re: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions Understood -- and I'm not debating these questions in light of RDMAoE/ IBoE/whateveroE. I'm asking how tightly these questions have been applied to all the other stacks in OFED. For example, RDS, SRP, and verbs itself. Are any of these published/standardized APIs other than the fact that they're in OFED? What does it mean to have a standard body "associated" with a proposal? Does "associated" mean that the proposal/API is a published standard in that standards body? Or some weaker definition? On Jun 16, 2009, at 12:37 PM, Ryan, Jim wrote: > Jeff, thanks, I'll try to keep this as brief as possible and offer > more offline if anyone wants it: > - The bylaws state the requirement for a standards body to be > associated with a proposal. It contains some examples, including the > IETF and the IBTA (and others) > > - I think it's fair to say there's some ambiguity around what a > "proposal" would consist of. That's part of the discussion I've > tried to get started in the past, unsuccessfully > > - The bylaws also say the Board can ID other such bodies as the need > is perceived > > - My best understanding is the IBTA is developing a spec that > addresses this area. Work has started, but I don't have a timeline > for delivery. > > An obvious concern to me is having code submitted prior to a spec > being agreed > > Jim > > -----Original Message----- > From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org > ] On Behalf Of Jeff Squyres > Sent: Tuesday, June 16, 2009 9:25 AM > To: OpenFabrics EWG; OpenFabrics General > Subject: Re: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over > Ethernet -- some procedural questions > > On Jun 15, 2009, at 9:54 PM, Ryan, Jim wrote: > > > - There was no standards body associated with the proposal. > > > > > Clarification question on this -- what exactly does this statement > mean? I ask because if you take that statement to the extreme, many > things in OFED may not qualify. > > For example, the uDAPL software API is standardized, so I assume it's > fine. The iWARP and IB hardware layers are standardized, but are > there standardized software APIs? Specifically: the verbs software > stack is not standardized by any standards body, is it? (what is the > exact definition of "standards body", anyway -- does consensus of OFED > members count?) > > These may be naive questions (I don't follow these APIs at all), but > is the RDS API standardized? Is SRP? Are the various tools that are > shipped in OFED standardized? > > I'm sure we don't want to take all these things out of OFED, but some > clarification of the rules might be useful. > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > -- Jeff Squyres Cisco Systems _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From arlin.r.davis at intel.com Tue Jun 16 12:17:36 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Tue, 16 Jun 2009 12:17:36 -0700 Subject: [ofa-general] [PATCH] dat 2.0 - scm: disconnect EP before cleaning up orphaned CR's during dat_ep_free Message-ID: <0288AB0BCD2048918A252656959784FC@amr.corp.intel.com> There is the possibility of dat_ep_free being called with RC CR's still in connected state. Call disconnect on the CR before marking for destroy. Destroy state change must be serialized via CR lock. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_cm.c | 3 ++- dapl/openib_scm/dapl_ib_qp.c | 3 +++ dapl/openib_scm/dapl_ib_util.h | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index d116c62..6d39a4a 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -308,8 +308,9 @@ static void dapli_cm_queue(struct ib_cm_handle *cm_ptr) /* * ACTIVE/PASSIVE: called from CR thread or consumer via ep_disconnect + * or from ep_free */ -static DAT_RETURN dapli_socket_disconnect(dp_ib_cm_handle_t cm_ptr) +DAT_RETURN dapli_socket_disconnect(dp_ib_cm_handle_t cm_ptr) { DAPL_EP *ep_ptr = cm_ptr->ep; DAT_UINT32 disc_data = htonl(0xdead); diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index f30da11..7c3c280 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -219,7 +219,10 @@ DAT_RETURN dapls_ib_qp_free(IN DAPL_IA * ia_ptr, IN DAPL_EP * ep_ptr) if (cr->ep == ep_ptr) { dapl_dbg_log(DAPL_DBG_TYPE_EP, " qp_free CR: ep %p cr %p\n", ep_ptr, cr); + dapli_socket_disconnect(cr); + dapl_os_lock(&cr->lock); cr->state = SCM_DESTROY; + dapl_os_unlock(&cr->lock); } } dapl_os_unlock(&ia_ptr->hca_ptr->ib_trans.lock); diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index e924572..ef98ab6 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -316,7 +316,7 @@ int32_t dapls_ib_release (void); void cr_thread(void *arg); int dapli_cq_thread_init(struct dapl_hca *hca_ptr); void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr); - +DAT_RETURN dapli_socket_disconnect(dp_ib_cm_handle_t cm_ptr); DAT_RETURN dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, -- 1.5.2.5 From jim.ryan at intel.com Tue Jun 16 12:31:51 2009 From: jim.ryan at intel.com (Ryan, Jim) Date: Tue, 16 Jun 2009 12:31:51 -0700 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions In-Reply-To: <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> Message-ID: <3F6F638B8D880340AB536D29CD4C1E194DAFF3ED@orsmsx501.amr.corp.intel.com> "Associated" is the strong form of the range you suggest. OFA is not in the spec creation biz. We look to other orgs to create the spec that we implement to. Again, I'm trying to tell you what our bylaws tell us. Certainly they can be changed, and I'd never claim we followed them with 100% diligence Jim -----Original Message----- From: Jeff Squyres [mailto:jsquyres at cisco.com] Sent: Tuesday, June 16, 2009 11:24 AM To: Ryan, Jim Cc: OpenFabrics EWG; OpenFabrics General Subject: Re: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions Understood -- and I'm not debating these questions in light of RDMAoE/ IBoE/whateveroE. I'm asking how tightly these questions have been applied to all the other stacks in OFED. For example, RDS, SRP, and verbs itself. Are any of these published/standardized APIs other than the fact that they're in OFED? What does it mean to have a standard body "associated" with a proposal? Does "associated" mean that the proposal/API is a published standard in that standards body? Or some weaker definition? On Jun 16, 2009, at 12:37 PM, Ryan, Jim wrote: > Jeff, thanks, I'll try to keep this as brief as possible and offer > more offline if anyone wants it: > - The bylaws state the requirement for a standards body to be > associated with a proposal. It contains some examples, including the > IETF and the IBTA (and others) > > - I think it's fair to say there's some ambiguity around what a > "proposal" would consist of. That's part of the discussion I've > tried to get started in the past, unsuccessfully > > - The bylaws also say the Board can ID other such bodies as the need > is perceived > > - My best understanding is the IBTA is developing a spec that > addresses this area. Work has started, but I don't have a timeline > for delivery. > > An obvious concern to me is having code submitted prior to a spec > being agreed > > Jim > > -----Original Message----- > From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org > ] On Behalf Of Jeff Squyres > Sent: Tuesday, June 16, 2009 9:25 AM > To: OpenFabrics EWG; OpenFabrics General > Subject: Re: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over > Ethernet -- some procedural questions > > On Jun 15, 2009, at 9:54 PM, Ryan, Jim wrote: > > > - There was no standards body associated with the proposal. > > > > > Clarification question on this -- what exactly does this statement > mean? I ask because if you take that statement to the extreme, many > things in OFED may not qualify. > > For example, the uDAPL software API is standardized, so I assume it's > fine. The iWARP and IB hardware layers are standardized, but are > there standardized software APIs? Specifically: the verbs software > stack is not standardized by any standards body, is it? (what is the > exact definition of "standards body", anyway -- does consensus of OFED > members count?) > > These may be naive questions (I don't follow these APIs at all), but > is the RDS API standardized? Is SRP? Are the various tools that are > shipped in OFED standardized? > > I'm sure we don't want to take all these things out of OFED, but some > clarification of the rules might be useful. > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > -- Jeff Squyres Cisco Systems From robert.j.woodruff at intel.com Tue Jun 16 13:46:29 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 16 Jun 2009 13:46:29 -0700 Subject: [ofa-general] Memory registration redux In-Reply-To: References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> Message-ID: <382A478CAD40FA4FB46605CF81FE39F43534AE20@orsmsx507.amr.corp.intel.com> Hay Roland, One question from my MPI guys. Looks like you have added the ability to have more than one version of the device to allow future versions, i.e., the .intf_version in the register call. struct ummunot_register_ioctl r = { .intf_version = UMMUNOT_INTF_VERSION, .start = (unsigned long) buf, .end = (unsigned long) buf + size, .user_cookie = cookie, }; I assume there will be some ioctl to allow a program to discover at runtime the version(s) of the device that are supported on a particular system ? woody -----Original Message----- From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Roland Dreier Sent: Tuesday, May 26, 2009 4:14 PM To: Jason Gunthorpe Cc: Pavel Shamis; Hans Westgaard Ry; Dontje; Lenny Verkhovsky; H??kon Bugge; Donald Kerr; OpenFabrics General; Supalov, Alexander Subject: Re: [ofa-general] Memory registration redux Here's the test program: #include #include #include #include #include #include #include #include #define UMMUNOT_INTF_VERSION 1 enum { UMMUNOT_EVENT_TYPE_INVAL = 0, UMMUNOT_EVENT_TYPE_LAST = 1, }; enum { UMMUNOT_EVENT_FLAG_HINT = 1 << 0, }; /* * If type field is INVAL, then user_cookie_counter holds the * user_cookie for the region being reported; if the HINT flag is set * then hint_start/hint_end hold the start and end of the mapping that * was invalidated. (If HINT is not set, then multiple events * invalidated parts of the registered range and hint_start/hint_end * should be ignored) * * If type is LAST, then the read operation has emptied the list of * invalidated regions, and user_cookie_counter holds the value of the * kernel's generation counter when the empty list occurred. The * other fields are not filled in for this event. */ struct ummunot_event { __u32 type; __u32 flags; __u64 hint_start; __u64 hint_end; __u64 user_cookie_counter; }; struct ummunot_register_ioctl { __u32 intf_version; /* in */ __u32 reserved1; __u64 start; /* in */ __u64 end; /* in */ __u64 user_cookie; /* in */ }; #define UMMUNOT_MAGIC 'U' #define UMMUNOT_REGISTER_REGION _IOWR(UMMUNOT_MAGIC, 1, \ struct ummunot_register_ioctl) #define UMMUNOT_UNREGISTER_REGION _IOW(UMMUNOT_MAGIC, 2, __u64) static int umn_fd; static volatile unsigned long long *umn_counter; static int umn_init(void) { umn_fd = open("/dev/ummunot", O_RDONLY); if (umn_fd < 0) { perror("open"); return 1; } umn_counter = mmap(NULL, sizeof *umn_counter, PROT_READ, MAP_SHARED, umn_fd, 0); if (umn_counter == MAP_FAILED) { perror("mmap"); return 1; } return 0; } static int umn_register(void *buf, size_t size, __u64 cookie) { struct ummunot_register_ioctl r = { .intf_version = UMMUNOT_INTF_VERSION, .start = (unsigned long) buf, .end = (unsigned long) buf + size, .user_cookie = cookie, }; if (ioctl(umn_fd, UMMUNOT_REGISTER_REGION, &r)) { perror("ioctl"); return 1; } return 0; } static int umn_unregister(__u64 cookie) { if (ioctl(umn_fd, UMMUNOT_UNREGISTER_REGION, &cookie)) { perror("ioctl"); return 1; } return 0; } int main(int argc, char *argv[]) { int page_size = sysconf(_SC_PAGESIZE); void *t; if (umn_init()) return 1; if (*umn_counter != 0) { fprintf(stderr, "counter = %lld (expected 0)\n", *umn_counter); return 1; } t = mmap(NULL, 3 * page_size, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0); if (umn_register(t, 3 * page_size, 123)) return 1; munmap(t + page_size, page_size); printf("ummunot events: %lld\n", *umn_counter); if (*umn_counter > 0) { struct ummunot_event ev[2]; int len; int i; len = read(umn_fd, &ev, sizeof ev); printf("read %d events (%d tot)\n", len / sizeof ev[0], len); for (i = 0; i < len / sizeof ev[0]; ++i) { switch (ev[i].type) { case UMMUNOT_EVENT_TYPE_INVAL: printf("[%3d]: inval cookie %lld\n", i, ev[i].user_cookie_counter); if (ev[i].flags & UMMUNOT_EVENT_FLAG_HINT) printf(" hint %llx...%lx\n", ev[i].hint_start, ev[i].hint_end); break; case UMMUNOT_EVENT_TYPE_LAST: printf("[%3d]: empty up to %lld\n", i, ev[i].user_cookie_counter); break; default: printf("[%3d]: unknown event type %d\n", i, ev[i].type); break; } } } umn_unregister(123); munmap(t, page_size); printf("ummunot events: %lld\n", *umn_counter); return 0; } _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jsquyres at cisco.com Tue Jun 16 13:54:45 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 16 Jun 2009 16:54:45 -0400 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions In-Reply-To: <382A478CAD40FA4FB46605CF81FE39F43534AC10@orsmsx507.amr.corp.intel.com> References: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> <382A478CAD40FA4FB46605CF81FE39F43534AC10@orsmsx507.amr.corp.intel.com> Message-ID: <1B1A21F2-0C8C-46AA-8D6E-BCADEB595266@cisco.com> On Jun 16, 2009, at 2:42 PM, Woodruff, Robert J wrote: > I think that in any case, OFA needs to have a consistent policy, and > if we allow something that is not a standard for one member, it > should be allowed for all members. Agreed; I think that this is my central point -- thanks for saying it succinctly! Regardless of whether OF "asked" Oracle to submit RDS or not, it's not associated with any standard (I'm not picking on RDS; I'm picking on the OF rules and selectively applying them). Therefore, either the bylaws are wrong of OF/EWG is wrong. I see several possibilities: 1. EWG needs to come into compliance with the OF bylaws 2. The bylaws need to be changed to match what EWG does (has done) 3. Throw the bylaws out (rules are useless if they don't guide what you do) 4. Some combination of 1 and 2 My $0.02. I leave it to others to figure out what to do. -- Jeff Squyres Cisco Systems From rdreier at cisco.com Tue Jun 16 13:55:08 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 16 Jun 2009 13:55:08 -0700 Subject: [ofa-general] Memory registration redux In-Reply-To: <382A478CAD40FA4FB46605CF81FE39F43534AE20@orsmsx507.amr.corp.intel.com> (Robert J. Woodruff's message of "Tue, 16 Jun 2009 13:46:29 -0700") References: <20090506214628.GM2590@obsidianresearch.com> <20090506222638.GA16280@obsidianresearch.com> <20090507000231.GB16280@obsidianresearch.com> <20090507224806.GF16280@obsidianresearch.com> <382A478CAD40FA4FB46605CF81FE39F43534AE20@orsmsx507.amr.corp.intel.com> Message-ID: > I assume there will be some ioctl to allow a program to discover at runtime > the version(s) of the device that are supported on a particular system ? Yeah, I guess. I haven't really thought through the forwards compat completely I guess. From jgunthorpe at obsidianresearch.com Tue Jun 16 13:56:13 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 16 Jun 2009 14:56:13 -0600 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> Message-ID: <20090616205613.GE18150@obsidianresearch.com> On Tue, Jun 16, 2009 at 09:32:25AM -0700, Sean Hefty wrote: > >RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over > >Ethernet, providing IB capabilities for Ethernet fabrics. The packets are > >standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport > >headers and payload. HCA RDMAoE ports are no different than regular IB ports > >from the RDMA stack perspective. > > I would refer to this as IBoE, not RDMAoE. I disagree - this is very clearly not IB. Without interop at the SA, CM and other management layers it is effectively a different protocol - that only shares the same wire format for established QPs as IB. Like FCoE the real challenge with something like this is all the surrounding management and interop/interwork with the established base (ie actual IB). As Jim said, IBTA is going to be taking this up, and I'd be really disappointed if the resulting specification was something that could not interwork with IB at all layers. > The RDMA stack should see these ports different than regular IB HCA ports. > There are a lot of differences that should not simply be hidden or incorrectly > assumed: QP0, QoS, multiple paths, routing(?), no SA, etc. .. Or it should actually do all this in some sane IB compatible way .. Jason From pgrun at systemfabricworks.com Tue Jun 16 14:07:03 2009 From: pgrun at systemfabricworks.com (Paul Grun) Date: Tue, 16 Jun 2009 14:07:03 -0700 Subject: [ewg] Re: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <20090616205613.GE18150@obsidianresearch.com> References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> <20090616205613.GE18150@obsidianresearch.com> Message-ID: <011401c9eec6$66f928b0$34eb7a10$@com> If I might chime in here...I've been working to actively squash the expression 'IBoE' or any variation that includes IB in the name. The reason is because the InfiniBand Architecture is defined as a cohesive solution that includes five layers of the OSI stack. Hence using the expression IBoE creates unnecessary confusion by implying that RDMAoE includes elements of IB from the five OSI layers, which is not an accurate description of the proposal. -Paul Grun -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Jason Gunthorpe Sent: Tuesday, June 16, 2009 1:56 PM To: Sean Hefty Cc: Roland Dreier; ewg; general-list Subject: [ewg] Re: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet On Tue, Jun 16, 2009 at 09:32:25AM -0700, Sean Hefty wrote: > >RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over > >Ethernet, providing IB capabilities for Ethernet fabrics. The packets are > >standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport > >headers and payload. HCA RDMAoE ports are no different than regular IB ports > >from the RDMA stack perspective. > > I would refer to this as IBoE, not RDMAoE. I disagree - this is very clearly not IB. Without interop at the SA, CM and other management layers it is effectively a different protocol - that only shares the same wire format for established QPs as IB. Like FCoE the real challenge with something like this is all the surrounding management and interop/interwork with the established base (ie actual IB). As Jim said, IBTA is going to be taking this up, and I'd be really disappointed if the resulting specification was something that could not interwork with IB at all layers. > The RDMA stack should see these ports different than regular IB HCA ports. > There are a lot of differences that should not simply be hidden or incorrectly > assumed: QP0, QoS, multiple paths, routing(?), no SA, etc. .. Or it should actually do all this in some sane IB compatible way .. Jason _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From rdreier at cisco.com Tue Jun 16 14:08:05 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 16 Jun 2009 14:08:05 -0700 Subject: [ofa-general] RE: [ewg] [PATCH 0/9] RDMAoE - RDMA over Ethernet-- some procedural questions In-Reply-To: <1B1A21F2-0C8C-46AA-8D6E-BCADEB595266@cisco.com> (Jeff Squyres's message of "Tue, 16 Jun 2009 16:54:45 -0400") References: <3F6F638B8D880340AB536D29CD4C1E194DAFF174@orsmsx501.amr.corp.intel.com> <1913D537-B5D1-4DC7-857F-3DD613D9188D@cisco.com> <382A478CAD40FA4FB46605CF81FE39F43534AC10@orsmsx507.amr.corp.intel.com> <1B1A21F2-0C8C-46AA-8D6E-BCADEB595266@cisco.com> Message-ID: > > I think that in any case, OFA needs to have a consistent policy, and > > if we allow something that is not a standard for one member, it > > should be allowed for all members. > Agreed; I think that this is my central point -- thanks for saying it > succinctly! Regardless of whether OF "asked" Oracle to submit RDS or > not, it's not associated with any standard (I'm not picking on RDS; > I'm picking on the OF rules and selectively applying them). > Therefore, either the bylaws are wrong of OF/EWG is wrong. The bylaw in question seems pretty silly given the lack of control or involvement that OFA has with Linux kernel development. Given that RDS is not standardized at the API or wire level and given that RDS is included in the Linux kernel, what options does the OFA have for enforcing its bylaws? Removing RDS from OFED? Once OFED has *fewer* features than the standard kernel it becomes pretty pointless; maybe the logical conclusion is that OFA should get out of the distribution business (my feelings about ending OFED are well-known I think). The same applies to IBoE; if (and that really is "if" since I don't think a conclusion about merging IBoE support into the kernel has been reached) IBoE goes into the kernel but OFED can't or won't distribute it, then the relevance of OFED becomes marginal. - R. From rdreier at cisco.com Tue Jun 16 14:13:39 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 16 Jun 2009 14:13:39 -0700 Subject: [ewg] Re: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <011401c9eec6$66f928b0$34eb7a10$@com> (Paul Grun's message of "Tue, 16 Jun 2009 14:07:03 -0700") References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> <20090616205613.GE18150@obsidianresearch.com> <011401c9eec6$66f928b0$34eb7a10$@com> Message-ID: > If I might chime in here...I've been working to actively squash the > expression 'IBoE' or any variation that includes IB in the name. The reason > is because the InfiniBand Architecture is defined as a cohesive solution > that includes five layers of the OSI stack. Hence using the expression IBoE > creates unnecessary confusion by implying that RDMAoE includes elements of > IB from the five OSI layers, which is not an accurate description of the > proposal. Seems pretty silly to me -- if I build an IB router that carries IB over the WAN on MPLS, is it not InfiniBand because it uses the wrong layer 1? Similarly the worlds seems to cope with talking about FCoE rather than SCSIoE even though FCoE does not share all the OSI layers of fibre channel. From faisal.latif at intel.com Tue Jun 16 14:22:38 2009 From: faisal.latif at intel.com (Faisal Latif) Date: Tue, 16 Jun 2009 16:22:38 -0500 Subject: [ofa-general] RDMA/nes: fix max_qp_init_rd_atom during query Message-ID: <20090616212238.GA9184@flatif-MOBL> max_qp_init_rd_atom is set in error to max_qp_wr. This was found when a test application had dapl async event error. Signed-off-by: Faisal Latif --- drivers/infiniband/hw/nes/nes_verbs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c index 64d5cfd..21e0fd3 100644 --- a/drivers/infiniband/hw/nes/nes_verbs.c +++ b/drivers/infiniband/hw/nes/nes_verbs.c @@ -654,7 +654,7 @@ static int nes_query_device(struct ib_device *ibdev, struct ib_device_attr *prop default: props->max_qp_rd_atom = 0; } - props->max_qp_init_rd_atom = props->max_qp_wr; + props->max_qp_init_rd_atom = props->max_qp_rd_atom; props->atomic_cap = IB_ATOMIC_NONE; props->max_map_per_fmr = 1; -- 1.5.3.3 From faisal.latif at intel.com Tue Jun 16 14:24:16 2009 From: faisal.latif at intel.com (Faisal Latif) Date: Tue, 16 Jun 2009 16:24:16 -0500 Subject: [ofa-general] RDMA/nes: FIN state handling under error conditions Message-ID: <20090616212416.GA6232@flatif-MOBL> During cluster testing, one qp is not closed as FIN is not handled properly when its rexmit count expires or in some cases when RST is is received after sending FIN. The reason is that cm_id does not get decremented under these conditions. Signed-off-by: Faisal Latif --- drivers/infiniband/hw/nes/nes_cm.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/nes/nes_cm.c b/drivers/infiniband/hw/nes/nes_cm.c index 11c7d66..114b802 100644 --- a/drivers/infiniband/hw/nes/nes_cm.c +++ b/drivers/infiniband/hw/nes/nes_cm.c @@ -472,6 +472,7 @@ int schedule_nes_timer(struct nes_cm_node *cm_node, struct sk_buff *skb, static void nes_retrans_expired(struct nes_cm_node *cm_node) { + struct iw_cm_id *cm_id = cm_node->cm_id; switch (cm_node->state) { case NES_CM_STATE_SYN_RCVD: case NES_CM_STATE_CLOSING: @@ -479,7 +480,9 @@ static void nes_retrans_expired(struct nes_cm_node *cm_node) break; case NES_CM_STATE_LAST_ACK: case NES_CM_STATE_FIN_WAIT1: - case NES_CM_STATE_MPAREJ_RCVD: + if (cm_node->cm_id) + cm_id->rem_ref(cm_id); + cm_node->state = NES_CM_STATE_CLOSED; send_reset(cm_node, NULL); break; default: @@ -1406,6 +1409,7 @@ static void handle_rst_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, case NES_CM_STATE_CLOSED: drop_packet(skb); break; + case NES_CM_STATE_FIN_WAIT1: case NES_CM_STATE_LAST_ACK: cm_node->cm_id->rem_ref(cm_node->cm_id); case NES_CM_STATE_TIME_WAIT: @@ -1413,8 +1417,6 @@ static void handle_rst_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, rem_ref_cm_node(cm_node->cm_core, cm_node); drop_packet(skb); break; - case NES_CM_STATE_FIN_WAIT1: - nes_debug(NES_DBG_CM, "Bad state %s[%u]\n", __func__, __LINE__); default: drop_packet(skb); break; -- 1.5.3.3 From jgunthorpe at obsidianresearch.com Tue Jun 16 14:33:09 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 16 Jun 2009 15:33:09 -0600 Subject: [ewg] Re: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <011401c9eec6$66f928b0$34eb7a10$@com> References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> <20090616205613.GE18150@obsidianresearch.com> <011401c9eec6$66f928b0$34eb7a10$@com> Message-ID: <20090616213309.GF18150@obsidianresearch.com> On Tue, Jun 16, 2009 at 02:07:03PM -0700, Paul Grun wrote: > If I might chime in here...I've been working to actively squash the > expression 'IBoE' or any variation that includes IB in the name. The reason > is because the InfiniBand Architecture is defined as a cohesive solution > that includes five layers of the OSI stack. Hence using the expression IBoE > creates unnecessary confusion by implying that RDMAoE includes elements of > IB from the five OSI layers, which is not an accurate description of the > proposal. I think stuff like this is creating alot of confusion about what the actual intent here is. I've seen your presentations at Sonoma, I've seen the stuff in IBTA - and I still don't know what the proponents of this want. Is this IBoE ala FCoE - in that gateways/routers can exist to create a heterogeneous IB fabric with Ethernet and IB physical transports? Or.. Is this ripping Chapter 9 out of IBA and mixing in a splash of Ethernet to give us something entirely new and different? This is exactly what Eli's patch set has done - and is certainly what the name RDMAoE means to me. Jason From pgrun at systemfabricworks.com Tue Jun 16 14:39:20 2009 From: pgrun at systemfabricworks.com (Paul Grun) Date: Tue, 16 Jun 2009 14:39:20 -0700 Subject: [ewg] Re: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> <20090616205613.GE18150@obsidianresearch.com> <011401c9eec6$66f928b0$34eb7a10$@com> Message-ID: <011901c9eeca$e939e5e0$bbadb1a0$@com> The bottom line is that we're talking about carrying an RDMA protocol over an Ethernet wire. It just so happens that this particular proposal uses IB's RDMA protocol. It does so for two reasons; 1. It already exists and has been validated and 2. it is relatively straightforward to implement. Your example of building an IB router for carrying IB over the WAN is an interesting one. It so happens that I've been working on just such a project; that project has moved away from calling the device an IB router in favor of calling it a WAN gateway. In any case, the analogy is imperfect since the function of the WAN gateway is to connect either two IB subnets, or in the case of the project I've been working on, it is to connect two segments of an IB subnet. In either case, it's clearly about IB. I'll concede the point about FCoE. Nevertheless, I see lots of merit, at least in this case, in using an expression to describe the technology which is crisp and unambiguous. -Paul -----Original Message----- From: Roland Dreier [mailto:rdreier at cisco.com] Sent: Tuesday, June 16, 2009 2:14 PM To: Paul Grun Cc: 'Jason Gunthorpe'; 'Sean Hefty'; 'ewg'; 'general-list' Subject: Re: [ewg] Re: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet > If I might chime in here...I've been working to actively squash the > expression 'IBoE' or any variation that includes IB in the name. The reason > is because the InfiniBand Architecture is defined as a cohesive solution > that includes five layers of the OSI stack. Hence using the expression IBoE > creates unnecessary confusion by implying that RDMAoE includes elements of > IB from the five OSI layers, which is not an accurate description of the > proposal. Seems pretty silly to me -- if I build an IB router that carries IB over the WAN on MPLS, is it not InfiniBand because it uses the wrong layer 1? Similarly the worlds seems to cope with talking about FCoE rather than SCSIoE even though FCoE does not share all the OSI layers of fibre channel. From rajouri.jammu at gmail.com Tue Jun 16 19:53:18 2009 From: rajouri.jammu at gmail.com (Rajouri Jammu) Date: Tue, 16 Jun 2009 19:53:18 -0700 Subject: [ofa-general] Compilation errors with OFED 1.4.1/ 1.4 Message-ID: <3307cdf90906161953l3d123c6bwfab1aa32a4e7d7@mail.gmail.com> Hi, I'm trying to build a kernel module using OFED 1.4.1 on SLES 10 Update 2 (2.6.16.60-0.21-smp) and I get the below compilation errors. I looked in the kernel headers but didn't find any references to things like "dma_map_single_attrs" etc. According to the OFED_release_notes SLES 10u2 is mentioned as supported. Note that I'm able to successfully compile with OFED 1.3.1 on the same system. Here are the errors. Any ideas? /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1725: warning: âstruct dma_attrsâ declared inside parameter list /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1725: warning: its scope is only this definition or declaration, which is probably not what you want /usr/src/ofa_kernel/include/rdma/ib_verbs.h: In function âib_dma_map_single_attrsâ: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1727: error: implicit declaration of function âdma_map_single_attrsâ /usr/src/ofa_kernel/include/rdma/ib_verbs.h: At top level: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1734: warning: âstruct dma_attrsâ declared inside parameter list /usr/src/ofa_kernel/include/rdma/ib_verbs.h: In function âib_dma_unmap_single_attrsâ: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1736: error: implicit declaration of function âdma_unmap_single_attrsâ /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1737: warning: âreturnâ with a value, in function returning void /usr/src/ofa_kernel/include/rdma/ib_verbs.h: At top level: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1812: warning: âstruct dma_attrsâ declared inside parameter list /usr/src/ofa_kernel/include/rdma/ib_verbs.h: In function âib_dma_map_sg_attrsâ: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1814: error: implicit declaration of function âdma_map_sg_attrsâ /usr/src/ofa_kernel/include/rdma/ib_verbs.h: At top level: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1820: warning: âstruct dma_attrsâ declared inside parameter list /usr/src/ofa_kernel/include/rdma/ib_verbs.h: In function âib_dma_unmap_sg_attrsâ: /usr/src/ofa_kernel/include/rdma/ib_verbs.h:1822: error: implicit declaration of function âdma_unmap_sg_attrsâ -------------- next part -------------- An HTML attachment was scrubbed... URL: From liranl at mellanox.co.il Wed Jun 17 01:32:42 2009 From: liranl at mellanox.co.il (Liran Liss) Date: Wed, 17 Jun 2009 11:32:42 +0300 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD030C3A22@mtlexch01.mtl.com> S.B. --Liran >RDMA over Ethernet (RDMAoE) allows running the IB transport protocol >over Ethernet, providing IB capabilities for Ethernet fabrics. The >packets are standard Ethernet frames with an Ethertype, an IB GRH, >unmodified IB transport headers and payload. HCA RDMAoE ports are no >different than regular IB ports from the RDMA stack perspective. I would refer to this as IBoE, not RDMAoE. The RDMA stack should see these ports different than regular IB HCA ports. There are a lot of differences that should not simply be hidden or incorrectly assumed: QP0, QoS, multiple paths, routing(?), no SA, etc. LL: the RDMA stack will see that the port has different link types. SLs map cleanly to VLAN user priorities. >IB subnet management and SA services are not required for RDMAoE >operation; Then I would not try to emulate it at all. As Hal mentioned in a separate post, there are too many ways to interact with the SA that an emulation won't cover. LL: you need to emulate *enough* so that typical applications don't need to worry about the link type. SA path queries is the best example. Otherwise, every RDMA application (not necessarily a CMA app) will need to have different code paths depending on the link type. >Ethernet management practices are used instead. In Ethernet, nodes are >commonly referred to by applications by means of an IP address. RDMAoE >treats IP addresses that were assigned to the corresponding Ethernet >port as GIDs, and makes use of the IP stack to bind a destination >address to the corresponding netdevice (just as the CMA does today for >IB and iWARP) and to obtain its L2 MAC addresses. Is the actual L3 address an IP address, or just an encoded IP address in an IBoE L3 address? What L3 protocol is being used and will it interoperate with some peer L3 protocol (IP or IB)? LL: RDMAoE uses GIDs that encoded IP addresses. For IPv6, this is straightforward. We use mapped address for IPv4 (::0xffff). Currently, RDMAoE is not routable, as the IB routing specs are not complete. However, nothing prohibits making it so in the future (either Eth to Eth or Eth to IB). - Sean _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From liranl at mellanox.co.il Wed Jun 17 01:41:28 2009 From: liranl at mellanox.co.il (Liran Liss) Date: Wed, 17 Jun 2009 11:41:28 +0300 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: References: <20090615073431.GC27543@mtls03> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> > Why not just use IP to MAC calls? Or use the MAC as the GUID? We do use standard OS services to map the IP addresses (that were encoded in the GID) to MACs. GIDs encode IP addresses rather than MACs to enable users to use the node names that they are used to. Specifically, we will feed in all IP addresses that were assigned to the Ethernet interface to the corresponding port GID table. This will also enable routing in the future. The only exception is IPv6 link-local addresses, which already encodes the MAC. In this case, a simple algorithmic operation extracts the MAC without requiring ARP, etc. > Do the GIDs follow the IB GID format? Yes. _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From liranl at mellanox.co.il Wed Jun 17 01:43:46 2009 From: liranl at mellanox.co.il (Liran Liss) Date: Wed, 17 Jun 2009 11:43:46 +0300 Subject: [ofa-general] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: <106107653A20408FB313639BF9294DD8@amr.corp.intel.com> References: <20090615073442.GD27543@mtls03> <106107653A20408FB313639BF9294DD8@amr.corp.intel.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD030C3A63@mtlexch01.mtl.com> > Which modules will use QP1 and for what purpose? I see sa_query/multicast, but there's not an actual SA. I'm guessing that the ib_cm works without changes. Currently, QP1 will be used only for the CM, which indeed doesn't require any changes for RDMAoE. However, we can gradually extend the support for additional QP1 services in the future. > To clarify, do all IBoE packets carry a GRH? Yes. _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From vlad at lists.openfabrics.org Wed Jun 17 02:27:44 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 17 Jun 2009 02:27:44 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090617-0200 daily build status Message-ID: <20090617092744.DD1EE1020383@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: Build failed on ia64 with linux-2.6.18 from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 Log: ---------------------------------------------------------------------------------- /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090617-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From eli at dev.mellanox.co.il Wed Jun 17 04:10:44 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 17 Jun 2009 14:10:44 +0300 Subject: [ofa-general] [PATCH] libibverbs: Add RDMAoE support In-Reply-To: <20090615163203.GF12430@obsidianresearch.com> References: <20090615134236.GA25353@mtls03> <20090615163203.GF12430@obsidianresearch.com> Message-ID: <20090617111044.GA29832@mtls03> On Mon, Jun 15, 2009 at 10:32:03AM -0600, Jason Gunthorpe wrote: > On Mon, Jun 15, 2009 at 04:42:36PM +0300, Eli Cohen wrote: > > Please use inet_pton(AF_INET6,grh,gid) for this? > I was searching for something like this... Thanks, I'll fix this. > > + if (ctx->dgid.global.interface_id) { > > + attr.ah_attr.is_global = 1; > > + attr.ah_attr.grh.hop_limit = 1; > > + attr.ah_attr.grh.dgid = ctx->dgid; > > + } > > We've been working to get rid of stuff like this for the IB router > work. Can you call into the SA path resolution stuff instead of doing > this? > Please note this is only an example which initially did not use SA services so I did not change that too. From eli at dev.mellanox.co.il Wed Jun 17 04:10:57 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 17 Jun 2009 14:10:57 +0300 Subject: [ofa-general] Re: [ewg] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: References: <20090615073442.GD27543@mtls03> Message-ID: <20090617111057.GB29832@mtls03> On Mon, Jun 15, 2009 at 02:26:42PM -0400, Hal Rosenstock wrote: > Should ib_post_send_mad return some error on QP0 sends on RDMAoE ports ? You can't send anything over QP0 because it is not created and so there are no data structs corresponding to it. > What QP1 sends are allowed ? Basically, all QP1 sends are allowed without any changes - QP1 functions as normal. However, RDMAoE will initially support only the CM. In the future, we can support additional QP1 services. > Is it only SA sends which are faked ? Yes > How > are others handled ? These questions are important to what happens to > the IB management/diag tools when invoked on an RDMAoE port. We need > to be able to handle that scenario cleanly. You should be able to to send MADs over QP1 from the kernel. I did not make any tests as for sending MADs from userspace but I can't think of a reason why this would be a problem. > > Is something similar needed in ib_mad_port_close for handling no QP0 > on RDMAoE ports in terms of destroy_mad_qp/cleanup_recv_queue calls ? > No, becuase it is handled inside destroy_mad_qp(): if (!qp_info->qp) return; From eli at dev.mellanox.co.il Wed Jun 17 04:11:07 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 17 Jun 2009 14:11:07 +0300 Subject: [ewg] RE: [ofa-general] [PATCH 4/9] ib_core: Add RDMAoE SA support In-Reply-To: References: <20090615073450.GE27543@mtls03> Message-ID: <20090617111107.GC29832@mtls03> On Tue, Jun 16, 2009 at 10:27:33AM -0700, Sean Hefty wrote: > >+ > >+ w->member->multicast.rec.qkey = cpu_to_be32(0xc2c); > > How can a user control this? An app needs the same qkey for unicast traffic. In RDMAoE, the qkey has a fixed well-known value, which will be returned both by multicast and path queries. > > >+ atomic_inc(&w->member->refcount); > > This needs to be moved below... I don't follow you - please explain. > >+static struct ib_sa_multicast * > >+eth_join_multicast(struct ib_sa_client *client, > >+ struct ib_device *device, u8 port_num, > >+ struct ib_sa_mcmember_rec *rec, > >+ ib_sa_comp_mask comp_mask, gfp_t gfp_mask, > >+ int (*callback)(int status, > >+ struct ib_sa_multicast *multicast), > >+ void *context) > >+{ > >+ struct mcast_device *dev; > >+ struct eth_work *w; > >+ struct mcast_member *member; > >+ int err; > >+ > >+ dev = ib_get_client_data(device, &mcast_client); > >+ if (!dev) > >+ return ERR_PTR(-ENODEV); > >+ > >+ member = kzalloc(sizeof *member, gfp_mask); > >+ if (!member) > >+ return ERR_PTR(-ENOMEM); > >+ > >+ w = kzalloc(sizeof *w, gfp_mask); > >+ if (!w) { > >+ err = -ENOMEM; > >+ goto out1; > >+ } > >+ w->member = member; > >+ w->device = device; > >+ w->port_num = port_num; > >+ > >+ member->multicast.context = context; > >+ member->multicast.callback = callback; > >+ member->client = client; > >+ member->multicast.rec.mgid = rec->mgid; > >+ init_completion(&member->comp); > >+ atomic_set(&member->refcount, 1); > >+ > >+ ib_sa_client_get(client); > >+ INIT_WORK(&w->work, eth_mcast_work_handler); > >+ queue_work(mcast_wq, &w->work); > >+ > >+ return &member->multicast; > > The user could leave/destroy the multicast join request before the queued work > item runs. We need to hold an additional reference on the member until the work > item completes. Yes, thanks for catching this. I'll fix and resend. > > There's substantial differences in functionality between an IB multicast group > and the Ethernet group. I would rather see the differences hidden by the > rdma_cm, than the IB SA module. > This is a question of transparency. The motivation is to enable non-cma apps that use the ib_sa_join_multicast() API to work without changes. > >diff --git a/drivers/infiniband/core/sa_query.c > >b/drivers/infiniband/core/sa_query.c > >index 1865049..7bf9b5c 100644 > >--- a/drivers/infiniband/core/sa_query.c > >+++ b/drivers/infiniband/core/sa_query.c > >@@ -45,6 +45,7 @@ > > There's not a userspace interface into the sa_query module. How will those apps > work, or apps that send MADs on QPs other than QP1? Currently, these apps won't work. Sending MADs directly on QPs other than QP1 will not work. However, we expect that a userpsace interface to the sa_query module will be implemented in the future, which will forward queries to the kernel module. Naturally, most kernel ULPs and user-level apps will use these standard interfaces instead of implementing MAD queries themselves. From eli at dev.mellanox.co.il Wed Jun 17 04:15:57 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 17 Jun 2009 14:15:57 +0300 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernety In-Reply-To: <4A37C55C.6030209@opengridcomputing.com> References: <20090615073408.GA27543@mtls03> <4A37C55C.6030209@opengridcomputing.com> Message-ID: <20090617111557.GD29832@mtls03> On Tue, Jun 16, 2009 at 11:16:28AM -0500, Steve Wise wrote: > Hey Eli, > > Does this series implement UD/multicast support? > > I didn't see it with a quick perusal of the patches. > Hi Steve, yes UD multicast is supported. Note that in this version of the pathces I use the broadcast MAC address (all FF's) for all multicast but this will be changed and GIDs will be mapped to multicast addresses. From hal.rosenstock at gmail.com Wed Jun 17 04:14:56 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 17 Jun 2009 07:14:56 -0400 Subject: [ofa-general] Re: [ewg] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: <20090617111057.GB29832@mtls03> References: <20090615073442.GD27543@mtls03> <20090617111057.GB29832@mtls03> Message-ID: On Wed, Jun 17, 2009 at 7:10 AM, Eli Cohen wrote: > On Mon, Jun 15, 2009 at 02:26:42PM -0400, Hal Rosenstock wrote: >> Should ib_post_send_mad return some error on QP0 sends on RDMAoE ports ? > You can't send anything over QP0 because it is not created and so > there are no data structs corresponding to it. Yes, I understand that's the intention but I didn't see where a MAD posted to QP0 returns an error. Does that occur ? Or is it just silently dropped ? >> What QP1 sends are allowed ? > Basically, all QP1 sends are allowed without any changes - QP1 > functions as normal. > However, RDMAoE will initially support only the CM. In the future, we > can support additional QP1 services. So what happens with other QP1 sends now ? Do they go into hyperspace and then timeout ? >> Is it only SA sends which are faked ? > Yes > >> How >> are others handled ? These questions are important to what happens to >> the IB management/diag tools when invoked on an RDMAoE port. We need >> to be able to handle that scenario cleanly. > You should be able to to send MADs over QP1 from the kernel. I did not > make any tests as for sending MADs from userspace but I can't think of > a reason why this would be a problem. There are a set of tools (infiniband-diags and ibutils) which do send MADs from userspace. I'm concerned that if someone tries these, the right thing will happen. -- Hal >> >> Is something similar needed in ib_mad_port_close for handling no QP0 >> on RDMAoE ports in terms of destroy_mad_qp/cleanup_recv_queue calls ? >> > No, becuase it is handled inside destroy_mad_qp(): > >        if (!qp_info->qp) >                return; > > From eli at dev.mellanox.co.il Wed Jun 17 04:57:51 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 17 Jun 2009 14:57:51 +0300 Subject: [ofa-general] Re: [ewg] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: References: <20090615073442.GD27543@mtls03> <20090617111057.GB29832@mtls03> Message-ID: <20090617115751.GE29832@mtls03> On Wed, Jun 17, 2009 at 07:14:56AM -0400, Hal Rosenstock wrote: > On Wed, Jun 17, 2009 at 7:10 AM, Eli Cohen wrote: > > On Mon, Jun 15, 2009 at 02:26:42PM -0400, Hal Rosenstock wrote: > >> Should ib_post_send_mad return some error on QP0 sends on RDMAoE ports ? > > You can't send anything over QP0 because it is not created and so > > there are no data structs corresponding to it. > > Yes, I understand that's the intention but I didn't see where a MAD > posted to QP0 returns an error. Does that occur ? Or is it just > silently dropped ? But you don't have a "struct ib_qp *" for QP0 that you could use to post MADs to QP0... > > >> What QP1 sends are allowed ? > > Basically, all QP1 sends are allowed without any changes - QP1 > > functions as normal. > > However, RDMAoE will initially support only the CM. In the future, we > > can support additional QP1 services. > > So what happens with other QP1 sends now ? Do they go into hyperspace > and then timeout ? SA joins and SA path queries are terminated at the driver. Otherwise, post sends on QP1 should be sent on the wire. > > There are a set of tools (infiniband-diags and ibutils) which do send > MADs from userspace. I'm concerned that if someone tries these, the > right thing will happen. > What exactly do you mean? From roel.kluin at gmail.com Wed Jun 17 07:11:23 2009 From: roel.kluin at gmail.com (Roel Kluin) Date: Wed, 17 Jun 2009 16:11:23 +0200 Subject: [ofa-general] Re: [PATCH] EHCA: Ensure that guid_entry index is positive In-Reply-To: <4A368D18.108@gmail.com> References: <4A368D18.108@gmail.com> Message-ID: <4A38F98B.5070907@gmail.com> This prevents the memcpy of a guid_entries element using a negative index. Signed-off-by: Roel Kluin --- Maybe this change then? diff --git a/drivers/infiniband/hw/ehca/ehca_hca.c b/drivers/infiniband/hw/ehca/ehca_hca.c index 9209c53..bd9414b 100644 --- a/drivers/infiniband/hw/ehca/ehca_hca.c +++ b/drivers/infiniband/hw/ehca/ehca_hca.c @@ -319,7 +319,7 @@ int ehca_query_gid(struct ib_device *ibdev, u8 port, ib_device); struct hipz_query_port *rblock; - if (index > 255) { + if (index > 255 || index < 0) { ehca_err(&shca->ib_device, "Invalid index: %x.", index); return -EINVAL; } From nicolas.morey-chaisemartin at ext.bull.net Wed Jun 17 05:46:19 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey-Chaisemartin) Date: Wed, 17 Jun 2009 14:46:19 +0200 Subject: [ofa-general] Xmit_wait counters in OpenSM Message-ID: <4A38E59B.4020000@ext.bull.net> Hi everyone, I was having a look at congestion management in IB when I realized OpenSM does not support Xmit_wait counters. Is there any good reason why? I'm ready to give it a try but I'd like to be sure there is no contraindications. Regards Nicolas From hal.rosenstock at gmail.com Wed Jun 17 06:12:13 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 17 Jun 2009 09:12:13 -0400 Subject: [ofa-general] Re: [ewg] [PATCH 3/9] ib_core: RDMAoE support only QP1 In-Reply-To: <20090617115751.GE29832@mtls03> References: <20090615073442.GD27543@mtls03> <20090617111057.GB29832@mtls03> <20090617115751.GE29832@mtls03> Message-ID: On Wed, Jun 17, 2009 at 7:57 AM, Eli Cohen wrote: > On Wed, Jun 17, 2009 at 07:14:56AM -0400, Hal Rosenstock wrote: >> On Wed, Jun 17, 2009 at 7:10 AM, Eli Cohen wrote: >> > On Mon, Jun 15, 2009 at 02:26:42PM -0400, Hal Rosenstock wrote: >> >> Should ib_post_send_mad return some error on QP0 sends on RDMAoE ports ? >> > You can't send anything over QP0 because it is not created and so >> > there are no data structs corresponding to it. >> >> Yes, I understand that's the intention but I didn't see where a MAD >> posted to QP0 returns an error. Does that occur ? Or is it just >> silently dropped ? > But you don't have a "struct ib_qp *" for QP0 that you could use to > post MADs to QP0... Understood. Doesn't an error need to be returned for certain cases of invoking ib_post_send_mad on this new port type (at least qp0 and maybe some qp1 things) ? Look at user_mad.c:umad_write calling ib_post_send_mad(). > >> >> >> What QP1 sends are allowed ? >> > Basically, all QP1 sends are allowed without any changes - QP1 >> > functions as normal. >> > However, RDMAoE will initially support only the CM. In the future, we >> > can support additional QP1 services. >> >> So what happens with other QP1 sends now ? Do they go into hyperspace >> and then timeout ? > SA joins and SA path queries are terminated at the driver. Otherwise, > post sends on QP1 should be sent on the wire. So these would just timeout if there's nothing there to consume them ? Seems better to error them out at the sender. >> >> There are a set of tools (infiniband-diags and ibutils) which do send >> MADs from userspace. I'm concerned that if someone tries these, the >> right thing will happen. >> > What exactly do you mean? opensm and any IB diag (relying on QP1 and/or QP0) should error out at the sender. >From your patches, I'm also not sure how user space sees this port type (umad needs to know about these port types). Maybe this needs ioctl exposure (and another change to umad API if it's done this way (rather than some other way which is what I think Sean was getting at in his email on node type)); sigh :-( -- Hal From yevgenyp at mellanox.co.il Wed Jun 17 07:22:01 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Wed, 17 Jun 2009 17:22:01 +0300 Subject: [ofa-general][PATCH] mlx4: ConnectX multi functional device support Message-ID: <4A38FC09.4010800@mellanox.co.il> The patch adds new device ID, Another change is that the device can open multiple physical functions. The current driver can only work with one (primary) pf. For all other functions, QUERY_FW command would fail with CMD_STAT_MULTI_FUNC_REQ error code. We should not work on those devices, but they should remain in the driver's ownership. Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/cmd.c | 5 ++++- drivers/net/mlx4/main.c | 21 +++++++++++++++++---- include/linux/mlx4/device.h | 1 + 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/drivers/net/mlx4/cmd.c b/drivers/net/mlx4/cmd.c index 2845a05..92d649c 100644 --- a/drivers/net/mlx4/cmd.c +++ b/drivers/net/mlx4/cmd.c @@ -80,7 +80,9 @@ enum { /* Bad management packet (silently discarded): */ CMD_STAT_BAD_PKT = 0x30, /* More outstanding CQEs in CQ than new CQ size: */ - CMD_STAT_BAD_SIZE = 0x40 + CMD_STAT_BAD_SIZE = 0x40, + /* Multi Function device support required: */ + CMD_STAT_MULTI_FUNC_REQ = 0x50 }; enum { @@ -128,6 +130,7 @@ static int mlx4_status_to_errno(u8 status) [CMD_STAT_LAM_NOT_PRE] = -EAGAIN, [CMD_STAT_BAD_PKT] = -EINVAL, [CMD_STAT_BAD_SIZE] = -ENOMEM, + [CMD_STAT_MULTI_FUNC_REQ] = -EACCES, }; if (status >= ARRAY_SIZE(trans_table) || diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 018348c..541243e 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -729,7 +729,10 @@ static int mlx4_init_hca(struct mlx4_dev *dev) err = mlx4_QUERY_FW(dev); if (err) { - mlx4_err(dev, "QUERY_FW command failed, aborting.\n"); + if (err == -EACCES) + mlx4_dbg(dev, "Function disabled.\n"); + else + mlx4_err(dev, "QUERY_FW command failed, aborting.\n"); return err; } @@ -1137,8 +1140,14 @@ static int __mlx4_init_one(struct pci_dev *pdev, const struct pci_device_id *id) } err = mlx4_init_hca(dev); - if (err) - goto err_cmd; + if (err) { + if (err == -EACCES) { + dev->flags |= MLX4_FLAG_NOT_PRIME; + pci_set_drvdata(pdev, dev); + return 0; + } else + goto err_cmd; + } err = mlx4_alloc_eq_table(dev); if (err) @@ -1234,6 +1243,8 @@ static void mlx4_remove_one(struct pci_dev *pdev) int p; if (dev) { + if (dev->flags & MLX4_FLAG_NOT_PRIME) + goto cmd_cleanup; mlx4_stop_sense(dev); mlx4_unregister_device(dev); @@ -1256,11 +1267,12 @@ static void mlx4_remove_one(struct pci_dev *pdev) mlx4_cleanup_uar_table(dev); mlx4_free_eq_table(dev); mlx4_close_hca(dev); - mlx4_cmd_cleanup(dev); if (dev->flags & MLX4_FLAG_MSI_X) pci_disable_msix(pdev); +cmd_cleanup: + mlx4_cmd_cleanup(dev); kfree(priv); pci_release_region(pdev, 2); pci_release_region(pdev, 0); @@ -1285,6 +1297,7 @@ static struct pci_device_id mlx4_pci_table[] = { { PCI_VDEVICE(MELLANOX, 0x6750) }, /* MT25408 "Hermon" EN 10GigE PCIe gen2 */ { PCI_VDEVICE(MELLANOX, 0x6372) }, /* MT25458 ConnectX EN 10GBASE-T 10GigE */ { PCI_VDEVICE(MELLANOX, 0x675a) }, /* MT25458 ConnectX EN 10GBASE-T+Gen2 10GigE */ + { PCI_VDEVICE(MELLANOX, 0x6764) }, /* MT26468 ConnectX EN 10GigE, PCIe, 2.0 5Gt/s */ { 0, } }; diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index ce7cc6c..637e72c 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -42,6 +42,7 @@ enum { MLX4_FLAG_MSI_X = 1 << 0, MLX4_FLAG_OLD_PORT_CMDS = 1 << 1, + MLX4_FLAG_NOT_PRIME = 1 << 2, }; enum { -- 1.6.0 From hnrose at comcast.net Wed Jun 17 06:40:46 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 17 Jun 2009 09:40:46 -0400 Subject: [ofa-general] [PATCH] opensm/osm_inform.c: For traps 64-67, use GID in DataDetails in log messages Message-ID: <20090617134046.GA10939@comcast.net> Issuer GID and LID are uninteresting for SM generated notices Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index 990f1e0..43676c2 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -561,15 +561,29 @@ ib_api_status_t osm_report_notice(IN osm_log_t * p_log, IN osm_subn_t * p_subn, /* an official Event information log */ if (ib_notice_is_generic(p_ntc)) - OSM_LOG(p_log, OSM_LOG_INFO, - "Reporting Generic Notice type:%u num:%u (%s)" - " from LID:%u GID:%s\n", - ib_notice_get_type(p_ntc), - cl_ntoh16(p_ntc->g_or_v.generic.trap_num), - ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), - cl_ntoh16(p_ntc->issuer_lid), - inet_ntop(AF_INET6, p_ntc->issuer_gid.raw, gid_str, - sizeof gid_str)); + if ((p_ntc->g_or_v.generic.trap_num == CL_HTON16(64)) || + (p_ntc->g_or_v.generic.trap_num == CL_HTON16(65)) || + (p_ntc->g_or_v.generic.trap_num == CL_HTON16(66)) || + (p_ntc->g_or_v.generic.trap_num == CL_HTON16(67))) + OSM_LOG(p_log, OSM_LOG_INFO, + "Reporting Generic Notice type:%u num:%u (%s)" + " GID:%s\n", + ib_notice_get_type(p_ntc), + cl_ntoh16(p_ntc->g_or_v.generic.trap_num), + ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), + inet_ntop(AF_INET6, + p_ntc->data_details.ntc_64_67.gid.raw, + gid_str, sizeof gid_str)); + else + OSM_LOG(p_log, OSM_LOG_INFO, + "Reporting Generic Notice type:%u num:%u (%s)" + " from LID:%u GID:%s\n", + ib_notice_get_type(p_ntc), + cl_ntoh16(p_ntc->g_or_v.generic.trap_num), + ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), + cl_ntoh16(p_ntc->issuer_lid), + inet_ntop(AF_INET6, p_ntc->issuer_gid.raw, + gid_str, sizeof gid_str)); else OSM_LOG(p_log, OSM_LOG_INFO, "Reporting Vendor Notice type:%u vend:%u dev:%u" From hal.rosenstock at gmail.com Wed Jun 17 06:47:26 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 17 Jun 2009 09:47:26 -0400 Subject: [ofa-general] Xmit_wait counters in OpenSM In-Reply-To: <4A38E59B.4020000@ext.bull.net> References: <4A38E59B.4020000@ext.bull.net> Message-ID: On Wed, Jun 17, 2009 at 8:46 AM, Nicolas Morey-Chaisemartin wrote: > Hi everyone, > > I was having a look at congestion management in IB when I realized OpenSM does not support Xmit_wait counters. > Is there any good reason why? > I'm ready to give it a try but I'd like to be sure there is no contraindications. FWIW, there's an outstanding errata not yet public against PortXmitWait. Also, there's more to congestion management than this and there are numerous errata against the CC Annex. The hope is that the fixes for these errata will be made public by the IBTA soon. -- Hal > Regards > > Nicolas > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Wed Jun 17 07:17:24 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 17 Jun 2009 10:17:24 -0400 Subject: [ofa-general] [PATCH] opensm/osm_mcast_mgr.c: In mcast_mgr_set_tbl, only return OSM_SIGNAL_DONE_PENDING if osm_req_set successful Message-ID: <20090617141724.GA12759@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index de0a8a5..44292db 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -3,6 +3,7 @@ * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -371,14 +372,14 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) cl_hton32(block_id_ho), CL_DISP_MSGID_NONE, &mad_context); - if (status != IB_SUCCESS) { + if (status == IB_SUCCESS) + signal = OSM_SIGNAL_DONE_PENDING; + else { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A02: " "Sending multicast fwd. tbl. block failed (%s)\n", ib_get_err_str(status)); } - signal = OSM_SIGNAL_DONE_PENDING; - if (++position > max_position) { position = 0; block_num++; From sashak at voltaire.com Wed Jun 17 07:38:25 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 17:38:25 +0300 Subject: [ofa-general] Re: [PATCH] Return single PR record for SubnAdmGet In-Reply-To: <4A266461.1000908@gmail.com> References: <4A266461.1000908@gmail.com> Message-ID: <20090617143825.GA6163@me> Hi Eli, On 14:54 Wed 03 Jun , Eli Dorfman (Voltaire) wrote: > Return single PR record for SubnAdmGet > > Fix to return single PathRecord for SubnAdmGet when SGID and/or DGID > are 0 in SA component mask (wildcard) > Also avoid iterating beyond requested number of paths. I agree with the idea. Couple of comments are below. > > Signed-off-by: Eli Dorfman > --- > opensm/opensm/osm_sa_path_record.c | 29 ++++++++++++++++++++--------- > 1 files changed, 20 insertions(+), 9 deletions(-) > > diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c > index 6e7d5f6..531115c 100644 > --- a/opensm/opensm/osm_sa_path_record.c > +++ b/opensm/opensm/osm_sa_path_record.c > @@ -83,6 +83,8 @@ typedef struct osm_path_parms { > boolean_t reversible; > } osm_path_parms_t; > > +#define OSM_SA_PR_MAX_NUM_PATH 127 As far as I understand this does nothing with SubnAdmGet(), but instead will limit number of returned paths for SubnAdmGetTable() when NumbPath is not selected. According to the spec SA requester shell select NumbPath when SubnAdmGetTable() is used, but now OpenSM also supports queries where NumbPath is not selected. Would we really like to change its default behavior in this case by limiting number of paths returned? > + > /********************************************************************** > **********************************************************************/ > static inline boolean_t sa_path_rec_is_tavor_port(IN const osm_port_t * p_port) > @@ -891,7 +893,7 @@ Exit: > > /********************************************************************** > **********************************************************************/ > -static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, > +static int pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, > IN const osm_madw_t * p_madw, > IN const osm_port_t * p_req_port, > IN const osm_port_t * p_src_port, > @@ -1020,14 +1022,14 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, > Preferred paths come first in OpenSM > */ > preference = 0; > - path_num = 0; > + path_num = cl_qlist_count(p_list); > > /* If SubnAdmGet, assume NumbPaths 1 (1.2 erratum) */ > if (p_sa_mad->method != IB_MAD_METHOD_GET) > if (comp_mask & IB_PR_COMPMASK_NUMBPATH) > iterations = ib_path_rec_num_path(p_pr); > else > - iterations = (uintn_t) (-1); > + iterations = OSM_SA_PR_MAX_NUM_PATH; > else > iterations = 1; > > @@ -1042,6 +1044,8 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, > comp_mask, preference); > > if (p_pr_item) { > + if (p_sa_mad->method == IB_MAD_METHOD_GET && cl_qlist_count(p_list) == 1) > + return -1; Are those lines really needed? 'path_num' is incremented anyway. And the case 'path_num == iterations' is handled after this loop. Maybe it is just simple to make this function (pr_rcv_get_port_pair_paths()) to return path_num? And then to check for non-zero status below? Sasha > cl_qlist_insert_tail(p_list, &p_pr_item->list_item); > ++path_num; > } > @@ -1105,6 +1109,8 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, > comp_mask, preference); > > if (p_pr_item) { > + if (p_sa_mad->method == IB_MAD_METHOD_GET && cl_qlist_count(p_list) == 1) > + return -1; > cl_qlist_insert_tail(p_list, &p_pr_item->list_item); > ++path_num; > } > @@ -1112,6 +1118,7 @@ static void pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, > > Exit: > OSM_LOG_EXIT(sa->p_log); > + return 0; > } > > /********************************************************************** > @@ -1350,9 +1357,10 @@ static void pr_rcv_process_world(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, > while (p_dest_port != (osm_port_t *) cl_qmap_end(p_tbl)) { > p_src_port = (osm_port_t *) cl_qmap_head(p_tbl); > while (p_src_port != (osm_port_t *) cl_qmap_end(p_tbl)) { > - pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, > + if (pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, > p_src_port, p_dest_port, > - p_dgid, comp_mask, p_list); > + p_dgid, comp_mask, p_list) < 0) > + return; > > p_src_port = > (osm_port_t *) cl_qmap_next(&p_src_port->map_item); > @@ -1393,9 +1401,10 @@ static void pr_rcv_process_half(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, > */ > p_port = (osm_port_t *) cl_qmap_head(p_tbl); > while (p_port != (osm_port_t *) cl_qmap_end(p_tbl)) { > - pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, > + if (pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, > p_src_port, p_port, p_dgid, > - comp_mask, p_list); > + comp_mask, p_list) < 0) > + goto Exit; > p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item); > } > } else { > @@ -1404,13 +1413,15 @@ static void pr_rcv_process_half(IN osm_sa_t * sa, IN const osm_madw_t * p_madw, > */ > p_port = (osm_port_t *) cl_qmap_head(p_tbl); > while (p_port != (osm_port_t *) cl_qmap_end(p_tbl)) { > - pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, > + if (pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, > p_port, p_dest_port, p_dgid, > - comp_mask, p_list); > + comp_mask, p_list) < 0) > + goto Exit; > p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item); > } > } > > +Exit: > OSM_LOG_EXIT(sa->p_log); > } > > -- > 1.5.5 > From sashak at voltaire.com Wed Jun 17 07:38:37 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 17:38:37 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_sa.c add attribute and component mask to error message In-Reply-To: <4A27BE6A.9090101@gmail.com> References: <4A27BE6A.9090101@gmail.com> Message-ID: <20090617143837.GB6163@me> On 15:30 Thu 04 Jun , Eli Dorfman (Voltaire) wrote: > add attribute and component mask to error message > about too many records for SubnAdmGet query > > Signed-off-by: Eli Dorfman Applied. Thanks. Sasha From sashak at voltaire.com Wed Jun 17 07:38:50 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 17:38:50 +0300 Subject: [ofa-general] Re: [PATCH management] Move _with_node_name_map rpm spec option from infiniband-diags to opensm. In-Reply-To: <20090604110951.3fe59f93.weiny2@llnl.gov> References: <20090604110951.3fe59f93.weiny2@llnl.gov> Message-ID: <20090617143850.GC6163@me> On 11:09 Thu 04 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Thu, 4 Jun 2009 11:08:27 -0700 > Subject: [PATCH] Move _with_node_name_map rpm spec option from infiniband-diags to opensm. > > The node name map is now part of libosmcomp. > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Wed Jun 17 07:39:00 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 17:39:00 +0300 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Determine SL properly In-Reply-To: References: <20090529153515.GA10301@comcast.net> <20090613103400.GJ10489@me> Message-ID: <20090617143900.GD6163@me> On 10:58 Sat 13 Jun , Hal Rosenstock wrote: > > > Instead of just resolving destination port's lid by guid using its port > > gid (and gid is derived from guid) > > to itself SA PR you are trying also to resolve SL which should be used > > from "self" to destination port. Right? > > Yes. > > > Assuming so. Would in this case "self" and srcport be same? > > Not sure what you mean by "same". They are different as to their > underlying type. I meant will it always describe the same port (I guess this should be since this is from where sa query will be issued)? Sasha From hal.rosenstock at gmail.com Wed Jun 17 07:42:28 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 17 Jun 2009 10:42:28 -0400 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Determine SL properly In-Reply-To: <20090617143900.GD6163@me> References: <20090529153515.GA10301@comcast.net> <20090613103400.GJ10489@me> <20090617143900.GD6163@me> Message-ID: On Wed, Jun 17, 2009 at 10:39 AM, Sasha Khapyorsky wrote: > On 10:58 Sat 13 Jun     , Hal Rosenstock wrote: >> >> > Instead of just resolving destination port's lid by guid using its port >> >>  gid (and gid is derived from guid) >> > to itself SA PR you are trying also to resolve SL which should be used >> > from "self" to destination port. Right? >> >> Yes. >> >> > Assuming so. Would in this case "self" and srcport be same? >> >> Not sure what you mean by "same". They are different as to their >> underlying type. > > I meant will it always describe the same port (I guess this should be > since this is from where sa query will be issued)? Yes, I think so but the variables are used for different purposes. -- Hal > > Sasha > From rdreier at cisco.com Wed Jun 17 09:12:45 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 09:12:45 -0700 Subject: [ofa-general][PATCH] mlx4: ConnectX multi functional device support In-Reply-To: <4A38FC09.4010800@mellanox.co.il> (Yevgeny Petrilin's message of "Wed, 17 Jun 2009 17:22:01 +0300") References: <4A38FC09.4010800@mellanox.co.il> Message-ID: > The patch adds new device ID, > Another change is that the device can open multiple physical functions. Two completely independent things, right? Please send as separate patches. - R. From sashak at voltaire.com Wed Jun 17 09:31:44 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:31:44 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_sa_node_record.c use comp mask to match by LID or GUID In-Reply-To: <4A2BD758.9080204@gmail.com> References: <4A2BD758.9080204@gmail.com> Message-ID: <20090617163144.GK6163@me> On 18:06 Sun 07 Jun , Eli Dorfman (Voltaire) wrote: > use comp mask to match by LID or GUID > > when matching node records use comp mask to filter > queries that use 0 lid or guid while mask is set > > Signed-off-by: Eli Dorfman Applied. Thanks. Sasha From Evan.Fraser at rms.com Wed Jun 17 09:33:57 2009 From: Evan.Fraser at rms.com (Evan Fraser) Date: Wed, 17 Jun 2009 17:33:57 +0100 Subject: [ofa-general] Multipath, srp_daemon help Message-ID: <9759F033B56A60469F217C49AC58E2BF085A5419@MAILUK2.rms.com> Hello folks, I'm trying to add some luns to a Linux server running SLES 10SP2 without rebooting it. I'm using OFED 1.4.1rc2, with a Voltaire dual port HBA and connecting to the luns from a DDN S2A9900. I've tried rescanning the SRP targets via: srp_daemon -o -e -n -d /dev/infiniband/umad0 srp_daemon -o -e -n -d /dev/infiniband/umad1 But multipath -ll still doesn't see the paths/luns. Could somebody please help me by pointing out what I'm overlooking/missing? Thanks very much, Evan. [cid:image001.gif at 01C9EF71.7973E0B0] Evan Fraser Senior Systems Analyst Peninsular House, 30 Monument Street London EC3R 8NB, United Kingdom Tel +44 20 7444 7860 Mobile +44 75 9024 5788 evan.fraser at rms.com www.rms.com This message and any attachments contain information that may be RMS Inc. confidential and/or privileged. If you are not the intended recipient (or authorized to receive for the intended recipient), and have received this message in error, any use, disclosure or distribution is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to the e-mail and permanently deleting the message from your computer and/or storage system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 961 bytes Desc: image001.gif URL: From sashak at voltaire.com Wed Jun 17 09:35:33 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:35:33 +0300 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Determine SL properly In-Reply-To: References: <20090529153515.GA10301@comcast.net> <20090613103400.GJ10489@me> <20090617143900.GD6163@me> Message-ID: <20090617163533.GL6163@me> On 10:42 Wed 17 Jun , Hal Rosenstock wrote: > > > > I meant will it always describe the same port (I guess this should be > > since this is from where sa query will be issued)? > > Yes, I think so but the variables are used for different purposes. Thanks for the explanation. Sasha From sashak at voltaire.com Wed Jun 17 09:35:51 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:35:51 +0300 Subject: [ofa-general] Re: [PATCH] libibmad/resolve.c: Determine SL properly In-Reply-To: <20090529153515.GA10301@comcast.net> References: <20090529153515.GA10301@comcast.net> Message-ID: <20090617163551.GM6163@me> On 11:35 Fri 29 May , Hal Rosenstock wrote: > > rather than assuming SL 0 > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Wed Jun 17 09:38:27 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:38:27 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_mesh.c: Use define rather than hard coded constant In-Reply-To: <20090522114110.GA29953@comcast.net> References: <20090522114110.GA29953@comcast.net> Message-ID: <20090617163827.GN6163@me> On 07:41 Fri 22 May , Hal Rosenstock wrote: > > Add LARGE define and use it > > Signed-off-by: Robert Pearson > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Wed Jun 17 09:39:08 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:39:08 +0300 Subject: [ofa-general] Re: [PATCH 1/2] [TRIVIAL] opensm/osm_ucast_lash.c: Fix commentary typo In-Reply-To: <20090522114234.GB29953@comcast.net> References: <20090522114234.GB29953@comcast.net> Message-ID: <20090617163908.GO6163@me> On 07:42 Fri 22 May , Hal Rosenstock wrote: > > Signed-off-by: Robert Pearson > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Wed Jun 17 09:40:26 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:40:26 +0300 Subject: [ofa-general] Re: [PATCH 2/2] opensm/osm_ucast_lash.c: Use calloc rather than malloc/memset In-Reply-To: <20090522114346.GC29953@comcast.net> References: <20090522114346.GC29953@comcast.net> Message-ID: <20090617164026.GP6163@me> On 07:43 Fri 22 May , Hal Rosenstock wrote: > > Signed-off-by: Robert Pearson > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From Evan.Fraser at rms.com Wed Jun 17 09:45:10 2009 From: Evan.Fraser at rms.com (Evan Fraser) Date: Wed, 17 Jun 2009 17:45:10 +0100 Subject: [ofa-general] Multipath, srp_daemon help Message-ID: <9759F033B56A60469F217C49AC58E2BF085A541A@MAILUK2.rms.com> Hello folks, I'm trying to add some luns to a Linux server running SLES 10SP2 without rebooting it. I'm using OFED 1.4.1rc2, with a Voltaire dual port HBA and connecting to the luns from a DDN S2A9900. I've tried rescanning the SRP targets via: srp_daemon -o -e -n -d /dev/infiniband/umad0 srp_daemon -o -e -n -d /dev/infiniband/umad1 But multipath -ll still doesn't see the paths/luns. Could somebody please help me by pointing out what I'm overlooking/missing? Thanks very much, Evan Fraser Senior Systems Analyst Peninsular House, 30 Monument Street London EC3R 8NB, United Kingdom Tel +44 20 7444 7860 Mobile +44 75 9024 5788 evan.fraser at rms.com www.rms.com This message and any attachments contain information that may be RMS Inc. confidential and/or privileged. If you are not the intended recipient (or authorized to receive for the intended recipient), and have received this message in error, any use, disclosure or distribution is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to the e-mail and permanently deleting the message from your computer and/or storage system. From sashak at voltaire.com Wed Jun 17 09:46:24 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 19:46:24 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_inform.c: For traps 64-67, use GID in DataDetails in log messages In-Reply-To: <20090617134046.GA10939@comcast.net> References: <20090617134046.GA10939@comcast.net> Message-ID: <20090617164624.GQ6163@me> On 09:40 Wed 17 Jun , Hal Rosenstock wrote: > > Issuer GID and LID are uninteresting for SM generated notices > > Signed-off-by: Hal Rosenstock > --- > diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c > index 990f1e0..43676c2 100644 > --- a/opensm/opensm/osm_inform.c > +++ b/opensm/opensm/osm_inform.c > @@ -561,15 +561,29 @@ ib_api_status_t osm_report_notice(IN osm_log_t * p_log, IN osm_subn_t * p_subn, > > /* an official Event information log */ > if (ib_notice_is_generic(p_ntc)) > - OSM_LOG(p_log, OSM_LOG_INFO, > - "Reporting Generic Notice type:%u num:%u (%s)" > - " from LID:%u GID:%s\n", > - ib_notice_get_type(p_ntc), > - cl_ntoh16(p_ntc->g_or_v.generic.trap_num), > - ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), > - cl_ntoh16(p_ntc->issuer_lid), > - inet_ntop(AF_INET6, p_ntc->issuer_gid.raw, gid_str, > - sizeof gid_str)); > + if ((p_ntc->g_or_v.generic.trap_num == CL_HTON16(64)) || > + (p_ntc->g_or_v.generic.trap_num == CL_HTON16(65)) || > + (p_ntc->g_or_v.generic.trap_num == CL_HTON16(66)) || > + (p_ntc->g_or_v.generic.trap_num == CL_HTON16(67))) > + OSM_LOG(p_log, OSM_LOG_INFO, > + "Reporting Generic Notice type:%u num:%u (%s)" > + " GID:%s\n", > + ib_notice_get_type(p_ntc), > + cl_ntoh16(p_ntc->g_or_v.generic.trap_num), > + ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), > + inet_ntop(AF_INET6, > + p_ntc->data_details.ntc_64_67.gid.raw, > + gid_str, sizeof gid_str)); > + else > + OSM_LOG(p_log, OSM_LOG_INFO, > + "Reporting Generic Notice type:%u num:%u (%s)" > + " from LID:%u GID:%s\n", > + ib_notice_get_type(p_ntc), > + cl_ntoh16(p_ntc->g_or_v.generic.trap_num), > + ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), > + cl_ntoh16(p_ntc->issuer_lid), > + inet_ntop(AF_INET6, p_ntc->issuer_gid.raw, > + gid_str, sizeof gid_str)); Maybe, but it seems easier for me to have unified log here (which is macro which will be filtered out when log level is lower) than add additional "if/else". Sasha From sashak at voltaire.com Wed Jun 17 10:03:12 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 17 Jun 2009 20:03:12 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_mcast_mgr.c: In mcast_mgr_set_tbl, only return OSM_SIGNAL_DONE_PENDING if osm_req_set successful In-Reply-To: <20090617141724.GA12759@comcast.net> References: <20090617141724.GA12759@comcast.net> Message-ID: <20090617170312.GR6163@me> On 10:17 Wed 17 Jun , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > --- > diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c > index de0a8a5..44292db 100644 > --- a/opensm/opensm/osm_mcast_mgr.c > +++ b/opensm/opensm/osm_mcast_mgr.c > @@ -3,6 +3,7 @@ > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > + * Copyright (c) 2009 HNR Consulting Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -371,14 +372,14 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) > cl_hton32(block_id_ho), CL_DISP_MSGID_NONE, > &mad_context); > > - if (status != IB_SUCCESS) { > + if (status == IB_SUCCESS) > + signal = OSM_SIGNAL_DONE_PENDING; > + else { > OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A02: " > "Sending multicast fwd. tbl. block failed (%s)\n", > ib_get_err_str(status)); > } > > - signal = OSM_SIGNAL_DONE_PENDING; > - The "signal" value doesn't matter anymore. Actually I think it would be nice to rework all "managers" to return integer 0, -1 values as success/failure status. Sasha From weiny2 at llnl.gov Wed Jun 17 10:08:12 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 17 Jun 2009 10:08:12 -0700 Subject: [ofa-general] [PATCH] iblinkinfo, ibqueryerrors: prevent core when switch is not found Message-ID: <20090617100812.52c87359.weiny2@llnl.gov> From: Ira Weiny Date: Thu, 14 May 2009 15:52:42 -0700 Subject: [PATCH] iblinkinfo, ibqueryerrors: prevent core when switch is not found If the switch is not found print nice error message instead of seg faulting Signed-off-by: Ira Weiny --- infiniband-diags/src/iblinkinfo.c | 11 +++++++++-- infiniband-diags/src/ibqueryerrors.c | 10 ++++++++-- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/infiniband-diags/src/iblinkinfo.c b/infiniband-diags/src/iblinkinfo.c index cf38ecb..367056c 100644 --- a/infiniband-diags/src/iblinkinfo.c +++ b/infiniband-diags/src/iblinkinfo.c @@ -395,11 +395,18 @@ main(int argc, char **argv) goto close_port; } - if (guid) { + if (guid_str) { ibnd_node_t *sw = ibnd_find_node_guid(fabric, guid); - print_switch(sw, NULL); + if (sw) + print_switch(sw, NULL); + else + fprintf(stderr, "Failed to find switch: %s\n", guid_str); } else if (dr_path) { ibnd_node_t *sw = ibnd_find_node_dr(fabric, dr_path); + if (sw) + print_switch(sw, NULL); + else + fprintf(stderr, "Failed to find switch: %s\n", dr_path); print_switch(sw, NULL); } else { ibnd_iter_nodes_type(fabric, print_switch, IB_NODE_SWITCH, NULL); diff --git a/infiniband-diags/src/ibqueryerrors.c b/infiniband-diags/src/ibqueryerrors.c index 525af70..999329e 100644 --- a/infiniband-diags/src/ibqueryerrors.c +++ b/infiniband-diags/src/ibqueryerrors.c @@ -445,10 +445,16 @@ main(int argc, char **argv) if (switch_guid) { ibnd_node_t *node = ibnd_find_node_guid(fabric, switch_guid); - print_node(node, NULL); + if (node) + print_node(node, NULL); + else + fprintf(stderr, "Failed to find node: %s\n", switch_guid_str); } else if (dr_path) { ibnd_node_t *node = ibnd_find_node_dr(fabric, dr_path); - print_node(node, NULL); + if (node) + print_node(node, NULL); + else + fprintf(stderr, "Failed to find node: %s\n", dr_path); } else ibnd_iter_nodes(fabric, print_node, NULL); -- 1.5.4.5 From weiny2 at llnl.gov Wed Jun 17 10:09:00 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 17 Jun 2009 10:09:00 -0700 Subject: [ofa-general] [PATCH] iblinkinfo: remove unused file pointer. Message-ID: <20090617100900.b25a0530.weiny2@llnl.gov> From: Ira Weiny Date: Thu, 14 May 2009 16:39:52 -0700 Subject: [PATCH] iblinkinfo: remove unused file pointer. Signed-off-by: Ira Weiny --- infiniband-diags/src/iblinkinfo.c | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/infiniband-diags/src/iblinkinfo.c b/infiniband-diags/src/iblinkinfo.c index 367056c..d422a2a 100644 --- a/infiniband-diags/src/iblinkinfo.c +++ b/infiniband-diags/src/iblinkinfo.c @@ -52,7 +52,6 @@ #include char *argv0 = "iblinkinfotest"; -static FILE *f; static char *node_name_map_file = NULL; static nn_map_t *node_name_map = NULL; @@ -294,8 +293,6 @@ main(int argc, char **argv) { 0 } }; - f = stdout; - argv0 = argv[0]; while (1) { @@ -357,9 +354,6 @@ main(int argc, char **argv) argc -= optind; argv += optind; - if (argc && !(f = fopen(argv[0], "w"))) - fprintf(stderr, "can't open file %s for writing", argv[0]); - ibmad_port = mad_rpc_open_port(ca, ca_port, mgmt_classes, 3); if (!ibmad_port) { fprintf(stderr, "Failed to open %s port %d", ca, ca_port); -- 1.5.4.5 From weiny2 at llnl.gov Wed Jun 17 10:10:24 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 17 Jun 2009 10:10:24 -0700 Subject: [ofa-general] [PATCH] iblinkinfo: convert to ibdiag_process_opts Message-ID: <20090617101024.cab3ea90.weiny2@llnl.gov> From: Ira Weiny Date: Thu, 14 May 2009 17:23:22 -0700 Subject: [PATCH] iblinkinfo: convert to ibdiag_process_opts Signed-off-by: Ira Weiny --- infiniband-diags/src/iblinkinfo.c | 190 +++++++++++++----------------------- 1 files changed, 69 insertions(+), 121 deletions(-) diff --git a/infiniband-diags/src/iblinkinfo.c b/infiniband-diags/src/iblinkinfo.c index d422a2a..ae6da27 100644 --- a/infiniband-diags/src/iblinkinfo.c +++ b/infiniband-diags/src/iblinkinfo.c @@ -51,12 +51,16 @@ #include #include -char *argv0 = "iblinkinfotest"; +#include "ibdiag_common.h" static char *node_name_map_file = NULL; static nn_map_t *node_name_map = NULL; -static int timeout_ms = 500; +static uint64_t guid = 0; +static char *guid_str = NULL; +static char *dr_path = NULL; +static char *from = NULL; +static int hops = 0; static int down_links_only = 0; static int line_mode = 0; @@ -223,140 +227,84 @@ print_switch(ibnd_node_t *node, void *user_data) free(remap); } -void -usage(void) +static int process_opt(void *context, int ch, char *optarg) { - fprintf(stderr, - "Usage: %s [-hclp -S -D -C -P ]\n" - " Report link speed and connection for each port of each switch which is active\n" - " -h This help message\n" - " -S output only the node specified by guid\n" - " -D print only node specified by \n" - " -f specify node to start \"from\"\n" - " -n Number of hops to include away from specified node\n" - " -d print only down links\n" - " -l (line mode) print all information for each link on each line\n" - " -p print additional switch settings (PktLifeTime,HoqLife,VLStallCount)\n" - - - " -t timeout for any single fabric query\n" - " -s show progress during scan\n" - " --node-name-map use specified node name map\n" - - " -C use selected Channel Adaptor name for queries\n" - " -P use selected channel adaptor port for queries\n" - " -g print port guids instead of node guids\n" - " --debug print debug messages\n" - " -R (this option is obsolete and does nothing)\n" - , - argv0); - exit(-1); + switch (ch) { + case 1: + node_name_map_file = strdup(optarg); + break; + case 'S': + guid_str = optarg; + guid = (uint64_t)strtoull(guid_str, 0, 0); + break; + case 'D': + dr_path = strdup(optarg); + break; + case 'f': + from = strdup(optarg); + break; + case 'n': + hops = (int)strtol(optarg, NULL, 0); + break; + case 'd': + down_links_only = 1; + break; + case 'l': + line_mode = 1; + break; + case 'p': + add_sw_settings = 1; + break; + case 'g': + print_port_guids = 1; + break; + case 'R': /* nop */ + break; + default: + return -1; + } + + return 0; } + int main(int argc, char **argv) { + int rc = 0; - char *ca = 0; - int ca_port = 0; ibnd_fabric_t *fabric = NULL; - uint64_t guid = 0; - char *guid_str = NULL; - char *dr_path = NULL; - char *from = NULL; - int hops = 0; - ib_portid_t port_id = {0}; - struct ibmad_port *ibmad_port; + ib_portid_t port_id = {0}; int mgmt_classes[3] = {IB_SMI_CLASS, IB_SMI_DIRECT_CLASS, IB_SA_CLASS}; - static char const str_opts[] = "S:D:n:C:P:t:sldgphuf:R"; - static const struct option long_opts[] = { - { "S", 1, 0, 'S'}, - { "D", 1, 0, 'D'}, - { "num-hops", 1, 0, 'n'}, - { "down-links-only", 0, 0, 'd'}, - { "line-mode", 0, 0, 'l'}, - { "ca-name", 1, 0, 'C'}, - { "ca-port", 1, 0, 'P'}, - { "timeout", 1, 0, 't'}, - { "show", 0, 0, 's'}, - { "print-port-guids", 0, 0, 'g'}, - { "print-additional", 0, 0, 'p'}, - { "help", 0, 0, 'h'}, - { "usage", 0, 0, 'u'}, - { "node-name-map", 1, 0, 1}, - { "debug", 0, 0, 2}, - { "compat", 0, 0, 3}, - { "from", 1, 0, 'f'}, - { "R", 0, 0, 'R'}, + const struct ibdiag_opt opts[] = { + { "node-name-map", 1, 1, "", "node name map file" }, + { "switch", 'S', 1, "", "query only (hex format)"}, + { "Direct", 'D', 1, "", "query only node specified by "}, + { "from", 'f', 1, "", "specify node to start \"\""}, + { "hops", 'n', 1, "", "Number of hops to include away from specified node"}, + { "down", 'd', 0, NULL, "print only down links"}, + { "line", 'l', 0, NULL, "(line mode) print all information for each link on a single line"}, + { "additional", 'p', 0, NULL, "print additional switch settings (PktLifeTime, HoqLife, VLStallCount)"}, + { "portguids", 'g', 0, NULL, "print port guids instead of node guids"}, + { "GNDN", 'R', 0, NULL, "(This option is obsolete and does nothing)"}, { 0 } }; + char usage_args[] = ""; + + ibdiag_process_opts(argc, argv, "sDLG", "SDfndlpgR", opts, process_opt, + usage_args, NULL); - argv0 = argv[0]; - - while (1) { - int ch = getopt_long(argc, argv, str_opts, long_opts, NULL); - if ( ch == -1 ) - break; - switch(ch) { - case 1: - node_name_map_file = strdup(optarg); - break; - case 2: - ibnd_debug(1); - break; - case 'f': - from = strdup(optarg); - break; - case 'C': - ca = strdup(optarg); - break; - case 'P': - ca_port = strtoul(optarg, 0, 0); - break; - case 'D': - dr_path = strdup(optarg); - break; - case 'n': - hops = (int)strtol(optarg, NULL, 0); - break; - case 'd': - down_links_only = 1; - break; - case 'l': - line_mode = 1; - break; - case 't': - timeout_ms = strtoul(optarg, 0, 0); - break; - case 's': - ibnd_show_progress(1); - break; - case 'g': - print_port_guids = 1; - break; - case 'S': - guid_str = optarg; - guid = (uint64_t)strtoull(guid_str, 0, 0); - break; - case 'p': - add_sw_settings = 1; - break; - case 'R': - /* GNDN */ - break; - default: - usage(); - break; - } - } argc -= optind; argv += optind; - ibmad_port = mad_rpc_open_port(ca, ca_port, mgmt_classes, 3); + if (ibverbose) + ibnd_debug(1); + + ibmad_port = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 3); if (!ibmad_port) { - fprintf(stderr, "Failed to open %s port %d", ca, ca_port); + fprintf(stderr, "Failed to open %s port %d", ibd_ca, ibd_ca_port); exit(1); } @@ -365,7 +313,7 @@ main(int argc, char **argv) if (from) { /* only scan part of the fabric */ str2drpath(&(port_id.drpath), from, 0, 0); - if ((fabric = ibnd_discover_fabric(ibmad_port, timeout_ms, &port_id, hops)) == NULL) { + if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, &port_id, hops)) == NULL) { fprintf(stderr, "discover failed\n"); rc = 1; goto close_port; @@ -375,7 +323,7 @@ main(int argc, char **argv) if (ib_resolve_portid_str_via(&port_id, guid_str, IB_DEST_GUID, NULL, ibmad_port) >= 0) { if ((fabric = ibnd_discover_fabric(ibmad_port, - timeout_ms, &port_id, 1)) == NULL) + ibd_timeout, &port_id, 1)) == NULL) IBWARN("Single node discover failed; attempting full scan\n"); } else IBWARN("Failed to resolve %s; attempting full scan\n", @@ -383,7 +331,7 @@ main(int argc, char **argv) } if (!fabric) /* do a full scan */ - if ((fabric = ibnd_discover_fabric(ibmad_port, timeout_ms, NULL, -1)) == NULL) { + if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, NULL, -1)) == NULL) { fprintf(stderr, "discover failed\n"); rc = 1; goto close_port; -- 1.5.4.5 From weiny2 at llnl.gov Wed Jun 17 10:10:56 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 17 Jun 2009 10:10:56 -0700 Subject: [ofa-general] [PATCH] ibqueryerrors: fix custom opts string Message-ID: <20090617101056.c1873840.weiny2@llnl.gov> From: Ira Weiny Date: Thu, 14 May 2009 17:24:29 -0700 Subject: [PATCH] ibqueryerrors: fix custom opts string Signed-off-by: Ira Weiny --- infiniband-diags/src/ibqueryerrors.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/infiniband-diags/src/ibqueryerrors.c b/infiniband-diags/src/ibqueryerrors.c index 999329e..36b8050 100644 --- a/infiniband-diags/src/ibqueryerrors.c +++ b/infiniband-diags/src/ibqueryerrors.c @@ -407,7 +407,7 @@ main(int argc, char **argv) }; char usage_args[] = ""; - ibdiag_process_opts(argc, argv, "sDLG", "snSrR", opts, process_opt, + ibdiag_process_opts(argc, argv, "sDLG", "scnSrR", opts, process_opt, usage_args, NULL); argc -= optind; -- 1.5.4.5 From Evan.Fraser at rms.com Wed Jun 17 10:14:32 2009 From: Evan.Fraser at rms.com (Evan Fraser) Date: Wed, 17 Jun 2009 18:14:32 +0100 Subject: [ofa-general] Compilation errors with OFED 1.4.1/ 1.4 Message-ID: <9759F033B56A60469F217C49AC58E2BF085A541B@MAILUK2.rms.com> Hi Rajouri, >I'm trying to build a kernel module using OFED 1.4.1 on SLES 10 Update 2 >(2.6.16.60-0.21-smp) and I get the below compilation errors. > OFED didn't compile for me on that kernel either. Apply the SLES updates to upgrade your kernel etc. I found it compiles fine against kernel 2.6.16.60-0.34-smp. Evan Fraser Senior Systems Analyst Peninsular House, 30 Monument Street London EC3R 8NB, United Kingdom Tel +44 20 7444 7860 Mobile +44 75 9024 5788 evan.fraser at rms.com www.rms.com This message and any attachments contain information that may be RMS Inc. confidential and/or privileged. If you are not the intended recipient (or authorized to receive for the intended recipient), and have received this message in error, any use, disclosure or distribution is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to the e-mail and permanently deleting the message from your computer and/or storage system. From sean.hefty at intel.com Wed Jun 17 11:11:40 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 17 Jun 2009 11:11:40 -0700 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD030C3A22@mtlexch01.mtl.com> References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A22@mtlexch01.mtl.com> Message-ID: >LL: the RDMA stack will see that the port has different link types. >SLs map cleanly to VLAN user priorities. >LL: you need to emulate *enough* so that typical applications don't need >to worry about the link type. SA path queries is the best example. >Otherwise, every RDMA application (not necessarily a CMA app) will need >to have different code paths depending on the link type. Let's just say that at this point I completely disagree with where these patches try to abstract the differences, which are many. RDMA apps that want to use this and IB without going through an abstraction will need different code -- just like they would for iWarp, which also provides RDMA over Ethernet, and is a standard. IB mad and SA query modules are not appropriate places for abstracting the differences between IB, iWarp, and whatever name we give this. This could change depending on whether this is really trying to be IB with a different L2, or is just another RDMA protocol that runs on Ethernet. - Sean From rdreier at cisco.com Wed Jun 17 11:17:46 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 11:17:46 -0700 Subject: [ofa-general] Re: [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <20090615073408.GA27543@mtls03> (Eli Cohen's message of "Mon, 15 Jun 2009 10:34:08 +0300") References: <20090615073408.GA27543@mtls03> Message-ID: A quick first reaction. I'm a little conflicted about this. On one hand, we definitely want to support existing hardware for users that see value in running this RDMA protocol on ethernet. On the other hand the way this support is added craps all over the RDMA core stuff so I'm not happy about adding it. So I don't know at the moment. - R. From jgunthorpe at obsidianresearch.com Wed Jun 17 11:17:53 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 17 Jun 2009 12:17:53 -0600 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> References: <20090615073431.GC27543@mtls03> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> Message-ID: <20090617181753.GI18150@obsidianresearch.com> On Wed, Jun 17, 2009 at 11:41:28AM +0300, Liran Liss wrote: > > > Why not just use IP to MAC calls? Or use the MAC as the GUID? > > We do use standard OS services to map the IP addresses (that were > encoded in the GID) to MACs. GIDs encode IP addresses rather than > MACs to enable users to use the node names that they are used to. > Specifically, we will feed in all IP addresses that were assigned to > the Ethernet interface to the corresponding port GID table. This > will also enable routing in the future. Hum, This is a very tricky subject. Co-mingling the IB GID address space and the IPv6 address space like this is not really something that was envisioned from the IBA side. Also, the use of IPv4 mapped addresses presents some interesting problems from an IB router perspective. > The only exception is IPv6 link-local addresses, which already > encodes the MAC. In this case, a simple algorithmic operation > extracts the MAC without requiring ARP, etc. You shouldn't do this, the IPv6 link local addresses may use the MAC, or maybe not. There are several additional IPv6 protocols like privacy extensions and SEND that break this assumption. You still must do the lookup if you are using IPv6 addresses. Jason From rdreier at cisco.com Wed Jun 17 11:19:04 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 11:19:04 -0700 Subject: [ofa-general] Re: [PATCH 2/9] ib_core: kernel API for GID --> MAC translations In-Reply-To: <20090615073431.GC27543@mtls03> (Eli Cohen's message of "Mon, 15 Jun 2009 10:34:31 +0300") References: <20090615073431.GC27543@mtls03> Message-ID: > diff --git a/include/rdma/ib_user_verbs.h b/include/rdma/ib_user_verbs.h > index a17f771..5bfa2e6 100644 > --- a/include/rdma/ib_user_verbs.h > +++ b/include/rdma/ib_user_verbs.h > @@ -81,7 +81,8 @@ enum { > IB_USER_VERBS_CMD_MODIFY_SRQ, > IB_USER_VERBS_CMD_QUERY_SRQ, > IB_USER_VERBS_CMD_DESTROY_SRQ, > - IB_USER_VERBS_CMD_POST_SRQ_RECV > + IB_USER_VERBS_CMD_POST_SRQ_RECV, > + IB_USER_VERBS_CMD_GET_MAC > }; > This breaks the ABI defined for XRC. So I guess these patches are not really intended to be applied? I definitely want to finish up XRC before starting to look at this series in detail anyway. - R. From sean.hefty at intel.com Wed Jun 17 11:19:53 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 17 Jun 2009 11:19:53 -0700 Subject: [ewg] RE: [ofa-general] [PATCH 4/9] ib_core: Add RDMAoE SA support In-Reply-To: <20090617111107.GC29832@mtls03> References: <20090615073450.GE27543@mtls03> <20090617111107.GC29832@mtls03> Message-ID: >> How can a user control this? An app needs the same qkey for unicast traffic. > >In RDMAoE, the qkey has a fixed well-known value, which will be >returned both by multicast and path queries. The rdma_cm defines and uses a different well-known qkey. From rdreier at cisco.com Wed Jun 17 11:20:26 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 11:20:26 -0700 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: <20090617181753.GI18150@obsidianresearch.com> (Jason Gunthorpe's message of "Wed, 17 Jun 2009 12:17:53 -0600") References: <20090615073431.GC27543@mtls03> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> <20090617181753.GI18150@obsidianresearch.com> Message-ID: > Hum, This is a very tricky subject. Co-mingling the IB GID address > space and the IPv6 address space like this is not really something > that was envisioned from the IBA side. Doesn't the IB spec say that an IB GID *is* an IPv6 address? So in theory it should be OK; however I don't think in practice anyone paid attention to making sure that the IB GID space works as an IPv6 address. - R. From rdreier at cisco.com Wed Jun 17 11:23:05 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 11:23:05 -0700 Subject: [ofa-general] [ANNOUNCE] libmlx4 1.0.1 released Message-ID: libmlx4 is a userspace driver for Mellanox ConnectX InfiniBand HCAs. It is a plug-in module for libibverbs that allows programs to use Mellanox hardware directly from userspace. A new stable release, libmlx4 1.0.1, is available from http://www.openfabrics.org/downloads/mlx4/libmlx4-1.0.1.tar.gz with sha1sum 1f1465664c19e263ef096e49128a7dfa1d086c73 libmlx4-1.0.1.tar.gz I also tagged the 1.0.1 release of libmlx4 and pushed it out to my git tree on kernel.org: git://git.kernel.org/pub/scm/libs/infiniband/libmlx4.git (the name of the tag is libmlx4-1.0.1). This is a minor update release of libmlx4, primarily to have a release with a fix to the BlueFlame memcpy problem, which causes problems when running under Valgrind. Things appear quite usable at the moment. Please test and let me know if you see anything that needs to be fixed. I have also started the process of getting updated libmlx4 packages into the main Debian and Fedora archives. The complete list of changes since 1.0 is: Dotan Barak (1): Fix value of pkey_index in QP1 completions Eli Cohen (1): Optimize QP stamping Jack Morgenstein (5): Clear context struct at allocation time Fix test for GRH presence in mlx4_poll_one() Don't use memcpy() to write blueflame sends Fix race between create QP and destroy QP Set ownership bit correctly when copying over CQEs during CQ resize Roland Dreier (16): Update Debian version to 1.0-1 Move to using cdbs to generate Debian/control for better Build-depends Add Closes: tag for Debian ITP bug Use real Homepage: tag instead of pseudo-header inside description Micro-optimize mlx4_poll_one() Revert "Don't add an extra entry to CQs" Spec file cleanups based on Fedora review Update RPM spec changelog to avoid a macro Add debian/watch file Change openib.org URLs to openfabrics.org URLs Fix CQ cleanup when QP is destroyed Update Debian policy version to 3.8.2 Update build system to use shave Move -dbg Debian package to section debug Merge spec file changelog from Fedora Roll libmlx4-1.0.1 release Vladimir Sokolovsky (1): Implement resize CQ From robert.j.woodruff at intel.com Wed Jun 17 11:25:13 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Wed, 17 Jun 2009 11:25:13 -0700 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: <20090615073408.GA27543@mtls03> References: <20090615073408.GA27543@mtls03> Message-ID: <382A478CAD40FA4FB46605CF81FE39F4353985E0@orsmsx507.amr.corp.intel.com> Eli Cohen wrote, >RDMA over Ethernet (RDMAoE) allows running the IB transport protocol over >Ethernet, providing IB capabilities for Ethernet fabrics. The packets are >standard Ethernet frames with an Ethertype, an IB GRH, unmodified IB transport >headers and payload. HCA RDMAoE ports are no different than regular IB ports >from the RDMA stack perspective. This seems like a hack to try and shoehorn this in under the IB portion of the stack. It makes more sense for it to be it's own transport type, a peer to IB and iWarp. >IB subnet management and SA services are not required for RDMAoE operation; >Ethernet management practices are used instead. In Ethernet, nodes are commonly >referred to by applications by means of an IP address. RDMAoE treats IP >addresses that were assigned to the corresponding Ethernet port as GIDs, and >makes use of the IP stack to bind a destination address to the corresponding >netdevice (just as the CMA does today for IB and iWARP) and to obtain its L2 >MAC addresses. Since it does not require SM or SA services, then this is another reason to make it it's own transport type rather than try to fit in under the IB core services and then emulate the expected behaviour that the core mid-layer and ULPs expect for things like SA queries. From jgunthorpe at obsidianresearch.com Wed Jun 17 11:31:23 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 17 Jun 2009 12:31:23 -0600 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: References: <20090615073431.GC27543@mtls03> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> <20090617181753.GI18150@obsidianresearch.com> Message-ID: <20090617183123.GJ18150@obsidianresearch.com> On Wed, Jun 17, 2009 at 11:20:26AM -0700, Roland Dreier wrote: > > > Hum, This is a very tricky subject. Co-mingling the IB GID address > > space and the IPv6 address space like this is not really something > > that was envisioned from the IBA side. > > Doesn't the IB spec say that an IB GID *is* an IPv6 address? So in > theory it should be OK; however I don't think in practice anyone paid > attention to making sure that the IB GID space works as an IPv6 address. It is like an IPv6 address but it was expressly envisioned to be a seperate space. The IBA authors copied many of the conventions from IPv6 for numbering this new space, like link local, and multicast prefixes, but it was not intended to be co-mingled. That said, because of the incredible similarity it could probably be co-mingled, with some research and validation, but IIRC the main reason this wasn't done from the start is that the IETF wasn't interested in supplying a protocol number and the definition work to make IB over IPv6 a standard. So instead we have IB over GRH, which is 99% the same.... So, I didn't look closely enough, but what was the ethertype that is used here in this patch set? Hopefully not IPv6. Therin is the oddness, if the main IPv6 routing table is used to direct packets that are not labeled with the IPv6 ethertype that is very confusing - on the other hand if the RDMA packets are labeled with the IPv6 type then you need IETF to supply a protocol number for this. In either case, this is a good point, it is difficult to imagine including this work in Linux until either IEEE or IETF supply a number.. Jason From rdreier at cisco.com Wed Jun 17 11:38:43 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 11:38:43 -0700 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: <20090617183123.GJ18150@obsidianresearch.com> (Jason Gunthorpe's message of "Wed, 17 Jun 2009 12:31:23 -0600") References: <20090615073431.GC27543@mtls03> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> <20090617181753.GI18150@obsidianresearch.com> <20090617183123.GJ18150@obsidianresearch.com> Message-ID: > It is like an IPv6 address but it was expressly envisioned to be a > seperate space. The IBA authors copied many of the conventions from > IPv6 for numbering this new space, like link local, and multicast > prefixes, but it was not intended to be co-mingled. Well (I've quoted this many times): IBA section 4.1: "A GID is a valid 128-bit IPv6 address (per RFC 2373) with additional properties / restrictions defined within IBA..." People often try to claim that this sentence doesn't mean what it very explicitly and clearly says, and certainly I believe that existing practice doesn't comply with the IBA spec, but I don't see how anyone can say that a truly compliant IB GID is not a real IPv6 address. > So, I didn't look closely enough, but what was the ethertype that is > used here in this patch set? Hopefully not IPv6. I don't think it's specified in the code -- presumably in HCA FW. Which is an issue as you say -- do we have an IEEE ethertype yet? And if we don't use the IPv6 ethertype, then is multicast going to work well (if the code is moved away from just broadcasting everything)? I doubt that switch IGMP snooping works well for non-IP ethertypes -- in fact I wonder how well existing switches handle IPv6 multicast ;) - R. From hnrose at comcast.net Wed Jun 17 13:09:11 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 17 Jun 2009 16:09:11 -0400 Subject: [ofa-general] [PATCH] opensm/osm_ucast_lash.(h c): Replace memory allocation by array Message-ID: <20090617200911.GB11146@comcast.net> lash's num_mst_in_lane array is indexed by lane nuber, not by switch index. Since it's a small number, get rid of extra memory allocation. Signed-off-by: Robert Pearson Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/opensm/osm_ucast_lash.h b/opensm/include/opensm/osm_ucast_lash.h index f48ea33..98034cc 100644 --- a/opensm/include/opensm/osm_ucast_lash.h +++ b/opensm/include/opensm/osm_ucast_lash.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008 System Fabric Works, Inc. + * Copyright (c) 2008,2009 System Fabric Works, Inc. All rights reserved. * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. @@ -90,7 +90,7 @@ typedef struct _lash { int balance_limit; switch_t **switches; cdg_vertex_t ****cdg_vertex_matrix; - int *num_mst_in_lane; + int num_mst_in_lane[IB_MAX_NUM_VLS]; int ***virtual_location; } lash_t; diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c index a987eb3..d8f0ac3 100644 --- a/opensm/opensm/osm_ucast_lash.c +++ b/opensm/opensm/osm_ucast_lash.c @@ -4,6 +4,7 @@ * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2007 Simula Research Laboratory. All rights reserved. * Copyright (c) 2007 Silicon Graphics Inc. All rights reserved. + * Copyright (c) 2008,2009 System Fabric Works, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -707,9 +708,6 @@ static void free_lash_structures(lash_t * p_lash) if (p_lash->virtual_location) free(p_lash->virtual_location); - if (p_lash->num_mst_in_lane) - free(p_lash->num_mst_in_lane); - OSM_LOG_EXIT(p_log); } @@ -778,11 +776,7 @@ static int init_lash_structures(lash_t * p_lash) } /* initialise num_mst_in_lane[num_switches], default 0 */ - p_lash->num_mst_in_lane = (int *)malloc(num_switches * sizeof(int)); - if (p_lash->num_mst_in_lane == NULL) - goto Exit_Mem_Error; - memset(p_lash->num_mst_in_lane, 0, - num_switches * sizeof(p_lash->num_mst_in_lane[0])); + memset(p_lash->num_mst_in_lane, 0, IB_MAX_NUM_VLS); goto Exit; From hnrose at comcast.net Wed Jun 17 13:08:11 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 17 Jun 2009 16:08:11 -0400 Subject: [ofa-general] [PATCHv2] libibmad/resolve.c: Remove compile warning on x86_64 arch Message-ID: <20090617200811.GA11146@comcast.net> resolve.c:106: warning: pointer targets in passing argument 1 of 'mad_decode_field' differ in signedness Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c index f17da11..643c187 100644 --- a/libibmad/src/resolve.c +++ b/libibmad/src/resolve.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2006 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -74,7 +75,7 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, const struct ibmad_port *srcport) { ib_portid_t sm_portid; - char buf[IB_SA_DATA_SIZE] = { 0 }; + uint8_t buf[IB_SA_DATA_SIZE] = { 0 }; ib_portid_t self = { 0 }; uint64_t selfguid; ibmad_gid_t selfgid; From hnrose at comcast.net Wed Jun 17 13:10:26 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 17 Jun 2009 16:10:26 -0400 Subject: [ofa-general] [PATCH] opensm/osm_mesh.c: Improve VL utilization Message-ID: <20090617201026.GC11146@comcast.net> Add steps to the algorithm to further condition the netlist of switches that is handed to lash to analyze. (1) The previous version sorted out all the links out of switches into dimension order’in the order the dimensions were found which is driven by the arbitrary choice of links out of the seed’switch. This takes the additional step of waiting until after the lengths along each dimension are measured then reordering the links out of each switch so that dimensions with the largest lengths are ordered first. i.e. if the mesh analysis discovers that the mesh is a 5x8x6 torus then the coordinates are reordered so that based on the order of links on each switch the mesh is an 8x6x5 torus. (2) The previous version left the order of switches in the switch list the same as when they were discovered by the SMPs which is some sort of depth first tree walk modified by the random return of MADs if multiples are outstanding. This takes the additional step of reordering the switches so that they are presented in odometer’order again with the dimension with the largest lengths changing fastest. With this algorithm, we always see what we believe is the optimal result for each size of toroidal mesh. As a footnote, when we later studied the effect of broken or imperfect meshes, we found that the results are dependent on the choice of origin for the mesh which clearly doesn't matter for a perfect torus with nodes sorted as described here. Thus there is some additional work required to try to understand how to select the optimal origin for meshes with defects. We saw that the number of VLs required was higher when the origin was near a defect so choosing one as far as possible from the defects seems to be the right idea. This still needs to be done since it will remove some fairly rare defect cases which now require 9 VLs to route. Signed-off-by: Robert Pearson Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_mesh.c b/opensm/opensm/osm_mesh.c index 1867876..f2dd2f8 100644 --- a/opensm/opensm/osm_mesh.c +++ b/opensm/opensm/osm_mesh.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008,2009 System Fabric Works, Inc. + * Copyright (c) 2008,2009 System Fabric Works, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -1026,12 +1026,15 @@ static inline int ltmag(int a, int b) return (a1 < b1) || (a1 == b1 && a > b); } +static int dim_order[MAX_DIMENSION]; +static int dim_reverse[MAX_DIMENSION]; + /* - * reorder_links + * reorder_node_links * * reorder the links out of a switch in sign/dimension order */ -static int reorder_links(lash_t *p_lash, int sw) +static int reorder_node_links(lash_t *p_lash, mesh_t *mesh, int sw) { osm_log_t *p_log = &p_lash->p_osm->log; switch_t *s = p_lash->switches[sw]; @@ -1039,9 +1042,10 @@ static int reorder_links(lash_t *p_lash, int sw) int n = node->num_links; link_t **links; int *axes; - int i, j; + int i, j, k, l; int c; int next = 0; + int dimension = mesh->dimension; if (!(links = calloc(n, sizeof(link_t *)))) { OSM_LOG(p_log, OSM_LOG_ERROR, "Failed allocating temp array - out of memory\n"); @@ -1057,19 +1061,23 @@ static int reorder_links(lash_t *p_lash, int sw) /* * find the links with axes */ - for (j = 1; j <= 2*node->dimension; j++) { - c = j; - if (node->coord[(c-1)/2] > 0) - c = opposite(s, c); + for (i = 0; i < dimension; i++) { + j = dim_order[i]; + for (k = 1; k <= 2; k++) { + c = 2*j + k; - for (i = 0; i < n; i++) { - if (!node->links[i]) - continue; - if (node->axes[i] == c) { - links[next] = node->links[i]; - axes[next] = node->axes[i]; - node->links[i] = NULL; - next++; + if (node->coord[j] > 0) + c = opposite(s, c); + + for (l = 0; l < n; l++) { + if (!node->links[l]) + continue; + if (node->axes[l] == c) { + links[next] = node->links[l]; + axes[next] = node->axes[l]; + node->links[l] = NULL; + next++; + } } } } @@ -1099,9 +1107,9 @@ static int reorder_links(lash_t *p_lash, int sw) } /* - * measure geometry + * make_coord */ -static int measure_geometry(lash_t *p_lash, mesh_t *mesh, int seed) +static int make_coord(lash_t *p_lash, mesh_t *mesh, int seed) { osm_log_t *p_log = &p_lash->p_osm->log; int i, j, k; @@ -1111,8 +1119,6 @@ static int measure_geometry(lash_t *p_lash, mesh_t *mesh, int seed) int dimension = mesh->dimension; int num_switches = p_lash->num_switches; int assigned_axes = 0, unassigned_axes = 0; - int max[MAX_DIMENSION]; - int min[MAX_DIMENSION]; OSM_LOG_ENTER(p_log); @@ -1120,6 +1126,13 @@ static int measure_geometry(lash_t *p_lash, mesh_t *mesh, int seed) s = p_lash->switches[sw]; s->node->coord = calloc(dimension, sizeof(int)); + if (!s->node->coord) { + OSM_LOG(p_log, OSM_LOG_ERROR, + "Failed allocating coord - out of memory\n"); + OSM_LOG_EXIT(p_log); + return -1; + } + for (i = 0; i < dimension; i++) s->node->coord[i] = (sw == seed) ? 0 : LARGE; @@ -1163,14 +1176,36 @@ static int measure_geometry(lash_t *p_lash, mesh_t *mesh, int seed) } } while (change); - for (sw = 0; sw < num_switches; sw++) { - if (reorder_links(p_lash, sw)) { - OSM_LOG_EXIT(p_log); - return -1; - } - } + OSM_LOG_EXIT(p_log); + return 0; +} + +/* + * measure geometry + */ +static int measure_geometry(lash_t *p_lash, mesh_t *mesh) +{ + osm_log_t *p_log = &p_lash->p_osm->log; + int i, j; + int sw; + switch_t *s; + int dimension = mesh->dimension; + int num_switches = p_lash->num_switches; + int max[MAX_DIMENSION]; + int min[MAX_DIMENSION]; + int size[MAX_DIMENSION]; + int max_size; + int max_index; + + OSM_LOG_ENTER(p_log); mesh->size = calloc(dimension, sizeof(int)); + if (!mesh->size) { + OSM_LOG(p_log, OSM_LOG_ERROR, + "Failed allocating size - out of memory\n"); + OSM_LOG_EXIT(p_log); + return -1; + } for (i = 0; i < dimension; i++) { max[i] = -LARGE; @@ -1191,7 +1226,49 @@ static int measure_geometry(lash_t *p_lash, mesh_t *mesh, int seed) } for (i = 0; i < dimension; i++) - mesh->size[i] = max[i] - min[i] + 1; + mesh->size[i] = size[i] = max[i] - min[i] + 1; + + /* + * find an order of dimensions that places largest + * sizes first since this seems to work best with LASH + */ + for (j = 0; j < dimension; j++) { + max_size = -1; + max_index = -1; + + for (i = 0; i < dimension; i++) { + if (size[i] > max_size) { + max_size = size[i]; + max_index = i; + } + } + + dim_order[j] = max_index; + dim_reverse[max_index] = j; + size[max_index] = -1; + } + + OSM_LOG_EXIT(p_log); + return 0; +} + +/* + * reorder links + */ +static int reorder_links(lash_t *p_lash, mesh_t *mesh) +{ + osm_log_t *p_log = &p_lash->p_osm->log; + int sw; + int num_switches = p_lash->num_switches; + + OSM_LOG_ENTER(p_log); + + for (sw = 0; sw < num_switches; sw++) { + if (reorder_node_links(p_lash, mesh, sw)) { + OSM_LOG_EXIT(p_log); + return -1; + } + } OSM_LOG_EXIT(p_log); return 0; @@ -1387,7 +1464,13 @@ int osm_do_mesh_analysis(lash_t *p_lash) if (s->node->type) { make_geometry(p_lash, max_class_type); - if (measure_geometry(p_lash, mesh, max_class_type)) + if (make_coord(p_lash, mesh, max_class_type)) + goto err; + + if (measure_geometry(p_lash, mesh)) + goto err; + + if (reorder_links(p_lash, mesh)) goto err; p = buf; From agshew at gmail.com Wed Jun 17 13:12:56 2009 From: agshew at gmail.com (Andrew Shewmaker) Date: Wed, 17 Jun 2009 14:12:56 -0600 Subject: [ofa-general] ofed-1.5-daily, linux kernel 2.6.30, and failure to read the uverbs ABI version Message-ID: Hi, Thanks in advance for any info and/or help. My understanding of the ofed-1.5-daily code was that it was based on 2.6.30 kernel code. However, after building the daily from yesterday, I see errors when loading the ib_umad module. IPOIB appears to work, but RDMA and verbs commands report "libibverbs: Fatal: couldn't read uverbs ABI version." What should I expect to work? I'm not actually expecting this to work yet, but I was testing pnfs and thought I would try ofed too. BTW, dmesg shows this upon loading the ib_umad module: kernel BUG at drivers/infiniband/core/verbs.c:91! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:80/0000:80:0e.0/class CPU 1 Modules linked in: ib_umad(+) nfsd lockd exportfs nfs_acl auth_rpcgss sunrpc autofs4 ib_ipoib ib_cm ib_sa ipv6 ipt_LOG xt_limit xt_multiport dm_multipath snd_pcsp ppdev snd_ seq_dummy parport_pc snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss parport floppy snd_pcm ib_mthca ib_mad snd_timer i2c_nforce2 snd ib_core i2c_core soundcore forcedeth snd_page_alloc k8temp hwmon sata_nv pata_acpi pata_amd ata_generic [last unloaded: scsi_wait_scan] Pid: 481, comm: modprobe Tainted: G W 2.6.30-6.pnfs.fc9.x86_64 #1 H8DCE RIP: 0010:[] [] rdma_node_get_transport+0x2d/0x47 [ib_core] RSP: 0018:ffff880042f0dbc8 EFLAGS: 00010297 RAX: 0000000000000000 RBX: ffff88007e26c000 RCX: 0000000000170001 RDX: 0000000000000000 RSI: ffffffff814be723 RDI: 0000000000000000 RBP: ffff880042f0dbd8 R08: 000000004e519652 R09: 0000000000000000 R10: ffff88013ae3b958 R11: 000000004e519652 R12: ffffffffa02b3900 R13: 0000000000000000 R14: 0000000001294120 R15: 0000000001294138 FS: 00007fff455526f0(0000) GS:ffffc200001d7000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fff3d7a8bf8 CR3: 0000000139971000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process modprobe (pid: 481, threadinfo ffff880042f0c000, task ffff88007e1c47c0) Stack: 000000004e519652 000000004e519652 ffff880042f0dc58 ffffffffa02b1afb 000000004e519652 000000004e519652 0000000000000246 ffff88007e26c090 ffff880042f0dc58 ffffffffa0057b7e 0000000001294120 0000000001294138 Call Trace: [] ib_umad_add_one+0x37/0x480 [ib_umad] [] ? add_client_context+0xdf/0x104 [ib_core] [] ib_register_client+0x70/0xc2 [ib_core] [] ib_umad_init+0xaa/0xf6 [ib_umad] [] ? marker_update_probe_range+0x3a/0x276 [] ? ib_umad_init+0x0/0xf6 [ib_umad] [] do_one_initcall+0x84/0x1ad [] ? print_lock_contention_bug+0x2d/0x133 [] ? __mutex_lock_common+0x362/0x3c3 [] ? __up_read+0x2d/0xaf [] ? trace_hardirqs_on_caller+0x32/0x172 [] ? trace_hardirqs_on+0x20/0x36 [] ? __up_read+0x2d/0xaf [] ? __up_read+0x2d/0xaf [] ? trace_hardirqs_on_caller+0x32/0x172 [] ? trace_hardirqs_on+0x20/0x36 [] ? __up_read+0x8f/0xaf [] ? up_read+0x3a/0x55 [] ? __blocking_notifier_call_chain+0x72/0x98 [] sys_init_module+0xbf/0x1fd [] system_call_fastpath+0x16/0x1b Code: 89 e5 48 83 ec 10 0f 1f 44 00 00 65 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 83 ff 01 72 0c 83 ff 03 76 0b 83 ff 04 b0 01 74 04 <0f> 0b eb fe 48 8b 55 f8 65 48 33 14 25 28 00 00 00 74 05 e8 fb RIP [] rdma_node_get_transport+0x2d/0x47 [ib_core] RSP ---[ end trace 4e46d3e076693c7d ]--- Thanks again. -- Andrew Shewmaker From hal.rosenstock at gmail.com Wed Jun 17 13:36:24 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 17 Jun 2009 16:36:24 -0400 Subject: [ofa-general] Re: [PATCH] opensm/osm_inform.c: For traps 64-67, use GID in DataDetails in log messages In-Reply-To: <20090617164624.GQ6163@me> References: <20090617134046.GA10939@comcast.net> <20090617164624.GQ6163@me> Message-ID: On Wed, Jun 17, 2009 at 12:46 PM, Sasha Khapyorsky wrote: > On 09:40 Wed 17 Jun     , Hal Rosenstock wrote: >> >> Issuer GID and LID are uninteresting for SM generated notices >> >> Signed-off-by: Hal Rosenstock >> --- >> diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c >> index 990f1e0..43676c2 100644 >> --- a/opensm/opensm/osm_inform.c >> +++ b/opensm/opensm/osm_inform.c >> @@ -561,15 +561,29 @@ ib_api_status_t osm_report_notice(IN osm_log_t * p_log, IN osm_subn_t * p_subn, >> >>       /* an official Event information log */ >>       if (ib_notice_is_generic(p_ntc)) >> -             OSM_LOG(p_log, OSM_LOG_INFO, >> -                     "Reporting Generic Notice type:%u num:%u (%s)" >> -                     " from LID:%u GID:%s\n", >> -                     ib_notice_get_type(p_ntc), >> -                     cl_ntoh16(p_ntc->g_or_v.generic.trap_num), >> -                     ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), >> -                     cl_ntoh16(p_ntc->issuer_lid), >> -                     inet_ntop(AF_INET6, p_ntc->issuer_gid.raw, gid_str, >> -                               sizeof gid_str)); >> +             if ((p_ntc->g_or_v.generic.trap_num == CL_HTON16(64)) || >> +                 (p_ntc->g_or_v.generic.trap_num == CL_HTON16(65)) || >> +                 (p_ntc->g_or_v.generic.trap_num == CL_HTON16(66)) || >> +                 (p_ntc->g_or_v.generic.trap_num == CL_HTON16(67))) >> +                     OSM_LOG(p_log, OSM_LOG_INFO, >> +                             "Reporting Generic Notice type:%u num:%u (%s)" >> +                             " GID:%s\n", >> +                             ib_notice_get_type(p_ntc), >> +                             cl_ntoh16(p_ntc->g_or_v.generic.trap_num), >> +                             ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), >> +                             inet_ntop(AF_INET6, >> +                                       p_ntc->data_details.ntc_64_67.gid.raw, >> +                                       gid_str, sizeof gid_str)); >> +             else >> +                     OSM_LOG(p_log, OSM_LOG_INFO, >> +                             "Reporting Generic Notice type:%u num:%u (%s)" >> +                             " from LID:%u GID:%s\n", >> +                             ib_notice_get_type(p_ntc), >> +                             cl_ntoh16(p_ntc->g_or_v.generic.trap_num), >> +                             ib_get_trap_str(p_ntc->g_or_v.generic.trap_num), >> +                             cl_ntoh16(p_ntc->issuer_lid), >> +                             inet_ntop(AF_INET6, p_ntc->issuer_gid.raw, >> +                                       gid_str, sizeof gid_str)); > > Maybe, but it seems easier for me to have unified log here (which is > macro which will be filtered out when log level is lower) than add > additional "if/else". Not sure what you mean by "unified log". -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From jgunthorpe at obsidianresearch.com Wed Jun 17 13:52:57 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 17 Jun 2009 14:52:57 -0600 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: References: <20090615073431.GC27543@mtls03> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> <20090617181753.GI18150@obsidianresearch.com> <20090617183123.GJ18150@obsidianresearch.com> Message-ID: <20090617205257.GL18150@obsidianresearch.com> On Wed, Jun 17, 2009 at 11:38:43AM -0700, Roland Dreier wrote: > > > It is like an IPv6 address but it was expressly envisioned to be a > > seperate space. The IBA authors copied many of the conventions from > > IPv6 for numbering this new space, like link local, and multicast > > prefixes, but it was not intended to be co-mingled. > > Well (I've quoted this many times): IBA section 4.1: > > "A GID is a valid 128-bit IPv6 address (per RFC 2373) with additional > properties / restrictions defined within IBA..." > > People often try to claim that this sentence doesn't mean what it very > explicitly and clearly says, and certainly I believe that existing > practice doesn't comply with the IBA spec, but I don't see how anyone > can say that a truly compliant IB GID is not a real IPv6 address. Yes, I know.. This is a crummy area in IBA, the counter quote to 4.1, is 5.2.2: ''Note however, that IBA does not define a relationship between a device GID and IPv6 address (ie there is no defined mapping between GID and IPv6 address for and IB device or port)'' Which I take to mean that the GID follows all the conventions of RFC 2373 but is a distinct address family. Fundamentally the question that is unanswered by IBA is if a GID and IPv6 address that have the same value represent the same interface and host. FWIW, I have long sought to keep the GID and IPv6 spaces as aligned and would prefer to be done with this ambiguity and have a 1:1 mapping.. > > So, I didn't look closely enough, but what was the ethertype that is > > used here in this patch set? Hopefully not IPv6. > I don't think it's specified in the code -- presumably in HCA FW. Which > is an issue as you say -- do we have an IEEE ethertype yet? And if we > don't use the IPv6 ethertype, then is multicast going to work well (if > the code is moved away from just broadcasting everything)? I doubt that > switch IGMP snooping works well for non-IP ethertypes -- in fact I > wonder how well existing switches handle IPv6 multicast ;) Hmm, murky indeed. Your point about IGMPv6 is well made. The problem is that IB GRHs are not IPv6 headers and have different numerology for the Next Header field. Ie in IPv6 Next Header 0x1B is RFC 908, while in GRH it is a BTH. Labeling GRHs with an IPv6 ethertype is fundamentally wrong. But also highly desirable for things like IGMPv6 snooping, etc. There is obviously alot of possible trade offs here, and it is impossible to judge any proposal until the issue of IBoE or RDMAoE and the intended goal of this thing is laid to rest. Jason From rdreier at cisco.com Wed Jun 17 14:10:06 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 17 Jun 2009 14:10:06 -0700 Subject: [ofa-general] [PATCH 2/9] ib_core: kernel API for GID -->MAC translations In-Reply-To: <20090617205257.GL18150@obsidianresearch.com> (Jason Gunthorpe's message of "Wed, 17 Jun 2009 14:52:57 -0600") References: <20090615073431.GC27543@mtls03> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A51@mtlexch01.mtl.com> <20090617181753.GI18150@obsidianresearch.com> <20090617183123.GJ18150@obsidianresearch.com> <20090617205257.GL18150@obsidianresearch.com> Message-ID: > Hmm, murky indeed. Your point about IGMPv6 is well made. The problem > is that IB GRHs are not IPv6 headers and have different numerology for > the Next Header field. Ie in IPv6 Next Header 0x1B is RFC 908, while > in GRH it is a BTH. Labeling GRHs with an IPv6 ethertype is > fundamentally wrong. Yes, but the next header is the only issue I know of. Since 0x1b is already assigned as an IPv6 next header protocol, we would have to get a new value assigned. However once a non-conflicting value is chosen, then an IB GRH really is an IPv6 header and in that case I think using the IPv6 ethertype too would make things work much better -- eg IB traffic actually could be forwarded by an IPv6 router with no additional work required. - R. From akepner at sgi.com Wed Jun 17 17:10:40 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Wed, 17 Jun 2009 17:10:40 -0700 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM In-Reply-To: <4A313AAE.2070108@voltaire.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> <4A313AAE.2070108@voltaire.com> Message-ID: <20090618001040.GI13049@sgi.com> On Thu, Jun 11, 2009 at 08:11:10PM +0300, Yossi Etigin wrote: > .... > How about checking the refcnt in the neigh_cleanup function, and if it's > not 1 give up the ipoib cleanup? > ipoib_neigh_cleanup() can be invoked even when there are > 1 references (e.g., if the interface is brought down that can happen). So using the refcnt in the neighbour isn't going to work. I'm going to send a very simple patch in just a minute. -- Arthur From akepner at sgi.com Wed Jun 17 17:44:22 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Wed, 17 Jun 2009 17:44:22 -0700 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in CM In-Reply-To: <20090618001040.GI13049@sgi.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> <4A313AAE.2070108@voltaire.com> <20090618001040.GI13049@sgi.com> Message-ID: <20090618004422.GJ13049@sgi.com> There is a race condition between the neighbour cleanup code, and some uses of the closely related ipoib_neigh structure when using IPoIB-CM. The root cause is that the struct ipoib_neigh pointer that's stashed away in struct neighbour is read (and subsequently used) in ipoib_neigh_cleanup() without using locking that's consistent with other reads/writes of this pointer. (The pointer must be read before it can be known which lock to use, so it's difficult to avoid.) Re-read the pointer in ipoib_neigh_cleanup() after the appropriate lock is acquired. Signed-off-by: Arthur Kepner --- ipoib_main.c | 29 +++++++++++++++++++++++------ 1 files changed, 23 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index ab2c192..901735b 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -841,10 +841,20 @@ static void ipoib_set_mcast_list(struct net_device *dev) static void ipoib_neigh_cleanup(struct neighbour *n) { struct ipoib_neigh *neigh; - struct ipoib_dev_priv *priv = netdev_priv(n->dev); + struct ipoib_dev_priv *priv; unsigned long flags; struct ipoib_ah *ah = NULL; + /* + * Note that the read of the neigh pointer below is not protected + * by a ipoib_dev_priv->lock (since we don't yet know which device's + * lock to use). Count on the fact that if ipoib_neigh_free() has + * already freed the struct ipoib_neigh, to_ipoib_neigh() will + * return NULL. + * + * If to_ipoib_neigh() does not return NULL, we'll re-read neigh + * under the appropriate lock below. + */ neigh = *to_ipoib_neigh(n); if (neigh) priv = netdev_priv(neigh->dev); @@ -856,11 +866,14 @@ static void ipoib_neigh_cleanup(struct neighbour *n) n->ha + 4); spin_lock_irqsave(&priv->lock, flags); - - if (neigh->ah) - ah = neigh->ah; - list_del(&neigh->list); - ipoib_neigh_free(n->dev, neigh); + /* re-read neigh under priv->lock */ + neigh = *to_ipoib_neigh(n); + if (neigh) { + if (neigh->ah) + ah = neigh->ah; + list_del(&neigh->list); + ipoib_neigh_free(n->dev, neigh); + } spin_unlock_irqrestore(&priv->lock, flags); @@ -888,7 +901,11 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, void ipoib_neigh_free(struct net_device *dev, struct ipoib_neigh *neigh) { + struct ipoib_dev_priv *priv = netdev_priv(neigh->dev); struct sk_buff *skb; + + BUG_ON(!spin_is_locked(&priv->lock)); + *to_ipoib_neigh(neigh->neighbour) = NULL; while ((skb = __skb_dequeue(&neigh->queue))) { ++dev->stats.tx_dropped; From acceptany at gmail.com Wed Jun 17 19:24:56 2009 From: acceptany at gmail.com (Jordan) Date: Thu, 18 Jun 2009 10:24:56 +0800 Subject: [ofa-general] What is the difference between forward table "lft" and "new_lft" ? Message-ID: <91fe68d50906171924v6a340427mbf2ee7eccd9268fc@mail.gmail.com> There are two forward tables "lft" and "new_lft" in the switch. It seems that the functions "do_ucast_file_load()" and "osm_ucast_mgr_set_fwd_table()" only update the table new_lft. Where the table "lft" is updated? Which table does the switch look up when a package is forwarded in switch? "lft" or "new_lft" ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From diego.guella at sircomtech.com Wed Jun 17 23:24:50 2009 From: diego.guella at sircomtech.com (Diego Guella) Date: Thu, 18 Jun 2009 08:24:50 +0200 Subject: [ofa-general] [PATCH] opensm/osm_ucast_lash.(h c): Replace memoryallocation by array References: <20090617200911.GB11146@comcast.net> Message-ID: <004701c9efdd$7c5c3b80$05c8a8c0@DIEGO> Hi Hal, >From: "Hal Rosenstock" > @@ -778,11 +776,7 @@ static int init_lash_structures(lash_t * p_lash) > } > > /* initialise num_mst_in_lane[num_switches], default 0 */ > - p_lash->num_mst_in_lane = (int *)malloc(num_switches * sizeof(int)); > - if (p_lash->num_mst_in_lane == NULL) > - goto Exit_Mem_Error; > - memset(p_lash->num_mst_in_lane, 0, > - num_switches * sizeof(p_lash->num_mst_in_lane[0])); > + memset(p_lash->num_mst_in_lane, 0, IB_MAX_NUM_VLS); Should it be memset(p_lash->num_mst_in_lane, 0, IB_MAX_NUM_VLS * sizeof(p_lash->num_mst_in_lane[0])); ? From amirv at mellanox.co.il Wed Jun 17 23:59:52 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 18 Jun 2009 09:59:52 +0300 Subject: [ofa-general] SDP - How to? In-Reply-To: <7d4423d30906172355x47040970lc11450afdf69a42d@mail.gmail.com> References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> <4A30A296.60109@mellanox.co.il> <7d4423d30906172355x47040970lc11450afdf69a42d@mail.gmail.com> Message-ID: <4A39E5E8.3080900@mellanox.co.il> HI, The number I sent you were measured using netperf TCP_STREAM test. With unmodified Mellanox ofed 1.4 running on high end hosts. - Amir On 06/18/2009 09:55 AM, Zafar Gilani wrote: > Hi Amir, > > I hope you are fine! I want to carry out benchmark testing to compare > IPoIB and SDP protocols on the cluster I am working on. Where can I > get test cases (such as pt2pt communications) to carry out these > tests? What approach did you use in your benchmark test that you sent > me (your sent message attached)? > > Thanks in advance! > > On Thu, Jun 11, 2009 at 11:22 AM, Amir Vadai > wrote: > > Hi Zafar, > > I am the maintainer of SDP in OFED. > > The current SDP implementation does not have ZCopy. What we have > is BCopy and something that we call BZcopy. > Small packets (<=32k) are sent using BCopy - meaning the SEND verb > is in use and data is not zero copied. > Packets bigger than 32K are sent using BZcopy - they are also sent > using the SEND verb but they are zero copied on the sender side. > on the RX side packets are memcpy'ed. > When we implement ZCopy it will be automatic too - when sending > packets bigger than a threshold the driver will automatically use > RDMA verb instead of the SEND. > > BTW, your numbers are different that what we measured. See > attached. Was measured on a high end host using QDR IB PCI gen 2 > and Mellanox ofed 1.4. > > - Amir > > > Best Regards, > Zafar > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zafargilani at gmail.com Wed Jun 17 23:55:34 2009 From: zafargilani at gmail.com (Zafar Gilani) Date: Thu, 18 Jun 2009 11:55:34 +0500 Subject: [ofa-general] SDP - How to? In-Reply-To: <4A30A296.60109@mellanox.co.il> References: <7d4423d30906100707h63c01095u18b3a53e8016fdb6@mail.gmail.com> <7d4423d30906100747l4dadfd53l9562ecaa492a4960@mail.gmail.com> <4A30A296.60109@mellanox.co.il> Message-ID: <7d4423d30906172355x47040970lc11450afdf69a42d@mail.gmail.com> Hi Amir, I hope you are fine! I want to carry out benchmark testing to compare IPoIB and SDP protocols on the cluster I am working on. Where can I get test cases (such as pt2pt communications) to carry out these tests? What approach did you use in your benchmark test that you sent me (your sent message attached)? Thanks in advance! On Thu, Jun 11, 2009 at 11:22 AM, Amir Vadai wrote: > Hi Zafar, > > I am the maintainer of SDP in OFED. > > The current SDP implementation does not have ZCopy. What we have is BCopy > and something that we call BZcopy. > Small packets (<=32k) are sent using BCopy - meaning the SEND verb is in > use and data is not zero copied. > Packets bigger than 32K are sent using BZcopy - they are also sent using > the SEND verb but they are zero copied on the sender side. on the RX side > packets are memcpy'ed. > When we implement ZCopy it will be automatic too - when sending packets > bigger than a threshold the driver will automatically use RDMA verb instead > of the SEND. > > BTW, your numbers are different that what we measured. See attached. Was > measured on a high end host using QDR IB PCI gen 2 and Mellanox ofed 1.4. > > - Amir > Best Regards, Zafar -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sdp_vs_ip_vs_10ge.png Type: image/png Size: 29636 bytes Desc: not available URL: From eli at dev.mellanox.co.il Thu Jun 18 01:46:06 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 18 Jun 2009 11:46:06 +0300 Subject: [ofa-general] Re: [ewg] Re: [PATCH 2/9] ib_core: kernel API for GID --> MAC translations In-Reply-To: References: <20090615073431.GC27543@mtls03> Message-ID: <20090618084606.GA24512@mtls03> On Wed, Jun 17, 2009 at 11:19:04AM -0700, Roland Dreier wrote: > > > diff --git a/include/rdma/ib_user_verbs.h b/include/rdma/ib_user_verbs.h > > index a17f771..5bfa2e6 100644 > > --- a/include/rdma/ib_user_verbs.h > > +++ b/include/rdma/ib_user_verbs.h > > @@ -81,7 +81,8 @@ enum { > > IB_USER_VERBS_CMD_MODIFY_SRQ, > > IB_USER_VERBS_CMD_QUERY_SRQ, > > IB_USER_VERBS_CMD_DESTROY_SRQ, > > - IB_USER_VERBS_CMD_POST_SRQ_RECV > > + IB_USER_VERBS_CMD_POST_SRQ_RECV, > > + IB_USER_VERBS_CMD_GET_MAC > > }; > > > > This breaks the ABI defined for XRC. So I guess these patches are not > really intended to be applied? > > I definitely want to finish up XRC before starting to look at this > series in detail anyway. > Roland, on top of which trees would like me to rebase the patches on and update the ABI version? From sebastien.dugue at bull.net Thu Jun 18 01:53:37 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Thu, 18 Jun 2009 10:53:37 +0200 Subject: [ofa-general] [PATCH 1/3] V2 libmthca - Optimize memory allocation of QP buffers In-Reply-To: <20090618105051.34c40d7f@frecb007965> References: <20090618105051.34c40d7f@frecb007965> Message-ID: <20090618105337.2c122df7@frecb007965> QP buffers are allocated with mthca_alloc_buf(), which rounds the buffers size to the page size and then allocates page aligned memory using posix_memalign(). However, this allocation is quite wasteful on architectures using 64K pages (ia64 for example) because we then hit glibc's MMAP_THRESHOLD malloc parameter and chunks are allocated using mmap. thus we end up allocating: (requested size rounded to the page size) + (page size) + (malloc overhead) rounded internally to the page size. So for example, if we request a buffer of page_size bytes, we end up consuming 3 pages. In short, for each QP buffer we allocate, there is an overhead of 2 pages. This is quite visible on large clusters especially where the number of QP can reach several thousands. This patch creates a new function mthca_alloc_page() for use by mthca_alloc_qp_buf() that does an mmap() instead of a posix_memalign(). Signed-off-by: Sebastien Dugue --- src/buf.c | 34 ++++++++++++++++++++++++++++++++-- src/mthca.h | 7 +++++++ src/qp.c | 7 ++++--- 3 files changed, 43 insertions(+), 5 deletions(-) diff --git a/src/buf.c b/src/buf.c index 6c1be4f..499edeb 100644 --- a/src/buf.c +++ b/src/buf.c @@ -35,6 +35,8 @@ #endif /* HAVE_CONFIG_H */ #include +#include +#include #include "mthca.h" @@ -69,8 +71,32 @@ int mthca_alloc_buf(struct mthca_buf *buf, size_t size, int page_size) if (ret) free(buf->buf); - if (!ret) + if (!ret) { buf->length = size; + buf->type = MTHCA_MALIGN; + } + + return ret; +} + +int mthca_alloc_page(struct mthca_buf *buf, size_t size, int page_size) +{ + int ret; + + /* Use mmap directly to allocate an aligned buffer */ + buf->buf = mmap(0 ,align(size, page_size) , PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + + if (buf->buf == MAP_FAILED) + return errno; + + ret = ibv_dontfork_range(buf->buf, size); + if (ret) + munmap(buf->buf, align(size, page_size)); + else { + buf->length = size; + buf->type = MTHCA_MMAP; + } return ret; } @@ -78,5 +104,9 @@ int mthca_alloc_buf(struct mthca_buf *buf, size_t size, int page_size) void mthca_free_buf(struct mthca_buf *buf) { ibv_dofork_range(buf->buf, buf->length); - free(buf->buf); + + if ( buf->type == MTHCA_MMAP ) + munmap(buf->buf, buf->length); + else + free(buf->buf); } diff --git a/src/mthca.h b/src/mthca.h index 66751f3..7db15a7 100644 --- a/src/mthca.h +++ b/src/mthca.h @@ -138,9 +138,15 @@ struct mthca_context { int qp_table_mask; }; +enum mthca_buf_type { + MTHCA_MMAP, + MTHCA_MALIGN +}; + struct mthca_buf { void *buf; size_t length; + enum mthca_buf_type type; }; struct mthca_pd { @@ -291,6 +297,7 @@ static inline int mthca_is_memfree(struct ibv_context *ibctx) } int mthca_alloc_buf(struct mthca_buf *buf, size_t size, int page_size); +int mthca_alloc_page(struct mthca_buf *buf, size_t size, int page_size); void mthca_free_buf(struct mthca_buf *buf); int mthca_alloc_db(struct mthca_db_table *db_tab, enum mthca_db_type type, diff --git a/src/qp.c b/src/qp.c index 84dd206..15f4805 100644 --- a/src/qp.c +++ b/src/qp.c @@ -848,9 +848,10 @@ int mthca_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *cap, qp->buf_size = qp->send_wqe_offset + (qp->sq.max << qp->sq.wqe_shift); - if (mthca_alloc_buf(&qp->buf, - align(qp->buf_size, to_mdev(pd->context->device)->page_size), - to_mdev(pd->context->device)->page_size)) { + if (mthca_alloc_page(&qp->buf, + align(qp->buf_size, + to_mdev(pd->context->device)->page_size), + to_mdev(pd->context->device)->page_size)) { free(qp->wrid); return -1; } -- 1.6.3.1 From sebastien.dugue at bull.net Thu Jun 18 01:50:51 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Thu, 18 Jun 2009 10:50:51 +0200 Subject: [ofa-general] [PATCH 0/3 RESEND] V2 - libmthca libmlx4 - Optimize memory allocation of QP buffers Message-ID: <20090618105051.34c40d7f@frecb007965> Hi, here is a resend of the QP buffers memory allocation optimization patches in which QP buffers are allocated using mmap() regardless of the page size. Changes V1 -> V2: ---------------- - Use mmap whatever the page size, not only with 64K pages. libmthca and libmlx4 allocate QP buffers using posix_memalign(), which results in big memory wastage on architectures with 64K pages. Replacing posix_memalign() with mmap() allows to fix this (more description in the patches themselves). Now, for some numbers, a micro benchmark I wrote shows the heap usage and the number of mmaped pages used with posix_memalign() and mmap() respectively for 1000, 2000, up to 8000 QP. MTHCA posix_memalign mmap QP heap mmaped(pages) heap mmaped(pages) 1000 838736 2988 576512 1000 2000 1751216 5973 1161264 2000 3000 2598144 8961 1746016 3000 4000 3510656 11946 2330704 4000 5000 4357616 14934 2915440 5000 6000 5270080 17919 3500176 6000 7000 6117056 20907 4084912 7000 8000 6963968 23895 4669632 8000 MLX4 posix_memalign mmap QP heap mmaped(pages) heap mmaped(pages) 1000 1469424 2982 1010544 1003 2000 2994048 5958 2010752 2003 3000 4518672 8934 3010960 3003 4000 5969520 11913 4002960 4003 5000 7494176 14889 5003168 5003 6000 8953248 17868 6003376 6003 7000 10477856 20844 7003584 7003 8000 12002496 23820 8003792 8003 This patchset consists in 3 patches: 1. Optimize memory allocation of QP buffers for libmthca 2. Optimize memory allocation of QP buffers for libmlx4 3. Fix the fixes patches for libmlx4 after having applied the previous patch. Sebastien Dugue From sebastien.dugue at bull.net Thu Jun 18 01:54:35 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Thu, 18 Jun 2009 10:54:35 +0200 Subject: [ofa-general] [PATCH 3/3] V2 libmlx4 - Fix fixes after QP buffers alloc optimization patch to allow build In-Reply-To: <20090618105051.34c40d7f@frecb007965> References: <20090618105051.34c40d7f@frecb007965> Message-ID: <20090618105435.6276903e@frecb007965> The patches in 'fixes/' need to be refreshed after the previous patch in order to build properly. Signed-off-by: Sebastien Dugue --- fixes/lim_qp_resources.patch | 20 ++++------- fixes/resize_cq_owner_bit.patch | 4 +-- fixes/userspace_dev_lims.patch | 12 ++---- fixes/xrc_consolidated_v2.patch | 68 ++++++++++++++------------------------ fixes/xrc_fix_close_domain.patch | 8 ++--- fixes/xrc_rcv_qp_v2.patch | 12 ++----- 6 files changed, 44 insertions(+), 80 deletions(-) diff --git a/fixes/lim_qp_resources.patch b/fixes/lim_qp_resources.patch index 1f89256..54cc63e 100644 --- a/fixes/lim_qp_resources.patch +++ b/fixes/lim_qp_resources.patch @@ -7,11 +7,9 @@ qp creation also lie within the reported device limits. Signed-off-by: Jack Morgenstein -Index: libmlx4/src/qp.c -=================================================================== ---- libmlx4.orig/src/qp.c 2008-06-04 08:24:45.000000000 +0300 -+++ libmlx4/src/qp.c 2008-06-04 08:24:49.000000000 +0300 -@@ -619,6 +619,7 @@ void mlx4_set_sq_sizes(struct mlx4_qp *q +--- a/src/qp.c ++++ b/src/qp.c +@@ -622,6 +622,7 @@ void mlx4_set_sq_sizes(struct mlx4_qp *q enum ibv_qp_type type) { int wqe_size; @@ -19,7 +17,7 @@ Index: libmlx4/src/qp.c wqe_size = (1 << qp->sq.wqe_shift) - sizeof (struct mlx4_wqe_ctrl_seg); switch (type) { -@@ -636,8 +637,9 @@ void mlx4_set_sq_sizes(struct mlx4_qp *q +@@ -639,8 +640,9 @@ void mlx4_set_sq_sizes(struct mlx4_qp *q } qp->sq.max_gs = wqe_size / sizeof (struct mlx4_wqe_data_seg); @@ -31,10 +29,8 @@ Index: libmlx4/src/qp.c cap->max_send_wr = qp->sq.max_post; /* -Index: libmlx4/src/verbs.c -=================================================================== ---- libmlx4.orig/src/verbs.c 2008-06-04 08:24:45.000000000 +0300 -+++ libmlx4/src/verbs.c 2008-06-04 08:24:49.000000000 +0300 +--- a/src/verbs.c ++++ b/src/verbs.c @@ -390,12 +390,14 @@ struct ibv_qp *mlx4_create_qp(struct ibv struct ibv_create_qp_resp resp; struct mlx4_qp *qp; @@ -54,9 +50,9 @@ Index: libmlx4/src/verbs.c attr->cap.max_inline_data > 1024) return NULL; -@@ -461,8 +463,14 @@ struct ibv_qp *mlx4_create_qp(struct ibv - if (ret) +@@ -464,8 +466,14 @@ struct ibv_qp *mlx4_create_qp(struct ibv goto err_destroy; + pthread_mutex_unlock(&to_mctx(pd->context)->qp_table_mutex); - qp->rq.wqe_cnt = qp->rq.max_post = attr->cap.max_recv_wr; + qp->rq.wqe_cnt = attr->cap.max_recv_wr; diff --git a/fixes/resize_cq_owner_bit.patch b/fixes/resize_cq_owner_bit.patch index 6557027..0a5b564 100644 --- a/fixes/resize_cq_owner_bit.patch +++ b/fixes/resize_cq_owner_bit.patch @@ -3,11 +3,9 @@ for the target buffer (and not left as it was in the source buffer). Signed-off-by: Jack Morgenstein -diff --git a/src/cq.c b/src/cq.c -index 68e16e9..8226b6b 100644 --- a/src/cq.c +++ b/src/cq.c -@@ -455,6 +455,8 @@ void mlx4_cq_resize_copy_cqes(struct mlx4_cq *cq, void *buf, int old_cqe) +@@ -478,6 +478,8 @@ void mlx4_cq_resize_copy_cqes(struct mlx cqe = get_cqe(cq, (i & old_cqe)); while ((cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) != MLX4_CQE_OPCODE_RESIZE) { diff --git a/fixes/userspace_dev_lims.patch b/fixes/userspace_dev_lims.patch index 07cf638..80d4d14 100644 --- a/fixes/userspace_dev_lims.patch +++ b/fixes/userspace_dev_lims.patch @@ -9,10 +9,8 @@ preferable to breaking the ABI. Signed-off-by: Jack Morgenstein -Index: libmlx4/src/mlx4.c -=================================================================== ---- libmlx4.orig/src/mlx4.c 2008-06-03 15:45:18.000000000 +0300 -+++ libmlx4/src/mlx4.c 2008-06-04 08:24:10.000000000 +0300 +--- a/src/mlx4.c ++++ b/src/mlx4.c @@ -104,6 +104,7 @@ static struct ibv_context *mlx4_alloc_co struct ibv_get_context cmd; struct mlx4_alloc_ucontext_resp resp; @@ -42,10 +40,8 @@ Index: libmlx4/src/mlx4.c err_free: free(context); return NULL; -Index: libmlx4/src/mlx4.h -=================================================================== ---- libmlx4.orig/src/mlx4.h 2008-06-03 15:45:18.000000000 +0300 -+++ libmlx4/src/mlx4.h 2008-06-04 08:24:10.000000000 +0300 +--- a/src/mlx4.h ++++ b/src/mlx4.h @@ -83,6 +83,20 @@ #define PFX "mlx4: " diff --git a/fixes/xrc_consolidated_v2.patch b/fixes/xrc_consolidated_v2.patch index 6fbd0a9..78a4f6c 100644 --- a/fixes/xrc_consolidated_v2.patch +++ b/fixes/xrc_consolidated_v2.patch @@ -18,8 +18,6 @@ V2: 2. Changed xrc_ops to more ops 3. Check for xrc verbs in ibv_more_ops via AC_CHECK_MEMBER -diff --git a/configure.in b/configure.in -index 25f27f7..46a3a64 100644 --- a/configure.in +++ b/configure.in @@ -42,6 +42,12 @@ AC_CHECK_HEADER(valgrind/memcheck.h, @@ -35,11 +33,9 @@ index 25f27f7..46a3a64 100644 dnl Checks for library functions AC_CHECK_FUNC(ibv_read_sysfs_file, [], -diff --git a/src/cq.c b/src/cq.c -index 68e16e9..c598b87 100644 --- a/src/cq.c +++ b/src/cq.c -@@ -194,8 +194,9 @@ static int mlx4_poll_one(struct mlx4_cq *cq, +@@ -194,8 +194,9 @@ static int mlx4_poll_one(struct mlx4_cq { struct mlx4_wq *wq; struct mlx4_cqe *cqe; @@ -50,7 +46,7 @@ index 68e16e9..c598b87 100644 uint32_t g_mlpath_rqpn; uint16_t wqe_index; int is_error; -@@ -221,20 +223,29 @@ static int mlx4_poll_one(struct mlx4_cq *cq, +@@ -221,20 +222,29 @@ static int mlx4_poll_one(struct mlx4_cq is_error = (cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) == MLX4_CQE_OPCODE_ERROR; @@ -84,7 +80,7 @@ index 68e16e9..c598b87 100644 if (is_send) { wq = &(*cur_qp)->sq; -@@ -242,6 +254,10 @@ static int mlx4_poll_one(struct mlx4_cq *cq, +@@ -242,6 +252,10 @@ static int mlx4_poll_one(struct mlx4_cq wq->tail += (uint16_t) (wqe_index - (uint16_t) wq->tail); wc->wr_id = wq->wrid[wq->tail & (wq->wqe_cnt - 1)]; ++wq->tail; @@ -95,7 +91,7 @@ index 68e16e9..c598b87 100644 } else if ((*cur_qp)->ibv_qp.srq) { srq = to_msrq((*cur_qp)->ibv_qp.srq); wqe_index = htons(cqe->wqe_index); -@@ -387,6 +403,10 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, uint32_t qpn, struct mlx4_srq *srq) +@@ -387,6 +401,10 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, uint32_t prod_index; uint8_t owner_bit; int nfreed = 0; @@ -106,7 +102,7 @@ index 68e16e9..c598b87 100644 /* * First we need to find the current producer index, so we -@@ -405,7 +425,12 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, uint32_t qpn, struct mlx4_srq *srq) +@@ -405,7 +423,12 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, */ while ((int) --prod_index - (int) cq->cons_index >= 0) { cqe = get_cqe(cq, prod_index & cq->ibv_cq.cqe); @@ -120,8 +116,6 @@ index 68e16e9..c598b87 100644 if (srq && !(cqe->owner_sr_opcode & MLX4_CQE_IS_SEND_MASK)) mlx4_free_srq_wqe(srq, ntohs(cqe->wqe_index)); ++nfreed; -diff --git a/src/mlx4-abi.h b/src/mlx4-abi.h -index 20a40c9..1b1253c 100644 --- a/src/mlx4-abi.h +++ b/src/mlx4-abi.h @@ -68,6 +68,14 @@ struct mlx4_resize_cq { @@ -152,8 +146,6 @@ index 20a40c9..1b1253c 100644 +#endif + #endif /* MLX4_ABI_H */ -diff --git a/src/mlx4.c b/src/mlx4.c -index 671e849..27ca75d 100644 --- a/src/mlx4.c +++ b/src/mlx4.c @@ -68,6 +68,16 @@ struct { @@ -173,7 +165,7 @@ index 671e849..27ca75d 100644 static struct ibv_context_ops mlx4_ctx_ops = { .query_device = mlx4_query_device, .query_port = mlx4_query_port, -@@ -124,6 +134,15 @@ static struct ibv_context *mlx4_alloc_context(struct ibv_device *ibdev, int cmd_ +@@ -124,6 +134,15 @@ static struct ibv_context *mlx4_alloc_co for (i = 0; i < MLX4_QP_TABLE_SIZE; ++i) context->qp_table[i].refcnt = 0; @@ -189,7 +181,7 @@ index 671e849..27ca75d 100644 for (i = 0; i < MLX4_NUM_DB_TYPE; ++i) context->db_list[i] = NULL; -@@ -156,6 +175,9 @@ static struct ibv_context *mlx4_alloc_context(struct ibv_device *ibdev, int cmd_ +@@ -156,6 +175,9 @@ static struct ibv_context *mlx4_alloc_co pthread_spin_init(&context->uar_lock, PTHREAD_PROCESS_PRIVATE); context->ibv_ctx.ops = mlx4_ctx_ops; @@ -199,8 +191,6 @@ index 671e849..27ca75d 100644 if (mlx4_query_device(&context->ibv_ctx, &dev_attrs)) goto query_free; -diff --git a/src/mlx4.h b/src/mlx4.h -index 8643d8f..3eadb98 100644 --- a/src/mlx4.h +++ b/src/mlx4.h @@ -79,6 +79,11 @@ @@ -248,7 +238,7 @@ index 8643d8f..3eadb98 100644 struct mlx4_db_page *db_list[MLX4_NUM_DB_TYPE]; pthread_mutex_t db_list_mutex; }; -@@ -260,6 +284,11 @@ struct mlx4_ah { +@@ -266,6 +290,11 @@ struct mlx4_ah { struct mlx4_av av; }; @@ -260,7 +250,7 @@ index 8643d8f..3eadb98 100644 static inline unsigned long align(unsigned long val, unsigned long align) { return (val + align - 1) & ~(align - 1); -@@ -304,6 +333,13 @@ static inline struct mlx4_ah *to_mah(struct ibv_ah *ibah) +@@ -310,6 +339,13 @@ static inline struct mlx4_ah *to_mah(str return to_mxxx(ah, ah); } @@ -272,9 +262,9 @@ index 8643d8f..3eadb98 100644 +#endif + int mlx4_alloc_buf(struct mlx4_buf *buf, size_t size, int page_size); + int mlx4_alloc_page(struct mlx4_buf *buf, size_t size, int page_size); void mlx4_free_buf(struct mlx4_buf *buf); - -@@ -350,6 +386,10 @@ void mlx4_free_srq_wqe(struct mlx4_srq *srq, int ind); +@@ -357,6 +393,10 @@ void mlx4_free_srq_wqe(struct mlx4_srq * int mlx4_post_srq_recv(struct ibv_srq *ibsrq, struct ibv_recv_wr *wr, struct ibv_recv_wr **bad_wr); @@ -285,7 +275,7 @@ index 8643d8f..3eadb98 100644 struct ibv_qp *mlx4_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *attr); int mlx4_query_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, -@@ -380,5 +420,16 @@ int mlx4_alloc_av(struct mlx4_pd *pd, struct ibv_ah_attr *attr, +@@ -387,5 +427,16 @@ int mlx4_alloc_av(struct mlx4_pd *pd, st void mlx4_free_av(struct mlx4_ah *ah); int mlx4_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); int mlx4_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); @@ -302,11 +292,9 @@ index 8643d8f..3eadb98 100644 + #endif /* MLX4_H */ -diff --git a/src/qp.c b/src/qp.c -index 01e8580..2f02430 100644 --- a/src/qp.c +++ b/src/qp.c -@@ -226,7 +226,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, +@@ -226,7 +226,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, ctrl = wqe = get_send_wqe(qp, ind & (qp->sq.wqe_cnt - 1)); qp->sq.wrid[ind & (qp->sq.wqe_cnt - 1)] = wr->wr_id; @@ -315,7 +303,7 @@ index 01e8580..2f02430 100644 (wr->send_flags & IBV_SEND_SIGNALED ? htonl(MLX4_WQE_CTRL_CQ_UPDATE) : 0) | (wr->send_flags & IBV_SEND_SOLICITED ? -@@ -243,6 +243,9 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, +@@ -243,6 +243,9 @@ int mlx4_post_send(struct ibv_qp *ibqp, size = sizeof *ctrl / 16; switch (ibqp->qp_type) { @@ -325,7 +313,7 @@ index 01e8580..2f02430 100644 case IBV_QPT_RC: case IBV_QPT_UC: switch (wr->opcode) { -@@ -543,6 +546,7 @@ void mlx4_calc_sq_wqe_size(struct ibv_qp_cap *cap, enum ibv_qp_type type, +@@ -543,6 +546,7 @@ void mlx4_calc_sq_wqe_size(struct ibv_qp size += sizeof (struct mlx4_wqe_raddr_seg); break; @@ -333,7 +321,7 @@ index 01e8580..2f02430 100644 case IBV_QPT_RC: size += sizeof (struct mlx4_wqe_raddr_seg); /* -@@ -631,6 +635,7 @@ void mlx4_set_sq_sizes(struct mlx4_qp *qp, struct ibv_qp_cap *cap, +@@ -632,6 +636,7 @@ void mlx4_set_sq_sizes(struct mlx4_qp *q case IBV_QPT_UC: case IBV_QPT_RC: @@ -341,11 +329,9 @@ index 01e8580..2f02430 100644 wqe_size -= sizeof (struct mlx4_wqe_raddr_seg); break; -diff --git a/src/srq.c b/src/srq.c -index ba2ceb9..1350792 100644 --- a/src/srq.c +++ b/src/srq.c -@@ -167,3 +167,53 @@ int mlx4_alloc_srq_buf(struct ibv_pd *pd, struct ibv_srq_attr *attr, +@@ -167,3 +167,53 @@ int mlx4_alloc_srq_buf(struct ibv_pd *pd return 0; } @@ -399,8 +385,6 @@ index ba2ceb9..1350792 100644 + pthread_mutex_unlock(&ctx->xrc_srq_table_mutex); +} + -diff --git a/src/verbs.c b/src/verbs.c -index 400050c..b7c9c8e 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -368,18 +368,36 @@ int mlx4_query_srq(struct ibv_srq *srq, @@ -447,7 +431,7 @@ index 400050c..b7c9c8e 100644 return 0; } -@@ -415,7 +433,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *attr) +@@ -415,7 +433,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv qp->sq.wqe_cnt = align_queue_size(attr->cap.max_send_wr + qp->sq_spare_wqes); qp->rq.wqe_cnt = align_queue_size(attr->cap.max_recv_wr); @@ -456,7 +440,7 @@ index 400050c..b7c9c8e 100644 attr->cap.max_recv_wr = qp->rq.wqe_cnt = 0; else { if (attr->cap.max_recv_sge < 1) -@@ -433,7 +451,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *attr) +@@ -433,7 +451,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv pthread_spin_init(&qp->rq.lock, PTHREAD_PROCESS_PRIVATE)) goto err_free; @@ -465,7 +449,7 @@ index 400050c..b7c9c8e 100644 qp->db = mlx4_alloc_db(to_mctx(pd->context), MLX4_DB_TYPE_RQ); if (!qp->db) goto err_free; -@@ -442,7 +460,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *attr) +@@ -442,7 +460,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv } cmd.buf_addr = (uintptr_t) qp->buf.buf; @@ -474,7 +458,7 @@ index 400050c..b7c9c8e 100644 cmd.db_addr = 0; else cmd.db_addr = (uintptr_t) qp->db; -@@ -485,7 +503,7 @@ err_destroy: +@@ -489,7 +507,7 @@ err_destroy: err_rq_db: pthread_mutex_unlock(&to_mctx(pd->context)->qp_table_mutex); @@ -483,7 +467,7 @@ index 400050c..b7c9c8e 100644 mlx4_free_db(to_mctx(pd->context), MLX4_DB_TYPE_RQ, qp->db); err_free: -@@ -544,7 +562,7 @@ int mlx4_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, +@@ -548,7 +566,7 @@ int mlx4_modify_qp(struct ibv_qp *qp, st mlx4_cq_clean(to_mcq(qp->send_cq), qp->qp_num, NULL); mlx4_init_qp_indices(to_mqp(qp)); @@ -492,16 +476,16 @@ index 400050c..b7c9c8e 100644 *to_mqp(qp)->db = 0; } -@@ -603,7 +621,7 @@ int mlx4_destroy_qp(struct ibv_qp *ibqp) - +@@ -611,7 +629,7 @@ int mlx4_destroy_qp(struct ibv_qp *ibqp) mlx4_unlock_cqs(ibqp); + pthread_mutex_unlock(&to_mctx(ibqp->context)->qp_table_mutex); - if (!ibqp->srq) + if (!ibqp->srq && ibqp->qp_type != IBV_QPT_XRC) mlx4_free_db(to_mctx(ibqp->context), MLX4_DB_TYPE_RQ, qp->db); free(qp->sq.wrid); if (qp->rq.wqe_cnt) -@@ -661,3 +679,103 @@ int mlx4_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid) +@@ -669,3 +687,103 @@ int mlx4_detach_mcast(struct ibv_qp *qp, { return ibv_cmd_detach_mcast(qp, gid, lid); } @@ -605,8 +589,6 @@ index 400050c..b7c9c8e 100644 + return 0; +} +#endif -diff --git a/src/wqe.h b/src/wqe.h -index 6f7f309..fa2f8ac 100644 --- a/src/wqe.h +++ b/src/wqe.h @@ -65,7 +65,7 @@ struct mlx4_wqe_ctrl_seg { diff --git a/fixes/xrc_fix_close_domain.patch b/fixes/xrc_fix_close_domain.patch index dfad7ac..3af2640 100644 --- a/fixes/xrc_fix_close_domain.patch +++ b/fixes/xrc_fix_close_domain.patch @@ -6,11 +6,9 @@ Need to pass this upward to caller. Signed-off-by: Jack Morgenstein -Index: libmlx4/src/verbs.c -=================================================================== ---- libmlx4.orig/src/verbs.c 2008-09-01 10:51:11.000000000 +0300 -+++ libmlx4/src/verbs.c 2008-09-01 10:52:40.000000000 +0300 -@@ -774,9 +774,11 @@ +--- a/src/verbs.c ++++ b/src/verbs.c +@@ -782,9 +782,11 @@ struct ibv_xrc_domain *mlx4_open_xrc_dom int mlx4_close_xrc_domain(struct ibv_xrc_domain *d) { diff --git a/fixes/xrc_rcv_qp_v2.patch b/fixes/xrc_rcv_qp_v2.patch index 311c500..00ffd53 100644 --- a/fixes/xrc_rcv_qp_v2.patch +++ b/fixes/xrc_rcv_qp_v2.patch @@ -5,11 +5,9 @@ Signed-off-by: Jack Morgenstein V2: 1. xrc_ops changed to more_ops -diff --git a/src/mlx4.c b/src/mlx4.c -index 27ca75d..e5ded78 100644 --- a/src/mlx4.c +++ b/src/mlx4.c -@@ -74,6 +74,11 @@ static struct ibv_more_ops mlx4_more_ops = { +@@ -74,6 +74,11 @@ static struct ibv_more_ops mlx4_more_ops .create_xrc_srq = mlx4_create_xrc_srq, .open_xrc_domain = mlx4_open_xrc_domain, .close_xrc_domain = mlx4_close_xrc_domain, @@ -21,11 +19,9 @@ index 27ca75d..e5ded78 100644 #endif }; #endif -diff --git a/src/mlx4.h b/src/mlx4.h -index 3eadb98..6307a2d 100644 --- a/src/mlx4.h +++ b/src/mlx4.h -@@ -429,6 +429,21 @@ struct ibv_xrc_domain *mlx4_open_xrc_domain(struct ibv_context *context, +@@ -436,6 +436,21 @@ struct ibv_xrc_domain *mlx4_open_xrc_dom int fd, int oflag); int mlx4_close_xrc_domain(struct ibv_xrc_domain *d); @@ -47,11 +43,9 @@ index 3eadb98..6307a2d 100644 #endif -diff --git a/src/verbs.c b/src/verbs.c -index b7c9c8e..8261eae 100644 --- a/src/verbs.c +++ b/src/verbs.c -@@ -778,4 +778,59 @@ int mlx4_close_xrc_domain(struct ibv_xrc_domain *d) +@@ -786,4 +786,59 @@ int mlx4_close_xrc_domain(struct ibv_xrc free(d); return 0; } -- 1.6.3.1 From sebastien.dugue at bull.net Thu Jun 18 01:53:49 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Thu, 18 Jun 2009 10:53:49 +0200 Subject: [ofa-general] [PATCH 2/3] V2 libmlx4 - Optimize memory allocation of QP buffers In-Reply-To: <20090618105051.34c40d7f@frecb007965> References: <20090618105051.34c40d7f@frecb007965> Message-ID: <20090618105349.4fcffb5f@frecb007965> QP buffers are allocated with mlx4_alloc_buf(), which rounds the buffers size to the page size and then allocates page aligned memory using posix_memalign(). However, this allocation is quite wasteful on architectures using 64K pages (ia64 for example) because we then hit glibc's MMAP_THRESHOLD malloc parameter and chunks are allocated using mmap. thus we end up allocating: (requested size rounded to the page size) + (page size) + (malloc overhead) rounded internally to the page size. So for example, if we request a buffer of page_size bytes, we end up consuming 3 pages. In short, for each QP buffer we allocate, there is an overhead of 2 pages. This is quite visible on large clusters especially where the number of QP can reach several thousands. This patch creates a new function mlx4_alloc_page() for use by mlx4_alloc_qp_buf() that does an mmap() instead of a posix_memalign(). Signed-off-by: Sebastien Dugue --- src/buf.c | 34 ++++++++++++++++++++++++++++++++-- src/mlx4.h | 7 +++++++ src/qp.c | 5 +++-- 3 files changed, 42 insertions(+), 4 deletions(-) diff --git a/src/buf.c b/src/buf.c index 0e5f9b6..73565e6 100644 --- a/src/buf.c +++ b/src/buf.c @@ -35,6 +35,8 @@ #endif /* HAVE_CONFIG_H */ #include +#include +#include #include "mlx4.h" @@ -69,14 +71,42 @@ int mlx4_alloc_buf(struct mlx4_buf *buf, size_t size, int page_size) if (ret) free(buf->buf); - if (!ret) + if (!ret) { buf->length = size; + buf->type = MLX4_MALIGN; + } return ret; } +int mlx4_alloc_page(struct mlx4_buf *buf, size_t size, int page_size) +{ + int ret; + + /* Use mmap directly to allocate an aligned buffer */ + buf->buf = mmap(0 ,align(size, page_size) , PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + + if (buf->buf == MAP_FAILED) + return errno; + + ret = ibv_dontfork_range(buf->buf, size); + if (ret) + munmap(buf->buf, align(size, page_size)); + else { + buf->length = size; + buf->type = MLX4_MMAP; + } + + return ret; + } + void mlx4_free_buf(struct mlx4_buf *buf) { ibv_dofork_range(buf->buf, buf->length); - free(buf->buf); + + if ( buf->type == MLX4_MMAP ) + munmap(buf->buf, buf->length); + else + free(buf->buf); } diff --git a/src/mlx4.h b/src/mlx4.h index 827a201..83547f5 100644 --- a/src/mlx4.h +++ b/src/mlx4.h @@ -161,9 +161,15 @@ struct mlx4_context { pthread_mutex_t db_list_mutex; }; +enum mlx4_buf_type { + MLX4_MMAP, + MLX4_MALIGN +}; + struct mlx4_buf { void *buf; size_t length; + enum mlx4_buf_type type; }; struct mlx4_pd { @@ -288,6 +294,7 @@ static inline struct mlx4_ah *to_mah(struct ibv_ah *ibah) } int mlx4_alloc_buf(struct mlx4_buf *buf, size_t size, int page_size); +int mlx4_alloc_page(struct mlx4_buf *buf, size_t size, int page_size); void mlx4_free_buf(struct mlx4_buf *buf); uint32_t *mlx4_alloc_db(struct mlx4_context *context, enum mlx4_db_type type); diff --git a/src/qp.c b/src/qp.c index d194ae3..557e255 100644 --- a/src/qp.c +++ b/src/qp.c @@ -604,8 +604,9 @@ int mlx4_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *cap, qp->sq.offset = 0; } - if (mlx4_alloc_buf(&qp->buf, - align(qp->buf_size, to_mdev(pd->context->device)->page_size), + if (mlx4_alloc_page(&qp->buf, + align(qp->buf_size, + to_mdev(pd->context->device)->page_size), to_mdev(pd->context->device)->page_size)) { free(qp->sq.wrid); free(qp->rq.wrid); -- 1.6.3.1 From vlad at lists.openfabrics.org Thu Jun 18 02:27:53 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 18 Jun 2009 02:27:53 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090618-0200 daily build status Message-ID: <20090618092754.12A1710205E8@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090618-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From sashak at voltaire.com Thu Jun 18 03:10:06 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:10:06 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_inform.c: For traps 64-67, use GID in DataDetails in log messages In-Reply-To: References: <20090617134046.GA10939@comcast.net> <20090617164624.GQ6163@me> Message-ID: <20090618101006.GS6163@me> On 16:36 Wed 17 Jun , Hal Rosenstock wrote: > > Not sure what you mean by "unified log". The one with single OSM_LOG() call, without flow around. Sasha From sashak at voltaire.com Thu Jun 18 03:16:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:16:52 +0300 Subject: [ofa-general] Re: [PATCH V2] ibnetdiscover: only report max hops discovered when requested In-Reply-To: <20090513093020.f85f2a0a.weiny2@llnl.gov> References: <20090513093020.f85f2a0a.weiny2@llnl.gov> Message-ID: <20090618101652.GT6163@me> On 09:30 Wed 13 May , Ira Weiny wrote: > Added "-m" flag to report this information if the user wants it. I also > changed the text in the message which says "reported max hops discovered". I > don't know if we want to change that text to something else but I wanted to > indicate this number is not constant and may change. This is true not just if > you change the algorithm of discovery but also if you run from different > nodes. > > Thoughts, > Ira > > > From: Ira Weiny > Date: Wed, 6 May 2009 17:56:23 -0700 > Subject: [PATCH] ibnetdiscover: only report max hops discovered when requested > > > Signed-off-by: Ira Weiny Applied. Thanks. I'm changing 'max_hops' variable name to 'report_max_hops'. Sasha From sashak at voltaire.com Thu Jun 18 03:20:09 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:20:09 +0300 Subject: [ofa-general] Re: [PATCH] iblinkinfo, ibqueryerrors: prevent core when switch is not found In-Reply-To: <20090514160417.e7505e06.weiny2@llnl.gov> References: <20090514160417.e7505e06.weiny2@llnl.gov> Message-ID: <20090618102009.GU6163@me> On 16:04 Thu 14 May , Ira Weiny wrote: > > From: Ira Weiny > Date: Thu, 14 May 2009 15:52:42 -0700 > Subject: [PATCH] iblinkinfo, ibqueryerrors: prevent core when switch is not found > > If the switch is not found print nice error message instead of seg faulting > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 18 03:20:30 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:20:30 +0300 Subject: [ofa-general] Re: [PATCH] iblinkinfo: remove unused file pointer. In-Reply-To: <20090514164210.8a42f37d.weiny2@llnl.gov> References: <20090514164210.8a42f37d.weiny2@llnl.gov> Message-ID: <20090618102030.GV6163@me> On 16:42 Thu 14 May , Ira Weiny wrote: > > From: Ira Weiny > Date: Thu, 14 May 2009 16:39:52 -0700 > Subject: [PATCH] iblinkinfo: remove unused file pointer. > > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 18 03:22:27 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:22:27 +0300 Subject: [ofa-general] Re: [PATCH] iblinkinfo: convert to ibdiag_process_opts In-Reply-To: <20090617101024.cab3ea90.weiny2@llnl.gov> References: <20090617101024.cab3ea90.weiny2@llnl.gov> Message-ID: <20090618102227.GW6163@me> On 10:10 Wed 17 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Thu, 14 May 2009 17:23:22 -0700 > Subject: [PATCH] iblinkinfo: convert to ibdiag_process_opts > > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 18 03:24:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:24:52 +0300 Subject: [ofa-general] Re: [PATCH] ibqueryerrors: fix custom opts string In-Reply-To: <20090617101056.c1873840.weiny2@llnl.gov> References: <20090617101056.c1873840.weiny2@llnl.gov> Message-ID: <20090618102452.GX6163@me> On 10:10 Wed 17 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Thu, 14 May 2009 17:24:29 -0700 > Subject: [PATCH] ibqueryerrors: fix custom opts string > > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 18 03:25:26 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 13:25:26 +0300 Subject: [ofa-general] Re: [PATCHv2] libibmad/resolve.c: Remove compile warning on x86_64 arch In-Reply-To: <20090617200811.GA11146@comcast.net> References: <20090617200811.GA11146@comcast.net> Message-ID: <20090618102526.GY6163@me> On 16:08 Wed 17 Jun , Hal Rosenstock wrote: > > resolve.c:106: warning: pointer targets in passing argument 1 of 'mad_decode_field' differ in signedness > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hal.rosenstock at gmail.com Thu Jun 18 04:28:06 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 18 Jun 2009 07:28:06 -0400 Subject: [ofa-general] [PATCH] opensm/osm_ucast_lash.(h c): Replace memoryallocation by array In-Reply-To: <004701c9efdd$7c5c3b80$05c8a8c0@DIEGO> References: <20090617200911.GB11146@comcast.net> <004701c9efdd$7c5c3b80$05c8a8c0@DIEGO> Message-ID: Hi Diego, On Thu, Jun 18, 2009 at 2:24 AM, Diego Guella wrote: > Hi Hal, > >> From: "Hal Rosenstock" >> @@ -778,11 +776,7 @@ static int init_lash_structures(lash_t * p_lash) >>  } >> >>  /* initialise num_mst_in_lane[num_switches], default 0 */ >> - p_lash->num_mst_in_lane = (int *)malloc(num_switches * sizeof(int)); >> - if (p_lash->num_mst_in_lane == NULL) >> - goto Exit_Mem_Error; >> - memset(p_lash->num_mst_in_lane, 0, >> -        num_switches * sizeof(p_lash->num_mst_in_lane[0])); >> + memset(p_lash->num_mst_in_lane, 0, IB_MAX_NUM_VLS); > > Should it be > memset(p_lash->num_mst_in_lane, 0, IB_MAX_NUM_VLS * > sizeof(p_lash->num_mst_in_lane[0])); > ? Good catch. v2 of this patch shortly. -- Hal > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Thu Jun 18 04:31:25 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 18 Jun 2009 07:31:25 -0400 Subject: [ofa-general] [PATCHv2] opensm/osm_ucast_lash.(h c): Replace memory allocation by array Message-ID: <20090618113125.GA21708@comcast.net> lash's num_mst_in_lane array is indexed by lane nuber, not by switch index. Since it's a small number, get rid of extra memory allocation. Signed-off-by: Robert Pearson Signed-off-by: Hal Rosenstock --- Changes since v1: Fixed memset in init_lash_structures pointed out by Diego Guella diff --git a/opensm/include/opensm/osm_ucast_lash.h b/opensm/include/opensm/osm_ucast_lash.h index f48ea33..98034cc 100644 --- a/opensm/include/opensm/osm_ucast_lash.h +++ b/opensm/include/opensm/osm_ucast_lash.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008 System Fabric Works, Inc. + * Copyright (c) 2008,2009 System Fabric Works, Inc. All rights reserved. * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. @@ -90,7 +90,7 @@ typedef struct _lash { int balance_limit; switch_t **switches; cdg_vertex_t ****cdg_vertex_matrix; - int *num_mst_in_lane; + int num_mst_in_lane[IB_MAX_NUM_VLS]; int ***virtual_location; } lash_t; diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c index a987eb3..5913434 100644 --- a/opensm/opensm/osm_ucast_lash.c +++ b/opensm/opensm/osm_ucast_lash.c @@ -4,6 +4,7 @@ * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2007 Simula Research Laboratory. All rights reserved. * Copyright (c) 2007 Silicon Graphics Inc. All rights reserved. + * Copyright (c) 2008,2009 System Fabric Works, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -707,9 +708,6 @@ static void free_lash_structures(lash_t * p_lash) if (p_lash->virtual_location) free(p_lash->virtual_location); - if (p_lash->num_mst_in_lane) - free(p_lash->num_mst_in_lane); - OSM_LOG_EXIT(p_log); } @@ -778,11 +776,8 @@ static int init_lash_structures(lash_t * p_lash) } /* initialise num_mst_in_lane[num_switches], default 0 */ - p_lash->num_mst_in_lane = (int *)malloc(num_switches * sizeof(int)); - if (p_lash->num_mst_in_lane == NULL) - goto Exit_Mem_Error; memset(p_lash->num_mst_in_lane, 0, - num_switches * sizeof(p_lash->num_mst_in_lane[0])); + IB_MAX_NUM_VLS * sizeof(p_lash->num_mst_in_lane[0])); goto Exit; From yevgenyp at mellanox.co.il Thu Jun 18 05:36:50 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Thu, 18 Jun 2009 15:36:50 +0300 Subject: [ofa-general][PATCH 1/2] mlx4: Support ConnectX new device ID: 0x6764 Message-ID: <4A3A34E2.8040500@mellanox.co.il> Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/main.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 018348c..57326f9 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -1285,6 +1285,7 @@ static struct pci_device_id mlx4_pci_table[] = { { PCI_VDEVICE(MELLANOX, 0x6750) }, /* MT25408 "Hermon" EN 10GigE PCIe gen2 */ { PCI_VDEVICE(MELLANOX, 0x6372) }, /* MT25458 ConnectX EN 10GBASE-T 10GigE */ { PCI_VDEVICE(MELLANOX, 0x675a) }, /* MT25458 ConnectX EN 10GBASE-T+Gen2 10GigE */ + { PCI_VDEVICE(MELLANOX, 0x6764) }, /* MT26468 ConnectX EN 10GigE, PCIe, 2.0 5Gt/s */ { 0, } }; -- 1.6.0 From yevgenyp at mellanox.co.il Thu Jun 18 05:37:09 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Thu, 18 Jun 2009 15:37:09 +0300 Subject: [ofa-general][PATCH 2/2] mlx4: ConnectX multi functional device support Message-ID: <4A3A34F5.9030800@mellanox.co.il> MT26468 (0x6764) device can open multiple physical functions. The current driver can only work with one (primary) pf. For all other functions, QUERY_FW command would fail with CMD_STAT_MULTI_FUNC_REQ error code. We should not work on those devices, but they should remain in the driver's ownership. Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/cmd.c | 5 ++++- drivers/net/mlx4/main.c | 20 ++++++++++++++++---- include/linux/mlx4/device.h | 1 + 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/drivers/net/mlx4/cmd.c b/drivers/net/mlx4/cmd.c index 2845a05..92d649c 100644 --- a/drivers/net/mlx4/cmd.c +++ b/drivers/net/mlx4/cmd.c @@ -80,7 +80,9 @@ enum { /* Bad management packet (silently discarded): */ CMD_STAT_BAD_PKT = 0x30, /* More outstanding CQEs in CQ than new CQ size: */ - CMD_STAT_BAD_SIZE = 0x40 + CMD_STAT_BAD_SIZE = 0x40, + /* Multi Function device support required: */ + CMD_STAT_MULTI_FUNC_REQ = 0x50 }; enum { @@ -128,6 +130,7 @@ static int mlx4_status_to_errno(u8 status) [CMD_STAT_LAM_NOT_PRE] = -EAGAIN, [CMD_STAT_BAD_PKT] = -EINVAL, [CMD_STAT_BAD_SIZE] = -ENOMEM, + [CMD_STAT_MULTI_FUNC_REQ] = -EACCES, }; if (status >= ARRAY_SIZE(trans_table) || diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 57326f9..541243e 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -729,7 +729,10 @@ static int mlx4_init_hca(struct mlx4_dev *dev) err = mlx4_QUERY_FW(dev); if (err) { - mlx4_err(dev, "QUERY_FW command failed, aborting.\n"); + if (err == -EACCES) + mlx4_dbg(dev, "Function disabled.\n"); + else + mlx4_err(dev, "QUERY_FW command failed, aborting.\n"); return err; } @@ -1137,8 +1140,14 @@ static int __mlx4_init_one(struct pci_dev *pdev, const struct pci_device_id *id) } err = mlx4_init_hca(dev); - if (err) - goto err_cmd; + if (err) { + if (err == -EACCES) { + dev->flags |= MLX4_FLAG_NOT_PRIME; + pci_set_drvdata(pdev, dev); + return 0; + } else + goto err_cmd; + } err = mlx4_alloc_eq_table(dev); if (err) @@ -1234,6 +1243,8 @@ static void mlx4_remove_one(struct pci_dev *pdev) int p; if (dev) { + if (dev->flags & MLX4_FLAG_NOT_PRIME) + goto cmd_cleanup; mlx4_stop_sense(dev); mlx4_unregister_device(dev); @@ -1256,11 +1267,12 @@ static void mlx4_remove_one(struct pci_dev *pdev) mlx4_cleanup_uar_table(dev); mlx4_free_eq_table(dev); mlx4_close_hca(dev); - mlx4_cmd_cleanup(dev); if (dev->flags & MLX4_FLAG_MSI_X) pci_disable_msix(pdev); +cmd_cleanup: + mlx4_cmd_cleanup(dev); kfree(priv); pci_release_region(pdev, 2); pci_release_region(pdev, 0); diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index ce7cc6c..637e72c 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -42,6 +42,7 @@ enum { MLX4_FLAG_MSI_X = 1 << 0, MLX4_FLAG_OLD_PORT_CMDS = 1 << 1, + MLX4_FLAG_NOT_PRIME = 1 << 2, }; enum { -- 1.6.0 From sashak at voltaire.com Thu Jun 18 05:27:39 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 15:27:39 +0300 Subject: [ofa-general] Re: [PATCH v2] opensm: MFT tables are not set after non full member re-join In-Reply-To: <4A35E52A.3020502@gmail.com> References: <4A1019F6.5060900@gmail.com> <4A127444.8080707@gmail.com> <4A35E52A.3020502@gmail.com> Message-ID: <20090618122739.GZ6163@me> Hi Eli, On 09:07 Mon 15 Jun , Eli Dorfman (Voltaire) wrote: > MFT tables are not set after non full member re-join > > In case of non full member re-join MFT tables are not set. > No need to set or check non full member reference to mlid (port->mcm_list). > This list should be used only for full members for cleanup when port goes down. > > A simple scenarion to reproduce this: > 1. Full member creates group > 2. Non-member join - MFT sent > 3. Full member leave > a. group is deleted but non member port has still reference to the MLID > 4. Full member re-creates the group > 5. Non member re-joins - MFT not sent to switches I agree about the issue. However have couple of comment about the patch. > > Signed-off-by: Eli Dorfman > --- > opensm/include/opensm/osm_sm.h | 3 ++- > opensm/opensm/osm_sa_mcmember_record.c | 6 +++--- > opensm/opensm/osm_sm.c | 22 +++++++++++++++++++++- > 3 files changed, 26 insertions(+), 5 deletions(-) > > diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h > index cc8321d..1a8a577 100644 > --- a/opensm/include/opensm/osm_sm.h > +++ b/opensm/include/opensm/osm_sm.h > @@ -539,7 +539,8 @@ osm_resp_send(IN osm_sm_t * sm, > ib_api_status_t > osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > IN const ib_net16_t mlid, > - IN const ib_net64_t port_guid); > + IN const ib_net64_t port_guid, > + IN uint8_t scope_state); Here and in other places in the patch please try to keep indentation (use opensm/osm_indent script when unsure). > /* > * PARAMETERS > * p_sm > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > index 5543221..fe29dd6 100644 > --- a/opensm/opensm/osm_sa_mcmember_record.c > +++ b/opensm/opensm/osm_sa_mcmember_record.c > @@ -1039,7 +1039,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > if (!p_mgrp) { > char gid_str[INET6_ADDRSTRLEN]; > CL_PLOCK_RELEASE(sa->p_lock); > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > + OSM_LOG(sa->p_log, OSM_LOG_INFO, How is this related to the patch? In general try to not use OSM_LOG_INFO for debug logging. > "Failed since multicast group %s not present\n", > inet_ntop(AF_INET6, p_recvd_mcmember_rec->mgid.raw, > gid_str, sizeof gid_str)); > @@ -1309,8 +1309,8 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > > /* do the actual routing (actually schedule the update) */ > status = osm_sm_mcgrp_join(sa->sm, mlid, > - p_recvd_mcmember_rec->port_gid.unicast. > - interface_id); > + p_recvd_mcmember_rec->port_gid.unicast.interface_id, > + p_recvd_mcmember_rec->scope_state); > > if (status != IB_SUCCESS) { > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " > diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c > index daa60ff..b334d39 100644 > --- a/opensm/opensm/osm_sm.c > +++ b/opensm/opensm/osm_sm.c > @@ -468,7 +468,7 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, > /********************************************************************** > **********************************************************************/ > ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > - IN const ib_net64_t port_guid) > + IN const ib_net64_t port_guid, IN uint8_t scope_state) > { > osm_mgrp_t *p_mgrp; > osm_port_t *p_port; > @@ -515,6 +515,26 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > goto Exit; > } > > + /* if there was no change from the last time > + * we processed the group we can skip doing anything > + */ > + if (p_mgrp->last_change_id == p_mgrp->last_tree_id) { As I said you don't need to perform this check here - 'last_change_id' *IS* verified against 'last_tree_id' in osm_mcast_mgr_process_mgroups() where multicast re-routing requests are processing and repeated requests will not be handled. And even if your goal here is optimization (so even re-routing requests will not scheduled) you likely will need to care about leave too, no? And then such check should go to common sm_mgrp_process() function (and since this does nothing with the issue described in the patch introduction it would be nice to have this as separate patch too). > + CL_PLOCK_RELEASE(p_sm->p_lock); > + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, Why VERBOSE? It looks like DEBUG prints (do we need them at all here?). > + "Skip processing mgrp with lid:0x%X last change id:%u\n", > + cl_ntoh16(mlid), p_mgrp->last_change_id); > + goto Exit; > + } else { > + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, > + "processing mgrp with lid:0x%X port: 0x%016" PRIx64 " last change id:%u tree id:%u\n", > + cl_ntoh16(mlid), cl_ntoh64(port_guid), > + p_mgrp->last_change_id, p_mgrp->last_tree_id); > + } > + > + /* add mgrp only to FULL member port. used for cleanup when port goes down */ > + if (!(scope_state & IB_JOIN_STATE_FULL)) > + goto MgrpProcess; > + Ok, so mlid will not be added to non member port's mcm_lid - that's fine IMO. But what will happen when full member downgrades the state to non-member? mlid will still be present in its mcm_list (and we are returning to the original bug), no? Sasha > /* > * Check if the object (according to mlid) already exists on this port. > * If it does - then no need to update it again, and no need to > @@ -543,6 +563,7 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > goto Exit; > } > > +MgrpProcess: > status = sm_mgrp_process(p_sm, p_mgrp); > CL_PLOCK_RELEASE(p_sm->p_lock); > > -- > 1.5.3.6 > From sashak at voltaire.com Thu Jun 18 05:31:03 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 15:31:03 +0300 Subject: [ofa-general] Re: [PATCHv2] opensm/osm_ucast_lash.(h c): Replace memory allocation by array In-Reply-To: <20090618113125.GA21708@comcast.net> References: <20090618113125.GA21708@comcast.net> Message-ID: <20090618123103.GA6163@me> On 07:31 Thu 18 Jun , Hal Rosenstock wrote: > > lash's num_mst_in_lane array is indexed by lane nuber, not by switch index. > Since it's a small number, get rid of extra memory allocation. > > Signed-off-by: Robert Pearson > Signed-off-by: Hal Rosenstock Applied. Thanks. Tiny comment is below. > @@ -778,11 +776,8 @@ static int init_lash_structures(lash_t * p_lash) > } > > /* initialise num_mst_in_lane[num_switches], default 0 */ > - p_lash->num_mst_in_lane = (int *)malloc(num_switches * sizeof(int)); > - if (p_lash->num_mst_in_lane == NULL) > - goto Exit_Mem_Error; > memset(p_lash->num_mst_in_lane, 0, > - num_switches * sizeof(p_lash->num_mst_in_lane[0])); > + IB_MAX_NUM_VLS * sizeof(p_lash->num_mst_in_lane[0])); sizeof(p_lash->num_mst_in_line) would be even simpler, but it is minor. Sasha From sashak at voltaire.com Thu Jun 18 05:56:46 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 15:56:46 +0300 Subject: [ofa-general] Re: [PATCH] Add LMC support to DOR routing In-Reply-To: <20090315144207.GA18027@sgi.com> References: <20090315144207.GA18027@sgi.com> Message-ID: <20090618125646.GC6163@me> On 09:42 Sun 15 Mar , Dale Purdy wrote: > > DOR routing equilizes traffic across redundant links representing > dimensions. This change makes DOR compatible with LMC and tries to > equilize traffic across redundant links between LMC aliased LIDs. > > Signed-off-by: Dale Purdy Rebased and applied. Thanks. Sasha From slavas at Voltaire.COM Thu Jun 18 05:52:10 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Thu, 18 Jun 2009 15:52:10 +0300 Subject: [ofa-general] [PATCH] opensm: multicast multiplexing many mgid to mlid Message-ID: <4A3A387A.2020509@Voltaire.COM> >From cc95291fc8c9a5e2f9bb1367ffbf5930509a30c1 Mon Sep 17 00:00:00 2001 From: Slava Strebkov Date: Thu, 18 Jun 2009 15:02:34 +0300 Subject: [PATCH] multicast multiplexing many mgid to mlid This patch implements multicast multiplexing as proposed in the thread entitled "IPv6 and IPoIB scalability issue". It includes both IPv6 SNM groups collapsing and IPv4 special groups (All hosts, routers, gateways) and muxing of many IPoIB MGIDs to single MLID according to mask of MGID least significant bits. Addition patch will allow mask configuration from opensm configuration file. Signed-off-by: Slava Strebkov --- opensm/include/opensm/osm_mgrp_holder.h | 428 +++++++++++++++++++++++++++++++ opensm/include/opensm/osm_multicast.h | 15 +- opensm/include/opensm/osm_sm.h | 8 +- opensm/include/opensm/osm_subnet.h | 6 +- opensm/opensm/Makefile.am | 1 + opensm/opensm/osm_drop_mgr.c | 11 +- opensm/opensm/osm_mcast_mgr.c | 211 +++++++++------- opensm/opensm/osm_mgrp_holder.c | 315 +++++++++++++++++++++++ opensm/opensm/osm_multicast.c | 57 +++-- opensm/opensm/osm_qos_policy.c | 42 ++-- opensm/opensm/osm_sa.c | 38 ++- opensm/opensm/osm_sa_mcmember_record.c | 144 ++++++----- opensm/opensm/osm_sa_path_record.c | 18 +- opensm/opensm/osm_sm.c | 73 +++++- opensm/opensm/osm_subnet.c | 11 +- 15 files changed, 1114 insertions(+), 264 deletions(-) create mode 100644 opensm/include/opensm/osm_mgrp_holder.h create mode 100644 opensm/opensm/osm_mgrp_holder.c diff --git a/opensm/include/opensm/osm_mgrp_holder.h b/opensm/include/opensm/osm_mgrp_holder.h new file mode 100644 index 0000000..05b1c14 --- /dev/null +++ b/opensm/include/opensm/osm_mgrp_holder.h @@ -0,0 +1,428 @@ +/* + * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. + * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ +#ifndef _OSM_MGRP_HOLDER_H_ +#define _OSM_MGRP_HOLDER_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef __cplusplus +# define BEGIN_C_DECLS extern "C" { +# define END_C_DECLS } +#else /* !__cplusplus */ +# define BEGIN_C_DECLS +# define END_C_DECLS +#endif /* __cplusplus */ + +BEGIN_C_DECLS + + + +/****s* OpenSM: Multicast Group/osm_mgrp_holder_t +* NAME +* osm_mgrp_holder_t +* +* DESCRIPTION +* Holder for mgroup. +* +* The osm_mgrp_t object should be treated as opaque and should +* be manipulated only through the provided functions. +* +* SYNOPSIS +*/ + +typedef struct osm_mgrp_holder { + cl_fmap_t mgrp_map; + cl_qmap_t mgrp_port_map; + ib_gid_t common_mgid; + osm_mtree_node_t *p_root; + boolean_t to_be_deleted; + uint32_t last_tree_id; + uint32_t last_change_id; + ib_net16_t mlid; +} osm_mgrp_holder_t; + +/* +* FIELDS +* mgrp_map +* Map for mgroups. Must be first element!! +* +* mgrp_port_map +* Map of all ports joined same mlid +* common_mgid +* mgid of mgroup, ANDed with bitmask. +* mgid of each mgroup in mgrp_map, ANDed with bitmask, +* see osm_mgrp_holder_prepare_common_mgid +* +* p_root +* Pointer to the root "tree node" in the single spanning tree +* for this multicast group holder. The nodes of the tree represent +* switches. Member ports are not represented in the tree. +* +* to_be_deleted +* Since holders are deleted when there are no mgroups in. +* +* last_change_id +* a counter for the number of changes applied to the group in this holder. +* This counter shuold be incremented on any modification +* to the group: joining or leaving of ports. +* +* last_tree_id +* the last change id used for building the current tree. +* +*/ + /****s* OpenSM: Multicast Group/osm_mgrp_port _t +* NAME +* osm_mgrp_port _t +* +* DESCRIPTION +* Holder for mgroups and port guid. +* +* +* SYNOPSIS +*/ +typedef struct _osm_mgrp_port { + cl_map_item_t guid_item; + cl_qlist_t mgroups; + ib_net64_t port_guid; +} osm_mgrp_port_t; +/* +* FIELDS +* guid_item +* Map for ports. Must be first element +* mgroups +* Map for mgroups opened by this port. +* +* portguid +* +*/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_new +* NAME +* osm_mgrp_holder_new +* +* DESCRIPTION +* Allocates and initializes a Multicast Group Holder for use. +* +* SYNOPSIS +*/ +osm_mgrp_holder_t *osm_mgrp_holder_new(IN osm_subn_t *p_subn, IN ib_gid_t * p_mgid, IN ib_net16_t mlid); +/* +* PARAMETERS +* p_subn +* (in) pointer to osm_subnet +* p_mgid +* (in) pointer to mgid +* mlid +* [in] Multicast LID for this multicast group holder. +* +* RETURN VALUES +* pointer to initialized osm_mgrp_holder_t +* or NULL, if unsuccessful +* +* SEE ALSO +* Multicast Group Holder, osm_mgrp_holder_delete +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_delete +* NAME +* osm_mgrp_holder_delete +* +* DESCRIPTION +* Removes entry from array of holders +* Removes port from mgroup port list +* +* SYNOPSIS +*/ +void osm_mgrp_holder_delete(IN osm_subn_t * p_subn, IN ib_net16_t mlid); +/* +* PARAMETERS +* +* p_subn +* [in] Pointer to osm_subnet +* +* mlid +* [in] holder's mlid +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* +*********/ + + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_add_mgrp_port +* NAME +* osm_mgrp_holder_add_mgrp_port +* +* DESCRIPTION +* Allocates osm_mgrp_port_t for new port joined to mgroup with mlid of this holder, +* (or / and) adds mgroup to mgroup map of existed osm_mgrp_port_t object. +* +* SYNOPSIS +*/ +ib_api_status_t osm_mgrp_holder_add_mgrp_port(IN osm_mgrp_holder_t *p_mgrp_holder, IN osm_mgrp_t *p_mgrp, + IN ib_net64_t port_guid); +/* +* PARAMETERS +* p_mgrp_holder +* (in) pointer to osm_mgrp_holder_t +* p_mgrp +* (in) pointer to osm_mgrp_t +* +* RETURN VALUES +* IB_SUCCESS or +* IB_INSUFFICIENT_MEMORY +* +* SEE ALSO +* Multicast Group Holder, osm_mgrp_holder_delete_mgrp_port +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_delete_mgrp_port +* NAME +* osm_mgrp_holder_delete_mgrp_port +* +* DESCRIPTION +* Deletes osm_mgrp_port_t for specified port +* +* SYNOPSIS +*/ +void osm_mgrp_holder_delete_mgrp_port(IN osm_mgrp_holder_t *p_mgrp_holder, IN osm_mgrp_t *p_mgrp, + IN ib_net64_t port_guid); +/* +* PARAMETERS +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* p_mgrp +* (in) Pointer to osm_mgrp_t object +* +* port_guid +* [in] Port guid of the departing port. +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO + Multicast Group Holder,osm_holder_add_mgrp_port +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_remove_port +* NAME +* osm_mgrp_holder_remove_port +* +* DESCRIPTION +* Removes osm_mgrp_port_t from mgrp_port_map of holder +* Removes port from mgroup port list +* +* SYNOPSIS +*/ +void osm_mgrp_holder_remove_port(IN osm_subn_t * const p_subn, + IN osm_log_t * const p_log, + IN osm_mgrp_holder_t * const p_mgrp_holder, + IN const ib_net64_t port_guid); +/* +* PARAMETERS +* +* p_subn +* [in] Pointer to the subnet object +* +* p_log +* [in] The log object pointer +* +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* port_guid +* [in] Port guid of the departing port. +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_add_mgrp +* NAME +* osm_mgrp_holder_add_mgrp +* +* DESCRIPTION +* Adds mgroup to holder according to its mgid +* +* +* SYNOPSIS +*/ +void osm_mgrp_holder_add_mgrp(IN osm_mgrp_holder_t * p_mgrp_holder, + IN osm_mgrp_t *p_mgrp, + IN osm_log_t * const p_log); +/* +* PARAMETERS +* +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* p_mgrp +* [in] mgroup to add. +* +* RETURN VALUES +* None. +* +* NOTES +* Updates common_mgid when holder is being reused +* SEE ALSO +* Multicast Group Holder,osm_mgrp_holder_delete_mgrp +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_delete_mgrp +* NAME +* osm_mgrp_holder_delete_mgrp +* +* DESCRIPTION +* Deletes mgroup from holder according to its mgid +* +* +* SYNOPSIS +*/ +void osm_mgrp_holder_delete_mgrp(IN osm_mgrp_holder_t *p_mgrp_holder, + IN osm_mgrp_t *p_mgrp); +/* +* PARAMETERS +* +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* p_mgrp +* [in] mgroup to delete. +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* Multicast Group Holder,osm_mgrp_holder_add_mgrp +*********/ + + + + +/* Multicast Group Holder, osm_mgrp_holder_delete_port +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_prepare_common_mgid +* NAME +* osm_mgrp_holder_prepare_common_mgid +* +* DESCRIPTION +* Prepares mgid, which is common for all mgroups in this holder +* +* SYNOPSIS +*/ +void osm_mgrp_holder_prepare_common_mgid(IN const ib_gid_t *const p_mgid, OUT ib_gid_t *p_common_mgid); +/* +* PARAMETERS +* +* p_mgid +* [in] Pointer to mgid +* +* p_common_mgid +* [out] common mgid +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_get_mlid_by_mgid +* NAME +* osm_mgrp_holder_get_mlid_by_mgid +* +* DESCRIPTION +* Searches holder which contains mgroup with given mgid +* Returns mlid of the found holder +* +* SYNOPSIS +*/ +ib_net16_t osm_mgrp_holder_get_mlid_by_mgid(IN osm_sa_t * sa, IN const ib_gid_t *const p_mgid); +/* +* PARAMETERS +* +* p_sa +* [in] Pointer to sa object +* +* p_mgid +* [in] pointer to mgid +* +* RETURN VALUES +* mlid of found holder, or zero. +* +* NOTES +* +* SEE ALSO +* +*********/ + +static inline ib_net16_t osm_mgrp_holder_get_mlid(IN osm_mgrp_holder_t * const p_mgrp_holder) +{ + return (p_mgrp_holder->mlid); +} + +static inline boolean_t osm_mgrp_holder_is_empty(IN const osm_mgrp_holder_t * const p_mgrp_holder) +{ + return (cl_qmap_count(&p_mgrp_holder->mgrp_port_map) == 0); +} +#endif + diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index a871306..b59f06e 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -44,6 +44,7 @@ #define _OSM_MULTICAST_H_ #include +#include #include #include #include @@ -107,6 +108,7 @@ typedef struct osm_mcast_mgr_ctxt { * SEE ALSO *********/ + /****s* OpenSM: Multicast Group/osm_mgrp_t * NAME * osm_mgrp_t @@ -117,12 +119,13 @@ typedef struct osm_mcast_mgr_ctxt { * The osm_mgrp_t object should be treated as opaque and should * be manipulated only through the provided functions. * -* SYNOPSIS +* SYNOPSIS */ typedef struct osm_mgrp { + cl_fmap_item_t mgid_item; + cl_list_item_t mgrp_item; cl_map_item_t map_item; ib_net16_t mlid; - osm_mtree_node_t *p_root; cl_qmap_t mcm_port_tbl; ib_member_rec_t mcmember_rec; boolean_t well_known; @@ -140,11 +143,6 @@ typedef struct osm_mgrp { * The network ordered LID of this Multicast Group (must be * >= 0xC000). * -* p_root -* Pointer to the root "tree node" in the single spanning tree -* for this multicast group. The nodes of the tree represent -* switches. Member ports are not represented in the tree. -* * mcm_port_tbl * Table (sorted by port GUID) of osm_mcm_port_t objects * representing the member ports of this multicast group. @@ -239,7 +237,8 @@ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid); * * SYNOPSIS */ -void osm_mgrp_delete(IN osm_mgrp_t * const p_mgrp); +void osm_mgrp_delete(IN osm_subn_t * p_subn, + IN osm_mgrp_t *const p_mgrp); /* * PARAMETERS * p_mgrp diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h index cc8321d..e828505 100644 --- a/opensm/include/opensm/osm_sm.h +++ b/opensm/include/opensm/osm_sm.h @@ -61,6 +61,7 @@ #include #include #include +#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -539,7 +540,8 @@ osm_resp_send(IN osm_sm_t * sm, ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid); + IN const ib_net64_t port_guid, + IN const ib_gid_t *p_mgid); /* * PARAMETERS * p_sm @@ -551,6 +553,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, * port_guid * [in] Port GUID to add to the group. * +* p_mgid +* [in] MGID to add to the group holder. * RETURN VALUES * None * @@ -572,7 +576,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, */ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, - IN const ib_net16_t mlid, IN const ib_net64_t port_guid); + IN osm_mgrp_t *p_mgrp, IN ib_net64_t port_guid); /* * PARAMETERS * p_sm diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 59a32ad..329c5a8 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -508,7 +508,7 @@ typedef struct osm_subn { boolean_t first_time_master_sweep; boolean_t coming_out_of_standby; unsigned need_update; - void *mgroups[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; + void *mgroup_holders[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; } osm_subn_t; /* * FIELDS @@ -942,9 +942,9 @@ struct osm_port *osm_get_port_by_guid(IN osm_subn_t const *p_subn, * SYNOPSIS */ static inline -struct osm_mgrp *osm_get_mgrp_by_mlid(osm_subn_t const *p_subn, ib_net16_t mlid) +struct osm_mgrp_holder *osm_get_mgrp_holder_by_mlid(osm_subn_t const *p_subn, ib_net16_t mlid) { - return p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + return p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; } /* * PARAMETERS diff --git a/opensm/opensm/Makefile.am b/opensm/opensm/Makefile.am index 2d67a95..9a4d2a6 100644 --- a/opensm/opensm/Makefile.am +++ b/opensm/opensm/Makefile.am @@ -52,6 +52,7 @@ opensm_SOURCES = main.c osm_console_io.c osm_console.c osm_db_files.c \ osm_subnet.c osm_sw_info_rcv.c osm_switch.c \ osm_prtn.c osm_prtn_config.c osm_qos.c osm_router.c \ osm_trap_rcv.c osm_ucast_mgr.c osm_ucast_updn.c \ + osm_mgrp_holder.c \ osm_ucast_lash.c osm_ucast_file.c osm_ucast_ftree.c \ osm_vl15intf.c osm_vl_arb_rcv.c \ st.c osm_perfmgr.c osm_perfmgr_db.c \ diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index c9a4f33..8becc2c 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -62,6 +62,7 @@ #include #include #include +#include /********************************************************************** **********************************************************************/ @@ -158,7 +159,6 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) osm_port_t *p_port_check; cl_qmap_t *p_sm_guid_tbl; osm_mcm_info_t *p_mcm; - osm_mgrp_t *p_mgrp; cl_ptr_vector_t *p_port_lid_tbl; uint16_t min_lid_ho; uint16_t max_lid_ho; @@ -168,6 +168,7 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) ib_gid_t port_gid; ib_mad_notice_attr_t notice; ib_api_status_t status; + osm_mgrp_holder_t *p_osm_mgrp_holder; OSM_LOG_ENTER(sm->p_log); @@ -212,10 +213,10 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { - p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, p_mcm->mlid); - if (p_mgrp) { - osm_mgrp_delete_port(sm->p_subn, sm->p_log, - p_mgrp, p_port->guid); + p_osm_mgrp_holder = osm_get_mgrp_holder_by_mlid(sm->p_subn, p_mcm->mlid); + if (p_osm_mgrp_holder) { + osm_mgrp_holder_remove_port(sm->p_subn, sm->p_log, + p_osm_mgrp_holder, p_port->guid); osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); } p_mcm = diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index de0a8a5..9a4adae 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -55,6 +55,8 @@ #include #include #include +#include +#include /********************************************************************** **********************************************************************/ @@ -111,14 +113,14 @@ static void mcast_mgr_purge_tree_node(IN osm_mtree_node_t * p_mtn) /********************************************************************** **********************************************************************/ -static void mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_t * p_mgrp) +static void mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_holder_t * p_mgrp_holder) { OSM_LOG_ENTER(sm->p_log); - if (p_mgrp->p_root) - mcast_mgr_purge_tree_node(p_mgrp->p_root); + if (p_mgrp_holder->p_root) + mcast_mgr_purge_tree_node(p_mgrp_holder->p_root); - p_mgrp->p_root = NULL; + p_mgrp_holder->p_root = NULL; OSM_LOG_EXIT(sm->p_log); } @@ -126,41 +128,37 @@ static void mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_t * p_mgrp) /********************************************************************** **********************************************************************/ static float osm_mcast_mgr_compute_avg_hops(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp, + const osm_mgrp_holder_t * p_mgrp_holder, const osm_switch_t * p_sw) { float avg_hops = 0; uint32_t hops = 0; uint32_t num_ports = 0; const osm_port_t *p_port; - const osm_mcm_port_t *p_mcm_port; - const cl_qmap_t *p_mcm_tbl; + const osm_mgrp_port_t *p_holder_port; OSM_LOG_ENTER(sm->p_log); - p_mcm_tbl = &p_mgrp->mcm_port_tbl; /* For each member of the multicast group, compute the number of hops to its base LID. */ - for (p_mcm_port = (osm_mcm_port_t *) cl_qmap_head(p_mcm_tbl); - p_mcm_port != (osm_mcm_port_t *) cl_qmap_end(p_mcm_tbl); - p_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item)) { + for (p_holder_port = (osm_mgrp_port_t *) cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + p_holder_port != (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map); + p_holder_port = + (osm_mgrp_port_t *) cl_qmap_next(&p_holder_port->guid_item)) { /* Acquire the port object for this port guid, then create the new worker object to build the list. */ p_port = osm_get_port_by_guid(sm->p_subn, - ib_gid_get_guid(&p_mcm_port-> - port_gid)); + p_holder_port->port_guid); if (!p_port) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A18: " "No port object for port 0x%016" PRIx64 "\n", - cl_ntoh64(ib_gid_get_guid - (&p_mcm_port->port_gid))); + cl_ntoh64(p_holder_port->port_guid)); continue; } @@ -185,40 +183,35 @@ static float osm_mcast_mgr_compute_avg_hops(osm_sm_t * sm, of the group HCAs **********************************************************************/ static float osm_mcast_mgr_compute_max_hops(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp, + const osm_mgrp_holder_t * p_mgrp_holder, const osm_switch_t * p_sw) { uint32_t max_hops = 0; uint32_t hops = 0; const osm_port_t *p_port; - const osm_mcm_port_t *p_mcm_port; - const cl_qmap_t *p_mcm_tbl; + const osm_mgrp_port_t *p_mgrp_holder_port; OSM_LOG_ENTER(sm->p_log); - p_mcm_tbl = &p_mgrp->mcm_port_tbl; /* For each member of the multicast group, compute the number of hops to its base LID. */ - for (p_mcm_port = (osm_mcm_port_t *) cl_qmap_head(p_mcm_tbl); - p_mcm_port != (osm_mcm_port_t *) cl_qmap_end(p_mcm_tbl); - p_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item)) { + for (p_mgrp_holder_port = (osm_mgrp_port_t *) cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + p_mgrp_holder_port != (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map); + p_mgrp_holder_port = + (osm_mgrp_port_t *) cl_qmap_next(&p_mgrp_holder_port->guid_item)) { /* Acquire the port object for this port guid, then create the new worker object to build the list. */ p_port = osm_get_port_by_guid(sm->p_subn, - ib_gid_get_guid(&p_mcm_port-> - port_gid)); - + p_mgrp_holder_port->port_guid); if (!p_port) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A1A: " "No port object for port 0x%016" PRIx64 "\n", - cl_ntoh64(ib_gid_get_guid - (&p_mcm_port->port_gid))); + cl_ntoh64(p_mgrp_holder_port->port_guid)); continue; } @@ -244,15 +237,15 @@ static float osm_mcast_mgr_compute_max_hops(osm_sm_t * sm, of the multicast group. **********************************************************************/ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp) + const osm_mgrp_holder_t *p_mgrp_holder) { cl_qmap_t *p_sw_tbl; - const osm_switch_t *p_sw; - const osm_switch_t *p_best_sw = NULL; + osm_switch_t *p_sw; + osm_switch_t *p_best_sw = NULL; float hops = 0; float best_hops = 10000; /* any big # will do */ #ifdef OSM_VENDOR_INTF_ANAFA - boolean_t use_avg_hops = TRUE; /* anafa2 - bug hca on switch *//* use max hops for root */ + boolean_t use_avg_hops = TRUE; /* anafa2 - bug hca on switch - use max hops for root */ #else boolean_t use_avg_hops = FALSE; /* use max hops for root */ #endif @@ -260,8 +253,7 @@ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, OSM_LOG_ENTER(sm->p_log); p_sw_tbl = &sm->p_subn->sw_guid_tbl; - - CL_ASSERT(!osm_mgrp_is_empty(p_mgrp)); + CL_ASSERT(!osm_mgrp_holder_is_empty(p_mgrp_holder)); for (p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl); @@ -270,13 +262,13 @@ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, continue; if (use_avg_hops) - hops = osm_mcast_mgr_compute_avg_hops(sm, p_mgrp, p_sw); + hops = osm_mcast_mgr_compute_avg_hops(sm, p_mgrp_holder, p_sw); else - hops = osm_mcast_mgr_compute_max_hops(sm, p_mgrp, p_sw); + hops = osm_mcast_mgr_compute_max_hops(sm, p_mgrp_holder, p_sw); OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Switch 0x%016" PRIx64 ", hops = %f\n", - cl_ntoh64(osm_node_get_node_guid(p_sw->p_node)), hops); + "Switch 0x%016" PRIx64 ", hops = %f\n", + cl_ntoh64(osm_node_get_node_guid(p_sw->p_node)), hops); if (hops < best_hops) { p_best_sw = p_sw; @@ -301,7 +293,7 @@ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, This function returns the existing or optimal root swtich for the tree. **********************************************************************/ static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp) + const osm_mgrp_holder_t * p_mgrp_holder) { const osm_switch_t *p_sw = NULL; @@ -313,7 +305,7 @@ static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, the root will be always on the first switch attached to it. - Very bad ... */ - p_sw = mcast_mgr_find_optimal_switch(sm, p_mgrp); + p_sw = mcast_mgr_find_optimal_switch(sm, p_mgrp_holder); OSM_LOG_EXIT(sm->p_log); return (osm_switch_t *) p_sw; @@ -394,7 +386,7 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) spanning tree that eminate from this switch. On input, the p_list contains the group members that must be routed from this switch. **********************************************************************/ -static void mcast_mgr_subdivide(osm_sm_t * sm, osm_mgrp_t * p_mgrp, +static void mcast_mgr_subdivide(osm_sm_t * sm, osm_mgrp_holder_t * p_mgrp_holder, osm_switch_t * p_sw, cl_qlist_t * p_list, cl_qlist_t * list_array, uint8_t array_size) { @@ -405,7 +397,7 @@ static void mcast_mgr_subdivide(osm_sm_t * sm, osm_mgrp_t * p_mgrp, OSM_LOG_ENTER(sm->p_log); - mlid_ho = cl_ntoh16(osm_mgrp_get_mlid(p_mgrp)); + mlid_ho = cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder)); /* For Multicast Groups, we want not to count on previous @@ -495,7 +487,7 @@ static void mcast_mgr_purge_list(osm_sm_t * sm, cl_qlist_t * p_list) The function returns the newly created mtree node element. **********************************************************************/ -static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, +static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_holder_t * p_mgrp_holder, osm_switch_t * p_sw, cl_qlist_t * p_list, uint8_t depth, uint8_t upstream_port, @@ -521,7 +513,7 @@ static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, node_guid = osm_node_get_node_guid(p_sw->p_node); node_guid_ho = cl_ntoh64(node_guid); - mlid_ho = cl_ntoh16(osm_mgrp_get_mlid(p_mgrp)); + mlid_ho = cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder)); OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Routing MLID 0x%X through switch 0x%" PRIx64 @@ -598,7 +590,7 @@ static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, for (i = 0; i < max_children; i++) cl_qlist_init(&list_array[i]); - mcast_mgr_subdivide(sm, p_mgrp, p_sw, p_list, list_array, max_children); + mcast_mgr_subdivide(sm, p_mgrp_holder, p_sw, p_list, list_array, max_children); p_tbl = osm_switch_get_mcast_tbl_ptr(p_sw); @@ -681,7 +673,7 @@ static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, CL_ASSERT(p_remote_physp); p_mtn->child_array[i] = - mcast_mgr_branch(sm, p_mgrp, p_remote_node->sw, + mcast_mgr_branch(sm, p_mgrp_holder, p_remote_node->sw, p_port_list, depth, osm_physp_get_port_num (p_remote_physp), p_max_depth); @@ -717,11 +709,10 @@ Exit: /********************************************************************** **********************************************************************/ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, - osm_mgrp_t * p_mgrp) + osm_mgrp_holder_t * p_mgrp_holder) { - const cl_qmap_t *p_mcm_tbl; const osm_port_t *p_port; - const osm_mcm_port_t *p_mcm_port; + const osm_mgrp_port_t *p_mgrp_port; uint32_t num_ports; cl_qlist_t port_list; osm_switch_t *p_sw; @@ -740,14 +731,13 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, on multicast forwarding table information if the user wants to preserve existing multicast routes. */ - mcast_mgr_purge_tree(sm, p_mgrp); + mcast_mgr_purge_tree(sm, p_mgrp_holder); - p_mcm_tbl = &p_mgrp->mcm_port_tbl; - num_ports = cl_qmap_count(p_mcm_tbl); + num_ports = cl_qmap_count(&p_mgrp_holder->mgrp_port_map); if (num_ports == 0) { OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "MLID 0x%X has no members - nothing to do\n", - cl_ntoh16(osm_mgrp_get_mlid(p_mgrp))); + cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder))); goto Exit; } @@ -767,11 +757,11 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, Locate the switch around which to create the spanning tree for this multicast group. */ - p_sw = mcast_mgr_find_root_switch(sm, p_mgrp); + p_sw = mcast_mgr_find_root_switch(sm, p_mgrp_holder); if (p_sw == NULL) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A08: " "Unable to locate a suitable switch for group 0x%X\n", - cl_ntoh16(osm_mgrp_get_mlid(p_mgrp))); + cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder))); status = IB_ERROR; goto Exit; } @@ -779,22 +769,19 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, /* Build the first "subset" containing all member ports. */ - for (p_mcm_port = (osm_mcm_port_t *) cl_qmap_head(p_mcm_tbl); - p_mcm_port != (osm_mcm_port_t *) cl_qmap_end(p_mcm_tbl); - p_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item)) { + for (p_mgrp_port = (osm_mgrp_port_t *) cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + p_mgrp_port != (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map); + p_mgrp_port = + (osm_mgrp_port_t *) cl_qmap_next(&p_mgrp_port->guid_item)) { /* Acquire the port object for this port guid, then create the new worker object to build the list. */ - p_port = osm_get_port_by_guid(sm->p_subn, - ib_gid_get_guid(&p_mcm_port-> - port_gid)); + p_port = osm_get_port_by_guid(sm->p_subn, p_mgrp_port->port_guid); if (!p_port) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A09: " "No port object for port 0x%016" PRIx64 "\n", - cl_ntoh64(ib_gid_get_guid - (&p_mcm_port->port_gid))); + cl_ntoh64(p_mgrp_port->port_guid)); continue; } @@ -803,7 +790,7 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A10: " "Insufficient memory to route port 0x%016" PRIx64 "\n", - cl_ntoh64(osm_port_get_guid(p_port))); + cl_ntoh64(p_mgrp_port->port_guid)); continue; } @@ -811,12 +798,12 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, } count = cl_qlist_count(&port_list); - p_mgrp->p_root = mcast_mgr_branch(sm, p_mgrp, p_sw, &port_list, 0, 0, + p_mgrp_holder->p_root = mcast_mgr_branch(sm, p_mgrp_holder, p_sw, &port_list, 0, 0, &max_depth); OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Configured MLID 0x%X for %u ports, max tree depth = %u\n", - cl_ntoh16(osm_mgrp_get_mlid(p_mgrp)), count, max_depth); + cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder)), count, max_depth); Exit: OSM_LOG_EXIT(sm->p_log); @@ -1023,14 +1010,16 @@ Exit: lock must already be held on entry **********************************************************************/ static ib_api_status_t osm_mcast_mgr_process_tree(osm_sm_t * sm, - IN osm_mgrp_t * p_mgrp) + IN osm_mgrp_holder_t * p_mgrp_holder) { ib_api_status_t status = IB_SUCCESS; ib_net16_t mlid; + osm_mgrp_t *p_mgrp; + unsigned has_full_members = 0; OSM_LOG_ENTER(sm->p_log); - mlid = osm_mgrp_get_mlid(p_mgrp); + mlid = osm_mgrp_holder_get_mlid(p_mgrp_holder); OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Processing multicast group 0x%X\n", cl_ntoh16(mlid)); @@ -1051,10 +1040,16 @@ static ib_api_status_t osm_mcast_mgr_process_tree(osm_sm_t * sm, */ mcast_mgr_clear(sm, cl_ntoh16(mlid)); - if (!p_mgrp->full_members) + p_mgrp = (osm_mgrp_t*)cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_mgrp != (osm_mgrp_t*)cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + if (0 != (has_full_members = p_mgrp->full_members)) + break; + p_mgrp = (osm_mgrp_t*)cl_fmap_next(&p_mgrp->mgid_item); + } + if (!has_full_members) goto Exit; - status = mcast_mgr_build_spanning_tree(sm, p_mgrp); + status = mcast_mgr_build_spanning_tree(sm, p_mgrp_holder); if (status != IB_SUCCESS) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A17: " "Unable to create spanning tree (%s)\n", @@ -1068,33 +1063,56 @@ Exit: } /********************************************************************** - Process the entire group. + Process the entire group holder. NOTE : The lock should be held externally! **********************************************************************/ static ib_api_status_t mcast_mgr_process_mgrp(osm_sm_t * sm, - IN osm_mgrp_t * p_mgrp) + IN osm_mgrp_holder_t * p_mgrp_holder) { ib_api_status_t status; + osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + ib_net16_t mlid; + char gid_str[INET6_ADDRSTRLEN]; OSM_LOG_ENTER(sm->p_log); - status = osm_mcast_mgr_process_tree(sm, p_mgrp); + status = osm_mcast_mgr_process_tree(sm, p_mgrp_holder); if (status != IB_SUCCESS) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A19: " "Unable to create spanning tree (%s)\n", ib_get_err_str(status)); goto Exit; } - p_mgrp->last_tree_id = p_mgrp->last_change_id; - - /* remove MCGRP if it is marked for deletion */ - if (p_mgrp->to_be_deleted) { + p_mgrp_holder->last_tree_id = p_mgrp_holder->last_change_id; + + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while(p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t*)p_fitem; + mlid = p_mgrp->mlid; + p_fitem = cl_fmap_next(p_fitem); + if (osm_is_debug()) { + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "holder mlid 0x%X has mgrp mgid = %s\n", + cl_ntoh16(p_mgrp_holder->mlid), + inet_ntop(AF_INET6, p_mgrp->mcmember_rec.mgid.raw, gid_str, sizeof (gid_str))); + } + /* remove MCGRP if it is marked for deletion */ + if (p_mgrp->to_be_deleted) { + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Destroying mgrp %s with lid:0x%x\n", + inet_ntop(AF_INET6, p_mgrp->mcmember_rec.mgid.raw, gid_str, sizeof (gid_str)), + cl_ntoh16(p_mgrp->mlid)); + osm_mgrp_delete(sm->p_subn,p_mgrp); + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + } + } + if (0 == cl_fmap_count(&p_mgrp_holder->mgrp_map)) { + /* no more groups on this mlid*/ OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Destroying mgrp with lid:0x%x\n", - cl_ntoh16(p_mgrp->mlid)); - sm->p_subn->mgroups[cl_ntoh16(p_mgrp->mlid) - - IB_LID_MCAST_START_HO] = NULL; - osm_mgrp_delete(p_mgrp); + "Destroying mgrp_holder with lid:0x%x\n", + cl_ntoh16(p_mgrp_holder->mlid)); + osm_mgrp_holder_delete(sm->p_subn,p_mgrp_holder->mlid); } Exit: @@ -1110,7 +1128,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; cl_qlist_t *p_list = &sm->mgrp_list; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; boolean_t pending_transactions = FALSE; int i; @@ -1132,9 +1150,10 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) of the subnet. Not due to a specific multicast request. So the request type is subnet_change and the port guid is 0. */ - p_mgrp = sm->p_subn->mgroups[i]; - if (p_mgrp) - mcast_mgr_process_mgrp(sm, p_mgrp); + p_mgrp_holder = sm->p_subn->mgroup_holders[i]; + if (p_mgrp_holder) { + mcast_mgr_process_mgrp(sm, p_mgrp_holder); + } } /* @@ -1172,7 +1191,7 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) cl_qlist_t *p_list = &sm->mgrp_list; osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; ib_net16_t mlid; osm_signal_t ret, signal = OSM_SIGNAL_DONE; osm_mcast_mgr_ctxt_t *ctx; @@ -1193,24 +1212,24 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) /* since we delayed the execution we prefer to pass the mlid as the mgrp identifier and then find it or abort */ - p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, mlid); - if (!p_mgrp) + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sm->p_subn, mlid); + if (!p_mgrp_holder) continue; /* if there was no change from the last time * we processed the group we can skip doing anything */ - if (p_mgrp->last_change_id == p_mgrp->last_tree_id) { + if (p_mgrp_holder->last_change_id == p_mgrp_holder->last_tree_id) { OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Skip processing mgrp with lid:0x%X change id:%u\n", - cl_ntoh16(mlid), p_mgrp->last_change_id); + "Skip processing p_mgrp_holder with lid:0x%X change id:%u\n", + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); continue; } OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Processing mgrp with lid:0x%X change id:%u\n", - cl_ntoh16(mlid), p_mgrp->last_change_id); - mcast_mgr_process_mgrp(sm, p_mgrp); + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); + mcast_mgr_process_mgrp(sm, p_mgrp_holder); } /* diff --git a/opensm/opensm/osm_mgrp_holder.c b/opensm/opensm/osm_mgrp_holder.c new file mode 100644 index 0000000..9962c3e --- /dev/null +++ b/opensm/opensm/osm_mgrp_holder.c @@ -0,0 +1,315 @@ +/* + * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. + * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +/* + * Abstract: + * Implementation of multicast holder functions. + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +/********************************************************************** + **********************************************************************/ +#define PREFIX_MASK_IP CL_HTON64(0xff10ffff0000ffffULL) +#define PREFIX_SIGNATURE_IPV4 CL_HTON64(0xff10401b00000000ULL) +#define INTERFACE_ID_IPV4 CL_HTON64(0x0000000fffffffffULL) +#define PREFIX_SIGNATURE_IPV6 CL_HTON64(0xff10601b00000000ULL) +#define INTERFACE_ID_ALL_NODES CL_HTON64(0x00000000ffffffffULL) +#define INTERFACE_ID_ALL_HOSTS CL_HTON64(0x0000000000000001ULL) +#define INTERFACE_ID_ALL_GATEWAYS CL_HTON64(0x000000000000004dULL) +#define INTERFACE_ID_ALL_ROUTERS CL_HTON64(0x0000000000000002ULL) +#define PREFIX_PKEY_MASK_OFF CL_HTON64(0xffffffff0000ffffULL) +#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) +#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) +#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) +#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) + +/********************************************************************** + **********************************************************************/ + /* mask used for multiplexing mgids to one mlid. Here default value (0xff) means that 8 lsb bits + of mgid will be masked off . + */ +static uint64_t MGID_MUX_MLID_MASK = CL_HTON64(0x00000000000000ffULL); + +void osm_mgrp_holder_prepare_common_mgid(IN const ib_gid_t *const p_mgid, OUT ib_gid_t *p_common_mgid) +{ + memcpy(p_common_mgid, p_mgid,sizeof(ib_gid_t)); + /* Don't mux non IPoIB mgids. When mux mask is zero, no multiplexing occures*/ + if ( ( (p_common_mgid->unicast.prefix & PREFIX_MASK_IP) != PREFIX_SIGNATURE_IPV4) && + ((p_common_mgid->unicast.prefix & PREFIX_MASK_IP) != PREFIX_SIGNATURE_IPV6) || + ( ! MGID_MUX_MLID_MASK)) + return; + + if ( ( (p_common_mgid->unicast.prefix & PREFIX_MASK_IP) == PREFIX_SIGNATURE_IPV4) && + ( (p_common_mgid->unicast.interface_id == INTERFACE_ID_ALL_NODES) || + (p_common_mgid->unicast.interface_id == INTERFACE_ID_ALL_HOSTS) || + (p_common_mgid->unicast.interface_id == INTERFACE_ID_ALL_GATEWAYS) || + (p_common_mgid->unicast.interface_id == INTERFACE_ID_ALL_ROUTERS) )) { + /* zero pkey bits - special groups will have same mlid for all PKEYs*/ + p_common_mgid->unicast.prefix &= PREFIX_PKEY_MASK_OFF; + } + else if ((p_common_mgid->unicast.prefix & PREFIX_MASK_IP) == PREFIX_SIGNATURE_IPV6 && + (p_common_mgid->unicast.interface_id & INT_ID_MASK) == INT_ID_SIGNATURE) { + /* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ + /* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ + /* Where Z is the scope, XXXX is the P_Key, and + * YYYYYY is the last 24 bits of the port guid */ + p_common_mgid->unicast.prefix &= PREFIX_MASK_IP; + p_common_mgid->unicast.interface_id &= INT_ID_MASK; + } + else { + /* zero mask bits*/ + p_common_mgid->unicast.interface_id &= (~MGID_MUX_MLID_MASK); + } +} + +static int64_t +__mgid_cmp( + IN const void* const p_gid1, + IN const void* const p_gid2 ) +{ + return memcmp( p_gid1, p_gid2, sizeof(ib_gid_t) ); +} + +static osm_mgrp_port_t *osm_mgrp_port_new(ib_net64_t port_guid) +{ + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t*)malloc(sizeof(osm_mgrp_port_t)); + if (!p_mgrp_port) { + return NULL; + } + memset(p_mgrp_port, 0, sizeof(*p_mgrp_port)); + p_mgrp_port->port_guid = port_guid; + cl_qlist_init(&p_mgrp_port->mgroups); + return p_mgrp_port; +} + +void osm_mgrp_holder_delete(IN osm_subn_t * p_subn, ib_net16_t mlid) +{ + osm_mgrp_port_t *p_osm_mgr_port; + cl_map_item_t *p_item; + cl_fmap_item_t *p_fitem; + osm_mgrp_t * p_mgrp; + + osm_mgrp_holder_t *p_mgrp_holder = + p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + p_item = cl_qmap_head( &p_mgrp_holder->mgrp_port_map); + while (p_item != cl_qmap_end( &p_mgrp_holder->mgrp_port_map)) { + p_osm_mgr_port = (osm_mgrp_port_t *)p_item; + cl_qlist_remove_all(&p_osm_mgr_port->mgroups); + cl_qmap_remove_item( &p_mgrp_holder->mgrp_port_map, p_item); + p_item = cl_qmap_head( &p_mgrp_holder->mgrp_port_map); + free(p_osm_mgr_port); + } + p_fitem = cl_fmap_head( &p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end( &p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t *)p_fitem; + osm_mgrp_delete(p_subn, p_mgrp); + p_fitem = cl_fmap_head( &p_mgrp_holder->mgrp_map); + } + /* destroy the mtree_node structure */ + osm_mtree_destroy(p_mgrp_holder->p_root); + p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; + free(p_mgrp_holder); +} + +/********************************************************************** + **********************************************************************/ +osm_mgrp_holder_t *osm_mgrp_holder_new(IN osm_subn_t *p_subn,ib_gid_t * p_mgid, ib_net16_t mlid) +{ + osm_mgrp_holder_t *p_mgrp_holder; + p_mgrp_holder = p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = + (osm_mgrp_holder_t *) malloc(sizeof(*p_mgrp_holder)); + if (!p_mgrp_holder) + return NULL; + + memset(p_mgrp_holder, 0, sizeof(*p_mgrp_holder)); + p_mgrp_holder->mlid = mlid; + cl_fmap_init(&p_mgrp_holder->mgrp_map, __mgid_cmp); + cl_qmap_init(&p_mgrp_holder->mgrp_port_map); + osm_mgrp_holder_prepare_common_mgid(p_mgid, &p_mgrp_holder->common_mgid); + return p_mgrp_holder; +} + +ib_net16_t osm_mgrp_holder_get_mlid_by_mgid(IN osm_sa_t * sa, IN const ib_gid_t *const p_mgid) +{ + int i; + ib_gid_t common_mgid; + osm_mgrp_holder_t * p_mgrp_holder; + + OSM_LOG_ENTER(sa->p_log); + + osm_mgrp_holder_prepare_common_mgid(p_mgid, &common_mgid); + for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; + i++) { + if (sa->p_subn->mgroup_holders[i]) { + p_mgrp_holder = (osm_mgrp_holder_t*)sa->p_subn->mgroup_holders[i]; + if ( !memcmp(&p_mgrp_holder->common_mgid, &common_mgid,sizeof(ib_gid_t))) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "Found holder 0x%X for MGID %s\n", + cl_ntoh16(p_mgrp_holder->mlid), + inet_ntop(AF_INET6, p_mgid->raw, gid_str, + sizeof (gid_str))); + OSM_LOG_EXIT(sa->p_log); + return p_mgrp_holder->mlid; + } + } + } + OSM_LOG_EXIT(sa->p_log); + return 0; +} + +/* drop the port */ +void osm_mgrp_holder_remove_port(osm_subn_t * subn, osm_log_t * p_log, osm_mgrp_holder_t * p_mgrp_holder, + ib_net64_t port_guid) +{ + osm_mgrp_t * p_mgrp; + cl_list_item_t *p_item; + + OSM_LOG_ENTER(p_log); + + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t*) + cl_qmap_remove(&p_mgrp_holder->mgrp_port_map, port_guid); + if (p_mgrp_port != (osm_mgrp_port_t*)cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(p_log, OSM_LOG_DEBUG,"port 0x%" PRIx64 " removed from mlid 0x%X\n", + port_guid,cl_ntoh16(p_mgrp_holder->mlid)); + while ((p_item = cl_qlist_remove_head(&p_mgrp_port->mgroups)) != + cl_qlist_end(&p_mgrp_port->mgroups)) { + p_mgrp = (osm_mgrp_t*) PARENT_STRUCT(p_item, osm_mgrp_t, mgrp_item); + OSM_LOG(p_log, OSM_LOG_DEBUG,"removing mgrp mgid %s from port 0x%" PRIx64 "\n", + inet_ntop(AF_INET6, p_mgrp->mcmember_rec.mgid.raw, gid_str, sizeof (gid_str)), + cl_ntoh64(port_guid)); + osm_mgrp_delete_port(subn, p_log, p_mgrp, port_guid); + } + free(p_mgrp_port); + } + OSM_LOG_EXIT(p_log); +} + + +void osm_mgrp_holder_add_mgrp(osm_mgrp_holder_t * p_mgrp_holder, osm_mgrp_t *p_mgrp,osm_log_t * p_log) +{ + cl_fmap_item_t *p_fitem; + char gid_str[INET6_ADDRSTRLEN]; + + OSM_LOG_ENTER(p_log); + + if (p_mgrp_holder->to_be_deleted) { + /* this is re-used mgrp_holder, need to update common_mgid*/ + osm_mgrp_holder_prepare_common_mgid(&p_mgrp->mcmember_rec.mgid, + &p_mgrp_holder->common_mgid); + } + p_fitem = cl_fmap_insert(&p_mgrp_holder->mgrp_map, + &p_mgrp->mcmember_rec.mgid ,&p_mgrp->mgid_item); + CL_ASSERT( p_item == &p_mgrp->mgid_item ); + + OSM_LOG(p_log, OSM_LOG_DEBUG, "mgrp with MGID:%s added to holder with mlid = 0x%X\n", + inet_ntop(AF_INET6, p_mgrp->mcmember_rec.mgid.raw, gid_str, + sizeof( gid_str)),cl_ntoh16(p_mgrp_holder->mlid)); + + p_mgrp_holder->last_change_id++; + OSM_LOG_EXIT(p_log); +} + +ib_api_status_t osm_mgrp_holder_add_mgrp_port(osm_mgrp_holder_t *p_mgrp_holder, osm_mgrp_t *p_mgrp, + ib_net64_t port_guid) +{ + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t*) + cl_qmap_get(&p_mgrp_holder->mgrp_port_map, port_guid); + if (p_mgrp_port == (osm_mgrp_port_t*)cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + /* new port to mlid*/ + p_mgrp_port = osm_mgrp_port_new(port_guid); + if (!p_mgrp_port) { + return IB_INSUFFICIENT_MEMORY; + } + cl_qmap_insert(&p_mgrp_holder->mgrp_port_map, + p_mgrp_port->port_guid, + &p_mgrp_port->guid_item); + cl_qlist_insert_tail(&p_mgrp_port->mgroups, + &p_mgrp->mgrp_item); + } + else { + cl_qlist_insert_tail(&p_mgrp_port->mgroups, &p_mgrp->mgrp_item); + } + return IB_SUCCESS; +} + +void osm_mgrp_holder_delete_mgrp_port(osm_mgrp_holder_t *p_mgrp_holder, osm_mgrp_t *p_mgrp, + ib_net64_t port_guid) +{ + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t*) + cl_qmap_get(&p_mgrp_holder->mgrp_port_map, port_guid); + if (p_mgrp_port != (osm_mgrp_port_t*)cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + cl_qlist_remove_item(&p_mgrp_port->mgroups, &p_mgrp->mgrp_item); + if (0 == cl_qlist_count(&p_mgrp_port->mgroups)) { + /* No mgroups registered on this port for current mlid*/ + cl_qmap_remove_item(&p_mgrp_holder->mgrp_port_map, &p_mgrp_port->guid_item); + free(p_mgrp_port); + } + p_mgrp_holder->last_change_id++; + } +} + +void osm_mgrp_holder_delete_mgrp(osm_mgrp_holder_t *p_mgrp_holder, osm_mgrp_t *p_mgrp) +{ + p_mgrp->to_be_deleted = 1; + cl_fmap_remove_item(&p_mgrp_holder->mgrp_map, &p_mgrp->mgid_item); + if (0 == cl_fmap_count(&p_mgrp_holder->mgrp_map)) { + /* No more mgroups on this mlid*/ + p_mgrp_holder->to_be_deleted = 1; + p_mgrp_holder->last_tree_id = 0; + p_mgrp_holder->last_change_id = 0; + memset(&p_mgrp_holder->common_mgid,0,sizeof(ib_gid_t)); + } +} +/********************************************************************** + **********************************************************************/ + diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index d2733c4..f57b3e9 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -41,23 +41,24 @@ #if HAVE_CONFIG_H # include #endif /* HAVE_CONFIG_H */ - +#include #include #include #include #include #include #include - +#include /********************************************************************** **********************************************************************/ -void osm_mgrp_delete(IN osm_mgrp_t * p_mgrp) +void osm_mgrp_delete(IN osm_subn_t * p_subn, IN osm_mgrp_t *p_mgrp) { osm_mcm_port_t *p_mcm_port; osm_mcm_port_t *p_next_mcm_port; CL_ASSERT(p_mgrp); + osm_mgrp_holder_t *p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_subn, p_mgrp->mlid); p_next_mcm_port = (osm_mcm_port_t *) cl_qmap_head(&p_mgrp->mcm_port_tbl); while (p_next_mcm_port != @@ -65,14 +66,13 @@ void osm_mgrp_delete(IN osm_mgrp_t * p_mgrp) p_mcm_port = p_next_mcm_port; p_next_mcm_port = (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item); + osm_mgrp_holder_delete_mgrp_port(p_mgrp_holder, p_mgrp, + p_mcm_port->port_gid.unicast.interface_id); osm_mcm_port_delete(p_mcm_port); } - /* destroy the mtree_node structure */ - osm_mtree_destroy(p_mgrp->p_root); - + osm_mgrp_holder_delete_mgrp(p_mgrp_holder, p_mgrp); free(p_mgrp); } - /********************************************************************** **********************************************************************/ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid) @@ -133,6 +133,7 @@ osm_mcm_port_t *osm_mgrp_add_port(IN osm_subn_t * subn, osm_log_t * log, ib_net64_t port_guid; osm_mcm_port_t *p_mcm_port; cl_map_item_t *prev_item; + osm_mgrp_holder_t *p_mgrp_holder; uint8_t prev_join_state = 0; uint8_t prev_scope; @@ -168,8 +169,25 @@ osm_mcm_port_t *osm_mgrp_add_port(IN osm_subn_t * subn, osm_log_t * log, ib_member_set_scope_state(prev_scope, prev_join_state | join_state); } else { - /* track the fact we modified the group ports */ - p_mgrp->last_change_id++; + p_mgrp_holder = subn->mgroup_holders[cl_ntoh16(p_mgrp->mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp_holder) + /* create new holder */ + p_mgrp_holder = + subn->mgroup_holders[cl_ntoh16(p_mgrp->mlid) - IB_LID_MCAST_START_HO] = + osm_mgrp_holder_new(subn,&p_mgrp->mcmember_rec.mgid,p_mgrp->mlid); + + if (p_mgrp_holder && + (IB_SUCCESS == osm_mgrp_holder_add_mgrp_port(p_mgrp_holder, + p_mgrp, port_guid)) ) { + /* track the fact we modified the group ports */ + p_mgrp->last_change_id++; + } + else { + /* if osm_mgrp_holder_new failed, remove port also from mcm_port_tbl */ + cl_qmap_remove_item(&p_mgrp->mcm_port_tbl, &p_mcm_port->map_item); + osm_mcm_port_delete(p_mcm_port); + return NULL; + } } if ((join_state & IB_JOIN_STATE_FULL) && @@ -190,7 +208,7 @@ int osm_mgrp_remove_port(osm_subn_t * subn, osm_log_t * log, osm_mgrp_t * mgrp, int ret; uint8_t port_join_state; uint8_t new_join_state; - + char gid_str[INET6_ADDRSTRLEN]; /* * according to the same o15-0.1.14 we get the stored * JoinState and the request JoinState and they must be @@ -198,7 +216,9 @@ int osm_mgrp_remove_port(osm_subn_t * subn, osm_log_t * log, osm_mgrp_t * mgrp, */ port_join_state = mcm->scope_state & 0x0F; new_join_state = port_join_state & ~join_state; - + OSM_LOG(log, OSM_LOG_DEBUG, "mgrp mgid %s well_known = %d, new_join_state = %d\n", + inet_ntop(AF_INET6, mgrp->mcmember_rec.mgid.raw, gid_str, sizeof( gid_str)), + mgrp->well_known,new_join_state); if (new_join_state) { mcm->scope_state = new_join_state | (mcm->scope_state & 0xf0); OSM_LOG(log, OSM_LOG_DEBUG, @@ -283,18 +303,3 @@ static void mgrp_apply_func_sub(const osm_mgrp_t * p_mgrp, } } -/********************************************************************** - **********************************************************************/ -void osm_mgrp_apply_func(const osm_mgrp_t * p_mgrp, osm_mgrp_func_t p_func, - void *context) -{ - osm_mtree_node_t *p_mtn; - - CL_ASSERT(p_mgrp); - CL_ASSERT(p_func); - - p_mtn = p_mgrp->p_root; - - if (p_mtn) - mgrp_apply_func_sub(p_mgrp, p_mtn, p_func, context); -} diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index 094fef2..1d15776 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -53,6 +53,7 @@ #include #include #include +#include extern osm_qos_level_t __default_simple_qos_level; @@ -796,6 +797,8 @@ static void __qos_policy_validate_pkey( uint32_t flow; uint8_t hop; osm_mgrp_t * p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; + cl_fmap_item_t *p_fitem; if (!p_qos_policy || !p_qos_match_rule || !p_prtn) return; @@ -819,28 +822,33 @@ static void __qos_policy_validate_pkey( if (!p_prtn->mlid) return; - p_mgrp = osm_get_mgrp_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); - if (!p_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); + if (!p_mgrp_holder) { OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, - "ERR AC16: MCast group for partition with " + "ERR AC16: MCast mgrp_holder for partition with " "pkey 0x%04X not found\n", cl_ntoh16(p_prtn->pkey)); return; } - - CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == - (cl_ntoh16(p_prtn->pkey) & 0x7fff)); - - ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, - &sl, &flow, &hop); - if (sl != p_prtn->sl) { - OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, - "Updating MCGroup (MLID 0x%04x) SL to " - "match partition SL (%u)\n", - cl_hton16(p_mgrp->mcmember_rec.mlid), - p_prtn->sl); - p_mgrp->mcmember_rec.sl_flow_hop = - ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t*)p_fitem; + p_fitem = cl_fmap_next(p_fitem); + + CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == + (cl_ntoh16(p_prtn->pkey) & 0x7fff)); + + ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, + &sl, &flow, &hop); + if (sl != p_prtn->sl) { + OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, + "Updating MCGroup (MLID 0x%04x) SL to " + "match partition SL (%u)\n", + cl_hton16(p_mgrp->mcmember_rec.mlid), + p_prtn->sl); + p_mgrp->mcmember_rec.sl_flow_hop = + ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); + } } } diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index 3521132..e32f5c5 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -67,6 +68,7 @@ #include #include #include +#include #define OSM_SA_INITIAL_TID_VALUE 0xabc @@ -705,6 +707,7 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) { struct opensm_dump_context dump_context; osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t * p_mgrp_holder; int i; dump_context.p_osm = p_osm; @@ -713,9 +716,14 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) cl_plock_acquire(&p_osm->lock); for (i = 0; i <= p_osm->subn.max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = p_osm->subn.mgroups[i]; - if (p_mgrp) + p_mgrp_holder = p_osm->subn.mgroup_holders[i]; + if (!p_mgrp_holder) + continue; + p_mgrp = (osm_mgrp_t*)cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_mgrp != (osm_mgrp_t*)cl_fmap_end(&p_mgrp_holder->mgrp_map)) { sa_dump_one_mgrp(p_mgrp, &dump_context); + p_mgrp = (osm_mgrp_t*)cl_fmap_next(&p_mgrp->mgid_item); + } } OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform\n"); cl_qlist_apply_func(&p_osm->subn.sa_infr_list, @@ -740,24 +748,28 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, { ib_net64_t comp_mask; osm_mgrp_t *p_mgrp; - + osm_mgrp_holder_t *p_mgrp_holder; + ib_gid_t common_mgid; cl_plock_excl_acquire(&p_osm->lock); - p_mgrp = osm_get_mgrp_by_mlid(&p_osm->subn, mlid); - if (p_mgrp) { - if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, - sizeof(ib_gid_t))) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(&p_osm->subn, mlid); + if (p_mgrp_holder) { + p_mgrp = (osm_mgrp_t*)cl_fmap_get(&p_mgrp_holder->mgrp_map ,&p_mcm_rec->mgid); + if (p_mgrp != (osm_mgrp_t*)cl_fmap_end(&p_mgrp_holder->mgrp_map)) { OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "mgrp %04x is already here.", cl_ntoh16(mlid)); goto _out; } - OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, - "mlid %04x is already used by another MC group. Will " - "request clients reregistration.\n", cl_ntoh16(mlid)); - p_mgrp = NULL; - goto _out; - } + osm_mgrp_holder_prepare_common_mgid(&p_mcm_rec->mgid, &common_mgid); + if (memcmp(&p_mgrp_holder->common_mgid, &common_mgid, sizeof(ib_gid_t))) { + OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, + "mlid %04x is already used by another MC group. Will " + "request clients reregistration.\n", cl_ntoh16(mlid)); + p_mgrp = NULL; + goto _out; + } + } comp_mask = IB_MCR_COMPMASK_MTU | IB_MCR_COMPMASK_MTU_SEL | IB_MCR_COMPMASK_RATE | IB_MCR_COMPMASK_RATE_SEL; if (osm_mcmr_rcv_find_or_create_new_mgrp(&p_osm->sa, diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 5543221..d431cfb 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -63,6 +63,7 @@ #include #include #include +#include #define JOIN_MC_COMP_MASK (IB_MCR_COMPMASK_MGID | \ IB_MCR_COMPMASK_PORT_GID | \ @@ -121,13 +122,13 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_mcast_lid_ho - && !osm_get_mgrp_by_mlid(p_subn, requested_mlid)) + && !osm_get_mgrp_holder_by_mlid(p_subn, requested_mlid)) return requested_mlid; max = p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO + 1; for (i = 0; i < max; i++) { - osm_mgrp_t *p_mgrp = sa->p_subn->mgroups[i]; - if (!p_mgrp || p_mgrp->to_be_deleted) + osm_mgrp_holder_t *p_mgrp_holder = (osm_mgrp_holder_t*)sa->p_subn->mgroup_holders[i]; + if (!p_mgrp_holder || p_mgrp_holder->to_be_deleted) return cl_hton16(i + IB_LID_MCAST_START_HO); } @@ -141,14 +142,16 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) we silently drop it. Since it was an intermediate group no need to re-route it. **********************************************************************/ -static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * mgrp) +static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * p_mgrp) { /* Remove MGRP only if osm_mcm_port_t count is 0 and not a well known group */ - if (cl_is_qmap_empty(&mgrp->mcm_port_tbl) && !mgrp->well_known) { - sa->p_subn->mgroups[cl_ntoh16(mgrp->mlid) - - IB_LID_MCAST_START_HO] = NULL; - osm_mgrp_delete(mgrp); + char gid_str[INET6_ADDRSTRLEN]; + if (cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && !p_mgrp->well_known) { + OSM_LOG(sa->p_log, OSM_LOG_DEBUG,"mgrp mgid %s will be deleted\n", + inet_ntop(AF_INET6, p_mgrp->mcmember_rec.mgid.raw, + gid_str, sizeof( gid_str))); + osm_mgrp_delete(sa->sm->p_subn,p_mgrp); } } @@ -202,7 +205,6 @@ static ib_api_status_t add_new_mgrp_port(osm_sa_t * sa, IN osm_mgrp_t * p_mgrp, return IB_INSUFFICIENT_MEMORY; } - return IB_SUCCESS; } @@ -806,11 +808,12 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, IN const osm_physp_t * p_physp, OUT osm_mgrp_t ** pp_mgrp) { - ib_net16_t mlid; + ib_net16_t mlid, existed_mlid; unsigned zero_mgid, i; uint8_t scope; + char gid_str[INET6_ADDRSTRLEN]; ib_gid_t *p_mgid; - osm_mgrp_t *p_prev_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; ib_api_status_t status = IB_SUCCESS; ib_member_rec_t mcm_rec = *p_recvd_mcmember_rec; /* copy for modifications */ @@ -823,11 +826,11 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, zero_mgid = 0; break; } - + p_mgid = &(mcm_rec.mgid); /* we allocate a new mlid number before we might use it for MGID ... - */ + */ mlid = get_new_mlid(sa, mcm_rec.mlid); if (mlid == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B19: " @@ -843,8 +846,6 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* we need to create the new MGID if it was not defined */ if (zero_mgid) { /* create a new MGID */ - char gid_str[INET6_ADDRSTRLEN]; - /* use the given scope state only if requested! */ if (comp_mask & IB_MCR_COMPMASK_SCOPE) ib_member_get_scope_state(p_recvd_mcmember_rec-> @@ -888,6 +889,12 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, goto Exit; } + if (0 != (existed_mlid = osm_mgrp_holder_get_mlid_by_mgid(sa,p_mgid))) { + mlid = existed_mlid; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "found existed mlid 0x%04x for mgid %s\n", + cl_ntoh16(mlid),inet_ntop(AF_INET6, p_mgid->raw, gid_str,sizeof gid_str)); + } /* create a new MC Group */ *pp_mgrp = osm_mgrp_new(mlid); if (*pp_mgrp == NULL) { @@ -915,17 +922,20 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* since we might have an old group by that mlid one whose deletion was delayed for an idle time we need to deallocate it first */ - p_prev_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); - if (p_prev_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sa->p_subn, mlid); + if (! p_mgrp_holder) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Found previous group for mlid:0x%04x - " - "Destroying it first\n", cl_ntoh16(mlid)); - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = - NULL; - osm_mgrp_delete(p_prev_mgrp); + "Creating new mgrp holder for mlid:0x%04x\n", cl_ntoh16(mlid)); + p_mgrp_holder = osm_mgrp_holder_new(sa->p_subn,p_mgid, mlid); } - - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; + if (!p_mgrp_holder) { + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B08: " + "osm_mgrp_holder_new failed\n"); + free_mlid(sa, mlid); + status = IB_INSUFFICIENT_MEMORY; + goto Exit; + } + osm_mgrp_holder_add_mgrp(p_mgrp_holder, *pp_mgrp, sa->p_log); Exit: OSM_LOG_EXIT(sa->p_log); @@ -934,56 +944,42 @@ Exit: /********************************************************************** *********************************************************************/ -static unsigned match_mgrp_by_mgid(IN osm_mgrp_t * p_mgrp, ib_gid_t * mgid) -{ - /* ignore groups marked for deletion */ - if (p_mgrp->to_be_deleted || - memcmp(&p_mgrp->mcmember_rec.mgid, mgid, sizeof(ib_gid_t))) - return 0; - else - return 1; -} - -/********************************************************************** - **********************************************************************/ -#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) -#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) -#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) -#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) - -/* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ -/* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ -/* Where Z is the scope, XXXX is the P_Key, and - * YYYYYY is the last 24 bits of the port guid */ -static unsigned match_and_update_ipv6_snm_mgid(ib_gid_t * mgid) +static osm_mgrp_t* match_mgrp_by_mgid(IN osm_mgrp_holder_t * p_mgrp_holder, ib_gid_t * p_mgid) { - if ((mgid->unicast.prefix & PREFIX_MASK) == PREFIX_SIGNATURE && - (mgid->unicast.interface_id & INT_ID_MASK) == INT_ID_SIGNATURE) { - mgid->unicast.prefix &= PREFIX_MASK; - mgid->unicast.interface_id &= INT_ID_MASK; - return 1; + osm_mgrp_t *p_mgrp; + ib_gid_t common_mgid; + + osm_mgrp_holder_prepare_common_mgid(p_mgid, &common_mgid); + + if (memcmp(&p_mgrp_holder->common_mgid, &common_mgid, sizeof(ib_gid_t))) + return NULL; + p_mgrp = (osm_mgrp_t*)cl_fmap_get(&p_mgrp_holder->mgrp_map,p_mgid); + if (p_mgrp != (osm_mgrp_t*)cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + /* ignore groups marked for deletion */ + if (p_mgrp->to_be_deleted) + p_mgrp = NULL; } - return 0; + else + p_mgrp = NULL; + return p_mgrp;; } osm_mgrp_t *osm_get_mgrp_by_mgid(IN osm_sa_t * sa, IN ib_gid_t * p_mgid) { int i; - - if (sa->p_subn->opt.consolidate_ipv6_snm_req && - match_and_update_ipv6_snm_mgid(p_mgid)) { - char gid_str[INET6_ADDRSTRLEN]; - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Special Case Solicited Node Mcast Join for MGID %s\n", - inet_ntop(AF_INET6, p_mgid->raw, gid_str, - sizeof gid_str)); - } - + osm_mgrp_t *p_mgrp; for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; - i++) - if (sa->p_subn->mgroups[i] && - match_mgrp_by_mgid(sa->p_subn->mgroups[i], p_mgid)) - return sa->p_subn->mgroups[i]; + i++) { + if (sa->p_subn->mgroup_holders[i] && + (p_mgrp = match_mgrp_by_mgid(sa->p_subn->mgroup_holders[i], p_mgid))) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "Found mgroup for MGID %s\n", + inet_ntop(AF_INET6, p_mgid->raw, gid_str, + sizeof gid_str)); + return p_mgrp; + } + } return NULL; } @@ -1080,7 +1076,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) CL_PLOCK_RELEASE(sa->p_lock); /* we can leave if port was deleted from MCG */ - if (removed && osm_sm_mcgrp_leave(sa->sm, mlid, portguid)) + if (removed && osm_sm_mcgrp_leave(sa->sm, p_mgrp, portguid)) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B09: " "osm_sm_mcgrp_leave failed\n"); @@ -1309,8 +1305,8 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* do the actual routing (actually schedule the update) */ status = osm_sm_mcgrp_join(sa->sm, mlid, - p_recvd_mcmember_rec->port_gid.unicast. - interface_id); + p_recvd_mcmember_rec->port_gid.unicast.interface_id, + &p_recvd_mcmember_rec->mgid); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " @@ -1555,6 +1551,7 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) osm_physp_t *p_req_physp; boolean_t trusted_req; osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t * p_mgrp_holder; int i; OSM_LOG_ENTER(sa->p_log); @@ -1586,10 +1583,15 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* simply go over all MCGs and match */ for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = sa->p_subn->mgroups[i]; - if (p_mgrp) + p_mgrp_holder = sa->p_subn->mgroup_holders[i]; + if (!p_mgrp_holder) + continue; + p_mgrp = (osm_mgrp_t*)cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_mgrp != (osm_mgrp_t*)cl_fmap_end(&p_mgrp_holder->mgrp_map)) { mcmr_by_comp_mask(sa, p_rcvd_rec, comp_mask, p_mgrp, p_req_physp, trusted_req, &rec_list); + p_mgrp = (osm_mgrp_t*)cl_fmap_next(&p_mgrp->mgid_item); + } } CL_PLOCK_RELEASE(sa->p_lock); diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 9b50deb..e1e324f 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -68,6 +68,7 @@ #include #include #include +#include typedef struct osm_pr_item { cl_list_item_t list_item; @@ -1438,6 +1439,7 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; ib_net64_t comp_mask; + osm_mgrp_holder_t *p_mgrp_holder; osm_mgrp_t *mgrp = NULL; p_sa_mad = osm_madw_get_sa_mad_ptr(p_madw); @@ -1468,11 +1470,17 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) mgrp = NULL; goto Exit; } - } else - if (!(mgrp = osm_get_mgrp_by_mlid(sa->p_subn, p_pr->dlid))) - OSM_LOG(sa->p_log, OSM_LOG_ERROR, - "ERR 1F11: " "No MC group found for PathRecord " - "destination LID 0x%x\n", p_pr->dlid); + } else { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sa->p_subn, p_pr->dlid); + if (p_mgrp_holder) { + mgrp = (osm_mgrp_t*)cl_fmap_get(&p_mgrp_holder->mgrp_map, &p_pr->dgid); + } + if (!p_mgrp_holder || mgrp == (osm_mgrp_t*)cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + OSM_LOG(sa->p_log, OSM_LOG_ERROR, + "ERR 1F11: " "No MC group found for PathRecord " + "destination LID 0x%x\n", p_pr->dlid); + } + } } Exit: diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index daa60ff..5b6d8cf 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -47,6 +47,7 @@ #include #include +#include #include #include #include @@ -60,6 +61,7 @@ #include #include #include +#include #define OSM_SM_INITIAL_TID_VALUE 0x1233 @@ -468,12 +470,14 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, /********************************************************************** **********************************************************************/ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid) + IN const ib_net64_t port_guid, IN const ib_gid_t * p_mgid) { - osm_mgrp_t *p_mgrp; + osm_mgrp_t *p_mgrp = NULL; osm_port_t *p_port; ib_api_status_t status = IB_SUCCESS; osm_mcm_info_t *p_mcm; + cl_fmap_item_t *p_item; + osm_mgrp_holder_t *p_mgrp_holder; OSM_LOG_ENTER(p_sm->p_log); @@ -498,8 +502,38 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, /* * If this multicast group does not already exist, create it. */ - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); - if (!p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); + if (p_mgrp_holder) { + char gid_str[INET6_ADDRSTRLEN]; + if (osm_is_debug()) { + size_t gr_count = cl_fmap_count(&p_mgrp_holder->mgrp_map); + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG,"mlid 0x%X has %d mgroups\n", + cl_ntoh16(mlid),gr_count); + if (gr_count) { + p_mgrp = (osm_mgrp_t*)cl_fmap_head( &p_mgrp_holder->mgrp_map); + while (p_mgrp != (osm_mgrp_t*)cl_fmap_end( &p_mgrp_holder->mgrp_map)) { + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "mlid 0x%X has mgrp with MGID: %s\n",cl_ntoh16(mlid), + inet_ntop(AF_INET6, + p_mgrp->mcmember_rec.mgid.raw, + gid_str, sizeof gid_str)); + p_mgrp = (osm_mgrp_t*)cl_fmap_next( &p_mgrp->mgid_item); + } + } + } + p_item = cl_fmap_get( &p_mgrp_holder->mgrp_map, p_mgid ); + if ( p_item == cl_fmap_end( &p_mgrp_holder->mgrp_map ) ) { + p_mgrp = NULL; + OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, + "group with MGID: %s not found on mlid 0x%X\n", + inet_ntop(AF_INET6, + p_mgid->raw, + gid_str, sizeof gid_str), cl_ntoh16(mlid)); + } + else + p_mgrp = (osm_mgrp_t *)p_item; + } + if (!p_mgrp_holder || !p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { /* * The group removed or the port is not a * member of the group, then fail immediately. @@ -514,7 +548,21 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, status = IB_NOT_FOUND; goto Exit; } - + /* if there was no change from the last time + * we processed the group we can skip doing anything + */ + if (p_mgrp_holder->last_change_id == p_mgrp_holder->last_tree_id) { + CL_PLOCK_RELEASE(p_sm->p_lock); + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, + "Skip processing mgrp holder with lid:0x%X last change id:%u\n", + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); + goto Exit; + } else { + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "processing mgrp holder with lid:0x%X port: 0x%016" PRIx64 " last change id:%u tree id:%u\n", + cl_ntoh16(mlid), cl_ntoh64(port_guid), + p_mgrp_holder->last_change_id, p_mgrp_holder->last_tree_id); + } /* * Check if the object (according to mlid) already exists on this port. * If it does - then no need to update it again, and no need to @@ -543,9 +591,6 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } - status = sm_mgrp_process(p_sm, p_mgrp); - CL_PLOCK_RELEASE(p_sm->p_lock); - Exit: OSM_LOG_EXIT(p_sm->p_log); return status; @@ -553,12 +598,13 @@ Exit: /********************************************************************** **********************************************************************/ -ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, +ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN osm_mgrp_t *p_mgrp, IN const ib_net64_t port_guid) { - osm_mgrp_t *p_mgrp; osm_port_t *p_port; ib_api_status_t status; + osm_mgrp_holder_t *p_mgrp_holder; + ib_net16_t mlid = p_mgrp->mlid; OSM_LOG_ENTER(p_sm->p_log); @@ -584,8 +630,8 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, /* * Get the multicast group object for this group. */ - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); - if (!p_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); + if (!p_mgrp_holder) { CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); @@ -593,11 +639,12 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } + osm_mgrp_holder_delete_mgrp_port(p_mgrp_holder, p_mgrp, port_guid); /* * Walk the list of ports in the group, and remove the appropriate one. */ osm_port_remove_mgrp(p_port, mlid); - + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG," Calling sm_mgrp_process for mgrp with mlid = 0x%X\n",cl_ntoh16(mlid)); status = sm_mgrp_process(p_sm, p_mgrp); CL_PLOCK_RELEASE(p_sm->p_lock); diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index ec15f8a..a6d4f4c 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -68,6 +68,7 @@ #include #include #include +#include static const char null_str[] = "(null)"; @@ -419,7 +420,7 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) osm_switch_t *p_sw, *p_next_sw; osm_remote_sm_t *p_rsm, *p_next_rsm; osm_prtn_t *p_prtn, *p_next_prtn; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_osm_mgrp_holder; osm_infr_t *p_infr, *p_next_infr; /* it might be a good idea to de-allocate all known objects */ @@ -464,10 +465,10 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) for (i = 0; i <= p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = p_subn->mgroups[i]; - p_subn->mgroups[i] = NULL; - if (p_mgrp) - osm_mgrp_delete(p_mgrp); + p_osm_mgrp_holder = p_subn->mgroup_holders[i]; + if (p_osm_mgrp_holder){ + osm_mgrp_holder_delete(p_subn, p_osm_mgrp_holder->mlid); + } } p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); -- 1.5.5 From hnrose at comcast.net Thu Jun 18 06:04:32 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 18 Jun 2009 09:04:32 -0400 Subject: [ofa-general] [PATCH] opensm/osm_ucast_lash.c: Remove some unused code Message-ID: <20090618130432.GA23147@comcast.net> Signed-off-by: Robert Pearson Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c index 5913434..12b5e34 100644 --- a/opensm/opensm/osm_ucast_lash.c +++ b/opensm/opensm/osm_ucast_lash.c @@ -478,7 +478,7 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) cdg_vertex_t ****cdg_vertex_matrix = p_lash->cdg_vertex_matrix; int *num_mst_in_lane = p_lash->num_mst_in_lane; int ***virtual_location = p_lash->virtual_location; - int min_filled_lane, max_filled_lane, medium_filled_lane, trials; + int min_filled_lane, max_filled_lane, trials; int old_min_filled_lane, old_max_filled_lane, new_num_min_lane, new_num_max_lane; unsigned int i, j; @@ -490,9 +490,6 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) max_filled_lane = 0; min_filled_lane = lanes_needed - 1; - if (max_filled_lane > 1) - medium_filled_lane = max_filled_lane - 1; - trials = num_mst_in_lane[max_filled_lane]; if (lanes_needed == 1) stop = 1; From liranl at mellanox.co.il Thu Jun 18 06:27:12 2009 From: liranl at mellanox.co.il (Liran Liss) Date: Thu, 18 Jun 2009 16:27:12 +0300 Subject: [ofa-general] [PATCH 0/9] RDMAoE - RDMA over Ethernet In-Reply-To: References: <20090615073408.GA27543@mtls03> <8B9659B85D3D4DD38ADD571935236F3A@amr.corp.intel.com> <5D49E7A8952DC44FB38C38FA0D758EAD030C3A22@mtlexch01.mtl.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD0EE0D4@mtlexch01.mtl.com> >Let's just say that at this point I completely disagree with where these patches try to abstract the differences, which are many. >RDMA apps that want to use this and IB without going through an abstraction will need different code -- just like they would for iWarp, which also provides RDMA over Ethernet, and is a standard. IB mad and SA query modules are not >appropriate places for abstracting the differences between IB, iWarp, and whatever name we give this. >This could change depending on whether this is really trying to be IB with a different L2, or is just another RDMA protocol that runs on Ethernet. >- Sean Sean, These are indeed real concerns; I know that the cma is the natural place for abstracting transport differences, but I am worried about non-cma Infiniband ULPs which can work just as well with RDMAoE (perhaps we can specifically expose RDMAoE "path queries" as a simple library function). We will rethink our approach to SA queries and post new patches shortly. Note that without SA query emulation, the RDMAoE patches really amount to just a few cosmetic changes to ib_core...:) Thanks for the feedback. --Liran From sashak at voltaire.com Thu Jun 18 06:27:25 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 16:27:25 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <4A30B3DA.30601@bull.net> References: <4A30B3DA.30601@bull.net> Message-ID: <20090618132725.GD6163@me> Hi Vincent, On 09:35 Thu 11 Jun , Vincent Ficet wrote: > > We worked on a scheme that allows to dynamically load external routing > engines. Hmm, and why is it really needed? Why to not include routing engine code into OpenSM? > The plugin system is very similar to the perfmgr plugin > implementation. Once it was perfmgr event plugin, now plugin API is generic so what is the reason to make yet another plugin interface? > It allows to provide customised vendor routing engines > and algorithms whithout having to recompile opensm: Recompiling OpenSM should not be a big deal. Assuming that it still be an open source project. Sasha From sashak at voltaire.com Thu Jun 18 06:48:00 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 16:48:00 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c: Enhanced Fat-Tree algorithm In-Reply-To: <4A3392C4.1090608@morey-chaisemartin.com> References: <4A3392C4.1090608@morey-chaisemartin.com> Message-ID: <20090618134800.GE6163@me> On 13:51 Sat 13 Jun , Nicolas Morey-Chaisemartin wrote: > This patch introduce many updates in the Fat-tree discussed here: > http://www.mail-archive.com/general at lists.openfabrics.org/msg22416.html > > Major changes are: > - Ensure minimum hop path is used > - Balance secondary routes (necessary as a secondary route may become primary if shorter) > - Optimizing balancing (quality and computation time) > > Squashed commits: > Added balanced minhop within Fat-Tree > Added qsort for choosing secondary routes order instead of simple index > Enhanced qsort compareason function so it looks at the next switch level > Enhanced qsort performances for 2nd level access > Removed copy of port group arrays to save some time > Replaced qsort by bubble sort > Inlining compare and sort functions > Updated bubble sort to allow inline compareason functions > Enhanced sorting and reduced necessary compareasons > Fixed increment to earn one useless loop when we are on the main path > > Signed-off-by: Nicolas Morey-Chaisemartin Applied. Thanks. Please use osm_indent to format the code. Sasha From sashak at voltaire.com Thu Jun 18 06:51:24 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 16:51:24 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_lash.c: Remove some unused code In-Reply-To: <20090618130432.GA23147@comcast.net> References: <20090618130432.GA23147@comcast.net> Message-ID: <20090618135124.GF6163@me> On 09:04 Thu 18 Jun , Hal Rosenstock wrote: > > Signed-off-by: Robert Pearson > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From jean-vincent.ficet at bull.net Thu Jun 18 06:54:07 2009 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Thu, 18 Jun 2009 15:54:07 +0200 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090618132725.GD6163@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> Message-ID: <4A3A46FF.3060001@bull.net> Hi Sasha, >> We worked on a scheme that allows to dynamically load external routing >> engines. >> > > Hmm, and why is it really needed? Why to not include routing engine code > into OpenSM? > The idea is to include non-open source routing algorithms into opensm on demand, which is permitted by the BSD license. It also enables to try a new algorithm on the fly without having to reinstall opensm, which can prove useful on a running cluster. >> The plugin system is very similar to the perfmgr plugin >> implementation. >> > Once it was perfmgr event plugin, now plugin API is generic so what is > the reason to make yet another plugin interface? > The event-based perfmgr plugin does not make it easy to integrate a new routing algorithm because the list of available routing algorithms is statically declared. >> It allows to provide customised vendor routing engines >> and algorithms whithout having to recompile opensm: >> > > Recompiling OpenSM should not be a big deal. Assuming that it still be > an open source project. > AFAIK, the latest Voltaire UFM also includes or will include proprietary routing algorithms (mesh + fat tree), so it could also be useful for Voltaire. Cheers, Vincent From sashak at voltaire.com Thu Jun 18 06:58:55 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 16:58:55 +0300 Subject: [ofa-general] Re: [PATCH v2] osm_ucast_ftree.c Allow horizontal links between switches of max rank In-Reply-To: <4A37997E.8080708@Sun.COM> References: <4A37997E.8080708@Sun.COM> Message-ID: <20090618135855.GG6163@me> Hi Line, On 15:09 Tue 16 Jun , Line.Holen at Sun.COM wrote: > This patch makes it legal to have cross links (horizontal links) between > switches at max rank. These switches do have same rank, so hop count cannot > be calculated based on rank anymore. > The horizontal links are treated as downlinks. Switch A has a downlink to B > while B has a downlink to A. Tests on lids and also number of hops makes sure > that we don't loop back and forth across the link. > > Signed-off-by: Frank Olaf Sem-Jacobsen > Signed-off-by: Line Holen This patch is conflicting with Nicolas's one (and his was first). Could you please rebase against the current master and resend. Thanks. Sasha From hnrose at comcast.net Thu Jun 18 07:14:30 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 18 Jun 2009 10:14:30 -0400 Subject: [ofa-general] [PATCH] opensm/lash: Set minimum VL for LASH to use Message-ID: <20090618141430.GA11942@comcast.net> rather than starting from VL 0 Signed-off-by: Robert Pearson Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 59a32ad..da8cc5e 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -3,6 +3,7 @@ * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -221,6 +222,7 @@ typedef struct osm_subn_opt { char *prefix_routes_file; boolean_t consolidate_ipv6_snm_req; struct osm_subn_opt *file_opts; /* used for update */ + uint8_t lash_start_vl; /* starting vl to use in lash */ } osm_subn_opt_t; /* * FIELDS diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index 66d2fe6..e8801c9 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -1,4 +1,4 @@ -.TH OPENSM 8 "April 22, 2009" "OpenIB" "OpenIB Management" +.TH OPENSM 8 "May 28, 2009" "OpenIB" "OpenIB Management" .SH NAME opensm \- InfiniBand subnet manager and administration (SM/SA) @@ -15,6 +15,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA) [\-r(eassign_lids)] [\-R | \-\-routing_engine ] [\-\-do_mesh_analysis] +[\-\-lash_start_vl ] [\-A | \-\-ucast_cache] [\-z | \-\-connect_roots] [\-M | \-\-lid_matrix_file ] @@ -147,6 +148,10 @@ This option enables additional analysis for the lash routing engine to precondition switch port assignments in regular cartesian meshes which may reduce the number of SLs required to give a deadlock free routing. .TP +\fB\-\-lash_start_vl\fR +This option sets the starting VL to use for the lash routing algorithm. +Defaults to 0. +.TP \fB\-A\fR, \fB\-\-ucast_cache\fR This option enables unicast routing cache and prevents routing recalculation (which is a heavy task in a large cluster) when diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 296d5d5..d682ff5 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -3,6 +3,7 @@ * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2009 HNR Consulting. All rights reserved. + * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -181,6 +182,9 @@ static void show_usage(void) " routing engine to precondition switch port assignments\n" " in regular cartesian meshes which may reduce the number\n" " of SLs required to give a deadlock free routing\n\n"); + printf("--lash_start_vl \n" + " Sets the starting VL to use for the lash routing algorithm.\n" + " Defaults to 0.\n"); printf("--connect_roots, -z\n" " This option enforces a routing engine (currently\n" " up/down only) to make connectivity between root switches\n" @@ -601,6 +605,7 @@ int main(int argc, char *argv[]) {"prefix_routes_file", 1, NULL, 3}, {"consolidate_ipv6_snm_req", 0, NULL, 4}, {"do_mesh_analysis", 0, NULL, 5}, + {"lash_start_vl", 1, NULL, 6}, {NULL, 0, NULL, 0} /* Required at the end of the array */ }; @@ -951,6 +956,15 @@ int main(int argc, char *argv[]) case 5: opt.do_mesh_analysis = TRUE; break; + case 6: + temp = strtol(optarg, NULL, 0); + if (temp < 0 || temp >= IB_MAX_NUM_VLS) { + fprintf(stderr, + "ERROR: starting lash vl must be between 0 and 15\n"); + return (-1); + } + opt.lash_start_vl = (uint8_t) temp; + break; case 'h': case '?': case ':': diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index ec15f8a..fda2eb0 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -3,6 +3,7 @@ * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -389,6 +390,7 @@ static const opt_rec_t opt_tbl[] = { { "no_clients_rereg", OPT_OFFSET(no_clients_rereg), opts_parse_boolean, NULL, 1 }, { "prefix_routes_file", OPT_OFFSET(prefix_routes_file), opts_parse_charp, NULL, 0 }, { "consolidate_ipv6_snm_req", OPT_OFFSET(consolidate_ipv6_snm_req), opts_parse_boolean, NULL, 1 }, + { "lash_start_vl", OPT_OFFSET(lash_start_vl), opts_parse_uint8, NULL, 1 }, {0} }; @@ -749,6 +751,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * const p_opt) p_opt->no_clients_rereg = FALSE; p_opt->prefix_routes_file = strdup(OSM_DEFAULT_PREFIX_ROUTES_FILE); p_opt->consolidate_ipv6_snm_req = FALSE; + p_opt->lash_start_vl = 0; subn_init_qos_options(&p_opt->qos_options, NULL); subn_init_qos_options(&p_opt->qos_ca_options, NULL); subn_init_qos_options(&p_opt->qos_sw0_options, NULL); @@ -1432,6 +1435,11 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t *const p_opts) p_opts->do_mesh_analysis ? "TRUE" : "FALSE"); fprintf(out, + "# Starting VL for LASH algorithm\n" + "lash_start_vl %d\n\n", + p_opts->lash_start_vl); + + fprintf(out, "# SA database file name\nsa_db_file %s\n\n", p_opts->sa_db_file ? p_opts->sa_db_file : null_str); diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c index 12b5e34..540214c 100644 --- a/opensm/opensm/osm_ucast_lash.c +++ b/opensm/opensm/osm_ucast_lash.c @@ -478,7 +478,7 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) cdg_vertex_t ****cdg_vertex_matrix = p_lash->cdg_vertex_matrix; int *num_mst_in_lane = p_lash->num_mst_in_lane; int ***virtual_location = p_lash->virtual_location; - int min_filled_lane, max_filled_lane, trials; + int min_filled_lane, max_filled_lane, trials, max_vl; int old_min_filled_lane, old_max_filled_lane, new_num_min_lane, new_num_max_lane; unsigned int i, j; @@ -486,9 +486,13 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) int next_switch2, output_link2; int stop = 0, cycle_found; int cycle_found2; + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; - max_filled_lane = 0; - min_filled_lane = lanes_needed - 1; + max_filled_lane = opt->lash_start_vl; + max_vl = lanes_needed + opt->lash_start_vl; + if (max_vl > IB_MAX_NUM_VLS) + max_vl = IB_MAX_NUM_VLS; + min_filled_lane = max_vl - 1; trials = num_mst_in_lane[max_filled_lane]; if (lanes_needed == 1) @@ -590,7 +594,7 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) new_num_min_lane = MAX_INT; new_num_max_lane = 0; - for (i = 0; i < lanes_needed; i++) { + for (i = opt->lash_start_vl; i < max_vl; i++) { if (num_mst_in_lane[i] < new_num_min_lane) { new_num_min_lane = num_mst_in_lane[i]; @@ -673,12 +677,18 @@ static void free_lash_structures(lash_t * p_lash) { unsigned int i, j, k; unsigned num_switches = p_lash->num_switches; + int max_vl; osm_log_t *p_log = &p_lash->p_osm->log; + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; OSM_LOG_ENTER(p_log); + max_vl = opt->lash_start_vl + p_lash->vl_min; + if (max_vl > IB_MAX_NUM_VLS) + max_vl = IB_MAX_NUM_VLS; + /* free cdg_vertex_matrix */ - for (i = 0; i < p_lash->vl_min; i++) { + for (i = opt->lash_start_vl; i < max_vl; i++) { for (j = 0; j < num_switches; j++) { for (k = 0; k < num_switches; k++) if (p_lash->cdg_vertex_matrix[i][j][k]) @@ -715,13 +725,19 @@ static int init_lash_structures(lash_t * p_lash) osm_log_t *p_log = &p_lash->p_osm->log; int status = 0; unsigned int i, j, k; + int max_vl; + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; OSM_LOG_ENTER(p_log); + max_vl = vl_min + opt->lash_start_vl; + if (max_vl > IB_MAX_NUM_VLS) + max_vl = IB_MAX_NUM_VLS; + /* initialise cdg_vertex_matrix[num_switches][num_switches][num_switches] */ p_lash->cdg_vertex_matrix = - (cdg_vertex_t ****) malloc(vl_min * sizeof(cdg_vertex_t ****)); - for (i = 0; i < vl_min; i++) { + (cdg_vertex_t ****) calloc(max_vl, sizeof(cdg_vertex_t ****)); + for (i = opt->lash_start_vl; i < max_vl; i++) { p_lash->cdg_vertex_matrix[i] = (cdg_vertex_t ***) malloc(num_switches * sizeof(cdg_vertex_t ***)); @@ -730,7 +746,7 @@ static int init_lash_structures(lash_t * p_lash) goto Exit_Mem_Error; } - for (i = 0; i < vl_min; i++) { + for (i = opt->lash_start_vl; i < max_vl; i++) { for (j = 0; j < num_switches; j++) { p_lash->cdg_vertex_matrix[i][j] = (cdg_vertex_t **) malloc(num_switches * @@ -763,7 +779,7 @@ static int init_lash_structures(lash_t * p_lash) for (i = 0; i < num_switches; i++) { for (j = 0; j < num_switches; j++) { p_lash->virtual_location[i][j] = - (int *)malloc(vl_min * sizeof(int *)); + (int *)calloc(max_vl, sizeof(int *)); if (p_lash->virtual_location[i][j] == NULL) goto Exit_Mem_Error; for (k = 0; k < vl_min; k++) { @@ -804,10 +820,12 @@ static int lash_core(lash_t * p_lash) int cycle_found2 = 0; int status = 0; int *switch_bitmap = NULL; /* Bitmap to check if we have processed this pair */ + int max_vl; + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; OSM_LOG_ENTER(p_log); - if (p_lash->p_osm->subn.opt.do_mesh_analysis && osm_do_mesh_analysis(p_lash)) { + if (opt->do_mesh_analysis && osm_do_mesh_analysis(p_lash)) { OSM_LOG(p_log, OSM_LOG_ERROR, "Mesh analysis failed\n"); goto Exit; } @@ -838,11 +856,14 @@ static int lash_core(lash_t * p_lash) } for (i = 0; i < num_switches; i++) { - for (dest_switch = 0; dest_switch < num_switches; dest_switch++) + for (dest_switch = 0; dest_switch < num_switches; dest_switch++) { + max_vl = lanes_needed + opt->lash_start_vl; + if (max_vl > IB_MAX_NUM_VLS) + max_vl = IB_MAX_NUM_VLS; if (dest_switch != i && switch_bitmap[i * num_switches + dest_switch] == 0) { - v_lane = 0; + v_lane = opt->lash_start_vl; stop = 0; - while (v_lane < lanes_needed && stop == 0) { + while (v_lane < max_vl && stop == 0) { generate_cdg_for_sp(p_lash, i, dest_switch, v_lane); generate_cdg_for_sp(p_lash, dest_switch, i, v_lane); @@ -906,7 +927,9 @@ static int lash_core(lash_t * p_lash) switches[dest_switch]->routing_table[i].lane = v_lane; if (cycle_found == 1 || cycle_found2 == 1) { - if (++lanes_needed > p_lash->vl_min) + lanes_needed++; + if (lanes_needed > p_lash->vl_min || + opt->lash_start_vl + lanes_needed - 1 >= IB_DROP_VL) goto Error_Not_Enough_Lanes; generate_cdg_for_sp(p_lash, i, dest_switch, v_lane); @@ -926,19 +949,24 @@ static int lash_core(lash_t * p_lash) switch_bitmap[i * num_switches + dest_switch] = 1; switch_bitmap[dest_switch * num_switches + i] = 1; } + } } OSM_LOG(p_log, OSM_LOG_INFO, "Lanes needed: %d, Balancing\n", lanes_needed); - for (i = 0; i < lanes_needed; i++) { + max_vl = lanes_needed + opt->lash_start_vl; + if (max_vl > IB_MAX_NUM_VLS) + max_vl = IB_MAX_NUM_VLS; + + for (i = opt->lash_start_vl; i < max_vl; i++) { OSM_LOG(p_log, OSM_LOG_INFO, "Lanes in layer %d: %d\n", i, p_lash->num_mst_in_lane[i]); } balance_virtual_lanes(p_lash, lanes_needed); - for (i = 0; i < lanes_needed; i++) { + for (i = opt->lash_start_vl; i < max_vl; i++) { OSM_LOG(p_log, OSM_LOG_INFO, "Lanes in layer %d: %d\n", i, p_lash->num_mst_in_lane[i]); } @@ -1271,6 +1299,7 @@ uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, const osm_port_t * p_src_port, unsigned dst_id; unsigned src_id; osm_switch_t *p_sw; + osm_subn_opt_t *opt = &p_osm->subn.opt; if (p_osm->routing_engine_used != OSM_ROUTING_ENGINE_TYPE_LASH) return OSM_DEFAULT_SL; @@ -1286,7 +1315,7 @@ uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, const osm_port_t * p_src_port, src_id = get_lash_id(p_sw); if (src_id == dst_id) - return OSM_DEFAULT_SL; + return opt->lash_start_vl; return (uint8_t) ((switch_t *) p_sw->priv)->routing_table[dst_id].lane; } From sashak at voltaire.com Thu Jun 18 07:36:29 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 17:36:29 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <4A3A46FF.3060001@bull.net> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> Message-ID: <20090618143629.GH6163@me> On 15:54 Thu 18 Jun , Vincent Ficet wrote: > The idea is to include non-open source routing algorithms into opensm on > demand, which is permitted by the BSD license. It is permitted, but I don't think that we as open source community need to support such efforts. I'm not a lawyer and cannot say for sure, but maybe in the case of dual GPL/BSD license choice you will need anyway to relicense and to redistribute OpenSM for doing this. > It also enables to try a new algorithm on the fly without having to > reinstall opensm, which can prove useful on a running cluster. Even with your patches you will need to restart OpenSM, so it should not be a big deal to build and run OpenSM even on a running cluster especially if you are going to experiment with new routing engines. Alternatively you can load routing tables from file. > The event-based perfmgr plugin does not make it easy to integrate a new > routing algorithm because the list of available routing algorithms is > statically declared. Eventually it is only link list with names and callbacks. It should be easy to add new entry there. > AFAIK, the latest Voltaire UFM also includes or will include proprietary > routing algorithms (mesh + fat tree), so it could also be useful for > Voltaire. I don't know about this, but even if it is so. It is not our charter as open source project to care about such practices and they (UFM) will need to support their stuff by themselves. Sasha From rdreier at cisco.com Thu Jun 18 08:59:45 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 18 Jun 2009 08:59:45 -0700 Subject: [ofa-general] Re: [PATCH 0/3 RESEND] V2 - libmthca libmlx4 - Optimize memory allocation of QP buffers In-Reply-To: <20090618105051.34c40d7f@frecb007965> (sebastien dugue's message of "Thu, 18 Jun 2009 10:50:51 +0200") References: <20090618105051.34c40d7f@frecb007965> Message-ID: Sorry I haven't applied this... I will get to it soon, just wanted to get libmlx4 1.0.1 out first, since I felt this was a little too risky right before a release. - R. From rdreier at cisco.com Thu Jun 18 09:04:21 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 18 Jun 2009 09:04:21 -0700 Subject: [ofa-general] Re: [ewg] Re: [PATCH 2/9] ib_core: kernel API for GID --> MAC translations In-Reply-To: <20090618084606.GA24512@mtls03> (Eli Cohen's message of "Thu, 18 Jun 2009 11:46:06 +0300") References: <20090615073431.GC27543@mtls03> <20090618084606.GA24512@mtls03> Message-ID: > on top of which trees would like me to rebase the patches on and update > the ABI version? I guess you could base it on my xrc branch but as I said I'm not planning on merging this until I'm done with the XRC stuff. From yosefe at voltaire.com Thu Jun 18 09:13:50 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Thu, 18 Jun 2009 19:13:50 +0300 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM In-Reply-To: <20090618001040.GI13049@sgi.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> <4A313AAE.2070108@voltaire.com> <20090618001040.GI13049@sgi.com> Message-ID: <4A3A67BE.8030808@voltaire.com> akepner at sgi.com wrote: > On Thu, Jun 11, 2009 at 08:11:10PM +0300, Yossi Etigin wrote: >> .... >> How about checking the refcnt in the neigh_cleanup function, and if it's >> not 1 give up the ipoib cleanup? >> > > ipoib_neigh_cleanup() can be invoked even when there are > 1 references > (e.g., if the interface is brought down that can happen). > > So using the refcnt in the neighbour isn't going to work. > > I'm going to send a very simple patch in just a minute. > If the interface is brought down, isn't ipoib cleaning the neighbours anyway? (ipoib_flush_paths) From yosefe at voltaire.com Thu Jun 18 09:54:22 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Thu, 18 Jun 2009 19:54:22 +0300 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in CM In-Reply-To: <20090618004422.GJ13049@sgi.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> <4A313AAE.2070108@voltaire.com> <20090618001040.GI13049@sgi.com> <20090618004422.GJ13049@sgi.com> Message-ID: <4A3A713E.6090109@voltaire.com> akepner at sgi.com wrote: > @@ -841,10 +841,20 @@ static void ipoib_set_mcast_list(struct net_device *dev) > static void ipoib_neigh_cleanup(struct neighbour *n) > { > struct ipoib_neigh *neigh; > - struct ipoib_dev_priv *priv = netdev_priv(n->dev); > + struct ipoib_dev_priv *priv; > unsigned long flags; > struct ipoib_ah *ah = NULL; > > + /* > + * Note that the read of the neigh pointer below is not protected > + * by a ipoib_dev_priv->lock (since we don't yet know which device's > + * lock to use). Count on the fact that if ipoib_neigh_free() has > + * already freed the struct ipoib_neigh, to_ipoib_neigh() will > + * return NULL. > + * > + * If to_ipoib_neigh() does not return NULL, we'll re-read neigh > + * under the appropriate lock below. > + */ > neigh = *to_ipoib_neigh(n); > if (neigh) > priv = netdev_priv(neigh->dev); What if someone frees the neighbour right after you assign it to 'neigh'? 'neigh->dev' may become invalid, and so is the 'priv' and 'priv->spinlock'. --Yossi From sean.hefty at intel.com Thu Jun 18 10:23:31 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 18 Jun 2009 10:23:31 -0700 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090618143629.GH6163@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> Message-ID: <1D2A1F3FAFCF4DE38A8A2F1D78D3A5BB@amr.corp.intel.com> >> The idea is to include non-open source routing algorithms into opensm on >> demand, which is permitted by the BSD license. > >It is permitted, but I don't think that we as open source community >need to support such efforts. I agree with this. This sets a precedence of opening up the source code to all sorts of changes that become difficult to test and maintain. Anyone is free to take opensm, integrate their own changes, and release separately, but the burden of maintaining those changes should not rest on the open source community at large. - Sean From akepner at sgi.com Thu Jun 18 11:04:17 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Thu, 18 Jun 2009 11:04:17 -0700 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in IPoIB-CM In-Reply-To: <4A3A67BE.8030808@voltaire.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> <4A313AAE.2070108@voltaire.com> <20090618001040.GI13049@sgi.com> <4A3A67BE.8030808@voltaire.com> Message-ID: <20090618180417.GB6597@sgi.com> On Thu, Jun 18, 2009 at 07:13:50PM +0300, Yossi Etigin wrote: > ... > If the interface is brought down, isn't ipoib cleaning the > neighbours anyway? (ipoib_flush_paths) Yes, but the point is that you can't rely on the neighbour refcnt to indicate when the ipoib_neigh structure should be cleaned up. -- Arthur From akepner at sgi.com Thu Jun 18 11:48:38 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Thu, 18 Jun 2009 11:48:38 -0700 Subject: [ofa-general] [PATCH] ipoib: racing uses of ipoib_neigh in CM In-Reply-To: <4A3A713E.6090109@voltaire.com> References: <20090610190728.GR13781@sgi.com> <4A300EFD.3050303@voltaire.com> <20090611163322.GB32601@sgi.com> <4A313AAE.2070108@voltaire.com> <20090618001040.GI13049@sgi.com> <20090618004422.GJ13049@sgi.com> <4A3A713E.6090109@voltaire.com> Message-ID: <20090618184838.GC6597@sgi.com> On Thu, Jun 18, 2009 at 07:54:22PM +0300, Yossi Etigin wrote: > akepner at sgi.com wrote: > > @@ -841,10 +841,20 @@ static void ipoib_set_mcast_list(struct net_device *dev) > > static void ipoib_neigh_cleanup(struct neighbour *n) > > { > > struct ipoib_neigh *neigh; > > - struct ipoib_dev_priv *priv = netdev_priv(n->dev); > > + struct ipoib_dev_priv *priv; > > unsigned long flags; > > struct ipoib_ah *ah = NULL; > > > > + /* > > + * Note that the read of the neigh pointer below is not protected > > + * by a ipoib_dev_priv->lock (since we don't yet know which device's > > + * lock to use). Count on the fact that if ipoib_neigh_free() has > > + * already freed the struct ipoib_neigh, to_ipoib_neigh() will > > + * return NULL. > > + * > > + * If to_ipoib_neigh() does not return NULL, we'll re-read neigh > > + * under the appropriate lock below. > > + */ > > neigh = *to_ipoib_neigh(n); > > if (neigh) > > priv = netdev_priv(neigh->dev); > > What if someone frees the neighbour right after you assign it to 'neigh'? > 'neigh->dev' may become invalid, and so is the 'priv' and 'priv->spinlock'. > There remains a window where that can happen, this patch just makes that window (quite a bit) smaller. The best idea I've got for entirely preventing the race was suggested here: http://lists.openfabrics.org/pipermail/general/2009-June/059841.html (Seems like alot of complexity to add for this, though....) Suggestions? -- Arthur From sashak at voltaire.com Thu Jun 18 12:12:59 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 18 Jun 2009 22:12:59 +0300 Subject: [ofa-general] [PATCH] opensm: sweep component processors return status value Message-ID: <20090618191259.GI6163@me> All sweep component processors (aka managers) will return its status value instead of OSM_SIGNAL_DONE|OSM_SIGNAL_DONE_PENDING status - waiting for outstanding MAD responses are automated now in wait_for_pending_transaction() function. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_base.h | 4 -- opensm/include/opensm/osm_lid_mgr.h | 18 ++------ opensm/opensm/osm_lid_mgr.c | 77 +++++++++++++---------------------- opensm/opensm/osm_link_mgr.c | 56 ++++++++++++------------- opensm/opensm/osm_mcast_mgr.c | 36 ++++++---------- opensm/opensm/osm_pkey_mgr.c | 74 ++++++++++++++++------------------ opensm/opensm/osm_qos.c | 33 ++++++++------- opensm/opensm/osm_state_mgr.c | 10 ++-- 8 files changed, 130 insertions(+), 178 deletions(-) diff --git a/opensm/include/opensm/osm_base.h b/opensm/include/opensm/osm_base.h index bca1133..03eed79 100644 --- a/opensm/include/opensm/osm_base.h +++ b/opensm/include/opensm/osm_base.h @@ -793,10 +793,6 @@ typedef enum _osm_thread_state { #define OSM_SIGNAL_PERFMGR_SWEEP 3 #define OSM_SIGNAL_MAX 3 -/* status values for sweep managers - can be removed later */ -#define OSM_SIGNAL_DONE 16 -#define OSM_SIGNAL_DONE_PENDING 17 - typedef unsigned int osm_signal_t; /***********/ diff --git a/opensm/include/opensm/osm_lid_mgr.h b/opensm/include/opensm/osm_lid_mgr.h index d6d1ab8..7e41c0c 100644 --- a/opensm/include/opensm/osm_lid_mgr.h +++ b/opensm/include/opensm/osm_lid_mgr.h @@ -96,7 +96,6 @@ typedef struct osm_lid_mgr { osm_db_t *p_db; osm_log_t *p_log; cl_plock_t *p_lock; - boolean_t send_set_reqs; osm_db_domain_t *p_g2l; cl_qlist_t free_ranges; uint8_t used_lids[IB_LID_UCAST_END_HO + 1]; @@ -118,9 +117,6 @@ typedef struct osm_lid_mgr { * p_lock * Pointer to the serializing lock. * -* send_set_reqs -* Boolean to indicate whether any set requests sent. -* * p_g2l * Pointer to the database domain storing guid to lid mapping. * @@ -239,17 +235,14 @@ osm_lid_mgr_init(IN osm_lid_mgr_t * const p_mgr, IN struct osm_sm * sm); * * SYNOPSIS */ -osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr); +int osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr); /* * PARAMETERS * p_mgr * [in] Pointer to an osm_lid_mgr_t object. * * RETURN VALUES -* Returns the appropriate signal to the caller: -* OSM_SIGNAL_DONE - operation is complete -* OSM_SIGNAL_DONE_PENDING - local operations are complete, but -* transactions are still pending on the wire. +* Returns 0 on success and non-zero value otherwise. * * NOTES * @@ -267,17 +260,14 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr); * * SYNOPSIS */ -osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr); +int osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr); /* * PARAMETERS * p_mgr * [in] Pointer to an osm_lid_mgr_t object. * * RETURN VALUES -* Returns the appropriate signal to the caller: -* OSM_SIGNAL_DONE - operation is complete -* OSM_SIGNAL_DONE_PENDING - local operations are complete, but -* transactions are still pending on the wire. +* Returns 0 on success and non-zero value otherwise. * * NOTES * diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index e1d5106..34625ba 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -821,10 +821,10 @@ static void lid_mgr_set_remote_pi_state_to_init(IN osm_lid_mgr_t * p_mgr, /********************************************************************** **********************************************************************/ -static boolean_t lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, - IN osm_port_t * p_port, - IN osm_physp_t * p_physp, - IN ib_net16_t const lid) +static int lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, + IN osm_port_t * p_port, + IN osm_physp_t * p_physp, + IN ib_net16_t const lid) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_port_info_t *p_pi = (ib_port_info_t *) payload; @@ -836,6 +836,7 @@ static boolean_t lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, uint8_t op_vls; uint8_t port_num; boolean_t send_set = FALSE; + int ret = 0; OSM_LOG_ENTER(p_mgr->p_log); @@ -1068,31 +1069,31 @@ static boolean_t lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, if (p_mgr->p_subn->first_time_master_sweep == TRUE) send_set = TRUE; - if (send_set) { - p_mgr->send_set_reqs = TRUE; - status = osm_req_set(p_mgr->sm, - osm_physp_get_dr_path_ptr(p_physp), - payload, sizeof(payload), - IB_MAD_ATTR_PORT_INFO, - cl_hton32(osm_physp_get_port_num(p_physp)), - CL_DISP_MSGID_NONE, &context); - } + if (!send_set) + goto Exit; + + status = osm_req_set(p_mgr->sm, osm_physp_get_dr_path_ptr(p_physp), + payload, sizeof(payload), IB_MAD_ATTR_PORT_INFO, + cl_hton32(osm_physp_get_port_num(p_physp)), + CL_DISP_MSGID_NONE, &context); + if (status != IB_SUCCESS) + ret = -1; Exit: OSM_LOG_EXIT(p_mgr->p_log); - return send_set; + return ret; } /********************************************************************** Processes our own node Lock must already be held. **********************************************************************/ -static boolean_t lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) +static int lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) { osm_port_t *p_port; uint16_t min_lid_ho; uint16_t max_lid_ho; - boolean_t res = TRUE; + int ret; OSM_LOG_ENTER(p_mgr->p_log); @@ -1105,7 +1106,7 @@ static boolean_t lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0308: " "Can't acquire SM's port object, GUID 0x%016" PRIx64 "\n", cl_ntoh64(p_mgr->p_subn->sm_port_guid)); - res = FALSE; + ret = -1; goto Exit; } @@ -1134,19 +1135,19 @@ static boolean_t lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) /* Set the PortInfo the Physical Port associated with this Port. */ - lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, - cl_hton16(min_lid_ho)); + ret = lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, + cl_hton16(min_lid_ho)); Exit: OSM_LOG_EXIT(p_mgr->p_log); - return res; + return ret; } /********************************************************************** **********************************************************************/ -osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) +int osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) { - osm_signal_t signal = OSM_SIGNAL_DONE_PENDING; + int ret; OSM_LOG_ENTER(p_mgr->p_log); @@ -1158,21 +1159,12 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) persistent db */ lid_mgr_init_sweep(p_mgr); - /* Set the send_set_reqs of the p_mgr to FALSE, and - we'll see if any set requests were sent. If not - - can signal OSM_SIGNAL_DONE */ - p_mgr->send_set_reqs = FALSE; - if (lid_mgr_process_our_sm_node(p_mgr) == FALSE) - /* The initialization failed */ - signal = OSM_SIGNAL_DONE; - - if (p_mgr->send_set_reqs == FALSE) - signal = OSM_SIGNAL_DONE; + ret = lid_mgr_process_our_sm_node(p_mgr); CL_PLOCK_RELEASE(p_mgr->p_lock); OSM_LOG_EXIT(p_mgr->p_log); - return (signal); + return ret; } /********************************************************************** @@ -1181,14 +1173,13 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) 1.2 if a change is required send the port info 2 if any change send the signal PENDING... **********************************************************************/ -osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) +int osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) { - osm_signal_t signal; cl_qmap_t *p_port_guid_tbl; osm_port_t *p_port; ib_net64_t port_guid; + int lid_changed, ret = 0; uint16_t min_lid_ho, max_lid_ho; - int lid_changed; CL_ASSERT(p_mgr); @@ -1198,11 +1189,6 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) CL_ASSERT(p_mgr->p_subn->sm_port_guid); - /* Set the send_set_reqs of the p_mgr to FALSE, and - we'll see if any set requests were sent. If not - - can signal OSM_SIGNAL_DONE */ - p_mgr->send_set_reqs = FALSE; - p_port_guid_tbl = &p_mgr->p_subn->port_guid_tbl; for (p_port = (osm_port_t *) cl_qmap_head(p_port_guid_tbl); @@ -1244,19 +1230,14 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) /* the proc returns the fact it sent a set port info */ if (lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, cl_hton16(min_lid_ho))) - p_mgr->send_set_reqs = TRUE; + ret = -1; } /* all ports */ /* store the guid to lid table in persistent db */ osm_db_store(p_mgr->p_g2l); - if (p_mgr->send_set_reqs == FALSE) - signal = OSM_SIGNAL_DONE; - else - signal = OSM_SIGNAL_DONE_PENDING; - CL_PLOCK_RELEASE(p_mgr->p_lock); OSM_LOG_EXIT(p_mgr->p_log); - return (signal); + return ret; } diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c index d189471..285096c 100644 --- a/opensm/opensm/osm_link_mgr.c +++ b/opensm/opensm/osm_link_mgr.c @@ -93,8 +93,8 @@ static uint8_t link_mgr_get_smsl(IN osm_sm_t * sm, IN osm_physp_t * p_physp) /********************************************************************** **********************************************************************/ -static boolean_t link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, - IN uint8_t port_state) +static int link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, + IN uint8_t port_state) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_port_info_t *p_pi = (ib_port_info_t *) payload; @@ -102,13 +102,10 @@ static boolean_t link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, osm_madw_context_t context; osm_node_t *p_node; ib_api_status_t status; - uint8_t port_num; - uint8_t mtu; - uint8_t smsl = OSM_DEFAULT_SL; - uint8_t op_vls; - boolean_t esp0 = FALSE; - boolean_t send_set = FALSE; + uint8_t port_num, mtu, op_vls, smsl = OSM_DEFAULT_SL; + boolean_t esp0 = FALSE, send_set = FALSE; osm_physp_t *p_remote_physp; + int ret = 0; OSM_LOG_ENTER(sm->p_log); @@ -391,28 +388,29 @@ Send: && sm->p_subn->first_time_master_sweep == TRUE) send_set = TRUE; - if (send_set) - status = osm_req_set(sm, osm_physp_get_dr_path_ptr(p_physp), - payload, sizeof(payload), - IB_MAD_ATTR_PORT_INFO, - cl_hton32(port_num), - CL_DISP_MSGID_NONE, &context); + if (!send_set) + goto Exit; + + status = osm_req_set(sm, osm_physp_get_dr_path_ptr(p_physp), + payload, sizeof(payload), IB_MAD_ATTR_PORT_INFO, + cl_hton32(port_num), CL_DISP_MSGID_NONE, &context); + if (status) + ret = -1; Exit: OSM_LOG_EXIT(sm->p_log); - return send_set; + return ret; } /********************************************************************** **********************************************************************/ -static osm_signal_t link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, - IN const uint8_t link_state) +static int link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, + IN const uint8_t link_state) { - uint32_t i; - uint32_t num_physp; osm_physp_t *p_physp; + uint32_t i, num_physp; + int ret = 0; uint8_t current_state; - osm_signal_t signal = OSM_SIGNAL_DONE; OSM_LOG_ENTER(sm->p_log); @@ -453,20 +451,20 @@ static osm_signal_t link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, p_physp->port_num, ib_get_port_state_str(current_state)); else if (link_mgr_set_physp_pi(sm, p_physp, link_state)) - signal = OSM_SIGNAL_DONE_PENDING; + ret = -1; } OSM_LOG_EXIT(sm->p_log); - return (signal); + return ret; } /********************************************************************** **********************************************************************/ -osm_signal_t osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) +int osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) { cl_qmap_t *p_node_guid_tbl; osm_node_t *p_node; - osm_signal_t signal = OSM_SIGNAL_DONE; + int ret = 0; OSM_LOG_ENTER(sm->p_log); @@ -476,14 +474,12 @@ osm_signal_t osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) for (p_node = (osm_node_t *) cl_qmap_head(p_node_guid_tbl); p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); - p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { - if (link_mgr_process_node(sm, p_node, link_state) == - OSM_SIGNAL_DONE_PENDING) - signal = OSM_SIGNAL_DONE_PENDING; - } + p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) + if (link_mgr_process_node(sm, p_node, link_state)) + ret = -1; CL_PLOCK_RELEASE(sm->p_lock); OSM_LOG_EXIT(sm->p_log); - return (signal); + return ret; } diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index de0a8a5..ea49588 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -321,7 +321,7 @@ static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, /********************************************************************** **********************************************************************/ -static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) +static int mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) { osm_node_t *p_node; osm_dr_path_t *p_path; @@ -333,7 +333,7 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) uint32_t max_position; osm_mcast_tbl_t *p_tbl; ib_net16_t block[IB_MCAST_BLOCK_SIZE]; - osm_signal_t signal = OSM_SIGNAL_DONE; + int ret = 0; CL_ASSERT(sm); @@ -375,10 +375,9 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A02: " "Sending multicast fwd. tbl. block failed (%s)\n", ib_get_err_str(status)); + ret = -1; } - signal = OSM_SIGNAL_DONE_PENDING; - if (++position > max_position) { position = 0; block_num++; @@ -386,7 +385,7 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) } OSM_LOG_EXIT(sm->p_log); - return signal; + return ret; } /********************************************************************** @@ -1104,15 +1103,13 @@ Exit: /********************************************************************** **********************************************************************/ -osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) +int osm_mcast_mgr_process(osm_sm_t * sm) { - osm_signal_t signal; osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; cl_qlist_t *p_list = &sm->mgrp_list; osm_mgrp_t *p_mgrp; - boolean_t pending_transactions = FALSE; - int i; + int i, ret = 0; OSM_LOG_ENTER(sm->p_log); @@ -1142,9 +1139,8 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) */ p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { - signal = mcast_mgr_set_tbl(sm, p_sw); - if (signal == OSM_SIGNAL_DONE_PENDING) - pending_transactions = TRUE; + if (mcast_mgr_set_tbl(sm, p_sw)) + ret = -1; p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item); } @@ -1157,25 +1153,22 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) OSM_LOG_EXIT(sm->p_log); - if (pending_transactions == TRUE) - return (OSM_SIGNAL_DONE_PENDING); - else - return (OSM_SIGNAL_DONE); + return ret; } /********************************************************************** This is the function that is invoked during idle time to handle the process request for mcast groups where join/leave/delete was required. **********************************************************************/ -osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) +int osm_mcast_mgr_process_mgroups(osm_sm_t * sm) { cl_qlist_t *p_list = &sm->mgrp_list; osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; osm_mgrp_t *p_mgrp; ib_net16_t mlid; - osm_signal_t ret, signal = OSM_SIGNAL_DONE; osm_mcast_mgr_ctxt_t *ctx; + int ret = 0; OSM_LOG_ENTER(sm->p_log); @@ -1219,9 +1212,8 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) p_sw_tbl = &sm->p_subn->sw_guid_tbl; p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { - ret = mcast_mgr_set_tbl(sm, p_sw); - if (ret == OSM_SIGNAL_DONE_PENDING) - signal = ret; + if (mcast_mgr_set_tbl(sm, p_sw)) + ret = -1; p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item); } @@ -1229,5 +1221,5 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) CL_PLOCK_RELEASE(sm->p_lock); OSM_LOG_EXIT(sm->p_log); - return signal; + return ret; } diff --git a/opensm/opensm/osm_pkey_mgr.c b/opensm/opensm/osm_pkey_mgr.c index 9b76ed7..6b23b27 100644 --- a/opensm/opensm/osm_pkey_mgr.c +++ b/opensm/opensm/osm_pkey_mgr.c @@ -189,7 +189,7 @@ pkey_mgr_update_pkey_entry(IN osm_sm_t * sm, /********************************************************************** **********************************************************************/ -static boolean_t +static ib_api_status_t pkey_mgr_enforce_partition(IN osm_log_t * p_log, osm_sm_t * sm, IN osm_physp_t * p_physp, IN const boolean_t enforce) { @@ -207,7 +207,7 @@ pkey_mgr_enforce_partition(IN osm_log_t * p_log, osm_sm_t * sm, cl_ntoh64(osm_node_get_node_guid (osm_physp_get_node_ptr(p_physp))), osm_physp_get_port_num(p_physp)); - return FALSE; + return IB_SUCCESS; } memset(payload, 0, IB_SMP_DATA_SIZE); @@ -233,28 +233,26 @@ pkey_mgr_enforce_partition(IN osm_log_t * p_log, osm_sm_t * sm, IB_MAD_ATTR_PORT_INFO, cl_hton32(osm_physp_get_port_num(p_physp)), CL_DISP_MSGID_NONE, &context); - if (status != IB_SUCCESS) { + if (status != IB_SUCCESS) OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0511: " "Failed to set PortInfo for " "node 0x%016" PRIx64 " port %u\n", cl_ntoh64(osm_node_get_node_guid (osm_physp_get_node_ptr(p_physp))), osm_physp_get_port_num(p_physp)); - return FALSE; - } else { + else OSM_LOG(p_log, OSM_LOG_DEBUG, "Set PortInfo for node 0x%016" PRIx64 " port %u\n", cl_ntoh64(osm_node_get_node_guid (osm_physp_get_node_ptr(p_physp))), osm_physp_get_port_num(p_physp)); - return TRUE; - } + return status; } /********************************************************************** **********************************************************************/ -static boolean_t pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, - const osm_port_t * const p_port) +static int pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, + const osm_port_t * const p_port) { osm_physp_t *p_physp; osm_node_t *p_node; @@ -267,10 +265,10 @@ static boolean_t pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, uint16_t num_of_blocks; uint16_t max_num_of_blocks; ib_api_status_t status; - boolean_t ret_val = FALSE; osm_pending_pkey_t *p_pending; boolean_t found; ib_pkey_table_t empty_block; + int ret = 0; memset(&empty_block, 0, sizeof(ib_pkey_table_t)); @@ -360,32 +358,32 @@ static boolean_t pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, status = pkey_mgr_update_pkey_entry(sm, p_physp, new_block, block_index); - if (status == IB_SUCCESS) { + if (status == IB_SUCCESS) OSM_LOG(p_log, OSM_LOG_DEBUG, "Updated pkey table block %d for node 0x%016" PRIx64 " port %u\n", block_index, cl_ntoh64(osm_node_get_node_guid(p_node)), osm_physp_get_port_num(p_physp)); - ret_val = TRUE; - } else { + else { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0506: " "pkey_mgr_update_pkey_entry() failed to update " "pkey table block %d for node 0x%016" PRIx64 " port %u\n", block_index, cl_ntoh64(osm_node_get_node_guid(p_node)), osm_physp_get_port_num(p_physp)); + ret = -1; } } - return ret_val; + return ret; } /********************************************************************** **********************************************************************/ -static boolean_t -pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, - const osm_subn_t * p_subn, - const osm_port_t * const p_port, boolean_t enforce) +static int pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, + const osm_subn_t * p_subn, + const osm_port_t * const p_port, + boolean_t enforce) { osm_physp_t *p_physp, *peer; osm_node_t *p_node; @@ -396,21 +394,20 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, uint16_t num_of_blocks; uint16_t peer_max_blocks; ib_api_status_t status = IB_SUCCESS; - boolean_t ret_val = FALSE; - boolean_t port_info_set = FALSE; ib_pkey_table_t empty_block; + int ret = 0; memset(&empty_block, 0, sizeof(ib_pkey_table_t)); p_physp = p_port->p_physp; if (!p_physp) - return FALSE; + return -1; peer = osm_physp_get_remote(p_physp); if (!peer) - return FALSE; + return -1; p_node = osm_physp_get_node_ptr(peer); if (!p_node->sw || !p_node->sw->switch_info.enforce_cap) - return FALSE; + return 0; p_pkey_tbl = osm_physp_get_pkey_tbl(p_physp); p_peer_pkey_tbl = &peer->pkeys; @@ -424,13 +421,14 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, cl_ntoh64(osm_node_get_node_guid(p_node)), osm_physp_get_port_num(peer)); enforce = FALSE; + ret = -1; } if (pkey_mgr_enforce_partition(p_log, sm, peer, enforce)) - port_info_set = TRUE; + ret = -1; if (enforce == FALSE) - return port_info_set; + return ret; p_peer_pkey_tbl->used_blocks = p_pkey_tbl->used_blocks; for (block_index = 0; block_index < p_pkey_tbl->used_blocks; @@ -443,12 +441,9 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, osm_pkey_tbl_block_get(p_peer_pkey_tbl, block_index); if (!peer_block || memcmp(peer_block, block, sizeof(*peer_block))) { - status = - pkey_mgr_update_pkey_entry(sm, peer, block, - block_index); - if (status == IB_SUCCESS) - ret_val = TRUE; - else + status = pkey_mgr_update_pkey_entry(sm, peer, block, + block_index); + if (status != IB_SUCCESS) { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0509: " "pkey_mgr_update_pkey_entry() failed to update " "pkey table block %d for node 0x%016" @@ -456,19 +451,19 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, cl_ntoh64(osm_node_get_node_guid (p_node)), osm_physp_get_port_num(peer)); + ret = -1; + } } } - if (ret_val) + if (!ret) OSM_LOG(p_log, OSM_LOG_DEBUG, "Pkey table was updated for node 0x%016" PRIx64 " port %u\n", cl_ntoh64(osm_node_get_node_guid(p_node)), osm_physp_get_port_num(peer)); - if (port_info_set) - return TRUE; - return ret_val; + return ret; } /********************************************************************** @@ -479,7 +474,7 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) cl_map_item_t *p_next; osm_prtn_t *p_prtn; osm_port_t *p_port; - osm_signal_t signal = OSM_SIGNAL_DONE; + int ret = 0; CL_ASSERT(p_osm); @@ -490,6 +485,7 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) if (osm_prtn_make_partitions(&p_osm->log, &p_osm->subn) != IB_SUCCESS) { OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 0510: " "osm_prtn_make_partitions() failed\n"); + ret = -1; goto _err; } @@ -512,17 +508,17 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) p_port = (osm_port_t *) p_next; p_next = cl_qmap_next(p_next); if (pkey_mgr_update_port(&p_osm->log, &p_osm->sm, p_port)) - signal = OSM_SIGNAL_DONE_PENDING; + ret = -1; if ((osm_node_get_type(p_port->p_node) != IB_NODE_TYPE_SWITCH) && pkey_mgr_update_peer_port(&p_osm->log, &p_osm->sm, &p_osm->subn, p_port, !p_osm->subn.opt. no_partition_enforcement)) - signal = OSM_SIGNAL_DONE_PENDING; + ret = -1; } _err: CL_PLOCK_RELEASE(&p_osm->lock); OSM_LOG_EXIT(&p_osm->log); - return (signal); + return ret; } diff --git a/opensm/opensm/osm_qos.c b/opensm/opensm/osm_qos.c index d0aa866..e3dfb58 100644 --- a/opensm/opensm/osm_qos.c +++ b/opensm/opensm/osm_qos.c @@ -222,10 +222,10 @@ static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, return IB_SUCCESS; } -static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, - osm_port_t * p_port, osm_physp_t * p, - uint8_t port_num, unsigned force_update, - const struct qos_config *qcfg) +static int qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, + osm_port_t * p_port, osm_physp_t * p, + uint8_t port_num, unsigned force_update, + const struct qos_config *qcfg) { ib_api_status_t status; @@ -241,7 +241,7 @@ static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, "failed to update VLArbitration tables " "for port %" PRIx64 " #%d\n", cl_ntoh64(p->port_guid), port_num); - return status; + return -1; } /* setup SL2VL tables */ @@ -251,13 +251,13 @@ static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, "failed to update SL2VLMapping tables " "for port %" PRIx64 " #%d\n", cl_ntoh64(p->port_guid), port_num); - return status; + return -1; } - return IB_SUCCESS; + return 0; } -osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) +int osm_qos_setup(osm_opensm_t * p_osm) { struct qos_config ca_config, sw0_config, swe_config, rtr_config; struct qos_config *cfg; @@ -267,12 +267,12 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) uint32_t num_physp; osm_physp_t *p_physp; osm_node_t *p_node; - ib_api_status_t status; unsigned force_update; + int ret = 0; uint8_t i; if (!p_osm->subn.opt.qos) - return OSM_SIGNAL_DONE; + return 0; OSM_LOG_ENTER(&p_osm->log); @@ -305,10 +305,10 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) continue; force_update = p_physp->need_update || p_osm->subn.need_update; - status = - qos_physp_setup(&p_osm->log, &p_osm->sm, + if (qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, p_physp, i, - force_update, &swe_config); + force_update, &swe_config)) + ret = -1; } /* skip base port 0 */ if (!ib_switch_info_is_enhanced_port0 @@ -326,14 +326,15 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) continue; force_update = p_physp->need_update || p_osm->subn.need_update; - status = qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, - p_physp, 0, force_update, cfg); + if (qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, p_physp, + 0, force_update, cfg)) + ret = -1; } cl_plock_release(&p_osm->lock); OSM_LOG_EXIT(&p_osm->log); - return OSM_SIGNAL_DONE; + return ret; } /* diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index a3e3609..adc39a0 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -64,11 +64,11 @@ #include extern void osm_drop_mgr_process(IN osm_sm_t * sm); -extern osm_signal_t osm_qos_setup(IN osm_opensm_t * p_osm); -extern osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm); -extern osm_signal_t osm_mcast_mgr_process(IN osm_sm_t * sm); -extern osm_signal_t osm_mcast_mgr_process_mgroups(IN osm_sm_t * sm); -extern osm_signal_t osm_link_mgr_process(IN osm_sm_t * sm, IN uint8_t state); +extern int osm_qos_setup(IN osm_opensm_t * p_osm); +extern int osm_pkey_mgr_process(IN osm_opensm_t * p_osm); +extern int osm_mcast_mgr_process(IN osm_sm_t * sm); +extern int osm_mcast_mgr_process_mgroups(IN osm_sm_t * sm); +extern int osm_link_mgr_process(IN osm_sm_t * sm, IN uint8_t state); /********************************************************************** **********************************************************************/ -- 1.6.3.1 From chu11 at llnl.gov Thu Jun 18 13:38:36 2009 From: chu11 at llnl.gov (Al Chu) Date: Thu, 18 Jun 2009 13:38:36 -0700 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090618143629.GH6163@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> Message-ID: <1245357516.31575.149.camel@auk31.llnl.gov> Hey Sasha, On Thu, 2009-06-18 at 17:36 +0300, Sasha Khapyorsky wrote: > On 15:54 Thu 18 Jun , Vincent Ficet wrote: > > The idea is to include non-open source routing algorithms into opensm on > > demand, which is permitted by the BSD license. > > It is permitted, but I don't think that we as open source community > need to support such efforts. I agree with this on principle, however, we already support a plugin system that allows users to develop proprietary perfmgr plugins. So I feel that we already are down this path. Is the primary issue that this module "imports" new behavior into opensm? I don't know the perfmgr plugin code that well, but my understanding is that you can "import new behavior" into opensm via the perfmgr plugin code too? So what is really different?? Al > I'm not a lawyer and cannot say for sure, but maybe in the case of dual > GPL/BSD license choice you will need anyway to relicense and to > redistribute OpenSM for doing this. > > > It also enables to try a new algorithm on the fly without having to > > reinstall opensm, which can prove useful on a running cluster. > > Even with your patches you will need to restart OpenSM, so it should not > be a big deal to build and run OpenSM even on a running cluster > especially if you are going to experiment with new routing engines. > Alternatively you can load routing tables from file. > > > The event-based perfmgr plugin does not make it easy to integrate a new > > routing algorithm because the list of available routing algorithms is > > statically declared. > > Eventually it is only link list with names and callbacks. It should be > easy to add new entry there. > > > AFAIK, the latest Voltaire UFM also includes or will include proprietary > > routing algorithms (mesh + fat tree), so it could also be useful for > > Voltaire. > > I don't know about this, but even if it is so. It is not our charter as > open source project to care about such practices and they (UFM) will need > to support their stuff by themselves. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > -- Albert Chu chu11 at llnl.gov Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From weiny2 at llnl.gov Thu Jun 18 14:04:48 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 18 Jun 2009 14:04:48 -0700 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <1245357516.31575.149.camel@auk31.llnl.gov> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <1245357516.31575.149.camel@auk31.llnl.gov> Message-ID: <20090618140448.d1e94167.weiny2@llnl.gov> On Thu, 18 Jun 2009 13:38:36 -0700 Al Chu wrote: > Hey Sasha, > > On Thu, 2009-06-18 at 17:36 +0300, Sasha Khapyorsky wrote: > > On 15:54 Thu 18 Jun , Vincent Ficet wrote: > > > The idea is to include non-open source routing algorithms into opensm on > > > demand, which is permitted by the BSD license. > > > > It is permitted, but I don't think that we as open source community > > need to support such efforts. > > I agree with this on principle, however, we already support a plugin > system that allows users to develop proprietary perfmgr plugins. So I > feel that we already are down this path. Also it should be noted that the "event plugin" (formerly perfmgr plugin) was added to reduce the dependencies OpenSM would require when more advanced monitoring was desired. My plugin (and others) use Open Source packages such as MySQL which could not have been added directly to OpenSM. The intent was never to facilitate proprietary extensions. However, we never thought to prevent them either. Ira > > Is the primary issue that this module "imports" new behavior into > opensm? I don't know the perfmgr plugin code that well, but my > understanding is that you can "import new behavior" into opensm via the > perfmgr plugin code too? So what is really different?? > > Al > > > I'm not a lawyer and cannot say for sure, but maybe in the case of dual > > GPL/BSD license choice you will need anyway to relicense and to > > redistribute OpenSM for doing this. > > > > > It also enables to try a new algorithm on the fly without having to > > > reinstall opensm, which can prove useful on a running cluster. > > > > Even with your patches you will need to restart OpenSM, so it should not > > be a big deal to build and run OpenSM even on a running cluster > > especially if you are going to experiment with new routing engines. > > Alternatively you can load routing tables from file. > > > > > The event-based perfmgr plugin does not make it easy to integrate a new > > > routing algorithm because the list of available routing algorithms is > > > statically declared. > > > > Eventually it is only link list with names and callbacks. It should be > > easy to add new entry there. > > > > > AFAIK, the latest Voltaire UFM also includes or will include proprietary > > > routing algorithms (mesh + fat tree), so it could also be useful for > > > Voltaire. > > > > I don't know about this, but even if it is so. It is not our charter as > > open source project to care about such practices and they (UFM) will need > > to support their stuff by themselves. > > > > Sasha > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > > > -- > Albert Chu > chu11 at llnl.gov > Computer Scientist > High Performance Systems Division > Lawrence Livermore National Laboratory > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab weiny2 at llnl.gov From nicolas.morey-chaisemartin at ext.bull.net Thu Jun 18 22:44:49 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey-Chaisemartin) Date: Fri, 19 Jun 2009 07:44:49 +0200 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090618143629.GH6163@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> Message-ID: <4A3B25D1.8030500@ext.bull.net> Hi Sasha, Le 18/06/2009 16:36, Sasha Khapyorsky a écrit : >> It also enables to try a new algorithm on the fly without having to >> reinstall opensm, which can prove useful on a running cluster. > > Even with your patches you will need to restart OpenSM, so it should not > be a big deal to build and run OpenSM even on a running cluster > especially if you are going to experiment with new routing engines. > Alternatively you can load routing tables from file. Well at least we wouldn't have to reinstall a new OpenSM (and its dependencies) each time. Problem is we are experimenting with several algorithms for different topologies and different usages and we don't necessary want to give them all to the clients at once. With this plugin API you simply need to override the old lib and restart OpenSM (I guess it can probably be made even more interactive by reloading the plugin without restarting). Most of our algorithm try to add a "dynamic" behaviour to IB routing so loading a file is definitly not a viable solution. > >> The event-based perfmgr plugin does not make it easy to integrate a new >> routing algorithm because the list of available routing algorithms is >> statically declared. > > Eventually it is only link list with names and callbacks. It should be > easy to add new entry there. > I guess it could be done but are you sure the event plugin is loaded and set up before the routing engine ? If it is, we could make the routing_engine list a bit more dynamic. However I'm not fully satisfied of the idea of inserting new engines through an "event" plugin. We could also make this interface a bit more generic and provides the means to add new features around OpenSM more easily. >> AFAIK, the latest Voltaire UFM also includes or will include proprietary >> routing algorithms (mesh + fat tree), so it could also be useful for >> Voltaire. > > I don't know about this, but even if it is so. It is not our charter as > open source project to care about such practices and they (UFM) will need > to support their stuff by themselves. > Fair enough but the cost of this plugin is quite low in terms of code quantity, so it shouldn't introduce too much trouble maintaning it. And as Ira, it reduces OpenSM dependencies for external algorithm. Some of ours for example rely on PostgreSQL drivers or libdbi, some own library (which uses libibnetdisc) and much more. Anyway, this was just a proposal as we thought it might be handful to other people than us. If you think it's not or don't agree with the code, we'll just keep it on our own git branches. Regards Nicolas From sebastien.dugue at bull.net Thu Jun 18 23:42:32 2009 From: sebastien.dugue at bull.net (sebastien dugue) Date: Fri, 19 Jun 2009 08:42:32 +0200 Subject: [ofa-general] Re: [PATCH 0/3 RESEND] V2 - libmthca libmlx4 - Optimize memory allocation of QP buffers In-Reply-To: References: <20090618105051.34c40d7f@frecb007965> Message-ID: <20090619084232.48a82fa7@frecb007965> Hi Roland, On Thu, 18 Jun 2009 08:59:45 -0700 Roland Dreier wrote: > Sorry I haven't applied this... No problem. > I will get to it soon, just wanted to > get libmlx4 1.0.1 out first, since I felt this was a little too risky > right before a release. Fair enough. Thanks for looking into this. Sebastien. From vlad at lists.openfabrics.org Fri Jun 19 02:24:04 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 19 Jun 2009 02:24:04 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090619-0200 daily build status Message-ID: <20090619092405.4D88E10208F7@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090619-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From rajouri.jammu at gmail.com Fri Jun 19 14:17:55 2009 From: rajouri.jammu at gmail.com (Rajouri Jammu) Date: Fri, 19 Jun 2009 14:17:55 -0700 Subject: [ofa-general] Compilation errors with OFED 1.4.1/ 1.4 In-Reply-To: <9759F033B56A60469F217C49AC58E2BF085A541B@MAILUK2.rms.com> References: <9759F033B56A60469F217C49AC58E2BF085A541B@MAILUK2.rms.com> Message-ID: <3307cdf90906191417y46a0142bs50f58c36860d33ab@mail.gmail.com> Thanks for the response Evan. It appears to me that this is a problem in OFED 1.4.1 since the header file in /usr/src/ofa_kernel/include/rdma/ib_verbs.h is referring to data structures that are not present in the SLES 10 u 2 kernel. However, someone else indicated it was a problem in our code. Btw, my code compiled fine with OFED 1.3.1 on the same kernel. What 'am I missing? Thanks. On Wed, Jun 17, 2009 at 10:14 AM, Evan Fraser wrote: > Hi Rajouri, > > > >I'm trying to build a kernel module using OFED 1.4.1 on SLES 10 Update 2 > >(2.6.16.60-0.21-smp) and I get the below compilation errors. > > > > OFED didn't compile for me on that kernel either. Apply the SLES updates > to upgrade your kernel etc. I found it compiles fine against kernel > 2.6.16.60-0.34-smp. > > Evan Fraser > > Senior Systems Analyst > Peninsular House, 30 Monument Street > London EC3R 8NB, United Kingdom > Tel +44 20 7444 7860 > Mobile +44 75 9024 5788 > evan.fraser at rms.com > www.rms.com > > > > > This message and any attachments contain information that may be RMS Inc. > confidential and/or privileged. If you are not the intended recipient > (or authorized to receive for the intended recipient), and have received > this message in error, any use, disclosure or distribution is strictly > prohibited. If you have received this message in error, please notify > the sender immediately by replying to the e-mail and permanently deleting > the message from your computer and/or storage system. > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adit.262 at gmail.com Fri Jun 19 15:14:01 2009 From: adit.262 at gmail.com (Adit Ranadive) Date: Fri, 19 Jun 2009 18:14:01 -0400 Subject: [ofa-general] Support for reg_phys_mr in mlx4_ib Message-ID: Is there any plan to implement the reg_phys_mr verb for the ConnectX cards in the newer OFED releases? If not whats the best option to register a list of physical pages as a memory region? Thanks, Adit ------ Adit Ranadive PhD CS Candidate Georgia Institute of Technology, Atlanta, GA -------------- next part -------------- An HTML attachment was scrubbed... URL: From arlin.r.davis at intel.com Fri Jun 19 23:22:43 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 19 Jun 2009 23:22:43 -0700 Subject: [ofa-general] [ANNOUNCE] uDAPL v2.0 - dapl-2.0.20 release Message-ID: New release for uDAPL 2.0 available on the OFA download page and in my git tree. md5sum: 49b26fb8c60f7ad010530bda8199338f dapl-2.0.20.tar.gz Summary of changes: common,scm: add debug capabilities to print in-process CM lists scm: disconnect EP before cleaning up orphaned CR's during dat_ep_free scm: private data is not handled properly via CR rejects. scm: cleanup orphaned UD CR's when destroying the EP scm: provider specific query for default UD MTU is wrong. scm: update CM code to shutdown before closing socket scm: multi-hca CM processing broken. Need cr thread wakeup mechanism per HCA. dapltest: windows scripts updated cma: add support for network devices based on index windows: missing build files for common and udapl sub-directories windows: add build files for openib_scm, remove /Wp64 build option. dtest: add connection timers on client side linux_osd: use pthread_self instead of getpid for debug messages windows_osd: new dapl_os_gettid macro to return thread id Vlad, please pull new v2 package into OFED 1.5 build and install the following: compat-dapl-1.2.14-1 compat-dapl-devel-1.2.14-1 dapl-2.0.20-1 dapl-utils-2.0.20-1 dapl-devel-2.0.20-1 dapl-debuginfo-2.0.20-1 See http://www.openfabrics.org/downloads/dapl/ more details. -arlin From vlad at lists.openfabrics.org Sat Jun 20 02:27:43 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 20 Jun 2009 02:27:43 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090620-0200 daily build status Message-ID: <20090620092743.E1F221020BF7@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090620-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From vlad at dev.mellanox.co.il Sun Jun 21 00:38:58 2009 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Sun, 21 Jun 2009 10:38:58 +0300 Subject: [ofa-general] Re: [ANNOUNCE] uDAPL v2.0 - dapl-2.0.20 release In-Reply-To: References: Message-ID: <4A3DE392.8020100@dev.mellanox.co.il> Davis, Arlin R wrote: > Vlad, please pull new v2 package into OFED 1.5 build and install the following: > > compat-dapl-1.2.14-1 > compat-dapl-devel-1.2.14-1 > dapl-2.0.20-1 > dapl-utils-2.0.20-1 > dapl-devel-2.0.20-1 > dapl-debuginfo-2.0.20-1 > > See http://www.openfabrics.org/downloads/dapl/ more details. > > -arlin > > > Done, Regards, Vladimir From monis at Voltaire.COM Sun Jun 21 07:38:11 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Sun, 21 Jun 2009 17:38:11 +0300 Subject: [ofa-general] [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections Message-ID: <4A3E45D3.3040405@Voltaire.COM> This patch was sent on Apr, 23 2009. Nothing was changed except for fixing offsets and verifying it under 2.6.30 -------------------- Create a virtual file under debugfs for each cma device and use it to print information about each rdma_id that is attached to this device. Here is an example of 'cat /sys/kernel/debug/rdma_cm/mthca0_rdma_id'. This example is for a host that runs a rping server (when a remote client is connected to it) and a rping client to a remote server. TYPE DEVICE PORT NET_DEV SRC_ADDR DST_ADDR SPACE STATE QP_NUM mthca0 0 0.0.0.0:7174 TCP LISTEN 0 IB mthca0 1 ib0 192.30.3.249:46079 192.30.3.248:7174 TCP CONNECT 132102 IB mthca0 1 ib0 192.30.3.249:7174 192.30.3.248:42561 TCP CONNECT 132103 Signed-off-by: Moni Shoua -- drivers/infiniband/core/cma.c | 183 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 851de83..746ab14 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -51,6 +51,9 @@ #include #include +#include +#include + MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("Generic RDMA CM Agent"); MODULE_LICENSE("Dual BSD/GPL"); @@ -59,6 +62,8 @@ MODULE_LICENSE("Dual BSD/GPL"); #define CMA_MAX_CM_RETRIES 15 #define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24) +static struct dentry *cma_root_dentry; + static void cma_add_one(struct ib_device *device); static void cma_remove_one(struct ib_device *device); @@ -86,6 +91,7 @@ struct cma_device { struct completion comp; atomic_t refcount; struct list_head id_list; + struct dentry *rdma_id_dentry; }; enum cma_state { @@ -102,6 +108,47 @@ enum cma_state { CMA_DESTROYING }; +static const char *format_cma_state(enum cma_state s) +{ + switch (s) { + case CMA_IDLE: return "IDLE"; + case CMA_ADDR_QUERY: return "ADDR_QUERY"; + case CMA_ADDR_RESOLVED: return "ADDR_RESOLVED"; + case CMA_ROUTE_QUERY: return "ROUTE_QUERY"; + case CMA_ROUTE_RESOLVED: return "ROUTE_RESOLVED"; + case CMA_CONNECT: return "CONNECT"; + case CMA_DISCONNECT: return "DISCONNECT"; + case CMA_ADDR_BOUND: return "ADDR_BOUND"; + case CMA_LISTEN: return "LISTEN"; + case CMA_DEVICE_REMOVAL: return "DEVICE_REMOVAL"; + case CMA_DESTROYING: return "DESTROYING"; + } + return ""; +} + +static const char *format_port_space(enum rdma_port_space ps) +{ + switch (ps) { + case RDMA_PS_SDP: return "SDP"; + case RDMA_PS_IPOIB: return "IPOIB"; + case RDMA_PS_TCP: return "TCP"; + case RDMA_PS_UDP: return "UDP"; + case RDMA_PS_SCTP: return "SCTP"; + } + return ""; +} + +static const char *format_node_type(enum rdma_node_type nt) +{ + if (nt) { + switch (rdma_node_get_transport(nt)) { + case RDMA_TRANSPORT_IB: return "IB"; + case RDMA_TRANSPORT_IWARP: return "IW"; + } + } + return ""; +} + struct rdma_bind_list { struct idr *ps; struct hlist_head owners; @@ -2867,6 +2914,131 @@ static struct notifier_block cma_nb = { .notifier_call = cma_netdev_callback }; +static void *cma_rdma_id_seq_start(struct seq_file *file, loff_t *pos) +{ + struct cma_device *cma_dev = file->private; + void *ret; + + mutex_lock(&lock); + if (*pos == 0) + return SEQ_START_TOKEN; + ret = seq_list_start_head(&cma_dev->id_list, *pos); + return ret; +} + +static void *cma_rdma_id_seq_next(struct seq_file *file, void *v, loff_t *pos) +{ + void *ret; + struct cma_device *cma_dev = file->private; + if (v == SEQ_START_TOKEN) { + ++*pos; + if (!list_empty(&cma_dev->id_list)) + ret = cma_dev->id_list.next; + else + ret = NULL; + } else { + ret = seq_list_next(v, &cma_dev->id_list, pos); + } + return ret; +} + +static void cma_rdma_id_seq_stop(struct seq_file *file, void *iter_ptr) +{ + mutex_unlock(&lock); +} + +static void format_addr(struct sockaddr *sa, char* buf) +{ + switch (sa->sa_family) { + case AF_INET: { + struct sockaddr_in *sin = (struct sockaddr_in *)sa; + sprintf(buf, "%pI4:%u", &sin->sin_addr.s_addr, + be16_to_cpu(cma_port(sa))); + break; + } + case AF_INET6: { + struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)sa; + sprintf(buf, "%pI6:%u", &sin6->sin6_addr, + be16_to_cpu(cma_port(sa))); + break; + } + default: + buf[0] = 0; + } +} + +static int cma_rdma_id_seq_show(struct seq_file *file, void *v) +{ + struct rdma_id_private *id_priv; + char local_addr[64], remote_addr[64]; + + if (!v) + return 0; + if (v == SEQ_START_TOKEN) { + seq_printf(file, + "%-5s %-8s %-5s %-8s %-52s %-52s %-6s %-15s %-8s \n", + "TYPE", "DEVICE", "PORT", "NET_DEV", "SRC_ADDR", "DST_ADDR", "SPACE", "STATE", "QP_NUM"); + } else { + id_priv = list_entry(v, struct rdma_id_private, list); + format_addr((struct sockaddr *)&id_priv->id.route.addr.src_addr, + local_addr); + format_addr((struct sockaddr *)&id_priv->id.route.addr.dst_addr, + remote_addr); + + seq_printf(file, + "%-5s %-8s %-5d %-8s %-52s %-52s %-6s %-15s %-8d \n", + format_node_type(id_priv->id.route.addr.dev_addr.dev_type), + (id_priv->id.device) ? id_priv->id.device->name : "", + id_priv->id.port_num, + (id_priv->id.route.addr.dev_addr.src_dev) ? id_priv->id.route.addr.dev_addr.src_dev->name : "", + local_addr, remote_addr, + format_port_space(id_priv->id.ps), + format_cma_state(id_priv->state), + id_priv->qp_num); + } + return 0; +} + +static const struct seq_operations cma_rdma_id_seq_ops = { + .start = cma_rdma_id_seq_start, + .next = cma_rdma_id_seq_next, + .stop = cma_rdma_id_seq_stop, + .show = cma_rdma_id_seq_show, +}; + +static int cma_rdma_id_open(struct inode *inode, struct file *file) +{ + struct seq_file *seq; + int ret; + + ret = seq_open(file, &cma_rdma_id_seq_ops); + if (ret) + return ret; + + seq = file->private_data; + seq->private = inode->i_private; + + return 0; +} + +static const struct file_operations cma_rdma_id_fops = { + .owner = THIS_MODULE, + .open = cma_rdma_id_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release +}; + +void cma_create_debug_files(struct cma_device *cma_dev) +{ + char name[IB_DEVICE_NAME_MAX + sizeof "_rdma_id"]; + snprintf(name, sizeof name, "%s_rdma_id", cma_dev->device->name); + cma_dev->rdma_id_dentry = debugfs_create_file(name, S_IFREG | S_IRUGO, + cma_root_dentry, cma_dev, &cma_rdma_id_fops); + if (!cma_dev->rdma_id_dentry) + printk(KERN_WARNING "RDMA CMA: failed to create debugfs file %s\n", name); +} + static void cma_add_one(struct ib_device *device) { struct cma_device *cma_dev; @@ -2888,6 +3060,7 @@ static void cma_add_one(struct ib_device *device) list_for_each_entry(id_priv, &listen_any_list, list) cma_listen_on_dev(id_priv, cma_dev); mutex_unlock(&lock); + cma_create_debug_files(cma_dev); } static int cma_remove_id_dev(struct rdma_id_private *id_priv) @@ -2922,6 +3095,8 @@ static void cma_process_remove(struct cma_device *cma_dev) int ret; mutex_lock(&lock); + if (cma_dev->rdma_id_dentry) + debugfs_remove(cma_dev->rdma_id_dentry); while (!list_empty(&cma_dev->id_list)) { id_priv = list_entry(cma_dev->id_list.next, struct rdma_id_private, list); @@ -2957,6 +3132,7 @@ static void cma_remove_one(struct ib_device *device) mutex_unlock(&lock); cma_process_remove(cma_dev); + kfree(cma_dev); } @@ -2964,6 +3140,12 @@ static int cma_init(void) { int ret, low, high, remaining; + cma_root_dentry = debugfs_create_dir("rdma_cm", NULL); + if (!cma_root_dentry) { + printk(KERN_ERR "RDMA CMA: failed to create debugfs dir\n"); + return -ENOMEM; + } + get_random_bytes(&next_port, sizeof next_port); inet_get_local_port_range(&low, &high); remaining = (high - low) + 1; @@ -3001,6 +3183,7 @@ static void cma_cleanup(void) idr_destroy(&tcp_ps); idr_destroy(&udp_ps); idr_destroy(&ipoib_ps); + debugfs_remove(cma_root_dentry); } module_init(cma_init); From tziporet at dev.mellanox.co.il Sun Jun 21 13:34:31 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 21 Jun 2009 23:34:31 +0300 Subject: [ofa-general] Compilation errors with OFED 1.4.1/ 1.4 In-Reply-To: <3307cdf90906191417y46a0142bs50f58c36860d33ab@mail.gmail.com> References: <9759F033B56A60469F217C49AC58E2BF085A541B@MAILUK2.rms.com> <3307cdf90906191417y46a0142bs50f58c36860d33ab@mail.gmail.com> Message-ID: <4A3E9957.50104@mellanox.co.il> Rajouri Jammu wrote: > Thanks for the response Evan. > > It appears to me that this is a problem in OFED 1.4.1 since the header > file in /usr/src/ofa_kernel/include/rdma/ib_verbs.h is referring to > data structures that are not present in the SLES 10 u 2 kernel. > > However, someone else indicated it was a problem in our code. > > Btw, my code compiled fine with OFED 1.3.1 on the same kernel. > We (Mellanox) are running regression testing on SLES10 SP2 with OFED 1.4.1 without any problem So I guess its something specific in your system, or order of include files you use Tziporet From an2355 at columbia.edu Sun Jun 21 16:43:59 2009 From: an2355 at columbia.edu (Ashwath Narasimhan) Date: Sun, 21 Jun 2009 19:43:59 -0400 Subject: [ofa-general] Infiniband Driver support for Mellanox HCA in Centos 5.3 Message-ID: Hi, I am a linux newbie and I am trying to run ipoib on linux Centos 5.3. My software driver does not recognize the hardware even though it is connected. 1. I am using the following infiniband HCA $ my prompt> lspci |grep Mellanox 01:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (rev a0) 2. I installed the infiniband utilities using ---> yum groupinstall 'OpenFabrics Enterprise Distribution' 3. I then started the driver in this way --> [root at localhost ]# /etc/init.d/openibd start Loading OpenIB kernel modules: [ OK ] [root at localhost ]# ibv_devinfo libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0 No IB devices found As you can see it says No IB devices found. I am able to detect the hardware using Windows, however I am not able to detect the same using Linux. Please guide me as to what I can do. 4. One more Question, As I dug into to the release notes, I find that Centos 5.3 is not supported in the distribution of OFED-1.4. Is this correct? Does this mean I have to downgrade to centos 5.2? 5. What could be the time frame by which Centos 5.3 is also supported? Thanks. regards, Ashwath -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfr at canb.auug.org.au Sun Jun 21 18:01:13 2009 From: sfr at canb.auug.org.au (Stephen Rothwell) Date: Mon, 22 Jun 2009 11:01:13 +1000 Subject: [ofa-general] linux-next: origin tree build warning Message-ID: <20090622110113.7cd89382.sfr@canb.auug.org.au> Hi all, Today's linux-next build (powerpc ppc64_defconfig) produced this warning: drivers/infiniband/hw/mthca/mthca_mr.c: In function 'mthca_arbel_write_mtt_seg': drivers/infiniband/hw/mthca/mthca_mr.c:358: warning: 'dma_sync_single' is deprecated (declared at /scratch/sfr/next/include/linux/dma-mapping.h:113) drivers/infiniband/hw/mthca/mthca_mr.c: In function 'mthca_arbel_map_phys_fmr': drivers/infiniband/hw/mthca/mthca_mr.c:810: warning: 'dma_sync_single' is deprecated (declared at /scratch/sfr/next/include/linux/dma-mapping.h:113) Introduced by commit dbe6f1869188b6e04e38aa861dd198befb08bcd7 ("dma-mapping: mark dma_sync_single and dma_sync_sg as deprecated"). -- Cheers, Stephen Rothwell sfr at canb.auug.org.au http://www.canb.auug.org.au/~sfr/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From amirv at mellanox.co.il Mon Jun 22 01:18:40 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:40 +0300 Subject: [ofa-general] Overhaul on SDP BCopy mode Message-ID: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> Following is a series of patches to improve SDP small-mid sized packets bandwidth performance. Patches were splitted from 1 big patch for readability. The full patch is at the end of the series. -- Amir Vadai Software Eng. Mellanox Technologies mailto: amirv at mellanox.co.il Tel +972-3-6259539 From amirv at mellanox.co.il Mon Jun 22 01:18:42 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:42 +0300 Subject: [ofa-general] [PATCH] sdp: Interrupts performance fixes In-Reply-To: <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> * use rcvbuf_initial - fix server to use rcvbuf_initial_size as initial * hw interrupt moderation * fix interrupts statistics * fix max frags bug Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 7 +++++-- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 1 + drivers/infiniband/ulp/sdp/sdp_cma.c | 21 +++++++++++++++------ drivers/infiniband/ulp/sdp/sdp_proc.c | 3 +-- 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index e36f786..a4d3aa2 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -83,8 +83,7 @@ struct sdpstats { u32 sendmsg_seglen[25]; u32 send_size[25]; u32 post_recv; - u32 rx_int_count; - u32 tx_int_count; + u32 int_count; u32 bzcopy_poll_miss; u32 send_wait_for_mem; u32 send_miss_no_credits; @@ -152,6 +151,10 @@ static inline void sdpstats_hist(u32 *h, u32 val, u32 maxidx, int is_log) #define SDP_OP_RECV 0x800000000LL #define SDP_OP_SEND 0x400000000LL +#ifndef MIN +#define MIN(a, b) (a < b ? a : b) +#endif + extern struct list_head sock_list; extern spinlock_t sock_list_lock; diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 7c7271c..c5a0ccf 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -917,6 +917,7 @@ void sdp_completion_handler(struct ib_cq *cq, void *cq_context) struct sock *sk = cq_context; struct sdp_sock *ssk = sdp_sk(sk); schedule_work(&ssk->work); + SDPSTATS_COUNTER_INC(int_count); } int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq) diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index b6bc4e5..3bc2b02 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -129,7 +129,12 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) goto err_cq; } - ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); + rc = ib_modify_cq(cq, 10, 200); + if (rc) { + sdp_warn(sk, "Unable to modify RX CQ: %d.\n", rc); + goto err_qp; + } + sdp_warn(sk, "Initialized CQ moderation\n"); qp_init_attr.send_cq = qp_init_attr.recv_cq = cq; @@ -207,10 +212,11 @@ static int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, sizeof(struct sdp_bsdh); sdp_sk(child)->send_frags = PAGE_ALIGN(sdp_sk(child)->xmit_size_goal) / PAGE_SIZE; - sdp_init_buffers(sdp_sk(child), ntohl(h->desremrcvsz)); + sdp_init_buffers(sdp_sk(child), rcvbuf_initial_size); - sdp_dbg(child, "%s bufs %d xmit_size_goal %d send trigger %d\n", + sdp_dbg(child, "%s recv_frags: %d bufs %d xmit_size_goal %d send trigger %d\n", __func__, + sdp_sk(child)->recv_frags, sdp_sk(child)->bufs, sdp_sk(child)->xmit_size_goal, sdp_sk(child)->min_bufs); @@ -251,13 +257,16 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, sdp_sk(sk)->min_bufs = sdp_sk(sk)->bufs / 4; sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - sizeof(struct sdp_bsdh); - sdp_sk(sk)->send_frags = PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / - PAGE_SIZE; + sdp_sk(sk)->send_frags = MIN(PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / + PAGE_SIZE, SDP_MAX_SEND_SKB_FRAGS); + sdp_sk(sk)->xmit_size_goal = MIN(sdp_sk(sk)->xmit_size_goal, + sdp_sk(sk)->send_frags * PAGE_SIZE); - sdp_dbg(sk, "%s bufs %d xmit_size_goal %d send trigger %d\n", + sdp_dbg(sk, "%s bufs %d xmit_size_goal %d send_frags: %d send trigger %d\n", __func__, sdp_sk(sk)->bufs, sdp_sk(sk)->xmit_size_goal, + sdp_sk(sk)->send_frags, sdp_sk(sk)->min_bufs); sdp_sk(sk)->poll_cq = 1; diff --git a/drivers/infiniband/ulp/sdp/sdp_proc.c b/drivers/infiniband/ulp/sdp/sdp_proc.c index 8971517..b8614a5 100644 --- a/drivers/infiniband/ulp/sdp/sdp_proc.c +++ b/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -262,8 +262,7 @@ static int sdpstats_seq_show(struct seq_file *seq, void *v) seq_printf(seq, "tx_poll_miss \t\t: %d\n", sdpstats.tx_poll_miss); seq_printf(seq, "CQ stats:\n"); - seq_printf(seq, "- RX interrupts\t\t: %d\n", sdpstats.rx_int_count); - seq_printf(seq, "- TX interrupts\t\t: %d\n", sdpstats.tx_int_count); + seq_printf(seq, "- interrupts\t\t: %d\n", sdpstats.int_count); return 0; } -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:45 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:45 +0300 Subject: [ofa-general] [PATCH] sdp: don't do nagle on first packet In-Reply-To: <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-6-git-send-email-amirv@mellanox.co.il> Send first packet immediately - even when nagle is on Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 5 +++ drivers/infiniband/ulp/sdp/sdp_bcopy.c | 48 ++++++++++++++++++++++++++++++-- drivers/infiniband/ulp/sdp/sdp_rx.c | 6 +++- drivers/infiniband/ulp/sdp/sdp_tx.c | 11 +++++-- 4 files changed, 63 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index 57265aa..4bc81e1 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -213,6 +213,7 @@ static inline void sdpstats_hist(u32 *h, u32 val, u32 maxidx, int is_log) instead of interrupts (in per-core Tx rings) - should be power of 2 */ #define SDP_TX_POLL_MODER 16 #define SDP_TX_POLL_TIMEOUT (HZ / 4) +#define SDP_NAGLE_TIMEOUT (HZ / 10) #define SDP_RESOLVE_TIMEOUT 1000 #define SDP_ROUTE_TIMEOUT 1000 @@ -406,6 +407,9 @@ struct sdp_sock { unsigned max_bufs; /* Initial buffers offered by other side */ unsigned min_bufs; /* Low water mark to wake senders */ + unsigned long nagle_last_unacked; /* mseq of lastest unacked packet */ + struct timer_list nagle_timer; /* timeout waiting for ack */ + atomic_t remote_credits; #define remote_credits(ssk) (atomic_read(&ssk->remote_credits)) int poll_cq; @@ -552,6 +556,7 @@ int sdp_xmit_poll(struct sdp_sock *ssk, int force); void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid); void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonagle); #define sdp_post_sends(ssk, nonagle) _sdp_post_sends(__func__, __LINE__, ssk, nonagle) +void sdp_nagle_timeout(unsigned long data); /* sdp_rx.c */ void sdp_rx_ring_init(struct sdp_sock *ssk); diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index b98171e..3465035 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -105,12 +105,54 @@ static inline void update_send_head(struct sock *sk, struct sk_buff *skb) static inline int sdp_nagle_off(struct sdp_sock *ssk, struct sk_buff *skb) { - return (ssk->nonagle & TCP_NAGLE_OFF) || + int send_now = + (ssk->nonagle & TCP_NAGLE_OFF) || + !ssk->nagle_last_unacked || skb->next != (struct sk_buff *)&ssk->isk.sk.sk_write_queue || skb->len + sizeof(struct sdp_bsdh) >= ssk->xmit_size_goal || - (ring_tail(ssk->tx_ring) == ring_head(ssk->tx_ring) && - !(ssk->nonagle & TCP_NAGLE_CORK)) || (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_PSH); + + if (send_now) { + unsigned long mseq = ring_head(ssk->tx_ring); + ssk->nagle_last_unacked = mseq; + + mod_timer(&ssk->nagle_timer, jiffies + SDP_NAGLE_TIMEOUT); + sdp_dbg_data(&ssk->isk.sk, "Starting nagle timer\n"); + } + sdp_dbg_data(&ssk->isk.sk, "send_now = %d last_unacked = %ld\n", + send_now, ssk->nagle_last_unacked); + + return send_now; +} + +void sdp_nagle_timeout(unsigned long data) +{ + struct sdp_sock *ssk = (struct sdp_sock *)data; + struct sock *sk = &ssk->isk.sk; + + sdp_dbg_data(&ssk->isk.sk, "last_unacked = %ld\n", ssk->nagle_last_unacked); + + if (!ssk->nagle_last_unacked) + return; + + /* Only process if the socket is not in use */ + bh_lock_sock(sk); + if (sock_owned_by_user(sk)) { + mod_timer(&ssk->nagle_timer, jiffies + SDP_NAGLE_TIMEOUT); + sdp_dbg_data(&ssk->isk.sk, "socket is busy - trying later\n"); + goto out; + } + + if (sk->sk_state == TCP_CLOSE) + goto out; + + ssk->nagle_last_unacked = 0; + sdp_post_sends(ssk, 0); + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + sk_stream_write_space(&ssk->isk.sk); +out: + bh_unlock_sock(sk); } int sdp_post_credits(struct sdp_sock *ssk) diff --git a/drivers/infiniband/ulp/sdp/sdp_rx.c b/drivers/infiniband/ulp/sdp/sdp_rx.c index 895ec12..05d8690 100644 --- a/drivers/infiniband/ulp/sdp/sdp_rx.c +++ b/drivers/infiniband/ulp/sdp/sdp_rx.c @@ -490,6 +490,7 @@ static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) struct sdp_bsdh *h; struct sock *sk = &ssk->isk.sk; int credits_before; + unsigned long mseq_ack; skb = sdp_recv_completion(ssk, wc->wr_id); if (unlikely(!skb)) @@ -536,9 +537,12 @@ static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) SDPSTATS_HIST_LINEAR(credits_before_update, tx_credits(ssk)); + mseq_ack = ntohl(h->mseq_ack); credits_before = tx_credits(ssk); - atomic_set(&ssk->tx_ring.credits, ntohl(h->mseq_ack) - ring_head(ssk->tx_ring) + 1 + + atomic_set(&ssk->tx_ring.credits, mseq_ack - ring_head(ssk->tx_ring) + 1 + ntohs(h->bufs)); + if (mseq_ack >= ssk->nagle_last_unacked) + ssk->nagle_last_unacked = 0; sdp_prf(&ssk->isk.sk, skb, "RX %s bufs=%d c before:%d after:%d " "mseq:%d, ack:%d", mid2str(h->mid), ntohs(h->bufs), credits_before, diff --git a/drivers/infiniband/ulp/sdp/sdp_tx.c b/drivers/infiniband/ulp/sdp/sdp_tx.c index 28720b9..a7d16fe 100644 --- a/drivers/infiniband/ulp/sdp/sdp_tx.c +++ b/drivers/infiniband/ulp/sdp/sdp_tx.c @@ -68,7 +68,7 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) { struct sdp_buf *tx_req; struct sdp_bsdh *h = (struct sdp_bsdh *)skb_push(skb, sizeof *h); - unsigned mseq = ring_head(ssk->tx_ring); + unsigned long mseq = ring_head(ssk->tx_ring); int i, rc, frags; u64 addr; struct ib_device *dev; @@ -78,7 +78,6 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) struct ib_sge *sge = ibsge; struct ib_send_wr tx_wr = { 0 }; - SDPSTATS_COUNTER_MID_INC(post_send, mid); SDPSTATS_HIST(send_size, skb->len); @@ -93,7 +92,7 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) h->mseq = htonl(mseq); h->mseq_ack = htonl(mseq_ack(ssk)); - sdp_prf(&ssk->isk.sk, skb, "TX: %s bufs: %d mseq:%d ack:%d", + sdp_prf(&ssk->isk.sk, skb, "TX: %s bufs: %d mseq:%ld ack:%d", mid2str(mid), ring_posted(ssk->rx_ring), mseq, ntohl(h->mseq_ack)); SDP_DUMP_PACKET(&ssk->isk.sk, "TX", skb, h); @@ -405,6 +404,10 @@ int sdp_tx_ring_create(struct sdp_sock *ssk, struct ib_device *device) ssk->tx_ring.timer.data = (unsigned long) ssk; ssk->tx_ring.poll_cnt = 0; + init_timer(&ssk->nagle_timer); + ssk->nagle_timer.function = sdp_nagle_timeout; + ssk->nagle_timer.data = (unsigned long) ssk; + return 0; err_cq: @@ -416,6 +419,8 @@ out: void sdp_tx_ring_destroy(struct sdp_sock *ssk) { + del_timer(&ssk->nagle_timer); + if (ssk->tx_ring.buffer) { sdp_tx_ring_purge(ssk); -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:47 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:47 +0300 Subject: [ofa-general] [PATCH] sdp: fix bad HELLO/HELLO_ACK buffer size matching + let recvmsg do posts on data packets too In-Reply-To: <1245658729-10642-7-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> <1245658729-10642-6-git-send-email-amirv@mellanox.co.il> <1245658729-10642-7-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-8-git-send-email-amirv@mellanox.co.il> Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp_cma.c | 7 ++++--- drivers/infiniband/ulp/sdp/sdp_rx.c | 3 +-- drivers/infiniband/ulp/sdp/sdp_tx.c | 2 ++ 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index bbc69c6..83e39dc 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -215,7 +215,7 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, sdp_sk(sk)->max_bufs = ntohs(h->bsdh.bufs); atomic_set(&sdp_sk(sk)->tx_ring.credits, sdp_sk(sk)->max_bufs); sdp_sk(sk)->min_bufs = tx_credits(sdp_sk(sk)) / 4; - sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - SDP_HEAD_SIZE; + sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - sizeof(struct sdp_bsdh); sdp_sk(sk)->send_frags = MIN(PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / PAGE_SIZE, MAX_SKB_FRAGS) + 1; /* The +1 is to conpensate on not aligned buffers */ sdp_sk(sk)->xmit_size_goal = MIN(sdp_sk(sk)->xmit_size_goal, @@ -352,7 +352,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) hh.majv_minv = SDP_MAJV_MINV; sdp_init_buffers(sdp_sk(sk), rcvbuf_initial_size); hh.localrcvsz = hh.desremrcvsz = htonl(sdp_sk(sk)->recv_frags * - PAGE_SIZE + SDP_HEAD_SIZE); + PAGE_SIZE + sizeof(struct sdp_bsdh)); hh.max_adverts = 0x1; inet_sk(sk)->saddr = inet_sk(sk)->rcv_saddr = ((struct sockaddr_in *)&id->route.addr.src_addr)->sin_addr.s_addr; @@ -385,7 +385,8 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) hah.majv_minv = SDP_MAJV_MINV; hah.ext_max_adverts = 1; /* Doesn't seem to be mandated by spec, but just in case */ - hah.actrcvsz = htonl(sdp_sk(child)->recv_frags * PAGE_SIZE + SDP_HEAD_SIZE); + hah.actrcvsz = htonl(sdp_sk(child)->recv_frags * PAGE_SIZE + + sizeof(struct sdp_bsdh)); memset(&conn_param, 0, sizeof conn_param); conn_param.private_data_len = sizeof hah; conn_param.private_data = &hah; diff --git a/drivers/infiniband/ulp/sdp/sdp_rx.c b/drivers/infiniband/ulp/sdp/sdp_rx.c index b3611ae..1ef28ac 100644 --- a/drivers/infiniband/ulp/sdp/sdp_rx.c +++ b/drivers/infiniband/ulp/sdp/sdp_rx.c @@ -753,8 +753,7 @@ static void sdp_rx_irq(struct ib_cq *cq, void *cq_context) credits_before, tx_credits(ssk)); if (posts_handler(ssk) || - (!skb_queue_empty(&ssk->rx_ctl_q) && - (sk->sk_socket && (sk->sk_socket->flags & SOCK_ASYNC_WAITDATA)))) { + (sk->sk_socket && test_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags))) { sdp_prf(&ssk->isk.sk, NULL, "Somebody is doing the post work for me. %d", posts_handler(ssk)); diff --git a/drivers/infiniband/ulp/sdp/sdp_tx.c b/drivers/infiniband/ulp/sdp/sdp_tx.c index bb0d81d..9cc37a3 100644 --- a/drivers/infiniband/ulp/sdp/sdp_tx.c +++ b/drivers/infiniband/ulp/sdp/sdp_tx.c @@ -205,6 +205,8 @@ static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) if (unlikely(wc->status)) { if (wc->status != IB_WC_WR_FLUSH_ERR) { struct sock *sk = &ssk->isk.sk; + sdp_prf(sk, skb, "Send completion with error. " + "Status %d", wc->status); sdp_warn(sk, "Send completion with error. " "Status %d\n", wc->status); sdp_set_error(sk, -ECONNRESET); -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:48 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:48 +0300 Subject: [ofa-general] [PATCH] sdp: make interrupt moderation adaptive In-Reply-To: <1245658729-10642-8-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> <1245658729-10642-6-git-send-email-amirv@mellanox.co.il> <1245658729-10642-7-git-send-email-amirv@mellanox.co.il> <1245658729-10642-8-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-9-git-send-email-amirv@mellanox.co.il> Make HW RC interrupt moderation adaptive. For latency sensitive traffic - no moderation. In BW mode - maximal moderation. Use heuristics to detect mode.:w Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 42 ++++++++ drivers/infiniband/ulp/sdp/sdp_cma.c | 3 + drivers/infiniband/ulp/sdp/sdp_main.c | 187 +++++++++++++++++++++++++++------ drivers/infiniband/ulp/sdp/sdp_rx.c | 40 ++------ drivers/infiniband/ulp/sdp/sdp_tx.c | 7 +- 5 files changed, 213 insertions(+), 66 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index 5d3e7d7..cc38f5b 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -37,6 +37,16 @@ spin_unlock_irqrestore(&ssk->rx_ring.lock, f); \ } while (0) +#define SDP_MODPARAM_SINT(var, def_val, msg) \ + static int var = def_val; \ + module_param_named(var, var, int, 0644); \ + MODULE_PARM_DESC(var, msg " [" #def_val "]"); \ + +#define SDP_MODPARAM_INT(var, def_val, msg) \ + int var = def_val; \ + module_param_named(var, var, int, 0644); \ + MODULE_PARM_DESC(var, msg " [" #def_val "]"); \ + #ifdef SDP_PROFILING struct sk_buff; struct sdpprf_log { @@ -238,6 +248,9 @@ static inline void sdpstats_hist(u32 *h, u32 val, u32 maxidx, int is_log) /* how long (in jiffies) to block sender till tx completion*/ #define SDP_BZCOPY_POLL_TIMEOUT (HZ / 10) +#define SDP_AUTO_CONF 0xffff +#define AUTO_MOD_DELAY (HZ / 4) + #define BZCOPY_STATE(skb) (*(struct bzcopy_state **)(skb->cb)) #ifndef MIN #define MIN(a, b) (a < b ? a : b) @@ -372,6 +385,27 @@ struct sdp_chrecvbuf { sdp_do_posts(ssk); \ }) +struct sdp_moderation { + unsigned long last_moder_packets; + unsigned long last_moder_tx_packets; + unsigned long last_moder_bytes; + unsigned long last_moder_jiffies; + int last_moder_time; + u16 rx_usecs; + u16 rx_frames; + u16 tx_usecs; + u32 pkt_rate_low; + u16 rx_usecs_low; + u32 pkt_rate_high; + u16 rx_usecs_high; + u16 sample_interval; + u16 adaptive_rx_coal; + u32 msg_enable; + + int moder_cnt; + int moder_time; +}; + struct sdp_sock { /* sk has to be the first member of inet_sock */ struct inet_sock isk; @@ -451,6 +485,12 @@ struct sdp_sock { int recv_frags; /* max skb frags in recv packets */ int send_frags; /* max skb frags in send packets */ + unsigned long tx_packets; + unsigned long rx_packets; + unsigned long tx_bytes; + unsigned long rx_bytes; + struct sdp_moderation auto_mod; + /* BZCOPY data */ int zcopy_thresh; }; @@ -568,6 +608,8 @@ void sdp_start_keepalive_timer(struct sock *sk); int sdp_init_sock(struct sock *sk); int __init sdp_proc_init(void); void sdp_proc_unregister(void); +/* sdp_main.c */ +void sdp_set_default_moderation(struct sdp_sock *ssk); /* sdp_tx.c */ int sdp_tx_ring_create(struct sdp_sock *ssk, struct ib_device *device); diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index 83e39dc..55ade25 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -203,6 +203,7 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, sdp_dbg(sk, "%s\n", __func__); sdp_exch_state(sk, TCPF_SYN_SENT, TCP_ESTABLISHED); + sdp_set_default_moderation(sdp_sk(sk)); if (sock_flag(sk, SOCK_KEEPOPEN)) sdp_start_keepalive_timer(sk); @@ -249,6 +250,8 @@ static int sdp_connected_handler(struct sock *sk, struct rdma_cm_event *event) sdp_exch_state(sk, TCPF_SYN_RECV, TCP_ESTABLISHED); + sdp_set_default_moderation(sdp_sk(sk)); + if (sock_flag(sk, SOCK_KEEPOPEN)) sdp_start_keepalive_timer(sk); diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index 64511fe..b34743d 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -79,41 +79,33 @@ MODULE_DESCRIPTION("InfiniBand SDP module"); MODULE_LICENSE("Dual BSD/GPL"); #ifdef CONFIG_INFINIBAND_SDP_DEBUG -int sdp_debug_level; - -module_param_named(debug_level, sdp_debug_level, int, 0644); -MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0."); +SDP_MODPARAM_INT(sdp_debug_level, 0, "Enable debug tracing if > 0."); #endif #ifdef CONFIG_INFINIBAND_SDP_DEBUG -int sdp_data_debug_level; - -module_param_named(data_debug_level, sdp_data_debug_level, int, 0644); -MODULE_PARM_DESC(data_debug_level, "Enable data path debug tracing if > 0."); +SDP_MODPARAM_INT(sdp_data_debug_level, 0, "Enable data path debug tracing if > 0."); #endif -static int recv_poll_hit; - -module_param_named(recv_poll_hit, recv_poll_hit, int, 0644); -MODULE_PARM_DESC(recv_poll_hit, "How many times recv poll helped."); - -static int recv_poll_miss; - -module_param_named(recv_poll_miss, recv_poll_miss, int, 0644); -MODULE_PARM_DESC(recv_poll_miss, "How many times recv poll missed."); - -static int recv_poll = 1000; - -module_param_named(recv_poll, recv_poll, int, 0644); -MODULE_PARM_DESC(recv_poll, "How many times to poll recv."); - -static unsigned int sdp_keepalive_time = SDP_KEEPALIVE_TIME; - -module_param_named(sdp_keepalive_time, sdp_keepalive_time, uint, 0644); -MODULE_PARM_DESC(sdp_keepalive_time, "Default idle time in seconds before keepalive probe sent."); - -static int sdp_zcopy_thresh = 65536; -module_param_named(sdp_zcopy_thresh, sdp_zcopy_thresh, int, 0644); -MODULE_PARM_DESC(sdp_zcopy_thresh, "Zero copy send threshold; 0=0ff."); +SDP_MODPARAM_SINT(recv_poll_hit, -1, "How many times recv poll helped."); +SDP_MODPARAM_SINT(recv_poll_miss, -1, "How many times recv poll missed."); +SDP_MODPARAM_SINT(recv_poll, 1000, "How many times to poll recv."); +SDP_MODPARAM_SINT(sdp_keepalive_time, SDP_KEEPALIVE_TIME, + "Default idle time in seconds before keepalive probe sent."); +SDP_MODPARAM_SINT(sdp_zcopy_thresh, 65536, "Zero copy send threshold; 0=0ff."); + +#define SDP_RX_COAL_TIME_HIGH 128 +SDP_MODPARAM_SINT(sdp_rx_coal_target, 0x50000, + "Target number of bytes to coalesce with interrupt moderation (bytes)."); +SDP_MODPARAM_SINT(sdp_rx_coal_time, 0x10, "rx coal time (jiffies)."); +SDP_MODPARAM_SINT(sdp_rx_rate_low, 80000, "rx_rate low (packets/sec)."); +SDP_MODPARAM_SINT(sdp_rx_coal_time_low, 0, "low moderation time val (usec)."); +SDP_MODPARAM_SINT(sdp_rx_rate_high, 100000, "rx_rate high (packets/sec)."); +SDP_MODPARAM_SINT(sdp_rx_coal_time_high, 128, "high moderation time val (usec)."); +SDP_MODPARAM_SINT(sdp_rx_rate_thresh, (200000 / SDP_RX_COAL_TIME_HIGH), + "rx rate thresh ()."); +SDP_MODPARAM_SINT(sdp_sample_interval, (HZ / 4), "sample interval (jiffies)."); + +SDP_MODPARAM_INT(hw_int_mod_count, -1, "forced hw int moderation val. -1 for auto (packets)."); +SDP_MODPARAM_INT(hw_int_mod_usec, -1, "forced hw int moderation val. -1 for auto (usec)."); struct workqueue_struct *sdp_wq; struct workqueue_struct *rx_comp_wq; @@ -298,6 +290,135 @@ void sdp_start_keepalive_timer(struct sock *sk) sdp_reset_keepalive_timer(sk, sdp_keepalive_time_when(sdp_sk(sk))); } +void sdp_set_default_moderation(struct sdp_sock *ssk) +{ + struct sdp_moderation *mod = &ssk->auto_mod; + int rx_buf_size; + + if (hw_int_mod_count > -1 || hw_int_mod_usec > -1) { + int err; + + mod->adaptive_rx_coal = 0; + + if (hw_int_mod_count > 0 && hw_int_mod_usec > 0) { + err = ib_modify_cq(ssk->rx_ring.cq, hw_int_mod_count, hw_int_mod_usec); + if (err) + sdp_warn(&ssk->isk.sk, "Failed modifying moderation for cq"); + else + sdp_dbg(&ssk->isk.sk, "Using fixed interrupt moderation\n"); + } + return; + } + + mod->adaptive_rx_coal = 1; + sdp_dbg(&ssk->isk.sk, "Using adaptive interrupt moderation\n"); + + /* If we haven't received a specific coalescing setting + * (module param), we set the moderation paramters as follows: + * - moder_cnt is set to the number of mtu sized packets to + * satisfy our coelsing target. + * - moder_time is set to a fixed value. + */ + rx_buf_size = (ssk->recv_frags * PAGE_SIZE) + sizeof(struct sdp_bsdh); + mod->moder_cnt = sdp_rx_coal_target / rx_buf_size + 1; + mod->moder_time = sdp_rx_coal_time; + sdp_dbg(&ssk->isk.sk, "Default coalesing params for buf size:%d - " + "moder_cnt:%d moder_time:%d\n", + rx_buf_size, mod->moder_cnt, mod->moder_time); + + /* Reset auto-moderation params */ + mod->pkt_rate_low = sdp_rx_rate_low; + mod->rx_usecs_low = sdp_rx_coal_time_low; + mod->pkt_rate_high = sdp_rx_rate_high; + mod->rx_usecs_high = sdp_rx_coal_time_high; + mod->sample_interval = sdp_sample_interval; + + mod->last_moder_time = SDP_AUTO_CONF; + mod->last_moder_jiffies = 0; + mod->last_moder_packets = 0; + mod->last_moder_tx_packets = 0; + mod->last_moder_bytes = 0; +} + +static void sdp_auto_moderation(struct sdp_sock *ssk) +{ + struct sdp_moderation *mod = &ssk->auto_mod; + + unsigned long period = (unsigned long) (jiffies - mod->last_moder_jiffies); + unsigned long packets; + unsigned long rate; + unsigned long avg_pkt_size; + unsigned long tx_pkt_diff; + unsigned long rx_pkt_diff; + int moder_time; + int err; + + if (!mod->adaptive_rx_coal) + return; + + if (period < mod->sample_interval) + return; + + if (!mod->last_moder_jiffies || !period) + goto out; + + tx_pkt_diff = ((unsigned long) (ssk->tx_packets - + mod->last_moder_tx_packets)); + rx_pkt_diff = ((unsigned long) (ssk->rx_packets - + mod->last_moder_packets)); + packets = max(tx_pkt_diff, rx_pkt_diff); + rate = packets * HZ / period; + avg_pkt_size = packets ? ((unsigned long) (ssk->rx_bytes - + mod->last_moder_bytes)) / packets : 0; + + /* Apply auto-moderation only when packet rate exceeds a rate that + * it matters */ + if (rate > sdp_rx_rate_thresh) { + /* If tx and rx packet rates are not balanced, assume that + * traffic is mainly BW bound and apply maximum moderation. + * Otherwise, moderate according to packet rate */ + if (2 * tx_pkt_diff > 3 * rx_pkt_diff || + 2 * rx_pkt_diff > 3 * tx_pkt_diff) { + moder_time = mod->rx_usecs_high; + } else { + if (rate < mod->pkt_rate_low) { + moder_time = mod->rx_usecs_low; + } else if (rate > mod->pkt_rate_high) + moder_time = mod->rx_usecs_high; + else + moder_time = (rate - mod->pkt_rate_low) * + (mod->rx_usecs_high - mod->rx_usecs_low) / + (mod->pkt_rate_high - mod->pkt_rate_low) + + mod->rx_usecs_low; + } + } else { + /* When packet rate is low, use default moderation rather than + * 0 to prevent interrupt storms if traffic suddenly increases */ + moder_time = mod->moder_time; + } + + sdp_dbg_data(&ssk->isk.sk, "tx rate:%lu rx_rate:%lu\n", + tx_pkt_diff * HZ / period, rx_pkt_diff * HZ / period); + + sdp_dbg_data(&ssk->isk.sk, "Rx moder_time changed from:%d to %d period:%lu " + "[jiff] packets:%lu avg_pkt_size:%lu rate:%lu [p/s])\n", + mod->last_moder_time, moder_time, period, packets, + avg_pkt_size, rate); + + if (moder_time != mod->last_moder_time) { + mod->last_moder_time = moder_time; + err = ib_modify_cq(ssk->rx_ring.cq, mod->moder_cnt, moder_time); + if (err) + sdp_dbg_data(&ssk->isk.sk, "Failed modifying moderation for cq"); + } + +out: + mod->last_moder_packets = ssk->rx_packets; + mod->last_moder_tx_packets = ssk->tx_packets; + mod->last_moder_bytes = ssk->rx_bytes; + mod->last_moder_jiffies = jiffies; +} + void sdp_reset_sk(struct sock *sk, int rc) { struct sdp_sock *ssk = sdp_sk(sk); @@ -1780,6 +1901,8 @@ out: posts_handler_put(ssk); + sdp_auto_moderation(ssk); + rdtscll(end); SDPSTATS_COUNTER_ADD(sendmsg_sum, end - start); release_sock(sk); @@ -2050,6 +2173,8 @@ out: posts_handler_put(ssk); + sdp_auto_moderation(ssk); + release_sock(sk); return err; diff --git a/drivers/infiniband/ulp/sdp/sdp_rx.c b/drivers/infiniband/ulp/sdp/sdp_rx.c index 1ef28ac..2c33af8 100644 --- a/drivers/infiniband/ulp/sdp/sdp_rx.c +++ b/drivers/infiniband/ulp/sdp/sdp_rx.c @@ -37,34 +37,15 @@ #include #include "sdp.h" -static int rcvbuf_scale = 0x10; - -int rcvbuf_initial_size = 32 * 1024; -module_param_named(rcvbuf_initial_size, rcvbuf_initial_size, int, 0644); -MODULE_PARM_DESC(rcvbuf_initial_size, "Receive buffer initial size in bytes."); - -module_param_named(rcvbuf_scale, rcvbuf_scale, int, 0644); -MODULE_PARM_DESC(rcvbuf_scale, "Receive buffer size scale factor."); - -static int top_mem_usage = 0; -module_param_named(top_mem_usage, top_mem_usage, int, 0644); -MODULE_PARM_DESC(top_mem_usage, "Top system wide sdp memory usage for recv (in MB)."); - -static int hw_int_mod_count = 10; -module_param_named(hw_int_mod_count, hw_int_mod_count, int, 0644); -MODULE_PARM_DESC(hw_int_mod_count, "HW interrupt moderation. int count"); - -static int hw_int_mod_msec = 200; -module_param_named(hw_int_mod_msec, hw_int_mod_msec, int, 0644); -MODULE_PARM_DESC(hw_int_mod_count, "HW interrupt moderation. mseq"); +SDP_MODPARAM_INT(rcvbuf_initial_size, 32 * 1024, "Receive buffer initial size in bytes."); +SDP_MODPARAM_SINT(rcvbuf_scale, 0x10, "Receive buffer size scale factor."); +SDP_MODPARAM_SINT(top_mem_usage, 0, "Top system wide sdp memory usage for recv (in MB)."); #ifdef CONFIG_PPC -static int max_large_sockets = 100; +SDP_MODPARAM_SINT(max_large_sockets, 100, "Max number of large sockets (32k buffers)."); #else -static int max_large_sockets = 1000; +SDP_MODPARAM_SINT(max_large_sockets, 1000, "Max number of large sockets (32k buffers)."); #endif -module_param_named(max_large_sockets, max_large_sockets, int, 0644); -MODULE_PARM_DESC(max_large_sockets, "Max number of large sockets (32k buffers)."); static int curr_large_sockets = 0; atomic_t sdp_current_mem_usage; @@ -580,6 +561,9 @@ static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) SDP_DUMP_PACKET(&ssk->isk.sk, "RX", skb, h); skb_reset_transport_header(skb); + ssk->rx_packets++; + ssk->rx_bytes += skb->len; + mseq = ntohl(h->mseq); atomic_set(&ssk->mseq_ack, mseq); if (mseq != (int)wc->wr_id) @@ -823,12 +807,6 @@ int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device) goto err_cq; } - rc = ib_modify_cq(rx_cq, hw_int_mod_count, hw_int_mod_msec); - if (rc) { - sdp_warn(&ssk->isk.sk, "Unable to modify RX CQ: %d.\n", rc); - goto err_mod; - } - sdp_warn(&ssk->isk.sk, "Initialized CQ moderation\n"); sdp_sk(&ssk->isk.sk)->rx_ring.cq = rx_cq; INIT_WORK(&ssk->rx_comp_work, sdp_rx_comp_work); @@ -837,8 +815,6 @@ int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device) goto out; -err_mod: - ib_destroy_cq(rx_cq); err_cq: kfree(ssk->rx_ring.buffer); ssk->rx_ring.buffer = NULL; diff --git a/drivers/infiniband/ulp/sdp/sdp_tx.c b/drivers/infiniband/ulp/sdp/sdp_tx.c index 9cc37a3..6fc4746 100644 --- a/drivers/infiniband/ulp/sdp/sdp_tx.c +++ b/drivers/infiniband/ulp/sdp/sdp_tx.c @@ -38,10 +38,8 @@ #include "sdp.h" #define sdp_cnt(var) do { (var)++; } while (0) -static unsigned sdp_keepalive_probes_sent = 0; -module_param_named(sdp_keepalive_probes_sent, sdp_keepalive_probes_sent, uint, 0644); -MODULE_PARM_DESC(sdp_keepalive_probes_sent, "Total number of keepalive probes sent."); +SDP_MODPARAM_SINT(sdp_keepalive_probes_sent, 0, "Total number of keepalive probes sent."); static int sdp_process_tx_cq(struct sdp_sock *ssk); @@ -83,6 +81,9 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) SDPSTATS_COUNTER_MID_INC(post_send, mid); SDPSTATS_HIST(send_size, skb->len); + ssk->tx_packets++; + ssk->tx_bytes += skb->len; + h->mid = mid; if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) h->flags = SDP_OOB_PRES | SDP_OOB_PEND; -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:46 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:46 +0300 Subject: [ofa-general] [PATCH] sdp: TX from 1 context only. RX with minimal context switches In-Reply-To: <1245658729-10642-6-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> <1245658729-10642-6-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-7-git-send-email-amirv@mellanox.co.il> Do TX work from sendmsg/recvmsg context when available, and don't use work queue. Now all work is done from one context and workqueue is almost not waiting for lock. Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 43 +-- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 3 +- drivers/infiniband/ulp/sdp/sdp_cma.c | 4 +- drivers/infiniband/ulp/sdp/sdp_main.c | 52 +-- drivers/infiniband/ulp/sdp/sdp_proc.c | 22 +- drivers/infiniband/ulp/sdp/sdp_rx.c | 334 +++++++++++--------- drivers/infiniband/ulp/sdp/sdp_tx.c | 6 +- .../2.6.16/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.16/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.16_sles10/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.16_sles10/sdp_0100_revert_to_2_6_23.patch | 26 +- .../sdp_0090_revert_to_2_6_24.patch | 39 +-- .../sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.17/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.17/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.18-EL5.1/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.18-EL5.1/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.18_FC6/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.18_FC6/sdp_0100_revert_to_2_6_23.patch | 26 +- .../sdp_0090_revert_to_2_6_24.patch | 39 +-- .../sdp_0100_revert_to_2_6_23.patch | 26 +- .../sdp_0090_revert_to_2_6_24.patch | 39 +-- .../sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.9_U4/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.9_U4/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.9_U5/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.9_U5/sdp_0100_revert_to_2_6_23.patch | 26 +- .../sdp_0090_revert_to_2_6_24.patch | 39 +-- .../sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.18-EL5.2/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.18-EL5.2/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.18-EL5.3/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.18-EL5.3/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.18/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.18/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.19/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.19/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.20/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.20/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.21/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.21/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.22/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.22/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.23/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.23/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.24/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.9_U6/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.9_U6/sdp_0100_revert_to_2_6_23.patch | 26 +- .../2.6.9_U7/sdp_0090_revert_to_2_6_24.patch | 39 +-- .../2.6.9_U7/sdp_0100_revert_to_2_6_23.patch | 26 +- 50 files changed, 858 insertions(+), 1010 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index f9b295e..25e1844 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -6,15 +6,22 @@ #include #include /* For urgent data flags */ #include +#include + +#undef SDPSTATS_ON +#undef SDP_PROFILING +#undef CONFIG_INFINIBAND_SDP_DEBUG_DATA +#undef CONFIG_INFINIBAND_SDP_DEBUG -#undef SDP_LOCKS_CHECK #define SDPSTATS_ON #define SDP_PROFILING +#define CONFIG_INFINIBAND_SDP_DEBUG_DATA +#define CONFIG_INFINIBAND_SDP_DEBUG #define _sdp_printk(func, line, level, sk, format, arg...) \ - printk(level "%s:%d sdp_sock(%5d %d:%d): " format, \ + printk(level "%s:%d sdp_sock(%5d:%d %d:%d): " format, \ func, line, \ - current->pid, \ + current->pid, smp_processor_id(), \ (sk) ? inet_sk(sk)->num : -1, \ (sk) ? ntohs(inet_sk(sk)->dport) : -1, ## arg) #define sdp_printk(level, sk, format, arg...) \ @@ -23,25 +30,6 @@ sdp_printk(KERN_WARNING, sk, format , ## arg) -#ifdef SDP_LOCKS_CHECK -#define WARN_ON_UNLOCKED(sk, l) do {\ - if (unlikely(!spin_is_locked(l))) { \ - sdp_warn(sk, "lock " #l " should be locked\n"); \ - WARN_ON(1); \ - } \ -} while (0) - -#define WARN_ON_LOCKED(sk, l) do {\ - if (unlikely(spin_is_locked(l))) { \ - sdp_warn(sk, "lock " #l " should be unlocked\n"); \ - WARN_ON(1); \ - } \ -} while (0) -#else -#define WARN_ON_UNLOCKED(sk, l) -#define WARN_ON_LOCKED(sk, l) -#endif - #define rx_ring_lock(ssk, f) do { \ spin_lock_irqsave(&ssk->rx_ring.lock, f); \ } while (0) @@ -55,6 +43,7 @@ struct sk_buff; struct sdpprf_log { int idx; int pid; + int cpu; int sk_num; int sk_dport; struct sk_buff *skb; @@ -83,6 +72,7 @@ static inline unsigned long long current_nsec(void) l->pid = current->pid; \ l->sk_num = (sk) ? inet_sk(sk)->num : -1; \ l->sk_dport = (sk) ? ntohs(inet_sk(sk)->dport) : -1; \ + l->cpu = smp_processor_id(); \ l->skb = s; \ snprintf(l->msg, sizeof(l->msg) - 1, format, ## arg); \ l->time = current_nsec(); \ @@ -360,7 +350,7 @@ struct sdp_sock { struct list_head sock_list; struct list_head accept_queue; struct list_head backlog_queue; - struct sk_buff_head rx_backlog; + struct sk_buff_head rx_ctl_q; struct sock *parent; struct work_struct rx_comp_work; @@ -373,7 +363,8 @@ struct sdp_sock { u16 urg_data; u32 urg_seq; u32 copied_seq; - u32 rcv_nxt; +#define rcv_nxt(ssk) atomic_read(&(ssk->rcv_nxt)) + atomic_t rcv_nxt; int write_seq; int pushed_seq; @@ -561,10 +552,10 @@ void sdp_nagle_timeout(unsigned long data); void sdp_rx_ring_init(struct sdp_sock *ssk); int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device); void sdp_rx_ring_destroy(struct sdp_sock *ssk); -int sdp_process_rx_q(struct sdp_sock *ssk); int sdp_resize_buffers(struct sdp_sock *ssk, u32 new_size); int sdp_init_buffers(struct sdp_sock *ssk, u32 new_size); -void sdp_post_recvs(struct sdp_sock *ssk); +void sdp_schedule_post_recvs(struct sdp_sock *ssk); +void sdp_rx_comp_full(struct sdp_sock *ssk); static inline void sdp_arm_rx_cq(struct sock *sk) { diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 6e0a038..8f6f0e2 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -157,7 +157,8 @@ void sdp_nagle_timeout(unsigned long data) out: bh_unlock_sock(sk); out2: - mod_timer(&ssk->nagle_timer, jiffies + SDP_NAGLE_TIMEOUT); + if (sk->sk_send_head) /* If has pending sends - rearm */ + mod_timer(&ssk->nagle_timer, jiffies + SDP_NAGLE_TIMEOUT); } int sdp_post_credits(struct sdp_sock *ssk) diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index a9dcf77..5726fe4 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -288,15 +288,13 @@ static int sdp_disconnected_handler(struct sock *sk) sdp_dbg(sk, "%s\n", __func__); - sdp_process_rx_q(ssk); - if (ssk->tx_ring.cq) sdp_xmit_poll(ssk, 1); if (sk->sk_state == TCP_SYN_RECV) { sdp_connected_handler(sk, NULL); - if (ssk->rcv_nxt) + if (rcv_nxt(ssk)) return 0; } diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index c6b17db..432b198 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -325,8 +325,6 @@ void sdp_reset_sk(struct sock *sk, int rc) read_lock(&device_removal_lock); - sdp_process_rx_q(ssk); - if (ssk->tx_ring.cq) sdp_xmit_poll(ssk, 1); @@ -505,7 +503,7 @@ static void sdp_close(struct sock *sk, long timeout) * descriptor close, not protocol-sourced closes, because the * reader process may not have drained the data yet! */ - while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) { + while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) { data_was_unread = 1; __kfree_skb(skb); } @@ -696,13 +694,9 @@ static int sdp_wait_for_connect(struct sock *sk, long timeo) TASK_INTERRUPTIBLE); release_sock(sk); if (list_empty(&ssk->accept_queue)) { - sdp_dbg(sk, "%s schedule_timeout\n", __func__); timeo = schedule_timeout(timeo); - sdp_dbg(sk, "%s schedule_timeout done\n", __func__); } - sdp_dbg(sk, "%s lock_sock\n", __func__); lock_sock(sk); - sdp_dbg(sk, "%s lock_sock done\n", __func__); err = 0; if (!list_empty(&ssk->accept_queue)) break; @@ -800,8 +794,8 @@ static int sdp_ioctl(struct sock *sk, int cmd, unsigned long arg) else if (sock_flag(sk, SOCK_URGINLINE) || !ssk->urg_data || before(ssk->urg_seq, ssk->copied_seq) || - !before(ssk->urg_seq, ssk->rcv_nxt)) { - answ = ssk->rcv_nxt - ssk->copied_seq; + !before(ssk->urg_seq, rcv_nxt(ssk))) { + answ = rcv_nxt(ssk) - ssk->copied_seq; /* Subtract 1, if FIN is in queue. */ if (answ && !skb_queue_empty(&sk->sk_receive_queue)) @@ -925,7 +919,7 @@ int sdp_init_sock(struct sock *sk) sk->sk_route_caps |= NETIF_F_SG | NETIF_F_NO_CSUM; - skb_queue_head_init(&ssk->rx_backlog); + skb_queue_head_init(&ssk->rx_ctl_q); atomic_set(&ssk->mseq_ack, 0); @@ -1138,7 +1132,7 @@ static inline int poll_recv_cq(struct sock *sk) { int i; for (i = 0; i < recv_poll; ++i) { - if (!sdp_process_rx_q(sdp_sk(sk))) { + if (!skb_queue_empty(&sk->sk_receive_queue)) { ++recv_poll_hit; return 0; } @@ -1214,12 +1208,6 @@ static int sdp_recv_urg(struct sock *sk, long timeo, return -EAGAIN; } -static void sdp_rcv_space_adjust(struct sock *sk) -{ - sdp_post_recvs(sdp_sk(sk)); - sdp_post_sends(sdp_sk(sk), 0); -} - static unsigned int sdp_current_mss(struct sock *sk, int large_allowed) { /* TODO */ @@ -1875,7 +1863,7 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, lock_sock(sk); sdp_dbg_data(sk, "%s\n", __func__); -// sdp_prf(sk, skb, "Read from user"); + sdp_prf(sk, skb, "Read from user"); err = -ENOTCONN; if (sk->sk_state == TCP_LISTEN) @@ -1912,12 +1900,12 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, if (!skb) break; - if ((skb_transport_header(skb))[0] == SDP_MID_DISCONN) - goto found_fin_ok; + BUG_ON((skb_transport_header(skb))[0] != SDP_MID_DATA); if (before(*seq, TCP_SKB_CB(skb)->seq)) { - printk(KERN_INFO "recvmsg bug: copied %X " - "seq %X\n", *seq, TCP_SKB_CB(skb)->seq); + sdp_warn(sk, "recvmsg bug: copied %X seq %X\n", + *seq, TCP_SKB_CB(skb)->seq); + sdp_reset(sk); break; } @@ -2034,7 +2022,7 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, sdp_dbg_data(sk, "%s: done copied %d target %d\n", __func__, copied, target); - sdp_rcv_space_adjust(sk); + sdp_schedule_post_recvs(sdp_sk(sk)); skip_copy: if (ssk->urg_data && after(ssk->copied_seq, ssk->urg_seq)) ssk->urg_data = 0; @@ -2042,21 +2030,13 @@ skip_copy: continue; offset = 0; - if (!(flags & MSG_PEEK)) - sk_eat_skb(sk, skb, 0); - - continue; -found_fin_ok: - ++*seq; - if (!(flags & MSG_PEEK)) - sk_eat_skb(sk, skb, 0); - - break; + if (!(flags & MSG_PEEK)) { + skb_unlink(skb, &sk->sk_receive_queue); + __kfree_skb(skb); + } } while (len > 0); - release_sock(sk); - return copied; - + err = copied; out: release_sock(sk); return err; diff --git a/drivers/infiniband/ulp/sdp/sdp_proc.c b/drivers/infiniband/ulp/sdp/sdp_proc.c index ec97854..a4a1a4d 100644 --- a/drivers/infiniband/ulp/sdp/sdp_proc.c +++ b/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -39,9 +39,6 @@ #define PROC_SDP_STATS "sdpstats" #define PROC_SDP_PERF "sdpprf" -struct sdpprf_log sdpprf_log[SDPPRF_LOG_SIZE]; -int sdpprf_log_count = 0; - /* just like TCP fs */ struct sdp_seq_afinfo { struct module *owner; @@ -57,9 +54,6 @@ struct sdp_iter_state { struct seq_operations seq_ops; }; -//struct sdpprf sdpprf = { { 0 } }; - - static void *sdp_get_idx(struct seq_file *seq, loff_t pos) { int i = 0; @@ -148,7 +142,7 @@ static int sdp_seq_show(struct seq_file *seq, void *v) srcp = ntohs(inet_sk(sk)->sport); uid = sock_i_uid(sk); inode = sock_i_ino(sk); - rx_queue = sdp_sk(sk)->rcv_nxt - sdp_sk(sk)->copied_seq; + rx_queue = rcv_nxt(sdp_sk(sk)) - sdp_sk(sk)->copied_seq; tx_queue = sdp_sk(sk)->write_seq - sdp_sk(sk)->tx_ring.una_seq; sprintf(tmpbuf, "%4d: %08X:%04X %08X:%04X %5d %lu %08X:%08X %X", @@ -273,6 +267,7 @@ static int sdpstats_seq_show(struct seq_file *seq, void *v) seq_printf(seq, "rx_poll_miss \t\t: %d\n", sdpstats.rx_poll_miss); seq_printf(seq, "tx_poll_miss \t\t: %d\n", sdpstats.tx_poll_miss); + seq_printf(seq, "tx_poll_busy \t\t: %d\n", sdpstats.tx_poll_busy); seq_printf(seq, "tx_poll_hit \t\t: %d\n", sdpstats.tx_poll_hit); seq_printf(seq, "CQ stats:\n"); @@ -306,6 +301,10 @@ static struct file_operations sdpstats_fops = { #endif +#ifdef SDP_PROFILING +struct sdpprf_log sdpprf_log[SDPPRF_LOG_SIZE]; +int sdpprf_log_count = 0; + unsigned long long start_t = 0; static int sdpprf_show(struct seq_file *m, void *v) @@ -321,9 +320,9 @@ static int sdpprf_show(struct seq_file *m, void *v) t = l->time - start_t; nsec_rem = do_div(t, 1000000000); - seq_printf(m, "%-6d: [%5lu.%06lu] %-50s - [%d %d:%d] skb: %p %s:%d\n", + seq_printf(m, "%-6d: [%5lu.%06lu] %-50s - [%d{%d} %d:%d] skb: %p %s:%d\n", l->idx, (unsigned long)t, nsec_rem/1000, - l->msg, l->pid, l->sk_num, l->sk_dport, + l->msg, l->pid, l->cpu, l->sk_num, l->sk_dport, l->skb, l->func, l->line); out: return 0; @@ -408,6 +407,7 @@ static struct file_operations sdpprf_fops = { .release = seq_release, .write = sdpprf_write, }; +#endif /* SDP_PROFILING */ int __init sdp_proc_init(void) { @@ -437,10 +437,12 @@ int __init sdp_proc_init(void) #endif +#ifdef SDP_PROFILING sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; +#endif return 0; no_mem: @@ -464,7 +466,9 @@ void sdp_proc_unregister(void) #ifdef SDPSTATS_ON proc_net_remove(&init_net, PROC_SDP_STATS); #endif +#ifdef SDP_PROFILING proc_net_remove(&init_net, PROC_SDP_PERF); +#endif } #else /* CONFIG_PROC_FS */ diff --git a/drivers/infiniband/ulp/sdp/sdp_rx.c b/drivers/infiniband/ulp/sdp/sdp_rx.c index 05d8690..7d5b33e 100644 --- a/drivers/infiniband/ulp/sdp/sdp_rx.c +++ b/drivers/infiniband/ulp/sdp/sdp_rx.c @@ -149,9 +149,6 @@ static void sdp_fin(struct sock *sk) } } -/* lock_sock must be taken before calling this - since rx_ring.head is not - * protected (although being atomic - */ static int sdp_post_recv(struct sdp_sock *ssk) { struct sdp_buf *rx_req; @@ -170,7 +167,6 @@ static int sdp_post_recv(struct sdp_sock *ssk) gfp_t gfp_page; int ret = 0; - WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); /* Now, allocate and repost recv */ /* TODO: allocate from cache */ @@ -184,6 +180,7 @@ static int sdp_post_recv(struct sdp_sock *ssk) gfp_page = GFP_HIGHUSER; } + sdp_prf(&ssk->isk.sk, skb, "Posting skb"); /* FIXME */ BUG_ON(!skb); h = (struct sdp_bsdh *)skb->head; @@ -231,53 +228,90 @@ static int sdp_post_recv(struct sdp_sock *ssk) rx_wr.sg_list = ibsge; rx_wr.num_sge = frags + 1; rc = ib_post_recv(ssk->qp, &rx_wr, &bad_wr); - SDPSTATS_COUNTER_INC(post_recv); atomic_inc(&ssk->rx_ring.head); if (unlikely(rc)) { sdp_warn(&ssk->isk.sk, "ib_post_recv failed with status %d\n", rc); + + lock_sock(&ssk->isk.sk); sdp_reset(&ssk->isk.sk); + release_sock(&ssk->isk.sk); + ret = -1; } + SDPSTATS_COUNTER_INC(post_recv); atomic_add(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); return ret; } -/* lock_sock must be taken before calling this */ -static void _sdp_post_recvs(struct sdp_sock *ssk) +static inline int sdp_post_recvs_needed(struct sdp_sock *ssk) { struct sock *sk = &ssk->isk.sk; int scale = ssk->rcvbuf_scale; - - WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); - - if (unlikely(!ssk->id || ((1 << sk->sk_state) & - (TCPF_CLOSE | TCPF_TIME_WAIT)))) { - return; - } + int buffer_size = SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE; + unsigned long max_bytes; if (top_mem_usage && (top_mem_usage * 0x100000) < atomic_read(&sdp_current_mem_usage) * PAGE_SIZE) scale = 1; - while ((likely(ring_posted(ssk->rx_ring) < SDP_RX_SIZE) && - (ring_posted(ssk->rx_ring) - SDP_MIN_TX_CREDITS) * - (SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE) + - ssk->rcv_nxt - ssk->copied_seq < sk->sk_rcvbuf * scale) || - unlikely(ring_posted(ssk->rx_ring) < SDP_MIN_TX_CREDITS)) { - if (sdp_post_recv(ssk)) - break; + max_bytes = sk->sk_rcvbuf * scale; + + if (unlikely(ring_posted(ssk->rx_ring) >= SDP_RX_SIZE)) { + sdp_prf(sk, NULL, "rx ring is full"); + return 0; } + + if (likely(ring_posted(ssk->rx_ring) > SDP_MIN_TX_CREDITS)) { + unsigned long bytes_in_process = + (ring_posted(ssk->rx_ring) - SDP_MIN_TX_CREDITS) * buffer_size; + bytes_in_process += rcv_nxt(ssk) - ssk->copied_seq; + + if (bytes_in_process >= max_bytes) { + sdp_prf(sk, NULL, "bytes_in_process:%ld > max_bytes:%ld", + bytes_in_process, max_bytes); + return 0; + } + } + + return 1; } -void sdp_post_recvs(struct sdp_sock *ssk) +static inline void sdp_post_recvs(struct sdp_sock *ssk) { - unsigned long flags; + int rc = 0; +again: + ssk->posts_in_process = 1; - rx_ring_lock(ssk, flags); - _sdp_post_recvs(ssk); - rx_ring_unlock(ssk, flags); + do { + if (!sdp_post_recvs_needed(ssk)) + break; + + rc = sdp_post_recv(ssk); + } while (!rc); + + sk_stream_mem_reclaim(&ssk->isk.sk); + + ssk->posts_in_process = 0; + if (sdp_post_recvs_needed(ssk)) + goto again; +} + +void sdp_schedule_post_recvs(struct sdp_sock *ssk) +{ + struct sock *sk = &ssk->isk.sk; + + WARN_ON_LOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); + + if (unlikely(!ssk->id || ((1 << sk->sk_state) & + (TCPF_CLOSE | TCPF_TIME_WAIT)))) { + sdp_prf(sk, NULL, "socket is closed - not posting"); + return; + } + + if (!ssk->posts_in_process && sdp_post_recvs_needed(ssk)) + queue_work(rx_comp_wq, &ssk->rx_comp_work); } static inline struct sk_buff *sdp_sock_queue_rcv_skb(struct sock *sk, @@ -285,7 +319,6 @@ static inline struct sk_buff *sdp_sock_queue_rcv_skb(struct sock *sk, { int skb_len; struct sdp_sock *ssk = sdp_sk(sk); - struct sk_buff *tail = NULL; /* not needed since sk_rmem_alloc is not currently used * TODO - remove this? @@ -293,16 +326,10 @@ static inline struct sk_buff *sdp_sock_queue_rcv_skb(struct sock *sk, skb_len = skb->len; - TCP_SKB_CB(skb)->seq = ssk->rcv_nxt; - ssk->rcv_nxt += skb_len; + TCP_SKB_CB(skb)->seq = rcv_nxt(ssk); + atomic_add(skb_len, &ssk->rcv_nxt); - if (likely(skb_len && (tail = skb_peek_tail(&sk->sk_receive_queue))) && - unlikely(skb_tailroom(tail) >= skb_len)) { - skb_copy_bits(skb, 0, skb_put(tail, skb_len), skb_len); - __kfree_skb(skb); - skb = tail; - } else - skb_queue_tail(&sk->sk_receive_queue, skb); + skb_queue_tail(&sk->sk_receive_queue, skb); if (!sock_flag(sk, SOCK_DEAD)) sk->sk_data_ready(sk, skb_len); @@ -412,71 +439,97 @@ static struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id) return skb; } -/* this must be called while sock_lock is taken */ -static int sdp_process_rx_skb(struct sdp_sock *ssk, struct sk_buff *skb) +/* socket lock should be taken before calling this */ +static int sdp_process_rx_ctl_skb(struct sdp_sock *ssk, struct sk_buff *skb) { + struct sdp_bsdh *h = (struct sdp_bsdh *)skb->data; struct sock *sk = &ssk->isk.sk; - int frags; - struct sdp_bsdh *h; - int pagesz, i; - - h = (struct sdp_bsdh *)skb->data; - - frags = skb_shinfo(skb)->nr_frags; - pagesz = PAGE_ALIGN(skb->data_len); - skb_shinfo(skb)->nr_frags = pagesz / PAGE_SIZE; - - for (i = skb_shinfo(skb)->nr_frags; - i < frags; ++i) { - put_page(skb_shinfo(skb)->frags[i].page); - skb->truesize -= PAGE_SIZE; - } - - if (unlikely(h->flags & SDP_OOB_PEND)) - sk_send_sigurg(sk); - - skb_pull(skb, sizeof(struct sdp_bsdh)); switch (h->mid) { case SDP_MID_DATA: - if (unlikely(skb->len <= 0)) { - __kfree_skb(skb); - break; - } + WARN_ON(!(sk->sk_shutdown & RCV_SHUTDOWN)); - if (unlikely(sk->sk_shutdown & RCV_SHUTDOWN)) { - /* got data in RCV_SHUTDOWN */ - if (sk->sk_state == TCP_FIN_WAIT1) { - /* go into abortive close */ - sdp_exch_state(sk, TCPF_FIN_WAIT1, - TCP_TIME_WAIT); + sdp_warn(sk, "DATA after socket rcv was shutdown\n"); - sk->sk_prot->disconnect(sk, 0); - } + /* got data in RCV_SHUTDOWN */ + if (sk->sk_state == TCP_FIN_WAIT1) { + sdp_warn(sk, "socket in shutdown, state = FIN_WAIT1 and got data\n"); + /* go into abortive close */ + sdp_exch_state(sk, TCPF_FIN_WAIT1, + TCP_TIME_WAIT); - __kfree_skb(skb); - break; + sk->sk_prot->disconnect(sk, 0); } - skb = sdp_sock_queue_rcv_skb(sk, skb); - if (unlikely(h->flags & SDP_OOB_PRES)) - sdp_urg(ssk, skb); + break; case SDP_MID_DISCONN: - __kfree_skb(skb); + sdp_dbg_data(sk, "Handling RX disconnect\n"); sdp_fin(sk); break; case SDP_MID_CHRCVBUF: + sdp_dbg_data(sk, "Handling RX CHRCVBUF\n"); sdp_handle_resize_request(ssk, (struct sdp_chrecvbuf *)skb->data); - __kfree_skb(skb); break; case SDP_MID_CHRCVBUF_ACK: + sdp_dbg_data(sk, "Handling RX CHRCVBUF_ACK\n"); sdp_handle_resize_ack(ssk, (struct sdp_chrecvbuf *)skb->data); - __kfree_skb(skb); break; default: /* TODO: Handle other messages */ - printk(KERN_WARNING "SDP: FIXME MID %d\n", h->mid); + sdp_warn(sk, "SDP: FIXME MID %d\n", h->mid); + } + + __kfree_skb(skb); + + return 0; +} + +static int sdp_process_rx_skb(struct sdp_sock *ssk, struct sk_buff *skb) +{ + struct sock *sk = &ssk->isk.sk; + int frags; + struct sdp_bsdh *h; + int pagesz, i; + unsigned long mseq_ack; + int credits_before; + + h = (struct sdp_bsdh *)skb->data; + + SDPSTATS_HIST_LINEAR(credits_before_update, tx_credits(ssk)); + + mseq_ack = ntohl(h->mseq_ack); + credits_before = tx_credits(ssk); + atomic_set(&ssk->tx_ring.credits, mseq_ack - ring_head(ssk->tx_ring) + 1 + + ntohs(h->bufs)); + if (mseq_ack >= ssk->nagle_last_unacked) + ssk->nagle_last_unacked = 0; + + sdp_prf(&ssk->isk.sk, skb, "RX %s +%d c:%d->%d mseq:%d ack:%d", + mid2str(h->mid), ntohs(h->bufs), credits_before, + tx_credits(ssk), ntohl(h->mseq), ntohl(h->mseq_ack)); + + frags = skb_shinfo(skb)->nr_frags; + pagesz = PAGE_ALIGN(skb->data_len); + skb_shinfo(skb)->nr_frags = pagesz / PAGE_SIZE; + + for (i = skb_shinfo(skb)->nr_frags; i < frags; ++i) { + put_page(skb_shinfo(skb)->frags[i].page); + skb->truesize -= PAGE_SIZE; + } + +/* if (unlikely(h->flags & SDP_OOB_PEND)) + sk_send_sigurg(sk);*/ + + if (h->mid != SDP_MID_DATA || unlikely(sk->sk_shutdown & RCV_SHUTDOWN)) { + sdp_warn(sk, "Control skb - queing to control queue\n"); + skb_queue_tail(&ssk->rx_ctl_q, skb); + return 0; + } + + skb_pull(skb, sizeof(struct sdp_bsdh)); + + if (unlikely(skb->len <= 0)) { __kfree_skb(skb); } @@ -489,8 +542,7 @@ static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) struct sk_buff *skb; struct sdp_bsdh *h; struct sock *sk = &ssk->isk.sk; - int credits_before; - unsigned long mseq_ack; + int mseq; skb = sdp_recv_completion(ssk, wc->wr_id); if (unlikely(!skb)) @@ -530,23 +582,12 @@ static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) h = (struct sdp_bsdh *)skb->data; SDP_DUMP_PACKET(&ssk->isk.sk, "RX", skb, h); skb_reset_transport_header(skb); - atomic_set(&ssk->mseq_ack, ntohl(h->mseq)); - if (mseq_ack(ssk) != (int)wc->wr_id) - printk(KERN_WARNING "SDP BUG! mseq %d != wrid %d\n", - mseq_ack(ssk), (int)wc->wr_id); - SDPSTATS_HIST_LINEAR(credits_before_update, tx_credits(ssk)); - - mseq_ack = ntohl(h->mseq_ack); - credits_before = tx_credits(ssk); - atomic_set(&ssk->tx_ring.credits, mseq_ack - ring_head(ssk->tx_ring) + 1 + - ntohs(h->bufs)); - if (mseq_ack >= ssk->nagle_last_unacked) - ssk->nagle_last_unacked = 0; - - sdp_prf(&ssk->isk.sk, skb, "RX %s bufs=%d c before:%d after:%d " - "mseq:%d, ack:%d", mid2str(h->mid), ntohs(h->bufs), credits_before, - tx_credits(ssk), ntohl(h->mseq), ntohl(h->mseq_ack)); + mseq = ntohl(h->mseq); + atomic_set(&ssk->mseq_ack, mseq); + if (mseq != (int)wc->wr_id) + sdp_warn(sk, "SDP BUG! mseq %d != wrid %d\n", + mseq, (int)wc->wr_id); return skb; } @@ -576,7 +617,6 @@ static void sdp_bzcopy_write_space(struct sdp_sock *ssk) } /* only from interrupt. - * drain rx cq into rx_backlog queue */ static int sdp_poll_rx_cq(struct sdp_sock *ssk) { struct ib_cq *cq = ssk->rx_ring.cq; @@ -596,7 +636,8 @@ static int sdp_poll_rx_cq(struct sdp_sock *ssk) skb = sdp_process_rx_wc(ssk, wc); if (!skb) continue; - skb_queue_tail(&ssk->rx_backlog, skb); + + sdp_process_rx_skb(ssk, skb); wc_processed++; } } while (n == SDP_NUM_WC); @@ -607,69 +648,55 @@ static int sdp_poll_rx_cq(struct sdp_sock *ssk) return wc_processed; } -int sdp_process_rx_q(struct sdp_sock *ssk) +void sdp_rx_comp_work(struct work_struct *work) { - struct sk_buff *skb; + struct sdp_sock *ssk = container_of(work, struct sdp_sock, rx_comp_work); struct sock *sk = &ssk->isk.sk; - unsigned long flags; + int xmit_poll_force; + struct sk_buff *skb; - if (!ssk->rx_backlog.next || !ssk->rx_backlog.prev) { - sdp_warn(&ssk->isk.sk, "polling a zeroed rx_backlog!!!! %p\n", &ssk->rx_backlog); - return 0; - } + sdp_prf(sk, NULL, "%s", __func__); + + if (unlikely(!ssk->rx_ring.cq)) + return; - if (skb_queue_empty(&ssk->rx_backlog)) { - SDPSTATS_COUNTER_INC(rx_poll_miss); - return -EAGAIN; + if (unlikely(!ssk->poll_cq)) { + struct rdma_cm_id *id = ssk->id; + sdp_warn(sk, "poll cq is 0. socket was reset or wasn't initialized\n"); + if (id && id->qp) + rdma_notify(id, RDMA_CM_EVENT_ESTABLISHED); + return; } - /* update credits */ - sdp_post_sends(ssk, 0); + sdp_post_recvs(ssk); - spin_lock_irqsave(&ssk->rx_backlog.lock, flags); - while ((skb = __skb_dequeue(&ssk->rx_backlog))) { - sdp_process_rx_skb(ssk, skb); + if (!(sk->sk_write_pending && (tx_credits(ssk) > SDP_MIN_TX_CREDITS)) && + !credit_update_needed(ssk) && + skb_queue_empty(&ssk->rx_ctl_q)) { + sdp_prf(&ssk->isk.sk, NULL, "only post recvs"); + return; } - spin_unlock_irqrestore(&ssk->rx_backlog.lock, flags); - if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) - sk_stream_write_space(&ssk->isk.sk); + lock_sock(sk); - return 0; -} + sdp_post_sends(ssk, 0); -static void sdp_rx_comp_work(struct work_struct *work) -{ - struct sdp_sock *ssk = container_of(work, struct sdp_sock, rx_comp_work); - struct sock *sk = &ssk->isk.sk; - struct ib_cq *rx_cq; + while ((skb = skb_dequeue(&ssk->rx_ctl_q))) { + sdp_process_rx_ctl_skb(ssk, skb); + } - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) - goto out; + sk_stream_mem_reclaim(sk); - if (unlikely(!ssk->poll_cq)) { - struct rdma_cm_id *id = ssk->id; - sdp_warn(sk, "poll cq is 0. socket was reset or wasn't initialized\n"); - if (id && id->qp) - rdma_notify(id, RDMA_CM_EVENT_ESTABLISHED); - goto out; - } + xmit_poll_force = sk->sk_write_pending && (tx_credits(ssk) > SDP_MIN_TX_CREDITS); - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); - sk_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) - goto out; - - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); + if (credit_update_needed(ssk) || xmit_poll_force) { + /* if has pending tx because run out of tx_credits - xmit it */ + sdp_prf(sk, NULL, "Processing to free pending sends"); + sdp_xmit_poll(ssk, xmit_poll_force); + sdp_prf(sk, NULL, "Sending credit update if needed"); + sdp_post_sends(ssk, 0); + } -out: release_sock(sk); } @@ -679,6 +706,7 @@ static void sdp_rx_irq(struct ib_cq *cq, void *cq_context) struct sdp_sock *ssk = sdp_sk(sk); unsigned long flags; int wc_processed = 0; + int credits_before; sdp_dbg_data(&ssk->isk.sk, "rx irq called\n"); @@ -690,6 +718,8 @@ static void sdp_rx_irq(struct ib_cq *cq, void *cq_context) rx_ring_lock(ssk, flags); + credits_before = tx_credits(ssk); + if (unlikely(!ssk->poll_cq)) sdp_warn(sk, "poll cq is 0. socket was reset or wasn't initialized\n"); @@ -703,15 +733,11 @@ static void sdp_rx_irq(struct ib_cq *cq, void *cq_context) sdp_prf(&ssk->isk.sk, NULL, "processed %d", wc_processed); if (wc_processed) { - _sdp_post_recvs(ssk); + sdp_prf(&ssk->isk.sk, NULL, "credits: %d -> %d", + credits_before, tx_credits(ssk)); - /* Best was to send credit update from here */ -/* sdp_post_credits(ssk); */ - - /* issue sdp_rx_comp_work() */ queue_work(rx_comp_wq, &ssk->rx_comp_work); } - sdp_arm_rx_cq(sk); out: @@ -764,7 +790,7 @@ int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device) } rx_cq = ib_create_cq(device, sdp_rx_irq, sdp_rx_cq_event_handler, - &ssk->isk.sk, SDP_RX_SIZE, 0); + &ssk->isk.sk, SDP_RX_SIZE, IB_CQ_VECTOR_LEAST_ATTACHED); if (IS_ERR(rx_cq)) { rc = PTR_ERR(rx_cq); diff --git a/drivers/infiniband/ulp/sdp/sdp_tx.c b/drivers/infiniband/ulp/sdp/sdp_tx.c index a7d16fe..95e3c82 100644 --- a/drivers/infiniband/ulp/sdp/sdp_tx.c +++ b/drivers/infiniband/ulp/sdp/sdp_tx.c @@ -287,15 +287,19 @@ static void sdp_poll_tx_timeout(unsigned long data) sdp_dbg_data(&ssk->isk.sk, "Polling tx cq. inflight=%d\n", (u32) ring_posted(ssk->tx_ring)); + sdp_prf(&ssk->isk.sk, NULL, "%s. inflight=%d", __func__, + (u32) ring_posted(ssk->tx_ring)); + /* Only process if the socket is not in use */ bh_lock_sock(sk); if (sock_owned_by_user(sk)) { mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); sdp_dbg_data(&ssk->isk.sk, "socket is busy - trying later\n"); + SDPSTATS_COUNTER_INC(tx_poll_busy); goto out; } - if (sk->sk_state == TCP_CLOSE) + if (unlikely(sk->sk_state == TCP_CLOSE)) goto out; wc_processed = sdp_process_tx_cq(ssk); diff --git a/kernel_patches/attic/backport/2.6.16/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.16/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.16/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.16/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.16/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.16/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.16/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.16/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.16_sles10/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.16_sles10/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.16_sles10/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.16_sles10/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.16_sles10/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.16_sles10/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.16_sles10/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.16_sles10/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.16_sles10_sp1/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.17/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.17/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.17/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.17/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.17/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.17/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.17/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.17/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.18-EL5.1/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.18_FC6/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.18_FC6/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.18_FC6/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.18_FC6/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.18_FC6/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.18_FC6/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.18_FC6/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.18_FC6/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.18_suse10_2/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.22_suse10_3/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.9_U4/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.9_U4/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.9_U4/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.9_U4/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.9_U4/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.9_U4/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.9_U4/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.9_U4/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/attic/backport/2.6.9_U5/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/attic/backport/2.6.9_U5/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/attic/backport/2.6.9_U5/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/attic/backport/2.6.9_U5/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/attic/backport/2.6.9_U5/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/attic/backport/2.6.9_U5/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/attic/backport/2.6.9_U5/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/attic/backport/2.6.9_U5/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.16_sles10_sp2/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.18-EL5.2/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.18-EL5.2/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.18-EL5.2/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.18-EL5.2/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.18-EL5.2/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.18-EL5.2/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.18-EL5.2/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.18-EL5.2/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.18-EL5.3/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.18-EL5.3/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.18-EL5.3/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.18-EL5.3/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.18-EL5.3/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.18-EL5.3/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.18-EL5.3/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.18-EL5.3/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.18/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.18/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.18/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.18/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.18/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.18/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.18/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.18/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.19/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.19/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.19/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.19/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.19/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.19/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.19/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.19/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.20/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.20/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.20/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.20/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.20/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.20/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.20/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.20/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.21/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.21/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.21/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.21/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.21/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.21/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.21/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.21/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.22/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.22/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.22/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.22/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.22/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.22/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.22/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.22/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.23/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.23/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.23/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.23/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.23/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.23/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.23/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.23/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.24/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.24/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.24/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.24/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.9_U6/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.9_U6/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.9_U6/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.9_U6/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.9_U6/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.9_U6/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.9_U6/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.9_U6/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ diff --git a/kernel_patches/backport/2.6.9_U7/sdp_0090_revert_to_2_6_24.patch b/kernel_patches/backport/2.6.9_U7/sdp_0090_revert_to_2_6_24.patch index 23d526a..2272473 100644 --- a/kernel_patches/backport/2.6.9_U7/sdp_0090_revert_to_2_6_24.patch +++ b/kernel_patches/backport/2.6.9_U7/sdp_0090_revert_to_2_6_24.patch @@ -9,7 +9,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp.h +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp.h -@@ -594,30 +594,4 @@ static inline char *mid2str(int mid) +@@ -584,30 +584,4 @@ static inline char *mid2str(int mid) return mid2str[mid]; } @@ -44,7 +44,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c -@@ -170,7 +170,7 @@ int sdp_post_credits(struct sdp_sock *ss +@@ -174,7 +174,7 @@ int sdp_post_credits(struct sdp_sock *ss if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; @@ -53,7 +53,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); if (!skb) -@@ -224,7 +224,7 @@ void _sdp_post_sends(const char *func, i +@@ -228,7 +228,7 @@ void _sdp_post_sends(const char *func, i sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -62,7 +62,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*resp_size), gfp_page); -@@ -259,7 +259,7 @@ void _sdp_post_sends(const char *func, i +@@ -263,7 +263,7 @@ void _sdp_post_sends(const char *func, i ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; @@ -71,7 +71,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh) + sizeof(*req_size), gfp_page); -@@ -282,7 +282,7 @@ void _sdp_post_sends(const char *func, i +@@ -286,7 +286,7 @@ void _sdp_post_sends(const char *func, i likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -80,7 +80,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_bcopy.c sizeof(struct sdp_bsdh), GFP_KERNEL); /* FIXME */ -@@ -301,7 +301,7 @@ void _sdp_post_sends(const char *func, i +@@ -305,7 +305,7 @@ void _sdp_post_sends(const char *func, i !ssk->isk.sk.sk_send_head && tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; @@ -93,7 +93,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c +++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -510,7 +510,7 @@ static void sdp_close(struct sock *sk, l +@@ -508,7 +508,7 @@ static void sdp_close(struct sock *sk, l __kfree_skb(skb); } @@ -102,7 +102,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c /* As outlined in draft-ietf-tcpimpl-prob-03.txt, section * 3.10, we send a RST here because data was lost. To -@@ -1241,7 +1241,7 @@ static inline void sdp_mark_urg(struct s +@@ -1229,7 +1229,7 @@ static inline void sdp_mark_urg(struct s { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -111,7 +111,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c } } -@@ -1258,8 +1258,7 @@ static inline void skb_entail(struct soc +@@ -1246,8 +1246,7 @@ static inline void skb_entail(struct soc { skb_header_release(skb); __skb_queue_tail(&sk->sk_write_queue, skb); @@ -121,7 +121,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk->sk_send_head) sk->sk_send_head = skb; if (ssk->nonagle & TCP_NAGLE_PUSH) -@@ -1432,7 +1431,7 @@ static inline int sdp_bcopy_get(struct s +@@ -1420,7 +1419,7 @@ static inline int sdp_bcopy_get(struct s if (copy > PAGE_SIZE - off) copy = PAGE_SIZE - off; @@ -130,7 +130,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; if (!page) { -@@ -1508,7 +1507,7 @@ static inline int sdp_bzcopy_get(struct +@@ -1496,7 +1495,7 @@ static inline int sdp_bzcopy_get(struct if (left <= this_page) this_page = left; @@ -139,7 +139,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c return SDP_DO_WAIT_MEM; get_page(bz->pages[bz->cur_page]); -@@ -1709,8 +1708,8 @@ new_segment: +@@ -1697,8 +1696,8 @@ new_segment: goto wait_for_sndbuf; } @@ -150,7 +150,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c if (!skb) goto wait_for_memory; -@@ -1743,7 +1742,7 @@ new_segment: +@@ -1731,7 +1730,7 @@ new_segment: /* OOB data byte should be the last byte of the data payload */ @@ -159,7 +159,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c !(flags & MSG_OOB)) { sdp_mark_push(ssk, skb); goto new_segment; -@@ -1826,7 +1825,7 @@ do_fault: +@@ -1814,7 +1813,7 @@ do_fault: if (sk->sk_send_head == skb) sk->sk_send_head = NULL; __skb_unlink(skb, &sk->sk_write_queue); @@ -181,7 +181,7 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); -@@ -175,11 +175,11 @@ static int sdp_post_recv(struct sdp_sock +@@ -171,11 +171,11 @@ static int sdp_post_recv(struct sdp_sock /* TODO: allocate from cache */ if (unlikely(ssk->isk.sk.sk_allocation)) { @@ -195,15 +195,6 @@ Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_rx.c GFP_KERNEL); gfp_page = GFP_HIGHUSER; } -@@ -660,7 +660,7 @@ static void sdp_rx_comp_work(struct work - sdp_process_rx_q(ssk); - sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ - release_sock(sk); -- sk_mem_reclaim(sk); -+ sk_stream_mem_reclaim(sk); - lock_sock(sk); - rx_cq = ssk->rx_ring.cq; - if (unlikely(!rx_cq)) Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_tx.c =================================================================== --- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_tx.c diff --git a/kernel_patches/backport/2.6.9_U7/sdp_0100_revert_to_2_6_23.patch b/kernel_patches/backport/2.6.9_U7/sdp_0100_revert_to_2_6_23.patch index 608dd83..90d8e67 100644 --- a/kernel_patches/backport/2.6.9_U7/sdp_0100_revert_to_2_6_23.patch +++ b/kernel_patches/backport/2.6.9_U7/sdp_0100_revert_to_2_6_23.patch @@ -2,11 +2,11 @@ drivers/infiniband/ulp/sdp/sdp_main.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_main.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c -@@ -2238,16 +2238,13 @@ static struct proto_ops sdp_proto_ops = +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_main.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_main.c +@@ -2218,16 +2218,13 @@ static struct proto_ops sdp_proto_ops = .sendpage = sock_no_sendpage, }; @@ -24,7 +24,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (sock->type != SOCK_STREAM) { sdp_warn(NULL, "SDP: unsupported type %d.\n", sock->type); return -ESOCKTNOSUPPORT; -@@ -2259,7 +2256,7 @@ static int sdp_create_socket(struct net +@@ -2239,7 +2236,7 @@ static int sdp_create_socket(struct net return -EPROTONOSUPPORT; } @@ -33,10 +33,10 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_main.c if (!sk) { sdp_warn(NULL, "SDP: failed to allocate socket.\n"); return -ENOMEM; -Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +Index: ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== ---- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/ulp/sdp/sdp_proc.c -+++ ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c +--- ofed_kernel.orig/drivers/infiniband/ulp/sdp/sdp_proc.c ++++ ofed_kernel/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -421,7 +421,7 @@ int __init sdp_proc_init(void) sdp_seq_afinfo.seq_fops->llseek = seq_lseek; sdp_seq_afinfo.seq_fops->release = seq_release_private; @@ -46,7 +46,7 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c sdp_seq_afinfo.seq_fops); if (p) p->data = &sdp_seq_afinfo; -@@ -430,14 +430,14 @@ int __init sdp_proc_init(void) +@@ -430,7 +430,7 @@ int __init sdp_proc_init(void) #ifdef SDPSTATS_ON @@ -55,15 +55,16 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c S_IRUGO | S_IWUGO, &sdpstats_fops); if (!sdpstats) goto no_mem; - +@@ -438,7 +438,7 @@ int __init sdp_proc_init(void) #endif + #ifdef SDP_PROFILING - sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + sdpprf = proc_net_fops_create(PROC_SDP_PERF, S_IRUGO | S_IWUGO, &sdpprf_fops); if (!sdpprf) goto no_mem; -@@ -445,26 +445,26 @@ int __init sdp_proc_init(void) +@@ -447,27 +447,27 @@ int __init sdp_proc_init(void) return 0; no_mem: if (sdpprf) @@ -91,8 +92,9 @@ Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/ulp/sdp/sdp_proc.c - proc_net_remove(&init_net, PROC_SDP_STATS); + proc_net_remove(PROC_SDP_STATS); #endif + #ifdef SDP_PROFILING - proc_net_remove(&init_net, PROC_SDP_PERF); + proc_net_remove(PROC_SDP_PERF); + #endif } - #else /* CONFIG_PROC_FS */ -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:44 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:44 +0300 Subject: [ofa-general] [PATCH] sdp: process RX CQ from interrupt In-Reply-To: <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> Queue RX packets in interrupt context - reduce unnecessary context switches. Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 140 +++++++---- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 66 ++---- drivers/infiniband/ulp/sdp/sdp_cma.c | 114 ++------- drivers/infiniband/ulp/sdp/sdp_main.c | 124 +++++----- drivers/infiniband/ulp/sdp/sdp_proc.c | 13 +- drivers/infiniband/ulp/sdp/sdp_rx.c | 442 ++++++++++++++++++++++--------- drivers/infiniband/ulp/sdp/sdp_tx.c | 160 ++++++++---- 7 files changed, 614 insertions(+), 445 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index 5e78282..57265aa 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -7,11 +7,12 @@ #include /* For urgent data flags */ #include +#undef SDP_LOCKS_CHECK #define SDPSTATS_ON -#undef CONFIG_INFINIBAND_SDP_DEBUG_DATA +#define SDP_PROFILING #define _sdp_printk(func, line, level, sk, format, arg...) \ - printk(level "%s:%d sdp_sock(%d %d:%d): " format, \ + printk(level "%s:%d sdp_sock(%5d %d:%d): " format, \ func, line, \ current->pid, \ (sk) ? inet_sk(sk)->num : -1, \ @@ -22,6 +23,34 @@ sdp_printk(KERN_WARNING, sk, format , ## arg) +#ifdef SDP_LOCKS_CHECK +#define WARN_ON_UNLOCKED(sk, l) do {\ + if (unlikely(!spin_is_locked(l))) { \ + sdp_warn(sk, "lock " #l " should be locked\n"); \ + WARN_ON(1); \ + } \ +} while (0) + +#define WARN_ON_LOCKED(sk, l) do {\ + if (unlikely(spin_is_locked(l))) { \ + sdp_warn(sk, "lock " #l " should be unlocked\n"); \ + WARN_ON(1); \ + } \ +} while (0) +#else +#define WARN_ON_UNLOCKED(sk, l) +#define WARN_ON_LOCKED(sk, l) +#endif + +#define rx_ring_lock(ssk, f) do { \ + spin_lock_irqsave(&ssk->rx_ring.lock, f); \ +} while (0) + +#define rx_ring_unlock(ssk, f) do { \ + spin_unlock_irqrestore(&ssk->rx_ring.lock, f); \ +} while (0) + +#ifdef SDP_PROFILING struct sk_buff; struct sdpprf_log { int idx; @@ -37,7 +66,7 @@ struct sdpprf_log { int line; }; -#define SDPPRF_LOG_SIZE 0x10000 /* must be a power of 2 */ +#define SDPPRF_LOG_SIZE 0x20000 /* must be a power of 2 */ extern struct sdpprf_log sdpprf_log[SDPPRF_LOG_SIZE]; extern int sdpprf_log_count; @@ -48,7 +77,6 @@ static inline unsigned long long current_nsec(void) getnstimeofday(&tv); return tv.tv_sec * NSEC_PER_SEC + tv.tv_nsec; } -#if 1 #define sdp_prf(sk, s, format, arg...) ({ \ struct sdpprf_log *l = &sdpprf_log[sdpprf_log_count++ & (SDPPRF_LOG_SIZE - 1)]; \ l->idx = sdpprf_log_count - 1; \ @@ -66,16 +94,6 @@ static inline unsigned long long current_nsec(void) #define sdp_prf(sk, s, format, arg...) #endif -#if 0 -#if 1 -#define sdp_prf_rx(sk, s, format, arg...) sdp_prf(sk, s, format, ## arg) -#define sdp_prf_tx(sk, s, format, arg...) -#else -#define sdp_prf_rx(sk, s, format, arg...) -#define sdp_prf_tx(sk, s, format, arg...) sdp_prf(sk, s, format, ## arg) -#endif -#endif - #ifdef CONFIG_INFINIBAND_SDP_DEBUG extern int sdp_debug_level; @@ -125,7 +143,6 @@ extern int sdp_data_debug_level; } while (0) #else #define sdp_dbg_data(priv, format, arg...) -// do { (void) (priv); } while (0) #define SDP_DUMP_PACKET(sk, str, skb, h) #endif @@ -300,23 +317,36 @@ struct sdp_buf { u64 mapping[SDP_MAX_SEND_SKB_FRAGS + 1]; }; +#define ring_head(ring) (atomic_read(&(ring).head)) +#define ring_tail(ring) (atomic_read(&(ring).tail)) +#define ring_posted(ring) (ring_head(ring) - ring_tail(ring)) struct sdp_tx_ring { struct sdp_buf *buffer; - unsigned head; - unsigned tail; + atomic_t head; + atomic_t tail; + struct ib_cq *cq; int una_seq; - unsigned credits; + atomic_t credits; +#define tx_credits(ssk) (atomic_read(&ssk->tx_ring.credits)) struct timer_list timer; u16 poll_cnt; +}; + +struct sdp_rx_ring { + struct sdp_buf *buffer; + atomic_t head; + atomic_t tail; struct ib_cq *cq; + + spinlock_t lock; }; static inline int sdp_tx_ring_slots_left(struct sdp_tx_ring *tx_ring) { - return SDP_TX_SIZE - (tx_ring->head - tx_ring->tail); + return SDP_TX_SIZE - ring_posted(*tx_ring); } struct sdp_chrecvbuf { @@ -329,6 +359,7 @@ struct sdp_sock { struct list_head sock_list; struct list_head accept_queue; struct list_head backlog_queue; + struct sk_buff_head rx_backlog; struct sock *parent; struct work_struct rx_comp_work; @@ -362,32 +393,27 @@ struct sdp_sock { int sdp_disconnect; int destruct_in_process; - + struct sdp_rx_ring rx_ring; + struct sdp_tx_ring tx_ring; /* Data below will be reset on error */ struct rdma_cm_id *id; struct ib_device *ib_device; /* SDP specific */ - struct ib_recv_wr rx_wr; - unsigned rx_head; - unsigned rx_tail; - unsigned mseq_ack; + atomic_t mseq_ack; +#define mseq_ack(ssk) (atomic_read(&ssk->mseq_ack)) unsigned max_bufs; /* Initial buffers offered by other side */ unsigned min_bufs; /* Low water mark to wake senders */ - int remote_credits; + atomic_t remote_credits; +#define remote_credits(ssk) (atomic_read(&ssk->remote_credits)) int poll_cq; /* rdma specific */ struct ib_qp *qp; - struct ib_cq *rx_cq; struct ib_mr *mr; - struct sdp_buf *rx_ring; - struct sdp_tx_ring tx_ring; - struct ib_send_wr tx_wr; - /* SDP slow start */ int rcvbuf_scale; /* local recv buf scale for each socket */ int sent_request_head; /* mark the tx_head of the last send resize @@ -402,8 +428,6 @@ struct sdp_sock { /* BZCOPY data */ int zcopy_thresh; - - struct ib_sge ibsge[SDP_MAX_SEND_SKB_FRAGS + 1]; }; /* Context used for synchronous zero copy bcopy (BZCOY) */ @@ -499,13 +523,6 @@ static inline void sdp_set_error(struct sock *sk, int err) sk->sk_error_report(sk); } -static inline void sdp_arm_rx_cq(struct sock *sk) -{ - sdp_dbg_data(sk, "ib_req_notify_cq on RX cq\n"); - - ib_req_notify_cq(sdp_sk(sk)->rx_cq, IB_CQ_NEXT_COMP); -} - #ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA void _dump_packet(const char *func, int line, struct sock *sk, char *str, struct sk_buff *skb, const struct sdp_bsdh *h); @@ -524,28 +541,53 @@ void sdp_remove_sock(struct sdp_sock *ssk); void sdp_remove_large_sock(struct sdp_sock *ssk); void sdp_post_keepalive(struct sdp_sock *ssk); void sdp_start_keepalive_timer(struct sock *sk); -void sdp_bzcopy_write_space(struct sdp_sock *ssk); int sdp_init_sock(struct sock *sk); int __init sdp_proc_init(void); void sdp_proc_unregister(void); +/* sdp_tx.c */ +int sdp_tx_ring_create(struct sdp_sock *ssk, struct ib_device *device); +void sdp_tx_ring_destroy(struct sdp_sock *ssk); int sdp_xmit_poll(struct sdp_sock *ssk, int force); -void sdp_tx_ring_purge(struct sdp_sock *ssk); void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid); void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonagle); #define sdp_post_sends(ssk, nonagle) _sdp_post_sends(__func__, __LINE__, ssk, nonagle) -void sdp_process_tx_wc_work(struct work_struct *work); -void sdp_poll_tx_cq(unsigned long data); -void _sdp_poll_tx_cq(unsigned long data); -void sdp_tx_irq(struct ib_cq *cq, void *cq_context); +/* sdp_rx.c */ +void sdp_rx_ring_init(struct sdp_sock *ssk); +int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device); +void sdp_rx_ring_destroy(struct sdp_sock *ssk); +int sdp_process_rx_q(struct sdp_sock *ssk); int sdp_resize_buffers(struct sdp_sock *ssk, u32 new_size); int sdp_init_buffers(struct sdp_sock *ssk, u32 new_size); -void sdp_rx_ring_purge(struct sdp_sock *ssk); void sdp_post_recvs(struct sdp_sock *ssk); -void sdp_rx_comp_work(struct work_struct *work); -int sdp_poll_rx_cq(struct sdp_sock *ssk); -void sdp_rx_irq(struct ib_cq *cq, void *cq_context); + +static inline void sdp_arm_rx_cq(struct sock *sk) +{ + sdp_prf(sk, NULL, "Arming RX cq"); + sdp_dbg_data(sk, "Arming RX cq\n"); + + ib_req_notify_cq(sdp_sk(sk)->rx_ring.cq, IB_CQ_NEXT_COMP); +} + +/* utilities */ +static inline char *mid2str(int mid) +{ +#define ENUM2STR(e) [e] = #e + static char *mid2str[] = { + ENUM2STR(SDP_MID_HELLO), + ENUM2STR(SDP_MID_HELLO_ACK), + ENUM2STR(SDP_MID_DISCONN), + ENUM2STR(SDP_MID_CHRCVBUF), + ENUM2STR(SDP_MID_CHRCVBUF_ACK), + ENUM2STR(SDP_MID_DATA), + }; + + if (mid >= ARRAY_SIZE(mid2str)) + return NULL; + + return mid2str[mid]; +} static inline struct sk_buff *sdp_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) { diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 5d4441a..b98171e 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -45,18 +45,9 @@ void _dump_packet(const char *func, int line, struct sock *sk, char *str, { int len = 0; char buf[256]; -#define ENUM2STR(e) [e] = #e - static char *mid2str[] = { - ENUM2STR(SDP_MID_HELLO), - ENUM2STR(SDP_MID_HELLO_ACK), - ENUM2STR(SDP_MID_DISCONN), - ENUM2STR(SDP_MID_CHRCVBUF), - ENUM2STR(SDP_MID_CHRCVBUF_ACK), - ENUM2STR(SDP_MID_DATA), - }; - len += snprintf(buf, 255-len, "skb: %p mid: %2x:%-20s flags: 0x%x bufs: %d " + len += snprintf(buf, 255-len, "%s skb: %p mid: %2x:%-20s flags: 0x%x bufs: %d " "len: %d mseq: %d mseq_ack: %d", - skb, h->mid, mid2str[h->mid], h->flags, + str, skb, h->mid, mid2str(h->mid), h->flags, ntohs(h->bufs), ntohl(h->len),ntohl(h->mseq), ntohl(h->mseq_ack)); @@ -117,7 +108,7 @@ static inline int sdp_nagle_off(struct sdp_sock *ssk, struct sk_buff *skb) return (ssk->nonagle & TCP_NAGLE_OFF) || skb->next != (struct sk_buff *)&ssk->isk.sk.sk_write_queue || skb->len + sizeof(struct sdp_bsdh) >= ssk->xmit_size_goal || - (ssk->tx_ring.tail == ssk->tx_ring.head && + (ring_tail(ssk->tx_ring) == ring_head(ssk->tx_ring) && !(ssk->nonagle & TCP_NAGLE_CORK)) || (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_PSH); } @@ -125,26 +116,14 @@ static inline int sdp_nagle_off(struct sdp_sock *ssk, struct sk_buff *skb) int sdp_post_credits(struct sdp_sock *ssk) { int post_count = 0; - struct sk_buff *skb; sdp_dbg_data(&ssk->isk.sk, "credits: %d remote credits: %d " "tx ring slots left: %d send_head: %p\n", - ssk->tx_ring.credits, ssk->remote_credits, + tx_credits(ssk), remote_credits(ssk), sdp_tx_ring_slots_left(&ssk->tx_ring), ssk->isk.sk.sk_send_head); - if (ssk->tx_ring.credits > SDP_MIN_TX_CREDITS && - sdp_tx_ring_slots_left(&ssk->tx_ring) && - (skb = ssk->isk.sk.sk_send_head) && - sdp_nagle_off(ssk, skb)) { - update_send_head(&ssk->isk.sk, skb); - __skb_dequeue(&ssk->isk.sk.sk_write_queue); - sdp_post_send(ssk, skb, SDP_MID_DATA); - post_count++; - goto out; - } - - if (likely(ssk->tx_ring.credits > 1) && + if (likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; skb = sdp_stream_alloc_skb(&ssk->isk.sk, @@ -156,7 +135,6 @@ int sdp_post_credits(struct sdp_sock *ssk) post_count++; } -out: if (post_count) sdp_xmit_poll(ssk, 0); return post_count; @@ -188,7 +166,7 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag gfp_page = GFP_KERNEL; sdp_dbg_data(&ssk->isk.sk, "credits: %d tx ring slots left: %d send_head: %p\n", - ssk->tx_ring.credits, sdp_tx_ring_slots_left(&ssk->tx_ring), + tx_credits(ssk), sdp_tx_ring_slots_left(&ssk->tx_ring), ssk->isk.sk.sk_send_head); if (sdp_tx_ring_slots_left(&ssk->tx_ring) < SDP_TX_SIZE / 2) { @@ -197,8 +175,8 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag } if (ssk->recv_request && - ssk->rx_tail >= ssk->recv_request_head && - ssk->tx_ring.credits >= SDP_MIN_TX_CREDITS && + ring_tail(ssk->rx_ring) >= ssk->recv_request_head && + tx_credits(ssk) >= SDP_MIN_TX_CREDITS && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; @@ -214,7 +192,7 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag post_count++; } - if (ssk->tx_ring.credits <= SDP_MIN_TX_CREDITS && + if (tx_credits(ssk) <= SDP_MIN_TX_CREDITS && sdp_tx_ring_slots_left(&ssk->tx_ring) && (skb = ssk->isk.sk.sk_send_head) && sdp_nagle_off(ssk, skb)) { @@ -222,7 +200,7 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag sdp_prf(&ssk->isk.sk, skb, "no credits. called from %s:%d", func, line); } - while (ssk->tx_ring.credits > SDP_MIN_TX_CREDITS && + while (tx_credits(ssk) > SDP_MIN_TX_CREDITS && sdp_tx_ring_slots_left(&ssk->tx_ring) && (skb = ssk->isk.sk.sk_send_head) && sdp_nagle_off(ssk, skb)) { @@ -232,9 +210,9 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag post_count++; } - if (0 && ssk->tx_ring.credits == SDP_MIN_TX_CREDITS && + if (0 && tx_credits(ssk) == SDP_MIN_TX_CREDITS && !ssk->sent_request && - ssk->tx_ring.head > ssk->sent_request_head + SDP_RESIZE_WAIT && + ring_head(ssk->tx_ring) > ssk->sent_request_head + SDP_RESIZE_WAIT && sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; skb = sdp_stream_alloc_skb(&ssk->isk.sk, @@ -244,19 +222,19 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag /* FIXME */ BUG_ON(!skb); ssk->sent_request = SDP_MAX_SEND_SKB_FRAGS * PAGE_SIZE; - ssk->sent_request_head = ssk->tx_ring.head; + ssk->sent_request_head = ring_head(ssk->tx_ring); req_size = (struct sdp_chrecvbuf *)skb_put(skb, sizeof *req_size); req_size->size = htonl(ssk->sent_request); sdp_post_send(ssk, skb, SDP_MID_CHRCVBUF); post_count++; } - c = ssk->remote_credits; + c = remote_credits(ssk); if (likely(c > SDP_MIN_TX_CREDITS)) c *= 2; - if (unlikely(c < ssk->rx_head - ssk->rx_tail) && - likely(ssk->tx_ring.credits > 1) && + if (unlikely(c < ring_posted(ssk->rx_ring)) && + likely(tx_credits(ssk) > 1) && likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { @@ -276,8 +254,8 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag * messages that provide additional credits and also do not contain ULP * payload. */ if (unlikely(ssk->sdp_disconnect) && - !ssk->isk.sk.sk_send_head && - ssk->tx_ring.credits > (ssk->remote_credits >= ssk->rx_head - ssk->rx_tail)) { + !ssk->isk.sk.sk_send_head && + tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; skb = sdp_stream_alloc_skb(&ssk->isk.sk, sizeof(struct sdp_bsdh), @@ -287,14 +265,10 @@ void _sdp_post_sends(const char *func, int line, struct sdp_sock *ssk, int nonag sdp_post_send(ssk, skb, SDP_MID_DISCONN); post_count++; } + if (post_count) { sdp_xmit_poll(ssk, 0); - sdp_prf(&ssk->isk.sk, NULL, "finshed polling from post_sends"); + sdp_prf(&ssk->isk.sk, NULL, "post_sends finished polling [%s:%d].", func, line); } } - -static inline int sdp_tx_qp_empty(struct sdp_sock *ssk) -{ - return (ssk->tx_ring.head - ssk->tx_ring.tail) == 0; -} diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index e0d0b20..a9dcf77 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -53,10 +53,6 @@ enum { SDP_HAH_SIZE = 180, }; -static void sdp_cq_event_handler(struct ib_event *event, void *data) -{ -} - static void sdp_qp_event_handler(struct ib_event *event, void *data) { } @@ -73,36 +69,12 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) .qp_type = IB_QPT_RC, }; struct ib_device *device = id->device; - struct ib_cq *rx_cq, *tx_cq; struct ib_mr *mr; struct ib_pd *pd; int rc; sdp_dbg(sk, "%s\n", __func__); - sdp_sk(sk)->tx_ring.head = 1; - sdp_sk(sk)->tx_ring.tail = 1; - sdp_sk(sk)->rx_head = 1; - sdp_sk(sk)->rx_tail = 1; - - sdp_sk(sk)->tx_ring.buffer = kmalloc(sizeof(*sdp_sk(sk)->tx_ring.buffer) * - (SDP_TX_SIZE + 1), GFP_KERNEL); - if (!sdp_sk(sk)->tx_ring.buffer) { - rc = -ENOMEM; - sdp_warn(sk, "Unable to allocate TX Ring size %zd.\n", - sizeof *sdp_sk(sk)->tx_ring.buffer * (SDP_TX_SIZE + 1)); - goto err_tx; - } - - sdp_sk(sk)->rx_ring = kmalloc(sizeof *sdp_sk(sk)->rx_ring * SDP_RX_SIZE, - GFP_KERNEL); - if (!sdp_sk(sk)->rx_ring) { - rc = -ENOMEM; - sdp_warn(sk, "Unable to allocate RX Ring size %zd.\n", - sizeof *sdp_sk(sk)->rx_ring * SDP_RX_SIZE); - goto err_rx; - } - pd = ib_alloc_pd(device); if (IS_ERR(pd)) { rc = PTR_ERR(pd); @@ -118,43 +90,15 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) } sdp_sk(sk)->mr = mr; - INIT_WORK(&sdp_sk(sk)->rx_comp_work, sdp_rx_comp_work); - - rx_cq = ib_create_cq(device, sdp_rx_irq, sdp_cq_event_handler, - sk, SDP_RX_SIZE, 0); - - if (IS_ERR(rx_cq)) { - rc = PTR_ERR(rx_cq); - sdp_warn(sk, "Unable to allocate RX CQ: %d.\n", rc); - goto err_rx_cq; - } - rc = ib_modify_cq(rx_cq, 10, 200); - if (rc) { - sdp_warn(sk, "Unable to modify RX CQ: %d.\n", rc); - goto err_tx_cq; - } - sdp_warn(sk, "Initialized CQ moderation\n"); - sdp_sk(sk)->rx_cq = rx_cq; - sdp_arm_rx_cq(sk); - qp_init_attr.recv_cq = rx_cq; - - tx_cq = ib_create_cq(device, sdp_tx_irq, sdp_cq_event_handler, - sk, SDP_TX_SIZE, 0); - - if (IS_ERR(tx_cq)) { - rc = PTR_ERR(tx_cq); - sdp_warn(sk, "Unable to allocate TX CQ: %d.\n", rc); - goto err_tx_cq; - } + if ((rc = sdp_rx_ring_create(sdp_sk(sk), device))) + goto err_rx; - init_timer(&sdp_sk(sk)->tx_ring.timer); - sdp_sk(sk)->tx_ring.timer.function = _sdp_poll_tx_cq; - sdp_sk(sk)->tx_ring.timer.data = (unsigned long) sdp_sk(sk); - sdp_sk(sk)->tx_ring.poll_cnt = 0; + if ((rc = sdp_tx_ring_create(sdp_sk(sk), device))) + goto err_tx; - sdp_sk(sk)->tx_ring.cq = tx_cq; - qp_init_attr.send_cq = tx_cq; + qp_init_attr.recv_cq = sdp_sk(sk)->rx_ring.cq; + qp_init_attr.send_cq = sdp_sk(sk)->tx_ring.cq; rc = rdma_create_qp(id, pd, &qp_init_attr); if (rc) { @@ -170,20 +114,14 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) return 0; err_qp: - ib_destroy_cq(tx_cq); -err_tx_cq: - ib_destroy_cq(rx_cq); -err_rx_cq: + sdp_tx_ring_destroy(sdp_sk(sk)); +err_tx: + sdp_rx_ring_destroy(sdp_sk(sk)); +err_rx: ib_dereg_mr(sdp_sk(sk)->mr); err_mr: ib_dealloc_pd(pd); err_pd: - kfree(sdp_sk(sk)->rx_ring); - sdp_sk(sk)->rx_ring = NULL; -err_rx: - kfree(sdp_sk(sk)->tx_ring.buffer); - sdp_sk(sk)->tx_ring.buffer = NULL; -err_tx: return rc; } @@ -225,8 +163,10 @@ static int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, sdp_add_sock(sdp_sk(child)); - sdp_sk(child)->max_bufs = sdp_sk(child)->tx_ring.credits = ntohs(h->bsdh.bufs); - sdp_sk(child)->min_bufs = sdp_sk(child)->tx_ring.credits / 4; + sdp_sk(child)->max_bufs = ntohs(h->bsdh.bufs); + atomic_set(&sdp_sk(child)->tx_ring.credits, sdp_sk(child)->max_bufs); + + sdp_sk(child)->min_bufs = tx_credits(sdp_sk(child)) / 4; sdp_sk(child)->xmit_size_goal = ntohl(h->localrcvsz) - sizeof(struct sdp_bsdh); sdp_sk(child)->send_frags = PAGE_ALIGN(sdp_sk(child)->xmit_size_goal) / @@ -236,7 +176,7 @@ static int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, sdp_dbg(child, "%s recv_frags: %d tx credits %d xmit_size_goal %d send trigger %d\n", __func__, sdp_sk(child)->recv_frags, - sdp_sk(child)->tx_ring.credits, + tx_credits(sdp_sk(child)), sdp_sk(child)->xmit_size_goal, sdp_sk(child)->min_bufs); @@ -272,8 +212,9 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, h = event->param.conn.private_data; SDP_DUMP_PACKET(sk, "RX", NULL, &h->bsdh); - sdp_sk(sk)->max_bufs = sdp_sk(sk)->tx_ring.credits = ntohs(h->bsdh.bufs); - sdp_sk(sk)->min_bufs = sdp_sk(sk)->tx_ring.credits / 4; + sdp_sk(sk)->max_bufs = ntohs(h->bsdh.bufs); + atomic_set(&sdp_sk(sk)->tx_ring.credits, sdp_sk(sk)->max_bufs); + sdp_sk(sk)->min_bufs = tx_credits(sdp_sk(sk)) / 4; sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - sizeof(struct sdp_bsdh); sdp_sk(sk)->send_frags = MIN(PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / @@ -282,14 +223,12 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, sdp_sk(sk)->send_frags * PAGE_SIZE); sdp_dbg(sk, "tx credits %d xmit_size_goal %d send_frags: %d credits update trigger %d\n", - sdp_sk(sk)->tx_ring.credits, + tx_credits(sdp_sk(sk)), sdp_sk(sk)->xmit_size_goal, sdp_sk(sk)->send_frags, sdp_sk(sk)->min_bufs); sdp_sk(sk)->poll_cq = 1; - sdp_arm_rx_cq(sk); - sdp_poll_rx_cq(sdp_sk(sk)); sk->sk_state_change(sk); sk_wake_async(sk, 0, POLL_OUT); @@ -349,8 +288,7 @@ static int sdp_disconnected_handler(struct sock *sk) sdp_dbg(sk, "%s\n", __func__); - if (ssk->rx_cq) - sdp_poll_rx_cq(ssk); + sdp_process_rx_q(ssk); if (ssk->tx_ring.cq) sdp_xmit_poll(ssk, 1); @@ -400,7 +338,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) rc = rdma_resolve_route(id, SDP_ROUTE_TIMEOUT); break; case RDMA_CM_EVENT_ADDR_ERROR: - sdp_dbg(sk, "RDMA_CM_EVENT_ADDR_ERROR\n"); + sdp_warn(sk, "RDMA_CM_EVENT_ADDR_ERROR\n"); rc = -ENETUNREACH; break; case RDMA_CM_EVENT_ROUTE_RESOLVED: @@ -408,11 +346,10 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) rc = sdp_init_qp(sk, id); if (rc) break; - sdp_sk(sk)->remote_credits = sdp_sk(sk)->rx_head - - sdp_sk(sk)->rx_tail; + atomic_set(&sdp_sk(sk)->remote_credits, ring_posted(sdp_sk(sk)->rx_ring)); memset(&hh, 0, sizeof hh); hh.bsdh.mid = SDP_MID_HELLO; - hh.bsdh.bufs = htons(sdp_sk(sk)->remote_credits); + hh.bsdh.bufs = htons(remote_credits(sdp_sk(sk))); hh.bsdh.len = htonl(sizeof(struct sdp_bsdh) + SDP_HH_SIZE); hh.max_adverts = 1; hh.majv_minv = SDP_MAJV_MINV; @@ -443,11 +380,10 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) break; } child = id->context; - sdp_sk(child)->remote_credits = sdp_sk(child)->rx_head - - sdp_sk(child)->rx_tail; + atomic_set(&sdp_sk(child)->remote_credits, ring_posted(sdp_sk(child)->rx_ring)); memset(&hah, 0, sizeof hah); hah.bsdh.mid = SDP_MID_HELLO_ACK; - hah.bsdh.bufs = htons(sdp_sk(child)->remote_credits); + hah.bsdh.bufs = htons(remote_credits(sdp_sk(child))); hah.bsdh.len = htonl(sizeof(struct sdp_bsdh) + SDP_HAH_SIZE); hah.majv_minv = SDP_MAJV_MINV; hah.ext_max_adverts = 1; /* Doesn't seem to be mandated by spec, diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index b5322da..c6b17db 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -206,31 +206,24 @@ static int sdp_get_port(struct sock *sk, unsigned short snum) static void sdp_destroy_qp(struct sdp_sock *ssk) { struct ib_pd *pd = NULL; - struct ib_cq *rx_cq = NULL; - struct ib_cq *tx_cq = NULL; + unsigned long flags; + + + sdp_dbg(&ssk->isk.sk, "destroying qp\n"); del_timer(&ssk->tx_ring.timer); + rx_ring_lock(ssk, flags); + + sdp_rx_ring_destroy(ssk); + sdp_tx_ring_destroy(ssk); + if (ssk->qp) { pd = ssk->qp->pd; - rx_cq = ssk->rx_cq; - ssk->rx_cq = NULL; - tx_cq = ssk->tx_ring.cq; - ssk->tx_ring.cq = NULL; ib_destroy_qp(ssk->qp); ssk->qp = NULL; - - sdp_rx_ring_purge(ssk); - sdp_tx_ring_purge(ssk); - } - - if (tx_cq) { - ib_destroy_cq(tx_cq); } - if (rx_cq) - ib_destroy_cq(rx_cq); - if (ssk->mr) { ib_dereg_mr(ssk->mr); ssk->mr = NULL; @@ -241,14 +234,8 @@ static void sdp_destroy_qp(struct sdp_sock *ssk) sdp_remove_large_sock(ssk); - if (ssk->rx_ring) { - kfree(ssk->rx_ring); - ssk->rx_ring = NULL; - } - if (ssk->tx_ring.buffer) { - kfree(ssk->tx_ring.buffer); - ssk->tx_ring.buffer = NULL; - } + rx_ring_unlock(ssk, flags); + } static void sdp_reset_keepalive_timer(struct sock *sk, unsigned long len) @@ -257,8 +244,8 @@ static void sdp_reset_keepalive_timer(struct sock *sk, unsigned long len) sdp_dbg(sk, "%s\n", __func__); - ssk->keepalive_tx_head = ssk->tx_ring.head; - ssk->keepalive_rx_head = ssk->rx_head; + ssk->keepalive_tx_head = ring_head(ssk->tx_ring); + ssk->keepalive_rx_head = ring_head(ssk->rx_ring); sk_reset_timer(sk, &sk->sk_timer, jiffies + len); } @@ -293,8 +280,8 @@ static void sdp_keepalive_timer(unsigned long data) sk->sk_state == TCP_CLOSE) goto out; - if (ssk->keepalive_tx_head == ssk->tx_ring.head && - ssk->keepalive_rx_head == ssk->rx_head) + if (ssk->keepalive_tx_head == ring_head(ssk->tx_ring) && + ssk->keepalive_rx_head == ring_head(ssk->rx_ring)) sdp_post_keepalive(ssk); sdp_reset_keepalive_timer(sk, sdp_keepalive_time_when(ssk)); @@ -338,17 +325,21 @@ void sdp_reset_sk(struct sock *sk, int rc) read_lock(&device_removal_lock); - if (ssk->rx_cq) - sdp_poll_rx_cq(ssk); + sdp_process_rx_q(ssk); if (ssk->tx_ring.cq) sdp_xmit_poll(ssk, 1); - if (!(sk->sk_shutdown & RCV_SHUTDOWN) || !sk_stream_memory_free(sk)) + if (!(sk->sk_shutdown & RCV_SHUTDOWN) || !sk_stream_memory_free(sk)) { + sdp_warn(sk, "setting state to error. shutdown: %d, mem_free: %d\n", + !(sk->sk_shutdown & RCV_SHUTDOWN), + !sk_stream_memory_free(sk)); sdp_set_error(sk, rc); + } sdp_destroy_qp(ssk); + sdp_dbg(sk, "memset on sdp_sock\n"); memset((void *)&ssk->id, 0, sizeof(*ssk) - offsetof(typeof(*ssk), id)); sk->sk_state_change(sk); @@ -488,6 +479,7 @@ static void sdp_close(struct sock *sk, long timeout) lock_sock(sk); sdp_dbg(sk, "%s\n", __func__); + sdp_prf(sk, NULL, __func__); sdp_delete_keepalive_timer(sk); @@ -773,10 +765,9 @@ out: release_sock(sk); if (newsk) { lock_sock(newsk); - if (newssk->rx_cq) { + if (newssk->rx_ring.cq) { newssk->poll_cq = 1; sdp_arm_rx_cq(&newssk->isk.sk); - sdp_poll_rx_cq(newssk); } release_sock(newsk); } @@ -934,7 +925,11 @@ int sdp_init_sock(struct sock *sk) sk->sk_route_caps |= NETIF_F_SG | NETIF_F_NO_CSUM; - ssk->rx_ring = NULL; + skb_queue_head_init(&ssk->rx_backlog); + + atomic_set(&ssk->mseq_ack, 0); + + sdp_rx_ring_init(ssk); ssk->tx_ring.buffer = NULL; ssk->sdp_disconnect = 0; ssk->destructed_already = 0; @@ -1142,14 +1137,13 @@ static int sdp_getsockopt(struct sock *sk, int level, int optname, static inline int poll_recv_cq(struct sock *sk) { int i; - if (sdp_sk(sk)->rx_cq) { - for (i = 0; i < recv_poll; ++i) - if (!sdp_poll_rx_cq(sdp_sk(sk))) { - ++recv_poll_hit; - return 0; - } - ++recv_poll_miss; + for (i = 0; i < recv_poll; ++i) { + if (!sdp_process_rx_q(sdp_sk(sk))) { + ++recv_poll_hit; + return 0; + } } + ++recv_poll_miss; return 1; } @@ -1551,8 +1545,8 @@ static inline int slots_free(struct sdp_sock *ssk) { int min_free; - min_free = MIN(ssk->tx_ring.credits, - SDP_TX_SIZE - (ssk->tx_ring.head - ssk->tx_ring.tail)); + min_free = MIN(tx_credits(ssk), + SDP_TX_SIZE - ring_posted(ssk->tx_ring)); if (min_free < SDP_MIN_TX_CREDITS) return 0; @@ -1608,6 +1602,9 @@ static int sdp_bzcopy_wait_memory(struct sdp_sock *ssk, long *timeo_p, set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); sk->sk_write_pending++; + sdp_prf(sk, NULL, "credits: %d, head: %d, tail: %d, busy: %d", + tx_credits(ssk), ring_head(ssk->tx_ring), ring_tail(ssk->tx_ring), + bz->busy); sk_wait_event(sk, ¤t_timeo, sdp_bzcopy_slots_avail(ssk, bz) && vm_wait); sk->sk_write_pending--; @@ -1627,24 +1624,6 @@ static int sdp_bzcopy_wait_memory(struct sdp_sock *ssk, long *timeo_p, return err; } -/* like sk_stream_write_space - execpt measures remote credits */ -void sdp_bzcopy_write_space(struct sdp_sock *ssk) -{ - struct sock *sk = &ssk->isk.sk; - struct socket *sock = sk->sk_socket; - - if (ssk->tx_ring.credits >= ssk->min_bufs && - ssk->tx_ring.head == ssk->tx_ring.tail && - sock != NULL) { - clear_bit(SOCK_NOSPACE, &sock->flags); - - if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) - wake_up_interruptible(sk->sk_sleep); - if (sock->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN)) - sock_wake_async(sock, 2, POLL_OUT); - } -} - /* Like tcp_sendmsg */ /* TODO: check locking */ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, @@ -1735,7 +1714,7 @@ new_segment: if (!skb) goto wait_for_memory; - sdp_prf(sk, skb, "Created"); +// sdp_prf(sk, skb, "Created"); BZCOPY_STATE(skb) = bz; @@ -1751,7 +1730,7 @@ new_segment: skb_entail(sk, ssk, skb); copy = size_goal; } else { - sdp_prf(sk, skb, "adding %d bytes", copy); +// sdp_prf(sk, skb, "adding %d bytes", copy); sdp_dbg_data(sk, "adding to existing skb: %p" " len = %d, sk_send_head: %p copy: %d\n", skb, skb->len, sk->sk_send_head, copy); @@ -1770,10 +1749,8 @@ new_segment: goto new_segment; } -// sdp_prf(sk, skb, "before memcpy %d bytes", copy); copy = (bz) ? sdp_bzcopy_get(sk, skb, from, copy, bz) : sdp_bcopy_get(sk, skb, from, copy); -// sdp_prf(sk, skb, "after memcpy. result: %d", copy); if (unlikely(copy < 0)) { if (!++copy) goto wait_for_memory; @@ -1863,6 +1840,20 @@ out_err: return err; } +int dummy_memcpy_toiovec(struct iovec *iov, int len) +{ + while (len > 0) { + if (iov->iov_len) { + int copy = min_t(unsigned int, iov->iov_len, len); + len -= copy; + iov->iov_len -= copy; + iov->iov_base += copy; + } + iov++; + } + + return 0; +} /* Like tcp_recvmsg */ /* Maybe use skb_recv_datagram here? */ /* Note this does not seem to handle vectored messages. Relevant? */ @@ -1884,7 +1875,7 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, lock_sock(sk); sdp_dbg_data(sk, "%s\n", __func__); - sdp_prf(sk, skb, "Read from user"); +// sdp_prf(sk, skb, "Read from user"); err = -ENOTCONN; if (sk->sk_state == TCP_LISTEN) @@ -2024,6 +2015,7 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, err = skb_copy_datagram_iovec(skb, offset, /* TODO: skip header? */ msg->msg_iov, used); +// err = dummy_memcpy_toiovec(msg->msg_iov, used); sdp_prf(sk, skb, "Copied to user %ld bytes. err = %d", used, err); if (err) { sdp_dbg(sk, "%s: skb_copy_datagram_iovec failed" diff --git a/drivers/infiniband/ulp/sdp/sdp_proc.c b/drivers/infiniband/ulp/sdp/sdp_proc.c index 3778c9a..537b3a6 100644 --- a/drivers/infiniband/ulp/sdp/sdp_proc.c +++ b/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -236,15 +236,6 @@ static void sdpstats_seq_hist(struct seq_file *seq, char *str, u32 *h, int n, in static int sdpstats_seq_show(struct seq_file *seq, void *v) { -#define ENUM2STR(e) [e] = #e - static char *mid2str[] = { - ENUM2STR(SDP_MID_HELLO), - ENUM2STR(SDP_MID_HELLO_ACK), - ENUM2STR(SDP_MID_DISCONN), - ENUM2STR(SDP_MID_CHRCVBUF), - ENUM2STR(SDP_MID_CHRCVBUF_ACK), - ENUM2STR(SDP_MID_DATA), - }; int i; seq_printf(seq, "SDP statistics:\n"); @@ -268,9 +259,9 @@ static int sdpstats_seq_show(struct seq_file *seq, void *v) seq_printf(seq, "memcpy_count \t\t: %u\n", sdpstats.memcpy_count); for (i = 0; i < ARRAY_SIZE(sdpstats.post_send); i++) { - if (mid2str[i]) { + if (mid2str(i)) { seq_printf(seq, "post_send %-20s\t: %d\n", - mid2str[i], sdpstats.post_send[i]); + mid2str(i), sdpstats.post_send[i]); } } diff --git a/drivers/infiniband/ulp/sdp/sdp_rx.c b/drivers/infiniband/ulp/sdp/sdp_rx.c index ba8e0fe..810dcbb 100644 --- a/drivers/infiniband/ulp/sdp/sdp_rx.c +++ b/drivers/infiniband/ulp/sdp/sdp_rx.c @@ -39,7 +39,7 @@ static int rcvbuf_scale = 0x10; -int rcvbuf_initial_size = SDP_HEAD_SIZE; +int rcvbuf_initial_size = 32 * 1024; module_param_named(rcvbuf_initial_size, rcvbuf_initial_size, int, 0644); MODULE_PARM_DESC(rcvbuf_initial_size, "Receive buffer initial size in bytes."); @@ -149,22 +149,28 @@ static void sdp_fin(struct sock *sk) } } - -static void sdp_post_recv(struct sdp_sock *ssk) +/* lock_sock must be taken before calling this - since rx_ring.head is not + * protected (although being atomic + */ +static int sdp_post_recv(struct sdp_sock *ssk) { struct sdp_buf *rx_req; int i, rc, frags; u64 addr; struct ib_device *dev; - struct ib_sge *sge; + struct ib_recv_wr rx_wr = { 0 }; + struct ib_sge ibsge[SDP_MAX_SEND_SKB_FRAGS + 1]; + struct ib_sge *sge = ibsge; struct ib_recv_wr *bad_wr; struct sk_buff *skb; struct page *page; skb_frag_t *frag; struct sdp_bsdh *h; - int id = ssk->rx_head; + int id = ring_head(ssk->rx_ring); gfp_t gfp_page; + int ret = 0; + WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); /* Now, allocate and repost recv */ /* TODO: allocate from cache */ @@ -194,10 +200,9 @@ static void sdp_post_recv(struct sdp_sock *ssk) skb->truesize += frag->size; } - rx_req = ssk->rx_ring + (id & (SDP_RX_SIZE - 1)); + rx_req = ssk->rx_ring.buffer + (id & (SDP_RX_SIZE - 1)); rx_req->skb = skb; dev = ssk->ib_device; - sge = ssk->ibsge; addr = ib_dma_map_single(dev, h, SDP_HEAD_SIZE, DMA_FROM_DEVICE); BUG_ON(ib_dma_mapping_error(dev, addr)); @@ -221,27 +226,32 @@ static void sdp_post_recv(struct sdp_sock *ssk) sge->lkey = ssk->mr->lkey; } - ssk->rx_wr.next = NULL; - ssk->rx_wr.wr_id = id | SDP_OP_RECV; - ssk->rx_wr.sg_list = ssk->ibsge; - ssk->rx_wr.num_sge = frags + 1; - rc = ib_post_recv(ssk->qp, &ssk->rx_wr, &bad_wr); - sdp_prf(&ssk->isk.sk, skb, "rx skb was posted"); + rx_wr.next = NULL; + rx_wr.wr_id = id | SDP_OP_RECV; + rx_wr.sg_list = ibsge; + rx_wr.num_sge = frags + 1; + rc = ib_post_recv(ssk->qp, &rx_wr, &bad_wr); SDPSTATS_COUNTER_INC(post_recv); - ++ssk->rx_head; + atomic_inc(&ssk->rx_ring.head); if (unlikely(rc)) { - sdp_dbg(&ssk->isk.sk, "ib_post_recv failed with status %d\n", rc); + sdp_warn(&ssk->isk.sk, "ib_post_recv failed with status %d\n", rc); sdp_reset(&ssk->isk.sk); + ret = -1; } atomic_add(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); + + return ret; } -void sdp_post_recvs(struct sdp_sock *ssk) +/* lock_sock must be taken before calling this */ +static void _sdp_post_recvs(struct sdp_sock *ssk) { struct sock *sk = &ssk->isk.sk; int scale = ssk->rcvbuf_scale; + WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); + if (unlikely(!ssk->id || ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_TIME_WAIT)))) { return; @@ -251,12 +261,23 @@ void sdp_post_recvs(struct sdp_sock *ssk) (top_mem_usage * 0x100000) < atomic_read(&sdp_current_mem_usage) * PAGE_SIZE) scale = 1; - while ((likely(ssk->rx_head - ssk->rx_tail < SDP_RX_SIZE) && - (ssk->rx_head - ssk->rx_tail - SDP_MIN_TX_CREDITS) * + while ((likely(ring_posted(ssk->rx_ring) < SDP_RX_SIZE) && + (ring_posted(ssk->rx_ring) - SDP_MIN_TX_CREDITS) * (SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE) + ssk->rcv_nxt - ssk->copied_seq < sk->sk_rcvbuf * scale) || - unlikely(ssk->rx_head - ssk->rx_tail < SDP_MIN_TX_CREDITS)) - sdp_post_recv(ssk); + unlikely(ring_posted(ssk->rx_ring) < SDP_MIN_TX_CREDITS)) { + if (sdp_post_recv(ssk)) + break; + } +} + +void sdp_post_recvs(struct sdp_sock *ssk) +{ + unsigned long flags; + + rx_ring_lock(ssk, flags); + _sdp_post_recvs(ssk); + rx_ring_unlock(ssk, flags); } static inline struct sk_buff *sdp_sock_queue_rcv_skb(struct sock *sk, @@ -328,9 +349,9 @@ int sdp_resize_buffers(struct sdp_sock *ssk, u32 new_size) static void sdp_handle_resize_request(struct sdp_sock *ssk, struct sdp_chrecvbuf *buf) { if (sdp_resize_buffers(ssk, ntohl(buf->size)) == 0) - ssk->recv_request_head = ssk->rx_head + 1; + ssk->recv_request_head = ring_head(ssk->rx_ring) + 1; else - ssk->recv_request_head = ssk->rx_tail; + ssk->recv_request_head = ring_tail(ssk->rx_ring); ssk->recv_request = 1; } @@ -347,23 +368,17 @@ static void sdp_handle_resize_ack(struct sdp_sock *ssk, struct sdp_chrecvbuf *bu ssk->sent_request = 0; } -static inline int credit_update_needed(struct sdp_sock *ssk, int wc_processed) +static inline int credit_update_needed(struct sdp_sock *ssk) { int c; - c = ssk->remote_credits; + c = remote_credits(ssk); if (likely(c > SDP_MIN_TX_CREDITS)) c += c/2; -/* sdp_warn(&ssk->isk.sk, "credits: %d remote credits: %d " - "tx ring slots left: %d send_head: %p\n", - ssk->tx_ring.credits, ssk->remote_credits, - sdp_tx_ring_slots_left(&ssk->tx_ring), - ssk->isk.sk.sk_send_head); -*/ - return (unlikely(c < ssk->rx_head - ssk->rx_tail + wc_processed) && - likely(ssk->tx_ring.credits > 1) && - likely(sdp_tx_ring_slots_left(&ssk->tx_ring))); + return unlikely(c < ring_posted(ssk->rx_ring)) && + likely(tx_credits(ssk) > 1) && + likely(sdp_tx_ring_slots_left(&ssk->tx_ring)); } @@ -374,14 +389,16 @@ static struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id) struct sk_buff *skb; int i, frags; - if (unlikely(id != ssk->rx_tail)) { + WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); + + if (unlikely(id != ring_tail(ssk->rx_ring))) { printk(KERN_WARNING "Bogus recv completion id %d tail %d\n", - id, ssk->rx_tail); + id, ring_tail(ssk->rx_ring)); return NULL; } dev = ssk->ib_device; - rx_req = &ssk->rx_ring[id & (SDP_RX_SIZE - 1)]; + rx_req = &ssk->rx_ring.buffer[id & (SDP_RX_SIZE - 1)]; skb = rx_req->skb; ib_dma_unmap_single(dev, rx_req->mapping[0], SDP_HEAD_SIZE, DMA_FROM_DEVICE); @@ -390,67 +407,20 @@ static struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id) ib_dma_unmap_page(dev, rx_req->mapping[i + 1], skb_shinfo(skb)->frags[i].size, DMA_FROM_DEVICE); - ++ssk->rx_tail; - --ssk->remote_credits; + atomic_inc(&ssk->rx_ring.tail); + atomic_dec(&ssk->remote_credits); return skb; } -static int sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) +/* this must be called while sock_lock is taken */ +static int sdp_process_rx_skb(struct sdp_sock *ssk, struct sk_buff *skb) { struct sock *sk = &ssk->isk.sk; int frags; - struct sk_buff *skb; struct sdp_bsdh *h; int pagesz, i; - skb = sdp_recv_completion(ssk, wc->wr_id); - if (unlikely(!skb)) - return -1; - - sdp_prf(sk, skb, "recv completion"); - - atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); - - if (unlikely(wc->status)) { - if (wc->status != IB_WC_WR_FLUSH_ERR) { - sdp_dbg(sk, "Recv completion with error. Status %d\n", - wc->status); - sdp_reset(sk); - } - __kfree_skb(skb); - return 0; - } - - sdp_dbg_data(sk, "Recv completion. ID %d Length %d\n", - (int)wc->wr_id, wc->byte_len); - if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { - printk(KERN_WARNING "SDP BUG! byte_len %d < %zd\n", - wc->byte_len, sizeof(struct sdp_bsdh)); - __kfree_skb(skb); - return -1; - } - skb->len = wc->byte_len; - if (likely(wc->byte_len > SDP_HEAD_SIZE)) - skb->data_len = wc->byte_len - SDP_HEAD_SIZE; - else - skb->data_len = 0; - skb->data = skb->head; -#ifdef NET_SKBUFF_DATA_USES_OFFSET - skb->tail = skb_headlen(skb); -#else - skb->tail = skb->head + skb_headlen(skb); -#endif h = (struct sdp_bsdh *)skb->data; - SDP_DUMP_PACKET(&ssk->isk.sk, "RX", skb, h); - skb_reset_transport_header(skb); - ssk->mseq_ack = ntohl(h->mseq); - if (ssk->mseq_ack != (int)wc->wr_id) - printk(KERN_WARNING "SDP BUG! mseq %d != wrid %d\n", - ssk->mseq_ack, (int)wc->wr_id); - - SDPSTATS_HIST_LINEAR(credits_before_update, ssk->tx_ring.credits); - ssk->tx_ring.credits = ntohl(h->mseq_ack) - ssk->tx_ring.head + 1 + - ntohs(h->bufs); frags = skb_shinfo(skb)->nr_frags; pagesz = PAGE_ALIGN(skb->data_len); @@ -513,13 +483,105 @@ static int sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) return 0; } -int sdp_poll_rx_cq(struct sdp_sock *ssk) +/* called only from irq */ +static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, struct ib_wc *wc) { - struct ib_cq *cq = ssk->rx_cq; + struct sk_buff *skb; + struct sdp_bsdh *h; + struct sock *sk = &ssk->isk.sk; + int credits_before; + + skb = sdp_recv_completion(ssk, wc->wr_id); + if (unlikely(!skb)) + return NULL; + + atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); + + if (unlikely(wc->status)) { + if (wc->status != IB_WC_WR_FLUSH_ERR) { + sdp_warn(sk, "Recv completion with error. Status %d\n", + wc->status); + sdp_reset(sk); + } + __kfree_skb(skb); + return NULL; + } + + sdp_dbg_data(sk, "Recv completion. ID %d Length %d\n", + (int)wc->wr_id, wc->byte_len); + if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { + printk(KERN_WARNING "SDP BUG! byte_len %d < %zd\n", + wc->byte_len, sizeof(struct sdp_bsdh)); + __kfree_skb(skb); + return NULL; + } + skb->len = wc->byte_len; + if (likely(wc->byte_len > SDP_HEAD_SIZE)) + skb->data_len = wc->byte_len - SDP_HEAD_SIZE; + else + skb->data_len = 0; + skb->data = skb->head; +#ifdef NET_SKBUFF_DATA_USES_OFFSET + skb->tail = skb_headlen(skb); +#else + skb->tail = skb->head + skb_headlen(skb); +#endif + h = (struct sdp_bsdh *)skb->data; + SDP_DUMP_PACKET(&ssk->isk.sk, "RX", skb, h); + skb_reset_transport_header(skb); + atomic_set(&ssk->mseq_ack, ntohl(h->mseq)); + if (mseq_ack(ssk) != (int)wc->wr_id) + printk(KERN_WARNING "SDP BUG! mseq %d != wrid %d\n", + mseq_ack(ssk), (int)wc->wr_id); + + SDPSTATS_HIST_LINEAR(credits_before_update, tx_credits(ssk)); + + credits_before = tx_credits(ssk); + atomic_set(&ssk->tx_ring.credits, ntohl(h->mseq_ack) - ring_head(ssk->tx_ring) + 1 + + ntohs(h->bufs)); + + sdp_prf(&ssk->isk.sk, skb, "RX %s bufs=%d c before:%d after:%d " + "mseq:%d, ack:%d", mid2str(h->mid), ntohs(h->bufs), credits_before, + tx_credits(ssk), ntohl(h->mseq), ntohl(h->mseq_ack)); + + return skb; +} + +/* like sk_stream_write_space - execpt measures remote credits */ +static void sdp_bzcopy_write_space(struct sdp_sock *ssk) +{ + struct sock *sk = &ssk->isk.sk; + struct socket *sock = sk->sk_socket; + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) { + sdp_prf(&ssk->isk.sk, NULL, "credits: %d, min_bufs: %d. tx_head: %d, tx_tail: %d", + tx_credits(ssk), ssk->min_bufs, + ring_head(ssk->tx_ring), ring_tail(ssk->tx_ring)); + } + + if (tx_credits(ssk) >= ssk->min_bufs && + ring_head(ssk->tx_ring) == ring_tail(ssk->tx_ring) && + sock != NULL) { + clear_bit(SOCK_NOSPACE, &sock->flags); + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + wake_up_interruptible(sk->sk_sleep); + if (sock->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN)) + sock_wake_async(sock, 2, POLL_OUT); + } +} + +/* only from interrupt. + * drain rx cq into rx_backlog queue */ +static int sdp_poll_rx_cq(struct sdp_sock *ssk) +{ + struct ib_cq *cq = ssk->rx_ring.cq; struct ib_wc ibwc[SDP_NUM_WC]; int n, i; - int ret = -EAGAIN; int wc_processed = 0; + struct sk_buff *skb; + + WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); do { n = ib_poll_cq(cq, SDP_NUM_WC, ibwc); @@ -527,102 +589,224 @@ int sdp_poll_rx_cq(struct sdp_sock *ssk) struct ib_wc *wc = &ibwc[i]; BUG_ON(!(wc->wr_id & SDP_OP_RECV)); - sdp_process_rx_wc(ssk, wc); + skb = sdp_process_rx_wc(ssk, wc); + if (!skb) + continue; + skb_queue_tail(&ssk->rx_backlog, skb); wc_processed++; - - if (credit_update_needed(ssk, wc_processed)) { - sdp_prf(&ssk->isk.sk, NULL, "credit update. remote_credits: %d, avail now: %d processed: %d", - ssk->remote_credits, - ssk->rx_head - ssk->rx_tail, - wc_processed); - sdp_post_recvs(ssk); - if (sdp_post_credits(ssk)) - wc_processed = 0; - } - - ret = 0; } } while (n == SDP_NUM_WC); - if (!ret) { - struct sock *sk = &ssk->isk.sk; + if (wc_processed) + sdp_bzcopy_write_space(ssk); - sdp_post_recvs(ssk); + return wc_processed; +} - /* update credits */ - sdp_post_sends(ssk, 0); +int sdp_process_rx_q(struct sdp_sock *ssk) +{ + struct sk_buff *skb; + struct sock *sk = &ssk->isk.sk; + unsigned long flags; - if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) - sk_stream_write_space(&ssk->isk.sk); - } else { + if (!ssk->rx_backlog.next || !ssk->rx_backlog.prev) { + sdp_warn(&ssk->isk.sk, "polling a zeroed rx_backlog!!!! %p\n", &ssk->rx_backlog); + return 0; + } + + if (skb_queue_empty(&ssk->rx_backlog)) { SDPSTATS_COUNTER_INC(rx_poll_miss); + return -EAGAIN; } - return ret; + /* update credits */ + sdp_post_sends(ssk, 0); + + spin_lock_irqsave(&ssk->rx_backlog.lock, flags); + while ((skb = __skb_dequeue(&ssk->rx_backlog))) { + sdp_process_rx_skb(ssk, skb); + } + spin_unlock_irqrestore(&ssk->rx_backlog.lock, flags); + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + sk_stream_write_space(&ssk->isk.sk); + + return 0; } -void sdp_rx_comp_work(struct work_struct *work) +static void sdp_rx_comp_work(struct work_struct *work) { struct sdp_sock *ssk = container_of(work, struct sdp_sock, rx_comp_work); struct sock *sk = &ssk->isk.sk; struct ib_cq *rx_cq; lock_sock(sk); - rx_cq = ssk->rx_cq; + rx_cq = ssk->rx_ring.cq; if (unlikely(!rx_cq)) goto out; if (unlikely(!ssk->poll_cq)) { struct rdma_cm_id *id = ssk->id; + sdp_warn(sk, "poll cq is 0. socket was reset or wasn't initialized\n"); if (id && id->qp) rdma_notify(id, RDMA_CM_EVENT_ESTABLISHED); goto out; } - sdp_poll_rx_cq(ssk); + sdp_process_rx_q(ssk); sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ release_sock(sk); sk_stream_mem_reclaim(sk); lock_sock(sk); - rx_cq = ssk->rx_cq; + rx_cq = ssk->rx_ring.cq; if (unlikely(!rx_cq)) goto out; - sdp_arm_rx_cq(sk); - sdp_poll_rx_cq(ssk); + sdp_process_rx_q(ssk); sdp_xmit_poll(ssk, 1); + out: release_sock(sk); } -void sdp_rx_irq(struct ib_cq *cq, void *cq_context) +static void sdp_rx_irq(struct ib_cq *cq, void *cq_context) { struct sock *sk = cq_context; struct sdp_sock *ssk = sdp_sk(sk); + unsigned long flags; + int wc_processed = 0; - WARN_ON(ssk->rx_cq && cq != ssk->rx_cq); + sdp_dbg_data(&ssk->isk.sk, "rx irq called\n"); - if (!ssk->rx_cq) - sdp_warn(&ssk->isk.sk, "WARNING: rx irq after cq destroyed\n"); + WARN_ON(cq != ssk->rx_ring.cq); SDPSTATS_COUNTER_INC(rx_int_count); - sdp_prf(sk, NULL, "rx completion"); + sdp_prf(sk, NULL, "rx irq"); + + rx_ring_lock(ssk, flags); + + if (unlikely(!ssk->poll_cq)) + sdp_warn(sk, "poll cq is 0. socket was reset or wasn't initialized\n"); + + if (!ssk->rx_ring.cq) { + sdp_warn(&ssk->isk.sk, "WARNING: rx irq after cq destroyed\n"); + + goto out; + } + + wc_processed = sdp_poll_rx_cq(ssk); + sdp_prf(&ssk->isk.sk, NULL, "processed %d", wc_processed); + + if (wc_processed) { + _sdp_post_recvs(ssk); + + /* Best was to send credit update from here */ +/* sdp_post_credits(ssk); */ - /* issue sdp_rx_comp_work() */ - queue_work(rx_comp_wq, &ssk->rx_comp_work); + /* issue sdp_rx_comp_work() */ + queue_work(rx_comp_wq, &ssk->rx_comp_work); + } + + sdp_arm_rx_cq(sk); + +out: + rx_ring_unlock(ssk, flags); } -void sdp_rx_ring_purge(struct sdp_sock *ssk) +static void sdp_rx_ring_purge(struct sdp_sock *ssk) { - struct sk_buff *skb; + WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); - while (ssk->rx_head != ssk->rx_tail) { + while (ring_posted(ssk->rx_ring) > 0) { struct sk_buff *skb; - skb = sdp_recv_completion(ssk, ssk->rx_tail); + skb = sdp_recv_completion(ssk, ring_tail(ssk->rx_ring)); if (!skb) break; atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); __kfree_skb(skb); } } + +void sdp_rx_ring_init(struct sdp_sock *ssk) +{ + ssk->rx_ring.buffer = NULL; + spin_lock_init(&ssk->rx_ring.lock); +} + +static void sdp_rx_cq_event_handler(struct ib_event *event, void *data) +{ +} + +int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device) +{ + struct ib_cq *rx_cq; + int rc = 0; + unsigned long flags; + + rx_ring_lock(ssk, flags); + + atomic_set(&ssk->rx_ring.head, 1); + atomic_set(&ssk->rx_ring.tail, 1); + + ssk->rx_ring.buffer = kmalloc(sizeof *ssk->rx_ring.buffer * SDP_RX_SIZE, + GFP_KERNEL); + if (!ssk->rx_ring.buffer) { + rc = -ENOMEM; + sdp_warn(&ssk->isk.sk, "Unable to allocate RX Ring size %zd.\n", + sizeof(*ssk->rx_ring.buffer) * SDP_RX_SIZE); + + goto out; + } + + rx_cq = ib_create_cq(device, sdp_rx_irq, sdp_rx_cq_event_handler, + &ssk->isk.sk, SDP_RX_SIZE, 0); + + if (IS_ERR(rx_cq)) { + rc = PTR_ERR(rx_cq); + sdp_warn(&ssk->isk.sk, "Unable to allocate RX CQ: %d.\n", rc); + goto err_cq; + } + + rc = ib_modify_cq(rx_cq, 10, 200); + if (rc) { + sdp_warn(&ssk->isk.sk, "Unable to modify RX CQ: %d.\n", rc); + goto err_mod; + } + sdp_warn(&ssk->isk.sk, "Initialized CQ moderation\n"); + sdp_sk(&ssk->isk.sk)->rx_ring.cq = rx_cq; + + INIT_WORK(&ssk->rx_comp_work, sdp_rx_comp_work); + + sdp_arm_rx_cq(&ssk->isk.sk); + + goto out; + +err_mod: + ib_destroy_cq(rx_cq); +err_cq: + kfree(ssk->rx_ring.buffer); + ssk->rx_ring.buffer = NULL; +out: + rx_ring_unlock(ssk, flags); + return rc; +} + +void sdp_rx_ring_destroy(struct sdp_sock *ssk) +{ + WARN_ON_UNLOCKED(&ssk->isk.sk, &ssk->rx_ring.lock); + + if (ssk->rx_ring.buffer) { + sdp_rx_ring_purge(ssk); + + kfree(ssk->rx_ring.buffer); + ssk->rx_ring.buffer = NULL; + } + + if (ssk->rx_ring.cq) { + ib_destroy_cq(ssk->rx_ring.cq); + ssk->rx_ring.cq = NULL; + } + + WARN_ON(ring_head(ssk->rx_ring) != ring_tail(ssk->rx_ring)); +} diff --git a/drivers/infiniband/ulp/sdp/sdp_tx.c b/drivers/infiniband/ulp/sdp/sdp_tx.c index 5e6a2dc..de9d792 100644 --- a/drivers/infiniband/ulp/sdp/sdp_tx.c +++ b/drivers/infiniband/ulp/sdp/sdp_tx.c @@ -55,8 +55,11 @@ int sdp_xmit_poll(struct sdp_sock *ssk, int force) mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); /* Poll the CQ every SDP_TX_POLL_MODER packets */ - if (force || (++ssk->tx_ring.poll_cnt & (SDP_TX_POLL_MODER - 1)) == 0) + if (force || (++ssk->tx_ring.poll_cnt & (SDP_TX_POLL_MODER - 1)) == 0) { wc_processed = sdp_process_tx_cq(ssk); + sdp_prf(&ssk->isk.sk, NULL, "processed %d wc's. inflight=%d", wc_processed, + ring_posted(ssk->tx_ring)); + } return wc_processed; } @@ -65,24 +68,16 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) { struct sdp_buf *tx_req; struct sdp_bsdh *h = (struct sdp_bsdh *)skb_push(skb, sizeof *h); - unsigned mseq = ssk->tx_ring.head; + unsigned mseq = ring_head(ssk->tx_ring); int i, rc, frags; u64 addr; struct ib_device *dev; - struct ib_sge *sge; struct ib_send_wr *bad_wr; -#define ENUM2STR(e) [e] = #e - static char *mid2str[] = { - ENUM2STR(SDP_MID_HELLO), - ENUM2STR(SDP_MID_HELLO_ACK), - ENUM2STR(SDP_MID_DISCONN), - ENUM2STR(SDP_MID_CHRCVBUF), - ENUM2STR(SDP_MID_CHRCVBUF_ACK), - ENUM2STR(SDP_MID_DATA), - }; - sdp_prf(&ssk->isk.sk, skb, "post_send mid = %s, bufs = %d", - mid2str[mid], ssk->rx_head - ssk->rx_tail); + struct ib_sge ibsge[SDP_MAX_SEND_SKB_FRAGS + 1]; + struct ib_sge *sge = ibsge; + struct ib_send_wr tx_wr = { 0 }; + SDPSTATS_COUNTER_MID_INC(post_send, mid); SDPSTATS_HIST(send_size, skb->len); @@ -93,16 +88,19 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) else h->flags = 0; - h->bufs = htons(ssk->rx_head - ssk->rx_tail); + h->bufs = htons(ring_posted(ssk->rx_ring)); h->len = htonl(skb->len); h->mseq = htonl(mseq); - h->mseq_ack = htonl(ssk->mseq_ack); + h->mseq_ack = htonl(mseq_ack(ssk)); + + sdp_prf(&ssk->isk.sk, skb, "TX: %s bufs: %d mseq:%d ack:%d", + mid2str(mid), ring_posted(ssk->rx_ring), mseq, ntohl(h->mseq_ack)); SDP_DUMP_PACKET(&ssk->isk.sk, "TX", skb, h); + tx_req = &ssk->tx_ring.buffer[mseq & (SDP_TX_SIZE - 1)]; tx_req->skb = skb; dev = ssk->ib_device; - sge = ssk->ibsge; addr = ib_dma_map_single(dev, skb->data, skb->len - skb->data_len, DMA_TO_DEVICE); tx_req->mapping[0] = addr; @@ -127,14 +125,14 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) sge->lkey = ssk->mr->lkey; } - ssk->tx_wr.next = NULL; - ssk->tx_wr.wr_id = ssk->tx_ring.head | SDP_OP_SEND; - ssk->tx_wr.sg_list = ssk->ibsge; - ssk->tx_wr.num_sge = frags + 1; - ssk->tx_wr.opcode = IB_WR_SEND; - ssk->tx_wr.send_flags = IB_SEND_SIGNALED; + tx_wr.next = NULL; + tx_wr.wr_id = ring_head(ssk->tx_ring) | SDP_OP_SEND; + tx_wr.sg_list = ibsge; + tx_wr.num_sge = frags + 1; + tx_wr.opcode = IB_WR_SEND; + tx_wr.send_flags = IB_SEND_SIGNALED; if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_URG)) - ssk->tx_wr.send_flags |= IB_SEND_SOLICITED; + tx_wr.send_flags |= IB_SEND_SOLICITED; { static unsigned long last_send = 0; @@ -145,19 +143,15 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) last_send = jiffies; } - rc = ib_post_send(ssk->qp, &ssk->tx_wr, &bad_wr); - ++ssk->tx_ring.head; - --ssk->tx_ring.credits; - ssk->remote_credits = ssk->rx_head - ssk->rx_tail; + rc = ib_post_send(ssk->qp, &tx_wr, &bad_wr); + atomic_inc(&ssk->tx_ring.head); + atomic_dec(&ssk->tx_ring.credits); + atomic_set(&ssk->remote_credits, ring_posted(ssk->rx_ring)); if (unlikely(rc)) { sdp_dbg(&ssk->isk.sk, "ib_post_send failed with status %d.\n", rc); sdp_set_error(&ssk->isk.sk, -ECONNRESET); wake_up(&ssk->wq); } - - if (ssk->tx_ring.credits <= SDP_MIN_TX_CREDITS) { - sdp_poll_rx_cq(ssk); - } } static struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq) @@ -168,9 +162,9 @@ static struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq) struct bzcopy_state *bz; int i, frags; struct sdp_tx_ring *tx_ring = &ssk->tx_ring; - if (unlikely(mseq != tx_ring->tail)) { + if (unlikely(mseq != ring_tail(*tx_ring))) { printk(KERN_WARNING "Bogus send completion id %d tail %d\n", - mseq, tx_ring->tail); + mseq, ring_tail(*tx_ring)); goto out; } @@ -193,7 +187,7 @@ static struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq) if (bz) bz->busy--; - ++tx_ring->tail; + atomic_inc(&tx_ring->tail); out: return skb; @@ -219,7 +213,11 @@ static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) } } - sdp_prf(&ssk->isk.sk, skb, "tx completion"); + { + struct sdp_bsdh *h = (struct sdp_bsdh *)skb->data; + sdp_prf(&ssk->isk.sk, skb, "tx completion. mseq:%d", ntohl(h->mseq)); + } + sk_stream_free_skb(&ssk->isk.sk, skb); return 0; @@ -281,15 +279,14 @@ static int sdp_process_tx_cq(struct sdp_sock *ssk) return wc_processed; } -void sdp_poll_tx_cq(unsigned long data) +static void sdp_poll_tx_timeout(unsigned long data) { struct sdp_sock *ssk = (struct sdp_sock *)data; struct sock *sk = &ssk->isk.sk; u32 inflight, wc_processed; - sdp_dbg_data(&ssk->isk.sk, "Polling tx cq. inflight=%d\n", - (u32) ssk->tx_ring.head - ssk->tx_ring.tail); + (u32) ring_posted(ssk->tx_ring)); /* Only process if the socket is not in use */ bh_lock_sock(sk); @@ -303,12 +300,14 @@ void sdp_poll_tx_cq(unsigned long data) goto out; wc_processed = sdp_process_tx_cq(ssk); + sdp_prf(&ssk->isk.sk, NULL, "processed %d wc's. inflight=%d", wc_processed, + ring_posted(ssk->tx_ring)); if (!wc_processed) SDPSTATS_COUNTER_INC(tx_poll_miss); else SDPSTATS_COUNTER_INC(tx_poll_hit); - inflight = (u32) ssk->tx_ring.head - ssk->tx_ring.tail; + inflight = (u32) ring_posted(ssk->tx_ring); /* If there are still packets in flight and the timer has not already * been scheduled by the Tx routine then schedule it here to guarantee @@ -322,17 +321,7 @@ out: bh_unlock_sock(sk); } -void _sdp_poll_tx_cq(unsigned long data) -{ - struct sdp_sock *ssk = (struct sdp_sock *)data; - struct sock *sk = &ssk->isk.sk; - - sdp_prf(sk, NULL, "sdp poll tx timeout expired"); - - sdp_poll_tx_cq(data); -} - -void sdp_tx_irq(struct ib_cq *cq, void *cq_context) +static void sdp_tx_irq(struct ib_cq *cq, void *cq_context) { struct sock *sk = cq_context; struct sdp_sock *ssk = sdp_sk(sk); @@ -344,11 +333,9 @@ void sdp_tx_irq(struct ib_cq *cq, void *cq_context) void sdp_tx_ring_purge(struct sdp_sock *ssk) { - struct sk_buff *skb; - - while (ssk->tx_ring.head != ssk->tx_ring.tail) { + while (ring_posted(ssk->tx_ring)) { struct sk_buff *skb; - skb = sdp_send_completion(ssk, ssk->tx_ring.tail); + skb = sdp_send_completion(ssk, ring_tail(ssk->tx_ring)); if (!skb) break; __kfree_skb(skb); @@ -380,3 +367,66 @@ void sdp_post_keepalive(struct sdp_sock *ssk) sdp_cnt(sdp_keepalive_probes_sent); } +static void sdp_tx_cq_event_handler(struct ib_event *event, void *data) +{ +} + +int sdp_tx_ring_create(struct sdp_sock *ssk, struct ib_device *device) +{ + struct ib_cq *tx_cq; + int rc = 0; + + atomic_set(&ssk->tx_ring.head, 1); + atomic_set(&ssk->tx_ring.tail, 1); + + ssk->tx_ring.buffer = kmalloc(sizeof *ssk->tx_ring.buffer * SDP_TX_SIZE, + GFP_KERNEL); + if (!ssk->tx_ring.buffer) { + rc = -ENOMEM; + sdp_warn(&ssk->isk.sk, "Unable to allocate TX Ring size %zd.\n", + sizeof(*ssk->tx_ring.buffer) * SDP_TX_SIZE); + + goto out; + } + + tx_cq = ib_create_cq(device, sdp_tx_irq, sdp_tx_cq_event_handler, + &ssk->isk.sk, SDP_TX_SIZE, 0); + + if (IS_ERR(tx_cq)) { + rc = PTR_ERR(tx_cq); + sdp_warn(&ssk->isk.sk, "Unable to allocate TX CQ: %d.\n", rc); + goto err_cq; + } + + sdp_sk(&ssk->isk.sk)->tx_ring.cq = tx_cq; + + init_timer(&ssk->tx_ring.timer); + ssk->tx_ring.timer.function = sdp_poll_tx_timeout; + ssk->tx_ring.timer.data = (unsigned long) ssk; + ssk->tx_ring.poll_cnt = 0; + + return 0; + +err_cq: + kfree(ssk->tx_ring.buffer); + ssk->tx_ring.buffer = NULL; +out: + return rc; +} + +void sdp_tx_ring_destroy(struct sdp_sock *ssk) +{ + if (ssk->tx_ring.buffer) { + sdp_tx_ring_purge(ssk); + + kfree(ssk->tx_ring.buffer); + ssk->tx_ring.buffer = NULL; + } + + if (ssk->tx_ring.cq) { + ib_destroy_cq(ssk->tx_ring.cq); + ssk->tx_ring.cq = NULL; + } + + WARN_ON(ring_head(ssk->tx_ring) != ring_tail(ssk->tx_ring)); +} -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:43 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:43 +0300 Subject: [ofa-general] [PATCH] sdp: no tx interrupts In-Reply-To: <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> poll tx cq with timer instead of interrupts Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 34 ++++-- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 222 ++++++++++++++++++++++++------- drivers/infiniband/ulp/sdp/sdp_cma.c | 72 +++++++---- drivers/infiniband/ulp/sdp/sdp_main.c | 83 ++++++++----- drivers/infiniband/ulp/sdp/sdp_proc.c | 4 +- 5 files changed, 297 insertions(+), 118 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index cbef0eb..8128065 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -83,12 +83,14 @@ struct sdpstats { u32 sendmsg_seglen[25]; u32 send_size[25]; u32 post_recv; - u32 int_count; + u32 rx_int_count; + u32 tx_int_count; u32 bzcopy_poll_miss; u32 send_wait_for_mem; u32 send_miss_no_credits; u32 rx_poll_miss; u32 tx_poll_miss; + u32 tx_poll_hit; u32 memcpy_count; u32 credits_before_update[64]; u32 send_interval[25]; @@ -132,6 +134,11 @@ static inline void sdpstats_hist(u32 *h, u32 val, u32 maxidx, int is_log) #define sock_put(sk, msg) sock_ref(sk, msg, sock_put) #define __sock_put(sk, msg) sock_ref(sk, msg, __sock_put) +/* Interval between sucessive polls in the Tx routine when polling is used + instead of interrupts (in per-core Tx rings) - should be power of 2 */ +#define SDP_TX_POLL_MODER 16 +#define SDP_TX_POLL_TIMEOUT (HZ / 4) + #define SDP_RESOLVE_TIMEOUT 1000 #define SDP_ROUTE_TIMEOUT 1000 #define SDP_RETRY_COUNT 5 @@ -242,7 +249,10 @@ struct sdp_tx_ring { int una_seq; unsigned credits; + + struct timer_list timer; u16 poll_cnt; + struct ib_cq *cq; }; static inline int sdp_tx_ring_slots_left(struct sdp_tx_ring *tx_ring) @@ -258,7 +268,7 @@ struct sdp_sock { struct list_head backlog_queue; struct sock *parent; - struct work_struct work; + struct work_struct rx_comp_work; wait_queue_head_t wq; struct delayed_work dreq_wait_work; @@ -308,7 +318,7 @@ struct sdp_sock { /* rdma specific */ struct ib_qp *qp; - struct ib_cq *cq; + struct ib_cq *rx_cq; struct ib_mr *mr; struct sdp_buf *rx_ring; @@ -349,7 +359,7 @@ struct bzcopy_state { extern int rcvbuf_initial_size; extern struct proto sdp_proto; -extern struct workqueue_struct *comp_wq; +extern struct workqueue_struct *rx_comp_wq; extern atomic_t sdp_current_mem_usage; extern spinlock_t sdp_large_sockets_lock; @@ -440,11 +450,11 @@ static inline void sdp_set_error(struct sock *sk, int err) sk->sk_error_report(sk); } -static inline void sdp_arm_cq(struct sock *sk) +static inline void sdp_arm_rx_cq(struct sock *sk) { - sdp_dbg_data(sk, "ib_req_notify_cq on cq\n"); + sdp_dbg_data(sk, "ib_req_notify_cq on RX cq\n"); - ib_req_notify_cq(sdp_sk(sk)->cq, IB_CQ_NEXT_COMP); + ib_req_notify_cq(sdp_sk(sk)->rx_cq, IB_CQ_NEXT_COMP); } #ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA @@ -453,12 +463,15 @@ void dump_packet(struct sock *sk, char *str, struct sk_buff *skb, const struct s int sdp_cma_handler(struct rdma_cm_id *, struct rdma_cm_event *); void sdp_reset(struct sock *sk); void sdp_reset_sk(struct sock *sk, int rc); -void sdp_completion_handler(struct ib_cq *cq, void *cq_context); -void sdp_work(struct work_struct *work); +void sdp_rx_irq(struct ib_cq *cq, void *cq_context); +void sdp_tx_irq(struct ib_cq *cq, void *cq_context); +void sdp_poll_tx_cq(unsigned long data); +void sdp_rx_comp_work(struct work_struct *work); +void sdp_process_tx_wc_work(struct work_struct *work); int sdp_post_credits(struct sdp_sock *ssk); void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid); void sdp_post_recvs(struct sdp_sock *ssk); -int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq); +int sdp_poll_rx_cq(struct sdp_sock *ssk); void sdp_post_sends(struct sdp_sock *ssk, int nonagle); void sdp_destroy_work(struct work_struct *work); void sdp_cancel_dreq_wait_timeout(struct sdp_sock *ssk); @@ -477,6 +490,7 @@ void sdp_bzcopy_write_space(struct sdp_sock *ssk); int sdp_init_sock(struct sock *sk); int __init sdp_proc_init(void); void sdp_proc_unregister(void); +int sdp_xmit_poll(struct sdp_sock *ssk, int force); static inline struct sk_buff *sdp_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) { diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 9f7f4a0..29c9761 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -244,6 +244,24 @@ void dump_packet(struct sock *sk, char *str, struct sk_buff *skb, const struct s } #endif +static int sdp_process_tx_cq(struct sdp_sock *ssk); + +int sdp_xmit_poll(struct sdp_sock *ssk, int force) +{ + int wc_processed = 0; + + /* If we don't have a pending timer, set one up to catch our recent + post in case the interface becomes idle */ + if (!timer_pending(&ssk->tx_ring.timer)) + mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); + + /* Poll the CQ every SDP_TX_POLL_MODER packets */ + if (force || (++ssk->tx_ring.poll_cnt & (SDP_TX_POLL_MODER - 1)) == 0) + wc_processed = sdp_process_tx_cq(ssk); + + return wc_processed; +} + void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) { struct sdp_buf *tx_req; @@ -573,6 +591,7 @@ int sdp_post_credits(struct sdp_sock *ssk) if (!skb) return -ENOMEM; sdp_post_send(ssk, skb, SDP_MID_DATA); + sdp_xmit_poll(ssk, 0); } return 0; } @@ -583,6 +602,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) struct sk_buff *skb; int c; gfp_t gfp_page; + int post_count = 0; if (unlikely(!ssk->id)) { if (ssk->isk.sk.sk_send_head) { @@ -602,7 +622,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) if (ssk->recv_request && ssk->rx_tail >= ssk->recv_request_head && ssk->tx_ring.credits >= SDP_MIN_TX_CREDITS && - ssk->tx_ring.head - ssk->tx_ring.tail < SDP_TX_SIZE) { + sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; skb = sdp_stream_alloc_skb(&ssk->isk.sk, @@ -614,6 +634,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) resp_size = (struct sdp_chrecvbuf *)skb_put(skb, sizeof *resp_size); resp_size->size = htonl(ssk->recv_frags * PAGE_SIZE); sdp_post_send(ssk, skb, SDP_MID_CHRCVBUF_ACK); + post_count++; } if (ssk->tx_ring.credits <= SDP_MIN_TX_CREDITS && @@ -634,12 +655,13 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) update_send_head(&ssk->isk.sk, skb); __skb_dequeue(&ssk->isk.sk.sk_write_queue); sdp_post_send(ssk, skb, SDP_MID_DATA); + post_count++; } if (ssk->tx_ring.credits == SDP_MIN_TX_CREDITS && !ssk->sent_request && ssk->tx_ring.head > ssk->sent_request_head + SDP_RESIZE_WAIT && - ssk->tx_ring.head - ssk->tx_ring.tail < SDP_TX_SIZE) { + sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *req_size; skb = sdp_stream_alloc_skb(&ssk->isk.sk, sizeof(struct sdp_bsdh) + @@ -652,6 +674,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) req_size = (struct sdp_chrecvbuf *)skb_put(skb, sizeof *req_size); req_size->size = htonl(ssk->sent_request); sdp_post_send(ssk, skb, SDP_MID_CHRCVBUF); + post_count++; } c = ssk->remote_credits; @@ -660,7 +683,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) if (unlikely(c < ssk->rx_head - ssk->rx_tail) && likely(ssk->tx_ring.credits > 1) && - likely(ssk->tx_ring.head - ssk->tx_ring.tail < SDP_TX_SIZE) && + likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { skb = sdp_stream_alloc_skb(&ssk->isk.sk, @@ -670,6 +693,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) BUG_ON(!skb); SDPSTATS_COUNTER_INC(post_send_credits); sdp_post_send(ssk, skb, SDP_MID_DATA); + post_count++; } /* send DisConn if needed @@ -687,7 +711,10 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) /* FIXME */ BUG_ON(!skb); sdp_post_send(ssk, skb, SDP_MID_DISCONN); + post_count++; } + if (post_count) + sdp_xmit_poll(ssk, 0); } int sdp_init_buffers(struct sdp_sock *ssk, u32 new_size) @@ -867,7 +894,6 @@ static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) { struct sk_buff *skb = NULL; - struct sdp_bsdh *h; skb = sdp_send_completion(ssk, wc->wr_id); if (unlikely(!skb)) @@ -881,72 +907,162 @@ static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) sdp_set_error(sk, -ECONNRESET); wake_up(&ssk->wq); - queue_work(comp_wq, &ssk->destroy_work); + queue_work(rx_comp_wq, &ssk->destroy_work); } - goto out; } - h = (struct sdp_bsdh *)skb->data; + sk_stream_free_skb(&ssk->isk.sk, skb); - if (likely(h->mid != SDP_MID_DISCONN)) - goto out; + return 0; +} + +void sdp_rx_irq(struct ib_cq *cq, void *cq_context) +{ + struct sock *sk = cq_context; + struct sdp_sock *ssk = sdp_sk(sk); + + WARN_ON(ssk->rx_cq && cq != ssk->rx_cq); + + if (!ssk->rx_cq) + sdp_warn(&ssk->isk.sk, "WARNING: rx irq after cq destroyed\n"); + + SDPSTATS_COUNTER_INC(rx_int_count); + + queue_work(rx_comp_wq, &ssk->rx_comp_work); +} - if ((1 << ssk->isk.sk.sk_state) & ~(TCPF_FIN_WAIT1 | TCPF_LAST_ACK)) { - sdp_dbg(&ssk->isk.sk, - "%s: sent DISCONNECT from unexpected state %d\n", - __func__, ssk->isk.sk.sk_state); +static inline void sdp_process_tx_wc(struct sdp_sock *ssk, struct ib_wc *wc) +{ + if (likely(wc->wr_id & SDP_OP_SEND)) { + sdp_handle_send_comp(ssk, wc); + return; } -out: sk_wmem_free_skb(&ssk->isk.sk, skb); - return 0; + /* Keepalive probe sent cleanup */ + sdp_cnt(sdp_keepalive_probes_sent); + + if (likely(!wc->status)) + return; + + sdp_dbg(&ssk->isk.sk, " %s consumes KEEPALIVE status %d\n", + __func__, wc->status); + + if (wc->status == IB_WC_WR_FLUSH_ERR) + return; + + sdp_set_error(&ssk->isk.sk, -ECONNRESET); + wake_up(&ssk->wq); } -static void sdp_handle_wc(struct sdp_sock *ssk, struct ib_wc *wc) +static int sdp_process_tx_cq(struct sdp_sock *ssk) { - if (wc->wr_id & SDP_OP_RECV) { - if (sdp_handle_recv_comp(ssk, wc)) - return; - } else if (likely(wc->wr_id & SDP_OP_SEND)) { - if (sdp_handle_send_comp(ssk, wc)) - return; - } else { - sdp_cnt(sdp_keepalive_probes_sent); + struct ib_wc ibwc[SDP_NUM_WC]; + int n, i; + int wc_processed = 0; - if (likely(!wc->status)) - return; + if (!ssk->tx_ring.cq) { + sdp_warn(&ssk->isk.sk, "WARNING: tx irq when tx_cq is destroyed\n"); + return 0; + } + + do { + n = ib_poll_cq(ssk->tx_ring.cq, SDP_NUM_WC, ibwc); + for (i = 0; i < n; ++i) { + sdp_process_tx_wc(ssk, ibwc + i); + wc_processed++; + } + } while (n == SDP_NUM_WC); - sdp_dbg(&ssk->isk.sk, " %s consumes KEEPALIVE status %d\n", - __func__, wc->status); + sdp_dbg_data(&ssk->isk.sk, "processed %d wc's\n", wc_processed); - if (wc->status == IB_WC_WR_FLUSH_ERR) - return; + if (wc_processed) { + struct sock *sk = &ssk->isk.sk; + sdp_post_sends(ssk, 0); - sdp_set_error(&ssk->isk.sk, -ECONNRESET); - wake_up(&ssk->wq); + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + sk_stream_write_space(&ssk->isk.sk); - return; } + + return wc_processed; } -void sdp_completion_handler(struct ib_cq *cq, void *cq_context) +void sdp_poll_tx_cq(unsigned long data) +{ + struct sdp_sock *ssk = (struct sdp_sock *)data; + struct sock *sk = &ssk->isk.sk; + u32 inflight, wc_processed; + + sdp_dbg_data(&ssk->isk.sk, "Polling tx cq. inflight=%d\n", + (u32) ssk->tx_ring.head - ssk->tx_ring.tail); + + /* Only process if the socket is not in use */ + bh_lock_sock(sk); + if (sock_owned_by_user(sk)) { + mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); + sdp_dbg_data(&ssk->isk.sk, "socket is busy - trying later\n"); + goto out; + } + + if (sk->sk_state == TCP_CLOSE) + goto out; + + wc_processed = sdp_process_tx_cq(ssk); + if (!wc_processed) + SDPSTATS_COUNTER_INC(tx_poll_miss); + else + SDPSTATS_COUNTER_INC(tx_poll_hit); + + inflight = (u32) ssk->tx_ring.head - ssk->tx_ring.tail; + + /* If there are still packets in flight and the timer has not already + * been scheduled by the Tx routine then schedule it here to guarantee + * completion processing of these packets */ + if (inflight) { /* TODO: make sure socket is not closed */ + sdp_dbg_data(sk, "arming timer for more polling\n"); + mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); + } + +out: + bh_unlock_sock(sk); +} + + +void sdp_tx_irq(struct ib_cq *cq, void *cq_context) { struct sock *sk = cq_context; struct sdp_sock *ssk = sdp_sk(sk); - schedule_work(&ssk->work); - SDPSTATS_COUNTER_INC(int_count); + + sdp_warn(sk, "Got tx comp interrupt\n"); + + mod_timer(&ssk->tx_ring.timer, jiffies + 1); } -int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq) + +int sdp_poll_rx_cq(struct sdp_sock *ssk) { + struct ib_cq *cq = ssk->rx_cq; struct ib_wc ibwc[SDP_NUM_WC]; int n, i; int ret = -EAGAIN; + int updated_credits = 0; + do { n = ib_poll_cq(cq, SDP_NUM_WC, ibwc); for (i = 0; i < n; ++i) { - sdp_handle_wc(ssk, &ibwc[i]); + struct ib_wc *wc = &ibwc[i]; + + BUG_ON(!(wc->wr_id & SDP_OP_RECV)); + sdp_handle_recv_comp(ssk, wc); + + if (!updated_credits) { + sdp_post_recvs(ssk); + sdp_post_sends(ssk, 0); + updated_credits = 1; + } + ret = 0; } } while (n == SDP_NUM_WC); @@ -968,17 +1084,20 @@ int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq) return ret; } -void sdp_work(struct work_struct *work) +static inline int sdp_tx_qp_empty(struct sdp_sock *ssk) { - struct sdp_sock *ssk = container_of(work, struct sdp_sock, work); - struct sock *sk = &ssk->isk.sk; - struct ib_cq *cq; + return (ssk->tx_ring.head - ssk->tx_ring.tail) == 0; +} - sdp_dbg_data(sk, "%s\n", __func__); +void sdp_rx_comp_work(struct work_struct *work) +{ + struct sdp_sock *ssk = container_of(work, struct sdp_sock, rx_comp_work); + struct sock *sk = &ssk->isk.sk; + struct ib_cq *rx_cq; lock_sock(sk); - cq = ssk->cq; - if (unlikely(!cq)) + rx_cq = ssk->rx_cq; + if (unlikely(!rx_cq)) goto out; if (unlikely(!ssk->poll_cq)) { @@ -988,15 +1107,18 @@ void sdp_work(struct work_struct *work) goto out; } - sdp_poll_cq(ssk, cq); + sdp_poll_rx_cq(ssk); + sdp_xmit_poll(ssk, 1); /* if has pending tx because run out of tx_credits - xmit it */ release_sock(sk); sk_mem_reclaim(sk); lock_sock(sk); - cq = ssk->cq; - if (unlikely(!cq)) + rx_cq = ssk->rx_cq; + if (unlikely(!rx_cq)) goto out; - sdp_arm_cq(sk); - sdp_poll_cq(ssk, cq); + + sdp_arm_rx_cq(sk); + sdp_poll_rx_cq(ssk); + sdp_xmit_poll(ssk, 1); out: release_sock(sk); } diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index 1e1ff9d..7c34637 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -73,7 +73,7 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) .qp_type = IB_QPT_RC, }; struct ib_device *device = id->device; - struct ib_cq *cq; + struct ib_cq *rx_cq, *tx_cq; struct ib_mr *mr; struct ib_pd *pd; int rc; @@ -118,32 +118,49 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) } sdp_sk(sk)->mr = mr; - INIT_WORK(&sdp_sk(sk)->work, sdp_work); + INIT_WORK(&sdp_sk(sk)->rx_comp_work, sdp_rx_comp_work); - cq = ib_create_cq(device, sdp_completion_handler, sdp_cq_event_handler, - sk, SDP_TX_SIZE + SDP_RX_SIZE, 0); + rx_cq = ib_create_cq(device, sdp_rx_irq, sdp_cq_event_handler, + sk, SDP_RX_SIZE, 0); - if (IS_ERR(cq)) { - rc = PTR_ERR(cq); - sdp_warn(sk, "Unable to allocate CQ: %d.\n", rc); - goto err_cq; + if (IS_ERR(rx_cq)) { + rc = PTR_ERR(rx_cq); + sdp_warn(sk, "Unable to allocate RX CQ: %d.\n", rc); + goto err_rx_cq; } - rc = ib_modify_cq(cq, 10, 200); + rc = ib_modify_cq(rx_cq, 10, 200); if (rc) { sdp_warn(sk, "Unable to modify RX CQ: %d.\n", rc); - goto err_qp; + goto err_tx_cq; } sdp_warn(sk, "Initialized CQ moderation\n"); + sdp_sk(sk)->rx_cq = rx_cq; + sdp_arm_rx_cq(sk); + qp_init_attr.recv_cq = rx_cq; + + tx_cq = ib_create_cq(device, sdp_tx_irq, sdp_cq_event_handler, + sk, SDP_TX_SIZE, 0); + + if (IS_ERR(tx_cq)) { + rc = PTR_ERR(tx_cq); + sdp_warn(sk, "Unable to allocate TX CQ: %d.\n", rc); + goto err_tx_cq; + } - qp_init_attr.send_cq = qp_init_attr.recv_cq = cq; + init_timer(&sdp_sk(sk)->tx_ring.timer); + sdp_sk(sk)->tx_ring.timer.function = sdp_poll_tx_cq; + sdp_sk(sk)->tx_ring.timer.data = (unsigned long) sdp_sk(sk); + sdp_sk(sk)->tx_ring.poll_cnt = 0; + + sdp_sk(sk)->tx_ring.cq = tx_cq; + qp_init_attr.send_cq = tx_cq; rc = rdma_create_qp(id, pd, &qp_init_attr); if (rc) { sdp_warn(sk, "Unable to create QP: %d.\n", rc); goto err_qp; } - sdp_sk(sk)->cq = cq; sdp_sk(sk)->qp = id->qp; sdp_sk(sk)->ib_device = device; @@ -153,8 +170,10 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) return 0; err_qp: - ib_destroy_cq(cq); -err_cq: + ib_destroy_cq(tx_cq); +err_tx_cq: + ib_destroy_cq(rx_cq); +err_rx_cq: ib_dereg_mr(sdp_sk(sk)->mr); err_mr: ib_dealloc_pd(pd); @@ -162,7 +181,6 @@ err_pd: kfree(sdp_sk(sk)->rx_ring); sdp_sk(sk)->rx_ring = NULL; err_rx: - WARN_ON(sdp_sk(sk)->tx_ring.head != sdp_sk(sk)->tx_ring.tail); kfree(sdp_sk(sk)->tx_ring.buffer); sdp_sk(sk)->tx_ring.buffer = NULL; err_tx: @@ -258,21 +276,20 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, sdp_sk(sk)->min_bufs = sdp_sk(sk)->tx_ring.credits / 4; sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - sizeof(struct sdp_bsdh); - sdp_sk(sk)->send_frags = MIN(PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / - PAGE_SIZE, SDP_MAX_SEND_SKB_FRAGS); - sdp_sk(sk)->xmit_size_goal = MIN(sdp_sk(sk)->xmit_size_goal, - sdp_sk(sk)->send_frags * PAGE_SIZE); + sdp_sk(sk)->send_frags = MIN(PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / + PAGE_SIZE, SDP_MAX_SEND_SKB_FRAGS); + sdp_sk(sk)->xmit_size_goal = MIN(sdp_sk(sk)->xmit_size_goal, + sdp_sk(sk)->send_frags * PAGE_SIZE); - sdp_dbg(sk, "%s bufs %d xmit_size_goal %d send_frags: %d send trigger %d\n", - __func__, + sdp_dbg(sk, "tx credits %d xmit_size_goal %d send_frags: %d credits update trigger %d\n", sdp_sk(sk)->tx_ring.credits, sdp_sk(sk)->xmit_size_goal, - sdp_sk(sk)->send_frags, + sdp_sk(sk)->send_frags, sdp_sk(sk)->min_bufs); sdp_sk(sk)->poll_cq = 1; - sdp_arm_cq(sk); - sdp_poll_cq(sdp_sk(sk), sdp_sk(sk)->cq); + sdp_arm_rx_cq(sk); + sdp_poll_rx_cq(sdp_sk(sk)); sk->sk_state_change(sk); sk_wake_async(sk, 0, POLL_OUT); @@ -332,8 +349,11 @@ static int sdp_disconnected_handler(struct sock *sk) sdp_dbg(sk, "%s\n", __func__); - if (ssk->cq) - sdp_poll_cq(ssk, ssk->cq); + if (ssk->rx_cq) + sdp_poll_rx_cq(ssk); + + if (ssk->tx_ring.cq) + sdp_xmit_poll(ssk, 1); if (sk->sk_state == TCP_SYN_RECV) { sdp_connected_handler(sk, NULL); diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index 6b7494f..454abeb 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -135,7 +135,8 @@ static int sdp_zcopy_thresh = 65536; module_param_named(sdp_zcopy_thresh, sdp_zcopy_thresh, int, 0644); MODULE_PARM_DESC(sdp_zcopy_thresh, "Zero copy send threshold; 0=0ff."); -struct workqueue_struct *comp_wq; +struct workqueue_struct *sdp_wq; +struct workqueue_struct *rx_comp_wq; struct list_head sock_list; spinlock_t sock_list_lock; @@ -205,12 +206,17 @@ static int sdp_get_port(struct sock *sk, unsigned short snum) static void sdp_destroy_qp(struct sdp_sock *ssk) { struct ib_pd *pd = NULL; - struct ib_cq *cq = NULL; + struct ib_cq *rx_cq = NULL; + struct ib_cq *tx_cq = NULL; + + del_timer(&ssk->tx_ring.timer); if (ssk->qp) { pd = ssk->qp->pd; - cq = ssk->cq; - ssk->cq = NULL; + rx_cq = ssk->rx_cq; + ssk->rx_cq = NULL; + tx_cq = ssk->tx_ring.cq; + ssk->tx_ring.cq = NULL; ib_destroy_qp(ssk->qp); ssk->qp = NULL; @@ -231,8 +237,12 @@ static void sdp_destroy_qp(struct sdp_sock *ssk) } } - if (cq) - ib_destroy_cq(cq); + if (tx_cq) { + ib_destroy_cq(tx_cq); + } + + if (rx_cq) + ib_destroy_cq(rx_cq); if (ssk->mr) { ib_dereg_mr(ssk->mr); @@ -341,8 +351,11 @@ void sdp_reset_sk(struct sock *sk, int rc) read_lock(&device_removal_lock); - if (ssk->cq) - sdp_poll_cq(ssk, ssk->cq); + if (ssk->rx_cq) + sdp_poll_rx_cq(ssk); + + if (ssk->tx_ring.cq) + sdp_xmit_poll(ssk, 1); if (!(sk->sk_shutdown & RCV_SHUTDOWN) || !sk_stream_memory_free(sk)) sdp_set_error(sk, rc); @@ -355,7 +368,7 @@ void sdp_reset_sk(struct sock *sk, int rc) /* Don't destroy socket before destroy work does its job */ sock_hold(sk, SOCK_REF_RESET); - queue_work(comp_wq, &ssk->destroy_work); + queue_work(sdp_wq, &ssk->destroy_work); read_unlock(&device_removal_lock); } @@ -773,11 +786,10 @@ out: release_sock(sk); if (newsk) { lock_sock(newsk); - if (newssk->cq) { - sdp_dbg(newsk, "%s: ib_req_notify_cq\n", __func__); + if (newssk->rx_cq) { newssk->poll_cq = 1; - sdp_arm_cq(&newssk->isk.sk); - sdp_poll_cq(newssk, newssk->cq); + sdp_arm_rx_cq(&newssk->isk.sk); + sdp_poll_rx_cq(newssk); } release_sock(newsk); } @@ -847,7 +859,7 @@ static inline void sdp_start_dreq_wait_timeout(struct sdp_sock *ssk, int timeo) { sdp_dbg(&ssk->isk.sk, "Starting dreq wait timeout\n"); - queue_delayed_work(comp_wq, &ssk->dreq_wait_work, timeo); + queue_delayed_work(sdp_wq, &ssk->dreq_wait_work, timeo); ssk->dreq_wait_timeout = 1; } @@ -1143,9 +1155,9 @@ static int sdp_getsockopt(struct sock *sk, int level, int optname, static inline int poll_recv_cq(struct sock *sk) { int i; - if (sdp_sk(sk)->cq) { + if (sdp_sk(sk)->rx_cq) { for (i = 0; i < recv_poll; ++i) - if (!sdp_poll_cq(sdp_sk(sk), sdp_sk(sk)->cq)) { + if (!sdp_poll_rx_cq(sdp_sk(sk))) { ++recv_poll_hit; return 0; } @@ -1157,9 +1169,9 @@ static inline int poll_recv_cq(struct sock *sk) static inline void poll_send_cq(struct sock *sk) { int i; - if (sdp_sk(sk)->cq) { + if (sdp_sk(sk)->tx_ring.cq) { for (i = 0; i < send_poll; ++i) - if (!sdp_poll_cq(sdp_sk(sk), sdp_sk(sk)->cq)) { + if (sdp_xmit_poll(sdp_sk(sk), 1)) { ++send_poll_hit; return; } @@ -1421,12 +1433,9 @@ static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, /* We can extend the last page * fragment. */ merge = 1; - } else if (i == ssk->send_frags || - (!i && - !(sk->sk_route_caps & NETIF_F_SG))) { + } else if (i == ssk->send_frags) { /* Need to add new fragment and cannot - * do this because interface is non-SG, - * or because all the page slots are + * do this because all the page slots are * busy. */ sdp_mark_push(ssk, skb); return SDP_NEW_SEG; @@ -1649,7 +1658,6 @@ void sdp_bzcopy_write_space(struct sdp_sock *ssk) } } - /* Like tcp_sendmsg */ /* TODO: check locking */ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, @@ -1812,7 +1820,9 @@ wait_for_sndbuf: wait_for_memory: SDPSTATS_COUNTER_INC(send_wait_for_mem); if (copied) - sdp_push(sk, ssk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); + sdp_push(sk, ssk, flags & ~MSG_MORE, PAGE_SIZE, TCP_NAGLE_PUSH); + + sdp_xmit_poll(ssk, 1); err = (bz) ? sdp_bzcopy_wait_memory(ssk, &timeo, bz) : sk_stream_wait_memory(sk, &timeo); @@ -2386,17 +2396,25 @@ static int __init sdp_init(void) sdp_proto.sockets_allocated = sockets_allocated; sdp_proto.orphan_count = orphan_count; - comp_wq = create_singlethread_workqueue("comp_wq"); - if (!comp_wq) + rx_comp_wq = create_singlethread_workqueue("rx_comp_wq"); + if (!rx_comp_wq) goto no_mem_rx_wq; + sdp_wq = create_singlethread_workqueue("sdp_wq"); + if (!sdp_wq) + goto no_mem_sdp_wq; + rc = proto_register(&sdp_proto, 1); - if (rc) + if (rc) { + printk(KERN_WARNING "%s: proto_register failed: %d\n", __func__, rc); goto error_proto_reg; + } rc = sock_register(&sdp_net_proto); - if (rc) + if (rc) { + printk(KERN_WARNING "%s: sock_register failed: %d\n", __func__, rc); goto error_sock_reg; + } sdp_proc_init(); @@ -2409,7 +2427,9 @@ static int __init sdp_init(void) error_sock_reg: proto_unregister(&sdp_proto); error_proto_reg: - destroy_workqueue(comp_wq); + destroy_workqueue(sdp_wq); +no_mem_sdp_wq: + destroy_workqueue(rx_comp_wq); no_mem_rx_wq: kfree(orphan_count); no_mem_orphan_count: @@ -2427,7 +2447,8 @@ static void __exit sdp_exit(void) printk(KERN_WARNING "%s: orphan_count %lld\n", __func__, percpu_counter_read_positive(orphan_count)); - destroy_workqueue(comp_wq); + destroy_workqueue(rx_comp_wq); + destroy_workqueue(sdp_wq); flush_scheduled_work(); diff --git a/drivers/infiniband/ulp/sdp/sdp_proc.c b/drivers/infiniband/ulp/sdp/sdp_proc.c index e759864..0971a49 100644 --- a/drivers/infiniband/ulp/sdp/sdp_proc.c +++ b/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -260,9 +260,11 @@ static int sdpstats_seq_show(struct seq_file *seq, void *v) seq_printf(seq, "rx_poll_miss \t\t: %d\n", sdpstats.rx_poll_miss); seq_printf(seq, "tx_poll_miss \t\t: %d\n", sdpstats.tx_poll_miss); + seq_printf(seq, "tx_poll_hit \t\t: %d\n", sdpstats.tx_poll_hit); seq_printf(seq, "CQ stats:\n"); - seq_printf(seq, "- interrupts\t\t: %d\n", sdpstats.int_count); + seq_printf(seq, "- RX interrupts\t\t: %d\n", sdpstats.rx_int_count); + seq_printf(seq, "- TX interrupts\t\t: %d\n", sdpstats.tx_int_count); return 0; } -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:41 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:41 +0300 Subject: [ofa-general] [PATCH] sdp: add /proc/net/sdpstats + packets dump In-Reply-To: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> * statistics infrastructure for sdp * RX/TX packets dumps Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/Makefile | 2 +- drivers/infiniband/ulp/sdp/sdp.h | 108 +++++++++- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 80 +++++++- drivers/infiniband/ulp/sdp/sdp_cma.c | 36 +--- drivers/infiniband/ulp/sdp/sdp_main.c | 217 ++----------------- drivers/infiniband/ulp/sdp/sdp_proc.c | 358 ++++++++++++++++++++++++++++++++ 6 files changed, 569 insertions(+), 232 deletions(-) create mode 100644 drivers/infiniband/ulp/sdp/sdp_proc.c diff --git a/drivers/infiniband/ulp/sdp/Makefile b/drivers/infiniband/ulp/sdp/Makefile index c889cce..5da4b7b 100644 --- a/drivers/infiniband/ulp/sdp/Makefile +++ b/drivers/infiniband/ulp/sdp/Makefile @@ -3,4 +3,4 @@ EXTRA_CFLAGS += -ggdb obj-$(CONFIG_INFINIBAND_SDP) += ib_sdp.o -ib_sdp-objs := sdp_main.o sdp_cma.o sdp_bcopy.o +ib_sdp-objs := sdp_main.o sdp_cma.o sdp_bcopy.o sdp_proc.o diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index f9c2421..e36f786 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -7,9 +7,12 @@ #include /* For urgent data flags */ #include +#define SDPSTATS_ON + #define sdp_printk(level, sk, format, arg...) \ - printk(level "%s:%d sdp_sock(%d:%d): " format, \ + printk(level "%s:%d sdp_sock(%d %d:%d): " format, \ __func__, __LINE__, \ + current->pid, \ (sk) ? inet_sk(sk)->num : -1, \ (sk) ? ntohs(inet_sk(sk)->dport) : -1, ## arg) #define sdp_warn(sk, format, arg...) \ @@ -50,15 +53,72 @@ extern int sdp_debug_level; #endif /* CONFIG_INFINIBAND_SDP_DEBUG */ #ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA + extern int sdp_data_debug_level; #define sdp_dbg_data(sk, format, arg...) \ do { \ - if (sdp_data_debug_level > 0) \ + if (sdp_data_debug_level & 0x2) \ sdp_printk(KERN_DEBUG, sk, format , ## arg); \ } while (0) +#define SDP_DUMP_PACKET(sk, str, skb, h) \ + do { \ + if (sdp_data_debug_level & 0x1) \ + dump_packet(sk, str, skb, h); \ + } while (0) #else -#define sdp_dbg_data(priv, format, arg...) \ - do { (void) (priv); } while (0) +#define sdp_dbg_data(priv, format, arg...) +// do { (void) (priv); } while (0) +#define SDP_DUMP_PACKET(sk, str, skb, h) +#endif + +#ifdef SDPSTATS_ON + +struct sdpstats { + u32 post_send[256]; + u32 sendmsg_bcopy_segment; + u32 sendmsg_bzcopy_segment; + u32 sendmsg; + u32 post_send_credits; + u32 sendmsg_nagle_skip; + u32 sendmsg_seglen[25]; + u32 send_size[25]; + u32 post_recv; + u32 rx_int_count; + u32 tx_int_count; + u32 bzcopy_poll_miss; + u32 send_wait_for_mem; + u32 send_miss_no_credits; + u32 rx_poll_miss; + u32 tx_poll_miss; + u32 memcpy_count; + u32 credits_before_update[64]; + u32 send_interval[25]; +}; +extern struct sdpstats sdpstats; + +static inline void sdpstats_hist(u32 *h, u32 val, u32 maxidx, int is_log) +{ + int idx = is_log ? ilog2(val) : val; + if (idx > maxidx) + idx = maxidx; + + h[idx]++; +} + +#define SDPSTATS_COUNTER_INC(stat) do {sdpstats.stat++;} while (0) +#define SDPSTATS_COUNTER_ADD(stat, val) do {sdpstats.stat+=val;} while (0) +#define SDPSTATS_COUNTER_MID_INC(stat, mid) do {sdpstats.stat[mid]++;} while (0) +#define SDPSTATS_HIST(stat, size) \ + sdpstats_hist(sdpstats.stat, size, ARRAY_SIZE(sdpstats.stat) - 1, 1) + +#define SDPSTATS_HIST_LINEAR(stat, size) \ + sdpstats_hist(sdpstats.stat, size, ARRAY_SIZE(sdpstats.stat) - 1, 0) + +#else +#define SDPSTATS_COUNTER_INC(stat) +#define SDPSTATS_COUNTER_ADD(stat, val) +#define SDPSTATS_COUNTER_MID_INC(stat, mid) +#define SDPSTATS_HIST(stat, size) #endif #define SOCK_REF_RESET "RESET" @@ -92,6 +152,9 @@ extern int sdp_data_debug_level; #define SDP_OP_RECV 0x800000000LL #define SDP_OP_SEND 0x400000000LL +extern struct list_head sock_list; +extern spinlock_t sock_list_lock; + enum sdp_mid { SDP_MID_HELLO = 0x0, SDP_MID_HELLO_ACK = 0x1, @@ -129,6 +192,38 @@ struct sdp_bsdh { __u32 mseq_ack; }; +union cma_ip_addr { + struct in6_addr ip6; + struct { + __u32 pad[3]; + __u32 addr; + } ip4; +}; + +/* TODO: too much? Can I avoid having the src/dst and port here? */ +struct sdp_hh { + struct sdp_bsdh bsdh; + u8 majv_minv; + u8 ipv_cap; + u8 rsvd1; + u8 max_adverts; + __u32 desremrcvsz; + __u32 localrcvsz; + __u16 port; + __u16 rsvd2; + union cma_ip_addr src_addr; + union cma_ip_addr dst_addr; +}; + +struct sdp_hah { + struct sdp_bsdh bsdh; + u8 majv_minv; + u8 ipv_cap; + u8 rsvd1; + u8 ext_max_adverts; + __u32 actrcvsz; +}; + struct sdp_buf { struct sk_buff *skb; u64 mapping[SDP_MAX_SEND_SKB_FRAGS + 1]; @@ -329,6 +424,9 @@ static inline void sdp_set_error(struct sock *sk, int err) extern struct workqueue_struct *sdp_workqueue; +#ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA +void dump_packet(struct sock *sk, char *str, struct sk_buff *skb, const struct sdp_bsdh *h); +#endif int sdp_cma_handler(struct rdma_cm_id *, struct rdma_cm_event *); void sdp_reset(struct sock *sk); void sdp_reset_sk(struct sock *sk, int rc); @@ -354,6 +452,8 @@ void sdp_post_keepalive(struct sdp_sock *ssk); void sdp_start_keepalive_timer(struct sock *sk); void sdp_bzcopy_write_space(struct sdp_sock *ssk); int sdp_init_sock(struct sock *sk); +int __init sdp_proc_init(void); +void sdp_proc_unregister(void); static inline struct sk_buff *sdp_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) { diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 475d7c3..7c7271c 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -186,6 +186,64 @@ void sdp_post_keepalive(struct sdp_sock *ssk) sdp_cnt(sdp_keepalive_probes_sent); } +#ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA +void dump_packet(struct sock *sk, char *str, struct sk_buff *skb, const struct sdp_bsdh *h) +{ + int len = 0; + char buf[256]; +#define ENUM2STR(e) [e] = #e + static char *mid2str[] = { + ENUM2STR(SDP_MID_HELLO), + ENUM2STR(SDP_MID_HELLO_ACK), + ENUM2STR(SDP_MID_DISCONN), + ENUM2STR(SDP_MID_CHRCVBUF), + ENUM2STR(SDP_MID_CHRCVBUF_ACK), + ENUM2STR(SDP_MID_DATA), + }; + len += snprintf(buf, 255-len, "skb: %p mid: %2x:%-20s flags: 0x%x bufs: %d " + "len: %d mseq: %d mseq_ack: %d", + skb, h->mid, mid2str[h->mid], h->flags, + ntohs(h->bufs), ntohl(h->len),ntohl(h->mseq), + ntohl(h->mseq_ack)); + + switch (h->mid) { + case SDP_MID_HELLO: + { + const struct sdp_hh *hh = (struct sdp_hh *)h; + len += snprintf(buf + len, 255-len, + " | max_adverts: %d majv_minv: %d localrcvsz: %d " + "desremrcvsz: %d |", + hh->max_adverts, + hh->majv_minv, + ntohl(hh->localrcvsz), + ntohl(hh->desremrcvsz)); + } + break; + case SDP_MID_HELLO_ACK: + { + const struct sdp_hah *hah = (struct sdp_hah *)h; + len += snprintf(buf + len, 255-len, " | actrcvz: %d |", + ntohl(hah->actrcvsz)); + } + break; + case SDP_MID_CHRCVBUF: + case SDP_MID_CHRCVBUF_ACK: + { + struct sdp_chrecvbuf *req_size = (struct sdp_chrecvbuf *)(h+1); + len += snprintf(buf + len, 255-len, + " | req_size: %d |", ntohl(req_size->size)); + } + break; + case SDP_MID_DATA: + len += snprintf(buf + len, 255-len, " | data_len: %ld |", ntohl(h->len) - sizeof(struct sdp_bsdh)); + default: + break; + } + buf[len] = 0; + sdp_warn(sk, "%s: %s\n", str, buf); +} +#endif + void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) { struct sdp_buf *tx_req; @@ -197,6 +255,9 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) struct ib_sge *sge; struct ib_send_wr *bad_wr; + SDPSTATS_COUNTER_MID_INC(post_send, mid); + SDPSTATS_HIST(send_size, skb->len); + h->mid = mid; if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) h->flags = SDP_OOB_PRES | SDP_OOB_PEND; @@ -208,6 +269,7 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) h->mseq = htonl(mseq); h->mseq_ack = htonl(ssk->mseq_ack); + SDP_DUMP_PACKET(&ssk->isk.sk, "TX", skb, h); tx_req = &ssk->tx_ring[mseq & (SDP_TX_SIZE - 1)]; tx_req->skb = skb; dev = ssk->ib_device; @@ -244,6 +306,16 @@ void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) ssk->tx_wr.send_flags = IB_SEND_SIGNALED; if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) ssk->tx_wr.send_flags |= IB_SEND_SOLICITED; + + { + static unsigned long last_send = 0; + int delta = jiffies - last_send; + + if (likely(last_send)) + SDPSTATS_HIST(send_interval, delta); + + last_send = jiffies; + } rc = ib_post_send(ssk->qp, &ssk->tx_wr, &bad_wr); ++ssk->tx_head; --ssk->bufs; @@ -292,7 +364,6 @@ struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq) return skb; } - static void sdp_post_recv(struct sdp_sock *ssk) { struct sdp_buf *rx_req; @@ -375,6 +446,7 @@ static void sdp_post_recv(struct sdp_sock *ssk) ssk->rx_wr.sg_list = ssk->ibsge; ssk->rx_wr.num_sge = frags + 1; rc = ib_post_recv(ssk->qp, &ssk->rx_wr, &bad_wr); + SDPSTATS_COUNTER_INC(post_recv); ++ssk->rx_head; if (unlikely(rc)) { sdp_dbg(&ssk->isk.sk, "ib_post_recv failed with status %d\n", rc); @@ -583,6 +655,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) GFP_KERNEL); /* FIXME */ BUG_ON(!skb); + SDPSTATS_COUNTER_INC(post_send_credits); sdp_post_send(ssk, skb, SDP_MID_DATA); } @@ -701,11 +774,14 @@ static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) skb->tail = skb->head + skb_headlen(skb); #endif h = (struct sdp_bsdh *)skb->data; + SDP_DUMP_PACKET(&ssk->isk.sk, "RX", skb, h); skb_reset_transport_header(skb); ssk->mseq_ack = ntohl(h->mseq); if (ssk->mseq_ack != (int)wc->wr_id) printk(KERN_WARNING "SDP BUG! mseq %d != wrid %d\n", ssk->mseq_ack, (int)wc->wr_id); + + SDPSTATS_HIST_LINEAR(credits_before_update, ssk->bufs); ssk->bufs = ntohl(h->mseq_ack) - ssk->tx_head + 1 + ntohs(h->bufs); @@ -782,7 +858,7 @@ static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) if (unlikely(wc->status)) { if (wc->status != IB_WC_WR_FLUSH_ERR) { struct sock *sk = &ssk->isk.sk; - sdp_dbg(sk, "Send completion with error. " + sdp_warn(sk, "Send completion with error. " "Status %d\n", wc->status); sdp_set_error(sk, -ECONNRESET); wake_up(&ssk->wq); diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index 96d65bd..b6bc4e5 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -46,40 +46,8 @@ #include "sdp_socket.h" #include "sdp.h" -union cma_ip_addr { - struct in6_addr ip6; - struct { - __u32 pad[3]; - __u32 addr; - } ip4; -}; - #define SDP_MAJV_MINV 0x22 -/* TODO: too much? Can I avoid having the src/dst and port here? */ -struct sdp_hh { - struct sdp_bsdh bsdh; - u8 majv_minv; - u8 ipv_cap; - u8 rsvd1; - u8 max_adverts; - __u32 desremrcvsz; - __u32 localrcvsz; - __u16 port; - __u16 rsvd2; - union cma_ip_addr src_addr; - union cma_ip_addr dst_addr; -}; - -struct sdp_hah { - struct sdp_bsdh bsdh; - u8 majv_minv; - u8 ipv_cap; - u8 rsvd1; - u8 ext_max_adverts; - __u32 actrcvsz; -}; - enum { SDP_HH_SIZE = 76, SDP_HAH_SIZE = 180, @@ -206,6 +174,7 @@ static int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, sdp_dbg(sk, "%s %p -> %p\n", __func__, sdp_sk(sk)->id, id); h = event->param.conn.private_data; + SDP_DUMP_PACKET(sk, "RX", NULL, &h->bsdh); if (!h->max_adverts) return -EINVAL; @@ -277,6 +246,7 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, return 0; h = event->param.conn.private_data; + SDP_DUMP_PACKET(sk, "RX", NULL, &h->bsdh); sdp_sk(sk)->max_bufs = sdp_sk(sk)->bufs = ntohs(h->bsdh.bufs); sdp_sk(sk)->min_bufs = sdp_sk(sk)->bufs / 4; sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - @@ -428,6 +398,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) conn_param.responder_resources = 4 /* TODO */; conn_param.initiator_depth = 4 /* TODO */; conn_param.retry_count = SDP_RETRY_COUNT; + SDP_DUMP_PACKET(NULL, "TX", NULL, &hh.bsdh); rc = rdma_connect(id, &conn_param); break; case RDMA_CM_EVENT_ROUTE_ERROR: @@ -458,6 +429,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) conn_param.responder_resources = 4 /* TODO */; conn_param.initiator_depth = 4 /* TODO */; conn_param.retry_count = SDP_RETRY_COUNT; + SDP_DUMP_PACKET(sk, "TX", NULL, &hah.bsdh); rc = rdma_accept(id, &conn_param); if (rc) { sdp_sk(child)->id = NULL; diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index 7a38c47..c9b5ba2 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -67,7 +67,6 @@ unsigned int csum_partial_copy_from_user_new (const char *src, char *dst, #include #include #include -#include #include #include /* TODO: remove when sdp_socket.h becomes part of include/linux/socket.h */ @@ -138,8 +137,8 @@ MODULE_PARM_DESC(sdp_zcopy_thresh, "Zero copy send threshold; 0=0ff."); struct workqueue_struct *sdp_workqueue; -static struct list_head sock_list; -static spinlock_t sock_list_lock; +struct list_head sock_list; +spinlock_t sock_list_lock; static DEFINE_RWLOCK(device_removal_lock); @@ -1288,6 +1287,8 @@ static inline struct bzcopy_state *sdp_bz_cleanup(struct bzcopy_state *bz) if (!bz->busy) break; + + SDPSTATS_COUNTER_INC(bzcopy_poll_miss); } if (bz->busy) @@ -1319,11 +1320,14 @@ static struct bzcopy_state *sdp_bz_setup(struct sdp_sock *ssk, int thresh; mm_segment_t cur_fs; - cur_fs = get_fs(); - thresh = ssk->zcopy_thresh ? : sdp_zcopy_thresh; - if (thresh == 0 || len < thresh || !capable(CAP_IPC_LOCK)) + if (thresh == 0 || len < thresh || !capable(CAP_IPC_LOCK)) { + SDPSTATS_COUNTER_INC(sendmsg_bcopy_segment); return NULL; + } + SDPSTATS_COUNTER_INC(sendmsg_bzcopy_segment); + + cur_fs = get_fs(); /* * Since we use the TCP segmentation fields of the skb to map user @@ -1445,6 +1449,8 @@ static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, /* Time to copy data. We are close to * the end! */ + SDPSTATS_COUNTER_ADD(memcpy_count, copy); + sdp_dbg_data(sk, "Copying from: %p offset: %d size: %d\n", from, off, copy); err = skb_copy_to_page(sk, from, skb, page, off, copy); if (err) { @@ -1463,6 +1469,8 @@ static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, skb_shinfo(skb)->frags[i - 1].size += copy; } else { + sdp_dbg_data(sk, "Adding to frage %d: page: %p off: %d size: %d\n", + i, page, off, copy); skb_fill_page_desc(skb, i, page, off, copy); if (TCP_PAGE(sk)) { get_page(page); @@ -1651,7 +1659,7 @@ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, int err, copied; long timeo; struct bzcopy_state *bz = NULL; - + SDPSTATS_COUNTER_INC(sendmsg); lock_sock(sk); sdp_dbg_data(sk, "%s\n", __func__); @@ -1688,6 +1696,8 @@ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, if (size_goal > SDP_MAX_PAYLOAD) size_goal = SDP_MAX_PAYLOAD; + SDPSTATS_HIST(sendmsg_seglen, seglen); + if (bz) sdp_bz_cleanup(bz); bz = sdp_bz_setup(ssk, from, seglen, size_goal); @@ -1737,8 +1747,14 @@ new_segment: NETIF_F_HW_CSUM)) skb->ip_summed = CHECKSUM_PARTIAL; + sdp_dbg_data(sk, "entail new skb: %p len = %d\n", skb, skb->len); skb_entail(sk, ssk, skb); copy = size_goal; + } else { + sdp_dbg_data(sk, "adding to existing skb: %p" + " len = %d, sk_send_head: %p copy: %d\n", + skb, skb->len, sk->sk_send_head, copy); + } /* Try to append data to the end of skb. */ @@ -1791,6 +1807,7 @@ new_segment: wait_for_sndbuf: set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); wait_for_memory: + SDPSTATS_COUNTER_INC(send_wait_for_mem); if (copied) sdp_push(sk, ssk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); @@ -2266,192 +2283,6 @@ static int sdp_create_socket(struct net *net, struct socket *sock, int protocol) return 0; } -#ifdef CONFIG_PROC_FS - -static void *sdp_get_idx(struct seq_file *seq, loff_t pos) -{ - int i = 0; - struct sdp_sock *ssk; - - if (!list_empty(&sock_list)) - list_for_each_entry(ssk, &sock_list, sock_list) { - if (i == pos) - return ssk; - i++; - } - - return NULL; -} - -static void *sdp_seq_start(struct seq_file *seq, loff_t *pos) -{ - void *start = NULL; - struct sdp_iter_state* st = seq->private; - - st->num = 0; - - if (!*pos) - return SEQ_START_TOKEN; - - spin_lock_irq(&sock_list_lock); - start = sdp_get_idx(seq, *pos - 1); - if (start) - sock_hold((struct sock *)start, SOCK_REF_SEQ); - spin_unlock_irq(&sock_list_lock); - - return start; -} - -static void *sdp_seq_next(struct seq_file *seq, void *v, loff_t *pos) -{ - struct sdp_iter_state* st = seq->private; - void *next = NULL; - - spin_lock_irq(&sock_list_lock); - if (v == SEQ_START_TOKEN) - next = sdp_get_idx(seq, 0); - else - next = sdp_get_idx(seq, *pos); - if (next) - sock_hold((struct sock *)next, SOCK_REF_SEQ); - spin_unlock_irq(&sock_list_lock); - - *pos += 1; - st->num++; - - return next; -} - -static void sdp_seq_stop(struct seq_file *seq, void *v) -{ -} - -#define TMPSZ 150 - -static int sdp_seq_show(struct seq_file *seq, void *v) -{ - struct sdp_iter_state* st; - struct sock *sk = v; - char tmpbuf[TMPSZ + 1]; - unsigned int dest; - unsigned int src; - int uid; - unsigned long inode; - __u16 destp; - __u16 srcp; - __u32 rx_queue, tx_queue; - - if (v == SEQ_START_TOKEN) { - seq_printf(seq, "%-*s\n", TMPSZ - 1, - " sl local_address rem_address uid inode" - " rx_queue tx_queue state"); - goto out; - } - - st = seq->private; - - dest = inet_sk(sk)->daddr; - src = inet_sk(sk)->rcv_saddr; - destp = ntohs(inet_sk(sk)->dport); - srcp = ntohs(inet_sk(sk)->sport); - uid = sock_i_uid(sk); - inode = sock_i_ino(sk); - rx_queue = sdp_sk(sk)->rcv_nxt - sdp_sk(sk)->copied_seq; - tx_queue = sdp_sk(sk)->write_seq - sdp_sk(sk)->snd_una; - - sprintf(tmpbuf, "%4d: %08X:%04X %08X:%04X %5d %lu %08X:%08X %X", - st->num, src, srcp, dest, destp, uid, inode, - rx_queue, tx_queue, sk->sk_state); - - seq_printf(seq, "%-*s\n", TMPSZ - 1, tmpbuf); - - sock_put(sk, SOCK_REF_SEQ); -out: - return 0; -} - -static int sdp_seq_open(struct inode *inode, struct file *file) -{ - struct sdp_seq_afinfo *afinfo = PDE(inode)->data; - struct seq_file *seq; - struct sdp_iter_state *s; - int rc; - - if (unlikely(afinfo == NULL)) - return -EINVAL; - - s = kzalloc(sizeof(*s), GFP_KERNEL); - if (!s) - return -ENOMEM; - s->family = afinfo->family; - s->seq_ops.start = sdp_seq_start; - s->seq_ops.next = sdp_seq_next; - s->seq_ops.show = afinfo->seq_show; - s->seq_ops.stop = sdp_seq_stop; - - rc = seq_open(file, &s->seq_ops); - if (rc) - goto out_kfree; - seq = file->private_data; - seq->private = s; -out: - return rc; -out_kfree: - kfree(s); - goto out; -} - - -static struct file_operations sdp_seq_fops; -static struct sdp_seq_afinfo sdp_seq_afinfo = { - .owner = THIS_MODULE, - .name = "sdp", - .family = AF_INET_SDP, - .seq_show = sdp_seq_show, - .seq_fops = &sdp_seq_fops, -}; - - -static int __init sdp_proc_init(void) -{ - int rc = 0; - struct proc_dir_entry *p; - - sdp_seq_afinfo.seq_fops->owner = sdp_seq_afinfo.owner; - sdp_seq_afinfo.seq_fops->open = sdp_seq_open; - sdp_seq_afinfo.seq_fops->read = seq_read; - sdp_seq_afinfo.seq_fops->llseek = seq_lseek; - sdp_seq_afinfo.seq_fops->release = seq_release_private; - - p = proc_net_fops_create(&init_net, sdp_seq_afinfo.name, S_IRUGO, - sdp_seq_afinfo.seq_fops); - if (p) - p->data = &sdp_seq_afinfo; - else - rc = -ENOMEM; - - return rc; -} - -static void sdp_proc_unregister(void) -{ - proc_net_remove(&init_net, sdp_seq_afinfo.name); - memset(sdp_seq_afinfo.seq_fops, 0, sizeof(*sdp_seq_afinfo.seq_fops)); -} - -#else /* CONFIG_PROC_FS */ - -static int __init sdp_proc_init(void) -{ - return 0; -} - -static void sdp_proc_unregister(void) -{ - -} -#endif /* CONFIG_PROC_FS */ - static void sdp_add_device(struct ib_device *device) { } diff --git a/drivers/infiniband/ulp/sdp/sdp_proc.c b/drivers/infiniband/ulp/sdp/sdp_proc.c new file mode 100644 index 0000000..8971517 --- /dev/null +++ b/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -0,0 +1,358 @@ +/* + * Copyright (c) 2008 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include "sdp_socket.h" +#include "sdp.h" + +#ifdef CONFIG_PROC_FS + +#define PROC_SDP_STATS "sdpstats" + +static void *sdp_get_idx(struct seq_file *seq, loff_t pos) +{ + int i = 0; + struct sdp_sock *ssk; + + if (!list_empty(&sock_list)) + list_for_each_entry(ssk, &sock_list, sock_list) { + if (i == pos) + return ssk; + i++; + } + + return NULL; +} + +static void *sdp_seq_start(struct seq_file *seq, loff_t *pos) +{ + void *start = NULL; + struct sdp_iter_state* st = seq->private; + + st->num = 0; + + if (!*pos) + return SEQ_START_TOKEN; + + spin_lock_irq(&sock_list_lock); + start = sdp_get_idx(seq, *pos - 1); + if (start) + sock_hold((struct sock *)start, SOCK_REF_SEQ); + spin_unlock_irq(&sock_list_lock); + + return start; +} + +static void *sdp_seq_next(struct seq_file *seq, void *v, loff_t *pos) +{ + struct sdp_iter_state* st = seq->private; + void *next = NULL; + + spin_lock_irq(&sock_list_lock); + if (v == SEQ_START_TOKEN) + next = sdp_get_idx(seq, 0); + else + next = sdp_get_idx(seq, *pos); + if (next) + sock_hold((struct sock *)next, SOCK_REF_SEQ); + spin_unlock_irq(&sock_list_lock); + + *pos += 1; + st->num++; + + return next; +} + +static void sdp_seq_stop(struct seq_file *seq, void *v) +{ +} + +#define TMPSZ 150 + +static int sdp_seq_show(struct seq_file *seq, void *v) +{ + struct sdp_iter_state* st; + struct sock *sk = v; + char tmpbuf[TMPSZ + 1]; + unsigned int dest; + unsigned int src; + int uid; + unsigned long inode; + __u16 destp; + __u16 srcp; + __u32 rx_queue, tx_queue; + + if (v == SEQ_START_TOKEN) { + seq_printf(seq, "%-*s\n", TMPSZ - 1, + " sl local_address rem_address uid inode" + " rx_queue tx_queue state"); + goto out; + } + + st = seq->private; + + dest = inet_sk(sk)->daddr; + src = inet_sk(sk)->rcv_saddr; + destp = ntohs(inet_sk(sk)->dport); + srcp = ntohs(inet_sk(sk)->sport); + uid = sock_i_uid(sk); + inode = sock_i_ino(sk); + rx_queue = sdp_sk(sk)->rcv_nxt - sdp_sk(sk)->copied_seq; + tx_queue = sdp_sk(sk)->write_seq - sdp_sk(sk)->snd_una; + + sprintf(tmpbuf, "%4d: %08X:%04X %08X:%04X %5d %lu %08X:%08X %X", + st->num, src, srcp, dest, destp, uid, inode, + rx_queue, tx_queue, sk->sk_state); + + seq_printf(seq, "%-*s\n", TMPSZ - 1, tmpbuf); + + sock_put(sk, SOCK_REF_SEQ); +out: + return 0; +} + +static int sdp_seq_open(struct inode *inode, struct file *file) +{ + struct sdp_seq_afinfo *afinfo = PDE(inode)->data; + struct seq_file *seq; + struct sdp_iter_state *s; + int rc; + + if (unlikely(afinfo == NULL)) + return -EINVAL; + + s = kzalloc(sizeof(*s), GFP_KERNEL); + if (!s) + return -ENOMEM; + s->family = afinfo->family; + s->seq_ops.start = sdp_seq_start; + s->seq_ops.next = sdp_seq_next; + s->seq_ops.show = afinfo->seq_show; + s->seq_ops.stop = sdp_seq_stop; + + rc = seq_open(file, &s->seq_ops); + if (rc) + goto out_kfree; + seq = file->private_data; + seq->private = s; +out: + return rc; +out_kfree: + kfree(s); + goto out; +} + + +static struct file_operations sdp_seq_fops; +static struct sdp_seq_afinfo sdp_seq_afinfo = { + .owner = THIS_MODULE, + .name = "sdp", + .family = AF_INET_SDP, + .seq_show = sdp_seq_show, + .seq_fops = &sdp_seq_fops, +}; + +#ifdef SDPSTATS_ON +struct sdpstats sdpstats = { { 0 } }; + +static void sdpstats_seq_hist(struct seq_file *seq, char *str, u32 *h, int n, int is_log) +{ + int i; + u32 max = 0; + + seq_printf(seq, "%s:\n", str); + + for (i = 0; i < n; i++) { + if (h[i] > max) + max = h[i]; + } + + if (max == 0) { + seq_printf(seq, " - all values are 0\n"); + return; + } + + for (i = 0; i < n; i++) { + char s[51]; + int j = 50 * h[i] / max; + int val = is_log ? (i == n-1 ? 0 : 1<owner = sdp_seq_afinfo.owner; + sdp_seq_afinfo.seq_fops->open = sdp_seq_open; + sdp_seq_afinfo.seq_fops->read = seq_read; + sdp_seq_afinfo.seq_fops->llseek = seq_lseek; + sdp_seq_afinfo.seq_fops->release = seq_release_private; + + p = proc_net_fops_create(sdp_seq_afinfo.name, S_IRUGO, + sdp_seq_afinfo.seq_fops); + if (p) + p->data = &sdp_seq_afinfo; + else + goto no_mem;; + +#ifdef SDPSTATS_ON + + sdpstats = proc_net_fops_create(PROC_SDP_STATS, + S_IRUGO | S_IWUGO, &sdpstats_fops); + if (!sdpstats) + goto no_mem;; + + return 0; +#endif + +no_mem: + if (sdpstats) + proc_net_remove(PROC_SDP_STATS); + + if (p) + proc_net_remove(sdp_seq_afinfo.name); + + return -ENOMEM; +} + +void sdp_proc_unregister(void) +{ + proc_net_remove(sdp_seq_afinfo.name); + memset(sdp_seq_afinfo.seq_fops, 0, sizeof(*sdp_seq_afinfo.seq_fops)); + +#ifdef SDPSTATS_ON + proc_net_remove(PROC_SDP_STATS); +#endif +} + +#else /* CONFIG_PROC_FS */ + +int __init sdp_proc_init(void) +{ + return 0; +} + +void sdp_proc_unregister(void) +{ + +} +#endif /* CONFIG_PROC_FS */ -- 1.5.3.7 From amirv at mellanox.co.il Mon Jun 22 01:18:49 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Mon, 22 Jun 2009 11:18:49 +0300 Subject: [ofa-general] [PATCH] sdp: all previous patches in one patch In-Reply-To: <1245658729-10642-9-git-send-email-amirv@mellanox.co.il> References: <1245658729-10642-1-git-send-email-amirv@mellanox.co.il> <1245658729-10642-2-git-send-email-amirv@mellanox.co.il> <1245658729-10642-3-git-send-email-amirv@mellanox.co.il> <1245658729-10642-4-git-send-email-amirv@mellanox.co.il> <1245658729-10642-5-git-send-email-amirv@mellanox.co.il> <1245658729-10642-6-git-send-email-amirv@mellanox.co.il> <1245658729-10642-7-git-send-email-amirv@mellanox.co.il> <1245658729-10642-8-git-send-email-amirv@mellanox.co.il> <1245658729-10642-9-git-send-email-amirv@mellanox.co.il> Message-ID: <1245658729-10642-10-git-send-email-amirv@mellanox.co.il> changes: * fixed coding style * make interrupt moderation adaptive * arm nagle timer on not sent packet instead of on sent packet * fix bad HELLO/HELLO_ACK buffer size matching + let recvmsg do posts on data packets too * make bzcopy poll timeout in jiffies instead of iterations count * fix bad handling for not aligned buffers in bzcopy + removed poll at end of send * TX from 1 context only. RX with minimal context switches * don't arm nagle timer for every sent packet * Do not nagle BZCopy packets * don't do nagle on first packet * process RX CQ from interrupt * move rx/tx stuff to dedicated files: sdp_rx and sdp_tx * /proc/net/sdpprf - performance utilities * no tx interrupts * move tx_ring into dedicated structre + many cosmetic fixes * Interrupts performance fixes * added /proc/net/sdpstats + packets dump Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/Makefile | 2 +- drivers/infiniband/ulp/sdp/sdp.h | 455 ++++++++++++++--- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 893 +++++-------------------------- drivers/infiniband/ulp/sdp/sdp_cma.c | 191 +++----- drivers/infiniband/ulp/sdp/sdp_main.c | 897 +++++++++++++++----------------- drivers/infiniband/ulp/sdp/sdp_proc.c | 496 ++++++++++++++++++ drivers/infiniband/ulp/sdp/sdp_rx.c | 850 ++++++++++++++++++++++++++++++ drivers/infiniband/ulp/sdp/sdp_tx.c | 442 ++++++++++++++++ 8 files changed, 2800 insertions(+), 1426 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/Makefile b/drivers/infiniband/ulp/sdp/Makefile index c889cce..b14a16a 100644 --- a/drivers/infiniband/ulp/sdp/Makefile +++ b/drivers/infiniband/ulp/sdp/Makefile @@ -3,4 +3,4 @@ EXTRA_CFLAGS += -ggdb obj-$(CONFIG_INFINIBAND_SDP) += ib_sdp.o -ib_sdp-objs := sdp_main.o sdp_cma.o sdp_bcopy.o +ib_sdp-objs := sdp_main.o sdp_cma.o sdp_bcopy.o sdp_proc.o sdp_tx.o sdp_rx.o diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index f9c2421..53f4b95 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -6,15 +6,91 @@ #include #include /* For urgent data flags */ #include +#include -#define sdp_printk(level, sk, format, arg...) \ - printk(level "%s:%d sdp_sock(%d:%d): " format, \ - __func__, __LINE__, \ +#define SDPSTATS_ON +/* #define SDP_PROFILING */ + +#define _sdp_printk(func, line, level, sk, format, arg...) \ + printk(level "%s:%d sdp_sock(%5d:%d %d:%d): " format, \ + func, line, \ + current->pid, smp_processor_id(), \ (sk) ? inet_sk(sk)->num : -1, \ (sk) ? ntohs(inet_sk(sk)->dport) : -1, ## arg) +#define sdp_printk(level, sk, format, arg...) \ + _sdp_printk(__func__, __LINE__, level, sk, format, ## arg) #define sdp_warn(sk, format, arg...) \ sdp_printk(KERN_WARNING, sk, format , ## arg) +#define rx_ring_lock(ssk, f) do { \ + spin_lock_irqsave(&ssk->rx_ring.lock, f); \ +} while (0) + +#define rx_ring_unlock(ssk, f) do { \ + spin_unlock_irqrestore(&ssk->rx_ring.lock, f); \ +} while (0) + +#define SDP_MODPARAM_SINT(var, def_val, msg) \ + static int var = def_val; \ + module_param_named(var, var, int, 0644); \ + MODULE_PARM_DESC(var, msg " [" #def_val "]"); \ + +#define SDP_MODPARAM_INT(var, def_val, msg) \ + int var = def_val; \ + module_param_named(var, var, int, 0644); \ + MODULE_PARM_DESC(var, msg " [" #def_val "]"); \ + +#ifdef SDP_PROFILING +struct sk_buff; +struct sdpprf_log { + int idx; + int pid; + int cpu; + int sk_num; + int sk_dport; + struct sk_buff *skb; + char msg[256]; + + unsigned long long time; + + const char *func; + int line; +}; + +#define SDPPRF_LOG_SIZE 0x20000 /* must be a power of 2 */ + +extern struct sdpprf_log sdpprf_log[SDPPRF_LOG_SIZE]; +extern int sdpprf_log_count; + +static inline unsigned long long current_nsec(void) +{ + struct timespec tv; + getnstimeofday(&tv); + return tv.tv_sec * NSEC_PER_SEC + tv.tv_nsec; +} +#define sdp_prf1(sk, s, format, arg...) ({ \ + struct sdpprf_log *l = \ + &sdpprf_log[sdpprf_log_count++ & (SDPPRF_LOG_SIZE - 1)]; \ + l->idx = sdpprf_log_count - 1; \ + l->pid = current->pid; \ + l->sk_num = (sk) ? inet_sk(sk)->num : -1; \ + l->sk_dport = (sk) ? ntohs(inet_sk(sk)->dport) : -1; \ + l->cpu = smp_processor_id(); \ + l->skb = s; \ + snprintf(l->msg, sizeof(l->msg) - 1, format, ## arg); \ + l->time = current_nsec(); \ + l->func = __func__; \ + l->line = __LINE__; \ + 1; \ +}) +#define sdp_prf(sk, s, format, arg...) +/* #define sdp_prf(sk, s, format, arg...) sdp_prf1(sk, s, format, ## arg) */ + +#else +#define sdp_prf1(sk, s, format, arg...) +#define sdp_prf(sk, s, format, arg...) +#endif + #ifdef CONFIG_INFINIBAND_SDP_DEBUG extern int sdp_debug_level; @@ -37,9 +113,9 @@ extern int sdp_debug_level; }) #define sk_common_release(sk) do { \ - sdp_dbg(sk, "%s:%d - sock_put(" SOCK_REF_BORN ") - refcount = %d " \ - "from withing sk_common_release\n",\ - __FUNCTION__, __LINE__, atomic_read(&(sk)->sk_refcnt)); \ + sdp_dbg(sk, "%s:%d - sock_put(" SOCK_REF_BORN \ + ") - refcount = %d from withing sk_common_release\n",\ + __func__, __LINE__, atomic_read(&(sk)->sk_refcnt));\ sk_common_release(sk); \ } while (0) @@ -50,15 +126,80 @@ extern int sdp_debug_level; #endif /* CONFIG_INFINIBAND_SDP_DEBUG */ #ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA + extern int sdp_data_debug_level; -#define sdp_dbg_data(sk, format, arg...) \ - do { \ - if (sdp_data_debug_level > 0) \ - sdp_printk(KERN_DEBUG, sk, format , ## arg); \ +#define sdp_dbg_data(sk, format, arg...) \ + do { \ + if (sdp_data_debug_level & 0x2) \ + sdp_printk(KERN_DEBUG, sk, format , ## arg); \ + } while (0) +#define SDP_DUMP_PACKET(sk, str, skb, h) \ + do { \ + if (sdp_data_debug_level & 0x1) \ + dump_packet(sk, str, skb, h); \ } while (0) #else -#define sdp_dbg_data(priv, format, arg...) \ - do { (void) (priv); } while (0) +#define sdp_dbg_data(priv, format, arg...) +#define SDP_DUMP_PACKET(sk, str, skb, h) +#endif + +#ifdef SDPSTATS_ON + +struct sdpstats { + u32 post_send[256]; + u32 sendmsg_bcopy_segment; + u32 sendmsg_bzcopy_segment; + u32 sendmsg; + u32 post_send_credits; + u32 sendmsg_nagle_skip; + u32 sendmsg_seglen[25]; + u32 send_size[25]; + u32 post_recv; + u32 rx_int_count; + u32 tx_int_count; + u32 bzcopy_poll_miss; + u32 send_wait_for_mem; + u32 send_miss_no_credits; + u32 rx_poll_miss; + u32 tx_poll_miss; + u32 tx_poll_hit; + u32 tx_poll_busy; + u32 memcpy_count; + u32 credits_before_update[64]; + u32 send_interval[25]; + + u32 bz_clean_sum; + u32 bz_setup_sum; + u32 tx_copy_sum; + u32 sendmsg_sum; +}; +extern struct sdpstats sdpstats; + +static inline void sdpstats_hist(u32 *h, u32 val, u32 maxidx, int is_log) +{ + int idx = is_log ? ilog2(val) : val; + if (idx > maxidx) + idx = maxidx; + + h[idx]++; +} + +#define SDPSTATS_COUNTER_INC(stat) do { sdpstats.stat++; } while (0) +#define SDPSTATS_COUNTER_ADD(stat, val) do { sdpstats.stat += val; } while (0) +#define SDPSTATS_COUNTER_MID_INC(stat, mid) do { sdpstats.stat[mid]++; } \ + while (0) +#define SDPSTATS_HIST(stat, size) \ + sdpstats_hist(sdpstats.stat, size, ARRAY_SIZE(sdpstats.stat) - 1, 1) + +#define SDPSTATS_HIST_LINEAR(stat, size) \ + sdpstats_hist(sdpstats.stat, size, ARRAY_SIZE(sdpstats.stat) - 1, 0) + +#else +#define SDPSTATS_COUNTER_INC(stat) +#define SDPSTATS_COUNTER_ADD(stat, val) +#define SDPSTATS_COUNTER_MID_INC(stat, mid) +#define SDPSTATS_HIST_LINEAR(stat, size) +#define SDPSTATS_HIST(stat, size) #endif #define SOCK_REF_RESET "RESET" @@ -72,6 +213,12 @@ extern int sdp_data_debug_level; #define sock_put(sk, msg) sock_ref(sk, msg, sock_put) #define __sock_put(sk, msg) sock_ref(sk, msg, __sock_put) +/* Interval between sucessive polls in the Tx routine when polling is used + instead of interrupts (in per-core Tx rings) - should be power of 2 */ +#define SDP_TX_POLL_MODER 16 +#define SDP_TX_POLL_TIMEOUT (HZ / 4) +#define SDP_NAGLE_TIMEOUT (HZ / 10) + #define SDP_RESOLVE_TIMEOUT 1000 #define SDP_ROUTE_TIMEOUT 1000 #define SDP_RETRY_COUNT 5 @@ -81,7 +228,8 @@ extern int sdp_data_debug_level; #define SDP_TX_SIZE 0x40 #define SDP_RX_SIZE 0x40 -#define SDP_MAX_SEND_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE) +#define SDP_MAX_RECV_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE) +#define SDP_MAX_SEND_SKB_FRAGS (SDP_MAX_RECV_SKB_FRAGS + 1) #define SDP_HEAD_SIZE (PAGE_SIZE / 2 + sizeof(struct sdp_bsdh)) #define SDP_NUM_WC 4 #define SDP_MAX_PAYLOAD ((1 << 16) - SDP_HEAD_SIZE) @@ -92,6 +240,21 @@ extern int sdp_data_debug_level; #define SDP_OP_RECV 0x800000000LL #define SDP_OP_SEND 0x400000000LL +/* how long (in jiffies) to block sender till tx completion*/ +#define SDP_BZCOPY_POLL_TIMEOUT (HZ / 10) + +#define SDP_AUTO_CONF 0xffff +#define AUTO_MOD_DELAY (HZ / 4) + +#define BZCOPY_STATE(skb) (*(struct bzcopy_state **)(skb->cb)) +#ifndef MIN +#define MIN(a, b) (a < b ? a : b) +#endif + +extern struct workqueue_struct *sdp_wq; +extern struct list_head sock_list; +extern spinlock_t sock_list_lock; + enum sdp_mid { SDP_MID_HELLO = 0x0, SDP_MID_HELLO_ACK = 0x1, @@ -107,7 +270,7 @@ enum sdp_flags { }; enum { - SDP_MIN_BUFS = 2 + SDP_MIN_TX_CREDITS = 2 }; enum { @@ -129,33 +292,132 @@ struct sdp_bsdh { __u32 mseq_ack; }; +union cma_ip_addr { + struct in6_addr ip6; + struct { + __u32 pad[3]; + __u32 addr; + } ip4; +}; + +/* TODO: too much? Can I avoid having the src/dst and port here? */ +struct sdp_hh { + struct sdp_bsdh bsdh; + u8 majv_minv; + u8 ipv_cap; + u8 rsvd1; + u8 max_adverts; + __u32 desremrcvsz; + __u32 localrcvsz; + __u16 port; + __u16 rsvd2; + union cma_ip_addr src_addr; + union cma_ip_addr dst_addr; +}; + +struct sdp_hah { + struct sdp_bsdh bsdh; + u8 majv_minv; + u8 ipv_cap; + u8 rsvd1; + u8 ext_max_adverts; + __u32 actrcvsz; +}; + struct sdp_buf { struct sk_buff *skb; u64 mapping[SDP_MAX_SEND_SKB_FRAGS + 1]; }; +#define ring_head(ring) (atomic_read(&(ring).head)) +#define ring_tail(ring) (atomic_read(&(ring).tail)) +#define ring_posted(ring) (ring_head(ring) - ring_tail(ring)) + +struct sdp_tx_ring { + struct sdp_buf *buffer; + atomic_t head; + atomic_t tail; + struct ib_cq *cq; + + int una_seq; + atomic_t credits; +#define tx_credits(ssk) (atomic_read(&ssk->tx_ring.credits)) + + struct timer_list timer; + u16 poll_cnt; +}; + +struct sdp_rx_ring { + struct sdp_buf *buffer; + atomic_t head; + atomic_t tail; + struct ib_cq *cq; + + spinlock_t lock; +}; + +static inline int sdp_tx_ring_slots_left(struct sdp_tx_ring *tx_ring) +{ + return SDP_TX_SIZE - ring_posted(*tx_ring); +} + +struct sdp_chrecvbuf { + u32 size; +}; + +#define posts_handler(ssk) atomic_read(&ssk->somebody_is_doing_posts) +#define posts_handler_get(ssk) atomic_inc(&ssk->somebody_is_doing_posts) +#define posts_handler_put(ssk) do {\ + atomic_dec(&ssk->somebody_is_doing_posts); \ + sdp_do_posts(ssk); \ +} while (0) + +struct sdp_moderation { + unsigned long last_moder_packets; + unsigned long last_moder_tx_packets; + unsigned long last_moder_bytes; + unsigned long last_moder_jiffies; + int last_moder_time; + u16 rx_usecs; + u16 rx_frames; + u16 tx_usecs; + u32 pkt_rate_low; + u16 rx_usecs_low; + u32 pkt_rate_high; + u16 rx_usecs_high; + u16 sample_interval; + u16 adaptive_rx_coal; + u32 msg_enable; + + int moder_cnt; + int moder_time; +}; + struct sdp_sock { /* sk has to be the first member of inet_sock */ struct inet_sock isk; struct list_head sock_list; struct list_head accept_queue; struct list_head backlog_queue; + struct sk_buff_head rx_ctl_q; struct sock *parent; - struct work_struct work; + struct work_struct rx_comp_work; wait_queue_head_t wq; struct delayed_work dreq_wait_work; struct work_struct destroy_work; + atomic_t somebody_is_doing_posts; + /* Like tcp_sock */ u16 urg_data; u32 urg_seq; u32 copied_seq; - u32 rcv_nxt; +#define rcv_nxt(ssk) atomic_read(&(ssk->rcv_nxt)) + atomic_t rcv_nxt; int write_seq; - int snd_una; int pushed_seq; int xmit_size_goal; int nonagle; @@ -174,32 +436,29 @@ struct sdp_sock { int sdp_disconnect; int destruct_in_process; - struct sdp_buf *rx_ring; - struct sdp_buf *tx_ring; + struct sdp_rx_ring rx_ring; + struct sdp_tx_ring tx_ring; - /* rdma specific */ - struct ib_qp *qp; - struct ib_cq *cq; - struct ib_mr *mr; /* Data below will be reset on error */ struct rdma_cm_id *id; struct ib_device *ib_device; /* SDP specific */ - struct ib_recv_wr rx_wr; - unsigned rx_head; - unsigned rx_tail; - unsigned mseq_ack; - unsigned bufs; + atomic_t mseq_ack; +#define mseq_ack(ssk) (atomic_read(&ssk->mseq_ack)) unsigned max_bufs; /* Initial buffers offered by other side */ unsigned min_bufs; /* Low water mark to wake senders */ - int remote_credits; + unsigned long nagle_last_unacked; /* mseq of lastest unacked packet */ + struct timer_list nagle_timer; /* timeout waiting for ack */ + + atomic_t remote_credits; +#define remote_credits(ssk) (atomic_read(&ssk->remote_credits)) int poll_cq; - unsigned tx_head; - unsigned tx_tail; - struct ib_send_wr tx_wr; + /* rdma specific */ + struct ib_qp *qp; + struct ib_mr *mr; /* SDP slow start */ int rcvbuf_scale; /* local recv buf scale for each socket */ @@ -213,11 +472,14 @@ struct sdp_sock { int recv_frags; /* max skb frags in recv packets */ int send_frags; /* max skb frags in send packets */ + unsigned long tx_packets; + unsigned long rx_packets; + unsigned long tx_bytes; + unsigned long rx_bytes; + struct sdp_moderation auto_mod; + /* BZCOPY data */ int zcopy_thresh; - - struct ib_sge ibsge[SDP_MAX_SEND_SKB_FRAGS + 1]; - struct ib_wc ibwc[SDP_NUM_WC]; }; /* Context used for synchronous zero copy bcopy (BZCOY) */ @@ -236,26 +498,11 @@ struct bzcopy_state { extern int rcvbuf_initial_size; extern struct proto sdp_proto; -extern struct workqueue_struct *sdp_workqueue; +extern struct workqueue_struct *rx_comp_wq; extern atomic_t sdp_current_mem_usage; extern spinlock_t sdp_large_sockets_lock; -/* just like TCP fs */ -struct sdp_seq_afinfo { - struct module *owner; - char *name; - sa_family_t family; - int (*seq_show) (struct seq_file *m, void *v); - struct file_operations *seq_fops; -}; - -struct sdp_iter_state { - sa_family_t family; - int num; - struct seq_operations seq_ops; -}; - static inline struct sdp_sock *sdp_sk(const struct sock *sk) { return (struct sdp_sock *)sk; @@ -290,9 +537,10 @@ static inline int _sdp_exch_state(const char *func, int line, struct sock *sk, int old; spin_lock_irqsave(&sdp_sk(sk)->lock, flags); - + sdp_dbg(sk, "%s:%d - set state: %s -> %s 0x%x\n", func, line, - sdp_state_str(sk->sk_state), sdp_state_str(state), from_states); + sdp_state_str(sk->sk_state), + sdp_state_str(state), from_states); if ((1 << sk->sk_state) & ~from_states) { sdp_warn(sk, "trying to exchange state from unexpected state " @@ -327,35 +575,93 @@ static inline void sdp_set_error(struct sock *sk, int err) sk->sk_error_report(sk); } -extern struct workqueue_struct *sdp_workqueue; +#ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA +void _dump_packet(const char *func, int line, struct sock *sk, char *str, + struct sk_buff *skb, const struct sdp_bsdh *h); +#define dump_packet(sk, str, skb, h) \ + _dump_packet(__func__, __LINE__, sk, str, skb, h) +#endif -int sdp_cma_handler(struct rdma_cm_id *, struct rdma_cm_event *); -void sdp_reset(struct sock *sk); +/* sdp_main.c */ +void sdp_set_default_moderation(struct sdp_sock *ssk); +int sdp_init_sock(struct sock *sk); +void sdp_start_keepalive_timer(struct sock *sk); +void sdp_remove_sock(struct sdp_sock *ssk); +void sdp_add_sock(struct sdp_sock *ssk); +void sdp_urg(struct sdp_sock *ssk, struct sk_buff *skb); +void sdp_dreq_wait_timeout_work(struct work_struct *work); +void sdp_cancel_dreq_wait_timeout(struct sdp_sock *ssk); +void sdp_destroy_work(struct work_struct *work); void sdp_reset_sk(struct sock *sk, int rc); -void sdp_completion_handler(struct ib_cq *cq, void *cq_context); -void sdp_work(struct work_struct *work); +void sdp_reset(struct sock *sk); + +/* sdp_proc.c */ +int __init sdp_proc_init(void); +void sdp_proc_unregister(void); + +/* sdp_cma.c */ +int sdp_cma_handler(struct rdma_cm_id *, struct rdma_cm_event *); + +/* sdp_bcopy.c */ int sdp_post_credits(struct sdp_sock *ssk); + +/* sdp_tx.c */ +int sdp_tx_ring_create(struct sdp_sock *ssk, struct ib_device *device); +void sdp_tx_ring_destroy(struct sdp_sock *ssk); +int sdp_xmit_poll(struct sdp_sock *ssk, int force); void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid); -void sdp_post_recvs(struct sdp_sock *ssk); -int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq); void sdp_post_sends(struct sdp_sock *ssk, int nonagle); -void sdp_destroy_work(struct work_struct *work); -void sdp_cancel_dreq_wait_timeout(struct sdp_sock *ssk); -void sdp_dreq_wait_timeout_work(struct work_struct *work); -struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id); -struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq); -void sdp_urg(struct sdp_sock *ssk, struct sk_buff *skb); -void sdp_add_sock(struct sdp_sock *ssk); -void sdp_remove_sock(struct sdp_sock *ssk); -void sdp_remove_large_sock(struct sdp_sock *ssk); +void sdp_nagle_timeout(unsigned long data); +void sdp_post_keepalive(struct sdp_sock *ssk); + +/* sdp_rx.c */ +void sdp_rx_ring_init(struct sdp_sock *ssk); +int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device); +void sdp_rx_ring_destroy(struct sdp_sock *ssk); int sdp_resize_buffers(struct sdp_sock *ssk, u32 new_size); int sdp_init_buffers(struct sdp_sock *ssk, u32 new_size); -void sdp_post_keepalive(struct sdp_sock *ssk); -void sdp_start_keepalive_timer(struct sock *sk); -void sdp_bzcopy_write_space(struct sdp_sock *ssk); -int sdp_init_sock(struct sock *sk); +void sdp_do_posts(struct sdp_sock *ssk); +void sdp_rx_comp_full(struct sdp_sock *ssk); +void sdp_remove_large_sock(struct sdp_sock *ssk); -static inline struct sk_buff *sdp_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) +static inline void sdp_arm_rx_cq(struct sock *sk) +{ + sdp_prf(sk, NULL, "Arming RX cq"); + sdp_dbg_data(sk, "Arming RX cq\n"); + + ib_req_notify_cq(sdp_sk(sk)->rx_ring.cq, IB_CQ_NEXT_COMP); +} + +static inline void sdp_arm_tx_cq(struct sock *sk) +{ + sdp_prf(sk, NULL, "Arming TX cq"); + sdp_dbg_data(sk, "Arming TX cq. credits: %d, posted: %d\n", + tx_credits(sdp_sk(sk)), ring_posted(sdp_sk(sk)->tx_ring)); + + ib_req_notify_cq(sdp_sk(sk)->tx_ring.cq, IB_CQ_NEXT_COMP); +} + +/* utilities */ +static inline char *mid2str(int mid) +{ +#define ENUM2STR(e) [e] = #e + static char *mid2str[] = { + ENUM2STR(SDP_MID_HELLO), + ENUM2STR(SDP_MID_HELLO_ACK), + ENUM2STR(SDP_MID_DISCONN), + ENUM2STR(SDP_MID_CHRCVBUF), + ENUM2STR(SDP_MID_CHRCVBUF_ACK), + ENUM2STR(SDP_MID_DATA), + }; + + if (mid >= ARRAY_SIZE(mid2str)) + return NULL; + + return mid2str[mid]; +} + +static inline struct sk_buff *sdp_stream_alloc_skb(struct sock *sk, int size, + gfp_t gfp) { struct sk_buff *skb; @@ -380,5 +686,4 @@ static inline struct sk_buff *sdp_stream_alloc_skb(struct sock *sk, int size, gf return NULL; } - #endif diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 475d7c3..e2c4ffd 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -39,459 +39,139 @@ #define SDP_RESIZE_WAIT 16 -struct sdp_chrecvbuf { - u32 size; -}; - -static int rcvbuf_scale = 0x10; - -int rcvbuf_initial_size = SDP_HEAD_SIZE; -module_param_named(rcvbuf_initial_size, rcvbuf_initial_size, int, 0644); -MODULE_PARM_DESC(rcvbuf_initial_size, "Receive buffer initial size in bytes."); - -module_param_named(rcvbuf_scale, rcvbuf_scale, int, 0644); -MODULE_PARM_DESC(rcvbuf_scale, "Receive buffer size scale factor."); - -static int top_mem_usage = 0; -module_param_named(top_mem_usage, top_mem_usage, int, 0644); -MODULE_PARM_DESC(top_mem_usage, "Top system wide sdp memory usage for recv (in MB)."); - -#ifdef CONFIG_PPC -static int max_large_sockets = 100; -#else -static int max_large_sockets = 1000; -#endif -module_param_named(max_large_sockets, max_large_sockets, int, 0644); -MODULE_PARM_DESC(max_large_sockets, "Max number of large sockets (32k buffers)."); - -#define sdp_cnt(var) do { (var)++; } while (0) -static unsigned sdp_keepalive_probes_sent = 0; - -module_param_named(sdp_keepalive_probes_sent, sdp_keepalive_probes_sent, uint, 0644); -MODULE_PARM_DESC(sdp_keepalive_probes_sent, "Total number of keepalive probes sent."); - -static int curr_large_sockets = 0; -atomic_t sdp_current_mem_usage; -spinlock_t sdp_large_sockets_lock; - -static int sdp_get_large_socket(struct sdp_sock *ssk) -{ - int count, ret; - - if (ssk->recv_request) - return 1; - - spin_lock_irq(&sdp_large_sockets_lock); - count = curr_large_sockets; - ret = curr_large_sockets < max_large_sockets; - if (ret) - curr_large_sockets++; - spin_unlock_irq(&sdp_large_sockets_lock); - - return ret; -} - -void sdp_remove_large_sock(struct sdp_sock *ssk) +#ifdef CONFIG_INFINIBAND_SDP_DEBUG_DATA +void _dump_packet(const char *func, int line, struct sock *sk, char *str, + struct sk_buff *skb, const struct sdp_bsdh *h) { - if (ssk->recv_frags) { - spin_lock_irq(&sdp_large_sockets_lock); - curr_large_sockets--; - spin_unlock_irq(&sdp_large_sockets_lock); - } -} - -/* Like tcp_fin - called when SDP_MID_DISCONNECT is received */ -static void sdp_fin(struct sock *sk) -{ - sdp_dbg(sk, "%s\n", __func__); + struct sdp_hh *hh; + struct sdp_hah *hah; + struct sdp_chrecvbuf *req_size; + int len = 0; + char buf[256]; + len += snprintf(buf, 255-len, "%s skb: %p mid: %2x:%-20s flags: 0x%x " + "bufs: %d len: %d mseq: %d mseq_ack: %d | ", + str, skb, h->mid, mid2str(h->mid), h->flags, + ntohs(h->bufs), ntohl(h->len), ntohl(h->mseq), + ntohl(h->mseq_ack)); - sk->sk_shutdown |= RCV_SHUTDOWN; - sock_set_flag(sk, SOCK_DONE); - - switch (sk->sk_state) { - case TCP_SYN_RECV: - case TCP_ESTABLISHED: - sdp_exch_state(sk, TCPF_SYN_RECV | TCPF_ESTABLISHED, - TCP_CLOSE_WAIT); + switch (h->mid) { + case SDP_MID_HELLO: + hh = (struct sdp_hh *)h; + len += snprintf(buf + len, 255-len, + "max_adverts: %d majv_minv: %d " + "localrcvsz: %d desremrcvsz: %d |", + hh->max_adverts, hh->majv_minv, + ntohl(hh->localrcvsz), + ntohl(hh->desremrcvsz)); break; - - case TCP_FIN_WAIT1: - /* Received a reply FIN - start Infiniband tear down */ - sdp_dbg(sk, "%s: Starting Infiniband tear down sending DREQ\n", - __func__); - - sdp_cancel_dreq_wait_timeout(sdp_sk(sk)); - - sdp_exch_state(sk, TCPF_FIN_WAIT1, TCP_TIME_WAIT); - - if (sdp_sk(sk)->id) { - rdma_disconnect(sdp_sk(sk)->id); - } else { - sdp_warn(sk, "%s: sdp_sk(sk)->id is NULL\n", __func__); - return; - } + case SDP_MID_HELLO_ACK: + hah = (struct sdp_hah *)h; + len += snprintf(buf + len, 255-len, "actrcvz: %d |", + ntohl(hah->actrcvsz)); break; - case TCP_TIME_WAIT: - /* This is a mutual close situation and we've got the DREQ from - the peer before the SDP_MID_DISCONNECT */ + case SDP_MID_CHRCVBUF: + case SDP_MID_CHRCVBUF_ACK: + req_size = (struct sdp_chrecvbuf *)(h+1); + len += snprintf(buf + len, 255-len, "req_size: %d |", + ntohl(req_size->size)); break; - case TCP_CLOSE: - /* FIN arrived after IB teardown started - do nothing */ - sdp_dbg(sk, "%s: fin in state %s\n", - __func__, sdp_state_str(sk->sk_state)); - return; + case SDP_MID_DATA: + len += snprintf(buf + len, 255-len, "data_len: %ld |", + ntohl(h->len) - sizeof(struct sdp_bsdh)); default: - sdp_warn(sk, "%s: FIN in unexpected state. sk->sk_state=%d\n", - __func__, sk->sk_state); break; } - - - sk_mem_reclaim(sk); - - if (!sock_flag(sk, SOCK_DEAD)) { - sk->sk_state_change(sk); - - /* Do not send POLL_HUP for half duplex close. */ - if (sk->sk_shutdown == SHUTDOWN_MASK || - sk->sk_state == TCP_CLOSE) - sk_wake_async(sk, 1, POLL_HUP); - else - sk_wake_async(sk, 1, POLL_IN); - } -} - -void sdp_post_keepalive(struct sdp_sock *ssk) -{ - int rc; - struct ib_send_wr wr, *bad_wr; - - sdp_dbg(&ssk->isk.sk, "%s\n", __func__); - - memset(&wr, 0, sizeof(wr)); - - wr.next = NULL; - wr.wr_id = 0; - wr.sg_list = NULL; - wr.num_sge = 0; - wr.opcode = IB_WR_RDMA_WRITE; - - rc = ib_post_send(ssk->qp, &wr, &bad_wr); - if (rc) { - sdp_dbg(&ssk->isk.sk, "ib_post_keepalive failed with status %d.\n", rc); - sdp_set_error(&ssk->isk.sk, -ECONNRESET); - wake_up(&ssk->wq); - } - - sdp_cnt(sdp_keepalive_probes_sent); + buf[len] = 0; + _sdp_printk(func, line, KERN_WARNING, sk, "%s: %s\n", str, buf); } +#endif -void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) +static inline void update_send_head(struct sock *sk, struct sk_buff *skb) { - struct sdp_buf *tx_req; - struct sdp_bsdh *h = (struct sdp_bsdh *)skb_push(skb, sizeof *h); - unsigned mseq = ssk->tx_head; - int i, rc, frags; - u64 addr; - struct ib_device *dev; - struct ib_sge *sge; - struct ib_send_wr *bad_wr; - - h->mid = mid; - if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) - h->flags = SDP_OOB_PRES | SDP_OOB_PEND; - else - h->flags = 0; - - h->bufs = htons(ssk->rx_head - ssk->rx_tail); - h->len = htonl(skb->len); - h->mseq = htonl(mseq); - h->mseq_ack = htonl(ssk->mseq_ack); - - tx_req = &ssk->tx_ring[mseq & (SDP_TX_SIZE - 1)]; - tx_req->skb = skb; - dev = ssk->ib_device; - sge = ssk->ibsge; - addr = ib_dma_map_single(dev, skb->data, skb->len - skb->data_len, - DMA_TO_DEVICE); - tx_req->mapping[0] = addr; - - /* TODO: proper error handling */ - BUG_ON(ib_dma_mapping_error(dev, addr)); - - sge->addr = addr; - sge->length = skb->len - skb->data_len; - sge->lkey = ssk->mr->lkey; - frags = skb_shinfo(skb)->nr_frags; - for (i = 0; i < frags; ++i) { - ++sge; - addr = ib_dma_map_page(dev, skb_shinfo(skb)->frags[i].page, - skb_shinfo(skb)->frags[i].page_offset, - skb_shinfo(skb)->frags[i].size, - DMA_TO_DEVICE); - BUG_ON(ib_dma_mapping_error(dev, addr)); - tx_req->mapping[i + 1] = addr; - sge->addr = addr; - sge->length = skb_shinfo(skb)->frags[i].size; - sge->lkey = ssk->mr->lkey; - } - - ssk->tx_wr.next = NULL; - ssk->tx_wr.wr_id = ssk->tx_head | SDP_OP_SEND; - ssk->tx_wr.sg_list = ssk->ibsge; - ssk->tx_wr.num_sge = frags + 1; - ssk->tx_wr.opcode = IB_WR_SEND; - ssk->tx_wr.send_flags = IB_SEND_SIGNALED; - if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) - ssk->tx_wr.send_flags |= IB_SEND_SOLICITED; - rc = ib_post_send(ssk->qp, &ssk->tx_wr, &bad_wr); - ++ssk->tx_head; - --ssk->bufs; - ssk->remote_credits = ssk->rx_head - ssk->rx_tail; - if (unlikely(rc)) { - sdp_dbg(&ssk->isk.sk, "ib_post_send failed with status %d.\n", rc); - sdp_set_error(&ssk->isk.sk, -ECONNRESET); - wake_up(&ssk->wq); + struct page *page; + sk->sk_send_head = skb->next; + if (sk->sk_send_head == (struct sk_buff *)&sk->sk_write_queue) { + sk->sk_send_head = NULL; + page = sk->sk_sndmsg_page; + if (page) { + put_page(page); + sk->sk_sndmsg_page = NULL; + } } } -struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq) +static inline int sdp_nagle_off(struct sdp_sock *ssk, struct sk_buff *skb) { - struct ib_device *dev; - struct sdp_buf *tx_req; - struct sk_buff *skb; - struct bzcopy_state *bz; - int i, frags; - - if (unlikely(mseq != ssk->tx_tail)) { - printk(KERN_WARNING "Bogus send completion id %d tail %d\n", - mseq, ssk->tx_tail); - return NULL; - } + int send_now = + BZCOPY_STATE(skb) || + (ssk->nonagle & TCP_NAGLE_OFF) || + !ssk->nagle_last_unacked || + skb->next != (struct sk_buff *)&ssk->isk.sk.sk_write_queue || + skb->len + sizeof(struct sdp_bsdh) >= ssk->xmit_size_goal || + (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_PSH); - dev = ssk->ib_device; - tx_req = &ssk->tx_ring[mseq & (SDP_TX_SIZE - 1)]; - skb = tx_req->skb; - ib_dma_unmap_single(dev, tx_req->mapping[0], skb->len - skb->data_len, - DMA_TO_DEVICE); - frags = skb_shinfo(skb)->nr_frags; - for (i = 0; i < frags; ++i) { - ib_dma_unmap_page(dev, tx_req->mapping[i + 1], - skb_shinfo(skb)->frags[i].size, - DMA_TO_DEVICE); + if (send_now) { + unsigned long mseq = ring_head(ssk->tx_ring); + ssk->nagle_last_unacked = mseq; + } else { + if (!timer_pending(&ssk->nagle_timer)) { + mod_timer(&ssk->nagle_timer, + jiffies + SDP_NAGLE_TIMEOUT); + sdp_dbg_data(&ssk->isk.sk, "Starting nagle timer\n"); + } } + sdp_dbg_data(&ssk->isk.sk, "send_now = %d last_unacked = %ld\n", + send_now, ssk->nagle_last_unacked); - ssk->snd_una += TCP_SKB_CB(skb)->end_seq; - ++ssk->tx_tail; - - /* TODO: AIO and real zcopy cdoe; add their context support here */ - bz = *(struct bzcopy_state **)skb->cb; - if (bz) - bz->busy--; - - return skb; + return send_now; } - -static void sdp_post_recv(struct sdp_sock *ssk) +void sdp_nagle_timeout(unsigned long data) { - struct sdp_buf *rx_req; - int i, rc, frags; - u64 addr; - struct ib_device *dev; - struct ib_sge *sge; - struct ib_recv_wr *bad_wr; - struct sk_buff *skb; - struct page *page; - skb_frag_t *frag; - struct sdp_bsdh *h; - int id = ssk->rx_head; - gfp_t gfp_page; - - /* Now, allocate and repost recv */ - /* TODO: allocate from cache */ - - if (unlikely(ssk->isk.sk.sk_allocation)) { - skb = sdp_stream_alloc_skb(&ssk->isk.sk, SDP_HEAD_SIZE, - ssk->isk.sk.sk_allocation); - gfp_page = ssk->isk.sk.sk_allocation | __GFP_HIGHMEM; - } else { - skb = sdp_stream_alloc_skb(&ssk->isk.sk, SDP_HEAD_SIZE, - GFP_KERNEL); - gfp_page = GFP_HIGHUSER; - } - - /* FIXME */ - BUG_ON(!skb); - h = (struct sdp_bsdh *)skb->head; - for (i = 0; i < ssk->recv_frags; ++i) { - page = alloc_pages(gfp_page, 0); - BUG_ON(!page); - frag = &skb_shinfo(skb)->frags[i]; - frag->page = page; - frag->page_offset = 0; - - /* Bugzilla 1311 */ - if ( sizeof(frag->size) < 4 ) - frag->size = min(PAGE_SIZE, SDP_MAX_PAYLOAD); - else - frag->size = PAGE_SIZE; - - ++skb_shinfo(skb)->nr_frags; - skb->len += frag->size; - skb->data_len += frag->size; - skb->truesize += frag->size; - } - - rx_req = ssk->rx_ring + (id & (SDP_RX_SIZE - 1)); - rx_req->skb = skb; - dev = ssk->ib_device; - sge = ssk->ibsge; - addr = ib_dma_map_single(dev, h, SDP_HEAD_SIZE, DMA_FROM_DEVICE); - BUG_ON(ib_dma_mapping_error(dev, addr)); + struct sdp_sock *ssk = (struct sdp_sock *)data; + struct sock *sk = &ssk->isk.sk; - rx_req->mapping[0] = addr; + sdp_dbg_data(sk, "last_unacked = %ld\n", ssk->nagle_last_unacked); - /* TODO: proper error handling */ - sge->addr = (u64)addr; - sge->length = SDP_HEAD_SIZE; - sge->lkey = ssk->mr->lkey; - frags = skb_shinfo(skb)->nr_frags; - for (i = 0; i < frags; ++i) { - ++sge; - addr = ib_dma_map_page(dev, skb_shinfo(skb)->frags[i].page, - skb_shinfo(skb)->frags[i].page_offset, - skb_shinfo(skb)->frags[i].size, - DMA_FROM_DEVICE); - BUG_ON(ib_dma_mapping_error(dev, addr)); - rx_req->mapping[i + 1] = addr; - sge->addr = addr; - sge->length = skb_shinfo(skb)->frags[i].size; - sge->lkey = ssk->mr->lkey; - } + if (!ssk->nagle_last_unacked) + goto out2; - ssk->rx_wr.next = NULL; - ssk->rx_wr.wr_id = id | SDP_OP_RECV; - ssk->rx_wr.sg_list = ssk->ibsge; - ssk->rx_wr.num_sge = frags + 1; - rc = ib_post_recv(ssk->qp, &ssk->rx_wr, &bad_wr); - ++ssk->rx_head; - if (unlikely(rc)) { - sdp_dbg(&ssk->isk.sk, "ib_post_recv failed with status %d\n", rc); - sdp_reset(&ssk->isk.sk); + /* Only process if the socket is not in use */ + bh_lock_sock(sk); + if (sock_owned_by_user(sk)) { + sdp_dbg_data(sk, "socket is busy - will try later\n"); + goto out; } - atomic_add(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); -} - -void sdp_post_recvs(struct sdp_sock *ssk) -{ - struct sock *sk = &ssk->isk.sk; - int scale = ssk->rcvbuf_scale; - - if (unlikely(!ssk->id || ((1 << sk->sk_state) & - (TCPF_CLOSE | TCPF_TIME_WAIT)))) { + if (sk->sk_state == TCP_CLOSE) { + bh_unlock_sock(sk); return; } - if (top_mem_usage && - (top_mem_usage * 0x100000) < atomic_read(&sdp_current_mem_usage) * PAGE_SIZE) - scale = 1; - - while ((likely(ssk->rx_head - ssk->rx_tail < SDP_RX_SIZE) && - (ssk->rx_head - ssk->rx_tail - SDP_MIN_BUFS) * - (SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE) + - ssk->rcv_nxt - ssk->copied_seq < sk->sk_rcvbuf * scale) || - unlikely(ssk->rx_head - ssk->rx_tail < SDP_MIN_BUFS)) - sdp_post_recv(ssk); -} - -struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id) -{ - struct sdp_buf *rx_req; - struct ib_device *dev; - struct sk_buff *skb; - int i, frags; - - if (unlikely(id != ssk->rx_tail)) { - printk(KERN_WARNING "Bogus recv completion id %d tail %d\n", - id, ssk->rx_tail); - return NULL; - } + ssk->nagle_last_unacked = 0; + sdp_post_sends(ssk, 0); - dev = ssk->ib_device; - rx_req = &ssk->rx_ring[id & (SDP_RX_SIZE - 1)]; - skb = rx_req->skb; - ib_dma_unmap_single(dev, rx_req->mapping[0], SDP_HEAD_SIZE, - DMA_FROM_DEVICE); - frags = skb_shinfo(skb)->nr_frags; - for (i = 0; i < frags; ++i) - ib_dma_unmap_page(dev, rx_req->mapping[i + 1], - skb_shinfo(skb)->frags[i].size, - DMA_FROM_DEVICE); - ++ssk->rx_tail; - --ssk->remote_credits; - return skb; + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + sk_stream_write_space(&ssk->isk.sk); +out: + bh_unlock_sock(sk); +out2: + if (sk->sk_send_head) /* If has pending sends - rearm */ + mod_timer(&ssk->nagle_timer, jiffies + SDP_NAGLE_TIMEOUT); } -/* Here because I do not want queue to fail. */ -static inline struct sk_buff *sdp_sock_queue_rcv_skb(struct sock *sk, - struct sk_buff *skb) +int sdp_post_credits(struct sdp_sock *ssk) { - int skb_len; - struct sdp_sock *ssk = sdp_sk(sk); - struct sk_buff *tail = NULL; - - /* not needed since sk_rmem_alloc is not currently used - * TODO - remove this? - skb_set_owner_r(skb, sk); */ - - skb_len = skb->len; + int post_count = 0; - TCP_SKB_CB(skb)->seq = ssk->rcv_nxt; - ssk->rcv_nxt += skb_len; + sdp_dbg_data(&ssk->isk.sk, "credits: %d remote credits: %d " + "tx ring slots left: %d send_head: %p\n", + tx_credits(ssk), remote_credits(ssk), + sdp_tx_ring_slots_left(&ssk->tx_ring), + ssk->isk.sk.sk_send_head); - if (likely(skb_len && (tail = skb_peek_tail(&sk->sk_receive_queue))) && - unlikely(skb_tailroom(tail) >= skb_len)) { - skb_copy_bits(skb, 0, skb_put(tail, skb_len), skb_len); - __kfree_skb(skb); - skb = tail; - } else - skb_queue_tail(&sk->sk_receive_queue, skb); - - if (!sock_flag(sk, SOCK_DEAD)) - sk->sk_data_ready(sk, skb_len); - return skb; -} - -static inline void update_send_head(struct sock *sk, struct sk_buff *skb) -{ - struct page *page; - sk->sk_send_head = skb->next; - if (sk->sk_send_head == (struct sk_buff *)&sk->sk_write_queue) { - sk->sk_send_head = NULL; - page = sk->sk_sndmsg_page; - if (page) { - put_page(page); - sk->sk_sndmsg_page = NULL; - } - } -} - -static inline int sdp_nagle_off(struct sdp_sock *ssk, struct sk_buff *skb) -{ - return (ssk->nonagle & TCP_NAGLE_OFF) || - skb->next != (struct sk_buff *)&ssk->isk.sk.sk_write_queue || - skb->len + sizeof(struct sdp_bsdh) >= ssk->xmit_size_goal || - (ssk->tx_tail == ssk->tx_head && - !(ssk->nonagle & TCP_NAGLE_CORK)) || - (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_PSH); -} - -int sdp_post_credits(struct sdp_sock *ssk) -{ - if (likely(ssk->bufs > 1) && - likely(ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE)) { + if (likely(tx_credits(ssk) > 1) && + likely(sdp_tx_ring_slots_left(&ssk->tx_ring))) { struct sk_buff *skb; skb = sdp_stream_alloc_skb(&ssk->isk.sk, sizeof(struct sdp_bsdh), @@ -499,8 +179,12 @@ int sdp_post_credits(struct sdp_sock *ssk) if (!skb) return -ENOMEM; sdp_post_send(ssk, skb, SDP_MID_DATA); + post_count++; } - return 0; + + if (post_count) + sdp_xmit_poll(ssk, 0); + return post_count; } void sdp_post_sends(struct sdp_sock *ssk, int nonagle) @@ -509,6 +193,7 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) struct sk_buff *skb; int c; gfp_t gfp_page; + int post_count = 0; if (unlikely(!ssk->id)) { if (ssk->isk.sk.sk_send_head) { @@ -525,10 +210,15 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) else gfp_page = GFP_KERNEL; + if (sdp_tx_ring_slots_left(&ssk->tx_ring) < SDP_TX_SIZE / 2) { + int wc_processed = sdp_xmit_poll(ssk, 1); + sdp_dbg_data(&ssk->isk.sk, "freed %d\n", wc_processed); + } + if (ssk->recv_request && - ssk->rx_tail >= ssk->recv_request_head && - ssk->bufs >= SDP_MIN_BUFS && - ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE) { + ring_tail(ssk->rx_ring) >= ssk->recv_request_head && + tx_credits(ssk) >= SDP_MIN_TX_CREDITS && + sdp_tx_ring_slots_left(&ssk->tx_ring)) { struct sdp_chrecvbuf *resp_size; ssk->recv_request = 0; skb = sdp_stream_alloc_skb(&ssk->isk.sk, @@ -537,45 +227,37 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) gfp_page); /* FIXME */ BUG_ON(!skb); - resp_size = (struct sdp_chrecvbuf *)skb_put(skb, sizeof *resp_size); + resp_size = (struct sdp_chrecvbuf *)skb_put(skb, + sizeof *resp_size); resp_size->size = htonl(ssk->recv_frags * PAGE_SIZE); sdp_post_send(ssk, skb, SDP_MID_CHRCVBUF_ACK); + post_count++; + } + + if (tx_credits(ssk) <= SDP_MIN_TX_CREDITS && + sdp_tx_ring_slots_left(&ssk->tx_ring) && + ssk->isk.sk.sk_send_head && + sdp_nagle_off(ssk, ssk->isk.sk.sk_send_head)) { + SDPSTATS_COUNTER_INC(send_miss_no_credits); } - while (ssk->bufs > SDP_MIN_BUFS && - ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE && + while (tx_credits(ssk) > SDP_MIN_TX_CREDITS && + sdp_tx_ring_slots_left(&ssk->tx_ring) && (skb = ssk->isk.sk.sk_send_head) && sdp_nagle_off(ssk, skb)) { update_send_head(&ssk->isk.sk, skb); __skb_dequeue(&ssk->isk.sk.sk_write_queue); sdp_post_send(ssk, skb, SDP_MID_DATA); + post_count++; } - if (ssk->bufs == SDP_MIN_BUFS && - !ssk->sent_request && - ssk->tx_head > ssk->sent_request_head + SDP_RESIZE_WAIT && - ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE) { - struct sdp_chrecvbuf *req_size; - skb = sdp_stream_alloc_skb(&ssk->isk.sk, - sizeof(struct sdp_bsdh) + - sizeof(*req_size), - gfp_page); - /* FIXME */ - BUG_ON(!skb); - ssk->sent_request = SDP_MAX_SEND_SKB_FRAGS * PAGE_SIZE; - ssk->sent_request_head = ssk->tx_head; - req_size = (struct sdp_chrecvbuf *)skb_put(skb, sizeof *req_size); - req_size->size = htonl(ssk->sent_request); - sdp_post_send(ssk, skb, SDP_MID_CHRCVBUF); - } - - c = ssk->remote_credits; - if (likely(c > SDP_MIN_BUFS)) + c = remote_credits(ssk); + if (likely(c > SDP_MIN_TX_CREDITS)) c *= 2; - if (unlikely(c < ssk->rx_head - ssk->rx_tail) && - likely(ssk->bufs > 1) && - likely(ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE) && + if (unlikely(c < ring_posted(ssk->rx_ring)) && + likely(tx_credits(ssk) > 1) && + likely(sdp_tx_ring_slots_left(&ssk->tx_ring)) && likely((1 << ssk->isk.sk.sk_state) & (TCPF_ESTABLISHED | TCPF_FIN_WAIT1))) { skb = sdp_stream_alloc_skb(&ssk->isk.sk, @@ -583,12 +265,19 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) GFP_KERNEL); /* FIXME */ BUG_ON(!skb); + SDPSTATS_COUNTER_INC(post_send_credits); sdp_post_send(ssk, skb, SDP_MID_DATA); + post_count++; } + /* send DisConn if needed + * Do not send DisConn if there is only 1 credit. Compliance with CA4-82 + * If one credit is available, an implementation shall only send SDP + * messages that provide additional credits and also do not contain ULP + * payload. */ if (unlikely(ssk->sdp_disconnect) && - !ssk->isk.sk.sk_send_head && - ssk->bufs > (ssk->remote_credits >= ssk->rx_head - ssk->rx_tail)) { + !ssk->isk.sk.sk_send_head && + tx_credits(ssk) > 1) { ssk->sdp_disconnect = 0; skb = sdp_stream_alloc_skb(&ssk->isk.sk, sizeof(struct sdp_bsdh), @@ -596,307 +285,9 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) /* FIXME */ BUG_ON(!skb); sdp_post_send(ssk, skb, SDP_MID_DISCONN); + post_count++; } -} - -int sdp_init_buffers(struct sdp_sock *ssk, u32 new_size) -{ - ssk->recv_frags = PAGE_ALIGN(new_size - SDP_HEAD_SIZE) / PAGE_SIZE; - if (ssk->recv_frags > SDP_MAX_SEND_SKB_FRAGS) - ssk->recv_frags = SDP_MAX_SEND_SKB_FRAGS; - ssk->rcvbuf_scale = rcvbuf_scale; - - sdp_post_recvs(ssk); - - return 0; -} - -int sdp_resize_buffers(struct sdp_sock *ssk, u32 new_size) -{ - skb_frag_t skb_frag; - u32 curr_size = SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE; - u32 max_size = SDP_HEAD_SIZE + SDP_MAX_SEND_SKB_FRAGS * PAGE_SIZE; - - /* Bugzilla 1311, Kernels using smaller fragments must reject - * re-size requests larger than 32k to prevent being sent - * fragment larger than the receive buffer fragment. - */ - if ( (sizeof(skb_frag.size) < 4) && (max_size > 0x8000)) - max_size = 0x8000; - - if (new_size > curr_size && new_size <= max_size && - sdp_get_large_socket(ssk)) { - ssk->rcvbuf_scale = rcvbuf_scale; - ssk->recv_frags = PAGE_ALIGN(new_size - SDP_HEAD_SIZE) / PAGE_SIZE; - if (ssk->recv_frags > SDP_MAX_SEND_SKB_FRAGS) - ssk->recv_frags = SDP_MAX_SEND_SKB_FRAGS; - return 0; - } else - return -1; -} - -static void sdp_handle_resize_request(struct sdp_sock *ssk, struct sdp_chrecvbuf *buf) -{ - if (sdp_resize_buffers(ssk, ntohl(buf->size)) == 0) - ssk->recv_request_head = ssk->rx_head + 1; - else - ssk->recv_request_head = ssk->rx_tail; - ssk->recv_request = 1; -} - -static void sdp_handle_resize_ack(struct sdp_sock *ssk, struct sdp_chrecvbuf *buf) -{ - u32 new_size = ntohl(buf->size); - - if (new_size > ssk->xmit_size_goal) { - ssk->sent_request = -1; - ssk->xmit_size_goal = new_size; - ssk->send_frags = - PAGE_ALIGN(ssk->xmit_size_goal) / PAGE_SIZE; - } else - ssk->sent_request = 0; -} - -static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) -{ - struct sock *sk = &ssk->isk.sk; - int frags; - struct sk_buff *skb; - struct sdp_bsdh *h; - int pagesz, i; - - skb = sdp_recv_completion(ssk, wc->wr_id); - if (unlikely(!skb)) - return -1; - - atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); - - if (unlikely(wc->status)) { - if (wc->status != IB_WC_WR_FLUSH_ERR) { - sdp_dbg(sk, "Recv completion with error. Status %d\n", - wc->status); - sdp_reset(sk); - } - __kfree_skb(skb); - return 0; - } - - sdp_dbg_data(sk, "Recv completion. ID %d Length %d\n", - (int)wc->wr_id, wc->byte_len); - if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { - printk(KERN_WARNING "SDP BUG! byte_len %d < %zd\n", - wc->byte_len, sizeof(struct sdp_bsdh)); - __kfree_skb(skb); - return -1; - } - skb->len = wc->byte_len; - if (likely(wc->byte_len > SDP_HEAD_SIZE)) - skb->data_len = wc->byte_len - SDP_HEAD_SIZE; - else - skb->data_len = 0; - skb->data = skb->head; -#ifdef NET_SKBUFF_DATA_USES_OFFSET - skb->tail = skb_headlen(skb); -#else - skb->tail = skb->head + skb_headlen(skb); -#endif - h = (struct sdp_bsdh *)skb->data; - skb_reset_transport_header(skb); - ssk->mseq_ack = ntohl(h->mseq); - if (ssk->mseq_ack != (int)wc->wr_id) - printk(KERN_WARNING "SDP BUG! mseq %d != wrid %d\n", - ssk->mseq_ack, (int)wc->wr_id); - ssk->bufs = ntohl(h->mseq_ack) - ssk->tx_head + 1 + - ntohs(h->bufs); - - frags = skb_shinfo(skb)->nr_frags; - pagesz = PAGE_ALIGN(skb->data_len); - skb_shinfo(skb)->nr_frags = pagesz / PAGE_SIZE; - - for (i = skb_shinfo(skb)->nr_frags; - i < frags; ++i) { - put_page(skb_shinfo(skb)->frags[i].page); - skb->truesize -= PAGE_SIZE; - } - - if (unlikely(h->flags & SDP_OOB_PEND)) - sk_send_sigurg(sk); - - skb_pull(skb, sizeof(struct sdp_bsdh)); - - switch (h->mid) { - case SDP_MID_DATA: - if (unlikely(skb->len <= 0)) { - __kfree_skb(skb); - break; - } - - if (unlikely(sk->sk_shutdown & RCV_SHUTDOWN)) { - /* got data in RCV_SHUTDOWN */ - if (sk->sk_state == TCP_FIN_WAIT1) { - /* go into abortive close */ - sdp_exch_state(sk, TCPF_FIN_WAIT1, - TCP_TIME_WAIT); - - sk->sk_prot->disconnect(sk, 0); - } - - __kfree_skb(skb); - break; - } - skb = sdp_sock_queue_rcv_skb(sk, skb); - if (unlikely(h->flags & SDP_OOB_PRES)) - sdp_urg(ssk, skb); - break; - case SDP_MID_DISCONN: - __kfree_skb(skb); - sdp_fin(sk); - break; - case SDP_MID_CHRCVBUF: - sdp_handle_resize_request(ssk, - (struct sdp_chrecvbuf *)skb->data); - __kfree_skb(skb); - break; - case SDP_MID_CHRCVBUF_ACK: - sdp_handle_resize_ack(ssk, (struct sdp_chrecvbuf *)skb->data); - __kfree_skb(skb); - break; - default: - /* TODO: Handle other messages */ - printk(KERN_WARNING "SDP: FIXME MID %d\n", h->mid); - __kfree_skb(skb); - } - - return 0; -} - -static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) -{ - struct sk_buff *skb; - struct sdp_bsdh *h; - - skb = sdp_send_completion(ssk, wc->wr_id); - if (unlikely(!skb)) - return -1; - - if (unlikely(wc->status)) { - if (wc->status != IB_WC_WR_FLUSH_ERR) { - struct sock *sk = &ssk->isk.sk; - sdp_dbg(sk, "Send completion with error. " - "Status %d\n", wc->status); - sdp_set_error(sk, -ECONNRESET); - wake_up(&ssk->wq); - - queue_work(sdp_workqueue, &ssk->destroy_work); - } - goto out; - } - - h = (struct sdp_bsdh *)skb->data; - - if (likely(h->mid != SDP_MID_DISCONN)) - goto out; - - if ((1 << ssk->isk.sk.sk_state) & ~(TCPF_FIN_WAIT1 | TCPF_LAST_ACK)) { - sdp_dbg(&ssk->isk.sk, - "%s: sent DISCONNECT from unexpected state %d\n", - __func__, ssk->isk.sk.sk_state); - } - -out: - sk_wmem_free_skb(&ssk->isk.sk, skb); - - return 0; -} - -static void sdp_handle_wc(struct sdp_sock *ssk, struct ib_wc *wc) -{ - if (wc->wr_id & SDP_OP_RECV) { - if (sdp_handle_recv_comp(ssk, wc)) - return; - } else if (likely(wc->wr_id & SDP_OP_SEND)) { - if (sdp_handle_send_comp(ssk, wc)) - return; - } else { - sdp_cnt(sdp_keepalive_probes_sent); - if (likely(!wc->status)) - return; - - sdp_dbg(&ssk->isk.sk, " %s consumes KEEPALIVE status %d\n", - __func__, wc->status); - - if (wc->status == IB_WC_WR_FLUSH_ERR) - return; - - sdp_set_error(&ssk->isk.sk, -ECONNRESET); - wake_up(&ssk->wq); - - return; - } -} - -void sdp_completion_handler(struct ib_cq *cq, void *cq_context) -{ - struct sock *sk = cq_context; - struct sdp_sock *ssk = sdp_sk(sk); - schedule_work(&ssk->work); -} - -int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq) -{ - int n, i; - int ret = -EAGAIN; - do { - n = ib_poll_cq(cq, SDP_NUM_WC, ssk->ibwc); - for (i = 0; i < n; ++i) { - sdp_handle_wc(ssk, ssk->ibwc + i); - ret = 0; - } - } while (n == SDP_NUM_WC); - - if (!ret) { - struct sock *sk = &ssk->isk.sk; - - sdp_post_recvs(ssk); - sdp_post_sends(ssk, 0); - - if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) - sk_stream_write_space(&ssk->isk.sk); - } - - return ret; -} - -void sdp_work(struct work_struct *work) -{ - struct sdp_sock *ssk = container_of(work, struct sdp_sock, work); - struct sock *sk = &ssk->isk.sk; - struct ib_cq *cq; - - sdp_dbg_data(sk, "%s\n", __func__); - - lock_sock(sk); - cq = ssk->cq; - if (unlikely(!cq)) - goto out; - - if (unlikely(!ssk->poll_cq)) { - struct rdma_cm_id *id = ssk->id; - if (id && id->qp) - rdma_notify(id, RDMA_CM_EVENT_ESTABLISHED); - goto out; - } - - sdp_poll_cq(ssk, cq); - release_sock(sk); - sk_mem_reclaim(sk); - lock_sock(sk); - cq = ssk->cq; - if (unlikely(!cq)) - goto out; - ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); - sdp_poll_cq(ssk, cq); -out: - release_sock(sk); + if (post_count) + sdp_xmit_poll(ssk, 0); } diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index 96d65bd..f7dc7c6 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -46,49 +46,13 @@ #include "sdp_socket.h" #include "sdp.h" -union cma_ip_addr { - struct in6_addr ip6; - struct { - __u32 pad[3]; - __u32 addr; - } ip4; -}; - #define SDP_MAJV_MINV 0x22 -/* TODO: too much? Can I avoid having the src/dst and port here? */ -struct sdp_hh { - struct sdp_bsdh bsdh; - u8 majv_minv; - u8 ipv_cap; - u8 rsvd1; - u8 max_adverts; - __u32 desremrcvsz; - __u32 localrcvsz; - __u16 port; - __u16 rsvd2; - union cma_ip_addr src_addr; - union cma_ip_addr dst_addr; -}; - -struct sdp_hah { - struct sdp_bsdh bsdh; - u8 majv_minv; - u8 ipv_cap; - u8 rsvd1; - u8 ext_max_adverts; - __u32 actrcvsz; -}; - enum { SDP_HH_SIZE = 76, SDP_HAH_SIZE = 180, }; -static void sdp_cq_event_handler(struct ib_event *event, void *data) -{ -} - static void sdp_qp_event_handler(struct ib_event *event, void *data) { } @@ -98,43 +62,19 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) struct ib_qp_init_attr qp_init_attr = { .event_handler = sdp_qp_event_handler, .cap.max_send_wr = SDP_TX_SIZE, - .cap.max_send_sge = SDP_MAX_SEND_SKB_FRAGS + 1, /* TODO */ + .cap.max_send_sge = SDP_MAX_SEND_SKB_FRAGS, .cap.max_recv_wr = SDP_RX_SIZE, - .cap.max_recv_sge = SDP_MAX_SEND_SKB_FRAGS + 1, /* TODO */ + .cap.max_recv_sge = SDP_MAX_RECV_SKB_FRAGS + 1, .sq_sig_type = IB_SIGNAL_REQ_WR, .qp_type = IB_QPT_RC, }; struct ib_device *device = id->device; - struct ib_cq *cq; struct ib_mr *mr; struct ib_pd *pd; int rc; sdp_dbg(sk, "%s\n", __func__); - sdp_sk(sk)->tx_head = 1; - sdp_sk(sk)->tx_tail = 1; - sdp_sk(sk)->rx_head = 1; - sdp_sk(sk)->rx_tail = 1; - - sdp_sk(sk)->tx_ring = kmalloc(sizeof *sdp_sk(sk)->tx_ring * SDP_TX_SIZE, - GFP_KERNEL); - if (!sdp_sk(sk)->tx_ring) { - rc = -ENOMEM; - sdp_warn(sk, "Unable to allocate TX Ring size %zd.\n", - sizeof *sdp_sk(sk)->tx_ring * SDP_TX_SIZE); - goto err_tx; - } - - sdp_sk(sk)->rx_ring = kmalloc(sizeof *sdp_sk(sk)->rx_ring * SDP_RX_SIZE, - GFP_KERNEL); - if (!sdp_sk(sk)->rx_ring) { - rc = -ENOMEM; - sdp_warn(sk, "Unable to allocate RX Ring size %zd.\n", - sizeof *sdp_sk(sk)->rx_ring * SDP_RX_SIZE); - goto err_rx; - } - pd = ib_alloc_pd(device); if (IS_ERR(pd)) { rc = PTR_ERR(pd); @@ -150,27 +90,23 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) } sdp_sk(sk)->mr = mr; - INIT_WORK(&sdp_sk(sk)->work, sdp_work); - - cq = ib_create_cq(device, sdp_completion_handler, sdp_cq_event_handler, - sk, SDP_TX_SIZE + SDP_RX_SIZE, 0); - if (IS_ERR(cq)) { - rc = PTR_ERR(cq); - sdp_warn(sk, "Unable to allocate CQ: %d.\n", rc); - goto err_cq; - } + rc = sdp_rx_ring_create(sdp_sk(sk), device); + if (rc) + goto err_rx; - ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); + rc = sdp_tx_ring_create(sdp_sk(sk), device); + if (rc) + goto err_tx; - qp_init_attr.send_cq = qp_init_attr.recv_cq = cq; + qp_init_attr.recv_cq = sdp_sk(sk)->rx_ring.cq; + qp_init_attr.send_cq = sdp_sk(sk)->tx_ring.cq; rc = rdma_create_qp(id, pd, &qp_init_attr); if (rc) { sdp_warn(sk, "Unable to create QP: %d.\n", rc); goto err_qp; } - sdp_sk(sk)->cq = cq; sdp_sk(sk)->qp = id->qp; sdp_sk(sk)->ib_device = device; @@ -180,18 +116,14 @@ static int sdp_init_qp(struct sock *sk, struct rdma_cm_id *id) return 0; err_qp: - ib_destroy_cq(cq); -err_cq: + sdp_tx_ring_destroy(sdp_sk(sk)); +err_tx: + sdp_rx_ring_destroy(sdp_sk(sk)); +err_rx: ib_dereg_mr(sdp_sk(sk)->mr); err_mr: ib_dealloc_pd(pd); err_pd: - kfree(sdp_sk(sk)->rx_ring); - sdp_sk(sk)->rx_ring = NULL; -err_rx: - kfree(sdp_sk(sk)->tx_ring); - sdp_sk(sk)->tx_ring = NULL; -err_tx: return rc; } @@ -206,6 +138,7 @@ static int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, sdp_dbg(sk, "%s %p -> %p\n", __func__, sdp_sk(sk)->id, id); h = event->param.conn.private_data; + SDP_DUMP_PACKET(sk, "RX", NULL, &h->bsdh); if (!h->max_adverts) return -EINVAL; @@ -232,24 +165,21 @@ static int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, sdp_add_sock(sdp_sk(child)); - sdp_sk(child)->max_bufs = sdp_sk(child)->bufs = ntohs(h->bsdh.bufs); - sdp_sk(child)->min_bufs = sdp_sk(child)->bufs / 4; + sdp_sk(child)->max_bufs = ntohs(h->bsdh.bufs); + atomic_set(&sdp_sk(child)->tx_ring.credits, sdp_sk(child)->max_bufs); + + sdp_sk(child)->min_bufs = tx_credits(sdp_sk(child)) / 4; sdp_sk(child)->xmit_size_goal = ntohl(h->localrcvsz) - sizeof(struct sdp_bsdh); sdp_sk(child)->send_frags = PAGE_ALIGN(sdp_sk(child)->xmit_size_goal) / - PAGE_SIZE; - sdp_init_buffers(sdp_sk(child), ntohl(h->desremrcvsz)); - - sdp_dbg(child, "%s bufs %d xmit_size_goal %d send trigger %d\n", - __func__, - sdp_sk(child)->bufs, - sdp_sk(child)->xmit_size_goal, - sdp_sk(child)->min_bufs); + PAGE_SIZE + 1; /* +1 to conpensate on not aligned buffers */ + sdp_init_buffers(sdp_sk(child), rcvbuf_initial_size); id->context = child; sdp_sk(child)->id = id; - list_add_tail(&sdp_sk(child)->backlog_queue, &sdp_sk(sk)->backlog_queue); + list_add_tail(&sdp_sk(child)->backlog_queue, + &sdp_sk(sk)->backlog_queue); sdp_sk(child)->parent = sk; sdp_exch_state(child, TCPF_LISTEN | TCPF_CLOSE, TCP_SYN_RECV); @@ -269,6 +199,7 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, sdp_dbg(sk, "%s\n", __func__); sdp_exch_state(sk, TCPF_SYN_SENT, TCP_ESTABLISHED); + sdp_set_default_moderation(sdp_sk(sk)); if (sock_flag(sk, SOCK_KEEPOPEN)) sdp_start_keepalive_timer(sk); @@ -277,22 +208,19 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, return 0; h = event->param.conn.private_data; - sdp_sk(sk)->max_bufs = sdp_sk(sk)->bufs = ntohs(h->bsdh.bufs); - sdp_sk(sk)->min_bufs = sdp_sk(sk)->bufs / 4; - sdp_sk(sk)->xmit_size_goal = ntohl(h->actrcvsz) - - sizeof(struct sdp_bsdh); - sdp_sk(sk)->send_frags = PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / - PAGE_SIZE; - - sdp_dbg(sk, "%s bufs %d xmit_size_goal %d send trigger %d\n", - __func__, - sdp_sk(sk)->bufs, - sdp_sk(sk)->xmit_size_goal, - sdp_sk(sk)->min_bufs); + SDP_DUMP_PACKET(sk, "RX", NULL, &h->bsdh); + sdp_sk(sk)->max_bufs = ntohs(h->bsdh.bufs); + atomic_set(&sdp_sk(sk)->tx_ring.credits, sdp_sk(sk)->max_bufs); + sdp_sk(sk)->min_bufs = tx_credits(sdp_sk(sk)) / 4; + sdp_sk(sk)->xmit_size_goal = + ntohl(h->actrcvsz) - sizeof(struct sdp_bsdh); + sdp_sk(sk)->send_frags = MIN(PAGE_ALIGN(sdp_sk(sk)->xmit_size_goal) / + PAGE_SIZE, MAX_SKB_FRAGS) + 1; /* +1 to conpensate on not */ + /* aligned buffers */ + sdp_sk(sk)->xmit_size_goal = MIN(sdp_sk(sk)->xmit_size_goal, + sdp_sk(sk)->send_frags * PAGE_SIZE); sdp_sk(sk)->poll_cq = 1; - ib_req_notify_cq(sdp_sk(sk)->cq, IB_CQ_NEXT_COMP); - sdp_poll_cq(sdp_sk(sk), sdp_sk(sk)->cq); sk->sk_state_change(sk); sk_wake_async(sk, 0, POLL_OUT); @@ -314,6 +242,8 @@ static int sdp_connected_handler(struct sock *sk, struct rdma_cm_event *event) sdp_exch_state(sk, TCPF_SYN_RECV, TCP_ESTABLISHED); + sdp_set_default_moderation(sdp_sk(sk)); + if (sock_flag(sk, SOCK_KEEPOPEN)) sdp_start_keepalive_timer(sk); @@ -325,18 +255,12 @@ static int sdp_connected_handler(struct sock *sk, struct rdma_cm_event *event) sdp_dbg(sk, "parent is going away.\n"); goto done; } -#if 0 - /* TODO: backlog */ - if (sk_acceptq_is_full(parent)) { - sdp_dbg(parent, "%s ECONNREFUSED: parent accept queue full: %d > %d\n", __func__, parent->sk_ack_backlog, parent->sk_max_ack_backlog); - release_sock(parent); - return -ECONNREFUSED; - } -#endif + sk_acceptq_added(parent); sdp_dbg(parent, "%s child connection established\n", __func__); list_del_init(&sdp_sk(sk)->backlog_queue); - list_add_tail(&sdp_sk(sk)->accept_queue, &sdp_sk(parent)->accept_queue); + list_add_tail(&sdp_sk(sk)->accept_queue, + &sdp_sk(parent)->accept_queue); parent->sk_state_change(parent); sk_wake_async(parent, 0, POLL_OUT); @@ -352,13 +276,13 @@ static int sdp_disconnected_handler(struct sock *sk) sdp_dbg(sk, "%s\n", __func__); - if (ssk->cq) - sdp_poll_cq(ssk, ssk->cq); + if (ssk->tx_ring.cq) + sdp_xmit_poll(ssk, 1); if (sk->sk_state == TCP_SYN_RECV) { sdp_connected_handler(sk, NULL); - if (ssk->rcv_nxt) + if (rcv_nxt(ssk)) return 0; } @@ -400,7 +324,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) rc = rdma_resolve_route(id, SDP_ROUTE_TIMEOUT); break; case RDMA_CM_EVENT_ADDR_ERROR: - sdp_dbg(sk, "RDMA_CM_EVENT_ADDR_ERROR\n"); + sdp_warn(sk, "RDMA_CM_EVENT_ADDR_ERROR\n"); rc = -ENETUNREACH; break; case RDMA_CM_EVENT_ROUTE_RESOLVED: @@ -408,17 +332,17 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) rc = sdp_init_qp(sk, id); if (rc) break; - sdp_sk(sk)->remote_credits = sdp_sk(sk)->rx_head - - sdp_sk(sk)->rx_tail; + atomic_set(&sdp_sk(sk)->remote_credits, + ring_posted(sdp_sk(sk)->rx_ring)); memset(&hh, 0, sizeof hh); hh.bsdh.mid = SDP_MID_HELLO; - hh.bsdh.bufs = htons(sdp_sk(sk)->remote_credits); + hh.bsdh.bufs = htons(remote_credits(sdp_sk(sk))); hh.bsdh.len = htonl(sizeof(struct sdp_bsdh) + SDP_HH_SIZE); hh.max_adverts = 1; hh.majv_minv = SDP_MAJV_MINV; sdp_init_buffers(sdp_sk(sk), rcvbuf_initial_size); hh.localrcvsz = hh.desremrcvsz = htonl(sdp_sk(sk)->recv_frags * - PAGE_SIZE + SDP_HEAD_SIZE); + PAGE_SIZE + sizeof(struct sdp_bsdh)); hh.max_adverts = 0x1; inet_sk(sk)->saddr = inet_sk(sk)->rcv_saddr = ((struct sockaddr_in *)&id->route.addr.src_addr)->sin_addr.s_addr; @@ -428,6 +352,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) conn_param.responder_resources = 4 /* TODO */; conn_param.initiator_depth = 4 /* TODO */; conn_param.retry_count = SDP_RETRY_COUNT; + SDP_DUMP_PACKET(NULL, "TX", NULL, &hh.bsdh); rc = rdma_connect(id, &conn_param); break; case RDMA_CM_EVENT_ROUTE_ERROR: @@ -442,22 +367,24 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) break; } child = id->context; - sdp_sk(child)->remote_credits = sdp_sk(child)->rx_head - - sdp_sk(child)->rx_tail; + atomic_set(&sdp_sk(child)->remote_credits, + ring_posted(sdp_sk(child)->rx_ring)); memset(&hah, 0, sizeof hah); hah.bsdh.mid = SDP_MID_HELLO_ACK; - hah.bsdh.bufs = htons(sdp_sk(child)->remote_credits); + hah.bsdh.bufs = htons(remote_credits(sdp_sk(child))); hah.bsdh.len = htonl(sizeof(struct sdp_bsdh) + SDP_HAH_SIZE); hah.majv_minv = SDP_MAJV_MINV; hah.ext_max_adverts = 1; /* Doesn't seem to be mandated by spec, but just in case */ - hah.actrcvsz = htonl(sdp_sk(child)->recv_frags * PAGE_SIZE + SDP_HEAD_SIZE); + hah.actrcvsz = htonl(sdp_sk(child)->recv_frags * PAGE_SIZE + + sizeof(struct sdp_bsdh)); memset(&conn_param, 0, sizeof conn_param); conn_param.private_data_len = sizeof hah; conn_param.private_data = &hah; conn_param.responder_resources = 4 /* TODO */; conn_param.initiator_depth = 4 /* TODO */; conn_param.retry_count = SDP_RETRY_COUNT; + SDP_DUMP_PACKET(sk, "TX", NULL, &hah.bsdh); rc = rdma_accept(id, &conn_param); if (rc) { sdp_sk(child)->id = NULL; @@ -475,7 +402,7 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) rc = rdma_accept(id, NULL); if (!rc) - rc = sdp_post_credits(sdp_sk(sk)); + rc = sdp_post_credits(sdp_sk(sk)) < 0 ?: 0; break; case RDMA_CM_EVENT_CONNECT_ERROR: sdp_dbg(sk, "RDMA_CM_EVENT_CONNECT_ERROR\n"); @@ -511,8 +438,9 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) if (sk->sk_state != TCP_TIME_WAIT) { if (sk->sk_state == TCP_CLOSE_WAIT) { - sdp_dbg(sk, "IB teardown while in TCP_CLOSE_WAIT " - "taking reference to let close() finish the work\n"); + sdp_dbg(sk, "IB teardown while in " + "TCP_CLOSE_WAIT taking reference to " + "let close() finish the work\n"); sock_hold(sk, SOCK_REF_CM_TW); } sdp_set_error(sk, EPIPE); @@ -550,7 +478,6 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) sdp_dbg(sk, "event %d done. status %d\n", event->event, rc); if (parent) { - sdp_dbg(parent, "deleting child %d done. status %d\n", event->event, rc); lock_sock(parent); if (!sdp_sk(parent)->id) { /* TODO: look at SOCK_DEAD? */ sdp_dbg(sk, "parent is going away.\n"); diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index 7a38c47..8e1a6d7 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -67,7 +67,6 @@ unsigned int csum_partial_copy_from_user_new (const char *src, char *dst, #include #include #include -#include #include #include /* TODO: remove when sdp_socket.h becomes part of include/linux/socket.h */ @@ -80,66 +79,42 @@ MODULE_DESCRIPTION("InfiniBand SDP module"); MODULE_LICENSE("Dual BSD/GPL"); #ifdef CONFIG_INFINIBAND_SDP_DEBUG -int sdp_debug_level; - -module_param_named(debug_level, sdp_debug_level, int, 0644); -MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0."); +SDP_MODPARAM_INT(sdp_debug_level, 0, "Enable debug tracing if > 0."); #endif #ifdef CONFIG_INFINIBAND_SDP_DEBUG -int sdp_data_debug_level; - -module_param_named(data_debug_level, sdp_data_debug_level, int, 0644); -MODULE_PARM_DESC(data_debug_level, "Enable data path debug tracing if > 0."); +SDP_MODPARAM_INT(sdp_data_debug_level, 0, + "Enable data path debug tracing if > 0."); #endif -static int send_poll_hit; - -module_param_named(send_poll_hit, send_poll_hit, int, 0644); -MODULE_PARM_DESC(send_poll_hit, "How many times send poll helped."); - -static int send_poll_miss; - -module_param_named(send_poll_miss, send_poll_miss, int, 0644); -MODULE_PARM_DESC(send_poll_miss, "How many times send poll missed."); - -static int recv_poll_hit; - -module_param_named(recv_poll_hit, recv_poll_hit, int, 0644); -MODULE_PARM_DESC(recv_poll_hit, "How many times recv poll helped."); - -static int recv_poll_miss; - -module_param_named(recv_poll_miss, recv_poll_miss, int, 0644); -MODULE_PARM_DESC(recv_poll_miss, "How many times recv poll missed."); - -static int send_poll = 100; - -module_param_named(send_poll, send_poll, int, 0644); -MODULE_PARM_DESC(send_poll, "How many times to poll send."); - -static int recv_poll = 1000; - -module_param_named(recv_poll, recv_poll, int, 0644); -MODULE_PARM_DESC(recv_poll, "How many times to poll recv."); - -static int send_poll_thresh = 8192; - -module_param_named(send_poll_thresh, send_poll_thresh, int, 0644); -MODULE_PARM_DESC(send_poll_thresh, "Send message size thresh hold over which to start polling."); - -static unsigned int sdp_keepalive_time = SDP_KEEPALIVE_TIME; - -module_param_named(sdp_keepalive_time, sdp_keepalive_time, uint, 0644); -MODULE_PARM_DESC(sdp_keepalive_time, "Default idle time in seconds before keepalive probe sent."); - -static int sdp_zcopy_thresh = 65536; -module_param_named(sdp_zcopy_thresh, sdp_zcopy_thresh, int, 0644); -MODULE_PARM_DESC(sdp_zcopy_thresh, "Zero copy send threshold; 0=0ff."); - -struct workqueue_struct *sdp_workqueue; - -static struct list_head sock_list; -static spinlock_t sock_list_lock; +SDP_MODPARAM_SINT(recv_poll_hit, -1, "How many times recv poll helped."); +SDP_MODPARAM_SINT(recv_poll_miss, -1, "How many times recv poll missed."); +SDP_MODPARAM_SINT(recv_poll, 1000, "How many times to poll recv."); +SDP_MODPARAM_SINT(sdp_keepalive_time, SDP_KEEPALIVE_TIME, + "Default idle time in seconds before keepalive probe sent."); +SDP_MODPARAM_SINT(sdp_zcopy_thresh, 65536, "Zero copy send threshold; 0=0ff."); + +#define SDP_RX_COAL_TIME_HIGH 128 +SDP_MODPARAM_SINT(sdp_rx_coal_target, 0x50000, + "Target number of bytes to coalesce with interrupt moderation."); +SDP_MODPARAM_SINT(sdp_rx_coal_time, 0x10, "rx coal time (jiffies)."); +SDP_MODPARAM_SINT(sdp_rx_rate_low, 80000, "rx_rate low (packets/sec)."); +SDP_MODPARAM_SINT(sdp_rx_coal_time_low, 0, "low moderation usec."); +SDP_MODPARAM_SINT(sdp_rx_rate_high, 100000, "rx_rate high (packets/sec)."); +SDP_MODPARAM_SINT(sdp_rx_coal_time_high, 128, "high moderation usec."); +SDP_MODPARAM_SINT(sdp_rx_rate_thresh, (200000 / SDP_RX_COAL_TIME_HIGH), + "rx rate thresh ()."); +SDP_MODPARAM_SINT(sdp_sample_interval, (HZ / 4), "sample interval (jiffies)."); + +SDP_MODPARAM_INT(hw_int_mod_count, -1, + "forced hw int moderation val. -1 for auto (packets)."); +SDP_MODPARAM_INT(hw_int_mod_usec, -1, + "forced hw int moderation val. -1 for auto (usec)."); + +struct workqueue_struct *sdp_wq; +struct workqueue_struct *rx_comp_wq; + +struct list_head sock_list; +spinlock_t sock_list_lock; static DEFINE_RWLOCK(device_removal_lock); @@ -206,52 +181,38 @@ static int sdp_get_port(struct sock *sk, unsigned short snum) static void sdp_destroy_qp(struct sdp_sock *ssk) { struct ib_pd *pd = NULL; - struct ib_cq *cq = NULL; + unsigned long flags; + + + sdp_dbg(&ssk->isk.sk, "destroying qp\n"); + sdp_prf(&ssk->isk.sk, NULL, "destroying qp"); + + del_timer(&ssk->tx_ring.timer); + + rx_ring_lock(ssk, flags); + + sdp_rx_ring_destroy(ssk); + sdp_tx_ring_destroy(ssk); if (ssk->qp) { pd = ssk->qp->pd; - cq = ssk->cq; - ssk->cq = NULL; ib_destroy_qp(ssk->qp); - - while (ssk->rx_head != ssk->rx_tail) { - struct sk_buff *skb; - skb = sdp_recv_completion(ssk, ssk->rx_tail); - if (!skb) - break; - atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); - __kfree_skb(skb); - } - while (ssk->tx_head != ssk->tx_tail) { - struct sk_buff *skb; - skb = sdp_send_completion(ssk, ssk->tx_tail); - if (!skb) - break; - __kfree_skb(skb); - } + ssk->qp = NULL; } - if (cq) - ib_destroy_cq(cq); - - if (ssk->mr) + if (ssk->mr) { ib_dereg_mr(ssk->mr); + ssk->mr = NULL; + } if (pd) ib_dealloc_pd(pd); sdp_remove_large_sock(ssk); - if (ssk->rx_ring) { - kfree(ssk->rx_ring); - ssk->rx_ring = NULL; - } - if (ssk->tx_ring) { - kfree(ssk->tx_ring); - ssk->tx_ring = NULL; - } -} + rx_ring_unlock(ssk, flags); +} static void sdp_reset_keepalive_timer(struct sock *sk, unsigned long len) { @@ -259,8 +220,8 @@ static void sdp_reset_keepalive_timer(struct sock *sk, unsigned long len) sdp_dbg(sk, "%s\n", __func__); - ssk->keepalive_tx_head = ssk->tx_head; - ssk->keepalive_rx_head = ssk->rx_head; + ssk->keepalive_tx_head = ring_head(ssk->tx_ring); + ssk->keepalive_rx_head = ring_head(ssk->rx_ring); sk_reset_timer(sk, &sk->sk_timer, jiffies + len); } @@ -295,8 +256,8 @@ static void sdp_keepalive_timer(unsigned long data) sk->sk_state == TCP_CLOSE) goto out; - if (ssk->keepalive_tx_head == ssk->tx_head && - ssk->keepalive_rx_head == ssk->rx_head) + if (ssk->keepalive_tx_head == ring_head(ssk->tx_ring) && + ssk->keepalive_rx_head == ring_head(ssk->rx_ring)) sdp_post_keepalive(ssk); sdp_reset_keepalive_timer(sk, sdp_keepalive_time_when(ssk)); @@ -306,7 +267,7 @@ out: sock_put(sk, SOCK_REF_BORN); } -static void sdp_init_timer(struct sock *sk) +static void sdp_init_keepalive_timer(struct sock *sk) { init_timer(&sk->sk_timer); @@ -332,6 +293,150 @@ void sdp_start_keepalive_timer(struct sock *sk) sdp_reset_keepalive_timer(sk, sdp_keepalive_time_when(sdp_sk(sk))); } +void sdp_set_default_moderation(struct sdp_sock *ssk) +{ + struct sock *sk = &ssk->isk.sk; + struct sdp_moderation *mod = &ssk->auto_mod; + int rx_buf_size; + + if (hw_int_mod_count > -1 || hw_int_mod_usec > -1) { + int err; + + mod->adaptive_rx_coal = 0; + + if (hw_int_mod_count > 0 && hw_int_mod_usec > 0) { + err = ib_modify_cq(ssk->rx_ring.cq, hw_int_mod_count, + hw_int_mod_usec); + if (err) + sdp_warn(sk, + "Failed modifying moderation for cq"); + else + sdp_dbg(sk, + "Using fixed interrupt moderation\n"); + } + return; + } + + mod->adaptive_rx_coal = 1; + sdp_dbg(sk, "Using adaptive interrupt moderation\n"); + + /* If we haven't received a specific coalescing setting + * (module param), we set the moderation paramters as follows: + * - moder_cnt is set to the number of mtu sized packets to + * satisfy our coelsing target. + * - moder_time is set to a fixed value. + */ + rx_buf_size = (ssk->recv_frags * PAGE_SIZE) + sizeof(struct sdp_bsdh); + mod->moder_cnt = sdp_rx_coal_target / rx_buf_size + 1; + mod->moder_time = sdp_rx_coal_time; + sdp_dbg(sk, "Default coalesing params for buf size:%d - " + "moder_cnt:%d moder_time:%d\n", + rx_buf_size, mod->moder_cnt, mod->moder_time); + + /* Reset auto-moderation params */ + mod->pkt_rate_low = sdp_rx_rate_low; + mod->rx_usecs_low = sdp_rx_coal_time_low; + mod->pkt_rate_high = sdp_rx_rate_high; + mod->rx_usecs_high = sdp_rx_coal_time_high; + mod->sample_interval = sdp_sample_interval; + + mod->last_moder_time = SDP_AUTO_CONF; + mod->last_moder_jiffies = 0; + mod->last_moder_packets = 0; + mod->last_moder_tx_packets = 0; + mod->last_moder_bytes = 0; +} + +/* If tx and rx packet rates are not balanced, assume that + * traffic is mainly BW bound and apply maximum moderation. + * Otherwise, moderate according to packet rate */ +static inline int calc_moder_time(int rate, struct sdp_moderation *mod, + int tx_pkt_diff, int rx_pkt_diff) +{ + if (2 * tx_pkt_diff > 3 * rx_pkt_diff || + 2 * rx_pkt_diff > 3 * tx_pkt_diff) + return mod->rx_usecs_high; + + if (rate < mod->pkt_rate_low) + return mod->rx_usecs_low; + + if (rate > mod->pkt_rate_high) + return mod->rx_usecs_high; + + return (rate - mod->pkt_rate_low) * + (mod->rx_usecs_high - mod->rx_usecs_low) / + (mod->pkt_rate_high - mod->pkt_rate_low) + + mod->rx_usecs_low; +} + +static void sdp_auto_moderation(struct sdp_sock *ssk) +{ + struct sdp_moderation *mod = &ssk->auto_mod; + + unsigned long period = jiffies - mod->last_moder_jiffies; + unsigned long packets; + unsigned long rate; + unsigned long avg_pkt_size; + unsigned long tx_pkt_diff; + unsigned long rx_pkt_diff; + int moder_time; + int err; + + if (!mod->adaptive_rx_coal) + return; + + if (period < mod->sample_interval) + return; + + if (!mod->last_moder_jiffies || !period) + goto out; + + tx_pkt_diff = ((unsigned long) (ssk->tx_packets - + mod->last_moder_tx_packets)); + rx_pkt_diff = ((unsigned long) (ssk->rx_packets - + mod->last_moder_packets)); + packets = max(tx_pkt_diff, rx_pkt_diff); + rate = packets * HZ / period; + avg_pkt_size = packets ? ((unsigned long) (ssk->rx_bytes - + mod->last_moder_bytes)) / packets : 0; + + /* Apply auto-moderation only when packet rate exceeds a rate that + * it matters */ + if (rate > sdp_rx_rate_thresh) { + moder_time = calc_moder_time(rate, mod, tx_pkt_diff, + rx_pkt_diff); + } else { + /* When packet rate is low, use default moderation rather + * than 0 to prevent interrupt storms if traffic suddenly + * increases */ + moder_time = mod->moder_time; + } + + sdp_dbg_data(&ssk->isk.sk, "tx rate:%lu rx_rate:%lu\n", + tx_pkt_diff * HZ / period, rx_pkt_diff * HZ / period); + + sdp_dbg_data(&ssk->isk.sk, "Rx moder_time changed from:%d to %d " + "period:%lu [jiff] packets:%lu avg_pkt_size:%lu " + "rate:%lu [p/s])\n", + mod->last_moder_time, moder_time, period, packets, + avg_pkt_size, rate); + + if (moder_time != mod->last_moder_time) { + mod->last_moder_time = moder_time; + err = ib_modify_cq(ssk->rx_ring.cq, mod->moder_cnt, moder_time); + if (err) { + sdp_dbg_data(&ssk->isk.sk, + "Failed modifying moderation for cq"); + } + } + +out: + mod->last_moder_packets = ssk->rx_packets; + mod->last_moder_tx_packets = ssk->tx_packets; + mod->last_moder_bytes = ssk->rx_bytes; + mod->last_moder_jiffies = jiffies; +} + void sdp_reset_sk(struct sock *sk, int rc) { struct sdp_sock *ssk = sdp_sk(sk); @@ -340,11 +445,15 @@ void sdp_reset_sk(struct sock *sk, int rc) read_lock(&device_removal_lock); - if (ssk->cq) - sdp_poll_cq(ssk, ssk->cq); + if (ssk->tx_ring.cq) + sdp_xmit_poll(ssk, 1); - if (!(sk->sk_shutdown & RCV_SHUTDOWN) || !sk_stream_memory_free(sk)) + if (!(sk->sk_shutdown & RCV_SHUTDOWN) || !sk_stream_memory_free(sk)) { + sdp_warn(sk, "setting state to error\n"); sdp_set_error(sk, rc); + } + + sdp_destroy_qp(ssk); memset((void *)&ssk->id, 0, sizeof(*ssk) - offsetof(typeof(*ssk), id)); @@ -352,7 +461,7 @@ void sdp_reset_sk(struct sock *sk, int rc) /* Don't destroy socket before destroy work does its job */ sock_hold(sk, SOCK_REF_RESET); - queue_work(sdp_workqueue, &ssk->destroy_work); + queue_work(sdp_wq, &ssk->destroy_work); read_unlock(&device_removal_lock); } @@ -430,7 +539,7 @@ static void sdp_destruct(struct sock *sk) ssk->destructed_already = 1; sdp_remove_sock(ssk); - + sdp_close_sk(sk); if (ssk->parent) @@ -485,6 +594,7 @@ static void sdp_close(struct sock *sk, long timeout) lock_sock(sk); sdp_dbg(sk, "%s\n", __func__); + sdp_prf(sk, NULL, __func__); sdp_delete_keepalive_timer(sk); @@ -510,7 +620,7 @@ static void sdp_close(struct sock *sk, long timeout) * descriptor close, not protocol-sourced closes, because the * reader process may not have drained the data yet! */ - while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) { + while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) { data_was_unread = 1; __kfree_skb(skb); } @@ -701,13 +811,9 @@ static int sdp_wait_for_connect(struct sock *sk, long timeo) TASK_INTERRUPTIBLE); release_sock(sk); if (list_empty(&ssk->accept_queue)) { - sdp_dbg(sk, "%s schedule_timeout\n", __func__); timeo = schedule_timeout(timeo); - sdp_dbg(sk, "%s schedule_timeout done\n", __func__); } - sdp_dbg(sk, "%s lock_sock\n", __func__); lock_sock(sk); - sdp_dbg(sk, "%s lock_sock done\n", __func__); err = 0; if (!list_empty(&ssk->accept_queue)) break; @@ -730,7 +836,7 @@ static int sdp_wait_for_connect(struct sock *sk, long timeo) /* Like inet_csk_accept */ static struct sock *sdp_accept(struct sock *sk, int flags, int *err) { - struct sdp_sock *newssk, *ssk; + struct sdp_sock *newssk = NULL, *ssk; struct sock *newsk; int error; @@ -761,7 +867,8 @@ static struct sock *sdp_accept(struct sock *sk, int flags, int *err) goto out_err; } - newssk = list_entry(ssk->accept_queue.next, struct sdp_sock, accept_queue); + newssk = list_entry(ssk->accept_queue.next, struct sdp_sock, + accept_queue); list_del_init(&newssk->accept_queue); newssk->parent = NULL; sk_acceptq_removed(sk); @@ -770,11 +877,9 @@ out: release_sock(sk); if (newsk) { lock_sock(newsk); - if (newssk->cq) { - sdp_dbg(newsk, "%s: ib_req_notify_cq\n", __func__); + if (newssk->rx_ring.cq) { newssk->poll_cq = 1; - ib_req_notify_cq(newssk->cq, IB_CQ_NEXT_COMP); - sdp_poll_cq(newssk, newssk->cq); + sdp_arm_rx_cq(&newssk->isk.sk); } release_sock(newsk); } @@ -807,8 +912,8 @@ static int sdp_ioctl(struct sock *sk, int cmd, unsigned long arg) else if (sock_flag(sk, SOCK_URGINLINE) || !ssk->urg_data || before(ssk->urg_seq, ssk->copied_seq) || - !before(ssk->urg_seq, ssk->rcv_nxt)) { - answ = ssk->rcv_nxt - ssk->copied_seq; + !before(ssk->urg_seq, rcv_nxt(ssk))) { + answ = rcv_nxt(ssk) - ssk->copied_seq; /* Subtract 1, if FIN is in queue. */ if (answ && !skb_queue_empty(&sk->sk_receive_queue)) @@ -829,7 +934,7 @@ static int sdp_ioctl(struct sock *sk, int cmd, unsigned long arg) if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) answ = 0; else - answ = ssk->write_seq - ssk->snd_una; + answ = ssk->write_seq - ssk->tx_ring.una_seq; break; default: return -ENOIOCTLCMD; @@ -837,14 +942,14 @@ static int sdp_ioctl(struct sock *sk, int cmd, unsigned long arg) /* TODO: Need to handle: case SIOCOUTQ: */ - return put_user(answ, (int __user *)arg); + return put_user(answ, (int __user *)arg); } static inline void sdp_start_dreq_wait_timeout(struct sdp_sock *ssk, int timeo) { sdp_dbg(&ssk->isk.sk, "Starting dreq wait timeout\n"); - queue_delayed_work(sdp_workqueue, &ssk->dreq_wait_work, timeo); + queue_delayed_work(sdp_wq, &ssk->dreq_wait_work, timeo); ssk->dreq_wait_timeout = 1; } @@ -866,7 +971,8 @@ void sdp_cancel_dreq_wait_timeout(struct sdp_sock *ssk) void sdp_destroy_work(struct work_struct *work) { - struct sdp_sock *ssk = container_of(work, struct sdp_sock, destroy_work); + struct sdp_sock *ssk = container_of(work, struct sdp_sock, + destroy_work); struct sock *sk = &ssk->isk.sk; sdp_dbg(sk, "%s: refcnt %d\n", __func__, atomic_read(&sk->sk_refcnt)); @@ -908,12 +1014,10 @@ void sdp_dreq_wait_timeout_work(struct work_struct *work) release_sock(sk); - if (sdp_sk(sk)->id) { + if (sdp_sk(sk)->id) rdma_disconnect(sdp_sk(sk)->id); - } else { - sdp_warn(sk, "NOT SENDING DREQ - no need to wait for timewait exit\n"); + else sock_put(sk, SOCK_REF_CM_TW); - } out: sock_put(sk, SOCK_REF_DREQ_TO); @@ -932,13 +1036,19 @@ int sdp_init_sock(struct sock *sk) sk->sk_route_caps |= NETIF_F_SG | NETIF_F_NO_CSUM; - ssk->rx_ring = NULL; - ssk->tx_ring = NULL; + skb_queue_head_init(&ssk->rx_ctl_q); + + atomic_set(&ssk->mseq_ack, 0); + + sdp_rx_ring_init(ssk); + ssk->tx_ring.buffer = NULL; ssk->sdp_disconnect = 0; ssk->destructed_already = 0; ssk->destruct_in_process = 0; spin_lock_init(&ssk->lock); + atomic_set(&ssk->somebody_is_doing_posts, 0); + return 0; } @@ -976,7 +1086,7 @@ static void sdp_mark_push(struct sdp_sock *ssk, struct sk_buff *skb) { TCP_SKB_CB(skb)->flags |= TCPCB_FLAG_PSH; ssk->pushed_seq = ssk->write_seq; - sdp_post_sends(ssk, 0); + sdp_do_posts(ssk); } static inline void sdp_push_pending_frames(struct sock *sk) @@ -984,7 +1094,6 @@ static inline void sdp_push_pending_frames(struct sock *sk) struct sk_buff *skb = sk->sk_send_head; if (skb) { sdp_mark_push(sdp_sk(sk), skb); - sdp_post_sends(sdp_sk(sk), 0); } } @@ -1072,8 +1181,11 @@ static int sdp_setsockopt(struct sock *sk, int level, int optname, ssk->keepalive_time = val * HZ; if (sock_flag(sk, SOCK_KEEPOPEN) && - !((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))) - sdp_reset_keepalive_timer(sk, ssk->keepalive_time); + !((1 << sk->sk_state) & + (TCPF_CLOSE | TCPF_LISTEN))) { + sdp_reset_keepalive_timer(sk, + ssk->keepalive_time); + } } break; case SDP_ZCOPY_THRESH: @@ -1140,30 +1252,16 @@ static int sdp_getsockopt(struct sock *sk, int level, int optname, static inline int poll_recv_cq(struct sock *sk) { int i; - if (sdp_sk(sk)->cq) { - for (i = 0; i < recv_poll; ++i) - if (!sdp_poll_cq(sdp_sk(sk), sdp_sk(sk)->cq)) { - ++recv_poll_hit; - return 0; - } - ++recv_poll_miss; + for (i = 0; i < recv_poll; ++i) { + if (!skb_queue_empty(&sk->sk_receive_queue)) { + ++recv_poll_hit; + return 0; + } } + ++recv_poll_miss; return 1; } -static inline void poll_send_cq(struct sock *sk) -{ - int i; - if (sdp_sk(sk)->cq) { - for (i = 0; i < send_poll; ++i) - if (!sdp_poll_cq(sdp_sk(sk), sdp_sk(sk)->cq)) { - ++send_poll_hit; - return; - } - ++send_poll_miss; - } -} - /* Like tcp_recv_urg */ /* * Handle reading urgent data. BSD has very simple semantics for @@ -1218,30 +1316,8 @@ static int sdp_recv_urg(struct sock *sk, long timeo, return -EAGAIN; } -static void sdp_rcv_space_adjust(struct sock *sk) -{ - sdp_post_recvs(sdp_sk(sk)); - sdp_post_sends(sdp_sk(sk), 0); -} - -static unsigned int sdp_current_mss(struct sock *sk, int large_allowed) -{ - /* TODO */ - return PAGE_SIZE; -} - -static int forced_push(struct sdp_sock *sk) -{ - /* TODO */ - return 0; -} - -static inline int select_size(struct sock *sk, struct sdp_sock *ssk) -{ - return 0; -} - -static inline void sdp_mark_urg(struct sock *sk, struct sdp_sock *ssk, int flags) +static inline void sdp_mark_urg(struct sock *sk, struct sdp_sock *ssk, + int flags) { if (unlikely(flags & MSG_OOB)) { struct sk_buff *skb = sk->sk_write_queue.prev; @@ -1249,12 +1325,11 @@ static inline void sdp_mark_urg(struct sock *sk, struct sdp_sock *ssk, int flags } } -static inline void sdp_push(struct sock *sk, struct sdp_sock *ssk, int flags, - int mss_now, int nonagle) +static inline void sdp_push(struct sock *sk, struct sdp_sock *ssk, int flags) { if (sk->sk_send_head) sdp_mark_urg(sk, ssk, flags); - sdp_post_sends(ssk, nonagle); + sdp_do_posts(sdp_sk(sk)); } static inline void skb_entail(struct sock *sk, struct sdp_sock *ssk, @@ -1270,24 +1345,21 @@ static inline void skb_entail(struct sock *sk, struct sdp_sock *ssk, ssk->nonagle &= ~TCP_NAGLE_PUSH; } -static void sdp_push_one(struct sock *sk, unsigned int mss_now) -{ -} - static inline struct bzcopy_state *sdp_bz_cleanup(struct bzcopy_state *bz) { - int i, max_retry; + int i; struct sdp_sock *ssk = (struct sdp_sock *)bz->ssk; /* Wait for in-flight sends; should be quick */ if (bz->busy) { struct sock *sk = &ssk->isk.sk; + unsigned long timeout = jiffies + SDP_BZCOPY_POLL_TIMEOUT; - for (max_retry = 0; max_retry < 10000; max_retry++) { - poll_send_cq(sk); - + while (jiffies < timeout) { + sdp_xmit_poll(sdp_sk(sk), 1); if (!bz->busy) break; + SDPSTATS_COUNTER_INC(bzcopy_poll_miss); } if (bz->busy) @@ -1319,11 +1391,14 @@ static struct bzcopy_state *sdp_bz_setup(struct sdp_sock *ssk, int thresh; mm_segment_t cur_fs; - cur_fs = get_fs(); - thresh = ssk->zcopy_thresh ? : sdp_zcopy_thresh; - if (thresh == 0 || len < thresh || !capable(CAP_IPC_LOCK)) + if (thresh == 0 || len < thresh || !capable(CAP_IPC_LOCK)) { + SDPSTATS_COUNTER_INC(sendmsg_bcopy_segment); return NULL; + } + SDPSTATS_COUNTER_INC(sendmsg_bzcopy_segment); + + cur_fs = get_fs(); /* * Since we use the TCP segmentation fields of the skb to map user @@ -1347,7 +1422,8 @@ static struct bzcopy_state *sdp_bz_setup(struct sdp_sock *ssk, bz->busy = 0; bz->ssk = ssk; bz->page_cnt = PAGE_ALIGN(len + bz->cur_offset) >> PAGE_SHIFT; - bz->pages = kcalloc(bz->page_cnt, sizeof(struct page *), GFP_KERNEL); + bz->pages = kcalloc(bz->page_cnt, sizeof(struct page *), + GFP_KERNEL); if (!bz->pages) { kfree(bz); @@ -1386,7 +1462,6 @@ static struct bzcopy_state *sdp_bz_setup(struct sdp_sock *ssk, return bz; } - #define TCP_PAGE(sk) (sk->sk_sndmsg_page) #define TCP_OFF(sk) (sk->sk_sndmsg_off) static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, @@ -1413,12 +1488,9 @@ static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, /* We can extend the last page * fragment. */ merge = 1; - } else if (i == ssk->send_frags || - (!i && - !(sk->sk_route_caps & NETIF_F_SG))) { + } else if (i == ssk->send_frags) { /* Need to add new fragment and cannot - * do this because interface is non-SG, - * or because all the page slots are + * do this because all the page slots are * busy. */ sdp_mark_push(ssk, skb); return SDP_NEW_SEG; @@ -1445,6 +1517,7 @@ static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, /* Time to copy data. We are close to * the end! */ + SDPSTATS_COUNTER_ADD(memcpy_count, copy); err = skb_copy_to_page(sk, from, skb, page, off, copy); if (err) { @@ -1478,7 +1551,6 @@ static inline int sdp_bcopy_get(struct sock *sk, struct sk_buff *skb, return copy; } - static inline int sdp_bzcopy_get(struct sock *sk, struct sk_buff *skb, unsigned char __user *from, int copy, struct bzcopy_state *bz) @@ -1539,23 +1611,27 @@ static inline int sdp_bzcopy_get(struct sock *sk, struct sk_buff *skb, return copy; } -static inline int slots_free(struct sdp_sock *ssk) +/* return the min of: + * - tx credits + * - free slots in tx_ring (not including SDP_MIN_TX_CREDITS + */ +static inline int tx_slots_free(struct sdp_sock *ssk) { int min_free; - min_free = SDP_TX_SIZE - (ssk->tx_head - ssk->tx_tail); - if (ssk->bufs < min_free) - min_free = ssk->bufs; - min_free -= (min_free < SDP_MIN_BUFS) ? min_free : SDP_MIN_BUFS; + min_free = MIN(tx_credits(ssk), + SDP_TX_SIZE - ring_posted(ssk->tx_ring)); + if (min_free < SDP_MIN_TX_CREDITS) + return 0; - return min_free; + return min_free - SDP_MIN_TX_CREDITS; }; /* like sk_stream_memory_free - except measures remote credits */ static inline int sdp_bzcopy_slots_avail(struct sdp_sock *ssk, struct bzcopy_state *bz) { - return slots_free(ssk) > bz->busy; + return tx_slots_free(ssk) > bz->busy; } /* like sk_stream_wait_memory - except waits on remote credits */ @@ -1595,14 +1671,24 @@ static int sdp_bzcopy_wait_memory(struct sdp_sock *ssk, long *timeo_p, clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); + posts_handler_put(ssk); + if (sdp_bzcopy_slots_avail(ssk, bz)) break; set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); sk->sk_write_pending++; + + if (tx_credits(ssk) > SDP_MIN_TX_CREDITS) + sdp_arm_tx_cq(sk); + sk_wait_event(sk, ¤t_timeo, sdp_bzcopy_slots_avail(ssk, bz) && vm_wait); sk->sk_write_pending--; + sdp_prf1(sk, NULL, "finished wait for mem"); + + posts_handler_get(ssk); + sdp_do_posts(ssk); if (vm_wait) { vm_wait -= current_timeo; @@ -1619,25 +1705,6 @@ static int sdp_bzcopy_wait_memory(struct sdp_sock *ssk, long *timeo_p, return err; } -/* like sk_stream_write_space - execpt measures remote credits */ -void sdp_bzcopy_write_space(struct sdp_sock *ssk) -{ - struct sock *sk = &ssk->isk.sk; - struct socket *sock = sk->sk_socket; - - if (ssk->bufs >= ssk->min_bufs && - ssk->tx_head == ssk->tx_tail && - sock != NULL) { - clear_bit(SOCK_NOSPACE, &sock->flags); - - if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) - wake_up_interruptible(sk->sk_sleep); - if (sock->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN)) - sock_wake_async(sock, 2, POLL_OUT); - } -} - - /* Like tcp_sendmsg */ /* TODO: check locking */ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, @@ -1647,14 +1714,20 @@ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, struct sdp_sock *ssk = sdp_sk(sk); struct sk_buff *skb; int iovlen, flags; - int mss_now, size_goal; + int size_goal; int err, copied; long timeo; struct bzcopy_state *bz = NULL; - + unsigned long long a, b, c; + unsigned long long start, end; + SDPSTATS_COUNTER_INC(sendmsg); lock_sock(sk); sdp_dbg_data(sk, "%s\n", __func__); + rdtscll(start); + + posts_handler_get(ssk); + flags = msg->msg_flags; timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); @@ -1666,7 +1739,6 @@ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, /* This should be in poll */ clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); - mss_now = sdp_current_mss(sk, !(flags&MSG_OOB)); size_goal = ssk->xmit_size_goal; /* Ok commence sending. */ @@ -1688,9 +1760,16 @@ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, if (size_goal > SDP_MAX_PAYLOAD) size_goal = SDP_MAX_PAYLOAD; + SDPSTATS_HIST(sendmsg_seglen, seglen); + + rdtscll(a); if (bz) sdp_bz_cleanup(bz); + rdtscll(b); bz = sdp_bz_setup(ssk, from, seglen, size_goal); + rdtscll(c); + SDPSTATS_COUNTER_ADD(bz_clean_sum, b - a); + SDPSTATS_COUNTER_ADD(bz_setup_sum, c - b); if (IS_ERR(bz)) { bz = NULL; err = PTR_ERR(bz); @@ -1704,8 +1783,7 @@ static int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, if (!sk->sk_send_head || (copy = size_goal - skb->len) <= 0 || - bz != *(struct bzcopy_state **)skb->cb) { - + bz != BZCOPY_STATE(skb)) { new_segment: /* * Allocate a new segment @@ -1722,12 +1800,12 @@ new_segment: goto wait_for_sndbuf; } - skb = sdp_stream_alloc_skb(sk, select_size(sk, ssk), - sk->sk_allocation); + skb = sdp_stream_alloc_skb(sk, 0, + sk->sk_allocation); if (!skb) goto wait_for_memory; - *((struct bzcopy_state **)skb->cb) = bz; + BZCOPY_STATE(skb) = bz; /* * Check whether we can use HW checksum. @@ -1739,6 +1817,11 @@ new_segment: skb_entail(sk, ssk, skb); copy = size_goal; + } else { + sdp_dbg_data(sk, "adding to existing skb: %p" + " len = %d, sk_send_head: %p " + "copy: %d\n", + skb, skb->len, sk->sk_send_head, copy); } /* Try to append data to the end of skb. */ @@ -1753,8 +1836,12 @@ new_segment: goto new_segment; } + rdtscll(a); copy = (bz) ? sdp_bzcopy_get(sk, skb, from, copy, bz) : sdp_bcopy_get(sk, skb, from, copy); + rdtscll(b); + if (copy > 0) + SDPSTATS_COUNTER_ADD(tx_copy_sum, b - a); if (unlikely(copy < 0)) { if (!++copy) goto wait_for_memory; @@ -1777,48 +1864,60 @@ new_segment: if ((seglen -= copy) == 0 && iovlen == 0) goto out; - if (skb->len < mss_now || (flags & MSG_OOB)) + if (skb->len < PAGE_SIZE || (flags & MSG_OOB)) continue; - - if (forced_push(ssk)) { - sdp_mark_push(ssk, skb); - /* TODO: and push pending frames mss_now */ - /* sdp_push_pending(sk, ssk, mss_now, TCP_NAGLE_PUSH); */ - } else if (skb == sk->sk_send_head) - sdp_push_one(sk, mss_now); continue; wait_for_sndbuf: set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); wait_for_memory: + sdp_prf(sk, skb, "wait for mem"); + SDPSTATS_COUNTER_INC(send_wait_for_mem); if (copied) - sdp_push(sk, ssk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); + sdp_push(sk, ssk, flags & ~MSG_MORE); + + sdp_xmit_poll(ssk, 1); + + if (bz) { + err = sdp_bzcopy_wait_memory(ssk, &timeo, bz); + } else { + posts_handler_put(ssk); + + sdp_arm_tx_cq(sk); + + err = sk_stream_wait_memory(sk, &timeo); + + posts_handler_get(ssk); + sdp_do_posts(ssk); + } - err = (bz) ? sdp_bzcopy_wait_memory(ssk, &timeo, bz) : - sk_stream_wait_memory(sk, &timeo); if (err) goto do_error; - mss_now = sdp_current_mss(sk, !(flags&MSG_OOB)); size_goal = ssk->xmit_size_goal; } } out: if (copied) { - sdp_push(sk, ssk, flags, mss_now, ssk->nonagle); + sdp_push(sk, ssk, flags); if (bz) bz = sdp_bz_cleanup(bz); - else - if (size > send_poll_thresh) - poll_send_cq(sk); } + posts_handler_put(ssk); + + sdp_auto_moderation(ssk); + + rdtscll(end); + SDPSTATS_COUNTER_ADD(sendmsg_sum, end - start); release_sock(sk); return copied; do_fault: + sdp_prf(sk, skb, "prepare fault"); + if (!skb->len) { if (sk->sk_send_head == skb) sk->sk_send_head = NULL; @@ -1833,6 +1932,9 @@ out_err: if (bz) bz = sdp_bz_cleanup(bz); err = sk_stream_error(sk, flags, err); + + posts_handler_put(ssk); + release_sock(sk); return err; } @@ -1841,7 +1943,7 @@ out_err: /* Maybe use skb_recv_datagram here? */ /* Note this does not seem to handle vectored messages. Relevant? */ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, - size_t len, int noblock, int flags, + size_t len, int noblock, int flags, int *addr_len) { struct sk_buff *skb = NULL; @@ -1858,6 +1960,10 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, lock_sock(sk); sdp_dbg_data(sk, "%s\n", __func__); + posts_handler_get(ssk); + + sdp_prf(sk, skb, "Read from user"); + err = -ENOTCONN; if (sk->sk_state == TCP_LISTEN) goto out; @@ -1878,12 +1984,14 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, do { u32 offset; - /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */ + /* Are we at urgent data? Stop if we have read anything or have + * SIGURG pending. */ if (ssk->urg_data && ssk->urg_seq == *seq) { if (copied) break; if (signal_pending(current)) { - copied = timeo ? sock_intr_errno(timeo) : -EAGAIN; + copied = timeo ? sock_intr_errno(timeo) : + -EAGAIN; break; } } @@ -1897,8 +2005,9 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, goto found_fin_ok; if (before(*seq, TCP_SKB_CB(skb)->seq)) { - printk(KERN_INFO "recvmsg bug: copied %X " - "seq %X\n", *seq, TCP_SKB_CB(skb)->seq); + sdp_warn(sk, "recvmsg bug: copied %X seq %X\n", + *seq, TCP_SKB_CB(skb)->seq); + sdp_reset(sk); break; } @@ -1962,15 +2071,24 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, release_sock(sk); lock_sock(sk); } else if (rc) { - sdp_dbg_data(sk, "%s: sk_wait_data %ld\n", __func__, timeo); + sdp_dbg_data(sk, "sk_wait_data %ld\n", timeo); + + posts_handler_put(ssk); + + /* socket lock is released inside sk_wait_data */ sk_wait_data(sk, &timeo); + + posts_handler_get(ssk); + sdp_prf(sk, NULL, "got data"); + + sdp_do_posts(ssk); } continue; found_ok_skb: - sdp_dbg_data(sk, "%s: found_ok_skb len %d\n", __func__, skb->len); - sdp_dbg_data(sk, "%s: len %Zd offset %d\n", __func__, len, offset); - sdp_dbg_data(sk, "%s: copied %d target %d\n", __func__, copied, target); + sdp_dbg_data(sk, "found_ok_skb len %d\n", skb->len); + sdp_dbg_data(sk, "len %Zd offset %d\n", len, offset); + sdp_dbg_data(sk, "copied %d target %d\n", copied, target); used = skb->len - offset; if (len < used) used = len; @@ -2011,9 +2129,9 @@ static int sdp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, len -= used; *seq += used; - sdp_dbg_data(sk, "%s: done copied %d target %d\n", __func__, copied, target); + sdp_dbg_data(sk, "done copied %d target %d\n", copied, target); - sdp_rcv_space_adjust(sk); + sdp_do_posts(sdp_sk(sk)); skip_copy: if (ssk->urg_data && after(ssk->copied_seq, ssk->urg_seq)) ssk->urg_data = 0; @@ -2021,22 +2139,32 @@ skip_copy: continue; offset = 0; - if (!(flags & MSG_PEEK)) - sk_eat_skb(sk, skb, 0); - + if (!(flags & MSG_PEEK)) { + struct sdp_bsdh *h; + h = (struct sdp_bsdh *)skb_transport_header(skb); + sdp_prf1(sk, skb, "READ finished. mseq: %d mseq_ack:%d", + ntohl(h->mseq), ntohl(h->mseq_ack)); + skb_unlink(skb, &sk->sk_receive_queue); + __kfree_skb(skb); + } continue; found_fin_ok: ++*seq; - if (!(flags & MSG_PEEK)) - sk_eat_skb(sk, skb, 0); - + if (!(flags & MSG_PEEK)) { + skb_unlink(skb, &sk->sk_receive_queue); + __kfree_skb(skb); + } break; - } while (len > 0); - release_sock(sk); - return copied; + } while (len > 0); + err = copied; out: + + posts_handler_put(ssk); + + sdp_auto_moderation(ssk); + release_sock(sk); return err; @@ -2128,8 +2256,8 @@ static unsigned int sdp_poll(struct file *file, struct socket *socket, /* * Adjust for memory in later kernels */ - if (!sk_stream_memory_free(sk) || !slots_free(ssk)) - mask &= ~(POLLOUT | POLLWRNORM | POLLWRBAND); + if (!sk_stream_memory_free(sk) || !tx_slots_free(ssk)) + mask &= ~(POLLOUT | POLLWRNORM | POLLWRBAND); /* TODO: Slightly ugly: it would be nicer if there was function * like datagram_poll that didn't include poll_wait, @@ -2221,7 +2349,7 @@ static int sdp_create_socket(struct net *net, struct socket *sock, int protocol) struct sock *sk; int rc; - sdp_dbg(NULL, "%s: type %d protocol %d\n", __func__, sock->type, protocol); + sdp_dbg(NULL, "type %d protocol %d\n", sock->type, protocol); if (net != &init_net) return -EAFNOSUPPORT; @@ -2256,7 +2384,7 @@ static int sdp_create_socket(struct net *net, struct socket *sock, int protocol) sk->sk_destruct = sdp_destruct; - sdp_init_timer(sk); + sdp_init_keepalive_timer(sk); sock->ops = &sdp_proto_ops; sock->state = SS_UNCONNECTED; @@ -2266,192 +2394,6 @@ static int sdp_create_socket(struct net *net, struct socket *sock, int protocol) return 0; } -#ifdef CONFIG_PROC_FS - -static void *sdp_get_idx(struct seq_file *seq, loff_t pos) -{ - int i = 0; - struct sdp_sock *ssk; - - if (!list_empty(&sock_list)) - list_for_each_entry(ssk, &sock_list, sock_list) { - if (i == pos) - return ssk; - i++; - } - - return NULL; -} - -static void *sdp_seq_start(struct seq_file *seq, loff_t *pos) -{ - void *start = NULL; - struct sdp_iter_state* st = seq->private; - - st->num = 0; - - if (!*pos) - return SEQ_START_TOKEN; - - spin_lock_irq(&sock_list_lock); - start = sdp_get_idx(seq, *pos - 1); - if (start) - sock_hold((struct sock *)start, SOCK_REF_SEQ); - spin_unlock_irq(&sock_list_lock); - - return start; -} - -static void *sdp_seq_next(struct seq_file *seq, void *v, loff_t *pos) -{ - struct sdp_iter_state* st = seq->private; - void *next = NULL; - - spin_lock_irq(&sock_list_lock); - if (v == SEQ_START_TOKEN) - next = sdp_get_idx(seq, 0); - else - next = sdp_get_idx(seq, *pos); - if (next) - sock_hold((struct sock *)next, SOCK_REF_SEQ); - spin_unlock_irq(&sock_list_lock); - - *pos += 1; - st->num++; - - return next; -} - -static void sdp_seq_stop(struct seq_file *seq, void *v) -{ -} - -#define TMPSZ 150 - -static int sdp_seq_show(struct seq_file *seq, void *v) -{ - struct sdp_iter_state* st; - struct sock *sk = v; - char tmpbuf[TMPSZ + 1]; - unsigned int dest; - unsigned int src; - int uid; - unsigned long inode; - __u16 destp; - __u16 srcp; - __u32 rx_queue, tx_queue; - - if (v == SEQ_START_TOKEN) { - seq_printf(seq, "%-*s\n", TMPSZ - 1, - " sl local_address rem_address uid inode" - " rx_queue tx_queue state"); - goto out; - } - - st = seq->private; - - dest = inet_sk(sk)->daddr; - src = inet_sk(sk)->rcv_saddr; - destp = ntohs(inet_sk(sk)->dport); - srcp = ntohs(inet_sk(sk)->sport); - uid = sock_i_uid(sk); - inode = sock_i_ino(sk); - rx_queue = sdp_sk(sk)->rcv_nxt - sdp_sk(sk)->copied_seq; - tx_queue = sdp_sk(sk)->write_seq - sdp_sk(sk)->snd_una; - - sprintf(tmpbuf, "%4d: %08X:%04X %08X:%04X %5d %lu %08X:%08X %X", - st->num, src, srcp, dest, destp, uid, inode, - rx_queue, tx_queue, sk->sk_state); - - seq_printf(seq, "%-*s\n", TMPSZ - 1, tmpbuf); - - sock_put(sk, SOCK_REF_SEQ); -out: - return 0; -} - -static int sdp_seq_open(struct inode *inode, struct file *file) -{ - struct sdp_seq_afinfo *afinfo = PDE(inode)->data; - struct seq_file *seq; - struct sdp_iter_state *s; - int rc; - - if (unlikely(afinfo == NULL)) - return -EINVAL; - - s = kzalloc(sizeof(*s), GFP_KERNEL); - if (!s) - return -ENOMEM; - s->family = afinfo->family; - s->seq_ops.start = sdp_seq_start; - s->seq_ops.next = sdp_seq_next; - s->seq_ops.show = afinfo->seq_show; - s->seq_ops.stop = sdp_seq_stop; - - rc = seq_open(file, &s->seq_ops); - if (rc) - goto out_kfree; - seq = file->private_data; - seq->private = s; -out: - return rc; -out_kfree: - kfree(s); - goto out; -} - - -static struct file_operations sdp_seq_fops; -static struct sdp_seq_afinfo sdp_seq_afinfo = { - .owner = THIS_MODULE, - .name = "sdp", - .family = AF_INET_SDP, - .seq_show = sdp_seq_show, - .seq_fops = &sdp_seq_fops, -}; - - -static int __init sdp_proc_init(void) -{ - int rc = 0; - struct proc_dir_entry *p; - - sdp_seq_afinfo.seq_fops->owner = sdp_seq_afinfo.owner; - sdp_seq_afinfo.seq_fops->open = sdp_seq_open; - sdp_seq_afinfo.seq_fops->read = seq_read; - sdp_seq_afinfo.seq_fops->llseek = seq_lseek; - sdp_seq_afinfo.seq_fops->release = seq_release_private; - - p = proc_net_fops_create(&init_net, sdp_seq_afinfo.name, S_IRUGO, - sdp_seq_afinfo.seq_fops); - if (p) - p->data = &sdp_seq_afinfo; - else - rc = -ENOMEM; - - return rc; -} - -static void sdp_proc_unregister(void) -{ - proc_net_remove(&init_net, sdp_seq_afinfo.name); - memset(sdp_seq_afinfo.seq_fops, 0, sizeof(*sdp_seq_afinfo.seq_fops)); -} - -#else /* CONFIG_PROC_FS */ - -static int __init sdp_proc_init(void) -{ - return 0; -} - -static void sdp_proc_unregister(void) -{ - -} -#endif /* CONFIG_PROC_FS */ - static void sdp_add_device(struct ib_device *device) { } @@ -2499,12 +2441,12 @@ kill_socks: sk->sk_shutdown |= RCV_SHUTDOWN; sdp_reset(sk); - if ((1 << sk->sk_state) & + if ((1 << sk->sk_state) & (TCPF_FIN_WAIT1 | TCPF_CLOSE_WAIT | TCPF_LAST_ACK | TCPF_TIME_WAIT)) { sock_put(sk, SOCK_REF_CM_TW); } - + schedule(); spin_lock_irq(&sock_list_lock); @@ -2532,39 +2474,44 @@ static struct ib_client sdp_client = { static int __init sdp_init(void) { - int rc; + int rc = -ENOMEM; INIT_LIST_HEAD(&sock_list); spin_lock_init(&sock_list_lock); spin_lock_init(&sdp_large_sockets_lock); sockets_allocated = kmalloc(sizeof(*sockets_allocated), GFP_KERNEL); + if (!sockets_allocated) + goto no_mem_sockets_allocated; + orphan_count = kmalloc(sizeof(*orphan_count), GFP_KERNEL); + if (!orphan_count) + goto no_mem_orphan_count; + percpu_counter_init(sockets_allocated, 0); percpu_counter_init(orphan_count, 0); sdp_proto.sockets_allocated = sockets_allocated; sdp_proto.orphan_count = orphan_count; + rx_comp_wq = create_singlethread_workqueue("rx_comp_wq"); + if (!rx_comp_wq) + goto no_mem_rx_wq; - sdp_workqueue = create_singlethread_workqueue("sdp"); - if (!sdp_workqueue) { - return -ENOMEM; - } + sdp_wq = create_singlethread_workqueue("sdp_wq"); + if (!sdp_wq) + goto no_mem_sdp_wq; rc = proto_register(&sdp_proto, 1); if (rc) { - printk(KERN_WARNING "%s: proto_register failed: %d\n", __func__, rc); - destroy_workqueue(sdp_workqueue); - return rc; + printk(KERN_WARNING "proto_register failed: %d\n", rc); + goto error_proto_reg; } rc = sock_register(&sdp_net_proto); if (rc) { - printk(KERN_WARNING "%s: sock_register failed: %d\n", __func__, rc); - proto_unregister(&sdp_proto); - destroy_workqueue(sdp_workqueue); - return rc; + printk(KERN_WARNING "sock_register failed: %d\n", rc); + goto error_sock_reg; } sdp_proc_init(); @@ -2574,6 +2521,19 @@ static int __init sdp_init(void) ib_register_client(&sdp_client); return 0; + +error_sock_reg: + proto_unregister(&sdp_proto); +error_proto_reg: + destroy_workqueue(sdp_wq); +no_mem_sdp_wq: + destroy_workqueue(rx_comp_wq); +no_mem_rx_wq: + kfree(orphan_count); +no_mem_orphan_count: + kfree(sockets_allocated); +no_mem_sockets_allocated: + return rc; } static void __exit sdp_exit(void) @@ -2584,7 +2544,10 @@ static void __exit sdp_exit(void) if (percpu_counter_read_positive(orphan_count)) printk(KERN_WARNING "%s: orphan_count %lld\n", __func__, percpu_counter_read_positive(orphan_count)); - destroy_workqueue(sdp_workqueue); + + destroy_workqueue(rx_comp_wq); + destroy_workqueue(sdp_wq); + flush_scheduled_work(); BUG_ON(!list_empty(&sock_list)); diff --git a/drivers/infiniband/ulp/sdp/sdp_proc.c b/drivers/infiniband/ulp/sdp/sdp_proc.c new file mode 100644 index 0000000..924e2ee --- /dev/null +++ b/drivers/infiniband/ulp/sdp/sdp_proc.c @@ -0,0 +1,496 @@ +/* + * Copyright (c) 2008 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include "sdp_socket.h" +#include "sdp.h" + +#ifdef CONFIG_PROC_FS + +#define PROC_SDP_STATS "sdpstats" +#define PROC_SDP_PERF "sdpprf" + +/* just like TCP fs */ +struct sdp_seq_afinfo { + struct module *owner; + char *name; + sa_family_t family; + int (*seq_show) (struct seq_file *m, void *v); + struct file_operations *seq_fops; +}; + +struct sdp_iter_state { + sa_family_t family; + int num; + struct seq_operations seq_ops; +}; + +static void *sdp_get_idx(struct seq_file *seq, loff_t pos) +{ + int i = 0; + struct sdp_sock *ssk; + + if (!list_empty(&sock_list)) + list_for_each_entry(ssk, &sock_list, sock_list) { + if (i == pos) + return ssk; + i++; + } + + return NULL; +} + +static void *sdp_seq_start(struct seq_file *seq, loff_t *pos) +{ + void *start = NULL; + struct sdp_iter_state *st = seq->private; + + st->num = 0; + + if (!*pos) + return SEQ_START_TOKEN; + + spin_lock_irq(&sock_list_lock); + start = sdp_get_idx(seq, *pos - 1); + if (start) + sock_hold((struct sock *)start, SOCK_REF_SEQ); + spin_unlock_irq(&sock_list_lock); + + return start; +} + +static void *sdp_seq_next(struct seq_file *seq, void *v, loff_t *pos) +{ + struct sdp_iter_state *st = seq->private; + void *next = NULL; + + spin_lock_irq(&sock_list_lock); + if (v == SEQ_START_TOKEN) + next = sdp_get_idx(seq, 0); + else + next = sdp_get_idx(seq, *pos); + if (next) + sock_hold((struct sock *)next, SOCK_REF_SEQ); + spin_unlock_irq(&sock_list_lock); + + *pos += 1; + st->num++; + + return next; +} + +static void sdp_seq_stop(struct seq_file *seq, void *v) +{ +} + +#define TMPSZ 150 + +static int sdp_seq_show(struct seq_file *seq, void *v) +{ + struct sdp_iter_state *st; + struct sock *sk = v; + char tmpbuf[TMPSZ + 1]; + unsigned int dest; + unsigned int src; + int uid; + unsigned long inode; + __u16 destp; + __u16 srcp; + __u32 rx_queue, tx_queue; + + if (v == SEQ_START_TOKEN) { + seq_printf(seq, "%-*s\n", TMPSZ - 1, + " sl local_address rem_address " + "uid inode rx_queue tx_queue state"); + goto out; + } + + st = seq->private; + + dest = inet_sk(sk)->daddr; + src = inet_sk(sk)->rcv_saddr; + destp = ntohs(inet_sk(sk)->dport); + srcp = ntohs(inet_sk(sk)->sport); + uid = sock_i_uid(sk); + inode = sock_i_ino(sk); + rx_queue = rcv_nxt(sdp_sk(sk)) - sdp_sk(sk)->copied_seq; + tx_queue = sdp_sk(sk)->write_seq - sdp_sk(sk)->tx_ring.una_seq; + + sprintf(tmpbuf, "%4d: %08X:%04X %08X:%04X %5d %lu %08X:%08X %X", + st->num, src, srcp, dest, destp, uid, inode, + rx_queue, tx_queue, sk->sk_state); + + seq_printf(seq, "%-*s\n", TMPSZ - 1, tmpbuf); + + sock_put(sk, SOCK_REF_SEQ); +out: + return 0; +} + +static int sdp_seq_open(struct inode *inode, struct file *file) +{ + struct sdp_seq_afinfo *afinfo = PDE(inode)->data; + struct seq_file *seq; + struct sdp_iter_state *s; + int rc; + + if (unlikely(afinfo == NULL)) + return -EINVAL; + + s = kzalloc(sizeof(*s), GFP_KERNEL); + if (!s) + return -ENOMEM; + s->family = afinfo->family; + s->seq_ops.start = sdp_seq_start; + s->seq_ops.next = sdp_seq_next; + s->seq_ops.show = afinfo->seq_show; + s->seq_ops.stop = sdp_seq_stop; + + rc = seq_open(file, &s->seq_ops); + if (rc) + goto out_kfree; + seq = file->private_data; + seq->private = s; +out: + return rc; +out_kfree: + kfree(s); + goto out; +} + + +static struct file_operations sdp_seq_fops; +static struct sdp_seq_afinfo sdp_seq_afinfo = { + .owner = THIS_MODULE, + .name = "sdp", + .family = AF_INET_SDP, + .seq_show = sdp_seq_show, + .seq_fops = &sdp_seq_fops, +}; + +#ifdef SDPSTATS_ON +struct sdpstats sdpstats = { { 0 } }; + +static void sdpstats_seq_hist(struct seq_file *seq, char *str, u32 *h, int n, + int is_log) +{ + int i; + u32 max = 0; + + seq_printf(seq, "%s:\n", str); + + for (i = 0; i < n; i++) { + if (h[i] > max) + max = h[i]; + } + + if (max == 0) { + seq_printf(seq, " - all values are 0\n"); + return; + } + + for (i = 0; i < n; i++) { + char s[51]; + int j = 50 * h[i] / max; + int val = is_log ? (i == n-1 ? 0 : 1<time - start_t; + nsec_rem = do_div(t, 1000000000); + + seq_printf(m, "%-6d: [%5lu.%06lu] %-50s - [%d{%d} %d:%d] " + "skb: %p %s:%d\n", + l->idx, (unsigned long)t, nsec_rem/1000, + l->msg, l->pid, l->cpu, l->sk_num, l->sk_dport, + l->skb, l->func, l->line); +out: + return 0; +} + +static void *sdpprf_start(struct seq_file *p, loff_t *pos) +{ + int idx = *pos; + + if (!*pos) { + if (!sdpprf_log_count) + return SEQ_START_TOKEN; + } + + if (*pos >= MIN(sdpprf_log_count, SDPPRF_LOG_SIZE - 1)) + return NULL; + + if (sdpprf_log_count >= SDPPRF_LOG_SIZE - 1) { + int off = sdpprf_log_count & (SDPPRF_LOG_SIZE - 1); + idx = (idx + off) & (SDPPRF_LOG_SIZE - 1); + + } + + if (!start_t) + start_t = sdpprf_log[idx].time; + return &sdpprf_log[idx]; +} + +static void *sdpprf_next(struct seq_file *p, void *v, loff_t *pos) +{ + struct sdpprf_log *l = v; + + if (++*pos >= MIN(sdpprf_log_count, SDPPRF_LOG_SIZE - 1)) + return NULL; + + ++l; + if (l - &sdpprf_log[0] >= SDPPRF_LOG_SIZE - 1) + return &sdpprf_log[0]; + + return l; +} + +static void sdpprf_stop(struct seq_file *p, void *v) +{ +} + + +const struct seq_operations sdpprf_ops = { + .start = sdpprf_start, + .stop = sdpprf_stop, + .next = sdpprf_next, + .show = sdpprf_show, +}; + +static int sdpprf_open(struct inode *inode, struct file *file) +{ + int res; + + res = seq_open(file, &sdpprf_ops); + + return res; +} + +static ssize_t sdpprf_write(struct file *file, const char __user *buf, + size_t count, loff_t *offs) +{ + sdpprf_log_count = 0; + printk(KERN_INFO "Cleared sdpprf statistics\n"); + + return count; +} + +static const struct file_operations sdpprf_fops = { + .open = sdpprf_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release, + .write = sdpprf_write, +}; +#endif /* SDP_PROFILING */ + +int __init sdp_proc_init(void) +{ + struct proc_dir_entry *p = NULL; + struct proc_dir_entry *sdpstats = NULL; + struct proc_dir_entry *sdpprf = NULL; + + sdp_seq_afinfo.seq_fops->owner = sdp_seq_afinfo.owner; + sdp_seq_afinfo.seq_fops->open = sdp_seq_open; + sdp_seq_afinfo.seq_fops->read = seq_read; + sdp_seq_afinfo.seq_fops->llseek = seq_lseek; + sdp_seq_afinfo.seq_fops->release = seq_release_private; + + p = proc_net_fops_create(&init_net, sdp_seq_afinfo.name, S_IRUGO, + sdp_seq_afinfo.seq_fops); + if (p) + p->data = &sdp_seq_afinfo; + else + goto no_mem; + +#ifdef SDPSTATS_ON + + sdpstats = proc_net_fops_create(&init_net, PROC_SDP_STATS, + S_IRUGO | S_IWUGO, &sdpstats_fops); + if (!sdpstats) + goto no_mem; + +#endif + +#ifdef SDP_PROFILING + sdpprf = proc_net_fops_create(&init_net, PROC_SDP_PERF, + S_IRUGO | S_IWUGO, &sdpprf_fops); + if (!sdpprf) + goto no_mem; +#endif + + return 0; +no_mem: + if (sdpprf) + proc_net_remove(&init_net, PROC_SDP_PERF); + + if (sdpstats) + proc_net_remove(&init_net, PROC_SDP_STATS); + + if (p) + proc_net_remove(&init_net, sdp_seq_afinfo.name); + + return -ENOMEM; +} + +void sdp_proc_unregister(void) +{ + proc_net_remove(&init_net, sdp_seq_afinfo.name); + memset(sdp_seq_afinfo.seq_fops, 0, sizeof(*sdp_seq_afinfo.seq_fops)); + +#ifdef SDPSTATS_ON + proc_net_remove(&init_net, PROC_SDP_STATS); +#endif +#ifdef SDP_PROFILING + proc_net_remove(&init_net, PROC_SDP_PERF); +#endif +} + +#else /* CONFIG_PROC_FS */ + +int __init sdp_proc_init(void) +{ + return 0; +} + +void sdp_proc_unregister(void) +{ + +} +#endif /* CONFIG_PROC_FS */ diff --git a/drivers/infiniband/ulp/sdp/sdp_rx.c b/drivers/infiniband/ulp/sdp/sdp_rx.c new file mode 100644 index 0000000..ed5c2ea --- /dev/null +++ b/drivers/infiniband/ulp/sdp/sdp_rx.c @@ -0,0 +1,850 @@ +/* + * Copyright (c) 2009 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +#include +#include +#include +#include +#include "sdp.h" + +SDP_MODPARAM_INT(rcvbuf_initial_size, 32 * 1024, + "Receive buffer initial size in bytes."); +SDP_MODPARAM_SINT(rcvbuf_scale, 0x10, + "Receive buffer size scale factor."); +SDP_MODPARAM_SINT(top_mem_usage, 0, + "Top system wide sdp memory usage for recv (in MB)."); + +#ifdef CONFIG_PPC +SDP_MODPARAM_SINT(max_large_sockets, 100, + "Max number of large sockets (32k buffers)."); +#else +SDP_MODPARAM_SINT(max_large_sockets, 1000, + "Max number of large sockets (32k buffers)."); +#endif + +static int curr_large_sockets; +atomic_t sdp_current_mem_usage; +spinlock_t sdp_large_sockets_lock; + +static int sdp_get_large_socket(struct sdp_sock *ssk) +{ + int count, ret; + + if (ssk->recv_request) + return 1; + + spin_lock_irq(&sdp_large_sockets_lock); + count = curr_large_sockets; + ret = curr_large_sockets < max_large_sockets; + if (ret) + curr_large_sockets++; + spin_unlock_irq(&sdp_large_sockets_lock); + + return ret; +} + +void sdp_remove_large_sock(struct sdp_sock *ssk) +{ + if (ssk->recv_frags) { + spin_lock_irq(&sdp_large_sockets_lock); + curr_large_sockets--; + spin_unlock_irq(&sdp_large_sockets_lock); + } +} + +/* Like tcp_fin - called when SDP_MID_DISCONNECT is received */ +static void sdp_fin(struct sock *sk) +{ + sdp_dbg(sk, "%s\n", __func__); + + sk->sk_shutdown |= RCV_SHUTDOWN; + sock_set_flag(sk, SOCK_DONE); + + switch (sk->sk_state) { + case TCP_SYN_RECV: + case TCP_ESTABLISHED: + sdp_exch_state(sk, TCPF_SYN_RECV | TCPF_ESTABLISHED, + TCP_CLOSE_WAIT); + break; + + case TCP_FIN_WAIT1: + /* Received a reply FIN - start Infiniband tear down */ + sdp_dbg(sk, "%s: Starting Infiniband tear down sending DREQ\n", + __func__); + + sdp_cancel_dreq_wait_timeout(sdp_sk(sk)); + + sdp_exch_state(sk, TCPF_FIN_WAIT1, TCP_TIME_WAIT); + + if (sdp_sk(sk)->id) { + rdma_disconnect(sdp_sk(sk)->id); + } else { + sdp_warn(sk, "%s: sdp_sk(sk)->id is NULL\n", __func__); + return; + } + break; + case TCP_TIME_WAIT: + /* This is a mutual close situation and we've got the DREQ from + the peer before the SDP_MID_DISCONNECT */ + break; + case TCP_CLOSE: + /* FIN arrived after IB teardown started - do nothing */ + sdp_dbg(sk, "%s: fin in state %s\n", + __func__, sdp_state_str(sk->sk_state)); + return; + default: + sdp_warn(sk, "%s: FIN in unexpected state. sk->sk_state=%d\n", + __func__, sk->sk_state); + break; + } + + + sk_mem_reclaim(sk); + + if (!sock_flag(sk, SOCK_DEAD)) { + sk->sk_state_change(sk); + + /* Do not send POLL_HUP for half duplex close. */ + if (sk->sk_shutdown == SHUTDOWN_MASK || + sk->sk_state == TCP_CLOSE) + sk_wake_async(sk, 1, POLL_HUP); + else + sk_wake_async(sk, 1, POLL_IN); + } +} + +static int sdp_post_recv(struct sdp_sock *ssk) +{ + struct sdp_buf *rx_req; + int i, rc, frags; + u64 addr; + struct ib_device *dev; + struct ib_recv_wr rx_wr = { 0 }; + struct ib_sge ibsge[SDP_MAX_RECV_SKB_FRAGS + 1]; + struct ib_sge *sge = ibsge; + struct ib_recv_wr *bad_wr; + struct sk_buff *skb; + struct page *page; + skb_frag_t *frag; + struct sdp_bsdh *h; + int id = ring_head(ssk->rx_ring); + gfp_t gfp_page; + int ret = 0; + + /* Now, allocate and repost recv */ + /* TODO: allocate from cache */ + + if (unlikely(ssk->isk.sk.sk_allocation)) { + skb = sdp_stream_alloc_skb(&ssk->isk.sk, SDP_HEAD_SIZE, + ssk->isk.sk.sk_allocation); + gfp_page = ssk->isk.sk.sk_allocation | __GFP_HIGHMEM; + } else { + skb = sdp_stream_alloc_skb(&ssk->isk.sk, SDP_HEAD_SIZE, + GFP_KERNEL); + gfp_page = GFP_HIGHUSER; + } + + sdp_prf(&ssk->isk.sk, skb, "Posting skb"); + /* FIXME */ + BUG_ON(!skb); + h = (struct sdp_bsdh *)skb->head; + for (i = 0; i < ssk->recv_frags; ++i) { + page = alloc_pages(gfp_page, 0); + BUG_ON(!page); + frag = &skb_shinfo(skb)->frags[i]; + frag->page = page; + frag->page_offset = 0; + frag->size = min(PAGE_SIZE, SDP_MAX_PAYLOAD); + ++skb_shinfo(skb)->nr_frags; + skb->len += frag->size; + skb->data_len += frag->size; + skb->truesize += frag->size; + } + + rx_req = ssk->rx_ring.buffer + (id & (SDP_RX_SIZE - 1)); + rx_req->skb = skb; + dev = ssk->ib_device; + addr = ib_dma_map_single(dev, h, SDP_HEAD_SIZE, DMA_FROM_DEVICE); + BUG_ON(ib_dma_mapping_error(dev, addr)); + + rx_req->mapping[0] = addr; + + /* TODO: proper error handling */ + sge->addr = (u64)addr; + sge->length = SDP_HEAD_SIZE; + sge->lkey = ssk->mr->lkey; + frags = skb_shinfo(skb)->nr_frags; + for (i = 0; i < frags; ++i) { + ++sge; + addr = ib_dma_map_page(dev, skb_shinfo(skb)->frags[i].page, + skb_shinfo(skb)->frags[i].page_offset, + skb_shinfo(skb)->frags[i].size, + DMA_FROM_DEVICE); + BUG_ON(ib_dma_mapping_error(dev, addr)); + rx_req->mapping[i + 1] = addr; + sge->addr = addr; + sge->length = skb_shinfo(skb)->frags[i].size; + sge->lkey = ssk->mr->lkey; + } + + rx_wr.next = NULL; + rx_wr.wr_id = id | SDP_OP_RECV; + rx_wr.sg_list = ibsge; + rx_wr.num_sge = frags + 1; + rc = ib_post_recv(ssk->qp, &rx_wr, &bad_wr); + atomic_inc(&ssk->rx_ring.head); + if (unlikely(rc)) { + sdp_warn(&ssk->isk.sk, "ib_post_recv failed. status %d\n", rc); + + lock_sock(&ssk->isk.sk); + sdp_reset(&ssk->isk.sk); + release_sock(&ssk->isk.sk); + + ret = -1; + } + + SDPSTATS_COUNTER_INC(post_recv); + atomic_add(ssk->recv_frags, &sdp_current_mem_usage); + + return ret; +} + +static inline int sdp_post_recvs_needed(struct sdp_sock *ssk) +{ + struct sock *sk = &ssk->isk.sk; + int scale = ssk->rcvbuf_scale; + int buffer_size = SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE; + unsigned long max_bytes; + + if (top_mem_usage && (top_mem_usage * 0x100000) < + atomic_read(&sdp_current_mem_usage) * PAGE_SIZE) { + scale = 1; + } + + max_bytes = sk->sk_rcvbuf * scale; + + if (unlikely(ring_posted(ssk->rx_ring) >= SDP_RX_SIZE)) + return 0; + + if (likely(ring_posted(ssk->rx_ring) > SDP_MIN_TX_CREDITS)) { + unsigned long bytes_in_process = + (ring_posted(ssk->rx_ring) - SDP_MIN_TX_CREDITS) * + buffer_size; + bytes_in_process += rcv_nxt(ssk) - ssk->copied_seq; + + if (bytes_in_process >= max_bytes) { + sdp_prf(sk, NULL, + "bytes_in_process:%ld > max_bytes:%ld", + bytes_in_process, max_bytes); + return 0; + } + } + + return 1; +} + +static inline void sdp_post_recvs(struct sdp_sock *ssk) +{ +again: + while (sdp_post_recvs_needed(ssk)) { + if (sdp_post_recv(ssk)) + goto out; + } + + sk_stream_mem_reclaim(&ssk->isk.sk); + + if (sdp_post_recvs_needed(ssk)) + goto again; +out: + sk_stream_mem_reclaim(&ssk->isk.sk); +} + +static inline struct sk_buff *sdp_sock_queue_rcv_skb(struct sock *sk, + struct sk_buff *skb) +{ + int skb_len; + struct sdp_sock *ssk = sdp_sk(sk); + + /* not needed since sk_rmem_alloc is not currently used + * TODO - remove this? + skb_set_owner_r(skb, sk); */ + + skb_len = skb->len; + + TCP_SKB_CB(skb)->seq = rcv_nxt(ssk); + atomic_add(skb_len, &ssk->rcv_nxt); + + skb_queue_tail(&sk->sk_receive_queue, skb); + + if (!sock_flag(sk, SOCK_DEAD)) + sk->sk_data_ready(sk, skb_len); + return skb; +} + +int sdp_init_buffers(struct sdp_sock *ssk, u32 new_size) +{ + ssk->recv_frags = PAGE_ALIGN(new_size - SDP_HEAD_SIZE) / PAGE_SIZE; + if (ssk->recv_frags > SDP_MAX_RECV_SKB_FRAGS) + ssk->recv_frags = SDP_MAX_RECV_SKB_FRAGS; + ssk->rcvbuf_scale = rcvbuf_scale; + + sdp_post_recvs(ssk); + + return 0; +} + +int sdp_resize_buffers(struct sdp_sock *ssk, u32 new_size) +{ + u32 curr_size = SDP_HEAD_SIZE + ssk->recv_frags * PAGE_SIZE; +#if defined(__ia64__) + /* for huge PAGE_SIZE systems, aka IA64, limit buffers size + [re-]negotiation to a known+working size that will not + trigger a HW error/rc to be interpreted as a IB_WC_LOC_LEN_ERR */ + u32 max_size = (SDP_HEAD_SIZE + SDP_MAX_RECV_SKB_FRAGS * PAGE_SIZE) <= + 32784 ? + (SDP_HEAD_SIZE + SDP_MAX_RECV_SKB_FRAGS * PAGE_SIZE) : 32784; +#else + u32 max_size = SDP_HEAD_SIZE + SDP_MAX_RECV_SKB_FRAGS * PAGE_SIZE; +#endif + + if (new_size > curr_size && new_size <= max_size && + sdp_get_large_socket(ssk)) { + ssk->rcvbuf_scale = rcvbuf_scale; + ssk->recv_frags = PAGE_ALIGN(new_size - SDP_HEAD_SIZE) / + PAGE_SIZE; + if (ssk->recv_frags > SDP_MAX_RECV_SKB_FRAGS) + ssk->recv_frags = SDP_MAX_RECV_SKB_FRAGS; + return 0; + } else + return -1; +} + +static void sdp_handle_resize_request(struct sdp_sock *ssk, + struct sdp_chrecvbuf *buf) +{ + if (sdp_resize_buffers(ssk, ntohl(buf->size)) == 0) + ssk->recv_request_head = ring_head(ssk->rx_ring) + 1; + else + ssk->recv_request_head = ring_tail(ssk->rx_ring); + ssk->recv_request = 1; +} + +static void sdp_handle_resize_ack(struct sdp_sock *ssk, + struct sdp_chrecvbuf *buf) +{ + u32 new_size = ntohl(buf->size); + + if (new_size > ssk->xmit_size_goal) { + ssk->sent_request = -1; + ssk->xmit_size_goal = new_size; + ssk->send_frags = + PAGE_ALIGN(ssk->xmit_size_goal) / PAGE_SIZE + 1; + } else + ssk->sent_request = 0; +} + +static inline int credit_update_needed(struct sdp_sock *ssk) +{ + int c; + + c = remote_credits(ssk); + if (likely(c > SDP_MIN_TX_CREDITS)) + c += c/2; + + return unlikely(c < ring_posted(ssk->rx_ring)) && + likely(tx_credits(ssk) > 1) && + likely(sdp_tx_ring_slots_left(&ssk->tx_ring)); +} + + +static struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id) +{ + struct sdp_buf *rx_req; + struct ib_device *dev; + struct sk_buff *skb; + int i, frags; + + if (unlikely(id != ring_tail(ssk->rx_ring))) { + printk(KERN_WARNING "Bogus recv completion id %d tail %d\n", + id, ring_tail(ssk->rx_ring)); + return NULL; + } + + dev = ssk->ib_device; + rx_req = &ssk->rx_ring.buffer[id & (SDP_RX_SIZE - 1)]; + skb = rx_req->skb; + ib_dma_unmap_single(dev, rx_req->mapping[0], SDP_HEAD_SIZE, + DMA_FROM_DEVICE); + frags = skb_shinfo(skb)->nr_frags; + for (i = 0; i < frags; ++i) + ib_dma_unmap_page(dev, rx_req->mapping[i + 1], + skb_shinfo(skb)->frags[i].size, + DMA_FROM_DEVICE); + atomic_inc(&ssk->rx_ring.tail); + atomic_dec(&ssk->remote_credits); + return skb; +} + +/* socket lock should be taken before calling this */ +static int sdp_process_rx_ctl_skb(struct sdp_sock *ssk, struct sk_buff *skb) +{ + struct sdp_bsdh *h = (struct sdp_bsdh *)skb_transport_header(skb); + struct sock *sk = &ssk->isk.sk; + + switch (h->mid) { + case SDP_MID_DATA: + WARN_ON(!(sk->sk_shutdown & RCV_SHUTDOWN)); + + sdp_warn(sk, "DATA after socket rcv was shutdown\n"); + + /* got data in RCV_SHUTDOWN */ + if (sk->sk_state == TCP_FIN_WAIT1) { + sdp_warn(sk, "RX data when state = FIN_WAIT1\n"); + /* go into abortive close */ + sdp_exch_state(sk, TCPF_FIN_WAIT1, + TCP_TIME_WAIT); + + sk->sk_prot->disconnect(sk, 0); + } + __kfree_skb(skb); + + break; + case SDP_MID_DISCONN: + sdp_dbg_data(sk, "Handling RX disconnect\n"); + sdp_prf(sk, NULL, "Handling RX disconnect"); + sdp_fin(sk); + sdp_prf(sk, NULL, "Queueing fin skb - release recvmsg"); + /* Enqueue fin skb to release sleeping recvmsg */ + sdp_sock_queue_rcv_skb(sk, skb); + break; + case SDP_MID_CHRCVBUF: + sdp_dbg_data(sk, "Handling RX CHRCVBUF\n"); + sdp_handle_resize_request(ssk, (struct sdp_chrecvbuf *)h); + __kfree_skb(skb); + break; + case SDP_MID_CHRCVBUF_ACK: + sdp_dbg_data(sk, "Handling RX CHRCVBUF_ACK\n"); + sdp_handle_resize_ack(ssk, (struct sdp_chrecvbuf *)h); + __kfree_skb(skb); + break; + default: + /* TODO: Handle other messages */ + sdp_warn(sk, "SDP: FIXME MID %d\n", h->mid); + __kfree_skb(skb); + } + + return 0; +} + +static int sdp_process_rx_skb(struct sdp_sock *ssk, struct sk_buff *skb) +{ + struct sock *sk = &ssk->isk.sk; + int frags; + struct sdp_bsdh *h; + int pagesz, i; + unsigned long mseq_ack; + int credits_before; + + h = (struct sdp_bsdh *)skb_transport_header(skb); + + SDPSTATS_HIST_LINEAR(credits_before_update, tx_credits(ssk)); + + mseq_ack = ntohl(h->mseq_ack); + credits_before = tx_credits(ssk); + atomic_set(&ssk->tx_ring.credits, mseq_ack - ring_head(ssk->tx_ring) + + 1 + ntohs(h->bufs)); + if (mseq_ack >= ssk->nagle_last_unacked) + ssk->nagle_last_unacked = 0; + + sdp_prf1(&ssk->isk.sk, skb, "RX %s +%d c:%d->%d mseq:%d ack:%d", + mid2str(h->mid), ntohs(h->bufs), credits_before, + tx_credits(ssk), ntohl(h->mseq), ntohl(h->mseq_ack)); + + frags = skb_shinfo(skb)->nr_frags; + pagesz = PAGE_ALIGN(skb->data_len); + skb_shinfo(skb)->nr_frags = pagesz / PAGE_SIZE; + + for (i = skb_shinfo(skb)->nr_frags; i < frags; ++i) { + put_page(skb_shinfo(skb)->frags[i].page); + skb->truesize -= PAGE_SIZE; + } + +/* if (unlikely(h->flags & SDP_OOB_PEND)) + sk_send_sigurg(sk);*/ + + skb_pull(skb, sizeof(struct sdp_bsdh)); + + if (h->mid != SDP_MID_DATA || + unlikely(sk->sk_shutdown & RCV_SHUTDOWN)) { + sdp_prf(sk, NULL, "Control skb - queing to control queue"); + skb_queue_tail(&ssk->rx_ctl_q, skb); + return 0; + } + + if (unlikely(skb->len <= 0)) { + __kfree_skb(skb); + return 0; + } + + sdp_prf(sk, NULL, "queueing a %s skb", + (h->mid == SDP_MID_DATA ? "data" : "disconnect")); + skb = sdp_sock_queue_rcv_skb(sk, skb); + +/* if (unlikely(h->flags & SDP_OOB_PRES)) + sdp_urg(ssk, skb);*/ + + return 0; +} + +/* called only from irq */ +static struct sk_buff *sdp_process_rx_wc(struct sdp_sock *ssk, + struct ib_wc *wc) +{ + struct sk_buff *skb; + struct sdp_bsdh *h; + struct sock *sk = &ssk->isk.sk; + int mseq; + + skb = sdp_recv_completion(ssk, wc->wr_id); + if (unlikely(!skb)) + return NULL; + + atomic_sub(skb_shinfo(skb)->nr_frags, &sdp_current_mem_usage); + + if (unlikely(wc->status)) { + if (wc->status != IB_WC_WR_FLUSH_ERR) { + sdp_warn(sk, "Recv completion with error. Status %d\n", + wc->status); + sdp_reset(sk); + } + __kfree_skb(skb); + return NULL; + } + + sdp_dbg_data(sk, "Recv completion. ID %d Length %d\n", + (int)wc->wr_id, wc->byte_len); + if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { + sdp_warn(sk, "SDP BUG! byte_len %d < %zd\n", + wc->byte_len, sizeof(struct sdp_bsdh)); + __kfree_skb(skb); + return NULL; + } + skb->len = wc->byte_len; + if (likely(wc->byte_len > SDP_HEAD_SIZE)) + skb->data_len = wc->byte_len - SDP_HEAD_SIZE; + else + skb->data_len = 0; + skb->data = skb->head; +#ifdef NET_SKBUFF_DATA_USES_OFFSET + skb->tail = skb_headlen(skb); +#else + skb->tail = skb->head + skb_headlen(skb); +#endif + h = (struct sdp_bsdh *)skb->data; + SDP_DUMP_PACKET(&ssk->isk.sk, "RX", skb, h); + skb_reset_transport_header(skb); + + ssk->rx_packets++; + ssk->rx_bytes += skb->len; + + mseq = ntohl(h->mseq); + atomic_set(&ssk->mseq_ack, mseq); + if (mseq != (int)wc->wr_id) + sdp_warn(sk, "SDP BUG! mseq %d != wrid %d\n", + mseq, (int)wc->wr_id); + + return skb; +} + +/* like sk_stream_write_space - execpt measures remote credits */ +static void sdp_bzcopy_write_space(struct sdp_sock *ssk) +{ + struct sock *sk = &ssk->isk.sk; + struct socket *sock = sk->sk_socket; + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) { + sdp_prf1(&ssk->isk.sk, NULL, "credits: %d, min_bufs: %d. " + "tx_head: %d, tx_tail: %d", + tx_credits(ssk), ssk->min_bufs, + ring_head(ssk->tx_ring), ring_tail(ssk->tx_ring)); + } + + if (tx_credits(ssk) >= ssk->min_bufs && sock != NULL) { + clear_bit(SOCK_NOSPACE, &sock->flags); + sdp_prf1(sk, NULL, "Waking up sleepers"); + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + wake_up_interruptible(sk->sk_sleep); + if (sock->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN)) + sock_wake_async(sock, 2, POLL_OUT); + } +} + +/* only from interrupt. */ +static int sdp_poll_rx_cq(struct sdp_sock *ssk) +{ + struct ib_cq *cq = ssk->rx_ring.cq; + struct ib_wc ibwc[SDP_NUM_WC]; + int n, i; + int wc_processed = 0; + struct sk_buff *skb; + + do { + n = ib_poll_cq(cq, SDP_NUM_WC, ibwc); + for (i = 0; i < n; ++i) { + struct ib_wc *wc = &ibwc[i]; + + BUG_ON(!(wc->wr_id & SDP_OP_RECV)); + skb = sdp_process_rx_wc(ssk, wc); + if (!skb) + continue; + + sdp_process_rx_skb(ssk, skb); + wc_processed++; + } + } while (n == SDP_NUM_WC); + + if (wc_processed) + sdp_bzcopy_write_space(ssk); + + return wc_processed; +} + +void sdp_rx_comp_work(struct work_struct *work) +{ + struct sdp_sock *ssk = container_of(work, struct sdp_sock, + rx_comp_work); + struct sock *sk = &ssk->isk.sk; + + sdp_prf(sk, NULL, "%s", __func__); + + if (unlikely(!ssk->qp)) { + sdp_prf(sk, NULL, "qp was destroyed"); + return; + } + if (unlikely(!ssk->rx_ring.cq)) { + sdp_prf(sk, NULL, "rx_ring.cq is NULL"); + return; + } + + if (unlikely(!ssk->poll_cq)) { + struct rdma_cm_id *id = ssk->id; + if (id && id->qp) + rdma_notify(id, RDMA_CM_EVENT_ESTABLISHED); + return; + } + + lock_sock(sk); + + sdp_do_posts(ssk); + + release_sock(sk); +} + +void sdp_do_posts(struct sdp_sock *ssk) +{ + struct sock *sk = &ssk->isk.sk; + int xmit_poll_force; + struct sk_buff *skb; + + while ((skb = skb_dequeue(&ssk->rx_ctl_q))) + sdp_process_rx_ctl_skb(ssk, skb); + + if (sk->sk_state == TCP_TIME_WAIT) + return; + + if (!ssk->rx_ring.cq || !ssk->tx_ring.cq) + return; + + sdp_post_recvs(ssk); + + if (ring_posted(ssk->tx_ring)) + sdp_xmit_poll(ssk, 1); + + sdp_post_sends(ssk, 0); + + sk_stream_mem_reclaim(sk); + + xmit_poll_force = sk->sk_write_pending && + (tx_credits(ssk) > SDP_MIN_TX_CREDITS); + + if (credit_update_needed(ssk) || xmit_poll_force) { + /* if has pending tx because run out of tx_credits - xmit it */ + sdp_prf(sk, NULL, "Processing to free pending sends"); + sdp_xmit_poll(ssk, xmit_poll_force); + sdp_prf(sk, NULL, "Sending credit update"); + sdp_post_sends(ssk, 0); + } + +} + +static void sdp_rx_irq(struct ib_cq *cq, void *cq_context) +{ + struct sock *sk = cq_context; + struct sdp_sock *ssk = sdp_sk(sk); + unsigned long flags; + int wc_processed = 0; + int credits_before; + + sdp_dbg_data(&ssk->isk.sk, "rx irq called\n"); + + if (cq != ssk->rx_ring.cq) { + sdp_warn(sk, "cq = %p, ssk->cq = %p\n", cq, ssk->rx_ring.cq); + return; + } + + SDPSTATS_COUNTER_INC(rx_int_count); + + sdp_prf(sk, NULL, "rx irq"); + + rx_ring_lock(ssk, flags); + + credits_before = tx_credits(ssk); + + if (!ssk->rx_ring.cq) { + sdp_warn(&ssk->isk.sk, "WARNING: rx irq after cq destroyed\n"); + + goto out; + } + + wc_processed = sdp_poll_rx_cq(ssk); + sdp_prf(&ssk->isk.sk, NULL, "processed %d", wc_processed); + + if (wc_processed) { + sdp_prf(&ssk->isk.sk, NULL, "credits: %d -> %d", + credits_before, tx_credits(ssk)); + + if (posts_handler(ssk) || (sk->sk_socket && + test_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags))) { + + sdp_prf(&ssk->isk.sk, NULL, + "Somebody is doing the post work for me. %d", + posts_handler(ssk)); + + } else { + sdp_prf(&ssk->isk.sk, NULL, "Queuing work. ctl_q: %d", + !skb_queue_empty(&ssk->rx_ctl_q)); + queue_work(rx_comp_wq, &ssk->rx_comp_work); + } + } + sdp_arm_rx_cq(sk); + +out: + rx_ring_unlock(ssk, flags); +} + +static void sdp_rx_ring_purge(struct sdp_sock *ssk) +{ + while (ring_posted(ssk->rx_ring) > 0) { + struct sk_buff *skb; + skb = sdp_recv_completion(ssk, ring_tail(ssk->rx_ring)); + if (!skb) + break; + atomic_sub(skb_shinfo(skb)->nr_frags, &sdp_current_mem_usage); + __kfree_skb(skb); + } +} + +void sdp_rx_ring_init(struct sdp_sock *ssk) +{ + ssk->rx_ring.buffer = NULL; + spin_lock_init(&ssk->rx_ring.lock); +} + +static void sdp_rx_cq_event_handler(struct ib_event *event, void *data) +{ +} + +int sdp_rx_ring_create(struct sdp_sock *ssk, struct ib_device *device) +{ + struct ib_cq *rx_cq; + int rc = 0; + unsigned long flags; + + rx_ring_lock(ssk, flags); + + atomic_set(&ssk->rx_ring.head, 1); + atomic_set(&ssk->rx_ring.tail, 1); + + ssk->rx_ring.buffer = kmalloc( + sizeof *ssk->rx_ring.buffer * SDP_RX_SIZE, GFP_KERNEL); + if (!ssk->rx_ring.buffer) { + rc = -ENOMEM; + sdp_warn(&ssk->isk.sk, + "Unable to allocate RX Ring size %zd.\n", + sizeof(*ssk->rx_ring.buffer) * SDP_RX_SIZE); + + goto out; + } + + /* TODO: use vector=IB_CQ_VECTOR_LEAST_ATTACHED when implemented + * in ofed-1.5 */ + rx_cq = ib_create_cq(device, sdp_rx_irq, sdp_rx_cq_event_handler, + &ssk->isk.sk, SDP_RX_SIZE, 0); + + if (IS_ERR(rx_cq)) { + rc = PTR_ERR(rx_cq); + sdp_warn(&ssk->isk.sk, "Unable to allocate RX CQ: %d.\n", rc); + goto err_cq; + } + + sdp_sk(&ssk->isk.sk)->rx_ring.cq = rx_cq; + + INIT_WORK(&ssk->rx_comp_work, sdp_rx_comp_work); + + sdp_arm_rx_cq(&ssk->isk.sk); + + goto out; + +err_cq: + kfree(ssk->rx_ring.buffer); + ssk->rx_ring.buffer = NULL; +out: + rx_ring_unlock(ssk, flags); + return rc; +} + +void sdp_rx_ring_destroy(struct sdp_sock *ssk) +{ + if (ssk->rx_ring.buffer) { + sdp_rx_ring_purge(ssk); + + kfree(ssk->rx_ring.buffer); + ssk->rx_ring.buffer = NULL; + } + + if (ssk->rx_ring.cq) { + ib_destroy_cq(ssk->rx_ring.cq); + ssk->rx_ring.cq = NULL; + } + + WARN_ON(ring_head(ssk->rx_ring) != ring_tail(ssk->rx_ring)); +} diff --git a/drivers/infiniband/ulp/sdp/sdp_tx.c b/drivers/infiniband/ulp/sdp/sdp_tx.c new file mode 100644 index 0000000..d53e838 --- /dev/null +++ b/drivers/infiniband/ulp/sdp/sdp_tx.c @@ -0,0 +1,442 @@ +/* + * Copyright (c) 2009 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +#include +#include +#include +#include +#include "sdp.h" + +#define sdp_cnt(var) do { (var)++; } while (0) + +SDP_MODPARAM_SINT(sdp_keepalive_probes_sent, 0, + "Total number of keepalive probes sent."); + +static int sdp_process_tx_cq(struct sdp_sock *ssk); + +int sdp_xmit_poll(struct sdp_sock *ssk, int force) +{ + int wc_processed = 0; + + sdp_prf(&ssk->isk.sk, NULL, "called from %s:%d", func, line); + + /* If we don't have a pending timer, set one up to catch our recent + post in case the interface becomes idle */ + if (!timer_pending(&ssk->tx_ring.timer)) + mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); + + /* Poll the CQ every SDP_TX_POLL_MODER packets */ + if (force || (++ssk->tx_ring.poll_cnt & (SDP_TX_POLL_MODER - 1)) == 0) + wc_processed = sdp_process_tx_cq(ssk); + + return wc_processed; +} + +static unsigned long last_send; + +void sdp_post_send(struct sdp_sock *ssk, struct sk_buff *skb, u8 mid) +{ + struct sdp_buf *tx_req; + struct sdp_bsdh *h = (struct sdp_bsdh *)skb_push(skb, sizeof *h); + unsigned long mseq = ring_head(ssk->tx_ring); + int i, rc, frags; + u64 addr; + struct ib_device *dev; + struct ib_send_wr *bad_wr; + int delta; + + struct ib_sge ibsge[SDP_MAX_SEND_SKB_FRAGS + 1]; + struct ib_sge *sge = ibsge; + struct ib_send_wr tx_wr = { 0 }; + + SDPSTATS_COUNTER_MID_INC(post_send, mid); + SDPSTATS_HIST(send_size, skb->len); + + ssk->tx_packets++; + ssk->tx_bytes += skb->len; + + h->mid = mid; + if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) + h->flags = SDP_OOB_PRES | SDP_OOB_PEND; + else + h->flags = 0; + + h->bufs = htons(ring_posted(ssk->rx_ring)); + h->len = htonl(skb->len); + h->mseq = htonl(mseq); + h->mseq_ack = htonl(mseq_ack(ssk)); + + sdp_prf1(&ssk->isk.sk, skb, "TX: %s bufs: %d mseq:%ld ack:%d", + mid2str(mid), ring_posted(ssk->rx_ring), mseq, + ntohl(h->mseq_ack)); + + SDP_DUMP_PACKET(&ssk->isk.sk, "TX", skb, h); + + tx_req = &ssk->tx_ring.buffer[mseq & (SDP_TX_SIZE - 1)]; + tx_req->skb = skb; + dev = ssk->ib_device; + addr = ib_dma_map_single(dev, skb->data, skb->len - skb->data_len, + DMA_TO_DEVICE); + tx_req->mapping[0] = addr; + + /* TODO: proper error handling */ + BUG_ON(ib_dma_mapping_error(dev, addr)); + + sge->addr = addr; + sge->length = skb->len - skb->data_len; + sge->lkey = ssk->mr->lkey; + frags = skb_shinfo(skb)->nr_frags; + for (i = 0; i < frags; ++i) { + ++sge; + addr = ib_dma_map_page(dev, skb_shinfo(skb)->frags[i].page, + skb_shinfo(skb)->frags[i].page_offset, + skb_shinfo(skb)->frags[i].size, + DMA_TO_DEVICE); + BUG_ON(ib_dma_mapping_error(dev, addr)); + tx_req->mapping[i + 1] = addr; + sge->addr = addr; + sge->length = skb_shinfo(skb)->frags[i].size; + sge->lkey = ssk->mr->lkey; + } + + tx_wr.next = NULL; + tx_wr.wr_id = ring_head(ssk->tx_ring) | SDP_OP_SEND; + tx_wr.sg_list = ibsge; + tx_wr.num_sge = frags + 1; + tx_wr.opcode = IB_WR_SEND; + tx_wr.send_flags = IB_SEND_SIGNALED; + if (unlikely(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) + tx_wr.send_flags |= IB_SEND_SOLICITED; + + delta = jiffies - last_send; + if (likely(last_send)) + SDPSTATS_HIST(send_interval, delta); + last_send = jiffies; + + rc = ib_post_send(ssk->qp, &tx_wr, &bad_wr); + atomic_inc(&ssk->tx_ring.head); + atomic_dec(&ssk->tx_ring.credits); + atomic_set(&ssk->remote_credits, ring_posted(ssk->rx_ring)); + if (unlikely(rc)) { + sdp_dbg(&ssk->isk.sk, + "ib_post_send failed with status %d.\n", rc); + sdp_set_error(&ssk->isk.sk, -ECONNRESET); + wake_up(&ssk->wq); + } +} + +static struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq) +{ + struct ib_device *dev; + struct sdp_buf *tx_req; + struct sk_buff *skb = NULL; + struct bzcopy_state *bz; + int i, frags; + struct sdp_tx_ring *tx_ring = &ssk->tx_ring; + if (unlikely(mseq != ring_tail(*tx_ring))) { + printk(KERN_WARNING "Bogus send completion id %d tail %d\n", + mseq, ring_tail(*tx_ring)); + goto out; + } + + dev = ssk->ib_device; + tx_req = &tx_ring->buffer[mseq & (SDP_TX_SIZE - 1)]; + skb = tx_req->skb; + ib_dma_unmap_single(dev, tx_req->mapping[0], skb->len - skb->data_len, + DMA_TO_DEVICE); + frags = skb_shinfo(skb)->nr_frags; + for (i = 0; i < frags; ++i) { + ib_dma_unmap_page(dev, tx_req->mapping[i + 1], + skb_shinfo(skb)->frags[i].size, + DMA_TO_DEVICE); + } + + tx_ring->una_seq += TCP_SKB_CB(skb)->end_seq; + + /* TODO: AIO and real zcopy code; add their context support here */ + bz = BZCOPY_STATE(skb); + if (bz) + bz->busy--; + + atomic_inc(&tx_ring->tail); + +out: + return skb; +} + +static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) +{ + struct sk_buff *skb = NULL; + + skb = sdp_send_completion(ssk, wc->wr_id); + if (unlikely(!skb)) + return -1; + + if (unlikely(wc->status)) { + if (wc->status != IB_WC_WR_FLUSH_ERR) { + struct sock *sk = &ssk->isk.sk; + sdp_prf(sk, skb, "Send completion with error. " + "Status %d", wc->status); + sdp_warn(sk, "Send completion with error. " + "Status %d\n", wc->status); + sdp_set_error(sk, -ECONNRESET); + wake_up(&ssk->wq); + + queue_work(sdp_wq, &ssk->destroy_work); + } + } + +#ifdef SDP_PROFILING +{ + struct sdp_bsdh *h = (struct sdp_bsdh *)skb->data; + sdp_prf1(&ssk->isk.sk, skb, "tx completion. mseq:%d", ntohl(h->mseq)); +} +#endif + sk_wmem_free_skb(&ssk->isk.sk, skb); + + return 0; +} + +static inline void sdp_process_tx_wc(struct sdp_sock *ssk, struct ib_wc *wc) +{ + if (likely(wc->wr_id & SDP_OP_SEND)) { + sdp_handle_send_comp(ssk, wc); + return; + } + + /* Keepalive probe sent cleanup */ + sdp_cnt(sdp_keepalive_probes_sent); + + if (likely(!wc->status)) + return; + + sdp_dbg(&ssk->isk.sk, " %s consumes KEEPALIVE status %d\n", + __func__, wc->status); + + if (wc->status == IB_WC_WR_FLUSH_ERR) + return; + + sdp_set_error(&ssk->isk.sk, -ECONNRESET); + wake_up(&ssk->wq); +} + +static int sdp_process_tx_cq(struct sdp_sock *ssk) +{ + struct ib_wc ibwc[SDP_NUM_WC]; + int n, i; + int wc_processed = 0; + + if (!ssk->tx_ring.cq) { + sdp_warn(&ssk->isk.sk, "tx irq on destroyed tx_cq\n"); + return 0; + } + + do { + n = ib_poll_cq(ssk->tx_ring.cq, SDP_NUM_WC, ibwc); + for (i = 0; i < n; ++i) { + sdp_process_tx_wc(ssk, ibwc + i); + wc_processed++; + } + } while (n == SDP_NUM_WC); + + sdp_dbg_data(&ssk->isk.sk, "processed %d wc's\n", wc_processed); + + if (wc_processed) { + struct sock *sk = &ssk->isk.sk; + sdp_post_sends(ssk, 0); + + if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) + sk_stream_write_space(&ssk->isk.sk); + } + + return wc_processed; +} + +static void sdp_poll_tx_timeout(unsigned long data) +{ + struct sdp_sock *ssk = (struct sdp_sock *)data; + struct sock *sk = &ssk->isk.sk; + u32 inflight, wc_processed; + + sdp_dbg_data(&ssk->isk.sk, "Polling tx cq. inflight=%d\n", + (u32) ring_posted(ssk->tx_ring)); + + sdp_prf(&ssk->isk.sk, NULL, "%s. inflight=%d", __func__, + (u32) ring_posted(ssk->tx_ring)); + + /* Only process if the socket is not in use */ + bh_lock_sock(sk); + if (sock_owned_by_user(sk)) { + mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); + sdp_dbg_data(&ssk->isk.sk, "socket is busy - trying later\n"); + SDPSTATS_COUNTER_INC(tx_poll_busy); + goto out; + } + + if (unlikely(sk->sk_state == TCP_CLOSE)) + goto out; + + wc_processed = sdp_process_tx_cq(ssk); + if (!wc_processed) + SDPSTATS_COUNTER_INC(tx_poll_miss); + else + SDPSTATS_COUNTER_INC(tx_poll_hit); + + inflight = (u32) ring_posted(ssk->tx_ring); + + /* If there are still packets in flight and the timer has not already + * been scheduled by the Tx routine then schedule it here to guarantee + * completion processing of these packets */ + if (inflight) { /* TODO: make sure socket is not closed */ + sdp_dbg_data(sk, "arming timer for more polling\n"); + mod_timer(&ssk->tx_ring.timer, jiffies + SDP_TX_POLL_TIMEOUT); + } + +out: + bh_unlock_sock(sk); +} + +static void sdp_tx_irq(struct ib_cq *cq, void *cq_context) +{ + struct sock *sk = cq_context; + struct sdp_sock *ssk = sdp_sk(sk); + + sdp_prf1(sk, NULL, "Got tx comp interrupt"); + + mod_timer(&ssk->tx_ring.timer, jiffies); +} + +void sdp_tx_ring_purge(struct sdp_sock *ssk) +{ + while (ring_posted(ssk->tx_ring)) { + struct sk_buff *skb; + skb = sdp_send_completion(ssk, ring_tail(ssk->tx_ring)); + if (!skb) + break; + __kfree_skb(skb); + } +} + +void sdp_post_keepalive(struct sdp_sock *ssk) +{ + int rc; + struct ib_send_wr wr, *bad_wr; + + sdp_dbg(&ssk->isk.sk, "%s\n", __func__); + + memset(&wr, 0, sizeof(wr)); + + wr.next = NULL; + wr.wr_id = 0; + wr.sg_list = NULL; + wr.num_sge = 0; + wr.opcode = IB_WR_RDMA_WRITE; + + rc = ib_post_send(ssk->qp, &wr, &bad_wr); + if (rc) { + sdp_dbg(&ssk->isk.sk, + "ib_post_keepalive failed with status %d.\n", rc); + sdp_set_error(&ssk->isk.sk, -ECONNRESET); + wake_up(&ssk->wq); + } + + sdp_cnt(sdp_keepalive_probes_sent); +} + +static void sdp_tx_cq_event_handler(struct ib_event *event, void *data) +{ +} + +int sdp_tx_ring_create(struct sdp_sock *ssk, struct ib_device *device) +{ + struct ib_cq *tx_cq; + int rc = 0; + + atomic_set(&ssk->tx_ring.head, 1); + atomic_set(&ssk->tx_ring.tail, 1); + + ssk->tx_ring.buffer = kmalloc( + sizeof *ssk->tx_ring.buffer * SDP_TX_SIZE, GFP_KERNEL); + if (!ssk->tx_ring.buffer) { + rc = -ENOMEM; + sdp_warn(&ssk->isk.sk, "Can't allocate TX Ring size %zd.\n", + sizeof(*ssk->tx_ring.buffer) * SDP_TX_SIZE); + + goto out; + } + + tx_cq = ib_create_cq(device, sdp_tx_irq, sdp_tx_cq_event_handler, + &ssk->isk.sk, SDP_TX_SIZE, 0); + + if (IS_ERR(tx_cq)) { + rc = PTR_ERR(tx_cq); + sdp_warn(&ssk->isk.sk, "Unable to allocate TX CQ: %d.\n", rc); + goto err_cq; + } + + sdp_sk(&ssk->isk.sk)->tx_ring.cq = tx_cq; + + init_timer(&ssk->tx_ring.timer); + ssk->tx_ring.timer.function = sdp_poll_tx_timeout; + ssk->tx_ring.timer.data = (unsigned long) ssk; + ssk->tx_ring.poll_cnt = 0; + + init_timer(&ssk->nagle_timer); + ssk->nagle_timer.function = sdp_nagle_timeout; + ssk->nagle_timer.data = (unsigned long) ssk; + + return 0; + +err_cq: + kfree(ssk->tx_ring.buffer); + ssk->tx_ring.buffer = NULL; +out: + return rc; +} + +void sdp_tx_ring_destroy(struct sdp_sock *ssk) +{ + del_timer(&ssk->nagle_timer); + + if (ssk->tx_ring.buffer) { + sdp_tx_ring_purge(ssk); + + kfree(ssk->tx_ring.buffer); + ssk->tx_ring.buffer = NULL; + } + + if (ssk->tx_ring.cq) { + ib_destroy_cq(ssk->tx_ring.cq); + ssk->tx_ring.cq = NULL; + } + + WARN_ON(ring_head(ssk->tx_ring) != ring_tail(ssk->tx_ring)); +} From bart.vanassche at gmail.com Sun Jun 21 23:19:06 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Mon, 22 Jun 2009 08:19:06 +0200 Subject: [ofa-general] Infiniband Driver support for Mellanox HCA in Centos 5.3 In-Reply-To: References: Message-ID: On Mon, Jun 22, 2009 at 1:43 AM, Ashwath Narasimhan wrote: > 4. One more Question, As I dug into to the release notes, I find that Centos 5.3 is not supported in the distribution of OFED-1.4. Is this correct? Does this mean I have to downgrade to centos 5.2? Please have a look at the OFED 1.4.1 release notes. Bart. From tziporet at dev.mellanox.co.il Mon Jun 22 01:26:36 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Mon, 22 Jun 2009 11:26:36 +0300 Subject: [ofa-general] Infiniband Driver support for Mellanox HCA in Centos 5.3 In-Reply-To: References: Message-ID: <4A3F403C.4000602@mellanox.co.il> Ashwath Narasimhan wrote: > Hi, > I am a linux newbie and I am trying to run ipoib on linux Centos 5.3. > My software driver does not recognize the hardware even though it is > connected. > > 1. I am using the following infiniband HCA > $ my prompt> lspci |grep Mellanox > 01:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex > (Tavor compatibility mode) (rev a0) > > 2. I installed the infiniband utilities using ---> yum groupinstall > 'OpenFabrics Enterprise Distribution' > > 3. I then started the driver in this way --> > > [root at localhost ]# /etc/init.d/openibd start > Loading OpenIB kernel modules: [ OK ] > [root at localhost ]# ibv_devinfo > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > No IB devices found > > As you can see it says No IB devices found. I am able to detect the > hardware using Windows, however I am not able to detect the same using > Linux. Please guide me as to what I can do. Please send output of /var/log/messages Also output of lsmod to see which modules are loaded > > 4. One more Question, As I dug into to the release notes, I find that > Centos 5.3 is not supported in the distribution of OFED-1.4. Is this > correct? Does this mean I have to downgrade to centos 5.2? > > 5. What could be the time frame by which Centos 5.3 is also supported? > Which OFED version are you using? We added RHEL 5.3 support in OFED 1.4.1, and usually when things works on it then it will work on CenOS too. Its just that CentOS 5.3 was not tested but it should work. Tziporet From tziporet at dev.mellanox.co.il Mon Jun 22 01:28:03 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Mon, 22 Jun 2009 11:28:03 +0300 Subject: [ofa-general] Support for reg_phys_mr in mlx4_ib In-Reply-To: References: Message-ID: <4A3F4093.9030105@mellanox.co.il> Adit Ranadive wrote: > Is there any plan to implement the reg_phys_mr verb for the ConnectX > cards in the newer OFED releases? > If not whats the best option to register a list of physical pages as > a memory region? > We don't have such a plan You can use FMRs to register physical pages as a memory regions You can look at SRP or NFS/RDMA code to see how is it done Tziporet From rajouri.jammu at gmail.com Sun Jun 21 21:57:41 2009 From: rajouri.jammu at gmail.com (Rajouri Jammu) Date: Sun, 21 Jun 2009 21:57:41 -0700 Subject: [ofa-general] Compilation errors with OFED 1.4.1/ 1.4 In-Reply-To: <4A3E9957.50104@mellanox.co.il> References: <9759F033B56A60469F217C49AC58E2BF085A541B@MAILUK2.rms.com> <3307cdf90906191417y46a0142bs50f58c36860d33ab@mail.gmail.com> <4A3E9957.50104@mellanox.co.il> Message-ID: <3307cdf90906212157o687a6929w49f9abc8b16fc131@mail.gmail.com> Tziporet, > We (Mellanox) are running regression testing on SLES10 SP2 with OFED 1.4.1 without any problem I don't think it's a runtime issue. The problem: /usr/src/ofa_kernel/include/rdma/ib_verbs.h in OFED 1.4.1 is referring to symbols like "dma_map_single_attrs" etc, but when I looked for those symbols in /usr/src/linux-2.6.16.60-0.21 I did not find any references. Can you suggest a file that I should include that will resolve this issue? Thanks. On Sun, Jun 21, 2009 at 1:34 PM, Tziporet Koren wrote: > Rajouri Jammu wrote: > >> Thanks for the response Evan. >> >> It appears to me that this is a problem in OFED 1.4.1 since the header >> file in /usr/src/ofa_kernel/include/rdma/ib_verbs.h is referring to data >> structures that are not present in the SLES 10 u 2 kernel. >> >> However, someone else indicated it was a problem in our code. >> >> Btw, my code compiled fine with OFED 1.3.1 on the same kernel. >> >> > We (Mellanox) are running regression testing on SLES10 SP2 with OFED 1.4.1 > without any problem > So I guess its something specific in your system, or order of include files > you use > > Tziporet > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tziporet at dev.mellanox.co.il Mon Jun 22 05:58:07 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Mon, 22 Jun 2009 15:58:07 +0300 Subject: [ofa-general] Infiniband Driver support for Mellanox HCA in Centos 5.3 In-Reply-To: References: Message-ID: <4A3F7FDF.9010302@mellanox.co.il> Bart Van Assche wrote: > On Mon, Jun 22, 2009 at 1:43 AM, Ashwath Narasimhan wrote: > >> 4. One more Question, As I dug into to the release notes, I find that Centos 5.3 is not supported in the distribution of OFED-1.4. Is this correct? Does this mean I have to downgrade to centos 5.2? >> > > Please have a look at the OFED 1.4.1 release notes. > > I know the RN since I wrote them I just say it should work, since till today anything that worked with RH worked with CentOS with the same version number Tziporet From fujita.tomonori at lab.ntt.co.jp Sun Jun 21 20:11:12 2009 From: fujita.tomonori at lab.ntt.co.jp (FUJITA Tomonori) Date: Mon, 22 Jun 2009 12:11:12 +0900 Subject: [ofa-general] Re: linux-next: origin tree build warning In-Reply-To: <20090622110113.7cd89382.sfr@canb.auug.org.au> References: <20090622110113.7cd89382.sfr@canb.auug.org.au> Message-ID: <20090622121054K.fujita.tomonori@lab.ntt.co.jp> On Mon, 22 Jun 2009 11:01:13 +1000 Stephen Rothwell wrote: > Hi all, > > Today's linux-next build (powerpc ppc64_defconfig) produced this warning: > > drivers/infiniband/hw/mthca/mthca_mr.c: In function 'mthca_arbel_write_mtt_seg': > drivers/infiniband/hw/mthca/mthca_mr.c:358: warning: 'dma_sync_single' is deprecated (declared at /scratch/sfr/next/include/linux/dma-mapping.h:113) > drivers/infiniband/hw/mthca/mthca_mr.c: In function 'mthca_arbel_map_phys_fmr': > drivers/infiniband/hw/mthca/mthca_mr.c:810: warning: 'dma_sync_single' is deprecated (declared at /scratch/sfr/next/include/linux/dma-mapping.h:113) > > Introduced by commit dbe6f1869188b6e04e38aa861dd198befb08bcd7 > ("dma-mapping: mark dma_sync_single and dma_sync_sg as deprecated"). Thanks, here's a fix for this: http://marc.info/?l=linux-kernel&m=124347325819016&w=2 The patch doesn't change the driver functionally but the driver looks broken wrt DMA synchronization. I guess that this issue is on Roland's todo list: http://marc.info/?l=linux-kernel&m=124354923710600&w=2 From hal.rosenstock at gmail.com Mon Jun 22 10:47:31 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 22 Jun 2009 13:47:31 -0400 Subject: [ofa-general] [PATCH] opensm: sweep component processors return status value In-Reply-To: <20090618191259.GI6163@me> References: <20090618191259.GI6163@me> Message-ID: On Thu, Jun 18, 2009 at 3:12 PM, Sasha Khapyorsky wrote: > > All sweep component processors (aka managers) will return its status > value instead of OSM_SIGNAL_DONE|OSM_SIGNAL_DONE_PENDING status - > waiting for outstanding MAD responses are automated now in > wait_for_pending_transaction() function. Some possible simplifications and some other comments below. -- Hal > > Signed-off-by: Sasha Khapyorsky > --- >  opensm/include/opensm/osm_base.h    |    4 -- >  opensm/include/opensm/osm_lid_mgr.h |   18 ++------ >  opensm/opensm/osm_lid_mgr.c         |   77 +++++++++++++---------------------- >  opensm/opensm/osm_link_mgr.c        |   56 ++++++++++++------------- >  opensm/opensm/osm_mcast_mgr.c       |   36 ++++++---------- >  opensm/opensm/osm_pkey_mgr.c        |   74 ++++++++++++++++------------------ >  opensm/opensm/osm_qos.c             |   33 ++++++++------- >  opensm/opensm/osm_state_mgr.c       |   10 ++-- >  8 files changed, 130 insertions(+), 178 deletions(-) > > diff --git a/opensm/include/opensm/osm_base.h b/opensm/include/opensm/osm_base.h > index bca1133..03eed79 100644 > --- a/opensm/include/opensm/osm_base.h > +++ b/opensm/include/opensm/osm_base.h > @@ -793,10 +793,6 @@ typedef enum _osm_thread_state { >  #define OSM_SIGNAL_PERFMGR_SWEEP               3 >  #define OSM_SIGNAL_MAX                         3 > > -/* status values for sweep managers - can be removed later */ > -#define OSM_SIGNAL_DONE                16 > -#define OSM_SIGNAL_DONE_PENDING        17 > - >  typedef unsigned int osm_signal_t; >  /***********/ > > diff --git a/opensm/include/opensm/osm_lid_mgr.h b/opensm/include/opensm/osm_lid_mgr.h > index d6d1ab8..7e41c0c 100644 > --- a/opensm/include/opensm/osm_lid_mgr.h > +++ b/opensm/include/opensm/osm_lid_mgr.h > @@ -96,7 +96,6 @@ typedef struct osm_lid_mgr { >        osm_db_t *p_db; >        osm_log_t *p_log; >        cl_plock_t *p_lock; > -       boolean_t send_set_reqs; >        osm_db_domain_t *p_g2l; >        cl_qlist_t free_ranges; >        uint8_t used_lids[IB_LID_UCAST_END_HO + 1]; > @@ -118,9 +117,6 @@ typedef struct osm_lid_mgr { >  *      p_lock >  *              Pointer to the serializing lock. >  * > -*      send_set_reqs > -*              Boolean to indicate whether any set requests sent. > -* >  *      p_g2l >  *              Pointer to the database domain storing guid to lid mapping. >  * > @@ -239,17 +235,14 @@ osm_lid_mgr_init(IN osm_lid_mgr_t * const p_mgr, IN struct osm_sm * sm); >  * >  * SYNOPSIS >  */ > -osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr); > +int osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr); >  /* >  * PARAMETERS >  *      p_mgr >  *              [in] Pointer to an osm_lid_mgr_t object. >  * >  * RETURN VALUES > -*      Returns the appropriate signal to the caller: > -*              OSM_SIGNAL_DONE - operation is complete > -*              OSM_SIGNAL_DONE_PENDING - local operations are complete, but > -*                      transactions are still pending on the wire. > +*      Returns 0 on success and non-zero value otherwise. >  * >  * NOTES >  * > @@ -267,17 +260,14 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr); >  * >  * SYNOPSIS >  */ > -osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr); > +int osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr); >  /* >  * PARAMETERS >  *      p_mgr >  *              [in] Pointer to an osm_lid_mgr_t object. >  * >  * RETURN VALUES > -*      Returns the appropriate signal to the caller: > -*              OSM_SIGNAL_DONE - operation is complete > -*              OSM_SIGNAL_DONE_PENDING - local operations are complete, but > -*                      transactions are still pending on the wire. > +*      Returns 0 on success and non-zero value otherwise. >  * >  * NOTES >  * > diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c > index e1d5106..34625ba 100644 > --- a/opensm/opensm/osm_lid_mgr.c > +++ b/opensm/opensm/osm_lid_mgr.c > @@ -821,10 +821,10 @@ static void lid_mgr_set_remote_pi_state_to_init(IN osm_lid_mgr_t * p_mgr, > >  /********************************************************************** >  **********************************************************************/ > -static boolean_t lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, > -                                     IN osm_port_t * p_port, > -                                     IN osm_physp_t * p_physp, > -                                     IN ib_net16_t const lid) > +static int lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, > +                               IN osm_port_t * p_port, > +                               IN osm_physp_t * p_physp, > +                               IN ib_net16_t const lid) >  { >        uint8_t payload[IB_SMP_DATA_SIZE]; >        ib_port_info_t *p_pi = (ib_port_info_t *) payload; > @@ -836,6 +836,7 @@ static boolean_t lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, >        uint8_t op_vls; >        uint8_t port_num; >        boolean_t send_set = FALSE; > +       int ret = 0; > >        OSM_LOG_ENTER(p_mgr->p_log); > > @@ -1068,31 +1069,31 @@ static boolean_t lid_mgr_set_physp_pi(IN osm_lid_mgr_t * p_mgr, >        if (p_mgr->p_subn->first_time_master_sweep == TRUE) >                send_set = TRUE; > > -       if (send_set) { > -               p_mgr->send_set_reqs = TRUE; > -               status = osm_req_set(p_mgr->sm, > -                                    osm_physp_get_dr_path_ptr(p_physp), > -                                    payload, sizeof(payload), > -                                    IB_MAD_ATTR_PORT_INFO, > -                                    cl_hton32(osm_physp_get_port_num(p_physp)), > -                                    CL_DISP_MSGID_NONE, &context); > -       } > +       if (!send_set) > +               goto Exit; > + > +       status = osm_req_set(p_mgr->sm, osm_physp_get_dr_path_ptr(p_physp), > +                            payload, sizeof(payload), IB_MAD_ATTR_PORT_INFO, > +                            cl_hton32(osm_physp_get_port_num(p_physp)), > +                            CL_DISP_MSGID_NONE, &context); > +       if (status != IB_SUCCESS) > +               ret = -1; > >  Exit: >        OSM_LOG_EXIT(p_mgr->p_log); > -       return send_set; > +       return ret; >  } > >  /********************************************************************** >  Processes our own node >  Lock must already be held. >  **********************************************************************/ > -static boolean_t lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) > +static int lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) >  { >        osm_port_t *p_port; >        uint16_t min_lid_ho; >        uint16_t max_lid_ho; > -       boolean_t res = TRUE; > +       int ret; > >        OSM_LOG_ENTER(p_mgr->p_log); > > @@ -1105,7 +1106,7 @@ static boolean_t lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) >                OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0308: " >                        "Can't acquire SM's port object, GUID 0x%016" PRIx64 >                        "\n", cl_ntoh64(p_mgr->p_subn->sm_port_guid)); > -               res = FALSE; > +               ret = -1; >                goto Exit; >        } > > @@ -1134,19 +1135,19 @@ static boolean_t lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * p_mgr) >        /* >           Set the PortInfo the Physical Port associated with this Port. >         */ > -       lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, > -                            cl_hton16(min_lid_ho)); > +       ret = lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, > +                                  cl_hton16(min_lid_ho)); > >  Exit: >        OSM_LOG_EXIT(p_mgr->p_log); > -       return res; > +       return ret; >  } > >  /********************************************************************** >  **********************************************************************/ > -osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) > +int osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) >  { > -       osm_signal_t signal = OSM_SIGNAL_DONE_PENDING; > +       int ret; > >        OSM_LOG_ENTER(p_mgr->p_log); > > @@ -1158,21 +1159,12 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) >           persistent db */ >        lid_mgr_init_sweep(p_mgr); > > -       /* Set the send_set_reqs of the p_mgr to FALSE, and > -          we'll see if any set requests were sent. If not - > -          can signal OSM_SIGNAL_DONE */ > -       p_mgr->send_set_reqs = FALSE; > -       if (lid_mgr_process_our_sm_node(p_mgr) == FALSE) > -               /* The initialization failed */ > -               signal = OSM_SIGNAL_DONE; > - > -       if (p_mgr->send_set_reqs == FALSE) > -               signal = OSM_SIGNAL_DONE; > +       ret = lid_mgr_process_our_sm_node(p_mgr); > >        CL_PLOCK_RELEASE(p_mgr->p_lock); > >        OSM_LOG_EXIT(p_mgr->p_log); > -       return (signal); > +       return ret; >  } > >  /********************************************************************** > @@ -1181,14 +1173,13 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * p_mgr) >  1.2 if a change is required send the port info >  2 if any change send the signal PENDING... >  **********************************************************************/ > -osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) > +int osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) >  { > -       osm_signal_t signal; >        cl_qmap_t *p_port_guid_tbl; >        osm_port_t *p_port; >        ib_net64_t port_guid; > +       int lid_changed, ret = 0; >        uint16_t min_lid_ho, max_lid_ho; > -       int lid_changed; > >        CL_ASSERT(p_mgr); > > @@ -1198,11 +1189,6 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) > >        CL_ASSERT(p_mgr->p_subn->sm_port_guid); > > -       /* Set the send_set_reqs of the p_mgr to FALSE, and > -          we'll see if any set requests were sent. If not - > -          can signal OSM_SIGNAL_DONE */ > -       p_mgr->send_set_reqs = FALSE; > - >        p_port_guid_tbl = &p_mgr->p_subn->port_guid_tbl; > >        for (p_port = (osm_port_t *) cl_qmap_head(p_port_guid_tbl); > @@ -1244,19 +1230,14 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) >                /* the proc returns the fact it sent a set port info */ >                if (lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, >                                         cl_hton16(min_lid_ho))) > -                       p_mgr->send_set_reqs = TRUE; > +                       ret = -1; >        }                       /* all ports */ > >        /* store the guid to lid table in persistent db */ >        osm_db_store(p_mgr->p_g2l); > > -       if (p_mgr->send_set_reqs == FALSE) > -               signal = OSM_SIGNAL_DONE; > -       else > -               signal = OSM_SIGNAL_DONE_PENDING; > - >        CL_PLOCK_RELEASE(p_mgr->p_lock); > >        OSM_LOG_EXIT(p_mgr->p_log); > -       return (signal); > +       return ret; >  } > diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c > index d189471..285096c 100644 > --- a/opensm/opensm/osm_link_mgr.c > +++ b/opensm/opensm/osm_link_mgr.c > @@ -93,8 +93,8 @@ static uint8_t link_mgr_get_smsl(IN osm_sm_t * sm, IN osm_physp_t * p_physp) > >  /********************************************************************** >  **********************************************************************/ > -static boolean_t link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, > -                                      IN uint8_t port_state) > +static int link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, > +                                IN uint8_t port_state) >  { >        uint8_t payload[IB_SMP_DATA_SIZE]; >        ib_port_info_t *p_pi = (ib_port_info_t *) payload; > @@ -102,13 +102,10 @@ static boolean_t link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, >        osm_madw_context_t context; >        osm_node_t *p_node; >        ib_api_status_t status; > -       uint8_t port_num; > -       uint8_t mtu; > -       uint8_t smsl = OSM_DEFAULT_SL; > -       uint8_t op_vls; > -       boolean_t esp0 = FALSE; > -       boolean_t send_set = FALSE; > +       uint8_t port_num, mtu, op_vls, smsl = OSM_DEFAULT_SL; > +       boolean_t esp0 = FALSE, send_set = FALSE; >        osm_physp_t *p_remote_physp; > +       int ret = 0; > >        OSM_LOG_ENTER(sm->p_log); > > @@ -391,28 +388,29 @@ Send: >            && sm->p_subn->first_time_master_sweep == TRUE) >                send_set = TRUE; > > -       if (send_set) > -               status = osm_req_set(sm, osm_physp_get_dr_path_ptr(p_physp), > -                                    payload, sizeof(payload), > -                                    IB_MAD_ATTR_PORT_INFO, > -                                    cl_hton32(port_num), > -                                    CL_DISP_MSGID_NONE, &context); > +       if (!send_set) > +               goto Exit; > + > +       status = osm_req_set(sm, osm_physp_get_dr_path_ptr(p_physp), > +                            payload, sizeof(payload), IB_MAD_ATTR_PORT_INFO, > +                            cl_hton32(port_num), CL_DISP_MSGID_NONE, &context); > +       if (status) > +               ret = -1; > >  Exit: >        OSM_LOG_EXIT(sm->p_log); > -       return send_set; > +       return ret; >  } > >  /********************************************************************** >  **********************************************************************/ > -static osm_signal_t link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, > -                                         IN const uint8_t link_state) > +static int link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, > +                                IN const uint8_t link_state) >  { > -       uint32_t i; > -       uint32_t num_physp; >        osm_physp_t *p_physp; > +       uint32_t i, num_physp; > +       int ret = 0; >        uint8_t current_state; > -       osm_signal_t signal = OSM_SIGNAL_DONE; > >        OSM_LOG_ENTER(sm->p_log); > > @@ -453,20 +451,20 @@ static osm_signal_t link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, >                                p_physp->port_num, >                                ib_get_port_state_str(current_state)); >                else if (link_mgr_set_physp_pi(sm, p_physp, link_state)) > -                       signal = OSM_SIGNAL_DONE_PENDING; > +                       ret = -1; Simpler as: else ret = link_mgr_set_physp_pi() ? >        } > >        OSM_LOG_EXIT(sm->p_log); > -       return (signal); > +       return ret; >  } > >  /********************************************************************** >  **********************************************************************/ > -osm_signal_t osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) > +int osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) >  { >        cl_qmap_t *p_node_guid_tbl; >        osm_node_t *p_node; > -       osm_signal_t signal = OSM_SIGNAL_DONE; > +       int ret = 0; > >        OSM_LOG_ENTER(sm->p_log); > > @@ -476,14 +474,12 @@ osm_signal_t osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) > >        for (p_node = (osm_node_t *) cl_qmap_head(p_node_guid_tbl); >             p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); > -            p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { > -               if (link_mgr_process_node(sm, p_node, link_state) == > -                   OSM_SIGNAL_DONE_PENDING) > -                       signal = OSM_SIGNAL_DONE_PENDING; > -       } > +            p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) > +               if (link_mgr_process_node(sm, p_node, link_state)) > +                       ret = -1; Simpler as: ret = link_mgr_process_node() ? > >        CL_PLOCK_RELEASE(sm->p_lock); > >        OSM_LOG_EXIT(sm->p_log); > -       return (signal); > +       return ret; >  } > diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c > index de0a8a5..ea49588 100644 > --- a/opensm/opensm/osm_mcast_mgr.c > +++ b/opensm/opensm/osm_mcast_mgr.c > @@ -321,7 +321,7 @@ static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, > >  /********************************************************************** >  **********************************************************************/ > -static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) > +static int mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) >  { >        osm_node_t *p_node; >        osm_dr_path_t *p_path; > @@ -333,7 +333,7 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) >        uint32_t max_position; >        osm_mcast_tbl_t *p_tbl; >        ib_net16_t block[IB_MCAST_BLOCK_SIZE]; > -       osm_signal_t signal = OSM_SIGNAL_DONE; > +       int ret = 0; > >        CL_ASSERT(sm); > > @@ -375,10 +375,9 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) >                        OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A02: " >                                "Sending multicast fwd. tbl. block failed (%s)\n", >                                ib_get_err_str(status)); > +                       ret = -1; >                } > > -               signal = OSM_SIGNAL_DONE_PENDING; > - >                if (++position > max_position) { >                        position = 0; >                        block_num++; > @@ -386,7 +385,7 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) >        } > >        OSM_LOG_EXIT(sm->p_log); > -       return signal; > +       return ret; >  } > >  /********************************************************************** > @@ -1104,15 +1103,13 @@ Exit: > >  /********************************************************************** >  **********************************************************************/ > -osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > +int osm_mcast_mgr_process(osm_sm_t * sm) >  { > -       osm_signal_t signal; >        osm_switch_t *p_sw; >        cl_qmap_t *p_sw_tbl; >        cl_qlist_t *p_list = &sm->mgrp_list; >        osm_mgrp_t *p_mgrp; > -       boolean_t pending_transactions = FALSE; > -       int i; > +       int i, ret = 0; > >        OSM_LOG_ENTER(sm->p_log); > > @@ -1142,9 +1139,8 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) >         */ >        p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); >        while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { > -               signal = mcast_mgr_set_tbl(sm, p_sw); > -               if (signal == OSM_SIGNAL_DONE_PENDING) > -                       pending_transactions = TRUE; > +               if (mcast_mgr_set_tbl(sm, p_sw)) > +                       ret = -1; Simpler as: ret = mcast_mgr_set_tbl() ? >                p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item); >        } > > @@ -1157,25 +1153,22 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > >        OSM_LOG_EXIT(sm->p_log); > > -       if (pending_transactions == TRUE) > -               return (OSM_SIGNAL_DONE_PENDING); > -       else > -               return (OSM_SIGNAL_DONE); > +       return ret; >  } > >  /********************************************************************** >   This is the function that is invoked during idle time to handle the >   process request for mcast groups where join/leave/delete was required. >  **********************************************************************/ > -osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) > +int osm_mcast_mgr_process_mgroups(osm_sm_t * sm) >  { >        cl_qlist_t *p_list = &sm->mgrp_list; >        osm_switch_t *p_sw; >        cl_qmap_t *p_sw_tbl; >        osm_mgrp_t *p_mgrp; >        ib_net16_t mlid; > -       osm_signal_t ret, signal = OSM_SIGNAL_DONE; >        osm_mcast_mgr_ctxt_t *ctx; > +       int ret = 0; > >        OSM_LOG_ENTER(sm->p_log); > > @@ -1219,9 +1212,8 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) >        p_sw_tbl = &sm->p_subn->sw_guid_tbl; >        p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); >        while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { > -               ret = mcast_mgr_set_tbl(sm, p_sw); > -               if (ret == OSM_SIGNAL_DONE_PENDING) > -                       signal = ret; > +               if (mcast_mgr_set_tbl(sm, p_sw)) > +                       ret = -1; >                p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item); >        } > > @@ -1229,5 +1221,5 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) > >        CL_PLOCK_RELEASE(sm->p_lock); >        OSM_LOG_EXIT(sm->p_log); > -       return signal; > +       return ret; >  } > diff --git a/opensm/opensm/osm_pkey_mgr.c b/opensm/opensm/osm_pkey_mgr.c > index 9b76ed7..6b23b27 100644 > --- a/opensm/opensm/osm_pkey_mgr.c > +++ b/opensm/opensm/osm_pkey_mgr.c > @@ -189,7 +189,7 @@ pkey_mgr_update_pkey_entry(IN osm_sm_t * sm, > >  /********************************************************************** >  **********************************************************************/ > -static boolean_t > +static ib_api_status_t >  pkey_mgr_enforce_partition(IN osm_log_t * p_log, osm_sm_t * sm, >                           IN osm_physp_t * p_physp, IN const boolean_t enforce) >  { > @@ -207,7 +207,7 @@ pkey_mgr_enforce_partition(IN osm_log_t * p_log, osm_sm_t * sm, >                        cl_ntoh64(osm_node_get_node_guid >                                  (osm_physp_get_node_ptr(p_physp))), >                        osm_physp_get_port_num(p_physp)); > -               return FALSE; > +               return IB_SUCCESS; >        } > >        memset(payload, 0, IB_SMP_DATA_SIZE); > @@ -233,28 +233,26 @@ pkey_mgr_enforce_partition(IN osm_log_t * p_log, osm_sm_t * sm, >                             IB_MAD_ATTR_PORT_INFO, >                             cl_hton32(osm_physp_get_port_num(p_physp)), >                             CL_DISP_MSGID_NONE, &context); > -       if (status != IB_SUCCESS) { > +       if (status != IB_SUCCESS) >                OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0511: " >                        "Failed to set PortInfo for " >                        "node 0x%016" PRIx64 " port %u\n", >                        cl_ntoh64(osm_node_get_node_guid >                                  (osm_physp_get_node_ptr(p_physp))), >                        osm_physp_get_port_num(p_physp)); > -               return FALSE; > -       } else { > +       else >                OSM_LOG(p_log, OSM_LOG_DEBUG, >                        "Set PortInfo for node 0x%016" PRIx64 " port %u\n", >                        cl_ntoh64(osm_node_get_node_guid >                                  (osm_physp_get_node_ptr(p_physp))), >                        osm_physp_get_port_num(p_physp)); > -               return TRUE; > -       } > +       return status; >  } > >  /********************************************************************** >  **********************************************************************/ > -static boolean_t pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, > -                                     const osm_port_t * const p_port) > +static int pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, > +                               const osm_port_t * const p_port) >  { >        osm_physp_t *p_physp; >        osm_node_t *p_node; > @@ -267,10 +265,10 @@ static boolean_t pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, >        uint16_t num_of_blocks; >        uint16_t max_num_of_blocks; >        ib_api_status_t status; > -       boolean_t ret_val = FALSE; >        osm_pending_pkey_t *p_pending; >        boolean_t found; >        ib_pkey_table_t empty_block; > +       int ret = 0; > >        memset(&empty_block, 0, sizeof(ib_pkey_table_t)); > > @@ -360,32 +358,32 @@ static boolean_t pkey_mgr_update_port(osm_log_t * p_log, osm_sm_t * sm, >                status = >                    pkey_mgr_update_pkey_entry(sm, p_physp, new_block, >                                               block_index); > -               if (status == IB_SUCCESS) { > +               if (status == IB_SUCCESS) >                        OSM_LOG(p_log, OSM_LOG_DEBUG, >                                "Updated pkey table block %d for node 0x%016" >                                PRIx64 " port %u\n", block_index, >                                cl_ntoh64(osm_node_get_node_guid(p_node)), >                                osm_physp_get_port_num(p_physp)); > -                       ret_val = TRUE; > -               } else { > +               else { >                        OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0506: " >                                "pkey_mgr_update_pkey_entry() failed to update " >                                "pkey table block %d for node 0x%016" PRIx64 >                                " port %u\n", block_index, >                                cl_ntoh64(osm_node_get_node_guid(p_node)), >                                osm_physp_get_port_num(p_physp)); > +                       ret = -1; >                } >        } > > -       return ret_val; > +       return ret; >  } > >  /********************************************************************** >  **********************************************************************/ > -static boolean_t > -pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, > -                         const osm_subn_t * p_subn, > -                         const osm_port_t * const p_port, boolean_t enforce) > +static int pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, > +                                    const osm_subn_t * p_subn, > +                                    const osm_port_t * const p_port, > +                                    boolean_t enforce) >  { >        osm_physp_t *p_physp, *peer; >        osm_node_t *p_node; > @@ -396,21 +394,20 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, >        uint16_t num_of_blocks; >        uint16_t peer_max_blocks; >        ib_api_status_t status = IB_SUCCESS; > -       boolean_t ret_val = FALSE; > -       boolean_t port_info_set = FALSE; >        ib_pkey_table_t empty_block; > +       int ret = 0; > >        memset(&empty_block, 0, sizeof(ib_pkey_table_t)); > >        p_physp = p_port->p_physp; >        if (!p_physp) > -               return FALSE; > +               return -1; >        peer = osm_physp_get_remote(p_physp); >        if (!peer) > -               return FALSE; > +               return -1; >        p_node = osm_physp_get_node_ptr(peer); >        if (!p_node->sw || !p_node->sw->switch_info.enforce_cap) > -               return FALSE; > +               return 0; Why are the FALSEs above changed inconsistently ? >        p_pkey_tbl = osm_physp_get_pkey_tbl(p_physp); >        p_peer_pkey_tbl = &peer->pkeys; > @@ -424,13 +421,14 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, >                        cl_ntoh64(osm_node_get_node_guid(p_node)), >                        osm_physp_get_port_num(peer)); >                enforce = FALSE; > +               ret = -1; >        } > >        if (pkey_mgr_enforce_partition(p_log, sm, peer, enforce)) > -               port_info_set = TRUE; > +               ret = -1; Simpler as: ret = pkey_mgr_enforce_partition() ? >        if (enforce == FALSE) > -               return port_info_set; > +               return ret; > >        p_peer_pkey_tbl->used_blocks = p_pkey_tbl->used_blocks; >        for (block_index = 0; block_index < p_pkey_tbl->used_blocks; > @@ -443,12 +441,9 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, >                    osm_pkey_tbl_block_get(p_peer_pkey_tbl, block_index); >                if (!peer_block >                    || memcmp(peer_block, block, sizeof(*peer_block))) { > -                       status = > -                           pkey_mgr_update_pkey_entry(sm, peer, block, > -                                                      block_index); > -                       if (status == IB_SUCCESS) > -                               ret_val = TRUE; > -                       else > +                       status = pkey_mgr_update_pkey_entry(sm, peer, block, > +                                                           block_index); > +                       if (status != IB_SUCCESS) { >                                OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0509: " >                                        "pkey_mgr_update_pkey_entry() failed to update " >                                        "pkey table block %d for node 0x%016" > @@ -456,19 +451,19 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, >                                        cl_ntoh64(osm_node_get_node_guid >                                                  (p_node)), >                                        osm_physp_get_port_num(peer)); > +                               ret = -1; > +                       } >                } >        } > > -       if (ret_val) > +       if (!ret) >                OSM_LOG(p_log, OSM_LOG_DEBUG, >                        "Pkey table was updated for node 0x%016" PRIx64 >                        " port %u\n", >                        cl_ntoh64(osm_node_get_node_guid(p_node)), >                        osm_physp_get_port_num(peer)); > > -       if (port_info_set) > -               return TRUE; > -       return ret_val; > +       return ret; >  } > >  /********************************************************************** > @@ -479,7 +474,7 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) >        cl_map_item_t *p_next; >        osm_prtn_t *p_prtn; >        osm_port_t *p_port; > -       osm_signal_t signal = OSM_SIGNAL_DONE; > +       int ret = 0; > >        CL_ASSERT(p_osm); > > @@ -490,6 +485,7 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) >        if (osm_prtn_make_partitions(&p_osm->log, &p_osm->subn) != IB_SUCCESS) { >                OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 0510: " >                        "osm_prtn_make_partitions() failed\n"); > +               ret = -1; >                goto _err; >        } > > @@ -512,17 +508,17 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) >                p_port = (osm_port_t *) p_next; >                p_next = cl_qmap_next(p_next); >                if (pkey_mgr_update_port(&p_osm->log, &p_osm->sm, p_port)) > -                       signal = OSM_SIGNAL_DONE_PENDING; > +                       ret = -1; Simpler as: ret = pkey_mgr_update_port() ? >                if ((osm_node_get_type(p_port->p_node) != IB_NODE_TYPE_SWITCH) >                    && pkey_mgr_update_peer_port(&p_osm->log, &p_osm->sm, >                                                 &p_osm->subn, p_port, >                                                 !p_osm->subn.opt. >                                                 no_partition_enforcement)) > -                       signal = OSM_SIGNAL_DONE_PENDING; > +                       ret = -1; Simpler as: if (osm_node_get_type()) ret = pkey_mgr_update_peer_port() ? >        } > >  _err: >        CL_PLOCK_RELEASE(&p_osm->lock); >        OSM_LOG_EXIT(&p_osm->log); > -       return (signal); > +       return ret; >  } > diff --git a/opensm/opensm/osm_qos.c b/opensm/opensm/osm_qos.c > index d0aa866..e3dfb58 100644 > --- a/opensm/opensm/osm_qos.c > +++ b/opensm/opensm/osm_qos.c > @@ -222,10 +222,10 @@ static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, >        return IB_SUCCESS; >  } > > -static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, > -                                      osm_port_t * p_port, osm_physp_t * p, > -                                      uint8_t port_num, unsigned force_update, > -                                      const struct qos_config *qcfg) > +static int qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, > +                          osm_port_t * p_port, osm_physp_t * p, > +                          uint8_t port_num, unsigned force_update, > +                          const struct qos_config *qcfg) >  { >        ib_api_status_t status; > > @@ -241,7 +241,7 @@ static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, >                        "failed to update VLArbitration tables " >                        "for port %" PRIx64 " #%d\n", >                        cl_ntoh64(p->port_guid), port_num); > -               return status; > +               return -1; >        } > >        /* setup SL2VL tables */ > @@ -251,13 +251,13 @@ static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, >                        "failed to update SL2VLMapping tables " >                        "for port %" PRIx64 " #%d\n", >                        cl_ntoh64(p->port_guid), port_num); > -               return status; > +               return -1; >        } > > -       return IB_SUCCESS; > +       return 0; >  } > > -osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) > +int osm_qos_setup(osm_opensm_t * p_osm) >  { >        struct qos_config ca_config, sw0_config, swe_config, rtr_config; >        struct qos_config *cfg; > @@ -267,12 +267,12 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) >        uint32_t num_physp; >        osm_physp_t *p_physp; >        osm_node_t *p_node; > -       ib_api_status_t status; >        unsigned force_update; > +       int ret = 0; >        uint8_t i; > >        if (!p_osm->subn.opt.qos) > -               return OSM_SIGNAL_DONE; > +               return 0; > >        OSM_LOG_ENTER(&p_osm->log); > > @@ -305,10 +305,10 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) >                                        continue; >                                force_update = p_physp->need_update || >                                    p_osm->subn.need_update; > -                               status = > -                                   qos_physp_setup(&p_osm->log, &p_osm->sm, > +                               if (qos_physp_setup(&p_osm->log, &p_osm->sm, >                                                    p_port, p_physp, i, > -                                                   force_update, &swe_config); > +                                                   force_update, &swe_config)) > +                                       ret = -1; Simpler as: ret = qos_physp_setup() ? >                        } >                        /* skip base port 0 */ >                        if (!ib_switch_info_is_enhanced_port0 > @@ -326,14 +326,15 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) >                        continue; > >                force_update = p_physp->need_update || p_osm->subn.need_update; > -               status = qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, > -                                        p_physp, 0, force_update, cfg); > +               if (qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, p_physp, > +                                   0, force_update, cfg)) > +                       ret = -1; Simpler as: ret = qos_physp_setup() ? >        } > >        cl_plock_release(&p_osm->lock); >        OSM_LOG_EXIT(&p_osm->log); > > -       return OSM_SIGNAL_DONE; > +       return ret; >  } > >  /* > diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c > index a3e3609..adc39a0 100644 > --- a/opensm/opensm/osm_state_mgr.c > +++ b/opensm/opensm/osm_state_mgr.c > @@ -64,11 +64,11 @@ >  #include > >  extern void osm_drop_mgr_process(IN osm_sm_t * sm); > -extern osm_signal_t osm_qos_setup(IN osm_opensm_t * p_osm); > -extern osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm); > -extern osm_signal_t osm_mcast_mgr_process(IN osm_sm_t * sm); > -extern osm_signal_t osm_mcast_mgr_process_mgroups(IN osm_sm_t * sm); > -extern osm_signal_t osm_link_mgr_process(IN osm_sm_t * sm, IN uint8_t state); > +extern int osm_qos_setup(IN osm_opensm_t * p_osm); > +extern int osm_pkey_mgr_process(IN osm_opensm_t * p_osm); > +extern int osm_mcast_mgr_process(IN osm_sm_t * sm); > +extern int osm_mcast_mgr_process_mgroups(IN osm_sm_t * sm); > +extern int osm_link_mgr_process(IN osm_sm_t * sm, IN uint8_t state); > >  /********************************************************************** >  **********************************************************************/ > -- > 1.6.3.1 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From ishai at mellanox.co.il Mon Jun 22 13:47:57 2009 From: ishai at mellanox.co.il (Ishai Rabinovitz) Date: Mon, 22 Jun 2009 23:47:57 +0300 Subject: [ofa-general] Multipath, srp_daemon help In-Reply-To: <9759F033B56A60469F217C49AC58E2BF085A541A@MAILUK2.rms.com> References: <9759F033B56A60469F217C49AC58E2BF085A541A@MAILUK2.rms.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD031CA7D5@mtlexch01.mtl.com> Hi Evan, Can you supply more data: 1) Do you see output when the srp_daemon is running? 2) Can you find ib_srp prints in /var/log/messages Currently I guess that he problem is in the configuration of multipath (maybe the disks are in the black list). I'm quoting the srp release notes: " To modify and tune multipath configuration, edit the file /etc/multipath.conf according to instructions and tips listed in /usr/share/doc/packages/multipath-tools/multipath.conf.*" I hope it helps. Ishai -----Original Message----- From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Evan Fraser Sent: Wednesday, June 17, 2009 7:45 PM To: 'general at lists.openfabrics.org' Subject: [ofa-general] Multipath, srp_daemon help Hello folks, I'm trying to add some luns to a Linux server running SLES 10SP2 without rebooting it. I'm using OFED 1.4.1rc2, with a Voltaire dual port HBA and connecting to the luns from a DDN S2A9900. I've tried rescanning the SRP targets via: srp_daemon -o -e -n -d /dev/infiniband/umad0 srp_daemon -o -e -n -d /dev/infiniband/umad1 But multipath -ll still doesn't see the paths/luns. Could somebody please help me by pointing out what I'm overlooking/missing? Thanks very much, Evan Fraser Senior Systems Analyst Peninsular House, 30 Monument Street London EC3R 8NB, United Kingdom Tel +44 20 7444 7860 Mobile +44 75 9024 5788 evan.fraser at rms.com www.rms.com This message and any attachments contain information that may be RMS Inc. confidential and/or privileged. If you are not the intended recipient (or authorized to receive for the intended recipient), and have received this message in error, any use, disclosure or distribution is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to the e-mail and permanently deleting the message from your computer and/or storage system. _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From an2355 at columbia.edu Mon Jun 22 19:02:42 2009 From: an2355 at columbia.edu (Ashwath Narasimhan) Date: Mon, 22 Jun 2009 22:02:42 -0400 Subject: [ofa-general] Infiniband Driver support for Mellanox HCA in Centos 5.3 In-Reply-To: <4A3F7FDF.9010302@mellanox.co.il> References: <4A3F7FDF.9010302@mellanox.co.il> Message-ID: Thank you so much everyone. I really appreciate it. I downloaded the 1.4.1 version and it works fine on Centos 5.3. best regards, Ashwath On Mon, Jun 22, 2009 at 8:58 AM, Tziporet Koren wrote: > Bart Van Assche wrote: > >> On Mon, Jun 22, 2009 at 1:43 AM, Ashwath Narasimhan >> wrote: >> >> >>> 4. One more Question, As I dug into to the release notes, I find that >>> Centos 5.3 is not supported in the distribution of OFED-1.4. Is this >>> correct? Does this mean I have to downgrade to centos 5.2? >>> >>> >> >> Please have a look at the OFED 1.4.1 release notes. >> >> >> > I know the RN since I wrote them > I just say it should work, since till today anything that worked with RH > worked with CentOS with the same version number > > Tziporet > -- regards, Ashwath -------------- next part -------------- An HTML attachment was scrubbed... URL: From tharindu at sentory.net Mon Jun 22 21:41:43 2009 From: tharindu at sentory.net (Tharindu Rukshan Bamunuarachchi) Date: Tue, 23 Jun 2009 10:11:43 +0530 Subject: [ofa-general] createQP failed when attr->cap.max_inline_data > 470 Message-ID: Hi all, createQP failed when max_inline_data > 470,but it define as a 1024 bytes in the libmthca library.we have using library comes with the suse_11. libmthca version is 1.0.5 thanks tharindu From rdreier at cisco.com Mon Jun 22 22:23:26 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 22 Jun 2009 22:23:26 -0700 Subject: [ofa-general] Re: [PATCH] EHCA: Ensure that guid_entry index is positive In-Reply-To: <4A38F98B.5070907@gmail.com> (Roel Kluin's message of "Wed, 17 Jun 2009 16:11:23 +0200") References: <4A368D18.108@gmail.com> <4A38F98B.5070907@gmail.com> Message-ID: Makes sense to me, applied. From dotanba at gmail.com Mon Jun 22 22:26:51 2009 From: dotanba at gmail.com (Dotan Barak) Date: Tue, 23 Jun 2009 08:26:51 +0300 Subject: [ofa-general] createQP failed when attr->cap.max_inline_data > 470 In-Reply-To: References: Message-ID: <2f3bf9a60906222226m45bad72k86f7b9bc5e98ed51@mail.gmail.com> You are welcome to create a QP with maximum number of sg enties and check the max_inline_data value of the newly created QP. The same library (libmthca) is being used for several HCAs with different capabilities. Dotan On Tue, Jun 23, 2009 at 7:41 AM, Tharindu Rukshan Bamunuarachchi wrote: > Hi all, > > createQP  failed when max_inline_data > 470,but it define as a 1024 > bytes in the libmthca library.we have using library comes with the > suse_11. > libmthca version is 1.0.5 > > > thanks > tharindu > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From acceptany at gmail.com Mon Jun 22 22:46:04 2009 From: acceptany at gmail.com (Jordan) Date: Tue, 23 Jun 2009 13:46:04 +0800 Subject: [ofa-general] Some confusion about DOR algorithm Message-ID: <91fe68d50906222246o73bad6fk1872bdb9cdf0df82@mail.gmail.com> I have some confusion about the DOR algorithm , in this algorithm , It chooses the outport throuth a "for" iteration :for (i = start_from; i < start_from + num_ports; i++). So the small port_num will be choosed firstly. But not always the small port_num connects the X-dimension or Y-dimension.In other words , using this method we can't always conform the DOR rule. By the way , in the specification, it says " Paths are grown from a destination back to a source using the lowest dimension (port) of available paths at each step. ". Maybe ,the lowest port connect to the X-dimension in one switch, but it connects to the Y-dimension in another switch. So when routing the package from A to B, the dimension may be X -Y- Y- X- X- Y, not X-X-X- Y-Y-Y . This will break the DOR rule. Can anybody explain this situation? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Mon Jun 22 22:53:54 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 22 Jun 2009 22:53:54 -0700 Subject: [ofa-general] RDMA/nes: FIN state handling under error conditions In-Reply-To: <20090616212416.GA6232@flatif-MOBL> (Faisal Latif's message of "Tue, 16 Jun 2009 16:24:16 -0500") References: <20090616212416.GA6232@flatif-MOBL> Message-ID: Thanks, applied both patches. From rdreier at cisco.com Mon Jun 22 23:10:17 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 22 Jun 2009 23:10:17 -0700 Subject: [ofa-general] Re: linux-next: origin tree build warning In-Reply-To: <20090622121054K.fujita.tomonori@lab.ntt.co.jp> (FUJITA Tomonori's message of "Mon, 22 Jun 2009 12:11:12 +0900") References: <20090622110113.7cd89382.sfr@canb.auug.org.au> <20090622121054K.fujita.tomonori@lab.ntt.co.jp> Message-ID: Thanks, I queued up: commit 99987bea474ceca8ec6fb05f81d7d188634cdffd Author: Roland Dreier Date: Mon Jun 22 23:04:13 2009 -0700 IB/mthca: Replace dma_sync_single() use with proper functions dma_sync_single() is deprecated now, and the use in mthca is wrong: there should be a dma_sync_single_for_cpu() before touching the memory from the CPU, and a dma_sync_single_for_device() afterwards. Fix this, prompted by a kick in the pants from a patch from FUJITA Tomonori . Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/hw/mthca/mthca_mr.c b/drivers/infiniband/hw/mthca/mthca_mr.c index d606edf..065b208 100644 --- a/drivers/infiniband/hw/mthca/mthca_mr.c +++ b/drivers/infiniband/hw/mthca/mthca_mr.c @@ -352,10 +352,14 @@ static void mthca_arbel_write_mtt_seg(struct mthca_dev *dev, BUG_ON(!mtts); + dma_sync_single_for_cpu(&dev->pdev->dev, dma_handle, + list_len * sizeof (u64), DMA_TO_DEVICE); + for (i = 0; i < list_len; ++i) mtts[i] = cpu_to_be64(buffer_list[i] | MTHCA_MTT_FLAG_PRESENT); - dma_sync_single(&dev->pdev->dev, dma_handle, list_len * sizeof (u64), DMA_TO_DEVICE); + dma_sync_single_for_device(&dev->pdev->dev, dma_handle, + list_len * sizeof (u64), DMA_TO_DEVICE); } int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, @@ -803,12 +807,15 @@ int mthca_arbel_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list, wmb(); + dma_sync_single_for_cpu(&dev->pdev->dev, fmr->mem.arbel.dma_handle, + list_len * sizeof(u64), DMA_TO_DEVICE); + for (i = 0; i < list_len; ++i) fmr->mem.arbel.mtts[i] = cpu_to_be64(page_list[i] | MTHCA_MTT_FLAG_PRESENT); - dma_sync_single(&dev->pdev->dev, fmr->mem.arbel.dma_handle, - list_len * sizeof(u64), DMA_TO_DEVICE); + dma_sync_single_for_device(&dev->pdev->dev, fmr->mem.arbel.dma_handle, + list_len * sizeof(u64), DMA_TO_DEVICE); fmr->mem.arbel.mpt->key = cpu_to_be32(key); fmr->mem.arbel.mpt->lkey = cpu_to_be32(key); From alexs at linux.vnet.ibm.com Tue Jun 23 00:49:13 2009 From: alexs at linux.vnet.ibm.com (Alexander Schmidt) Date: Tue, 23 Jun 2009 09:49:13 +0200 Subject: [ofa-general] Re: [PATCH] EHCA: Ensure that guid_entry index is positive In-Reply-To: <4A38F98B.5070907@gmail.com> References: <4A368D18.108@gmail.com> <4A38F98B.5070907@gmail.com> Message-ID: <20090623094913.7deba579@BL3D1974.boeblingen.de.ibm.com> On Wed, 17 Jun 2009 16:11:23 +0200 Roel Kluin wrote: > This prevents the memcpy of a guid_entries element using a negative index. > > Signed-off-by: Roel Kluin Acked-by: Alexander Schmidt > --- > Maybe this change then? > > diff --git a/drivers/infiniband/hw/ehca/ehca_hca.c b/drivers/infiniband/hw/ehca/ehca_hca.c > index 9209c53..bd9414b 100644 > --- a/drivers/infiniband/hw/ehca/ehca_hca.c > +++ b/drivers/infiniband/hw/ehca/ehca_hca.c > @@ -319,7 +319,7 @@ int ehca_query_gid(struct ib_device *ibdev, u8 port, > ib_device); > struct hipz_query_port *rblock; > > - if (index > 255) { > + if (index > 255 || index < 0) { > ehca_err(&shca->ib_device, "Invalid index: %x.", index); > return -EINVAL; > } > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From dorfman.eli at gmail.com Tue Jun 23 02:03:47 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Tue, 23 Jun 2009 12:03:47 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <4A3B25D1.8030500@ext.bull.net> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> Message-ID: <4A409A73.9030101@gmail.com> Nicolas Morey-Chaisemartin wrote: > Hi Sasha, > > Le 18/06/2009 16:36, Sasha Khapyorsky a écrit : >>> It also enables to try a new algorithm on the fly without having to >>> reinstall opensm, which can prove useful on a running cluster. >> Even with your patches you will need to restart OpenSM, so it should not >> be a big deal to build and run OpenSM even on a running cluster >> especially if you are going to experiment with new routing engines. >> Alternatively you can load routing tables from file. > > Well at least we wouldn't have to reinstall a new OpenSM (and its dependencies) each time. > Problem is we are experimenting with several algorithms for different topologies and different usages and we don't necessary want to give them all to the clients at once. > With this plugin API you simply need to override the old lib and restart OpenSM (I guess it can probably be made even more interactive by reloading the plugin without restarting). > Most of our algorithm try to add a "dynamic" behaviour to IB routing so loading a file is definitly not a viable solution. There are pending patches to allow dynamic change of opensm routing engine (should fix few comments). I think that using pluggable routing engines is better approach than the current implementation. > >>> The event-based perfmgr plugin does not make it easy to integrate a new >>> routing algorithm because the list of available routing algorithms is >>> statically declared. >> Eventually it is only link list with names and callbacks. It should be >> easy to add new entry there. >> > I guess it could be done but are you sure the event plugin is loaded and set up before the routing engine ? > If it is, we could make the routing_engine list a bit more dynamic. However I'm not fully satisfied of the idea of inserting new engines through an "event" plugin. > We could also make this interface a bit more generic and provides the means to add new features around OpenSM more easily. I think that the routing plugin is a cleaner approach for new routing algorithms. > >>> AFAIK, the latest Voltaire UFM also includes or will include proprietary >>> routing algorithms (mesh + fat tree), so it could also be useful for >>> Voltaire. >> I don't know about this, but even if it is so. It is not our charter as >> open source project to care about such practices and they (UFM) will need >> to support their stuff by themselves. >> > > Fair enough but the cost of this plugin is quite low in terms of code quantity, > so it shouldn't introduce too much trouble maintaning it. It will not add any burden on the open source community - on the contrary. Eli > And as Ira, it reduces OpenSM dependencies for external algorithm. > Some of ours for example rely on PostgreSQL drivers or libdbi, some own library (which uses libibnetdisc) and much more. > > Anyway, this was just a proposal as we thought it might be handful to other people than us. If you think it's not or don't agree with the code, we'll just keep it on our own git branches. > > Regards > > Nicolas > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From devel-ofed at morey-chaisemartin.com Tue Jun 23 06:53:55 2009 From: devel-ofed at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Tue, 23 Jun 2009 15:53:55 +0200 Subject: [ofa-general] Re: [PATCH v2] osm_ucast_ftree.c Allow horizontal links between switches of max rank In-Reply-To: <20090618135855.GG6163@me> References: <4A37997E.8080708@Sun.COM> <20090618135855.GG6163@me> Message-ID: <4A40DE73.9070502@morey-chaisemartin.com> Hi, As I discussed with Line, this patch also present a problem with topolgy using reverse hops and has a negative impact on routing quality. Moreover, I gave it some thoughts and I think there is a flaw in this approach. For example L1-1 L1-2 | | \ | | ---- | | \ _|___|___|_ L2 | QNEM | |\\ ----------- |\\\ //// \\\\ |\\\\ N[1-12] N[13-24] M[1-18] In this example the horizontal link is at the last level of the tree so it is allowed by Line's patch. As horizontal port are treated as downlinks from both sides they can only be used when creating upwards routes (algorithm going down). So in this example nodes N 13 to 24 will have full connectivity to nodes M 1 to 18 and nodes N 1 to 12 will have a route to nodes M 1 to 18. However nodes M 1 to 18 won't have any routes to nodes N 1 to 12 because the algorithm starting from node N1, from example, cannot reach switch L1-2 as the horizontal link is seen as a down link. With reverse hop it should be possible to use one horizontal link and go one but reverse hop are not well balanced and should not be used for compute nodes. What I think should be done is have a more generalized approached. The Ftree algorithm already manages an enum with direction. We could add a 3rd direction (horizontal) and treat links as both upward and downwards. With the patch I've pushed (the one conflicting), we already ensure minimum hop path is used and secondary routes are more balanced so it should work quite nicely. I also discussed with Line about it but I'd like to have other opinions. Why should this be restricted to max rank switches? Voltaire for example is selling 648 port switches (2x 324 with horizontal links) which could use such an update in the Fat-Tree algorithm. Regards Nicolas Le 18/06/2009 15:58, Sasha Khapyorsky a écrit : > Hi Line, > > On 15:09 Tue 16 Jun , Line.Holen at Sun.COM wrote: >> This patch makes it legal to have cross links (horizontal links) between >> switches at max rank. These switches do have same rank, so hop count cannot >> be calculated based on rank anymore. >> The horizontal links are treated as downlinks. Switch A has a downlink to B >> while B has a downlink to A. Tests on lids and also number of hops makes sure >> that we don't loop back and forth across the link. >> >> Signed-off-by: Frank Olaf Sem-Jacobsen >> Signed-off-by: Line Holen > > This patch is conflicting with Nicolas's one (and his was first). Could > you please rebase against the current master and resend. Thanks. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From rdreier at cisco.com Tue Jun 23 10:32:21 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 23 Jun 2009 10:32:21 -0700 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <4A3E45D3.3040405@Voltaire.COM> (Moni Shoua's message of "Sun, 21 Jun 2009 17:38:11 +0300") References: <4A3E45D3.3040405@Voltaire.COM> Message-ID: > This patch was sent on Apr, 23 2009. Nothing was changed except for fixing offsets and verifying it under 2.6.30 Jason had some concerns about this as I recall -- did you every reach consensus? From rdreier at cisco.com Tue Jun 23 10:39:51 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 23 Jun 2009 10:39:51 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This has a few more fixes for 2.6.31-rc1; however everything here should be a fix appropriate for -rc2 as well, so if this misses the end of the merge window then I think it should still be OK to pull. Alexander Schmidt (1): IB/ehca: Bump version number Faisal Latif (2): RDMA/nes: Fix max_qp_init_rd_atom returned from query device RDMA/nes: Fix FIN state handling under error conditions Hannes Hering (1): IB/ehca: Tolerate dynamic memory operations before driver load Peter Huewe (1): RDMA: Add __init/__exit macros to addr.c and cma.c Roel Kluin (1): IB/ehca: Ensure that guid_entry index is not negative Roland Dreier (3): IB/mthca: Replace dma_sync_single() use with proper functions mlx4_core: Fix dma_sync_single_for_cpu() with matching for_device() calls Merge branches 'ehca', 'misc', 'mlx4', 'mthca' and 'nes' into for-linus drivers/infiniband/core/addr.c | 4 +- drivers/infiniband/core/cma.c | 4 +- drivers/infiniband/hw/ehca/ehca_hca.c | 2 +- drivers/infiniband/hw/ehca/ehca_main.c | 20 +- drivers/infiniband/hw/ehca/ehca_mrmw.c | 508 +++++++++++++++++++++++++++++++- drivers/infiniband/hw/ehca/ehca_mrmw.h | 13 +- drivers/infiniband/hw/mthca/mthca_mr.c | 13 +- drivers/infiniband/hw/nes/nes_cm.c | 8 +- drivers/infiniband/hw/nes/nes_verbs.c | 2 +- drivers/net/mlx4/mr.c | 14 +- 10 files changed, 554 insertions(+), 34 deletions(-) From adit.262 at gmail.com Tue Jun 23 11:01:22 2009 From: adit.262 at gmail.com (Adit Ranadive) Date: Tue, 23 Jun 2009 14:01:22 -0400 Subject: [ofa-general] Support for reg_phys_mr in mlx4_ib In-Reply-To: <4A3F5AA6.6000500@voltaire.com> References: <4A3F4093.9030105@mellanox.co.il> <4A3F5AA6.6000500@voltaire.com> Message-ID: Is there a difference with HCAs that the reg_phys_mr verb call isnt supported? Do the FMRs provide the same functionality as the physical MR verb? Thanks, Adit Adit Ranadive PhD CS Candidate Georgia Institute of Technology, Atlanta, GA On Mon, Jun 22, 2009 at 6:19 AM, Or Gerlitz wrote: > > Tziporet Koren wrote: > >> You can use FMRs to register physical pages as a memory regions You can >> look at SRP or NFS/RDMA code to see how is it done >> > and iser as well :) > > Or. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Tue Jun 23 11:27:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 23 Jun 2009 21:27:52 +0300 Subject: [ofa-general] [PATCH] opensm: sweep component processors return status value In-Reply-To: References: <20090618191259.GI6163@me> Message-ID: <20090623182752.GA12955@me> Hi Hal, On 13:47 Mon 22 Jun , Hal Rosenstock wrote: > > Some possible simplifications and some other comments below. Thanks for the comments. > > @@ -453,20 +451,20 @@ static osm_signal_t link_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??p_physp->port_num, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??ib_get_port_state_str(current_state)); > > ?? ?? ?? ?? ?? ?? ?? ??else if (link_mgr_set_physp_pi(sm, p_physp, link_state)) > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? signal = OSM_SIGNAL_DONE_PENDING; > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > else ret = link_mgr_set_physp_pi() ? This function is called in loop for each node port: for (i = 0; i < num_physp; i++) , so if link_mgr_set_physp_pi() will fail for one or more ports the return status will remain -1. In your simplification it can be overwritten to '0' by following successful runs. > > @@ -476,14 +474,12 @@ osm_signal_t osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) > > > > ?? ?? ?? ??for (p_node = (osm_node_t *) cl_qmap_head(p_node_guid_tbl); > > ?? ?? ?? ?? ?? ?? p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); > > - ?? ?? ?? ?? ?? ??p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { > > - ?? ?? ?? ?? ?? ?? ?? if (link_mgr_process_node(sm, p_node, link_state) == > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? OSM_SIGNAL_DONE_PENDING) > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? signal = OSM_SIGNAL_DONE_PENDING; > > - ?? ?? ?? } > > + ?? ?? ?? ?? ?? ??p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) > > + ?? ?? ?? ?? ?? ?? ?? if (link_mgr_process_node(sm, p_node, link_state)) > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > ret = link_mgr_process_node() ? Same as above, but in this case the loop is for each node. > > @@ -1142,9 +1139,8 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > > ?? ?? ?? ?? */ > > ?? ?? ?? ??p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); > > ?? ?? ?? ??while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { > > - ?? ?? ?? ?? ?? ?? ?? signal = mcast_mgr_set_tbl(sm, p_sw); > > - ?? ?? ?? ?? ?? ?? ?? if (signal == OSM_SIGNAL_DONE_PENDING) > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? pending_transactions = TRUE; > > + ?? ?? ?? ?? ?? ?? ?? if (mcast_mgr_set_tbl(sm, p_sw)) > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > ret = mcast_mgr_set_tbl() ? Same as above. > > @@ -396,21 +394,20 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, > > ?? ?? ?? ??uint16_t num_of_blocks; > > ?? ?? ?? ??uint16_t peer_max_blocks; > > ?? ?? ?? ??ib_api_status_t status = IB_SUCCESS; > > - ?? ?? ?? boolean_t ret_val = FALSE; > > - ?? ?? ?? boolean_t port_info_set = FALSE; > > ?? ?? ?? ??ib_pkey_table_t empty_block; > > + ?? ?? ?? int ret = 0; > > > > ?? ?? ?? ??memset(&empty_block, 0, sizeof(ib_pkey_table_t)); > > > > ?? ?? ?? ??p_physp = p_port->p_physp; > > ?? ?? ?? ??if (!p_physp) > > - ?? ?? ?? ?? ?? ?? ?? return FALSE; > > + ?? ?? ?? ?? ?? ?? ?? return -1; > > ?? ?? ?? ??peer = osm_physp_get_remote(p_physp); > > ?? ?? ?? ??if (!peer) > > - ?? ?? ?? ?? ?? ?? ?? return FALSE; > > + ?? ?? ?? ?? ?? ?? ?? return -1; > > ?? ?? ?? ??p_node = osm_physp_get_node_ptr(peer); > > ?? ?? ?? ??if (!p_node->sw || !p_node->sw->switch_info.enforce_cap) > > - ?? ?? ?? ?? ?? ?? ?? return FALSE; > > + ?? ?? ?? ?? ?? ?? ?? return 0; > > Why are the FALSEs above changed inconsistently ? It is because previously this function returned an indication that something was sent or not, regardless to the reason - errors or not. Now it returns the status of success including the case when no MADs sending is needed. > > @@ -424,13 +421,14 @@ pkey_mgr_update_peer_port(osm_log_t * p_log, osm_sm_t * sm, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??cl_ntoh64(osm_node_get_node_guid(p_node)), > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??osm_physp_get_port_num(peer)); > > ?? ?? ?? ?? ?? ?? ?? ??enforce = FALSE; > > + ?? ?? ?? ?? ?? ?? ?? ret = -1; > > ?? ?? ?? ??} > > > > ?? ?? ?? ??if (pkey_mgr_enforce_partition(p_log, sm, peer, enforce)) > > - ?? ?? ?? ?? ?? ?? ?? port_info_set = TRUE; > > + ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > ret = pkey_mgr_enforce_partition() ? Similar to a loop cases, such statement may overwrite ret value if it indicates previous errors. > > @@ -512,17 +508,17 @@ osm_signal_t osm_pkey_mgr_process(IN osm_opensm_t * p_osm) > > ?? ?? ?? ?? ?? ?? ?? ??p_port = (osm_port_t *) p_next; > > ?? ?? ?? ?? ?? ?? ?? ??p_next = cl_qmap_next(p_next); > > ?? ?? ?? ?? ?? ?? ?? ??if (pkey_mgr_update_port(&p_osm->log, &p_osm->sm, p_port)) > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? signal = OSM_SIGNAL_DONE_PENDING; > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > ret = pkey_mgr_update_port() ? This is a loop case. > > > ?? ?? ?? ?? ?? ?? ?? ??if ((osm_node_get_type(p_port->p_node) != IB_NODE_TYPE_SWITCH) > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ??&& pkey_mgr_update_peer_port(&p_osm->log, &p_osm->sm, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? &p_osm->subn, p_port, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? !p_osm->subn.opt. > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? no_partition_enforcement)) > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? signal = OSM_SIGNAL_DONE_PENDING; > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > if (osm_node_get_type()) > ret = pkey_mgr_update_peer_port() ? Ditto. > > @@ -305,10 +305,10 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??continue; > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??force_update = p_physp->need_update || > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??p_osm->subn.need_update; > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? status = > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? qos_physp_setup(&p_osm->log, &p_osm->sm, > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? if (qos_physp_setup(&p_osm->log, &p_osm->sm, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??p_port, p_physp, i, > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? force_update, &swe_config); > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? force_update, &swe_config)) > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > ret = qos_physp_setup() ? > > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??} > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??/* skip base port 0 */ > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??if (!ib_switch_info_is_enhanced_port0 > > @@ -326,14 +326,15 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??continue; > > > > ?? ?? ?? ?? ?? ?? ?? ??force_update = p_physp->need_update || p_osm->subn.need_update; > > - ?? ?? ?? ?? ?? ?? ?? status = qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??p_physp, 0, force_update, cfg); > > + ?? ?? ?? ?? ?? ?? ?? if (qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, p_physp, > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 0, force_update, cfg)) > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ret = -1; > > Simpler as: > ret = qos_physp_setup() ? Same as above - it is under loop. Sasha From sashak at voltaire.com Tue Jun 23 12:08:15 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 23 Jun 2009 22:08:15 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <4A3B25D1.8030500@ext.bull.net> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> Message-ID: <20090623190815.GB4324@me> Hi Nicolas, On 07:44 Fri 19 Jun , Nicolas Morey-Chaisemartin wrote: > > > > Eventually it is only link list with names and callbacks. It should be > > easy to add new entry there. > > > I guess it could be done but are you sure the event plugin is loaded and set up before the routing engine ? Yes. > If it is, we could make the routing_engine list a bit more dynamic. However I'm not fully satisfied of the idea of inserting new engines through an "event" plugin. This called "event" for historical reasons - now it is completely generic plugin interface. > We could also make this interface a bit more generic and provides the means to add new features around OpenSM more easily. For example? Reload plugin on the fly? That is fine for me. > > >> AFAIK, the latest Voltaire UFM also includes or will include proprietary > >> routing algorithms (mesh + fat tree), so it could also be useful for > >> Voltaire. > > > > I don't know about this, but even if it is so. It is not our charter as > > open source project to care about such practices and they (UFM) will need > > to support their stuff by themselves. > > > > Fair enough but the cost of this plugin is quite low in terms of code quantity, > so it shouldn't introduce too much trouble maintaning it. > And as Ira, it reduces OpenSM dependencies for external algorithm. > Some of ours for example rely on PostgreSQL drivers or libdbi, some own library (which uses libibnetdisc) and much more. I understand such arguments, but up to now I didn't see any good reason to have more than one plugin interface in OpenSM. Sasha From sashak at voltaire.com Tue Jun 23 12:14:32 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 23 Jun 2009 22:14:32 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <1245357516.31575.149.camel@auk31.llnl.gov> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <1245357516.31575.149.camel@auk31.llnl.gov> Message-ID: <20090623191432.GC4324@me> Hi Al, On 13:38 Thu 18 Jun , Al Chu wrote: > > I agree with this on principle, however, we already support a plugin > system that allows users to develop proprietary perfmgr plugins. So I > feel that we already are down this path. It is not about allowing or not - I thought about our support for doing such things. > Is the primary issue that this module "imports" new behavior into > opensm? I don't know the perfmgr plugin code that well, but my > understanding is that you can "import new behavior" into opensm via the > perfmgr plugin code too? So what is really different?? Right, nothing different. And this is exactly my point - why yet another plugin interface is needed? Sasha From weiny2 at llnl.gov Tue Jun 23 13:19:39 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 23 Jun 2009 13:19:39 -0700 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090623190815.GB4324@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> <20090623190815.GB4324@me> Message-ID: <20090623131939.e464afb2.weiny2@llnl.gov> Hi Sasha, Question at the bottom. On Tue, 23 Jun 2009 22:08:15 +0300 Sasha Khapyorsky wrote: > Hi Nicolas, > > On 07:44 Fri 19 Jun , Nicolas Morey-Chaisemartin wrote: > > > > > > Eventually it is only link list with names and callbacks. It should be > > > easy to add new entry there. > > > > > I guess it could be done but are you sure the event plugin is loaded and > > set up before the routing engine ? > > Yes. > > > If it is, we could make the routing_engine list a bit more dynamic. > > However I'm not fully satisfied of the idea of inserting new engines > > through an "event" plugin. > > This called "event" for historical reasons - now it is completely generic > plugin interface. > > > We could also make this interface a bit more generic and provides the > > means to add new features around OpenSM more easily. > > For example? Reload plugin on the fly? That is fine for me. > > > > > >> AFAIK, the latest Voltaire UFM also includes or will include > > >> proprietary routing algorithms (mesh + fat tree), so it could also be > > >> useful for Voltaire. > > > > > > I don't know about this, but even if it is so. It is not our charter as > > > open source project to care about such practices and they (UFM) will > > > need to support their stuff by themselves. > > > > > > > Fair enough but the cost of this plugin is quite low in terms of code > > quantity, so it shouldn't introduce too much trouble maintaning it. And > > as Ira, it reduces OpenSM dependencies for external algorithm. Some of > > ours for example rely on PostgreSQL drivers or libdbi, some own library > > (which uses libibnetdisc) and much more. > > I understand such arguments, but up to now I didn't see any good reason to > have more than one plugin interface in OpenSM. > Does this mean you are open to the idea now? Or are you still not convinced? Ira From sashak at voltaire.com Tue Jun 23 13:43:24 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 23 Jun 2009 23:43:24 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090623131939.e464afb2.weiny2@llnl.gov> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> <20090623190815.GB4324@me> <20090623131939.e464afb2.weiny2@llnl.gov> Message-ID: <20090623204324.GE4324@me> Hi Ira, On 13:19 Tue 23 Jun , Ira Weiny wrote: > > Does this mean you are open to the idea now? Or are you still not convinced? Which idea? To have a plugin interface? We have it already. Sasha From sashak at voltaire.com Tue Jun 23 14:13:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 24 Jun 2009 00:13:04 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_mesh.c: Improve VL utilization In-Reply-To: <20090617201026.GC11146@comcast.net> References: <20090617201026.GC11146@comcast.net> Message-ID: <20090623211304.GF4324@me> Hi Hal, On 16:10 Wed 17 Jun , Hal Rosenstock wrote: > diff --git a/opensm/opensm/osm_mesh.c b/opensm/opensm/osm_mesh.c > index 1867876..f2dd2f8 100644 > --- a/opensm/opensm/osm_mesh.c > +++ b/opensm/opensm/osm_mesh.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2008,2009 System Fabric Works, Inc. > + * Copyright (c) 2008,2009 System Fabric Works, Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -1026,12 +1026,15 @@ static inline int ltmag(int a, int b) > return (a1 < b1) || (a1 == b1 && a > b); > } > > +static int dim_order[MAX_DIMENSION]; > +static int dim_reverse[MAX_DIMENSION]; This makes the code non-reenterable. Wouldn't it better to move this under 'struct mesh'? > @@ -1191,7 +1226,49 @@ static int measure_geometry(lash_t *p_lash, mesh_t *mesh, int seed) > } > > for (i = 0; i < dimension; i++) > - mesh->size[i] = max[i] - min[i] + 1; > + mesh->size[i] = size[i] = max[i] - min[i] + 1; > + > + /* > + * find an order of dimensions that places largest > + * sizes first since this seems to work best with LASH > + */ > + for (j = 0; j < dimension; j++) { > + max_size = -1; > + max_index = -1; > + > + for (i = 0; i < dimension; i++) { > + if (size[i] > max_size) { > + max_size = size[i]; > + max_index = i; > + } > + } > + > + dim_order[j] = max_index; > + dim_reverse[max_index] = j; Where is 'dim_reverse' used, I didn't find. Sasha From sashak at voltaire.com Tue Jun 23 14:49:28 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 24 Jun 2009 00:49:28 +0300 Subject: [ofa-general] Re: [PATCH] opensm/lash: Set minimum VL for LASH to use In-Reply-To: <20090618141430.GA11942@comcast.net> References: <20090618141430.GA11942@comcast.net> Message-ID: <20090623214928.GG4324@me> Hi Hal, On 10:14 Thu 18 Jun , Hal Rosenstock wrote: > > rather than starting from VL 0 > > Signed-off-by: Robert Pearson > Signed-off-by: Hal Rosenstock See the comments below. > diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c > index 12b5e34..540214c 100644 > --- a/opensm/opensm/osm_ucast_lash.c > +++ b/opensm/opensm/osm_ucast_lash.c > @@ -478,7 +478,7 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) > cdg_vertex_t ****cdg_vertex_matrix = p_lash->cdg_vertex_matrix; > int *num_mst_in_lane = p_lash->num_mst_in_lane; > int ***virtual_location = p_lash->virtual_location; > - int min_filled_lane, max_filled_lane, trials; > + int min_filled_lane, max_filled_lane, trials, max_vl; > int old_min_filled_lane, old_max_filled_lane, new_num_min_lane, > new_num_max_lane; > unsigned int i, j; > @@ -486,9 +486,13 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) > int next_switch2, output_link2; > int stop = 0, cycle_found; > int cycle_found2; > + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; In this function 'opt' is used only to get 'lash_start_vl', so wouldn't it be better to do something like: unsigned start_vl = &p_lash->p_osm->subn.opt.lash_start_vl; ? This is relevant for all other similar places in the patch. > > - max_filled_lane = 0; > - min_filled_lane = lanes_needed - 1; > + max_filled_lane = opt->lash_start_vl; > + max_vl = lanes_needed + opt->lash_start_vl; > + if (max_vl > IB_MAX_NUM_VLS) > + max_vl = IB_MAX_NUM_VLS; Isn't this case where LASH requires more VLs than available in IB? > + min_filled_lane = max_vl - 1; > > trials = num_mst_in_lane[max_filled_lane]; > if (lanes_needed == 1) > @@ -590,7 +594,7 @@ static void balance_virtual_lanes(lash_t * p_lash, unsigned lanes_needed) > new_num_min_lane = MAX_INT; > new_num_max_lane = 0; > > - for (i = 0; i < lanes_needed; i++) { > + for (i = opt->lash_start_vl; i < max_vl; i++) { > > if (num_mst_in_lane[i] < new_num_min_lane) { > new_num_min_lane = num_mst_in_lane[i]; > @@ -673,12 +677,18 @@ static void free_lash_structures(lash_t * p_lash) > { > unsigned int i, j, k; > unsigned num_switches = p_lash->num_switches; > + int max_vl; > osm_log_t *p_log = &p_lash->p_osm->log; > + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; > > OSM_LOG_ENTER(p_log); > > + max_vl = opt->lash_start_vl + p_lash->vl_min; > + if (max_vl > IB_MAX_NUM_VLS) > + max_vl = IB_MAX_NUM_VLS; > + > /* free cdg_vertex_matrix */ > - for (i = 0; i < p_lash->vl_min; i++) { > + for (i = opt->lash_start_vl; i < max_vl; i++) { > for (j = 0; j < num_switches; j++) { > for (k = 0; k < num_switches; k++) > if (p_lash->cdg_vertex_matrix[i][j][k]) > @@ -715,13 +725,19 @@ static int init_lash_structures(lash_t * p_lash) > osm_log_t *p_log = &p_lash->p_osm->log; > int status = 0; > unsigned int i, j, k; > + int max_vl; > + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; > > OSM_LOG_ENTER(p_log); > > + max_vl = vl_min + opt->lash_start_vl; I'm not following... vl_min is total number of VLs available in a fabric, lash_start_vl is VL number where LASH may start to use VLs, so VLs available for LASH are lash_start_vl..vl_min. Assuming so what does 'max_vl = vl_min + lash_start_vl' represent? It is always overflow in terms of available VLs, no? > + if (max_vl > IB_MAX_NUM_VLS) > + max_vl = IB_MAX_NUM_VLS; > + > /* initialise cdg_vertex_matrix[num_switches][num_switches][num_switches] */ > p_lash->cdg_vertex_matrix = > - (cdg_vertex_t ****) malloc(vl_min * sizeof(cdg_vertex_t ****)); > - for (i = 0; i < vl_min; i++) { > + (cdg_vertex_t ****) calloc(max_vl, sizeof(cdg_vertex_t ****)); > + for (i = opt->lash_start_vl; i < max_vl; i++) { > p_lash->cdg_vertex_matrix[i] = > (cdg_vertex_t ***) malloc(num_switches * > sizeof(cdg_vertex_t ***)); > @@ -730,7 +746,7 @@ static int init_lash_structures(lash_t * p_lash) > goto Exit_Mem_Error; > } > > - for (i = 0; i < vl_min; i++) { > + for (i = opt->lash_start_vl; i < max_vl; i++) { > for (j = 0; j < num_switches; j++) { > p_lash->cdg_vertex_matrix[i][j] = > (cdg_vertex_t **) malloc(num_switches * > @@ -763,7 +779,7 @@ static int init_lash_structures(lash_t * p_lash) > for (i = 0; i < num_switches; i++) { > for (j = 0; j < num_switches; j++) { > p_lash->virtual_location[i][j] = > - (int *)malloc(vl_min * sizeof(int *)); > + (int *)calloc(max_vl, sizeof(int *)); > if (p_lash->virtual_location[i][j] == NULL) > goto Exit_Mem_Error; > for (k = 0; k < vl_min; k++) { > @@ -804,10 +820,12 @@ static int lash_core(lash_t * p_lash) > int cycle_found2 = 0; > int status = 0; > int *switch_bitmap = NULL; /* Bitmap to check if we have processed this pair */ > + int max_vl; > + osm_subn_opt_t *opt = &p_lash->p_osm->subn.opt; > > OSM_LOG_ENTER(p_log); > > - if (p_lash->p_osm->subn.opt.do_mesh_analysis && osm_do_mesh_analysis(p_lash)) { > + if (opt->do_mesh_analysis && osm_do_mesh_analysis(p_lash)) { > OSM_LOG(p_log, OSM_LOG_ERROR, "Mesh analysis failed\n"); > goto Exit; > } > @@ -838,11 +856,14 @@ static int lash_core(lash_t * p_lash) > } > > for (i = 0; i < num_switches; i++) { > - for (dest_switch = 0; dest_switch < num_switches; dest_switch++) > + for (dest_switch = 0; dest_switch < num_switches; dest_switch++) { > + max_vl = lanes_needed + opt->lash_start_vl; > + if (max_vl > IB_MAX_NUM_VLS) > + max_vl = IB_MAX_NUM_VLS; Shouldn't comparison be done against lsh->vl_min (number VLs available in a fabric)? And return an error in case if it is exceeded (which means LASH requires more VLs than available)? > if (dest_switch != i && switch_bitmap[i * num_switches + dest_switch] == 0) { > - v_lane = 0; > + v_lane = opt->lash_start_vl; > stop = 0; > - while (v_lane < lanes_needed && stop == 0) { > + while (v_lane < max_vl && stop == 0) { > generate_cdg_for_sp(p_lash, i, dest_switch, v_lane); > generate_cdg_for_sp(p_lash, dest_switch, i, v_lane); > > @@ -906,7 +927,9 @@ static int lash_core(lash_t * p_lash) > switches[dest_switch]->routing_table[i].lane = v_lane; > > if (cycle_found == 1 || cycle_found2 == 1) { > - if (++lanes_needed > p_lash->vl_min) > + lanes_needed++; > + if (lanes_needed > p_lash->vl_min || > + opt->lash_start_vl + lanes_needed - 1 >= IB_DROP_VL) Shouldn't be just if (start_vl + lanes_needed > p_lash->vl_min) ? > goto Error_Not_Enough_Lanes; > > generate_cdg_for_sp(p_lash, i, dest_switch, v_lane); > @@ -926,19 +949,24 @@ static int lash_core(lash_t * p_lash) > switch_bitmap[i * num_switches + dest_switch] = 1; > switch_bitmap[dest_switch * num_switches + i] = 1; > } > + } > } > > OSM_LOG(p_log, OSM_LOG_INFO, > "Lanes needed: %d, Balancing\n", lanes_needed); > > - for (i = 0; i < lanes_needed; i++) { > + max_vl = lanes_needed + opt->lash_start_vl; > + if (max_vl > IB_MAX_NUM_VLS) Shouldn't this be compared with p_lash->vl_min (which is maximum number of VLs available in a fabric) and not with IB_MAX_NUM_VLS? > + max_vl = IB_MAX_NUM_VLS; When max_vl is larger than number of available VLs (IOW lash needs more VLs than fabric has) shouldn't it fallback to another routing algorithm? > + > + for (i = opt->lash_start_vl; i < max_vl; i++) { > OSM_LOG(p_log, OSM_LOG_INFO, "Lanes in layer %d: %d\n", > i, p_lash->num_mst_in_lane[i]); > } > > balance_virtual_lanes(p_lash, lanes_needed); > > - for (i = 0; i < lanes_needed; i++) { > + for (i = opt->lash_start_vl; i < max_vl; i++) { > OSM_LOG(p_log, OSM_LOG_INFO, "Lanes in layer %d: %d\n", > i, p_lash->num_mst_in_lane[i]); > } > @@ -1271,6 +1299,7 @@ uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, const osm_port_t * p_src_port, > unsigned dst_id; > unsigned src_id; > osm_switch_t *p_sw; > + osm_subn_opt_t *opt = &p_osm->subn.opt; > > if (p_osm->routing_engine_used != OSM_ROUTING_ENGINE_TYPE_LASH) > return OSM_DEFAULT_SL; > @@ -1286,7 +1315,7 @@ uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, const osm_port_t * p_src_port, > > src_id = get_lash_id(p_sw); > if (src_id == dst_id) > - return OSM_DEFAULT_SL; > + return opt->lash_start_vl; Is this correct? As far as I understand this is SL for paths between CA ports connected to the same switch (which should be safe in sense of credit loops). Should lash be involved here? Sasha From weiny2 at llnl.gov Tue Jun 23 15:29:35 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 23 Jun 2009 15:29:35 -0700 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090623204324.GE4324@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> <20090623190815.GB4324@me> <20090623131939.e464afb2.weiny2@llnl.gov> <20090623204324.GE4324@me> Message-ID: <20090623152935.b4f64a1c.weiny2@llnl.gov> On Tue, 23 Jun 2009 23:43:24 +0300 Sasha Khapyorsky wrote: > Hi Ira, > > On 13:19 Tue 23 Jun , Ira Weiny wrote: > > > > Does this mean you are open to the idea now? Or are you still not convinced? > > Which idea? To have a plugin interface? We have it already. No the idea of a "plugable routing engine". I misunderstood you, sorry. From your previous response are we now going to guarantee that all the plugins will be loaded before opensm does anything? ie all plugins will be able to hook into the code before anything else begins. I am not saying this does not happen now but it will be something we have to keep in mind going forward. Ira > > Sasha -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab weiny2 at llnl.gov From sashak at voltaire.com Wed Jun 24 02:23:43 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 24 Jun 2009 12:23:43 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090623152935.b4f64a1c.weiny2@llnl.gov> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> <20090623190815.GB4324@me> <20090623131939.e464afb2.weiny2@llnl.gov> <20090623204324.GE4324@me> <20090623152935.b4f64a1c.weiny2@llnl.gov> Message-ID: <20090624092343.GA1479@me> On 15:29 Tue 23 Jun , Ira Weiny wrote: > > No the idea of a "plugable routing engine". Ok. Let's try to summarize. 1. We have pretty generic plugin interface right now and users may use it as they want (including routing engines obviously). If improvements are needed in this area (such as plugin(s) reload on the fly or any other) let's go ahead and do it. 2. I don't see why we need yet another plugin interface specific to routing engines - (1) should be good enough for such purposes. 3. Regardless to above I don't think that we as open source community should support any sort of "proprietary additions". Yes, it is allowed for third parties, but it is not our charter. Sasha From myxiplx at googlemail.com Wed Jun 24 04:21:37 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Wed, 24 Jun 2009 12:21:37 +0100 Subject: [ofa-general] Installing OFED 1.4.1 in XenServer Message-ID: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> Hi guys, I'm very new to Linux, and trying to install OFED in XenServer, is anybody able to help out? XenServer is based on CentOS 5.2, and I'm informed that it should be possible to install this version of OFED, but I'm struggling to get the installation to compile. I believe the proper way to compile the drivers is to use the Citrix Driver Development Kit, which includes all the source for XenServer. However that's supplied as a 2GB virtual machine, and runs out of disk space if I try to install the OFED prerequisits. So instead I'm trying to compile it directly on Dom0 of a XenServer. The prerequisits have installed fine, but the install.pl script fails with: "/lib/modules/2.6.18-92.1.10.el5.xs5.0.0.426.647xen/build/scripts is required to build kernel-ib RPM. Please install the corresponding kernel-source or kernel-devel RPM." On the DDK, I can see that those folders all exist, but I can't find how to install those on a live server. I've found kernel-2.6.18.92.1.10.el5.xs5.0.0.426.647.src.rpm on the DDK source disk, but installing that does not help. If I were to simply copy the files from the DDK to the live server, could somebody advise me on what I would need to copy. Do I need anything outside of the /lib/modules/... directory? thanks, Ross From dorfman.eli at gmail.com Wed Jun 24 04:59:13 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 24 Jun 2009 14:59:13 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <20090624092343.GA1479@me> References: <4A30B3DA.30601@bull.net> <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> <20090623190815.GB4324@me> <20090623131939.e464afb2.weiny2@llnl.gov> <20090623204324.GE4324@me> <20090623152935.b4f64a1c.weiny2@llnl.gov> <20090624092343.GA1479@me> Message-ID: <4A421511.6000004@gmail.com> Sasha Khapyorsky wrote: > On 15:29 Tue 23 Jun , Ira Weiny wrote: >> No the idea of a "plugable routing engine". > > Ok. Let's try to summarize. > > 1. We have pretty generic plugin interface right now and users may use > it as they want (including routing engines obviously). If improvements > are needed in this area (such as plugin(s) reload on the fly or any > other) let's go ahead and do it. How can this plugin interface enable configuration of routing engines order? > > 2. I don't see why we need yet another plugin interface specific to > routing engines - (1) should be good enough for such purposes. > > 3. Regardless to above I don't think that we as open source community > should support any sort of "proprietary additions". Yes, it is allowed > for third parties, but it is not our charter. These are not proprietary additions but only an infrastructure to enable proprietary implementations and to reduce opensm dependency (e.g. plugin mysql). Eli From myxiplx at googlemail.com Wed Jun 24 08:09:06 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Wed, 24 Jun 2009 16:09:06 +0100 Subject: [ofa-general] Re: Installing OFED 1.4.1 in XenServer In-Reply-To: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> References: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> Message-ID: <7b160d240906240809v13e46f8er6bc6a8d371c79c3f@mail.gmail.com> Replying to my own email since I've made progress and now have a new question. OFED 1.4.1 compiles and installs find under the XenServer 5.5 DDK. My question now is how do I transfer these drivers to a live server? Are there details anywhere about what files need transferring, and how they should be loaded? Alternatively, do I need to attempt to transfer the relevant kernel source files from the XenServer DDK to a live server? thanks, Ross On Wed, Jun 24, 2009 at 12:21 PM, Ross Smith wrote: > Hi guys, > > I'm very new to Linux, and trying to install OFED in XenServer, is > anybody able to help out? > > XenServer is based on CentOS 5.2, and I'm informed that it should be > possible to install this version of OFED, but I'm struggling to get > the installation to compile. > > I believe the proper way to compile the drivers is to use the Citrix > Driver Development Kit, which includes all the source for XenServer. > However that's supplied as a 2GB virtual machine, and runs out of disk > space if I try to install the OFED prerequisits. > > So instead I'm trying to compile it directly on Dom0 of a XenServer. > The prerequisits have installed fine, but the install.pl script fails > with: > > "/lib/modules/2.6.18-92.1.10.el5.xs5.0.0.426.647xen/build/scripts is > required to build kernel-ib RPM. > Please install the corresponding kernel-source or kernel-devel RPM." > > On the DDK, I can see that those folders all exist, but I can't find > how to install those on a live server.  I've found > kernel-2.6.18.92.1.10.el5.xs5.0.0.426.647.src.rpm on the DDK source > disk, but installing that does not help. > > If I were to simply copy the files from the DDK to the live server, > could somebody advise me on what I would need to copy.  Do I need > anything outside of the /lib/modules/... directory? > > thanks, > > Ross > From vlad at dev.mellanox.co.il Wed Jun 24 08:30:28 2009 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Wed, 24 Jun 2009 18:30:28 +0300 Subject: [ofa-general] Re: Installing OFED 1.4.1 in XenServer In-Reply-To: <7b160d240906240809v13e46f8er6bc6a8d371c79c3f@mail.gmail.com> References: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> <7b160d240906240809v13e46f8er6bc6a8d371c79c3f@mail.gmail.com> Message-ID: <4A424694.5030706@dev.mellanox.co.il> Ross Smith wrote: > Replying to my own email since I've made progress and now have a new question. > > OFED 1.4.1 compiles and installs find under the XenServer 5.5 DDK. My > question now is how do I transfer these drivers to a live server? > > Are there details anywhere about what files need transferring, and how > they should be loaded? > > Alternatively, do I need to attempt to transfer the relevant kernel > source files from the XenServer DDK to a live server? > > thanks, > > Ross > Hi Ross, After installing OFED-1.4.1 you will have binary RPMs under OFED-1.4.1/RPMS/... directory. So, to install OFED-1.4.1 on the live server you should run the same install command (from the same OFED-1.4.1 directory; you can copy it to the live server if it is not accessible through NFS) on your server and it will install binary RPMs that were previously created. It will not require kernel sources on the live server. Regards, Vladimir > > > On Wed, Jun 24, 2009 at 12:21 PM, Ross Smith wrote: > >> Hi guys, >> >> I'm very new to Linux, and trying to install OFED in XenServer, is >> anybody able to help out? >> >> XenServer is based on CentOS 5.2, and I'm informed that it should be >> possible to install this version of OFED, but I'm struggling to get >> the installation to compile. >> >> I believe the proper way to compile the drivers is to use the Citrix >> Driver Development Kit, which includes all the source for XenServer. >> However that's supplied as a 2GB virtual machine, and runs out of disk >> space if I try to install the OFED prerequisits. >> >> So instead I'm trying to compile it directly on Dom0 of a XenServer. >> The prerequisits have installed fine, but the install.pl script fails >> with: >> >> "/lib/modules/2.6.18-92.1.10.el5.xs5.0.0.426.647xen/build/scripts is >> required to build kernel-ib RPM. >> Please install the corresponding kernel-source or kernel-devel RPM." >> >> On the DDK, I can see that those folders all exist, but I can't find >> how to install those on a live server. I've found >> kernel-2.6.18.92.1.10.el5.xs5.0.0.426.647.src.rpm on the DDK source >> disk, but installing that does not help. >> >> If I were to simply copy the files from the DDK to the live server, >> could somebody advise me on what I would need to copy. Do I need >> anything outside of the /lib/modules/... directory? >> >> thanks, >> >> Ross >> >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From myxiplx at googlemail.com Wed Jun 24 08:41:18 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Wed, 24 Jun 2009 16:41:18 +0100 Subject: [ofa-general] Re: Installing OFED 1.4.1 in XenServer In-Reply-To: <4A424694.5030706@dev.mellanox.co.il> References: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> <7b160d240906240809v13e46f8er6bc6a8d371c79c3f@mail.gmail.com> <4A424694.5030706@dev.mellanox.co.il> Message-ID: <7b160d240906240841g6e8846b9w45a295965a6c7335@mail.gmail.com> Thanks Vladimir, that sounds very neat and tidy :-) I'll be sure to let the list know how my testing goes with XenServer. On Wed, Jun 24, 2009 at 4:30 PM, Vladimir Sokolovsky wrote: > Ross Smith wrote: >> >> Replying to my own email since I've made progress and now have a new >> question. >> >> OFED 1.4.1 compiles and installs find under the XenServer 5.5 DDK.  My >> question now is how do I transfer these drivers to a live server? >> >> Are there details anywhere about what files need transferring, and how >> they should be loaded? >> >> Alternatively, do I need to attempt to transfer the relevant kernel >> source files from the XenServer DDK to a live server? >> >> thanks, >> >> Ross >> > > Hi Ross, > After installing OFED-1.4.1 you will have binary RPMs under > OFED-1.4.1/RPMS/... directory. > So, to install OFED-1.4.1 on the live server you should run the > same install command (from the same OFED-1.4.1 directory; > you can copy it to the live server if it is not accessible through NFS) > on your server and it will install binary RPMs that were previously > created. > It will not require kernel sources on the live server. > > Regards, > Vladimir > >> >> >> On Wed, Jun 24, 2009 at 12:21 PM, Ross Smith >> wrote: >> >>> >>> Hi guys, >>> >>> I'm very new to Linux, and trying to install OFED in XenServer, is >>> anybody able to help out? >>> >>> XenServer is based on CentOS 5.2, and I'm informed that it should be >>> possible to install this version of OFED, but I'm struggling to get >>> the installation to compile. >>> >>> I believe the proper way to compile the drivers is to use the Citrix >>> Driver Development Kit, which includes all the source for XenServer. >>> However that's supplied as a 2GB virtual machine, and runs out of disk >>> space if I try to install the OFED prerequisits. >>> >>> So instead I'm trying to compile it directly on Dom0 of a XenServer. >>> The prerequisits have installed fine, but the install.pl script fails >>> with: >>> >>> "/lib/modules/2.6.18-92.1.10.el5.xs5.0.0.426.647xen/build/scripts is >>> required to build kernel-ib RPM. >>> Please install the corresponding kernel-source or kernel-devel RPM." >>> >>> On the DDK, I can see that those folders all exist, but I can't find >>> how to install those on a live server.  I've found >>> kernel-2.6.18.92.1.10.el5.xs5.0.0.426.647.src.rpm on the DDK source >>> disk, but installing that does not help. >>> >>> If I were to simply copy the files from the DDK to the live server, >>> could somebody advise me on what I would need to copy.  Do I need >>> anything outside of the /lib/modules/... directory? >>> >>> thanks, >>> >>> Ross >>> >>> >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> > > From monis at Voltaire.COM Wed Jun 24 08:55:43 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Wed, 24 Jun 2009 18:55:43 +0300 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: References: <4A3E45D3.3040405@Voltaire.COM> Message-ID: <4A424C7F.4010006@Voltaire.COM> Roland Dreier wrote: > > This patch was sent on Apr, 23 2009. Nothing was changed except for fixing offsets and verifying it under 2.6.30 > > Jason had some concerns about this as I recall -- did you every reach consensus? > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > I believe that Jason and I still disagree but... Jason suggests that I implement this feature with netlink. This approach might have an advantage but if I understand it right this approach requires a patch also to some user application in order to take advantage of this patch. Also, I think that there is a value for a virtual text file under debugfs to monitor rdma_cm connections easily and without any other special app besides 'cat' (just like in IPoIB). Finally, this implementation doesn't contradict netlink implementation in the future (and it won't be the first time). BTW, this feature was suggested several times (first under procfs and later under debugfs) and Jason is the only one that objected to the concept, other agreed that the place should be a file system and only had comments to the implementation itself and to these comments I already answered. From purdy at sgi.com Wed Jun 24 08:58:05 2009 From: purdy at sgi.com (Dale Purdy) Date: Wed, 24 Jun 2009 10:58:05 -0500 Subject: [ofa-general] Some confusion about DOR algorithm In-Reply-To: <91fe68d50906222246o73bad6fk1872bdb9cdf0df82@mail.gmail.com> References: <91fe68d50906222246o73bad6fk1872bdb9cdf0df82@mail.gmail.com> Message-ID: <20090624155805.GA6407@sgi.com> On Tue, Jun 23, 2009 at 01:46:04PM +0800, Jordan wrote: > I have some confusion about the DOR algorithm , in this algorithm , It > chooses the outport throuth a "for" iteration :for (i = start_from; i < > start_from + num_ports; i++). So the small port_num will be choosed > firstly. > But not always the small port_num connects the X-dimension or Y-dimension.In > other words , using this method we can't always conform the DOR rule. > By the way , in the specification, it says " Paths are grown from a > destination back to a source using the lowest dimension (port) of available > paths at each step. ". Maybe ,the lowest port connect to the X-dimension in > one switch, but it connects to the Y-dimension in another switch. So when > routing the package from A to B, the dimension may be X -Y- Y- X- X- Y, not > X-X-X- Y-Y-Y . This will break the DOR rule. > > Can anybody explain this situation? You will also note that the description of the DOR Routing Algorithm indicates that the same port must be used throughout the fabric to represent the hypercube dimension. The fabric must be consistently cabled such that each switch to switch port # is in one to one correspondence with a hypercube dimension. Similarly for (consecutive) port pairs representing a dimension of a mesh. With a fabric cabled as described, following port order is the same as following dimension order. Perhaps someone could relax these requirements using the new osm_mesh.c routines. -- Dale From chocapiiic.tiery at gmail.com Wed Jun 24 09:03:58 2009 From: chocapiiic.tiery at gmail.com (Thierry) Date: Wed, 24 Jun 2009 18:03:58 +0200 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <4A424C7F.4010006@Voltaire.COM> References: <4A3E45D3.3040405@Voltaire.COM> <4A424C7F.4010006@Voltaire.COM> Message-ID: <8d9c773c0906240903h51872440g9f07889379eb8137@mail.gmail.com> Hi, I have already worked on monitoring rdma connection from kernel (and also user) part. I used netlink to send data to userland and I made a simple program to get theses information and print it. As it is a very simple implementation (I am still a student and I just started to work in kernelland few months ago), it should not be a very strong code but it can give ideas, if never you need... Kind regards, Thierry On Wed, Jun 24, 2009 at 5:55 PM, Moni Shoua wrote: > Roland Dreier wrote: >>  > This patch was sent on Apr, 23 2009. Nothing was changed except for fixing offsets and verifying it under 2.6.30 >> >> Jason had some concerns about this as I recall -- did you every reach consensus? >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> > > I believe that Jason and I still disagree but... > > Jason suggests that I implement this feature with netlink. This approach might have an advantage but if I understand it right this approach requires a patch also to some user application in order to take advantage of this patch. > > Also, I think that there is a value for a virtual text file under debugfs to monitor rdma_cm connections easily and without any other special app besides 'cat' (just like in IPoIB). > > Finally, this implementation doesn't contradict  netlink implementation in the future (and it won't be the first time). > > BTW, this feature was suggested several times (first under procfs and later under debugfs) and Jason is the only one that objected to the concept, other agreed that the place should be a file system and only had comments to the implementation itself and to these comments I already answered. > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From monis at Voltaire.COM Wed Jun 24 09:15:12 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Wed, 24 Jun 2009 19:15:12 +0300 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <8d9c773c0906240903h51872440g9f07889379eb8137@mail.gmail.com> References: <4A3E45D3.3040405@Voltaire.COM> <4A424C7F.4010006@Voltaire.COM> <8d9c773c0906240903h51872440g9f07889379eb8137@mail.gmail.com> Message-ID: <4A425110.6070709@Voltaire.COM> Thierry wrote: > Hi, > > I have already worked on monitoring rdma connection from kernel (and > also user) part. I used netlink to send data to userland and I made a > simple program to get theses information and print it. As it is a very > simple implementation (I am still a student and I just started to work > in kernelland few months ago), it should not be a very strong code but > it can give ideas, if never you need... > > Kind regards, > > Thierry Hi It would be nice if you send your patch also. I don't have much experience with netlink so it would be interesting for me to look at you work. After all, as I said, I'm not against netlink and I think that both ways can live together. thanks MoniS From myxiplx at googlemail.com Wed Jun 24 10:09:49 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Wed, 24 Jun 2009 18:09:49 +0100 Subject: [ofa-general] Re: Installing OFED 1.4.1 in XenServer In-Reply-To: <7b160d240906240841g6e8846b9w45a295965a6c7335@mail.gmail.com> References: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> <7b160d240906240809v13e46f8er6bc6a8d371c79c3f@mail.gmail.com> <4A424694.5030706@dev.mellanox.co.il> <7b160d240906240841g6e8846b9w45a295965a6c7335@mail.gmail.com> Message-ID: <7b160d240906241009m353fcf3fnd209ff08fac48fb7@mail.gmail.com> Me again, Well, the installation went pretty well but there was an error during the installation with iscsi: "Failed to install iscsi-initiator-utils RPM See /tmp/OFED.6760.logs/iscsi-initiator-utils.rpminstall.log " Removing open-iscsi seemed to get around this, and the version included with OFED is slighly newer, so fingers crossed it won't cause any problems: # yum remove open-iscsi Tomorrow I'll see if it plays nicely with Xen. Ross On Wed, Jun 24, 2009 at 4:41 PM, Ross Smith wrote: > Thanks Vladimir, that sounds very neat and tidy :-) > > I'll be sure to let the list know how my testing goes with XenServer. > > > > On Wed, Jun 24, 2009 at 4:30 PM, Vladimir > Sokolovsky wrote: >> Ross Smith wrote: >>> >>> Replying to my own email since I've made progress and now have a new >>> question. >>> >>> OFED 1.4.1 compiles and installs find under the XenServer 5.5 DDK.  My >>> question now is how do I transfer these drivers to a live server? >>> >>> Are there details anywhere about what files need transferring, and how >>> they should be loaded? >>> >>> Alternatively, do I need to attempt to transfer the relevant kernel >>> source files from the XenServer DDK to a live server? >>> >>> thanks, >>> >>> Ross >>> >> >> Hi Ross, >> After installing OFED-1.4.1 you will have binary RPMs under >> OFED-1.4.1/RPMS/... directory. >> So, to install OFED-1.4.1 on the live server you should run the >> same install command (from the same OFED-1.4.1 directory; >> you can copy it to the live server if it is not accessible through NFS) >> on your server and it will install binary RPMs that were previously >> created. >> It will not require kernel sources on the live server. >> >> Regards, >> Vladimir >> >>> >>> >>> On Wed, Jun 24, 2009 at 12:21 PM, Ross Smith >>> wrote: >>> >>>> >>>> Hi guys, >>>> >>>> I'm very new to Linux, and trying to install OFED in XenServer, is >>>> anybody able to help out? >>>> >>>> XenServer is based on CentOS 5.2, and I'm informed that it should be >>>> possible to install this version of OFED, but I'm struggling to get >>>> the installation to compile. >>>> >>>> I believe the proper way to compile the drivers is to use the Citrix >>>> Driver Development Kit, which includes all the source for XenServer. >>>> However that's supplied as a 2GB virtual machine, and runs out of disk >>>> space if I try to install the OFED prerequisits. >>>> >>>> So instead I'm trying to compile it directly on Dom0 of a XenServer. >>>> The prerequisits have installed fine, but the install.pl script fails >>>> with: >>>> >>>> "/lib/modules/2.6.18-92.1.10.el5.xs5.0.0.426.647xen/build/scripts is >>>> required to build kernel-ib RPM. >>>> Please install the corresponding kernel-source or kernel-devel RPM." >>>> >>>> On the DDK, I can see that those folders all exist, but I can't find >>>> how to install those on a live server.  I've found >>>> kernel-2.6.18.92.1.10.el5.xs5.0.0.426.647.src.rpm on the DDK source >>>> disk, but installing that does not help. >>>> >>>> If I were to simply copy the files from the DDK to the live server, >>>> could somebody advise me on what I would need to copy.  Do I need >>>> anything outside of the /lib/modules/... directory? >>>> >>>> thanks, >>>> >>>> Ross >>>> >>>> >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general >>> >>> >> >> > From jgunthorpe at obsidianresearch.com Wed Jun 24 10:55:27 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 24 Jun 2009 11:55:27 -0600 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <4A424C7F.4010006@Voltaire.COM> References: <4A3E45D3.3040405@Voltaire.COM> <4A424C7F.4010006@Voltaire.COM> Message-ID: <20090624175527.GU18150@obsidianresearch.com> On Wed, Jun 24, 2009 at 06:55:43PM +0300, Moni Shoua wrote: > I believe that Jason and I still disagree but... > Jason suggests that I implement this feature with netlink. This > approach might have an advantage but if I understand it right this > approach requires a patch also to some user application in order to > take advantage of this patch. The kernel devs have made it clear that the preferred way to export this information is through netlink and/or a file in /proc/net/.. You never got an answer if /proc/net is truely discouraged or not. debugfs is completely useless because it cannot be used for actual end-user interrogation and the rules are it should not be used as a stable kernel-user interface. (ie is NOT a new dumping ground like /proc/ was) > Also, I think that there is a value for a virtual text file under > debugfs to monitor rdma_cm connections easily and without any other > special app besides 'cat' (just like in IPoIB). If you have the netlink based program there really is no point in including something under debugfs. It is just bloat. > Finally, this implementation doesn't contradict netlink > implementation in the future (and it won't be the first time). As I said before, the stack has reached a level of maturity that new stuff going in should meaningfully 'move the ball forward' toward a mature and complete stack. In the context of state reporting that means a stable user interface that programs like lsof, netstat, etc, can rely upon. Putting something under debugfs clearly does nothing to advance that goal. Further, there is clearly alot of state information we should be exporting to userspace. RDMA-CM stuff is only a tiny portion. netlink is the kernel devs answer to all of this. Jason From weiny2 at llnl.gov Wed Jun 24 18:04:05 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 24 Jun 2009 18:04:05 -0700 Subject: [ofa-general] [PATCH] iblinkinfo: optomize -D operation Message-ID: <20090624180405.076e8555.weiny2@llnl.gov> From: Ira Weiny Date: Thu, 18 Jun 2009 14:07:13 -0700 Subject: [PATCH] iblinkinfo: optomize -D operation Remove -f option and combine with -D only scan part of the fabric when -D is specified. add -a option which allows for printing "all" of the switches found when performing a partial scan Signed-off-by: Ira Weiny --- infiniband-diags/src/iblinkinfo.c | 58 +++++++++++++++++++----------------- 1 files changed, 31 insertions(+), 27 deletions(-) diff --git a/infiniband-diags/src/iblinkinfo.c b/infiniband-diags/src/iblinkinfo.c index ae6da27..f2dca6b 100644 --- a/infiniband-diags/src/iblinkinfo.c +++ b/infiniband-diags/src/iblinkinfo.c @@ -59,7 +59,7 @@ static nn_map_t *node_name_map = NULL; static uint64_t guid = 0; static char *guid_str = NULL; static char *dr_path = NULL; -static char *from = NULL; +static int all = 0; static int hops = 0; static int down_links_only = 0; @@ -240,8 +240,8 @@ static int process_opt(void *context, int ch, char *optarg) case 'D': dr_path = strdup(optarg); break; - case 'f': - from = strdup(optarg); + case 'a': + all = 1; break; case 'n': hops = (int)strtol(optarg, NULL, 0); @@ -271,8 +271,8 @@ static int process_opt(void *context, int ch, char *optarg) int main(int argc, char **argv) { - int rc = 0; + int resolved = -1; ibnd_fabric_t *fabric = NULL; struct ibmad_port *ibmad_port; ib_portid_t port_id = {0}; @@ -282,7 +282,7 @@ main(int argc, char **argv) { "node-name-map", 1, 1, "", "node name map file" }, { "switch", 'S', 1, "", "query only (hex format)"}, { "Direct", 'D', 1, "", "query only node specified by "}, - { "from", 'f', 1, "", "specify node to start \"\""}, + { "all", 'a', 0, NULL, "print all switches found in a partial fabric scan"}, { "hops", 'n', 1, "", "Number of hops to include away from specified node"}, { "down", 'd', 0, NULL, "print only down links"}, { "line", 'l', 0, NULL, "(line mode) print all information for each link on a single line"}, @@ -293,7 +293,7 @@ main(int argc, char **argv) }; char usage_args[] = ""; - ibdiag_process_opts(argc, argv, "sDLG", "SDfndlpgR", opts, process_opt, + ibdiag_process_opts(argc, argv, "sDLG", "SDandlpgR", opts, process_opt, usage_args, NULL); argc -= optind; @@ -310,49 +310,53 @@ main(int argc, char **argv) node_name_map = open_node_name_map(node_name_map_file); - if (from) { + if (dr_path) { /* only scan part of the fabric */ - str2drpath(&(port_id.drpath), from, 0, 0); - if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, &port_id, hops)) == NULL) { - fprintf(stderr, "discover failed\n"); - rc = 1; - goto close_port; - } - guid = 0; + if ((resolved = ib_resolve_portid_str_via(&port_id, dr_path, IB_DEST_DRPATH, + NULL, ibmad_port)) < 0) + IBWARN("Failed to resolve %s; attempting full scan\n", + dr_path); } else if (guid_str) { - if (ib_resolve_portid_str_via(&port_id, guid_str, IB_DEST_GUID, - NULL, ibmad_port) >= 0) { - if ((fabric = ibnd_discover_fabric(ibmad_port, - ibd_timeout, &port_id, 1)) == NULL) - IBWARN("Single node discover failed; attempting full scan\n"); - } else + if ((resolved = ib_resolve_portid_str_via(&port_id, guid_str, IB_DEST_GUID, + NULL, ibmad_port)) < 0) IBWARN("Failed to resolve %s; attempting full scan\n", guid_str); } - if (!fabric) /* do a full scan */ + if (resolved >= 0) + if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, &port_id, + hops)) == NULL) + IBWARN("Single node discover failed; attempting full scan\n"); + + if (!fabric) if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, NULL, -1)) == NULL) { fprintf(stderr, "discover failed\n"); rc = 1; goto close_port; } - if (guid_str) { + if (!all && guid_str) { ibnd_node_t *sw = ibnd_find_node_guid(fabric, guid); if (sw) print_switch(sw, NULL); else fprintf(stderr, "Failed to find switch: %s\n", guid_str); - } else if (dr_path) { - ibnd_node_t *sw = ibnd_find_node_dr(fabric, dr_path); + } else if (!all && dr_path) { + ibnd_node_t *sw = NULL; + uint8_t ni[IB_SMP_DATA_SIZE]; + + if (!smp_query_via(ni, &port_id, IB_ATTR_NODE_INFO, 0, + ibd_timeout, ibmad_port)) + return -1; + mad_decode_field(ni, IB_NODE_GUID_F, &(guid)); + + sw = ibnd_find_node_guid(fabric, guid); if (sw) print_switch(sw, NULL); else fprintf(stderr, "Failed to find switch: %s\n", dr_path); - print_switch(sw, NULL); - } else { + } else ibnd_iter_nodes_type(fabric, print_switch, IB_NODE_SWITCH, NULL); - } ibnd_destroy_fabric(fabric); -- 1.5.4.5 From weiny2 at llnl.gov Wed Jun 24 18:04:06 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 24 Jun 2009 18:04:06 -0700 Subject: [ofa-general] [PATCH] ibqueryerrors: optomize -D operation Message-ID: <20090624180406.fa03303a.weiny2@llnl.gov> From: Ira Weiny Date: Wed, 24 Jun 2009 17:52:37 -0700 Subject: [PATCH] ibqueryerrors: optomize -D operation only scan part of the fabric when -D is specified. Signed-off-by: Ira Weiny --- infiniband-diags/src/ibqueryerrors.c | 35 ++++++++++++++++++++++++--------- 1 files changed, 25 insertions(+), 10 deletions(-) diff --git a/infiniband-diags/src/ibqueryerrors.c b/infiniband-diags/src/ibqueryerrors.c index 36b8050..d443f1d 100644 --- a/infiniband-diags/src/ibqueryerrors.c +++ b/infiniband-diags/src/ibqueryerrors.c @@ -388,6 +388,8 @@ static int process_opt(void *context, int ch, char *optarg) int main(int argc, char **argv) { + int resolved = -1; + ib_portid_t portid = {0}; int rc = 0; ibnd_fabric_t *fabric = NULL; @@ -422,18 +424,23 @@ main(int argc, char **argv) node_name_map = open_node_name_map(node_name_map_file); - if (switch_guid) { - /* limit the scan the fabric around the target */ - ib_portid_t portid = {0}; - - if (ib_resolve_portid_str_via(&portid, switch_guid_str, IB_DEST_GUID, - ibd_sm_id, ibmad_port) >= 0) { - if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, &portid, 1)) == NULL) - IBWARN("Single node discover failed; attempting full scan\n"); - } else + /* limit the scan the fabric around the target */ + if (dr_path) { + if ((resolved = ib_resolve_portid_str_via(&portid, dr_path, IB_DEST_DRPATH, + NULL, ibmad_port)) < 0) + IBWARN("Failed to resolve %s; attempting full scan\n", + dr_path); + } else if (switch_guid_str) { + if ((resolved = ib_resolve_portid_str_via(&portid, switch_guid_str, IB_DEST_GUID, + ibd_sm_id, ibmad_port)) >= 0) IBWARN("Failed to resolve %s; attempting full scan\n", switch_guid_str); } + if (resolved >= 0) + if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, &portid, + 0)) == NULL) + IBWARN("Single node discover failed; attempting full scan\n"); + if (!fabric) /* do a full scan */ if ((fabric = ibnd_discover_fabric(ibmad_port, ibd_timeout, NULL, -1)) == NULL) { fprintf(stderr, "discover failed\n"); @@ -443,7 +450,7 @@ main(int argc, char **argv) report_suppressed(); - if (switch_guid) { + if (switch_guid_str) { ibnd_node_t *node = ibnd_find_node_guid(fabric, switch_guid); if (node) print_node(node, NULL); @@ -451,6 +458,14 @@ main(int argc, char **argv) fprintf(stderr, "Failed to find node: %s\n", switch_guid_str); } else if (dr_path) { ibnd_node_t *node = ibnd_find_node_dr(fabric, dr_path); + uint8_t ni[IB_SMP_DATA_SIZE]; + + if (!smp_query_via(ni, &portid, IB_ATTR_NODE_INFO, 0, + ibd_timeout, ibmad_port)) + return -1; + mad_decode_field(ni, IB_NODE_GUID_F, &(switch_guid)); + + node = ibnd_find_node_guid(fabric, switch_guid); if (node) print_node(node, NULL); else -- 1.5.4.5 From weiny2 at llnl.gov Wed Jun 24 18:04:08 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 24 Jun 2009 18:04:08 -0700 Subject: [ofa-general] [PATCH] ibprintca.pl: fix corner case: print switch information for CA's found at the end of the cache file. Message-ID: <20090624180408.1550f55c.weiny2@llnl.gov> From: Ira Weiny Date: Tue, 23 Jun 2009 13:29:19 -0700 Subject: [PATCH] ibprintca.pl: fix corner case: print switch information for CA's found at the end of the cache file. Signed-off-by: Ira Weiny --- infiniband-diags/scripts/ibprintca.pl | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/infiniband-diags/scripts/ibprintca.pl b/infiniband-diags/scripts/ibprintca.pl index 3cac9b4..9832ac0 100755 --- a/infiniband-diags/scripts/ibprintca.pl +++ b/infiniband-diags/scripts/ibprintca.pl @@ -122,6 +122,13 @@ sub main } } + + if ($in_hca eq "yes") { + foreach my $port (sort { $a <=> $b } (keys %ports)) { + print $ports{$port}; + } + } + if ($found_hca == 0) { die "\"$target_hca\" not found\n" . " Try running with the \"-R\" option.\n" . -- 1.5.4.5 From rdreier at cisco.com Wed Jun 24 23:18:55 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 24 Jun 2009 23:18:55 -0700 Subject: [ofa-general][PATCH 2/2] mlx4: ConnectX multi functional device support In-Reply-To: <4A3A34F5.9030800@mellanox.co.il> (Yevgeny Petrilin's message of "Thu, 18 Jun 2009 15:37:09 +0300") References: <4A3A34F5.9030800@mellanox.co.il> Message-ID: > MT26468 (0x6764) device can open multiple physical functions. > The current driver can only work with one (primary) pf. > For all other functions, QUERY_FW command would fail with > CMD_STAT_MULTI_FUNC_REQ error code. We should not work on those > devices, but they should remain in the driver's ownership. Seems this patch should really be 1/2, since we want the driver to be able to handle multi-func devices before we add the PCI ID for such devices. Also, it didn't occur to me before, but why does the driver need to keep ownership of the non-primary functions? It seems we could avoid having the NOT_PRIME flag and all of that if we just gave up on a device when QUERY_FW told us it wasn't the primary function. Also from my naive point of view at least, it seems your hardware interface could be simpler for software to handle if you just used a different PCI ID for the non-primary physical function. Not sure if it's too late to change that (and maybe there's a reason I'm missing to use the same PCI ID for functions that behave differently and require different driver behavior to handle them??). - R. From weiny2 at llnl.gov Wed Jun 24 18:22:35 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 24 Jun 2009 18:22:35 -0700 Subject: [ofa-general] [PATCH] iblinkinfo, ibqueryerrors: clean up ibdiag_process_opts call Message-ID: <20090624182235.601261e3.weiny2@llnl.gov> From: Ira Weiny Date: Wed, 24 Jun 2009 18:22:14 -0700 Subject: [PATCH] iblinkinfo, ibqueryerrors: clean up ibdiag_process_opts call Signed-off-by: Ira Weiny --- infiniband-diags/src/iblinkinfo.c | 2 +- infiniband-diags/src/ibqueryerrors.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/infiniband-diags/src/iblinkinfo.c b/infiniband-diags/src/iblinkinfo.c index f2dca6b..bba5c5c 100644 --- a/infiniband-diags/src/iblinkinfo.c +++ b/infiniband-diags/src/iblinkinfo.c @@ -293,7 +293,7 @@ main(int argc, char **argv) }; char usage_args[] = ""; - ibdiag_process_opts(argc, argv, "sDLG", "SDandlpgR", opts, process_opt, + ibdiag_process_opts(argc, argv, NULL, "SDandlpgRGL", opts, process_opt, usage_args, NULL); argc -= optind; diff --git a/infiniband-diags/src/ibqueryerrors.c b/infiniband-diags/src/ibqueryerrors.c index d443f1d..09f57c5 100644 --- a/infiniband-diags/src/ibqueryerrors.c +++ b/infiniband-diags/src/ibqueryerrors.c @@ -409,7 +409,7 @@ main(int argc, char **argv) }; char usage_args[] = ""; - ibdiag_process_opts(argc, argv, "sDLG", "scnSrR", opts, process_opt, + ibdiag_process_opts(argc, argv, NULL, "scnSrRDGL", opts, process_opt, usage_args, NULL); argc -= optind; -- 1.5.4.5 From rdreier at cisco.com Wed Jun 24 23:14:30 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 24 Jun 2009 23:14:30 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git In-Reply-To: <4A4309B0.7020400@mellanox.co.il> (Tziporet Koren's message of "Thu, 25 Jun 2009 08:22:56 +0300") References: <4A4309B0.7020400@mellanox.co.il> Message-ID: > What about the patches from Yevgeny supporting our new ConnextX HW > coming soon? > > I talk about: > http://lists.openfabrics.org/pipermail/general/2009-June/060225.html > http://lists.openfabrics.org/pipermail/general/2009-June/060226.html Sorry, missed that going by the first time, will reply to the messages. From tziporet at dev.mellanox.co.il Wed Jun 24 22:22:56 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 25 Jun 2009 08:22:56 +0300 Subject: [ofa-general] [GIT PULL] please pull infiniband.git In-Reply-To: References: Message-ID: <4A4309B0.7020400@mellanox.co.il> Roland Dreier wrote: > Linus, please pull from > > master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus > > This tree is also available from kernel.org mirrors at: > > git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus > > This has a few more fixes for 2.6.31-rc1; however everything here > should be a fix appropriate for -rc2 as well, so if this misses the > end of the merge window then I think it should still be OK to pull. > > Roland What about the patches from Yevgeny supporting our new ConnextX HW coming soon? I talk about: http://lists.openfabrics.org/pipermail/general/2009-June/060225.html http://lists.openfabrics.org/pipermail/general/2009-June/060226.html Since device ID is already in pci.ids I think it is important to have this code too, otherwise the driver will not load :-( Thanks Tziporet From vlad at lists.openfabrics.org Thu Jun 25 02:26:37 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 25 Jun 2009 02:26:37 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090625-0200 daily build status Message-ID: <20090625092637.E41D21020286@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090625-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From myxiplx at googlemail.com Thu Jun 25 03:41:26 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Thu, 25 Jun 2009 11:41:26 +0100 Subject: [ofa-general] Re: Installing OFED 1.4.1 in XenServer In-Reply-To: <7b160d240906241009m353fcf3fnd209ff08fac48fb7@mail.gmail.com> References: <7b160d240906240421j79a621c1n75d8dbfe3e235152@mail.gmail.com> <7b160d240906240809v13e46f8er6bc6a8d371c79c3f@mail.gmail.com> <4A424694.5030706@dev.mellanox.co.il> <7b160d240906240841g6e8846b9w45a295965a6c7335@mail.gmail.com> <7b160d240906241009m353fcf3fnd209ff08fac48fb7@mail.gmail.com> Message-ID: <7b160d240906250341i20511558xa7eed8e20b4d0b25@mail.gmail.com> Ok, Infiniband is working fine with XenServer, but I'm getting poor performance from NFS so want to try NFS-RDMA. It doesn't look like the default install has included this, so I could do with some help getting this running. My first attempt was to run install.pl again, and choose all modules. That needed some more prerequisites, so I installed pciutils-devel, tcl-devel, tk and tcsh. However building all with install.pl now fails with the error: "Failed to build ofa_kernel RPM See /tmp/OFED.10263.logs/ofa_kernel.rpmbuild.log" I then tried a custom build, but no matter what I tried, it didn't seem to register that I had chosen nfs-rdma (I'm guessing though that it's included in kernel-ib and just isn't shown on the confirmation screen). However, the custom build also ended with that same error, even after I removed the extre prerequisites. I eventually did get it to build, but I don't remember which pieces I picked from the custom build, and there are so many I don't actually know which I do or don't need. I also now can't work out which .rpm file it created for nfsrdma. Can anybody help me understand what I need to do to get the .rpm for nfsrdma so I can install it on the live server for testing? All I want are the default modules plus nfsrdma. thanks, Ross PS. In case it helps, the last 20 lines of that error log are: # tail /tmp/OFED.10263.logs/ofa_kernel.rpmbuild.log -n 20 from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband/ulp/qlgc_vnic/vnic_main.c:35: include/linux/sysctl.h:58:1: warning: this is the location of the previous definition In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h:38, from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband/ulp/qlgc_vnic/vnic_main.c:46: /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h:49: error: redefinition of 'struct mc_info' make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband/ulp/qlgc_vnic/vnic_main.o] Error 1 make[3]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband/ulp/qlgc_vnic] Error 2 make[2]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/drivers/infiniband] Error 2 make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1] Error 2 make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.1.6.el5.xs5.5.0.496.1012xen-i686' make: *** [kernel] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.6235 (%build) RPM build errors: user vlad does not exist - using root group vlad does not exist - using root user vlad does not exist - using root group vlad does not exist - using root Bad exit status from /var/tmp/rpm-tmp.6235 (%build) On Wed, Jun 24, 2009 at 6:09 PM, Ross Smith wrote: > Me again, > > Well, the installation went pretty well but there was an error during > the installation with iscsi: > "Failed to install iscsi-initiator-utils RPM > See /tmp/OFED.6760.logs/iscsi-initiator-utils.rpminstall.log " > > Removing open-iscsi seemed to get around this, and the version > included with OFED is slighly newer, so fingers crossed it won't cause > any problems: > # yum remove open-iscsi > > Tomorrow I'll see if it plays nicely with Xen. > > Ross > > > On Wed, Jun 24, 2009 at 4:41 PM, Ross Smith wrote: >> Thanks Vladimir, that sounds very neat and tidy :-) >> >> I'll be sure to let the list know how my testing goes with XenServer. >> >> >> >> On Wed, Jun 24, 2009 at 4:30 PM, Vladimir >> Sokolovsky wrote: >>> Ross Smith wrote: >>>> >>>> Replying to my own email since I've made progress and now have a new >>>> question. >>>> >>>> OFED 1.4.1 compiles and installs find under the XenServer 5.5 DDK.  My >>>> question now is how do I transfer these drivers to a live server? >>>> >>>> Are there details anywhere about what files need transferring, and how >>>> they should be loaded? >>>> >>>> Alternatively, do I need to attempt to transfer the relevant kernel >>>> source files from the XenServer DDK to a live server? >>>> >>>> thanks, >>>> >>>> Ross >>>> >>> >>> Hi Ross, >>> After installing OFED-1.4.1 you will have binary RPMs under >>> OFED-1.4.1/RPMS/... directory. >>> So, to install OFED-1.4.1 on the live server you should run the >>> same install command (from the same OFED-1.4.1 directory; >>> you can copy it to the live server if it is not accessible through NFS) >>> on your server and it will install binary RPMs that were previously >>> created. >>> It will not require kernel sources on the live server. >>> >>> Regards, >>> Vladimir >>> >>>> >>>> >>>> On Wed, Jun 24, 2009 at 12:21 PM, Ross Smith >>>> wrote: >>>> >>>>> >>>>> Hi guys, >>>>> >>>>> I'm very new to Linux, and trying to install OFED in XenServer, is >>>>> anybody able to help out? >>>>> >>>>> XenServer is based on CentOS 5.2, and I'm informed that it should be >>>>> possible to install this version of OFED, but I'm struggling to get >>>>> the installation to compile. >>>>> >>>>> I believe the proper way to compile the drivers is to use the Citrix >>>>> Driver Development Kit, which includes all the source for XenServer. >>>>> However that's supplied as a 2GB virtual machine, and runs out of disk >>>>> space if I try to install the OFED prerequisits. >>>>> >>>>> So instead I'm trying to compile it directly on Dom0 of a XenServer. >>>>> The prerequisits have installed fine, but the install.pl script fails >>>>> with: >>>>> >>>>> "/lib/modules/2.6.18-92.1.10.el5.xs5.0.0.426.647xen/build/scripts is >>>>> required to build kernel-ib RPM. >>>>> Please install the corresponding kernel-source or kernel-devel RPM." >>>>> >>>>> On the DDK, I can see that those folders all exist, but I can't find >>>>> how to install those on a live server.  I've found >>>>> kernel-2.6.18.92.1.10.el5.xs5.0.0.426.647.src.rpm on the DDK source >>>>> disk, but installing that does not help. >>>>> >>>>> If I were to simply copy the files from the DDK to the live server, >>>>> could somebody advise me on what I would need to copy.  Do I need >>>>> anything outside of the /lib/modules/... directory? >>>>> >>>>> thanks, >>>>> >>>>> Ross >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >>>> >>> >>> >> > From hnrose at comcast.net Thu Jun 25 05:54:08 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 25 Jun 2009 08:54:08 -0400 Subject: [ofa-general] [PATCH] opensm/man/osmtest.8: Update email address Message-ID: <20090625125408.GB15384@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/man/osmtest.8 b/opensm/man/osmtest.8 index c814b49..fa0cd52 100644 --- a/opensm/man/osmtest.8 +++ b/opensm/man/osmtest.8 @@ -184,7 +184,7 @@ Display this usage info then exit. .SH AUTHORS .TP Hal Rosenstock -.RI < hal at xsigo.com > +.RI < hal.rosenstock at gmail.com > .TP Eitan Zahavi .RI < eitan at mellanox.co.il > From hnrose at comcast.net Thu Jun 25 05:53:16 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 25 Jun 2009 08:53:16 -0400 Subject: [ofa-general] [PATCH] opensm/libvendor/osm_vendor_(ibumad mlx)_sa.c: Handle malloc failure in __osmv_send_sa_req Message-ID: <20090625125316.GA15384@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/libvendor/osm_vendor_ibumad_sa.c b/opensm/libvendor/osm_vendor_ibumad_sa.c index 218cce2..24146a7 100644 --- a/opensm/libvendor/osm_vendor_ibumad_sa.c +++ b/opensm/libvendor/osm_vendor_ibumad_sa.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -437,6 +438,13 @@ __osmv_send_sa_req(IN osmv_sa_bind_info_t * p_bind, p_madw->context.ni_context.node_guid */ p_query_req_copy = malloc(sizeof(*p_query_req_copy)); + if (!p_query_req_copy) { + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 5511: " + "Unable to acquire memory for query copy\n"); + osm_mad_pool_put(p_bind->p_mad_pool, p_madw); + status = IB_INSUFFICIENT_RESOURCES; + goto Exit; + } *p_query_req_copy = *p_query_req; p_madw->context.ni_context.node_guid = (ib_net64_t) (long)p_query_req_copy; diff --git a/opensm/libvendor/osm_vendor_mlx_sa.c b/opensm/libvendor/osm_vendor_mlx_sa.c index 7bd5aea..23e16ea 100644 --- a/opensm/libvendor/osm_vendor_mlx_sa.c +++ b/opensm/libvendor/osm_vendor_mlx_sa.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -552,6 +553,13 @@ __osmv_send_sa_req(IN osmv_sa_bind_info_t * p_bind, p_madw->context.arb_context.context1 */ p_query_req_copy = malloc(sizeof(*p_query_req_copy)); + if (!p_query_req_copy) { + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0511: " + "Unable to acquire memory for query copy\n"); + osm_mad_pool_put(p_bind->p_mad_pool, p_madw); + status = IB_INSUFFICIENT_RESOURCES; + goto Exit; + } *p_query_req_copy = *p_query_req; p_madw->context.arb_context.context1 = p_query_req_copy; From chocapiiic.tiery at gmail.com Thu Jun 25 07:16:49 2009 From: chocapiiic.tiery at gmail.com (Thierry) Date: Thu, 25 Jun 2009 16:16:49 +0200 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <20090624175527.GU18150@obsidianresearch.com> References: <4A3E45D3.3040405@Voltaire.COM> <4A424C7F.4010006@Voltaire.COM> <20090624175527.GU18150@obsidianresearch.com> Message-ID: <8d9c773c0906250716v411e8615j6745aa018e6798ac@mail.gmail.com> Hi, Here is my patch, I have worked on kernel 2.6.18 on centos 5.2 I get some trouble while accessing struc sg_list, but this implementention allow forwarding information from struct ib_send_wr in function post_send (in mlx4/qp.c). The program in userland is also attached but it only print informations on console. I will keep looking at this, Regards, Thierry On Wed, Jun 24, 2009 at 7:55 PM, Jason Gunthorpe wrote: > On Wed, Jun 24, 2009 at 06:55:43PM +0300, Moni Shoua wrote: > >> I believe that Jason and I still disagree but... > >> Jason suggests that I implement this feature with netlink. This >> approach might have an advantage but if I understand it right this >> approach requires a patch also to some user application in order to >> take advantage of this patch. > > The kernel devs have made it clear that the preferred way to export > this information is through netlink and/or a file in /proc/net/.. > > You never got an answer if /proc/net is truely discouraged or not. > > debugfs is completely useless because it cannot be used for actual > end-user interrogation and the rules are it should not be used as a > stable kernel-user interface. (ie is NOT a new dumping ground like > /proc/ was) > >> Also, I think that there is a value for a virtual text file under >> debugfs to monitor rdma_cm connections easily and without any other >> special app besides 'cat' (just like in IPoIB). > > If you have the netlink based program there really is no point in > including something under debugfs. It is just bloat. > >> Finally, this implementation doesn't contradict netlink >> implementation in the future (and it won't be the first time). > > As I said before, the stack has reached a level of maturity that new > stuff going in should meaningfully 'move the ball forward' toward a > mature and complete stack. In the context of state reporting that > means a stable user interface that programs like lsof, netstat, etc, > can rely upon. > > Putting something under debugfs clearly does nothing to advance that > goal. > > Further, there is clearly alot of state information we should be > exporting to userspace. RDMA-CM stuff is only a tiny portion. netlink > is the kernel devs answer to all of this. > > Jason > -------------- next part -------------- A non-text attachment was scrubbed... Name: netlink_trap.patch Type: text/x-patch Size: 9335 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: receiver.c Type: text/x-csrc Size: 3818 bytes Desc: not available URL: From monis at Voltaire.COM Thu Jun 25 08:23:27 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Thu, 25 Jun 2009 18:23:27 +0300 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <20090624175527.GU18150@obsidianresearch.com> References: <4A3E45D3.3040405@Voltaire.COM> <4A424C7F.4010006@Voltaire.COM> <20090624175527.GU18150@obsidianresearch.com> Message-ID: <4A43966F.70605@Voltaire.COM> Jason Gunthorpe wrote: > On Wed, Jun 24, 2009 at 06:55:43PM +0300, Moni Shoua wrote: > >> I believe that Jason and I still disagree but... > >> Jason suggests that I implement this feature with netlink. This >> approach might have an advantage but if I understand it right this >> approach requires a patch also to some user application in order to >> take advantage of this patch. > > The kernel devs have made it clear that the preferred way to export > this information is through netlink and/or a file in /proc/net/.. > > You never got an answer if /proc/net is truely discouraged or not. As I remember I got opinions against /proc/net. I'm willing to move the virtual files to there if this is acceptable by all. Otherwise I will find myself in the same situation. Roland, Sean - what do you think? > > debugfs is completely useless because it cannot be used for actual > end-user interrogation and the rules are it should not be used as a > stable kernel-user interface. (ie is NOT a new dumping ground like > /proc/ was) > I don't know about that. The only fault of debugfs that I'm aware of is that distros don't mount it by default >> Also, I think that there is a value for a virtual text file under >> debugfs to monitor rdma_cm connections easily and without any other >> special app besides 'cat' (just like in IPoIB). > > If you have the netlink based program there really is no point in > including something under debugfs. It is just bloat. > >> Finally, this implementation doesn't contradict netlink >> implementation in the future (and it won't be the first time). > > As I said before, the stack has reached a level of maturity that new > stuff going in should meaningfully 'move the ball forward' toward a > mature and complete stack. In the context of state reporting that > means a stable user interface that programs like lsof, netstat, etc, > can rely upon. > > Putting something under debugfs clearly does nothing to advance that > goal. > > Further, there is clearly alot of state information we should be > exporting to userspace. RDMA-CM stuff is only a tiny portion. netlink > is the kernel devs answer to all of this. > I just want to emphasize that my goal for now is to supply a simple way to monitor rdma_cm connections, nothing more and nothing less. I don't want it to depend on special userspace application and in fact, I just want to add a capability like we have in many other modules. This is a feature that we always get request for from customers and developers. From monis at Voltaire.COM Thu Jun 25 08:24:35 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Thu, 25 Jun 2009 18:24:35 +0300 Subject: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to monitor rdma_cm connections In-Reply-To: <8d9c773c0906250716v411e8615j6745aa018e6798ac@mail.gmail.com> References: <4A3E45D3.3040405@Voltaire.COM> <4A424C7F.4010006@Voltaire.COM> <20090624175527.GU18150@obsidianresearch.com> <8d9c773c0906250716v411e8615j6745aa018e6798ac@mail.gmail.com> Message-ID: <4A4396B3.9050601@Voltaire.COM> Thanks :) From bsl at mek.dtu.dk Thu Jun 25 11:40:38 2009 From: bsl at mek.dtu.dk (Boyan Lazarov) Date: Thu, 25 Jun 2009 20:40:38 +0200 Subject: [ofa-general] OFEM installation problem Message-ID: <200906252040.39051.bsl@mek.dtu.dk> Hello, We have installed OFED 1.4.1 on Scientific Linux 53 (RH derivative). When I run the osu_* tests (OpenMPI) as root it seems that everything is running fine and the mpi is using openib for communications. However if I run the test as an ordinary user I am getting the following: WARNING: There was an error initializing an OpenFabrics device. and openmpi is using tcp for communications - the speed is much lower. Anyone with similar problem and any suggestions? Best regards, Boyan From Line.Holen at Sun.COM Fri Jun 26 00:24:02 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Fri, 26 Jun 2009 09:24:02 +0200 Subject: [ofa-general] Re: [PATCH v2] osm_ucast_ftree.c Allow horizontal links between switches of max rank In-Reply-To: <4A40DE73.9070502@morey-chaisemartin.com> References: <4A37997E.8080708@Sun.COM> <20090618135855.GG6163@me> <4A40DE73.9070502@morey-chaisemartin.com> Message-ID: <4A447792.4090102@Sun.COM> On 06/23/09 03:53 PM, Nicolas Morey-Chaisemartin wrote: > Hi, > > As I discussed with Line, this patch also present a problem with topolgy using reverse hops and has a negative impact on routing quality. OK, if thats still the case then you need to supply an example showing this. Otherwise I have no way to verify and fix the issues you are facing. Without the reverse hops the results of the routing before and after the horizontal support are identical. > Moreover, I gave it some thoughts and I think there is a flaw in this approach. > For example > > > L1-1 L1-2 > | | \ > | | ---- > | | \ > _|___|___|_ L2 > | QNEM | |\\ > ----------- |\\\ > //// \\\\ |\\\\ > N[1-12] N[13-24] M[1-18] > > In this example the horizontal link is at the last level of the tree so it is allowed by Line's patch. > As horizontal port are treated as downlinks from both sides they can only be used when creating upwards routes (algorithm going down). > So in this example nodes N 13 to 24 will have full connectivity to nodes M 1 to 18 and nodes N 1 to 12 will have a route to nodes M 1 to 18. > However nodes M 1 to 18 won't have any routes to nodes N 1 to 12 because the algorithm starting from node N1, from example, > cannot reach switch L1-2 as the horizontal link is seen as a down link. With reverse hop it should be possible to use one horizontal link and go one but reverse hop are not well balanced and should not be used for compute nodes. > Nicolas, you are right. In this example you would not have full connectivity, N1-12 would not reach M1-18 because that would represent a down-up-down path. Your example represents a highly degraded system with no common core switch for L2 and the leftmost switch in the QNEM. In this case, full connectivity could be reached with alternative routing algorithms but that should be an administrative decision. > What I think should be done is have a more generalized approached. The Ftree algorithm already manages an enum with direction. We could add a 3rd direction (horizontal) and treat links as both upward and downwards. With the patch I've pushed (the one conflicting), we already ensure minimum hop path is used and secondary routes are more balanced so it should work quite nicely. > Handling the horizontal links as both up and down would give full connectivity in your example. But there are other examples that would introduce deadlock situations with your approach. Level 1 x-1 y-1 | \ / | | \ / | Level 2 x-2 x-3 y-2 y-3 | \ / | | \/ | | /\ | | / \ | leafs A -- B C -- D Clarifications: x1-3 is a subset of switches in a core ftree y1-3 is a subset of switches in another core ftree A+B are two interconnected leafs (or a QNEM if you like) C+D are also two interconnected leafs B is connected to y-2, and C to x-3 A, B, C and D all have CAs connected If you are allowed to use the horizontal link both on your way up and on your way down you could get the following path scenario all with valid shortest path: A to D : A --> x-2 --> x-1 --> x-3 --> C --> D (horizontal = DOWN) C to B : C --> D --> y-3 --> y-1 --> y-2 --> B (horizontal = UP) D to A : D --> y-3 --> y-1 --> y-2 --> B --> A (horizontal = DOWN) B to C : B --> A --> x-2 --> x-1 --> x-3 --> C (horizontal = UP) Voila - you have potential deadlock for CN to CN communication. The deadlock is removed if you only allow to use the horizontal link in one direction, either up or down. So there is a conflict here between avoiding deadlock and being able to have full connectivity in a degraded "ftree topology". And maybe also a conflict with system quality if reverse hops are used, I can't tell. The latter can be fixed by introducing a configuration parameter to control whether horizontal links should be accepted or not. > I also discussed with Line about it but I'd like to have other opinions. Why should this be restricted to max rank switches? Voltaire for example is selling 648 port switches (2x 324 with horizontal links) which could use such an update in the Fat-Tree algorithm. > Support for cross-links (as well as down-up type paths) will in general require more than one VL for deadlock avoidance. Restrictions on where (and under what conditions) cross-links can be introduced without causing deadlock potential will allow selected cross-link scenarios to be supported with only one VL. Such support could be introduced in several steps. > Regards > > Nicolas > Line From nicolas.morey-chaisemartin at ext.bull.net Fri Jun 26 01:27:37 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey-Chaisemartin) Date: Fri, 26 Jun 2009 10:27:37 +0200 Subject: [ofa-general] Re: [PATCH v2] osm_ucast_ftree.c Allow horizontal links between switches of max rank In-Reply-To: <4A447792.4090102@Sun.COM> References: <4A37997E.8080708@Sun.COM> <20090618135855.GG6163@me> <4A40DE73.9070502@morey-chaisemartin.com> <4A447792.4090102@Sun.COM> Message-ID: <4A448679.6090804@ext.bull.net> Hi , Le 26/06/2009 09:24, Line.Holen at Sun.COM a écrit : >> Moreover, I gave it some thoughts and I think there is a flaw in this approach. >> For example >> >> >> L1-1 L1-2 >> | | \ >> | | ---- >> | | \ >> _|___|___|_ L2 >> | QNEM | |\\ >> ----------- |\\\ >> //// \\\\ |\\\\ >> N[1-12] N[13-24] M[1-18] >> > > Nicolas, you are right. In this example you would not have > full connectivity, N1-12 would not reach M1-18 because that > would represent a down-up-down path. > Your example represents a highly degraded system with no > common core switch for L2 and the leftmost switch in the QNEM. > In this case, full connectivity could be reached with alternative > routing algorithms but that should be an administrative decision. > It's not necessary degraded. I took switches and many nodes for an example but you could as simply put a service node (admin?) directly under the L1-2 switch and have exactly the same problem. Connecting a whole cluster like this would be stupid, but add I/O or service nodes makes sense. > Handling the horizontal links as both up and down would give > full connectivity in your example. But there are other examples > that would introduce deadlock situations with your approach. > > > Level 1 x-1 y-1 > | \ / | > | \ / | > Level 2 x-2 x-3 y-2 y-3 > | \ / | > | \/ | > | /\ | > | / \ | > leafs A -- B C -- D > > Clarifications: > x1-3 is a subset of switches in a core ftree > y1-3 is a subset of switches in another core ftree > A+B are two interconnected leafs (or a QNEM if you like) > C+D are also two interconnected leafs > B is connected to y-2, and C to x-3 > A, B, C and D all have CAs connected > > If you are allowed to use the horizontal link both on your way up > and on your way down you could get the following path scenario > all with valid shortest path: > > A to D : A --> x-2 --> x-1 --> x-3 --> C --> D (horizontal = DOWN) > C to B : C --> D --> y-3 --> y-1 --> y-2 --> B (horizontal = UP) > D to A : D --> y-3 --> y-1 --> y-2 --> B --> A (horizontal = DOWN) > B to C : B --> A --> x-2 --> x-1 --> x-3 --> C (horizontal = UP) > > Voila - you have potential deadlock for CN to CN communication. > The deadlock is removed if you only allow to use the horizontal > link in one direction, either up or down. > It won't route this way if you optimize thing. You have to consider that you can take both up and down but they are not part of both up port groups and down port groups. By exploring in the right order (up before horizontal and down before horizontal) you can avoid that. Bascially if possible you'll always use the horizontal link last (in the algorithm which means first on the route). So here we would have A to D: A --> B --> y-2 --> y-1 --> y-3 --> D B to C: B --> A --> x-2 --> x-1 --> x-3 --> C C to B: C --> D --> y-3 --> y-1 --> y-2 --> B D to A: D --> C --> x-3 --> x-1 --> x-2 --> A Which is basically what you obtain with your patch Nicolas From vlad at lists.openfabrics.org Fri Jun 26 02:24:17 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 26 Jun 2009 02:24:17 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090626-0200 daily build status Message-ID: <20090626092418.0C9B6102021C@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090626-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From sashak at voltaire.com Fri Jun 26 03:03:43 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 26 Jun 2009 13:03:43 +0300 Subject: [ofa-general] Re: [PATCH 0/2] Opensm support for external routing engines In-Reply-To: <4A421511.6000004@gmail.com> References: <20090618132725.GD6163@me> <4A3A46FF.3060001@bull.net> <20090618143629.GH6163@me> <4A3B25D1.8030500@ext.bull.net> <20090623190815.GB4324@me> <20090623131939.e464afb2.weiny2@llnl.gov> <20090623204324.GE4324@me> <20090623152935.b4f64a1c.weiny2@llnl.gov> <20090624092343.GA1479@me> <4A421511.6000004@gmail.com> Message-ID: <20090626100343.GA20617@me> On 14:59 Wed 24 Jun , Eli Dorfman (Voltaire) wrote: > > > > 1. We have pretty generic plugin interface right now and users may use > > it as they want (including routing engines obviously). If improvements > > are needed in this area (such as plugin(s) reload on the fly or any > > other) let's go ahead and do it. > > How can this plugin interface enable configuration of routing engines order? Is it related to the order? The order is defined by configuration parameter with plugin or without it. > > 2. I don't see why we need yet another plugin interface specific to > > routing engines - (1) should be good enough for such purposes. > > > > 3. Regardless to above I don't think that we as open source community > > should support any sort of "proprietary additions". Yes, it is allowed > > for third parties, but it is not our charter. > > These are not proprietary additions but only an infrastructure to enable proprietary implementations And I don't think that we need to care about such infrastructure. > and to reduce opensm dependency (e.g. plugin mysql). That's different story - see (1) and (2). Sasha From myxiplx at googlemail.com Fri Jun 26 03:35:38 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Fri, 26 Jun 2009 11:35:38 +0100 Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 Message-ID: <7b160d240906260335l59ca2ff2j1be2df02c0f38c4a@mail.gmail.com> Hello everyone, Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually Citrix XenServer), with OFED-1.4.1? The default install of OFED doesn't include nfs-rdma, and I'm struggling to work out which components I do and don't need for a custom build. What I would like to include is: - Core infiniband components - Drivers for Mellanox Infinihost III Lx (MHES14) - IPoIB - nfs-rdma - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, ibhosts, etc...) My last attempt was following the instructions from OFED 1.4 - I edited ofed.conf to add nfsrdma=y, and attempted a default install. However after running install.pl, that change appears to have been overwritten. I am also having problems updating mount.nfs on this machine. Both nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: "blkid/blkid.h: No such file or directory", and I cannot yet find a source for this file. Finally, my understanding is that nfs-rdma is in beta status for CentOS 5.2, but I couldn't see any open bugs on the tracker, are there any issues I should be aware of? thanks, Ross From myxiplx at googlemail.com Fri Jun 26 04:25:10 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Fri, 26 Jun 2009 12:25:10 +0100 Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 In-Reply-To: References: <7b160d240906260335l59ca2ff2j1be2df02c0f38c4a@mail.gmail.com> Message-ID: <7b160d240906260425i1ddf7083yc63c4e803d0cafe4@mail.gmail.com> Thanks Robert, Looking through your notes, we've followed similar steps. I should mention that I do have Infiniband working on this server, including ipoib. It's purely the nfs-rdma part I'm struggling with. My latest attempt has been to attempt a custom install, selecting nfs-rdma manually. That worked fine on the development server, but when I attempted to run it on the live server I had an error: "/lib/modules/....../build/scripts is required to build kernel-ib RPM. Please install the corresponding kernel-source or kernel-devel RPM. tk rpm is required to install ibutils" The problem is, for XenServer, you can't get the kernel-source on the live server, instead Citrix ship a DDK (driver development kit) virtual machine, which is a pre-built system with all the kernel sources you need. OFED builds fine on the DDK machine, and for the default installation, I can compile everything there and then transfer the OFED folder (with the compiled RPM files) to the live server. At that point install.pl runs fine, without needing to re-compile things. However, it appears that when I choose some elements in the custom install, it is not using these RPM files, and is trying to compile kernel-ib again from scratch. I have attached a file listing the custom modules I have chosen in case it helps. Ross On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley wrote: > Hi Ross, > > I installed OFED 1.3.1 (Older I know) with other options OK on a Centos > 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is > some notes I took (I used the GUI installer) > > Hope this helps, > > Rob > > 1.      Download the OFED package from > http://www.openfabrics.org/downloads/OFED/ > 2.      Extract it: tar -xvzf OFED-1.3.1.tgz > 3.      Check for the dependencies by running ./install.pl and choosing > documentation ("0"). > 4.      Install all required dependencies using yum install package nam. > For Centos dependencies try: yum install gcc libstdc++-devel > libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build > gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel > zlib-devel You may need to run the yum command three times (It takes a > few runs for all the dependencies to sort themselves out) > 5.      It is recommended you now upgrade the kernel to make sure the > sources match it. For centos do: yum upgrade kernel (Or yum upgrade > kernel-xen for Xen setups) > 6.      Run ./install.pl again but this time choose to install ("2") and > choose custom install ("4").  See the appendix for a recommendation on > which modules to install. > 7.      Install should now complete, if it fails check the log (Most > likely you have not installed a required software package - go back to > step 3) > 8.      Say yes to configuring IPOIB (If you plan to use it) and > manually set the IP and subnet. When it asks for a network it wants the > base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 > subnet then the "Network" would be 192.168.10.0) > 9.      Reboot > 10.     Check the install using /etc/init.d/openibd status. This should > show one interface for each Infiniband port (First will be "ib0"). You > should also see quite a few modules loaded, the important ones are > (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core > & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - > IPOIB and ib_uverbs - Direct/Verb) > 11.     Run ibstat - This should display the installed devices and their > firmware versions and hardware / GUI IDs. If you need to update the > firmware for Mellanox see > http://www.mellanox.com/support/firmware_download.php I find it easier > to update using Windows and the WinOF software (Windows equivalent of > OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). > > -----Original Message----- > From: general-bounces at lists.openfabrics.org > [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith > Sent: 26 June 2009 11:36 > To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org > Cc: Yehonatan Yossef > Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 > > Hello everyone, > > Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually > Citrix XenServer), with OFED-1.4.1? > > The default install of OFED doesn't include nfs-rdma, and I'm > struggling to work out which components I do and don't need for a > custom build. > > What I would like to include is: > - Core infiniband components > - Drivers for Mellanox Infinihost III Lx (MHES14) > - IPoIB > - nfs-rdma > - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, > ibhosts, etc...) > > My last attempt was following the instructions from OFED 1.4 - I > edited ofed.conf to add nfsrdma=y, and attempted a default install. > However after running install.pl, that change appears to have been > overwritten. > > I am also having problems updating mount.nfs on this machine.  Both > nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: > "blkid/blkid.h: No such file or directory", and I cannot yet find a > source for this file. > > Finally, my understanding is that nfs-rdma is in beta status for > CentOS 5.2, but I couldn't see any open bugs on the tracker, are there > any issues I should be aware of? > > thanks, > > Ross > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > The SAQ Group > > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ > SAQ is the trading name of SEMTEC Limited. Registered in England & Wales > Company Number: 06481952 > > http://www.saqnet.co.uk AS29219 > > SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. > > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. > > ISPA Member > > -------------- next part -------------- packages to install: Select Option [1-4]:4 Install kernel-ib? [y/N]:y Install core module? [y/N]:y Install mthca module? [y/N]:y Install mlx4 module? [y/N]:n Install mlx4_en module? [y/N]:n Install cxgb3 module? [y/N]:n Install nes module? [y/N]:n Install ipoib module? [y/N]:y Install sdp module? [y/N]:n Install srp module? [y/N]:n Install srpt module? [y/N]:n Install rds module? [y/N]:n Install qlgc_vnic module? [y/N]:n Install iser module? (open-iscsi will also be installed) [y/N]:n Install nfsrdma module? [y/N]:y Install kernel-ib-devel? [y/N]:n Install ib-bonding? [y/N]:n Install ib-bonding-debuginfo? [y/N]:n Install libibverbs? [y/N]:y Install libibverbs-devel? [y/N]:y Install libibverbs-devel-static? [y/N]:n Install libibverbs-utils? [y/N]:y Install libibverbs-debuginfo? [y/N]:n Install libmthca? [y/N]:y Install libmthca-devel-static? [y/N]:n Install libmthca-debuginfo? [y/N]:n Install libmlx4? [y/N]:n Install libmlx4-devel? [y/N]:n Install libmlx4-debuginfo? [y/N]:n Install libcxgb3? [y/N]:n Install libcxgb3-devel? [y/N]:n Install libcxgb3-debuginfo? [y/N]:n Install libnes? [y/N]:n Install libnes-devel-static? [y/N]:n Install libnes-debuginfo? [y/N]:n Install libibcm? [y/N]:n Install libibcm-devel? [y/N]:n Install libibcm-debuginfo? [y/N]:n Install libibcommon? [y/N]:y Install libibcommon-devel? [y/N]:n Install libibcommon-static? [y/N]:n Install libibcommon-debuginfo? [y/N]:n Install libibumad? [y/N]:y Install libibumad-devel? [y/N]:n Install libibumad-static? [y/N]:n Install libibumad-debuginfo? [y/N]:n Install libibmad? [y/N]:n Install libibmad-devel? [y/N]:n Install libibmad-static? [y/N]:n Install libibmad-debuginfo? [y/N]:n Install ibsim? [y/N]:n Install ibsim-debuginfo? [y/N]:n Install librdmacm? [y/N]:y Install librdmacm-utils? [y/N]:y Install librdmacm-devel? [y/N]:n Install librdmacm-debuginfo? [y/N]:n Install libsdp? [y/N]:n Install libsdp-devel? [y/N]:n Install libsdp-debuginfo? [y/N]:n Install opensm? [y/N]:n Install opensm-libs? [y/N]:n Install opensm-devel? [y/N]:n Install opensm-debuginfo? [y/N]:n Install opensm-static? [y/N]:n Install compat-dapl? [y/N]:n Install compat-dapl-devel? [y/N]:n Install dapl? [y/N]:n Install dapl-devel? [y/N]:n Install dapl-devel-static? [y/N]:n Install dapl-utils? [y/N]:n Install dapl-debuginfo? [y/N]:n Install perftest? [y/N]:n Install mstflint? [y/N]:y Install tvflash? [y/N]:n Install qlvnictools? [y/N]:n Install sdpnetstat? [y/N]:n Install srptools? [y/N]:n Install rds-tools? [y/N]:n Install rnfs-utils? [y/N]:y Install ibutils? [y/N]:y Install infiniband-diags? [y/N]:y Install qperf? [y/N]:n Install qperf-debuginfo? [y/N]:n Install ofed-docs? [y/N]:y Install ofed-scripts? [y/N]:y Install tgt-generic? [y/N]:n Install mpi-selector? [y/N]:n Install openmpi_gcc? [y/N]:n Install mpitests_openmpi_gcc? [y/N]:n Please enter the OFED installation directory: [/usr]: From myxiplx at googlemail.com Fri Jun 26 06:06:11 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Fri, 26 Jun 2009 14:06:11 +0100 Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 In-Reply-To: <7b160d240906260425i1ddf7083yc63c4e803d0cafe4@mail.gmail.com> References: <7b160d240906260335l59ca2ff2j1be2df02c0f38c4a@mail.gmail.com> <7b160d240906260425i1ddf7083yc63c4e803d0cafe4@mail.gmail.com> Message-ID: <7b160d240906260606r2fd4c483lb66c248e3163bcd@mail.gmail.com> Ok, I've made a little more progress. The issue with nfs-utils was solved by installing efsprogs-devel, and mount.nfs is now reporting that it's running v1.1.6. I still can't install OFED with nfs-rdma on the live server though, it appears that it's unable to use the rpm package to install with nfs support. I've now discovered how to use the ofed.conf file, and this is the procedure I'm following: - On the development server, I run install.pl and perform a standard install - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y - Still on the development server I run: # install.pl -c ofed.conf - That can be seen to install most items from the rpm, and to recompile kernel-ib with the nfsrdma option. - I then package up the OFED folder with tar, and send it to the live server - On the live server I run the same command: # install.pl -c ofed.conf Despite it running with exactly the same options, I get the error: "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM. Please install the corresponding kernel-source or kerenel-devel RPM." If I exclude the nfsrdma option and repeat all the above steps, the script installs fine from the pre-compiled RPM. It looks to me like a problem with the install script when choosing nfs-rdma. Does anybody have any suggestions as to how I can work around this? Am I able to simply install the RPM files manually? Ross On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith wrote: > Thanks Robert, > > Looking through your notes, we've followed similar steps.  I should > mention that I do have Infiniband working on this server, including > ipoib.  It's purely the nfs-rdma part I'm struggling with. > > My latest attempt has been to attempt a custom install, selecting > nfs-rdma manually.  That worked fine on the development server, but > when I attempted to run it on the live server I had an error: > > "/lib/modules/....../build/scripts is required to build kernel-ib RPM. > Please install the corresponding kernel-source or kernel-devel RPM. > tk rpm is required to install ibutils" > > The problem is, for XenServer, you can't get the kernel-source on the > live server, instead Citrix ship a DDK (driver development kit) > virtual machine, which is a pre-built system with all the kernel > sources you need. > > OFED builds fine on the DDK machine, and for the default installation, > I can compile everything there and then transfer the OFED folder (with > the compiled RPM files) to the live server.  At that point install.pl > runs fine, without needing to re-compile things. > > However, it appears that when I choose some elements in the custom > install, it is not using these RPM files, and is trying to compile > kernel-ib again from scratch. > > I have attached a file listing the custom modules I have chosen in > case it helps. > > Ross > > > > On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley wrote: >> Hi Ross, >> >> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos >> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is >> some notes I took (I used the GUI installer) >> >> Hope this helps, >> >> Rob >> >> 1.      Download the OFED package from >> http://www.openfabrics.org/downloads/OFED/ >> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz >> 3.      Check for the dependencies by running ./install.pl and choosing >> documentation ("0"). >> 4.      Install all required dependencies using yum install package nam. >> For Centos dependencies try: yum install gcc libstdc++-devel >> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build >> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel >> zlib-devel You may need to run the yum command three times (It takes a >> few runs for all the dependencies to sort themselves out) >> 5.      It is recommended you now upgrade the kernel to make sure the >> sources match it. For centos do: yum upgrade kernel (Or yum upgrade >> kernel-xen for Xen setups) >> 6.      Run ./install.pl again but this time choose to install ("2") and >> choose custom install ("4").  See the appendix for a recommendation on >> which modules to install. >> 7.      Install should now complete, if it fails check the log (Most >> likely you have not installed a required software package - go back to >> step 3) >> 8.      Say yes to configuring IPOIB (If you plan to use it) and >> manually set the IP and subnet. When it asks for a network it wants the >> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 >> subnet then the "Network" would be 192.168.10.0) >> 9.      Reboot >> 10.     Check the install using /etc/init.d/openibd status. This should >> show one interface for each Infiniband port (First will be "ib0"). You >> should also see quite a few modules loaded, the important ones are >> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core >> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - >> IPOIB and ib_uverbs - Direct/Verb) >> 11.     Run ibstat - This should display the installed devices and their >> firmware versions and hardware / GUI IDs. If you need to update the >> firmware for Mellanox see >> http://www.mellanox.com/support/firmware_download.php I find it easier >> to update using Windows and the WinOF software (Windows equivalent of >> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). >> >> -----Original Message----- >> From: general-bounces at lists.openfabrics.org >> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith >> Sent: 26 June 2009 11:36 >> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org >> Cc: Yehonatan Yossef >> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 >> >> Hello everyone, >> >> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually >> Citrix XenServer), with OFED-1.4.1? >> >> The default install of OFED doesn't include nfs-rdma, and I'm >> struggling to work out which components I do and don't need for a >> custom build. >> >> What I would like to include is: >> - Core infiniband components >> - Drivers for Mellanox Infinihost III Lx (MHES14) >> - IPoIB >> - nfs-rdma >> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, >> ibhosts, etc...) >> >> My last attempt was following the instructions from OFED 1.4 - I >> edited ofed.conf to add nfsrdma=y, and attempted a default install. >> However after running install.pl, that change appears to have been >> overwritten. >> >> I am also having problems updating mount.nfs on this machine.  Both >> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: >> "blkid/blkid.h: No such file or directory", and I cannot yet find a >> source for this file. >> >> Finally, my understanding is that nfs-rdma is in beta status for >> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there >> any issues I should be aware of? >> >> thanks, >> >> Ross >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> The SAQ Group >> >> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ >> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales >> Company Number: 06481952 >> >> http://www.saqnet.co.uk AS29219 >> >> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. >> >> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. >> >> ISPA Member >> >> > From chocapiiic.tiery at gmail.com Fri Jun 26 06:07:46 2009 From: chocapiiic.tiery at gmail.com (Thierry) Date: Fri, 26 Jun 2009 15:07:46 +0200 Subject: [ofa-general] Sending two integers via RDMA_WRITE In-Reply-To: <7d4423d30905252245o869fd93p96facb1d1b2b94e6@mail.gmail.com> References: <7d4423d30905250836r431e7d1eh170f3bc22b731a55@mail.gmail.com> <4A1AD987.2080200@gmail.com> <7d4423d30905251108m5bc951ach4da2027698b0ffb6@mail.gmail.com> <4A1AF13D.8040506@gmail.com> <7d4423d30905252245o869fd93p96facb1d1b2b94e6@mail.gmail.com> Message-ID: <8d9c773c0906260607y7aa4a1a6y55742c45ba9751ed@mail.gmail.com> Hi, Have you fixed your problem ? I am also trying to pool CQ (associated with the receive part) with an other program, but I get nothing. Regards, Thierry On Tue, May 26, 2009 at 7:45 AM, Zafar Gilani wrote: > To make things more clear, why won't you just poll the CQ for completion > directly? (without using the CQ events) > > I believe that you will get a completion with error... > > Yes I tried polling directly and it returns a negative number. What is the > remedy for this? Is my for loop logically correct (client.c)? I also tried > polling the server CQ directly (server.c) and polling here also returns a > negative number, which means that data write is not working properly thus no > completion events. What do you suggest I do? I am obviously lost! :( > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From michael.heinz at qlogic.com Fri Jun 26 06:37:06 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Fri, 26 Jun 2009 08:37:06 -0500 Subject: [ofa-general] Cannot bring up mellanox HCAs on HP Nahalem machines running RHEL 5 Message-ID: We've open an OFED PR ( https://bugs.openfabrics.org/show_bug.cgi?id=1663 ) on this issue, but I'm wondering if anyone else has seen this: We just got some brand new HP "gen6" machines - and we cannot get any Mellanox HCA, DDR or QDR to come up on those machines - if they are running RHEL 5.3. Machines running SLES11 appear to work fine. The same HCAs work fine on slower machines and on machines running SLES11 - has anyone else seen this? We're running OFED 1.4.1 GA. From myxiplx at googlemail.com Fri Jun 26 07:31:50 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Fri, 26 Jun 2009 15:31:50 +0100 Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 In-Reply-To: <7b160d240906260606r2fd4c483lb66c248e3163bcd@mail.gmail.com> References: <7b160d240906260335l59ca2ff2j1be2df02c0f38c4a@mail.gmail.com> <7b160d240906260425i1ddf7083yc63c4e803d0cafe4@mail.gmail.com> <7b160d240906260606r2fd4c483lb66c248e3163bcd@mail.gmail.com> Message-ID: <7b160d240906260731o4e47d19bwa492771a71258f32@mail.gmail.com> Latest news: I've bodged the installation, but I'm having problems with nfs.mount now. Since everything worked with nfs-rdma on the development server, I manually copied the kerel-ib rpm file from there, and installed that on top of the default infiniband install on the live server, using the command: # rpm -iv --replacepkgs kernel-ib....... With that done modprobe xprtrdma now works. Since the kernel-ib file appears to be the only one modified as nfsrdma support is added, I assume this is all that's needed. Can anybody confirm if this is ok, or whether I'm going to break things horribly by doing this? I also have nfs v1.1.6 installed, but it doesn't seem to understand the rdma option. The system is still using tcp by default, and if I try to use rdma, I get this: # mount -o rdma, port=20049 server:path /mnt mount.nfs: Unsupported nfs mount option: rdma However, nfs is definitely the correct version: # mount.nfs -V mount.nfs (linux nfs-utils 1.1.6) Even using the mount.rnfs utility supplied with OFED doesn't help: # rpm -iv rnfs-utils-1.1.5-2.OFED.i386.rpm # mount.rnfs server:path /mnt -o rdma,port=20049 mount.rnfs: Unsupported nfs mount option: rdma And if I use mount.rnfs without the rdma option, "cat /proc/mounts" reports that it's just using tcp. I'm getting read speeds of around 400MB/s using regular NFS which isn't bad, but I know I can achieve over 900MB/s with this server over Infiniband, so there's still some way to go. Ross On Fri, Jun 26, 2009 at 2:06 PM, Ross Smith wrote: > Ok, I've made a little more progress. > > The issue with nfs-utils was solved by installing efsprogs-devel, and > mount.nfs is now reporting that it's running v1.1.6. > > I still can't install OFED with nfs-rdma on the live server though, it > appears that it's unable to use the rpm package to install with nfs > support. > > I've now discovered how to use the ofed.conf file, and this is the > procedure I'm following: > > - On the development server, I run install.pl and perform a standard install > - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y > - Still on the development server I run: >  # install.pl -c ofed.conf > - That can be seen to install most items from the rpm, and to > recompile kernel-ib with the nfsrdma option. > - I then package up the OFED folder with tar, and send it to the live server > - On the live server I run the same command: >  # install.pl -c ofed.conf > > Despite it running with exactly the same options, I get the error: > > "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM. > Please install the corresponding kernel-source or kerenel-devel RPM." > > If I exclude the nfsrdma option and repeat all the above steps, the > script installs fine from the pre-compiled RPM. > > It looks to me like a problem with the install script when choosing nfs-rdma. > > Does anybody have any suggestions as to how I can work around this? > Am I able to simply install the RPM files manually? > > Ross > > > > On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith wrote: >> Thanks Robert, >> >> Looking through your notes, we've followed similar steps.  I should >> mention that I do have Infiniband working on this server, including >> ipoib.  It's purely the nfs-rdma part I'm struggling with. >> >> My latest attempt has been to attempt a custom install, selecting >> nfs-rdma manually.  That worked fine on the development server, but >> when I attempted to run it on the live server I had an error: >> >> "/lib/modules/....../build/scripts is required to build kernel-ib RPM. >> Please install the corresponding kernel-source or kernel-devel RPM. >> tk rpm is required to install ibutils" >> >> The problem is, for XenServer, you can't get the kernel-source on the >> live server, instead Citrix ship a DDK (driver development kit) >> virtual machine, which is a pre-built system with all the kernel >> sources you need. >> >> OFED builds fine on the DDK machine, and for the default installation, >> I can compile everything there and then transfer the OFED folder (with >> the compiled RPM files) to the live server.  At that point install.pl >> runs fine, without needing to re-compile things. >> >> However, it appears that when I choose some elements in the custom >> install, it is not using these RPM files, and is trying to compile >> kernel-ib again from scratch. >> >> I have attached a file listing the custom modules I have chosen in >> case it helps. >> >> Ross >> >> >> >> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley wrote: >>> Hi Ross, >>> >>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos >>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is >>> some notes I took (I used the GUI installer) >>> >>> Hope this helps, >>> >>> Rob >>> >>> 1.      Download the OFED package from >>> http://www.openfabrics.org/downloads/OFED/ >>> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz >>> 3.      Check for the dependencies by running ./install.pl and choosing >>> documentation ("0"). >>> 4.      Install all required dependencies using yum install package nam. >>> For Centos dependencies try: yum install gcc libstdc++-devel >>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build >>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel >>> zlib-devel You may need to run the yum command three times (It takes a >>> few runs for all the dependencies to sort themselves out) >>> 5.      It is recommended you now upgrade the kernel to make sure the >>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade >>> kernel-xen for Xen setups) >>> 6.      Run ./install.pl again but this time choose to install ("2") and >>> choose custom install ("4").  See the appendix for a recommendation on >>> which modules to install. >>> 7.      Install should now complete, if it fails check the log (Most >>> likely you have not installed a required software package - go back to >>> step 3) >>> 8.      Say yes to configuring IPOIB (If you plan to use it) and >>> manually set the IP and subnet. When it asks for a network it wants the >>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 >>> subnet then the "Network" would be 192.168.10.0) >>> 9.      Reboot >>> 10.     Check the install using /etc/init.d/openibd status. This should >>> show one interface for each Infiniband port (First will be "ib0"). You >>> should also see quite a few modules loaded, the important ones are >>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core >>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - >>> IPOIB and ib_uverbs - Direct/Verb) >>> 11.     Run ibstat - This should display the installed devices and their >>> firmware versions and hardware / GUI IDs. If you need to update the >>> firmware for Mellanox see >>> http://www.mellanox.com/support/firmware_download.php I find it easier >>> to update using Windows and the WinOF software (Windows equivalent of >>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). >>> >>> -----Original Message----- >>> From: general-bounces at lists.openfabrics.org >>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith >>> Sent: 26 June 2009 11:36 >>> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org >>> Cc: Yehonatan Yossef >>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 >>> >>> Hello everyone, >>> >>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually >>> Citrix XenServer), with OFED-1.4.1? >>> >>> The default install of OFED doesn't include nfs-rdma, and I'm >>> struggling to work out which components I do and don't need for a >>> custom build. >>> >>> What I would like to include is: >>> - Core infiniband components >>> - Drivers for Mellanox Infinihost III Lx (MHES14) >>> - IPoIB >>> - nfs-rdma >>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, >>> ibhosts, etc...) >>> >>> My last attempt was following the instructions from OFED 1.4 - I >>> edited ofed.conf to add nfsrdma=y, and attempted a default install. >>> However after running install.pl, that change appears to have been >>> overwritten. >>> >>> I am also having problems updating mount.nfs on this machine.  Both >>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: >>> "blkid/blkid.h: No such file or directory", and I cannot yet find a >>> source for this file. >>> >>> Finally, my understanding is that nfs-rdma is in beta status for >>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there >>> any issues I should be aware of? >>> >>> thanks, >>> >>> Ross >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general >>> >>> The SAQ Group >>> >>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ >>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales >>> Company Number: 06481952 >>> >>> http://www.saqnet.co.uk AS29219 >>> >>> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. >>> >>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. >>> >>> ISPA Member >>> >>> >> > From Jeffrey.C.Becker at nasa.gov Fri Jun 26 10:15:14 2009 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Fri, 26 Jun 2009 10:15:14 -0700 Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 In-Reply-To: <7b160d240906260731o4e47d19bwa492771a71258f32@mail.gmail.com> References: <7b160d240906260335l59ca2ff2j1be2df02c0f38c4a@mail.gmail.com> <7b160d240906260425i1ddf7083yc63c4e803d0cafe4@mail.gmail.com> <7b160d240906260606r2fd4c483lb66c248e3163bcd@mail.gmail.com> <7b160d240906260731o4e47d19bwa492771a71258f32@mail.gmail.com> Message-ID: <4A450222.6040104@nasa.gov> Hi Ross. The easiest way to install NFSRDMA is to do a custom install and select it manually from the kernel-ib modules. Also be sure to select rnfs-utils since that includes a special mount.rnfs that is needed to understand RDMA mounts. -jeff Ross Smith wrote: > Latest news: I've bodged the installation, but I'm having problems > with nfs.mount now. > > Since everything worked with nfs-rdma on the development server, I > manually copied the kerel-ib rpm file from there, and installed that > on top of the default infiniband install on the live server, using the > command: > > # rpm -iv --replacepkgs kernel-ib....... > > With that done modprobe xprtrdma now works. > > Since the kernel-ib file appears to be the only one modified as > nfsrdma support is added, I assume this is all that's needed. Can > anybody confirm if this is ok, or whether I'm going to break things > horribly by doing this? > > I also have nfs v1.1.6 installed, but it doesn't seem to understand > the rdma option. The system is still using tcp by default, and if I > try to use rdma, I get this: > > # mount -o rdma, port=20049 server:path /mnt > mount.nfs: Unsupported nfs mount option: rdma > > However, nfs is definitely the correct version: > # mount.nfs -V > mount.nfs (linux nfs-utils 1.1.6) > > Even using the mount.rnfs utility supplied with OFED doesn't help: > # rpm -iv rnfs-utils-1.1.5-2.OFED.i386.rpm > # mount.rnfs server:path /mnt -o rdma,port=20049 > mount.rnfs: Unsupported nfs mount option: rdma > > And if I use mount.rnfs without the rdma option, "cat /proc/mounts" > reports that it's just using tcp. > > I'm getting read speeds of around 400MB/s using regular NFS which > isn't bad, but I know I can achieve over 900MB/s with this server over > Infiniband, so there's still some way to go. > > Ross > > > > On Fri, Jun 26, 2009 at 2:06 PM, Ross Smith wrote: > >> Ok, I've made a little more progress. >> >> The issue with nfs-utils was solved by installing efsprogs-devel, and >> mount.nfs is now reporting that it's running v1.1.6. >> >> I still can't install OFED with nfs-rdma on the live server though, it >> appears that it's unable to use the rpm package to install with nfs >> support. >> >> I've now discovered how to use the ofed.conf file, and this is the >> procedure I'm following: >> >> - On the development server, I run install.pl and perform a standard install >> - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y >> - Still on the development server I run: >> # install.pl -c ofed.conf >> - That can be seen to install most items from the rpm, and to >> recompile kernel-ib with the nfsrdma option. >> - I then package up the OFED folder with tar, and send it to the live server >> - On the live server I run the same command: >> # install.pl -c ofed.conf >> >> Despite it running with exactly the same options, I get the error: >> >> "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM. >> Please install the corresponding kernel-source or kerenel-devel RPM." >> >> If I exclude the nfsrdma option and repeat all the above steps, the >> script installs fine from the pre-compiled RPM. >> >> It looks to me like a problem with the install script when choosing nfs-rdma. >> >> Does anybody have any suggestions as to how I can work around this? >> Am I able to simply install the RPM files manually? >> >> Ross >> >> >> >> On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith wrote: >> >>> Thanks Robert, >>> >>> Looking through your notes, we've followed similar steps. I should >>> mention that I do have Infiniband working on this server, including >>> ipoib. It's purely the nfs-rdma part I'm struggling with. >>> >>> My latest attempt has been to attempt a custom install, selecting >>> nfs-rdma manually. That worked fine on the development server, but >>> when I attempted to run it on the live server I had an error: >>> >>> "/lib/modules/....../build/scripts is required to build kernel-ib RPM. >>> Please install the corresponding kernel-source or kernel-devel RPM. >>> tk rpm is required to install ibutils" >>> >>> The problem is, for XenServer, you can't get the kernel-source on the >>> live server, instead Citrix ship a DDK (driver development kit) >>> virtual machine, which is a pre-built system with all the kernel >>> sources you need. >>> >>> OFED builds fine on the DDK machine, and for the default installation, >>> I can compile everything there and then transfer the OFED folder (with >>> the compiled RPM files) to the live server. At that point install.pl >>> runs fine, without needing to re-compile things. >>> >>> However, it appears that when I choose some elements in the custom >>> install, it is not using these RPM files, and is trying to compile >>> kernel-ib again from scratch. >>> >>> I have attached a file listing the custom modules I have chosen in >>> case it helps. >>> >>> Ross >>> >>> >>> >>> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley wrote: >>> >>>> Hi Ross, >>>> >>>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos >>>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is >>>> some notes I took (I used the GUI installer) >>>> >>>> Hope this helps, >>>> >>>> Rob >>>> >>>> 1. Download the OFED package from >>>> http://www.openfabrics.org/downloads/OFED/ >>>> 2. Extract it: tar -xvzf OFED-1.3.1.tgz >>>> 3. Check for the dependencies by running ./install.pl and choosing >>>> documentation ("0"). >>>> 4. Install all required dependencies using yum install package nam. >>>> For Centos dependencies try: yum install gcc libstdc++-devel >>>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build >>>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel >>>> zlib-devel You may need to run the yum command three times (It takes a >>>> few runs for all the dependencies to sort themselves out) >>>> 5. It is recommended you now upgrade the kernel to make sure the >>>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade >>>> kernel-xen for Xen setups) >>>> 6. Run ./install.pl again but this time choose to install ("2") and >>>> choose custom install ("4"). See the appendix for a recommendation on >>>> which modules to install. >>>> 7. Install should now complete, if it fails check the log (Most >>>> likely you have not installed a required software package - go back to >>>> step 3) >>>> 8. Say yes to configuring IPOIB (If you plan to use it) and >>>> manually set the IP and subnet. When it asks for a network it wants the >>>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 >>>> subnet then the "Network" would be 192.168.10.0) >>>> 9. Reboot >>>> 10. Check the install using /etc/init.d/openibd status. This should >>>> show one interface for each Infiniband port (First will be "ib0"). You >>>> should also see quite a few modules loaded, the important ones are >>>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core >>>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - >>>> IPOIB and ib_uverbs - Direct/Verb) >>>> 11. Run ibstat - This should display the installed devices and their >>>> firmware versions and hardware / GUI IDs. If you need to update the >>>> firmware for Mellanox see >>>> http://www.mellanox.com/support/firmware_download.php I find it easier >>>> to update using Windows and the WinOF software (Windows equivalent of >>>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). >>>> >>>> -----Original Message----- >>>> From: general-bounces at lists.openfabrics.org >>>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith >>>> Sent: 26 June 2009 11:36 >>>> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org >>>> Cc: Yehonatan Yossef >>>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 >>>> >>>> Hello everyone, >>>> >>>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually >>>> Citrix XenServer), with OFED-1.4.1? >>>> >>>> The default install of OFED doesn't include nfs-rdma, and I'm >>>> struggling to work out which components I do and don't need for a >>>> custom build. >>>> >>>> What I would like to include is: >>>> - Core infiniband components >>>> - Drivers for Mellanox Infinihost III Lx (MHES14) >>>> - IPoIB >>>> - nfs-rdma >>>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, >>>> ibhosts, etc...) >>>> >>>> My last attempt was following the instructions from OFED 1.4 - I >>>> edited ofed.conf to add nfsrdma=y, and attempted a default install. >>>> However after running install.pl, that change appears to have been >>>> overwritten. >>>> >>>> I am also having problems updating mount.nfs on this machine. Both >>>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: >>>> "blkid/blkid.h: No such file or directory", and I cannot yet find a >>>> source for this file. >>>> >>>> Finally, my understanding is that nfs-rdma is in beta status for >>>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there >>>> any issues I should be aware of? >>>> >>>> thanks, >>>> >>>> Ross >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >>>> The SAQ Group >>>> >>>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ >>>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales >>>> Company Number: 06481952 >>>> >>>> http://www.saqnet.co.uk AS29219 >>>> >>>> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. >>>> >>>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. >>>> >>>> ISPA Member >>>> >>>> >>>> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From myxiplx at googlemail.com Fri Jun 26 11:03:14 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Fri, 26 Jun 2009 19:03:14 +0100 Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 In-Reply-To: <4A450222.6040104@nasa.gov> References: <7b160d240906260335l59ca2ff2j1be2df02c0f38c4a@mail.gmail.com> <7b160d240906260425i1ddf7083yc63c4e803d0cafe4@mail.gmail.com> <7b160d240906260606r2fd4c483lb66c248e3163bcd@mail.gmail.com> <7b160d240906260731o4e47d19bwa492771a71258f32@mail.gmail.com> <4A450222.6040104@nasa.gov> Message-ID: <7b160d240906261103g3473f461qf964f1211324a6a6@mail.gmail.com> Hi Jeff, Yeah, I tried that, but while it seemed to work ok on the server with the development tools installed, it didn't work on the live server. The install script kept trying to recompile the kernel-ib package which isn't possible on the live environment. Copying the pre-compiled kernel-ip package from the development to the live server, and manually installing that seemed to work, but like I said above, the problem now seems to be with the mount.nfs and mount.rnfs commands - neither understood the rdma option. Ross On Fri, Jun 26, 2009 at 6:15 PM, Jeff Becker wrote: > Hi Ross. The easiest way to install NFSRDMA is to do a custom install > and select it manually from the kernel-ib modules. Also be sure to > select rnfs-utils since that includes a special mount.rnfs that is > needed to understand RDMA mounts. > > -jeff > > Ross Smith wrote: >> Latest news:  I've bodged the installation, but I'm having problems >> with nfs.mount now. >> >> Since everything worked with nfs-rdma on the development server, I >> manually copied the kerel-ib rpm file from there, and installed that >> on top of the default infiniband install on the live server, using the >> command: >> >> # rpm -iv --replacepkgs kernel-ib....... >> >> With that done modprobe xprtrdma now works. >> >> Since the kernel-ib file appears to be the only one modified as >> nfsrdma support is added, I assume this is all that's needed.  Can >> anybody confirm if this is ok, or whether I'm going to break things >> horribly by doing this? >> >> I also have nfs v1.1.6 installed, but it doesn't seem to understand >> the rdma option.  The system is still using tcp by default, and if I >> try to use rdma, I get this: >> >> # mount -o rdma, port=20049 server:path /mnt >> mount.nfs: Unsupported nfs mount option: rdma >> >> However, nfs is definitely the correct version: >> # mount.nfs -V >> mount.nfs (linux nfs-utils 1.1.6) >> >> Even using the mount.rnfs utility supplied with OFED doesn't help: >> # rpm -iv rnfs-utils-1.1.5-2.OFED.i386.rpm >> # mount.rnfs server:path /mnt -o rdma,port=20049 >> mount.rnfs: Unsupported nfs mount option: rdma >> >> And if I use mount.rnfs without the rdma option, "cat /proc/mounts" >> reports that it's just using tcp. >> >> I'm getting read speeds of around 400MB/s using regular NFS which >> isn't bad, but I know I can achieve over 900MB/s with this server over >> Infiniband, so there's still some way to go. >> >> Ross >> >> >> >> On Fri, Jun 26, 2009 at 2:06 PM, Ross Smith wrote: >> >>> Ok, I've made a little more progress. >>> >>> The issue with nfs-utils was solved by installing efsprogs-devel, and >>> mount.nfs is now reporting that it's running v1.1.6. >>> >>> I still can't install OFED with nfs-rdma on the live server though, it >>> appears that it's unable to use the rpm package to install with nfs >>> support. >>> >>> I've now discovered how to use the ofed.conf file, and this is the >>> procedure I'm following: >>> >>> - On the development server, I run install.pl and perform a standard install >>> - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y >>> - Still on the development server I run: >>>  # install.pl -c ofed.conf >>> - That can be seen to install most items from the rpm, and to >>> recompile kernel-ib with the nfsrdma option. >>> - I then package up the OFED folder with tar, and send it to the live server >>> - On the live server I run the same command: >>>  # install.pl -c ofed.conf >>> >>> Despite it running with exactly the same options, I get the error: >>> >>> "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM. >>> Please install the corresponding kernel-source or kerenel-devel RPM." >>> >>> If I exclude the nfsrdma option and repeat all the above steps, the >>> script installs fine from the pre-compiled RPM. >>> >>> It looks to me like a problem with the install script when choosing nfs-rdma. >>> >>> Does anybody have any suggestions as to how I can work around this? >>> Am I able to simply install the RPM files manually? >>> >>> Ross >>> >>> >>> >>> On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith wrote: >>> >>>> Thanks Robert, >>>> >>>> Looking through your notes, we've followed similar steps.  I should >>>> mention that I do have Infiniband working on this server, including >>>> ipoib.  It's purely the nfs-rdma part I'm struggling with. >>>> >>>> My latest attempt has been to attempt a custom install, selecting >>>> nfs-rdma manually.  That worked fine on the development server, but >>>> when I attempted to run it on the live server I had an error: >>>> >>>> "/lib/modules/....../build/scripts is required to build kernel-ib RPM. >>>> Please install the corresponding kernel-source or kernel-devel RPM. >>>> tk rpm is required to install ibutils" >>>> >>>> The problem is, for XenServer, you can't get the kernel-source on the >>>> live server, instead Citrix ship a DDK (driver development kit) >>>> virtual machine, which is a pre-built system with all the kernel >>>> sources you need. >>>> >>>> OFED builds fine on the DDK machine, and for the default installation, >>>> I can compile everything there and then transfer the OFED folder (with >>>> the compiled RPM files) to the live server.  At that point install.pl >>>> runs fine, without needing to re-compile things. >>>> >>>> However, it appears that when I choose some elements in the custom >>>> install, it is not using these RPM files, and is trying to compile >>>> kernel-ib again from scratch. >>>> >>>> I have attached a file listing the custom modules I have chosen in >>>> case it helps. >>>> >>>> Ross >>>> >>>> >>>> >>>> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley wrote: >>>> >>>>> Hi Ross, >>>>> >>>>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos >>>>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is >>>>> some notes I took (I used the GUI installer) >>>>> >>>>> Hope this helps, >>>>> >>>>> Rob >>>>> >>>>> 1.      Download the OFED package from >>>>> http://www.openfabrics.org/downloads/OFED/ >>>>> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz >>>>> 3.      Check for the dependencies by running ./install.pl and choosing >>>>> documentation ("0"). >>>>> 4.      Install all required dependencies using yum install package nam. >>>>> For Centos dependencies try: yum install gcc libstdc++-devel >>>>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build >>>>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel >>>>> zlib-devel You may need to run the yum command three times (It takes a >>>>> few runs for all the dependencies to sort themselves out) >>>>> 5.      It is recommended you now upgrade the kernel to make sure the >>>>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade >>>>> kernel-xen for Xen setups) >>>>> 6.      Run ./install.pl again but this time choose to install ("2") and >>>>> choose custom install ("4").  See the appendix for a recommendation on >>>>> which modules to install. >>>>> 7.      Install should now complete, if it fails check the log (Most >>>>> likely you have not installed a required software package - go back to >>>>> step 3) >>>>> 8.      Say yes to configuring IPOIB (If you plan to use it) and >>>>> manually set the IP and subnet. When it asks for a network it wants the >>>>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0 >>>>> subnet then the "Network" would be 192.168.10.0) >>>>> 9.      Reboot >>>>> 10.     Check the install using /etc/init.d/openibd status. This should >>>>> show one interface for each Infiniband port (First will be "ib0"). You >>>>> should also see quite a few modules loaded, the important ones are >>>>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core >>>>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib - >>>>> IPOIB and ib_uverbs - Direct/Verb) >>>>> 11.     Run ibstat - This should display the installed devices and their >>>>> firmware versions and hardware / GUI IDs. If you need to update the >>>>> firmware for Mellanox see >>>>> http://www.mellanox.com/support/firmware_download.php I find it easier >>>>> to update using Windows and the WinOF software (Windows equivalent of >>>>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ). >>>>> >>>>> -----Original Message----- >>>>> From: general-bounces at lists.openfabrics.org >>>>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith >>>>> Sent: 26 June 2009 11:36 >>>>> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org >>>>> Cc: Yehonatan Yossef >>>>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1 >>>>> >>>>> Hello everyone, >>>>> >>>>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually >>>>> Citrix XenServer), with OFED-1.4.1? >>>>> >>>>> The default install of OFED doesn't include nfs-rdma, and I'm >>>>> struggling to work out which components I do and don't need for a >>>>> custom build. >>>>> >>>>> What I would like to include is: >>>>> - Core infiniband components >>>>> - Drivers for Mellanox Infinihost III Lx (MHES14) >>>>> - IPoIB >>>>> - nfs-rdma >>>>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches, >>>>> ibhosts, etc...) >>>>> >>>>> My last attempt was following the instructions from OFED 1.4 - I >>>>> edited ofed.conf to add nfsrdma=y, and attempted a default install. >>>>> However after running install.pl, that change appears to have been >>>>> overwritten. >>>>> >>>>> I am also having problems updating mount.nfs on this machine.  Both >>>>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error: >>>>> "blkid/blkid.h: No such file or directory", and I cannot yet find a >>>>> source for this file. >>>>> >>>>> Finally, my understanding is that nfs-rdma is in beta status for >>>>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there >>>>> any issues I should be aware of? >>>>> >>>>> thanks, >>>>> >>>>> Ross >>>>> _______________________________________________ >>>>> general mailing list >>>>> general at lists.openfabrics.org >>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>>> >>>>> To unsubscribe, please visit >>>>> http://openib.org/mailman/listinfo/openib-general >>>>> >>>>> The SAQ Group >>>>> >>>>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ >>>>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales >>>>> Company Number: 06481952 >>>>> >>>>> http://www.saqnet.co.uk AS29219 >>>>> >>>>> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. >>>>> >>>>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. >>>>> >>>>> ISPA Member >>>>> >>>>> >>>>> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> > > From zafargilani at gmail.com Fri Jun 26 11:03:45 2009 From: zafargilani at gmail.com (Zafar Gilani) Date: Fri, 26 Jun 2009 23:03:45 +0500 Subject: [ofa-general] Sending two integers via RDMA_WRITE In-Reply-To: <8d9c773c0906260607y7aa4a1a6y55742c45ba9751ed@mail.gmail.com> References: <7d4423d30905250836r431e7d1eh170f3bc22b731a55@mail.gmail.com> <4A1AD987.2080200@gmail.com> <7d4423d30905251108m5bc951ach4da2027698b0ffb6@mail.gmail.com> <4A1AF13D.8040506@gmail.com> <7d4423d30905252245o869fd93p96facb1d1b2b94e6@mail.gmail.com> <8d9c773c0906260607y7aa4a1a6y55742c45ba9751ed@mail.gmail.com> Message-ID: <7d4423d30906261103t2a0dc33cp14e04742cd0086ff@mail.gmail.com> Dear Thierry, I did not try it any further, since my basic aim was to run this with JNI. However Verbs code crashes the JVM and I am still trying to figure that out. Besides this I have now used SDP to run Java code on InfiniBand. On Fri, Jun 26, 2009 at 6:07 PM, Thierry wrote: > Hi, > > Have you fixed your problem ? I am also trying to pool CQ (associated > with the receive part) with an other program, but I get nothing. > > Regards, > > Thierry > > On Tue, May 26, 2009 at 7:45 AM, Zafar Gilani > wrote: > > To make things more clear, why won't you just poll the CQ for completion > > directly? (without using the CQ events) > > > > I believe that you will get a completion with error... > > > > Yes I tried polling directly and it returns a negative number. What is > the > > remedy for this? Is my for loop logically correct (client.c)? I also > tried > > polling the server CQ directly (server.c) and polling here also returns a > > negative number, which means that data write is not working properly thus > no > > completion events. What do you suggest I do? I am obviously lost! :( > > > > > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > -- Syed Zafar ul Hussan Gilani | BIT-7 Research Student | CHPSC MSP 2008-09 NUST SEECS | http://hpc.niit.edu.pk/~zafar -------------- next part -------------- An HTML attachment was scrubbed... URL: From akepner at sgi.com Fri Jun 26 13:29:42 2009 From: akepner at sgi.com (akepner at sgi.com) Date: Fri, 26 Jun 2009 13:29:42 -0700 Subject: [ofa-general] [PATCH] ipoib: fix racing uses of ipoib_neigh in CM with RCU Message-ID: <20090626202942.GO6566@sgi.com> There is a race between the neighbour cleanup code, and some uses of the closely related ipoib_neigh structure when using IPoIB-CM. For example, a IPoIB-CM connection may time out, invoking ipoib_neigh_free(), which cleans up and frees the struct ipoib_neigh, while at the same time a neighbour entry times out, invoking ipoib_neigh_cleanup(), cleaning up and freeing the ipoib_neigh structure a second time. The root cause is that the struct ipoib_neigh pointer that's stashed away in struct neighbour is read (and subsequently used) in ipoib_neigh_cleanup() without using locking that's consistent with other reads/writes of this pointer. (The pointer must be read before it can be known which lock to use, so it's difficult to avoid.) To fix this, use RCU methods to read/write the struct ipoib_neigh pointer that's stashed away in struct neighbour, cleanup (most of) the struct ipoib_neigh's internal state, and free the structure in an RCU callback. Updates of the struct ipoib_neigh pointer are protected by struct ipoib_dev_priv's lock. The pointer can be read, and most of the internal state of the ipoib_neigh can be used, without locking, inside an RCU read-side critical section. The internal state of the struct ipoib_neigh which requires mutual exclusion (the list_head element) is protected by a new spinlock. Signed-off-by: Arthur Kepner --- ipoib.h | 38 ++++++++++++++-- ipoib_cm.c | 23 +-------- ipoib_main.c | 127 +++++++++++++++++++++++++++++------------------------- ipoib_multicast.c | 56 ++++++++++++----------- 4 files changed, 138 insertions(+), 106 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 753a983..c84ab84 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -105,6 +105,7 @@ enum { MAX_SEND_CQE = 16, IPOIB_CM_COPYBREAK = 256, + IPOIB_FLAG_CM_CLEAR = 1, }; #define IPOIB_OP_RECV (1ul << 31) @@ -382,6 +383,10 @@ struct ipoib_path { }; struct ipoib_neigh { + spinlock_t lock; + int dead; + int rcu_flags; /* flags used during rcu callback */ + struct rcu_head rcu; struct ipoib_ah *ah; #ifdef CONFIG_INFINIBAND_IPOIB_CM struct ipoib_cm_tx *cm; @@ -409,15 +414,42 @@ static inline int ipoib_ud_need_sg(unsigned int ib_mtu) * sure that this pointer is stored aligned so that an unaligned * load is not needed to dereference it. */ -static inline struct ipoib_neigh **to_ipoib_neigh(struct neighbour *neigh) +static inline struct ipoib_neigh* ipoib_neigh_retrieve(struct neighbour *n) +{ + struct ipoib_neigh **np; + + np = (void*) n + ALIGN(offsetof(struct neighbour, ha) + + INFINIBAND_ALEN, sizeof(void *)); + + return rcu_dereference(*np); +} + +static inline void ipoib_neigh_assign(struct neighbour *n, + struct ipoib_neigh *in) +{ + struct ipoib_neigh **np; + + np = (void*) n + ALIGN(offsetof(struct neighbour, ha) + + INFINIBAND_ALEN, sizeof(void *)); + + rcu_assign_pointer(*np, in); +} + +static inline int ipoib_cm_get_rcu_flag(struct ipoib_neigh *neigh, int flag) +{ + return !!(neigh->rcu_flags & flag); +} + +static inline void ipoib_cm_set_rcu_flag(struct ipoib_neigh *neigh, int flag) { - return (void*) neigh + ALIGN(offsetof(struct neighbour, ha) + - INFINIBAND_ALEN, sizeof(void *)); + neigh->rcu_flags |= flag; } struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neigh, struct net_device *dev); void ipoib_neigh_free(struct net_device *dev, struct ipoib_neigh *neigh); +void ipoib_neigh_tidy(struct net_device *dev, struct ipoib_neigh *neigh, + int cm); extern struct workqueue_struct *ipoib_workqueue; diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 181b1f3..f3eead1 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -806,12 +806,7 @@ void ipoib_cm_handle_tx_wc(struct net_device *dev, struct ib_wc *wc) neigh = tx->neigh; if (neigh) { - neigh->cm = NULL; - list_del(&neigh->list); - if (neigh->ah) - ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); - + ipoib_neigh_tidy(dev, neigh, 1); tx->neigh = NULL; } @@ -1226,12 +1221,7 @@ static int ipoib_cm_tx_handler(struct ib_cm_id *cm_id, neigh = tx->neigh; if (neigh) { - neigh->cm = NULL; - list_del(&neigh->list); - if (neigh->ah) - ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); - + ipoib_neigh_tidy(dev, neigh, 1); tx->neigh = NULL; } @@ -1315,13 +1305,8 @@ static void ipoib_cm_tx_start(struct work_struct *work) if (ret) { neigh = p->neigh; - if (neigh) { - neigh->cm = NULL; - list_del(&neigh->list); - if (neigh->ah) - ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); - } + if (neigh) + ipoib_neigh_tidy(dev, neigh, 1); list_del(&p->list); kfree(p); } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index e319d91..409453a 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -273,18 +273,8 @@ static void path_free(struct net_device *dev, struct ipoib_path *path) spin_lock_irqsave(&priv->lock, flags); - list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) { - /* - * It's safe to call ipoib_put_ah() inside priv->lock - * here, because we know that path->ah will always - * hold one more reference, so ipoib_put_ah() will - * never do more than decrement the ref count. - */ - if (neigh->ah) - ipoib_put_ah(neigh->ah); - + list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) ipoib_neigh_free(dev, neigh); - } spin_unlock_irqrestore(&priv->lock, flags); @@ -466,10 +456,7 @@ static void path_rec_completion(int status, path, neigh)); if (!ipoib_cm_get(neigh)) { - list_del(&neigh->list); - if (neigh->ah) - ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); + ipoib_neigh_tidy(dev, neigh, 0); continue; } } @@ -561,15 +548,16 @@ static void neigh_add_path(struct sk_buff *skb, struct net_device *dev) struct ipoib_neigh *neigh; unsigned long flags; + spin_lock_irqsave(&priv->lock, flags); + neigh = ipoib_neigh_alloc(skb_dst(skb)->neighbour, skb->dev); if (!neigh) { + spin_unlock_irqrestore(&priv->lock, flags); ++dev->stats.tx_dropped; dev_kfree_skb_any(skb); return; } - spin_lock_irqsave(&priv->lock, flags); - path = __path_find(dev, skb_dst(skb)->neighbour->ha + 4); if (!path) { path = path_rec_create(dev, skb_dst(skb)->neighbour->ha + 4); @@ -591,10 +579,7 @@ static void neigh_add_path(struct sk_buff *skb, struct net_device *dev) if (!ipoib_cm_get(neigh)) ipoib_cm_set(neigh, ipoib_cm_create_tx(dev, path, neigh)); if (!ipoib_cm_get(neigh)) { - list_del(&neigh->list); - if (neigh->ah) - ipoib_put_ah(neigh->ah); - ipoib_neigh_free(dev, neigh); + ipoib_neigh_tidy(dev, neigh, 0); goto err_drop; } if (skb_queue_len(&neigh->queue) < IPOIB_MAX_PATH_REC_QUEUE) @@ -709,30 +694,22 @@ static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev) unsigned long flags; if (likely(skb_dst(skb) && skb_dst(skb)->neighbour)) { - if (unlikely(!*to_ipoib_neigh(skb_dst(skb)->neighbour))) { + struct neighbour *n = skb_dst(skb)->neighbour; + rcu_read_lock(); + neigh = ipoib_neigh_retrieve(n); + if (unlikely(!neigh)) { + rcu_read_unlock(); ipoib_path_lookup(skb, dev); return NETDEV_TX_OK; } - neigh = *to_ipoib_neigh(skb_dst(skb)->neighbour); - - if (unlikely((memcmp(&neigh->dgid.raw, - skb_dst(skb)->neighbour->ha + 4, + if (unlikely((memcmp(&neigh->dgid.raw, n->ha +4, sizeof(union ib_gid))) || (neigh->dev != dev))) { spin_lock_irqsave(&priv->lock, flags); - /* - * It's safe to call ipoib_put_ah() inside - * priv->lock here, because we know that - * path->ah will always hold one more reference, - * so ipoib_put_ah() will never do more than - * decrement the ref count. - */ - if (neigh->ah) - ipoib_put_ah(neigh->ah); - list_del(&neigh->list); - ipoib_neigh_free(dev, neigh); + ipoib_neigh_tidy(dev, neigh, 0); spin_unlock_irqrestore(&priv->lock, flags); + rcu_read_unlock(); ipoib_path_lookup(skb, dev); return NETDEV_TX_OK; } @@ -740,10 +717,12 @@ static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev) if (ipoib_cm_get(neigh)) { if (ipoib_cm_up(neigh)) { ipoib_cm_send(dev, skb, ipoib_cm_get(neigh)); + rcu_read_unlock(); return NETDEV_TX_OK; } } else if (neigh->ah) { - ipoib_send(dev, skb, neigh->ah, IPOIB_QPN(skb_dst(skb)->neighbour->ha)); + ipoib_send(dev, skb, neigh->ah, IPOIB_QPN(n->ha)); + rcu_read_unlock(); return NETDEV_TX_OK; } @@ -755,6 +734,7 @@ static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev) ++dev->stats.tx_dropped; dev_kfree_skb_any(skb); } + rcu_read_unlock(); } else { struct ipoib_pseudoheader *phdr = (struct ipoib_pseudoheader *) skb->data; @@ -843,62 +823,91 @@ static void ipoib_neigh_cleanup(struct neighbour *n) struct ipoib_neigh *neigh; struct ipoib_dev_priv *priv = netdev_priv(n->dev); unsigned long flags; - struct ipoib_ah *ah = NULL; - neigh = *to_ipoib_neigh(n); + rcu_read_lock(); + + neigh = ipoib_neigh_retrieve(n); if (neigh) priv = netdev_priv(neigh->dev); - else + else { + rcu_read_unlock(); return; + } ipoib_dbg(priv, "neigh_cleanup for %06x %pI6\n", IPOIB_QPN(n->ha), n->ha + 4); spin_lock_irqsave(&priv->lock, flags); - - if (neigh->ah) - ah = neigh->ah; - list_del(&neigh->list); - ipoib_neigh_free(n->dev, neigh); - + ipoib_neigh_tidy(n->dev, neigh, 0); spin_unlock_irqrestore(&priv->lock, flags); - - if (ah) - ipoib_put_ah(ah); + rcu_read_unlock(); } -struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, +struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *n, struct net_device *dev) { struct ipoib_neigh *neigh; - neigh = kmalloc(sizeof *neigh, GFP_ATOMIC); + neigh = kzalloc(sizeof *neigh, GFP_ATOMIC); if (!neigh) return NULL; - neigh->neighbour = neighbour; + spin_lock_init(&neigh->lock); + neigh->neighbour = n; neigh->dev = dev; - *to_ipoib_neigh(neighbour) = neigh; skb_queue_head_init(&neigh->queue); - ipoib_cm_set(neigh, NULL); - + ipoib_neigh_assign(n, neigh); return neigh; } -void ipoib_neigh_free(struct net_device *dev, struct ipoib_neigh *neigh) +static void ipoib_neigh_free_rcu(struct rcu_head *head) { + struct ipoib_neigh *neigh = container_of(head, struct ipoib_neigh, rcu); + struct net_device *dev = neigh->dev; struct sk_buff *skb; - *to_ipoib_neigh(neigh->neighbour) = NULL; + while ((skb = __skb_dequeue(&neigh->queue))) { ++dev->stats.tx_dropped; dev_kfree_skb_any(skb); } + + if (ipoib_cm_get_rcu_flag(neigh, IPOIB_FLAG_CM_CLEAR)) + ipoib_cm_set(neigh, NULL); + if (ipoib_cm_get(neigh)) ipoib_cm_destroy_tx(ipoib_cm_get(neigh)); + + if (neigh->ah) + ipoib_put_ah(neigh->ah); + kfree(neigh); } +void ipoib_neigh_free(struct net_device *dev, struct ipoib_neigh *neigh) +{ + struct neighbour *n = neigh->neighbour; + ipoib_neigh_assign(n, NULL); + + call_rcu(&neigh->rcu, ipoib_neigh_free_rcu); +} + +void ipoib_neigh_tidy(struct net_device *dev, struct ipoib_neigh *neigh, int cm) +{ + spin_lock(&neigh->lock); + if (neigh->dead) { + /* lost the race */ + spin_unlock(&neigh->lock); + return; + } + neigh->dead = 1; + list_del(&neigh->list); + spin_unlock(&neigh->lock); + if (cm) + ipoib_cm_set_rcu_flag(neigh, IPOIB_FLAG_CM_CLEAR); + ipoib_neigh_free(dev, neigh); +} + static int ipoib_neigh_setup_dev(struct net_device *dev, struct neigh_parms *parms) { parms->neigh_cleanup = ipoib_neigh_cleanup; @@ -1376,6 +1385,8 @@ static void ipoib_remove_one(struct ib_device *device) dev_list = ib_get_client_data(device, &ipoib_client); + rcu_barrier(); + list_for_each_entry_safe(priv, tmp, dev_list, list) { ib_unregister_event_handler(&priv->event_handler); diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c index a0e9753..ec96f82 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c @@ -76,17 +76,8 @@ static void ipoib_mcast_free(struct ipoib_mcast *mcast) spin_lock_irq(&priv->lock); - list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) { - /* - * It's safe to call ipoib_put_ah() inside priv->lock - * here, because we know that mcast->ah will always - * hold one more reference, so ipoib_put_ah() will - * never do more than decrement the ref count. - */ - if (neigh->ah) - ipoib_put_ah(neigh->ah); + list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) ipoib_neigh_free(dev, neigh); - } spin_unlock_irq(&priv->lock); @@ -658,9 +649,7 @@ void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb) if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags) || !priv->broadcast || !test_bit(IPOIB_MCAST_FLAG_ATTACHED, &priv->broadcast->flags)) { - ++dev->stats.tx_dropped; - dev_kfree_skb_any(skb); - goto unlock; + goto drop; } mcast = __ipoib_mcast_find(dev, mgid); @@ -673,9 +662,7 @@ void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb) if (!mcast) { ipoib_warn(priv, "unable to allocate memory for " "multicast structure\n"); - ++dev->stats.tx_dropped; - dev_kfree_skb_any(skb); - goto out; + goto drop; } set_bit(IPOIB_MCAST_FLAG_SENDONLY, &mcast->flags); @@ -705,25 +692,42 @@ void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb) mcast = NULL; } -out: if (mcast && mcast->ah) { - if (skb_dst(skb) && - skb_dst(skb)->neighbour && - !*to_ipoib_neigh(skb_dst(skb)->neighbour)) { - struct ipoib_neigh *neigh = ipoib_neigh_alloc(skb_dst(skb)->neighbour, - skb->dev); + if (skb_dst(skb) && skb_dst(skb)->neighbour) { + struct neighbour *n = skb_dst(skb)->neighbour; + struct ipoib_neigh *neigh; + int drop = 0; + + rcu_read_lock(); + neigh = ipoib_neigh_retrieve(n); + + if (!neigh) + neigh = ipoib_neigh_alloc(n, skb->dev); if (neigh) { - kref_get(&mcast->ah->ref); - neigh->ah = mcast->ah; - list_add_tail(&neigh->list, &mcast->neigh_list); + spin_lock(&neigh->lock); + if (!neigh->dead) { + kref_get(&mcast->ah->ref); + neigh->ah = mcast->ah; + list_add_tail(&neigh->list, &mcast->neigh_list); + } else + drop = 1; + spin_unlock(&neigh->lock); } + rcu_read_unlock(); + if (drop) + goto drop; } ipoib_send(dev, skb, mcast->ah, IB_MULTICAST_QPN); } -unlock: + spin_unlock_irqrestore(&priv->lock, flags); + return; + +drop: + ++dev->stats.tx_dropped; + dev_kfree_skb_any(skb); spin_unlock_irqrestore(&priv->lock, flags); } From sashak at voltaire.com Fri Jun 26 14:22:33 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 27 Jun 2009 00:22:33 +0300 Subject: [ofa-general] Re: [PATCH] iblinkinfo: optomize -D operation In-Reply-To: <20090624180405.076e8555.weiny2@llnl.gov> References: <20090624180405.076e8555.weiny2@llnl.gov> Message-ID: <20090626212233.GC20617@me> On 18:04 Wed 24 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Thu, 18 Jun 2009 14:07:13 -0700 > Subject: [PATCH] iblinkinfo: optomize -D operation > > Remove -f option and combine with -D > only scan part of the fabric when -D is specified. > add -a option which allows for printing "all" of the switches found when > performing a partial scan > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 26 14:22:53 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 27 Jun 2009 00:22:53 +0300 Subject: [ofa-general] Re: [PATCH] ibqueryerrors: optomize -D operation In-Reply-To: <20090624180406.fa03303a.weiny2@llnl.gov> References: <20090624180406.fa03303a.weiny2@llnl.gov> Message-ID: <20090626212253.GD20617@me> On 18:04 Wed 24 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Wed, 24 Jun 2009 17:52:37 -0700 > Subject: [PATCH] ibqueryerrors: optomize -D operation > > only scan part of the fabric when -D is specified. > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 26 14:23:22 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 27 Jun 2009 00:23:22 +0300 Subject: [ofa-general] Re: [PATCH] ibprintca.pl: fix corner case: print switch information for CA's found at the end of the cache file. In-Reply-To: <20090624180408.1550f55c.weiny2@llnl.gov> References: <20090624180408.1550f55c.weiny2@llnl.gov> Message-ID: <20090626212322.GE20617@me> On 18:04 Wed 24 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Tue, 23 Jun 2009 13:29:19 -0700 > Subject: [PATCH] ibprintca.pl: fix corner case: print switch information for CA's found at the end of the cache file. > > > Signed-off-by: Ira Weiny Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 26 14:27:20 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 27 Jun 2009 00:27:20 +0300 Subject: [ofa-general] Re: [PATCH] iblinkinfo, ibqueryerrors: clean up ibdiag_process_opts call In-Reply-To: <20090624182235.601261e3.weiny2@llnl.gov> References: <20090624182235.601261e3.weiny2@llnl.gov> Message-ID: <20090626212720.GF20617@me> On 18:22 Wed 24 Jun , Ira Weiny wrote: > > From: Ira Weiny > Date: Wed, 24 Jun 2009 18:22:14 -0700 > Subject: [PATCH] iblinkinfo, ibqueryerrors: clean up ibdiag_process_opts call > > > Signed-off-by: Ira Weiny Applied. Thanks. The comment is below. > --- > infiniband-diags/src/iblinkinfo.c | 2 +- > infiniband-diags/src/ibqueryerrors.c | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/infiniband-diags/src/iblinkinfo.c b/infiniband-diags/src/iblinkinfo.c > index f2dca6b..bba5c5c 100644 > --- a/infiniband-diags/src/iblinkinfo.c > +++ b/infiniband-diags/src/iblinkinfo.c > @@ -293,7 +293,7 @@ main(int argc, char **argv) > }; > char usage_args[] = ""; > > - ibdiag_process_opts(argc, argv, "sDLG", "SDandlpgR", opts, process_opt, > + ibdiag_process_opts(argc, argv, NULL, "SDandlpgRGL", opts, process_opt, > usage_args, NULL); This is correct. And actually it is not needed to specify custom options letters in 'exclude_string' (4th parameter), only ones which can conflict with common options should be enough. Sasha From sashak at voltaire.com Fri Jun 26 14:29:38 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 27 Jun 2009 00:29:38 +0300 Subject: [ofa-general] Re: [PATCH] opensm/libvendor/osm_vendor_(ibumad mlx)_sa.c: Handle malloc failure in __osmv_send_sa_req In-Reply-To: <20090625125316.GA15384@comcast.net> References: <20090625125316.GA15384@comcast.net> Message-ID: <20090626212938.GG20617@me> On 08:53 Thu 25 Jun , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 26 14:30:02 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 27 Jun 2009 00:30:02 +0300 Subject: [ofa-general] Re: [PATCH] opensm/man/osmtest.8: Update email address In-Reply-To: <20090625125408.GB15384@comcast.net> References: <20090625125408.GB15384@comcast.net> Message-ID: <20090626213002.GH20617@me> On 08:54 Thu 25 Jun , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From kamran at wbitt.com Fri Jun 26 14:47:29 2009 From: kamran at wbitt.com (Muhammad Kamran Azeem) Date: Sat, 27 Jun 2009 00:47:29 +0300 Subject: [ofa-general] unsubscribe In-Reply-To: <20090625190003.4712E102029A@openfabrics.org> References: <20090625190003.4712E102029A@openfabrics.org> Message-ID: <88444da4a67da1cf3474f11a19d7fcf7.squirrel@mail.wbitt.com> On Thu, June 25, 2009 10:00 pm, general-request at lists.openfabrics.org wrote: > Send general mailing list submissions to > general at lists.openfabrics.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > or, via email, send a message with subject or body 'help' to > general-request at lists.openfabrics.org > > You can reach the person managing the list at > general-owner at lists.openfabrics.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of general digest..." > > > Today's Topics: > > 1. Re: Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to > monitor rdma_cm connections (Moni Shoua) > 2. Re: Re: [PATCH v2 RESEND] rdma_cm: Add debugfs entries to > monitor rdma_cm connections (Moni Shoua) > 3. OFEM installation problem (Boyan Lazarov) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 25 Jun 2009 18:23:27 +0300 > From: Moni Shoua > Subject: Re: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs > entries to monitor rdma_cm connections > To: Jason Gunthorpe > Cc: Roland Dreier , anatolyg at voltaire.com, > OpenFabrics General > Message-ID: <4A43966F.70605 at Voltaire.COM> > Content-Type: text/plain; charset=us-ascii > > Jason Gunthorpe wrote: >> On Wed, Jun 24, 2009 at 06:55:43PM +0300, Moni Shoua wrote: >> >>> I believe that Jason and I still disagree but... >> >>> Jason suggests that I implement this feature with netlink. This >>> approach might have an advantage but if I understand it right this >>> approach requires a patch also to some user application in order to >>> take advantage of this patch. >> >> The kernel devs have made it clear that the preferred way to export >> this information is through netlink and/or a file in /proc/net/.. >> >> You never got an answer if /proc/net is truely discouraged or not. > As I remember I got opinions against /proc/net. I'm willing to move the > virtual files to there if this is acceptable by all. Otherwise I will > find myself in the same situation. > > Roland, Sean - what do you think? > >> >> debugfs is completely useless because it cannot be used for actual >> end-user interrogation and the rules are it should not be used as a >> stable kernel-user interface. (ie is NOT a new dumping ground like >> /proc/ was) >> > I don't know about that. The only fault of debugfs that I'm aware of is > that distros don't mount it by default > >>> Also, I think that there is a value for a virtual text file under >>> debugfs to monitor rdma_cm connections easily and without any other >>> special app besides 'cat' (just like in IPoIB). >> >> If you have the netlink based program there really is no point in >> including something under debugfs. It is just bloat. >> >>> Finally, this implementation doesn't contradict netlink >>> implementation in the future (and it won't be the first time). >> >> As I said before, the stack has reached a level of maturity that new >> stuff going in should meaningfully 'move the ball forward' toward a >> mature and complete stack. In the context of state reporting that >> means a stable user interface that programs like lsof, netstat, etc, >> can rely upon. >> >> Putting something under debugfs clearly does nothing to advance that >> goal. >> >> Further, there is clearly alot of state information we should be >> exporting to userspace. RDMA-CM stuff is only a tiny portion. netlink >> is the kernel devs answer to all of this. >> > I just want to emphasize that my goal for now is to supply a simple way to > monitor rdma_cm connections, nothing more and nothing less. I don't want > it to depend on special userspace application and in fact, I just want to > add a capability like we have in many other modules. This is a feature > that we always get request for from customers and developers. > > > > > ------------------------------ > > Message: 2 > Date: Thu, 25 Jun 2009 18:24:35 +0300 > From: Moni Shoua > Subject: Re: [ofa-general] Re: [PATCH v2 RESEND] rdma_cm: Add debugfs > entries to monitor rdma_cm connections > To: Thierry > Cc: Roland Dreier , anatolyg at voltaire.com, > OpenFabrics General > Message-ID: <4A4396B3.9050601 at Voltaire.COM> > Content-Type: text/plain; charset=UTF-8 > > Thanks :) > > > > ------------------------------ > > Message: 3 > Date: Thu, 25 Jun 2009 20:40:38 +0200 > From: Boyan Lazarov > Subject: [ofa-general] OFEM installation problem > To: general at lists.openfabrics.org > Message-ID: <200906252040.39051.bsl at mek.dtu.dk> > Content-Type: text/plain; charset="us-ascii" > > Hello, > > We have installed OFED 1.4.1 on Scientific Linux 53 (RH derivative). When > I > run the osu_* tests (OpenMPI) as root it seems that everything is running > fine and the mpi is using openib for communications. However if I run the > test as an ordinary user I am getting the following: > > WARNING: There was an error initializing an OpenFabrics device. > > and openmpi is using tcp for communications - the speed is much lower. > > Anyone with similar problem and any suggestions? > > Best regards, > Boyan > > > ------------------------------ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > End of general Digest, Vol 29, Issue 96 > *************************************** > -- http://www.wbitt.com Computer bugs are like ship-wrecks. They are not found, unless they want to be found. From arlin.r.davis at intel.com Fri Jun 26 16:46:27 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Fri, 26 Jun 2009 16:46:27 -0700 Subject: [ofa-general] [PATCH] dtestcm: new test to measure dapl connection rates. Message-ID: <9BD79D3A2315439CBD6AB463D89C8FC7@amr.corp.intel.com> new dapl client/server test added to measure connection rates of providers. Used to compare cma, scm, and other providers under development. dtestcm USAGE s: server c: connections (default = 1000) b: burst rate of conn_reqs (default = 100) m: multi-listens (set to burst setting ) v: verbose w: wait on event (default, polling) d: delay before accept h: hostname/address of server, specified on client P: provider name (default = OpenIB-v2-ib0) Signed-off-by: Arlin Davis --- test/dtest/Makefile.am | 6 +- test/dtest/dtestcm.c | 1087 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1092 insertions(+), 1 deletions(-) create mode 100644 test/dtest/dtestcm.c diff --git a/test/dtest/Makefile.am b/test/dtest/Makefile.am index a023c19..801d704 100755 --- a/test/dtest/Makefile.am +++ b/test/dtest/Makefile.am @@ -1,7 +1,10 @@ -bin_PROGRAMS = dtest +bin_PROGRAMS = dtest dtestcm dtest_SOURCES = dtest.c dtest_CFLAGS = -g -Wall -D_GNU_SOURCE +dtestcm_SOURCES = dtestcm.c +dtestcm_CFLAGS = -g -Wall -D_GNU_SOURCE + if EXT_TYPE_IB bin_PROGRAMS += dtestx dtestx_SOURCES = dtestx.c @@ -11,4 +14,5 @@ endif INCLUDES = -I $(srcdir)/../../dat/include dtest_LDADD = $(top_builddir)/dat/udat/libdat2.la +dtestcm_LDADD = $(top_builddir)/dat/udat/libdat2.la diff --git a/test/dtest/dtestcm.c b/test/dtest/dtestcm.c new file mode 100644 index 0000000..7bfe342 --- /dev/null +++ b/test/dtest/dtestcm.c @@ -0,0 +1,1087 @@ +/* + * Copyright (c) 2009 Intel Corporation. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: $ + */ +#include +#include +#include + +#ifdef DAPL_PROVIDER +#undef DAPL_PROVIDER +#endif + +#if defined(_WIN32) || defined(_WIN64) + +#include +#include +#include +#include +#include +#include +#include "..\..\..\..\etc\user\getopt.c" + +#define ((int)GetCurrentProcessId()) +#define F64x "%I64x" +#define F64d "%I64d" + +#ifdef DBG +#define DAPL_PROVIDER "ibnic0v2d" +#else +#define DAPL_PROVIDER "ibnic0v2" +#endif + +#define ntohll _byteswap_uint64 +#define htonll _byteswap_uint64 + +#else // _WIN32 || _WIN64 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define DAPL_PROVIDER "ofa-v2-mlx4_0-1" + +#define F64x "%"PRIx64"" +#define F64d "%"PRId64"" + + +#if __BYTE_ORDER == __BIG_ENDIAN +#define htonll(x) (x) +#define ntohll(x) (x) +#elif __BYTE_ORDER == __LITTLE_ENDIAN +#define htonll(x) bswap_64(x) +#define ntohll(x) bswap_64(x) +#endif + +#endif // _WIN32 || _WIN64 + +#define MAX_POLLING_CNT 50000 + +/* Header files needed for DAT/uDAPL */ +#include "dat2/udat.h" + +/* definitions */ +#define SERVER_CONN_QUAL 45248 +#define CONN_TIMEOUT (1000*1000*100) +#define CR_TIMEOUT DAT_TIMEOUT_INFINITE + +/* Global DAT vars */ +static DAT_IA_HANDLE h_ia = DAT_HANDLE_NULL; +static DAT_PZ_HANDLE h_pz = DAT_HANDLE_NULL; +static DAT_EP_HANDLE *h_ep; +static DAT_PSP_HANDLE *h_psp; +static DAT_CR_HANDLE h_cr = DAT_HANDLE_NULL; + +static DAT_EVD_HANDLE h_async_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_dto_req_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_dto_rcv_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_cr_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_conn_evd = DAT_HANDLE_NULL; + +static DAT_EP_ATTR ep_attr; +char hostname[256] = { 0 }; +char provider[64] = DAPL_PROVIDER; +char addr_str[INET_ADDRSTRLEN]; + +int status; + +/* timers */ +double start, stop, total_us, total_sec; + +struct dt_time { + double total; + double open; + double reg; + double unreg; + double pzc; + double pzf; + double evdc; + double evdf; + double cnoc; + double cnof; + double epc; + double epf; + double rtt; + double close; + double conn; +}; + +struct dt_time time; + +/* defaults */ +static int connected = 0; +static int multi_listens = 0; +static int server = 1; +static int waiting = 0; +static int verbose = 0; +static int cr_poll_count = 0; +static int conn_poll_count = 0; +static int delay = 0; +static int connections = 1000; +static int burst = 100; +static int port_id = SERVER_CONN_QUAL; + +/* forward prototypes */ +const char *DT_RetToString(DAT_RETURN ret_value); +const char *DT_EventToSTr(DAT_EVENT_NUMBER event_code); +void print_usage(void); +double get_time(void); +DAT_RETURN conn_client(void); +DAT_RETURN conn_server(void); +DAT_RETURN disconnect_eps(void); +DAT_RETURN create_events(void); +DAT_RETURN destroy_events(void); + +#define LOGPRINTF if (verbose) printf + +void flush_evds(void) +{ + DAT_EVENT event; + + /* Flush async error queue */ + printf(" ERR: Checking ASYNC EVD...\n"); + while (dat_evd_dequeue(h_async_evd, &event) == DAT_SUCCESS) { + printf(" ASYNC EVD ENTRY: handle=%p reason=%d\n", + event.event_data.asynch_error_event_data.dat_handle, + event.event_data.asynch_error_event_data.reason); + } +} + +int main(int argc, char **argv) +{ + int i, c, len; + DAT_RETURN ret; + + /* parse arguments */ + while ((c = getopt(argc, argv, "smwvb:c:d:h:P:p:")) != -1) { + switch (c) { + case 's': + server = 1; + break; + case 'm': + multi_listens = 1; + break; + case 'w': + waiting = 1; + break; + case 'c': + connections = atoi(optarg); + break; + case 'p': + port_id = atoi(optarg); + break; + case 'v': + verbose = 1; + fflush(stdout); + break; + case 'd': + delay = atoi(optarg); + break; + case 'b': + burst = atoi(optarg); + break; + case 'h': + server = 0; + strcpy(hostname, optarg); + break; + case 'P': + strcpy(provider, optarg); + break; + default: + print_usage(); + exit(-12); + } + } + +#if defined(_WIN32) || defined(_WIN64) + { + WSADATA wsaData; + + i = WSAStartup(MAKEWORD(2, 2), &wsaData); + if (i != 0) { + printf("%s WSAStartup(2.2) failed? (0x%x)\n", argv[0], + i); + fflush(stdout); + exit(1); + } + } +#endif + + if (!server) { + printf(" Running client on %s with %d connections\n", + provider, connections); + } else { + printf(" Running server on %s with %d connections\n", + provider, connections); + } + fflush(stdout); + + if (burst > connections) + burst = connections; + + + /* allocate EP handles for all connections */ + h_ep = (DAT_EP_HANDLE*)malloc(connections * sizeof(DAT_EP_HANDLE)); + if (h_ep == NULL) { + perror("malloc ep"); + exit(1); + } + memset(h_ep, 0, (burst * sizeof(DAT_PSP_HANDLE))); + + /* allocate PSP handles, check for multi-listens */ + if (multi_listens) + len = burst * sizeof(DAT_PSP_HANDLE); + else + len = sizeof(DAT_PSP_HANDLE); + + h_psp = (DAT_PSP_HANDLE*)malloc(len); + if (h_psp == NULL) { + perror("malloc psp"); + exit(1); + } + memset(h_psp, 0, len); + memset(&time, 0, sizeof(struct dt_time)); + + /* dat_ia_open, dat_pz_create */ + h_async_evd = DAT_HANDLE_NULL; + start = get_time(); + ret = dat_ia_open(provider, 8, &h_async_evd, &h_ia); + stop = get_time(); + time.open += ((stop - start) * 1.0e6); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error Adaptor open: %s\n", + DT_RetToString(ret)); + exit(1); + } else + LOGPRINTF(" Opened Interface Adaptor\n"); + + /* Create Protection Zone */ + start = get_time(); + LOGPRINTF(" Create Protection Zone\n"); + ret = dat_pz_create(h_ia, &h_pz); + stop = get_time(); + time.pzc += ((stop - start) * 1.0e6); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error creating Protection Zone: %s\n", + DT_RetToString(ret)); + exit(1); + } else + LOGPRINTF(" Created Protection Zone\n"); + + LOGPRINTF(" Create events\n"); + ret = create_events(); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error creating events: %s\n", + DT_RetToString(ret)); + goto cleanup; + } else { + LOGPRINTF(" Create events done\n"); + } + + /* create EP */ + memset(&ep_attr, 0, sizeof(ep_attr)); + ep_attr.service_type = DAT_SERVICE_TYPE_RC; + ep_attr.max_rdma_size = 0x10000; + ep_attr.max_recv_dtos = 1; + ep_attr.max_request_dtos = 1; + ep_attr.max_recv_iov = 1; + ep_attr.max_request_iov = 1; + ep_attr.request_completion_flags = DAT_COMPLETION_DEFAULT_FLAG; + + start = get_time(); + for (i = 0; i < connections; i++) { + ret = dat_ep_create(h_ia, h_pz, h_dto_rcv_evd, + h_dto_req_evd, h_conn_evd, + &ep_attr, &h_ep[i]); + } + stop = get_time(); + time.epc += ((stop - start) * 1.0e6); + time.total += time.epc; + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_ep_create: %s\n", + DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF(" EP created %p \n", h_ep[i]); + + /* create the service point for server listen */ + if (server) { + LOGPRINTF(" Creating server service point(s)\n"); + for (i = 0; i < burst; i++) { + ret = dat_psp_create(h_ia, + port_id+i, + h_cr_evd, + DAT_PSP_CONSUMER_FLAG, + &h_psp[i]); + + if (ret != DAT_SUCCESS) { + fprintf(stderr, " ERR psp_create: %s\n", + DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF(" psp_created for listen\n"); + + printf(" Server ready on port %d\n", + port_id+i); + + if (!multi_listens) + break; + } + } + + /* Connect all */ + if (server) + ret = conn_server(); + else + ret = conn_client(); + + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error %s: %s\n", + server ? "server()" : "client()", + DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF(" connect_ep complete\n"); + + connected = 1; + goto complete; + +cleanup: + flush_evds(); + goto bail; +complete: + + /* disconnect and free EP resources */ + if (h_ep[0]) { + /* unregister message buffers and tear down connection */ + LOGPRINTF(" Disconnect EPs\n"); + ret = disconnect_eps(); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error disconnect_eps: %s\n", + DT_RetToString(ret)); + goto bail; + } else { + LOGPRINTF(" disconnect_eps complete\n"); + } + } + + /* destroy server service point(s) */ + if ((server) && (h_psp[0] != DAT_HANDLE_NULL)) { + for (i = 0; i < burst; i++) { + ret = dat_psp_free(h_psp[i]); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_psp_free: %s\n", + DT_RetToString(ret)); + goto bail; + } else { + LOGPRINTF(" psp_free[%d] complete\n",i); + } + if (!multi_listens) + break; + } + } + + /* free EVDs */ + LOGPRINTF(" destroy events\n"); + ret = destroy_events(); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error destroy_events: %s\n", + DT_RetToString(ret)); + goto bail; + } else + LOGPRINTF(" destroy events done\n"); + + + /* Free protection domain */ + LOGPRINTF(" Freeing pz\n"); + start = get_time(); + ret = dat_pz_free(h_pz); + stop = get_time(); + time.pzf += ((stop - start) * 1.0e6); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error freeing PZ: %s\n", + DT_RetToString(ret)); + goto bail; + } else { + LOGPRINTF(" Freed pz\n"); + h_pz = NULL; + } + + /* close the device */ + LOGPRINTF(" Closing Interface Adaptor\n"); + start = get_time(); + ret = dat_ia_close(h_ia, DAT_CLOSE_ABRUPT_FLAG); + stop = get_time(); + time.close += ((stop - start) * 1.0e6); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error Adaptor close: %s\n", + DT_RetToString(ret)); + goto bail; + } else + LOGPRINTF(" Closed Interface Adaptor\n"); + + printf(" DAPL Connection Test Complete.\n"); + printf(" open: %10.2lf usec\n", time.open); + printf(" close: %10.2lf usec\n", time.close); + printf(" PZ create: %10.2lf usec\n", time.pzc); + printf(" PZ free: %10.2lf usec\n", time.pzf); + printf(" LMR create:%10.2lf usec\n", time.reg); + printf(" LMR free: %10.2lf usec\n", time.unreg); + printf(" EVD create:%10.2lf usec\n", time.evdc); + printf(" EVD free: %10.2lf usec\n", time.evdf); + printf(" EP create: %10.2lf usec avg\n", time.epc/connections); + printf(" EP free: %10.2lf usec avg\n", time.epf/connections); + if (!server) { + printf(" Connections: %8.2lf usec, CPS %7.2lf " + "Total %4.2lf secs, poll_cnt=%u, Num=%d\n", + (double)(time.conn/connections), + (double)(1/(time.conn/1000000/connections)), + (double)(time.conn/1000000), + conn_poll_count, connections); + } + printf(" TOTAL: %4.2lf sec\n", time.total/1000000); + fflush(stderr); fflush(stdout); +bail: + free(h_ep); + free(h_psp); + +#if defined(_WIN32) || defined(_WIN64) + WSACleanup(); +#endif + return (0); +} + +#if defined(_WIN32) || defined(_WIN64) + +void gettimeofday(struct timeval *t, char *jnk) +{ + SYSTEMTIME now; + GetLocalTime(&now); + t->tv_sec = now.wMinute * 60; + t->tv_sec += now.wSecond; + t->tv_usec = now.wMilliseconds; +} + +#endif + +double get_time(void) +{ + struct timeval tp; + + gettimeofday(&tp, NULL); + return ((double)tp.tv_sec + (double)tp.tv_usec * 1e-6); +} + +DAT_RETURN conn_server() +{ + DAT_RETURN ret; + DAT_EVENT event; + DAT_COUNT nmore; + DAT_CR_ARRIVAL_EVENT_DATA *cr_event = + &event.event_data.cr_arrival_event_data; + int i,bi; + unsigned char *buf; + DAT_CR_PARAM cr_param = { 0 }; + + printf(" Accepting...\n"); + for (i = 0; i < connections; i++) { + + /* poll for CR's */ + if (!waiting) { + cr_poll_count = 0; + while (DAT_GET_TYPE(dat_evd_dequeue(h_cr_evd, &event)) + == DAT_QUEUE_EMPTY) + cr_poll_count++; + } else { + ret = dat_evd_wait(h_cr_evd, CR_TIMEOUT, + 1, &event, &nmore); + if (ret != DAT_SUCCESS) { + fprintf(stderr, + " ERR: CR dat_evd_wait() %s\n", + DT_RetToString(ret)); + return ret; + } + } + + if (event.event_number != DAT_CONNECTION_REQUEST_EVENT) { + fprintf(stderr, " Error unexpected CR event : %s\n", + DT_EventToSTr(event.event_number)); + return (DAT_ABORT); + } + + + /* use to test rdma_cma timeout logic */ +#if defined(_WIN32) || defined(_WIN64) + if (delay) { + printf(" Accept delayed by %d seconds...\n", delay); + Sleep(delay * 1000); + } +#else + if (delay) { + printf(" Accept delayed by %d seconds...\n", delay); + sleep(delay); + } +#endif + /* accept connect request from client */ + h_cr = cr_event->cr_handle; + LOGPRINTF(" Accepting connect request from client\n"); + + /* private data - check and send it back */ + dat_cr_query(h_cr, DAT_CSP_FIELD_ALL, &cr_param); + + buf = (unsigned char *)cr_param.private_data; + LOGPRINTF(" CONN REQUEST Private Data %p[0]=%d [47]=%d\n", + buf, buf[0], buf[47]); + + for (bi = 0; bi < 48; bi++) { + if (buf[bi] != bi + 1) { + fprintf(stderr, " ERR on CONNECT REQUEST" + " private data: %p[%d]=%d s/be %d\n", + buf, bi, buf[bi], bi + 1); + dat_cr_reject(h_cr, 0, NULL); + return (DAT_ABORT); + } + buf[bi]++; /* change for trip back */ + } + +#ifdef TEST_REJECT_WITH_PRIVATE_DATA + printf(" REJECT request with 48 bytes of private data\n"); + ret = dat_cr_reject(h_cr, 48, cr_param.private_data); + printf("\n DAPL Test Complete. %s\n\n", + ret ? "FAILED" : "PASSED"); + exit(0); +#endif + ret = dat_cr_accept(h_cr, h_ep[i], 48, + cr_param.private_data); + + if (ret != DAT_SUCCESS) { + fprintf(stderr, " ERR dat_cr_accept: %s\n", + DT_RetToString(ret)); + return (ret); + } else + LOGPRINTF(" accept[%d] complete\n", i); + + event.event_number = 0; + } + + /* process the RTU, ESTABLISHMENT event */ + printf(" Completing...\n"); + for (i=0;iai_addr)->sin_addr.s_addr; +#else + if (getaddrinfo(hostname, NULL, NULL, &target) != 0) { + perror("\n remote name resolution failed!"); + exit(1); + } + rval = ((struct sockaddr_in *)target->ai_addr)->sin_addr.s_addr; +#endif + printf(" Connecting to Server: %s \n", hostname); + printf(" Address: %d.%d.%d.%d port %d\n", + (rval >> 0) & 0xff, (rval >> 8) & 0xff, + (rval >> 16) & 0xff, (rval >> 24) & 0xff, + port_id); + + raddr = *((DAT_IA_ADDRESS_PTR)target->ai_addr); + freeaddrinfo(target); + + for (i = 0; i < 48; i++) /* simple pattern in private data */ + pdata[i] = i + 1; + + printf(" Connecting...\n"); + start = get_time(); + for (i = 0; i < connections; i += burst) { + for (ii = 0; ii < burst; ii++) { /* conn_reqs */ + if (multi_listens) + conn_id = port_id + ii; + else + conn_id = port_id; + + ret = dat_ep_connect(h_ep[i+ii], &raddr, + conn_id, CONN_TIMEOUT, + 48, (DAT_PVOID) pdata, 0, + DAT_CONNECT_DEFAULT_FLAG); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " ERR dat_ep_connect: %s\n", + DT_RetToString(ret)); + return (ret); + } else + LOGPRINTF(" dat_ep_connect[%d] complete\n", + i+ii); + + } + for (ii = 0; ii < burst; ii++) { /* conn_events */ + if (!waiting) { + conn_poll_count = 0; + while (DAT_GET_TYPE(dat_evd_dequeue( + h_conn_evd, &event)) == + DAT_QUEUE_EMPTY) + conn_poll_count++; + } else { + ret = dat_evd_wait(h_conn_evd, CONN_TIMEOUT, + 1, &event, &nmore); + + if (ret != DAT_SUCCESS) { + fprintf(stderr, + " ERR: CONN evd_wait() %s\n", + DT_RetToString(ret)); + return ret; + } + } + +#ifdef TEST_REJECT_WITH_PRIVATE_DATA + if (event.event_number != + DAT_CONNECTION_EVENT_PEER_REJECTED) { + fprintf(stderr, " expected conn reject " + "event : %s\n", + DT_EventToSTr(event.event_number)); + return (DAT_ABORT); + } + + /* get the reject private data and validate */ + buf = (unsigned char *)conn_event->private_data; + printf(" Recv REJ with pdata %p[0]=%d [47]=%d\n", + buf, buf[0], buf[47]); + for (bi = 0; bi < 48; bi++) { + if (buf[bi] != idx + 2) { + fprintf(stderr, " client: Error" + " with REJECT event private" + " data: %p[%d]=%d s/be %d\n", + buf, bi, + buf[bi], bi + 2); + dat_ep_disconnect(h_ep[i+ii], 0); + return (DAT_ABORT); + } + } + printf("\n Rej Test Done. PASSED\n\n"); + exit(0); +#endif + if (event.event_number != + DAT_CONNECTION_EVENT_ESTABLISHED) { + fprintf(stderr, " Error unexpected conn " + "event : 0x%x %s\n", + event.event_number, + DT_EventToSTr(event.event_number)); + return (DAT_ABORT); + } + + /* check private data back from server */ + buf = (unsigned char *)conn_event->private_data; + + LOGPRINTF(" CONN[%d] Private Data " + "%p[0]=%d [47]=%d\n", + i+ii, buf, buf[0], buf[47]); + + for (bi = 0; bi < 48; bi++) { + if (buf[bi] != bi + 2) { + DAT_COUNT nmore; + fprintf(stderr, " ERR CONN event" + " pdata: %p[%d]=%d s/be %d\n", + buf, bi, buf[bi], + bi + 2); + dat_ep_disconnect(h_ep[i+ii], + DAT_CLOSE_ABRUPT_FLAG); + LOGPRINTF(" waiting for disc...\n"); + dat_evd_wait(h_conn_evd, + DAT_TIMEOUT_INFINITE, + 1, &event, &nmore); + return (DAT_ABORT); + } + } + event.event_number = 0; + } + } + + stop = get_time(); + time.conn += ((stop - start) * 1.0e6); + + printf("\n ALL %d CONNECTED on Client!\n\n", connections); + + return (DAT_SUCCESS); +} + +/* validate disconnected EP's and free them */ +DAT_RETURN disconnect_eps(void) +{ + DAT_RETURN ret; + DAT_EVENT event; + DAT_COUNT nmore; + int i,ii; + DAT_CONNECTION_EVENT_DATA *conn_event = + &event.event_data.connect_event_data; + + if (!connected) + return DAT_SUCCESS; + + /* + * Only the client needs to call disconnect. The server _should_ be able + * to just wait on the EVD associated with connection events for a + * disconnect request and then exit. + */ + if (!server) { + for (i = 0; i < connections; i++) { + LOGPRINTF(" dat_ep_disconnect\n"); + ret = dat_ep_disconnect(h_ep[i], + DAT_CLOSE_DEFAULT); + if (ret != DAT_SUCCESS) { + fprintf(stderr, + " Error disconnect: %s\n", + DT_RetToString(ret)); + return ret; + } else { + LOGPRINTF(" disconnect completed\n"); + } + } + } else { + LOGPRINTF(" Server waiting for disconnect...\n"); + } + + LOGPRINTF(" Wait for Disc event, free EPs as completed\n"); + start = get_time(); + for (i = 0; i < connections; i++) { + event.event_number = 0; + conn_event->ep_handle = NULL; + ret = dat_evd_wait(h_conn_evd, DAT_TIMEOUT_INFINITE, + 1, &event, &nmore); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_evd_wait: %s\n", + DT_RetToString(ret)); + return ret; + } else { + LOGPRINTF(" disc event[%d] complete," + " check for valid EP...\n", i); + } + + /* check for valid EP in creation list */ + for (ii = 0; ii < connections; ii++) { + if (h_ep[ii] == conn_event->ep_handle) { + LOGPRINTF(" valid EP[%d] %p !\n", + ii, h_ep[ii]); + ret = dat_ep_free(h_ep[ii]); + if (ret != DAT_SUCCESS) { + fprintf(stderr, + " ERR free EP[%d] %p: %s\n", + i, h_ep[ii], + DT_RetToString(ret)); + } else { + LOGPRINTF(" Freed EP[%d] %p\n", + i, h_ep[ii]); + h_ep[ii] = DAT_HANDLE_NULL; + } + break; + } else { + continue; + } + } + if (ii == connections) { + LOGPRINTF(" %s: invalid EP[%d] %p via DISC event!\n", + server ? "Server" : "Client", + i, conn_event->ep_handle); + return DAT_INVALID_HANDLE; + } + } + /* free EPs */ + stop = get_time(); + time.epf += ((stop - start) * 1.0e6); + time.total += time.epf; + return DAT_SUCCESS; +} + + + /* + * Create CR, CONN, and DTO events + */ +DAT_RETURN create_events(void) +{ + DAT_RETURN ret; + + /* create cr EVD */ + start = get_time(); + ret = dat_evd_create(h_ia, connections, DAT_HANDLE_NULL, + DAT_EVD_CR_FLAG, &h_cr_evd); + stop = get_time(); + time.evdc += ((stop - start) * 1.0e6); + time.total += time.evdc; + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_evd_create: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" cr_evd created %p\n", h_cr_evd); + } + + /* create conn EVD */ + ret = dat_evd_create(h_ia, + connections*2, + DAT_HANDLE_NULL, + DAT_EVD_CONNECTION_FLAG, &h_conn_evd); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_evd_create: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" con_evd created %p\n", h_conn_evd); + } + + /* create dto SND EVD */ + ret = dat_evd_create(h_ia, 1, NULL, + DAT_EVD_DTO_FLAG, &h_dto_req_evd); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_evd_create REQ: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" dto_req_evd created %p\n", + h_dto_req_evd); + } + + /* create dto RCV EVD */ + ret = dat_evd_create(h_ia, 1, NULL, + DAT_EVD_DTO_FLAG, &h_dto_rcv_evd); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error dat_evd_create RCV: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" dto_rcv_evd created %p\n", + h_dto_rcv_evd); + } + return DAT_SUCCESS; +} + +/* + * Destroy CR, CONN, CNO, and DTO events + */ + +DAT_RETURN destroy_events(void) +{ + DAT_RETURN ret; + + /* free cr EVD */ + if (h_cr_evd != DAT_HANDLE_NULL) { + LOGPRINTF(" Free cr EVD %p \n", h_cr_evd); + ret = dat_evd_free(h_cr_evd); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error freeing cr EVD: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" Freed cr EVD\n"); + h_cr_evd = DAT_HANDLE_NULL; + } + } + + /* free conn EVD */ + if (h_conn_evd != DAT_HANDLE_NULL) { + LOGPRINTF(" Free conn EVD %p\n", h_conn_evd); + ret = dat_evd_free(h_conn_evd); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error freeing conn EVD: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" Freed conn EVD\n"); + h_conn_evd = DAT_HANDLE_NULL; + } + } + + /* free RCV dto EVD */ + if (h_dto_rcv_evd != DAT_HANDLE_NULL) { + LOGPRINTF(" Free RCV dto EVD %p\n", h_dto_rcv_evd); + start = get_time(); + ret = dat_evd_free(h_dto_rcv_evd); + stop = get_time(); + time.evdf += ((stop - start) * 1.0e6); + time.total += time.evdf; + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error freeing dto EVD: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" Freed dto EVD\n"); + h_dto_rcv_evd = DAT_HANDLE_NULL; + } + } + + /* free REQ dto EVD */ + if (h_dto_req_evd != DAT_HANDLE_NULL) { + LOGPRINTF(" Free REQ dto EVD %p\n", h_dto_req_evd); + ret = dat_evd_free(h_dto_req_evd); + if (ret != DAT_SUCCESS) { + fprintf(stderr, " Error freeing dto EVD: %s\n", + DT_RetToString(ret)); + return (ret); + } else { + LOGPRINTF(" Freed dto EVD\n"); + h_dto_req_evd = DAT_HANDLE_NULL; + } + } + + return DAT_SUCCESS; +} + +/* + * Map DAT_RETURN values to readable strings, + * but don't assume the values are zero-based or contiguous. + */ +char errmsg[512] = { 0 }; +const char *DT_RetToString(DAT_RETURN ret_value) +{ + const char *major_msg, *minor_msg; + + dat_strerror(ret_value, &major_msg, &minor_msg); + + strcpy(errmsg, major_msg); + strcat(errmsg, " "); + strcat(errmsg, minor_msg); + + return errmsg; +} + +/* + * Map DAT_EVENT_CODE values to readable strings + */ +const char *DT_EventToSTr(DAT_EVENT_NUMBER event_code) +{ + unsigned int i; + static struct { + const char *name; + DAT_RETURN value; + } dat_events[] = { +# define DATxx(x) { # x, x } + DATxx(DAT_DTO_COMPLETION_EVENT), + DATxx(DAT_RMR_BIND_COMPLETION_EVENT), + DATxx(DAT_CONNECTION_REQUEST_EVENT), + DATxx(DAT_CONNECTION_EVENT_ESTABLISHED), + DATxx(DAT_CONNECTION_EVENT_PEER_REJECTED), + DATxx(DAT_CONNECTION_EVENT_NON_PEER_REJECTED), + DATxx(DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR), + DATxx(DAT_CONNECTION_EVENT_DISCONNECTED), + DATxx(DAT_CONNECTION_EVENT_BROKEN), + DATxx(DAT_CONNECTION_EVENT_TIMED_OUT), + DATxx(DAT_CONNECTION_EVENT_UNREACHABLE), + DATxx(DAT_ASYNC_ERROR_EVD_OVERFLOW), + DATxx(DAT_ASYNC_ERROR_IA_CATASTROPHIC), + DATxx(DAT_ASYNC_ERROR_EP_BROKEN), + DATxx(DAT_ASYNC_ERROR_TIMED_OUT), + DATxx(DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR), + DATxx(DAT_SOFTWARE_EVENT) +# undef DATxx + }; +# define NUM_EVENTS (sizeof(dat_events)/sizeof(dat_events[0])) + + for (i = 0; i < NUM_EVENTS; i++) { + if (dat_events[i].value == event_code) { + return (dat_events[i].name); + } + } + + return ("Invalid_DAT_EVENT_NUMBER"); +} + +void print_usage(void) +{ + printf("\n DAPL USAGE \n\n"); + printf("s: server\n"); + printf("c: connections (default = 1000)\n"); + printf("v: verbose\n"); + printf("w: wait on event (default, polling)\n"); + printf("d: delay before accept\n"); + printf("h: hostname/address of server, specified on client\n"); + printf("P: provider name (default = OpenIB-v2-ib0)\n"); + printf("\n"); +} + -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 26 16:47:42 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 26 Jun 2009 16:47:42 -0700 Subject: [ofa-general] [PATCH] dtestx: fix conn establishment event checking Message-ID: test is not catching error cases on client side when checking for event number and UD type && should have been || Signed-off-by: Arlin Davis --- test/dtest/dtestx.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/test/dtest/dtestx.c b/test/dtest/dtestx.c index 2d062f1..a14785b 100755 --- a/test/dtest/dtestx.c +++ b/test/dtest/dtestx.c @@ -344,12 +344,13 @@ void process_conn(int idx) status = dat_evd_wait(con_evd, CONN_TIMEOUT, 1, &event, &nmore); _OK(status, "CONN dat_evd_wait"); - LOGPRINTF("%s got connect[%d] event, pdata %p sz=%d\n", + LOGPRINTF("%s got connect[%d] event 0x%x, pdata %p sz=%d\n", server ? "Server" : "Client", idx, - conn_event->private_data, conn_event->private_data_size); + event.event_number, conn_event->private_data, + conn_event->private_data_size); /* Waiting on CR's or CONN_EST */ - if (event.event_number != DAT_CONNECTION_EVENT_ESTABLISHED && + if (event.event_number != DAT_CONNECTION_EVENT_ESTABLISHED || (ud_test && event.event_number != DAT_IB_UD_CONNECTION_EVENT_ESTABLISHED)) { printf("unexpected event, !conn established: 0x%x\n", -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 26 16:48:23 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 26 Jun 2009 16:48:23 -0700 Subject: [ofa-general] [PATCH] scm: EP reference in CR should be cleared during ep_destroy Message-ID: The EP reference in the CR should be set to null during the EP free call to insure no further reference back to a mem freed EP. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_qp.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index 7c3c280..f943ff8 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -221,6 +221,7 @@ DAT_RETURN dapls_ib_qp_free(IN DAPL_IA * ia_ptr, IN DAPL_EP * ep_ptr) " qp_free CR: ep %p cr %p\n", ep_ptr, cr); dapli_socket_disconnect(cr); dapl_os_lock(&cr->lock); + cr->ep = NULL; cr->state = SCM_DESTROY; dapl_os_unlock(&cr->lock); } -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 26 16:49:15 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 26 Jun 2009 16:49:15 -0700 Subject: [ofa-general] [PATCH] scm: double the default fd_set_size, linux provider only Message-ID: Signed-off-by: Arlin Davis --- dapl/openib_scm/linux/openib_osd.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/dapl/openib_scm/linux/openib_osd.h b/dapl/openib_scm/linux/openib_osd.h index 235a82e..a015188 100644 --- a/dapl/openib_scm/linux/openib_osd.h +++ b/dapl/openib_scm/linux/openib_osd.h @@ -14,7 +14,7 @@ #define DAPL_SOCKET int #define DAPL_INVALID_SOCKET -1 -#define DAPL_FD_SETSIZE 8192 +#define DAPL_FD_SETSIZE 16384 #define closesocket close -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 26 16:50:00 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 26 Jun 2009 16:50:00 -0700 Subject: [ofa-general] [PATCH] scm: fixes and optimizations for connection scaling Message-ID: Prioritize accepts on listen ports via FD_READ process the accepts ahead of other work to avoid socket half_connection (SYN_RECV) stalls. Fix dapl_poll to return DAPL_FD_ERROR on all event error types. Add new state for socket released, but CR not yet destroyed. This enables scm to release the socket resources immediately after exchanging all QP information. Also, add state to str call. Only add the CR reference to the EP if it is RC type. UD has multiple CR's per EP so when a UD EP disconnect_clean was called, from a timeout, it destroyed the wrong CR. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_cm.c | 176 +++++++++++++++++++++++----------------- dapl/openib_scm/dapl_ib_util.h | 4 +- 2 files changed, 103 insertions(+), 77 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index 27defb6..90d6d27 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -213,12 +213,14 @@ static enum DAPL_FD_EVENTS dapl_poll(DAPL_SOCKET s, enum DAPL_FD_EVENTS event) fds.events = event; fds.revents = 0; ret = poll(&fds, 1, 0); - dapl_dbg_log(DAPL_DBG_TYPE_CM, " dapl_poll: ret=%d, events=0x%x\n", - ret, fds.revents); - if (ret <= 0) - return ret; - - return fds.revents; + dapl_log(DAPL_DBG_TYPE_CM, " dapl_poll: fd=%d ret=%d, evnts=0x%x\n", + s, ret, fds.revents); + if (ret == 0) + return 0; + else if (fds.revents & (POLLERR | POLLHUP | POLLNVAL)) + return DAPL_FD_ERROR; + else + return fds.revents; } static int dapl_select(struct dapl_fd_set *set) @@ -271,8 +273,10 @@ static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) dapl_os_lock(&cm_ptr->lock); cm_ptr->state = SCM_DESTROY; - if ((cm_ptr->ep) && (cm_ptr->ep->cm_handle == cm_ptr)) + if ((cm_ptr->ep) && (cm_ptr->ep->cm_handle == cm_ptr)) { cm_ptr->ep->cm_handle = IB_INVALID_HANDLE; + cm_ptr->ep = NULL; + } /* close socket if still active */ if (cm_ptr->socket != DAPL_INVALID_SOCKET) { @@ -369,11 +373,14 @@ static void dapli_socket_connected(dp_ib_cm_handle_t cm_ptr, int err) if (err) { dapl_log(DAPL_DBG_TYPE_ERR, - " CONN_PENDING: socket ERR %s -> %s\n", - strerror(err), inet_ntoa(((struct sockaddr_in *) - ep_ptr->param. - remote_ia_address_ptr)-> - sin_addr)); + " CONN_PENDING: %s ERR %s -> %s %d\n", + err == -1 ? "POLL" : "SOCKOPT", + err == -1 ? strerror(errno) : strerror(err), + inet_ntoa(((struct sockaddr_in *) + ep_ptr->param. + remote_ia_address_ptr)->sin_addr), + ntohs(((struct sockaddr_in *) + &cm_ptr->dst.ia_address)->sin_port)); goto bail; } dapl_dbg_log(DAPL_DBG_TYPE_EP, @@ -486,6 +493,9 @@ dapli_socket_connect(DAPL_EP * ep_ptr, cm_ptr->hca = ia_ptr->hca_ptr; cm_ptr->ep = ep_ptr; cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; + ((struct sockaddr_in *) + &cm_ptr->dst.ia_address)->sin_port = ntohs(r_qual); + if (p_size) { cm_ptr->dst.p_size = htonl(p_size); dapl_os_memcpy(cm_ptr->p_data, p_data, p_size); @@ -642,7 +652,6 @@ static void dapli_socket_connect_rtu(dp_ib_cm_handle_t cm_ptr) goto bail; } /* init cm_handle and post the event with private data */ - ep_ptr->cm_handle = cm_ptr; cm_ptr->state = SCM_CONNECTED; event = IB_CME_CONNECTED; dapl_dbg_log(DAPL_DBG_TYPE_EP, " ACTIVE: connected!\n"); @@ -677,11 +686,15 @@ ud_bail: /* done with socket, don't destroy cm_ptr, need pdata */ closesocket(cm_ptr->socket); cm_ptr->socket = DAPL_INVALID_SOCKET; + cm_ptr->state = SCM_RELEASED; } else #endif + { + ep_ptr->cm_handle = cm_ptr; /* only RC, multi CR's on UD */ dapl_evd_connection_callback(cm_ptr, IB_CME_CONNECTED, cm_ptr->p_data, ep_ptr); + } return; bail: @@ -769,34 +782,36 @@ static void dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) int len; dapl_dbg_log(DAPL_DBG_TYPE_EP, " socket_accept\n"); + + /* + * Accept all CR's on this port to avoid half-connection (SYN_RCV) + * stalls with many to one connection storms + */ + do { + /* Allocate accept CM and initialize */ + if ((acm_ptr = dapli_cm_create()) == NULL) + return; + + acm_ptr->sp = cm_ptr->sp; + acm_ptr->hca = cm_ptr->hca; + + len = sizeof(acm_ptr->dst.ia_address); + acm_ptr->socket = accept(cm_ptr->socket, + (struct sockaddr *) + &acm_ptr->dst.ia_address, + (socklen_t *) & len); + if (acm_ptr->socket == DAPL_INVALID_SOCKET) { + dapl_log(DAPL_DBG_TYPE_ERR, + " accept: ERR %s on FD %d l_cr %p\n", + strerror(errno), cm_ptr->socket, cm_ptr); + dapli_cm_destroy(acm_ptr); + return; + } - /* Allocate accept CM and initialize */ - if ((acm_ptr = dapli_cm_create()) == NULL) - return; - - acm_ptr->sp = cm_ptr->sp; - acm_ptr->hca = cm_ptr->hca; - - len = sizeof(acm_ptr->dst.ia_address); - acm_ptr->socket = accept(cm_ptr->socket, - (struct sockaddr *)&acm_ptr->dst.ia_address, - (socklen_t *) & len); - if (acm_ptr->socket == DAPL_INVALID_SOCKET) { - dapl_log(DAPL_DBG_TYPE_ERR, - " accept: ERR %s on FD %d l_cr %p\n", - strerror(errno), cm_ptr->socket, cm_ptr); - goto bail; - } - - dapl_dbg_log(DAPL_DBG_TYPE_EP, - " socket accepted, queue new cm %p\n", acm_ptr); - - acm_ptr->state = SCM_ACCEPTING; - dapli_cm_queue(acm_ptr); - return; - bail: - /* close socket, free cm structure, active will see socket close as reject */ - dapli_cm_destroy(acm_ptr); + acm_ptr->state = SCM_ACCEPTING; + dapli_cm_queue(acm_ptr); + + } while (dapl_poll(cm_ptr->socket, DAPL_FD_READ) == DAPL_FD_READ); } /* @@ -964,6 +979,9 @@ dapli_socket_accept_usr(DAPL_EP * ep_ptr, local.lid = ia_ptr->hca_ptr->ib_trans.lid; local.gid = ia_ptr->hca_ptr->ib_trans.gid; local.ia_address = ia_ptr->hca_ptr->hca_address; + ((struct sockaddr_in *)&local.ia_address)->sin_port = + ntohs(cm_ptr->sp->conn_qual); + local.p_size = htonl(p_size); iov[0].iov_base = (void *)&local; iov[0].iov_len = sizeof(ib_qp_cm_t); @@ -1059,6 +1077,7 @@ void dapli_socket_accept_rtu(dp_ib_cm_handle_t cm_ptr) /* done with socket, don't destroy cm_ptr, need pdata */ closesocket(cm_ptr->socket); cm_ptr->socket = DAPL_INVALID_SOCKET; + cm_ptr->state = SCM_RELEASED; } else #endif dapls_cr_callback(cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp); @@ -1146,7 +1165,8 @@ dapls_ib_disconnect(IN DAPL_EP * ep_ptr, IN DAT_CLOSE_FLAGS close_flags) * * Clean up outstanding connection data. This routine is invoked * after the final disconnect callback has occurred. Only on the - * ACTIVE side of a connection. + * ACTIVE side of a connection. It is also called if dat_ep_connect + * times out using the consumer supplied timeout value. * * Input: * ep_ptr DAPL_EP @@ -1164,6 +1184,14 @@ dapls_ib_disconnect_clean(IN DAPL_EP * ep_ptr, IN DAT_BOOLEAN active, IN const ib_cm_events_t ib_cm_event) { + /* NOTE: SCM will only initialize cm_handle with RC type + * + * For UD there can many in-flight CR's so you + * cannot cleanup timed out CR's with EP reference + * alone since they share the same EP. The common + * code that handles connection timeout logic needs + * updated for UD support. + */ if (ep_ptr->cm_handle) dapli_cm_destroy(ep_ptr->cm_handle); @@ -1633,7 +1661,7 @@ void cr_thread(void *arg) " poll ret=0x%x cr->state=%d socket=%d\n", ret, cr->state, cr->socket); - /* data on listen, qp exchange, and on disconnect request */ + /* data on listen, qp exchange, and on disc req */ if (ret == DAPL_FD_READ) { if (cr->socket != DAPL_INVALID_SOCKET) { switch (cr->state) { @@ -1656,40 +1684,29 @@ void cr_thread(void *arg) break; } } - /* connect socket is writable, check status */ - } else if (ret == DAPL_FD_WRITE || ret == DAPL_FD_ERROR) { - if (cr->state == SCM_CONN_PENDING) { - opt = 0; - opt_len = sizeof(opt); - ret = getsockopt(cr->socket, SOL_SOCKET, - SO_ERROR, (char *)&opt, - &opt_len); - if (!ret) - dapli_socket_connected(cr, opt); - else - dapli_socket_connected(cr, - errno); - } else { - dapl_log(DAPL_DBG_TYPE_CM, - " CM poll ERR, wrong state(%d) -> %s SKIP\n", - cr->state, - inet_ntoa(((struct sockaddr_in - *)&cr->dst. - ia_address)-> - sin_addr)); - } - } else if (ret != 0) { - dapl_log(DAPL_DBG_TYPE_CM, - " CM poll warning %s, ret=%d st=%d -> %s\n", - strerror(errno), ret, cr->state, - inet_ntoa(((struct sockaddr_in *) - &cr->dst.ia_address)-> - sin_addr)); - - /* POLLUP, NVAL, or poll error. - DISC */ + /* connect socket is writable, check status */ + } else if (ret == DAPL_FD_WRITE || + (cr->state == SCM_CONN_PENDING && + ret == DAPL_FD_ERROR)) { + opt = 0; + opt_len = sizeof(opt); + ret = getsockopt(cr->socket, SOL_SOCKET, + SO_ERROR, (char *)&opt, + &opt_len); + if (!ret) + dapli_socket_connected(cr, opt); + else + dapli_socket_connected(cr, errno); + + /* POLLUP, ERR, NVAL, or poll error - DISC */ + } else if (ret < 0 || ret == DAPL_FD_ERROR) { + dapl_log(DAPL_DBG_TYPE_WARN, + " poll=%d cr->st=%s sk=%d ep %p, %d\n", + ret, dapl_cm_state_str(cr->state), + cr->socket, cr->ep, + cr->ep ? cr->ep->param.ep_state:0); dapli_socket_disconnect(cr); } - dapl_os_lock(&hca_ptr->ib_trans.lock); } @@ -1748,12 +1765,19 @@ void dapls_print_cm_list(IN DAPL_IA *ia_ptr) &ia_ptr->hca_ptr->ib_trans.list, (DAPL_LLIST_ENTRY*)&cr->entry); - printf( " CONN[%d]: sp %p ep %p sock %d %s %s -> %s\n", + printf( " CONN[%d]: sp %p ep %p sock %d %s %s %s %s %d\n", i, cr->sp, cr->ep, cr->socket, cr->dst.qp_type == IBV_QPT_RC ? "RC" : "UD", dapl_cm_state_str(cr->state), + cr->sp ? "<-" : "->", + cr->state == SCM_LISTEN ? + inet_ntoa(((struct sockaddr_in *) + &ia_ptr->hca_ptr->hca_address)->sin_addr) : inet_ntoa(((struct sockaddr_in *) - &cr->dst.ia_address)->sin_addr)); + &cr->dst.ia_address)->sin_addr), + cr->sp ? (int)cr->sp->conn_qual : + ntohs(((struct sockaddr_in *) + &cr->dst.ia_address)->sin_port)); i++; } printf("\n"); diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index 294ef3d..a668af7 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -99,6 +99,7 @@ typedef enum scm_state SCM_ACCEPTED, SCM_REJECTED, SCM_CONNECTED, + SCM_RELEASED, SCM_DISCONNECTED, SCM_DESTROY } SCM_STATE; @@ -382,10 +383,11 @@ STATIC _INLINE_ char * dapl_cm_state_str(IN int st) "SCM_ACCEPTED", "SCM_REJECTED", "SCM_CONNECTED", + "SCM_RELEASED", "SCM_DISCONNECTED", "SCM_DESTROY" }; - return ((st < 0 || st > 10) ? "Invalid CM state?" : cm_state[st]); + return ((st < 0 || st > 11) ? "Invalid CM state?" : cm_state[st]); } /* -- 1.5.2.5 From friedman at ucla.edu Fri Jun 26 17:58:24 2009 From: friedman at ucla.edu (Scott A. Friedman) Date: Fri, 26 Jun 2009 17:58:24 -0700 Subject: [ofa-general] verb level interoperability between vendor's hcas Message-ID: <4A456EB0.5050102@ucla.edu> We have a largish cluster that has nodes with HCAs from multiple vendors - specifically, Mellanox and QLogic. The cluster mostly consists of Mellanox cards and a much smaller number of QLogic cards (all DDR). We have noticed that there are sometimes problems (crashing) running jobs between nodes that mix vendors - when using verbs or using libraries that use verbs (e.g. MPI). The question I have is actually more general for the recipients of this list - what kinds and how much interop testing is there? Is this information available someplace and I am just missing it? Is a mixed HCA environment cluster not ready for prime time - yet? Anyone with experiences or comments they would like to share? Thanks, Scott From vlad at lists.openfabrics.org Sat Jun 27 02:24:35 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 27 Jun 2009 02:24:35 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090627-0200 daily build status Message-ID: <20090627092435.DC5CF102021B@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090627-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From sean.hefty at intel.com Sat Jun 27 23:33:18 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Sat, 27 Jun 2009 23:33:18 -0700 Subject: [ofa-general] verb level interoperability between vendor's hcas In-Reply-To: <4A456EB0.5050102@ucla.edu> References: <4A456EB0.5050102@ucla.edu> Message-ID: >Is a mixed HCA environment cluster not ready for prime time - yet? Are the crashes in the kernel or userspace? Is there a specific HCA on the nodes that crash? Interop testing is done, but I do not know the details of the configurations and tests that are run. From jackm at dev.mellanox.co.il Sun Jun 28 01:17:34 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 28 Jun 2009 11:17:34 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition Message-ID: <200906281117.34440.jackm@dev.mellanox.co.il> We have seen the following kernel Oops on IPoIB: ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22 Unable to handle kernel paging request for data at address 0x00000054 adFaulting instruction address: 0xe60b43c4 Oops: Kernel access of bad area, sig: 11 [#1] ... NIP [e60b43c4] ib_sa_free_multicast+0x14/0x148 [ib_sa] LR [e601fa24] ipoib_mcast_leave+0x140/0x144 [ib_ipoib] Call Trace: [de6a1ce0] [1000d3c0] 0x1000d3c0 (unreliable) [de6a1d00] [e601fa24] ipoib_mcast_leave+0x140/0x144 [ib_ipoib] [de6a1d60] [e6020904] ipoib_mcast_dev_flush+0x164/0x1a4 [ib_ipoib] [de6a1db0] [e601e2a4] ipoib_ib_dev_down+0x78/0x130 [ib_ipoib] [de6a1dd0] [e601c030] ipoib_stop+0xec/0x19c [ib_ipoib] [de6a1df0] [c01b0e14] dev_close+0x88/0xd8 [de6a1e00] [c01b0c84] dev_change_flags+0x154/0x1a8 [de6a1e20] [c01f6144] devinet_ioctl+0x62c/0x81c [de6a1e90] [c01f69fc] inet_ioctl+0xcc/0xf8 [de6a1ea0] [c01a12f0] sock_ioctl+0x60/0x2ec [de6a1ec0] [c008b1d4] vfs_ioctl+0x40/0xc0 [de6a1ee0] [c008b580] do_vfs_ioctl+0x32c/0x498 [de6a1f10] [c008b72c] sys_ioctl+0x40/0x74 [de6a1f40] [c000e780] ret_from_syscall+0x0/0x3c Scenario was that someone performed "ifconfig ib0 down" while IPoIB was coming up. Problem: a race-condition hole: - procedure ipoib_mcast_join(): set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, &rec, comp_mask, GFP_KERNEL, ipoib_mcast_join_complete, mcast); if (IS_ERR(mcast->mc)) { clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); - procedure ipoib_mcast_leave(): if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) ib_sa_free_multicast(mcast->mc); Problem: IPOIB_MCAST_FLAG_BUSY is set in join() before the mcast->mc pointer is valid. If leave() is called after the busy-flag is set, but before ib_sa_join_multicast returns, mcast->mc will not be valid (and ib_sa_free_multicast() assumes that mcast->mc is valid). In fact, the "hole" is as follows: set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); ******* hole starts here *********** mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, &rec, comp_mask, GFP_KERNEL, ipoib_mcast_join_complete, mcast); if (IS_ERR(mcast->mc)) { clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); ******* hole ends here *********** I don't yet see a clean fix for this. We need the busy flag set where it is so that we do not do multiple joins, and do not do ipoib_send(). Using another flag, say IPOIB_MCAST_FLAG_VALID, and setting it after mcast->mc is successfully assigned (and testing for that in leave() ) is a problem, because it may lead to memory leaks. We basically need atomicity here, and spinlocks are not an option. Any ideas? -Jack From vlad at lists.openfabrics.org Sun Jun 28 02:24:40 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 28 Jun 2009 02:24:40 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090628-0200 daily build status Message-ID: <20090628092440.BC8751020326@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090628-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From jackm at dev.mellanox.co.il Sun Jun 28 02:21:51 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 28 Jun 2009 12:21:51 +0300 Subject: [ofa-general] Write combining on PPC64 Message-ID: <200906281221.52028.jackm@dev.mellanox.co.il> Hi Shirley, I was reviewing write-combining for the PPC on kernel 2.6.30, and noticed the following in file arch/powerpc/include/asm/pgtable.h: #define _PAGE_CACHE_CTL (_PAGE_COHERENT | _PAGE_GUARDED | _PAGE_NO_CACHE | \ _PAGE_WRITETHRU) ... #define pgprot_noncached_wc(prot) (__pgprot((pgprot_val(prot) & ~_PAGE_CACHE_CTL) | \ _PAGE_NO_CACHE)) It seems to me that this gives the same result as the fix I put in OFED 1.5 for the PPC: (kernel_patches/fixes/mlx4_0010_add_wc.patch): +pgprot_t pgprot_wc(pgprot_t _prot) +{ + return __pgprot((pgprot_val(_prot) | _PAGE_NO_CACHE) & + ~(pgprot_t)_PAGE_GUARDED); +} ( That is, PAGE_NO_CACHE set, and _PAGE_GUARDED cleared). Differences are that _PAGE_COHERENT and _PAGE_WRITETHRU are also cleared in pgprot_noncached_wc() and the patch I put in is only for PPC64, not all PPCs. Questions: 1. Can I use the pgprot_noncached_wc() macro to implement write-combining on all PPCs? (it is defined thus in arch/powerpc/include/asm/pgtable.h for all PPC platforms) 2. Is there any reason that there is no define: #define pgprot_writecombine pgprot_noncached_wc in file arch/powerpc/include/asm/pgtable.h (so that the general pgprot_writecombine() call can be used for PPCs as well). -Jack From slavas at Voltaire.COM Sun Jun 28 03:36:34 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Sun, 28 Jun 2009 13:36:34 +0300 Subject: [ofa-general] [PATCH 1/4] multicast multiplexing - definitions for new data types, and infrastructure functions In-Reply-To: <4A3A387A.2020509@Voltaire.COM> References: <4A3A387A.2020509@Voltaire.COM> Message-ID: <4A4747B2.2030907@Voltaire.COM> [PATCH 1/4] Patch implements multicast multiplexing as proposed in the thread entitled "IPv6 and IPoIB scalability issue". This first patch contains definitions for new data types and infrastructure functions. Signed-off-by: Slava Strebkov --- opensm/include/opensm/osm_multicast.h | 94 +++++++++++++++++++++++++++++++-- opensm/opensm/osm_multicast.c | 82 ++++++++++++++++++++++++++++- 2 files changed, 170 insertions(+), 6 deletions(-) diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index a871306..02b63bd 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * @@ -45,6 +45,7 @@ #include #include +#include #include #include #include @@ -93,7 +94,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct osm_mcast_mgr_ctxt { + typedef struct osm_mcast_mgr_ctxt { cl_list_item_t list_item; ib_net16_t mlid; } osm_mcast_mgr_ctxt_t; @@ -107,6 +108,89 @@ typedef struct osm_mcast_mgr_ctxt { * SEE ALSO *********/ +/****s* OpenSM: Multicast Group Holder/osm_mgrp_holder_t +* NAME +* osm_mgrp_holder_t +* +* DESCRIPTION +* Holder for mgroups. +* +* The osm_mgrp_t object should be treated as opaque and should +* be manipulated only through the provided functions. +* +* SYNOPSIS +*/ + +typedef struct osm_mgrp_holder { + cl_fmap_t mgrp_map; + cl_qmap_t mgrp_port_map; + ib_gid_t common_mgid; + osm_mtree_node_t *p_root; + boolean_t to_be_deleted; + uint32_t last_tree_id; + uint32_t last_change_id; + ib_net16_t mlid; +} osm_mgrp_holder_t; + +/* +* FIELDS +* mgrp_map +* Map for mgroups. Must be first element!! +* +* mgrp_port_map +* Map of all ports joined same mlid +* +* common_mgid +* mgid of mgroup, ANDed with bitmask. +* mgid of each mgroup in mgrp_map, ANDed with bitmask, +* see osm_mgrp_holder_prepare_common_mgid +* +* p_root +* Pointer to the root "tree node" in the single spanning tree +* for this multicast group holder. The nodes of the tree represent +* switches. Member ports are not represented in the tree. +* +* to_be_deleted +* Since holders are deleted when there are no mgroups in. +* +* last_change_id +* a counter for the number of changes applied to the group in this holder. +* This counter shuold be incremented on any modification +* to the group: joining or leaving of ports. +* +* last_tree_id +* the last change id used for building the current tree. +* +* mlid +* mlid of current group holder +*/ + /****s* OpenSM: Multicast group Port /osm_mgrp_port _t +* NAME +* osm_mgrp_port _t +* +* DESCRIPTION +* Holder for pointers to mgroups and port guid. +* +* +* SYNOPSIS +*/ +typedef struct _osm_mgrp_port { + cl_map_item_t guid_item; + cl_qlist_t mgroups; + ib_net64_t port_guid; +} osm_mgrp_port_t; +/* +* FIELDS +* guid_item +* Map for ports. Must be first element +* +* mgroups +* Map for mgroups opened by this port. +* +* portguid +* guid of port representing current structure +*/ + /****s* OpenSM: Multicast Group/osm_mgrp_t * NAME * osm_mgrp_t @@ -355,7 +439,7 @@ static inline ib_net16_t osm_mgrp_get_mlid(IN const osm_mgrp_t * const p_mgrp) * * SYNOPSIS */ -osm_mcm_port_t *osm_mgrp_add_port(osm_subn_t *subn, osm_log_t *log, +osm_mcm_port_t *osm_mgrp_add_port(osm_subn_t * subn, osm_log_t * log, IN osm_mgrp_t * const p_mgrp, IN const ib_gid_t * const p_port_gid, IN const uint8_t join_state, @@ -452,8 +536,8 @@ osm_mgrp_delete_port(IN osm_subn_t * const p_subn, * SEE ALSO *********/ -int osm_mgrp_remove_port(osm_subn_t *subn, osm_log_t *log, osm_mgrp_t *mgrp, - osm_mcm_port_t *mcm, uint8_t join_state); +int osm_mgrp_remove_port(osm_subn_t * subn, osm_log_t * log, osm_mgrp_t * mgrp, + osm_mcm_port_t * mcm, uint8_t join_state); /****f* OpenSM: Multicast Group/osm_mgrp_apply_func * NAME diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index d2733c4..ae0a818 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * @@ -48,6 +48,7 @@ #include #include #include +#include /********************************************************************** **********************************************************************/ @@ -298,3 +299,82 @@ void osm_mgrp_apply_func(const osm_mgrp_t * p_mgrp, osm_mgrp_func_t p_func, if (p_mtn) mgrp_apply_func_sub(p_mgrp, p_mtn, p_func, context); } + +/********************************************************************** + **********************************************************************/ +#define PREFIX_MASK_IP CL_HTON64(0xff10ffff0000ffffULL) +#define PREFIX_SIGNATURE_IPV4 CL_HTON64(0xff10401b00000000ULL) +#define INTERFACE_ID_IPV4 CL_HTON64(0x0000000fffffffffULL) +#define PREFIX_SIGNATURE_IPV6 CL_HTON64(0xff10601b00000000ULL) +#define INTERFACE_ID_ALL_NODES CL_HTON64(0x00000000ffffffffULL) +#define INTERFACE_ID_ALL_HOSTS CL_HTON64(0x0000000000000001ULL) +#define INTERFACE_ID_ALL_GATEWAYS CL_HTON64(0x000000000000004dULL) +#define INTERFACE_ID_ALL_ROUTERS CL_HTON64(0x0000000000000002ULL) +#define PREFIX_PKEY_MASK_OFF CL_HTON64(0xffffffff0000ffffULL) +#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) +#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) +#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) +#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) + +/********************************************************************** + **********************************************************************/ + /* mask used for multiplexing mgids to one mlid. Here default value (0xff) means that 8 lsb bits + of mgid will be masked off . + */ +static uint64_t MGID_MUX_MLID_MASK = CL_HTON64(0x00000000000000ffULL); +void osm_mgrp_holder_prepare_common_mgid(IN const ib_gid_t * const p_mgid, + OUT ib_gid_t * p_common_mgid) +{ + memcpy(p_common_mgid, p_mgid, sizeof(ib_gid_t)); + /* Don't mux non IPoIB mgids. When mux mask is zero, no multiplexing occures */ + if ((((p_common_mgid->unicast.prefix & PREFIX_MASK_IP) != + PREFIX_SIGNATURE_IPV4) + && ((p_common_mgid->unicast.prefix & PREFIX_MASK_IP) != + PREFIX_SIGNATURE_IPV6)) || (!MGID_MUX_MLID_MASK)) + return; + + if (((p_common_mgid->unicast.prefix & PREFIX_MASK_IP) == + PREFIX_SIGNATURE_IPV4) + && ((p_common_mgid->unicast.interface_id == INTERFACE_ID_ALL_NODES) + || (p_common_mgid->unicast.interface_id == + INTERFACE_ID_ALL_HOSTS) + || (p_common_mgid->unicast.interface_id == + INTERFACE_ID_ALL_GATEWAYS) + || (p_common_mgid->unicast.interface_id == + INTERFACE_ID_ALL_ROUTERS))) { + /* zero pkey bits - special groups will have same mlid for all PKEYs */ + p_common_mgid->unicast.prefix &= PREFIX_PKEY_MASK_OFF; + } else if ((p_common_mgid->unicast.prefix & PREFIX_MASK_IP) == + PREFIX_SIGNATURE_IPV6 + && (p_common_mgid->unicast.interface_id & INT_ID_MASK) == + INT_ID_SIGNATURE) { + /* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ + /* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ + /* Where Z is the scope, XXXX is the P_Key, and + * YYYYYY is the last 24 bits of the port guid */ + p_common_mgid->unicast.prefix &= PREFIX_MASK_IP; + p_common_mgid->unicast.interface_id &= INT_ID_MASK; + } else { + /* zero mask bits */ + p_common_mgid->unicast.interface_id &= (~MGID_MUX_MLID_MASK); + } +} + +static int64_t +__mgid_cmp(IN const void *const p_gid1, IN const void *const p_gid2) +{ + return memcmp(p_gid1, p_gid2, sizeof(ib_gid_t)); +} + +static osm_mgrp_port_t *osm_mgrp_port_new(ib_net64_t port_guid) +{ + osm_mgrp_port_t *p_mgrp_port = + (osm_mgrp_port_t *) malloc(sizeof(osm_mgrp_port_t)); + if (!p_mgrp_port) { + return NULL; + } + memset(p_mgrp_port, 0, sizeof(*p_mgrp_port)); + p_mgrp_port->port_guid = port_guid; + cl_qlist_init(&p_mgrp_port->mgroups); + return p_mgrp_port; +} -- 1.5.5 From slavas at Voltaire.COM Sun Jun 28 03:39:28 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Sun, 28 Jun 2009 13:39:28 +0300 Subject: [ofa-general] Re: [PATCH] opensm: multicast multiplexing many mgid to mlid In-Reply-To: <4A3A387A.2020509@Voltaire.COM> References: <4A3A387A.2020509@Voltaire.COM> Message-ID: <4A474860.7000702@Voltaire.COM> Subject: [PATCH 3/4] Patch implements multicast multiplexing as proposed in the thread entitled "IPv6 and IPoIB scalability issue". This part (3) introduces usage of new infrastructure for mgid->mlid compression Signed-off-by: Slava Strebkov --- opensm/include/opensm/osm_multicast.h | 5 +- opensm/include/opensm/osm_sm.h | 23 ++- opensm/opensm/osm_drop_mgr.c | 14 +- opensm/opensm/osm_mcast_mgr.c | 2 +- opensm/opensm/osm_multicast.c | 2 +- opensm/opensm/osm_qos_policy.c | 360 ++++++++++++++++++-------------- opensm/opensm/osm_sa.c | 65 ++++-- opensm/opensm/osm_sa_mcmember_record.c | 161 ++++++++------- opensm/opensm/osm_sa_path_record.c | 32 +++- opensm/opensm/osm_sm.c | 92 +++++++-- opensm/opensm/osm_subnet.c | 12 +- 11 files changed, 474 insertions(+), 294 deletions(-) diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index 9f0cd96..567a989 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -331,12 +331,15 @@ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid); * * SYNOPSIS */ -void osm_mgrp_delete(IN osm_mgrp_t * const p_mgrp); +void osm_mgrp_delete_group(IN osm_subn_t * p_subn, IN osm_mgrp_t * const p_mgrp); /* * PARAMETERS * p_mgrp * [in] Pointer to an osm_mgrp_t object. * +* p_subn +* [in] Pointer to an osm_subn_t object. +* * RETURN VALUES * None. * diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h index cc8321d..26530f3 100644 --- a/opensm/include/opensm/osm_sm.h +++ b/opensm/include/opensm/osm_sm.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * @@ -61,6 +61,7 @@ #include #include #include +#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -101,7 +102,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct osm_sm { + typedef struct osm_sm { osm_thread_state_t thread_state; unsigned signal_mask; cl_spinlock_t signal_lock; @@ -539,7 +540,7 @@ osm_resp_send(IN osm_sm_t * sm, ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid); + IN const ib_net64_t port_guid, IN const ib_gid_t * p_mgid); /* * PARAMETERS * p_sm @@ -551,6 +552,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, * port_guid * [in] Port GUID to add to the group. * +* p_mgid +* [in] MGID to add to the group holder. * RETURN VALUES * None * @@ -572,7 +575,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, */ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, - IN const ib_net16_t mlid, IN const ib_net64_t port_guid); + IN osm_mgrp_t * p_mgrp, IN ib_net64_t port_guid); /* * PARAMETERS * p_sm @@ -689,7 +692,7 @@ osm_sm_is_greater_than(IN const uint8_t l_priority, * * SYNOPSIS */ -ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t *sm, +ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t * sm, IN osm_sm_signal_t signal); /* * PARAMETERS @@ -718,7 +721,7 @@ ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t *sm, * * SYNOPSIS */ -void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t *sm); +void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t * sm); /* * PARAMETERS * sm @@ -743,7 +746,7 @@ void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t *sm); * * SYNOPSIS */ -ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t *sm, +ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t * sm, IN osm_sm_signal_t signal); /* * PARAMETERS @@ -762,7 +765,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t *sm, * State Manager *********/ -void osm_report_sm_state(osm_sm_t *sm); +void osm_report_sm_state(osm_sm_t * sm); /****f* OpenSM: SM State Manager/osm_send_trap144 * NAME @@ -773,7 +776,7 @@ void osm_report_sm_state(osm_sm_t *sm); * * SYNOPSIS */ -int osm_send_trap144(osm_sm_t *sm, ib_net16_t local); +int osm_send_trap144(osm_sm_t * sm, ib_net16_t local); /* * PARAMETERS * sm @@ -787,7 +790,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local); * *********/ -void osm_set_sm_priority(osm_sm_t *sm, uint8_t priority); +void osm_set_sm_priority(osm_sm_t * sm, uint8_t priority); END_C_DECLS #endif /* _OSM_SM_H_ */ diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index c9a4f33..d92fdfc 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -158,7 +158,6 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) osm_port_t *p_port_check; cl_qmap_t *p_sm_guid_tbl; osm_mcm_info_t *p_mcm; - osm_mgrp_t *p_mgrp; cl_ptr_vector_t *p_port_lid_tbl; uint16_t min_lid_ho; uint16_t max_lid_ho; @@ -168,6 +167,7 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) ib_gid_t port_gid; ib_mad_notice_attr_t notice; ib_api_status_t status; + osm_mgrp_holder_t *p_osm_mgrp_holder; OSM_LOG_ENTER(sm->p_log); @@ -212,10 +212,12 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { - p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, p_mcm->mlid); - if (p_mgrp) { - osm_mgrp_delete_port(sm->p_subn, sm->p_log, - p_mgrp, p_port->guid); + p_osm_mgrp_holder = + osm_get_mgrp_holder_by_mlid(sm->p_subn, p_mcm->mlid); + if (p_osm_mgrp_holder) { + osm_mgrp_holder_remove_port(sm->p_subn, sm->p_log, + p_osm_mgrp_holder, + p_port->guid); osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); } p_mcm = diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index de0a8a5..e16c2e7 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index d2a19ea..e2e588c 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -41,7 +41,7 @@ #if HAVE_CONFIG_H # include #endif /* HAVE_CONFIG_H */ - +#include #include #include #include diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index 094fef2..07b05f5 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -59,11 +59,10 @@ extern osm_qos_level_t __default_simple_qos_level; /*************************************************** ***************************************************/ -static void -__build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) +static void __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) { - osm_node_t * p_node; - cl_qmap_t * p_node_guid_tbl = &p_qos_policy->p_subn->node_guid_tbl; + osm_node_t *p_node; + cl_qmap_t *p_node_guid_tbl = &p_qos_policy->p_subn->node_guid_tbl; p_qos_policy->p_node_hash = st_init_strtable(); CL_ASSERT(p_qos_policy->p_node_hash); @@ -75,10 +74,10 @@ __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { if (!st_lookup(p_qos_policy->p_node_hash, - (st_data_t)p_node->print_desc, NULL)) + (st_data_t) p_node->print_desc, NULL)) st_insert(p_qos_policy->p_node_hash, - (st_data_t)p_node->print_desc, - (st_data_t)p_node); + (st_data_t) p_node->print_desc, + (st_data_t) p_node); } } @@ -87,7 +86,7 @@ __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) static boolean_t __is_num_in_range_arr(uint64_t ** range_arr, - unsigned range_arr_len, uint64_t num) + unsigned range_arr_len, uint64_t num) { unsigned ind_1 = 0; unsigned ind_2 = range_arr_len - 1; @@ -97,22 +96,23 @@ __is_num_in_range_arr(uint64_t ** range_arr, return FALSE; while (ind_1 <= ind_2) { - if (num < range_arr[ind_1][0] || num > range_arr[ind_2][1]) - return FALSE; - else if (num <= range_arr[ind_1][1] || num >= range_arr[ind_2][0]) - return TRUE; - - ind_mid = ind_1 + (ind_2 - ind_1 + 1)/2; - - if (num < range_arr[ind_mid][0]) - ind_2 = ind_mid; - else if (num > range_arr[ind_mid][1]) - ind_1 = ind_mid; - else - return TRUE; - - ind_1++; - ind_2--; + if (num < range_arr[ind_1][0] || num > range_arr[ind_2][1]) + return FALSE; + else if (num <= range_arr[ind_1][1] + || num >= range_arr[ind_2][0]) + return TRUE; + + ind_mid = ind_1 + (ind_2 - ind_1 + 1) / 2; + + if (num < range_arr[ind_mid][0]) + ind_2 = ind_mid; + else if (num > range_arr[ind_mid][1]) + ind_1 = ind_mid; + else + return TRUE; + + ind_1++; + ind_2--; } return FALSE; @@ -130,10 +130,9 @@ static void __free_single_element(void *p_element, void *context) /*************************************************** ***************************************************/ -osm_qos_port_t *osm_qos_policy_port_create(osm_physp_t *p_physp) +osm_qos_port_t *osm_qos_policy_port_create(osm_physp_t * p_physp) { - osm_qos_port_t *p = - (osm_qos_port_t *) malloc(sizeof(osm_qos_port_t)); + osm_qos_port_t *p = (osm_qos_port_t *) malloc(sizeof(osm_qos_port_t)); if (!p) return NULL; memset(p, 0, sizeof(osm_qos_port_t)); @@ -163,8 +162,8 @@ osm_qos_port_group_t *osm_qos_policy_port_group_create() void osm_qos_policy_port_group_destroy(osm_qos_port_group_t * p) { - osm_qos_port_t * p_port; - osm_qos_port_t * p_old_port; + osm_qos_port_t *p_port; + osm_qos_port_t *p_old_port; if (!p) return; @@ -175,8 +174,7 @@ void osm_qos_policy_port_group_destroy(osm_qos_port_group_t * p) free(p->use); p_port = (osm_qos_port_t *) cl_qmap_head(&p->port_map); - while (p_port != (osm_qos_port_t *) cl_qmap_end(&p->port_map)) - { + while (p_port != (osm_qos_port_t *) cl_qmap_end(&p->port_map)) { p_old_port = p_port; p_port = (osm_qos_port_t *) cl_qmap_next(&p_port->map_item); free(p_old_port); @@ -424,9 +422,10 @@ void osm_qos_policy_match_rule_destroy(osm_qos_match_rule_t * p) /*************************************************** ***************************************************/ -osm_qos_policy_t * osm_qos_policy_create(osm_subn_t * p_subn) +osm_qos_policy_t *osm_qos_policy_create(osm_subn_t * p_subn) { - osm_qos_policy_t * p_qos_policy = (osm_qos_policy_t *)malloc(sizeof(osm_qos_policy_t)); + osm_qos_policy_t *p_qos_policy = + (osm_qos_policy_t *) malloc(sizeof(osm_qos_policy_t)); if (!p_qos_policy) return NULL; @@ -544,8 +543,8 @@ __qos_policy_is_port_in_group(osm_subn_t * p_subn, /* check whether this port's type matches any of group's types */ - if ( p_port_group->node_types & - (((uint8_t)1)<node_types & + (((uint8_t) 1) << osm_node_get_type(p_node))) return TRUE; /* check whether this port's guid is in group's port map */ @@ -585,24 +584,33 @@ __qos_policy_is_port_in_group_list(const osm_qos_policy_t * p_qos_policy, /*************************************************** ***************************************************/ -static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( - const osm_qos_policy_t * p_qos_policy, - uint64_t service_id, - uint16_t qos_class, - uint16_t pkey, - const osm_physp_t * p_src_physp, - const osm_physp_t * p_dest_physp, - ib_net64_t comp_mask) +static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params(const + osm_qos_policy_t + * + p_qos_policy, + uint64_t + service_id, + uint16_t + qos_class, + uint16_t + pkey, + const + osm_physp_t * + p_src_physp, + const + osm_physp_t * + p_dest_physp, + ib_net64_t + comp_mask) { osm_qos_match_rule_t *p_qos_match_rule = NULL; cl_list_iterator_t list_iterator; - osm_log_t * p_log = &p_qos_policy->p_subn->p_osm->log; + osm_log_t *p_log = &p_qos_policy->p_subn->p_osm->log; boolean_t matched_by_sguid = FALSE, - matched_by_dguid = FALSE, - matched_by_class = FALSE, - matched_by_sid = FALSE, - matched_by_pkey = FALSE; + matched_by_dguid = FALSE, + matched_by_class = FALSE, + matched_by_sid = FALSE, matched_by_pkey = FALSE; if (!cl_list_count(&p_qos_policy->qos_match_rules)) return NULL; @@ -698,8 +706,7 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( if (!__is_num_in_range_arr (p_qos_match_rule->pkey_range_arr, - p_qos_match_rule->pkey_range_len, - pkey & 0x7FFF)) { + p_qos_match_rule->pkey_range_len, pkey & 0x7FFF)) { list_iterator = cl_list_next(list_iterator); continue; } @@ -717,15 +724,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( OSM_LOG(p_log, OSM_LOG_DEBUG, "request matched rule (%s) by:%s%s%s%s%s\n", (p_qos_match_rule->use) ? - p_qos_match_rule->use : "no description", + p_qos_match_rule->use : "no description", (matched_by_sguid) ? " SGUID" : "", (matched_by_dguid) ? " DGUID" : "", (matched_by_class) ? " QoS_Class" : "", - (matched_by_sid) ? " ServiceID" : "", - (matched_by_pkey) ? " PKey" : ""); + (matched_by_sid) ? " ServiceID" : "", + (matched_by_pkey) ? " PKey" : ""); else - OSM_LOG(p_log, OSM_LOG_DEBUG, - "request not matched any rule\n"); + OSM_LOG(p_log, OSM_LOG_DEBUG, "request not matched any rule\n"); OSM_LOG_EXIT(p_log); return p_qos_match_rule; @@ -734,9 +740,10 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( /*************************************************** ***************************************************/ -static osm_qos_level_t *__qos_policy_get_qos_level_by_name( - const osm_qos_policy_t * p_qos_policy, - char *name) +static osm_qos_level_t *__qos_policy_get_qos_level_by_name(const + osm_qos_policy_t * + p_qos_policy, + char *name) { osm_qos_level_t *p_qos_level = NULL; cl_list_iterator_t list_iterator; @@ -760,9 +767,11 @@ static osm_qos_level_t *__qos_policy_get_qos_level_by_name( /*************************************************** ***************************************************/ -static osm_qos_port_group_t *__qos_policy_get_port_group_by_name( - const osm_qos_policy_t * p_qos_policy, - const char *const name) +static osm_qos_port_group_t *__qos_policy_get_port_group_by_name(const + osm_qos_policy_t + * p_qos_policy, + const char + *const name) { osm_qos_port_group_t *p_port_group = NULL; cl_list_iterator_t list_iterator; @@ -787,15 +796,16 @@ static osm_qos_port_group_t *__qos_policy_get_port_group_by_name( /*************************************************** ***************************************************/ -static void __qos_policy_validate_pkey( - osm_qos_policy_t * p_qos_policy, - osm_qos_match_rule_t * p_qos_match_rule, - osm_prtn_t * p_prtn) +static void __qos_policy_validate_pkey(osm_qos_policy_t * p_qos_policy, + osm_qos_match_rule_t * p_qos_match_rule, + osm_prtn_t * p_prtn) { uint8_t sl; uint32_t flow; uint8_t hop; - osm_mgrp_t * p_mgrp; + osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; + cl_fmap_item_t *p_fitem; if (!p_qos_policy || !p_qos_match_rule || !p_prtn) return; @@ -812,43 +822,49 @@ static void __qos_policy_validate_pkey( p_qos_match_rule->p_qos_level->sl); p_prtn->sl = p_qos_match_rule->p_qos_level->sl; - /* If this partition is an IPoIB partition, there should be a matching MCast group. Fix this group's SL too */ if (!p_prtn->mlid) return; - p_mgrp = osm_get_mgrp_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); - if (!p_mgrp) { + p_mgrp_holder = + osm_get_mgrp_holder_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); + if (!p_mgrp_holder) { OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, - "ERR AC16: MCast group for partition with " - "pkey 0x%04X not found\n", - cl_ntoh16(p_prtn->pkey)); + "ERR AC16: MCast mgrp_holder for partition with " + "pkey 0x%04X not found\n", cl_ntoh16(p_prtn->pkey)); return; } - - CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == - (cl_ntoh16(p_prtn->pkey) & 0x7fff)); - - ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, - &sl, &flow, &hop); - if (sl != p_prtn->sl) { - OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, - "Updating MCGroup (MLID 0x%04x) SL to " - "match partition SL (%u)\n", - cl_hton16(p_mgrp->mcmember_rec.mlid), - p_prtn->sl); - p_mgrp->mcmember_rec.sl_flow_hop = - ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + p_fitem = cl_fmap_next(p_fitem); + + CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == + (cl_ntoh16(p_prtn->pkey) & 0x7fff)); + + ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, + &sl, &flow, &hop); + if (sl != p_prtn->sl) { + OSM_LOG(&p_qos_policy->p_subn->p_osm->log, + OSM_LOG_DEBUG, + "Updating MCGroup (MLID 0x%04x) SL to " + "match partition SL (%u)\n", + cl_hton16(p_mgrp->mcmember_rec.mlid), + p_prtn->sl); + p_mgrp->mcmember_rec.sl_flow_hop = + ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); + } } } /*************************************************** ***************************************************/ -int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, - osm_log_t *p_log) +int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, osm_log_t * p_log) { cl_list_iterator_t match_rules_list_iterator; cl_list_iterator_t list_iterator; @@ -859,22 +875,23 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, int res = 0; uint64_t pkey_64; ib_net16_t pkey; - osm_prtn_t * p_prtn; + osm_prtn_t *p_prtn; OSM_LOG_ENTER(p_log); /* set default qos level */ p_qos_policy->p_default_qos_level = - __qos_policy_get_qos_level_by_name(p_qos_policy, OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); + __qos_policy_get_qos_level_by_name(p_qos_policy, + OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); if (!p_qos_policy->p_default_qos_level) { /* There's no default QoS level in the usual qos-level section. Check whether the 'simple' default QoS level that can be defined in the qos-ulp section exists */ if (__default_simple_qos_level.sl_set) { - p_qos_policy->p_default_qos_level = &__default_simple_qos_level; - } - else { + p_qos_policy->p_default_qos_level = + &__default_simple_qos_level; + } else { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC10: " "Default qos-level (%s) not defined.\n", OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); @@ -891,8 +908,7 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, cl_list_head(&p_qos_policy->qos_match_rules); while (match_rules_list_iterator != cl_list_end(&p_qos_policy->qos_match_rules)) { - p_qos_match_rule = - (osm_qos_match_rule_t *) + p_qos_match_rule = (osm_qos_match_rule_t *) cl_list_obj(match_rules_list_iterator); CL_ASSERT(p_qos_match_rule); @@ -900,8 +916,9 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, if (!p_qos_match_rule->p_qos_level) p_qos_match_rule->p_qos_level = - __qos_policy_get_qos_level_by_name(p_qos_policy, - p_qos_match_rule->qos_level_name); + __qos_policy_get_qos_level_by_name(p_qos_policy, + p_qos_match_rule-> + qos_level_name); if (!p_qos_match_rule->p_qos_level) { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC11: " @@ -922,9 +939,11 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, CL_ASSERT(str); p_port_group = - __qos_policy_get_port_group_by_name(p_qos_policy, str); + __qos_policy_get_port_group_by_name + (p_qos_policy, str); if (!p_port_group) { - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC12: " + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR AC12: " "qos-match-rule num %u: source port-group '%s' not found\n", i, str); res = 1; @@ -951,9 +970,11 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, CL_ASSERT(str); p_port_group = - __qos_policy_get_port_group_by_name(p_qos_policy,str); + __qos_policy_get_port_group_by_name + (p_qos_policy, str); if (!p_port_group) { - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC13: " + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR AC13: " "qos-match-rule num %u: destination port-group '%s' not found\n", i, str); res = 1; @@ -977,24 +998,30 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, */ for (j = 0; j < p_qos_match_rule->pkey_range_len; j++) { - for ( pkey_64 = p_qos_match_rule->pkey_range_arr[j][0]; - pkey_64 <= p_qos_match_rule->pkey_range_arr[j][1]; - pkey_64++) { - pkey = cl_hton16((uint16_t)(pkey_64 & 0x7fff)); - p_prtn = (osm_prtn_t *)cl_qmap_get( - &p_qos_policy->p_subn->prtn_pkey_tbl, pkey); - - if (p_prtn == (osm_prtn_t *)cl_qmap_end( - &p_qos_policy->p_subn->prtn_pkey_tbl)) + for (pkey_64 = p_qos_match_rule->pkey_range_arr[j][0]; + pkey_64 <= p_qos_match_rule->pkey_range_arr[j][1]; + pkey_64++) { + pkey = cl_hton16((uint16_t) (pkey_64 & 0x7fff)); + p_prtn = + (osm_prtn_t *) cl_qmap_get(&p_qos_policy-> + p_subn-> + prtn_pkey_tbl, + pkey); + + if (p_prtn == + (osm_prtn_t *) cl_qmap_end(&p_qos_policy-> + p_subn-> + prtn_pkey_tbl)) /* partition for this pkey not found */ - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC14: " + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR AC14: " "pkey 0x%04X in match rule - " "partition doesn't exist\n", cl_ntoh16(pkey)); else __qos_policy_validate_pkey(p_qos_policy, - p_qos_match_rule, - p_prtn); + p_qos_match_rule, + p_prtn); } } @@ -1013,53 +1040,70 @@ Exit: /*************************************************** ***************************************************/ -static osm_qos_level_t * __qos_policy_get_qos_level_by_params( - IN const osm_qos_policy_t * p_qos_policy, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN uint64_t service_id, - IN uint16_t qos_class, - IN uint16_t pkey, - IN ib_net64_t comp_mask) +static osm_qos_level_t *__qos_policy_get_qos_level_by_params(IN const + osm_qos_policy_t * + p_qos_policy, + IN const + osm_physp_t * + p_src_physp, + IN const + osm_physp_t * + p_dest_physp, + IN uint64_t + service_id, + IN uint16_t + qos_class, + IN uint16_t pkey, + IN ib_net64_t + comp_mask) { osm_qos_match_rule_t *p_qos_match_rule = NULL; if (!p_qos_policy) return NULL; - p_qos_match_rule = __qos_policy_get_match_rule_by_params( - p_qos_policy, service_id, qos_class, pkey, - p_src_physp, p_dest_physp, comp_mask); + p_qos_match_rule = + __qos_policy_get_match_rule_by_params(p_qos_policy, service_id, + qos_class, pkey, p_src_physp, + p_dest_physp, comp_mask); return p_qos_match_rule ? p_qos_match_rule->p_qos_level : - p_qos_policy->p_default_qos_level; + p_qos_policy->p_default_qos_level; } /* __qos_policy_get_qos_level_by_params() */ /*************************************************** ***************************************************/ -osm_qos_level_t * osm_qos_policy_get_qos_level_by_pr( - IN const osm_qos_policy_t * p_qos_policy, - IN const ib_path_rec_t * p_pr, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN ib_net64_t comp_mask) +osm_qos_level_t *osm_qos_policy_get_qos_level_by_pr(IN const osm_qos_policy_t * + p_qos_policy, + IN const ib_path_rec_t * + p_pr, + IN const osm_physp_t * + p_src_physp, + IN const osm_physp_t * + p_dest_physp, + IN ib_net64_t comp_mask) { - return __qos_policy_get_qos_level_by_params( - p_qos_policy, p_src_physp, p_dest_physp, - cl_ntoh64(p_pr->service_id), ib_path_rec_qos_class(p_pr), - cl_ntoh16(p_pr->pkey), comp_mask); + return __qos_policy_get_qos_level_by_params(p_qos_policy, p_src_physp, + p_dest_physp, + cl_ntoh64(p_pr->service_id), + ib_path_rec_qos_class(p_pr), + cl_ntoh16(p_pr->pkey), + comp_mask); } /*************************************************** ***************************************************/ -osm_qos_level_t * osm_qos_policy_get_qos_level_by_mpr( - IN const osm_qos_policy_t * p_qos_policy, - IN const ib_multipath_rec_t * p_mpr, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN ib_net64_t comp_mask) +osm_qos_level_t *osm_qos_policy_get_qos_level_by_mpr(IN const osm_qos_policy_t * + p_qos_policy, + IN const ib_multipath_rec_t + * p_mpr, + IN const osm_physp_t * + p_src_physp, + IN const osm_physp_t * + p_dest_physp, + IN ib_net64_t comp_mask) { ib_net64_t pr_comp_mask = 0; @@ -1071,20 +1115,24 @@ osm_qos_level_t * osm_qos_policy_get_qos_level_by_mpr( * compmask. Note that only relevant bits are set. */ pr_comp_mask = - ((comp_mask & IB_MPR_COMPMASK_QOS_CLASS) ? - IB_PR_COMPMASK_QOS_CLASS : 0) | - ((comp_mask & IB_MPR_COMPMASK_PKEY) ? - IB_PR_COMPMASK_PKEY : 0) | - ((comp_mask & IB_MPR_COMPMASK_SERVICEID_MSB) ? - IB_PR_COMPMASK_SERVICEID_MSB : 0) | - ((comp_mask & IB_MPR_COMPMASK_SERVICEID_LSB) ? - IB_PR_COMPMASK_SERVICEID_LSB : 0); - - return __qos_policy_get_qos_level_by_params( - p_qos_policy, p_src_physp, p_dest_physp, - cl_ntoh64(ib_multipath_rec_service_id(p_mpr)), - ib_multipath_rec_qos_class(p_mpr), - cl_ntoh16(p_mpr->pkey), pr_comp_mask); + ((comp_mask & IB_MPR_COMPMASK_QOS_CLASS) ? + IB_PR_COMPMASK_QOS_CLASS : 0) | + ((comp_mask & IB_MPR_COMPMASK_PKEY) ? + IB_PR_COMPMASK_PKEY : 0) | + ((comp_mask & IB_MPR_COMPMASK_SERVICEID_MSB) ? + IB_PR_COMPMASK_SERVICEID_MSB : 0) | + ((comp_mask & IB_MPR_COMPMASK_SERVICEID_LSB) ? + IB_PR_COMPMASK_SERVICEID_LSB : 0); + + return __qos_policy_get_qos_level_by_params(p_qos_policy, p_src_physp, + p_dest_physp, + cl_ntoh64 + (ib_multipath_rec_service_id + (p_mpr)), + ib_multipath_rec_qos_class + (p_mpr), + cl_ntoh16(p_mpr->pkey), + pr_comp_mask); } /*************************************************** diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index fcc3f27..0c24a49 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -317,7 +318,7 @@ Exit: return (status); } -ib_api_status_t osm_sa_send(osm_sa_t *sa, +ib_api_status_t osm_sa_send(osm_sa_t * sa, IN osm_madw_t * const p_madw, IN boolean_t const resp_expected) { @@ -397,8 +398,8 @@ Exit: OSM_LOG_EXIT(sa->p_log); } -void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, - cl_qlist_t *list) +void osm_sa_respond(osm_sa_t * sa, osm_madw_t * madw, size_t attr_size, + cl_qlist_t * list) { struct item_data { cl_list_item_t list; @@ -422,13 +423,13 @@ void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, */ if (sa_mad->method == IB_MAD_METHOD_GET && num_rec > 1) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4C05: " - "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" PRIx64 "\n", - num_rec, ib_get_sa_attr_str(sa_mad->attr_id), + "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" + PRIx64 "\n", num_rec, + ib_get_sa_attr_str(sa_mad->attr_id), cl_ntoh64(sa_mad->comp_mask)); osm_sa_send_error(sa, madw, IB_SA_MAD_STATUS_TOO_MANY_RECORDS); goto Exit; } - #ifndef VENDOR_RMPP_SUPPORT trim_num_rec = (MAD_BLOCK_SIZE - IB_SA_MAD_HDR_SIZE) / attr_size; if (trim_num_rec < num_rec) { @@ -564,7 +565,7 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) p_mcm_port->scope_state, p_mcm_port->proxy_join); } -static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) +static void sa_dump_one_mgrp(osm_mgrp_t * p_mgrp, void *cxt) { struct opensm_dump_context dump_context; osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; @@ -706,6 +707,8 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) { struct opensm_dump_context dump_context; osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; int i; dump_context.p_osm = p_osm; @@ -714,9 +717,17 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) cl_plock_acquire(&p_osm->lock); for (i = 0; i <= p_osm->subn.max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = p_osm->subn.mgroups[i]; - if (p_mgrp) + p_mgrp_holder = p_osm->subn.mgroup_holders[i]; + if (!p_mgrp_holder) + continue; + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); sa_dump_one_mgrp(p_mgrp, &dump_context); + p_fitem = cl_fmap_next(&p_mgrp->mgid_item); + } } OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform\n"); cl_qlist_apply_func(&p_osm->subn.sa_infr_list, @@ -740,25 +751,35 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, unsigned well_known) { ib_net64_t comp_mask; - osm_mgrp_t *p_mgrp; - + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; + ib_gid_t common_mgid; + osm_mgrp_t *p_mgrp = NULL; cl_plock_excl_acquire(&p_osm->lock); - p_mgrp = osm_get_mgrp_by_mlid(&p_osm->subn, mlid); - if (p_mgrp) { - if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, - sizeof(ib_gid_t))) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(&p_osm->subn, mlid); + if (p_mgrp_holder) { + p_fitem = + cl_fmap_get(&p_mgrp_holder->mgrp_map, &p_mcm_rec->mgid); + if (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "mgrp %04x is already here.", cl_ntoh16(mlid)); goto _out; } - OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, - "mlid %04x is already used by another MC group. Will " - "request clients reregistration.\n", cl_ntoh16(mlid)); - p_mgrp = NULL; - goto _out; - } + osm_mgrp_holder_prepare_common_mgid(&p_mcm_rec->mgid, + &common_mgid); + if (memcmp + (&p_mgrp_holder->common_mgid, &common_mgid, + sizeof(ib_gid_t))) { + OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, + "mlid %04x is already used by another MC group. Will " + "request clients reregistration.\n", + cl_ntoh16(mlid)); + p_mgrp = NULL; + goto _out; + } + } comp_mask = IB_MCR_COMPMASK_MTU | IB_MCR_COMPMASK_MTU_SEL | IB_MCR_COMPMASK_RATE | IB_MCR_COMPMASK_RATE_SEL; if (osm_mcmr_rcv_find_or_create_new_mgrp(&p_osm->sa, diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 5543221..23c5107 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -63,6 +63,7 @@ #include #include #include +#include #define JOIN_MC_COMP_MASK (IB_MCR_COMPMASK_MGID | \ IB_MCR_COMPMASK_PORT_GID | \ @@ -121,13 +122,14 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_mcast_lid_ho - && !osm_get_mgrp_by_mlid(p_subn, requested_mlid)) + && !osm_get_mgrp_holder_by_mlid(p_subn, requested_mlid)) return requested_mlid; max = p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO + 1; for (i = 0; i < max; i++) { - osm_mgrp_t *p_mgrp = sa->p_subn->mgroups[i]; - if (!p_mgrp || p_mgrp->to_be_deleted) + osm_mgrp_holder_t *p_mgrp_holder = + (osm_mgrp_holder_t *) sa->p_subn->mgroup_holders[i]; + if (!p_mgrp_holder || p_mgrp_holder->to_be_deleted) return cl_hton16(i + IB_LID_MCAST_START_HO); } @@ -141,14 +143,21 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) we silently drop it. Since it was an intermediate group no need to re-route it. **********************************************************************/ -static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * mgrp) +static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * p_mgrp) { /* Remove MGRP only if osm_mcm_port_t count is 0 and not a well known group */ - if (cl_is_qmap_empty(&mgrp->mcm_port_tbl) && !mgrp->well_known) { - sa->p_subn->mgroups[cl_ntoh16(mgrp->mlid) - - IB_LID_MCAST_START_HO] = NULL; - osm_mgrp_delete(mgrp); + char gid_str[INET6_ADDRSTRLEN]; + if (cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && !p_mgrp->well_known) { + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "mgrp mgid %s will be deleted\n", inet_ntop(AF_INET6, + p_mgrp-> + mcmember_rec. + mgid.raw, + gid_str, + sizeof + (gid_str))); + osm_mgrp_delete_group(sa->sm->p_subn, p_mgrp); } } @@ -202,7 +211,6 @@ static ib_api_status_t add_new_mgrp_port(osm_sa_t * sa, IN osm_mgrp_t * p_mgrp, return IB_INSUFFICIENT_MEMORY; } - return IB_SUCCESS; } @@ -806,11 +814,12 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, IN const osm_physp_t * p_physp, OUT osm_mgrp_t ** pp_mgrp) { - ib_net16_t mlid; + ib_net16_t mlid, existed_mlid; unsigned zero_mgid, i; uint8_t scope; + char gid_str[INET6_ADDRSTRLEN]; ib_gid_t *p_mgid; - osm_mgrp_t *p_prev_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; ib_api_status_t status = IB_SUCCESS; ib_member_rec_t mcm_rec = *p_recvd_mcmember_rec; /* copy for modifications */ @@ -823,7 +832,7 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, zero_mgid = 0; break; } - + p_mgid = &(mcm_rec.mgid); /* we allocate a new mlid number before we might use it for MGID ... @@ -843,8 +852,6 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* we need to create the new MGID if it was not defined */ if (zero_mgid) { /* create a new MGID */ - char gid_str[INET6_ADDRSTRLEN]; - /* use the given scope state only if requested! */ if (comp_mask & IB_MCR_COMPMASK_SCOPE) ib_member_get_scope_state(p_recvd_mcmember_rec-> @@ -888,6 +895,13 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, goto Exit; } + if (0 != (existed_mlid = osm_mgrp_holder_get_mlid_by_mgid(sa, p_mgid))) { + mlid = existed_mlid; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "found existed mlid 0x%04x for mgid %s\n", + cl_ntoh16(mlid), inet_ntop(AF_INET6, p_mgid->raw, + gid_str, sizeof gid_str)); + } /* create a new MC Group */ *pp_mgrp = osm_mgrp_new(mlid); if (*pp_mgrp == NULL) { @@ -915,17 +929,21 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* since we might have an old group by that mlid one whose deletion was delayed for an idle time we need to deallocate it first */ - p_prev_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); - if (p_prev_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sa->p_subn, mlid); + if (!p_mgrp_holder) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Found previous group for mlid:0x%04x - " - "Destroying it first\n", cl_ntoh16(mlid)); - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = - NULL; - osm_mgrp_delete(p_prev_mgrp); + "Creating new mgrp holder for mlid:0x%04x\n", + cl_ntoh16(mlid)); + p_mgrp_holder = osm_mgrp_holder_new(sa->p_subn, p_mgid, mlid); } - - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; + if (!p_mgrp_holder) { + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B08: " + "osm_mgrp_holder_new failed\n"); + free_mlid(sa, mlid); + status = IB_INSUFFICIENT_MEMORY; + goto Exit; + } + osm_mgrp_holder_add_mgrp(p_mgrp_holder, *pp_mgrp, sa->p_log); Exit: OSM_LOG_EXIT(sa->p_log); @@ -934,56 +952,47 @@ Exit: /********************************************************************** *********************************************************************/ -static unsigned match_mgrp_by_mgid(IN osm_mgrp_t * p_mgrp, ib_gid_t * mgid) -{ - /* ignore groups marked for deletion */ - if (p_mgrp->to_be_deleted || - memcmp(&p_mgrp->mcmember_rec.mgid, mgid, sizeof(ib_gid_t))) - return 0; - else - return 1; -} - -/********************************************************************** - **********************************************************************/ -#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) -#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) -#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) -#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) - -/* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ -/* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ -/* Where Z is the scope, XXXX is the P_Key, and - * YYYYYY is the last 24 bits of the port guid */ -static unsigned match_and_update_ipv6_snm_mgid(ib_gid_t * mgid) +static osm_mgrp_t *match_mgrp_by_mgid(IN osm_mgrp_holder_t * p_mgrp_holder, + ib_gid_t * p_mgid) { - if ((mgid->unicast.prefix & PREFIX_MASK) == PREFIX_SIGNATURE && - (mgid->unicast.interface_id & INT_ID_MASK) == INT_ID_SIGNATURE) { - mgid->unicast.prefix &= PREFIX_MASK; - mgid->unicast.interface_id &= INT_ID_MASK; - return 1; - } - return 0; + osm_mgrp_t *p_mgrp; + ib_gid_t common_mgid; + cl_fmap_item_t *p_fitem; + + osm_mgrp_holder_prepare_common_mgid(p_mgid, &common_mgid); + + if (memcmp(&p_mgrp_holder->common_mgid, &common_mgid, sizeof(ib_gid_t))) + return NULL; + p_fitem = cl_fmap_get(&p_mgrp_holder->mgrp_map, p_mgid); + if (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + /* ignore groups marked for deletion */ + if (p_mgrp->to_be_deleted) + p_mgrp = NULL; + } else + p_mgrp = NULL; + return p_mgrp;; } osm_mgrp_t *osm_get_mgrp_by_mgid(IN osm_sa_t * sa, IN ib_gid_t * p_mgid) { int i; - - if (sa->p_subn->opt.consolidate_ipv6_snm_req && - match_and_update_ipv6_snm_mgid(p_mgid)) { - char gid_str[INET6_ADDRSTRLEN]; - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Special Case Solicited Node Mcast Join for MGID %s\n", - inet_ntop(AF_INET6, p_mgid->raw, gid_str, - sizeof gid_str)); - } - + osm_mgrp_t *p_mgrp; for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; - i++) - if (sa->p_subn->mgroups[i] && - match_mgrp_by_mgid(sa->p_subn->mgroups[i], p_mgid)) - return sa->p_subn->mgroups[i]; + i++) { + if (sa->p_subn->mgroup_holders[i] && + (p_mgrp = + match_mgrp_by_mgid(sa->p_subn->mgroup_holders[i], + p_mgid))) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "Found mgroup for MGID %s\n", + inet_ntop(AF_INET6, p_mgid->raw, gid_str, + sizeof gid_str)); + return p_mgrp; + } + } return NULL; } @@ -1080,7 +1089,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) CL_PLOCK_RELEASE(sa->p_lock); /* we can leave if port was deleted from MCG */ - if (removed && osm_sm_mcgrp_leave(sa->sm, mlid, portguid)) + if (removed && osm_sm_mcgrp_leave(sa->sm, p_mgrp, portguid)) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B09: " "osm_sm_mcgrp_leave failed\n"); @@ -1310,7 +1319,7 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* do the actual routing (actually schedule the update) */ status = osm_sm_mcgrp_join(sa->sm, mlid, p_recvd_mcmember_rec->port_gid.unicast. - interface_id); + interface_id, &p_recvd_mcmember_rec->mgid); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " @@ -1555,6 +1564,8 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) osm_physp_t *p_req_physp; boolean_t trusted_req; osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; int i; OSM_LOG_ENTER(sa->p_log); @@ -1586,10 +1597,18 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* simply go over all MCGs and match */ for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = sa->p_subn->mgroups[i]; - if (p_mgrp) + p_mgrp_holder = sa->p_subn->mgroup_holders[i]; + if (!p_mgrp_holder) + continue; + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); mcmr_by_comp_mask(sa, p_rcvd_rec, comp_mask, p_mgrp, p_req_physp, trusted_req, &rec_list); + p_fitem = cl_fmap_next(p_fitem); + } } CL_PLOCK_RELEASE(sa->p_lock); diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 9b50deb..3271289 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -1438,6 +1438,7 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; ib_net64_t comp_mask; + osm_mgrp_holder_t *p_mgrp_holder; osm_mgrp_t *mgrp = NULL; p_sa_mad = osm_madw_get_sa_mad_ptr(p_madw); @@ -1468,11 +1469,30 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) mgrp = NULL; goto Exit; } - } else - if (!(mgrp = osm_get_mgrp_by_mlid(sa->p_subn, p_pr->dlid))) - OSM_LOG(sa->p_log, OSM_LOG_ERROR, - "ERR 1F11: " "No MC group found for PathRecord " - "destination LID 0x%x\n", p_pr->dlid); + } else { + cl_fmap_item_t *p_fitem; + p_mgrp_holder = + osm_get_mgrp_holder_by_mlid(sa->p_subn, p_pr->dlid); + if (p_mgrp_holder) { + p_fitem = + cl_fmap_get(&p_mgrp_holder->mgrp_map, + &p_pr->dgid); + } + if (!p_mgrp_holder + || p_fitem == + cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + OSM_LOG(sa->p_log, OSM_LOG_ERROR, + "ERR 1F11: " + "No MC group found for PathRecord " + "destination LID 0x%x\n", p_pr->dlid); + } else { + mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, + osm_mgrp_t, + mgid_item); + } + + } } Exit: diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index daa60ff..459e2cc 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -47,6 +47,7 @@ #include #include +#include #include #include #include @@ -468,12 +469,15 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, /********************************************************************** **********************************************************************/ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid) + IN const ib_net64_t port_guid, + IN const ib_gid_t * p_mgid) { - osm_mgrp_t *p_mgrp; + osm_mgrp_t *p_mgrp = NULL; osm_port_t *p_port; ib_api_status_t status = IB_SUCCESS; osm_mcm_info_t *p_mcm; + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; OSM_LOG_ENTER(p_sm->p_log); @@ -498,8 +502,51 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, /* * If this multicast group does not already exist, create it. */ - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); - if (!p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); + if (p_mgrp_holder) { + char gid_str[INET6_ADDRSTRLEN]; + if (osm_is_debug()) { + size_t gr_count = + cl_fmap_count(&p_mgrp_holder->mgrp_map); + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "mlid 0x%X has %d mgroups\n", cl_ntoh16(mlid), + gr_count); + if (gr_count) { + p_fitem = + cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != + cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) + PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "mlid 0x%X has mgrp with MGID: %s\n", + cl_ntoh16(mlid), + inet_ntop(AF_INET6, + p_mgrp->mcmember_rec. + mgid.raw, gid_str, + sizeof gid_str)); + p_fitem = cl_fmap_next(p_fitem); + } + } + } + p_fitem = cl_fmap_get(&p_mgrp_holder->mgrp_map, p_mgid); + if (p_fitem == cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = NULL; + OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, + "group with MGID: %s not found on mlid 0x%X\n", + inet_ntop(AF_INET6, + p_mgid->raw, + gid_str, sizeof gid_str), + cl_ntoh16(mlid)); + } else { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + } + } + if (!p_mgrp_holder || !p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { /* * The group removed or the port is not a * member of the group, then fail immediately. @@ -514,7 +561,23 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, status = IB_NOT_FOUND; goto Exit; } - + /* if there was no change from the last time + * we processed the group we can skip doing anything + */ + if (p_mgrp_holder->last_change_id == p_mgrp_holder->last_tree_id) { + CL_PLOCK_RELEASE(p_sm->p_lock); + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, + "Skip processing mgrp holder with lid:0x%X last change id:%u\n", + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); + goto Exit; + } else { + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "processing mgrp holder with lid:0x%X port: 0x%016" + PRIx64 " last change id:%u tree id:%u\n", + cl_ntoh16(mlid), cl_ntoh64(port_guid), + p_mgrp_holder->last_change_id, + p_mgrp_holder->last_tree_id); + } /* * Check if the object (according to mlid) already exists on this port. * If it does - then no need to update it again, and no need to @@ -543,9 +606,6 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } - status = sm_mgrp_process(p_sm, p_mgrp); - CL_PLOCK_RELEASE(p_sm->p_lock); - Exit: OSM_LOG_EXIT(p_sm->p_log); return status; @@ -553,12 +613,13 @@ Exit: /********************************************************************** **********************************************************************/ -ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, +ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN osm_mgrp_t * p_mgrp, IN const ib_net64_t port_guid) { - osm_mgrp_t *p_mgrp; osm_port_t *p_port; ib_api_status_t status; + osm_mgrp_holder_t *p_mgrp_holder; + ib_net16_t mlid = p_mgrp->mlid; OSM_LOG_ENTER(p_sm->p_log); @@ -584,8 +645,8 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, /* * Get the multicast group object for this group. */ - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); - if (!p_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); + if (!p_mgrp_holder) { CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); @@ -593,11 +654,14 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } + osm_mgrp_holder_delete_mgrp_port(p_mgrp_holder, p_mgrp, port_guid); /* * Walk the list of ports in the group, and remove the appropriate one. */ osm_port_remove_mgrp(p_port, mlid); - + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + " Calling sm_mgrp_process for mgrp with mlid = 0x%X\n", + cl_ntoh16(mlid)); status = sm_mgrp_process(p_sm, p_mgrp); CL_PLOCK_RELEASE(p_sm->p_lock); diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index ec15f8a..e6e624d 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -419,7 +419,7 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) osm_switch_t *p_sw, *p_next_sw; osm_remote_sm_t *p_rsm, *p_next_rsm; osm_prtn_t *p_prtn, *p_next_prtn; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_osm_mgrp_holder; osm_infr_t *p_infr, *p_next_infr; /* it might be a good idea to de-allocate all known objects */ @@ -464,10 +464,10 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) for (i = 0; i <= p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = p_subn->mgroups[i]; - p_subn->mgroups[i] = NULL; - if (p_mgrp) - osm_mgrp_delete(p_mgrp); + p_osm_mgrp_holder = p_subn->mgroup_holders[i]; + if (p_osm_mgrp_holder){ + osm_mgrp_holder_delete(p_subn, p_osm_mgrp_holder->mlid); + } } p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); -- 1.5.5 From slavas at Voltaire.COM Sun Jun 28 03:57:05 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Sun, 28 Jun 2009 13:57:05 +0300 Subject: [ofa-general] [PATCH 2/4 v2] multicast multiplexing - implementation of new infrastructure In-Reply-To: <4A3A387A.2020509@Voltaire.COM> References: <4A3A387A.2020509@Voltaire.COM> Message-ID: <4A474C81.2020602@Voltaire.COM> Subject: [PATCH 2/4] Patch implements multicast multiplexing as proposed in the thread entitled "IPv6 and IPoIB scalability issue". This second part contains functions for new infrastructure management Signed-off-by: Slava Strebkov --- opensm/include/opensm/osm_mcm_info.h | 35 ++++- opensm/include/opensm/osm_multicast.h | 297 +++++++++++++++++++++++++++++++++ opensm/include/opensm/osm_subnet.h | 7 +- opensm/opensm/osm_multicast.c | 227 +++++++++++++++++++++++++ 4 files changed, 562 insertions(+), 4 deletions(-) diff --git a/opensm/include/opensm/osm_mcm_info.h b/opensm/include/opensm/osm_mcm_info.h index dec607f..2310d25 100644 --- a/opensm/include/opensm/osm_mcm_info.h +++ b/opensm/include/opensm/osm_mcm_info.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * @@ -47,6 +47,7 @@ #include #include #include +#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -71,7 +72,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct osm_mcm_info { + typedef struct osm_mcm_info { cl_list_item_t list_item; ib_net16_t mlid; } osm_mcm_info_t; @@ -131,6 +132,34 @@ void osm_mcm_info_delete(IN osm_mcm_info_t * const p_mcm); * * SEE ALSO *********/ - +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_get_mlid_by_mgid +* NAME +* osm_mgrp_holder_get_mlid_by_mgid +* +* DESCRIPTION +* Searches holder which contains mgroup with given mgid +* Returns mlid of the found holder +* +* SYNOPSIS +*/ +ib_net16_t osm_mgrp_holder_get_mlid_by_mgid(IN osm_sa_t * sa, + IN const ib_gid_t * const p_mgid); +/* +* PARAMETERS +* +* p_sa +* [in] Pointer to sa object +* +* p_mgid +* [in] pointer to mgid +* +* RETURN VALUES +* mlid of found holder, or zero. +* +* NOTES +* +* SEE ALSO +* +*********/ END_C_DECLS #endif /* _OSM_MCM_INFO_H_ */ diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index 02b63bd..9f0cd96 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -214,6 +214,8 @@ typedef struct osm_mgrp { uint32_t last_change_id; uint32_t last_tree_id; unsigned full_members; + cl_fmap_item_t mgid_item; + cl_list_item_t mgrp_item; } osm_mgrp_t; /* * FIELDS @@ -254,6 +256,12 @@ typedef struct osm_mgrp { * last_tree_id * the last change id used for building the current tree. * +* mgid_item +* fleximap item for map linkage, sorted by mgid. +* +* mgrp_item +* list item for mgroups pointer list. +* * SEE ALSO *********/ @@ -572,6 +580,295 @@ osm_mgrp_apply_func(const osm_mgrp_t * const p_mgrp, * SEE ALSO * Multicast Group *********/ +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_new +* NAME +* osm_mgrp_holder_new +* +* DESCRIPTION +* Allocates and initializes a Multicast Group Holder for use. +* +* SYNOPSIS +*/ +osm_mgrp_holder_t *osm_mgrp_holder_new(IN osm_subn_t * p_subn, + IN ib_gid_t * p_mgid, + IN ib_net16_t mlid); +/* +* PARAMETERS +* p_subn +* (in) pointer to osm_subnet +* p_mgid +* (in) pointer to mgid +* mlid +* [in] Multicast LID for this multicast group holder. +* +* RETURN VALUES +* pointer to initialized osm_mgrp_holder_t +* or NULL, if unsuccessful +* +* SEE ALSO +* Multicast Group Holder, osm_mgrp_holder_delete +*********/ +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_delete +* NAME +* osm_mgrp_holder_delete +* +* DESCRIPTION +* Removes entry from array of holders +* Removes port from mgroup port list +* +* SYNOPSIS +*/ +void osm_mgrp_holder_delete(IN osm_subn_t * p_subn, IN ib_net16_t mlid); +/* +* PARAMETERS +* +* p_subn +* [in] Pointer to osm_subnet +* +* mlid +* [in] holder's mlid +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* +*********/ +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_add_mgrp_port +* NAME +* osm_mgrp_holder_add_mgrp_port +* +* DESCRIPTION +* Allocates osm_mgrp_port_t for new port joined to mgroup with mlid of this holder, +* (or / and) adds mgroup to mgroup map of existed osm_mgrp_port_t object. +* +* SYNOPSIS +*/ +ib_api_status_t osm_mgrp_holder_add_mgrp_port(IN osm_mgrp_holder_t * + p_mgrp_holder, + IN osm_mgrp_t * p_mgrp, + IN ib_net64_t port_guid); +/* +* PARAMETERS +* p_mgrp_holder +* (in) pointer to osm_mgrp_holder_t +* p_mgrp +* (in) pointer to osm_mgrp_t +* +* RETURN VALUES +* IB_SUCCESS or +* IB_INSUFFICIENT_MEMORY +* +* SEE ALSO +* Multicast Group Holder, osm_mgrp_holder_delete_mgrp_port +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_delete_mgrp_port +* NAME +* osm_mgrp_holder_delete_mgrp_port +* +* DESCRIPTION +* Deletes osm_mgrp_port_t for specified port +* +* SYNOPSIS +*/ +void osm_mgrp_holder_delete_mgrp_port(IN osm_mgrp_holder_t * p_mgrp_holder, + IN osm_mgrp_t * p_mgrp, + IN ib_net64_t port_guid); +/* +* PARAMETERS +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* p_mgrp +* (in) Pointer to osm_mgrp_t object +* +* port_guid +* [in] Port guid of the departing port. +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO + Multicast Group Holder,osm_holder_add_mgrp_port +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_remove_port +* NAME +* osm_mgrp_holder_remove_port +* +* DESCRIPTION +* Removes osm_mgrp_port_t from mgrp_port_map of holder +* Removes port from mgroup port list +* +* SYNOPSIS +*/ +void osm_mgrp_holder_remove_port(IN osm_subn_t * const p_subn, + IN osm_log_t * const p_log, + IN osm_mgrp_holder_t * const p_mgrp_holder, + IN const ib_net64_t port_guid); +/* +* PARAMETERS +* +* p_subn +* [in] Pointer to the subnet object +* +* p_log +* [in] The log object pointer +* +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* port_guid +* [in] Port guid of the departing port. +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_add_mgrp +* NAME +* osm_mgrp_holder_add_mgrp +* +* DESCRIPTION +* Adds mgroup to holder according to its mgid +* +* +* SYNOPSIS +*/ +void osm_mgrp_holder_add_mgrp(IN osm_mgrp_holder_t * p_mgrp_holder, + IN osm_mgrp_t * p_mgrp, + IN osm_log_t * const p_log); +/* +* PARAMETERS +* +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* p_mgrp +* [in] mgroup to add. +* +* RETURN VALUES +* None. +* +* NOTES +* Updates common_mgid when holder is being reused +* SEE ALSO +* Multicast Group Holder,osm_mgrp_holder_delete_mgrp +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_delete_mgrp +* NAME +* osm_mgrp_holder_delete_mgrp +* +* DESCRIPTION +* Deletes mgroup from holder according to its mgid +* +* +* SYNOPSIS +*/ +void osm_mgrp_holder_delete_mgrp(IN osm_mgrp_holder_t * p_mgrp_holder, + IN osm_mgrp_t * p_mgrp); +/* +* PARAMETERS +* +* p_mgrp_holder +* [in] Pointer to an osm_mgrp_holder_t object. +* +* p_mgrp +* [in] mgroup to delete. +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* Multicast Group Holder,osm_mgrp_holder_add_mgrp +*********/ + +/* Multicast Group Holder, osm_mgrp_holder_delete_port +*********/ + +/****f* OpenSM: Multicast Group Holder /osm_mgrp_holder_prepare_common_mgid +* NAME +* osm_mgrp_holder_prepare_common_mgid +* +* DESCRIPTION +* Prepares mgid, which is common for all mgroups in this holder +* +* SYNOPSIS +*/ +void osm_mgrp_holder_prepare_common_mgid(IN const ib_gid_t * const p_mgid, + OUT ib_gid_t * p_common_mgid); +/* +* PARAMETERS +* +* p_mgid +* [in] Pointer to mgid +* +* p_common_mgid +* [out] common mgid +* +* RETURN VALUES +* None. +* +* NOTES +* +* SEE ALSO +* +*********/ +/****f* OpenSM: Subnet/osm_get_mgrp_by_mlid +* NAME +* osm_get_mgrp_by_mlid +* +* DESCRIPTION +* The looks for the given multicast group in the subnet table by mlid. +* NOTE: this code is not thread safe. Need to grab the lock before +* calling it. +* +* SYNOPSIS +*/ +static inline + struct osm_mgrp_holder *osm_get_mgrp_holder_by_mlid(osm_subn_t const + *p_subn, + ib_net16_t mlid) +{ + return p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; +} + +/* +* PARAMETERS +* p_subn +* [in] Pointer to an osm_subn_t object +* +* mlid +* [in] The multicast group mlid in network order +* +* RETURN VALUES +* The multicast group structure pointer if found. NULL otherwise. +*********/ + +static inline ib_net16_t osm_mgrp_holder_get_mlid(IN osm_mgrp_holder_t * + const p_mgrp_holder) +{ + return (p_mgrp_holder->mlid); +} + +static inline boolean_t osm_mgrp_holder_is_empty(IN const osm_mgrp_holder_t * + const p_mgrp_holder) +{ + return (cl_qmap_count(&p_mgrp_holder->mgrp_port_map) == 0); +} END_C_DECLS #endif /* _OSM_MULTICAST_H_ */ diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 59a32ad..6713d4d 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -509,6 +509,7 @@ typedef struct osm_subn { boolean_t coming_out_of_standby; unsigned need_update; void *mgroups[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; + void *mgroup_holders[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; } osm_subn_t; /* * FIELDS @@ -633,6 +634,10 @@ typedef struct osm_subn { * Array of pointers to all Multicast Group objects in the subnet. * Indexed by MLID offset from base MLID. * +* mgroup_holders +* Array of pointers to all Multicast Group Holder objects in the subnet. +* Indexed by MLID offset from base MLID. +* * SEE ALSO * Subnet object *********/ diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index ae0a818..d2a19ea 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -44,10 +44,12 @@ #include #include +#include #include #include #include #include +#include #include /********************************************************************** @@ -74,6 +76,31 @@ void osm_mgrp_delete(IN osm_mgrp_t * p_mgrp) free(p_mgrp); } +void osm_mgrp_delete_group(IN osm_subn_t * p_subn, IN osm_mgrp_t * p_mgrp) +{ + osm_mcm_port_t *p_mcm_port; + osm_mcm_port_t *p_next_mcm_port; + + CL_ASSERT(p_mgrp); + + osm_mgrp_holder_t *p_mgrp_holder = + osm_get_mgrp_holder_by_mlid(p_subn, p_mgrp->mlid); + p_next_mcm_port = + (osm_mcm_port_t *) cl_qmap_head(&p_mgrp->mcm_port_tbl); + while (p_next_mcm_port != + (osm_mcm_port_t *) cl_qmap_end(&p_mgrp->mcm_port_tbl)) { + p_mcm_port = p_next_mcm_port; + p_next_mcm_port = + (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item); + osm_mgrp_holder_delete_mgrp_port(p_mgrp_holder, p_mgrp, + p_mcm_port->port_gid.unicast. + interface_id); + osm_mcm_port_delete(p_mcm_port); + } + osm_mgrp_holder_delete_mgrp(p_mgrp_holder, p_mgrp); + free(p_mgrp); +} + /********************************************************************** **********************************************************************/ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid) @@ -378,3 +405,203 @@ static osm_mgrp_port_t *osm_mgrp_port_new(ib_net64_t port_guid) cl_qlist_init(&p_mgrp_port->mgroups); return p_mgrp_port; } + +void osm_mgrp_holder_delete(IN osm_subn_t * p_subn, ib_net16_t mlid) +{ + osm_mgrp_port_t *p_osm_mgr_port; + cl_map_item_t *p_item; + cl_fmap_item_t *p_fitem; + osm_mgrp_t *p_mgrp; + + osm_mgrp_holder_t *p_mgrp_holder = + p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + p_item = cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + while (p_item != cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + p_osm_mgr_port = (osm_mgrp_port_t *) p_item; + cl_qlist_remove_all(&p_osm_mgr_port->mgroups); + cl_qmap_remove_item(&p_mgrp_holder->mgrp_port_map, p_item); + p_item = cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + free(p_osm_mgr_port); + } + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + osm_mgrp_delete_group(p_subn, p_mgrp); + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + } + /* destroy the mtree_node structure */ + osm_mtree_destroy(p_mgrp_holder->p_root); + p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; + free(p_mgrp_holder); +} + +osm_mgrp_holder_t *osm_mgrp_holder_new(IN osm_subn_t * p_subn, + ib_gid_t * p_mgid, ib_net16_t mlid) +{ + osm_mgrp_holder_t *p_mgrp_holder; + p_mgrp_holder = + p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = + (osm_mgrp_holder_t *) malloc(sizeof(*p_mgrp_holder)); + if (!p_mgrp_holder) + return NULL; + + memset(p_mgrp_holder, 0, sizeof(*p_mgrp_holder)); + p_mgrp_holder->mlid = mlid; + cl_fmap_init(&p_mgrp_holder->mgrp_map, __mgid_cmp); + cl_qmap_init(&p_mgrp_holder->mgrp_port_map); + osm_mgrp_holder_prepare_common_mgid(p_mgid, + &p_mgrp_holder->common_mgid); + return p_mgrp_holder; +} + +ib_net16_t osm_mgrp_holder_get_mlid_by_mgid(IN osm_sa_t * sa, + IN const ib_gid_t * const p_mgid) +{ + int i; + ib_gid_t common_mgid; + osm_mgrp_holder_t *p_mgrp_holder; + + OSM_LOG_ENTER(sa->p_log); + + osm_mgrp_holder_prepare_common_mgid(p_mgid, &common_mgid); + for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; + i++) { + if (sa->p_subn->mgroup_holders[i]) { + p_mgrp_holder = + (osm_mgrp_holder_t *) sa->p_subn->mgroup_holders[i]; + if (!memcmp + (&p_mgrp_holder->common_mgid, &common_mgid, + sizeof(ib_gid_t))) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "Found holder 0x%X for MGID %s\n", + cl_ntoh16(p_mgrp_holder->mlid), + inet_ntop(AF_INET6, p_mgid->raw, + gid_str, sizeof(gid_str))); + OSM_LOG_EXIT(sa->p_log); + return p_mgrp_holder->mlid; + } + } + } + OSM_LOG_EXIT(sa->p_log); + return 0; +} + +void osm_mgrp_holder_remove_port(osm_subn_t * subn, osm_log_t * p_log, + osm_mgrp_holder_t * p_mgrp_holder, + ib_net64_t port_guid) +{ + osm_mgrp_t *p_mgrp; + cl_list_item_t *p_item; + + OSM_LOG_ENTER(p_log); + + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t *) + cl_qmap_remove(&p_mgrp_holder->mgrp_port_map, port_guid); + if (p_mgrp_port != + (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(p_log, OSM_LOG_DEBUG, + "port 0x%" PRIx64 " removed from mlid 0x%X\n", + port_guid, cl_ntoh16(p_mgrp_holder->mlid)); + while ((p_item = + cl_qlist_remove_head(&p_mgrp_port->mgroups)) != + cl_qlist_end(&p_mgrp_port->mgroups)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_item, osm_mgrp_t, + mgrp_item); + OSM_LOG(p_log, OSM_LOG_DEBUG, + "removing mgrp mgid %s from port 0x%" PRIx64 + "\n", inet_ntop(AF_INET6, + p_mgrp->mcmember_rec.mgid.raw, + gid_str, sizeof(gid_str)), + cl_ntoh64(port_guid)); + osm_mgrp_delete_port(subn, p_log, p_mgrp, port_guid); + } + free(p_mgrp_port); + } + OSM_LOG_EXIT(p_log); +} + +void osm_mgrp_holder_add_mgrp(osm_mgrp_holder_t * p_mgrp_holder, + osm_mgrp_t * p_mgrp, osm_log_t * p_log) +{ + cl_fmap_item_t *p_fitem; + char gid_str[INET6_ADDRSTRLEN]; + + OSM_LOG_ENTER(p_log); + + if (p_mgrp_holder->to_be_deleted) { + /* this is re-used mgrp_holder, need to update common_mgid */ + osm_mgrp_holder_prepare_common_mgid(&p_mgrp->mcmember_rec.mgid, + &p_mgrp_holder-> + common_mgid); + } + p_fitem = cl_fmap_insert(&p_mgrp_holder->mgrp_map, + &p_mgrp->mcmember_rec.mgid, + &p_mgrp->mgid_item); + CL_ASSERT(p_item == &p_mgrp->mgid_item); + + OSM_LOG(p_log, OSM_LOG_DEBUG, + "mgrp with MGID:%s added to holder with mlid = 0x%X\n", + inet_ntop(AF_INET6, p_mgrp->mcmember_rec.mgid.raw, gid_str, + sizeof(gid_str)), cl_ntoh16(p_mgrp_holder->mlid)); + p_mgrp_holder->last_change_id++; + OSM_LOG_EXIT(p_log); +} + +ib_api_status_t osm_mgrp_holder_add_mgrp_port(osm_mgrp_holder_t * p_mgrp_holder, + osm_mgrp_t * p_mgrp, + ib_net64_t port_guid) +{ + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t *) + cl_qmap_get(&p_mgrp_holder->mgrp_port_map, port_guid); + if (p_mgrp_port == + (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + /* new port to mlid */ + p_mgrp_port = osm_mgrp_port_new(port_guid); + if (!p_mgrp_port) { + return IB_INSUFFICIENT_MEMORY; + } + cl_qmap_insert(&p_mgrp_holder->mgrp_port_map, + p_mgrp_port->port_guid, &p_mgrp_port->guid_item); + cl_qlist_insert_tail(&p_mgrp_port->mgroups, &p_mgrp->mgrp_item); + } else { + cl_qlist_insert_tail(&p_mgrp_port->mgroups, &p_mgrp->mgrp_item); + } + return IB_SUCCESS; +} + +void osm_mgrp_holder_delete_mgrp_port(osm_mgrp_holder_t * p_mgrp_holder, + osm_mgrp_t * p_mgrp, ib_net64_t port_guid) +{ + osm_mgrp_port_t *p_mgrp_port = (osm_mgrp_port_t *) + cl_qmap_get(&p_mgrp_holder->mgrp_port_map, port_guid); + if (p_mgrp_port != + (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map)) { + cl_qlist_remove_item(&p_mgrp_port->mgroups, &p_mgrp->mgrp_item); + if (0 == cl_qlist_count(&p_mgrp_port->mgroups)) { + /* No mgroups registered on this port for current mlid */ + cl_qmap_remove_item(&p_mgrp_holder->mgrp_port_map, + &p_mgrp_port->guid_item); + free(p_mgrp_port); + } + p_mgrp_holder->last_change_id++; + } +} + +void osm_mgrp_holder_delete_mgrp(osm_mgrp_holder_t * p_mgrp_holder, + osm_mgrp_t * p_mgrp) +{ + p_mgrp->to_be_deleted = 1; + cl_fmap_remove_item(&p_mgrp_holder->mgrp_map, &p_mgrp->mgid_item); + if (0 == cl_fmap_count(&p_mgrp_holder->mgrp_map)) { + /* No more mgroups on this mlid */ + p_mgrp_holder->to_be_deleted = 1; + p_mgrp_holder->last_tree_id = 0; + p_mgrp_holder->last_change_id = 0; + memset(&p_mgrp_holder->common_mgid, 0, sizeof(ib_gid_t)); + } +} -- 1.5.5 From slavas at Voltaire.COM Sun Jun 28 03:58:32 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Sun, 28 Jun 2009 13:58:32 +0300 Subject: [ofa-general] [PATCH 3/4 v2] multicast multiplexing - usage of new infrastructure, for mgid to mlid compression In-Reply-To: <4A3A387A.2020509@Voltaire.COM> References: <4A3A387A.2020509@Voltaire.COM> Message-ID: <4A474CD8.3050107@Voltaire.COM> Subject: [PATCH 3/4] Patch implements multicast multiplexing as proposed in the thread entitled "IPv6 and IPoIB scalability issue". This part (3) introduces usage of new infrastructure for mgid->mlid compression Signed-off-by: Slava Strebkov --- opensm/include/opensm/osm_multicast.h | 5 +- opensm/include/opensm/osm_sm.h | 23 ++- opensm/opensm/osm_drop_mgr.c | 14 +- opensm/opensm/osm_mcast_mgr.c | 2 +- opensm/opensm/osm_multicast.c | 2 +- opensm/opensm/osm_qos_policy.c | 360 ++++++++++++++++++-------------- opensm/opensm/osm_sa.c | 65 ++++-- opensm/opensm/osm_sa_mcmember_record.c | 161 ++++++++------- opensm/opensm/osm_sa_path_record.c | 32 +++- opensm/opensm/osm_sm.c | 92 +++++++-- opensm/opensm/osm_subnet.c | 12 +- 11 files changed, 474 insertions(+), 294 deletions(-) diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index 9f0cd96..567a989 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -331,12 +331,15 @@ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid); * * SYNOPSIS */ -void osm_mgrp_delete(IN osm_mgrp_t * const p_mgrp); +void osm_mgrp_delete_group(IN osm_subn_t * p_subn, IN osm_mgrp_t * const p_mgrp); /* * PARAMETERS * p_mgrp * [in] Pointer to an osm_mgrp_t object. * +* p_subn +* [in] Pointer to an osm_subn_t object. +* * RETURN VALUES * None. * diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h index cc8321d..26530f3 100644 --- a/opensm/include/opensm/osm_sm.h +++ b/opensm/include/opensm/osm_sm.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * @@ -61,6 +61,7 @@ #include #include #include +#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -101,7 +102,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct osm_sm { + typedef struct osm_sm { osm_thread_state_t thread_state; unsigned signal_mask; cl_spinlock_t signal_lock; @@ -539,7 +540,7 @@ osm_resp_send(IN osm_sm_t * sm, ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid); + IN const ib_net64_t port_guid, IN const ib_gid_t * p_mgid); /* * PARAMETERS * p_sm @@ -551,6 +552,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, * port_guid * [in] Port GUID to add to the group. * +* p_mgid +* [in] MGID to add to the group holder. * RETURN VALUES * None * @@ -572,7 +575,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, */ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, - IN const ib_net16_t mlid, IN const ib_net64_t port_guid); + IN osm_mgrp_t * p_mgrp, IN ib_net64_t port_guid); /* * PARAMETERS * p_sm @@ -689,7 +692,7 @@ osm_sm_is_greater_than(IN const uint8_t l_priority, * * SYNOPSIS */ -ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t *sm, +ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t * sm, IN osm_sm_signal_t signal); /* * PARAMETERS @@ -718,7 +721,7 @@ ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t *sm, * * SYNOPSIS */ -void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t *sm); +void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t * sm); /* * PARAMETERS * sm @@ -743,7 +746,7 @@ void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t *sm); * * SYNOPSIS */ -ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t *sm, +ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t * sm, IN osm_sm_signal_t signal); /* * PARAMETERS @@ -762,7 +765,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t *sm, * State Manager *********/ -void osm_report_sm_state(osm_sm_t *sm); +void osm_report_sm_state(osm_sm_t * sm); /****f* OpenSM: SM State Manager/osm_send_trap144 * NAME @@ -773,7 +776,7 @@ void osm_report_sm_state(osm_sm_t *sm); * * SYNOPSIS */ -int osm_send_trap144(osm_sm_t *sm, ib_net16_t local); +int osm_send_trap144(osm_sm_t * sm, ib_net16_t local); /* * PARAMETERS * sm @@ -787,7 +790,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local); * *********/ -void osm_set_sm_priority(osm_sm_t *sm, uint8_t priority); +void osm_set_sm_priority(osm_sm_t * sm, uint8_t priority); END_C_DECLS #endif /* _OSM_SM_H_ */ diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index c9a4f33..d92fdfc 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -158,7 +158,6 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) osm_port_t *p_port_check; cl_qmap_t *p_sm_guid_tbl; osm_mcm_info_t *p_mcm; - osm_mgrp_t *p_mgrp; cl_ptr_vector_t *p_port_lid_tbl; uint16_t min_lid_ho; uint16_t max_lid_ho; @@ -168,6 +167,7 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) ib_gid_t port_gid; ib_mad_notice_attr_t notice; ib_api_status_t status; + osm_mgrp_holder_t *p_osm_mgrp_holder; OSM_LOG_ENTER(sm->p_log); @@ -212,10 +212,12 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { - p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, p_mcm->mlid); - if (p_mgrp) { - osm_mgrp_delete_port(sm->p_subn, sm->p_log, - p_mgrp, p_port->guid); + p_osm_mgrp_holder = + osm_get_mgrp_holder_by_mlid(sm->p_subn, p_mcm->mlid); + if (p_osm_mgrp_holder) { + osm_mgrp_holder_remove_port(sm->p_subn, sm->p_log, + p_osm_mgrp_holder, + p_port->guid); osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); } p_mcm = diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index de0a8a5..e16c2e7 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index d2a19ea..e2e588c 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -41,7 +41,7 @@ #if HAVE_CONFIG_H # include #endif /* HAVE_CONFIG_H */ - +#include #include #include #include diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index 094fef2..07b05f5 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -59,11 +59,10 @@ extern osm_qos_level_t __default_simple_qos_level; /*************************************************** ***************************************************/ -static void -__build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) +static void __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) { - osm_node_t * p_node; - cl_qmap_t * p_node_guid_tbl = &p_qos_policy->p_subn->node_guid_tbl; + osm_node_t *p_node; + cl_qmap_t *p_node_guid_tbl = &p_qos_policy->p_subn->node_guid_tbl; p_qos_policy->p_node_hash = st_init_strtable(); CL_ASSERT(p_qos_policy->p_node_hash); @@ -75,10 +74,10 @@ __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { if (!st_lookup(p_qos_policy->p_node_hash, - (st_data_t)p_node->print_desc, NULL)) + (st_data_t) p_node->print_desc, NULL)) st_insert(p_qos_policy->p_node_hash, - (st_data_t)p_node->print_desc, - (st_data_t)p_node); + (st_data_t) p_node->print_desc, + (st_data_t) p_node); } } @@ -87,7 +86,7 @@ __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) static boolean_t __is_num_in_range_arr(uint64_t ** range_arr, - unsigned range_arr_len, uint64_t num) + unsigned range_arr_len, uint64_t num) { unsigned ind_1 = 0; unsigned ind_2 = range_arr_len - 1; @@ -97,22 +96,23 @@ __is_num_in_range_arr(uint64_t ** range_arr, return FALSE; while (ind_1 <= ind_2) { - if (num < range_arr[ind_1][0] || num > range_arr[ind_2][1]) - return FALSE; - else if (num <= range_arr[ind_1][1] || num >= range_arr[ind_2][0]) - return TRUE; - - ind_mid = ind_1 + (ind_2 - ind_1 + 1)/2; - - if (num < range_arr[ind_mid][0]) - ind_2 = ind_mid; - else if (num > range_arr[ind_mid][1]) - ind_1 = ind_mid; - else - return TRUE; - - ind_1++; - ind_2--; + if (num < range_arr[ind_1][0] || num > range_arr[ind_2][1]) + return FALSE; + else if (num <= range_arr[ind_1][1] + || num >= range_arr[ind_2][0]) + return TRUE; + + ind_mid = ind_1 + (ind_2 - ind_1 + 1) / 2; + + if (num < range_arr[ind_mid][0]) + ind_2 = ind_mid; + else if (num > range_arr[ind_mid][1]) + ind_1 = ind_mid; + else + return TRUE; + + ind_1++; + ind_2--; } return FALSE; @@ -130,10 +130,9 @@ static void __free_single_element(void *p_element, void *context) /*************************************************** ***************************************************/ -osm_qos_port_t *osm_qos_policy_port_create(osm_physp_t *p_physp) +osm_qos_port_t *osm_qos_policy_port_create(osm_physp_t * p_physp) { - osm_qos_port_t *p = - (osm_qos_port_t *) malloc(sizeof(osm_qos_port_t)); + osm_qos_port_t *p = (osm_qos_port_t *) malloc(sizeof(osm_qos_port_t)); if (!p) return NULL; memset(p, 0, sizeof(osm_qos_port_t)); @@ -163,8 +162,8 @@ osm_qos_port_group_t *osm_qos_policy_port_group_create() void osm_qos_policy_port_group_destroy(osm_qos_port_group_t * p) { - osm_qos_port_t * p_port; - osm_qos_port_t * p_old_port; + osm_qos_port_t *p_port; + osm_qos_port_t *p_old_port; if (!p) return; @@ -175,8 +174,7 @@ void osm_qos_policy_port_group_destroy(osm_qos_port_group_t * p) free(p->use); p_port = (osm_qos_port_t *) cl_qmap_head(&p->port_map); - while (p_port != (osm_qos_port_t *) cl_qmap_end(&p->port_map)) - { + while (p_port != (osm_qos_port_t *) cl_qmap_end(&p->port_map)) { p_old_port = p_port; p_port = (osm_qos_port_t *) cl_qmap_next(&p_port->map_item); free(p_old_port); @@ -424,9 +422,10 @@ void osm_qos_policy_match_rule_destroy(osm_qos_match_rule_t * p) /*************************************************** ***************************************************/ -osm_qos_policy_t * osm_qos_policy_create(osm_subn_t * p_subn) +osm_qos_policy_t *osm_qos_policy_create(osm_subn_t * p_subn) { - osm_qos_policy_t * p_qos_policy = (osm_qos_policy_t *)malloc(sizeof(osm_qos_policy_t)); + osm_qos_policy_t *p_qos_policy = + (osm_qos_policy_t *) malloc(sizeof(osm_qos_policy_t)); if (!p_qos_policy) return NULL; @@ -544,8 +543,8 @@ __qos_policy_is_port_in_group(osm_subn_t * p_subn, /* check whether this port's type matches any of group's types */ - if ( p_port_group->node_types & - (((uint8_t)1)<node_types & + (((uint8_t) 1) << osm_node_get_type(p_node))) return TRUE; /* check whether this port's guid is in group's port map */ @@ -585,24 +584,33 @@ __qos_policy_is_port_in_group_list(const osm_qos_policy_t * p_qos_policy, /*************************************************** ***************************************************/ -static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( - const osm_qos_policy_t * p_qos_policy, - uint64_t service_id, - uint16_t qos_class, - uint16_t pkey, - const osm_physp_t * p_src_physp, - const osm_physp_t * p_dest_physp, - ib_net64_t comp_mask) +static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params(const + osm_qos_policy_t + * + p_qos_policy, + uint64_t + service_id, + uint16_t + qos_class, + uint16_t + pkey, + const + osm_physp_t * + p_src_physp, + const + osm_physp_t * + p_dest_physp, + ib_net64_t + comp_mask) { osm_qos_match_rule_t *p_qos_match_rule = NULL; cl_list_iterator_t list_iterator; - osm_log_t * p_log = &p_qos_policy->p_subn->p_osm->log; + osm_log_t *p_log = &p_qos_policy->p_subn->p_osm->log; boolean_t matched_by_sguid = FALSE, - matched_by_dguid = FALSE, - matched_by_class = FALSE, - matched_by_sid = FALSE, - matched_by_pkey = FALSE; + matched_by_dguid = FALSE, + matched_by_class = FALSE, + matched_by_sid = FALSE, matched_by_pkey = FALSE; if (!cl_list_count(&p_qos_policy->qos_match_rules)) return NULL; @@ -698,8 +706,7 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( if (!__is_num_in_range_arr (p_qos_match_rule->pkey_range_arr, - p_qos_match_rule->pkey_range_len, - pkey & 0x7FFF)) { + p_qos_match_rule->pkey_range_len, pkey & 0x7FFF)) { list_iterator = cl_list_next(list_iterator); continue; } @@ -717,15 +724,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( OSM_LOG(p_log, OSM_LOG_DEBUG, "request matched rule (%s) by:%s%s%s%s%s\n", (p_qos_match_rule->use) ? - p_qos_match_rule->use : "no description", + p_qos_match_rule->use : "no description", (matched_by_sguid) ? " SGUID" : "", (matched_by_dguid) ? " DGUID" : "", (matched_by_class) ? " QoS_Class" : "", - (matched_by_sid) ? " ServiceID" : "", - (matched_by_pkey) ? " PKey" : ""); + (matched_by_sid) ? " ServiceID" : "", + (matched_by_pkey) ? " PKey" : ""); else - OSM_LOG(p_log, OSM_LOG_DEBUG, - "request not matched any rule\n"); + OSM_LOG(p_log, OSM_LOG_DEBUG, "request not matched any rule\n"); OSM_LOG_EXIT(p_log); return p_qos_match_rule; @@ -734,9 +740,10 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( /*************************************************** ***************************************************/ -static osm_qos_level_t *__qos_policy_get_qos_level_by_name( - const osm_qos_policy_t * p_qos_policy, - char *name) +static osm_qos_level_t *__qos_policy_get_qos_level_by_name(const + osm_qos_policy_t * + p_qos_policy, + char *name) { osm_qos_level_t *p_qos_level = NULL; cl_list_iterator_t list_iterator; @@ -760,9 +767,11 @@ static osm_qos_level_t *__qos_policy_get_qos_level_by_name( /*************************************************** ***************************************************/ -static osm_qos_port_group_t *__qos_policy_get_port_group_by_name( - const osm_qos_policy_t * p_qos_policy, - const char *const name) +static osm_qos_port_group_t *__qos_policy_get_port_group_by_name(const + osm_qos_policy_t + * p_qos_policy, + const char + *const name) { osm_qos_port_group_t *p_port_group = NULL; cl_list_iterator_t list_iterator; @@ -787,15 +796,16 @@ static osm_qos_port_group_t *__qos_policy_get_port_group_by_name( /*************************************************** ***************************************************/ -static void __qos_policy_validate_pkey( - osm_qos_policy_t * p_qos_policy, - osm_qos_match_rule_t * p_qos_match_rule, - osm_prtn_t * p_prtn) +static void __qos_policy_validate_pkey(osm_qos_policy_t * p_qos_policy, + osm_qos_match_rule_t * p_qos_match_rule, + osm_prtn_t * p_prtn) { uint8_t sl; uint32_t flow; uint8_t hop; - osm_mgrp_t * p_mgrp; + osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; + cl_fmap_item_t *p_fitem; if (!p_qos_policy || !p_qos_match_rule || !p_prtn) return; @@ -812,43 +822,49 @@ static void __qos_policy_validate_pkey( p_qos_match_rule->p_qos_level->sl); p_prtn->sl = p_qos_match_rule->p_qos_level->sl; - /* If this partition is an IPoIB partition, there should be a matching MCast group. Fix this group's SL too */ if (!p_prtn->mlid) return; - p_mgrp = osm_get_mgrp_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); - if (!p_mgrp) { + p_mgrp_holder = + osm_get_mgrp_holder_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); + if (!p_mgrp_holder) { OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, - "ERR AC16: MCast group for partition with " - "pkey 0x%04X not found\n", - cl_ntoh16(p_prtn->pkey)); + "ERR AC16: MCast mgrp_holder for partition with " + "pkey 0x%04X not found\n", cl_ntoh16(p_prtn->pkey)); return; } - - CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == - (cl_ntoh16(p_prtn->pkey) & 0x7fff)); - - ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, - &sl, &flow, &hop); - if (sl != p_prtn->sl) { - OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, - "Updating MCGroup (MLID 0x%04x) SL to " - "match partition SL (%u)\n", - cl_hton16(p_mgrp->mcmember_rec.mlid), - p_prtn->sl); - p_mgrp->mcmember_rec.sl_flow_hop = - ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + p_fitem = cl_fmap_next(p_fitem); + + CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == + (cl_ntoh16(p_prtn->pkey) & 0x7fff)); + + ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, + &sl, &flow, &hop); + if (sl != p_prtn->sl) { + OSM_LOG(&p_qos_policy->p_subn->p_osm->log, + OSM_LOG_DEBUG, + "Updating MCGroup (MLID 0x%04x) SL to " + "match partition SL (%u)\n", + cl_hton16(p_mgrp->mcmember_rec.mlid), + p_prtn->sl); + p_mgrp->mcmember_rec.sl_flow_hop = + ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); + } } } /*************************************************** ***************************************************/ -int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, - osm_log_t *p_log) +int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, osm_log_t * p_log) { cl_list_iterator_t match_rules_list_iterator; cl_list_iterator_t list_iterator; @@ -859,22 +875,23 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, int res = 0; uint64_t pkey_64; ib_net16_t pkey; - osm_prtn_t * p_prtn; + osm_prtn_t *p_prtn; OSM_LOG_ENTER(p_log); /* set default qos level */ p_qos_policy->p_default_qos_level = - __qos_policy_get_qos_level_by_name(p_qos_policy, OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); + __qos_policy_get_qos_level_by_name(p_qos_policy, + OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); if (!p_qos_policy->p_default_qos_level) { /* There's no default QoS level in the usual qos-level section. Check whether the 'simple' default QoS level that can be defined in the qos-ulp section exists */ if (__default_simple_qos_level.sl_set) { - p_qos_policy->p_default_qos_level = &__default_simple_qos_level; - } - else { + p_qos_policy->p_default_qos_level = + &__default_simple_qos_level; + } else { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC10: " "Default qos-level (%s) not defined.\n", OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); @@ -891,8 +908,7 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, cl_list_head(&p_qos_policy->qos_match_rules); while (match_rules_list_iterator != cl_list_end(&p_qos_policy->qos_match_rules)) { - p_qos_match_rule = - (osm_qos_match_rule_t *) + p_qos_match_rule = (osm_qos_match_rule_t *) cl_list_obj(match_rules_list_iterator); CL_ASSERT(p_qos_match_rule); @@ -900,8 +916,9 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, if (!p_qos_match_rule->p_qos_level) p_qos_match_rule->p_qos_level = - __qos_policy_get_qos_level_by_name(p_qos_policy, - p_qos_match_rule->qos_level_name); + __qos_policy_get_qos_level_by_name(p_qos_policy, + p_qos_match_rule-> + qos_level_name); if (!p_qos_match_rule->p_qos_level) { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC11: " @@ -922,9 +939,11 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, CL_ASSERT(str); p_port_group = - __qos_policy_get_port_group_by_name(p_qos_policy, str); + __qos_policy_get_port_group_by_name + (p_qos_policy, str); if (!p_port_group) { - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC12: " + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR AC12: " "qos-match-rule num %u: source port-group '%s' not found\n", i, str); res = 1; @@ -951,9 +970,11 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, CL_ASSERT(str); p_port_group = - __qos_policy_get_port_group_by_name(p_qos_policy,str); + __qos_policy_get_port_group_by_name + (p_qos_policy, str); if (!p_port_group) { - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC13: " + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR AC13: " "qos-match-rule num %u: destination port-group '%s' not found\n", i, str); res = 1; @@ -977,24 +998,30 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, */ for (j = 0; j < p_qos_match_rule->pkey_range_len; j++) { - for ( pkey_64 = p_qos_match_rule->pkey_range_arr[j][0]; - pkey_64 <= p_qos_match_rule->pkey_range_arr[j][1]; - pkey_64++) { - pkey = cl_hton16((uint16_t)(pkey_64 & 0x7fff)); - p_prtn = (osm_prtn_t *)cl_qmap_get( - &p_qos_policy->p_subn->prtn_pkey_tbl, pkey); - - if (p_prtn == (osm_prtn_t *)cl_qmap_end( - &p_qos_policy->p_subn->prtn_pkey_tbl)) + for (pkey_64 = p_qos_match_rule->pkey_range_arr[j][0]; + pkey_64 <= p_qos_match_rule->pkey_range_arr[j][1]; + pkey_64++) { + pkey = cl_hton16((uint16_t) (pkey_64 & 0x7fff)); + p_prtn = + (osm_prtn_t *) cl_qmap_get(&p_qos_policy-> + p_subn-> + prtn_pkey_tbl, + pkey); + + if (p_prtn == + (osm_prtn_t *) cl_qmap_end(&p_qos_policy-> + p_subn-> + prtn_pkey_tbl)) /* partition for this pkey not found */ - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC14: " + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR AC14: " "pkey 0x%04X in match rule - " "partition doesn't exist\n", cl_ntoh16(pkey)); else __qos_policy_validate_pkey(p_qos_policy, - p_qos_match_rule, - p_prtn); + p_qos_match_rule, + p_prtn); } } @@ -1013,53 +1040,70 @@ Exit: /*************************************************** ***************************************************/ -static osm_qos_level_t * __qos_policy_get_qos_level_by_params( - IN const osm_qos_policy_t * p_qos_policy, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN uint64_t service_id, - IN uint16_t qos_class, - IN uint16_t pkey, - IN ib_net64_t comp_mask) +static osm_qos_level_t *__qos_policy_get_qos_level_by_params(IN const + osm_qos_policy_t * + p_qos_policy, + IN const + osm_physp_t * + p_src_physp, + IN const + osm_physp_t * + p_dest_physp, + IN uint64_t + service_id, + IN uint16_t + qos_class, + IN uint16_t pkey, + IN ib_net64_t + comp_mask) { osm_qos_match_rule_t *p_qos_match_rule = NULL; if (!p_qos_policy) return NULL; - p_qos_match_rule = __qos_policy_get_match_rule_by_params( - p_qos_policy, service_id, qos_class, pkey, - p_src_physp, p_dest_physp, comp_mask); + p_qos_match_rule = + __qos_policy_get_match_rule_by_params(p_qos_policy, service_id, + qos_class, pkey, p_src_physp, + p_dest_physp, comp_mask); return p_qos_match_rule ? p_qos_match_rule->p_qos_level : - p_qos_policy->p_default_qos_level; + p_qos_policy->p_default_qos_level; } /* __qos_policy_get_qos_level_by_params() */ /*************************************************** ***************************************************/ -osm_qos_level_t * osm_qos_policy_get_qos_level_by_pr( - IN const osm_qos_policy_t * p_qos_policy, - IN const ib_path_rec_t * p_pr, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN ib_net64_t comp_mask) +osm_qos_level_t *osm_qos_policy_get_qos_level_by_pr(IN const osm_qos_policy_t * + p_qos_policy, + IN const ib_path_rec_t * + p_pr, + IN const osm_physp_t * + p_src_physp, + IN const osm_physp_t * + p_dest_physp, + IN ib_net64_t comp_mask) { - return __qos_policy_get_qos_level_by_params( - p_qos_policy, p_src_physp, p_dest_physp, - cl_ntoh64(p_pr->service_id), ib_path_rec_qos_class(p_pr), - cl_ntoh16(p_pr->pkey), comp_mask); + return __qos_policy_get_qos_level_by_params(p_qos_policy, p_src_physp, + p_dest_physp, + cl_ntoh64(p_pr->service_id), + ib_path_rec_qos_class(p_pr), + cl_ntoh16(p_pr->pkey), + comp_mask); } /*************************************************** ***************************************************/ -osm_qos_level_t * osm_qos_policy_get_qos_level_by_mpr( - IN const osm_qos_policy_t * p_qos_policy, - IN const ib_multipath_rec_t * p_mpr, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN ib_net64_t comp_mask) +osm_qos_level_t *osm_qos_policy_get_qos_level_by_mpr(IN const osm_qos_policy_t * + p_qos_policy, + IN const ib_multipath_rec_t + * p_mpr, + IN const osm_physp_t * + p_src_physp, + IN const osm_physp_t * + p_dest_physp, + IN ib_net64_t comp_mask) { ib_net64_t pr_comp_mask = 0; @@ -1071,20 +1115,24 @@ osm_qos_level_t * osm_qos_policy_get_qos_level_by_mpr( * compmask. Note that only relevant bits are set. */ pr_comp_mask = - ((comp_mask & IB_MPR_COMPMASK_QOS_CLASS) ? - IB_PR_COMPMASK_QOS_CLASS : 0) | - ((comp_mask & IB_MPR_COMPMASK_PKEY) ? - IB_PR_COMPMASK_PKEY : 0) | - ((comp_mask & IB_MPR_COMPMASK_SERVICEID_MSB) ? - IB_PR_COMPMASK_SERVICEID_MSB : 0) | - ((comp_mask & IB_MPR_COMPMASK_SERVICEID_LSB) ? - IB_PR_COMPMASK_SERVICEID_LSB : 0); - - return __qos_policy_get_qos_level_by_params( - p_qos_policy, p_src_physp, p_dest_physp, - cl_ntoh64(ib_multipath_rec_service_id(p_mpr)), - ib_multipath_rec_qos_class(p_mpr), - cl_ntoh16(p_mpr->pkey), pr_comp_mask); + ((comp_mask & IB_MPR_COMPMASK_QOS_CLASS) ? + IB_PR_COMPMASK_QOS_CLASS : 0) | + ((comp_mask & IB_MPR_COMPMASK_PKEY) ? + IB_PR_COMPMASK_PKEY : 0) | + ((comp_mask & IB_MPR_COMPMASK_SERVICEID_MSB) ? + IB_PR_COMPMASK_SERVICEID_MSB : 0) | + ((comp_mask & IB_MPR_COMPMASK_SERVICEID_LSB) ? + IB_PR_COMPMASK_SERVICEID_LSB : 0); + + return __qos_policy_get_qos_level_by_params(p_qos_policy, p_src_physp, + p_dest_physp, + cl_ntoh64 + (ib_multipath_rec_service_id + (p_mpr)), + ib_multipath_rec_qos_class + (p_mpr), + cl_ntoh16(p_mpr->pkey), + pr_comp_mask); } /*************************************************** diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index fcc3f27..0c24a49 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -317,7 +318,7 @@ Exit: return (status); } -ib_api_status_t osm_sa_send(osm_sa_t *sa, +ib_api_status_t osm_sa_send(osm_sa_t * sa, IN osm_madw_t * const p_madw, IN boolean_t const resp_expected) { @@ -397,8 +398,8 @@ Exit: OSM_LOG_EXIT(sa->p_log); } -void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, - cl_qlist_t *list) +void osm_sa_respond(osm_sa_t * sa, osm_madw_t * madw, size_t attr_size, + cl_qlist_t * list) { struct item_data { cl_list_item_t list; @@ -422,13 +423,13 @@ void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, */ if (sa_mad->method == IB_MAD_METHOD_GET && num_rec > 1) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4C05: " - "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" PRIx64 "\n", - num_rec, ib_get_sa_attr_str(sa_mad->attr_id), + "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" + PRIx64 "\n", num_rec, + ib_get_sa_attr_str(sa_mad->attr_id), cl_ntoh64(sa_mad->comp_mask)); osm_sa_send_error(sa, madw, IB_SA_MAD_STATUS_TOO_MANY_RECORDS); goto Exit; } - #ifndef VENDOR_RMPP_SUPPORT trim_num_rec = (MAD_BLOCK_SIZE - IB_SA_MAD_HDR_SIZE) / attr_size; if (trim_num_rec < num_rec) { @@ -564,7 +565,7 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) p_mcm_port->scope_state, p_mcm_port->proxy_join); } -static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) +static void sa_dump_one_mgrp(osm_mgrp_t * p_mgrp, void *cxt) { struct opensm_dump_context dump_context; osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; @@ -706,6 +707,8 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) { struct opensm_dump_context dump_context; osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; int i; dump_context.p_osm = p_osm; @@ -714,9 +717,17 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) cl_plock_acquire(&p_osm->lock); for (i = 0; i <= p_osm->subn.max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = p_osm->subn.mgroups[i]; - if (p_mgrp) + p_mgrp_holder = p_osm->subn.mgroup_holders[i]; + if (!p_mgrp_holder) + continue; + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); sa_dump_one_mgrp(p_mgrp, &dump_context); + p_fitem = cl_fmap_next(&p_mgrp->mgid_item); + } } OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform\n"); cl_qlist_apply_func(&p_osm->subn.sa_infr_list, @@ -740,25 +751,35 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, unsigned well_known) { ib_net64_t comp_mask; - osm_mgrp_t *p_mgrp; - + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; + ib_gid_t common_mgid; + osm_mgrp_t *p_mgrp = NULL; cl_plock_excl_acquire(&p_osm->lock); - p_mgrp = osm_get_mgrp_by_mlid(&p_osm->subn, mlid); - if (p_mgrp) { - if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, - sizeof(ib_gid_t))) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(&p_osm->subn, mlid); + if (p_mgrp_holder) { + p_fitem = + cl_fmap_get(&p_mgrp_holder->mgrp_map, &p_mcm_rec->mgid); + if (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "mgrp %04x is already here.", cl_ntoh16(mlid)); goto _out; } - OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, - "mlid %04x is already used by another MC group. Will " - "request clients reregistration.\n", cl_ntoh16(mlid)); - p_mgrp = NULL; - goto _out; - } + osm_mgrp_holder_prepare_common_mgid(&p_mcm_rec->mgid, + &common_mgid); + if (memcmp + (&p_mgrp_holder->common_mgid, &common_mgid, + sizeof(ib_gid_t))) { + OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, + "mlid %04x is already used by another MC group. Will " + "request clients reregistration.\n", + cl_ntoh16(mlid)); + p_mgrp = NULL; + goto _out; + } + } comp_mask = IB_MCR_COMPMASK_MTU | IB_MCR_COMPMASK_MTU_SEL | IB_MCR_COMPMASK_RATE | IB_MCR_COMPMASK_RATE_SEL; if (osm_mcmr_rcv_find_or_create_new_mgrp(&p_osm->sa, diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 5543221..23c5107 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -63,6 +63,7 @@ #include #include #include +#include #define JOIN_MC_COMP_MASK (IB_MCR_COMPMASK_MGID | \ IB_MCR_COMPMASK_PORT_GID | \ @@ -121,13 +122,14 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_mcast_lid_ho - && !osm_get_mgrp_by_mlid(p_subn, requested_mlid)) + && !osm_get_mgrp_holder_by_mlid(p_subn, requested_mlid)) return requested_mlid; max = p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO + 1; for (i = 0; i < max; i++) { - osm_mgrp_t *p_mgrp = sa->p_subn->mgroups[i]; - if (!p_mgrp || p_mgrp->to_be_deleted) + osm_mgrp_holder_t *p_mgrp_holder = + (osm_mgrp_holder_t *) sa->p_subn->mgroup_holders[i]; + if (!p_mgrp_holder || p_mgrp_holder->to_be_deleted) return cl_hton16(i + IB_LID_MCAST_START_HO); } @@ -141,14 +143,21 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) we silently drop it. Since it was an intermediate group no need to re-route it. **********************************************************************/ -static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * mgrp) +static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * p_mgrp) { /* Remove MGRP only if osm_mcm_port_t count is 0 and not a well known group */ - if (cl_is_qmap_empty(&mgrp->mcm_port_tbl) && !mgrp->well_known) { - sa->p_subn->mgroups[cl_ntoh16(mgrp->mlid) - - IB_LID_MCAST_START_HO] = NULL; - osm_mgrp_delete(mgrp); + char gid_str[INET6_ADDRSTRLEN]; + if (cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && !p_mgrp->well_known) { + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "mgrp mgid %s will be deleted\n", inet_ntop(AF_INET6, + p_mgrp-> + mcmember_rec. + mgid.raw, + gid_str, + sizeof + (gid_str))); + osm_mgrp_delete_group(sa->sm->p_subn, p_mgrp); } } @@ -202,7 +211,6 @@ static ib_api_status_t add_new_mgrp_port(osm_sa_t * sa, IN osm_mgrp_t * p_mgrp, return IB_INSUFFICIENT_MEMORY; } - return IB_SUCCESS; } @@ -806,11 +814,12 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, IN const osm_physp_t * p_physp, OUT osm_mgrp_t ** pp_mgrp) { - ib_net16_t mlid; + ib_net16_t mlid, existed_mlid; unsigned zero_mgid, i; uint8_t scope; + char gid_str[INET6_ADDRSTRLEN]; ib_gid_t *p_mgid; - osm_mgrp_t *p_prev_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; ib_api_status_t status = IB_SUCCESS; ib_member_rec_t mcm_rec = *p_recvd_mcmember_rec; /* copy for modifications */ @@ -823,7 +832,7 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, zero_mgid = 0; break; } - + p_mgid = &(mcm_rec.mgid); /* we allocate a new mlid number before we might use it for MGID ... @@ -843,8 +852,6 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* we need to create the new MGID if it was not defined */ if (zero_mgid) { /* create a new MGID */ - char gid_str[INET6_ADDRSTRLEN]; - /* use the given scope state only if requested! */ if (comp_mask & IB_MCR_COMPMASK_SCOPE) ib_member_get_scope_state(p_recvd_mcmember_rec-> @@ -888,6 +895,13 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, goto Exit; } + if (0 != (existed_mlid = osm_mgrp_holder_get_mlid_by_mgid(sa, p_mgid))) { + mlid = existed_mlid; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "found existed mlid 0x%04x for mgid %s\n", + cl_ntoh16(mlid), inet_ntop(AF_INET6, p_mgid->raw, + gid_str, sizeof gid_str)); + } /* create a new MC Group */ *pp_mgrp = osm_mgrp_new(mlid); if (*pp_mgrp == NULL) { @@ -915,17 +929,21 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* since we might have an old group by that mlid one whose deletion was delayed for an idle time we need to deallocate it first */ - p_prev_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); - if (p_prev_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sa->p_subn, mlid); + if (!p_mgrp_holder) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Found previous group for mlid:0x%04x - " - "Destroying it first\n", cl_ntoh16(mlid)); - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = - NULL; - osm_mgrp_delete(p_prev_mgrp); + "Creating new mgrp holder for mlid:0x%04x\n", + cl_ntoh16(mlid)); + p_mgrp_holder = osm_mgrp_holder_new(sa->p_subn, p_mgid, mlid); } - - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; + if (!p_mgrp_holder) { + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B08: " + "osm_mgrp_holder_new failed\n"); + free_mlid(sa, mlid); + status = IB_INSUFFICIENT_MEMORY; + goto Exit; + } + osm_mgrp_holder_add_mgrp(p_mgrp_holder, *pp_mgrp, sa->p_log); Exit: OSM_LOG_EXIT(sa->p_log); @@ -934,56 +952,47 @@ Exit: /********************************************************************** *********************************************************************/ -static unsigned match_mgrp_by_mgid(IN osm_mgrp_t * p_mgrp, ib_gid_t * mgid) -{ - /* ignore groups marked for deletion */ - if (p_mgrp->to_be_deleted || - memcmp(&p_mgrp->mcmember_rec.mgid, mgid, sizeof(ib_gid_t))) - return 0; - else - return 1; -} - -/********************************************************************** - **********************************************************************/ -#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) -#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) -#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) -#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) - -/* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ -/* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ -/* Where Z is the scope, XXXX is the P_Key, and - * YYYYYY is the last 24 bits of the port guid */ -static unsigned match_and_update_ipv6_snm_mgid(ib_gid_t * mgid) +static osm_mgrp_t *match_mgrp_by_mgid(IN osm_mgrp_holder_t * p_mgrp_holder, + ib_gid_t * p_mgid) { - if ((mgid->unicast.prefix & PREFIX_MASK) == PREFIX_SIGNATURE && - (mgid->unicast.interface_id & INT_ID_MASK) == INT_ID_SIGNATURE) { - mgid->unicast.prefix &= PREFIX_MASK; - mgid->unicast.interface_id &= INT_ID_MASK; - return 1; - } - return 0; + osm_mgrp_t *p_mgrp; + ib_gid_t common_mgid; + cl_fmap_item_t *p_fitem; + + osm_mgrp_holder_prepare_common_mgid(p_mgid, &common_mgid); + + if (memcmp(&p_mgrp_holder->common_mgid, &common_mgid, sizeof(ib_gid_t))) + return NULL; + p_fitem = cl_fmap_get(&p_mgrp_holder->mgrp_map, p_mgid); + if (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + /* ignore groups marked for deletion */ + if (p_mgrp->to_be_deleted) + p_mgrp = NULL; + } else + p_mgrp = NULL; + return p_mgrp;; } osm_mgrp_t *osm_get_mgrp_by_mgid(IN osm_sa_t * sa, IN ib_gid_t * p_mgid) { int i; - - if (sa->p_subn->opt.consolidate_ipv6_snm_req && - match_and_update_ipv6_snm_mgid(p_mgid)) { - char gid_str[INET6_ADDRSTRLEN]; - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Special Case Solicited Node Mcast Join for MGID %s\n", - inet_ntop(AF_INET6, p_mgid->raw, gid_str, - sizeof gid_str)); - } - + osm_mgrp_t *p_mgrp; for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; - i++) - if (sa->p_subn->mgroups[i] && - match_mgrp_by_mgid(sa->p_subn->mgroups[i], p_mgid)) - return sa->p_subn->mgroups[i]; + i++) { + if (sa->p_subn->mgroup_holders[i] && + (p_mgrp = + match_mgrp_by_mgid(sa->p_subn->mgroup_holders[i], + p_mgid))) { + char gid_str[INET6_ADDRSTRLEN]; + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "Found mgroup for MGID %s\n", + inet_ntop(AF_INET6, p_mgid->raw, gid_str, + sizeof gid_str)); + return p_mgrp; + } + } return NULL; } @@ -1080,7 +1089,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) CL_PLOCK_RELEASE(sa->p_lock); /* we can leave if port was deleted from MCG */ - if (removed && osm_sm_mcgrp_leave(sa->sm, mlid, portguid)) + if (removed && osm_sm_mcgrp_leave(sa->sm, p_mgrp, portguid)) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B09: " "osm_sm_mcgrp_leave failed\n"); @@ -1310,7 +1319,7 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* do the actual routing (actually schedule the update) */ status = osm_sm_mcgrp_join(sa->sm, mlid, p_recvd_mcmember_rec->port_gid.unicast. - interface_id); + interface_id, &p_recvd_mcmember_rec->mgid); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " @@ -1555,6 +1564,8 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) osm_physp_t *p_req_physp; boolean_t trusted_req; osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; int i; OSM_LOG_ENTER(sa->p_log); @@ -1586,10 +1597,18 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) /* simply go over all MCGs and match */ for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = sa->p_subn->mgroups[i]; - if (p_mgrp) + p_mgrp_holder = sa->p_subn->mgroup_holders[i]; + if (!p_mgrp_holder) + continue; + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); mcmr_by_comp_mask(sa, p_rcvd_rec, comp_mask, p_mgrp, p_req_physp, trusted_req, &rec_list); + p_fitem = cl_fmap_next(p_fitem); + } } CL_PLOCK_RELEASE(sa->p_lock); diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 9b50deb..3271289 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -1438,6 +1438,7 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; ib_net64_t comp_mask; + osm_mgrp_holder_t *p_mgrp_holder; osm_mgrp_t *mgrp = NULL; p_sa_mad = osm_madw_get_sa_mad_ptr(p_madw); @@ -1468,11 +1469,30 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) mgrp = NULL; goto Exit; } - } else - if (!(mgrp = osm_get_mgrp_by_mlid(sa->p_subn, p_pr->dlid))) - OSM_LOG(sa->p_log, OSM_LOG_ERROR, - "ERR 1F11: " "No MC group found for PathRecord " - "destination LID 0x%x\n", p_pr->dlid); + } else { + cl_fmap_item_t *p_fitem; + p_mgrp_holder = + osm_get_mgrp_holder_by_mlid(sa->p_subn, p_pr->dlid); + if (p_mgrp_holder) { + p_fitem = + cl_fmap_get(&p_mgrp_holder->mgrp_map, + &p_pr->dgid); + } + if (!p_mgrp_holder + || p_fitem == + cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + OSM_LOG(sa->p_log, OSM_LOG_ERROR, + "ERR 1F11: " + "No MC group found for PathRecord " + "destination LID 0x%x\n", p_pr->dlid); + } else { + mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, + osm_mgrp_t, + mgid_item); + } + + } } Exit: diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index daa60ff..459e2cc 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -47,6 +47,7 @@ #include #include +#include #include #include #include @@ -468,12 +469,15 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, /********************************************************************** **********************************************************************/ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, - IN const ib_net64_t port_guid) + IN const ib_net64_t port_guid, + IN const ib_gid_t * p_mgid) { - osm_mgrp_t *p_mgrp; + osm_mgrp_t *p_mgrp = NULL; osm_port_t *p_port; ib_api_status_t status = IB_SUCCESS; osm_mcm_info_t *p_mcm; + cl_fmap_item_t *p_fitem; + osm_mgrp_holder_t *p_mgrp_holder; OSM_LOG_ENTER(p_sm->p_log); @@ -498,8 +502,51 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, /* * If this multicast group does not already exist, create it. */ - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); - if (!p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); + if (p_mgrp_holder) { + char gid_str[INET6_ADDRSTRLEN]; + if (osm_is_debug()) { + size_t gr_count = + cl_fmap_count(&p_mgrp_holder->mgrp_map); + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "mlid 0x%X has %d mgroups\n", cl_ntoh16(mlid), + gr_count); + if (gr_count) { + p_fitem = + cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != + cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = + (osm_mgrp_t *) + PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "mlid 0x%X has mgrp with MGID: %s\n", + cl_ntoh16(mlid), + inet_ntop(AF_INET6, + p_mgrp->mcmember_rec. + mgid.raw, gid_str, + sizeof gid_str)); + p_fitem = cl_fmap_next(p_fitem); + } + } + } + p_fitem = cl_fmap_get(&p_mgrp_holder->mgrp_map, p_mgid); + if (p_fitem == cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = NULL; + OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, + "group with MGID: %s not found on mlid 0x%X\n", + inet_ntop(AF_INET6, + p_mgid->raw, + gid_str, sizeof gid_str), + cl_ntoh16(mlid)); + } else { + p_mgrp = + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + } + } + if (!p_mgrp_holder || !p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { /* * The group removed or the port is not a * member of the group, then fail immediately. @@ -514,7 +561,23 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, status = IB_NOT_FOUND; goto Exit; } - + /* if there was no change from the last time + * we processed the group we can skip doing anything + */ + if (p_mgrp_holder->last_change_id == p_mgrp_holder->last_tree_id) { + CL_PLOCK_RELEASE(p_sm->p_lock); + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, + "Skip processing mgrp holder with lid:0x%X last change id:%u\n", + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); + goto Exit; + } else { + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + "processing mgrp holder with lid:0x%X port: 0x%016" + PRIx64 " last change id:%u tree id:%u\n", + cl_ntoh16(mlid), cl_ntoh64(port_guid), + p_mgrp_holder->last_change_id, + p_mgrp_holder->last_tree_id); + } /* * Check if the object (according to mlid) already exists on this port. * If it does - then no need to update it again, and no need to @@ -543,9 +606,6 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } - status = sm_mgrp_process(p_sm, p_mgrp); - CL_PLOCK_RELEASE(p_sm->p_lock); - Exit: OSM_LOG_EXIT(p_sm->p_log); return status; @@ -553,12 +613,13 @@ Exit: /********************************************************************** **********************************************************************/ -ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, +ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN osm_mgrp_t * p_mgrp, IN const ib_net64_t port_guid) { - osm_mgrp_t *p_mgrp; osm_port_t *p_port; ib_api_status_t status; + osm_mgrp_holder_t *p_mgrp_holder; + ib_net16_t mlid = p_mgrp->mlid; OSM_LOG_ENTER(p_sm->p_log); @@ -584,8 +645,8 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, /* * Get the multicast group object for this group. */ - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); - if (!p_mgrp) { + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); + if (!p_mgrp_holder) { CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); @@ -593,11 +654,14 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, goto Exit; } + osm_mgrp_holder_delete_mgrp_port(p_mgrp_holder, p_mgrp, port_guid); /* * Walk the list of ports in the group, and remove the appropriate one. */ osm_port_remove_mgrp(p_port, mlid); - + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, + " Calling sm_mgrp_process for mgrp with mlid = 0x%X\n", + cl_ntoh16(mlid)); status = sm_mgrp_process(p_sm, p_mgrp); CL_PLOCK_RELEASE(p_sm->p_lock); diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index ec15f8a..e6e624d 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. @@ -419,7 +419,7 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) osm_switch_t *p_sw, *p_next_sw; osm_remote_sm_t *p_rsm, *p_next_rsm; osm_prtn_t *p_prtn, *p_next_prtn; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_osm_mgrp_holder; osm_infr_t *p_infr, *p_next_infr; /* it might be a good idea to de-allocate all known objects */ @@ -464,10 +464,10 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) for (i = 0; i <= p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { - p_mgrp = p_subn->mgroups[i]; - p_subn->mgroups[i] = NULL; - if (p_mgrp) - osm_mgrp_delete(p_mgrp); + p_osm_mgrp_holder = p_subn->mgroup_holders[i]; + if (p_osm_mgrp_holder){ + osm_mgrp_holder_delete(p_subn, p_osm_mgrp_holder->mlid); + } } p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); -- 1.5.5 From slavas at Voltaire.COM Sun Jun 28 04:00:03 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Sun, 28 Jun 2009 14:00:03 +0300 Subject: [ofa-general] [PATCH 4/4 v2] multicast multiplexing - building spanning tree and removes obsolete code. In-Reply-To: <4A3A387A.2020509@Voltaire.COM> References: <4A3A387A.2020509@Voltaire.COM> Message-ID: <4A474D33.9010502@Voltaire.COM> Subject: [PATCH 4/4] Patch implements multicast multiplexing as proposed in the thread entitled "IPv6 and IPoIB scalability issue". This part (4) implements infrastructure management for building spanning tree and removes obsolete parts of code. Signed-off-by: Slava Strebkov --- opensm/include/opensm/osm_multicast.h | 14 +- opensm/include/opensm/osm_subnet.h | 33 ----- opensm/opensm/osm_mcast_mgr.c | 247 ++++++++++++++++++++------------- opensm/opensm/osm_multicast.c | 47 ------ opensm/opensm/osm_sm.c | 5 +- 5 files changed, 156 insertions(+), 190 deletions(-) diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index 567a989..626c5af 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -206,13 +206,10 @@ typedef struct _osm_mgrp_port { typedef struct osm_mgrp { cl_map_item_t map_item; ib_net16_t mlid; - osm_mtree_node_t *p_root; cl_qmap_t mcm_port_tbl; ib_member_rec_t mcmember_rec; boolean_t well_known; boolean_t to_be_deleted; - uint32_t last_change_id; - uint32_t last_tree_id; unsigned full_members; cl_fmap_item_t mgid_item; cl_list_item_t mgrp_item; @@ -331,7 +328,8 @@ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid); * * SYNOPSIS */ -void osm_mgrp_delete_group(IN osm_subn_t * p_subn, IN osm_mgrp_t * const p_mgrp); +void osm_mgrp_delete_group(IN osm_subn_t * p_subn, + IN osm_mgrp_t * const p_mgrp); /* * PARAMETERS * p_mgrp @@ -841,10 +839,10 @@ void osm_mgrp_holder_prepare_common_mgid(IN const ib_gid_t * const p_mgid, * * SYNOPSIS */ -static inline - struct osm_mgrp_holder *osm_get_mgrp_holder_by_mlid(osm_subn_t const - *p_subn, - ib_net16_t mlid) +static inline struct osm_mgrp_holder *osm_get_mgrp_holder_by_mlid(osm_subn_t const + *p_subn, + ib_net16_t + mlid) { return p_subn->mgroup_holders[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; } diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 6713d4d..cafdd6b 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -508,7 +508,6 @@ typedef struct osm_subn { boolean_t first_time_master_sweep; boolean_t coming_out_of_standby; unsigned need_update; - void *mgroups[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; void *mgroup_holders[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; } osm_subn_t; /* @@ -630,10 +629,6 @@ typedef struct osm_subn { * This flag should be on during first non-master heavy * (including pre-master discovery stage) * -* mgroups -* Array of pointers to all Multicast Group objects in the subnet. -* Indexed by MLID offset from base MLID. -* * mgroup_holders * Array of pointers to all Multicast Group Holder objects in the subnet. * Indexed by MLID offset from base MLID. @@ -935,34 +930,6 @@ struct osm_port *osm_get_port_by_guid(IN osm_subn_t const *p_subn, * osm_port_t *********/ -/****f* OpenSM: Subnet/osm_get_mgrp_by_mlid -* NAME -* osm_get_mgrp_by_mlid -* -* DESCRIPTION -* The looks for the given multicast group in the subnet table by mlid. -* NOTE: this code is not thread safe. Need to grab the lock before -* calling it. -* -* SYNOPSIS -*/ -static inline -struct osm_mgrp *osm_get_mgrp_by_mlid(osm_subn_t const *p_subn, ib_net16_t mlid) -{ - return p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; -} -/* -* PARAMETERS -* p_subn -* [in] Pointer to an osm_subn_t object -* -* mlid -* [in] The multicast group mlid in network order -* -* RETURN VALUES -* The multicast group structure pointer if found. NULL otherwise. -*********/ - /****f* OpenSM: Helper/osm_get_physp_by_mad_addr * NAME * osm_get_physp_by_mad_addr diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index e16c2e7..1bc980c 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -55,6 +55,7 @@ #include #include #include +#include /********************************************************************** **********************************************************************/ @@ -111,14 +112,15 @@ static void mcast_mgr_purge_tree_node(IN osm_mtree_node_t * p_mtn) /********************************************************************** **********************************************************************/ -static void mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_t * p_mgrp) +static void mcast_mgr_purge_tree(osm_sm_t * sm, + IN osm_mgrp_holder_t * p_mgrp_holder) { OSM_LOG_ENTER(sm->p_log); - if (p_mgrp->p_root) - mcast_mgr_purge_tree_node(p_mgrp->p_root); + if (p_mgrp_holder->p_root) + mcast_mgr_purge_tree_node(p_mgrp_holder->p_root); - p_mgrp->p_root = NULL; + p_mgrp_holder->p_root = NULL; OSM_LOG_EXIT(sm->p_log); } @@ -126,41 +128,39 @@ static void mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_t * p_mgrp) /********************************************************************** **********************************************************************/ static float osm_mcast_mgr_compute_avg_hops(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp, + const osm_mgrp_holder_t * + p_mgrp_holder, const osm_switch_t * p_sw) { float avg_hops = 0; uint32_t hops = 0; uint32_t num_ports = 0; const osm_port_t *p_port; - const osm_mcm_port_t *p_mcm_port; - const cl_qmap_t *p_mcm_tbl; + const osm_mgrp_port_t *p_holder_port; OSM_LOG_ENTER(sm->p_log); - p_mcm_tbl = &p_mgrp->mcm_port_tbl; - /* For each member of the multicast group, compute the number of hops to its base LID. */ - for (p_mcm_port = (osm_mcm_port_t *) cl_qmap_head(p_mcm_tbl); - p_mcm_port != (osm_mcm_port_t *) cl_qmap_end(p_mcm_tbl); - p_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item)) { + for (p_holder_port = + (osm_mgrp_port_t *) cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + p_holder_port != + (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map); + p_holder_port = + (osm_mgrp_port_t *) cl_qmap_next(&p_holder_port->guid_item)) { /* Acquire the port object for this port guid, then create the new worker object to build the list. */ p_port = osm_get_port_by_guid(sm->p_subn, - ib_gid_get_guid(&p_mcm_port-> - port_gid)); + p_holder_port->port_guid); if (!p_port) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A18: " "No port object for port 0x%016" PRIx64 "\n", - cl_ntoh64(ib_gid_get_guid - (&p_mcm_port->port_gid))); + cl_ntoh64(p_holder_port->port_guid)); continue; } @@ -185,40 +185,37 @@ static float osm_mcast_mgr_compute_avg_hops(osm_sm_t * sm, of the group HCAs **********************************************************************/ static float osm_mcast_mgr_compute_max_hops(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp, + const osm_mgrp_holder_t * + p_mgrp_holder, const osm_switch_t * p_sw) { uint32_t max_hops = 0; uint32_t hops = 0; const osm_port_t *p_port; - const osm_mcm_port_t *p_mcm_port; - const cl_qmap_t *p_mcm_tbl; + const osm_mgrp_port_t *p_mgrp_holder_port; OSM_LOG_ENTER(sm->p_log); - p_mcm_tbl = &p_mgrp->mcm_port_tbl; - /* For each member of the multicast group, compute the number of hops to its base LID. */ - for (p_mcm_port = (osm_mcm_port_t *) cl_qmap_head(p_mcm_tbl); - p_mcm_port != (osm_mcm_port_t *) cl_qmap_end(p_mcm_tbl); - p_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item)) { + for (p_mgrp_holder_port = + (osm_mgrp_port_t *) cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + p_mgrp_holder_port != + (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map); + p_mgrp_holder_port = + (osm_mgrp_port_t *) cl_qmap_next(&p_mgrp_holder_port->guid_item)) { /* Acquire the port object for this port guid, then create the new worker object to build the list. */ p_port = osm_get_port_by_guid(sm->p_subn, - ib_gid_get_guid(&p_mcm_port-> - port_gid)); - + p_mgrp_holder_port->port_guid); if (!p_port) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A1A: " "No port object for port 0x%016" PRIx64 "\n", - cl_ntoh64(ib_gid_get_guid - (&p_mcm_port->port_gid))); + cl_ntoh64(p_mgrp_holder_port->port_guid)); continue; } @@ -244,15 +241,16 @@ static float osm_mcast_mgr_compute_max_hops(osm_sm_t * sm, of the multicast group. **********************************************************************/ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp) + const osm_mgrp_holder_t * + p_mgrp_holder) { cl_qmap_t *p_sw_tbl; - const osm_switch_t *p_sw; - const osm_switch_t *p_best_sw = NULL; + osm_switch_t *p_sw; + osm_switch_t *p_best_sw = NULL; float hops = 0; float best_hops = 10000; /* any big # will do */ #ifdef OSM_VENDOR_INTF_ANAFA - boolean_t use_avg_hops = TRUE; /* anafa2 - bug hca on switch *//* use max hops for root */ + boolean_t use_avg_hops = TRUE; /* anafa2 - bug hca on switch - use max hops for root */ #else boolean_t use_avg_hops = FALSE; /* use max hops for root */ #endif @@ -260,8 +258,7 @@ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, OSM_LOG_ENTER(sm->p_log); p_sw_tbl = &sm->p_subn->sw_guid_tbl; - - CL_ASSERT(!osm_mgrp_is_empty(p_mgrp)); + CL_ASSERT(!osm_mgrp_holder_is_empty(p_mgrp_holder)); for (p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl); @@ -270,9 +267,13 @@ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, continue; if (use_avg_hops) - hops = osm_mcast_mgr_compute_avg_hops(sm, p_mgrp, p_sw); + hops = + osm_mcast_mgr_compute_avg_hops(sm, p_mgrp_holder, + p_sw); else - hops = osm_mcast_mgr_compute_max_hops(sm, p_mgrp, p_sw); + hops = + osm_mcast_mgr_compute_max_hops(sm, p_mgrp_holder, + p_sw); OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Switch 0x%016" PRIx64 ", hops = %f\n", @@ -301,7 +302,8 @@ static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, This function returns the existing or optimal root swtich for the tree. **********************************************************************/ static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, - const osm_mgrp_t * p_mgrp) + const osm_mgrp_holder_t * + p_mgrp_holder) { const osm_switch_t *p_sw = NULL; @@ -313,7 +315,7 @@ static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, the root will be always on the first switch attached to it. - Very bad ... */ - p_sw = mcast_mgr_find_optimal_switch(sm, p_mgrp); + p_sw = mcast_mgr_find_optimal_switch(sm, p_mgrp_holder); OSM_LOG_EXIT(sm->p_log); return (osm_switch_t *) p_sw; @@ -394,7 +396,8 @@ static osm_signal_t mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * p_sw) spanning tree that eminate from this switch. On input, the p_list contains the group members that must be routed from this switch. **********************************************************************/ -static void mcast_mgr_subdivide(osm_sm_t * sm, osm_mgrp_t * p_mgrp, +static void mcast_mgr_subdivide(osm_sm_t * sm, + osm_mgrp_holder_t * p_mgrp_holder, osm_switch_t * p_sw, cl_qlist_t * p_list, cl_qlist_t * list_array, uint8_t array_size) { @@ -405,7 +408,7 @@ static void mcast_mgr_subdivide(osm_sm_t * sm, osm_mgrp_t * p_mgrp, OSM_LOG_ENTER(sm->p_log); - mlid_ho = cl_ntoh16(osm_mgrp_get_mlid(p_mgrp)); + mlid_ho = cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder)); /* For Multicast Groups, we want not to count on previous @@ -495,7 +498,8 @@ static void mcast_mgr_purge_list(osm_sm_t * sm, cl_qlist_t * p_list) The function returns the newly created mtree node element. **********************************************************************/ -static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, +static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, + osm_mgrp_holder_t * p_mgrp_holder, osm_switch_t * p_sw, cl_qlist_t * p_list, uint8_t depth, uint8_t upstream_port, @@ -521,7 +525,7 @@ static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, node_guid = osm_node_get_node_guid(p_sw->p_node); node_guid_ho = cl_ntoh64(node_guid); - mlid_ho = cl_ntoh16(osm_mgrp_get_mlid(p_mgrp)); + mlid_ho = cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder)); OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Routing MLID 0x%X through switch 0x%" PRIx64 @@ -598,7 +602,8 @@ static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, for (i = 0; i < max_children; i++) cl_qlist_init(&list_array[i]); - mcast_mgr_subdivide(sm, p_mgrp, p_sw, p_list, list_array, max_children); + mcast_mgr_subdivide(sm, p_mgrp_holder, p_sw, p_list, list_array, + max_children); p_tbl = osm_switch_get_mcast_tbl_ptr(p_sw); @@ -681,8 +686,9 @@ static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, osm_mgrp_t * p_mgrp, CL_ASSERT(p_remote_physp); p_mtn->child_array[i] = - mcast_mgr_branch(sm, p_mgrp, p_remote_node->sw, - p_port_list, depth, + mcast_mgr_branch(sm, p_mgrp_holder, + p_remote_node->sw, p_port_list, + depth, osm_physp_get_port_num (p_remote_physp), p_max_depth); } else { @@ -717,11 +723,11 @@ Exit: /********************************************************************** **********************************************************************/ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, - osm_mgrp_t * p_mgrp) + osm_mgrp_holder_t * + p_mgrp_holder) { - const cl_qmap_t *p_mcm_tbl; const osm_port_t *p_port; - const osm_mcm_port_t *p_mcm_port; + const osm_mgrp_port_t *p_mgrp_port; uint32_t num_ports; cl_qlist_t port_list; osm_switch_t *p_sw; @@ -740,14 +746,13 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, on multicast forwarding table information if the user wants to preserve existing multicast routes. */ - mcast_mgr_purge_tree(sm, p_mgrp); + mcast_mgr_purge_tree(sm, p_mgrp_holder); - p_mcm_tbl = &p_mgrp->mcm_port_tbl; - num_ports = cl_qmap_count(p_mcm_tbl); + num_ports = cl_qmap_count(&p_mgrp_holder->mgrp_port_map); if (num_ports == 0) { OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "MLID 0x%X has no members - nothing to do\n", - cl_ntoh16(osm_mgrp_get_mlid(p_mgrp))); + cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder))); goto Exit; } @@ -767,11 +772,11 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, Locate the switch around which to create the spanning tree for this multicast group. */ - p_sw = mcast_mgr_find_root_switch(sm, p_mgrp); + p_sw = mcast_mgr_find_root_switch(sm, p_mgrp_holder); if (p_sw == NULL) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A08: " "Unable to locate a suitable switch for group 0x%X\n", - cl_ntoh16(osm_mgrp_get_mlid(p_mgrp))); + cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder))); status = IB_ERROR; goto Exit; } @@ -779,22 +784,22 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, /* Build the first "subset" containing all member ports. */ - for (p_mcm_port = (osm_mcm_port_t *) cl_qmap_head(p_mcm_tbl); - p_mcm_port != (osm_mcm_port_t *) cl_qmap_end(p_mcm_tbl); - p_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item)) { + for (p_mgrp_port = + (osm_mgrp_port_t *) cl_qmap_head(&p_mgrp_holder->mgrp_port_map); + p_mgrp_port != + (osm_mgrp_port_t *) cl_qmap_end(&p_mgrp_holder->mgrp_port_map); + p_mgrp_port = + (osm_mgrp_port_t *) cl_qmap_next(&p_mgrp_port->guid_item)) { /* Acquire the port object for this port guid, then create the new worker object to build the list. */ - p_port = osm_get_port_by_guid(sm->p_subn, - ib_gid_get_guid(&p_mcm_port-> - port_gid)); + p_port = + osm_get_port_by_guid(sm->p_subn, p_mgrp_port->port_guid); if (!p_port) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A09: " "No port object for port 0x%016" PRIx64 "\n", - cl_ntoh64(ib_gid_get_guid - (&p_mcm_port->port_gid))); + cl_ntoh64(p_mgrp_port->port_guid)); continue; } @@ -802,8 +807,7 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, if (p_wobj == NULL) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A10: " "Insufficient memory to route port 0x%016" - PRIx64 "\n", - cl_ntoh64(osm_port_get_guid(p_port))); + PRIx64 "\n", cl_ntoh64(p_mgrp_port->port_guid)); continue; } @@ -811,12 +815,14 @@ static ib_api_status_t mcast_mgr_build_spanning_tree(osm_sm_t * sm, } count = cl_qlist_count(&port_list); - p_mgrp->p_root = mcast_mgr_branch(sm, p_mgrp, p_sw, &port_list, 0, 0, - &max_depth); + p_mgrp_holder->p_root = + mcast_mgr_branch(sm, p_mgrp_holder, p_sw, &port_list, 0, 0, + &max_depth); OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Configured MLID 0x%X for %u ports, max tree depth = %u\n", - cl_ntoh16(osm_mgrp_get_mlid(p_mgrp)), count, max_depth); + cl_ntoh16(osm_mgrp_holder_get_mlid(p_mgrp_holder)), count, + max_depth); Exit: OSM_LOG_EXIT(sm->p_log); @@ -1023,14 +1029,18 @@ Exit: lock must already be held on entry **********************************************************************/ static ib_api_status_t osm_mcast_mgr_process_tree(osm_sm_t * sm, - IN osm_mgrp_t * p_mgrp) + IN osm_mgrp_holder_t * + p_mgrp_holder) { ib_api_status_t status = IB_SUCCESS; ib_net16_t mlid; + osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + unsigned has_full_members = 0; OSM_LOG_ENTER(sm->p_log); - mlid = osm_mgrp_get_mlid(p_mgrp); + mlid = osm_mgrp_holder_get_mlid(p_mgrp_holder); OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Processing multicast group 0x%X\n", cl_ntoh16(mlid)); @@ -1051,10 +1061,18 @@ static ib_api_status_t osm_mcast_mgr_process_tree(osm_sm_t * sm, */ mcast_mgr_clear(sm, cl_ntoh16(mlid)); - if (!p_mgrp->full_members) + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + if (0 != (has_full_members = p_mgrp->full_members)) + break; + p_fitem = cl_fmap_next(p_fitem); + } + if (!has_full_members) goto Exit; - status = mcast_mgr_build_spanning_tree(sm, p_mgrp); + status = mcast_mgr_build_spanning_tree(sm, p_mgrp_holder); if (status != IB_SUCCESS) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A17: " "Unable to create spanning tree (%s)\n", @@ -1068,33 +1086,62 @@ Exit: } /********************************************************************** - Process the entire group. + Process the entire group holder. NOTE : The lock should be held externally! **********************************************************************/ static ib_api_status_t mcast_mgr_process_mgrp(osm_sm_t * sm, - IN osm_mgrp_t * p_mgrp) + IN osm_mgrp_holder_t * + p_mgrp_holder) { ib_api_status_t status; + osm_mgrp_t *p_mgrp; + cl_fmap_item_t *p_fitem; + ib_net16_t mlid; + char gid_str[INET6_ADDRSTRLEN]; OSM_LOG_ENTER(sm->p_log); - status = osm_mcast_mgr_process_tree(sm, p_mgrp); + status = osm_mcast_mgr_process_tree(sm, p_mgrp_holder); if (status != IB_SUCCESS) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A19: " "Unable to create spanning tree (%s)\n", ib_get_err_str(status)); goto Exit; } - p_mgrp->last_tree_id = p_mgrp->last_change_id; - - /* remove MCGRP if it is marked for deletion */ - if (p_mgrp->to_be_deleted) { + p_mgrp_holder->last_tree_id = p_mgrp_holder->last_change_id; + + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { + p_mgrp = (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, + mgid_item); + mlid = p_mgrp->mlid; + if (osm_is_debug()) { + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "holder mlid 0x%X has mgrp mgid = %s\n", + cl_ntoh16(p_mgrp_holder->mlid), + inet_ntop(AF_INET6, + p_mgrp->mcmember_rec.mgid.raw, + gid_str, sizeof(gid_str))); + } + /* remove MCGRP if it is marked for deletion */ + if (p_mgrp->to_be_deleted) { + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Destroying mgrp %s with lid:0x%x\n", + inet_ntop(AF_INET6, + p_mgrp->mcmember_rec.mgid.raw, + gid_str, sizeof(gid_str)), + cl_ntoh16(p_mgrp->mlid)); + osm_mgrp_delete_group(sm->p_subn, p_mgrp); + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); + } else + p_fitem = cl_fmap_next(p_fitem); + } + if (0 == cl_fmap_count(&p_mgrp_holder->mgrp_map)) { + /* no more groups on this mlid */ OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Destroying mgrp with lid:0x%x\n", - cl_ntoh16(p_mgrp->mlid)); - sm->p_subn->mgroups[cl_ntoh16(p_mgrp->mlid) - - IB_LID_MCAST_START_HO] = NULL; - osm_mgrp_delete(p_mgrp); + "Destroying mgrp_holder with lid:0x%x\n", + cl_ntoh16(p_mgrp_holder->mlid)); + osm_mgrp_holder_delete(sm->p_subn, p_mgrp_holder->mlid); } Exit: @@ -1110,7 +1157,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; cl_qlist_t *p_list = &sm->mgrp_list; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; boolean_t pending_transactions = FALSE; int i; @@ -1132,9 +1179,10 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) of the subnet. Not due to a specific multicast request. So the request type is subnet_change and the port guid is 0. */ - p_mgrp = sm->p_subn->mgroups[i]; - if (p_mgrp) - mcast_mgr_process_mgrp(sm, p_mgrp); + p_mgrp_holder = sm->p_subn->mgroup_holders[i]; + if (p_mgrp_holder) { + mcast_mgr_process_mgrp(sm, p_mgrp_holder); + } } /* @@ -1172,7 +1220,7 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) cl_qlist_t *p_list = &sm->mgrp_list; osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; - osm_mgrp_t *p_mgrp; + osm_mgrp_holder_t *p_mgrp_holder; ib_net16_t mlid; osm_signal_t ret, signal = OSM_SIGNAL_DONE; osm_mcast_mgr_ctxt_t *ctx; @@ -1193,24 +1241,25 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) /* since we delayed the execution we prefer to pass the mlid as the mgrp identifier and then find it or abort */ - p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, mlid); - if (!p_mgrp) + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sm->p_subn, mlid); + if (!p_mgrp_holder) continue; /* if there was no change from the last time * we processed the group we can skip doing anything */ - if (p_mgrp->last_change_id == p_mgrp->last_tree_id) { + if (p_mgrp_holder->last_change_id == + p_mgrp_holder->last_tree_id) { OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Skip processing mgrp with lid:0x%X change id:%u\n", - cl_ntoh16(mlid), p_mgrp->last_change_id); + "Skip processing p_mgrp_holder with lid:0x%X change id:%u\n", + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); continue; } OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Processing mgrp with lid:0x%X change id:%u\n", - cl_ntoh16(mlid), p_mgrp->last_change_id); - mcast_mgr_process_mgrp(sm, p_mgrp); + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); + mcast_mgr_process_mgrp(sm, p_mgrp_holder); } /* diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index e2e588c..e687ca8 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -54,28 +54,6 @@ /********************************************************************** **********************************************************************/ -void osm_mgrp_delete(IN osm_mgrp_t * p_mgrp) -{ - osm_mcm_port_t *p_mcm_port; - osm_mcm_port_t *p_next_mcm_port; - - CL_ASSERT(p_mgrp); - - p_next_mcm_port = - (osm_mcm_port_t *) cl_qmap_head(&p_mgrp->mcm_port_tbl); - while (p_next_mcm_port != - (osm_mcm_port_t *) cl_qmap_end(&p_mgrp->mcm_port_tbl)) { - p_mcm_port = p_next_mcm_port; - p_next_mcm_port = - (osm_mcm_port_t *) cl_qmap_next(&p_mcm_port->map_item); - osm_mcm_port_delete(p_mcm_port); - } - /* destroy the mtree_node structure */ - osm_mtree_destroy(p_mgrp->p_root); - - free(p_mgrp); -} - void osm_mgrp_delete_group(IN osm_subn_t * p_subn, IN osm_mgrp_t * p_mgrp) { osm_mcm_port_t *p_mcm_port; @@ -114,10 +92,7 @@ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid) memset(p_mgrp, 0, sizeof(*p_mgrp)); cl_qmap_init(&p_mgrp->mcm_port_tbl); p_mgrp->mlid = mlid; - p_mgrp->last_change_id = 0; - p_mgrp->last_tree_id = 0; p_mgrp->to_be_deleted = FALSE; - return p_mgrp; } @@ -195,11 +170,7 @@ osm_mcm_port_t *osm_mgrp_add_port(IN osm_subn_t * subn, osm_log_t * log, p_mcm_port->scope_state = ib_member_set_scope_state(prev_scope, prev_join_state | join_state); - } else { - /* track the fact we modified the group ports */ - p_mgrp->last_change_id++; } - if ((join_state & IB_JOIN_STATE_FULL) && !(prev_join_state & IB_JOIN_STATE_FULL) && (++p_mgrp->full_members == 1)) { @@ -239,8 +210,6 @@ int osm_mgrp_remove_port(osm_subn_t * subn, osm_log_t * log, osm_mgrp_t * mgrp, OSM_LOG(log, OSM_LOG_DEBUG, "removing port 0x%" PRIx64 "\n", cl_ntoh64(mcm->port_gid.unicast.interface_id)); osm_mcm_port_delete(mcm); - /* track the fact we modified the group */ - mgrp->last_change_id++; ret = 1; } @@ -313,22 +282,6 @@ static void mgrp_apply_func_sub(const osm_mgrp_t * p_mgrp, /********************************************************************** **********************************************************************/ -void osm_mgrp_apply_func(const osm_mgrp_t * p_mgrp, osm_mgrp_func_t p_func, - void *context) -{ - osm_mtree_node_t *p_mtn; - - CL_ASSERT(p_mgrp); - CL_ASSERT(p_func); - - p_mtn = p_mgrp->p_root; - - if (p_mtn) - mgrp_apply_func_sub(p_mgrp, p_mtn, p_func, context); -} - -/********************************************************************** - **********************************************************************/ #define PREFIX_MASK_IP CL_HTON64(0xff10ffff0000ffffULL) #define PREFIX_SIGNATURE_IPV4 CL_HTON64(0xff10401b00000000ULL) #define INTERFACE_ID_IPV4 CL_HTON64(0x0000000fffffffffULL) diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index 459e2cc..d74b4df 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -509,15 +509,14 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, size_t gr_count = cl_fmap_count(&p_mgrp_holder->mgrp_map); OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, - "mlid 0x%X has %d mgroups\n", cl_ntoh16(mlid), + "mlid 0x%X has %lu mgroups\n", cl_ntoh16(mlid), gr_count); if (gr_count) { p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { - p_mgrp = - (osm_mgrp_t *) + p_mgrp = (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, mgid_item); OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, -- 1.5.5 From slavas at Voltaire.COM Sun Jun 28 04:02:12 2009 From: slavas at Voltaire.COM (Slava Strebkov) Date: Sun, 28 Jun 2009 14:02:12 +0300 Subject: [ofa-general] Re: [PATCH] opensm: multicast multiplexing many mgid to mlid In-Reply-To: <4A474860.7000702@Voltaire.COM> References: <4A3A387A.2020509@Voltaire.COM> <4A474860.7000702@Voltaire.COM> Message-ID: <4A474DB4.7070009@Voltaire.COM> Please ignore this message - wrongly sent. Only review numbered patches 1-4 in this thread. Slava Slava Strebkov wrote: > Subject: [PATCH 3/4] Patch implements multicast multiplexing as proposed in the > thread entitled "IPv6 and IPoIB scalability issue". > This part (3) introduces usage of new infrastructure > for mgid->mlid compression > Signed-off-by: Slava Strebkov > > --- > opensm/include/opensm/osm_multicast.h | 5 +- > opensm/include/opensm/osm_sm.h | 23 ++- > opensm/opensm/osm_drop_mgr.c | 14 +- > opensm/opensm/osm_mcast_mgr.c | 2 +- > opensm/opensm/osm_multicast.c | 2 +- > opensm/opensm/osm_qos_policy.c | 360 ++++++++++++++++++-------------- > opensm/opensm/osm_sa.c | 65 ++++-- > opensm/opensm/osm_sa_mcmember_record.c | 161 ++++++++------- > opensm/opensm/osm_sa_path_record.c | 32 +++- > opensm/opensm/osm_sm.c | 92 +++++++-- > opensm/opensm/osm_subnet.c | 12 +- > 11 files changed, 474 insertions(+), 294 deletions(-) > > diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h > index 9f0cd96..567a989 100644 > --- a/opensm/include/opensm/osm_multicast.h > +++ b/opensm/include/opensm/osm_multicast.h > @@ -331,12 +331,15 @@ osm_mgrp_t *osm_mgrp_new(IN const ib_net16_t mlid); > * > * SYNOPSIS > */ > -void osm_mgrp_delete(IN osm_mgrp_t * const p_mgrp); > +void osm_mgrp_delete_group(IN osm_subn_t * p_subn, IN osm_mgrp_t * const p_mgrp); > /* > * PARAMETERS > * p_mgrp > * [in] Pointer to an osm_mgrp_t object. > * > +* p_subn > +* [in] Pointer to an osm_subn_t object. > +* > * RETURN VALUES > * None. > * > diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h > index cc8321d..26530f3 100644 > --- a/opensm/include/opensm/osm_sm.h > +++ b/opensm/include/opensm/osm_sm.h > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * > @@ -61,6 +61,7 @@ > #include > #include > #include > +#include > > #ifdef __cplusplus > # define BEGIN_C_DECLS extern "C" { > @@ -101,7 +102,7 @@ BEGIN_C_DECLS > * > * SYNOPSIS > */ > -typedef struct osm_sm { > + typedef struct osm_sm { > osm_thread_state_t thread_state; > unsigned signal_mask; > cl_spinlock_t signal_lock; > @@ -539,7 +540,7 @@ osm_resp_send(IN osm_sm_t * sm, > ib_api_status_t > osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > IN const ib_net16_t mlid, > - IN const ib_net64_t port_guid); > + IN const ib_net64_t port_guid, IN const ib_gid_t * p_mgid); > /* > * PARAMETERS > * p_sm > @@ -551,6 +552,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > * port_guid > * [in] Port GUID to add to the group. > * > +* p_mgid > +* [in] MGID to add to the group holder. > * RETURN VALUES > * None > * > @@ -572,7 +575,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > */ > ib_api_status_t > osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, > - IN const ib_net16_t mlid, IN const ib_net64_t port_guid); > + IN osm_mgrp_t * p_mgrp, IN ib_net64_t port_guid); > /* > * PARAMETERS > * p_sm > @@ -689,7 +692,7 @@ osm_sm_is_greater_than(IN const uint8_t l_priority, > * > * SYNOPSIS > */ > -ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t *sm, > +ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t * sm, > IN osm_sm_signal_t signal); > /* > * PARAMETERS > @@ -718,7 +721,7 @@ ib_api_status_t osm_sm_state_mgr_process(IN osm_sm_t *sm, > * > * SYNOPSIS > */ > -void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t *sm); > +void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t * sm); > /* > * PARAMETERS > * sm > @@ -743,7 +746,7 @@ void osm_sm_state_mgr_signal_master_is_alive(IN osm_sm_t *sm); > * > * SYNOPSIS > */ > -ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t *sm, > +ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t * sm, > IN osm_sm_signal_t signal); > /* > * PARAMETERS > @@ -762,7 +765,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(IN osm_sm_t *sm, > * State Manager > *********/ > > -void osm_report_sm_state(osm_sm_t *sm); > +void osm_report_sm_state(osm_sm_t * sm); > > /****f* OpenSM: SM State Manager/osm_send_trap144 > * NAME > @@ -773,7 +776,7 @@ void osm_report_sm_state(osm_sm_t *sm); > * > * SYNOPSIS > */ > -int osm_send_trap144(osm_sm_t *sm, ib_net16_t local); > +int osm_send_trap144(osm_sm_t * sm, ib_net16_t local); > /* > * PARAMETERS > * sm > @@ -787,7 +790,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local); > * > *********/ > > -void osm_set_sm_priority(osm_sm_t *sm, uint8_t priority); > +void osm_set_sm_priority(osm_sm_t * sm, uint8_t priority); > > END_C_DECLS > #endif /* _OSM_SM_H_ */ > diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c > index c9a4f33..d92fdfc 100644 > --- a/opensm/opensm/osm_drop_mgr.c > +++ b/opensm/opensm/osm_drop_mgr.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -158,7 +158,6 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > osm_port_t *p_port_check; > cl_qmap_t *p_sm_guid_tbl; > osm_mcm_info_t *p_mcm; > - osm_mgrp_t *p_mgrp; > cl_ptr_vector_t *p_port_lid_tbl; > uint16_t min_lid_ho; > uint16_t max_lid_ho; > @@ -168,6 +167,7 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > ib_gid_t port_gid; > ib_mad_notice_attr_t notice; > ib_api_status_t status; > + osm_mgrp_holder_t *p_osm_mgrp_holder; > > OSM_LOG_ENTER(sm->p_log); > > @@ -212,10 +212,12 @@ static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > > p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); > while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { > - p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, p_mcm->mlid); > - if (p_mgrp) { > - osm_mgrp_delete_port(sm->p_subn, sm->p_log, > - p_mgrp, p_port->guid); > + p_osm_mgrp_holder = > + osm_get_mgrp_holder_by_mlid(sm->p_subn, p_mcm->mlid); > + if (p_osm_mgrp_holder) { > + osm_mgrp_holder_remove_port(sm->p_subn, sm->p_log, > + p_osm_mgrp_holder, > + p_port->guid); > osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > } > p_mcm = > diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c > index de0a8a5..e16c2e7 100644 > --- a/opensm/opensm/osm_mcast_mgr.c > +++ b/opensm/opensm/osm_mcast_mgr.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c > index d2a19ea..e2e588c 100644 > --- a/opensm/opensm/osm_multicast.c > +++ b/opensm/opensm/osm_multicast.c > @@ -41,7 +41,7 @@ > #if HAVE_CONFIG_H > # include > #endif /* HAVE_CONFIG_H */ > - > +#include > #include > #include > #include > diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c > index 094fef2..07b05f5 100644 > --- a/opensm/opensm/osm_qos_policy.c > +++ b/opensm/opensm/osm_qos_policy.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -59,11 +59,10 @@ extern osm_qos_level_t __default_simple_qos_level; > /*************************************************** > ***************************************************/ > > -static void > -__build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) > +static void __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) > { > - osm_node_t * p_node; > - cl_qmap_t * p_node_guid_tbl = &p_qos_policy->p_subn->node_guid_tbl; > + osm_node_t *p_node; > + cl_qmap_t *p_node_guid_tbl = &p_qos_policy->p_subn->node_guid_tbl; > > p_qos_policy->p_node_hash = st_init_strtable(); > CL_ASSERT(p_qos_policy->p_node_hash); > @@ -75,10 +74,10 @@ __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) > p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); > p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { > if (!st_lookup(p_qos_policy->p_node_hash, > - (st_data_t)p_node->print_desc, NULL)) > + (st_data_t) p_node->print_desc, NULL)) > st_insert(p_qos_policy->p_node_hash, > - (st_data_t)p_node->print_desc, > - (st_data_t)p_node); > + (st_data_t) p_node->print_desc, > + (st_data_t) p_node); > } > } > > @@ -87,7 +86,7 @@ __build_nodebyname_hash(osm_qos_policy_t * p_qos_policy) > > static boolean_t > __is_num_in_range_arr(uint64_t ** range_arr, > - unsigned range_arr_len, uint64_t num) > + unsigned range_arr_len, uint64_t num) > { > unsigned ind_1 = 0; > unsigned ind_2 = range_arr_len - 1; > @@ -97,22 +96,23 @@ __is_num_in_range_arr(uint64_t ** range_arr, > return FALSE; > > while (ind_1 <= ind_2) { > - if (num < range_arr[ind_1][0] || num > range_arr[ind_2][1]) > - return FALSE; > - else if (num <= range_arr[ind_1][1] || num >= range_arr[ind_2][0]) > - return TRUE; > - > - ind_mid = ind_1 + (ind_2 - ind_1 + 1)/2; > - > - if (num < range_arr[ind_mid][0]) > - ind_2 = ind_mid; > - else if (num > range_arr[ind_mid][1]) > - ind_1 = ind_mid; > - else > - return TRUE; > - > - ind_1++; > - ind_2--; > + if (num < range_arr[ind_1][0] || num > range_arr[ind_2][1]) > + return FALSE; > + else if (num <= range_arr[ind_1][1] > + || num >= range_arr[ind_2][0]) > + return TRUE; > + > + ind_mid = ind_1 + (ind_2 - ind_1 + 1) / 2; > + > + if (num < range_arr[ind_mid][0]) > + ind_2 = ind_mid; > + else if (num > range_arr[ind_mid][1]) > + ind_1 = ind_mid; > + else > + return TRUE; > + > + ind_1++; > + ind_2--; > } > > return FALSE; > @@ -130,10 +130,9 @@ static void __free_single_element(void *p_element, void *context) > /*************************************************** > ***************************************************/ > > -osm_qos_port_t *osm_qos_policy_port_create(osm_physp_t *p_physp) > +osm_qos_port_t *osm_qos_policy_port_create(osm_physp_t * p_physp) > { > - osm_qos_port_t *p = > - (osm_qos_port_t *) malloc(sizeof(osm_qos_port_t)); > + osm_qos_port_t *p = (osm_qos_port_t *) malloc(sizeof(osm_qos_port_t)); > if (!p) > return NULL; > memset(p, 0, sizeof(osm_qos_port_t)); > @@ -163,8 +162,8 @@ osm_qos_port_group_t *osm_qos_policy_port_group_create() > > void osm_qos_policy_port_group_destroy(osm_qos_port_group_t * p) > { > - osm_qos_port_t * p_port; > - osm_qos_port_t * p_old_port; > + osm_qos_port_t *p_port; > + osm_qos_port_t *p_old_port; > > if (!p) > return; > @@ -175,8 +174,7 @@ void osm_qos_policy_port_group_destroy(osm_qos_port_group_t * p) > free(p->use); > > p_port = (osm_qos_port_t *) cl_qmap_head(&p->port_map); > - while (p_port != (osm_qos_port_t *) cl_qmap_end(&p->port_map)) > - { > + while (p_port != (osm_qos_port_t *) cl_qmap_end(&p->port_map)) { > p_old_port = p_port; > p_port = (osm_qos_port_t *) cl_qmap_next(&p_port->map_item); > free(p_old_port); > @@ -424,9 +422,10 @@ void osm_qos_policy_match_rule_destroy(osm_qos_match_rule_t * p) > /*************************************************** > ***************************************************/ > > -osm_qos_policy_t * osm_qos_policy_create(osm_subn_t * p_subn) > +osm_qos_policy_t *osm_qos_policy_create(osm_subn_t * p_subn) > { > - osm_qos_policy_t * p_qos_policy = (osm_qos_policy_t *)malloc(sizeof(osm_qos_policy_t)); > + osm_qos_policy_t *p_qos_policy = > + (osm_qos_policy_t *) malloc(sizeof(osm_qos_policy_t)); > if (!p_qos_policy) > return NULL; > > @@ -544,8 +543,8 @@ __qos_policy_is_port_in_group(osm_subn_t * p_subn, > > /* check whether this port's type matches any of group's types */ > > - if ( p_port_group->node_types & > - (((uint8_t)1)< + if (p_port_group->node_types & > + (((uint8_t) 1) << osm_node_get_type(p_node))) > return TRUE; > > /* check whether this port's guid is in group's port map */ > @@ -585,24 +584,33 @@ __qos_policy_is_port_in_group_list(const osm_qos_policy_t * p_qos_policy, > /*************************************************** > ***************************************************/ > > -static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( > - const osm_qos_policy_t * p_qos_policy, > - uint64_t service_id, > - uint16_t qos_class, > - uint16_t pkey, > - const osm_physp_t * p_src_physp, > - const osm_physp_t * p_dest_physp, > - ib_net64_t comp_mask) > +static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params(const > + osm_qos_policy_t > + * > + p_qos_policy, > + uint64_t > + service_id, > + uint16_t > + qos_class, > + uint16_t > + pkey, > + const > + osm_physp_t * > + p_src_physp, > + const > + osm_physp_t * > + p_dest_physp, > + ib_net64_t > + comp_mask) > { > osm_qos_match_rule_t *p_qos_match_rule = NULL; > cl_list_iterator_t list_iterator; > - osm_log_t * p_log = &p_qos_policy->p_subn->p_osm->log; > + osm_log_t *p_log = &p_qos_policy->p_subn->p_osm->log; > > boolean_t matched_by_sguid = FALSE, > - matched_by_dguid = FALSE, > - matched_by_class = FALSE, > - matched_by_sid = FALSE, > - matched_by_pkey = FALSE; > + matched_by_dguid = FALSE, > + matched_by_class = FALSE, > + matched_by_sid = FALSE, matched_by_pkey = FALSE; > > if (!cl_list_count(&p_qos_policy->qos_match_rules)) > return NULL; > @@ -698,8 +706,7 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( > > if (!__is_num_in_range_arr > (p_qos_match_rule->pkey_range_arr, > - p_qos_match_rule->pkey_range_len, > - pkey & 0x7FFF)) { > + p_qos_match_rule->pkey_range_len, pkey & 0x7FFF)) { > list_iterator = cl_list_next(list_iterator); > continue; > } > @@ -717,15 +724,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( > OSM_LOG(p_log, OSM_LOG_DEBUG, > "request matched rule (%s) by:%s%s%s%s%s\n", > (p_qos_match_rule->use) ? > - p_qos_match_rule->use : "no description", > + p_qos_match_rule->use : "no description", > (matched_by_sguid) ? " SGUID" : "", > (matched_by_dguid) ? " DGUID" : "", > (matched_by_class) ? " QoS_Class" : "", > - (matched_by_sid) ? " ServiceID" : "", > - (matched_by_pkey) ? " PKey" : ""); > + (matched_by_sid) ? " ServiceID" : "", > + (matched_by_pkey) ? " PKey" : ""); > else > - OSM_LOG(p_log, OSM_LOG_DEBUG, > - "request not matched any rule\n"); > + OSM_LOG(p_log, OSM_LOG_DEBUG, "request not matched any rule\n"); > > OSM_LOG_EXIT(p_log); > return p_qos_match_rule; > @@ -734,9 +740,10 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( > /*************************************************** > ***************************************************/ > > -static osm_qos_level_t *__qos_policy_get_qos_level_by_name( > - const osm_qos_policy_t * p_qos_policy, > - char *name) > +static osm_qos_level_t *__qos_policy_get_qos_level_by_name(const > + osm_qos_policy_t * > + p_qos_policy, > + char *name) > { > osm_qos_level_t *p_qos_level = NULL; > cl_list_iterator_t list_iterator; > @@ -760,9 +767,11 @@ static osm_qos_level_t *__qos_policy_get_qos_level_by_name( > /*************************************************** > ***************************************************/ > > -static osm_qos_port_group_t *__qos_policy_get_port_group_by_name( > - const osm_qos_policy_t * p_qos_policy, > - const char *const name) > +static osm_qos_port_group_t *__qos_policy_get_port_group_by_name(const > + osm_qos_policy_t > + * p_qos_policy, > + const char > + *const name) > { > osm_qos_port_group_t *p_port_group = NULL; > cl_list_iterator_t list_iterator; > @@ -787,15 +796,16 @@ static osm_qos_port_group_t *__qos_policy_get_port_group_by_name( > /*************************************************** > ***************************************************/ > > -static void __qos_policy_validate_pkey( > - osm_qos_policy_t * p_qos_policy, > - osm_qos_match_rule_t * p_qos_match_rule, > - osm_prtn_t * p_prtn) > +static void __qos_policy_validate_pkey(osm_qos_policy_t * p_qos_policy, > + osm_qos_match_rule_t * p_qos_match_rule, > + osm_prtn_t * p_prtn) > { > uint8_t sl; > uint32_t flow; > uint8_t hop; > - osm_mgrp_t * p_mgrp; > + osm_mgrp_t *p_mgrp; > + osm_mgrp_holder_t *p_mgrp_holder; > + cl_fmap_item_t *p_fitem; > > if (!p_qos_policy || !p_qos_match_rule || !p_prtn) > return; > @@ -812,43 +822,49 @@ static void __qos_policy_validate_pkey( > p_qos_match_rule->p_qos_level->sl); > p_prtn->sl = p_qos_match_rule->p_qos_level->sl; > > - > /* If this partition is an IPoIB partition, there should > be a matching MCast group. Fix this group's SL too */ > > if (!p_prtn->mlid) > return; > > - p_mgrp = osm_get_mgrp_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); > - if (!p_mgrp) { > + p_mgrp_holder = > + osm_get_mgrp_holder_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); > + if (!p_mgrp_holder) { > OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, > - "ERR AC16: MCast group for partition with " > - "pkey 0x%04X not found\n", > - cl_ntoh16(p_prtn->pkey)); > + "ERR AC16: MCast mgrp_holder for partition with " > + "pkey 0x%04X not found\n", cl_ntoh16(p_prtn->pkey)); > return; > } > - > - CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == > - (cl_ntoh16(p_prtn->pkey) & 0x7fff)); > - > - ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, > - &sl, &flow, &hop); > - if (sl != p_prtn->sl) { > - OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, > - "Updating MCGroup (MLID 0x%04x) SL to " > - "match partition SL (%u)\n", > - cl_hton16(p_mgrp->mcmember_rec.mlid), > - p_prtn->sl); > - p_mgrp->mcmember_rec.sl_flow_hop = > - ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); > + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); > + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + p_mgrp = > + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, > + mgid_item); > + p_fitem = cl_fmap_next(p_fitem); > + > + CL_ASSERT((cl_ntoh16(p_mgrp->mcmember_rec.pkey) & 0x7fff) == > + (cl_ntoh16(p_prtn->pkey) & 0x7fff)); > + > + ib_member_get_sl_flow_hop(p_mgrp->mcmember_rec.sl_flow_hop, > + &sl, &flow, &hop); > + if (sl != p_prtn->sl) { > + OSM_LOG(&p_qos_policy->p_subn->p_osm->log, > + OSM_LOG_DEBUG, > + "Updating MCGroup (MLID 0x%04x) SL to " > + "match partition SL (%u)\n", > + cl_hton16(p_mgrp->mcmember_rec.mlid), > + p_prtn->sl); > + p_mgrp->mcmember_rec.sl_flow_hop = > + ib_member_set_sl_flow_hop(p_prtn->sl, flow, hop); > + } > } > } > > /*************************************************** > ***************************************************/ > > -int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > - osm_log_t *p_log) > +int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, osm_log_t * p_log) > { > cl_list_iterator_t match_rules_list_iterator; > cl_list_iterator_t list_iterator; > @@ -859,22 +875,23 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > int res = 0; > uint64_t pkey_64; > ib_net16_t pkey; > - osm_prtn_t * p_prtn; > + osm_prtn_t *p_prtn; > > OSM_LOG_ENTER(p_log); > > /* set default qos level */ > > p_qos_policy->p_default_qos_level = > - __qos_policy_get_qos_level_by_name(p_qos_policy, OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); > + __qos_policy_get_qos_level_by_name(p_qos_policy, > + OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); > if (!p_qos_policy->p_default_qos_level) { > /* There's no default QoS level in the usual qos-level section. > Check whether the 'simple' default QoS level that can be > defined in the qos-ulp section exists */ > if (__default_simple_qos_level.sl_set) { > - p_qos_policy->p_default_qos_level = &__default_simple_qos_level; > - } > - else { > + p_qos_policy->p_default_qos_level = > + &__default_simple_qos_level; > + } else { > OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC10: " > "Default qos-level (%s) not defined.\n", > OSM_QOS_POLICY_DEFAULT_LEVEL_NAME); > @@ -891,8 +908,7 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > cl_list_head(&p_qos_policy->qos_match_rules); > while (match_rules_list_iterator != > cl_list_end(&p_qos_policy->qos_match_rules)) { > - p_qos_match_rule = > - (osm_qos_match_rule_t *) > + p_qos_match_rule = (osm_qos_match_rule_t *) > cl_list_obj(match_rules_list_iterator); > CL_ASSERT(p_qos_match_rule); > > @@ -900,8 +916,9 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > > if (!p_qos_match_rule->p_qos_level) > p_qos_match_rule->p_qos_level = > - __qos_policy_get_qos_level_by_name(p_qos_policy, > - p_qos_match_rule->qos_level_name); > + __qos_policy_get_qos_level_by_name(p_qos_policy, > + p_qos_match_rule-> > + qos_level_name); > > if (!p_qos_match_rule->p_qos_level) { > OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC11: " > @@ -922,9 +939,11 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > CL_ASSERT(str); > > p_port_group = > - __qos_policy_get_port_group_by_name(p_qos_policy, str); > + __qos_policy_get_port_group_by_name > + (p_qos_policy, str); > if (!p_port_group) { > - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC12: " > + OSM_LOG(p_log, OSM_LOG_ERROR, > + "ERR AC12: " > "qos-match-rule num %u: source port-group '%s' not found\n", > i, str); > res = 1; > @@ -951,9 +970,11 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > CL_ASSERT(str); > > p_port_group = > - __qos_policy_get_port_group_by_name(p_qos_policy,str); > + __qos_policy_get_port_group_by_name > + (p_qos_policy, str); > if (!p_port_group) { > - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC13: " > + OSM_LOG(p_log, OSM_LOG_ERROR, > + "ERR AC13: " > "qos-match-rule num %u: destination port-group '%s' not found\n", > i, str); > res = 1; > @@ -977,24 +998,30 @@ int osm_qos_policy_validate(osm_qos_policy_t * p_qos_policy, > */ > > for (j = 0; j < p_qos_match_rule->pkey_range_len; j++) { > - for ( pkey_64 = p_qos_match_rule->pkey_range_arr[j][0]; > - pkey_64 <= p_qos_match_rule->pkey_range_arr[j][1]; > - pkey_64++) { > - pkey = cl_hton16((uint16_t)(pkey_64 & 0x7fff)); > - p_prtn = (osm_prtn_t *)cl_qmap_get( > - &p_qos_policy->p_subn->prtn_pkey_tbl, pkey); > - > - if (p_prtn == (osm_prtn_t *)cl_qmap_end( > - &p_qos_policy->p_subn->prtn_pkey_tbl)) > + for (pkey_64 = p_qos_match_rule->pkey_range_arr[j][0]; > + pkey_64 <= p_qos_match_rule->pkey_range_arr[j][1]; > + pkey_64++) { > + pkey = cl_hton16((uint16_t) (pkey_64 & 0x7fff)); > + p_prtn = > + (osm_prtn_t *) cl_qmap_get(&p_qos_policy-> > + p_subn-> > + prtn_pkey_tbl, > + pkey); > + > + if (p_prtn == > + (osm_prtn_t *) cl_qmap_end(&p_qos_policy-> > + p_subn-> > + prtn_pkey_tbl)) > /* partition for this pkey not found */ > - OSM_LOG(p_log, OSM_LOG_ERROR, "ERR AC14: " > + OSM_LOG(p_log, OSM_LOG_ERROR, > + "ERR AC14: " > "pkey 0x%04X in match rule - " > "partition doesn't exist\n", > cl_ntoh16(pkey)); > else > __qos_policy_validate_pkey(p_qos_policy, > - p_qos_match_rule, > - p_prtn); > + p_qos_match_rule, > + p_prtn); > } > } > > @@ -1013,53 +1040,70 @@ Exit: > /*************************************************** > ***************************************************/ > > -static osm_qos_level_t * __qos_policy_get_qos_level_by_params( > - IN const osm_qos_policy_t * p_qos_policy, > - IN const osm_physp_t * p_src_physp, > - IN const osm_physp_t * p_dest_physp, > - IN uint64_t service_id, > - IN uint16_t qos_class, > - IN uint16_t pkey, > - IN ib_net64_t comp_mask) > +static osm_qos_level_t *__qos_policy_get_qos_level_by_params(IN const > + osm_qos_policy_t * > + p_qos_policy, > + IN const > + osm_physp_t * > + p_src_physp, > + IN const > + osm_physp_t * > + p_dest_physp, > + IN uint64_t > + service_id, > + IN uint16_t > + qos_class, > + IN uint16_t pkey, > + IN ib_net64_t > + comp_mask) > { > osm_qos_match_rule_t *p_qos_match_rule = NULL; > > if (!p_qos_policy) > return NULL; > > - p_qos_match_rule = __qos_policy_get_match_rule_by_params( > - p_qos_policy, service_id, qos_class, pkey, > - p_src_physp, p_dest_physp, comp_mask); > + p_qos_match_rule = > + __qos_policy_get_match_rule_by_params(p_qos_policy, service_id, > + qos_class, pkey, p_src_physp, > + p_dest_physp, comp_mask); > > return p_qos_match_rule ? p_qos_match_rule->p_qos_level : > - p_qos_policy->p_default_qos_level; > + p_qos_policy->p_default_qos_level; > } /* __qos_policy_get_qos_level_by_params() */ > > /*************************************************** > ***************************************************/ > > -osm_qos_level_t * osm_qos_policy_get_qos_level_by_pr( > - IN const osm_qos_policy_t * p_qos_policy, > - IN const ib_path_rec_t * p_pr, > - IN const osm_physp_t * p_src_physp, > - IN const osm_physp_t * p_dest_physp, > - IN ib_net64_t comp_mask) > +osm_qos_level_t *osm_qos_policy_get_qos_level_by_pr(IN const osm_qos_policy_t * > + p_qos_policy, > + IN const ib_path_rec_t * > + p_pr, > + IN const osm_physp_t * > + p_src_physp, > + IN const osm_physp_t * > + p_dest_physp, > + IN ib_net64_t comp_mask) > { > - return __qos_policy_get_qos_level_by_params( > - p_qos_policy, p_src_physp, p_dest_physp, > - cl_ntoh64(p_pr->service_id), ib_path_rec_qos_class(p_pr), > - cl_ntoh16(p_pr->pkey), comp_mask); > + return __qos_policy_get_qos_level_by_params(p_qos_policy, p_src_physp, > + p_dest_physp, > + cl_ntoh64(p_pr->service_id), > + ib_path_rec_qos_class(p_pr), > + cl_ntoh16(p_pr->pkey), > + comp_mask); > } > > /*************************************************** > ***************************************************/ > > -osm_qos_level_t * osm_qos_policy_get_qos_level_by_mpr( > - IN const osm_qos_policy_t * p_qos_policy, > - IN const ib_multipath_rec_t * p_mpr, > - IN const osm_physp_t * p_src_physp, > - IN const osm_physp_t * p_dest_physp, > - IN ib_net64_t comp_mask) > +osm_qos_level_t *osm_qos_policy_get_qos_level_by_mpr(IN const osm_qos_policy_t * > + p_qos_policy, > + IN const ib_multipath_rec_t > + * p_mpr, > + IN const osm_physp_t * > + p_src_physp, > + IN const osm_physp_t * > + p_dest_physp, > + IN ib_net64_t comp_mask) > { > ib_net64_t pr_comp_mask = 0; > > @@ -1071,20 +1115,24 @@ osm_qos_level_t * osm_qos_policy_get_qos_level_by_mpr( > * compmask. Note that only relevant bits are set. > */ > pr_comp_mask = > - ((comp_mask & IB_MPR_COMPMASK_QOS_CLASS) ? > - IB_PR_COMPMASK_QOS_CLASS : 0) | > - ((comp_mask & IB_MPR_COMPMASK_PKEY) ? > - IB_PR_COMPMASK_PKEY : 0) | > - ((comp_mask & IB_MPR_COMPMASK_SERVICEID_MSB) ? > - IB_PR_COMPMASK_SERVICEID_MSB : 0) | > - ((comp_mask & IB_MPR_COMPMASK_SERVICEID_LSB) ? > - IB_PR_COMPMASK_SERVICEID_LSB : 0); > - > - return __qos_policy_get_qos_level_by_params( > - p_qos_policy, p_src_physp, p_dest_physp, > - cl_ntoh64(ib_multipath_rec_service_id(p_mpr)), > - ib_multipath_rec_qos_class(p_mpr), > - cl_ntoh16(p_mpr->pkey), pr_comp_mask); > + ((comp_mask & IB_MPR_COMPMASK_QOS_CLASS) ? > + IB_PR_COMPMASK_QOS_CLASS : 0) | > + ((comp_mask & IB_MPR_COMPMASK_PKEY) ? > + IB_PR_COMPMASK_PKEY : 0) | > + ((comp_mask & IB_MPR_COMPMASK_SERVICEID_MSB) ? > + IB_PR_COMPMASK_SERVICEID_MSB : 0) | > + ((comp_mask & IB_MPR_COMPMASK_SERVICEID_LSB) ? > + IB_PR_COMPMASK_SERVICEID_LSB : 0); > + > + return __qos_policy_get_qos_level_by_params(p_qos_policy, p_src_physp, > + p_dest_physp, > + cl_ntoh64 > + (ib_multipath_rec_service_id > + (p_mpr)), > + ib_multipath_rec_qos_class > + (p_mpr), > + cl_ntoh16(p_mpr->pkey), > + pr_comp_mask); > } > > /*************************************************** > diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c > index fcc3f27..0c24a49 100644 > --- a/opensm/opensm/osm_sa.c > +++ b/opensm/opensm/osm_sa.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -52,6 +52,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -317,7 +318,7 @@ Exit: > return (status); > } > > -ib_api_status_t osm_sa_send(osm_sa_t *sa, > +ib_api_status_t osm_sa_send(osm_sa_t * sa, > IN osm_madw_t * const p_madw, > IN boolean_t const resp_expected) > { > @@ -397,8 +398,8 @@ Exit: > OSM_LOG_EXIT(sa->p_log); > } > > -void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, > - cl_qlist_t *list) > +void osm_sa_respond(osm_sa_t * sa, osm_madw_t * madw, size_t attr_size, > + cl_qlist_t * list) > { > struct item_data { > cl_list_item_t list; > @@ -422,13 +423,13 @@ void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, > */ > if (sa_mad->method == IB_MAD_METHOD_GET && num_rec > 1) { > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4C05: " > - "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" PRIx64 "\n", > - num_rec, ib_get_sa_attr_str(sa_mad->attr_id), > + "Got %u records for SubnAdmGet(%s) comp_mask 0x%016" > + PRIx64 "\n", num_rec, > + ib_get_sa_attr_str(sa_mad->attr_id), > cl_ntoh64(sa_mad->comp_mask)); > osm_sa_send_error(sa, madw, IB_SA_MAD_STATUS_TOO_MANY_RECORDS); > goto Exit; > } > - > #ifndef VENDOR_RMPP_SUPPORT > trim_num_rec = (MAD_BLOCK_SIZE - IB_SA_MAD_HDR_SIZE) / attr_size; > if (trim_num_rec < num_rec) { > @@ -564,7 +565,7 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) > p_mcm_port->scope_state, p_mcm_port->proxy_join); > } > > -static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) > +static void sa_dump_one_mgrp(osm_mgrp_t * p_mgrp, void *cxt) > { > struct opensm_dump_context dump_context; > osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; > @@ -706,6 +707,8 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) > { > struct opensm_dump_context dump_context; > osm_mgrp_t *p_mgrp; > + cl_fmap_item_t *p_fitem; > + osm_mgrp_holder_t *p_mgrp_holder; > int i; > > dump_context.p_osm = p_osm; > @@ -714,9 +717,17 @@ static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) > cl_plock_acquire(&p_osm->lock); > for (i = 0; i <= p_osm->subn.max_mcast_lid_ho - IB_LID_MCAST_START_HO; > i++) { > - p_mgrp = p_osm->subn.mgroups[i]; > - if (p_mgrp) > + p_mgrp_holder = p_osm->subn.mgroup_holders[i]; > + if (!p_mgrp_holder) > + continue; > + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); > + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + p_mgrp = > + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, > + mgid_item); > sa_dump_one_mgrp(p_mgrp, &dump_context); > + p_fitem = cl_fmap_next(&p_mgrp->mgid_item); > + } > } > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform\n"); > cl_qlist_apply_func(&p_osm->subn.sa_infr_list, > @@ -740,25 +751,35 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, > unsigned well_known) > { > ib_net64_t comp_mask; > - osm_mgrp_t *p_mgrp; > - > + cl_fmap_item_t *p_fitem; > + osm_mgrp_holder_t *p_mgrp_holder; > + ib_gid_t common_mgid; > + osm_mgrp_t *p_mgrp = NULL; > cl_plock_excl_acquire(&p_osm->lock); > > - p_mgrp = osm_get_mgrp_by_mlid(&p_osm->subn, mlid); > - if (p_mgrp) { > - if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, > - sizeof(ib_gid_t))) { > + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(&p_osm->subn, mlid); > + if (p_mgrp_holder) { > + p_fitem = > + cl_fmap_get(&p_mgrp_holder->mgrp_map, &p_mcm_rec->mgid); > + if (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, > "mgrp %04x is already here.", cl_ntoh16(mlid)); > goto _out; > } > - OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, > - "mlid %04x is already used by another MC group. Will " > - "request clients reregistration.\n", cl_ntoh16(mlid)); > - p_mgrp = NULL; > - goto _out; > - } > > + osm_mgrp_holder_prepare_common_mgid(&p_mcm_rec->mgid, > + &common_mgid); > + if (memcmp > + (&p_mgrp_holder->common_mgid, &common_mgid, > + sizeof(ib_gid_t))) { > + OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, > + "mlid %04x is already used by another MC group. Will " > + "request clients reregistration.\n", > + cl_ntoh16(mlid)); > + p_mgrp = NULL; > + goto _out; > + } > + } > comp_mask = IB_MCR_COMPMASK_MTU | IB_MCR_COMPMASK_MTU_SEL > | IB_MCR_COMPMASK_RATE | IB_MCR_COMPMASK_RATE_SEL; > if (osm_mcmr_rcv_find_or_create_new_mgrp(&p_osm->sa, > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > index 5543221..23c5107 100644 > --- a/opensm/opensm/osm_sa_mcmember_record.c > +++ b/opensm/opensm/osm_sa_mcmember_record.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -63,6 +63,7 @@ > #include > #include > #include > +#include > > #define JOIN_MC_COMP_MASK (IB_MCR_COMPMASK_MGID | \ > IB_MCR_COMPMASK_PORT_GID | \ > @@ -121,13 +122,14 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) > > if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO > && cl_ntoh16(requested_mlid) <= p_subn->max_mcast_lid_ho > - && !osm_get_mgrp_by_mlid(p_subn, requested_mlid)) > + && !osm_get_mgrp_holder_by_mlid(p_subn, requested_mlid)) > return requested_mlid; > > max = p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO + 1; > for (i = 0; i < max; i++) { > - osm_mgrp_t *p_mgrp = sa->p_subn->mgroups[i]; > - if (!p_mgrp || p_mgrp->to_be_deleted) > + osm_mgrp_holder_t *p_mgrp_holder = > + (osm_mgrp_holder_t *) sa->p_subn->mgroup_holders[i]; > + if (!p_mgrp_holder || p_mgrp_holder->to_be_deleted) > return cl_hton16(i + IB_LID_MCAST_START_HO); > } > > @@ -141,14 +143,21 @@ static ib_net16_t get_new_mlid(osm_sa_t * sa, ib_net16_t requested_mlid) > we silently drop it. Since it was an intermediate group no need to > re-route it. > **********************************************************************/ > -static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * mgrp) > +static void cleanup_mgrp(IN osm_sa_t * sa, osm_mgrp_t * p_mgrp) > { > /* Remove MGRP only if osm_mcm_port_t count is 0 and > not a well known group */ > - if (cl_is_qmap_empty(&mgrp->mcm_port_tbl) && !mgrp->well_known) { > - sa->p_subn->mgroups[cl_ntoh16(mgrp->mlid) - > - IB_LID_MCAST_START_HO] = NULL; > - osm_mgrp_delete(mgrp); > + char gid_str[INET6_ADDRSTRLEN]; > + if (cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && !p_mgrp->well_known) { > + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > + "mgrp mgid %s will be deleted\n", inet_ntop(AF_INET6, > + p_mgrp-> > + mcmember_rec. > + mgid.raw, > + gid_str, > + sizeof > + (gid_str))); > + osm_mgrp_delete_group(sa->sm->p_subn, p_mgrp); > } > } > > @@ -202,7 +211,6 @@ static ib_api_status_t add_new_mgrp_port(osm_sa_t * sa, IN osm_mgrp_t * p_mgrp, > > return IB_INSUFFICIENT_MEMORY; > } > - > return IB_SUCCESS; > } > > @@ -806,11 +814,12 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > IN const osm_physp_t * p_physp, > OUT osm_mgrp_t ** pp_mgrp) > { > - ib_net16_t mlid; > + ib_net16_t mlid, existed_mlid; > unsigned zero_mgid, i; > uint8_t scope; > + char gid_str[INET6_ADDRSTRLEN]; > ib_gid_t *p_mgid; > - osm_mgrp_t *p_prev_mgrp; > + osm_mgrp_holder_t *p_mgrp_holder; > ib_api_status_t status = IB_SUCCESS; > ib_member_rec_t mcm_rec = *p_recvd_mcmember_rec; /* copy for modifications */ > > @@ -823,7 +832,7 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > zero_mgid = 0; > break; > } > - > + p_mgid = &(mcm_rec.mgid); > /* > we allocate a new mlid number before we might use it > for MGID ... > @@ -843,8 +852,6 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > /* we need to create the new MGID if it was not defined */ > if (zero_mgid) { > /* create a new MGID */ > - char gid_str[INET6_ADDRSTRLEN]; > - > /* use the given scope state only if requested! */ > if (comp_mask & IB_MCR_COMPMASK_SCOPE) > ib_member_get_scope_state(p_recvd_mcmember_rec-> > @@ -888,6 +895,13 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > goto Exit; > } > > + if (0 != (existed_mlid = osm_mgrp_holder_get_mlid_by_mgid(sa, p_mgid))) { > + mlid = existed_mlid; > + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > + "found existed mlid 0x%04x for mgid %s\n", > + cl_ntoh16(mlid), inet_ntop(AF_INET6, p_mgid->raw, > + gid_str, sizeof gid_str)); > + } > /* create a new MC Group */ > *pp_mgrp = osm_mgrp_new(mlid); > if (*pp_mgrp == NULL) { > @@ -915,17 +929,21 @@ ib_api_status_t osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > /* since we might have an old group by that mlid > one whose deletion was delayed for an idle time > we need to deallocate it first */ > - p_prev_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); > - if (p_prev_mgrp) { > + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(sa->p_subn, mlid); > + if (!p_mgrp_holder) { > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > - "Found previous group for mlid:0x%04x - " > - "Destroying it first\n", cl_ntoh16(mlid)); > - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = > - NULL; > - osm_mgrp_delete(p_prev_mgrp); > + "Creating new mgrp holder for mlid:0x%04x\n", > + cl_ntoh16(mlid)); > + p_mgrp_holder = osm_mgrp_holder_new(sa->p_subn, p_mgid, mlid); > } > - > - sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; > + if (!p_mgrp_holder) { > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B08: " > + "osm_mgrp_holder_new failed\n"); > + free_mlid(sa, mlid); > + status = IB_INSUFFICIENT_MEMORY; > + goto Exit; > + } > + osm_mgrp_holder_add_mgrp(p_mgrp_holder, *pp_mgrp, sa->p_log); > > Exit: > OSM_LOG_EXIT(sa->p_log); > @@ -934,56 +952,47 @@ Exit: > > /********************************************************************** > *********************************************************************/ > -static unsigned match_mgrp_by_mgid(IN osm_mgrp_t * p_mgrp, ib_gid_t * mgid) > -{ > - /* ignore groups marked for deletion */ > - if (p_mgrp->to_be_deleted || > - memcmp(&p_mgrp->mcmember_rec.mgid, mgid, sizeof(ib_gid_t))) > - return 0; > - else > - return 1; > -} > - > -/********************************************************************** > - **********************************************************************/ > -#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) > -#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) > -#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) > -#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) > - > -/* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ > -/* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ > -/* Where Z is the scope, XXXX is the P_Key, and > - * YYYYYY is the last 24 bits of the port guid */ > -static unsigned match_and_update_ipv6_snm_mgid(ib_gid_t * mgid) > +static osm_mgrp_t *match_mgrp_by_mgid(IN osm_mgrp_holder_t * p_mgrp_holder, > + ib_gid_t * p_mgid) > { > - if ((mgid->unicast.prefix & PREFIX_MASK) == PREFIX_SIGNATURE && > - (mgid->unicast.interface_id & INT_ID_MASK) == INT_ID_SIGNATURE) { > - mgid->unicast.prefix &= PREFIX_MASK; > - mgid->unicast.interface_id &= INT_ID_MASK; > - return 1; > - } > - return 0; > + osm_mgrp_t *p_mgrp; > + ib_gid_t common_mgid; > + cl_fmap_item_t *p_fitem; > + > + osm_mgrp_holder_prepare_common_mgid(p_mgid, &common_mgid); > + > + if (memcmp(&p_mgrp_holder->common_mgid, &common_mgid, sizeof(ib_gid_t))) > + return NULL; > + p_fitem = cl_fmap_get(&p_mgrp_holder->mgrp_map, p_mgid); > + if (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + p_mgrp = (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, > + mgid_item); > + /* ignore groups marked for deletion */ > + if (p_mgrp->to_be_deleted) > + p_mgrp = NULL; > + } else > + p_mgrp = NULL; > + return p_mgrp;; > } > > osm_mgrp_t *osm_get_mgrp_by_mgid(IN osm_sa_t * sa, IN ib_gid_t * p_mgid) > { > int i; > - > - if (sa->p_subn->opt.consolidate_ipv6_snm_req && > - match_and_update_ipv6_snm_mgid(p_mgid)) { > - char gid_str[INET6_ADDRSTRLEN]; > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > - "Special Case Solicited Node Mcast Join for MGID %s\n", > - inet_ntop(AF_INET6, p_mgid->raw, gid_str, > - sizeof gid_str)); > - } > - > + osm_mgrp_t *p_mgrp; > for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; > - i++) > - if (sa->p_subn->mgroups[i] && > - match_mgrp_by_mgid(sa->p_subn->mgroups[i], p_mgid)) > - return sa->p_subn->mgroups[i]; > + i++) { > + if (sa->p_subn->mgroup_holders[i] && > + (p_mgrp = > + match_mgrp_by_mgid(sa->p_subn->mgroup_holders[i], > + p_mgid))) { > + char gid_str[INET6_ADDRSTRLEN]; > + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > + "Found mgroup for MGID %s\n", > + inet_ntop(AF_INET6, p_mgid->raw, gid_str, > + sizeof gid_str)); > + return p_mgrp; > + } > + } > > return NULL; > } > @@ -1080,7 +1089,7 @@ static void mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > CL_PLOCK_RELEASE(sa->p_lock); > > /* we can leave if port was deleted from MCG */ > - if (removed && osm_sm_mcgrp_leave(sa->sm, mlid, portguid)) > + if (removed && osm_sm_mcgrp_leave(sa->sm, p_mgrp, portguid)) > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B09: " > "osm_sm_mcgrp_leave failed\n"); > > @@ -1310,7 +1319,7 @@ static void mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > /* do the actual routing (actually schedule the update) */ > status = osm_sm_mcgrp_join(sa->sm, mlid, > p_recvd_mcmember_rec->port_gid.unicast. > - interface_id); > + interface_id, &p_recvd_mcmember_rec->mgid); > > if (status != IB_SUCCESS) { > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B14: " > @@ -1555,6 +1564,8 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > osm_physp_t *p_req_physp; > boolean_t trusted_req; > osm_mgrp_t *p_mgrp; > + cl_fmap_item_t *p_fitem; > + osm_mgrp_holder_t *p_mgrp_holder; > int i; > > OSM_LOG_ENTER(sa->p_log); > @@ -1586,10 +1597,18 @@ static void mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * p_madw) > /* simply go over all MCGs and match */ > for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; > i++) { > - p_mgrp = sa->p_subn->mgroups[i]; > - if (p_mgrp) > + p_mgrp_holder = sa->p_subn->mgroup_holders[i]; > + if (!p_mgrp_holder) > + continue; > + p_fitem = cl_fmap_head(&p_mgrp_holder->mgrp_map); > + while (p_fitem != cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + p_mgrp = > + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, > + mgid_item); > mcmr_by_comp_mask(sa, p_rcvd_rec, comp_mask, p_mgrp, > p_req_physp, trusted_req, &rec_list); > + p_fitem = cl_fmap_next(p_fitem); > + } > } > > CL_PLOCK_RELEASE(sa->p_lock); > diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c > index 9b50deb..3271289 100644 > --- a/opensm/opensm/osm_sa_path_record.c > +++ b/opensm/opensm/osm_sa_path_record.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -1438,6 +1438,7 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) > ib_path_rec_t *p_pr; > const ib_sa_mad_t *p_sa_mad; > ib_net64_t comp_mask; > + osm_mgrp_holder_t *p_mgrp_holder; > osm_mgrp_t *mgrp = NULL; > > p_sa_mad = osm_madw_get_sa_mad_ptr(p_madw); > @@ -1468,11 +1469,30 @@ static osm_mgrp_t *pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) > mgrp = NULL; > goto Exit; > } > - } else > - if (!(mgrp = osm_get_mgrp_by_mlid(sa->p_subn, p_pr->dlid))) > - OSM_LOG(sa->p_log, OSM_LOG_ERROR, > - "ERR 1F11: " "No MC group found for PathRecord " > - "destination LID 0x%x\n", p_pr->dlid); > + } else { > + cl_fmap_item_t *p_fitem; > + p_mgrp_holder = > + osm_get_mgrp_holder_by_mlid(sa->p_subn, p_pr->dlid); > + if (p_mgrp_holder) { > + p_fitem = > + cl_fmap_get(&p_mgrp_holder->mgrp_map, > + &p_pr->dgid); > + } > + if (!p_mgrp_holder > + || p_fitem == > + cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, > + "ERR 1F11: " > + "No MC group found for PathRecord " > + "destination LID 0x%x\n", p_pr->dlid); > + } else { > + mgrp = > + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, > + osm_mgrp_t, > + mgid_item); > + } > + > + } > } > > Exit: > diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c > index daa60ff..459e2cc 100644 > --- a/opensm/opensm/osm_sm.c > +++ b/opensm/opensm/osm_sm.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -47,6 +47,7 @@ > > #include > #include > +#include > #include > #include > #include > @@ -468,12 +469,15 @@ static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm, > /********************************************************************** > **********************************************************************/ > ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > - IN const ib_net64_t port_guid) > + IN const ib_net64_t port_guid, > + IN const ib_gid_t * p_mgid) > { > - osm_mgrp_t *p_mgrp; > + osm_mgrp_t *p_mgrp = NULL; > osm_port_t *p_port; > ib_api_status_t status = IB_SUCCESS; > osm_mcm_info_t *p_mcm; > + cl_fmap_item_t *p_fitem; > + osm_mgrp_holder_t *p_mgrp_holder; > > OSM_LOG_ENTER(p_sm->p_log); > > @@ -498,8 +502,51 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > /* > * If this multicast group does not already exist, create it. > */ > - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); > - if (!p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { > + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); > + if (p_mgrp_holder) { > + char gid_str[INET6_ADDRSTRLEN]; > + if (osm_is_debug()) { > + size_t gr_count = > + cl_fmap_count(&p_mgrp_holder->mgrp_map); > + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, > + "mlid 0x%X has %d mgroups\n", cl_ntoh16(mlid), > + gr_count); > + if (gr_count) { > + p_fitem = > + cl_fmap_head(&p_mgrp_holder->mgrp_map); > + while (p_fitem != > + cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + p_mgrp = > + (osm_mgrp_t *) > + PARENT_STRUCT(p_fitem, osm_mgrp_t, > + mgid_item); > + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, > + "mlid 0x%X has mgrp with MGID: %s\n", > + cl_ntoh16(mlid), > + inet_ntop(AF_INET6, > + p_mgrp->mcmember_rec. > + mgid.raw, gid_str, > + sizeof gid_str)); > + p_fitem = cl_fmap_next(p_fitem); > + } > + } > + } > + p_fitem = cl_fmap_get(&p_mgrp_holder->mgrp_map, p_mgid); > + if (p_fitem == cl_fmap_end(&p_mgrp_holder->mgrp_map)) { > + p_mgrp = NULL; > + OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, > + "group with MGID: %s not found on mlid 0x%X\n", > + inet_ntop(AF_INET6, > + p_mgid->raw, > + gid_str, sizeof gid_str), > + cl_ntoh16(mlid)); > + } else { > + p_mgrp = > + (osm_mgrp_t *) PARENT_STRUCT(p_fitem, osm_mgrp_t, > + mgid_item); > + } > + } > + if (!p_mgrp_holder || !p_mgrp || !osm_mgrp_is_guid(p_mgrp, port_guid)) { > /* > * The group removed or the port is not a > * member of the group, then fail immediately. > @@ -514,7 +561,23 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > status = IB_NOT_FOUND; > goto Exit; > } > - > + /* if there was no change from the last time > + * we processed the group we can skip doing anything > + */ > + if (p_mgrp_holder->last_change_id == p_mgrp_holder->last_tree_id) { > + CL_PLOCK_RELEASE(p_sm->p_lock); > + OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, > + "Skip processing mgrp holder with lid:0x%X last change id:%u\n", > + cl_ntoh16(mlid), p_mgrp_holder->last_change_id); > + goto Exit; > + } else { > + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, > + "processing mgrp holder with lid:0x%X port: 0x%016" > + PRIx64 " last change id:%u tree id:%u\n", > + cl_ntoh16(mlid), cl_ntoh64(port_guid), > + p_mgrp_holder->last_change_id, > + p_mgrp_holder->last_tree_id); > + } > /* > * Check if the object (according to mlid) already exists on this port. > * If it does - then no need to update it again, and no need to > @@ -543,9 +606,6 @@ ib_api_status_t osm_sm_mcgrp_join(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > goto Exit; > } > > - status = sm_mgrp_process(p_sm, p_mgrp); > - CL_PLOCK_RELEASE(p_sm->p_lock); > - > Exit: > OSM_LOG_EXIT(p_sm->p_log); > return status; > @@ -553,12 +613,13 @@ Exit: > > /********************************************************************** > **********************************************************************/ > -ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > +ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN osm_mgrp_t * p_mgrp, > IN const ib_net64_t port_guid) > { > - osm_mgrp_t *p_mgrp; > osm_port_t *p_port; > ib_api_status_t status; > + osm_mgrp_holder_t *p_mgrp_holder; > + ib_net16_t mlid = p_mgrp->mlid; > > OSM_LOG_ENTER(p_sm->p_log); > > @@ -584,8 +645,8 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > /* > * Get the multicast group object for this group. > */ > - p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); > - if (!p_mgrp) { > + p_mgrp_holder = osm_get_mgrp_holder_by_mlid(p_sm->p_subn, mlid); > + if (!p_mgrp_holder) { > CL_PLOCK_RELEASE(p_sm->p_lock); > OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " > "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); > @@ -593,11 +654,14 @@ ib_api_status_t osm_sm_mcgrp_leave(IN osm_sm_t * p_sm, IN const ib_net16_t mlid, > goto Exit; > } > > + osm_mgrp_holder_delete_mgrp_port(p_mgrp_holder, p_mgrp, port_guid); > /* > * Walk the list of ports in the group, and remove the appropriate one. > */ > osm_port_remove_mgrp(p_port, mlid); > - > + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG, > + " Calling sm_mgrp_process for mgrp with mlid = 0x%X\n", > + cl_ntoh16(mlid)); > status = sm_mgrp_process(p_sm, p_mgrp); > CL_PLOCK_RELEASE(p_sm->p_lock); > > diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c > index ec15f8a..e6e624d 100644 > --- a/opensm/opensm/osm_subnet.c > +++ b/opensm/opensm/osm_subnet.c > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > @@ -419,7 +419,7 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) > osm_switch_t *p_sw, *p_next_sw; > osm_remote_sm_t *p_rsm, *p_next_rsm; > osm_prtn_t *p_prtn, *p_next_prtn; > - osm_mgrp_t *p_mgrp; > + osm_mgrp_holder_t *p_osm_mgrp_holder; > osm_infr_t *p_infr, *p_next_infr; > > /* it might be a good idea to de-allocate all known objects */ > @@ -464,10 +464,10 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) > > for (i = 0; i <= p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; > i++) { > - p_mgrp = p_subn->mgroups[i]; > - p_subn->mgroups[i] = NULL; > - if (p_mgrp) > - osm_mgrp_delete(p_mgrp); > + p_osm_mgrp_holder = p_subn->mgroup_holders[i]; > + if (p_osm_mgrp_holder){ > + osm_mgrp_holder_delete(p_subn, p_osm_mgrp_holder->mlid); > + } > } > > p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); From tziporet at dev.mellanox.co.il Sun Jun 28 08:12:25 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 28 Jun 2009 18:12:25 +0300 Subject: [ofa-general] Cannot bring up mellanox HCAs on HP Nahalem machines running RHEL 5 In-Reply-To: References: Message-ID: <4A478859.8020600@mellanox.co.il> Mike Heinz wrote: > We've open an OFED PR ( https://bugs.openfabrics.org/show_bug.cgi?id=1663 ) > on this issue, but I'm wondering if anyone else has seen this: We just got > some brand new HP "gen6" machines - and we cannot get any Mellanox HCA, DDR > or QDR to come up on those machines - if they are running RHEL 5.3. Machines > running SLES11 appear to work fine. > > The same HCAs work fine on slower machines and on machines running SLES11 - > has anyone else seen this? > > We're running OFED 1.4.1 GA. > > This should be a firmware issue Please contact your Mellanox support to get a FW with a fix Tziporet From tziporet at dev.mellanox.co.il Sun Jun 28 08:16:32 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 28 Jun 2009 18:16:32 +0300 Subject: [ofa-general] OFEM installation problem In-Reply-To: <200906252040.39051.bsl@mek.dtu.dk> References: <200906252040.39051.bsl@mek.dtu.dk> Message-ID: <4A478950.4000005@mellanox.co.il> Boyan Lazarov wrote: > Hello, > > We have installed OFED 1.4.1 on Scientific Linux 53 (RH derivative). When I > run the osu_* tests (OpenMPI) as root it seems that everything is running > fine and the mpi is using openib for communications. However if I run the > test as an ordinary user I am getting the following: > > WARNING: There was an error initializing an OpenFabrics device. > > and openmpi is using tcp for communications - the speed is much lower. > > Anyone with similar problem and any suggestions? > > Can you send output of /var/log/messages? Also it could be that you have this problem: 2. Memory registration by the user is limited according to administrator setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for system configuration. =============================================================================== 3. Pinning (Locking) User Memory Pages =============================================================================== Memory locking is managed by the kernel on a per user basis. Regular users (as opposed to root) have a limited number of pages which they may pin, where the limit is pre-set by the administrator. Registering memory for IB verbs requires pinning memory, thus an application cannot register more memory than it is allowed to pin. The user can change the system per-process memory lock limit by adding the following two lines to the file /etc/security/limits.conf: * soft memlock * hard memlock where denotes the number of KBytes that may be locked by a user process. The above change to /etc/security/limits.conf will allow any user process in the system to lock up to KBytes of memory. On some systems, it may be possible to use "unlimited" for the size to disable these limits entirely. Note: The file /etc/security/limits.conf contains further documentation. Tziporet From monis at Voltaire.COM Sun Jun 28 09:09:57 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Sun, 28 Jun 2009 19:09:57 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906281117.34440.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> Message-ID: <4A4795D5.5040808@Voltaire.COM> Jack Morgenstein wrote: > We have seen the following kernel Oops on IPoIB: > ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22 > Unable to handle kernel paging request for data at address 0x00000054 > adFaulting instruction address: 0xe60b43c4 > Oops: Kernel access of bad area, sig: 11 [#1] > ... > NIP [e60b43c4] ib_sa_free_multicast+0x14/0x148 [ib_sa] > LR [e601fa24] ipoib_mcast_leave+0x140/0x144 [ib_ipoib] > Call Trace: > [de6a1ce0] [1000d3c0] 0x1000d3c0 (unreliable) > [de6a1d00] [e601fa24] ipoib_mcast_leave+0x140/0x144 [ib_ipoib] > [de6a1d60] [e6020904] ipoib_mcast_dev_flush+0x164/0x1a4 [ib_ipoib] > [de6a1db0] [e601e2a4] ipoib_ib_dev_down+0x78/0x130 [ib_ipoib] > [de6a1dd0] [e601c030] ipoib_stop+0xec/0x19c [ib_ipoib] > [de6a1df0] [c01b0e14] dev_close+0x88/0xd8 > [de6a1e00] [c01b0c84] dev_change_flags+0x154/0x1a8 > [de6a1e20] [c01f6144] devinet_ioctl+0x62c/0x81c > [de6a1e90] [c01f69fc] inet_ioctl+0xcc/0xf8 > [de6a1ea0] [c01a12f0] sock_ioctl+0x60/0x2ec > [de6a1ec0] [c008b1d4] vfs_ioctl+0x40/0xc0 > [de6a1ee0] [c008b580] do_vfs_ioctl+0x32c/0x498 > [de6a1f10] [c008b72c] sys_ioctl+0x40/0x74 > [de6a1f40] [c000e780] ret_from_syscall+0x0/0x3c > > Scenario was that someone performed "ifconfig ib0 down" while IPoIB was coming up. > Problem: a race-condition hole: > > - procedure ipoib_mcast_join(): > set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); > mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, > &rec, comp_mask, GFP_KERNEL, > ipoib_mcast_join_complete, mcast); > if (IS_ERR(mcast->mc)) { > clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); > > - procedure ipoib_mcast_leave(): > if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) > ib_sa_free_multicast(mcast->mc); > > Problem: IPOIB_MCAST_FLAG_BUSY is set in join() before the mcast->mc pointer is valid. If leave() is called > after the busy-flag is set, but before ib_sa_join_multicast returns, mcast->mc will not be valid > (and ib_sa_free_multicast() assumes that mcast->mc is valid). > > In fact, the "hole" is as follows: > set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); > ******* hole starts here *********** > mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, > &rec, comp_mask, GFP_KERNEL, > ipoib_mcast_join_complete, mcast); > if (IS_ERR(mcast->mc)) { > clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); > ******* hole ends here *********** > > I don't yet see a clean fix for this. We need the busy flag set where it is so that we do not do multiple joins, > and do not do ipoib_send(). Using another flag, say IPOIB_MCAST_FLAG_VALID, and setting it after mcast->mc > is successfully assigned (and testing for that in leave() ) is a problem, because it may lead to memory leaks. > We basically need atomicity here, and spinlocks are not an option. > > Any ideas? > -Jack > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > maybe synchronizing the race with a completion var (like IPoIB does in struct ipoib_path) will help. I think this will work. I can send a patch if you want unless you see this idea doesn't work for this case. MoniS From jackm at dev.mellanox.co.il Sun Jun 28 09:35:20 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 28 Jun 2009 19:35:20 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A4795D5.5040808@Voltaire.COM> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A4795D5.5040808@Voltaire.COM> Message-ID: <200906281935.21124.jackm@dev.mellanox.co.il> On Sunday 28 June 2009 19:09, Moni Shoua wrote: > maybe synchronizing the race with a completion var > (like IPoIB does in struct ipoib_path) will help. I think this will work. > I can send a patch if you want unless you see this idea doesn't work for this case. > Please do send a patch. Thanks! -Jack From friedman at ucla.edu Sun Jun 28 11:03:10 2009 From: friedman at ucla.edu (Scott A. Friedman) Date: Sun, 28 Jun 2009 11:03:10 -0700 Subject: [ofa-general] verb level interoperability between vendor's hcas In-Reply-To: References: <4A456EB0.5050102@ucla.edu> Message-ID: <4A47B05E.3000206@ucla.edu> We have had several tickets submitted by users since we have started adding Qlogic 7240 cards into our cluster which is mostly Mellanox (we have a couple different cards). We have looked at the codes (MPI based) and they do run fine when the Qlogic cards are excluded. Qlogic suggests using PSM or IPoIB on our cluster - both of which seem like a punt to us as PSM doesn't make sense with Mellanox and IPofIB is not a solution. Right now, we are trying to figure out where the problem is - it is not at the application level as we have distilled down to a specific case which will cause a problem (MPI all-to-all, for example). However, some things seem clearer to us. 1. test case works when using verbs using Mellanox only 2. test case works ok when we use PSM on Qlogic only 3. test case fails when using verbs between Mellanox and Qlogic 4. test case fails when using verbs on Qlogic Is this a verb level issue with the ipath stuff or an mpi problem? Or, is the issue someplace else? There had been some discussion of a mixed environment early this year on the OMPI list but the thread petered out. We would be happy to share our failing test case with whomever does the interop testing - if it could shed some light on the problem we see. The point is that we would like to know that different IB cards work together (like ethernet) so we can have a choice. Sean Hefty wrote: >> Is a mixed HCA environment cluster not ready for prime time - yet? > > Are the crashes in the kernel or userspace? Is there a specific HCA on the > nodes that crash? > > Interop testing is done, but I do not know the details of the configurations and > tests that are run. > From jackm at dev.mellanox.co.il Sun Jun 28 11:18:22 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 28 Jun 2009 21:18:22 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A4795D5.5040808@Voltaire.COM> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A4795D5.5040808@Voltaire.COM> Message-ID: <200906282118.22534.jackm@dev.mellanox.co.il> On Sunday 28 June 2009 19:09, Moni Shoua wrote: > maybe synchronizing the race with a completion var (like IPoIB does in struct ipoib_path) will help. I think this will work. I can send a patch if you want unless you see this idea doesn't work for this case. > > MoniS I just looked at the ipoib_path struct. I assume that you mean the "valid" field. I do not think this will work here. We need to wait when doing ifconfig ib0 down before testing the busy flag: *** We need to wait at this point in ipoib_mcast_leave() before testing FLAG_BUSY ***       set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags);       mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port,                                        &rec, comp_mask, GFP_KERNEL,                                        ipoib_mcast_join_complete, mcast);       if (IS_ERR(mcast->mc)) {               clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); *** Need to wait until this point -- the above set_bit/clear_bit pair needs to be atomic .... } else { *** Wait until this point in ipoib_mcast_leave() so that the busy flag is set, and mcast->mc is successfully assigned -- i.e., non-null **** } in ipoib_mcast_leave(): *** NEED TO WAIT HERE BEFORE CONTINUING (so that BUSY is cleared (mcast->mc is in error), *** or BUSY flag is set and mcast->mc is a valid, non-NULL pointer ****         if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags))               ib_sa_free_multicast(mcast->mc); - Jack From yossi.openib at gmail.com Sun Jun 28 13:04:53 2009 From: yossi.openib at gmail.com (Yossi Etigin) Date: Sun, 28 Jun 2009 23:04:53 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906282118.22534.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A4795D5.5040808@Voltaire.COM> <200906282118.22534.jackm@dev.mellanox.co.il> Message-ID: <4A47CCE5.4010408@gmail.com> Jack Morgenstein wrote: > in ipoib_mcast_leave(): > *** NEED TO WAIT HERE BEFORE CONTINUING (so that BUSY is cleared (mcast->mc is in error), > *** or BUSY flag is set and mcast->mc is a valid, non-NULL pointer **** > if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) > ib_sa_free_multicast(mcast->mc); > > - Jack How about making the leave/free mcast operation take place on the ipoib_workqueue, on which the join operation takes place? this way we can avoid this race, and more potential races of this kind. --Yossi From jackm at dev.mellanox.co.il Sun Jun 28 21:01:10 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 29 Jun 2009 07:01:10 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A47CCE5.4010408@gmail.com> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906282118.22534.jackm@dev.mellanox.co.il> <4A47CCE5.4010408@gmail.com> Message-ID: <200906290701.10656.jackm@dev.mellanox.co.il> On Sunday 28 June 2009 23:04, Yossi Etigin wrote: > Jack Morgenstein wrote: > > in ipoib_mcast_leave(): > > *** NEED TO WAIT HERE BEFORE CONTINUING (so that BUSY is cleared (mcast->mc is in error), > > *** or BUSY flag is set and mcast->mc is a valid, non-NULL pointer **** > > if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) > > ib_sa_free_multicast(mcast->mc); > > > > - Jack > > How about making the leave/free mcast operation take place on the ipoib_workqueue, on which > the join operation takes place? this way we can avoid this race, and more potential races > of this kind. > > --Yossi > > It may be possible. The only place where the leave does not take place on the ipoib_workqueue is ipoib_stop (where, of course, the crash occurs). Maybe we can transfer some of the stop operation to the workqueue? static int ipoib_stop(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); ipoib_dbg(priv, "stopping interface\n"); clear_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags); napi_disable(&priv->napi); netif_stop_queue(dev); ************* Transfer to work queue starting here **************** ipoib_ib_dev_down(dev, 0); ipoib_ib_dev_stop(dev, 0); if (!test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) { struct ipoib_dev_priv *cpriv; /* Bring down any child interfaces too */ mutex_lock(&priv->vlan_mutex); list_for_each_entry(cpriv, &priv->child_intfs, list) { int flags; flags = cpriv->dev->flags; if (!(flags & IFF_UP)) continue; dev_change_flags(cpriv->dev, flags & ~IFF_UP); } mutex_unlock(&priv->vlan_mutex); } ****************** transfer to here ******* return 0; } Or, maybe change the order of things as well, as below: static int ipoib_stop(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); ipoib_dbg(priv, "stopping interface\n"); clear_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags); napi_disable(&priv->napi); netif_stop_queue(dev); if (!test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) { struct ipoib_dev_priv *cpriv; /* Bring down any child interfaces too */ mutex_lock(&priv->vlan_mutex); list_for_each_entry(cpriv, &priv->child_intfs, list) { int flags; flags = cpriv->dev->flags; if (!(flags & IFF_UP)) continue; dev_change_flags(cpriv->dev, flags & ~IFF_UP); } mutex_unlock(&priv->vlan_mutex); } ************* Transfer to work queue starting here **************** ipoib_ib_dev_down(dev, 0); ipoib_ib_dev_stop(dev, 0); ****************** transfer to here ******* return 0; } Yossi, Roland, what do you think? From jackm at dev.mellanox.co.il Sun Jun 28 21:14:22 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 29 Jun 2009 07:14:22 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A47CCE5.4010408@gmail.com> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906282118.22534.jackm@dev.mellanox.co.il> <4A47CCE5.4010408@gmail.com> Message-ID: <200906290714.22571.jackm@dev.mellanox.co.il> On Sunday 28 June 2009 23:04, Yossi Etigin wrote: > How about making the leave/free mcast operation take place on the ipoib_workqueue, on which > the join operation takes place? this way we can avoid this race, and more potential races > of this kind. > On second thought, maybe it would be simpler to just create an ipoib_stop_task(), and do everything ipoib_stop() does in that workqueue task. leave would thus always be executed in the workqueue. Thoughts? From jackm at dev.mellanox.co.il Sun Jun 28 23:26:55 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 29 Jun 2009 09:26:55 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906290714.22571.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A47CCE5.4010408@gmail.com> <200906290714.22571.jackm@dev.mellanox.co.il> Message-ID: <200906290926.56014.jackm@dev.mellanox.co.il> On Monday 29 June 2009 07:14, Jack Morgenstein wrote: > > > On second thought, maybe it would be simpler to just create an ipoib_stop_task(), > and do everything ipoib_stop() does in that workqueue task. leave would thus always > be executed in the workqueue. > > Thoughts? > Looked at this again. There is a problem with this approach -- If I do ipoib_stop in the workqueue, I need to do ipoib_open in the workqueue as well. In ipoib_open, I will lose all the error returns if I take this approach: int ipoib_open(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); ipoib_dbg(priv, "bringing up interface\n"); napi_enable(&priv->napi); set_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags); if (ipoib_pkey_dev_delay_open(dev)) return 0; if (ipoib_ib_dev_open(dev)) { napi_disable(&priv->napi); return -EINVAL; } if (ipoib_ib_dev_up(dev)) { ipoib_ib_dev_stop(dev, 1); napi_disable(&priv->napi); return -EINVAL; } if (!test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) { struct ipoib_dev_priv *cpriv; /* Bring up any child interfaces too */ mutex_lock(&priv->vlan_mutex); list_for_each_entry(cpriv, &priv->child_intfs, list) { int flags; flags = cpriv->dev->flags; if (flags & IFF_UP) continue; dev_change_flags(cpriv->dev, flags | IFF_UP); } mutex_unlock(&priv->vlan_mutex); } netif_start_queue(dev); return 0; } I'm very worried that if we do things asymmetrically we will introduce instability. Regarding doing only the multicast leave/free in the workqueue -- I do not see how to do this cleanly (i.e., everything as is, but simply scheduling a leave-free task). Yossi, Roland, Eli? -Jack From acceptany at gmail.com Sun Jun 28 23:45:27 2009 From: acceptany at gmail.com (Jordan) Date: Mon, 29 Jun 2009 14:45:27 +0800 Subject: [ofa-general] Do all the ports within the same switch have the same LID? Message-ID: <91fe68d50906282345t243315f8x80d84ae4e0202a5c@mail.gmail.com> Since a switch only has one LID, the ports binding to it have the same LID , Is this right ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From monis at Voltaire.COM Mon Jun 29 00:27:25 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Mon, 29 Jun 2009 10:27:25 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906282118.22534.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A4795D5.5040808@Voltaire.COM> <200906282118.22534.jackm@dev.mellanox.co.il> Message-ID: <4A486CDD.9020902@Voltaire.COM> Jack Morgenstein wrote: > On Sunday 28 June 2009 19:09, Moni Shoua wrote: >> maybe synchronizing the race with a completion var (like IPoIB does in struct ipoib_path) will help. I think this will work. I can send a patch if you want unless you see this idea doesn't work for this case. >> >> MoniS > I just looked at the ipoib_path struct. I assume that you mean the "valid" field. I do not think this will work here. > We need to wait when doing ifconfig ib0 down before testing the busy flag: I actually meant the complete field. I'll think again and try to come up with a suggestion with a patch. From acceptany at gmail.com Mon Jun 29 01:25:49 2009 From: acceptany at gmail.com (Jordan) Date: Mon, 29 Jun 2009 16:25:49 +0800 Subject: [ofa-general] Some problem about the LID selection. Message-ID: <91fe68d50906290125ube7b08bu6af898b314ae4b4a@mail.gmail.com> When LMC > 0 , there are 2^LMC LIDs assigned to a port. When choosing different LID, there maybe different paths to get to this port. In other words, different LIDs represent different paths. But which path should I choose to get to the destination? It seems that OpenSM does not provide such algorithm . Does OFED provide such algorithm to manage which path should to choose ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at lists.openfabrics.org Mon Jun 29 02:25:40 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 29 Jun 2009 02:25:40 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090629-0200 daily build status Message-ID: <20090629092540.6EDCD10201C9@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090629-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From dorfman.eli at gmail.com Mon Jun 29 03:05:43 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 29 Jun 2009 13:05:43 +0300 Subject: [ofa-general] Xmit_wait counters in OpenSM In-Reply-To: <4A38E59B.4020000@ext.bull.net> References: <4A38E59B.4020000@ext.bull.net> Message-ID: <4A4891F7.2070907@gmail.com> Nicolas Morey-Chaisemartin wrote: > Hi everyone, > > I was having a look at congestion management in IB when I realized OpenSM does not support Xmit_wait counters. > Is there any good reason why? > I'm ready to give it a try but I'd like to be sure there is no contraindications. > Xmit_wait counters are supported for ConnectX HCA and IS4 switches with the following patch: commit d3d073e2ff76dc7b757b6d73839bd65cbc712986 Author: Eli Dorfman (Voltaire) Date: Tue Jan 13 16:56:57 2009 +0200 infiniband-diags support PortXmitWait get and set support PortXmitWait get and set Signed-off-by: Eli Dorfman Signed-off-by: Sasha Khapyorsky From jsquyres at cisco.com Mon Jun 29 03:55:54 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 29 Jun 2009 06:55:54 -0400 Subject: [ofa-general] verb level interoperability between vendor's hcas In-Reply-To: <4A47B05E.3000206@ucla.edu> References: <4A456EB0.5050102@ucla.edu> <4A47B05E.3000206@ucla.edu> Message-ID: There are definite things that won't work by default in Open MPI with mixed vendor HCAs. As you mentioned, the Open MPI IB vendors committed to making this work, but the effort kinda died. You might want to ping them again to remind them...? On Jun 28, 2009, at 2:03 PM, Scott A. Friedman wrote: > We have had several tickets submitted by users since we have started > adding Qlogic 7240 cards into our cluster which is mostly Mellanox (we > have a couple different cards). We have looked at the codes (MPI > based) > and they do run fine when the Qlogic cards are excluded. Qlogic > suggests > using PSM or IPoIB on our cluster - both of which seem like a punt > to us > as PSM doesn't make sense with Mellanox and IPofIB is not a solution. > > Right now, we are trying to figure out where the problem is - it is > not > at the application level as we have distilled down to a specific case > which will cause a problem (MPI all-to-all, for example). However, > some > things seem clearer to us. > > 1. test case works when using verbs using Mellanox only > 2. test case works ok when we use PSM on Qlogic only > 3. test case fails when using verbs between Mellanox and Qlogic > 4. test case fails when using verbs on Qlogic > > Is this a verb level issue with the ipath stuff or an mpi problem? Or, > is the issue someplace else? There had been some discussion of a mixed > environment early this year on the OMPI list but the thread petered > out. > > We would be happy to share our failing test case with whomever does > the > interop testing - if it could shed some light on the problem we see. > > The point is that we would like to know that different IB cards work > together (like ethernet) so we can have a choice. > > Sean Hefty wrote: > >> Is a mixed HCA environment cluster not ready for prime time - yet? > > > > Are the crashes in the kernel or userspace? Is there a specific > HCA on the > > nodes that crash? > > > > Interop testing is done, but I do not know the details of the > configurations and > > tests that are run. > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > -- Jeff Squyres Cisco Systems From sashak at voltaire.com Mon Jun 29 03:58:12 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 29 Jun 2009 13:58:12 +0300 Subject: [ofa-general] [PATCH] complib/Makefile.am: prevent file duplications Message-ID: <20090629105812.GF5307@me> After upgrading to 64-bit Gentoo with recent auto/etc-tools I've got this error: usr/bin/install: will not overwrite just-created `/usr/local/include/infiniband/complib/cl_threadpool.h' with `./../include/complib/cl_threadpool.h' /usr/bin/install: will not overwrite just-created `/usr/local/include/infiniband/complib/cl_timer.h' with `./../include/complib/cl_timer.h' /usr/bin/install: will not overwrite just-created `/usr/local/include/infiniband/complib/cl_timer_osd.h' with `./../include/complib/cl_timer_osd.h' /usr/bin/install: will not overwrite just-created `/usr/local/include/infiniband/complib/cl_types.h' with `./../include/complib/cl_types.h' /usr/bin/install: will not overwrite just-created `/usr/local/include/infiniband/complib/cl_types_osd.h' with `./../include/complib/cl_types_osd.h' make[2]: *** [install-libosmcompincludeHEADERS] Error 1 make[2]: Leaving directory `/home/sashak/src/m/opensm/complib' make[1]: *** [install-am] Error 2 make[1]: Leaving directory `/home/sashak/src/m/opensm/complib' make: *** [install-recursive] Error 1 , which is resulted by duplicated listing of those filei in libosmcompinclude_HEADERS definition - cleaning this up. Signed-off-by: Sasha Khapyorsky --- opensm/complib/Makefile.am | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/opensm/complib/Makefile.am b/opensm/complib/Makefile.am index 6f2f203..3ee90e8 100644 --- a/opensm/complib/Makefile.am +++ b/opensm/complib/Makefile.am @@ -70,11 +70,6 @@ libosmcompinclude_HEADERS = $(srcdir)/../include/complib/cl_atomic.h \ $(srcdir)/../include/complib/cl_timer_osd.h \ $(srcdir)/../include/complib/cl_types.h \ $(srcdir)/../include/complib/cl_types_osd.h \ - $(srcdir)/../include/complib/cl_threadpool.h \ - $(srcdir)/../include/complib/cl_timer.h \ - $(srcdir)/../include/complib/cl_timer_osd.h \ - $(srcdir)/../include/complib/cl_types.h \ - $(srcdir)/../include/complib/cl_types_osd.h \ $(srcdir)/../include/complib/cl_vector.h # headers are distributed as part of the include dir -- 1.6.3.3 From myxiplx at googlemail.com Mon Jun 29 04:00:51 2009 From: myxiplx at googlemail.com (Ross Smith) Date: Mon, 29 Jun 2009 12:00:51 +0100 Subject: [ofa-general] verb level interoperability between vendor's hcas In-Reply-To: <4A47B05E.3000206@ucla.edu> References: <4A456EB0.5050102@ucla.edu> <4A47B05E.3000206@ucla.edu> Message-ID: <7b160d240906290400r5ecb1881gaa854fc5ad0a57ab@mail.gmail.com> Guys, I might be misunderstanding, but: > 1. test case works when using verbs using Mellanox only > 2. test case works ok when we use PSM on Qlogic only > 3. test case fails when using verbs between Mellanox and Qlogic > 4. test case fails when using verbs on Qlogic Doesn't this show that verbs / MPI fail full stop with Qlogic? That's doesn't look like vendor interoperability so much as a bug with verbs and Qlogic hardware. Ross (Infiniband newbie) On Sun, Jun 28, 2009 at 7:03 PM, Scott A. Friedman wrote: > We have had several tickets submitted by users since we have started adding > Qlogic 7240 cards into our cluster which is mostly Mellanox (we have a > couple different cards). We have looked at the codes (MPI based) and they do > run fine when the Qlogic cards are excluded. Qlogic suggests using PSM or > IPoIB on our cluster - both of which seem like a punt to us as PSM doesn't > make sense with Mellanox and IPofIB is not a solution. > > Right now, we are trying to figure out where the problem is - it is not at > the application level as we have distilled down to a specific case which > will cause a problem (MPI all-to-all, for example). However, some things > seem clearer to us. > > 1. test case works when using verbs using Mellanox only > 2. test case works ok when we use PSM on Qlogic only > 3. test case fails when using verbs between Mellanox and Qlogic > 4. test case fails when using verbs on Qlogic > > Is this a verb level issue with the ipath stuff or an mpi problem? Or, is > the issue someplace else? There had been some discussion of a mixed > environment early this year on the OMPI list but the thread petered out. > > We would be happy to share our failing test case with whomever does the > interop testing - if it could shed some light on the problem we see. > > The point is that we would like to know that different IB cards work > together (like ethernet) so we can have a choice. > > Sean Hefty wrote: >>> >>> Is a mixed HCA environment cluster not ready for prime time - yet? >> >> Are the crashes in the kernel or userspace?  Is there a specific HCA on >> the >> nodes that crash? >> >> Interop testing is done, but I do not know the details of the >> configurations and >> tests that are run. > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From fenkes at de.ibm.com Mon Jun 29 05:10:30 2009 From: fenkes at de.ibm.com (Joachim Fenkes) Date: Mon, 29 Jun 2009 14:10:30 +0200 Subject: [ofa-general] [PATCH] libibmad: Handle MAD redirection Message-ID: <200906291410.33477.fenkes@de.ibm.com> Previously, libibmad reacted to GSI MAD responses with a "redirect" status by throwing an error. IBM eHCA adapters use redirection, so most infiniband_diags tools didn't work against eHCA. Fix: Modify mad_rpc() so that it resends the request to the redirection target if a "redirect" GS response is received. This is repeated until no "redirect" response is received, allowing for multiple levels of indirection. The dport argument is updated with the redirection target, so subsequent MADs will not go through the redirection process again but reach the target directly. Tested using perfquery between ehca, mlx4 and mthca in all possible combinations. Signed-off-by: Joachim Fenkes --- Vlad, please queue this patch for OFED 1.5 if Hal doesn't object -- thanks! Regards, Joachim libibmad/include/infiniband/mad.h | 9 +++++++ libibmad/src/gs.c | 6 +++- libibmad/src/rpc.c | 47 +++++++++++++++++++++++++----------- 3 files changed, 45 insertions(+), 17 deletions(-) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index aa27eb5..bdf5158 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -115,6 +115,8 @@ enum MAD_ATTR_ID { enum MAD_STATUS { IB_MAD_STS_OK = (0 << 2), + IB_MAD_STS_BUSY = (1 << 0), + IB_MAD_STS_REDIRECT = (1 << 1), IB_MAD_STS_BAD_BASE_VER_OR_CLASS = (1 << 2), IB_MAD_STS_METHOD_NOT_SUPPORTED = (2 << 2), IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), @@ -783,8 +785,15 @@ MAD_EXPORT int madrpc_set_timeout(int timeout); MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, int *mgmt_classes, int num_classes); MAD_EXPORT void mad_rpc_close_port(struct ibmad_port *srcport); + +/* + * On redirection, the dport argument is updated with the redirection target, + * so subsequent MADs will not go through the redirection process again but + * reach the target directly. + */ MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, void *payload, void *rcvdata); + MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *data); diff --git a/libibmad/src/gs.c b/libibmad/src/gs.c index f3d245e..c7e4ff6 100644 --- a/libibmad/src/gs.c +++ b/libibmad/src/gs.c @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, rpc.datasz = IB_PC_DATA_SZ; rpc.dataoffs = IB_PC_DATA_OFFS; - dest->qp = 1; + if (!dest->qp) + dest->qp = 1; if (!dest->qkey) dest->qkey = IB_DEFAULT_QP1_QKEY; @@ -109,7 +110,8 @@ uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, rpc.timeout = timeout; rpc.datasz = IB_PC_DATA_SZ; rpc.dataoffs = IB_PC_DATA_OFFS; - dest->qp = 1; + if (!dest->qp) + dest->qp = 1; if (!dest->qkey) dest->qkey = IB_DEFAULT_QP1_QKEY; diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 07b623d..c809060 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -189,27 +189,44 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, int status, len; uint8_t sndbuf[1024], rcvbuf[1024], *mad; int timeout, retries; + int redirect = 1; - len = 0; - memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); + while (redirect) { + len = 0; + memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); - if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) - return 0; + if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) + return 0; - timeout = rpc->timeout ? rpc->timeout : - port->timeout ? port->timeout : madrpc_timeout; - retries = port->retries ? port->retries : madrpc_retries; + timeout = rpc->timeout ? rpc->timeout : + port->timeout ? port->timeout : madrpc_timeout; + retries = port->retries ? port->retries : madrpc_retries; - if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, - port->class_agents[rpc->mgtclass], - len, timeout, retries)) < 0) { - IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); - return 0; - } + if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, + port->class_agents[rpc->mgtclass], + len, timeout, retries)) < 0) { + IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); + return 0; + } - mad = umad_get_mad(rcvbuf); + mad = umad_get_mad(rcvbuf); + status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); + + /* check for exact match instead of only the redirect bit; + * that way, weird statuses cause an error, too */ + if (status == IB_MAD_STS_REDIRECT) { + /* update dport for next request and retry */ + dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); + dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); + dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); + if (ibdebug) + IBWARN("redirected to lid 0x%x, qp 0x%x, qkey 0x%x, pkey 0x%x", + dport->lid, dport->qp, dport->qkey, dport->pkey_idx); + } else + redirect = 0; + } - if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { + if (status != 0) { ERRS("MAD completed with error status 0x%x; dport (%s)", status, portid2str(dport)); return 0; -- 1.5.5 From nicolas.morey-chaisemartin at ext.bull.net Mon Jun 29 06:48:06 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey-Chaisemartin) Date: Mon, 29 Jun 2009 15:48:06 +0200 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: Fixed issue with reverse_hops introduced by commit 1d7dd18b531c1d6370f80cc7303493d6f3e3e777. Message-ID: <4A48C616.1090402@ext.bull.net> When using reverse hop with nodes on root switches, origin switch is explored multiple times with wrong hop numbers creating loops in the network. This is fixed by checking before going up (even on the master path) that the remote switch is either not configured or with a longer path (second test was missing). Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 57 +++++++++++++++++++++++--------------- 1 files changed, 34 insertions(+), 23 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 6c695de..5a9eeff 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -2396,38 +2396,49 @@ fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, if (is_real_lid) { /* This LID may already be in the LFT in the reverse_hop feature is used */ /* We update the LFT only if this LID isn't already present. */ - if (p_remote_sw->p_osm_sw-> - new_lft[target_lid] == OSM_NO_PATH) { - p_remote_sw->p_osm_sw-> - new_lft[target_lid] = + + /* skip if target lid has been already set on remote switch fwd tbl (with a bigger hop count) */ + if ((p_remote_sw->p_osm_sw->new_lft[target_lid] == + OSM_NO_PATH) + || + ((p_remote_sw->p_osm_sw->new_lft[target_lid] != + OSM_NO_PATH) + && + ((target_rank - p_remote_sw->rank + + 2 * reverse_hops) < + sw_get_least_hops(p_remote_sw, target_lid)))) { + + p_remote_sw->p_osm_sw->new_lft[target_lid] = p_min_port->remote_port_num; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch %s: set path to CA LID %u through port %u\n", tuple_to_str(p_remote_sw->tuple), target_lid, p_min_port->remote_port_num); - } - /* On the remote switch that is pointed by the min_group, - set hops for ALL the ports in the remote group. */ - set_hops_on_remote_sw(p_min_group, target_lid, - target_rank - p_remote_sw->rank + - 2 * reverse_hops, is_target_a_sw); - } + /* On the remote switch that is pointed by the min_group, + set hops for ALL the ports in the remote group. */ - /* Recursion step: - Assign downgoing ports by stepping up, starting on REMOTE switch. */ - created_route |= fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ - reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ - } + set_hops_on_remote_sw(p_min_group, target_lid, + target_rank - + p_remote_sw->rank + + 2 * reverse_hops, + is_target_a_sw); + /* Recursion step: + Assign downgoing ports by stepping up, starting on REMOTE switch. */ + created_route |= fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ + reverse_hop_credit, /* Remaining reverse_hops allowed */ + reverse_hops); /* Number of reverse_hops done up to this point */ + } + } + } /* we're done for the third case */ if (!is_real_lid) return created_route; -- 1.6.3.1 From monis at Voltaire.COM Mon Jun 29 08:06:48 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Mon, 29 Jun 2009 18:06:48 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906282118.22534.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A4795D5.5040808@Voltaire.COM> <200906282118.22534.jackm@dev.mellanox.co.il> Message-ID: <4A48D888.1020601@Voltaire.COM> Jack Morgenstein wrote: > On Sunday 28 June 2009 19:09, Moni Shoua wrote: >> maybe synchronizing the race with a completion var (like IPoIB does in struct ipoib_path) will help. I think this will work. I can send a patch if you want unless you see this idea doesn't work for this case. >> >> MoniS What do you think of this (see below)? I used a completion struct to synchronize between leave and join in a similar way that ipoib_path is using the field 'done'. There is one difference though from the original code (I don't know if that's acceptable), here we wait for the join request to finish before we destroy the mcast struct and not canceling it. Please note that I didn't compile this code. I just want to hear opinions first before I make a real patch out of this. diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 753a983..a551258 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -145,6 +145,7 @@ struct ipoib_mcast { struct sk_buff_head pkt_queue; struct net_device *dev; + struct completion join_req; }; struct ipoib_rx_buf { diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c index a0e9753..afe72dd 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c @@ -117,6 +117,7 @@ static struct ipoib_mcast *ipoib_mcast_alloc(struct net_device *dev, mcast->dev = dev; mcast->created = jiffies; mcast->backoff = 1; + complete(&mcast->join_req); INIT_LIST_HEAD(&mcast->list); INIT_LIST_HEAD(&mcast->neigh_list); @@ -286,6 +287,7 @@ ipoib_mcast_sendonly_join_complete(int status, if (status == -ENETRESET) return 0; + complete(&mcast->join_req); if (!status) status = ipoib_mcast_join_finish(mcast, &multicast->rec); @@ -336,6 +338,7 @@ static int ipoib_mcast_sendonly_join(struct ipoib_mcast *mcast) rec.port_gid = priv->local_gid; rec.pkey = cpu_to_be16(priv->pkey); + init_completion(&mcast->join_req); mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, &rec, IB_SA_MCMEMBER_REC_MGID | @@ -384,6 +387,7 @@ static int ipoib_mcast_join_complete(int status, mcast->mcmember.mgid.raw, status); /* We trap for port events ourselves. */ + complete(&mcast->join_req); if (status == -ENETRESET) return 0; @@ -482,10 +486,12 @@ static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast, } set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); + init_completion(&mcast->join_req); mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, &rec, comp_mask, GFP_KERNEL, ipoib_mcast_join_complete, mcast); if (IS_ERR(mcast->mc)) { + complete(&mcast->join_req); clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); ret = PTR_ERR(mcast->mc); ipoib_warn(priv, "ib_sa_join_multicast failed, status %d\n", ret); @@ -630,8 +636,7 @@ static int ipoib_mcast_leave(struct net_device *dev, struct ipoib_mcast *mcast) struct ipoib_dev_priv *priv = netdev_priv(dev); int ret = 0; - if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) - ib_sa_free_multicast(mcast->mc); + wait_for_completion(mcast->join_req); if (test_and_clear_bit(IPOIB_MCAST_FLAG_ATTACHED, &mcast->flags)) { ipoib_dbg_mcast(priv, "leaving MGID %pI6\n", From tziporet at mellanox.co.il Mon Jun 29 08:27:05 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 29 Jun 2009 18:27:05 +0300 Subject: [ofa-general] EWG/OFED agenda for today (June 29) Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD0334A01D@mtlexch01.mtl.com> This is the agenda for today 1. OFED 1.5 feature freeze date All - please send what are the main features that are still under development and you wish to get them into 1.5 2. OFED 1.4.2 - there is a request from Sun to have bug fixes only release since there is a critical bug that prevent Lustre to run over OFED 1.4 and 1.4.1 3. RDMAoE 4. MPI in OFED: decision in XWG was not to change for 1.5 and bring it as a BOF in SC09 with the customers Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.j.woodruff at intel.com Mon Jun 29 09:00:59 2009 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Mon, 29 Jun 2009 09:00:59 -0700 Subject: [ofa-general] RE: EWG/OFED agenda for today (June 29) In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD0334A01D@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD0334A01D@mtlexch01.mtl.com> Message-ID: <382A478CAD40FA4FB46605CF81FE39F4356007BE@orsmsx507.amr.corp.intel.com> Can someone send the bridge information ? woody ________________________________ From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Tziporet Koren Sent: Monday, June 29, 2009 8:27 AM To: ewg at lists.openfabrics.org Cc: general at lists.openfabrics.org Subject: [ofa-general] EWG/OFED agenda for today (June 29) This is the agenda for today 1. OFED 1.5 feature freeze date All - please send what are the main features that are still under development and you wish to get them into 1.5 2. OFED 1.4.2 - there is a request from Sun to have bug fixes only release since there is a critical bug that prevent Lustre to run over OFED 1.4 and 1.4.1 3. RDMAoE 4. MPI in OFED: decision in XWG was not to change for 1.5 and bring it as a BOF in SC09 with the customers Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From yiftahs at voltaire.com Mon Jun 29 09:15:48 2009 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Mon, 29 Jun 2009 19:15:48 +0300 Subject: [ofa-general] RE: [ewg] RE: EWG/OFED agenda for today (June 29) In-Reply-To: <382A478CAD40FA4FB46605CF81FE39F4356007BE@orsmsx507.amr.corp.intel.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD0334A01D@mtlexch01.mtl.com> <382A478CAD40FA4FB46605CF81FE39F4356007BE@orsmsx507.amr.corp.intel.com> Message-ID: <39C75744D164D948A170E9792AF8E7CA025899D2@exil.voltaire.com> Unfortunately the below is not working for us :-( Moni S. & Yiftah **** HIT "3" WHEN YOU DIAL IN TO GET TO WEBEX!! **** HIT "3" WHEN YOU DIAL IN TO GET TO WEBEX!! Friendly reminder: the OFED teleconference is *today* (1 June 2009). Next few scheduled teleconferences: 1. Monday, June 1, noon US Eastern / 9am US Pacific / 7pm Israel 2. Monday, June 15, noon US Eastern / 9am US Pacific / 7pm Israel Dial in code: 203 675 633 Meeting password: OFED (should only be necessary if you use the web part; but the meeting will be audio only, so using the web part won't be useful) US/Canada: +1.866.432.9903 India: +91.80.4350.1111 Israel: +972.9.892.7026 ________________________________ From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Woodruff, Robert J Sent: Monday, June 29, 2009 19:01 To: Tziporet Koren; ewg at lists.openfabrics.org Cc: general at lists.openfabrics.org Subject: [ewg] RE: EWG/OFED agenda for today (June 29) Can someone send the bridge information ? woody ________________________________ From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Tziporet Koren Sent: Monday, June 29, 2009 8:27 AM To: ewg at lists.openfabrics.org Cc: general at lists.openfabrics.org Subject: [ofa-general] EWG/OFED agenda for today (June 29) This is the agenda for today 1. OFED 1.5 feature freeze date All - please send what are the main features that are still under development and you wish to get them into 1.5 2. OFED 1.4.2 - there is a request from Sun to have bug fixes only release since there is a critical bug that prevent Lustre to run over OFED 1.4 and 1.4.1 3. RDMAoE 4. MPI in OFED: decision in XWG was not to change for 1.5 and bring it as a BOF in SC09 with the customers Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at cisco.com Mon Jun 29 09:25:00 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 29 Jun 2009 12:25:00 -0400 Subject: [ofa-general] Re: [ewg] RE: EWG/OFED agenda for today (June 29) In-Reply-To: <39C75744D164D948A170E9792AF8E7CA025899D2@exil.voltaire.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD0334A01D@mtlexch01.mtl.com> <382A478CAD40FA4FB46605CF81FE39F4356007BE@orsmsx507.amr.corp.intel.com> <39C75744D164D948A170E9792AF8E7CA025899D2@exil.voltaire.com> Message-ID: Please look at the outlook invite that I sent yesterday and the reminder I sent after the meeting started this morning (sorry; I thought everyone would use the Outlook invite). http://lists.openfabrics.org/pipermail/ewg/2009-June/013462.html On Jun 29, 2009, at 12:15 PM, Yiftah Shahar wrote: > Unfortunately the below is not working for us L > Moni S. & Yiftah > > **** HIT "3" WHEN YOU DIAL IN TO GET TO WEBEX!! > **** HIT "3" WHEN YOU DIAL IN TO GET TO WEBEX!! > > Friendly reminder: the OFED teleconference is *today* (1 June 2009). > > Next few scheduled teleconferences: > > 1. Monday, June 1, noon US Eastern / 9am US Pacific / 7pm Israel > 2. Monday, June 15, noon US Eastern / 9am US Pacific / 7pm Israel > > Dial in code: 203 675 633 > > Meeting password: OFED (should only be necessary if you use the web > part; but the meeting will be audio only, so using the web part > won't be useful) > > US/Canada: +1.866.432.9903 > India: +91.80.4350.1111 > Israel: +972.9.892.7026 > > > From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org > ] On Behalf Of Woodruff, Robert J > Sent: Monday, June 29, 2009 19:01 > To: Tziporet Koren; ewg at lists.openfabrics.org > Cc: general at lists.openfabrics.org > Subject: [ewg] RE: EWG/OFED agenda for today (June 29) > > Can someone send the bridge information ? > woody > > > From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org > ] On Behalf Of Tziporet Koren > Sent: Monday, June 29, 2009 8:27 AM > To: ewg at lists.openfabrics.org > Cc: general at lists.openfabrics.org > Subject: [ofa-general] EWG/OFED agenda for today (June 29) > > This is the agenda for today > > 1. OFED 1.5 feature freeze date > > All - please send what are the main features that are still under > > development and you wish to get them into 1.5 > > 2. OFED 1.4.2 - there is a request from Sun to have bug fixes only > > release since there is a critical bug that prevent Lustre to run over > > OFED 1.4 and 1.4.1 > > 3. RDMAoE > > 4. MPI in OFED: decision in XWG was not to change for 1.5 and bring it > > as a BOF in SC09 with the customers > > Tziporet > -- Jeff Squyres Cisco Systems From sean.hefty at intel.com Mon Jun 29 10:54:50 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 29 Jun 2009 10:54:50 -0700 Subject: [ofa-general] [PATCH] dapl/windows: remove dlist.c Message-ID: <28A14EAB40AA4FA98D72BD0CDCD0D90C@amr.corp.intel.com> All dlist functions have been moved to the header file. Remove references to dlist.c. Signed-off-by: Sean Hefty --- dapl/openib_cma/dapl_ib_util.c | 1 - dapl/openib_scm/dapl_ib_cq.c | 1 - 2 files changed, 0 insertions(+), 2 deletions(-) diff --git a/dapl/openib_cma/dapl_ib_util.c b/dapl/openib_cma/dapl_ib_util.c index bf23d43..f48c1cb 100755 --- a/dapl/openib_cma/dapl_ib_util.c +++ b/dapl/openib_cma/dapl_ib_util.c @@ -56,7 +56,6 @@ struct dapl_llist_entry *g_hca_list; #if defined(_WIN64) || defined(_WIN32) #include "..\..\..\..\..\etc\user\comp_channel.cpp" -#include "..\..\..\..\..\etc\user\dlist.c" #include struct ibvw_windata windata; diff --git a/dapl/openib_scm/dapl_ib_cq.c b/dapl/openib_scm/dapl_ib_cq.c index 2af1889..8a9a2ab 100644 --- a/dapl/openib_scm/dapl_ib_cq.c +++ b/dapl/openib_scm/dapl_ib_cq.c @@ -55,7 +55,6 @@ #if defined(_WIN64) || defined(_WIN32) #include "..\..\..\..\..\etc\user\comp_channel.cpp" -#include "..\..\..\..\..\etc\user\dlist.c" void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr) { From hnrose at comcast.net Mon Jun 29 11:44:57 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 29 Jun 2009 14:44:57 -0400 Subject: [ofa-general] [PATCH] opensm/osm_switch.c: In osm_switch_prepare_path_rebuild, use realloc Message-ID: <20090629184457.GA13787@comcast.net> rather than malloc/memset/free Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index d97ec99..ce1ca63 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -529,18 +530,13 @@ osm_switch_prepare_path_rebuild(IN osm_switch_t * p_sw, IN uint16_t max_lids) p_sw->hops = hops; p_sw->num_hops = max_lids + 1; } else if (max_lids + 1 > p_sw->num_hops) { - uint8_t **old_hops; - - hops = malloc((max_lids + 1) * sizeof(hops[0])); + hops = realloc(p_sw->hops, (max_lids + 1) * sizeof(hops[0])); if (!hops) return -1; - memcpy(hops, p_sw->hops, p_sw->num_hops * sizeof(hops[0])); memset(hops + p_sw->num_hops, 0, (max_lids + 1 - p_sw->num_hops) * sizeof(hops[0])); - old_hops = p_sw->hops; p_sw->hops = hops; p_sw->num_hops = max_lids + 1; - free(old_hops); } p_sw->max_lid_ho = max_lids; From sashak at voltaire.com Mon Jun 29 12:02:29 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 29 Jun 2009 22:02:29 +0300 Subject: [ofa-general] Re: Missing printf() format specifier in osm_subnet.c In-Reply-To: <200906291301.36847.mbrown@fensystems.co.uk> References: <200906282305.58878.mbrown@fensystems.co.uk> <20090629103428.GE5307@me> <200906291301.36847.mbrown@fensystems.co.uk> Message-ID: <20090629190229.GG5307@me> [adding mailing list...] On 13:01 Mon 29 Jun , Michael Brown wrote: > > Thanks. I had the git://git.openfabrics.org/ofed_1_4/management.git tree > checked out, but not git://git.openfabrics.org/~sashak/management.git (which > I assumed was a personal, non-official tree). This is a master management repository. > Is there a definitive list anywhere of which trees on git.openfabrics.org are > the "official" trees? (i.e. if I want to test against or provide a patch > against the latest tree for some OpenFabrics component, how do I tell which > git tree I should be looking at?) This is in the wiki: https://wiki.openfabrics.org/tiki-index.php?page=Downloading+Code+From+the+OFA+git+Repositories , and IB management specific information: http://www.openfabrics.org/downloads/management/README And I agree that it is not easy to navigate to this from OFA front page. Tziporet, Jeff, Could we do more intuitive front page on the OFA site with direct links to installation guides and download pages? Sasha From jackm at dev.mellanox.co.il Mon Jun 29 12:29:30 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 29 Jun 2009 22:29:30 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A48D888.1020601@Voltaire.COM> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906282118.22534.jackm@dev.mellanox.co.il> <4A48D888.1020601@Voltaire.COM> Message-ID: <200906292229.31129.jackm@dev.mellanox.co.il> On Monday 29 June 2009 18:06, Moni Shoua wrote: > Jack Morgenstein wrote: > > On Sunday 28 June 2009 19:09, Moni Shoua wrote: > >> maybe synchronizing the race with a completion var (like IPoIB does in struct ipoib_path) will help. I think this will work. I can send a patch if you want unless you see this idea doesn't work for this case. > >> > >> MoniS > > What do you think of this (see below)? > I used a completion struct to synchronize between leave and join in a similar way that ipoib_path is using the field 'done'. > There is one difference though from the original code (I don't know if that's acceptable), here we wait for the join request to finish before we destroy the mcast struct and not canceling it. > Please note that I didn't compile this code. I just want to hear opinions first before I make a real patch out of this. > I have to think about this some. I'm a bit nervous about 2 things: 1. Somehow, the complete never arrives, and the "leave" gets stuck forever waiting. 2. What if someone does another ifconfig ib0 up while we are waiting for the complete? (I think it is disregarded, but I'm not sure -- with this change and all). Nice idea, though. -Jack > > diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h > index 753a983..a551258 100644 > --- a/drivers/infiniband/ulp/ipoib/ipoib.h > +++ b/drivers/infiniband/ulp/ipoib/ipoib.h > @@ -145,6 +145,7 @@ struct ipoib_mcast { > struct sk_buff_head pkt_queue; > > struct net_device *dev; > + struct completion join_req; > }; > > struct ipoib_rx_buf { > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c > index a0e9753..afe72dd 100644 > --- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c > +++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c > @@ -117,6 +117,7 @@ static struct ipoib_mcast *ipoib_mcast_alloc(struct net_device *dev, > mcast->dev = dev; > mcast->created = jiffies; > mcast->backoff = 1; > + complete(&mcast->join_req); > > INIT_LIST_HEAD(&mcast->list); > INIT_LIST_HEAD(&mcast->neigh_list); > @@ -286,6 +287,7 @@ ipoib_mcast_sendonly_join_complete(int status, > if (status == -ENETRESET) > return 0; > > + complete(&mcast->join_req); > if (!status) > status = ipoib_mcast_join_finish(mcast, &multicast->rec); > > @@ -336,6 +338,7 @@ static int ipoib_mcast_sendonly_join(struct ipoib_mcast *mcast) > rec.port_gid = priv->local_gid; > rec.pkey = cpu_to_be16(priv->pkey); > > + init_completion(&mcast->join_req); > mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, > priv->port, &rec, > IB_SA_MCMEMBER_REC_MGID | > @@ -384,6 +387,7 @@ static int ipoib_mcast_join_complete(int status, > mcast->mcmember.mgid.raw, status); > > /* We trap for port events ourselves. */ > + complete(&mcast->join_req); > if (status == -ENETRESET) > return 0; > > @@ -482,10 +486,12 @@ static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast, > } > > set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); > + init_completion(&mcast->join_req); > mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, > &rec, comp_mask, GFP_KERNEL, > ipoib_mcast_join_complete, mcast); > if (IS_ERR(mcast->mc)) { > + complete(&mcast->join_req); > clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); > ret = PTR_ERR(mcast->mc); > ipoib_warn(priv, "ib_sa_join_multicast failed, status %d\n", ret); > @@ -630,8 +636,7 @@ static int ipoib_mcast_leave(struct net_device *dev, struct ipoib_mcast *mcast) > struct ipoib_dev_priv *priv = netdev_priv(dev); > int ret = 0; > > - if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) > - ib_sa_free_multicast(mcast->mc); > + wait_for_completion(mcast->join_req); > > if (test_and_clear_bit(IPOIB_MCAST_FLAG_ATTACHED, &mcast->flags)) { > ipoib_dbg_mcast(priv, "leaving MGID %pI6\n", > From tziporet at mellanox.co.il Mon Jun 29 12:29:36 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 29 Jun 2009 22:29:36 +0300 Subject: [ofa-general] EWG/OFED meeting minutes fo June 29 09 Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD0334A18E@mtlexch01.mtl.com> These are the meeting minutes for EWG/OFED meeting for June 29 09 Meeting summary ============= 1. OFED 1.5 pre-alpha release will be done next Monday (July 6) 2. OFED 1.4.2: Will be done with critical Lustre and NFS/RDMA fixes only. Will have only partial QA. 3. RDMAoE: We will wait to see the progress with Linux kernel Details: ====== 1. OFED 1.5 feature freeze date Decided to wait with feature freeze till we know what is the status of RDMAoE and new ipath driver. All other components are ready for feature freeze (major features are in, bug fixes will be done later) In order to enable testing to start we will create pre-alpha release on Monday next week. AIs: - Qlogic to provide a date for ipath driver to be ready - Mellanox to work with kernel maintainers on RDMAoE - Tziporet to check the status of management package with Sasha the maintainer 2. OFED 1.4.2 - there is a request from Sun to have bug fixes only release since there are critical bugs that prevent Lustre to run over OFED 1.4 and 1.4.1 Decision: We will do 1.4.2 but only with the needed bug fixes. Only partial QA will run not to impact OFED 1.5 release. Jon for OGC will look into the bugs. Note: The interop was already done with 1.4.1 and will not be done for 1.4.2. 1.4.2 will be recommended only for customers that will need the new bug fixes only. 3. RDMAoE: There was a long discussion on RDMAoE in OFED 1.5 I will not summary all the discussion, since many items were already raised on the list. Outcome: We wish to see progress with the kernel maintainers. Decision will be delayed in few weeks based on this progress. We do not wish to start voting on features 4. MPI in OFED: Decision in XWG was not to change for 1.5 and bring it as a BOF in SC09 with the customers MPI versions to be in OFED 1.5: - Open MPI: 1.3.x or 1.4 (based on Open MPI latest version) - MVAPICH: 1.2 - MVAPICH2: 1.4 New MMU notification: - New code is being tested with Roland. There is still a bug with Fortran that is under debug. - Kernel code was not yet submitted to the kernel. - After issue is resolved and if new code will be submitted we can consider if to take it to OFED too Tziporet From bsl at mek.dtu.dk Mon Jun 29 12:49:29 2009 From: bsl at mek.dtu.dk (Boyan Lazarov) Date: Mon, 29 Jun 2009 21:49:29 +0200 Subject: [ofa-general] OFEM installation problem In-Reply-To: <4A478950.4000005@mellanox.co.il> References: <200906252040.39051.bsl@mek.dtu.dk> <4A478950.4000005@mellanox.co.il> Message-ID: <200906292149.29549.bsl@mek.dtu.dk> Hello, Thanks for the answer! Yes, the problem was the memory locking for the users. The root does not have this problem. Best regards, Boyan On Sunday 28 June 2009 17:16:32 Tziporet Koren wrote: > Boyan Lazarov wrote: > > Hello, > > > > We have installed OFED 1.4.1 on Scientific Linux 53 (RH derivative). > > When I run the osu_* tests (OpenMPI) as root it seems that everything is > > running fine and the mpi is using openib for communications. However if > > I run the test as an ordinary user I am getting the following: > > > > WARNING: There was an error initializing an OpenFabrics device. > > > > and openmpi is using tcp for communications - the speed is much lower. > > > > Anyone with similar problem and any suggestions? > > Can you send output of /var/log/messages? > Also it could be that you have this problem: > 2. Memory registration by the user is limited according to administrator > setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for > system configuration. > > =========================================================================== >==== 3. Pinning (Locking) User Memory Pages > =========================================================================== >==== > > Memory locking is managed by the kernel on a per user basis. Regular > users (as > opposed to root) have a limited number of pages which they may pin, where > the limit is pre-set by the administrator. Registering memory for IB verbs > requires pinning memory, thus an application cannot register more memory > than > it is allowed to pin. > > The user can change the system per-process memory lock limit by adding > the following two lines to the file /etc/security/limits.conf: > > * soft memlock > * hard memlock > > where denotes the number of KBytes that may be locked by a > user process. > > The above change to /etc/security/limits.conf will allow any user > process in the > system to lock up to KBytes of memory. > > On some systems, it may be possible to use "unlimited" for the size to > disable > these limits entirely. > > Note: The file /etc/security/limits.conf contains further documentation. > > Tziporet From yossi.openib at gmail.com Mon Jun 29 12:58:58 2009 From: yossi.openib at gmail.com (Yossi Etigin) Date: Mon, 29 Jun 2009 22:58:58 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906290926.56014.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <4A47CCE5.4010408@gmail.com> <200906290714.22571.jackm@dev.mellanox.co.il> <200906290926.56014.jackm@dev.mellanox.co.il> Message-ID: <4A491D02.5020103@gmail.com> Jack Morgenstein wrote: > I'm very worried that if we do things asymmetrically we will introduce instability. > Regarding doing only the multicast leave/free in the workqueue -- I do not see how to do > this cleanly (i.e., everything as is, but simply scheduling a leave-free task). > > Yossi, Roland, Eli? > > -Jack I think we could put just the leave/free on the WQ. It would be symmetrical to join which takes place on the WQ as well. We would probably need to add a "to-free" list to ipoib, the opposite of the join list (multicast_list) which is currently there. Instead of doing free to mcast, we list_add it to some priv->mcast_leave_list and queue the work. I think it can be safe to defer multicast leave/free, as long as it's removed from the mcast tree on time. --Yossi From hal.rosenstock at gmail.com Mon Jun 29 15:01:44 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 29 Jun 2009 18:01:44 -0400 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: <200906291410.33477.fenkes@de.ibm.com> References: <200906291410.33477.fenkes@de.ibm.com> Message-ID: On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: > Previously, libibmad reacted to GSI MAD responses with a "redirect" status > by throwing an error. IBM eHCA adapters use redirection, so most > infiniband_diags tools didn't work against eHCA. Are there GS classes other than PerfMgt which would be redirected by eHCA ? > Fix: Modify mad_rpc() so that it resends the request to the redirection > target if a "redirect" GS response is received. This is repeated until no > "redirect" response is received, allowing for multiple levels of > indirection. > > The dport argument is updated with the redirection target, so subsequent > MADs will not go through the redirection process again but reach the target > directly. > > Tested using perfquery between ehca, mlx4 and mthca in all possible > combinations. > > Signed-off-by: Joachim Fenkes > --- > > Vlad, please queue this patch for OFED 1.5 if Hal doesn't object -- thanks! See other comments below. -- Hal > Regards, >  Joachim > >  libibmad/include/infiniband/mad.h |    9 +++++++ >  libibmad/src/gs.c                 |    6 +++- >  libibmad/src/rpc.c                |   47 +++++++++++++++++++++++++----------- >  3 files changed, 45 insertions(+), 17 deletions(-) > > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index aa27eb5..bdf5158 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -115,6 +115,8 @@ enum MAD_ATTR_ID { > >  enum MAD_STATUS { >        IB_MAD_STS_OK                        = (0 << 2), > +       IB_MAD_STS_BUSY                      = (1 << 0), > +       IB_MAD_STS_REDIRECT                  = (1 << 1), >        IB_MAD_STS_BAD_BASE_VER_OR_CLASS     = (1 << 2), >        IB_MAD_STS_METHOD_NOT_SUPPORTED      = (2 << 2), >        IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), > @@ -783,8 +785,15 @@ MAD_EXPORT int madrpc_set_timeout(int timeout); >  MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, >                        int *mgmt_classes, int num_classes); >  MAD_EXPORT void mad_rpc_close_port(struct ibmad_port *srcport); > + > +/* > + * On redirection, the dport argument is updated with the redirection target, > + * so subsequent MADs will not go through the redirection process again but > + * reach the target directly. > + */ >  MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, >                        ib_portid_t * dport, void *payload, void *rcvdata); > + >  MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, >                              ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, >                              void *data); > diff --git a/libibmad/src/gs.c b/libibmad/src/gs.c > index f3d245e..c7e4ff6 100644 > --- a/libibmad/src/gs.c > +++ b/libibmad/src/gs.c > @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, >        rpc.datasz = IB_PC_DATA_SZ; >        rpc.dataoffs = IB_PC_DATA_OFFS; > > -       dest->qp = 1; > +       if (!dest->qp) > +               dest->qp = 1; Is this change part of this patch or unrelated/separate ? >        if (!dest->qkey) >                dest->qkey = IB_DEFAULT_QP1_QKEY; > > @@ -109,7 +110,8 @@ uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, >        rpc.timeout = timeout; >        rpc.datasz = IB_PC_DATA_SZ; >        rpc.dataoffs = IB_PC_DATA_OFFS; > -       dest->qp = 1; > +       if (!dest->qp) > +               dest->qp = 1; Same as above. >        if (!dest->qkey) >                dest->qkey = IB_DEFAULT_QP1_QKEY; > diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c > index 07b623d..c809060 100644 > --- a/libibmad/src/rpc.c > +++ b/libibmad/src/rpc.c > @@ -189,27 +189,44 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, >        int status, len; >        uint8_t sndbuf[1024], rcvbuf[1024], *mad; >        int timeout, retries; > +       int redirect = 1; > > -       len = 0; > -       memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > +       while (redirect) { > +               len = 0; > +               memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > > -       if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) > -               return 0; > +               if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) > +                       return 0; > > -       timeout = rpc->timeout ? rpc->timeout : > -           port->timeout ? port->timeout : madrpc_timeout; > -       retries = port->retries ? port->retries : madrpc_retries; > +               timeout = rpc->timeout ? rpc->timeout : > +                       port->timeout ? port->timeout : madrpc_timeout; > +               retries = port->retries ? port->retries : madrpc_retries; > > -       if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > -                             port->class_agents[rpc->mgtclass], > -                             len, timeout, retries)) < 0) { > -               IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > -               return 0; > -       } > +               if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > +                                     port->class_agents[rpc->mgtclass], > +                                     len, timeout, retries)) < 0) { > +                       IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > +                       return 0; > +               } > > -       mad = umad_get_mad(rcvbuf); > +               mad = umad_get_mad(rcvbuf); > +               status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); > + > +               /* check for exact match instead of only the redirect bit; > +                * that way, weird statuses cause an error, too */ > +               if (status == IB_MAD_STS_REDIRECT) { > +                       /* update dport for next request and retry */ > +                       dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); > +                       dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); > +                       dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); Are those the only 3 fields which eHCA changes on a redirect ? There may be others we would want to add in here (PKey, SL, ...) ? Also, are the offsets above correct ? > +                       if (ibdebug) > +                               IBWARN("redirected to lid 0x%x, qp 0x%x, qkey 0x%x, pkey 0x%x", > +                                      dport->lid, dport->qp, dport->qkey, dport->pkey_idx); > +               } else > +                       redirect = 0; > +       } > > -       if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { > +       if (status != 0) { >                ERRS("MAD completed with error status 0x%x; dport (%s)", >                     status, portid2str(dport)); >                return 0; Depending on which GS classes are to be supported for redirection, we may want to do something similar to the rmpp equivalent of this routine too. > -- > 1.5.5 > > > > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > From hal.rosenstock at gmail.com Mon Jun 29 15:02:08 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 29 Jun 2009 18:02:08 -0400 Subject: [ofa-general] Do all the ports within the same switch have the same LID? In-Reply-To: <91fe68d50906282345t243315f8x80d84ae4e0202a5c@mail.gmail.com> References: <91fe68d50906282345t243315f8x80d84ae4e0202a5c@mail.gmail.com> Message-ID: On Mon, Jun 29, 2009 at 2:45 AM, Jordan wrote: > Since a switch only has one LID, the ports binding to it have the same LID , > Is this right ? Switch physical/external ports are addressed via switch port 0 LID + port number. Is that what you mean ? -- Hal > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Mon Jun 29 15:02:28 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 29 Jun 2009 18:02:28 -0400 Subject: [ofa-general] Some problem about the LID selection. In-Reply-To: <91fe68d50906290125ube7b08bu6af898b314ae4b4a@mail.gmail.com> References: <91fe68d50906290125ube7b08bu6af898b314ae4b4a@mail.gmail.com> Message-ID: On Mon, Jun 29, 2009 at 4:25 AM, Jordan wrote: > When LMC > 0 , there are 2^LMC LIDs assigned to a port.  When choosing > different LID, there maybe different paths to get to this port. > In other words, different LIDs represent different paths. But which path > should I choose to  get to the destination? It seems that OpenSM > does not provide such algorithm . > > Does OFED provide such algorithm to manage  which path should to choose ? Path(s) that the SA return to the client are based on the type of request (e.g. Get v. GetTable, and which components are matched (LIDs, GIDs, NumbPaths if GetTable, etc.). If the request results in multiple path records in the response, the client is free to choose one of those. -- Hal > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From Ted.Kim at Sun.COM Mon Jun 29 16:26:51 2009 From: Ted.Kim at Sun.COM (Ted H. Kim) Date: Mon, 29 Jun 2009 16:26:51 -0700 Subject: [ofa-general] Does IPoIB automatically created MCGs? Message-ID: <4A494DBB.6010306@sun.com> Folks, When IPoIB starts up and can't find the corresponding IB multi-cast group, does it automatically create it? -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim at sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From jackm at dev.mellanox.co.il Mon Jun 29 23:55:09 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Tue, 30 Jun 2009 09:55:09 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A491D02.5020103@gmail.com> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906290926.56014.jackm@dev.mellanox.co.il> <4A491D02.5020103@gmail.com> Message-ID: <200906300955.10131.jackm@dev.mellanox.co.il> On Monday 29 June 2009 22:58, Yossi Etigin wrote: > I think we could put just the leave/free on the WQ. It would be symmetrical to join which > takes place on the WQ as well. We would probably need to add a "to-free" list to ipoib, > the opposite of the join list (multicast_list) which is currently there. > Instead of doing free to mcast, we list_add it to some priv->mcast_leave_list and queue > the work. I think it can be safe to defer multicast leave/free, as long as it's removed from > the mcast tree on time. > Is the patch below what you mean? I think this will work. (It is compiled, but not yet tested). I've opened Bugzilla 1666 on this issue. -Jack ================================================================================================================== ipoib: fix Oops in multicast leave flow. To avoid race conditions (which may lead to a kernel Oops) between multicast join and multicast leave, we transfer leave processing to the workqueue (rather than do it in place). This fixes Bugzilla 1666. This fix was suggested by Yossi Etigin of Voltaire. Signed-off-by: Jack Morgenstein Index: is4_kernel/drivers/infiniband/ulp/ipoib/ipoib.h =================================================================== --- is4_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib.h 2009-06-30 09:44:48.434443000 +0300 +++ is4_kernel/drivers/infiniband/ulp/ipoib/ipoib.h 2009-06-30 09:44:49.004011000 +0300 @@ -288,10 +288,12 @@ struct ipoib_dev_priv { struct ipoib_mcast *broadcast; struct list_head multicast_list; + struct list_head mcast_remove_list; struct rb_root multicast_tree; struct delayed_work pkey_poll_task; struct delayed_work mcast_task; + struct work_struct mcast_remove_task; struct work_struct carrier_on_task; struct work_struct flush_light; struct work_struct flush_normal; @@ -463,6 +465,7 @@ int ipoib_dev_init(struct net_device *de void ipoib_dev_cleanup(struct net_device *dev); void ipoib_mcast_join_task(struct work_struct *work); +void ipoib_mcast_remove_task(struct work_struct *work); void ipoib_mcast_carrier_on_task(struct work_struct *work); void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb); Index: is4_kernel/drivers/infiniband/ulp/ipoib/ipoib_main.c =================================================================== --- is4_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib_main.c 2009-06-30 09:44:48.897904000 +0300 +++ is4_kernel/drivers/infiniband/ulp/ipoib/ipoib_main.c 2009-06-30 09:44:49.011018000 +0300 @@ -1096,9 +1096,11 @@ static void ipoib_setup(struct net_devic INIT_LIST_HEAD(&priv->child_intfs); INIT_LIST_HEAD(&priv->dead_ahs); INIT_LIST_HEAD(&priv->multicast_list); + INIT_LIST_HEAD(&priv->mcast_remove_list); INIT_DELAYED_WORK(&priv->pkey_poll_task, ipoib_pkey_poll); INIT_DELAYED_WORK(&priv->mcast_task, ipoib_mcast_join_task); + INIT_WORK(&priv->mcast_remove_task, ipoib_mcast_remove_task); INIT_WORK(&priv->carrier_on_task, ipoib_mcast_carrier_on_task); INIT_WORK(&priv->flush_light, ipoib_ib_dev_flush_light); INIT_WORK(&priv->flush_normal, ipoib_ib_dev_flush_normal); @@ -1468,6 +1470,7 @@ err_fs: static void __exit ipoib_cleanup_module(void) { ib_unregister_client(&ipoib_client); + flush_workqueue(ipoib_workqueue); ib_sa_unregister_client(&ipoib_sa_client); ipoib_unregister_debugfs(); destroy_workqueue(ipoib_workqueue); Index: is4_kernel/drivers/infiniband/ulp/ipoib/ipoib_multicast.c =================================================================== --- is4_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib_multicast.c 2009-06-30 09:44:48.955963000 +0300 +++ is4_kernel/drivers/infiniband/ulp/ipoib/ipoib_multicast.c 2009-06-30 09:45:44.990941000 +0300 @@ -657,6 +657,31 @@ static int ipoib_mcast_leave(struct net_ return 0; } +void ipoib_mcast_remove_task(struct work_struct *work) +{ + struct ipoib_dev_priv *priv = + container_of(work, struct ipoib_dev_priv, mcast_remove_task); + struct net_device *dev = priv->dev; + + LIST_HEAD(remove_list); + struct ipoib_mcast *mcast, *tmcast; + unsigned long flags; + + ipoib_dbg_mcast(priv, "flushing multicast list\n"); + + spin_lock_irqsave(&priv->lock, flags); + list_for_each_entry_safe(mcast, tmcast, &priv->mcast_remove_list, list) { + list_del(&mcast->list); + list_add_tail(&mcast->list, &remove_list); + } + spin_unlock_irqrestore(&priv->lock, flags); + + list_for_each_entry_safe(mcast, tmcast, &remove_list, list) { + ipoib_mcast_leave(dev, mcast); + ipoib_mcast_free(mcast); + } +} + void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -740,7 +765,6 @@ unlock: void ipoib_mcast_dev_flush(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); - LIST_HEAD(remove_list); struct ipoib_mcast *mcast, *tmcast; unsigned long flags; @@ -751,21 +775,18 @@ void ipoib_mcast_dev_flush(struct net_de list_for_each_entry_safe(mcast, tmcast, &priv->multicast_list, list) { list_del(&mcast->list); rb_erase(&mcast->rb_node, &priv->multicast_tree); - list_add_tail(&mcast->list, &remove_list); + list_add_tail(&mcast->list, &priv->mcast_remove_list); } if (priv->broadcast) { rb_erase(&priv->broadcast->rb_node, &priv->multicast_tree); - list_add_tail(&priv->broadcast->list, &remove_list); + list_add_tail(&priv->broadcast->list, &priv->mcast_remove_list); priv->broadcast = NULL; } spin_unlock_irqrestore(&priv->lock, flags); - list_for_each_entry_safe(mcast, tmcast, &remove_list, list) { - ipoib_mcast_leave(dev, mcast); - ipoib_mcast_free(mcast); - } + queue_work(ipoib_workqueue, &priv->mcast_remove_task); } void ipoib_mcast_restart_task(struct work_struct *work) @@ -775,7 +796,6 @@ void ipoib_mcast_restart_task(struct wor struct net_device *dev = priv->dev; struct dev_mc_list *mclist; struct ipoib_mcast *mcast, *tmcast; - LIST_HEAD(remove_list); unsigned long flags; struct ib_sa_mcmember_rec rec; @@ -831,7 +851,7 @@ void ipoib_mcast_restart_task(struct wor if (mcast) { /* Destroy the send only entry */ - list_move_tail(&mcast->list, &remove_list); + list_move_tail(&mcast->list, &priv->mcast_remove_list); rb_replace_node(&mcast->rb_node, &nmcast->rb_node, @@ -856,7 +876,7 @@ void ipoib_mcast_restart_task(struct wor rb_erase(&mcast->rb_node, &priv->multicast_tree); /* Move to the remove list */ - list_move_tail(&mcast->list, &remove_list); + list_move_tail(&mcast->list, &priv->mcast_remove_list); } } @@ -864,11 +884,7 @@ void ipoib_mcast_restart_task(struct wor netif_addr_unlock(dev); local_irq_restore(flags); - /* We have to cancel outside of the spinlock */ - list_for_each_entry_safe(mcast, tmcast, &remove_list, list) { - ipoib_mcast_leave(mcast->dev, mcast); - ipoib_mcast_free(mcast); - } + queue_work(ipoib_workqueue, &priv->mcast_remove_task); if (test_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags)) ipoib_mcast_start_thread(dev); From vlad at lists.openfabrics.org Tue Jun 30 02:24:19 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 30 Jun 2009 02:24:19 -0700 (PDT) Subject: [ofa-general] ofa_1_5_kernel 20090630-0200 daily build status Message-ID: <20090630092419.C7929E302C5@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.26 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.24 Build failed on i686 with linux-2.6.27 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-128.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:394: error: 'for_each_netdev' undeclared (first use in this function) /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.27 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.27_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.27' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-78.ELsmp Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1383: error: 'struct ib_device' has no member named 'dev' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c: In function 'iwch_unregister_device': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:1402: error: 'struct ib_device' has no member named 'dev' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.9-78.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-78.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:592: error: implicit declaration of function 'pcie_set_readrq' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c: In function 'nes_store_wqm_quanta': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:1119: error: implicit declaration of function 'strict_strtoul' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.c:36: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h: In function 'clear_bit_unlock': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check/kernel_addons/backport/2.6.22/include/asm/bitops.h:8: error: implicit declaration of function 'smp_mb' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.26 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.26_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.26' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.25 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.25_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: In file included from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/common.h:645, from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:523: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.c:529: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2624: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2682: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:376: error: implicit declaration of function 'ipv4_is_loopback' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:395: error: expected ';' before 'if' /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.c:410: error: implicit declaration of function 'ipv6_addr_loopback' make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: from /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.c:59: /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:194: error: field 'napi' has incomplete type /home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check/drivers/net/cxgb3/adapter.h:198: error: field 'lro_frag_tbl' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/iwch_provider.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_5_kernel-20090630-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From eli at dev.mellanox.co.il Tue Jun 30 04:31:58 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 30 Jun 2009 14:31:58 +0300 Subject: [ofa-general] XRC userspace git trees Message-ID: <20090630113158.GA31362@mtls03> Roland, where can I find your userspace trees with XRC support? From monis at Voltaire.COM Tue Jun 30 04:51:58 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Tue, 30 Jun 2009 14:51:58 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906292229.31129.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906282118.22534.jackm@dev.mellanox.co.il> <4A48D888.1020601@Voltaire.COM> <200906292229.31129.jackm@dev.mellanox.co.il> Message-ID: <4A49FC5E.9040704@Voltaire.COM> Jack Morgenstein wrote: > On Monday 29 June 2009 18:06, Moni Shoua wrote: >> Jack Morgenstein wrote: >>> On Sunday 28 June 2009 19:09, Moni Shoua wrote: >>>> maybe synchronizing the race with a completion var (like IPoIB does in struct ipoib_path) will help. I think this will work. I can send a patch if you want unless you see this idea doesn't work for this case. >>>> >>>> MoniS >> What do you think of this (see below)? >> I used a completion struct to synchronize between leave and join in a similar way that ipoib_path is using the field 'done'. >> There is one difference though from the original code (I don't know if that's acceptable), here we wait for the join request to finish before we destroy the mcast struct and not canceling it. >> Please note that I didn't compile this code. I just want to hear opinions first before I make a real patch out of this. >> > I have to think about this some. I'm a bit nervous about 2 things: > 1. Somehow, the complete never arrives, and the "leave" gets stuck forever waiting. Is that a concern or an experience after trying this patch? If complete never arrives then this a bug that can be fixed (so I believe). > 2. What if someone does another ifconfig ib0 up while we are waiting for the complete? Even so, I don't see a path that leads from ifconfig up to changing the status of join_req for a group that alreay exists. > (I think it is disregarded, but I'm not sure -- with this change and all). > > Nice idea, though. > -Jack >> diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h >> index 753a983..a551258 100644 >> --- a/drivers/infiniband/ulp/ipoib/ipoib.h >> +++ b/drivers/infiniband/ulp/ipoib/ipoib.h >> @@ -145,6 +145,7 @@ struct ipoib_mcast { >> struct sk_buff_head pkt_queue; >> >> struct net_device *dev; >> + struct completion join_req; >> }; >> >> struct ipoib_rx_buf { >> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c >> index a0e9753..afe72dd 100644 >> --- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c >> +++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c >> @@ -117,6 +117,7 @@ static struct ipoib_mcast *ipoib_mcast_alloc(struct net_device *dev, >> mcast->dev = dev; >> mcast->created = jiffies; >> mcast->backoff = 1; >> + complete(&mcast->join_req); >> >> INIT_LIST_HEAD(&mcast->list); >> INIT_LIST_HEAD(&mcast->neigh_list); >> @@ -286,6 +287,7 @@ ipoib_mcast_sendonly_join_complete(int status, >> if (status == -ENETRESET) >> return 0; >> >> + complete(&mcast->join_req); >> if (!status) >> status = ipoib_mcast_join_finish(mcast, &multicast->rec); >> >> @@ -336,6 +338,7 @@ static int ipoib_mcast_sendonly_join(struct ipoib_mcast *mcast) >> rec.port_gid = priv->local_gid; >> rec.pkey = cpu_to_be16(priv->pkey); >> >> + init_completion(&mcast->join_req); >> mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, >> priv->port, &rec, >> IB_SA_MCMEMBER_REC_MGID | >> @@ -384,6 +387,7 @@ static int ipoib_mcast_join_complete(int status, >> mcast->mcmember.mgid.raw, status); >> >> /* We trap for port events ourselves. */ >> + complete(&mcast->join_req); >> if (status == -ENETRESET) >> return 0; >> >> @@ -482,10 +486,12 @@ static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast, >> } >> >> set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); >> + init_completion(&mcast->join_req); >> mcast->mc = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port, >> &rec, comp_mask, GFP_KERNEL, >> ipoib_mcast_join_complete, mcast); >> if (IS_ERR(mcast->mc)) { >> + complete(&mcast->join_req); >> clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); >> ret = PTR_ERR(mcast->mc); >> ipoib_warn(priv, "ib_sa_join_multicast failed, status %d\n", ret); >> @@ -630,8 +636,7 @@ static int ipoib_mcast_leave(struct net_device *dev, struct ipoib_mcast *mcast) >> struct ipoib_dev_priv *priv = netdev_priv(dev); >> int ret = 0; >> >> - if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags)) >> - ib_sa_free_multicast(mcast->mc); >> + wait_for_completion(mcast->join_req); >> >> if (test_and_clear_bit(IPOIB_MCAST_FLAG_ATTACHED, &mcast->flags)) { >> ipoib_dbg_mcast(priv, "leaving MGID %pI6\n", >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From fenkes at de.ibm.com Tue Jun 30 05:04:03 2009 From: fenkes at de.ibm.com (Joachim Fenkes) Date: Tue, 30 Jun 2009 14:04:03 +0200 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: References: <200906291410.33477.fenkes@de.ibm.com> Message-ID: <200906301404.05895.fenkes@de.ibm.com> On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote: > On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: > > Previously, libibmad reacted to GSI MAD responses with a "redirect" status > > by throwing an error. IBM eHCA adapters use redirection, so most > > infiniband_diags tools didn't work against eHCA. > > Are there GS classes other than PerfMgt which would be redirected by eHCA ? Not right now, no. If you're interested in the details of how and when the eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c. > > --- a/libibmad/src/gs.c > > +++ b/libibmad/src/gs.c > > @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, > >        rpc.datasz = IB_PC_DATA_SZ; > >        rpc.dataoffs = IB_PC_DATA_OFFS; > > > > -       dest->qp = 1; > > +       if (!dest->qp) > > +               dest->qp = 1; > > Is this change part of this patch or unrelated/separate ? Part of the patch. Without this change, pma_query_via() would overwrite the redirected QP with QP1 again, and the MAD would never arrive at the right destination. > > +               /* check for exact match instead of only the redirect bit; > > +                * that way, weird statuses cause an error, too */ > > +               if (status == IB_MAD_STS_REDIRECT) { > > +                       /* update dport for next request and retry */ > > +                       dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); > > +                       dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); > > +                       dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); > > Are those the only 3 fields which eHCA changes on a redirect ? There > may be others we would want to add in here (PKey, SL, ...) ? Yeah, I agree on the SL, I can add it to the patch. At first, I also tried to set the PKey, but ClassPortInfo specifies a PKey while ib_portid_t needs a PKey Index, and I found no way of converting between the two, so I left it at zero. Incidentally, there isn't a single code line in management.git that actually changes the pkey_index from its init value of 0, so I figured that omission couldn't be too bad. Then there's the GRH stuff, but I refrained from coding that because I wouldn't be able to test it -- InfiniBand isn't going to evolve beyond a single subnet any time toon, is it? > Also, are the offsets above correct ? Yes, they are, I tested. The ClassPortInfo data starts at offset 64 in the MAD, and I didn't find a constant for this in mad.h. > Depending on which GS classes are to be supported for redirection, we > may want to do something similar to the rmpp equivalent of this > routine too. The spec says in 13.5.2 that "The SA as well as each GSA may individually support this mechanism or not", so we should probably be prepared for any GS class to redirect. I don't care much about RMPP, though, so I left it alone. Regards, Joachim From yosefe at voltaire.com Tue Jun 30 06:27:31 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Tue, 30 Jun 2009 16:27:31 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <200906300955.10131.jackm@dev.mellanox.co.il> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906290926.56014.jackm@dev.mellanox.co.il> <4A491D02.5020103@gmail.com> <200906300955.10131.jackm@dev.mellanox.co.il> Message-ID: <4A4A12C3.5060604@voltaire.com> Jack Morgenstein wrote: > Is the patch below what you mean? I think this will work. > (It is compiled, but not yet tested). > I've opened Bugzilla 1666 on this issue. Yes, this is what exactly I've meant. However please see some remarks below. > --- is4_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib.h 2009-06-30 09:44:48.434443000 +0300 > +++ is4_kernel/drivers/infiniband/ulp/ipoib/ipoib.h 2009-06-30 09:44:49.004011000 +0300 > @@ -288,10 +288,12 @@ struct ipoib_dev_priv { > > struct ipoib_mcast *broadcast; > struct list_head multicast_list; > + struct list_head mcast_remove_list; > struct rb_root multicast_tree; > > struct delayed_work pkey_poll_task; > struct delayed_work mcast_task; > + struct work_struct mcast_remove_task; > struct work_struct carrier_on_task; > struct work_struct flush_light; > struct work_struct flush_normal; What do you think about renaming the mcast_task to mcast_join_task and multicast_list to mcast_join_list? It will make the purpose and the analogy between the two more obvious. > static void __exit ipoib_cleanup_module(void) > { > ib_unregister_client(&ipoib_client); > + flush_workqueue(ipoib_workqueue); > ib_sa_unregister_client(&ipoib_sa_client); > ipoib_unregister_debugfs(); > destroy_workqueue(ipoib_workqueue); This is already done in ipoib_remove_one(). > --- is4_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib_multicast.c 2009-06-30 09:44:48.955963000 +0300 > +++ is4_kernel/drivers/infiniband/ulp/ipoib/ipoib_multicast.c 2009-06-30 09:45:44.990941000 +0300 > @@ -657,6 +657,31 @@ static int ipoib_mcast_leave(struct net_ > return 0; > } > > +void ipoib_mcast_remove_task(struct work_struct *work) > +{ > + struct ipoib_dev_priv *priv = > + container_of(work, struct ipoib_dev_priv, mcast_remove_task); > + struct net_device *dev = priv->dev; > + > + LIST_HEAD(remove_list); > + struct ipoib_mcast *mcast, *tmcast; > + unsigned long flags; > + > + ipoib_dbg_mcast(priv, "flushing multicast list\n"); > + > + spin_lock_irqsave(&priv->lock, flags); > + list_for_each_entry_safe(mcast, tmcast, &priv->mcast_remove_list, list) { > + list_del(&mcast->list); > + list_add_tail(&mcast->list, &remove_list); > + } > + spin_unlock_irqrestore(&priv->lock, flags); I think you could use list_splice() here. > @@ -864,11 +884,7 @@ void ipoib_mcast_restart_task(struct wor > netif_addr_unlock(dev); > local_irq_restore(flags); > > - /* We have to cancel outside of the spinlock */ > - list_for_each_entry_safe(mcast, tmcast, &remove_list, list) { > - ipoib_mcast_leave(mcast->dev, mcast); > - ipoib_mcast_free(mcast); > - } > + queue_work(ipoib_workqueue, &priv->mcast_remove_task); > > if (test_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags)) > ipoib_mcast_start_thread(dev); Modifying ipoib_mcast_restart_task() is not really needed since it runs on the WQ anyway. On the other hand, for compatibility (?) with other changes it may be a good thing to do. --Yossi From hal.rosenstock at gmail.com Tue Jun 30 07:14:13 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 30 Jun 2009 10:14:13 -0400 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: <200906301404.05895.fenkes@de.ibm.com> References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> Message-ID: On Tue, Jun 30, 2009 at 8:04 AM, Joachim Fenkes wrote: > On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote: >> On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: > >> > Previously, libibmad reacted to GSI MAD responses with a "redirect" status >> > by throwing an error. IBM eHCA adapters use redirection, so most >> > infiniband_diags tools didn't work against eHCA. >> >> Are there GS classes other than PerfMgt which would be redirected by eHCA ? > > Not right now, no. If you're interested in the details of how and when the > eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c. SL is always set to 0 and RespTimeValue is hardcoded. >> > --- a/libibmad/src/gs.c >> > +++ b/libibmad/src/gs.c >> > @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, >> >        rpc.datasz = IB_PC_DATA_SZ; >> >        rpc.dataoffs = IB_PC_DATA_OFFS; >> > >> > -       dest->qp = 1; >> > +       if (!dest->qp) >> > +               dest->qp = 1; >> >> Is this change part of this patch or unrelated/separate ? > > Part of the patch. Without this change, pma_query_via() would overwrite the > redirected QP with QP1 again, and the MAD would never arrive at the right > destination. > >> > +               /* check for exact match instead of only the redirect bit; >> > +                * that way, weird statuses cause an error, too */ >> > +               if (status == IB_MAD_STS_REDIRECT) { >> > +                       /* update dport for next request and retry */ >> > +                       dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); >> > +                       dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); >> > +                       dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); >> >> Are those the only 3 fields which eHCA changes on a redirect ? There >> may be others we would want to add in here (PKey, SL, ...) ? > > Yeah, I agree on the SL, I can add it to the patch. > > At first, I also tried to set the PKey, but ClassPortInfo specifies a PKey > while ib_portid_t needs a PKey Index, and I found no way of converting > between the two, It's available via libibumad. Note that umad version 5 is needed for pkey index support. > so I left it at zero. Incidentally, there isn't a single > code line in management.git that actually changes the pkey_index from its > init value of 0, so I figured that omission couldn't be too bad. Agreed. I think you're referring to infiniband-diags and not opensm. > Then there's the GRH stuff, but I refrained from coding that because I > wouldn't be able to test it That's fine for now IMO. I think there's only some minimal GRH support now elsewhere in the diags and this needs more work in general. > -- InfiniBand isn't going to evolve beyond a > single subnet any time toon, is it? There's ongoing work in this area but not sure how soon is soon... >> Also, are the offsets above correct ? > > Yes, they are, I tested. The ClassPortInfo data starts at offset 64 in the > MAD, and I didn't find a constant for this in mad.h. > >> Depending on which GS classes are to be supported for redirection, we >> may want to do something similar to the rmpp equivalent of this >> routine too. > > The spec says in 13.5.2 that "The SA as well as each GSA may individually > support this mechanism or not", so we should probably be prepared for any GS > class to redirect. I don't care much about RMPP, though, so I left it alone. Understood. -- Hal > Regards, >  Joachim > > > > From jackm at dev.mellanox.co.il Tue Jun 30 07:26:48 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Tue, 30 Jun 2009 17:26:48 +0300 Subject: [ofa-general] IPoIB kernel Oops -- race condition In-Reply-To: <4A4A12C3.5060604@voltaire.com> References: <200906281117.34440.jackm@dev.mellanox.co.il> <200906300955.10131.jackm@dev.mellanox.co.il> <4A4A12C3.5060604@voltaire.com> Message-ID: <200906301726.48965.jackm@dev.mellanox.co.il> On Tuesday 30 June 2009 16:27, Yossi Etigin wrote: > > What do you think about renaming the mcast_task to mcast_join_task > and multicast_list to mcast_join_list? It will make the purpose and > the analogy between the two more obvious. I'll do that. > > static void __exit ipoib_cleanup_module(void) > > { > > ib_unregister_client(&ipoib_client); > > + flush_workqueue(ipoib_workqueue); > > ib_sa_unregister_client(&ipoib_sa_client); > > ipoib_unregister_debugfs(); > > destroy_workqueue(ipoib_workqueue); > > This is already done in ipoib_remove_one(). Correct, I will remove the flush. > > --- is4_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib_multicast.c 2009-06-30 09:44:48.955963000 +0300 > > +++ is4_kernel/drivers/infiniband/ulp/ipoib/ipoib_multicast.c 2009-06-30 09:45:44.990941000 +0300 > > @@ -657,6 +657,31 @@ static int ipoib_mcast_leave(struct net_ > > return 0; > > } > > > > +void ipoib_mcast_remove_task(struct work_struct *work) > > +{ > > + struct ipoib_dev_priv *priv = > > + container_of(work, struct ipoib_dev_priv, mcast_remove_task); > > + struct net_device *dev = priv->dev; > > + > > + LIST_HEAD(remove_list); > > + struct ipoib_mcast *mcast, *tmcast; > > + unsigned long flags; > > + > > + ipoib_dbg_mcast(priv, "flushing multicast list\n"); > > + > > + spin_lock_irqsave(&priv->lock, flags); > > + list_for_each_entry_safe(mcast, tmcast, &priv->mcast_remove_list, list) { > > + list_del(&mcast->list); > > + list_add_tail(&mcast->list, &remove_list); > > + } > > + spin_unlock_irqrestore(&priv->lock, flags); > > I think you could use list_splice() here. I'll look into that. > > @@ -864,11 +884,7 @@ void ipoib_mcast_restart_task(struct wor > > netif_addr_unlock(dev); > > local_irq_restore(flags); > > > > - /* We have to cancel outside of the spinlock */ > > - list_for_each_entry_safe(mcast, tmcast, &remove_list, list) { > > - ipoib_mcast_leave(mcast->dev, mcast); > > - ipoib_mcast_free(mcast); > > - } > > + queue_work(ipoib_workqueue, &priv->mcast_remove_task); > > > > if (test_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags)) > > ipoib_mcast_start_thread(dev); > > Modifying ipoib_mcast_restart_task() is not really needed since it runs on the WQ > anyway. You're right, I was just working on auto-pilot here and automatically made the change. I will eliminate the change in the restart_task, because there is no reason to complicate the flow unnecessarily. > On the other hand, for compatibility (?) with other changes it may be a good > thing to do. > > --Yossi > From chocapiiic.tiery at gmail.com Tue Jun 30 07:29:57 2009 From: chocapiiic.tiery at gmail.com (Thierry) Date: Tue, 30 Jun 2009 16:29:57 +0200 Subject: [ofa-general] Sending two integers via RDMA_WRITE In-Reply-To: <7d4423d30906261103t2a0dc33cp14e04742cd0086ff@mail.gmail.com> References: <7d4423d30905250836r431e7d1eh170f3bc22b731a55@mail.gmail.com> <4A1AD987.2080200@gmail.com> <7d4423d30905251108m5bc951ach4da2027698b0ffb6@mail.gmail.com> <4A1AF13D.8040506@gmail.com> <7d4423d30905252245o869fd93p96facb1d1b2b94e6@mail.gmail.com> <8d9c773c0906260607y7aa4a1a6y55742c45ba9751ed@mail.gmail.com> <7d4423d30906261103t2a0dc33cp14e04742cd0086ff@mail.gmail.com> Message-ID: <8d9c773c0906300729s4d27e11am98c83b15a9ba9af3@mail.gmail.com> hi, I am still working on this problem, but can't solve it. The problem is: I want to use completion queue element on completion queue associated with received queue (on remote hca) to allow reading databuffer. But I get nothing from the completion queue. In the specification, it says that a CQE should be created (in the remote hca) after performing a rdma write The function ibv_get_cq_event return nothings (it is like while(1) in ibv_get_cq_event) My code is attached, if you have any ideas. Regards, Thierry On Fri, Jun 26, 2009 at 8:03 PM, Zafar Gilani wrote: > Dear Thierry, > > I did not try it any further, since my basic aim was to run this with JNI. > However Verbs code crashes the JVM and I am still trying to figure that out. > Besides this I have now used SDP to run Java code on InfiniBand. > > On Fri, Jun 26, 2009 at 6:07 PM, Thierry wrote: >> >> Hi, >> >> Have you fixed your problem ? I am also trying to pool CQ (associated >> with the receive part) with an other program, but I get nothing. >> >> Regards, >> >> Thierry >> >> On Tue, May 26, 2009 at 7:45 AM, Zafar Gilani >> wrote: >> > To make things more clear, why won't you just poll the CQ for completion >> > directly? (without using the CQ events) >> > >> > I believe that you will get a completion with error... >> > >> > Yes I tried polling directly and it returns a negative number. What is >> > the >> > remedy for this? Is my for loop logically correct (client.c)? I also >> > tried >> > polling the server CQ directly (server.c) and polling here also returns >> > a >> > negative number, which means that data write is not working properly >> > thus no >> > completion events. What do you suggest I do? I am obviously lost! :( >> > >> > >> > >> > _______________________________________________ >> > general mailing list >> > general at lists.openfabrics.org >> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> > >> > To unsubscribe, please visit >> > http://openib.org/mailman/listinfo/openib-general >> > > > > > -- > Syed Zafar ul Hussan Gilani | BIT-7 > Research Student | CHPSC > MSP 2008-09 > NUST SEECS | http://hpc.niit.edu.pk/~zafar > -------------- next part -------------- A non-text attachment was scrubbed... Name: rdma_write.tgz Type: application/x-gzip Size: 4144 bytes Desc: not available URL: From sean.hefty at intel.com Tue Jun 30 07:46:57 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 30 Jun 2009 07:46:57 -0700 Subject: [ofa-general] Sending two integers via RDMA_WRITE In-Reply-To: <8d9c773c0906300729s4d27e11am98c83b15a9ba9af3@mail.gmail.com> References: <7d4423d30905250836r431e7d1eh170f3bc22b731a55@mail.gmail.com> <4A1AD987.2080200@gmail.com> <7d4423d30905251108m5bc951ach4da2027698b0ffb6@mail.gmail.com> <4A1AF13D.8040506@gmail.com> <7d4423d30905252245o869fd93p96facb1d1b2b94e6@mail.gmail.com> <8d9c773c0906260607y7aa4a1a6y55742c45ba9751ed@mail.gmail.com> <7d4423d30906261103t2a0dc33cp14e04742cd0086ff@mail.gmail.com> <8d9c773c0906300729s4d27e11am98c83b15a9ba9af3@mail.gmail.com> Message-ID: <7B7B10E08FA841F7AC7E856B1321CAB6@amr.corp.intel.com> >I want to use completion queue element on completion queue associated >with received queue (on remote hca) to allow reading databuffer. >But I get nothing from the completion queue. You need to send immediate data with an RDMA write to generate a completion on the remote side. Otherwise, a receive work request is not consumed. >In the specification, it says that a CQE should be created (in the >remote hca) after performing a rdma write See C10-87 (page 511) From balaji at mcs.anl.gov Tue Jun 30 09:39:10 2009 From: balaji at mcs.anl.gov (Pavan Balaji) Date: Tue, 30 Jun 2009 11:39:10 -0500 Subject: [ofa-general] [hpc-announce] Submission deadline for HPIDC '09 workshop extended to July 10th, 2009 Message-ID: <4A4A3FAE.3080603@mcs.anl.gov> Dear Colleague: This is to inform you that the International Workshop on High Performance Interconnects for Distributed Computing (HPIDC) has extended the paper submission deadline to July 10th, 2009. A full CFP can be found below. Regards, -- Ada & Pavan ---------------------------------------------------------------------- Workshop on High Performance Interconnects for Distributed Computing (HPI-DC'09) in conjunction with Cluster 2009 August 31, 2009, New Orleans, Louisiana http://www.cercs.gatech.edu/hpidc2009 ********************************************************************* Call for Papers ********************************************************************* The emergence of 10.0 GigE and above, InfiniBand and other high-performance interconnection technologies, programmable NICs and networking platforms, and protocols like DDP and RDMA over IP, make it possible to create tightly linked systems across physical distances that exceed those of traditional single cluster or server systems. These technologies can deliver communication capabilities that achieve the performance levels needed by high end applications in enterprise systems and like those produced by the high performance computing community. Furthermore, the manycore nature of next generation platforms and the creation of distributed cloud computing infrastructure will greatly increase the demand for high performance communication capabilities over wide area distances. The purpose of this workshop is to explore the confluence of distributed computing and communications technologies with high performance interconnects, as applicable or applied to realistic high end applications. The intent is to create a venue that will act as a bridge between researchers developing tools and platforms for high-performance distributed computing, end user applications seeking high performance solutions, and technology providers aiming to improve interconnect and networking technologies for future systems. The hope is to foster knowledge creation and intellectual interchanges between HPC and Cloud computing end users and technology developers, in the specific domain of high performance distributed interconnects. Topics of interest include but are not limited to: # Hardware/software architectures for communication infrastructures for HPC and Cloud Computing # Data and control protocols for interactive and large data volume applications # Novel devices and technologies to enhance interconnect properties # Interconnect-level issues when extending high performance beyond single machines, including architecture, protocols, services, QoS, and security # Remote storage (like iSCSI), remote databases, and datacenters, etc. # Development tools, programming environments and models (like PGAS, OpenShmem, Hadoop, etc.), ranging from programming language support to simulation environments. PAPER SUBMISSIONS: HPI-DC invites authors to submit original and unpublished work. Please submit extended abstracts or full papers, not exceeding 8 double-column pages in 10 point font or larger, in IEEE format. Electronic submission is strongly encouraged. Hard copies will be accepted only if electronic submission is not possible. Submission implies the willingness of at least one of the authors to register and present the paper. Any questions concerning hardcopy submissions or any other issues may be directed to the Program Co-Chairs. IMPORTANT DATES: # Paper submission: July 10th, 2009 # Notification of acceptance: July 22nd, 2009 # Final manuscript due: July 29th, 2009 # Workshop date: Aug. 31st, 2009 ORGANIZATION: General Chair # Steve Poole, Oak Ridge National Lab Program Co-Chairs # Pavan Balaji, Argonne National Lab # Ada Gavrilovska, Georgia Institute of Technology Technical Program Committee: # Ahmad Afsahi, Queen's University, Canada # Taisuke Boku, University of Tsukuba, Japan # Ron Brightwell, Sandia National Laboratory # Patrick Geoffray, Myricom # Kei Hiraki, University of Tokyo, Japan # Hyun-wook Jin, Konkuk University, Korea # Pankaj Mehra, HP Research # Guillaume Mercier, INRIA, France # Scott Pakin, Los Alamos National Laboratory # D. K. Panda, Ohio State University # Fabrizio Petrini, IBM Research # Karsten Schwan, Georgia Tech # Jesper Traeff, NEC, Europe # Sudhakar Yalamanchili, Georgia Tech # Weikuan Yu, Auburn University If you have any questions about the workshop, please contact us at hpidc09-chairs at mcs.anl.gov. ==================================================================== If you do not wish to receive any more emails on this list, you can unsubscribe here: https://lists.mcs.anl.gov/mailman/listinfo/hpc-announce ==================================================================== -- Pavan Balaji http://www.mcs.anl.gov/~balaji From fenkes at de.ibm.com Tue Jun 30 09:53:41 2009 From: fenkes at de.ibm.com (Joachim Fenkes) Date: Tue, 30 Jun 2009 18:53:41 +0200 Subject: [ofa-general] [PATCH v2] libibmad: Handle MAD redirection In-Reply-To: References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> Message-ID: <200906301853.44361.fenkes@de.ibm.com> Previously, libibmad reacted to GSI MAD responses with a "redirect" status by throwing an error. IBM eHCA adapters use redirection, so most infiniband_diags tools didn't work against eHCA. Fix: Modify mad_rpc() so that it resends the request to the redirection target if a "redirect" GS response is received. This is repeated until no "redirect" response is received, allowing for multiple levels of indirection. The dport argument is updated with the redirection target, so subsequent MADs will not go through the redirection process again but reach the target directly. Tested using perfquery between ehca, mlx4 and mthca in all possible combinations. Signed-off-by: Joachim Fenkes --- On Tuesday 30 June 2009 16:14, Hal Rosenstock wrote: > On Tue, Jun 30, 2009 at 8:04 AM, Joachim Fenkes wrote: > > On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote: > >> On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: > >> Are there GS classes other than PerfMgt which would be redirected by eHCA ? > > > > Not right now, no. If you're interested in the details of how and when the > > eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c. > > SL is always set to 0 and RespTimeValue is hardcoded. The hypervisor only gives us a QP#, and we hardcode the rest because it never changes. > > so I left it at zero. Incidentally, there isn't a single > > code line in management.git that actually changes the pkey_index from its > > init value of 0, so I figured that omission couldn't be too bad. > > Agreed. I think you're referring to infiniband-diags and not opensm. Yes, sorry -- none of {infiniband_diags,libibumad,libibmad} make any effort of setting up the pkey_index or even a default value for it, so I thought "why bother" =) Here's an updated patch with the SL set and a comment stating that we should look at the P_Key one day. What do you think of it? Regards, Joachim libibmad/include/infiniband/mad.h | 9 +++++++ libibmad/src/gs.c | 6 +++- libibmad/src/rpc.c | 49 +++++++++++++++++++++++++----------- 3 files changed, 47 insertions(+), 17 deletions(-) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index aa27eb5..bdf5158 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -115,6 +115,8 @@ enum MAD_ATTR_ID { enum MAD_STATUS { IB_MAD_STS_OK = (0 << 2), + IB_MAD_STS_BUSY = (1 << 0), + IB_MAD_STS_REDIRECT = (1 << 1), IB_MAD_STS_BAD_BASE_VER_OR_CLASS = (1 << 2), IB_MAD_STS_METHOD_NOT_SUPPORTED = (2 << 2), IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), @@ -783,8 +785,15 @@ MAD_EXPORT int madrpc_set_timeout(int timeout); MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, int *mgmt_classes, int num_classes); MAD_EXPORT void mad_rpc_close_port(struct ibmad_port *srcport); + +/* + * On redirection, the dport argument is updated with the redirection target, + * so subsequent MADs will not go through the redirection process again but + * reach the target directly. + */ MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, void *payload, void *rcvdata); + MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *data); diff --git a/libibmad/src/gs.c b/libibmad/src/gs.c index f3d245e..c7e4ff6 100644 --- a/libibmad/src/gs.c +++ b/libibmad/src/gs.c @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, rpc.datasz = IB_PC_DATA_SZ; rpc.dataoffs = IB_PC_DATA_OFFS; - dest->qp = 1; + if (!dest->qp) + dest->qp = 1; if (!dest->qkey) dest->qkey = IB_DEFAULT_QP1_QKEY; @@ -109,7 +110,8 @@ uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, rpc.timeout = timeout; rpc.datasz = IB_PC_DATA_SZ; rpc.dataoffs = IB_PC_DATA_OFFS; - dest->qp = 1; + if (!dest->qp) + dest->qp = 1; if (!dest->qkey) dest->qkey = IB_DEFAULT_QP1_QKEY; diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 07b623d..f2e6d5f 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -189,27 +189,46 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, int status, len; uint8_t sndbuf[1024], rcvbuf[1024], *mad; int timeout, retries; + int redirect = 1; - len = 0; - memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); + while (redirect) { + len = 0; + memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); - if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) - return 0; + if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) + return 0; - timeout = rpc->timeout ? rpc->timeout : - port->timeout ? port->timeout : madrpc_timeout; - retries = port->retries ? port->retries : madrpc_retries; + timeout = rpc->timeout ? rpc->timeout : + port->timeout ? port->timeout : madrpc_timeout; + retries = port->retries ? port->retries : madrpc_retries; - if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, - port->class_agents[rpc->mgtclass], - len, timeout, retries)) < 0) { - IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); - return 0; - } + if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, + port->class_agents[rpc->mgtclass], + len, timeout, retries)) < 0) { + IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); + return 0; + } - mad = umad_get_mad(rcvbuf); + mad = umad_get_mad(rcvbuf); + status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); + + /* check for exact match instead of only the redirect bit; + * that way, weird statuses cause an error, too */ + if (status == IB_MAD_STS_REDIRECT) { + /* update dport for next request and retry */ + dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); + dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); + dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); + dport->sl = mad_get_field(mad, 64, IB_CPI_REDIRECT_SL_F); + /* TODO: Reverse map redirection P_Key to P_Key index */ + if (ibdebug) + IBWARN("redirected to lid 0x%x, qp 0x%x, qkey 0x%x, pkey 0x%x", + dport->lid, dport->qp, dport->qkey, dport->pkey_idx); + } else + redirect = 0; + } - if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { + if (status != 0) { ERRS("MAD completed with error status 0x%x; dport (%s)", status, portid2str(dport)); return 0; -- 1.5.5 From jgunthorpe at obsidianresearch.com Tue Jun 30 11:00:03 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 30 Jun 2009 12:00:03 -0600 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: <200906301404.05895.fenkes@de.ibm.com> References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> Message-ID: <20090630180003.GD2982@obsidianresearch.com> On Tue, Jun 30, 2009 at 02:04:03PM +0200, Joachim Fenkes wrote: > On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote: > > On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: > > > > Previously, libibmad reacted to GSI MAD responses with a "redirect" status > > > by throwing an error. IBM eHCA adapters use redirection, so most > > > infiniband_diags tools didn't work against eHCA. > > > > Are there GS classes other than PerfMgt which would be redirected by eHCA ? > > Not right now, no. If you're interested in the details of how and when the > eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c. Hmm.. That definately doesn't look right overall. You are not forming the redirect reply in a way that will work with all possible fabrics. You can't just return a 0 SL and the default PKey and assume things will work out. It looks like all you want to do is redirect to a different QPN? If so I recommend you copy all the values from the incoming MAD's LRH and, if present, GRH into the ClassPortInfo reply. Copy the PKey too. This way you have the best chance of sending back the right information. If there is no GRH then you can use GID index 0 and a 0 TC and a 0 FL. According to the spec returning the port GID is NOT optional. > > > + /* update dport for next request and retry */ > > > + dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); > > > + dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); > > > + dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); This code sould also check for 0 LID and bail. Jason From hal.rosenstock at gmail.com Tue Jun 30 11:37:26 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 30 Jun 2009 14:37:26 -0400 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: <20090630180003.GD2982@obsidianresearch.com> References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> <20090630180003.GD2982@obsidianresearch.com> Message-ID: On Tue, Jun 30, 2009 at 2:00 PM, Jason Gunthorpe wrote: > On Tue, Jun 30, 2009 at 02:04:03PM +0200, Joachim Fenkes wrote: >> On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote: >> > On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: >> >> > > Previously, libibmad reacted to GSI MAD responses with a "redirect" status >> > > by throwing an error. IBM eHCA adapters use redirection, so most >> > > infiniband_diags tools didn't work against eHCA. >> > >> > Are there GS classes other than PerfMgt which would be redirected by eHCA ? >> >> Not right now, no. If you're interested in the details of how and when the >> eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c. > > Hmm.. That definately doesn't look right overall. You are not forming > the redirect reply in a way that will work with all possible > fabrics. You can't just return a 0 SL and the default PKey and assume > things will work out. > > It looks like all you want to do is redirect to a different QPN? If so > I recommend you copy all the values from the incoming MAD's LRH and, > if present, GRH into the ClassPortInfo reply. Copy the PKey too. > > This way you have the best chance of sending back the right information. Agreed. > If there is no GRH then you can use GID index 0 and a 0 TC and a 0 > FL. According to the spec returning the port GID is NOT optional. These are not needed when using LID based redirection (see ClassPortInfo RedirectLID description). -- Hal >> > > +            /* update dport for next request and retry */ >> > > +            dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); >> > > +            dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); >> > > +            dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); > > This code sould also check for 0 LID and bail. > > Jason > From hal.rosenstock at gmail.com Tue Jun 30 11:44:47 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 30 Jun 2009 14:44:47 -0400 Subject: [ofa-general] Re: [PATCH v2] libibmad: Handle MAD redirection In-Reply-To: <200906301853.44361.fenkes@de.ibm.com> References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> <200906301853.44361.fenkes@de.ibm.com> Message-ID: On Tue, Jun 30, 2009 at 12:53 PM, Joachim Fenkes wrote: > Previously, libibmad reacted to GSI MAD responses with a "redirect" status > by throwing an error. IBM eHCA adapters use redirection, so most > infiniband_diags tools didn't work against eHCA. > > Fix: Modify mad_rpc() so that it resends the request to the redirection > target if a "redirect" GS response is received. This is repeated until no > "redirect" response is received, allowing for multiple levels of > indirection. > > The dport argument is updated with the redirection target, so subsequent > MADs will not go through the redirection process again but reach the target > directly. > > Tested using perfquery between ehca, mlx4 and mthca in all possible > combinations. > > Signed-off-by: Joachim Fenkes > --- > > On Tuesday 30 June 2009 16:14, Hal Rosenstock wrote: >> On Tue, Jun 30, 2009 at 8:04 AM, Joachim Fenkes wrote: >> > On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote: >> >> On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes wrote: > >> >> Are there GS classes other than PerfMgt which would be redirected by eHCA ? >> > >> > Not right now, no. If you're interested in the details of how and when the >> > eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c. >> >> SL is always set to 0 and RespTimeValue is hardcoded. > > The hypervisor only gives us a QP#, and we hardcode the rest because it > never changes. I don't think that SL nor RespTimeValue will work in all cases. I think there are better ways to allow for more flexibility here. >> > so I left it at zero. Incidentally, there isn't a single >> > code line in management.git that actually changes the pkey_index from its >> > init value of 0, so I figured that omission couldn't be too bad. >> >> Agreed. I think you're referring to infiniband-diags and not opensm. > > Yes, sorry -- none of {infiniband_diags,libibumad,libibmad} make any effort > of setting up the pkey_index or even a default value for it, so I thought > "why bother" =) > > Here's an updated patch with the SL set and a comment stating that we should > look at the P_Key one day. What do you think of it? Looks OK to me with one comment below. I think the eHCA redirector side can be improved too. -- Hal > Regards, >  Joachim > > >  libibmad/include/infiniband/mad.h |    9 +++++++ >  libibmad/src/gs.c                 |    6 +++- >  libibmad/src/rpc.c                |   49 +++++++++++++++++++++++++----------- >  3 files changed, 47 insertions(+), 17 deletions(-) > > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index aa27eb5..bdf5158 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -115,6 +115,8 @@ enum MAD_ATTR_ID { > >  enum MAD_STATUS { >        IB_MAD_STS_OK                        = (0 << 2), > +       IB_MAD_STS_BUSY                      = (1 << 0), > +       IB_MAD_STS_REDIRECT                  = (1 << 1), >        IB_MAD_STS_BAD_BASE_VER_OR_CLASS     = (1 << 2), >        IB_MAD_STS_METHOD_NOT_SUPPORTED      = (2 << 2), >        IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), > @@ -783,8 +785,15 @@ MAD_EXPORT int madrpc_set_timeout(int timeout); >  MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, >                        int *mgmt_classes, int num_classes); >  MAD_EXPORT void mad_rpc_close_port(struct ibmad_port *srcport); > + > +/* > + * On redirection, the dport argument is updated with the redirection target, > + * so subsequent MADs will not go through the redirection process again but > + * reach the target directly. > + */ >  MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, >                        ib_portid_t * dport, void *payload, void *rcvdata); > + >  MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, >                              ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, >                              void *data); > diff --git a/libibmad/src/gs.c b/libibmad/src/gs.c > index f3d245e..c7e4ff6 100644 > --- a/libibmad/src/gs.c > +++ b/libibmad/src/gs.c > @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, >        rpc.datasz = IB_PC_DATA_SZ; >        rpc.dataoffs = IB_PC_DATA_OFFS; > > -       dest->qp = 1; > +       if (!dest->qp) > +               dest->qp = 1; >        if (!dest->qkey) >                dest->qkey = IB_DEFAULT_QP1_QKEY; > > @@ -109,7 +110,8 @@ uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, >        rpc.timeout = timeout; >        rpc.datasz = IB_PC_DATA_SZ; >        rpc.dataoffs = IB_PC_DATA_OFFS; > -       dest->qp = 1; > +       if (!dest->qp) > +               dest->qp = 1; >        if (!dest->qkey) >                dest->qkey = IB_DEFAULT_QP1_QKEY; > > diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c > index 07b623d..f2e6d5f 100644 > --- a/libibmad/src/rpc.c > +++ b/libibmad/src/rpc.c > @@ -189,27 +189,46 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, >        int status, len; >        uint8_t sndbuf[1024], rcvbuf[1024], *mad; >        int timeout, retries; > +       int redirect = 1; > > -       len = 0; > -       memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > +       while (redirect) { > +               len = 0; > +               memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > > -       if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) > -               return 0; > +               if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) > +                       return 0; > > -       timeout = rpc->timeout ? rpc->timeout : > -           port->timeout ? port->timeout : madrpc_timeout; > -       retries = port->retries ? port->retries : madrpc_retries; > +               timeout = rpc->timeout ? rpc->timeout : > +                       port->timeout ? port->timeout : madrpc_timeout; > +               retries = port->retries ? port->retries : madrpc_retries; > > -       if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > -                             port->class_agents[rpc->mgtclass], > -                             len, timeout, retries)) < 0) { > -               IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > -               return 0; > -       } > +               if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > +                                     port->class_agents[rpc->mgtclass], > +                                     len, timeout, retries)) < 0) { > +                       IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > +                       return 0; > +               } > > -       mad = umad_get_mad(rcvbuf); > +               mad = umad_get_mad(rcvbuf); > +               status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); > + > +               /* check for exact match instead of only the redirect bit; > +                * that way, weird statuses cause an error, too */ > +               if (status == IB_MAD_STS_REDIRECT) { Note here that only LID based redirection supported currently (and perhaps GID based redirection is a TODO). > +                       /* update dport for next request and retry */ > +                       dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F); > +                       dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F); > +                       dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F); > +                       dport->sl = mad_get_field(mad, 64, IB_CPI_REDIRECT_SL_F); > +                       /* TODO: Reverse map redirection P_Key to P_Key index */ > +                       if (ibdebug) > +                               IBWARN("redirected to lid 0x%x, qp 0x%x, qkey 0x%x, pkey 0x%x", > +                                      dport->lid, dport->qp, dport->qkey, dport->pkey_idx); > +               } else > +                       redirect = 0; > +       } > > -       if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { > +       if (status != 0) { >                ERRS("MAD completed with error status 0x%x; dport (%s)", >                     status, portid2str(dport)); >                return 0; > -- > 1.5.5 > > > > From jgunthorpe at obsidianresearch.com Tue Jun 30 11:53:45 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 30 Jun 2009 12:53:45 -0600 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> <20090630180003.GD2982@obsidianresearch.com> Message-ID: <20090630185345.GE2982@obsidianresearch.com> On Tue, Jun 30, 2009 at 02:37:26PM -0400, Hal Rosenstock wrote: > > If there is no GRH then you can use GID index 0 and a 0 TC and a 0 > > FL. According to the spec returning the port GID is NOT optional. > > These are not needed when using LID based redirection (see > ClassPortInfo RedirectLID description). Hmm? RedirectGID RO 128 64 The GID a requester shall use as the destination GID in the GRH of messages used to access redirected class services. If redirection is not being performed, this shall be set to zero. They *might* not be used when the LID is returned, but they still must be set. -- Jason Gunthorpe (780)4406067x832 Chief Technology Officer, Obsidian Research Corp Edmonton, Canada From hal.rosenstock at gmail.com Tue Jun 30 11:57:50 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 30 Jun 2009 14:57:50 -0400 Subject: [ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection In-Reply-To: <20090630185345.GE2982@obsidianresearch.com> References: <200906291410.33477.fenkes@de.ibm.com> <200906301404.05895.fenkes@de.ibm.com> <20090630180003.GD2982@obsidianresearch.com> <20090630185345.GE2982@obsidianresearch.com> Message-ID: On Tue, Jun 30, 2009 at 2:53 PM, Jason Gunthorpe wrote: > On Tue, Jun 30, 2009 at 02:37:26PM -0400, Hal Rosenstock wrote: > >> > If there is no GRH then you can use GID index 0 and a 0 TC and a 0 >> > FL. According to the spec returning the port GID is NOT optional. >> >> These are not needed when using LID based redirection (see >> ClassPortInfo RedirectLID description). > > Hmm? > > RedirectGID RO 128 64 The GID a requester shall use as the destination >                      GID in the GRH of messages used to access redirected class >                      services. If redirection is not being >                      performed, this shall be set to zero. > > They *might* not be used when the LID is returned, but they still > must be set. Sure; they can all be set to 0 as eHCA is doing now. That was all I meant to say. -- Hal > > -- > Jason Gunthorpe        (780)4406067x832 > Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada > From sdietrich at novell.com Tue Jun 30 12:40:07 2009 From: sdietrich at novell.com (Sven-Thorsten Dietrich) Date: Tue, 30 Jun 2009 12:40:07 -0700 Subject: [ofa-general] Re: [ewg] RFC: Do we wish to take MPI out of OFED? Message-ID: <1246390807.7748.20.camel@sven.thebigcorporation.com> Hi, I have just read over the lengthy discussion on the subject, and we have had some internal discussion on this topic since the OFA meeting earlier in the year. Without going into the many individual points: A Linux system in general has many many interdependencies, which are overseen by individual package and subsystem maintainers. They generally seem to manage just fine to keep things working seamlessly. When this is not the case, the issues are usually addressed based on direct input from customers, distro, or dependent system maintainers. Things become confusing when there are multiple POCs for a package, as with MPI in OFED. In regards to integrated QA, it should suffice to specify a version for MPI that have been tested with OFED/RDMA, and this version is something that OFED vendors can sort out, and communicate to distros. The specific source, however does not need to be re-packaged. Not packaging MPI in OFED does not eliminate the ability of the OFED maintainers and OFED users to interact with MPI maintainers to report issues. This notably, happens even with QA'd code, as its generally impossible to anticipate all issues at this level of complexity What is eliminated, is having to re-release the entire OFED package because of changes or updates in MPI. In summary, decoupling as much as possible, while referencing specific versions in QA testing, is fundamental for any integrated system, and I see no need for OFED to odd-man-out on this issue. At Novell we are therefore in favor of decoupling MPI from OFED. Regards, Sven From rajouri.jammu at gmail.com Tue Jun 30 19:41:29 2009 From: rajouri.jammu at gmail.com (Rajouri Jammu) Date: Tue, 30 Jun 2009 19:41:29 -0700 Subject: [ofa-general] Unable to ping/rping on Chelsio cards Message-ID: <3307cdf90906301941q66534a88i99a5b4f31392bde9@mail.gmail.com> Hi, I'm new to Chelsio. I think I followed all the steps mentioned in OFED-1.4.1/docs/cxgb3_release_notes.txt including upgrading the firmware, eeprom etc. The ports on the two nodes appear to be ACTIVE and the IP interfaces seem to be up. However, I'm unable to ping the two nodes and rping is also failing. Do I need to turn on the "peer2peer" option? Any tips or ideas would be much appreciated. One of the nodes had this in /var/log/messages but the port seems to be Active. xgb3 0000:13:00.0: TP parity error (0x8000000) cxgb3 0000:13:00.0: encountered fatal error, operation suspended cxgb3 0000:13:00.0: FW status: 0x0, 0x0, 0x0, 0x0 iw_cxgb3: Chelsio T3 RDMA Driver - version 1.1 iw_cxgb3: Initialized device 0000:13:00.0 ADDRCONF(NETDEV_UP): eth2: link is not ready eth2: link up, 10Gbps, full-duplex ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready eth2: no IPv6 routers present ADDRCONF(NETDEV_UP): eth2: link is not ready eth2: link up, 10Gbps, full-duplex ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready eth2: no IPv6 routers present [root at lv1 cxgb3_0]# lspci | grep Chelsio 13:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter [root at lv1 cxgb3_0]# cat fw_ver T 7.4.0 TP 1.1.0 [root at lv1 ~]# lsmod | grep cxgb iw_cxgb3 107476 1 ib_core 109568 16 rdma_ucm,qlgc_vnic,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_uverbs,ib_umad,iw_nes,iw_cxgb3,ib_ipath,mlx4_ib,ib_mthca,ib_mad cxgb3 161008 1 iw_cxgb3 [root at lv1 chelsio]# cat /sys/class/infiniband/cxgb3_0/ports/1/state 4: ACTIVE eth2 Link encap:Ethernet HWaddr 00:07:43:05:C9:FB inet addr:192.168.10.30 Bcast:192.168.10.255 Mask:255.255.255.0 inet6 addr: fe80::207:43ff:fe05:c9fb/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1038 (1.0 KiB) TX bytes:0 (0.0 b) Interrupt:82 Memory:fdff0000-fdff0fff -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Tue Jun 30 20:16:51 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 30 Jun 2009 20:16:51 -0700 Subject: [ofa-general] Re: XRC userspace git trees In-Reply-To: <20090630113158.GA31362@mtls03> (Eli Cohen's message of "Tue, 30 Jun 2009 14:31:58 +0300") References: <20090630113158.GA31362@mtls03> Message-ID: > where can I find your userspace trees with XRC support? I haven't even started looking at userspace support -- still trying to find time for the kernel side. From acceptany at gmail.com Tue Jun 30 23:00:55 2009 From: acceptany at gmail.com (Jordan) Date: Wed, 1 Jul 2009 14:00:55 +0800 Subject: [ofa-general] Is that difficult to implement a dynamic load balancing in Infiniband? Message-ID: <91fe68d50906302300k2ade4a8cxae7a5f65b427047c@mail.gmail.com> OpenSM supports multiple paths through LMC > 0, but it uses the static load balancing method. This can't always choose the best path to route . I want to implement a dynamic load balancing algorithm in OpenSM so that the multi-path can be used appropriately. Can anybody give me some suggestion ? -------------- next part -------------- An HTML attachment was scrubbed... URL: