From tziporet at mellanox.co.il  Tue Feb  1 02:21:13 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Tue, 1 Feb 2005 12:21:13 +0200 
Subject: [openib-general] roland-uverbs: CQ consumer index
Message-ID: <506C3D7B14CDD411A52C00025558DED6064BEEE0@mtlex01.yok.mtl.com>

Hi,

We thought about it a lot and decided to have two different drivers but with
as much as possible code sharing.
Since realy most code is the same and we did not wanted to duplicate logic
and bug fixes.

The way we did the code sharing is not complaint with Linux but I think one
can come up with another method that will enable to have different code only
to the objects that are required to be different from HW interface.

Tziporet 

-----Original Message-----
From: Roland Dreier [mailto:roland at topspin.com]
Sent: Monday, January 31, 2005 9:33 PM
To: Libor Michalek
Cc: openib-general at openib.org
Subject: Re: [openib-general] roland-uverbs: CQ consumer index


    Libor>   Without knowing how much code difference there will be
    Libor> between Tavor and Arbel native mode, I'm wondering if it
    Libor> makes sense at some point to have seperate hardware drivers
    Libor> for the two parts? Just curious what the your thinking is
    Libor> on the value of maintaining the same code base for the two.

I've gone back and forth between whether it's better to split mthca
into two drivers or keep it as one driver that supports both Tavor and
Arbel modes.  A lot of the code is common but there are also a lot of
differences.  Right now I think keeping one driver is a little better
because it's fairly easy to keep the Arbel-specific code in mthca, and
I keep coming up with places where it would be very awkward to turn
the common code into a library.  However I've gone back and forth in
my mind a bunch of times so we may end up with two drivers before it's
all over.

 - R.
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050201/04d782a2/attachment.html>

From mst at mellanox.co.il  Tue Feb  1 04:32:55 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 1 Feb 2005 14:32:55 +0200
Subject: [openib-general] Re: roland-uverbs: CQ consumer index
In-Reply-To: <52zmyps8av.fsf@topspin.com>
References: <20050131064742.GA8980@mellanox.co.il> <524qgxtnkp.fsf@topspin.com>
	<20050131112711.A29802@topspin.com> <52zmyps8av.fsf@topspin.com>
Message-ID: <20050201123255.GB27600@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: roland-uverbs: CQ consumer index
> 
>     Libor>   Without knowing how much code difference there will be
>     Libor> between Tavor and Arbel native mode, I'm wondering if it
>     Libor> makes sense at some point to have seperate hardware drivers
>     Libor> for the two parts? Just curious what the your thinking is
>     Libor> on the value of maintaining the same code base for the two.
> 
> I've gone back and forth between whether it's better to split mthca
> into two drivers or keep it as one driver that supports both Tavor and
> Arbel modes.  A lot of the code is common but there are also a lot of
> differences.  Right now I think keeping one driver is a little better
> because it's fairly easy to keep the Arbel-specific code in mthca, and
> I keep coming up with places where it would be very awkward to turn
> the common code into a library.  However I've gone back and forth in
> my mind a bunch of times so we may end up with two drivers before it's
> all over.
> 
>  - R.

So far the shared code looks good, so this simple approach seems to work out
well.
We are paying a bit in memory consumption, but nothing
drastic. I see no adverse effect on performance, at least as far as
IP over IB is concerned.


-- 
MST - Michael S. Tsirkin


From krause at cup.hp.com  Tue Feb  1 07:38:35 2005
From: krause at cup.hp.com (Michael Krause)
Date: Tue, 01 Feb 2005 07:38:35 -0800
Subject: [openib-general] [PATCH] roland-uverbs: possible race condition
In-Reply-To: <20050201072411.GB24472@esmail.cup.hp.com>
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<20050201072411.GB24472@esmail.cup.hp.com>
Message-ID: <6.2.0.14.2.20050201073513.02a82248@esmail.cup.hp.com>

At 11:24 PM 1/31/2005, Grant Grundler wrote:
>On Mon, Jan 31, 2005 at 02:40:18PM -0800, Michael Krause wrote:
> > - INTx are treated as writes from a PCIe transaction perspective.
>
>I remember the INTx are transactions once they hit the PCIe bus.
>But do PCIe native devices emit those INTx transactions directly?
>
>For some reaon I was thinking INTx was still a physical IRQ line for
>legacy PCI compatibility.

All PCIe transactions are in-band TLP.  Hence, an INTx is a special write 
TLP but still a TLP even though they are a PCI compatibility function.


>...
> > - In general, all interrupts (line or MSI/MSI-X) should be strongly 
> ordered
> > relative to other write operations to avoid silent data corruption from
> > occurring.  As such, an interrupt should not pass a memory write when 
> being
> > processed by the chipset.
>
>Ok...I need to keep in mind your reply is for PCIe where "interrupts" 
>means "INTx transaction".

Correct.

>BTW, OS's using legacy PCI OS drivers already do protect themselves from 
>the DMA vs IRQ line delivery races. Or at least linux does. MSI/MSI-X 
>support was only introduce a year or so ago - 4 or 5 years after MSI was 
>introduced into PCI 2.2 spec.

No one had really implemented a credible MSI.  HP proposed MSI-X as a 
natural extension of something we already developed 15+ years ago and 
worked with others in the SIG to get it in place.  I know of several IHV 
who have now implemented and multiple OS are doing so as well.  Hopefully 
it will catch on and become the primary method for interrupt delivery.  It 
will also benefit new technology such as virtualization where a MSI-X 
vector can now target a specific guest as opposed to just a data value that 
has to be handled by a hypervisor.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050201/c6a8e614/attachment.html>

From sean.hefty at intel.com  Tue Feb  1 09:05:35 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 1 Feb 2005 09:05:35 -0800
Subject: [openib-general] CM MRA handling
In-Reply-To: <000201c507c2$9aa9b430$8d5aa8c0@infiniconsys.com>
Message-ID: <ORSMSX401FRaqbC8wSA00000048@orsmsx401.amr.corp.intel.com>

>I was thinking about how to handle receipt of the MRA CM message.  Rather
>than add timer logic to the CM code base, why not leverage the timeout
>facility of the MAD agent?  When receiving an MRA, cancel whatever MAD is
>in progress (REQ, REP, or LAP), and reissue the exact same MAD with a 
>timeout as requested in the MRA.  This allows using the same MAD timeout
>logic for MRA_XXX_RCVD as for the original message.
>
>The down side of this is that you end up issuing an extra message for every
>MRA received (in effect responding to each MRA) with a repeat of the
>message that was MRA'd).

I thought about doing this exact same thing.  But I didn't like the idea of
sending a response to the MRA, which could generate a second MRA.  Also, I'm
guessing that handling timewait will require adding timer logic to the CM
code base anyway, so I'm hoping to leverage that.

- Sean 



From ftillier at infiniconsys.com  Tue Feb  1 09:07:45 2005
From: ftillier at infiniconsys.com (Fab Tillier)
Date: Tue, 1 Feb 2005 09:07:45 -0800
Subject: [openib-general] CM MRA handling
In-Reply-To: <ORSMSX401FRaqbC8wSA00000048@orsmsx401.amr.corp.intel.com>
Message-ID: <000301c50880$8d3d0630$8d5aa8c0@infiniconsys.com>

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> Sent: Tuesday, February 01, 2005 9:06 AM
> 
> >I was thinking about how to handle receipt of the MRA CM message.  Rather
> >than add timer logic to the CM code base, why not leverage the timeout
> >facility of the MAD agent?  When receiving an MRA, cancel whatever MAD is
> >in progress (REQ, REP, or LAP), and reissue the exact same MAD with a
> >timeout as requested in the MRA.  This allows using the same MAD timeout
> >logic for MRA_XXX_RCVD as for the original message.
> >
> >The down side of this is that you end up issuing an extra message for
> every
> >MRA received (in effect responding to each MRA) with a repeat of the
> >message that was MRA'd).
> 
> I thought about doing this exact same thing.  But I didn't like the idea
> of
> sending a response to the MRA, which could generate a second MRA.  Also,
> I'm
> guessing that handling timewait will require adding timer logic to the CM
> code base anyway, so I'm hoping to leverage that.
> 

Looking through the spec some more, it's pretty clear that the CM should not
resend the message for which the MRA was received until the indicated time
in the MRA plus the packet lifetime for the path.

- Fab



From rpearson at systemfabricworks.com  Tue Feb  1 14:20:53 2005
From: rpearson at systemfabricworks.com (Robert Pearson)
Date: Tue, 1 Feb 2005 16:20:53 -0600
Subject: [openib-general] Disabling IRQ #201 message
Message-ID: <001501c508ac$4b89db10$6a01a8c0@sfw.int>

Am running current version of openib on a 2.6.11-rc1 kernel on a NewIsis
dual Opteron system. Every 15-20 minutes the following occurs. Have
others seen this behavior? Is the system misconfigured?

 

Feb  1 16:13:25 localhost kernel: irq 201: nobody cared!

Feb  1 16:13:25 localhost kernel:

Feb  1 16:13:25 localhost kernel: Call Trace:<IRQ>
<ffffffff80159c80>{__report_bad_irq+48}
<ffffffff80159d59>{note_interrupt+89}

Feb  1 16:13:25 localhost kernel:
<ffffffff80159629>{__do_IRQ+281} <ffffffff80111542>{do_IRQ+66}

Feb  1 16:13:25 localhost kernel:
<ffffffff8010eac7>{ret_from_intr+0}  <EOI>
<ffffffff8010c720>{default_idle+0}

Feb  1 16:13:25 localhost kernel:
<ffffffff8010c740>{default_idle+32} <ffffffff8010c88f>{cpu_idle+63}

Feb  1 16:13:25 localhost kernel:

Feb  1 16:13:25 localhost kernel: handlers:

Feb  1 16:13:25 localhost kernel: [<ffffffff880e6a30>]
(mthca_interrupt+0x0/0xa0 [ib_mthca])

Feb  1 16:13:25 localhost kernel: Disabling IRQ #201

 

Bob Pearson

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050201/573a8482/attachment.html>

From roland at topspin.com  Tue Feb  1 16:30:57 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 01 Feb 2005 16:30:57 -0800
Subject: [openib-general] Disabling IRQ #201 message
In-Reply-To: <001501c508ac$4b89db10$6a01a8c0@sfw.int> (Robert Pearson's
	message of "Tue, 1 Feb 2005 16:20:53 -0600")
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
Message-ID: <52brb3dcqm.fsf@topspin.com>

    Robert> Am running current version of openib on a 2.6.11-rc1
    Robert> kernel on a NewIsis dual Opteron system. Every 15-20
    Robert> minutes the following occurs. Have others seen this
    Robert> behavior? Is the system misconfigured?

Do the drivers work other than this messsage?

It seems occasionally an interrupt occurs but the driver is not
finding an events in any of the event queues.  I've never seen this
but on the other hand I've not done much testing on the
Opteron/AMD-8131 platform.

 - R.


From roland at topspin.com  Tue Feb  1 19:27:17 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 01 Feb 2005 19:27:17 -0800
Subject: [openib-general] [PATCH] roland-uverbs: possible race condition
In-Reply-To: <6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com> (Michael
	Krause's message of "Mon, 31 Jan 2005 14:40:18 -0800")
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
Message-ID: <521xbz1w16.fsf@topspin.com>

    Michael> Ordering matrix is documented within the PCIe base 1.0a
    Michael> specification.  The rules are fairly straight forward
    Michael> depending upon the capabilities being accessed.  Too much
    Michael> to type in here but to give you an idea:

    Michael> - INTx are treated as writes from a PCIe transaction
    Michael> perspective.

Thanks, I had found that matrix.  However I was not able to find any
language about ordering of interrupts and writes outside of the PCI
Express domain -- the question is whether an interrupt could pass a
memory write transaction upstream of the root complex.  For example,
one could imagine a PCIe host bridge for a CPU that has no provision
for in-band interrupt signaling, so the host bridge must signal all
interrupts by asserting an interrupt pin that is independent of the
CPU's memory bus.  In this case what guarantees ordering?  Another
similar situation would be a PCI-PCIe reverse bridge where the bridge
has to convert PCIe INTx messages to real PCI interrupt pins -- it
seems impossible for any ordering to be guaranteed.

In fact, section 6.1.3 of the PCIe spec says

    "Note that similarly to physical interrupt signals, the INTx
     emulation mechanism may potentially cause spurious interrupts that
     must be handled by the system software."

which is conspicuously silent on ordering issues but seems to me to be
saying "watch out."

Thanks,
  Roland


From mst at mellanox.co.il  Wed Feb  2 05:37:05 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 15:37:05 +0200
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <52brb3dcqm.fsf@topspin.com>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com>
Message-ID: <20050202133705.GD5037@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: Disabling IRQ #201 message
> 
>     Robert> Am running current version of openib on a 2.6.11-rc1
>     Robert> kernel on a NewIsis dual Opteron system. Every 15-20
>     Robert> minutes the following occurs. Have others seen this
>     Robert> behavior? Is the system misconfigured?
> 
> Do the drivers work other than this messsage?
> 
> It seems occasionally an interrupt occurs but the driver is not
> finding an events in any of the event queues.  I've never seen this
> but on the other hand I've not done much testing on the
> Opteron/AMD-8131 platform.
> 
>  - R.

In my opinion, this is unavoidable on any platform in mthca
unless each EQ has a separate interrupt.
Roland, the reason you dont see it may be because you test more
with MSI.

Pls consider:

EQ0 drives interrupt (low)
IRQ handler clears interrupt (high)
EQ1 drives interrupt (low)
IRQ handler reads ECR
IRQ handler serves EQ0 and EQ1
IRQ handler responds with IRQ handled

all EQs are now empty, but interrupt line is low (asserted).

I propose returning IRQ_HANDLED *always* in regular interrupt.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_eq.c
===================================================================
--- hw/mthca/mthca_eq.c	(revision 1673)
+++ hw/mthca/mthca_eq.c	(working copy)
@@ -360,14 +360,12 @@ static irqreturn_t mthca_interrupt(int i
 {
 	struct mthca_dev *dev = dev_ptr;
 	u32 ecr;
-	int work = 0;
 	int i;
 
 	if (dev->eq_table.clr_mask)
 		writel(dev->eq_table.clr_mask, dev->eq_table.clr_int);
 
 	if ((ecr = readl(dev->ecr_base + 4)) != 0) {
-		work = 1;
 
 		writel(ecr, dev->ecr_base +
 		       MTHCA_ECR_CLR_BASE - MTHCA_ECR_BASE + 4);
@@ -377,7 +375,7 @@ static irqreturn_t mthca_interrupt(int i
 				mthca_eq_int(dev, &dev->eq_table.eq[i]);
 	}
 
-	return IRQ_RETVAL(work);
+	return IRQ_HANDLED;
 }
 
 static irqreturn_t mthca_msi_x_interrupt(int irq, void *eq_ptr,
-- 
MST - Michael S. Tsirkin


From rpearson at systemfabricworks.com  Wed Feb  2 08:15:50 2005
From: rpearson at systemfabricworks.com (Robert Pearson)
Date: Wed, 2 Feb 2005 10:15:50 -0600
Subject: [openib-general] Disabling IRQ #201 message
In-Reply-To: <52brb3dcqm.fsf@topspin.com>
Message-ID: <000601c50942$767ef350$6a01a8c0@sfw.int>

I'm debugging some code that is reading files in /sys/class/infiniband/.
Other than that the HCA isn't doing anything at all. The dropped
interrupt occurs whether or not I am doing anything. I can reboot the
machine and just let it sit there and the message will occur after a
while. After the message, files which require interacting with the HCA
e.g. /sys/class/infiniband/mthca0/ports/1/state become unreadable. Read
calls block for a long time and finally timeout with an EOF indication.

> -----Original Message-----
> From: Roland Dreier [mailto:roland at topspin.com]
> Sent: Tuesday, February 01, 2005 6:31 PM
> To: Robert Pearson
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] Disabling IRQ #201 message
> 
>     Robert> Am running current version of openib on a 2.6.11-rc1
>     Robert> kernel on a NewIsis dual Opteron system. Every 15-20
>     Robert> minutes the following occurs. Have others seen this
>     Robert> behavior? Is the system misconfigured?
> 
> Do the drivers work other than this messsage?
> 
> It seems occasionally an interrupt occurs but the driver is not
> finding an events in any of the event queues.  I've never seen this
> but on the other hand I've not done much testing on the
> Opteron/AMD-8131 platform.
> 
>  - R.






From mst at mellanox.co.il  Wed Feb  2 09:08:21 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 19:08:21 +0200
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <000601c50942$767ef350$6a01a8c0@sfw.int>
References: <52brb3dcqm.fsf@topspin.com>
	<000601c50942$767ef350$6a01a8c0@sfw.int>
Message-ID: <20050202170821.GA6058@mellanox.co.il>


> > -----Original Message-----
> > From: Roland Dreier [mailto:roland at topspin.com]
> > Sent: Tuesday, February 01, 2005 6:31 PM
> > To: Robert Pearson
> > Cc: openib-general at openib.org
> > Subject: Re: [openib-general] Disabling IRQ #201 message
> > 
> >     Robert> Am running current version of openib on a 2.6.11-rc1
> >     Robert> kernel on a NewIsis dual Opteron system. Every 15-20
> >     Robert> minutes the following occurs. Have others seen this
> >     Robert> behavior? Is the system misconfigured?
> > 
> > Do the drivers work other than this messsage?
> > 
> > It seems occasionally an interrupt occurs but the driver is not
> > finding an events in any of the event queues.  I've never seen this
> > but on the other hand I've not done much testing on the
> > Opteron/AMD-8131 platform.
> > 
> >  - R.
> 

Quoting r. Robert Pearson <rpearson at systemfabricworks.com>:
> Subject: RE: Disabling IRQ #201 message
> 
> I'm debugging some code that is reading files in /sys/class/infiniband/.
> Other than that the HCA isn't doing anything at all. The dropped
> interrupt occurs whether or not I am doing anything. I can reboot the
> machine and just let it sit there and the message will occur after a
> while. After the message, files which require interacting with the HCA
> e.g. /sys/class/infiniband/mthca0/ports/1/state become unreadable. Read
> calls block for a long time and finally timeout with an EOF indication.

Do you have ip over ib loaded, and/or is sm running on the subnet?
If yes, things are happening every now and then: arp, sm sweep ... .

-- 
MST - Michael S. Tsirkin


From shaharf at voltaire.com  Wed Feb  2 09:43:46 2005
From: shaharf at voltaire.com (shaharf)
Date: Wed, 2 Feb 2005 19:43:46 +0200
Subject: [openib-general] New utility
Message-ID: <D4F8F0B3820E754C887699BEF26A894054C91F@taurus.voltaire.com>

New utility named ibroute is added. This utility shows the linear
forwarding configuration and multicast forwarding configuration of a
switch.

 

Any comments and suggestions are welcomed.

 

Shahar

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050202/0b5e1ed8/attachment.html>

From roland at topspin.com  Wed Feb  2 11:07:01 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 11:07:01 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202133705.GD5037@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 2 Feb 2005 15:37:05 +0200")
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
Message-ID: <52y8e6ye5m.fsf@topspin.com>

    mst> Pls consider:

    mst> EQ0 drives interrupt (low)
    mst> IRQ handler clears interrupt (high)
    mst> EQ1 drives interrupt (low)
    mst> IRQ handler reads ECR
    mst> IRQ handler serves EQ0 and EQ1
    mst> IRQ handler responds with IRQ handled

    mst> all EQs are now empty, but interrupt line is low (asserted).

I agree that this scenario is possible and will probably result in a
return of IRQ_NONE from the next call to the interrupt handler.
However, this should be OK since about half of the interrupts will be
handled.

    mst> I propose returning IRQ_HANDLED *always* in regular interrupt.

We may be forced to do that, but it's pretty ugly (since it defeats
the kernel's detection of stuck interrupts).

Hmm, looking at kernel/irq/spurious.c I think I see a bug... I'll take
this up on lkml.

 - R.


From roland at topspin.com  Wed Feb  2 11:08:35 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 11:08:35 -0800
Subject: [openib-general] Disabling IRQ #201 message
In-Reply-To: <000601c50942$767ef350$6a01a8c0@sfw.int> (Robert Pearson's
	message of "Wed, 2 Feb 2005 10:15:50 -0600")
References: <000601c50942$767ef350$6a01a8c0@sfw.int>
Message-ID: <52u0ouye30.fsf@topspin.com>

    Robert> I'm debugging some code that is reading files in
    Robert> /sys/class/infiniband/.  Other than that the HCA isn't
    Robert> doing anything at all. The dropped interrupt occurs
    Robert> whether or not I am doing anything. I can reboot the
    Robert> machine and just let it sit there and the message will
    Robert> occur after a while. After the message, files which
    Robert> require interacting with the HCA
    Robert> e.g. /sys/class/infiniband/mthca0/ports/1/state become
    Robert> unreadable. Read calls block for a long time and finally
    Robert> timeout with an EOF indication.

I think I may see a kernel bug that can cause this.  In the meantime
booting with the "noirqdebug" kernel option should work around the
problem.

 - R.


From roland at topspin.com  Wed Feb  2 11:12:09 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 11:12:09 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <52y8e6ye5m.fsf@topspin.com> (Roland Dreier's message of "Wed,
	02 Feb 2005 11:07:01 -0800")
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<52y8e6ye5m.fsf@topspin.com>
Message-ID: <52psziydx2.fsf@topspin.com>

    Roland> Hmm, looking at kernel/irq/spurious.c I think I see a
    Roland> bug... I'll take this up on lkml.

Atually I misread the code.  But I still don't understand how we could
ever return IRQ_NONE for 99900 out of 100000 calls to our interrupt
handler -- that would seem to require the HCA to generate many
interrupts in a row with all EQs empty.

 - R.


From iod00d at hp.com  Wed Feb  2 11:27:33 2005
From: iod00d at hp.com (Grant Grundler)
Date: Wed, 2 Feb 2005 11:27:33 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202133705.GD5037@mellanox.co.il>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
Message-ID: <20050202192733.GB29821@esmail.cup.hp.com>

On Wed, Feb 02, 2005 at 03:37:05PM +0200, Michael S. Tsirkin wrote:
> Pls consider:
> 
> EQ0 drives interrupt (low)
> IRQ handler clears interrupt (high)
> EQ1 drives interrupt (low)
> IRQ handler reads ECR
> IRQ handler serves EQ0 and EQ1
> IRQ handler responds with IRQ handled

I think a step is missing here.

> 
> all EQs are now empty, but interrupt line is low (asserted).

If the IRQ line is asserted and we get another interrupt for
something we've already handled, then it means the regular
interrupt code really hasn't cleaned up sufficiently.
There has to be some way to tell the card what work has
been done *OR* flush all inflight DMA (MMIO read - Ouch!).
The interrupt handler needs to guarantee it has handled
pending work and communicatied that back to the card so
IRQ line is not asserted.

> I propose returning IRQ_HANDLED *always* in regular interrupt.

This totally breaks Shared IRQ support.
If the driver asserts it must have a private IRQ line
in it's request_irq() call, then I would be ok with this.

grant


From rpearson at systemfabricworks.com  Wed Feb  2 11:31:46 2005
From: rpearson at systemfabricworks.com (Robert Pearson)
Date: Wed, 2 Feb 2005 13:31:46 -0600
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <52psziydx2.fsf@topspin.com>
Message-ID: <003501c5095d$d6222e60$6a01a8c0@sfw.int>

Michael,

I ran an experiment where I rebooted the machine, waited for an hour and
then looked at /var/log/messages which I copy below. OpenSM is not
running and IPoverIB is not loaded. The dropped interrupt occurs about
30 minutes after the reboot. I do not know what if anything is causing
activity in the HCA. Port 1 is cabled to an unmanaged switch but stuck
in INIT state since there is no running SM.

I was mistaken about the machine. We have two dual Opteron machines, but
this one is an AMD engineering machine, not the NewIsys machine.

There is a message below about some device that does not call
pci_enable_device() which may be interesting. 

Feb  2 11:20:48 localhost syslogd 1.4.1: restart.
Feb  2 11:20:48 localhost syslog: syslogd startup succeeded
Feb  2 11:20:48 localhost kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Feb  2 11:20:48 localhost kernel: Bootdata ok (command line is ro
root=/dev/VolGroup00/LogVol00 rhgb quiet)
Feb  2 11:20:48 localhost kernel: Linux version 2.6.11-rc1rbp
(root at localhost.localdomain) (gcc version 3.4.2 20041017 (Red Hat
3.4.2-6.fc3)) #1 SMP Fri Jan 21 11:23:08 CST 2005
Feb  2 11:20:48 localhost kernel: BIOS-provided physical RAM map:
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 0000000000000000 -
000000000009fc00 (usable)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 000000000009fc00 -
00000000000a0000 (reserved)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 00000000000e0000 -
0000000000100000 (reserved)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 0000000000100000 -
00000000e7ff0000 (usable)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 00000000e7ff0000 -
00000000e7fff000 (ACPI data)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 00000000e7fff000 -
00000000e8000000 (ACPI NVS)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 00000000ff7c0000 -
0000000100000000 (reserved)
Feb  2 11:20:48 localhost kernel:  BIOS-e820: 0000000100000000 -
0000000400000000 (usable)
Feb  2 11:20:48 localhost syslog: klogd startup succeeded
Feb  2 11:20:48 localhost portmap: portmap startup succeeded
Feb  2 11:20:48 localhost rpc.statd[2680]: Version 1.0.6 Starting
Feb  2 11:20:48 localhost kernel: Scanning NUMA topology in Northbridge
24
Feb  2 11:20:48 localhost nfslock: rpc.statd startup succeeded
Feb  2 11:20:48 localhost kernel: Number of nodes 2 (10010)
Feb  2 11:20:48 localhost kernel: Node 0 already present. Skipping
Feb  2 11:20:48 localhost kernel: Node 1 already present. Skipping
Feb  2 11:20:48 localhost kernel: No NUMA configuration found
Feb  2 11:20:48 localhost kernel: Faking a node at
0000000000000000-0000000400000000
Feb  2 11:20:48 localhost kernel: Bootmem setup node 0
0000000000000000-0000000400000000
Feb  2 11:20:48 localhost kernel: ACPI: LAPIC (acpi_id[0x01]
lapic_id[0x00] enabled)
Feb  2 11:20:48 localhost kernel: Processor #0 15:5 APIC version 16
Feb  2 11:20:48 localhost kernel: ACPI: LAPIC (acpi_id[0x02]
lapic_id[0x01] enabled)
Feb  2 11:20:48 localhost rpcidmapd: rpc.idmapd startup succeeded
Feb  2 11:20:48 localhost kernel: Processor #1 15:5 APIC version 16
Feb  2 11:20:49 localhost kernel: ACPI: IOAPIC (id[0x02]
address[0xfec00000] gsi_base[0])
Feb  2 11:20:49 localhost kernel: IOAPIC[0]: apic_id 2, version 17,
address 0xfec00000, GSI 0-23
Feb  2 11:20:49 localhost kernel: ACPI: IOAPIC (id[0x03]
address[0xfebfe000] gsi_base[24])
Feb  2 11:20:49 localhost kernel: IOAPIC[1]: apic_id 3, version 17,
address 0xfebfe000, GSI 24-27
Feb  2 11:20:49 localhost netfs: Mounting other filesystems:  succeeded
Feb  2 11:20:49 localhost kernel: ACPI: IOAPIC (id[0x04]
address[0xfebff000] gsi_base[28])
Feb  2 11:20:49 localhost rc: Starting lm_sensors:  succeeded
Feb  2 11:20:49 localhost kernel: IOAPIC[2]: apic_id 4, version 17,
address 0xfebff000, GSI 28-31
Feb  2 11:20:49 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0
global_irq 2 dfl dfl)
Feb  2 11:20:49 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0
global_irq 2 dfl dfl)
Feb  2 11:20:49 localhost autofs: automount startup succeeded
Feb  2 11:20:49 localhost kernel: Setting APIC routing to flat
Feb  2 11:20:49 localhost kernel: Using ACPI (MADT) for SMP
configuration information
Feb  2 11:20:49 localhost kernel: Checking aperture...
Feb  2 11:20:49 localhost kernel: CPU 0: aperture @ 10000000 size 32 MB
Feb  2 11:20:49 localhost kernel: Aperture from northbridge cpu 0 too
small (32 MB)
Feb  2 11:20:49 localhost kernel: No AGP bridge found
Feb  2 11:20:49 localhost kernel: Your BIOS doesn't leave a aperture
memory hole
Feb  2 11:20:49 localhost kernel: Please enable the IOMMU option in the
BIOS setup
Feb  2 11:20:49 localhost kernel: This costs you 64 MB of RAM
Feb  2 11:20:49 localhost kernel: Mapping aperture over 65536 KB of RAM
@ 10000000
Feb  2 11:20:49 localhost kernel: Built 1 zonelists
Feb  2 11:20:49 localhost kernel: Kernel command line: ro
root=/dev/VolGroup00/LogVol00 rhgb quiet console=tty0
Feb  2 11:20:49 localhost kernel: Initializing CPU#0
Feb  2 11:20:49 localhost mDNSResponder:  startup succeeded
Feb  2 11:20:49 localhost kernel: PID hash table entries: 4096 (order:
12, 131072 bytes)
Feb  2 11:20:49 localhost kernel: time.c: Using 1.193182 MHz PIT timer.
Feb  2 11:20:49 localhost kernel: time.c: Detected 2191.515 MHz
processor.
Feb  2 11:20:49 localhost kernel: Console: colour VGA+ 80x25
Feb  2 11:20:49 localhost kernel: Dentry cache hash table entries:
4194304 (order: 13, 33554432 bytes)
Feb  2 11:20:49 localhost kernel: Inode-cache hash table entries:
2097152 (order: 12, 16777216 bytes)
Feb  2 11:20:49 localhost kernel: Memory: 16033072k/16777216k available
(2208k kernel code, 0k reserved, 1168k data, 220k init)
Feb  2 11:20:49 localhost smartd[2834]: smartd version 5.33
[x86_64-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen 
Feb  2 11:20:50 localhost smartd[2834]: Home page is
http://smartmontools.sourceforge.net/  
Feb  2 11:20:50 localhost kernel: Security Framework v1.0.0 initialized
Feb  2 11:20:50 localhost smartd[2834]: Opened configuration file
/etc/smartd.conf 
Feb  2 11:20:50 localhost kernel: SELinux:  Initializing.
Feb  2 11:20:50 localhost smartd[2834]: Configuration file
/etc/smartd.conf parsed. 
Feb  2 11:20:50 localhost kernel: SELinux:  Starting in permissive mode
Feb  2 11:20:50 localhost smartd[2834]: Device: /dev/sda, opened 
Feb  2 11:20:50 localhost kernel: selinux_register_security:
Registering secondary module capability
Feb  2 11:20:50 localhost kernel: Capability LSM initialized as
secondary
Feb  2 11:20:50 localhost kernel: Mount-cache hash table entries: 256
(order: 0, 4096 bytes)
Feb  2 11:20:50 localhost smartd[2834]: Device: /dev/sda, is SMART
capable. Adding to "monitor" list. 
Feb  2 11:20:50 localhost kernel: CPU: L1 I Cache: 64K (64 bytes/line),
D cache 64K (64 bytes/line)
Feb  2 11:20:50 localhost smartd[2834]: Monitoring 0 ATA and 1 SCSI
devices 
Feb  2 11:20:50 localhost kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Feb  2 11:20:50 localhost smartd[2834]: Device: /dev/sda, initial
Temperature is 27 Celsius 
Feb  2 11:20:50 localhost kernel: CPU 0(1) -> Node 0
Feb  2 11:20:50 localhost smartd[2834]:     [trip Temperature is 68
Celsius] 
Feb  2 11:20:50 localhost kernel: Using local APIC NMI watchdog using
perfctr0
Feb  2 11:20:50 localhost smartd[2836]: smartd has fork()ed into
background mode. New PID=2836. 
Feb  2 11:20:50 localhost smartd: smartd startup succeeded
Feb  2 11:20:50 localhost kernel: CPU: L1 I Cache: 64K (64 bytes/line),
D cache 64K (64 bytes/line)
Feb  2 11:20:50 localhost kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Feb  2 11:20:50 localhost acpid: acpid startup succeeded
Feb  2 11:20:50 localhost kernel: CPU 0(1) -> Node 0
Feb  2 11:20:50 localhost kernel: CPU0: AMD Opteron(tm) Processor 838
stepping 0a
Feb  2 11:20:48 localhost NET: /sbin/dhclient-script : updated
/etc/resolv.conf 
Feb  2 11:20:50 localhost kernel: per-CPU timeslice cutoff: 1024.16
usecs.
Feb  2 11:20:48 localhost dhclient: bound to 192.168.1.119 -- renewal in
4608396 seconds. 
Feb  2 11:20:51 localhost kernel: task migration cache decay timeout: 2
msecs.
Feb  2 11:20:48 localhost ifup:  done. 
Feb  2 11:20:51 localhost kernel: Booting processor 1/1 rip 6000 rsp
ffff81000f025f58
Feb  2 11:20:48 localhost network: Bringing up interface eth0:
succeeded 
Feb  2 11:20:51 localhost kernel: Initializing CPU#1
Feb  2 11:20:52 localhost kernel: CPU: L1 I Cache: 64K (64 bytes/line),
D cache 64K (64 bytes/line)
Feb  2 11:20:52 localhost cups: cupsd startup succeeded
Feb  2 11:20:52 localhost kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Feb  2 11:20:52 localhost kernel: CPU 1(1) -> Node 0
Feb  2 11:20:52 localhost sshd:  succeeded
Feb  2 11:20:52 localhost kernel: AMD Opteron(tm) Processor 838 stepping
0a
Feb  2 11:20:52 localhost kernel: Total of 2 processors activated
(8683.52 BogoMIPS).
Feb  2 11:20:52 localhost xinetd: xinetd startup succeeded
Feb  2 11:20:52 localhost kernel: Using local APIC timer interrupts.
Feb  2 11:20:52 localhost kernel: Detected 12.451 MHz APIC timer.
Feb  2 11:20:52 localhost kernel: checking TSC synchronization across 2
CPUs: passed.
Feb  2 11:20:52 localhost kernel: time.c: Using PIT/TSC based
timekeeping.
Feb  2 11:20:52 localhost kernel: Brought up 2 CPUs
Feb  2 11:20:52 localhost sendmail: sendmail startup succeeded
Feb  2 11:20:53 localhost sendmail: sm-client startup succeeded
Feb  2 11:20:53 localhost gpm[2946]: *** info [startup.c(95)]: 
Feb  2 11:20:53 localhost gpm[2946]: Started gpm successfully. Entered
daemon mode.
Feb  2 11:20:53 localhost kernel: checking if image is initramfs... it
is
Feb  2 11:20:53 localhost kernel: NET: Registered protocol family 16
Feb  2 11:20:53 localhost kernel: PCI: Using configuration type 1
Feb  2 11:20:53 localhost kernel: mtrr: v2.0 (20020519)
Feb  2 11:20:53 localhost kernel: ACPI: Subsystem revision 20041210
Feb  2 11:20:53 localhost kernel:     ACPI-1138: *** Error: Method
execution failed [\MCTH] (Node ffff810037e67440), AE_AML_BUFFER_LIMIT
Feb  2 11:20:53 localhost kernel:     ACPI-1138: *** Error: Method
execution failed [\OSFL] (Node ffff810037e67480), AE_AML_BUFFER_LIMIT
Feb  2 11:20:53 localhost kernel:     ACPI-1138: *** Error: Method
execution failed [\GPRW] (Node ffff810037e67400), AE_AML_BUFFER_LIMIT
Feb  2 11:20:53 localhost kernel:     ACPI-1138: *** Error: Method
execution failed [\_SB_.PCI0.PCI1._PRW] (Node ffff810037e65c40),
AE_AML_BUFFER_LIMIT
Feb  2 11:20:53 localhost kernel:     ACPI-0158: *** Error: Method
execution failed [\_SB_.PCI0.PCI1._PRW] (Node ffff810037e65c40),
AE_AML_BUFFER_LIMIT
Feb  2 11:20:53 localhost kernel: ACPI: Interpreter enabled
Feb  2 11:20:53 localhost kernel: ACPI: Using IOAPIC for interrupt
routing
Feb  2 11:20:53 localhost kernel: ACPI: PCI Root Bridge [PCI0] (00:00)
Feb  2 11:20:53 localhost gpm[2946]: *** info [mice.c(1766)]: 
Feb  2 11:20:53 localhost kernel: PCI: Probing PCI hardware (bus 00)
Feb  2 11:20:53 localhost gpm[2946]: imps2: Auto-detected intellimouse
PS/2
Feb  2 11:20:53 localhost kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs
3 4 5 6 7 9 10 *11 12 14 15)
Feb  2 11:20:53 localhost kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs
3 4 *5 6 7 9 10 11 12 14 15)
Feb  2 11:20:53 localhost kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs
3 4 5 6 7 9 *10 11 12 14 15)
Feb  2 11:20:53 localhost kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs
3 4 5 6 7 *9 10 11 12 14 15)
Feb  2 11:20:53 localhost kernel: PCI: Using ACPI for IRQ routing
Feb  2 11:20:53 localhost kernel: ** PCI interrupts are no longer routed
automatically.  If this
Feb  2 11:20:53 localhost kernel: ** causes a device to stop working, it
is probably because the
Feb  2 11:20:53 localhost kernel: ** driver failed to call
pci_enable_device().  As a temporary
Feb  2 11:20:53 localhost kernel: ** workaround, the "pci=routeirq"
argument restores the old
Feb  2 11:20:53 localhost kernel: ** behavior.  If this argument makes
the device work again,
Feb  2 11:20:53 localhost xinetd[2906]: xinetd Version 2.3.13 started
with libwrap loadavg options compiled in.
Feb  2 11:20:53 localhost gpm: gpm startup succeeded
Feb  2 11:20:53 localhost kernel: ** please email the output of "lspci"
to bjorn.helgaas at hp.com
Feb  2 11:20:53 localhost xinetd[2906]: Started working: 0 available
services
Feb  2 11:20:53 localhost kernel: ** so I can fix the driver.
Feb  2 11:20:53 localhost kernel: PCI-DMA: Disabling AGP.
Feb  2 11:20:53 localhost crond: crond startup succeeded
Feb  2 11:20:53 localhost kernel: PCI-DMA: aperture base @ 10000000 size
65536 KB
Feb  2 11:20:53 localhost kernel: PCI-DMA: Reserving 64MB of IOMMU area
in the AGP aperture
Feb  2 11:20:53 localhost kernel: IA32 emulation $Id: sys_ia32.c,v 1.32
2002/03/24 13:02:28 ak Exp $
Feb  2 11:20:53 localhost kernel: audit: initializing netlink socket
(disabled)
Feb  2 11:20:53 localhost kernel: audit(1107343159.528:0): initialized
Feb  2 11:20:54 localhost kernel: Total HugeTLB memory allocated, 0
Feb  2 11:20:54 localhost xfs: xfs startup succeeded
Feb  2 11:20:54 localhost kernel: VFS: Disk quotas dquot_6.5.1
Feb  2 11:20:54 localhost xfs[2982]: ignoring font path element
/usr/X11R6/lib/X11/fonts/Speedo (unreadable) 
Feb  2 11:20:54 localhost kernel: Dquot-cache hash table entries: 512
(order 0, 4096 bytes)
Feb  2 11:20:54 localhost anacron: anacron startup succeeded
Feb  2 11:20:54 localhost kernel: SELinux:  Registering netfilter hooks
Feb  2 11:20:54 localhost kernel: Initializing Cryptographic API
Feb  2 11:20:54 localhost atd: atd startup succeeded
Feb  2 11:20:54 localhost kernel: pci_hotplug: PCI Hot Plug PCI Core
version: 0.5
Feb  2 11:20:54 localhost kernel: ACPI: Processor [CPU1] (supports 8
throttling states)
Feb  2 11:20:54 localhost readahead: Starting background readahead: 
Feb  2 11:20:54 localhost kernel: Real Time Clock Driver v1.12
Feb  2 11:20:54 localhost kernel: hpet_acpi_add: no address or irqs in
_CRS
Feb  2 11:20:54 localhost kernel: Linux agpgart interface v0.100 (c)
Dave Jones
Feb  2 11:20:54 localhost kernel: [drm] Initialized drm 1.0.0 20040925
Feb  2 11:20:55 localhost kernel: ACPI: PS/2 Keyboard Controller [PS2K]
at I/O 0x60, 0x64, irq 1
Feb  2 11:20:55 localhost rc: Starting readahead:  succeeded
Feb  2 11:20:55 localhost kernel: ACPI: PS/2 Mouse Controller [PS2M] at
irq 12
Feb  2 11:20:55 localhost messagebus: messagebus startup succeeded
Feb  2 11:20:55 localhost kernel: serio: i8042 AUX port at 0x60,0x64 irq
12
Feb  2 11:20:56 localhost cups-config-daemon: cups-config-daemon startup
succeeded
Feb  2 11:20:56 localhost kernel: serio: i8042 KBD port at 0x60,0x64 irq
1
Feb  2 11:20:56 localhost haldaemon: haldaemon startup succeeded
Feb  2 11:20:56 localhost kernel: Serial: 8250/16550 driver $Revision:
1.90 $ 76 ports, IRQ sharing enabled
Feb  2 11:20:56 localhost fstab-sync[3046]: removed all generated mount
points
Feb  2 11:20:57 localhost kernel: ttyS0 at I/O 0x3f8 (irq = 4) is a
16550A
Feb  2 11:20:58 localhost kernel: io scheduler noop registered
Feb  2 11:20:58 localhost kernel: io scheduler anticipatory registered
Feb  2 11:20:58 localhost kernel: io scheduler deadline registered
Feb  2 11:20:58 localhost kernel: io scheduler cfq registered
Feb  2 11:20:58 localhost kernel: RAMDISK driver initialized: 16 RAM
disks of 16384K size 1024 blocksize
Feb  2 11:20:58 localhost kernel: Uniform Multi-Platform E-IDE driver
Revision: 7.00alpha2
Feb  2 11:20:58 localhost kernel: ide: Assuming 33MHz system bus speed
for PIO modes; override with idebus=xx
Feb  2 11:20:58 localhost kernel: AMD8111: IDE controller at PCI slot
0000:00:07.1
Feb  2 11:20:58 localhost kernel: AMD8111: chipset revision 3
Feb  2 11:20:58 localhost kernel: AMD8111: not 100%% native mode: will
probe irqs later
Feb  2 11:20:58 localhost kernel: AMD8111: 0000:00:07.1 (rev 03) UDMA100
controller
Feb  2 11:20:58 localhost kernel:     ide0: BM-DMA at 0xffa0-0xffa7,
BIOS settings: hda:pio, hdb:pio
Feb  2 11:20:58 localhost kernel:     ide1: BM-DMA at 0xffa8-0xffaf,
BIOS settings: hdc:pio, hdd:DMA
Feb  2 11:20:58 localhost kernel: hdd: DV-28E-B, ATAPI CD/DVD-ROM drive
Feb  2 11:20:58 localhost kernel: elevator: using anticipatory as
default io scheduler
Feb  2 11:20:58 localhost kernel: ide1 at 0x170-0x177,0x376 on irq 15
Feb  2 11:20:58 localhost fstab-sync[3329]: added mount point
/media/cdrom for /dev/hdd
Feb  2 11:20:59 localhost kernel: hdd: ATAPI 24X DVD-ROM drive, 256kB
Cache, UDMA(33)
Feb  2 11:20:59 localhost kernel: Uniform CD-ROM driver Revision: 3.20
Feb  2 11:20:59 localhost kernel: ide-floppy driver 0.99.newide
Feb  2 11:20:59 localhost kernel: mice: PS/2 mouse device common for all
mice
Feb  2 11:20:59 localhost kernel: input: AT Translated Set 2 keyboard on
isa0060/serio0
Feb  2 11:20:59 localhost kernel: input: ImExPS/2 Generic Explorer Mouse
on isa0060/serio1
Feb  2 11:20:59 localhost kernel: md: md driver 0.90.1 MAX_MD_DEVS=256,
MD_SB_DISKS=27
Feb  2 11:20:59 localhost kernel: NET: Registered protocol family 2
Feb  2 11:20:59 localhost kernel: IP: routing cache hash table of 65536
buckets, 1024Kbytes
Feb  2 11:20:59 localhost kernel: TCP established hash table entries:
262144 (order: 10, 4194304 bytes)
Feb  2 11:20:59 localhost kernel: TCP bind hash table entries: 65536
(order: 8, 1048576 bytes)
Feb  2 11:20:59 localhost kernel: TCP: Hash tables configured
(established 262144 bind 65536)
Feb  2 11:20:59 localhost kernel: Initializing IPsec netlink socket
Feb  2 11:20:59 localhost kernel: NET: Registered protocol family 1
Feb  2 11:20:59 localhost kernel: NET: Registered protocol family 17
Feb  2 11:20:59 localhost kernel: powernow-k8: Found 2 AMD Athlon 64 /
Opteron processors (version 1.00.09e)
Feb  2 11:20:59 localhost kernel: powernow-k8: MP systems not supported
by PSB BIOS structure
Feb  2 11:20:59 localhost kernel: powernow-k8: MP systems not supported
by PSB BIOS structure
Feb  2 11:20:59 localhost kernel: ACPI wakeup devices: 
Feb  2 11:20:59 localhost kernel: PCI1 USB0 ETHR USB1 GOLA GOLB PWRB 
Feb  2 11:20:59 localhost kernel: ACPI: (supports S0 S1 S5)
Feb  2 11:20:59 localhost kernel: Freeing unused kernel memory: 220k
freed
Feb  2 11:20:59 localhost kernel: SCSI subsystem initialized
Feb  2 11:20:59 localhost kernel: Fusion MPT base driver 3.01.18
Feb  2 11:20:59 localhost kernel: Copyright (c) 1999-2004 LSI Logic
Corporation
Feb  2 11:20:59 localhost kernel: ACPI: PCI interrupt 0000:02:01.0[A] ->
GSI 29 (level, low) -> IRQ 169
Feb  2 11:20:59 localhost kernel: mptbase: Initiating ioc0 bringup
Feb  2 11:20:59 localhost kernel: ioc0: 53C1030:
Capabilities={Initiator}
Feb  2 11:20:59 localhost kernel: ACPI: PCI interrupt 0000:02:01.1[B] ->
GSI 30 (level, low) -> IRQ 177
Feb  2 11:20:59 localhost kernel: mptbase: Initiating ioc1 bringup
Feb  2 11:20:59 localhost kernel: ioc1: 53C1030:
Capabilities={Initiator}
Feb  2 11:20:59 localhost kernel: Fusion MPT SCSI Host driver 3.01.18
Feb  2 11:20:59 localhost kernel: scsi0 : ioc0: LSI53C1030,
FwRev=01030600h, Ports=1, MaxQ=255, IRQ=169
Feb  2 11:20:59 localhost kernel: scsi1 : ioc1: LSI53C1030,
FwRev=01030600h, Ports=1, MaxQ=255, IRQ=177
Feb  2 11:20:59 localhost kernel:   Vendor: SEAGATE   Model: ST336607LC
Rev: 0007
Feb  2 11:20:59 localhost kernel:   Type:   Direct-Access
ANSI SCSI revision: 03
Feb  2 11:20:59 localhost kernel: SCSI device sda: 71687372 512-byte
hdwr sectors (36704 MB)
Feb  2 11:20:59 localhost kernel: SCSI device sda: drive cache: write
back
Feb  2 11:20:59 localhost kernel: SCSI device sda: 71687372 512-byte
hdwr sectors (36704 MB)
Feb  2 11:20:59 localhost kernel: SCSI device sda: drive cache: write
back
Feb  2 11:20:59 localhost kernel:  sda: sda1 sda2
Feb  2 11:20:59 localhost kernel: Attached scsi disk sda at scsi1,
channel 0, id 0, lun 0
Feb  2 11:20:59 localhost kernel:   Vendor: SDR       Model: GEM318P
Rev: 1   
Feb  2 11:20:59 localhost kernel:   Type:   Processor
ANSI SCSI revision: 02
Feb  2 11:20:59 localhost kernel: device-mapper: 4.3.0-ioctl
(2004-09-30) initialised: dm-devel at redhat.com
Feb  2 11:21:00 localhost kernel: cdrom: open failed.
Feb  2 11:21:00 localhost kernel: kjournald starting.  Commit interval 5
seconds
Feb  2 11:21:00 localhost kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Feb  2 11:21:00 localhost kernel: SELinux:  Disabled at runtime.
Feb  2 11:21:01 localhost kernel: SELinux:  Unregistering netfilter
hooks
Feb  2 11:21:01 localhost kernel: Floppy drive(s): fd0 is 1.44M
Feb  2 11:21:01 localhost kernel: FDC 0 is a post-1991 82077
Feb  2 11:21:01 localhost kernel: tg3.c:v3.15 (January 6, 2005)
Feb  2 11:21:01 localhost kernel: ACPI: PCI interrupt 0000:02:03.0[A] ->
GSI 31 (level, low) -> IRQ 185
Feb  2 11:21:01 localhost kernel: eth0: Tigon3 [partno(BCM95704) rev
2002 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet
00:00:1a:18:f6:62
Feb  2 11:21:01 localhost kernel: eth0: RXcsums[1] LinkChgREG[0]
MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
Feb  2 11:21:01 localhost kernel: ACPI: PCI interrupt 0000:02:03.1[B] ->
GSI 28 (level, low) -> IRQ 193
Feb  2 11:21:01 localhost kernel: eth1: Tigon3 [partno(BCM95704) rev
2002 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet
00:00:1a:18:f6:63
Feb  2 11:21:01 localhost kernel: eth1: RXcsums[1] LinkChgREG[0]
MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
Feb  2 11:21:01 localhost kernel: ib_mthca: Mellanox InfiniBand HCA
driver v0.06-pre (November 8, 2004)
Feb  2 11:21:01 localhost kernel: ib_mthca: Initializing  (0000:04:00.0)
Feb  2 11:21:01 localhost kernel: ACPI: PCI interrupt 0000:04:00.0[A] ->
GSI 25 (level, low) -> IRQ 201
Feb  2 11:21:01 localhost kernel: md: Autodetecting RAID arrays.
Feb  2 11:21:01 localhost kernel: md: autorun ...
Feb  2 11:21:01 localhost kernel: md: ... autorun DONE.
Feb  2 11:21:01 localhost kernel: NET: Registered protocol family 10
Feb  2 11:21:01 localhost kernel: Disabled Privacy Extensions on device
ffffffff8040c380(lo)
Feb  2 11:21:01 localhost kernel: IPv6 over IPv4 tunneling driver
Feb  2 11:21:01 localhost kernel: mtrr: type mismatch for
fc000000,800000 old: uncachable new: write-combining
Feb  2 11:21:01 localhost kernel: mtrr: type mismatch for
fc000000,800000 old: uncachable new: write-combining
Feb  2 11:21:01 localhost kernel: ACPI: Power Button (FF) [PWRF]
Feb  2 11:21:01 localhost kernel: ibm_acpi: ec object not found
Feb  2 11:21:01 localhost kernel: EXT3 FS on dm-0, internal journal
Feb  2 11:21:01 localhost kernel: cdrom: open failed.
Feb  2 11:21:01 localhost kernel: kjournald starting.  Commit interval 5
seconds
Feb  2 11:21:01 localhost kernel: EXT3 FS on sda1, internal journal
Feb  2 11:21:01 localhost kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Feb  2 11:21:01 localhost kernel: Adding 2031608k swap on
/dev/VolGroup00/LogVol01.  Priority:-1 extents:1
Feb  2 11:21:01 localhost kernel: Linux Kernel Card Services
Feb  2 11:21:01 localhost kernel:   options:  [pci] [cardbus] [pm]
Feb  2 11:21:02 localhost kernel: ip_tables: (C) 2000-2002 Netfilter
core team
Feb  2 11:21:02 localhost kernel: ip_tables: (C) 2000-2002 Netfilter
core team
Feb  2 11:21:02 localhost kernel: tg3: eth0: Link is up at 100 Mbps,
full duplex.
Feb  2 11:21:02 localhost kernel: tg3: eth0: Flow control is on for TX
and on for RX.
Feb  2 11:21:02 localhost kernel: mtrr: type mismatch for
fc000000,800000 old: uncachable new: write-combining
Feb  2 11:21:02 localhost kernel: mtrr: type mismatch for
fc000000,800000 old: uncachable new: write-combining
Feb  2 11:50:06 localhost kernel: irq 201: nobody cared!
Feb  2 11:50:06 localhost kernel: 
Feb  2 11:50:06 localhost kernel: Call Trace:<IRQ>
<ffffffff80159c80>{__report_bad_irq+48}
<ffffffff80159d59>{note_interrupt+89} 
Feb  2 11:50:06 localhost kernel:
<ffffffff80159629>{__do_IRQ+281} <ffffffff80111542>{do_IRQ+66} 
Feb  2 11:50:06 localhost kernel:
<ffffffff8010eac7>{ret_from_intr+0}  <EOI>
<ffffffff8010c720>{default_idle+0} 
Feb  2 11:50:06 localhost kernel:
<ffffffff8010c740>{default_idle+32} <ffffffff8010c88f>{cpu_idle+63} 
Feb  2 11:50:06 localhost kernel:        
Feb  2 11:50:06 localhost kernel: handlers:
Feb  2 11:50:06 localhost kernel: [<ffffffff880e6a30>]
(mthca_interrupt+0x0/0xa0 [ib_mthca])
Feb  2 11:50:06 localhost kernel: Disabling IRQ #201





From libor at topspin.com  Wed Feb  2 11:40:17 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 2 Feb 2005 11:40:17 -0800
Subject: [openib-general] CM oops in cm_free_msg()
In-Reply-To: <ORSMSX401FRaqbC8wSA00000040@orsmsx401.amr.corp.intel.com>;
	from sean.hefty@intel.com on Wed, Jan 26, 2005 at 08:49:27PM -0800
References: <20050126180609.K30310@topspin.com>
	<ORSMSX401FRaqbC8wSA00000040@orsmsx401.amr.corp.intel.com>
Message-ID: <20050202114017.A31792@topspin.com>

On Wed, Jan 26, 2005 at 08:49:27PM -0800, Sean Hefty wrote:
>
> >  On another note, calling ib_cm_establish from CQ callback context, I
> >would expect to see a callback from the CM that I've entered ESTABLISHED
> >state, but looking at the code, that does not appear to be the case. Is
> >this something that wasn't intended, or just isn't there yet?
> 
> This wasn't intended.  The assumption being that the user knows that the
> connection was already established.  I don't think that it would be too hard
> to add.  (The only issue that jumps to mind at the moment could be a
> potential race condition processing a DREQ.)  Is this something that you
> need?  Would you need the callback from one of the CM's threads?

Sean,

  Aside from not getting a callback after issuing a ib_cm_establish()
The other holdup on the active connect side, is that on the old CM I
would get a established callback based on the RTU being sent. If either
of these calls (ib_cm_establish or ib_send_cm_rtu) generated an
established callback, I would not have noticed that the other wasn't
generating a callback. Can we have callbacks for both?

-Libor


From roland at topspin.com  Wed Feb  2 11:41:58 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 11:41:58 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <003501c5095d$d6222e60$6a01a8c0@sfw.int> (Robert Pearson's
	message of "Wed, 2 Feb 2005 13:31:46 -0600")
References: <003501c5095d$d6222e60$6a01a8c0@sfw.int>
Message-ID: <52is5aycjd.fsf@topspin.com>

    Robert> I was mistaken about the machine. We have two dual Opteron
    Robert> machines, but this one is an AMD engineering machine, not
    Robert> the NewIsys machine.

Yikes -- I'd be worried that this problem may be caused by hardware or
BIOS problems in an "engineering machine."

    Robert> There is a message below about some device that does not
    Robert> call pci_enable_device() which may be interesting.

Nope, that's just a generic warning about a kernel change that was
made recently.  mthca has always done pci_enable_device correctly.

 - R.


From roland at topspin.com  Wed Feb  2 11:46:39 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 11:46:39 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202192733.GB29821@esmail.cup.hp.com> (Grant Grundler's
	message of "Wed, 2 Feb 2005 11:27:33 -0800")
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
Message-ID: <52ekfyycbk.fsf@topspin.com>

    Grant> If the IRQ line is asserted and we get another interrupt
    Grant> for something we've already handled, then it means the
    Grant> regular interrupt code really hasn't cleaned up
    Grant> sufficiently.  There has to be some way to tell the card
    Grant> what work has been done *OR* flush all inflight DMA (MMIO
    Grant> read - Ouch!).  The interrupt handler needs to guarantee it
    Grant> has handled pending work and communicatied that back to the
    Grant> card so IRQ line is not asserted.

Due to the way interrupts are handled in the Mellanox HCA, I think it
will be quite inefficient to guarantee that the interrupt line is not
asserted when the interrupt handler exits, because:

 A. the HCA has multiple "event queues" (interrupt sources)
 B. clearing interrupts is done by writing to a register (no
    "clear-on-read" when reading the interrupt cause register)

So closing the race between clearing the interrupts and reading the
interrupt cause register is very expensive -- we would end up doing
twice as many MMIO reads as necessary in the common case.

However it should be harmless to handle a "spurious" interrupt
occasionally.  The kernel should only complain if 99900 out of the
last 100000 interrupts are spurious, which we shouldn't see.

 - R.


From mst at mellanox.co.il  Wed Feb  2 11:47:52 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 21:47:52 +0200
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202192733.GB29821@esmail.cup.hp.com>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
Message-ID: <20050202194752.GC6316@mellanox.co.il>

Quoting r. Grant Grundler <iod00d at hp.com>:
> Subject: Re: [openib-general] Re: Disabling IRQ #201 message
> 
> On Wed, Feb 02, 2005 at 03:37:05PM +0200, Michael S. Tsirkin wrote:
> > Pls consider:
> > 
> > EQ0 drives interrupt (low)
> > IRQ handler clears interrupt (high)
> > EQ1 drives interrupt (low)
> > IRQ handler reads ECR
> > IRQ handler serves EQ0 and EQ1
> > IRQ handler responds with IRQ handled
> 
> I think a step is missing here.
> 
> > 
> > all EQs are now empty, but interrupt line is low (asserted).
> 
> If the IRQ line is asserted and we get another interrupt for
> something we've already handled, then it means the regular
> interrupt code really hasn't cleaned up sufficiently.
> There has to be some way to tell the card what work has
> been done *OR* flush all inflight DMA (MMIO read - Ouch!).
> The interrupt handler needs to guarantee it has handled
> pending work and communicatied that back to the card so
> IRQ line is not asserted.

Well, the handler did this, but while it was doing other things
another interrupt got asserted.


> > I propose returning IRQ_HANDLED *always* in regular interrupt.
> 
> This totally breaks Shared IRQ support.

Why does it "totally break" it? We did suppress the interrupt
(write to clear interrupt register) so we return "HANDLED".
Its unavoidable that hardware may re-assert it while
the handler is still running.

> If the driver asserts it must have a private IRQ line
> in it's request_irq() call, then I would be ok with this.
> 
> grant
> 

-- 
MST - Michael S. Tsirkin


From mshefty at ichips.intel.com  Wed Feb  2 11:49:17 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 02 Feb 2005 11:49:17 -0800
Subject: [openib-general] CM oops in cm_free_msg()
In-Reply-To: <20050202114017.A31792@topspin.com>
References: <20050126180609.K30310@topspin.com>
	<ORSMSX401FRaqbC8wSA00000040@orsmsx401.amr.corp.intel.com>
	<20050202114017.A31792@topspin.com>
Message-ID: <42012EBD.3090508@ichips.intel.com>

Libor Michalek wrote:

>   Aside from not getting a callback after issuing a ib_cm_establish()
> The other holdup on the active connect side, is that on the old CM I
> would get a established callback based on the RTU being sent. If either
> of these calls (ib_cm_establish or ib_send_cm_rtu) generated an
> established callback, I would not have noticed that the other wasn't
> generating a callback. Can we have callbacks for both?

I'm almost done putting into place the callback for the established. 
It doesn't seem like you need a callback after calling ib_send_cm_rtu. 
  If that call returns success, then the RTU has been sent.

Also, if the active side is going to establish immediately after 
sending the RTU, then it seems like it can do so after getting the REP.

- Sean


From roland at topspin.com  Wed Feb  2 11:53:04 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 11:53:04 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202194752.GC6316@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 2 Feb 2005 21:47:52 +0200")
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
	<20050202194752.GC6316@mellanox.co.il>
Message-ID: <52acqmyc0v.fsf@topspin.com>

    Michael> Why does it "totally break" it? We did suppress the
    Michael> interrupt (write to clear interrupt register) so we
    Michael> return "HANDLED".  Its unavoidable that hardware may
    Michael> re-assert it while the handler is still running.

Because the purpose of the IRQ_HANDLED return value is to tell the
kernel that our device did indeed generate the interrupt.  If the
interrupt is shared with another device with a stuck interrupt then
always returning IRQ_HANDLED from our handler will prevent the kernel
from detecting the stuck interrupt.

 - R.


From mst at mellanox.co.il  Wed Feb  2 11:54:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 21:54:08 +0200
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <52ekfyycbk.fsf@topspin.com>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
	<52ekfyycbk.fsf@topspin.com>
Message-ID: <20050202195408.GD6316@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] Re: Disabling IRQ #201 message
> 
>     Grant> If the IRQ line is asserted and we get another interrupt
>     Grant> for something we've already handled, then it means the
>     Grant> regular interrupt code really hasn't cleaned up
>     Grant> sufficiently.
>     Grant> There has to be some way to tell the card
>     Grant> what work has been done *OR* flush all inflight DMA (MMIO
>     Grant> read - Ouch!).  The interrupt handler needs to guarantee it
>     Grant> has handled pending work and communicatied that back to the
>     Grant> card so IRQ line is not asserted.
> 
> Due to the way interrupts are handled in the Mellanox HCA, I think it
> will be quite inefficient to guarantee that the interrupt line is not
> asserted when the interrupt handler exits, because:
> 
>  A. the HCA has multiple "event queues" (interrupt sources)
>  B. clearing interrupts is done by writing to a register (no
>     "clear-on-read" when reading the interrupt cause register)
> 
> So closing the race between clearing the interrupts and reading the
> interrupt cause register is very expensive -- we would end up doing
> twice as many MMIO reads as necessary in the common case.

I dont see how our HCA is different here.
Whatever way you choose to clear the interrupt, any device
may re-assert it before you get out of the handler.


> However it should be harmless to handle a "spurious" interrupt
> occasionally.  The kernel should only complain if 99900 out of the
> last 100000 interrupts are spurious, which we shouldn't see.

We may get up to 2 spurious interrupts per one regular one (
one per EQ).
So that is probably some other hardware problem.

>  - R.
> 

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  2 11:59:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 21:59:08 +0200
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <52acqmyc0v.fsf@topspin.com>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
	<20050202194752.GC6316@mellanox.co.il> <52acqmyc0v.fsf@topspin.com>
Message-ID: <20050202195908.GE6316@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] Re: Disabling IRQ #201 message
> 
>     Michael> Why does it "totally break" it? We did suppress the
>     Michael> interrupt (write to clear interrupt register) so we
>     Michael> return "HANDLED".  Its unavoidable that hardware may
>     Michael> re-assert it while the handler is still running.
> 
> Because the purpose of the IRQ_HANDLED return value is to tell the
> kernel that our device did indeed generate the interrupt.  If the
> interrupt is shared with another device with a stuck interrupt then
> always returning IRQ_HANDLED from our handler will prevent the kernel
> from detecting the stuck interrupt.
> 
>  - R.
> 

Roland, I know, but I believe thats a far cry from "totally break it".
Reson being, after the kernel disables a stuck interrupt, any device sharing it
will not function properly anyway.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  2 12:03:01 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 22:03:01 +0200
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <003501c5095d$d6222e60$6a01a8c0@sfw.int>
References: <52psziydx2.fsf@topspin.com>
	<003501c5095d$d6222e60$6a01a8c0@sfw.int>
Message-ID: <20050202200301.GF6316@mellanox.co.il>

Quoting r. Robert Pearson <rpearson at systemfabricworks.com>:
> Subject: RE: [openib-general] Re: Disabling IRQ #201 message
> 
> Michael,
> 
> I ran an experiment where I rebooted the machine, waited for an hour and
> then looked at /var/log/messages which I copy below. OpenSM is not
> running and IPoverIB is not loaded. The dropped interrupt occurs about
> 30 minutes after the reboot. I do not know what if anything is causing
> activity in the HCA. Port 1 is cabled to an unmanaged switch but stuck
> in INIT state since there is no running SM.
> 
> I was mistaken about the machine. We have two dual Opteron machines, but
> this one is an AMD engineering machine, not the NewIsys machine.
> 
> There is a message below about some device that does not call
> pci_enable_device() which may be interesting. 

Roland here claims its a board/BIOS issue at your end.
Try to remove the HCA and put another card in the slot
(btw , is it pci-x or express?)

If you decide its a board/BIOS issue, try with my patch as a work-around.

-- 
MST - Michael S. Tsirkin


From rpearson at systemfabricworks.com  Wed Feb  2 12:21:16 2005
From: rpearson at systemfabricworks.com (Robert Pearson)
Date: Wed, 2 Feb 2005 14:21:16 -0600
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202200301.GF6316@mellanox.co.il>
Message-ID: <003901c50964$bfe59180$6a01a8c0@sfw.int>



> If you decide its a board/BIOS issue, try with my patch as a
work-around.

Sorry, which patch was that?






From roland at topspin.com  Wed Feb  2 12:28:08 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 12:28:08 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <003901c50964$bfe59180$6a01a8c0@sfw.int> (Robert Pearson's
	message of "Wed, 2 Feb 2005 14:21:16 -0600")
References: <003901c50964$bfe59180$6a01a8c0@sfw.int>
Message-ID: <52zmymwvtz.fsf@topspin.com>

    Robert> Sorry, which patch was that?

It's probably easier just to boot with "noirqdebug" on the kernel
command line.  (Michael's patch basically just changes the return
value of mthca_eq.c:mthca_interrupt() from IRQ_RETVAL(work) to
IRQ_HANDLED, which should work just as well)

 - R.


From mst at mellanox.co.il  Wed Feb  2 12:28:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 22:28:12 +0200
Subject: [openib-general] ip over ib tx/rx split
Message-ID: <20050202202812.GA7171@mellanox.co.il>


Hi!
The following simple patch splits the tx and rx completions
for ip over ib to separate cqs.
This shall get better performance on SMP due to reduced cq lock contention.

I dont realy see any negative effect this may have.

Later I plan to look into switching the send cq to polling mode
to reduce interrupts, along the lines of what Ido proposed here.

With this approach, it may be also possible split the completion routine
to send a receive side, thus saving some conditional branches on interrupt path.
I left that for later, too.

Unfortunately I only have one machine alive at the moment,
so it had only limited testing :)
Could someone please tell me what effect it has?

Comments are welcome.

MST

Index: ulp/ipoib/ipoib_verbs.c
===================================================================
--- ulp/ipoib/ipoib_verbs.c	(revision 1716)
+++ ulp/ipoib/ipoib_verbs.c	(working copy)
@@ -175,24 +175,31 @@ int ipoib_transport_dev_init(struct net_
 		return -ENODEV;
 	}
 
-	priv->cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL, dev,
-				IPOIB_TX_RING_SIZE + IPOIB_RX_RING_SIZE + 1);
-	if (IS_ERR(priv->cq)) {
-		printk(KERN_WARNING "%s: failed to create CQ\n", ca->name);
+	priv->tx_cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL, dev,
+				IPOIB_TX_RING_SIZE);
+	if (IS_ERR(priv->tx_cq)) {
+		printk(KERN_WARNING "%s: failed to create TX CQ\n", ca->name);
 		goto out_free_pd;
 	}
 
-	if (ib_req_notify_cq(priv->cq, IB_CQ_NEXT_COMP))
-		goto out_free_cq;
+	priv->rx_cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL, dev,
+				IPOIB_RX_RING_SIZE);
+	if (IS_ERR(priv->rx_cq)) {
+		printk(KERN_WARNING "%s: failed to create RX CQ\n", ca->name);
+		goto out_free_tx_cq;
+	}
+
+	if (ib_req_notify_cq(priv->rx_cq, IB_CQ_NEXT_COMP))
+		goto out_free_rx_cq;
 
 	priv->mr = ib_get_dma_mr(priv->pd, IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(priv->mr)) {
 		printk(KERN_WARNING "%s: ib_get_dma_mr failed\n", ca->name);
-		goto out_free_cq;
+		goto out_free_rx_cq;
 	}
 
-	init_attr.send_cq = priv->cq;
-	init_attr.recv_cq = priv->cq,
+	init_attr.send_cq = priv->tx_cq;
+	init_attr.recv_cq = priv->rx_cq,
 
 	priv->qp = ib_create_qp(priv->pd, &init_attr);
 	if (IS_ERR(priv->qp)) {
@@ -216,8 +223,11 @@ int ipoib_transport_dev_init(struct net_
 out_free_mr:
 	ib_dereg_mr(priv->mr);
 
-out_free_cq:
-	ib_destroy_cq(priv->cq);
+out_free_rx_cq:
+	ib_destroy_cq(priv->rx_cq);
+
+out_free_tx_cq:
+	ib_destroy_cq(priv->tx_cq);
 
 out_free_pd:
 	ib_dealloc_pd(priv->pd);
@@ -239,8 +249,11 @@ void ipoib_transport_dev_cleanup(struct 
 	if (ib_dereg_mr(priv->mr))
 		ipoib_warn(priv, "ib_dereg_mr failed\n");
 
-	if (ib_destroy_cq(priv->cq))
-		ipoib_warn(priv, "ib_cq_destroy failed\n");
+	if (ib_destroy_cq(priv->rx_cq))
+		ipoib_warn(priv, "ib_cq_destroy for rx cq failed\n");
+
+	if (ib_destroy_cq(priv->tx_cq))
+		ipoib_warn(priv, "ib_cq_destroy for tx cq failed\n");
 
 	if (ib_dealloc_pd(priv->pd))
 		ipoib_warn(priv, "ib_dealloc_pd failed\n");
Index: ulp/ipoib/ipoib.h
===================================================================
--- ulp/ipoib/ipoib.h	(revision 1716)
+++ ulp/ipoib/ipoib.h	(working copy)
@@ -137,7 +137,8 @@ struct ipoib_dev_priv {
 	u16           	  pkey;
 	struct ib_pd  	 *pd;
 	struct ib_mr  	 *mr;
-	struct ib_cq  	 *cq;
+	struct ib_cq  	 *tx_cq;
+	struct ib_cq  	 *rx_cq;
 	struct ib_qp  	 *qp;
 	u32           	  qkey;
 
-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  2 12:37:30 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 22:37:30 +0200
Subject: [openib-general] Re: Re: Disabling IRQ #201 message
In-Reply-To: <52zmymwvtz.fsf@topspin.com>
References: <003901c50964$bfe59180$6a01a8c0@sfw.int>
	<52zmymwvtz.fsf@topspin.com>
Message-ID: <20050202203730.GC6455@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: Re: Disabling IRQ #201 message
> 
>     Robert> Sorry, which patch was that?
> 
> It's probably easier just to boot with "noirqdebug" on the kernel
> command line.  (Michael's patch basically just changes the return
> value of mthca_eq.c:mthca_interrupt() from IRQ_RETVAL(work) to
> IRQ_HANDLED, which should work just as well)
> 
>  - R.
> 

By the way, the documentation for "noirqdebug" implies its IA-32 only,
but it seems to be all in the generic code.
Probably a documentation bug?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  2 12:37:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 22:37:49 +0200
Subject: [openib-general] Re: Re: Disabling IRQ #201 message
In-Reply-To: <52zmymwvtz.fsf@topspin.com>
References: <003901c50964$bfe59180$6a01a8c0@sfw.int>
	<52zmymwvtz.fsf@topspin.com>
Message-ID: <20050202203749.GB7171@mellanox.co.il>


Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: Re: Disabling IRQ #201 message
> 
>     Robert> Sorry, which patch was that?

Look in the archives, for subject "Disabling IRQ #201 message",
or just change mthca_eq.c:mthca_interrupt to always return IRQ_HANDLED.

But pls understand it is just a work-around.

> It's probably easier just to boot with "noirqdebug" on the kernel
> command line.  (Michael's patch basically just changes the return
> value of mthca_eq.c:mthca_interrupt() from IRQ_RETVAL(work) to
> IRQ_HANDLED, which should work just as well)
> 
>  - R.
> 

By the way, am I right that with noirqdebug the return value
of the handler is basicaly ignored?

-- 
MST - Michael S. Tsirkin


From mshefty at ichips.intel.com  Wed Feb  2 13:42:41 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 2 Feb 2005 13:42:41 -0800
Subject: [openib-general] [PATCH] CM: add callback for user establishment
Message-ID: <20050202134241.506ab6e2.mshefty@ichips.intel.com>

This patch invokes a callback after a user calls ib_cm_establish.
Users will either receive either an IB_CM_RTU_RECEIVED or
IB_CM_USER_ESTABLISHED event indicating that a connection has been
established on the passive side.

The only item of note is that the connection state goes to ESTABLISHED
in the call to ib_cm_establish.  This is necessary to avoid timimg out
the connection from a missing RTU, but implies that a
DREQ message could be received and processed before the ESTABLISHED
callback is invoked.  (In which case there will not be an ESTABLISH
callback.)

- Sean

Index: include/ib_cm.h
===================================================================
--- include/ib_cm.h	(revision 1720)
+++ include/ib_cm.h	(working copy)
@@ -72,6 +72,7 @@
 	IB_CM_REP_ERROR,
 	IB_CM_REP_RECEIVED,
 	IB_CM_RTU_RECEIVED,
+	IB_CM_USER_ESTABLISHED,
 	IB_CM_DREQ_ERROR,
 	IB_CM_DREQ_RECEIVED,
 	IB_CM_DREP_RECEIVED,
Index: core/cm.c
===================================================================
--- core/cm.c	(revision 1720)
+++ core/cm.c	(working copy)
@@ -36,6 +36,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/err.h>
 #include <linux/idr.h>
+#include <linux/interrupt.h>
 #include <linux/pci.h>
 #include <linux/rbtree.h>
 #include <linux/spinlock.h>
@@ -133,7 +134,8 @@
 	struct work_struct work;
 	struct list_head list;
 	struct cm_port *port;
-	struct ib_mad_recv_wc *mad_recv_wc;
+	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
+	u32 local_id;				/* Established */
 	struct ib_cm_event cm_event;
 	struct ib_sa_path_rec path[];
 };
@@ -922,7 +924,6 @@
 	struct ib_cm_req_event_param *param;
 
 	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
-	work->cm_event.event = IB_CM_REQ_RECEIVED;
 	param = &work->cm_event.param.req_rcvd;
 	param->listen_id = listen_id;
 	param->device = cm_id_priv->av.port->mad_agent->device;
@@ -1262,7 +1263,6 @@
 	struct ib_cm_rep_event_param *param;
 
 	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
-	work->cm_event.event = IB_CM_REP_RECEIVED;
 	param = &work->cm_event.param.rep_rcvd;
 	param->remote_ca_guid = rep_msg->local_ca_guid;
 	param->remote_qkey = be32_to_cpu(rep_msg->local_qkey);
@@ -1341,29 +1341,23 @@
 	return -EINVAL;
 }
 
-static int cm_rtu_handler(struct cm_work *work)
+static int cm_establish_handler(struct cm_work *work)
 {
 	struct cm_id_private *cm_id_priv;
-	struct cm_rtu_msg *rtu_msg;
 	unsigned long flags;
 	u64 wr_id;
 	int ret;
 
-	rtu_msg = (struct cm_rtu_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(rtu_msg->remote_comm_id);
+	/* See comment in ib_cm_establish about lookup. */
+	cm_id_priv = cm_acquire_id_by_local_id(work->local_id);
 	if (!cm_id_priv)
 		return -EINVAL;
 
-	work->cm_event.event = IB_CM_RTU_RECEIVED;
-	work->cm_event.private_data = &rtu_msg->private_data;
-
 	spin_lock_irqsave(&cm_id_priv->lock, flags);
-	if (cm_id_priv->id.state != IB_CM_REP_SENT &&
-	    cm_id_priv->id.state != IB_CM_MRA_REP_RCVD) {
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED) {
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 		goto out;
 	}
-	cm_id_priv->id.state = IB_CM_ESTABLISHED;
 
 	wr_id = (unsigned long) cm_id_priv->msg;
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
@@ -1382,32 +1376,45 @@
 	return -EINVAL;
 }
 
-int ib_cm_establish(struct ib_cm_id *cm_id)
+static int cm_rtu_handler(struct cm_work *work)
 {
 	struct cm_id_private *cm_id_priv;
+	struct cm_rtu_msg *rtu_msg;
 	unsigned long flags;
-	int ret = 0;
+	u64 wr_id;
+	int ret;
 
-	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	rtu_msg = (struct cm_rtu_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id_by_local_id(rtu_msg->remote_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &rtu_msg->private_data;
 
 	spin_lock_irqsave(&cm_id_priv->lock, flags);
-	switch (cm_id->state)
-	{
-	case IB_CM_REP_SENT:
-	case IB_CM_MRA_REP_RCVD:
-		cm_id->state = IB_CM_ESTABLISHED;
-		break;
-	case IB_CM_ESTABLISHED:
-		ret = -EISCONN;
-		break;
-	default:
-		ret = -EINVAL;
-		break;
+	if (cm_id_priv->id.state != IB_CM_REP_SENT &&
+	    cm_id_priv->id.state != IB_CM_MRA_REP_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
 	}
+	cm_id_priv->id.state = IB_CM_ESTABLISHED;
+
+	wr_id = (unsigned long) cm_id_priv->msg;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-	return ret;
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
 }
-EXPORT_SYMBOL(ib_cm_establish);
 
 static void cm_format_dreq(struct cm_dreq_msg *dreq_msg,
 			  struct cm_id_private *cm_id_priv,
@@ -1557,7 +1564,6 @@
 	if (!cm_id_priv)
 		return -EINVAL;
 
-	work->cm_event.event = IB_CM_DREQ_RECEIVED;
 	work->cm_event.private_data = &dreq_msg->private_data;
 
 	spin_lock_irqsave(&cm_id_priv->lock, flags);
@@ -1608,7 +1614,6 @@
 	if (!cm_id_priv)
 		return -EINVAL;
 
-	work->cm_event.event = IB_CM_DREP_RECEIVED;
 	work->cm_event.private_data = &drep_msg->private_data;
 
 	spin_lock_irqsave(&cm_id_priv->lock, flags);
@@ -1960,7 +1965,6 @@
 	if (!cm_id_priv)
 		return -EINVAL;
 	
-	work->cm_event.event = IB_CM_LAP_RECEIVED;
 	param = &work->cm_event.param.lap_rcvd;
 	param->alternate_path = &work->path[0];
 	cm_format_path_from_lap(param->alternate_path, lap_msg);
@@ -2071,7 +2075,6 @@
 	if (!cm_id_priv)
 		return -EINVAL; /* Unmatched reply. */
 
-	work->cm_event.event = IB_CM_APR_RECEIVED;
 	work->cm_event.param.apr_rcvd.ap_status = apr_msg->ap_status;
 	work->cm_event.param.apr_rcvd.apr_info = &apr_msg->info;
 	work->cm_event.param.apr_rcvd.info_len = apr_msg->info_length;
@@ -2185,7 +2188,6 @@
 
 	sidr_req_msg = (struct cm_sidr_req_msg *)
 				work->mad_recv_wc->recv_buf.mad;
-	work->cm_event.event = IB_CM_SIDR_REQ_RECEIVED;
 	param = &work->cm_event.param.sidr_req_rcvd;
 	param->pkey = sidr_req_msg->pkey;
 	param->listen_id = listen_id;
@@ -2324,7 +2326,6 @@
 
 	sidr_rep_msg = (struct cm_sidr_rep_msg *)
 				work->mad_recv_wc->recv_buf.mad;
-	work->cm_event.event = IB_CM_SIDR_REP_RECEIVED;
 	param = &work->cm_event.param.sidr_rep_rcvd;
 	param->status = sidr_rep_msg->status;
 	param->qkey = be32_to_cpu(sidr_rep_msg->qkey);
@@ -2465,38 +2466,41 @@
 	struct cm_work *work = data;
 	int ret;
 
-	switch (work->mad_recv_wc->recv_buf.mad->mad_hdr.attr_id) {
-	case CM_REQ_ATTR_ID:
+	switch (work->cm_event.event) {
+	case IB_CM_REQ_RECEIVED:
 		ret = cm_req_handler(work);
 		break;
-	case CM_MRA_ATTR_ID:
+	case IB_CM_MRA_RECEIVED:
 		ret = cm_mra_handler(work);
 		break;
-	case CM_REJ_ATTR_ID:
+	case IB_CM_REJ_RECEIVED:
 		ret = cm_rej_handler(work);
 		break;
-	case CM_REP_ATTR_ID:
+	case IB_CM_REP_RECEIVED:
 		ret = cm_rep_handler(work);
 		break;
-	case CM_RTU_ATTR_ID:
+	case IB_CM_RTU_RECEIVED:
 		ret = cm_rtu_handler(work);
 		break;
-	case CM_DREQ_ATTR_ID:
+	case IB_CM_USER_ESTABLISHED:
+		ret = cm_establish_handler(work);
+		break;
+	case IB_CM_DREQ_RECEIVED:
 		ret = cm_dreq_handler(work);
 		break;
-	case CM_DREP_ATTR_ID:
+	case IB_CM_DREP_RECEIVED:
 		ret = cm_drep_handler(work);
 		break;
-	case CM_SIDR_REQ_ATTR_ID:
+	case IB_CM_SIDR_REQ_RECEIVED:
 		ret = cm_sidr_req_handler(work);
 		break;
-	case CM_SIDR_REP_ATTR_ID:
+	case IB_CM_SIDR_REP_RECEIVED:
 		ret = cm_sidr_rep_handler(work);
 		break;
-	case CM_LAP_ATTR_ID:
+	case IB_CM_LAP_RECEIVED:
 		ret = cm_lap_handler(work);
 		break;
-	case CM_APR_ATTR_ID:
+	case IB_CM_APR_RECEIVED:
 		ret = cm_apr_handler(work);
 		break;
 	default:
@@ -2507,23 +2511,102 @@
 		cm_free_work(work);
 }
 
+int ib_cm_establish(struct ib_cm_id *cm_id)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_work *work;
+	unsigned long flags;
+	int ret = 0;
+
+	work = kmalloc(sizeof *work, (in_atomic() || irqs_disabled()) ?
+				      GFP_ATOMIC : GFP_KERNEL);
+	if (!work)
+		return -ENOMEM;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state)
+	{
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		cm_id->state = IB_CM_ESTABLISHED;
+		break;
+	case IB_CM_ESTABLISHED:
+		ret = -EISCONN;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret) {
+		kfree(work);
+		goto out;
+	}
+
+	/*
+	 * The CM worker thread may try to destroy the cm_id before it
+	 * can execute this work item.  To prevent potential deadlock,
+	 * we need to find the cm_id once we're in the context of the 
+	 * worker thread, rather than holding a reference on it.
+	 */
+	INIT_WORK(&work->work, cm_work_handler, work);
+	work->local_id = cm_id->local_id;
+	work->cm_event.event = IB_CM_USER_ESTABLISHED;
+	queue_work(cm.wq, &work->work);
+out:
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_establish);
+
 static void cm_recv_handler(struct ib_mad_agent *mad_agent,
 			    struct ib_mad_recv_wc *mad_recv_wc)
 {
 	struct cm_work *work;
-	int paths;
+	enum ib_cm_event_type event;
+	int paths = 0;
 
 	switch (mad_recv_wc->recv_buf.mad->mad_hdr.attr_id) {
 	case CM_REQ_ATTR_ID:
 		paths = 1 + (((struct cm_req_msg *) mad_recv_wc->recv_buf.mad)->
 						    alt_local_lid != 0);
+		event = IB_CM_REQ_RECEIVED;
+		break;
+	case CM_MRA_ATTR_ID:
+		event = IB_CM_MRA_RECEIVED;
+		break;
+	case CM_REJ_ATTR_ID:
+		event = IB_CM_REJ_RECEIVED;
+		break;
+	case CM_REP_ATTR_ID:
+		event = IB_CM_REP_RECEIVED;
+		break;
+	case CM_RTU_ATTR_ID:
+		event = IB_CM_RTU_RECEIVED;
+		break;
+	case CM_DREQ_ATTR_ID:
+		event = IB_CM_DREQ_RECEIVED;
+		break;
+	case CM_DREP_ATTR_ID:
+		event = IB_CM_DREP_RECEIVED;
+		break;
+	case CM_SIDR_REQ_ATTR_ID:
+		event = IB_CM_SIDR_REQ_RECEIVED;
+		break;
+	case CM_SIDR_REP_ATTR_ID:
+		event = IB_CM_SIDR_REP_RECEIVED;
 		break;
 	case CM_LAP_ATTR_ID:
 		paths = 1;
+		event = IB_CM_LAP_RECEIVED;
 		break;
-	default:
-		paths = 0;
+	case CM_APR_ATTR_ID:
+		event = IB_CM_APR_RECEIVED;
 		break;
+	default:
+		ib_free_recv_mad(mad_recv_wc);
+		return;
 	}
 
 	work = kmalloc(sizeof *work + sizeof(struct ib_sa_path_rec) * paths,
@@ -2534,6 +2617,7 @@
 	}
 
 	INIT_WORK(&work->work, cm_work_handler, work);
+	work->cm_event.event = event;
 	work->mad_recv_wc = mad_recv_wc;
 	work->port = (struct cm_port *)mad_agent->context;
 	queue_work(cm.wq, &work->work);


From ftillier at infiniconsys.com  Wed Feb  2 13:05:10 2005
From: ftillier at infiniconsys.com (Fab Tillier)
Date: Wed, 2 Feb 2005 13:05:10 -0800
Subject: [openib-general] CM comments
Message-ID: <000001c5096a$e28a8780$8d5aa8c0@infiniconsys.com>

The state diagrams in the spec seem to imply that the MRA gets resent for
every subsequent message received for which the MRA was originally sent (see
the MRA Sent states).  So a REP received in the MRA(REP) Sent state results
in a resend of the MRA.  Likewise, a REQ received in the MRA(REQ) Sent state
also results in a resend of the MRA.

Are the IB_CM_MRA_XXX_SENT states needed?  I don't see anything in the IB
spec that prohibits a client from sending multiple MRA messages, but maybe I
missed it.  If the CM repeats the MRA as described above, then I think only
allowing the client to send one MRA is fine.

I don't think the LAP message should be retried by the CM.  Section 12.9.8.6
of the spec states that the REQ, REP, and DREQ can be retried.  The LAP does
have a timeout for a response, but no retries.

- Fab



From krause at cup.hp.com  Wed Feb  2 12:54:01 2005
From: krause at cup.hp.com (Michael Krause)
Date: Wed, 02 Feb 2005 12:54:01 -0800
Subject: [openib-general] [PATCH] roland-uverbs: possible race condition
In-Reply-To: <521xbz1w16.fsf@topspin.com>
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
Message-ID: <6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>

At 07:27 PM 2/1/2005, Roland Dreier wrote:
>     Michael> Ordering matrix is documented within the PCIe base 1.0a
>     Michael> specification.  The rules are fairly straight forward
>     Michael> depending upon the capabilities being accessed.  Too much
>     Michael> to type in here but to give you an idea:
>
>     Michael> - INTx are treated as writes from a PCIe transaction
>     Michael> perspective.
>
>Thanks, I had found that matrix.  However I was not able to find any
>language about ordering of interrupts and writes outside of the PCI
>Express domain -- the question is whether an interrupt could pass a
>memory write transaction upstream of the root complex.  For example,
>one could imagine a PCIe host bridge for a CPU that has no provision
>for in-band interrupt signaling, so the host bridge must signal all
>interrupts by asserting an interrupt pin that is independent of the
>CPU's memory bus.

This is a platform / chipset issue and not a PCIe protocol / specification 
issue since the problem is outside the scope of the root complex / root 
port (RC/RP).

>In this case what guarantees ordering?  Another similar situation would be 
>a PCI-PCIe reverse bridge where the bridge has to convert PCIe INTx 
>messages to real PCI interrupt pins -- it seems impossible for any 
>ordering to be guaranteed.

Again, this is a platform / chipset issue.  A reverse bridge would be 
required to maintain the ordering of the PCIe transactions and signal the 
PCI/PCI-X INTx accordingly.  Given the problem is solved for PCI/PCI-X at 
the chipset / platform level, there really isn't anything that PCIe can do 
in terms of specification or rules.


>In fact, section 6.1.3 of the PCIe spec says
>
>     "Note that similarly to physical interrupt signals, the INTx 
> emulation mechanism may potentially cause spurious interrupts that must 
> be handled by the system software."
>
>which is conspicuously silent on ordering issues but seems to me to be 
>saying "watch out."

It is silent because it is outside the scope of PCIe technology.  The 
informational note is there to act as a warning for those that may not be 
experienced in designing these types of solutions.

Let's take a step back and focus on what people believe is a problem for 
OpenIB to solve here.  What do people believe is the root cause?

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050202/a8a20eb3/attachment.html>

From mshefty at ichips.intel.com  Wed Feb  2 13:30:21 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 02 Feb 2005 13:30:21 -0800
Subject: [openib-general] CM comments
In-Reply-To: <000001c5096a$e28a8780$8d5aa8c0@infiniconsys.com>
References: <000001c5096a$e28a8780$8d5aa8c0@infiniconsys.com>
Message-ID: <4201466D.80908@ichips.intel.com>

Fab Tillier wrote:
> The state diagrams in the spec seem to imply that the MRA gets resent for
> every subsequent message received for which the MRA was originally sent (see
> the MRA Sent states).  So a REP received in the MRA(REP) Sent state results
> in a resend of the MRA.  Likewise, a REQ received in the MRA(REQ) Sent state
> also results in a resend of the MRA.

This is my interpretation as well.

> Are the IB_CM_MRA_XXX_SENT states needed?  I don't see anything in the IB
> spec that prohibits a client from sending multiple MRA messages, but maybe I
> missed it.  If the CM repeats the MRA as described above, then I think only
> allowing the client to send one MRA is fine.

For the CM to repeat the MRA, I think it needs these states.  I'm not 
sure what you mean by "multiple MRA messages".

> I don't think the LAP message should be retried by the CM.  Section 12.9.8.6
> of the spec states that the REQ, REP, and DREQ can be retried.  The LAP does
> have a timeout for a response, but no retries.

I think allowing the LAP to be retried by the CM is fine.  Section 
12.9.8 is dealing with connection establishment/teardown, and does not 
include path migration.  Section 12.8 says that the LAP "may be re-sent 
if there is no response".

- Sean


From mst at mellanox.co.il  Wed Feb  2 13:39:32 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 2 Feb 2005 23:39:32 +0200
Subject: [openib-general] Re: [PATCH] roland-uverbs: possible race condition
In-Reply-To: <6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
Message-ID: <20050202213932.GD6455@mellanox.co.il>

Quoting r. Michael Krause <krause at cup.hp.com>:
>     In fact, section 6.1.3 of the PCIe spec says
>    
>         "Note that similarly to physical interrupt signals, the INTx emulation
>     mechanism may potentially cause spurious interrupts that must be handled by
>     the system software."
>    
>     which is conspicuously silent on ordering issues but seems to me to be
>     saying "watch out."
> 
> 
> It is silent because it is outside the scope of PCIe technology.  The
> informational note is there to act as a warning for those that may not be
> experienced in designing these types of solutions. 
> 
> Let's take a step back and focus on what people believe is a problem for OpenIB
> to solve here.  What do people believe is the root cause?
> 
> Mike
> 

Mike, what do you mean by the root cause?

If the CPU may start serving the interrupt
while DMA is not yet committed to memory, we may not rely on the
interrupt/DMA ordering, and its irrelevant whether its PCI-express
or chipset issue.

MST

-- 
MST - Michael S. Tsirkin


From ftillier at infiniconsys.com  Wed Feb  2 13:56:19 2005
From: ftillier at infiniconsys.com (Fab Tillier)
Date: Wed, 2 Feb 2005 13:56:19 -0800
Subject: [openib-general] CM comments
In-Reply-To: <4201466D.80908@ichips.intel.com>
Message-ID: <000101c50972$073961d0$8d5aa8c0@infiniconsys.com>

> From: Sean Hefty [mailto:mshefty at ichips.intel.com]
> Sent: Wednesday, February 02, 2005 1:30 PM
> 
> Fab Tillier wrote:
> > Are the IB_CM_MRA_XXX_SENT states needed?  I don't see anything in
> > the IB spec that prohibits a client from sending multiple MRA
> > messages, but maybe I missed it.  If the CM repeats the MRA as
> > described above, then I think only allowing the client to send
> > one MRA is fine.
> 
> For the CM to repeat the MRA, I think it needs these states.  I'm not
> sure what you mean by "multiple MRA messages".

I meant a client calling ib_cm_send_mra multiple times in a row to keep
extending the timeout.  In any case, you're right that if the CM is to
repeat those MADs, then it needs those states.

- Fab



From mshefty at ichips.intel.com  Wed Feb  2 14:04:19 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 02 Feb 2005 14:04:19 -0800
Subject: [openib-general] CM comments
In-Reply-To: <000101c50972$073961d0$8d5aa8c0@infiniconsys.com>
References: <000101c50972$073961d0$8d5aa8c0@infiniconsys.com>
Message-ID: <42014E63.9050808@ichips.intel.com>

Fab Tillier wrote:

> I meant a client calling ib_cm_send_mra multiple times in a row to keep
> extending the timeout.  In any case, you're right that if the CM is to
> repeat those MADs, then it needs those states.

I would interpret that additional calls are invalid, since no 
transitions are defined after the first MRA has been sent.

- Sean


From roland at topspin.com  Wed Feb  2 14:32:56 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 14:32:56 -0800
Subject: [openib-general] [PATCH] roland-uverbs: possible race condition
In-Reply-To: <6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com> (Michael
	Krause's message of "Wed, 02 Feb 2005 12:54:01 -0800")
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
Message-ID: <52vf9awq1z.fsf@topspin.com>

    Michael> This is a platform / chipset issue and not a PCIe
    Michael> protocol / specification issue since the problem is
    Michael> outside the scope of the root complex / root port
    Michael> (RC/RP).

In that case a driver cannot rely on ordering for PCIe INTx messages
and and must do something like an MMIO read to ensure that the
interrupt has not passed any other writes.  The fact that the spec
seems silent on this point makes me believe this is in fact the case.
Note that this is quite different from MSI/MSI-X, where I see language
in the MSI-X ECN placing ordering requirements on the platform:

    An MSI or MSI-X message, by virtue of being a posted memory write
    (PMW) transaction, is prohibited by PCI ordering rules from
    passing PMW transactions sent earlier by the function. The system
    must guarantee that an interrupt service routine invoked as a
    result of a given message will observe any updates performed by
    PMW transactions arriving prior to that message. Thus, the
    interrupt service routine of a device driver is not required to
    read from a device register in order to ensure data consistency
    with previous PMW transactions.

Since there seems to be no analogous statement about INTx messages in
the PCIe spec, there is no guarantee that an ISR will see all PMW
transactions issued prior to the INTx message.

Thanks,
  Roland


From roland at topspin.com  Wed Feb  2 14:35:47 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 14:35:47 -0800
Subject: [openib-general] ip over ib tx/rx split
In-Reply-To: <20050202202812.GA7171@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 2 Feb 2005 22:28:12 +0200")
References: <20050202202812.GA7171@mellanox.co.il>
Message-ID: <52r7jywpx8.fsf@topspin.com>

    Michael> Hi!  The following simple patch splits the tx and rx
    Michael> completions for ip over ib to separate cqs.  This shall
    Michael> get better performance on SMP due to reduced cq lock
    Michael> contention.

I tried something similar and even split ipoib_ib_completion() into
two functions (to save dispatching between RX and TX completions).  I
was not able to measure any performance improvement.

In fact I'm not convinced that this should be better, since it may
cause more interrupts and reduce the natural interrupt coalescing
effect that having all completions go to the same queue has.

 - Roland


From krause at cup.hp.com  Wed Feb  2 16:19:45 2005
From: krause at cup.hp.com (Michael Krause)
Date: Wed, 02 Feb 2005 16:19:45 -0800
Subject: [openib-general] Re: [PATCH] roland-uverbs: possible race condition
In-Reply-To: <20050202213932.GD6455@mellanox.co.il>
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
	<20050202213932.GD6455@mellanox.co.il>
Message-ID: <6.2.0.14.2.20050202161754.02aec990@esmail.cup.hp.com>

At 01:39 PM 2/2/2005, Michael S. Tsirkin wrote:
>Quoting r. Michael Krause <krause at cup.hp.com>:
> >     In fact, section 6.1.3 of the PCIe spec says
> >
> >         "Note that similarly to physical interrupt signals, the INTx 
> emulation
> >     mechanism may potentially cause spurious interrupts that must be 
> handled by
> >     the system software."
> >
> >     which is conspicuously silent on ordering issues but seems to me to be
> >     saying "watch out."
> >
> >
> > It is silent because it is outside the scope of PCIe technology.  The
> > informational note is there to act as a warning for those that may not be
> > experienced in designing these types of solutions.
> >
> > Let's take a step back and focus on what people believe is a problem 
> for OpenIB
> > to solve here.  What do people believe is the root cause?
> >
> > Mike
> >
>
>Mike, what do you mean by the root cause?
>
>If the CPU may start serving the interrupt while DMA is not yet committed 
>to memory, we may not rely on the
>interrupt/DMA ordering, and its irrelevant whether its PCI-express or 
>chipset issue.

Every driver needs to deal with spurious interrupts irrespective of whether 
it is a a PCIe or chipset issue.  A good h/w / s/w design should not 
require PIO Read to ascertain whether something real or spurious 
happened.   If that is the only concern, then it does not sound like there 
is any real problem here only clarification of our reality being sought.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050202/c1eab249/attachment.html>

From mshefty at ichips.intel.com  Wed Feb  2 16:42:14 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 02 Feb 2005 16:42:14 -0800
Subject: [openib-general] CM timewait issue
Message-ID: <42017366.8070305@ichips.intel.com>

I'm trying to determine the timeout value to use for CM timewait 
states.  Here's what I think so far:

It should be the maximum timewait value for any path ever loaded by the 
client.

For the sender of a REQ: use the path record: packet life time * 2 + 
the target ACK delay returned in the REP.

For the receiver of the REQ, use the local ACK delay.

For the receiver of a LAP: use the local ACK delay.

What should the sender of the LAP use?  The sender will have the path 
record: packet life time, but may not know the remote CA ACK delay.

Should the CM just use the local ACK delay sent or received in the 
REQ/LAP in all cases for simplicity?

- Sean


From libor at topspin.com  Wed Feb  2 16:47:16 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 2 Feb 2005 16:47:16 -0800
Subject: [openib-general] more CM byte ordering.
In-Reply-To: <42017366.8070305@ichips.intel.com>;
	from mshefty@ichips.intel.com on Wed, Feb 02, 2005 at 04:42:14PM
	-0800
References: <42017366.8070305@ichips.intel.com>
Message-ID: <20050202164716.B31792@topspin.com>

Sean,

  I found a couple more spots in the active connect case of incorrect
byte ordering, this time around starting psn. Here is a tested patch.

-Libor


Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1725)
+++ infiniband/core/cm.c	(working copy)
@@ -711,7 +711,7 @@
 				       param->remote_cm_response_timeout);
 	cm_req_set_qp_type(req_msg, param->qp->qp_type);
 	cm_req_set_flow_ctrl(req_msg, param->flow_control);
-	cm_req_set_starting_psn(req_msg, cpu_to_be32(param->starting_psn));
+	cm_req_set_starting_psn(req_msg, param->starting_psn);
 	cm_req_set_local_resp_timeout(req_msg,
 				      param->local_cm_response_timeout);
 	cm_req_set_retry_count(req_msg, param->retry_count);
Index: infiniband/core/cm_msgs.h
===================================================================
--- infiniband/core/cm_msgs.h	(revision 1725)
+++ infiniband/core/cm_msgs.h	(working copy)
@@ -529,7 +529,7 @@
 
 static inline u32 cm_rep_get_starting_psn(struct cm_rep_msg *rep_msg)
 {
-	return cpu_to_be32(be32_to_cpu(rep_msg->offset20) >> 8);
+	return be32_to_cpu(rep_msg->offset20) >> 8;
 }
 
 static inline void cm_rep_set_starting_psn(struct cm_rep_msg *rep_msg,


From libor at topspin.com  Wed Feb  2 18:43:22 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 2 Feb 2005 18:43:22 -0800
Subject: [openib-general] [PATCH] CM: add callback for user establishment
In-Reply-To: <20050202134241.506ab6e2.mshefty@ichips.intel.com>;
	from mshefty@ichips.intel.com on Wed, Feb 02, 2005 at 01:42:41PM
	-0800
References: <20050202134241.506ab6e2.mshefty@ichips.intel.com>
Message-ID: <20050202184322.C31792@topspin.com>

On Wed, Feb 02, 2005 at 01:42:41PM -0800, Sean Hefty wrote:
> This patch invokes a callback after a user calls ib_cm_establish.
> Users will either receive either an IB_CM_RTU_RECEIVED or
> IB_CM_USER_ESTABLISHED event indicating that a connection has been
> established on the passive side.

  Thanks Sean. With this change I can get far enough to run some basic
performance tests. On a pair of 2 CPU 2.4GHz Xeons w/ 2GBs PCI-X 133
systems SDP is getting 610 MB/s throughput and 24 usec latency.

-Libor
















From roland at topspin.com  Wed Feb  2 21:40:16 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 02 Feb 2005 21:40:16 -0800
Subject: [openib-general] [PATCH] InfiniBand: add missing break between cases
Message-ID: <52y8e65hhb.fsf@topspin.com>

From: Libor Michalek <libor at topspin.com>

Add a missing break statement between RC and UD cases in mthca_post_send().
This fixes a possible oops for protocols that use the RC transport.

Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-bk.orig/drivers/infiniband/hw/mthca/mthca_qp.c	2005-01-28 11:11:02.000000000 -0800
+++ linux-bk/drivers/infiniband/hw/mthca/mthca_qp.c	2005-02-02 21:35:09.683871535 -0800
@@ -1323,6 +1323,8 @@
 				break;
 			}
 
+			break;
+
 		case UD:
 			((struct mthca_ud_seg *) wqe)->lkey =
 				cpu_to_be32(to_mah(wr->wr.ud.ah)->key);


From mst at mellanox.co.il  Wed Feb  2 21:50:43 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 07:50:43 +0200
Subject: [openib-general] Re: [PATCH] InfiniBand: add missing break between
	cases
In-Reply-To: <52y8e65hhb.fsf@topspin.com>
References: <52y8e65hhb.fsf@topspin.com>
Message-ID: <20050203055043.GA19348@mellanox.co.il>

BTW, is the lack of break after
case MTHCA_RES_UDAV intentional in mthca_make_profile?
Isnt this what was intended ( I noticed this when comparing trunk with
roland-uverbs)?

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_profile.c
===================================================================
--- hw/mthca/mthca_profile.c	(revision 1673)
+++ hw/mthca/mthca_profile.c	(working copy)
@@ -241,6 +241,7 @@ u64 mthca_make_profile(struct mthca_dev 
 		case MTHCA_RES_UDAV:
 			dev->av_table.ddr_av_base = profile[i].start;
 			dev->av_table.num_ddr_avs = profile[i].num;
+			break;
 		case MTHCA_RES_UARC:
 			init_hca->uarc_base   = profile[i].start;
 			init_hca->log_uarc_sz = ffs(request->uarc_size) - 13;

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  2 22:01:34 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 08:01:34 +0200
Subject: [openib-general] Re: ip over ib tx/rx split
In-Reply-To: <52r7jywpx8.fsf@topspin.com>
References: <20050202202812.GA7171@mellanox.co.il> <52r7jywpx8.fsf@topspin.com>
Message-ID: <20050203060134.GA19563@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: ip over ib tx/rx split
> 
>     Michael> Hi!  The following simple patch splits the tx and rx
>     Michael> completions for ip over ib to separate cqs.  This shall
>     Michael> get better performance on SMP due to reduced cq lock
>     Michael> contention.
> 
> I tried something similar and even split ipoib_ib_completion() into
> two functions (to save dispatching between RX and TX completions).  I
> was not able to measure any performance improvement.

This wa slikely before we removed the qptable lock in cq poll?

> In fact I'm not convinced that this should be better, since it may
> cause more interrupts and reduce the natural interrupt coalescing
> effect that having all completions go to the same queue has.

It cant , interrupt coalescing is done by eq, not by cq.

>  - Roland
> 

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 01:34:05 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 11:34:05 +0200
Subject: [openib-general] [PATCH] bugfix: remove unbalanced qp refcount
Message-ID: <20050203093404.GF28018@mellanox.co.il>

Hi!
It seems when the qp lock removal patch was applied, one chunk
was missing, so we get an unbalanced refcount decrement.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 1725)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -412,8 +412,6 @@ static inline int mthca_poll_one(struct 
 				*freed = 0;
 			}
 			spin_unlock(&(*cur_qp)->lock);
-			if (atomic_dec_and_test(&(*cur_qp)->refcount))
-				wake_up(&(*cur_qp)->wait);
 		}
 
 		/*


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 08:10:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 18:10:49 +0200
Subject: [openib-general] Re: [PATCH] roland-uverbs: possible race condition
In-Reply-To: <52vf9awq1z.fsf@topspin.com>
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
	<52vf9awq1z.fsf@topspin.com>
Message-ID: <20050203161049.GD2536@mellanox.co.il>


Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] roland-uverbs: possible race condition
>
> Since there seems to be no analogous statement about INTx messages in
> the PCIe spec, there is no guarantee that an ISR will see all PMW
> transactions issued prior to the INTx message.
> 
> Thanks,
>   Roland

OK, so are we all in agreement now, that an empty eq, too, must
be re-armed in memfree mode when working in regular interrupt mode?

Please note how that has no extra cost, since as
my patch that started this thread does, you can arm all eqs in one
fell swoop, pre-computing the mask.


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 09:25:59 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 19:25:59 +0200
Subject: [openib-general] Re: Re: [PATCH] (fixed) reduce qp locking on cq
	poll
In-Reply-To: <20050126223101.GA3765@esmail.cup.hp.com>
References: <20050126135451.GI3791@mellanox.co.il> <523bwo2ka2.fsf@topspin.com>
	<20050126171902.GC10743@mellanox.co.il>
	<20050126192810.GA18168@esmail.cup.hp.com>
	<20050126193749.GH10743@mellanox.co.il>
	<20050126195502.GD18168@esmail.cup.hp.com>
	<20050126201322.GD12344@mellanox.co.il>
	<20050126223101.GA3765@esmail.cup.hp.com>
Message-ID: <20050203172559.GF2536@mellanox.co.il>

Quoting r. Grant Grundler <iod00d at hp.com>:
> Subject: Re: Re: [PATCH] (fixed) reduce qp locking on cq poll
> 
> On Wed, Jan 26, 2005 at 10:13:22PM +0200, Michael S. Tsirkin wrote:
> > Its not the speed of hardware, my problem is the variations.
> 
> Ok. Your "log comment" was entirely misleading. It should read
> 	I can't measure a performance difference becuase of my
> 	test results vary by X%.
> 
> Substitute the right value for X.
> 
> > > Did you see the difference in IPoIB numbers that I posted
> > > earlier where I pegged the test processes to the same CPU
> > > taking interrupts or a different CPU? 
> >
> > No - link?
> 
> Sorry - I don't have the URL - just the original mail.
> I posted this on "Date: Thu, 13 Jan 2005 10:43:34 -0800"
> and you should be able to find this in the archives.
> 
> Here is the table again:
> netperf/irq same, netserver/irq same: 1591 Mbps
> netperf/irq diff, netserver/irq same: 1565 Mbps
> netperf/irq same, netserver/irq diff: 1507 Mbps
> netperf/irq diff, netserver/irq diff: 1510 Mbps
> 
> (1591-1510)/1510 is about 5%.
> If you use taskset on your platforms, run a similar table to the
> above, we could compare and see if ia64 is more sensitive to this
> than other arches.
> 
> hth,
> grant
> 

I finally found some time to do this.
taskset does not seem to help:

swlab155:~ # taskset 1 netperf -f M  -c -C -H 11.4.8.156 -- -m 100000
TCP STREAM TEST to 11.4.8.156
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB

 87380  16384 100000    10.00       301.38   44.20    92.95    2.865   6.024
swlab155:~ # taskset 1 netperf -f M  -c -C -H 11.4.8.156 -- -m 100000
TCP STREAM TEST to 11.4.8.156
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB

 87380  16384 100000    10.00       290.86   47.80    89.85    3.210   6.033
swlab155:~ # taskset 1 netperf -f M  -c -C -H 11.4.8.156 -- -m 100000
TCP STREAM TEST to 11.4.8.156
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB

 87380  16384 100000    10.00       295.07   43.46    90.45    2.876   5.987


Bandwidth went anywhere from 301.38 to 295.07.


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 09:54:19 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 19:54:19 +0200
Subject: [openib-general] [PATCH] 1/2 separate locking for send and receive
	q in mthca
Message-ID: <20050203175419.GJ2536@mellanox.co.il>

Hello!
Send and receive q posting both serialise on the same spinlock in mthca.
Not good for SMP.  The following patch solves this.
Together with my rx/tx split patch, I get about 2% more bandwidth on
ip over ib with this.

Applies on top of the "bugfix: remove unbalanced qp refcount" patch.

Now that I see real effect, I'll repost the ip over ib rx/tx split patch, too,
for completeness.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 1725)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -147,6 +147,7 @@ struct mthca_cq {
 };
 
 struct mthca_wq {
+	spinlock_t lock;
 	int   max;
 	int   cur;
 	int   next;
@@ -159,7 +160,6 @@ struct mthca_wq {
 
 struct mthca_qp {
 	struct ib_qp           ibqp;
-	spinlock_t             lock;
 	atomic_t               refcount;
 	u32                    qpn;
 	int                    is_direct;
Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 1725)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -370,12 +370,14 @@ static void dump_cqe(struct mthca_cqe *c
 
 static inline int mthca_poll_one(struct mthca_dev *dev,
 				 struct mthca_cq *cq,
-				 struct mthca_qp **cur_qp,
+				 struct mthca_wq **cur_wq,
 				 int *freed,
 				 struct ib_wc *entry)
 {
 	struct mthca_wq *wq;
+	struct mthca_qp *qp;
 	struct mthca_cqe *cqe;
+	u32 qpn;
 	int wqe_index;
 	int is_error;
 	int is_send;
@@ -392,9 +394,11 @@ static inline int mthca_poll_one(struct 
 	 */
 	rmb();
 
+	qpn = be32_to_cpu(cqe->my_qpn);
+
 	if (0) {
 		mthca_dbg(dev, "%x/%d: CQE -> QPN %06x, WQE @ %08x\n",
-			  cq->cqn, cq->cons_index, be32_to_cpu(cqe->my_qpn),
+			  cq->cqn, cq->cons_index, qpn,
 			  be32_to_cpu(cqe->wqe));
 
 		dump_cqe(cqe);
@@ -404,14 +408,21 @@ static inline int mthca_poll_one(struct 
 		MTHCA_ERROR_CQE_OPCODE_MASK;
 	is_send  = is_error ? cqe->opcode & 0x01 : cqe->is_send & 0x80;
 
-	if (!*cur_qp || be32_to_cpu(cqe->my_qpn) != (*cur_qp)->qpn) {
-		if (*cur_qp) {
+	wq = *cur_wq;
+
+	if (is_send)
+		qp = wq ? container_of(wq, struct mthca_qp, sq) : NULL;
+	else
+		qp = wq ? container_of(wq, struct mthca_qp, rq) : NULL;
+
+	if (!qp || be32_to_cpu(cqe->my_qpn) != qp->qpn) {
+		if (qp) {
 			if (*freed) {
 				wmb();
 				inc_cons_index(dev, cq, *freed);
 				*freed = 0;
 			}
-			spin_unlock(&(*cur_qp)->lock);
+			spin_unlock(&wq->lock);
 		}
 
 		/*
@@ -421,31 +430,29 @@ static inline int mthca_poll_one(struct 
 		 * because CQs will be locked while QPs are removed
 		 * from the table.
 		 */
-		*cur_qp = mthca_array_get(&dev->qp_table.qp,
+		qp = mthca_array_get(&dev->qp_table.qp,
 					  be32_to_cpu(cqe->my_qpn) &
 					  (dev->limits.num_qps - 1));
-		if (!*cur_qp) {
+		if (!qp) {
 			mthca_warn(dev, "CQ entry for unknown QP %06x\n",
 				   be32_to_cpu(cqe->my_qpn) & 0xffffff);
 			err = -EINVAL;
 			goto out;
 		}
 
-		spin_lock(&(*cur_qp)->lock);
+		*cur_wq = wq = is_send ? &qp->sq : &qp->rq;
+		spin_lock(&wq->lock);
 	}
 
-	entry->qp_num = (*cur_qp)->qpn;
+	entry->qp_num = qpn;
 
 	if (is_send) {
-		wq = &(*cur_qp)->sq;
-		wqe_index = ((be32_to_cpu(cqe->wqe) - (*cur_qp)->send_wqe_offset)
+		wqe_index = ((be32_to_cpu(cqe->wqe) - qp->send_wqe_offset)
 			     >> wq->wqe_shift);
-		entry->wr_id = (*cur_qp)->wrid[wqe_index +
-					       (*cur_qp)->rq.max];
+		entry->wr_id = qp->wrid[wqe_index + qp->rq.max];
 	} else {
-		wq = &(*cur_qp)->rq;
 		wqe_index = be32_to_cpu(cqe->wqe) >> wq->wqe_shift;
-		entry->wr_id = (*cur_qp)->wrid[wqe_index];
+		entry->wr_id = qp->wrid[wqe_index];
 	}
 
 	if (wq->last_comp < wqe_index)
@@ -458,10 +465,10 @@ static inline int mthca_poll_one(struct 
 	if (0)
 		mthca_dbg(dev, "%s completion for QP %06x, index %d (nr %d)\n",
 			  is_send ? "Send" : "Receive",
-			  (*cur_qp)->qpn, wqe_index, wq->max);
+			  qp->qpn, wqe_index, wq->max);
 
 	if (is_error) {
-		err = handle_error_cqe(dev, cq, *cur_qp, wqe_index, is_send,
+		err = handle_error_cqe(dev, cq, qp, wqe_index, is_send,
 				       (struct mthca_err_cqe *) cqe,
 				       entry, &free_cqe);
 		goto out;
@@ -515,7 +522,7 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 {
 	struct mthca_dev *dev = to_mdev(ibcq->device);
 	struct mthca_cq *cq = to_mcq(ibcq);
-	struct mthca_qp *qp = NULL;
+	struct mthca_wq *wq = NULL;
 	unsigned long flags;
 	int err = 0;
 	int freed = 0;
@@ -524,7 +531,7 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 	spin_lock_irqsave(&cq->lock, flags);
 
 	for (npolled = 0; npolled < num_entries; ++npolled) {
-		err = mthca_poll_one(dev, cq, &qp,
+		err = mthca_poll_one(dev, cq, &wq,
 				     &freed, entry + npolled);
 		if (err)
 			break;
@@ -535,8 +542,8 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 		inc_cons_index(dev, cq, freed);
 	}
 
-	if (qp)
-		spin_unlock(&qp->lock);
+	if (wq)
+		spin_unlock(&wq->lock);
 
 	spin_unlock_irqrestore(&cq->lock, flags);
 
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 1725)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -552,9 +552,11 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 		else
 			cur_state = attr->cur_qp_state;
 	} else {
-		spin_lock_irq(&qp->lock);
+		spin_lock_irq(&qp->sq.lock);
+		spin_lock(&qp->rq.lock);
 		cur_state = qp->state;
-		spin_unlock_irq(&qp->lock);
+		spin_unlock(&qp->rq.lock);
+		spin_unlock_irq(&qp->sq.lock);
 	}
 
 	if (attr_mask & IB_QP_STATE) {
@@ -982,7 +984,8 @@ static int mthca_alloc_qp_common(struct 
 {
 	int err;
 
-	spin_lock_init(&qp->lock);
+	spin_lock_init(&qp->sq.lock);
+	spin_lock_init(&qp->rq.lock);
 	atomic_set(&qp->refcount, 1);
 	qp->state    	 = IB_QPS_RESET;
 	qp->atomic_rd_en = 0;
@@ -1272,7 +1275,7 @@ int mthca_post_send(struct ib_qp *ibqp, 
 		[IB_WR_ATOMIC_FETCH_AND_ADD] = MTHCA_OPCODE_ATOMIC_FA,
 	};
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->sq.lock, flags);
 
 	/* XXX check that state is OK to post send */
 
@@ -1455,7 +1458,7 @@ out:
 	qp->sq.cur += nreq;
 	qp->sq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->sq.lock, flags);
 	return err;
 }
 
@@ -1474,7 +1477,7 @@ int mthca_post_receive(struct ib_qp *ibq
 	void *wqe;
 	void *prev_wqe;
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->rq.lock, flags);
 
 	/* XXX check that state is OK to post receive */
 
@@ -1554,7 +1557,7 @@ out:
 	qp->rq.cur += nreq;
 	qp->rq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->rq.lock, flags);
 	return err;
 }
 
-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 10:02:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 20:02:33 +0200
Subject: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
Message-ID: <20050203180233.GK2536@mellanox.co.il>

I'm reposting this patch, its now tested. Together with the first part:
mthca send/receive q lock split, I am getting about 2% more bandwidth
with this patch.
I also checked interrupt rate and it is about the same, which is not
surprising, since interrupts from both cqs are coalesced by the eq polling
in mthca.

The next week I plan to implement avoiding cq req_notif for send cq,
along the lines that Ido suggested, for which this split is a prerequisite,
so I'll be very happy if at least this piece goes in. Without the mthca
piece, my experiments show this is a no pain no gain patch.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/ipoib/ipoib_verbs.c
===================================================================
--- ulp/ipoib/ipoib_verbs.c	(revision 1725)
+++ ulp/ipoib/ipoib_verbs.c	(working copy)
@@ -175,24 +175,31 @@ int ipoib_transport_dev_init(struct net_
 		return -ENODEV;
 	}
 
-	priv->cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL, dev,
-				IPOIB_TX_RING_SIZE + IPOIB_RX_RING_SIZE + 1);
-	if (IS_ERR(priv->cq)) {
-		printk(KERN_WARNING "%s: failed to create CQ\n", ca->name);
+	priv->tx_cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL, dev,
+				IPOIB_TX_RING_SIZE);
+	if (IS_ERR(priv->tx_cq)) {
+		printk(KERN_WARNING "%s: failed to create TX CQ\n", ca->name);
 		goto out_free_pd;
 	}
 
-	if (ib_req_notify_cq(priv->cq, IB_CQ_NEXT_COMP))
-		goto out_free_cq;
+	priv->rx_cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL, dev,
+				IPOIB_RX_RING_SIZE);
+	if (IS_ERR(priv->rx_cq)) {
+		printk(KERN_WARNING "%s: failed to create RX CQ\n", ca->name);
+		goto out_free_tx_cq;
+	}
+
+	if (ib_req_notify_cq(priv->rx_cq, IB_CQ_NEXT_COMP))
+		goto out_free_rx_cq;
 
 	priv->mr = ib_get_dma_mr(priv->pd, IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(priv->mr)) {
 		printk(KERN_WARNING "%s: ib_get_dma_mr failed\n", ca->name);
-		goto out_free_cq;
+		goto out_free_rx_cq;
 	}
 
-	init_attr.send_cq = priv->cq;
-	init_attr.recv_cq = priv->cq,
+	init_attr.send_cq = priv->tx_cq;
+	init_attr.recv_cq = priv->rx_cq,
 
 	priv->qp = ib_create_qp(priv->pd, &init_attr);
 	if (IS_ERR(priv->qp)) {
@@ -216,8 +223,11 @@ int ipoib_transport_dev_init(struct net_
 out_free_mr:
 	ib_dereg_mr(priv->mr);
 
-out_free_cq:
-	ib_destroy_cq(priv->cq);
+out_free_rx_cq:
+	ib_destroy_cq(priv->rx_cq);
+
+out_free_tx_cq:
+	ib_destroy_cq(priv->tx_cq);
 
 out_free_pd:
 	ib_dealloc_pd(priv->pd);
@@ -239,8 +249,11 @@ void ipoib_transport_dev_cleanup(struct 
 	if (ib_dereg_mr(priv->mr))
 		ipoib_warn(priv, "ib_dereg_mr failed\n");
 
-	if (ib_destroy_cq(priv->cq))
-		ipoib_warn(priv, "ib_cq_destroy failed\n");
+	if (ib_destroy_cq(priv->rx_cq))
+		ipoib_warn(priv, "ib_cq_destroy for rx cq failed\n");
+
+	if (ib_destroy_cq(priv->tx_cq))
+		ipoib_warn(priv, "ib_cq_destroy for tx cq failed\n");
 
 	if (ib_dealloc_pd(priv->pd))
 		ipoib_warn(priv, "ib_dealloc_pd failed\n");
Index: ulp/ipoib/ipoib.h
===================================================================
--- ulp/ipoib/ipoib.h	(revision 1725)
+++ ulp/ipoib/ipoib.h	(working copy)
@@ -137,7 +137,8 @@ struct ipoib_dev_priv {
 	u16           	  pkey;
 	struct ib_pd  	 *pd;
 	struct ib_mr  	 *mr;
-	struct ib_cq  	 *cq;
+	struct ib_cq  	 *tx_cq;
+	struct ib_cq  	 *rx_cq;
 	struct ib_qp  	 *qp;
 	u32           	  qkey;
 

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 10:09:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 20:09:08 +0200
Subject: [openib-general] taskset + CPU taking interrupts: some data
In-Reply-To: <20050203172559.GF2536@mellanox.co.il>
References: <20050126135451.GI3791@mellanox.co.il> <523bwo2ka2.fsf@topspin.com>
	<20050126171902.GC10743@mellanox.co.il>
	<20050126192810.GA18168@esmail.cup.hp.com>
	<20050126193749.GH10743@mellanox.co.il>
	<20050126195502.GD18168@esmail.cup.hp.com>
	<20050126201322.GD12344@mellanox.co.il>
	<20050126223101.GA3765@esmail.cup.hp.com>
	<20050203172559.GF2536@mellanox.co.il>
Message-ID: <20050203180908.GL2536@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> > > > Did you see the difference in IPoIB numbers that I posted
> > > > earlier where I pegged the test processes to the same CPU
> > > > taking interrupts or a different CPU? 
> > >
> > > No - link?
> > 
> > Sorry - I don't have the URL - just the original mail.
> > I posted this on "Date: Thu, 13 Jan 2005 10:43:34 -0800"
> > and you should be able to find this in the archives.
> > 
> > Here is the table again:
> > netperf/irq same, netserver/irq same: 1591 Mbps
> > netperf/irq diff, netserver/irq same: 1565 Mbps
> > netperf/irq same, netserver/irq diff: 1507 Mbps
> > netperf/irq diff, netserver/irq diff: 1510 Mbps
> > 
> > (1591-1510)/1510 is about 5%.
> > If you use taskset on your platforms, run a similar table to the
> > above, we could compare and see if ia64 is more sensitive to this
> > than other arches.
> > 
> > hth,
> > grant
> > 

OK, here is my table:
> Recv   Send    Send                          Utilization       Service Demand
> Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
> Size   Size    Size     Time     Throughput  local    remote   local   remote
> bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB

Same:
  87380  16384 100000    10.00       301.38   44.20    92.95    2.865   6.024
  87380  16384 100000    10.00       290.86   47.80    89.85    3.210   6.033
  87380  16384 100000    10.00       295.07   43.46    90.45    2.876   5.987

Different:
 87380  16384 100000    10.00       303.82   42.56    44.70    2.736   2.874
 87380  16384 100000    10.00       305.32   42.51    45.65    2.719   2.920
 87380  16384 100000    10.00       304.76   42.46    47.30    2.721   3.032

(304-295)/300 is about 3%

Hmm.

-- 
MST - Michael S. Tsirkin


From mshefty at ichips.intel.com  Thu Feb  3 10:18:04 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 03 Feb 2005 10:18:04 -0800
Subject: [openib-general] Re: more CM byte ordering.
In-Reply-To: <20050202164716.B31792@topspin.com>
References: <42017366.8070305@ichips.intel.com>
	<20050202164716.B31792@topspin.com>
Message-ID: <42026ADC.9020409@ichips.intel.com>

Libor Michalek wrote:
>   I found a couple more spots in the active connect case of incorrect
> byte ordering, this time around starting psn. Here is a tested patch.

Thanks for finding this.

> -	cm_req_set_starting_psn(req_msg, cpu_to_be32(param->starting_psn));
> +	cm_req_set_starting_psn(req_msg, param->starting_psn);

This fix implies that the user is giving the CM the starting PSN in 
network-byte order, which probably isn't the case.  Looking at where 
the PSN values are set, I think it's swapped in 3 places.  (For 
example, I don't think it's being swapped on the receiving end of the 
REQ, which is why this patch works.)

I'm re-working the PSN handling portions now for a correct patch.

- Sean


From mshefty at ichips.intel.com  Thu Feb  3 11:35:36 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 3 Feb 2005 11:35:36 -0800
Subject: [openib-general] [PATCH] CM PSN byte ordering
Message-ID: <20050203113536.6c923fb0.mshefty@ichips.intel.com>

Patch fixes PSN byte ordering issues in the CM.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: core/cm.c
===================================================================
--- core/cm.c	(revision 1725)
+++ core/cm.c	(working copy)
@@ -1104,7 +1104,7 @@ static void cm_format_rep(struct cm_rep_
 	rep_msg->local_comm_id = cm_id_priv->id.local_id;
 	rep_msg->remote_comm_id = cm_id_priv->id.remote_id;
 	cm_rep_set_local_qpn(rep_msg, cpu_to_be32(param->qp->qp_num));
-	cm_rep_set_starting_psn(rep_msg, param->starting_psn);
+	cm_rep_set_starting_psn(rep_msg, cpu_to_be32(param->starting_psn));
 	rep_msg->resp_resources = param->responder_resources;
 	rep_msg->initiator_depth = param->initiator_depth;
 	cm_rep_set_target_ack_delay(rep_msg, param->target_ack_delay);
@@ -2718,7 +2718,7 @@ static int cm_init_qp_rts_attr(struct cm
 		qp_attr->timeout = cm_id_priv->local_ack_timeout;
 		qp_attr->retry_cnt = cm_id_priv->retry_count;
 		qp_attr->rnr_retry = cm_id_priv->rnr_retry_count;
-		qp_attr->sq_psn = cm_id_priv->sq_psn;
+		qp_attr->sq_psn = be32_to_cpu(cm_id_priv->sq_psn);
 		qp_attr->max_rd_atomic = cm_id_priv->initiator_depth;
 		if (cm_id_priv->alt_av.ah_attr.dlid) {
 			*qp_attr_mask |= IB_QP_PATH_MIG_STATE;
Index: core/cm_msgs.h
===================================================================
--- core/cm_msgs.h	(revision 1720)
+++ core/cm_msgs.h	(working copy)
@@ -210,7 +210,7 @@ static inline void cm_req_set_flow_ctrl(
 
 static inline u32 cm_req_get_starting_psn(struct cm_req_msg *req_msg)
 {
-	return be32_to_cpu(req_msg->offset44) >> 8;
+	return cpu_to_be32(be32_to_cpu(req_msg->offset44) >> 8);
 }
 
 static inline void cm_req_set_starting_psn(struct cm_req_msg *req_msg,


From libor at topspin.com  Thu Feb  3 10:47:30 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 3 Feb 2005 10:47:30 -0800
Subject: [openib-general] Re: more CM byte ordering.
In-Reply-To: <42026ADC.9020409@ichips.intel.com>;
	from mshefty@ichips.intel.com on Thu, Feb 03, 2005 at 10:18:04AM
	-0800
References: <42017366.8070305@ichips.intel.com>
	<20050202164716.B31792@topspin.com>
	<42026ADC.9020409@ichips.intel.com>
Message-ID: <20050203104730.A2418@topspin.com>

On Thu, Feb 03, 2005 at 10:18:04AM -0800, Sean Hefty wrote:
> Libor Michalek wrote:
> >   I found a couple more spots in the active connect case of incorrect
> > byte ordering, this time around starting psn. Here is a tested patch.
> 
> Thanks for finding this.
> 
> > -	cm_req_set_starting_psn(req_msg, cpu_to_be32(param->starting_psn));
> > +	cm_req_set_starting_psn(req_msg, param->starting_psn);
> 
> This fix implies that the user is giving the CM the starting PSN in 
> network-byte order, which probably isn't the case.  Looking at where 
> the PSN values are set, I think it's swapped in 3 places.  (For 
> example, I don't think it's being swapped on the receiving end of the 
> REQ, which is why this patch works.)

  The only reason I assumed that the user was providing the value in
network byte order, is that the Service ID is expected in network byte
order, as well as the staring PSN in ib_send_cm_rep is expected in
network byte order as well.

  I agree the host byte order makes sense, since the event data is
provided to the user in host byte order. Looks like there are three
fields for which the change would have to be made:

  struct ib_cm_req_param {
        u64                     service_id;
        u32                     starting_psn;
  }
  struct ib_cm_rep_param {
        u32             starting_psn;
  }

Plus the fields in the SIDR params...

-Libor


From mshefty at ichips.intel.com  Thu Feb  3 11:03:12 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 03 Feb 2005 11:03:12 -0800
Subject: [openib-general] Re: more CM byte ordering.
In-Reply-To: <20050203104730.A2418@topspin.com>
References: <42017366.8070305@ichips.intel.com>
	<20050202164716.B31792@topspin.com>
	<42026ADC.9020409@ichips.intel.com>
	<20050203104730.A2418@topspin.com>
Message-ID: <42027570.9030700@ichips.intel.com>

Libor Michalek wrote:
>   The only reason I assumed that the user was providing the value in
> network byte order, is that the Service ID is expected in network byte
> order, as well as the staring PSN in ib_send_cm_rep is expected in
> network byte order as well.

I was trying to make the CM interface use byte ordering consistent with 
related interfaces.  For PSN, I tried to use host-ordering to match the 
  QP modify call.  There was a bug in ib_send_cm_rep; it should have 
expected host order.  (The cm_msgs interface should be defined with all 
input/output in network byte order.)

>   I agree the host byte order makes sense, since the event data is
> provided to the user in host byte order. Looks like there are three
> fields for which the change would have to be made:

As it looks now, the only fields into the CM that should be in host 
order are: PSN, QPN, and QKey.

- Sean


From roland at topspin.com  Thu Feb  3 11:29:03 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:29:03 -0800
Subject: [openib-general] Re: [PATCH] InfiniBand: add missing break
	between cases
In-Reply-To: <20050203055043.GA19348@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 07:50:43 +0200")
References: <52y8e65hhb.fsf@topspin.com>
	<20050203055043.GA19348@mellanox.co.il>
Message-ID: <52vf99igsg.fsf@topspin.com>

>>>>> "Michael" == Michael S Tsirkin <mst at mellanox.co.il> writes:

    Michael> BTW, is the lack of break after case MTHCA_RES_UDAV
    Michael> intentional in mthca_make_profile?  Isnt this what was
    Michael> intended ( I noticed this when comparing trunk with
    Michael> roland-uverbs)?

Yes, and the next case should also have a break.  (Both are harmless
right now -- the first one because mem-free support is turned off on
the trunk, the second because it's followed by the default case)

 - R.


From roland at topspin.com  Thu Feb  3 11:33:48 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:33:48 -0800
Subject: [openib-general] [PATCH] bugfix: remove unbalanced qp refcount
In-Reply-To: <20050203093404.GF28018@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 11:34:05 +0200")
References: <20050203093404.GF28018@mellanox.co.il>
Message-ID: <52r7jxigkj.fsf@topspin.com>

Thanks, applied.

 - R.


From roland at topspin.com  Thu Feb  3 11:34:10 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:34:10 -0800
Subject: [openib-general] Re: [PATCH] roland-uverbs: possible race condition
In-Reply-To: <20050203161049.GD2536@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 18:10:49 +0200")
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
	<52vf9awq1z.fsf@topspin.com> <20050203161049.GD2536@mellanox.co.il>
Message-ID: <52mzuligjx.fsf@topspin.com>

    Michael> OK, so are we all in agreement now, that an empty eq,
    Michael> too, must be re-armed in memfree mode when working in
    Michael> regular interrupt mode?

Yes, I guess so.  I'll make the change.

 - R.


From roland at topspin.com  Thu Feb  3 11:39:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:39:53 -0800
Subject: [openib-general] [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <20050203175419.GJ2536@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 19:54:19 +0200")
References: <20050203175419.GJ2536@mellanox.co.il>
Message-ID: <52is59igae.fsf@topspin.com>

This makes sense.  Let me look at the implementation and think about
it a little (I probably won't be able to apply any substantial patches
until after the Sonoma workshop).

 - R.


From roland at topspin.com  Thu Feb  3 11:44:10 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:44:10 -0800
Subject: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
In-Reply-To: <20050203180233.GK2536@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 20:02:33 +0200")
References: <20050203180233.GK2536@mellanox.co.il>
Message-ID: <52ekfxig39.fsf@topspin.com>

    Michael> I'm reposting this patch, its now tested. Together with
    Michael> the first part: mthca send/receive q lock split, I am
    Michael> getting about 2% more bandwidth with this patch.  I also
    Michael> checked interrupt rate and it is about the same, which is
    Michael> not surprising, since interrupts from both cqs are
    Michael> coalesced by the eq polling in mthca.

Do you see any improvement with just the QP lock split and a single
CQ?  It seems to me that the boost comes from being able to handle
completions from one work queue while posting work requests to the
other queue.  I don't see how having two CQs helps reduce lock
contention or improve parallelism at all.  There's still only one
interrupt handler and one IPoIB completion event handler, which can
only run on one CPU.  So it seems what will happen is:

    EQ event
    Find EQE for receive CQ
    Lock receive CQ, process completions
    Unlock receive CQ
    Find EQE for send CQ
    Lock send CQ, process completions
    Unlock send CQ

The only difference is that we unlock the receive CQ and lock the send
CQ in the middle.

Especially since you say the interrupt rate is the same, it seems that
almost all the time, the completion event handler is finding
completions in both queues anyway.

 - R.


From roland at topspin.com  Thu Feb  3 11:46:01 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:46:01 -0800
Subject: [openib-general] Re: [PATCH] roland-uverbs: possible race
	condition
In-Reply-To: <52mzuligjx.fsf@topspin.com> (Roland Dreier's message of "Thu,
	03 Feb 2005 11:34:10 -0800")
References: <20050130143445.GH26505@mellanox.co.il>
	<52is5etabn.fsf@topspin.com> <20050131063819.GA3832@mellanox.co.il>
	<52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
	<52vf9awq1z.fsf@topspin.com> <20050203161049.GD2536@mellanox.co.il>
	<52mzuligjx.fsf@topspin.com>
Message-ID: <52acqlig06.fsf@topspin.com>

Does this look reasonable?

Index: infiniband/hw/mthca/mthca_dev.h
===================================================================
--- infiniband/hw/mthca/mthca_dev.h	(revision 1732)
+++ infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -171,6 +171,7 @@ struct mthca_eq_table {
 	struct mthca_alloc alloc;
 	void __iomem      *clr_int;
 	u32                clr_mask;
+	u32                arm_mask;
 	struct mthca_eq    eq[MTHCA_NUM_EQ];
 	u64                icm_virt;
 	struct page       *icm_page;
Index: infiniband/hw/mthca/mthca_eq.c
===================================================================
--- infiniband/hw/mthca/mthca_eq.c	(revision 1732)
+++ infiniband/hw/mthca/mthca_eq.c	(working copy)
@@ -430,6 +430,7 @@ static irqreturn_t mthca_arbel_interrupt
 {
 	struct mthca_dev *dev = dev_ptr;
 	u32 arm = 0;
+	int work = 0;
 	int i;
 
 	if (dev->eq_table.clr_mask)
@@ -437,14 +438,14 @@ static irqreturn_t mthca_arbel_interrupt
 
 	for (i = 0; i < MTHCA_NUM_EQ; ++i)
 		if (mthca_eq_int(dev, &dev->eq_table.eq[i])) {
+			work = 1;
 			arbel_set_eq_ci(dev, &dev->eq_table.eq[i],
 					dev->eq_table.eq[i].cons_index);
-			arm |= dev->eq_table.eq[i].eqn_mask;
 		}
 
-	arbel_eq_req_not(dev, arm);
+	arbel_eq_req_not(dev, dev->eq_table.arm_mask);
 
-	return IRQ_RETVAL(arm);
+	return IRQ_RETVAL(work);
 }
 
 static irqreturn_t mthca_arbel_msi_x_interrupt(int irq, void *eq_ptr,
@@ -568,6 +569,8 @@ static int __devinit mthca_create_eq(str
 	eq->eqn_mask   = swab32(1 << eq->eqn);
 	eq->cons_index = 0;
 
+	dev->eq_table.arm_mask |= eq->eqn_mask;
+
 	mthca_dbg(dev, "Allocated EQ %d with %d entries\n",
 		  eq->eqn, nent);
 
@@ -618,6 +621,8 @@ static void mthca_free_eq(struct mthca_d
 		mthca_warn(dev, "HW2SW_EQ returned status 0x%02x\n",
 			   status);
 
+	dev->eq_table.arm_mask &= ~eq->eqn_mask;
+
 	if (0) {
 		mthca_dbg(dev, "Dumping EQ context %02x:\n", eq->eqn);
 		for (i = 0; i < sizeof (struct mthca_eq_context) / 4; ++i) {
@@ -846,6 +851,8 @@ int __devinit mthca_init_eq_table(struct
 			(dev->eq_table.inta_pin < 31 ? 4 : 0);
 	}
 
+	dev->eq_table.arm_mask = 0;
+
 	intr = (dev->mthca_flags & MTHCA_FLAG_MSI) ?
 		128 : dev->eq_table.inta_pin;
 


From roland at topspin.com  Thu Feb  3 11:48:52 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 11:48:52 -0800
Subject: [openib-general] [PATCH] InfiniBand: remove unbalance refcnt
	decrement
In-Reply-To: <52y8e65hhb.fsf@topspin.com> (Roland Dreier's message of "Wed,
	02 Feb 2005 21:40:16 -0800")
References: <52y8e65hhb.fsf@topspin.com>
Message-ID: <526519ifvf.fsf@topspin.com>

From: Michael S. Tsirkin <mst at mellanox.co.il>

Fix unbalanced QP reference count decrement (introduced with QP lock
optimization patch)

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-bk.orig/drivers/infiniband/hw/mthca/mthca_cq.c	2005-01-28 11:11:03.000000000 -0800
+++ linux-bk/drivers/infiniband/hw/mthca/mthca_cq.c	2005-02-03 11:47:39.300426349 -0800
@@ -422,8 +422,6 @@
 				*freed = 0;
 			}
 			spin_unlock(&(*cur_qp)->lock);
-			if (atomic_dec_and_test(&(*cur_qp)->refcount))
-				wake_up(&(*cur_qp)->wait);
 		}
 
 		spin_lock(&dev->qp_table.lock);


From mst at mellanox.co.il  Thu Feb  3 12:30:50 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 22:30:50 +0200
Subject: [openib-general] Re: [PATCH] roland-uverbs: possible race
	condition
In-Reply-To: <52acqlig06.fsf@topspin.com>
References: <20050131063819.GA3832@mellanox.co.il> <52r7k1s4iw.fsf@topspin.com>
	<20050131221202.GP22123@esmail.cup.hp.com>
	<6.2.0.14.2.20050131142946.02afc608@esmail.cup.hp.com>
	<521xbz1w16.fsf@topspin.com>
	<6.2.0.14.2.20050202124949.027a1498@esmail.cup.hp.com>
	<52vf9awq1z.fsf@topspin.com> <20050203161049.GD2536@mellanox.co.il>
	<52mzuligjx.fsf@topspin.com> <52acqlig06.fsf@topspin.com>
Message-ID: <20050203203050.GA5108@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] Re: [PATCH] roland-uverbs: possible race condition
> 
> Does this look reasonable?
> 
> Index: infiniband/hw/mthca/mthca_dev.h
> ===================================================================
> --- infiniband/hw/mthca/mthca_dev.h	(revision 1732)
> +++ infiniband/hw/mthca/mthca_dev.h	(working copy)
> @@ -171,6 +171,7 @@ struct mthca_eq_table {
>  	struct mthca_alloc alloc;
>  	void __iomem      *clr_int;
>  	u32                clr_mask;
> +	u32                arm_mask;
>  	struct mthca_eq    eq[MTHCA_NUM_EQ];
>  	u64                icm_virt;
>  	struct page       *icm_page;
> Index: infiniband/hw/mthca/mthca_eq.c
> ===================================================================
> --- infiniband/hw/mthca/mthca_eq.c	(revision 1732)
> +++ infiniband/hw/mthca/mthca_eq.c	(working copy)
> @@ -430,6 +430,7 @@ static irqreturn_t mthca_arbel_interrupt
>  {
>  	struct mthca_dev *dev = dev_ptr;
>  	u32 arm = 0;
> +	int work = 0;
>  	int i;
>  
>  	if (dev->eq_table.clr_mask)
> @@ -437,14 +438,14 @@ static irqreturn_t mthca_arbel_interrupt
>  
>  	for (i = 0; i < MTHCA_NUM_EQ; ++i)
>  		if (mthca_eq_int(dev, &dev->eq_table.eq[i])) {
> +			work = 1;
>  			arbel_set_eq_ci(dev, &dev->eq_table.eq[i],
>  					dev->eq_table.eq[i].cons_index);
> -			arm |= dev->eq_table.eq[i].eqn_mask;
>  		}
>  
> -	arbel_eq_req_not(dev, arm);
> +	arbel_eq_req_not(dev, dev->eq_table.arm_mask);
>  
> -	return IRQ_RETVAL(arm);
> +	return IRQ_RETVAL(work);
>  }
>  
>  static irqreturn_t mthca_arbel_msi_x_interrupt(int irq, void *eq_ptr,
> @@ -568,6 +569,8 @@ static int __devinit mthca_create_eq(str
>  	eq->eqn_mask   = swab32(1 << eq->eqn);
>  	eq->cons_index = 0;
>  
> +	dev->eq_table.arm_mask |= eq->eqn_mask;
> +
>  	mthca_dbg(dev, "Allocated EQ %d with %d entries\n",
>  		  eq->eqn, nent);
>  
> @@ -618,6 +621,8 @@ static void mthca_free_eq(struct mthca_d
>  		mthca_warn(dev, "HW2SW_EQ returned status 0x%02x\n",
>  			   status);
>  
> +	dev->eq_table.arm_mask &= ~eq->eqn_mask;
> +
>  	if (0) {
>  		mthca_dbg(dev, "Dumping EQ context %02x:\n", eq->eqn);
>  		for (i = 0; i < sizeof (struct mthca_eq_context) / 4; ++i) {
> @@ -846,6 +851,8 @@ int __devinit mthca_init_eq_table(struct
>  			(dev->eq_table.inta_pin < 31 ? 4 : 0);
>  	}
>  
> +	dev->eq_table.arm_mask = 0;
> +
>  	intr = (dev->mthca_flags & MTHCA_FLAG_MSI) ?
>  		128 : dev->eq_table.inta_pin;
>  
> 

Yes, very close to my patch.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 12:48:51 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 22:48:51 +0200
Subject: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
In-Reply-To: <52ekfxig39.fsf@topspin.com>
References: <20050203180233.GK2536@mellanox.co.il> <52ekfxig39.fsf@topspin.com>
Message-ID: <20050203204851.GB5108@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
> 
>     Michael> I'm reposting this patch, its now tested. Together with
>     Michael> the first part: mthca send/receive q lock split, I am
>     Michael> getting about 2% more bandwidth with this patch.  I also
>     Michael> checked interrupt rate and it is about the same, which is
>     Michael> not surprising, since interrupts from both cqs are
>     Michael> coalesced by the eq polling in mthca.
> 
> Do you see any improvement with just the QP lock split and a single
> CQ?  It seems to me that the boost comes from being able to handle
> completions from one work queue while posting work requests to the
> other queue.

Didnt test that.

> I don't see how having two CQs helps reduce lock
> contention or improve parallelism at all.  There's still only one
> interrupt handler and one IPoIB completion event handler, which can
> only run on one CPU.  So it seems what will happen is:
> 
>     EQ event
>     Find EQE for receive CQ
>     Lock receive CQ, process completions
>     Unlock receive CQ
>     Find EQE for send CQ
>     Lock send CQ, process completions
>     Unlock send CQ
> 
> The only difference is that we unlock the receive CQ and lock the send
> CQ in the middle.

Are you sure?
I *do* have msi-x on these machines.

> 
> Especially since you say the interrupt rate is the same, it seems that
> almost all the time, the completion event handler is finding
> completions in both queues anyway.
> 
>  - R.
> 


-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Thu Feb  3 12:58:04 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 12:58:04 -0800
Subject: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
In-Reply-To: <20050203204851.GB5108@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 22:48:51 +0200")
References: <20050203180233.GK2536@mellanox.co.il>
	<52ekfxig39.fsf@topspin.com> <20050203204851.GB5108@mellanox.co.il>
Message-ID: <52wttpgy3n.fsf@topspin.com>

    Roland> The only difference is that we unlock the receive CQ and
    Roland> lock the send CQ in the middle.

    Michael> Are you sure?  I *do* have msi-x on these machines.

Of course I'm not sure, but as far as I know, with the current drivers
even with MSI-X there's still only one completion interrupt handler,
which can only run on one CPU at a time.

Of course allowing different CQs to be bound to different interrupt
handlers by creating multiple completion EQs is an good extension that
I plan on implementing, but it's not written yet.  Even when it's done
I'm not sure that having IPoIB run multiple interrupt handlers at once
is worth it.  It's fun to get huge numbers out of IPoIB but presumably
people want some CPU left over to do real work.

 - R.


From tduffy at sun.com  Thu Feb  3 13:10:10 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 03 Feb 2005 13:10:10 -0800
Subject: [openib-general] [PATCH] add infiniband device rules to udev
Message-ID: <1107465010.6531.6.camel@duffman>

Signed-off-by: Tom Duffy <tduffy at sun.com>

diff -Nur udev-bk/etc/udev/udev.rules udev-bk-ib/etc/udev/udev.rules
--- udev-bk/etc/udev/udev.rules	2004-04-21 17:00:42.000000000 -0700
+++ udev-bk-ib/etc/udev/udev.rules	2005-02-03 11:54:55.190634797 -0800
@@ -45,3 +45,7 @@
 
 # raw devices
 KERNEL="raw[0-9]*",	NAME="raw/%k" 
+
+# infiniband devices
+KERNEL="umad*",		NAME="infiniband/%k"
+KERNEL="issm*",		NAME="infiniband/%k"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050203/077cb556/attachment.sig>

From roland at topspin.com  Thu Feb  3 13:25:28 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 13:25:28 -0800
Subject: [openib-general] [PATCH] add infiniband device rules to udev
In-Reply-To: <1107465010.6531.6.camel@duffman> (Tom Duffy's message of "Thu,
	03 Feb 2005 13:10:10 -0800")
References: <1107465010.6531.6.camel@duffman>
Message-ID: <52pszhgwtz.fsf@topspin.com>

Cool, thanks for pushing this sort of stuff upstream and with distros.

 - R.


From mst at mellanox.co.il  Thu Feb  3 13:33:41 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 23:33:41 +0200
Subject: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
In-Reply-To: <52wttpgy3n.fsf@topspin.com>
References: <20050203180233.GK2536@mellanox.co.il> <52ekfxig39.fsf@topspin.com>
	<20050203204851.GB5108@mellanox.co.il> <52wttpgy3n.fsf@topspin.com>
Message-ID: <20050203213341.GC5108@mellanox.co.il>


Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
> 
>     Roland> The only difference is that we unlock the receive CQ and
>     Roland> lock the send CQ in the middle.
> 
>     Michael> Are you sure?  I *do* have msi-x on these machines.
> 
> Of course I'm not sure, but as far as I know, with the current drivers
> even with MSI-X there's still only one completion interrupt handler,
> which can only run on one CPU at a time.
> 
> Of course allowing different CQs to be bound to different interrupt
> handlers by creating multiple completion EQs is an good extension that
> I plan on implementing, but it's not written yet.  Even when it's done
> I'm not sure that having IPoIB run multiple interrupt handlers at once
> is worth it.  It's fun to get huge numbers out of IPoIB but presumably
> people want some CPU left over to do real work.
> 
>  - R.
> 

Sounds convincing. So the improvement is probably due to
better send/receive parallelism we are getting in mthca.


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb  3 13:34:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 23:34:33 +0200
Subject: [openib-general] [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <52is59igae.fsf@topspin.com>
References: <20050203175419.GJ2536@mellanox.co.il> <52is59igae.fsf@topspin.com>
Message-ID: <20050203213433.GD5108@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] 1/2 separate locking for send and receive q in mthca
> 
> This makes sense.  Let me look at the implementation and think about
> it a little (I probably won't be able to apply any substantial patches
> until after the Sonoma workshop).
> 
>  - R.
> 

Ok.
Its quite straight-forward - move lock to the wq structure,
and use it from there.

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Thu Feb  3 13:33:37 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 13:33:37 -0800
Subject: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
In-Reply-To: <20050203213341.GC5108@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 23:33:41 +0200")
References: <20050203180233.GK2536@mellanox.co.il>
	<52ekfxig39.fsf@topspin.com> <20050203204851.GB5108@mellanox.co.il>
	<52wttpgy3n.fsf@topspin.com> <20050203213341.GC5108@mellanox.co.il>
Message-ID: <52lla5gwge.fsf@topspin.com>

    Michael> Sounds convincing. So the improvement is probably due to
    Michael> better send/receive parallelism we are getting in mthca.

I won't have time to try it until next week but it should be easy to
test the QP lock split without the IPoIB CQ split and see how that compares.

 - R.


From roland at topspin.com  Thu Feb  3 13:35:17 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 03 Feb 2005 13:35:17 -0800
Subject: [openib-general] [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <20050203213433.GD5108@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 23:34:33 +0200")
References: <20050203175419.GJ2536@mellanox.co.il>
	<52is59igae.fsf@topspin.com> <20050203213433.GD5108@mellanox.co.il>
Message-ID: <52hdktgwdm.fsf@topspin.com>

    Michael> Ok.  Its quite straight-forward - move lock to the wq
    Michael> structure, and use it from there.

Right, but I'm not sure if that's the best way to go... there may be
better ways to lay out the structure, and we may want to mark things
with __cacheline_aligned_in_smp to make sure we get what we want.

 - R.


From mst at mellanox.co.il  Thu Feb  3 13:39:57 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 3 Feb 2005 23:39:57 +0200
Subject: [openib-general] Re: [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <52hdktgwdm.fsf@topspin.com>
References: <20050203175419.GJ2536@mellanox.co.il> <52is59igae.fsf@topspin.com>
	<20050203213433.GD5108@mellanox.co.il> <52hdktgwdm.fsf@topspin.com>
Message-ID: <20050203213957.GE5108@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] 1/2 separate locking for send and receive q in mthca
> 
>     Michael> Ok.  Its quite straight-forward - move lock to the wq
>     Michael> structure, and use it from there.
> 
> Right, but I'm not sure if that's the best way to go... there may be
> better ways to lay out the structure, and we may want to mark things
> with __cacheline_aligned_in_smp to make sure we get what we want.
> 
>  - R.
> 

Aha. Good.
Another thing maybe worth it looking into was the idea
to use atomics instead of the lock for cq poll ... we'll see.

-- 
MST - Michael S. Tsirkin


From tduffy at sun.com  Thu Feb  3 13:54:49 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 03 Feb 2005 13:54:49 -0800
Subject: [openib-general] [PATCH] add infiniband device rules to udev
In-Reply-To: <52pszhgwtz.fsf@topspin.com>
References: <1107465010.6531.6.camel@duffman>  <52pszhgwtz.fsf@topspin.com>
Message-ID: <1107467689.6531.14.camel@duffman>

On Thu, 2005-02-03 at 13:25 -0800, Roland Dreier wrote:
> Cool, thanks for pushing this sort of stuff upstream and with distros.

BTW, if anybody want to track these:

Fedora: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=147035
Gentoo: http://bugs.gentoo.org/show_bug.cgi?id=80631
Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=293493

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050203/182e67f5/attachment.sig>

From tduffy at sun.com  Thu Feb  3 14:50:11 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 03 Feb 2005 14:50:11 -0800
Subject: [openib-general] Re: Bug#293493: [RFE] add infiniband devices to
	udev device rules
In-Reply-To: <20050203223646.GC18609@wonderland.linux.it>
References: <1107466481.6531.10.camel@duffman>
	<20050203223646.GC18609@wonderland.linux.it>
Message-ID: <1107471011.6531.19.camel@duffman>

On Thu, 2005-02-03 at 23:36 +0100, Marco d'Itri wrote:
> Thank you, they will be in the next upload.
> Are these devices in the "block" SUBSYSTEM? It's what I will use to give
> devices the default "disk" group (a udev.conf subsystem is the same
> thing of the /sbin/hotplug argument).

Infiniband is a network and interconnect technology.  These devices are
character devices.

[root at flopteron ~]# ls -l /dev/infiniband/
total 0
crw-------  1 root root 231, 64 Feb  1 15:54 issm0
crw-------  1 root root 231, 65 Feb  1 15:54 issm1
crw-------  1 root root 231,  0 Feb  1 15:54 umad0
crw-------  1 root root 231,  1 Feb  1 15:54 umad1

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050203/2c6b54d5/attachment.sig>

From boas1 at llnl.gov  Thu Feb  3 15:10:55 2005
From: boas1 at llnl.gov (Bill Boas)
Date: Thu, 03 Feb 2005 15:10:55 -0800
Subject: [openib-general] Please bring a USB memory stick to the Workshop
Message-ID: <6.1.2.0.2.20050203150923.04020598@mail-lc.llnl.gov>

This is how we will distribute the talks.

Bill Boas                               bboas at llnl.gov
ICCD LLNL, B-113, R-2018                Wk: 925-422-4110
7000 East Ave                           Cell: 925-337-2224
Livermore, CA 94551                     Pgr: 877-203-2248 




From mshefty at ichips.intel.com  Thu Feb  3 16:45:36 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 03 Feb 2005 16:45:36 -0800
Subject: [openib-general] [FYI] Update to kernel CM test module
Message-ID: <4202C5B0.3080505@ichips.intel.com>

I've updated a kernel CM test module under:

gen2/utils/src/linux-kernel/infiniband/util/cmpost

Two systems are required to execute any CM tests, with one loaded as
the server, and the second the client.  The test executes as soon as
the modules are loaded.

The module will stress connection establishment, perform data
transfers, and disconnect.  The module is for debug testing only, but
can server as a simple example as how to establish a connection.  My
goal is to allow a user-mode test module perform more strenuous testing.

I was able to stress the CM with up to 8000 simultaneous connections.
Beyond that, my systems ran out of memory.

- Sean


From halr at voltaire.com  Fri Feb  4 07:00:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 04 Feb 2005 10:00:10 -0500
Subject: [openib-general] [ANNOUNCE] full cutover to autotools for
	userspace/management
Message-ID: <1107529002.4734.10.camel@localhost.localdomain>

Hi,

I just checked in the rest of the conversion to autotools for the
userspace/management subdirectory. OpenSM and its libraries (osmcomp and
osmvendor) are now using autotools (and added in the 2 newer diagnostics
(ibroute and ibtracert) as well).

The next step in this evolution will be to make the build more automagic
(hopefully as painless as it was before).

Note that all the original Makefiles are present in tree as
Makefile.orig if you don't want to build this way.

-- Hal



From iod00d at hp.com  Fri Feb  4 09:48:03 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 4 Feb 2005 09:48:03 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202194752.GC6316@mellanox.co.il>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
	<20050202194752.GC6316@mellanox.co.il>
Message-ID: <20050204174803.GC4504@esmail.cup.hp.com>

On Wed, Feb 02, 2005 at 09:47:52PM +0200, Michael S. Tsirkin wrote:
> > The interrupt handler needs to guarantee it has handled
> > pending work and communicatied that back to the card so
> > IRQ line is not asserted.
> 
> Well, the handler did this, but while it was doing other things
> another interrupt got asserted.

I realize there is a race condition between handling events
and getting interrupts.

> > > I propose returning IRQ_HANDLED *always* in regular interrupt.
> > 
> > This totally breaks Shared IRQ support.
> 
> Why does it "totally break" it?

Sorry - my brainfart - I'm wrong. It's doesn't "totally" break IRQ handling.
I had to look at kernel/handle_IRQ_event() to realize this though.

I did notice a different issue though.
A "hung" shared IRQ (live lock the CPU) will never get flagged if
the IB driver always claims it was handled.

> (write to clear interrupt register) so we return "HANDLED".
> Its unavoidable that hardware may re-assert it while
> the handler is still running.

I agree the race condition exists.
The issue is how it should be handled.
If it can not be handled cleanly, then my original
suggestion of requesting an exclusive interrupt
would be worth considering.

I agree with Roland that occasional spurious interrupts
are ok. But it's probably worth capturing this thread and
adding it to the FAQ. It would be convenient to point at everytime
someone asks about "spurious interrupts" in their syslog.

grant


From iod00d at hp.com  Fri Feb  4 09:50:51 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 4 Feb 2005 09:50:51 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050202195908.GE6316@mellanox.co.il>
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
	<20050202194752.GC6316@mellanox.co.il> <52acqmyc0v.fsf@topspin.com>
	<20050202195908.GE6316@mellanox.co.il>
Message-ID: <20050204175051.GE4504@esmail.cup.hp.com>

On Wed, Feb 02, 2005 at 09:59:08PM +0200, Michael S. Tsirkin wrote:
> Roland, I know, but I believe thats a far cry from "totally break it".

agreed - sorry.

> Reson being, after the kernel disables a stuck interrupt, any device
> sharing it will not function properly anyway.

The kernel will never know it's stuck IFF mthca always returns HANDLED.
ie it won't get disabled.

grant


From roland at topspin.com  Fri Feb  4 09:52:01 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 04 Feb 2005 09:52:01 -0800
Subject: [openib-general] Re: Disabling IRQ #201 message
In-Reply-To: <20050204174803.GC4504@esmail.cup.hp.com> (Grant Grundler's
	message of "Fri, 4 Feb 2005 09:48:03 -0800")
References: <001501c508ac$4b89db10$6a01a8c0@sfw.int>
	<52brb3dcqm.fsf@topspin.com> <20050202133705.GD5037@mellanox.co.il>
	<20050202192733.GB29821@esmail.cup.hp.com>
	<20050202194752.GC6316@mellanox.co.il>
	<20050204174803.GC4504@esmail.cup.hp.com>
Message-ID: <52brb09pry.fsf@topspin.com>

    Grant> I agree with Roland that occasional spurious interrupts are
    Grant> ok. But it's probably worth capturing this thread and
    Grant> adding it to the FAQ. It would be convenient to point at
    Grant> everytime someone asks about "spurious interrupts" in their
    Grant> syslog.

I don't think that there should be any "spurious interrupt" messages
under any normal circumstances.  The kernel will only complain if
99900 out of 100000 interrupts are unhandled, and if that happens
something is actually badly wrong.

 - R.


From iod00d at hp.com  Fri Feb  4 10:54:31 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 4 Feb 2005 10:54:31 -0800
Subject: [openib-general] Re: Re: [PATCH] (fixed) reduce qp locking on cq
	poll
In-Reply-To: <20050203172559.GF2536@mellanox.co.il>
References: <20050126135451.GI3791@mellanox.co.il> <523bwo2ka2.fsf@topspin.com>
	<20050126171902.GC10743@mellanox.co.il>
	<20050126192810.GA18168@esmail.cup.hp.com>
	<20050126193749.GH10743@mellanox.co.il>
	<20050126195502.GD18168@esmail.cup.hp.com>
	<20050126201322.GD12344@mellanox.co.il>
	<20050126223101.GA3765@esmail.cup.hp.com>
	<20050203172559.GF2536@mellanox.co.il>
Message-ID: <20050204185431.GM4504@esmail.cup.hp.com>

On Thu, Feb 03, 2005 at 07:25:59PM +0200, Michael S. Tsirkin wrote:
> Bandwidth went anywhere from 301.38 to 295.07.

That would imply the problem is on the "netserver" side and
not on the client side.

A new netserver process is started on the "-H" every time netperf
is invoked. That process exits when netperf finishes.

hth,
grant


From libor at topspin.com  Fri Feb  4 11:21:15 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 4 Feb 2005 11:21:15 -0800
Subject: [openib-general] RFC on SDP checkin
Message-ID: <20050204112115.A1592@topspin.com>


  With the CM comming along quickly and SDP now able to establish
connections and transfer data, I thought it might be time to get
it into the repository. I see at least three options for checkin
and would like to get feedback on what people feel is the best
start:

  1) Create a directory outside of the main source tree.
     (e.g. gen2/users/libor/sdp)

  2) Create a barnch of the existing tree and add the code to
     this tree. (e.g. gen2/branches/libor-sdp)

  3) Since the rest of the tree is not destabilized or effected in
     any other way by the code, just check it into tree itself.
     (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)

  Anyone have thoughts on this? Personally I'm leaning towards #3,
but that's because it's the least amount of work for me. :)

  The first performance number I've been able to collect are on a
pair of 2 CPU 2.4GHz Xeons w/ 2GBs PCI-X 133 systems, SDP is getting
610 MB/s throughput (sync. send/recv) and 24 usec latency.

Thanks.

-Libor


From mshefty at ichips.intel.com  Fri Feb  4 11:23:20 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 04 Feb 2005 11:23:20 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <20050204112115.A1592@topspin.com>
References: <20050204112115.A1592@topspin.com>
Message-ID: <4203CBA8.1080606@ichips.intel.com>

Libor Michalek wrote:
>   3) Since the rest of the tree is not destabilized or effected in
>      any other way by the code, just check it into tree itself.
>      (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)

I prefer this myself.

- Sean


From eitan at mellanox.co.il  Fri Feb  4 11:24:17 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Fri, 4 Feb 2005 21:24:17 +0200 
Subject: [openib-general] OpenIB OpenSM License
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EED84@mtlex01.yok.mtl.com>

Hi Hal, Woody

Regarding the OpenSM license, 
Mellanox legal finally returned the updated license. The only change is the
addition of the second 
copyright line.

Please use this one. 

Thanks.

EZ

/*
 * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved.
 * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved.
 * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.
 *
 * This software is available to you under a choice of one of two
 * licenses.  You may choose to be licensed under the terms of the GNU
 * General Public License (GPL) Version 2, available from the file
 * COPYING in the main directory of this source tree, or the
 * OpenIB.org BSD license below:
 *
 *     Redistribution and use in source and binary forms, with or
 *     without modification, are permitted provided that the following
 *     conditions are met:
 *
 *      - Redistributions of source code must retain the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer.
 *
 *      - Redistributions in binary form must reproduce the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer in the documentation and/or other materials
 *        provided with the distribution.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
 * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
 * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 *
 * $Id:
 */

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050204/3471008d/attachment.html>

From iod00d at hp.com  Fri Feb  4 11:27:18 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 4 Feb 2005 11:27:18 -0800
Subject: [openib-general] Please bring a USB memory stick to the Workshop
In-Reply-To: <6.1.2.0.2.20050203150923.04020598@mail-lc.llnl.gov>
References: <6.1.2.0.2.20050203150923.04020598@mail-lc.llnl.gov>
Message-ID: <20050204192718.GQ4504@esmail.cup.hp.com>

On Thu, Feb 03, 2005 at 03:10:55PM -0800, Bill Boas wrote:
> This is how we will distribute the talks.

I would also be happy to host ftp/http server on my
laptop via the wireless in case someone doesn't have
a USB storage device (don't forget USB camera might work too)

grant


From tduffy at sun.com  Fri Feb  4 11:33:18 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 04 Feb 2005 11:33:18 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <20050204112115.A1592@topspin.com>
References: <20050204112115.A1592@topspin.com>
Message-ID: <1107545598.5982.34.camel@duffman>

On Fri, 2005-02-04 at 11:21 -0800, Libor Michalek wrote:
>   3) Since the rest of the tree is not destabilized or effected in
>      any other way by the code, just check it into tree itself.
>      (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)
> 
>   Anyone have thoughts on this? Personally I'm leaning towards #3,
> but that's because it's the least amount of work for me. :)

I would like #3 as well.  Libor, would it be possible to get a preview
of what is going to be checked in?  Maybe with a small one-pager about
how to use it, etc?

Thanks,

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050204/bdcb6907/attachment.sig>

From libor at topspin.com  Fri Feb  4 11:42:07 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 4 Feb 2005 11:42:07 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1107545598.5982.34.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 04, 2005 at 11:33:18AM -0800
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
Message-ID: <20050204114207.B1592@topspin.com>

On Fri, Feb 04, 2005 at 11:33:18AM -0800, Tom Duffy wrote:
> On Fri, 2005-02-04 at 11:21 -0800, Libor Michalek wrote:
> >   3) Since the rest of the tree is not destabilized or effected in
> >      any other way by the code, just check it into tree itself.
> >      (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)
> > 
> >   Anyone have thoughts on this? Personally I'm leaning towards #3,
> > but that's because it's the least amount of work for me. :)
> 
> I would like #3 as well.  Libor, would it be possible to get a preview
> of what is going to be checked in?  Maybe with a small one-pager about
> how to use it, etc?

  You mean a quick description of the code, like the primary contents of
each file and how to get it to do something ?

-Libor




From paul.baxter at dsl.pipex.com  Fri Feb  4 12:04:10 2005
From: paul.baxter at dsl.pipex.com (Paul Baxter)
Date: Fri, 4 Feb 2005 20:04:10 -0000
Subject: [openib-general] RFC on SDP checkin
References: <20050204112115.A1592@topspin.com><1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
Message-ID: <00f001c50af4$b1cfa2c0$8000000a@blorp>

> On Fri, Feb 04, 2005 at 11:33:18AM -0800, Tom Duffy wrote:
>> On Fri, 2005-02-04 at 11:21 -0800, Libor Michalek wrote:
>> >   3) Since the rest of the tree is not destabilized or effected in
>> >      any other way by the code, just check it into tree itself.
>> >      (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)
>> >
>> >   Anyone have thoughts on this? Personally I'm leaning towards #3,
>> > but that's because it's the least amount of work for me. :)
>>
>> I would like #3 as well.  Libor, would it be possible to get a preview
>> of what is going to be checked in?  Maybe with a small one-pager about
>> how to use it, etc?
>
>  You mean a quick description of the code, like the primary contents of
> each file and how to get it to do something ?
>
> -Libor

I would be interested in knowing whether zero copy, asynchronous sockets 
made it into the code and, much like the SDP performance tests last year, a 
set of netperf figures and a comparison of cpu utilisation with bandwidth 
would be nice to whet my appetite.

I guess the main issue with location is in regard to what constitutes the 
kernel submission. AFAIK its still uncertain as to the legal position of 
using SDP code, is that right? or did it get resolved to everyone's 
satisfaction (esp greg kh :) )

Great work everybody





From mlleinin at hpcn.ca.sandia.gov  Fri Feb  4 12:11:31 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Fri, 04 Feb 2005 12:11:31 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <20050204114207.B1592@topspin.com>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
Message-ID: <1107547891.2543.463.camel@localhost>

On Fri, 2005-02-04 at 11:42 -0800, Libor Michalek wrote:
> On Fri, Feb 04, 2005 at 11:33:18AM -0800, Tom Duffy wrote:
> > On Fri, 2005-02-04 at 11:21 -0800, Libor Michalek wrote:
> > >   3) Since the rest of the tree is not destabilized or effected in
> > >      any other way by the code, just check it into tree itself.
> > >      (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)
> > > 
> > >   Anyone have thoughts on this? Personally I'm leaning towards #3,
> > > but that's because it's the least amount of work for me. :)
> > 
> > I would like #3 as well.  Libor, would it be possible to get a preview
> > of what is going to be checked in?  Maybe with a small one-pager about
> > how to use it, etc?
> 
>   You mean a quick description of the code, like the primary contents of
> each file and how to get it to do something ?
> 
  How about a description of the code, a todo list, and an sdp_faq we
can put on the website and in svn.  

 Thanks,

   - Matt




From libor at topspin.com  Fri Feb  4 12:31:07 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 4 Feb 2005 12:31:07 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <00f001c50af4$b1cfa2c0$8000000a@blorp>;
	from paul.baxter@dsl.pipex.com on Fri, Feb 04, 2005 at 08:04:10PM
	-0000
References: <20050204112115.A1592@topspin.com><1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<00f001c50af4$b1cfa2c0$8000000a@blorp>
Message-ID: <20050204123107.C1592@topspin.com>

On Fri, Feb 04, 2005 at 08:04:10PM -0000, Paul Baxter wrote:
> > On Fri, Feb 04, 2005 at 11:33:18AM -0800, Tom Duffy wrote:
> >> On Fri, 2005-02-04 at 11:21 -0800, Libor Michalek wrote:
> >> >   3) Since the rest of the tree is not destabilized or effected in
> >> >      any other way by the code, just check it into tree itself.
> >> >      (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)
> >> >
> >> >   Anyone have thoughts on this? Personally I'm leaning towards #3,
> >> > but that's because it's the least amount of work for me. :)
> >>
> >> I would like #3 as well.  Libor, would it be possible to get a preview
> >> of what is going to be checked in?  Maybe with a small one-pager about
> >> how to use it, etc?
> >
> >  You mean a quick description of the code, like the primary contents of
> > each file and how to get it to do something ?
> 
> I would be interested in knowing whether zero copy, asynchronous sockets 
> made it into the code and, much like the SDP performance tests last year, a 
> set of netperf figures and a comparison of cpu utilisation with bandwidth 
> would be nice to whet my appetite.

  Linux AIO sockets support is in place, but AIO plus RDMA is waiting
on FMR support in mthca. Once the intial code is in place one of the
features I was planning in this area was SDP RDMA support for blocking
synchronous sockets. This should be much easier in 2.6 since the sync
and async APIs now have the same socket entry points.

  I was hoping to do the performance figures on PCIe systems, but I
seem to be having 2.6 sata support issues on the systems I have, so I
might provide some numbers on the older PCI-X systems.

> I guess the main issue with location is in regard to what constitutes the 
> kernel submission. AFAIK its still uncertain as to the legal position of 
> using SDP code, is that right? or did it get resolved to everyone's 
> satisfaction (esp greg kh :) )

  Well, unlike the previous submission, I was planning on making this
submission using the exact same license as the rest of the OpenIB code,
without reference to any other "possible" IP. 

  I'm not sure what will be required for different people's satisfaction
on the legal position, and I suspect that it will evolve over time.
However, I believe the best way to make that happen is for everyone to be
able to look at and comment of the code itself. 


-Libor




From tduffy at sun.com  Fri Feb  4 13:26:32 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 04 Feb 2005 13:26:32 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <20050204114207.B1592@topspin.com>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
Message-ID: <1107552392.2435.6.camel@duffman>

On Fri, 2005-02-04 at 11:42 -0800, Libor Michalek wrote:
>   You mean a quick description of the code, like the primary contents of
> each file and how to get it to do something ?

That would be great start.

Is there a userland LD_PRELOAD library required?  Is there a new PF_SDP
family or does it transparently work if you are on top of IB hardware
with PF_INET?  What is the opt-[in|out] mechanism?

If a socket is listen()'ing, can it accept() connections on both TCP and
SDP?  Or does an app need to choose SDP or TCP?  Same goes for connect
().

Thanks,

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050204/5adf6787/attachment.sig>

From libor at topspin.com  Fri Feb  4 14:20:20 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 4 Feb 2005 14:20:20 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1107552392.2435.6.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 04, 2005 at 01:26:32PM -0800
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
Message-ID: <20050204142020.D1592@topspin.com>

On Fri, Feb 04, 2005 at 01:26:32PM -0800, Tom Duffy wrote:
> On Fri, 2005-02-04 at 11:42 -0800, Libor Michalek wrote:
> >   You mean a quick description of the code, like the primary contents of
> > each file and how to get it to do something ?
> 
> That would be great start.
> 
> Is there a userland LD_PRELOAD library required?  Is there a new PF_SDP
> family or does it transparently work if you are on top of IB hardware
> with PF_INET?  What is the opt-[in|out] mechanism?
> 
> If a socket is listen()'ing, can it accept() connections on both TCP and
> SDP?  Or does an app need to choose SDP or TCP?  Same goes for connect().

  The SDP module itself provides no transparency, it is implemented as a
seperate protocol family. I can provide a userspace LD_PRELOAD library for
configurable limited transparency. However, I think the best longer term
solution is a kernel level socket switch, which it would be nice to have
independent of the protocol modules themselves. There are a some people
here who have some good ideas and experience in this. The other option to
explore is the SDP port mapper protocol from the RDMAC.

-Libor


From robert.j.woodruff at intel.com  Fri Feb  4 14:32:20 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Fri, 4 Feb 2005 14:32:20 -0800
Subject: [openib-general] RFC on SDP checkin
Message-ID: <1AC79F16F5C5284499BB9591B33D6F00037B09A9@orsmsx408>

 
>The SDP module itself provides no transparency, it is implemented as a
>seperate protocol family. I can provide a userspace LD_PRELOAD library
for
>configurable limited transparency. However, I think the best longer
term
>solution is a kernel level socket switch, which it would be nice to
have
>independent of the protocol modules themselves. There are a some people
>here who have some good ideas and experience in this. The other option
to
>explore is the SDP port mapper protocol from the RDMAC.

>-Libor

There is a socket switch in the sourceforge code if people want to take
a 
look at an example. I think however that as a first step,
we go with the separate address family as it would be less controversial
and would give people some time to be come comfortable with the SDP
idea, then perhaps discuss with Dave Miller on how best to implement a 
transparent socket switch. 

woody


From nauzad at gmail.com  Fri Feb  4 17:17:28 2005
From: nauzad at gmail.com (Nauzad Sadry)
Date: Fri, 4 Feb 2005 17:17:28 -0800
Subject: [openib-general] OpenIB Architecture, kernel mode driver examples
Message-ID: <aa7b01ec05020417176d492238@mail.gmail.com>

Hello folks

I am a newbie in Infiniband. I managed to install & get 2 machines
with Mellanox MT23108 HCAs to connect through a Mellanox switch
MTS2400. I am currently using the Mellanox IBGD-1.6.1 drivers. I have
the following questions

1. I wanted to understand the Software Architecure of the OpenIB. Do
we have any document that describes this architecture (like a Theory
of Operations). An architecture document is available in the
Infiniband Linux Project supported on sourceforge. I was wondering if
there is/are such document(s) for OpenIB.

2. What are the APIs exposed by OpenIB Access Layer ?? I am not
referring to the VAPI_XXX APIs but the ib_XXX APIs ?? Can someone
refer me to some use-cases ??

3. I want to write a simple Linux kernel mode driver that uses the
APIs exposed by the OpenIB Access Layer (NOT the VAPI). The driver
does simple send/receive of messages. It does not need to use IPoIB or
SDP or other kernel level services. Is there anyway I can get access
to Sample code for such a driver.

Your help will be really appreciated

Thanks 

Nauzad


From iod00d at hp.com  Fri Feb  4 17:31:28 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 4 Feb 2005 17:31:28 -0800
Subject: [openib-general] OpenIB Architecture, kernel mode driver examples
In-Reply-To: <aa7b01ec05020417176d492238@mail.gmail.com>
References: <aa7b01ec05020417176d492238@mail.gmail.com>
Message-ID: <20050205013128.GI5568@esmail.cup.hp.com>

On Fri, Feb 04, 2005 at 05:17:28PM -0800, Nauzad Sadry wrote:
> 1. I wanted to understand the Software Architecure of the OpenIB. Do
> we have any document that describes this architecture (like a Theory
> of Operations). An architecture document is available in the
> Infiniband Linux Project supported on sourceforge. I was wondering if
> there is/are such document(s) for OpenIB.

See "Documentation" link on www.openib.org.
You might also look at http://www.infinibandta.org/home

> 2. What are the APIs exposed by OpenIB Access Layer ?? I am not
> referring to the VAPI_XXX APIs but the ib_XXX APIs ?? Can someone
> refer me to some use-cases ??

I'm hoping that is covered under "Documentation"...if not...

> 3. I want to write a simple Linux kernel mode driver that uses the
> APIs exposed by the OpenIB Access Layer (NOT the VAPI). The driver
> does simple send/receive of messages. It does not need to use IPoIB or
> SDP or other kernel level services. Is there anyway I can get access
> to Sample code for such a driver.

You can get the actual code. See "Developement tools" and how
to pull code from the subversion source code repository.
The current IPoIB driver is included.

hth,
grant


From iod00d at hp.com  Fri Feb  4 21:14:42 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 4 Feb 2005 21:14:42 -0800
Subject: [openib-general] OpenIB Architecture, kernel mode driver examples
In-Reply-To: <aa7b01ec050204175074f006e9@mail.gmail.com>
References: <aa7b01ec05020417176d492238@mail.gmail.com>
	<20050205013128.GI5568@esmail.cup.hp.com>
	<aa7b01ec050204175074f006e9@mail.gmail.com>
Message-ID: <20050205051442.GA9072@esmail.cup.hp.com>

On Fri, Feb 04, 2005 at 05:50:59PM -0800, Nauzad Sadry wrote:
> The architecture available in the openib.org website
> OpenIB-Arch-122104.pdf is just a set of PPTs & does not explain
> anything in detail. It just shows me the System Architecture in
> general with no use-cases or theory of operations.

correct - it's just an overview document.
I gather you are looking for the "Openib Driver Programing Guide".

Just curious: have you ever worked on an open source project?

> It does NOT contain any reference to the APIs exposed by OpenIB Access Layer.

The only other docs I'm aware of specific to openib.org are in SVN:
grundler at gsyprf3:/usr/src/openib_gen2/src/linux-kernel/docs$ ls
ipoib.txt  ipoib_faq.txt  sysfs.txt  user_mad.txt

But I don't think those will cover what you want either.
You really need to look at the header files (source code) in SVN:
grundler at gsyprf3:/usr/src/openib_gen2/src/linux-kernel/infiniband/include$ ls
ib_cache.h     ib_mad.h   ib_sa.h   ib_user_mad.h
ib_fmr_pool.h  ib_pack.h  ib_smi.h  ib_verbs.h

Are the API's you need defined in ib-verbs.h:struct ib_device?
(Hint: there is a very long list of function pointers)


> Do the documents specifed in the http://infiniband.sourceforge.net
> hold any relevance with openib ??

For "theory of operation", I would think yes. Infiniband implementations
can vary but ultimately they have to do the same basic things because
the HW design (and architecture) drives that.

> The code involved in SRP, SDP & IPoIB is too complicated &
> undocumented to try & understand the use-cases & APIs.

If you were talking about the original implementation a bunch
of linux kernel folks would agree with you.

The entire IPoIB driver is < 4k lines of code.
There is no SRP or SDP driver (yet).

Modern NIC drivers are substantially more.
tg3 is > 8K lines.  e1000 is over 10K lines.
acenic and e100 drivers are ~3000 lines of code as well but
are much simpler implementations (e.g. no TSO).

I don't find any of them particularly complex for basic operation.
The initialization sequences are usually the ugliest bits that
are hard to sort out.

> If someone has
> a sample kernel mode driver using the AccessLayer APIs, then I would
> really appreciate it.

IPoIB *is* the sample code.  If something isn't clear about the
IPoIB driver, folks on this list can probably answer your questions.

But you might do well to use google to search the mailing list
archives a bit before asking alot of questions.

hth,
grant


From halr at voltaire.com  Sat Feb  5 05:24:57 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Feb 2005 08:24:57 -0500
Subject: [openib-general] OpenIB Architecture, kernel mode driver examples
In-Reply-To: <aa7b01ec05020417176d492238@mail.gmail.com>
References: <aa7b01ec05020417176d492238@mail.gmail.com>
Message-ID: <1107609770.7142.23.camel@localhost.localdomain>

On Fri, 2005-02-04 at 20:17, Nauzad Sadry wrote:
> 2. What are the APIs exposed by OpenIB Access Layer ?? I am not
> referring to the VAPI_XXX APIs but the ib_XXX APIs ?? Can someone
> refer me to some use-cases ??

What ib_XXX calls are you referring to ? Are these in Mellanox Gold
1.6.1 or IBAL ? Can you provide some description of what you are looking
for ?

Thanks.

-- Hal



From halr at voltaire.com  Sat Feb  5 05:38:15 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Feb 2005 08:38:15 -0500
Subject: [openib-general] osm_sa_mc_member_record question
Message-ID: <1107610480.7142.50.camel@localhost.localdomain>

Hi Eitan,

In osm_sa_mc_member_record.c, at line 1526 there is the following code
in __osm_sa_mcm_by_comp_mask_cb:

  if (IB_MCR_COMPMASK_SL & comp_mask )
    sl_flow_hop_mask = sl_flow_hop_mask | 0x000F;

  if (IB_MCR_COMPMASK_FLOW & comp_mask)
    sl_flow_hop_mask = sl_flow_hop_mask | 0x0FF0;

  if (IB_MCR_COMPMASK_HOP & comp_mask)
    sl_flow_hop_mask = sl_flow_hop_mask | 0xF000;

Shouldn't Flow Label should be 20 bits (not 8) and Hop Limit 8 bits
rather than 4 ?

Thanks.

-- Hal



From nauzad at gmail.com  Sat Feb  5 16:26:46 2005
From: nauzad at gmail.com (Nauzad Sadry)
Date: Sat, 5 Feb 2005 16:26:46 -0800
Subject: [openib-general] OpenIB Architecture, kernel mode driver examples
In-Reply-To: <1107609770.7142.23.camel@localhost.localdomain>
References: <aa7b01ec05020417176d492238@mail.gmail.com>
	<1107609770.7142.23.camel@localhost.localdomain>
Message-ID: <aa7b01ec0502051626411ec8e2@mail.gmail.com>

Hi Hal

I was referring to the APIs that are a part of the OpenIB Access Layer
as distributed by Mellanox Gold 1.6.1. I believe this includes OpenIB
gen1 & not gen2

The APIs I was looking for were like ib_device_properties_get,
ib_pd_create, ib_qp_create, ib_send etc. No where in the source-code
do I see any description of what each function does.

I was primarily interested in identifying the use-cases for lifecycle,
connection establishment, send/receive data, memory/buffer management
for any Linux kernel mode driver that uses the OpenIB Access Layer (as
distributed by Mellanox Gold 1.6.1). This would have helped me in
starting my driver implementation

Judging by the response I got from Grant, I believe reviewing the
source code is the only option available to me. Also the document
available on the Infiniband Linux project on SourceForge should be a
decent guide to explain the Theory of Operations.

If there is anything else that I should be knowing

Thanks 

Nauzad

On Sat, 05 Feb 2005 05:34:09 -0800 (PST), Hal Rosenstock
<halr at voltaire.com> wrote:
> On Fri, 2005-02-04 at 20:17, Nauzad Sadry wrote:
> > 2. What are the APIs exposed by OpenIB Access Layer ?? I am not
> > referring to the VAPI_XXX APIs but the ib_XXX APIs ?? Can someone
> > refer me to some use-cases ??
> 
> What ib_XXX calls are you referring to ? Are these in Mellanox Gold
> 1.6.1 or IBAL ? Can you provide some description of what you are looking
> for ?
> 
> Thanks.
> 
> -- Hal
> 
>


From sean.hefty at intel.com  Sat Feb  5 17:11:53 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Sat, 5 Feb 2005 17:11:53 -0800
Subject: [openib-general] OpenIB Architecture, kernel mode driver examples
In-Reply-To: <aa7b01ec05020417176d492238@mail.gmail.com>
Message-ID: <ORSMSX401A1XvpFVjCR0000004e@orsmsx401.amr.corp.intel.com>

>3. I want to write a simple Linux kernel mode driver that uses the
>APIs exposed by the OpenIB Access Layer (NOT the VAPI). The driver
>does simple send/receive of messages. It does not need to use IPoIB or
>SDP or other kernel level services. Is there anyway I can get access
>to Sample code for such a driver.

Try looking at
https://openib.org/svn/gen2/utils/src/linux-kernel/infiniband/util/cmpost/cm
post.c for a simple (relative to IBA anyway) example program that connects
and sends/receive data.

- Sean



From shaharf at voltaire.com  Sun Feb  6 00:56:16 2005
From: shaharf at voltaire.com (shaharf)
Date: Sun, 6 Feb 2005 10:56:16 +0200
Subject: [openib-general] [ANNOUNCE] new tracert utility
Message-ID: <D4F8F0B3820E754C887699BEF26A894054CAA0@taurus.voltaire.com>

Hi all,

 

            A new utility is added to the management tree
(gen2/userspace/management/diags/net/ibtracert). Tracert shows the path
between two lids. It can show also multicast path, once the mild is
given (see the usage text).

 

            As a general note, please report any problem you find using
these utilities. I know about problems with ia32 machines concerning few
utilities. If someone knows about additional problems, please let me
know.

 

            Any additional comments and suggestions are welcomed.

 

Shahar

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050206/014b38c0/attachment.html>

From mst at mellanox.co.il  Sun Feb  6 06:14:56 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 6 Feb 2005 16:14:56 +0200
Subject: [openib-general] Re: [PATCH] 2/2 : ip over ib tx/rx split
In-Reply-To: <20050203213341.GC5108@mellanox.co.il>
References: <20050203180233.GK2536@mellanox.co.il> <52ekfxig39.fsf@topspin.com>
	<20050203204851.GB5108@mellanox.co.il> <52wttpgy3n.fsf@topspin.com>
	<20050203213341.GC5108@mellanox.co.il>
Message-ID: <20050206141455.GD5860@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: Re: [PATCH] 2/2 : ip over ib tx/rx split
> 
> 
> Quoting r. Roland Dreier <roland at topspin.com>:
> > Subject: Re: [openib-general] [PATCH] 2/2 : ip over ib tx/rx split
> > 
> >     Roland> The only difference is that we unlock the receive CQ and
> >     Roland> lock the send CQ in the middle.
> > 
> >     Michael> Are you sure?  I *do* have msi-x on these machines.
> > 
> > Of course I'm not sure, but as far as I know, with the current drivers
> > even with MSI-X there's still only one completion interrupt handler,
> > which can only run on one CPU at a time.
> > 
> 
> Sounds convincing. So the improvement is probably due to
> better send/receive parallelism we are getting in mthca.

Roland, I just made some measurements.

With the split lock in mthca, but a shared cq in ip over ib, bandwidth goes
up from around 300 to around 313 Mb/sec.
So for the mthca patch, I see almost 4% improvement - seems
like a good thing, so please consider applying it.

My ip over ib patch seems to need a bit more work before its actually
useful.

-- 
MST - Michael S. Tsirkin


From danb at voltaire.com  Mon Feb  7 07:06:52 2005
From: danb at voltaire.com (Dan Bar Dov)
Date: Mon, 7 Feb 2005 17:06:52 +0200
Subject: [openib-general] Announcement: iSER (iSCSI RDMA Extension) code
	added to OpenIB
Message-ID: <D4F8F0B3820E754C887699BEF26A894054CC6B@taurus.voltaire.com>

I am glad to announce the posting of iSER to the OpenIB svn. 

The code is now available under:   https://openib.org/svn/gen2/ulps/iser/

iSER is an important addition to OpenIB, adding significant capabilities and performance to InfiniBand enabled block and object storage

Mr. Yaron Haviv will cover in detail the iSER code architecture and implementation, during his Tuesday morning session (D9) at the 2005 OpenIB Developers Workshop.

 
About OpenIB-iSER:

OpenIB iSER is a fully functional and tested implementation, following all the latest iSCSI, DA, and iSER IETF draft RFC�s (including the latest generalization of iSER to InfiniBand).

iSER is a new direct access transport under iSCSI in addition to the TCP one, it is part of the new iSCSI Datamover Architecture (DA) separating the iSCSI control and management functionality from the actual SCSI Commands+Data.

iSCSI/iSER can be used for native InfiniBand storage and InfiniBand to GbE or FC gateways, where the implementation can leverage on all the iSCSI infrastructure including naming, discovery, security, notifications, error recovery, etc� in a transparent manner. 

The actual transport (TCP or iSER) can be automatically selected based on the available hardware and path to target, enabling a single driver for GbE, IB, and FC which is centrally provisioned through iSCSI mechanisms such as iSNS and SLP.

OpenIB-iSER  conforms to the  Datamover  Architecture , it works with the Linux-iSCSI  4.2 branch, or with the latest  head  patch from Voltaire adding the Datamover Architecture support (an API  layer  for pluggable Datamover modules), it can also be used by other block or object (or cluster file system) iSCSI based implementations by simply adding the Datamover layer (the Linux-iSCSI DA patch changes/adds ~1000 LOC, such an adaptation can be done in few weeks)
 

More details on iSER, and links can be found in:

http://www.voltaire.com/iser.htm


Dan Bar Dov
Project Manager
Infiniband Storage Solutions
T +972-9-9717636
C +972-54-8080505
www.voltaire.com
The Grid Interconnect Company
 


From csndvl at theteleport.com  Mon Feb  7 08:12:35 2005
From: csndvl at theteleport.com (Staint Wallace Clinic)
Date: Mon, 07 Feb 2005 08:12:35 -0800
Subject: [openib-general] Prompt V1@gra, X@NAX, Valiu/m/
Message-ID: <20050207161408.66FEA2283D4@openib.ca.sandia.gov>

http://zcpbqlzkhrqdwl182.fdnchxjdyrt.com/
Memory is a child walking along a seashore. You never can tell what small pebble it will pick up and store away among its treasured things. I'd rather have a bottle in front of me, than a frontal lobotomy. Why is it drug addicts and computer afficionados are both called users? Change is inevitable, except from vending machines. The great thing about a computer notebook is that no matter how much you stuff into it, it doesn't get bigger or heavier. Without question, the greatest invention in the history of mankind is beer. Oh, I grant you that the wheel was also a fine invention, but the wheel does not go nearly as well with pizza. If you find a path with no obstacles, it probably doesn't lead anywhere. Always listen to experts. They'll tell you what can't be done, and why. Then do it. Always listen to experts. They'll tell you what can't be done, and why. Then do it. Memory is a child walking along a seashore. You never can tell what small pebble it will pick up and store away among its treasured things. A happy person is not a person in a certain set of circumstances, but rather a person with a certain set of attitudes. You can't be a real country unless you have a beer and an airline. It helps if you have some kind of a football team, or some nuclear weapons, but at the very least you need a beer. A woman knows the face of the man she loves as a sailor knows the open sea. Change is inevitable, except from vending machines. Love is life. And if you miss love, you miss life. I would like to believe when I die that I have given myself away like a tree that sows seed every spring and never counts the loss, because it is not loss, it is adding to future life.  It is the tree's way of being.  Strongly rooted perhaps, but spilling out its treasure on the wind. Drinking provides a beautiful excuse to pursue the one activity that truly gives me pleasure -- hooking up with fat, hairy girls. Touch is the most fundamental sense. A baby experiences it, all over, before he is born and long after he learns to use sight, hearing, or taste, and no human ever ceases to need it. Keep your children short on pocket money - but long on hugs. The problem with some people is that when they aren't drunk, they're sober. Who would give a law to lovers? Love is unto itself a higher law. The quality of the moment is more important than the number of our days. Love is life. And if you miss love, you miss life. Who so loves believes the impossible. All love shifts and changes. I don't know if you can be wholeheartedly in love all the time. Blessed is the man who, having nothing to say, abstains from giving us wordy evidence of the fact. Hatch a dream and then believe it. Who so loves believes the impossible. Books are the shoes with which we tread the footsteps of great minds. You can widen your life by yourself, but to deepen it you need a friend. Each encounter that becomes a friendship turns into a lifeline. One can never have too many, only too many to properly take care of. Love is the energy of the soul. Love is what heals the personality. There is nothing that cannot be healed by love. There is nothing but love. Who would give a law to lovers? Love is unto itself a higher law. I'd rather have a bottle in front of me than a ... brain operation. Reduce the complexity of life by eliminating the needless wants of life, and the labors of life reduce themselves. The man who says I told you so is making an excuse for not expressing his position clearly in the first place. Love is the energy of the soul. Love is what heals the personality. There is nothing that cannot be healed by love. There is nothing but love. I would like to believe when I die that I have given myself away like a tree that sows seed every spring and never counts the loss, because it is not loss, it is adding to future life.  It is the tree's way of being.  Strongly rooted perhaps, but spilling out its treasure on the wind. Love is the beauty of the soul. Can miles truly separate you from friends... If you want to be with someone you love, aren't you already there? Reduce the complexity of life by eliminating the needless wants of life, and the labors of life reduce themselves. You can't be a real country unless you have a beer and an airline. It helps if you have some kind of a football team, or some nuclear weapons, but at the very least you need a beer. I feel sorry for people who don't drink. When they wake up in the morning, that's as good as they're going to feel all day. The ultimate measure of a man is not where he stands in moments of comfort and convenience, but where he stands in times of challenge and controversy. Compassion is the antitoxin of the soul: where there is compassion even the most poisonous impulses remain relatively harmless. Love takes off masks that we fear we cannot live without and know we cannot live within. A kiss is a lovely trick designed by nature to stop speech when words become superfluous. If you find a path with no obstacles, it probably doesn't lead anywhere. Very little is needed to make a happy life. It is all within yourself, in your way of thinking. It's not that chocolates are a substitute for love. Love is a substitute for chocolate. Chocolate is, let's face it, far more reliable than a man. The problem with some people is that when they aren't drunk, they're sober. Beer is proof that God loves us and wants us to be happy. No dictator, no invader, can hold an imprisoned population by force of arms forever. There is no greater power in the universe than the need for freedom. Against that power, governments and tyrants and armies cannot stand. What the world really needs is more love and less paper work. Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. The great thing about television is that if something important happens anywhere in the world, day or night, you can always change the channel. You can't be a real country unless you have a beer and an airline. It helps if you have some kind of a football team, or some nuclear weapons, but at the very least you need a beer. Solitude, if rightly used, becomes not only a privilege but a necessity. Only a superficial soul fears to fraternize with itself. Life is a waste of time, time is a waste of life, so get wasted all of the time and have the time of your life. Your living is determined not so much by what life brings to you as by the attitude you bring to life; not so much by what happens to you as by the way your mind looks at what happens. A positive attitude will not solve all your problems, but it will annoy enough people to make it worth the effort. The problem with some people is that when they aren't drunk, they're sober. Not all chemicals are bad. Without chemicals such as hydrogen and oxygen, for example, there would be no way to make water, a vital ingredient in beer. 24 hours in a day, 24 beers in a case. Coincidence? Not all chemicals are bad. Without chemicals such as hydrogen and oxygen, for example, there would be no way to make water, a vital ingredient in beer. The best proof of love is trust. Always remember that I have taken more out of alcohol than alcohol has taken out of me. Too often we give our children answers to remember rather than problems to solve. Iron rusts from disuse; stagnant water loses its purity and in cold weather becomes frozen; even so does inaction sap the vigor of the mind. Love is life. And if you miss love, you miss life. An intelligent man is sometimes forced to be drunk to spend time with his fools. Who would give a law to lovers? Love is unto itself a higher law. Beer is proof that God loves us and wants us to be happy. Love does not begin and end the way we seem to think it does. Love is a battle, love is a war; love is a growing up. Abstainer: a weak person who yields to the temptation of denying himself a pleasure. The problem with the world is that everyone is a few drinks behind. Before I got married I had six theories about bringing up children; now I have six children and no theories. I discovered I always have choices and sometimes it's only a choice of attitude. A positive attitude will not solve all your problems, but it will annoy enough people to make it worth the effort. Always remember that I have taken more out of alcohol than alcohol has taken out of me. Life is a waste of time, time is a waste of life, so get wasted all of the time and have the time of your life. A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a shp, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects. No matter what looms ahead, if you can eat today, enjoy today, mix good cheer with friends today enjoy it and bless God for it. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050207/36528b26/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gcngophp.jpg
Type: image/jpeg
Size: 15902 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050207/36528b26/attachment.jpg>

From tduffy at sun.com  Mon Feb  7 08:15:38 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 07 Feb 2005 08:15:38 -0800
Subject: [openib-general] [Fwd: Bug#293493 acknowledged by developer
	(Bug#293493: fixed in udev 0.051-1)]
Message-ID: <1107792938.24090.0.camel@duffman>


-------------- next part --------------
An embedded message was scrubbed...
From: owner at bugs.debian.org (Debian Bug Tracking System)
Subject: Bug#293493 acknowledged by developer         (Bug#293493: fixed in	udev 0.051-1)
Date: Sat, 05 Feb 2005 09:49:13 -0800
Size: 6299
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050207/29183d20/attachment.mht>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050207/29183d20/attachment.sig>

From mst at mellanox.co.il  Mon Feb  7 08:26:57 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 7 Feb 2005 18:26:57 +0200
Subject: [openib-general] un-signaled completions in mthca
Message-ID: <20050207162657.GA2141@mellanox.co.il>

Hi, Roland!
I tried clearing the IB_SEND_SIGNALED in some work requests, and I seem
to be still getting all the completions.

Have you seen something like this?
This is on Arbel, Tavor mode, x86_64.

MST

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb  7 08:38:01 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 7 Feb 2005 18:38:01 +0200
Subject: [openib-general] Re: un-signaled completions in mthca
In-Reply-To: <20050207162657.GA2141@mellanox.co.il>
References: <20050207162657.GA2141@mellanox.co.il>
Message-ID: <20050207163801.GC2141@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: un-signaled completions in mthca
> 
> Hi, Roland!
> I tried clearing the IB_SEND_SIGNALED in some work requests, and I seem
> to be still getting all the completions.
> 
> Have you seen something like this?
> This is on Arbel, Tavor mode, x86_64.
> 
> MST

Sorry, my bad, I set the wrong policy.
-- 
MST - Michael S. Tsirkin


From shaharf at voltaire.com  Mon Feb  7 09:25:28 2005
From: shaharf at voltaire.com (shaharf)
Date: Mon, 7 Feb 2005 19:25:28 +0200
Subject: [openib-general] [ANNOUNCE] New utilities - ibaddr, perfquery
Message-ID: <D4F8F0B3820E754C887699BEF26A894054CCA5@taurus.voltaire.com>

Hi all,

 

            I added two new utilities to the management tree
(gen2/userspace/management):

 

            ibaddr: shows the lid range and the default GID of the
target, the default is the local port.

 

            perfquery: dump (and optionally clears) the performance
counters (including error counters) of the destination port.

 

 

            I also improved the gen2/userspace/management/Makefile to
support the automake environment. 3 new targets are added:

 

            svnclean: remove any non svn files from the tree (be
carefull!)

 

            automake: configure and build the automake system and build
everything. After the initial automake, "make && make install" should be
enough, unless libraries are changed.

 

            origmake: restore the original make file system. After that,
"make && make install"  will build and install everything.

 

Shahar

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050207/0d1a0f79/attachment.html>

From tduffy at sun.com  Mon Feb  7 13:22:54 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 07 Feb 2005 13:22:54 -0800
Subject: [openib-general] [PATCH] teach ifconfig about ib [WAS: Latest IPoIB
	Bringup Questions]
In-Reply-To: <1098985903.17991.74.camel@hpc-1>
References: <1098985903.17991.74.camel@hpc-1>
Message-ID: <1107811374.6917.6.camel@duffman>

[Responding to an old message]

On Thu, 2004-10-28 at 13:51 -0400, Hal Rosenstock wrote:
> Should we teach ifconfig to display Link Encap: INFINIBAND ?

Still has the problem of truncating the address to the first 14 bytes.

diff -Nur net-tools-1.60/config.in net-tools-1.60-ib/config.in
--- net-tools-1.60/config.in	2000-05-21 07:32:12.000000000 -0700
+++ net-tools-1.60-ib/config.in	2005-02-07 10:45:14.108286619 -0800
@@ -82,6 +82,7 @@
 bool '(Cisco)-HDLC/LAPB support' HAVE_HWHDLCLAPB n
 bool 'IrDA support' HAVE_HWIRDA y
 bool 'Econet hardware support' HAVE_HWEC n
+bool 'InfiniBand hardware support' HAVE_HWIB y
 *
 *
 *           Other Features.
diff -Nur net-tools-1.60/config.make net-tools-1.60-ib/config.make
--- net-tools-1.60/config.make	2005-02-07 11:58:18.536146922 -0800
+++ net-tools-1.60-ib/config.make	2005-02-07 12:04:03.596462891 -0800
@@ -30,6 +30,7 @@
 HAVE_HWHDLCLAPB=1
 HAVE_HWIRDA=1
 HAVE_HWEC=1
+HAVE_HWIB=1
 HAVE_FW_MASQUERADE=1
 HAVE_IP_TOOLS=1
 HAVE_MII=1
Binary files net-tools-1.60/ipmaddr and net-tools-1.60-ib/ipmaddr differ
Binary files net-tools-1.60/iptunnel and net-tools-1.60-ib/iptunnel differ
diff -Nur net-tools-1.60/lib/hw.c net-tools-1.60-ib/lib/hw.c
--- net-tools-1.60/lib/hw.c	2000-05-20 11:27:25.000000000 -0700
+++ net-tools-1.60-ib/lib/hw.c	2005-02-07 09:56:22.315428035 -0800
@@ -73,6 +73,8 @@
 
 extern struct hwtype ec_hwtype;
 
+extern struct hwtype ib_hwtype;
+
 static struct hwtype *hwtypes[] =
 {
 
@@ -144,6 +146,9 @@
 #if HAVE_HWX25
     &x25_hwtype,
 #endif
+#if HAVE_HWIB
+    &ib_hwtype,
+#endif
     &unspec_hwtype,
     NULL
 };
@@ -217,6 +222,9 @@
 #if HAVE_HWEC
     ec_hwtype.title = _("Econet");
 #endif
+#if HAVE_HWIB
+    ib_hwtype.title = _("InfiniBand");
+#endif
     sVhwinit = 1;
 }
 
diff -Nur net-tools-1.60/lib/ib.c net-tools-1.60-ib/lib/ib.c
--- net-tools-1.60/lib/ib.c	1969-12-31 16:00:00.000000000 -0800
+++ net-tools-1.60-ib/lib/ib.c	2005-02-07 12:55:04.635559244 -0800
@@ -0,0 +1,147 @@
+/*
+ * lib/ib.c        This file contains an implementation of the "Infiniband"
+ *              support functions.
+ *
+ * Version:     $Id: ib.c,v 1.1 2005/02/06 11:00:47 tduffy Exp $
+ *
+ * Author:      Fred N. van Kempen, <waltje at uwalt.nl.mugnet.org>
+ *              Copyright 1993 MicroWalt Corporation
+ *		Tom Duffy <tduffy at sun.com>
+ *
+ *              This program is free software; you can redistribute it
+ *              and/or  modify it under  the terms of  the GNU General
+ *              Public  License as  published  by  the  Free  Software
+ *              Foundation;  either  version 2 of the License, or  (at
+ *              your option) any later version.
+ */
+#include "config.h"
+
+#if HAVE_HWIB
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <net/if_arp.h>
+#include <linux/if_infiniband.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <errno.h>
+#include <ctype.h>
+#include <string.h>
+#include <unistd.h>
+#include "net-support.h"
+#include "pathnames.h"
+#include "intl.h"
+#include "util.h"
+
+extern struct hwtype ib_hwtype;
+
+
+/* Display an InfiniBand address in readable format. */
+static char *pr_ib(unsigned char *ptr)
+{
+    static char buff[128];
+    char *pos;
+    unsigned int i;
+
+    pos = buff;
+    for (i = 0; i < INFINIBAND_ALEN; i++) {
+	pos += sprintf(pos, "%02X:", (*ptr++ & 0377));
+    }
+    buff[strlen(buff) - 1] = '\0';
+
+    /* snprintf(buff, sizeof(buff), "%02X:%02X:%02X:%02X:%02X:%02X",
+	     (ptr[0] & 0377), (ptr[1] & 0377), (ptr[2] & 0377),
+	     (ptr[3] & 0377), (ptr[4] & 0377), (ptr[5] & 0377)
+	);
+    */
+    return (buff);
+}
+
+
+/* Input an Infiniband address and convert to binary. */
+static int in_ib(char *bufp, struct sockaddr *sap)
+{
+    unsigned char *ptr;
+    char c, *orig;
+    int i;
+    unsigned val;
+
+    sap->sa_family = ib_hwtype.type;
+    ptr = sap->sa_data;
+
+    i = 0;
+    orig = bufp;
+    while ((*bufp != '\0') && (i < INFINIBAND_ALEN)) {
+	val = 0;
+	c = *bufp++;
+	if (isdigit(c))
+	    val = c - '0';
+	else if (c >= 'a' && c <= 'f')
+	    val = c - 'a' + 10;
+	else if (c >= 'A' && c <= 'F')
+	    val = c - 'A' + 10;
+	else {
+#ifdef DEBUG
+	    fprintf(stderr, _("in_ib(%s): invalid infiniband address!\n"), orig);
+#endif
+	    errno = EINVAL;
+	    return (-1);
+	}
+	val <<= 4;
+	c = *bufp;
+	if (isdigit(c))
+	    val |= c - '0';
+	else if (c >= 'a' && c <= 'f')
+	    val |= c - 'a' + 10;
+	else if (c >= 'A' && c <= 'F')
+	    val |= c - 'A' + 10;
+	else if (c == ':' || c == 0)
+	    val >>= 4;
+	else {
+#ifdef DEBUG
+	    fprintf(stderr, _("in_ib(%s): invalid infiniband address!\n"), orig);
+#endif
+	    errno = EINVAL;
+	    return (-1);
+	}
+	if (c != 0)
+	    bufp++;
+	*ptr++ = (unsigned char) (val & 0377);
+	i++;
+
+	/* We might get a semicolon here - not required. */
+	if (*bufp == ':') {
+	    if (i == INFINIBAND_ALEN) {
+#ifdef DEBUG
+		fprintf(stderr, _("in_ib(%s): trailing : ignored!\n"),
+			orig)
+#endif
+		    ;		/* nothing */
+	    }
+	    bufp++;
+	}
+    }
+
+    /* That's it.  Any trailing junk? */
+    if ((i == INFINIBAND_ALEN) && (*bufp != '\0')) {
+#ifdef DEBUG
+	fprintf(stderr, _("in_ib(%s): trailing junk!\n"), orig);
+	errno = EINVAL;
+	return (-1);
+#endif
+    }
+#ifdef DEBUG
+    fprintf(stderr, "in_ib(%s): %s\n", orig, pr_ib(sap->sa_data));
+#endif
+
+    return (0);
+}
+
+
+struct hwtype ib_hwtype =
+{
+    "infiniband", NULL, ARPHRD_INFINIBAND, INFINIBAND_ALEN,
+    pr_ib, in_ib, NULL
+};
+
+
+#endif				/* HAVE_HWETHER */
diff -Nur net-tools-1.60/lib/Makefile net-tools-1.60-ib/lib/Makefile
--- net-tools-1.60/lib/Makefile	2000-10-28 03:59:42.000000000 -0700
+++ net-tools-1.60-ib/lib/Makefile	2005-02-07 10:02:14.662640164 -0800
@@ -16,7 +16,7 @@
 #
 
 
-HWOBJS	 = hw.o loopback.o slip.o ether.o ax25.o ppp.o arcnet.o tr.o tunnel.o frame.o sit.o rose.o ash.o fddi.o hippi.o hdlclapb.o strip.o irda.o ec_hw.o x25.o
+HWOBJS	 = hw.o loopback.o slip.o ether.o ax25.o ppp.o arcnet.o tr.o tunnel.o frame.o sit.o rose.o ash.o fddi.o hippi.o hdlclapb.o strip.o irda.o ec_hw.o x25.o ib.o
 AFOBJS	 = unix.o inet.o inet6.o ax25.o ipx.o ddp.o ipx.o netrom.o af.o rose.o econet.o x25.o
 AFGROBJS = inet_gr.o inet6_gr.o ipx_gr.o ddp_gr.o netrom_gr.o ax25_gr.o rose_gr.o getroute.o x25_gr.o
 AFSROBJS = inet_sr.o inet6_sr.o netrom_sr.o ipx_sr.o setroute.o x25_sr.o
Binary files net-tools-1.60/mii-tool and net-tools-1.60-ib/mii-tool differ



From tduffy at sun.com  Mon Feb  7 15:46:29 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 07 Feb 2005 15:46:29 -0800
Subject: [openib-general] [PATCH] use standard path names in automake for
	managment
Message-ID: <1107819989.6917.16.camel@duffman>

Use standard paths for autotools.  You can always go back to using the
directory by specifying ./configure --exec-prefix=/usr/local/ib/ if you
want.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: diags/net/ibtracert/Makefile.am
===================================================================
--- diags/net/ibtracert/Makefile.am	(revision 1757)
+++ diags/net/ibtracert/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/net/ibnetdiscover/Makefile.am
===================================================================
--- diags/net/ibnetdiscover/Makefile.am	(revision 1757)
+++ diags/net/ibnetdiscover/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/net/perfquery/Makefile.am
===================================================================
--- diags/net/perfquery/Makefile.am	(revision 1757)
+++ diags/net/perfquery/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/net/smpdump/Makefile.am
===================================================================
--- diags/net/smpdump/Makefile.am	(revision 1757)
+++ diags/net/smpdump/Makefile.am	(working copy)
@@ -1,6 +1,6 @@
 
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband
Index: diags/net/smpquery/Makefile.am
===================================================================
--- diags/net/smpquery/Makefile.am	(revision 1757)
+++ diags/net/smpquery/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/net/ibaddr/Makefile.am
===================================================================
--- diags/net/ibaddr/Makefile.am	(revision 1757)
+++ diags/net/ibaddr/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/net/ibroute/Makefile.am
===================================================================
--- diags/net/ibroute/Makefile.am	(revision 1757)
+++ diags/net/ibroute/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/net/sminfo/Makefile.am
===================================================================
--- diags/net/sminfo/Makefile.am	(revision 1757)
+++ diags/net/sminfo/Makefile.am	(working copy)
@@ -1,6 +1,6 @@
 
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband \
Index: diags/host/ibstat/Makefile.am
===================================================================
--- diags/host/ibstat/Makefile.am	(revision 1757)
+++ diags/host/ibstat/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../../libibumad/include/infiniband
Index: diags/host/scripts/Makefile.am
===================================================================
--- diags/host/scripts/Makefile.am	(revision 1757)
+++ diags/host/scripts/Makefile.am	(working copy)
@@ -1,5 +1,4 @@
-
-bindir = ${exec_prefix}/ib/bin
+bindir = @bindir@
 
 bin_SCRIPTS = ibstatus
 
Index: libibumad/Makefile.am
===================================================================
--- libibumad/Makefile.am	(revision 1757)
+++ libibumad/Makefile.am	(working copy)
@@ -1,5 +1,4 @@
-
-libdir = ${exec_prefix}/ib/lib
+libdir = @libdir@
 
 SUBDIRS = .
 
Index: libibcommon/Makefile.am
===================================================================
--- libibcommon/Makefile.am	(revision 1757)
+++ libibcommon/Makefile.am	(working copy)
@@ -1,5 +1,4 @@
-
-libdir = ${exec_prefix}/ib/lib
+libdir = @libdir@
 
 SUBDIRS = .
 
Index: osm/opensm/Makefile.am
===================================================================
--- osm/opensm/Makefile.am	(revision 1757)
+++ osm/opensm/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../include \
 	   -I$(srcdir)/../../libibcommon/include/infiniband \
Index: osm/libvendor/Makefile.am
===================================================================
--- osm/libvendor/Makefile.am	(revision 1757)
+++ osm/libvendor/Makefile.am	(working copy)
@@ -1,5 +1,4 @@
-
-libdir = ${exec_prefix}/ib/lib
+libdir = @libdir@
 
 SUBDIRS = .
 
Index: osm/complib/Makefile.am
===================================================================
--- osm/complib/Makefile.am	(revision 1757)
+++ osm/complib/Makefile.am	(working copy)
@@ -1,5 +1,4 @@
-
-libdir = ${exec_prefix}/ib/lib
+libdir = @libdir@
 
 SUBDIRS = .
 
Index: libibmad/Makefile.am
===================================================================
--- libibmad/Makefile.am	(revision 1757)
+++ libibmad/Makefile.am	(working copy)
@@ -1,5 +1,4 @@
-
-libdir = ${exec_prefix}/ib/lib
+libdir = @libdir@
 
 SUBDIRS = .
 
Index: util/mad_test/Makefile.am
===================================================================
--- util/mad_test/Makefile.am	(revision 1757)
+++ util/mad_test/Makefile.am	(working copy)
@@ -1,6 +1,5 @@
-
-bindir = ${exec_prefix}/ib/bin
-libdir = ${exec_prefix}/ib/lib
+bindir = @bindir@
+libdir = @libdir@
 
 INCLUDES = -I$(srcdir)/../../libibcommon/include/infiniband \
 	   -I$(srcdir)/../../libibumad/include/infiniband



From mst at mellanox.co.il  Tue Feb  8 02:51:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 8 Feb 2005 12:51:12 +0200
Subject: [openib-general] [PATCH] ipoib: dont lock tx on completion
Message-ID: <20050208105112.GM2141@mellanox.co.il>

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

By testing again, and waking the net queue if completions arrived,
we can avoid taking the tx lock on send completion in ip over ib,
reducing the lock contention.

Index: ulp/ipoib/ipoib_ib.c
===================================================================
--- ulp/ipoib/ipoib_ib.c	(revision 1747)
+++ ulp/ipoib/ipoib_ib.c	(working copy)
@@ -225,7 +225,6 @@ static void ipoib_ib_handle_wc(struct ne
 
 	} else {
 		struct ipoib_buf *tx_req;
-		unsigned long flags;
 
 		if (wr_id >= IPOIB_TX_RING_SIZE) {
 			ipoib_warn(priv, "completion event with wrid %d (> %d)\n",
@@ -247,12 +246,10 @@ static void ipoib_ib_handle_wc(struct ne
 
 		dev_kfree_skb_any(tx_req->skb);
 
-		spin_lock_irqsave(&priv->tx_lock, flags);
 		++priv->tx_tail;
 		if (netif_queue_stopped(dev) &&
 		    priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2)
 			netif_wake_queue(dev);
-		spin_unlock_irqrestore(&priv->tx_lock, flags);
 
 		if (wc->status != IB_WC_SUCCESS &&
 		    wc->status != IB_WC_WR_FLUSH_ERR)
@@ -341,6 +338,14 @@ void ipoib_send(struct net_device *dev, 
 		if (priv->tx_head - priv->tx_tail == IPOIB_TX_RING_SIZE) {
 			ipoib_dbg(priv, "TX ring full, stopping kernel net queue\n");
 			netif_stop_queue(dev);
+
+			/* Make sure the comletion routine will see
+			   the interface stopped */
+			wmb();
+
+			/* Meanwhile, a completion may have updated tx_tail. */
+			if (priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2)
+				netif_wake_queue(dev);
 		}
 	}
 }

-- 
MST - Michael S. Tsirkin


From tduffy at sun.com  Tue Feb  8 07:09:47 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 08 Feb 2005 07:09:47 -0800
Subject: [openib-general] [PATCH] teach ifconfig about ib [WAS: Latest
	IPoIB Bringup Questions]
In-Reply-To: <20050208133725.GA10155@aon.at>
References: <1098985903.17991.74.camel@hpc-1>
	<1107811374.6917.6.camel@duffman>  <20050208133725.GA10155@aon.at>
Message-ID: <1107875387.19201.2.camel@duffman>

[ Please keep discussion on list ]

On Tue, 2005-02-08 at 14:37 +0100, Bernhard Fischer wrote:
> Sorry to address you..
> 
> Why don't you place the DEBUG #ifdefs like below thus saving a couple of
> IMHO unneeded instructions?
> Just curious, but TIA for clarification.

That block of code is pretty much copied directly from ether.c.  I don't
see why I couldn't add in what you are have suggested, but I am not sure
if it really that big a deal.

-tduffy

> >diff -Nur net-tools-1.60/lib/ib.c net-tools-1.60-ib/lib/ib.c
> >--- net-tools-1.60/lib/ib.c	1969-12-31 16:00:00.000000000 -0800
> >+++ net-tools-1.60-ib/lib/ib.c	2005-02-07 12:55:04.635559244 -0800
> [snip]
> >+static int in_ib(char *bufp, struct sockaddr *sap)
> [snip]
> >+    while ((*bufp != '\0') && (i < INFINIBAND_ALEN)) {
> >+	val = 0;
> >+	c = *bufp++;
> >+	if (isdigit(c))
> >+	    val = c - '0';
> >+	else if (c >= 'a' && c <= 'f')
> >+	    val = c - 'a' + 10;
> >+	else if (c >= 'A' && c <= 'F')
> >+	    val = c - 'A' + 10;
> >+	else {
> >+#ifdef DEBUG
> >+	    fprintf(stderr, _("in_ib(%s): invalid infiniband address!\n"), orig);
> >+#endif
> >+	    errno = EINVAL;
> >+	    return (-1);
> >+	}
> >+	val <<= 4;
> >+	c = *bufp;
> >+	if (isdigit(c))
> >+	    val |= c - '0';
> >+	else if (c >= 'a' && c <= 'f')
> >+	    val |= c - 'a' + 10;
> >+	else if (c >= 'A' && c <= 'F')
> >+	    val |= c - 'A' + 10;
> >+	else if (c == ':' || c == 0)
> >+	    val >>= 4;
> >+	else {
> >+#ifdef DEBUG
> >+	    fprintf(stderr, _("in_ib(%s): invalid infiniband address!\n"), orig);
> >+#endif
> >+	    errno = EINVAL;
> >+	    return (-1);
> >+	}
> >+	if (c != 0)
> >+	    bufp++;
> >+	*ptr++ = (unsigned char) (val & 0377);
> >+	i++;
> >+
> >+	/* We might get a semicolon here - not required. */
> >+	if (*bufp == ':') {
> +>+#ifdef DEBUG
> >+	    if (i == INFINIBAND_ALEN) {
> ->+#ifdef DEBUG
> >+		fprintf(stderr, _("in_ib(%s): trailing : ignored!\n"),
> >+			orig)
> ->+#endif
> >+		    ;		/* nothing */
> >+	    }
> +>+#endif
> >+	    bufp++;
> >+	}
> >+    }
> >+
> +>+#ifdef DEBUG
> >+    /* That's it.  Any trailing junk? */
> >+    if ((i == INFINIBAND_ALEN) && (*bufp != '\0')) {
> ->+#ifdef DEBUG
> >+	fprintf(stderr, _("in_ib(%s): trailing junk!\n"), orig);
> >+	errno = EINVAL;
> >+	return (-1);
> ->+#endif
> >+    }
> ->+#ifdef DEBUG
> >+    fprintf(stderr, "in_ib(%s): %s\n", orig, pr_ib(sap->sa_data));
> >+#endif
> >+
> >+    return (0);
> >+}

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050208/a2d7f9ba/attachment.sig>

From halr at voltaire.com  Tue Feb  8 09:59:37 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: Tue, 8 Feb 2005 19:59:37 +0200
Subject: [openib-general] RE: [PATCH] teach ifconfig about ib [WAS: Latest
	IPoIB BringupQuestions]
Message-ID: <5CE025EE7D88BA4599A2C8FEFCF226F5175B53@taurus.voltaire.com>

[Tom Duffy wrote:]

On Thu, 2004-10-28 at 13:51 -0400, Hal Rosenstock wrote:
> Should we teach ifconfig to display Link Encap: INFINIBAND ?

Still has the problem of truncating the address to the first 14 bytes.

diff -Nur net-tools-1.60/config.in net-tools-1.60-ib/config.in
--- net-tools-1.60/config.in    2000-05-21 07:32:12.000000000 -0700
+++ net-tools-1.60-ib/config.in 2005-02-07 10:45:14.108286619 -0800
@@ -82,6 +82,7 @@
 bool '(Cisco)-HDLC/LAPB support' HAVE_HWHDLCLAPB n
 bool 'IrDA support' HAVE_HWIRDA y
 bool 'Econet hardware support' HAVE_HWEC n
+bool 'InfiniBand hardware support' HAVE_HWIB y
...
 
Cool. Any idea if the "same thing" will be done for the arp command too ?
If my memory serves me correctly, I believe it had a similar problem. 
 
Thanks.
 
-- Hal
 


From tduffy at sun.com  Tue Feb  8 10:18:52 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 08 Feb 2005 10:18:52 -0800
Subject: [openib-general] RE: [PATCH] teach ifconfig about ib [WAS: Latest
	IPoIB BringupQuestions]
In-Reply-To: <5CE025EE7D88BA4599A2C8FEFCF226F5175B53@taurus.voltaire.com>
References: <5CE025EE7D88BA4599A2C8FEFCF226F5175B53@taurus.voltaire.com>
Message-ID: <1107886732.13786.6.camel@duffman>

On Tue, 2005-02-08 at 19:59 +0200, Hal Rosenstock wrote: 
> Cool. Any idea if the "same thing" will be done for the arp command too ?
> If my memory serves me correctly, I believe it had a similar problem. 

This same patch fixes arp as well.

Before:

[root at flopteron net-tools-1.60-ib2]# arp
Address                  HWtype  HWaddress           Flags Mask            Ifac
10.6.98.1                ether   00:00:0C:07:AC:00   C                     eth0
192.168.0.62             ether   00:00:00:14:FE:80:00C                     ib0
sins-nap.SFBay.Sun.COM   ether   00:03:47:22:85:FC   C                     eth0

After:

[root at flopteron net-tools-1.60-ib2]# ./arp
Address                  HWtype  HWaddress           Flags Mask            Ifac
10.6.98.1                ether   00:00:0C:07:AC:00   C                     eth0
192.168.0.62             infiniba00:00:00:14:FE:80:00C                     ib0
sins-nap.SFBay.Sun.COM   ether   00:03:47:22:85:FC   C                     eth0

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050208/3ffca80f/attachment.sig>

From mashirle at us.ibm.com  Tue Feb  8 13:45:12 2005
From: mashirle at us.ibm.com (Shirley Ma)
Date: Tue, 8 Feb 2005 13:45:12 -0800
Subject: [openib-general] [PATCH]memory leak in ipoib_ib_post_receive()
Message-ID: <200502081345.12755.mashirle@us.ibm.com>

Here is the patch.

diff 
-urN /home/mshirley/iba/gen2/trunk/src/linux-kernel/infiniband/ulp/ipoib/ipoib_ib.c 
infiniband-patch/ulp/ipoib/ipoib_ib.c
--- /home/mshirley/iba/gen2/trunk/src/linux-kernel/infiniband/ulp/ipoib/ipoib_ib.c 
2005-02-04 16:05:13.000000000 -0800
+++ infiniband-patch/ulp/ipoib/ipoib_ib.c 2005-02-08 11:46:18.516256344 -0800
@@ -137,6 +137,7 @@
  if (ret) {
   ipoib_warn(priv, "ipoib_ib_receive failed for buf %d (%d)\n",
       id, ret);
+  dev_kfree_skb_any(skb);
   priv->rx_ring[id].skb = NULL;
  }
 
-- 
Thanks
Shirley Ma
IBM Linux Technology Center


From tduffy at sun.com  Tue Feb  8 13:46:27 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 08 Feb 2005 13:46:27 -0800
Subject: [openib-general] [Fwd: [Bug 80631] [RFE] add infiniband devices to
	udev device rules]
Message-ID: <1107899187.31874.4.camel@duffman>

What should the default permissions be for /dev/infiniband/issm*
or /dev/infiniband/umad*?

I am guessing root:root 660 are fine...

BTW, Gentoo and Debian (both community driven distros) have been the
*most* receptive to supporting InfiniBand.

-tduffy
-------------- next part --------------
An embedded message was scrubbed...
From: bugzilla-daemon at gentoo.org
Subject: [Bug 80631] [RFE] add infiniband devices to udev device rules
Date: Tue, 08 Feb 2005 21:37:50 +0000
Size: 2994
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050208/b47f02f6/attachment.mht>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050208/b47f02f6/attachment.sig>

From mst at mellanox.co.il  Wed Feb  9 03:21:40 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 9 Feb 2005 13:21:40 +0200
Subject: [openib-general] mthca: post_receive
Message-ID: <20050209112140.GE18350@mellanox.co.il>


                if (qp->rq.cur + nreq >= qp->rq.max) {
                        mthca_err(dev, "RQ %06x full\n", qp->qpn);
                        err = -ENOMEM;
                        *bad_wr = wr;
                        goto out;
                }

Shouldnt that be:

                if (qp->rq.cur + nreq > qp->rq.max)

If I want to allow rq.max WQEs - should not I be allowed to post
that much?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  9 06:58:35 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 9 Feb 2005 16:58:35 +0200
Subject: [openib-general] remove selective signalling flags for ip over ib
Message-ID: <20050209145835.GK18350@mellanox.co.il>

Since currently the qp policy in ip over ib is SIGNAL_ALL,
we dont need to set signalling flags in each wqe.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/ipoib/ipoib_ib.c
===================================================================
--- ulp/ipoib/ipoib_ib.c	(revision 1758)
+++ ulp/ipoib/ipoib_ib.c	(working copy)
@@ -105,7 +105,6 @@ static inline int ipoib_ib_receive(struc
 		.wr_id 	    = wr_id | IPOIB_OP_RECV,
 		.sg_list    = &list,
 		.num_sge    = 1,
-		.recv_flags = IB_RECV_SIGNALED
 	};
 	struct ib_recv_wr *bad_wr;
 

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb  9 07:03:10 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 9 Feb 2005 17:03:10 +0200
Subject: [openib-general] ib_device_cap_flags
Message-ID: <20050209150310.GL18350@mellanox.co.il>

Hi!
enum ib_device_cap_flags from ib_verbs.h does not seem to be used anywhere.
Specifically there does not seem to exist a way to find out the device
capabilities.

# grep -rIi ib_device_cap_flags .
./include/ib_verbs.h:enum ib_device_cap_flags {

???

MST

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Wed Feb  9 08:42:13 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 08:42:13 -0800
Subject: [openib-general] [PATCH] ipoib: dont lock tx on completion
In-Reply-To: <20050208105112.GM2141@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 8 Feb 2005 12:51:12 +0200")
References: <20050208105112.GM2141@mellanox.co.il>
Message-ID: <527jlhfzx6.fsf@topspin.com>

    Michael> By testing again, and waking the net queue if completions
    Michael> arrived, we can avoid taking the tx lock on send
    Michael> completion in ip over ib, reducing the lock contention.

I really like the clever way of retesting after putting the queue to
sleep.  However I can't convince myself that some test or other will
be wrong because of stale values of tx_head or tx_tail, and I'm always
scared of code that requires multiple barriers and so on.  So I'm
going to defer applying this until I have time to think hard and make
sure there aren't any subtle races.

 - R.



From roland at topspin.com  Wed Feb  9 08:49:30 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 08:49:30 -0800
Subject: [openib-general] [PATCH]memory leak in ipoib_ib_post_receive()
In-Reply-To: <200502081345.12755.mashirle@us.ibm.com> (Shirley Ma's message
	of "Tue, 8 Feb 2005 13:45:12 -0800")
References: <200502081345.12755.mashirle@us.ibm.com>
Message-ID: <523bw5fzl1.fsf@topspin.com>

    Shirley> Here is the patch.

Thanks, good catch.  Unfortunately your patch got whitespace-mangled
somewhere along the way.  Also I think we also need to undo the DMA
mapping in addition to freeing the skb, like this:

Index: infiniband/ulp/ipoib/ipoib_ib.c
===================================================================
--- infiniband/ulp/ipoib/ipoib_ib.c	(revision 1758)
+++ infiniband/ulp/ipoib/ipoib_ib.c	(working copy)
@@ -137,6 +137,9 @@ static int ipoib_ib_post_receive(struct 
 	if (ret) {
 		ipoib_warn(priv, "ipoib_ib_receive failed for buf %d (%d)\n",
 			   id, ret);
+		dma_unmap_single(priv->ca->dma_device, addr,
+				 IPOIB_BUF_SIZE, DMA_FROM_DEVICE);
+		dev_kfree_skb_any(skb);
 		priv->rx_ring[id].skb = NULL;
 	}
 



From roland at topspin.com  Wed Feb  9 08:51:58 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 08:51:58 -0800
Subject: [openib-general] [Fwd: [Bug 80631] [RFE] add infiniband devices
	to udev device rules]
In-Reply-To: <1107899187.31874.4.camel@duffman> (Tom Duffy's message of
	"Tue, 08 Feb 2005 13:46:27 -0800")
References: <1107899187.31874.4.camel@duffman>
Message-ID: <52y8dxekwh.fsf@topspin.com>

    Tom> What should the default permissions be for
    Tom> /dev/infiniband/issm* or /dev/infiniband/umad*?

    Tom> I am guessing root:root 660 are fine...

For now I think root:root 0660 is fine (although not much different
from root and 0600...).  Longer term someone may want to come up with
some best practices for groups/permissions so that it's not necessary
to run everything as root.  This is especially important for userspace
verbs, of course.

 - R.


From roland at topspin.com  Wed Feb  9 08:54:22 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 08:54:22 -0800
Subject: [openib-general] mthca: post_receive
In-Reply-To: <20050209112140.GE18350@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 9 Feb 2005 13:21:40 +0200")
References: <20050209112140.GE18350@mellanox.co.il>
Message-ID: <52u0oleksh.fsf@topspin.com>

    Michael> Shouldnt that be:

    Michael>                 if (qp->rq.cur + nreq > qp->rq.max)

    Michael> If I want to allow rq.max WQEs - should not I be allowed
    Michael> to post that much?

The code is correct but perhaps the name "nreq" could be improved.  If
you look at the code in context:

	for (nreq = 0; wr; ++nreq, wr = wr->next) {
		if (unlikely(qp->rq.cur + nreq >= qp->rq.max)) {

you can see that nreq is a 0-based index of the work request we're
working on.  So for example, if max is 3 and cur is 1 (so we have 2
slots open in our WQ), then the test will fail when trying to post the
third work request, which is correct.

 - R.


From roland at topspin.com  Wed Feb  9 08:56:50 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 08:56:50 -0800
Subject: [openib-general] ib_device_cap_flags
In-Reply-To: <20050209150310.GL18350@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 9 Feb 2005 17:03:10 +0200")
References: <20050209150310.GL18350@mellanox.co.il>
Message-ID: <52psz9ekod.fsf@topspin.com>

    Michael> Hi!  enum ib_device_cap_flags from ib_verbs.h does not
    Michael> seem to be used anywhere.  Specifically there does not
    Michael> seem to exist a way to find out the device capabilities.

    Michael> # grep -rIi ib_device_cap_flags .
    Michael> ./include/ib_verbs.h:enum ib_device_cap_flags {

This is partially because of Sean's somewhat unfortunate habit of
defining enums but then just using "int" everywhere in the API.  The
way to get the capabilities is to use ib_query_device -- the flags
will be returned in the device_cap_flags member of struct
ib_device_attr.

However with that said, mthca doesn't implement setting that field and
in fact, looking at the code, will return garbage in that field, so it
can't really be used yet.  That's mostly because no one has cared to
test any of those capabilities yet.

 - R.


From tduffy at sun.com  Wed Feb  9 08:59:48 2005
From: tduffy at sun.com (Tom Duffy)
Date: Wed, 09 Feb 2005 08:59:48 -0800
Subject: [openib-general] [Fwd: [Bug 147396] [RFE] teach ifconfig about
	infiniband]
Message-ID: <1107968388.2620.1.camel@duffman>

Also, I've opened bug 147590 against glibc-kernheaders.
-------------- next part --------------
An embedded message was scrubbed...
From: bugzilla at redhat.com
Subject: [Bug 147396] [RFE] teach ifconfig about infiniband
Date: Wed, 09 Feb 2005 04:20:39 -0500
Size: 3704
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050209/3e018746/attachment.mht>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050209/3e018746/attachment.sig>

From roland at topspin.com  Wed Feb  9 09:12:03 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 09:12:03 -0800
Subject: [openib-general] [PATCH/iproute2] Add support for IP-over-InfiniBand
In-Reply-To: <20050207110042.166752a3@dxpl.pdx.osdl.net> (Stephen
	Hemminger's message of "Mon, 7 Feb 2005 11:00:42 -0800")
References: <20050207110042.166752a3@dxpl.pdx.osdl.net>
Message-ID: <527jlhejz0.fsf@topspin.com>

This trivial patch adds support for displaying InfiniBand link
addresses.  With the patch, output of "ip addr" changes from

    15: ib0: <BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast qlen 128
        link/[32] 00:00:04:04:fe:80:00:00:00:00:00:00:00:02:c9:01:07:8c:e4:61 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

to

    15: ib0: <BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast qlen 128
        link/infiniband 00:00:04:04:fe:80:00:00:00:00:00:00:00:02:c9:01:07:8c:e4:61 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

(Recent libc headers have ARPHRD_INFINIBAND defined -- for example,
they are in libc6-dev on my Debian Sarge system)

Thanks,
  Roland

--- iproute2-050207.orig/lib/ll_types.c	2005-02-07 10:37:20.000000000 -0800
+++ iproute2-050207/lib/ll_types.c	2005-02-08 21:40:10.000000000 -0800
@@ -55,6 +55,9 @@
 #ifdef ARPHRD_IEEE1394
 __PF(IEEE1394,ieee1394)
 #endif
+#ifdef ARPHRD_INFINIBAND
+__PF(INFINIBAND,infiniband)
+#endif
 
 __PF(SLIP,slip)
 __PF(CSLIP,cslip)


From mst at mellanox.co.il  Wed Feb  9 09:26:35 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 9 Feb 2005 19:26:35 +0200
Subject: [openib-general] Re: [PATCH] ipoib: dont lock tx on completion
In-Reply-To: <527jlhfzx6.fsf@topspin.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
Message-ID: <20050209172635.GQ18350@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] ipoib: dont lock tx on completion
> 
>     Michael> By testing again, and waking the net queue if completions
>     Michael> arrived, we can avoid taking the tx lock on send
>     Michael> completion in ip over ib, reducing the lock contention.
> 
> I really like the clever way of retesting after putting the queue to
> sleep.  However I can't convince myself that some test or other will
> be wrong because of stale values of tx_head or tx_tail,

Not sure what is meant here by "stale values".
This is the usual consumer/producer approach ...

> and I'm always
> scared of code that requires multiple barriers and so on.  So I'm
> going to defer applying this until I have time to think hard and make
> sure there aren't any subtle races.
> 
>  - R.
> 

Okay.
Also note that the exact value of tx_tail/tx_head should not practially matter,
since to change the state you have to go from half-full TX queue to full
TX queue, or back.
There will be a huge number barriers, locks etc done by that time.

MST

-- 
MST - Michael S. Tsirkin


From jlentini at netapp.com  Wed Feb  9 09:30:09 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 9 Feb 2005 12:30:09 -0500 (EST)
Subject: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <Pine.LNX.4.21.0502091144550.20317-100000@orca-linux1.nane.netapp.com>


DAPL BOF Meeting Minutes
2/8/2005

- Discussed which version of DAPL (1.1 or 1.2) should initially be
  submitted to OpenIB. Concluded that 1.2 should be submitted.

- DAPL License: Discussed the terms under which the DAPL OpenIB code
  should be licensed. Concluded that DAPL should bear the same
  licenses, GPL and BSD, as other the other components in the OpenIB  
  tree. NetApp raised the issue of how a GPL only patch to the Linux  
  kernel would be handled. The group concluded that this is an issue
  for all of the OpenIB code base and not specific to DAPL.

- uDAPL Platform independent features can be left intact. Only kernel 
  components need to be Linux specific.

- Group requested a link to the DAPL kernel code. The code is located
  on the DAPL SourceForge project web site. See:

  http://sourceforge.net/projects/dapl

  The CVS repository is browsable via the web. See:

  http://cvs.sourceforge.net/viewcvs.py/dapl/

  The kernel components are located in the dat and dapl 
  directories. 

  The dat directory contains the DAT registry. All 
  files and subdirectories in 

  dat/common
  dat/include
  dat/kdat

  are part of the kernel component.

  The dapl directory contains the reference provider library. All 
  of the files and subdirectories in 

  dapl/common
  dapl/include
  dapl/kdapl
  
  are part of the kernel component. In addition, a new directory 
  called dapl/openib_gen_two will be added and become part of the 
  kernel component.

- Tasks needed to integrate DAPL into OpenIB

  - Add OpenIB Gen 2 support for both uDAPL and kDAPL
  - Determine if a uDAPL kernel component is necessary to support 
    shared memory registration, ATS, etc. on OpenIB gen 2. Add 
    support if needed.
  - Cleanup kDAPL for Linux: need to remove platform independent 
    layer and adhere to Linux kernel coding style
  - Update kDAPL build procedures to use autotools like the other 
    OpenIB kernel components.

  Several individuals agreed to help, but no commitments were made.



From xma at us.ibm.com  Wed Feb  9 09:43:30 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Wed, 9 Feb 2005 09:43:30 -0800
Subject: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <Pine.LNX.4.21.0502091144550.20317-100000@orca-linux1.nane.netapp.com>
Message-ID: <OF26C5CB90.F1647006-ON87256FA3.0060BF4A-88256FA3.00615E79@us.ibm.com>

Thank you so much. Hope more OpenIB Workshop sections will be posted here. 
It's really useful for people not in the workshop. Do you mind to share 
your presentation here? Or a new documentation directory for technical 
presentations could be generated somewhere?

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050209/c38b4daf/attachment.html>

From robert.j.woodruff at intel.com  Wed Feb  9 09:47:25 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Wed, 9 Feb 2005 09:47:25 -0800
Subject: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <1AC79F16F5C5284499BB9591B33D6F0003866213@orsmsx408>


> Thank you so much. Hope more OpenIB Workshop sections will be posted
here. It's 
> really useful for people not i  n   the workshop. Do you mind to share
your 
>presentation here? Or a new documentation directory for technical
presentations could >be generated somewhere? 
>Phone(Fax): (503) 578-7638

Matt plans to post the presentations on the openib.org website within
the 
next couple of days.

woody




From mst at mellanox.co.il  Wed Feb  9 10:41:32 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 9 Feb 2005 20:41:32 +0200
Subject: [openib-general] [PATCH (updated)] ipoib: dont lock tx on completion
In-Reply-To: <527jlhfzx6.fsf@topspin.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
Message-ID: <20050209184132.GB19608@mellanox.co.il>


Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] ipoib: dont lock tx on completion
> 
>     Michael> By testing again, and waking the net queue if completions
>     Michael> arrived, we can avoid taking the tx lock on send
>     Michael> completion in ip over ib, reducing the lock contention.
> 
> I really like the clever way of retesting after putting the queue to
> sleep.  However I can't convince myself that some test or other will
> be wrong because of stale values of tx_head or tx_tail, and I'm always
> scared of code that requires multiple barriers and so on.  So I'm
> going to defer applying this until I have time to think hard and make
> sure there aren't any subtle races.
> 
>  - R.
> 

Hmm, stared at it some more and I think I see what you mean.
Here's another attempt:

By testing again, and waking the net queue if completions
arrived, we can avoid taking the tx lock on send
completion in ip over ib, reducing the lock contention.

An exception is when the interface is locked: in that case
we have to look at the head and tail values, so its easier
to take the lock than worry about smp issues.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/ipoib/ipoib_ib.c
===================================================================
--- ulp/ipoib/ipoib_ib.c	(revision 1747)
+++ ulp/ipoib/ipoib_ib.c	(working copy)
@@ -225,7 +225,6 @@ static void ipoib_ib_handle_wc(struct ne
 
 	} else {
 		struct ipoib_buf *tx_req;
-		unsigned long flags;
 
 		if (wr_id >= IPOIB_TX_RING_SIZE) {
 			ipoib_warn(priv, "completion event with wrid %d (> %d)\n",
@@ -247,12 +246,19 @@ static void ipoib_ib_handle_wc(struct ne
 
 		dev_kfree_skb_any(tx_req->skb);
 
-		spin_lock_irqsave(&priv->tx_lock, flags);
 		++priv->tx_tail;
-		if (netif_queue_stopped(dev) &&
-		    priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2)
-			netif_wake_queue(dev);
-		spin_unlock_irqrestore(&priv->tx_lock, flags);
+
+		/* Make sure the send routine will see
+		   the tail updated. */
+		wmb();
+
+		if (netif_queue_stopped(dev)) {
+			unsigned long flags;
+			spin_lock_irqsave(&priv->tx_lock, flags);
+			if (priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2)
+				netif_wake_queue(dev);
+			spin_unlock_irqrestore(&priv->tx_lock, flags);
+		}
 
 		if (wc->status != IB_WC_SUCCESS &&
 		    wc->status != IB_WC_WR_FLUSH_ERR)
@@ -341,6 +347,14 @@ void ipoib_send(struct net_device *dev, 
 		if (priv->tx_head - priv->tx_tail == IPOIB_TX_RING_SIZE) {
 			ipoib_dbg(priv, "TX ring full, stopping kernel net queue\n");
 			netif_stop_queue(dev);
+
+			/* Make sure the comletion routine will see
+			   the interface stopped */
+			wmb();
+
+			/* Meanwhile, a completion may have updated tx_tail. */
+			if (priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2)
+				netif_wake_queue(dev);
 		}
 	}
 }

-- 
MST - Michael S. Tsirkin


From mashirle at us.ibm.com  Wed Feb  9 11:47:57 2005
From: mashirle at us.ibm.com (Shirley Ma)
Date: Wed, 9 Feb 2005 11:47:57 -0800
Subject: [openib-general] [PATCH]rmmod could crash in tx_ring[].skb &
	rx_ring[].skb.
Message-ID: <200502091147.57206.mashirle@us.ibm.com>

I am going through the IPoIB code. I found rmmod could cause 
kernel crash. It's easy to fix tx_ring with tx_lock. rx_ring[].skb has 
the same problem, inducing another lock in ipoib_ib_handle_wc()
just for rmmod is too costly. Any other suggestion? 

Here is the patch for tx_ring[].skb.

---------- 
Thanks
Shirley Ma
IBM Linux Technology Center
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rmmod.tx_ring
Type: text/x-diff
Size: 1014 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050209/fd9ed5aa/attachment.diff>

From jice at pantasys.com  Wed Feb  9 12:39:53 2005
From: jice at pantasys.com (Jean-Christophe Hugly)
Date: Wed, 09 Feb 2005 12:39:53 -0800
Subject: [openib-general] RFC on SDP checkin
Message-ID: <1107981593.18825.48.camel@jhugly.pantasys.com>

[where to put gen2 SDP]

If you're still waiting for comments on that, here's my vote:
I'd prefer #3 as well. Less work for me (a consumer of your great work)
too :-). I can just update and use.

Thanks !

J-C




From roland at topspin.com  Wed Feb  9 14:08:09 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 14:08:09 -0800
Subject: [openib-general] [PATCH]rmmod could crash in tx_ring[].skb &
	rx_ring[].skb.
In-Reply-To: <200502091147.57206.mashirle@us.ibm.com> (Shirley Ma's message
	of "Wed, 9 Feb 2005 11:47:57 -0800")
References: <200502091147.57206.mashirle@us.ibm.com>
Message-ID: <52ll9xcrp2.fsf@topspin.com>

    Shirley> I am going through the IPoIB code. I found rmmod could
    Shirley> cause kernel crash. It's easy to fix tx_ring with
    Shirley> tx_lock. rx_ring[].skb has the same problem, inducing
    Shirley> another lock in ipoib_ib_handle_wc() just for rmmod is
    Shirley> too costly. Any other suggestion?

Sorry to be so dense, but could you explain the race you're protecting
against?  The code you're putting the lock around can only run 5
seconds after the interface has been shut down, so we're assuming the
hardware is wedged.  But in any case the net queue is stopped so no
new transmits should be able to be posted.

 - R.


From roland.list at gmail.com  Wed Feb  9 13:57:18 2005
From: roland.list at gmail.com (Roland Dreier)
Date: Wed, 9 Feb 2005 13:57:18 -0800
Subject: [openib-general] has anyone tried MSI-X on Opteron?
In-Reply-To: <52u0pkmuls.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED6064BEDB5@mtlex01.yok.mtl.com>
	<523bx5s5p1.fsf@topspin.com> <m3mzvcdcy6.fsf@maxwell.lnxi.com>
	<52u0pkmuls.fsf@topspin.com>
Message-ID: <f8ca0a150502091357b61fedf@mail.gmail.com>

> If anyone has an nForce4 mobo, I would be _very_ interested to see the
> output of lspci -vvxxx, especially using lspci from the latest
> pciutils 2.1.99 (available from /pub/software/utils/pciutils/alpha on
> ftp.kernel.org).

For what it's worth at this point, I finally got an nForce4 mobo.  The
nForce4 PCI bridge does support MSI:

00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size 08
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-0000000000000000
	Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [40] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [58] HyperTransport: MSI Mapping
	Capabilities: [80] Express Root Port (Slot+) IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <512ns, L1 <4us
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
		Link: Supported Speed 2.5Gb/s, Width x2, ASPM L0s, Port 3
		Link: Latency L0s <512ns, L1 <4us
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x4 0x1041
		Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug- Surpise-
		Slot: Number 8, PowerLimit 25.000000
		Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
		Slot: AttnInd Off, PwrInd On, Power-
		Root: Correctable- Non-Fatal- Fatal- PME-

and in fact the current Linux MSI-X code works fine.  On that system
with a PCIe HCA, I have the following in /proc/interrupts:

           CPU0
  0:    1463994    IO-APIC-edge  timer
  7:          2    IO-APIC-edge  parport0
  8:          0    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 15:      12942    IO-APIC-edge  ide1
 50:          0       PCI-MSI-X  ib_mthca (async)
 58:        321       PCI-MSI-X  ib_mthca (cmd)
 66:          0   IO-APIC-level  SysKonnect SK-98xx
177:        525   IO-APIC-level  libata, NVidia CK804
185:     188960   IO-APIC-level  libata, eth0
193:      90623   IO-APIC-level  nvidia
201:      27005   IO-APIC-level  ohci_hcd
209:          2   IO-APIC-level  ehci_hcd
217:          3   IO-APIC-level  ohci1394
233:          0       PCI-MSI-X  ib_mthca (comp)
NMI:        173
LOC:    1463664
ERR:          0
MIS:          0

(nothing connected to the HCA yet, so no interrupts other
than command completions)

 - R.


From roland.list at gmail.com  Wed Feb  9 13:57:18 2005
From: roland.list at gmail.com (Roland Dreier)
Date: Wed, 9 Feb 2005 13:57:18 -0800
Subject: [openib-general] has anyone tried MSI-X on Opteron?
In-Reply-To: <52u0pkmuls.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED6064BEDB5@mtlex01.yok.mtl.com>
	<523bx5s5p1.fsf@topspin.com> <m3mzvcdcy6.fsf@maxwell.lnxi.com>
	<52u0pkmuls.fsf@topspin.com>
Message-ID: <f8ca0a150502091357b61fedf@mail.gmail.com>

> If anyone has an nForce4 mobo, I would be _very_ interested to see the
> output of lspci -vvxxx, especially using lspci from the latest
> pciutils 2.1.99 (available from /pub/software/utils/pciutils/alpha on
> ftp.kernel.org).

For what it's worth at this point, I finally got an nForce4 mobo.  The
nForce4 PCI bridge does support MSI:

00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size 08
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-0000000000000000
	Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [40] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [58] HyperTransport: MSI Mapping
	Capabilities: [80] Express Root Port (Slot+) IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <512ns, L1 <4us
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
		Link: Supported Speed 2.5Gb/s, Width x2, ASPM L0s, Port 3
		Link: Latency L0s <512ns, L1 <4us
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x4 0x1041
		Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug- Surpise-
		Slot: Number 8, PowerLimit 25.000000
		Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
		Slot: AttnInd Off, PwrInd On, Power-
		Root: Correctable- Non-Fatal- Fatal- PME-

and in fact the current Linux MSI-X code works fine.  On that system
with a PCIe HCA, I have the following in /proc/interrupts:

           CPU0
  0:    1463994    IO-APIC-edge  timer
  7:          2    IO-APIC-edge  parport0
  8:          0    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 15:      12942    IO-APIC-edge  ide1
 50:          0       PCI-MSI-X  ib_mthca (async)
 58:        321       PCI-MSI-X  ib_mthca (cmd)
 66:          0   IO-APIC-level  SysKonnect SK-98xx
177:        525   IO-APIC-level  libata, NVidia CK804
185:     188960   IO-APIC-level  libata, eth0
193:      90623   IO-APIC-level  nvidia
201:      27005   IO-APIC-level  ohci_hcd
209:          2   IO-APIC-level  ehci_hcd
217:          3   IO-APIC-level  ohci1394
233:          0       PCI-MSI-X  ib_mthca (comp)
NMI:        173
LOC:    1463664
ERR:          0
MIS:          0

(nothing connected to the HCA yet, so no interrupts other
than command completions)

 - R.


From roland at topspin.com  Wed Feb  9 16:29:50 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 09 Feb 2005 16:29:50 -0800
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <20050209184132.GB19608@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 9 Feb 2005 20:41:32 +0200")
References: <20050208105112.GM2141@mellanox.co.il>
	<527jlhfzx6.fsf@topspin.com> <20050209184132.GB19608@mellanox.co.il>
Message-ID: <52d5v9cl4x.fsf@topspin.com>

    Michael> By testing again, and waking the net queue if completions
    Michael> arrived, we can avoid taking the tx lock on send
    Michael> completion in ip over ib, reducing the lock contention.

    Michael> An exception is when the interface is locked: in that
    Michael> case we have to look at the head and tail values, so its
    Michael> easier to take the lock than worry about smp issues.

Hmm, I still need to think about this for a while.  For example, I
can't convince myself easily that nothing bad can happen if we test
netif_queue_stopped() outside of a locked region.

 - R.


From mst at mellanox.co.il  Thu Feb 10 00:38:21 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 10:38:21 +0200
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <52d5v9cl4x.fsf@topspin.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
	<20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com>
Message-ID: <20050210083821.GV18350@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH (updated)] ipoib: dont lock tx on completion
> 
>     Michael> By testing again, and waking the net queue if completions
>     Michael> arrived, we can avoid taking the tx lock on send
>     Michael> completion in ip over ib, reducing the lock contention.
> 
>     Michael> An exception is when the interface is locked: in that
>     Michael> case we have to look at the head and tail values, so its
>     Michael> easier to take the lock than worry about smp issues.
> 
> Hmm, I still need to think about this for a while.  For example, I
> can't convince myself easily that nothing bad can happen if we test
> netif_queue_stopped() outside of a locked region.
> 
>  - R.
> 

We are now only looking at tx_tail and netif_queue_stopped
outside the lock (tx_head is only used inside the lock):

We have a write barrier in between, so
its guaranteed tx_tail is written before we test netif_queue_stopped.

On the other hand, the send code has a barrier after it stops the
interface, so this bit is written before we test
tx_tail the second time.

So, in the scenario where send stops the interface, either
the completion code will see it stopped, or the send code will see
tx tail updated.

OK?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb 10 02:15:19 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 12:15:19 +0200
Subject: [openib-general] Re: FW: summary of my understanding on our common
	work on openib.org
In-Reply-To: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
Message-ID: <20050210101519.GG18350@mellanox.co.il>



Quoting r. Tziporet Koren <tziporet at mellanox.co.il>:
> Subject: FW: summary of my understanding on our common work on openib.org
> 
> please do it
> 
> -----Original Message-----
> From: Roland Dreier [mailto:roland at topspin.com]
> Sent: Wednesday, February 09, 2005 9:00 AM
> To: Tziporet Koren
> Subject: Re: summary of my understanding on our common work on
> openib.org
> 
> 
>     Tziporet> sure.  Just one note - we are going to remove the
>     Tziporet> IB_SIGNAL_REQ_WR for receive QP.  I talked about it with
>     Tziporet> Sean and he agree its OK since its something we added in
>     Tziporet> Tavor which is not IB compliant and not going to support
>     Tziporet> in Arbel.  I asked Michael to prepare this patch so I
>     Tziporet> hope it will be ready this week.  Or - do you prefer to
>     Tziporet> do it yourself?
> 
> It would be great for Michael to prepare the patch.
> 
> Thanks,
>   Roland

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/ipoib/ipoib_verbs.c
===================================================================
--- ulp/ipoib/ipoib_verbs.c	(revision 1759)
+++ ulp/ipoib/ipoib_verbs.c	(working copy)
@@ -165,7 +165,6 @@ int ipoib_transport_dev_init(struct net_
 			.max_recv_sge = 1
 		},
 		.sq_sig_type = IB_SIGNAL_ALL_WR,
-		.rq_sig_type = IB_SIGNAL_ALL_WR,
 		.qp_type     = IB_QPT_UD
 	};
 
Index: ulp/ipoib/ipoib_ib.c
===================================================================
--- ulp/ipoib/ipoib_ib.c	(revision 1759)
+++ ulp/ipoib/ipoib_ib.c	(working copy)
@@ -105,7 +105,6 @@ static inline int ipoib_ib_receive(struc
 		.wr_id 	    = wr_id | IPOIB_OP_RECV,
 		.sg_list    = &list,
 		.num_sge    = 1,
-		.recv_flags = IB_RECV_SIGNALED
 	};
 	struct ib_recv_wr *bad_wr;
 
Index: include/ib_verbs.h
===================================================================
--- include/ib_verbs.h	(revision 1759)
+++ include/ib_verbs.h	(working copy)
@@ -73,7 +73,6 @@ enum ib_device_cap_flags {
 	IB_DEVICE_RC_RNR_NAK_GEN	= (1<<12),
 	IB_DEVICE_SRQ_RESIZE		= (1<<13),
 	IB_DEVICE_N_NOTIFY_CQ		= (1<<14),
-	IB_DEVICE_RQ_SIG_TYPE		= (1<<15)
 };
 
 enum ib_atomic_cap {
@@ -408,7 +407,6 @@ struct ib_qp_init_attr {
 	struct ib_srq	       *srq;
 	struct ib_qp_cap	cap;
 	enum ib_sig_type	sq_sig_type;
-	enum ib_sig_type	rq_sig_type;
 	enum ib_qp_type		qp_type;
 	u8			port_num; /* special QP types only */
 };
@@ -533,10 +531,6 @@ enum ib_send_flags {
 	IB_SEND_INLINE		= (1<<3)
 };
 
-enum ib_recv_flags {
-	IB_RECV_SIGNALED	= 1
-};
-
 struct ib_sge {
 	u64	addr;
 	u32	length;
@@ -579,7 +573,6 @@ struct ib_recv_wr {
 	u64			wr_id;
 	struct ib_sge	       *sg_list;
 	int			num_sge;
-	int			recv_flags;
 };
 
 enum ib_access_flags {
Index: core/mad.c
===================================================================
--- core/mad.c	(revision 1759)
+++ core/mad.c	(working copy)
@@ -2186,7 +2186,6 @@ static int ib_mad_post_receive_mads(stru
 	recv_wr.next = NULL;
 	recv_wr.sg_list = &sg_list;
 	recv_wr.num_sge = 1;
-	recv_wr.recv_flags = IB_RECV_SIGNALED;
 
 	do {
 		/* Allocate and map receive buffer */
@@ -2381,7 +2380,6 @@ static int create_mad_qp(struct ib_mad_q
 	qp_init_attr.send_cq = qp_info->port_priv->cq;
 	qp_init_attr.recv_cq = qp_info->port_priv->cq;
 	qp_init_attr.sq_sig_type = IB_SIGNAL_ALL_WR;
-	qp_init_attr.rq_sig_type = IB_SIGNAL_ALL_WR;
 	qp_init_attr.cap.max_send_wr = IB_MAD_QP_SEND_SIZE;
 	qp_init_attr.cap.max_recv_wr = IB_MAD_QP_RECV_SIZE;
 	qp_init_attr.cap.max_send_sge = IB_MAD_SEND_REQ_MAX_SG;
Index: hw/mthca/mthca_dev.h
===================================================================
--- hw/mthca/mthca_dev.h	(revision 1759)
+++ hw/mthca/mthca_dev.h	(working copy)
@@ -369,14 +369,12 @@ int mthca_alloc_qp(struct mthca_dev *dev
 		   struct mthca_cq *recv_cq,
 		   enum ib_qp_type type,
 		   enum ib_sig_type send_policy,
-		   enum ib_sig_type recv_policy,
 		   struct mthca_qp *qp);
 int mthca_alloc_sqp(struct mthca_dev *dev,
 		    struct mthca_pd *pd,
 		    struct mthca_cq *send_cq,
 		    struct mthca_cq *recv_cq,
 		    enum ib_sig_type send_policy,
-		    enum ib_sig_type recv_policy,
 		    int qpn,
 		    int port,
 		    struct mthca_sqp *sqp);
Index: hw/mthca/mthca_provider.c
===================================================================
--- hw/mthca/mthca_provider.c	(revision 1759)
+++ hw/mthca/mthca_provider.c	(working copy)
@@ -343,7 +343,7 @@ static struct ib_qp *mthca_create_qp(str
 				     to_mcq(init_attr->send_cq),
 				     to_mcq(init_attr->recv_cq),
 				     init_attr->qp_type, init_attr->sq_sig_type,
-				     init_attr->rq_sig_type, qp);
+				     qp);
 		qp->ibqp.qp_num = qp->qpn;
 		break;
 	}
@@ -364,7 +364,7 @@ static struct ib_qp *mthca_create_qp(str
 		err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd),
 				      to_mcq(init_attr->send_cq),
 				      to_mcq(init_attr->recv_cq),
-				      init_attr->sq_sig_type, init_attr->rq_sig_type,
+				      init_attr->sq_sig_type,
 				      qp->ibqp.qp_num, init_attr->port_num,
 				      to_msqp(qp));
 		break;
Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 1759)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -154,7 +154,6 @@ struct mthca_wq {
 	void *last;
 	int   max_gs;
 	int   wqe_shift;
-	enum ib_sig_type policy;
 };
 
 struct mthca_qp {
@@ -172,6 +171,7 @@ struct mthca_qp {
 
 	struct mthca_wq        rq;
 	struct mthca_wq        sq;
+	enum ib_sig_type       sq_policy;
 	int                    send_wqe_offset;
 
 	u64                   *wrid;
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 1759)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -690,7 +690,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 					     MTHCA_QP_BIT_SRE           |
 					     MTHCA_QP_BIT_SWE           |
 					     MTHCA_QP_BIT_SAE);
-	if (qp->sq.policy == IB_SIGNAL_ALL_WR)
+	if (qp->sq_policy == IB_SIGNAL_ALL_WR)
 		qp_context->params1 |= cpu_to_be32(MTHCA_QP_BIT_SSC);
 	if (attr_mask & IB_QP_RETRY_CNT) {
 		qp_context->params1 |= cpu_to_be32(attr->retry_cnt << 16);
@@ -778,8 +778,8 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 		qp->resp_depth = attr->max_rd_atomic;
 	}
 
-	if (qp->rq.policy == IB_SIGNAL_ALL_WR)
-		qp_context->params2 |= cpu_to_be32(MTHCA_QP_BIT_RSC);
+	qp_context->params2 |= cpu_to_be32(MTHCA_QP_BIT_RSC);
+
 	if (attr_mask & IB_QP_MIN_RNR_TIMER) {
 		qp_context->rnr_nextrecvpsn |= cpu_to_be32(attr->min_rnr_timer << 24);
 		qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_RNR_TIMEOUT);
@@ -977,7 +977,6 @@ static int mthca_alloc_qp_common(struct 
 				 struct mthca_cq *send_cq,
 				 struct mthca_cq *recv_cq,
 				 enum ib_sig_type send_policy,
-				 enum ib_sig_type recv_policy,
 				 struct mthca_qp *qp)
 {
 	int err;
@@ -987,8 +986,7 @@ static int mthca_alloc_qp_common(struct 
 	qp->state    	 = IB_QPS_RESET;
 	qp->atomic_rd_en = 0;
 	qp->resp_depth   = 0;
-	qp->sq.policy    = send_policy;
-	qp->rq.policy    = recv_policy;
+	qp->sq_policy    = send_policy;
 	qp->rq.cur       = 0;
 	qp->sq.cur       = 0;
 	qp->rq.next      = 0;
@@ -1008,7 +1006,6 @@ int mthca_alloc_qp(struct mthca_dev *dev
 		   struct mthca_cq *recv_cq,
 		   enum ib_qp_type type,
 		   enum ib_sig_type send_policy,
-		   enum ib_sig_type recv_policy,
 		   struct mthca_qp *qp)
 {
 	int err;
@@ -1025,7 +1022,7 @@ int mthca_alloc_qp(struct mthca_dev *dev
 		return -ENOMEM;
 
 	err = mthca_alloc_qp_common(dev, pd, send_cq, recv_cq,
-				    send_policy, recv_policy, qp);
+				    send_policy, qp);
 	if (err) {
 		mthca_free(&dev->qp_table.alloc, qp->qpn);
 		return err;
@@ -1044,7 +1041,6 @@ int mthca_alloc_sqp(struct mthca_dev *de
 		    struct mthca_cq *send_cq,
 		    struct mthca_cq *recv_cq,
 		    enum ib_sig_type send_policy,
-		    enum ib_sig_type recv_policy,
 		    int qpn,
 		    int port,
 		    struct mthca_sqp *sqp)
@@ -1073,8 +1069,7 @@ int mthca_alloc_sqp(struct mthca_dev *de
 	sqp->qp.transport = MLX;
 
 	err = mthca_alloc_qp_common(dev, pd, send_cq, recv_cq,
-				    send_policy, recv_policy,
-				    &sqp->qp);
+				    send_policy, &sqp->qp);
 	if (err)
 		goto err_out_free;
 
@@ -1495,9 +1490,7 @@ int mthca_post_receive(struct ib_qp *ibq
 		((struct mthca_next_seg *) wqe)->nda_op = 0;
 		((struct mthca_next_seg *) wqe)->ee_nds =
 			cpu_to_be32(MTHCA_NEXT_DBD);
-		((struct mthca_next_seg *) wqe)->flags =
-			(wr->recv_flags & IB_RECV_SIGNALED) ?
-			cpu_to_be32(MTHCA_NEXT_CQ_UPDATE) : 0;
+		((struct mthca_next_seg *) wqe)->flags = 0;
 
 		wqe += sizeof (struct mthca_next_seg);
 		size = sizeof (struct mthca_next_seg) / 16;
-- 
MST - Michael S. Tsirkin


From hch at lst.de  Thu Feb 10 05:48:35 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 10 Feb 2005 14:48:35 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
Message-ID: <20050210134835.GA5887@lst.de>

On Thu, Feb 10, 2005 at 03:44:56PM +0200, Yaron Haviv wrote:
> Christoph,
> 
> See the following email from OpenIB mailing list
> 
> Yaron

Umm, you IB guys had lots of strange ideas on what code was acceptable
until Roland stepped ahead and started the rewrite of the codebase.

Similarly even if we agree on a higher-level API not much of the
existing code would be of any use.

> -----Original Message-----
> From: openib-general-bounces at openib.org
> [mailto:openib-general-bounces at openib.org] On Behalf Of James Lentini
> Sent: Wednesday, February 09, 2005 7:30 PM
> To: openib-general at openib.org; dapl-devel at lists.sourceforge.net
> Subject: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
> 
> 
> DAPL BOF Meeting Minutes
> 2/8/2005
> 
> - Discussed which version of DAPL (1.1 or 1.2) should initially be
>   submitted to OpenIB. Concluded that 1.2 should be submitted.
> 
> - DAPL License: Discussed the terms under which the DAPL OpenIB code
>   should be licensed. Concluded that DAPL should bear the same
>   licenses, GPL and BSD, as other the other components in the OpenIB  
>   tree. NetApp raised the issue of how a GPL only patch to the Linux  
>   kernel would be handled. The group concluded that this is an issue
>   for all of the OpenIB code base and not specific to DAPL.
> 
> - uDAPL Platform independent features can be left intact. Only kernel 
>   components need to be Linux specific.
> 
> - Group requested a link to the DAPL kernel code. The code is located
>   on the DAPL SourceForge project web site. See:
> 
>   http://sourceforge.net/projects/dapl
> 
>   The CVS repository is browsable via the web. See:
> 
>   http://cvs.sourceforge.net/viewcvs.py/dapl/
> 
>   The kernel components are located in the dat and dapl 
>   directories. 
> 
>   The dat directory contains the DAT registry. All 
>   files and subdirectories in 
> 
>   dat/common
>   dat/include
>   dat/kdat
> 
>   are part of the kernel component.
> 
>   The dapl directory contains the reference provider library. All 
>   of the files and subdirectories in 
> 
>   dapl/common
>   dapl/include
>   dapl/kdapl
>   
>   are part of the kernel component. In addition, a new directory 
>   called dapl/openib_gen_two will be added and become part of the 
>   kernel component.
> 
> - Tasks needed to integrate DAPL into OpenIB
> 
>   - Add OpenIB Gen 2 support for both uDAPL and kDAPL
>   - Determine if a uDAPL kernel component is necessary to support 
>     shared memory registration, ATS, etc. on OpenIB gen 2. Add 
>     support if needed.
>   - Cleanup kDAPL for Linux: need to remove platform independent 
>     layer and adhere to Linux kernel coding style
>   - Update kDAPL build procedures to use autotools like the other 
>     OpenIB kernel components.
> 
>   Several individuals agreed to help, but no commitments were made.
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
---end quoted text---


From roland at topspin.com  Thu Feb 10 09:13:22 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 10 Feb 2005 09:13:22 -0800
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <20050210083821.GV18350@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 10 Feb 2005 10:38:21 +0200")
References: <20050208105112.GM2141@mellanox.co.il>
	<527jlhfzx6.fsf@topspin.com> <20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com> <20050210083821.GV18350@mellanox.co.il>
Message-ID: <52zmycbaod.fsf@topspin.com>

    Michael> We are now only looking at tx_tail and
    Michael> netif_queue_stopped outside the lock (tx_head is only
    Michael> used inside the lock):

Thanks, that helps.

    Michael> We have a write barrier in between, so its guaranteed
    Michael> tx_tail is written before we test netif_queue_stopped.

    Michael> On the other hand, the send code has a barrier after it
    Michael> stops the interface, so this bit is written before we
    Michael> test tx_tail the second time.

    Michael> So, in the scenario where send stops the interface,
    Michael> either the completion code will see it stopped, or the
    Michael> send code will see tx tail updated.

You may be right but I'm having a hard time convincing myself.  I
don't think just the write barriers are enough -- they just mean that
the CPU where the writes are being done will complete the writes in
order.  Without a read barrier, the CPU doing the tests can execute
the read whenever it wants.  I've seen the PPC 970 move reads absurdly
far, and the compiler can reorder things even more.

 - R.



From mst at mellanox.co.il  Thu Feb 10 09:23:06 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 19:23:06 +0200
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <52zmycbaod.fsf@topspin.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
	<20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com>
	<20050210083821.GV18350@mellanox.co.il>
	<52zmycbaod.fsf@topspin.com>
Message-ID: <20050210172306.GA28294@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH (updated)] ipoib: dont lock tx on completion
> 
>     Michael> We are now only looking at tx_tail and
>     Michael> netif_queue_stopped outside the lock (tx_head is only
>     Michael> used inside the lock):
> 
> Thanks, that helps.
> 
>     Michael> We have a write barrier in between, so its guaranteed
>     Michael> tx_tail is written before we test netif_queue_stopped.
> 
>     Michael> On the other hand, the send code has a barrier after it
>     Michael> stops the interface, so this bit is written before we
>     Michael> test tx_tail the second time.
> 
>     Michael> So, in the scenario where send stops the interface,
>     Michael> either the completion code will see it stopped, or the
>     Michael> send code will see tx tail updated.
> 
> You may be right but I'm having a hard time convincing myself.  I
> don't think just the write barriers are enough -- they just mean that
> the CPU where the writes are being done will complete the writes in
> order.

I know. I put it there so that read coming after that
wont be done after the write.

> Without a read barrier, the CPU doing the tests can execute
> the read whenever it wants.

Isnt the read barrier basically the same, just ensuring reads
are done in order?

> I've seen the PPC 970 move reads absurdly
> far, and the compiler can reorder things even more.
> 
>  - R.
> 

Hmm, I think I have it covered.

If the read of netif_queue_stopped is delayed by a huge margin,
we see the interface stopped and wake it up.

If the read of tx_tail is delayed, we see the tail updated and
wake it up in the send routine.

No? What is the race you see?

-- 
MST - Michael S. Tsirkin


From tduffy at sun.com  Thu Feb 10 09:28:43 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 09:28:43 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210134835.GA5887@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de>
Message-ID: <1108056523.18794.2.camel@duffman>

On Thu, 2005-02-10 at 14:48 +0100, Christoph Hellwig wrote:
> Umm, you IB guys had lots of strange ideas on what code was acceptable
> until Roland stepped ahead and started the rewrite of the codebase.

Don't worry.  We have learned.  We will clean it up.  The code with be
in kernel normal form before ever seeing lkml.

> Similarly even if we agree on a higher-level API not much of the
> existing code would be of any use.

The *DAPL API is already decided in a spec.  If we change it, it will
become lose compliance.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050210/d97de8b4/attachment.sig>

From hch at lst.de  Thu Feb 10 09:34:21 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 10 Feb 2005 18:34:21 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108056523.18794.2.camel@duffman>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
Message-ID: <20050210173421.GA10760@lst.de>

On Thu, Feb 10, 2005 at 09:28:43AM -0800, Tom Duffy wrote:
> On Thu, 2005-02-10 at 14:48 +0100, Christoph Hellwig wrote:
> > Umm, you IB guys had lots of strange ideas on what code was acceptable
> > until Roland stepped ahead and started the rewrite of the codebase.
> 
> Don't worry.  We have learned.  We will clean it up.  The code with be
> in kernel normal form before ever seeing lkml.
> 
> > Similarly even if we agree on a higher-level API not much of the
> > existing code would be of any use.
> 
> The *DAPL API is already decided in a spec.  If we change it, it will
> become lose compliance.

Who cares?  Specs don't matter at all for kernel APIs.  The kDAPL API
as-is is won't go in the kernel, and no amount of cosmetic cleanup
can change that.



From rminnich at lanl.gov  Thu Feb 10 09:38:30 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Thu, 10 Feb 2005 10:38:30 -0700 (MST)
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210173421.GA10760@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
Message-ID: <Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>



On Thu, 10 Feb 2005, Christoph Hellwig wrote:

> > The *DAPL API is already decided in a spec.  If we change it, it will
> > become lose compliance.
> 
> Who cares?  Specs don't matter at all for kernel APIs.  The kDAPL API
> as-is is won't go in the kernel, and no amount of cosmetic cleanup
> can change that.
> 

we're back into the "we did a spec, so the code can't change" loop. 

To break out of the loop, one need merely:
- repeat "spec? what spec? we don't need no steenking spec" out loud*
- burn the spec in your fireplace or wood stove
- write code

then it all works out.

ron
* see google for the movie reference


From hch at lst.de  Thu Feb 10 09:40:59 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 10 Feb 2005 18:40:59 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
Message-ID: <20050210174059.GA10907@lst.de>

On Thu, Feb 10, 2005 at 10:38:30AM -0700, Ronald G. Minnich wrote:
> we're back into the "we did a spec, so the code can't change" loop. 
> 
> To break out of the loop, one need merely:
> - repeat "spec? what spec? we don't need no steenking spec" out loud*
> - burn the spec in your fireplace or wood stove
> - write code
> 
> then it all works out.
> 
> ron
> * see google for the movie reference

*grin*, thanks Ronald.

btw, what other RDMA transports that DAPL is supposed to layer above are
actually available for Linux? (as in we'll see support for them in
kernel.org soon)


From tduffy at sun.com  Thu Feb 10 09:43:07 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 09:43:07 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
Message-ID: <1108057387.18794.10.camel@duffman>

On Thu, 2005-02-10 at 10:38 -0700, Ronald G. Minnich wrote:
> we're back into the "we did a spec, so the code can't change" loop. 
> burn the spec in your fireplace or wood stove

Fair enough, the spec isn't a gold standard.  But, let's not burn the
baby as well.  Why don't we clean up the code and see where it gets us?
Incremental changes to a working implementation seem far more palatable
than slash-and-burn.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050210/ceb8178a/attachment.sig>

From hch at lst.de  Thu Feb 10 09:46:16 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 10 Feb 2005 18:46:16 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108057387.18794.10.camel@duffman>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman>
Message-ID: <20050210174616.GA10990@lst.de>

On Thu, Feb 10, 2005 at 09:43:07AM -0800, Tom Duffy wrote:
> On Thu, 2005-02-10 at 10:38 -0700, Ronald G. Minnich wrote:
> > we're back into the "we did a spec, so the code can't change" loop. 
> > burn the spec in your fireplace or wood stove
> 
> Fair enough, the spec isn't a gold standard.  But, let's not burn the
> baby as well.  Why don't we clean up the code and see where it gets us?
> Incremental changes to a working implementation seem far more palatable
> than slash-and-burn.

Maybe you should lay down the requirement first.

 - why do we need an intermediate API
 - what differences does it
 - what devices does it abstract
 - what are the users



From mshefty at ichips.intel.com  Thu Feb 10 09:58:11 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 10 Feb 2005 09:58:11 -0800
Subject: [openib-general] ib_device_cap_flags
In-Reply-To: <52psz9ekod.fsf@topspin.com>
References: <20050209150310.GL18350@mellanox.co.il>
	<52psz9ekod.fsf@topspin.com>
Message-ID: <420BA0B3.80700@ichips.intel.com>

Roland Dreier wrote:
>     Michael> Hi!  enum ib_device_cap_flags from ib_verbs.h does not
>     Michael> seem to be used anywhere.  Specifically there does not
>     Michael> seem to exist a way to find out the device capabilities.
> 
>     Michael> # grep -rIi ib_device_cap_flags .
>     Michael> ./include/ib_verbs.h:enum ib_device_cap_flags {
> 
> This is partially because of Sean's somewhat unfortunate habit of
> defining enums but then just using "int" everywhere in the API.  The
> way to get the capabilities is to use ib_query_device -- the flags
> will be returned in the device_cap_flags member of struct
> ib_device_attr.

Int is used because the returned value is a bitwise OR of the valid 
capabilities, and is not a value defined by the enum.  We can change 
the enum values to #define if that would make it any easier.

- Sean


From mst at mellanox.co.il  Thu Feb 10 10:07:22 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 20:07:22 +0200
Subject: [openib-general] Re: ib_device_cap_flags
In-Reply-To: <420BA0B3.80700@ichips.intel.com>
References: <20050209150310.GL18350@mellanox.co.il>
	<52psz9ekod.fsf@topspin.com> <420BA0B3.80700@ichips.intel.com>
Message-ID: <20050210180722.GD28294@mellanox.co.il>

Quoting r. Sean Hefty <mshefty at ichips.intel.com>:
> Subject: Re: ib_device_cap_flags
> 
> Roland Dreier wrote:
> >    Michael> Hi!  enum ib_device_cap_flags from ib_verbs.h does not
> >    Michael> seem to be used anywhere.  Specifically there does not
> >    Michael> seem to exist a way to find out the device capabilities.
> >
> >    Michael> # grep -rIi ib_device_cap_flags .
> >    Michael> ./include/ib_verbs.h:enum ib_device_cap_flags {
> >
> >This is partially because of Sean's somewhat unfortunate habit of
> >defining enums but then just using "int" everywhere in the API.  The
> >way to get the capabilities is to use ib_query_device -- the flags
> >will be returned in the device_cap_flags member of struct
> >ib_device_attr.
> 
> Int is used because the returned value is a bitwise OR of the valid 
> capabilities, and is not a value defined by the enum.  We can change 
> the enum values to #define if that would make it any easier.
> 
> - Sean
> 

No, probably not :)

-- 
MST - Michael S. Tsirkin


From iod00d at hp.com  Thu Feb 10 10:15:00 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 10:15:00 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210174059.GA10907@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<20050210174059.GA10907@lst.de>
Message-ID: <20050210181500.GB28732@esmail.cup.hp.com>

On Thu, Feb 10, 2005 at 06:40:59PM +0100, Christoph Hellwig wrote:
> btw, what other RDMA transports that DAPL is supposed to layer above are
> actually available for Linux? (as in we'll see support for them in
> kernel.org soon)

I don't know for which definition of "soon": http://www.openrdma.org/

grant


From iod00d at hp.com  Thu Feb 10 10:19:47 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 10:19:47 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
Message-ID: <20050210181947.GC28732@esmail.cup.hp.com>

On Thu, Feb 10, 2005 at 10:38:30AM -0700, Ronald G. Minnich wrote:
> we're back into the "we did a spec, so the code can't change" loop. 

No we aren't. I consider it just a starting point.
If someone volunteers a different starting point (like roland
did for mthca), then let's use that. Either way, incremental
change from a working code base is a good way to boot strap
a project.

Not everyone is happy with gen2. Alot of functionality got
dropped when compared to gen1. This means the size of the
community that is interested in helping test got alot smaller.

> To break out of the loop, one need merely:
> - repeat "spec? what spec? we don't need no steenking spec" out loud*
> - burn the spec in your fireplace or wood stove
> - write code

Well, that works best IFF one has time and a clue what to write.

grant


From hch at lst.de  Thu Feb 10 10:25:37 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 10 Feb 2005 19:25:37 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210181500.GB28732@esmail.cup.hp.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<20050210174059.GA10907@lst.de>
	<20050210181500.GB28732@esmail.cup.hp.com>
Message-ID: <20050210182537.GA12005@lst.de>

On Thu, Feb 10, 2005 at 10:15:00AM -0800, Grant Grundler wrote:
> On Thu, Feb 10, 2005 at 06:40:59PM +0100, Christoph Hellwig wrote:
> > btw, what other RDMA transports that DAPL is supposed to layer above are
> > actually available for Linux? (as in we'll see support for them in
> > kernel.org soon)
> 
> I don't know for which definition of "soon": http://www.openrdma.org/

I might be doing wrong to someone, but this looks like the usual
buzzword masturbation to me..



From rminnich at lanl.gov  Thu Feb 10 10:28:54 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Thu, 10 Feb 2005 11:28:54 -0700 (MST)
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210181947.GC28732@esmail.cup.hp.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<20050210181947.GC28732@esmail.cup.hp.com>
Message-ID: <Pine.LNX.4.58.0502101125550.31745@enigma.lanl.gov>



On Thu, 10 Feb 2005, Grant Grundler wrote:

> Well, that works best IFF one has time and a clue what to write.

oops, that whole 'you have to have a clue' thing just ruled me out.

To a large extent, I am writing tongue-in-cheek about the 'burn the spec'
idea. 

My concern is that we avoid a slavish adherence to the spec that might
rule out good ideas. This is probably reaction from my checkered past of
involvement with the ISO/OSI protocol stack disaster of the late 80s ...  
in any event, I would hate to see us avoiding good ideas because they are
not 'in the spec'.

ron


From mshefty at ichips.intel.com  Thu Feb 10 10:37:55 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 10 Feb 2005 10:37:55 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our common
	work on openib.org
In-Reply-To: <20050210101519.GG18350@mellanox.co.il>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
Message-ID: <420BAA03.4040305@ichips.intel.com>

Michael S. Tsirkin wrote:

> Index: include/ib_verbs.h
> ===================================================================
> --- include/ib_verbs.h	(revision 1759)
> +++ include/ib_verbs.h	(working copy)
> @@ -73,7 +73,6 @@ enum ib_device_cap_flags {
>  	IB_DEVICE_RC_RNR_NAK_GEN	= (1<<12),
>  	IB_DEVICE_SRQ_RESIZE		= (1<<13),
>  	IB_DEVICE_N_NOTIFY_CQ		= (1<<14),
> -	IB_DEVICE_RQ_SIG_TYPE		= (1<<15)
>  };

It looks like this leaves an extra comma at the end of the enum.  Did 
you compile/test these changes?

- Sean


From iod00d at hp.com  Thu Feb 10 10:39:22 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 10:39:22 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <Pine.LNX.4.58.0502101125550.31745@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<20050210181947.GC28732@esmail.cup.hp.com>
	<Pine.LNX.4.58.0502101125550.31745@enigma.lanl.gov>
Message-ID: <20050210183922.GD28732@esmail.cup.hp.com>

On Thu, Feb 10, 2005 at 11:28:54AM -0700, Ronald G. Minnich wrote:
> 
> 
> On Thu, 10 Feb 2005, Grant Grundler wrote:
> 
> > Well, that works best IFF one has time and a clue what to write.
> 
> oops, that whole 'you have to have a clue' thing just ruled me out.

yeah, <aol>Me too</aol>. But I can hammer something that works
into shape within a few weeks I think.

> To a large extent, I am writing tongue-in-cheek about the 'burn the spec'
> idea. 
> 
> My concern is that we avoid a slavish adherence to the spec that might
> rule out good ideas. This is probably reaction from my checkered past of
> involvement with the ISO/OSI protocol stack disaster of the late 80s ...  
> in any event, I would hate to see us avoiding good ideas because they are
> not 'in the spec'.

*nod*

grant


From mst at mellanox.co.il  Thu Feb 10 10:40:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 20:40:54 +0200
Subject: [openib-general] Re: Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <420BAA03.4040305@ichips.intel.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<420BAA03.4040305@ichips.intel.com>
Message-ID: <20050210184054.GE28294@mellanox.co.il>

Quoting r. Sean Hefty <mshefty at ichips.intel.com>:
> Subject: Re: Re: FW: summary of my understanding on our common work on openib.org
> 
> Michael S. Tsirkin wrote:
> 
> >Index: include/ib_verbs.h
> >===================================================================
> >--- include/ib_verbs.h	(revision 1759)
> >+++ include/ib_verbs.h	(working copy)
> >@@ -73,7 +73,6 @@ enum ib_device_cap_flags {
> > 	IB_DEVICE_RC_RNR_NAK_GEN	= (1<<12),
> > 	IB_DEVICE_SRQ_RESIZE		= (1<<13),
> > 	IB_DEVICE_N_NOTIFY_CQ		= (1<<14),
> >-	IB_DEVICE_RQ_SIG_TYPE		= (1<<15)
> > };
> 
> It looks like this leaves an extra comma at the end of the enum.

gcc seems to allow that.

> Did 
> you compile/test these changes?
> 
> - Sean
> 

Yes.

-- 
MST - Michael S. Tsirkin


From tduffy at sun.com  Thu Feb 10 10:44:06 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 10:44:06 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <420BAA03.4040305@ichips.intel.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<420BAA03.4040305@ichips.intel.com>
Message-ID: <1108061046.18794.28.camel@duffman>

On Thu, 2005-02-10 at 10:37 -0800, Sean Hefty wrote:
> It looks like this leaves an extra comma at the end of the enum.  Did 
> you compile/test these changes?

In fact, I think this is preferred since then a new flag patch only
needs to + one line.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050210/29a64bde/attachment.sig>

From tduffy at sun.com  Thu Feb 10 10:51:25 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 10:51:25 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210174616.GA10990@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman>  <20050210174616.GA10990@lst.de>
Message-ID: <1108061485.18794.35.camel@duffman>

On Thu, 2005-02-10 at 18:46 +0100, Christoph Hellwig wrote:
> Maybe you should lay down the requirement first.

I'll take a crack at it.  Let me know where I am off base.

>  - why do we need an intermediate API

To get things working today with the code that people have already
written.  To create a proof of concept.

In any event, I am not proposing this API get kernel inclusion.  Only a
wider audience than in the sourceforge DAPL project cvs tree.

>  - what differences does it

??

>  - what devices does it abstract

IB for now, any other RDMA capable transports later.

>  - what are the users

NFS over RDMA, maybe -- so RPC.  Anybody else know others?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050210/8e5b511d/attachment.sig>

From roland at topspin.com  Thu Feb 10 11:21:48 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 10 Feb 2005 11:21:48 -0800
Subject: [openib-general] ib_device_cap_flags
In-Reply-To: <420BA0B3.80700@ichips.intel.com> (Sean Hefty's message of
	"Thu, 10 Feb 2005 09:58:11 -0800")
References: <20050209150310.GL18350@mellanox.co.il>
	<52psz9ekod.fsf@topspin.com> <420BA0B3.80700@ichips.intel.com>
Message-ID: <52vf90b4qb.fsf@topspin.com>

    Sean> Int is used because the returned value is a bitwise OR of
    Sean> the valid capabilities, and is not a value defined by the
    Sean> enum.  We can change the enum values to #define if that
    Sean> would make it any easier.

An enum type is an integer type, so it can represent the bitwise OR of
any of the defined enum values with no problem.

 - R.



From roland at topspin.com  Thu Feb 10 11:23:46 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 10 Feb 2005 11:23:46 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <20050210101519.GG18350@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 10 Feb 2005 12:15:19 +0200")
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
Message-ID: <52r7job4n1.fsf@topspin.com>

Does this seem OK to apply to everyone?  (The basic idea is to remove
the rq_sig_type QP attribute, since it is a Tavor-specific extension
that is not part of the IB spec, is currently unused, and that
Mellanox doesn't want to support)

 - R.


From mshefty at ichips.intel.com  Thu Feb 10 11:27:40 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 10 Feb 2005 11:27:40 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our	common
	work on openib.org
In-Reply-To: <52r7job4n1.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>	<20050210101519.GG18350@mellanox.co.il>
	<52r7job4n1.fsf@topspin.com>
Message-ID: <420BB5AC.2070709@ichips.intel.com>

Roland Dreier wrote:

> Does this seem OK to apply to everyone?  (The basic idea is to remove
> the rq_sig_type QP attribute, since it is a Tavor-specific extension
> that is not part of the IB spec, is currently unused, and that
> Mellanox doesn't want to support)

I say go ahead and apply it.

- Sean


From roland at topspin.com  Thu Feb 10 11:30:32 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 10 Feb 2005 11:30:32 -0800
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <20050210172306.GA28294@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 10 Feb 2005 19:23:06 +0200")
References: <20050208105112.GM2141@mellanox.co.il>
	<527jlhfzx6.fsf@topspin.com> <20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com> <20050210083821.GV18350@mellanox.co.il>
	<52zmycbaod.fsf@topspin.com> <20050210172306.GA28294@mellanox.co.il>
Message-ID: <52mzucb4br.fsf@topspin.com>

    Michael> Hmm, I think I have it covered.

    Michael> If the read of netif_queue_stopped is delayed by a huge
    Michael> margin, we see the interface stopped and wake it up.

    Michael> If the read of tx_tail is delayed, we see the tail
    Michael> updated and wake it up in the send routine.

    Michael> No? What is the race you see?

I think the wmb() has to at least become barrier() -- otherwise the
send routine has no reason to reread tx_tail after it stops the queue.

I don't see a specific race -- I'm just not totally comfortable yet,
even with that fixed.  The issue is not necessarily reads being
delayed, since both an out-of-order CPU and a compiler may move reads
much _earlier_ than they appear in the code (for example a read may be
speculatively executed because of branch prediction for a test in an
if statement).

 - R.


From mst at mellanox.co.il  Thu Feb 10 11:40:56 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 21:40:56 +0200
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <52mzucb4br.fsf@topspin.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
	<20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com>
	<20050210083821.GV18350@mellanox.co.il>
	<52zmycbaod.fsf@topspin.com>
	<20050210172306.GA28294@mellanox.co.il>
	<52mzucb4br.fsf@topspin.com>
Message-ID: <20050210194056.GF28294@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH (updated)] ipoib: dont lock tx on completion
> 
>     Michael> Hmm, I think I have it covered.
> 
>     Michael> If the read of netif_queue_stopped is delayed by a huge
>     Michael> margin, we see the interface stopped and wake it up.
> 
>     Michael> If the read of tx_tail is delayed, we see the tail
>     Michael> updated and wake it up in the send routine.
> 
>     Michael> No? What is the race you see?
> 
> I think the wmb() has to at least become barrier() -- otherwise the
> send routine has no reason to reread tx_tail after it stops the queue.
> 
> I don't see a specific race -- I'm just not totally comfortable yet,
> even with that fixed.  The issue is not necessarily reads being
> delayed, since both an out-of-order CPU and a compiler may move reads
> much _earlier_ than they appear in the code (for example a read may be
> speculatively executed because of branch prediction for a test in an
> if statement).
> 
>  - R.
> 

Can they be moved to before the wmb? I think not.

-- 
MST - Michael S. Tsirkin


From blist at aon.at  Thu Feb 10 11:44:04 2005
From: blist at aon.at (Bernhard Fischer)
Date: Thu, 10 Feb 2005 20:44:04 +0100
Subject: [openib-general] opensm segfaults after CL_INSUFFICIENT_MEMORY
Message-ID: <20050210194404.GB21097@aon.at>

Hi,

I'm seeing the segfault below when i try to run opensm.

gen2 as of 09.02.2005, only thing i changed in order to try to
circumvent those
ib_mthca 0000:04:00.0: CQ overrun on CQN 00000082
was setting IPOIB_NUM_WC to 1 as noted here:
http://openib.org/pipermail/openib-general/2004-December/007147.html
(which didn't help).

Any ideas?
(gdb) run
Starting program: /usr/local/ib/bin/opensm 
[Thread debugging using libthread_db enabled]
[New Thread -1209392032 (LWP 7991)]
__init: failed to create timer provider status (CL_INSUFFICIENT_MEMORY) 

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209392032 (LWP 7991)]
0xb7f0b597 in memset () from /lib/tls/libc.so.6
(gdb) bt
#0  0xb7f0b597 in memset () from /lib/tls/libc.so.6
#1  0xb7fcd000 in ?? ()
#2  0xb7fd2e4a in cl_memset (p_memory=Variable "p_memory" is not
available.
) at cl_memory_osd.c:78
#3  0xb7fd488e in __cl_sys_callback_construct () at cl_memory.h:553
#4  0xb7fd48d0 in __cl_sys_callback_init () at cl_syscallback.c:113
#5  0xb7fd09b6 in complib_init () at cl_complib.c:106
#6  0xb7fd5dc7 in __do_global_ctors_aux () from
/usr/local/ib/lib/libosmcomp.so.1
#7  0xb7fcfed5 in _init () from /usr/local/ib/lib/libosmcomp.so.1
#8  0xb7ff716c in call_init () from /lib/ld-linux.so.2
#9  0xb7ff7252 in _dl_init_internal () from /lib/ld-linux.so.2
#10 0xb7fea9c5 in _dl_start_user () from /lib/ld-linux.so.2

Thank you,


From mshefty at ichips.intel.com  Thu Feb 10 11:44:39 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 10 Feb 2005 11:44:39 -0800
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <20050210194056.GF28294@mellanox.co.il>
References: <20050208105112.GM2141@mellanox.co.il>
	<527jlhfzx6.fsf@topspin.com>	<20050209184132.GB19608@mellanox.co.il>	<52d5v9cl4x.fsf@topspin.com>	<20050210083821.GV18350@mellanox.co.il>	<52zmycbaod.fsf@topspin.com>	<20050210172306.GA28294@mellanox.co.il>	<52mzucb4br.fsf@topspin.com>
	<20050210194056.GF28294@mellanox.co.il>
Message-ID: <420BB9A7.8040603@ichips.intel.com>

Michael S. Tsirkin wrote:

>>I don't see a specific race -- I'm just not totally comfortable yet,
>>even with that fixed.  The issue is not necessarily reads being
>>delayed, since both an out-of-order CPU and a compiler may move reads
>>much _earlier_ than they appear in the code (for example a read may be
>>speculatively executed because of branch prediction for a test in an
>>if statement).
>>
>> - R.
>>
> Can they be moved to before the wmb? I think not.

Given that there's some concern here regarding races, what sort of 
performance improvement do we see with this change?

- Sean


From roland at topspin.com  Thu Feb 10 12:04:48 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 10 Feb 2005 12:04:48 -0800
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <20050210194056.GF28294@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 10 Feb 2005 21:40:56 +0200")
References: <20050208105112.GM2141@mellanox.co.il>
	<527jlhfzx6.fsf@topspin.com> <20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com> <20050210083821.GV18350@mellanox.co.il>
	<52zmycbaod.fsf@topspin.com> <20050210172306.GA28294@mellanox.co.il>
	<52mzucb4br.fsf@topspin.com> <20050210194056.GF28294@mellanox.co.il>
Message-ID: <52ekfob2qn.fsf@topspin.com>

    Michael> Can they be moved to before the wmb? I think not.

Why not?  All wmb() does is guarantee that stores issued before the
wmb() will complete before stores issued after the wmb().  It doesn't
even say anything about when the stores have to complete, let alone
the order of loads.  The most obvious definition is for sparc64, where
we have

	#define wmb()           membar("#StoreStore")

which my Sparc reference manual says only means

    Forces all stores after the MEMBAR to wait until all stores before
    the MEMBAR have reached global visibility.

ppc is similar, where wmb() is just "eieio."

 - R.


From mlleinin at hpcn.ca.sandia.gov  Thu Feb 10 12:05:58 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Thu, 10 Feb 2005 12:05:58 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108061485.18794.35.camel@duffman>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman>  <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
Message-ID: <1108065958.7776.10.camel@localhost>

On Thu, 2005-02-10 at 10:51 -0800, Tom Duffy wrote:
> On Thu, 2005-02-10 at 18:46 +0100, Christoph Hellwig wrote:
> > Maybe you should lay down the requirement first.
> 
> I'll take a crack at it.  Let me know where I am off base.
> 
> >  - why do we need an intermediate API
> 
> To get things working today with the code that people have already
> written.  To create a proof of concept.
> 
> In any event, I am not proposing this API get kernel inclusion.  Only a
> wider audience than in the sourceforge DAPL project cvs tree.
> 
> >  - what differences does it
> 
> ??
> 
> >  - what devices does it abstract
> 
> IB for now, any other RDMA capable transports later.
> 
> >  - what are the users
> 
> NFS over RDMA, maybe -- so RPC.  Anybody else know others?
> 
  uDAPL - Oracle, MPI
  kDAPL - iSER, NFS over RDMA, Lustre?

  - Matt



From hch at lst.de  Thu Feb 10 12:11:16 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 10 Feb 2005 21:11:16 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108065958.7776.10.camel@localhost>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
Message-ID: <20050210201116.GA13657@lst.de>

On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
>   kDAPL - iSER, NFS over RDMA, Lustre?

Okay, we have iSER code and maybe there will be NFS code.  Lustre
doesn't matter at all for any possible design because it's not freely
available.


From iod00d at hp.com  Thu Feb 10 12:25:18 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 12:25:18 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <52acqcb2or.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<52r7job4n1.fsf@topspin.com>
	<20050210193142.GA29078@esmail.cup.hp.com>
	<52is50b3uj.fsf@topspin.com>
	<20050210194801.GD29078@esmail.cup.hp.com>
	<52acqcb2or.fsf@topspin.com>
Message-ID: <20050210202518.GA29349@esmail.cup.hp.com>

Offline, I exchange email with Roland about this attribute.
Seems like we want to know the answer to these questions:

o is there any code outside of openib.org using this attribute?
o if so, how useful is it?

Roland is certain none of the protocols in the proprietary Topspin stack
uses this feature.

The reasons roland gave all seemed valid for removal except the fact
the product has been shipping for a while and thus it's available to
users regardless of if mellanox supports it or not.
It's reasonable to remove if no one can show a use case.

grant


From iod00d at hp.com  Thu Feb 10 12:27:04 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 12:27:04 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108065958.7776.10.camel@localhost>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
Message-ID: <20050210202704.GB29349@esmail.cup.hp.com>

On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
>   uDAPL - Oracle, MPI
>   kDAPL - iSER, NFS over RDMA, Lustre?

Lustre will use Sandia Portals AFAIK.
Anyone know what Portals will use?
They might directly program to VAPI or something.

grant


From libor at topspin.com  Thu Feb 10 12:31:17 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 10 Feb 2005 12:31:17 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108065958.7776.10.camel@localhost>;
	from mlleinin@hpcn.ca.sandia.gov on Thu, Feb 10, 2005 at
	12:05:58PM -0800
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
Message-ID: <20050210123117.A11665@topspin.com>

On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> On Thu, 2005-02-10 at 10:51 -0800, Tom Duffy wrote:
> > On Thu, 2005-02-10 at 18:46 +0100, Christoph Hellwig wrote:
> > 
> > >  - what are the users
> > 
> > NFS over RDMA, maybe -- so RPC.  Anybody else know others?
> > 
>   uDAPL - Oracle, MPI
>   kDAPL - iSER, NFS over RDMA, Lustre?

  uDAPL in this context is not relevant, since it's entirely in
userspace and does not depend on kDAPL in the kernel. 

    
-Libor


From halr at voltaire.com  Thu Feb 10 12:26:45 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 10 Feb 2005 15:26:45 -0500
Subject: [openib-general] opensm segfaults after CL_INSUFFICIENT_MEMORY
In-Reply-To: <20050210194404.GB21097@aon.at>
References: <20050210194404.GB21097@aon.at>
Message-ID: <1108067205.4649.19.camel@localhost.localdomain>

On Thu, 2005-02-10 at 14:44, Bernhard Fischer wrote:
> Hi,
> 
> I'm seeing the segfault below when i try to run opensm.
> 
> gen2 as of 09.02.2005, only thing i changed in order to try to
> circumvent those
> ib_mthca 0000:04:00.0: CQ overrun on CQN 00000082
> was setting IPOIB_NUM_WC to 1 as noted here:
> http://openib.org/pipermail/openib-general/2004-December/007147.html
> (which didn't help).

This is totally separate.

> Any ideas?
> (gdb) run
> Starting program: /usr/local/ib/bin/opensm 
> [Thread debugging using libthread_db enabled]
> [New Thread -1209392032 (LWP 7991)]
> __init: failed to create timer provider status (CL_INSUFFICIENT_MEMORY) 

First malloc is failing to create the timer

> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1209392032 (LWP 7991)]
> 0xb7f0b597 in memset () from /lib/tls/libc.so.6
> (gdb) bt
> #0  0xb7f0b597 in memset () from /lib/tls/libc.so.6
> #1  0xb7fcd000 in ?? ()
> #2  0xb7fd2e4a in cl_memset (p_memory=Variable "p_memory" is not
> available.
> ) at cl_memory_osd.c:78
> #3  0xb7fd488e in __cl_sys_callback_construct () at cl_memory.h:553
> #4  0xb7fd48d0 in __cl_sys_callback_init () at cl_syscallback.c:113
> #5  0xb7fd09b6 in complib_init () at cl_complib.c:106
> #6  0xb7fd5dc7 in __do_global_ctors_aux () from
> /usr/local/ib/lib/libosmcomp.so.1
> #7  0xb7fcfed5 in _init () from /usr/local/ib/lib/libosmcomp.so.1
> #8  0xb7ff716c in call_init () from /lib/ld-linux.so.2
> #9  0xb7ff7252 in _dl_init_internal () from /lib/ld-linux.so.2
> #10 0xb7fea9c5 in _dl_start_user () from /lib/ld-linux.so.2

Initialization continues (not sure it should) and attempting to clear a
static object fails. Even if initialization didn't continue, opensm
would not run properly.

The first problem is why the malloc fails for the timer provider. The
second problem is pretty strange as the system callback object is local
and should not be an address that causes a segmentation violation.

Is this built with the autotools version ?

Could you revert back to the old Makefiles and rebuild if so and see if
you still have the same problem ?

Thanks.

-- Hal

> Thank you,
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From johannes at erdfelt.com  Thu Feb 10 12:39:41 2005
From: johannes at erdfelt.com (Johannes Erdfelt)
Date: Thu, 10 Feb 2005 12:39:41 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <20050210202518.GA29349@esmail.cup.hp.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<52r7job4n1.fsf@topspin.com>
	<20050210193142.GA29078@esmail.cup.hp.com>
	<52is50b3uj.fsf@topspin.com>
	<20050210194801.GD29078@esmail.cup.hp.com>
	<52acqcb2or.fsf@topspin.com>
	<20050210202518.GA29349@esmail.cup.hp.com>
Message-ID: <20050210203941.GS25370@sventech.com>

On Thu, Feb 10, 2005, Grant Grundler <iod00d at hp.com> wrote:
> Offline, I exchange email with Roland about this attribute.
> Seems like we want to know the answer to these questions:
> 
> o is there any code outside of openib.org using this attribute?
> o if so, how useful is it?
> 
> Roland is certain none of the protocols in the proprietary Topspin stack
> uses this feature.
> 
> The reasons roland gave all seemed valid for removal except the fact
> the product has been shipping for a while and thus it's available to
> users regardless of if mellanox supports it or not.
> It's reasonable to remove if no one can show a use case.

Or reasonable to add back if someone shows a use case.

The kernel code should be lean and unless there is an immediate or short
term use for a feature, it probably shouldn't be there. We should be
asking "why keep it in the code?", not "why should we remove it from
the code?"

If we're a little bit too aggressive with removing stuff, just add it
back when you need it.

JE



From mlleinin at hpcn.ca.sandia.gov  Thu Feb 10 12:36:39 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Thu, 10 Feb 2005 12:36:39 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210202704.GB29349@esmail.cup.hp.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
	<20050210202704.GB29349@esmail.cup.hp.com>
Message-ID: <1108067799.7746.14.camel@localhost>

On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> >   uDAPL - Oracle, MPI
> >   kDAPL - iSER, NFS over RDMA, Lustre?
> 
> Lustre will use Sandia Portals AFAIK.
> Anyone know what Portals will use?
> They might directly program to VAPI or something.
> 
  There will be a Portals over verbs.  At some point there may be a
Portals over kDAPL to support both RDMA ethernet and IB.  

   - Matt



From mst at mellanox.co.il  Thu Feb 10 12:42:04 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 22:42:04 +0200
Subject: [openib-general] Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <52ekfob2qn.fsf@topspin.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
	<20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com>
	<20050210083821.GV18350@mellanox.co.il>
	<52zmycbaod.fsf@topspin.com>
	<20050210172306.GA28294@mellanox.co.il>
	<52mzucb4br.fsf@topspin.com>
	<20050210194056.GF28294@mellanox.co.il>
	<52ekfob2qn.fsf@topspin.com>
Message-ID: <20050210204204.GG28294@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH (updated)] ipoib: dont lock tx on completion
> 
>     Michael> Can they be moved to before the wmb? I think not.
> 
> Why not?  All wmb() does is guarantee that stores issued before the
> wmb() will complete before stores issued after the wmb().  It doesn't
> even say anything about when the stores have to complete, let alone
> the order of loads.  The most obvious definition is for sparc64, where
> we have
> 
> 	#define wmb()           membar("#StoreStore")
> 
> which my Sparc reference manual says only means
> 
>     Forces all stores after the MEMBAR to wait until all stores before
>     the MEMBAR have reached global visibility.
> 
> ppc is similar, where wmb() is just "eieio."
> 
>  - R.
> 

OK, so means we must make them both mb(). Sigh.

-- 
MST - Michael S. Tsirkin


From mlleinin at hpcn.ca.sandia.gov  Thu Feb 10 12:39:17 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Thu, 10 Feb 2005 12:39:17 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050210201116.GA13657@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
	<20050210201116.GA13657@lst.de>
Message-ID: <1108067957.7776.18.camel@localhost>

On Thu, 2005-02-10 at 21:11 +0100, Christoph Hellwig wrote:
> On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> >   kDAPL - iSER, NFS over RDMA, Lustre?
> 
> Okay, we have iSER code and maybe there will be NFS code.  Lustre
> doesn't matter at all for any possible design because it's not freely
> available.

   Good point.  Lustre is open source, but only after the code is old.
I wish they would open up the project and do real open development.  Oh
well.

   - Matt




From mst at mellanox.co.il  Thu Feb 10 12:44:58 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 10 Feb 2005 22:44:58 +0200
Subject: [openib-general] Re: Re: [PATCH (updated)] ipoib: dont lock tx on
	completion
In-Reply-To: <420BB9A7.8040603@ichips.intel.com>
References: <20050208105112.GM2141@mellanox.co.il> <527jlhfzx6.fsf@topspin.com>
	<20050209184132.GB19608@mellanox.co.il>
	<52d5v9cl4x.fsf@topspin.com>
	<20050210083821.GV18350@mellanox.co.il>
	<52zmycbaod.fsf@topspin.com>
	<20050210172306.GA28294@mellanox.co.il>
	<52mzucb4br.fsf@topspin.com>
	<20050210194056.GF28294@mellanox.co.il>
	<420BB9A7.8040603@ichips.intel.com>
Message-ID: <20050210204458.GH28294@mellanox.co.il>

Quoting r. Sean Hefty <mshefty at ichips.intel.com>:
> Subject: Re: Re: [PATCH (updated)] ipoib: dont lock tx on completion
> 
> Michael S. Tsirkin wrote:
> 
> >>I don't see a specific race -- I'm just not totally comfortable yet,
> >>even with that fixed.  The issue is not necessarily reads being
> >>delayed, since both an out-of-order CPU and a compiler may move reads
> >>much _earlier_ than they appear in the code (for example a read may be
> >>speculatively executed because of branch prediction for a test in an
> >>if statement).
> >>
> >>- R.
> >>
> >Can they be moved to before the wmb? I think not.
> 
> Given that there's some concern here regarding races, what sort of 
> performance improvement do we see with this change?
> 
> - Sean
> 

Nothing major, I expect, especially given that only the send flow is
affected, and that is already apparently using less CPU than receive.

But I am just thinking, with tx_head/tx_taiil we have a circular
buffer here, how come we can not use it without races and locks?
There must be a way ...

-- 
MST - Michael S. Tsirkin


From iod00d at hp.com  Thu Feb 10 12:59:03 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 12:59:03 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <20050210203941.GS25370@sventech.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<52r7job4n1.fsf@topspin.com>
	<20050210193142.GA29078@esmail.cup.hp.com>
	<52is50b3uj.fsf@topspin.com>
	<20050210194801.GD29078@esmail.cup.hp.com>
	<52acqcb2or.fsf@topspin.com>
	<20050210202518.GA29349@esmail.cup.hp.com>
	<20050210203941.GS25370@sventech.com>
Message-ID: <20050210205903.GC29349@esmail.cup.hp.com>

On Thu, Feb 10, 2005 at 12:39:41PM -0800, Johannes Erdfelt wrote:
> The kernel code should be lean and unless there is an immediate or short
> term use for a feature, it probably shouldn't be there. We should be
> asking "why keep it in the code?", not "why should we remove it from
> the code?"

enums and typedefs are a (weak) form of documentation.
If people don't know the feature exists they won't know to look for it
in better forms of documentation that are privately available.

I'm generalizing since I have NFC what this attribute does specifically.
If someone publicly can say "Don't use this because..." then that would
be a good reason to remove the declaration too.

grant


From iod00d at hp.com  Thu Feb 10 13:01:52 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 10 Feb 2005 13:01:52 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108067957.7776.18.camel@localhost>
References: <20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
	<20050210201116.GA13657@lst.de>
	<1108067957.7776.18.camel@localhost>
Message-ID: <20050210210152.GD29349@esmail.cup.hp.com>

On Thu, Feb 10, 2005 at 12:39:17PM -0800, Matt Leininger wrote:
>    Good point.  Lustre is open source, but only after the code is old.
> I wish they would open up the project and do real open development.  Oh
> well.

Ugh. Chrisptoph is right. I was under the illusion Lustre
was a "real" open source project. A quick visit to lustre.org
leaves one with that impression. Trying to download the source
leads one to clusterfs.com where the bubble gets popped pretty
quickly.

grant


From libor at topspin.com  Thu Feb 10 13:02:48 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 10 Feb 2005 13:02:48 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108067799.7746.14.camel@localhost>;
	from mlleinin@hpcn.ca.sandia.gov on Thu, Feb 10, 2005 at
	12:36:39PM -0800
References: <20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
	<20050210202704.GB29349@esmail.cup.hp.com>
	<1108067799.7746.14.camel@localhost>
Message-ID: <20050210130248.B11665@topspin.com>

On Thu, Feb 10, 2005 at 12:36:39PM -0800, Matt Leininger wrote:
> On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > >   uDAPL - Oracle, MPI
> > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > 
> > Lustre will use Sandia Portals AFAIK.
> > Anyone know what Portals will use?
> > They might directly program to VAPI or something.
> > 
>   There will be a Portals over verbs.  At some point there may be a
> Portals over kDAPL to support both RDMA ethernet and IB.  

  Yup, that's one of the bigger questions, can it abstract away the
differences between two different RDMA technologies? Having a RDMA
ethernet and IB providers for kDAPL is insufficient, one would need
to show an actual, non-trivial, protocol that works ontop of either
provider with no, or little, modification/ifdef'ing.

-Libor




From johannes at erdfelt.com  Thu Feb 10 13:04:25 2005
From: johannes at erdfelt.com (Johannes Erdfelt)
Date: Thu, 10 Feb 2005 13:04:25 -0800
Subject: [openib-general] Re: FW: summary of my understanding on our
	common work on openib.org
In-Reply-To: <20050210205903.GC29349@esmail.cup.hp.com>
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<52r7job4n1.fsf@topspin.com>
	<20050210193142.GA29078@esmail.cup.hp.com>
	<52is50b3uj.fsf@topspin.com>
	<20050210194801.GD29078@esmail.cup.hp.com>
	<52acqcb2or.fsf@topspin.com>
	<20050210202518.GA29349@esmail.cup.hp.com>
	<20050210203941.GS25370@sventech.com>
	<20050210205903.GC29349@esmail.cup.hp.com>
Message-ID: <20050210210425.GT25370@sventech.com>

On Thu, Feb 10, 2005, Grant Grundler <iod00d at hp.com> wrote:
> On Thu, Feb 10, 2005 at 12:39:41PM -0800, Johannes Erdfelt wrote:
> > The kernel code should be lean and unless there is an immediate or short
> > term use for a feature, it probably shouldn't be there. We should be
> > asking "why keep it in the code?", not "why should we remove it from
> > the code?"
> 
> enums and typedefs are a (weak) form of documentation.
> If people don't know the feature exists they won't know to look for it
> in better forms of documentation that are privately available.

Good point, that makes sense too.

> I'm generalizing since I have NFC what this attribute does specifically.
> If someone publicly can say "Don't use this because..." then that would
> be a good reason to remove the declaration too.

It sounds like Mellanox doesn't want to support it and it's nonstandard
anyway.

JE


From Arkady.Kanevsky at netapp.com  Thu Feb 10 15:49:02 2005
From: Arkady.Kanevsky at netapp.com (Kanevsky, Arkady)
Date: Thu, 10 Feb 2005 18:49:02 -0500
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <C98692FD98048C41885E0B0FACD9DFB86EB15B@exnane01.hq.netapp.com>

For kDAPL:
The iSER has been submitted to Open Ib by Voltaire already.
NFS-RDMA is at http://sourceforge.net/projects/nfs-rdma/.

For uDAPL: Oracle, DB2 and MPI.
I am not aware if there is an open source MPI version on uDAPL.

These are publicly known.

As far as changing the uDAPL or kDAPL APIs.
There are application already writen to them.
There are implementation of these APIs on other platforms besides Linux.
It is in nobody's interest to splinter the user community.
We need the same API on all platforms.
If there is a good technical reason to change some specific APIs we
should consider it.
But the "burn the spec" approach is not a rationale one.
If we need to change implementation or some definitions in header files
it is feasible.

As far as other transport. As people already mentioned iWARP (IETF
RDDP).
IBM talked at the BOF about RNIC PI which is being developed as a level
of
abstraction on the lower end to "discover" all the need info about
RNIC/HCA.
It is still no ready so we will start with gen2.
But lets not loose site of what DAPL brings:
OS independent,
Transport independent,
RDMA APIs!!!

Thanks for jumping on the code so quickly.
Arkady
Chair of DAT Collaborative

Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance                     phone: 781-768-5395
375 Totten Pond Rd.                  Fax: 781-895-1195
Waltham, MA 02451-2010          central phone: 781-768-5300
 


> -----Original Message-----
> From: Libor Michalek [mailto:libor at topspin.com] 
> Sent: Thursday, February 10, 2005 4:03 PM
> To: Matt Leininger
> Cc: Christoph Hellwig; openib-general at openib.org; Tom Duffy
> Subject: Re: FW: [openib-general] Minutes from DAPL BOF at 
> OpenIB Workshop
> 
> 
> On Thu, Feb 10, 2005 at 12:36:39PM -0800, Matt Leininger wrote:
> > On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > > >   uDAPL - Oracle, MPI
> > > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > > 
> > > Lustre will use Sandia Portals AFAIK.
> > > Anyone know what Portals will use?
> > > They might directly program to VAPI or something.
> > > 
> >   There will be a Portals over verbs.  At some point there may be a 
> > Portals over kDAPL to support both RDMA ethernet and IB.
> 
>   Yup, that's one of the bigger questions, can it abstract 
> away the differences between two different RDMA technologies? 
> Having a RDMA ethernet and IB providers for kDAPL is 
> insufficient, one would need to show an actual, non-trivial, 
> protocol that works ontop of either provider with no, or 
> little, modification/ifdef'ing.
> 
> -Libor
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org 
> http://openib.org/mailman/listinfo/openib-> general
> 
> To 
> unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
> 


From panda at cse.ohio-state.edu  Thu Feb 10 19:01:17 2005
From: panda at cse.ohio-state.edu (Dhabaleswar Panda)
Date: Thu, 10 Feb 2005 22:01:17 -0500 (EST)
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <C98692FD98048C41885E0B0FACD9DFB86EB15B@exnane01.hq.netapp.com>
	from "Kanevsky, Arkady" at Feb 10, 2005 06:49:02 PM
Message-ID: <200502110301.j1B31HLx000444@xi.cse.ohio-state.edu>

> For kDAPL:
> The iSER has been submitted to Open Ib by Voltaire already.
> NFS-RDMA is at http://sourceforge.net/projects/nfs-rdma/.
> 
> For uDAPL: Oracle, DB2 and MPI.
> I am not aware if there is an open source MPI version on uDAPL.

As I indicated in my talk at the OpenIB workshop, we are providing a
port of our open source MPI (MVAPICH2) on uDAPL. I also presented some
initial performance results of MVAPICH2/uDAPL/VAPI on IBA and
MVAPICH2/uDAPL/GM on Myrinet. We are currently testing/tuning it on
uDAPL support on other interconnects.  We will be releasing this
version of MPI in the next few weeks. Once the uDAPL layer is
available for OpenIB, this MPI version can also run on top of that
stack.

This will allow the community to have an open source MPI version on
uDAPL for various (existing and emerging) interconnects.

Thanks, 

DK Panda
OSU

> These are publicly known.
> 
> As far as changing the uDAPL or kDAPL APIs.
> There are application already writen to them.
> There are implementation of these APIs on other platforms besides Linux.
> It is in nobody's interest to splinter the user community.
> We need the same API on all platforms.
> If there is a good technical reason to change some specific APIs we
> should consider it.
> But the "burn the spec" approach is not a rationale one.
> If we need to change implementation or some definitions in header files
> it is feasible.
> 
> As far as other transport. As people already mentioned iWARP (IETF
> RDDP).
> IBM talked at the BOF about RNIC PI which is being developed as a level
> of
> abstraction on the lower end to "discover" all the need info about
> RNIC/HCA.
> It is still no ready so we will start with gen2.
> But lets not loose site of what DAPL brings:
> OS independent,
> Transport independent,
> RDMA APIs!!!
> 
> Thanks for jumping on the code so quickly.
> Arkady
> Chair of DAT Collaborative
> 
> Arkady Kanevsky                       email: arkady at netapp.com
> Network Appliance                     phone: 781-768-5395
> 375 Totten Pond Rd.                  Fax: 781-895-1195
> Waltham, MA 02451-2010          central phone: 781-768-5300
>  
> 
> 
> > -----Original Message-----
> > From: Libor Michalek [mailto:libor at topspin.com] 
> > Sent: Thursday, February 10, 2005 4:03 PM
> > To: Matt Leininger
> > Cc: Christoph Hellwig; openib-general at openib.org; Tom Duffy
> > Subject: Re: FW: [openib-general] Minutes from DAPL BOF at 
> > OpenIB Workshop
> > 
> > 
> > On Thu, Feb 10, 2005 at 12:36:39PM -0800, Matt Leininger wrote:
> > > On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > > > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > > > >   uDAPL - Oracle, MPI
> > > > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > > > 
> > > > Lustre will use Sandia Portals AFAIK.
> > > > Anyone know what Portals will use?
> > > > They might directly program to VAPI or something.
> > > > 
> > >   There will be a Portals over verbs.  At some point there may be a 
> > > Portals over kDAPL to support both RDMA ethernet and IB.
> > 
> >   Yup, that's one of the bigger questions, can it abstract 
> > away the differences between two different RDMA technologies? 
> > Having a RDMA ethernet and IB providers for kDAPL is 
> > insufficient, one would need to show an actual, non-trivial, 
> > protocol that works ontop of either provider with no, or 
> > little, modification/ifdef'ing.
> > 
> > -Libor
> > 
> > 
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org 
> > http://openib.org/mailman/listinfo/openib-> general
> > 
> > To 
> > unsubscribe, please visit 
> > http://openib.org/mailman/listinfo/openib-general
> > 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 



From libor at topspin.com  Thu Feb 10 19:09:04 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 10 Feb 2005 19:09:04 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1107552392.2435.6.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 04, 2005 at 01:26:32PM -0800
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
Message-ID: <20050210190904.C11665@topspin.com>

On Fri, Feb 04, 2005 at 01:26:32PM -0800, Tom Duffy wrote:
> On Fri, 2005-02-04 at 11:42 -0800, Libor Michalek wrote:
> >   You mean a quick description of the code, like the primary contents of
> > each file and how to get it to do something ?
> 
> That would be great start.

  OK, I've gone with option number 3. Here is a one or two line
description of each file, and a TODO list. Both are checked in
along with the code in the infiniband/ulp/sdp directory.

-Libor


FILES
-----
sdp_advt.[ch]  - Data structure. 
                 RDMA advertisments representing remote IOCBs. Used to 
                 track RDMA read and write requests initiated in the data
                 path.
sdp_actv.c     - Control path.
                 Active connection establishment interface to Connection
                 Manager.
sdp_buff.[ch]  - Data structure. 
                 Global pool of buffers used for data and control messages.
sdp_conn.[ch]  - Data structure. 
                 Connection data, including all state and information. 
                 Global connection tables.
sdp_event.c    - Control and Data path. 
                 Event notification for Connection Manager state changes. 
                 Completion Queue notification.
sdp_inet.c     - Control path. 
                 Interface to Linux AF_INET/SOCK_STREAM sockets. 
sdp_iocb.[ch]  - Data structure.
                 RDMA IO control structure representing local buffers which
                 can be used as RDMA targets by the remote connection peer
                 for RDMA read and write requests.
sdp_kvec.c     - AIO IOCB cancellation code. (TODO)
sdp_link.c     - Control path.
                 Active connection establishment interface to Address
                 Resolution and Path Record lookups.
sdp_pass.c     - Control path.
                 Passive connection establishment interface to Connection
                 Manager.
sdp_post.c     - Control path.
                 Requests made to Connection Manager common for active
                 and passive connections.
sdp_proc.[ch]  - Misc.
                 Linux /proc filesystem interface for reporting internal
                 state to userspace.
sdp_queue.[ch] - Data structure.
                 Generic FIFO/LILO queue and element definition used for
                 BUFF, ADVT, and IOCB data structures.
sdp_rcvd.c     - Data path.
                 Process receive CQ notifications. Member of the receive
                 half of the data path.
sdp_read.c     - Data path.
                 Process send CQ RDMA Read notifications. Member of the
                 receive half of the data path.
sdp_recv.c     - Data path.
                 Member of the receive half of the data path. Processes data 
                 from upper OS interface to the posting of buffers to a QPs
                 receive or send WQ.
sdp_send.c     - Data path.
                 Member of the send half of the data path. Processes data from
                 upper OS interface to the posting of buffers to a QPs send WQ.
sdp_sent.c     - Data path.
                 Process send CQ notifications. Member of the send half of
                 the data path.
sdp_wall.c     - Control path.
                 Transition between Operating specific state and protocol
                 control state. (TODO)
sdp_write.c    - Data path.
                 Process send CQ RDMA Write notifications. Member of the
                 send half of the data path.

sdp_buff_p.h   - Data structure. 
                 Global pool of buffers.
sdp_dev.h      - Misc.
                 Global definitions for limits and constants
sdp_main.h     - Misc.
                 Top level SDP include.
sdp_msgs.h     - Data structure.
                 IBTA specification SDP messages and headers.
sdp_proto.h    - Misc.
                 Function prototype definitions.

TODO
----
SDP 

  - Cosmetic cleanup as needed.

  - Reduce connection state machine complexity. Specifically there exists
    redundant tracking of connection state which should be merged for
    improved readability and maintainability.

  - Add full connection cleanup once Connection Manager IDLE is in place.

  - Reduce code bloat.

  - Enable FMRs once mthca support is provided.

  - Provide real support for SO_{RCV,SND}BUF
    - internal posted buffers
    - QP WQ size
    - CQ size

  - Create slow start algorithm for posted receive buffers to improve
    memory consumption in large connection count applications, especially
    for connections that are predominately sending data.

  - sdp_link needs locking protection.

  - sdp_link request cancellation support.

  - SDP RDMA support for blocking socket operations.

  - IPv6 addressing

  - Alternate Path Migration (APM) support.

  - Keepalive (zero byte RDMA write)

  - Socket duplication (i.e. suspend)

Possible Issues

  - SDP address resolution requires looking at the IPoIB device's private
    data structure inorder to determine interface's SGID, Port, and PKey.

  - Memory locking for AIO requires a call to do_mlock() which is not a
    kernel exported function, the method for calling the function is not
    standard.

Other

  - Kernel level transparent socket switch for dynamic TCP/SDP selection.
    (Possible SDP port mapper protocol from RDMAC)

  - Extended Sockets API (ES-API). Initial Linux AIO based implemetation?

  - New SOCK_DGRAM protocol?

  - NFS over SDP.

  - QP striping?


From tduffy at sun.com  Thu Feb 10 19:39:56 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 19:39:56 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <20050210190904.C11665@topspin.com>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman> <20050210190904.C11665@topspin.com>
Message-ID: <1108093196.28588.3.camel@duffman>

On Thu, 2005-02-10 at 19:09 -0800, Libor Michalek wrote:
>   OK, I've gone with option number 3. Here is a one or two line
> description of each file, and a TODO list. Both are checked in
> along with the code in the infiniband/ulp/sdp directory.

Of course, you will need this patch to plumb SDP into the build.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/Kconfig
===================================================================
--- drivers/infiniband/Kconfig	(revision 1766)
+++ drivers/infiniband/Kconfig	(working copy)
@@ -11,4 +11,6 @@ source "drivers/infiniband/hw/mthca/Kcon
 
 source "drivers/infiniband/ulp/ipoib/Kconfig"
 
+source "drivers/infiniband/ulp/sdp/Kconfig"
+
 endmenu
Index: drivers/infiniband/Makefile
===================================================================
--- drivers/infiniband/Makefile	(revision 1766)
+++ drivers/infiniband/Makefile	(working copy)
@@ -1,3 +1,4 @@
 obj-$(CONFIG_INFINIBAND)		+= core/
 obj-$(CONFIG_INFINIBAND_MTHCA)		+= hw/mthca/
 obj-$(CONFIG_INFINIBAND_IPOIB)		+= ulp/ipoib/
+obj-$(CONFIG_INFINIBAND_SDP)		+= ulp/sdp/



From roland at topspin.com  Thu Feb 10 19:53:04 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 10 Feb 2005 19:53:04 -0800
Subject: [openib-general] IB now in Ubuntu
Message-ID: <52d5v792hr.fsf@topspin.com>

Ubuntu just pulled in a 2.6.11 snapshot for their next release, which
is due in April:

    http://lists.ubuntu.com/archives/hoary-changes/2005-February/002512.html

I checked and they have enabled IB in their configs.  So now both
Fedora and Ubuntu will be shipping IB.

 - R.


From tduffy at sun.com  Thu Feb 10 19:53:30 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 19:53:30 -0800
Subject: [openib-general] [PATCH] Change AF_INET_SDP to 27
Message-ID: <1108094010.28588.6.camel@duffman>

It appears AF_LLC is already 26 in include/linux/socket.h.  Change SDP
to 27.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_sock.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_sock.h	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_sock.h	(working copy)
@@ -28,7 +28,7 @@
  * functions (e.g bind, connect, etc.) Either AF_INET or AF_INET_SDP can
  * be used with a SDP socket.
  */
-#define AF_INET_SDP   26
+#define AF_INET_SDP   27
 #define PF_INET_SDP   AF_INET_SDP
 /*
  * Socket option level for SDP specific parameters.



From tduffy at sun.com  Thu Feb 10 19:57:43 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 19:57:43 -0800
Subject: [openib-general] [PATCH] Fix init and exit functions in SDP
Message-ID: <1108094263.28588.8.camel@duffman>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -1707,7 +1707,7 @@ static struct net_proto_family _sdp_prot
 /* --------------------------------------------------------------------- */
 /* ========================================================================= */
 /*..sdp_init -- initialize the sdp host module */
-int __init sdp_init(void)
+static int __init sdp_init(void)
 {
 	int result = 0;
 
@@ -1812,7 +1812,7 @@ error_proc:
 
 /* ========================================================================= */
 /*..sdp_exit -- cleanup the sdp host module */
-void sdp_exit(void)
+static void __exit sdp_exit(void)
 {
 	sdp_dbg_init("INIT: SDP module unload.");
 	/*



From tduffy at sun.com  Thu Feb 10 20:01:01 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 20:01:01 -0800
Subject: [openib-general] [PATCH][Take2] Change AF_INET_SDP to 27
In-Reply-To: <1108094010.28588.6.camel@duffman>
References: <1108094010.28588.6.camel@duffman>
Message-ID: <1108094461.28588.10.camel@duffman>

Oops.  I only changed it one place.  Why is it defined in two places?

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_sock.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_sock.h	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_sock.h	(working copy)
@@ -28,7 +28,7 @@
  * functions (e.g bind, connect, etc.) Either AF_INET or AF_INET_SDP can
  * be used with a SDP socket.
  */
-#define AF_INET_SDP   26
+#define AF_INET_SDP   27
 #define PF_INET_SDP   AF_INET_SDP
 /*
  * Socket option level for SDP specific parameters.
Index: drivers/infiniband/ulp/sdp/sdp_inet.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_inet.h	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_inet.h	(working copy)
@@ -27,7 +27,7 @@
 /*
  * constants shared between user and kernel space.
  */
-#define AF_INET_SDP 26		/* SDP socket protocol family */
+#define AF_INET_SDP 27		/* SDP socket protocol family */
 #define AF_INET_STR "AF_INET_SDP"	/* SDP enabled environment variable */
 
 #endif /* _SDP_INET_H */



From tduffy at sun.com  Thu Feb 10 20:15:22 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 20:15:22 -0800
Subject: [openib-general] [PATCH] use module_param() instead of
	MODULE_PARM() in SDP
Message-ID: <1108095322.28588.14.camel@duffman>

use module_param() instead of MODULE_PARM()

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -46,16 +46,16 @@ MODULE_AUTHOR("Libor Michalek");
 MODULE_DESCRIPTION("InfiniBand SDP module");
 MODULE_LICENSE("Dual BSD/GPL");
 
-MODULE_PARM(_proto_family, "i");
-MODULE_PARM(_buff_min, "i");
-MODULE_PARM(_buff_max, "i");
-MODULE_PARM(_conn_size, "i");
-MODULE_PARM(_recv_post_max, "i");
-MODULE_PARM(_recv_buff_max, "i");
-MODULE_PARM(_send_post_max, "i");
-MODULE_PARM(_send_buff_max, "i");
-MODULE_PARM(_send_usig_max, "i");
-MODULE_PARM(debug_level, "i");
+module_param(_proto_family, int, 0);
+module_param(_buff_min, int, 0);
+module_param(_buff_max, int, 0);
+module_param(_conn_size, int, 0);
+module_param(_recv_post_max, int, 0);
+module_param(_recv_buff_max, int, 0);
+module_param(_send_post_max, int, 0);
+module_param(_send_buff_max, int, 0);
+module_param(_send_usig_max, int, 0);
+module_param(debug_level, int, 0);
 
 /*
  * socket structure relevant fields:



From tduffy at sun.com  Thu Feb 10 22:04:23 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 10 Feb 2005 22:04:23 -0800
Subject: [openib-general] enhanced libsdp
Message-ID: <1108101863.28588.17.camel@duffman>

At the workshop this week, Libor mentioned to me that somebody (can't
remember who, maybe at Mellanox?) had modified libsdp to create a
transparent connect() and listen().  It would be *really cool* if we
could get this into the repository.  A pointer to the work would be ok
too.

Thanks,

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050210/6fd9acdd/attachment.sig>

From ebiederman at lnxi.com  Thu Feb 10 23:36:17 2005
From: ebiederman at lnxi.com (Eric W. Biederman)
Date: 11 Feb 2005 00:36:17 -0700
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <C98692FD98048C41885E0B0FACD9DFB86EB15B@exnane01.hq.netapp.com>
References: <C98692FD98048C41885E0B0FACD9DFB86EB15B@exnane01.hq.netapp.com>
Message-ID: <m34qgjblam.fsf@maxwell.lnxi.com>

"Kanevsky, Arkady" <Arkady.Kanevsky at netapp.com> writes:

> For kDAPL:
> The iSER has been submitted to Open Ib by Voltaire already.
> NFS-RDMA is at http://sourceforge.net/projects/nfs-rdma/.
> 
> For uDAPL: Oracle, DB2 and MPI.
> I am not aware if there is an open source MPI version on uDAPL.
> 
> These are publicly known.
> 
> As far as changing the uDAPL or kDAPL APIs.
> There are application already writen to them.

uDAPL at the API level is safe, it is in user space and it is not a
kernel concern.

Looking at just the API header files kDAPL is a non-starter for
kernel inclusion.

> There are implementation of these APIs on other platforms besides Linux.
> It is in nobody's interest to splinter the user community.

So why are these not sane extensions to the standard sockets APIs?
I do agree it does not make sense to further fragment things.

> We need the same API on all platforms.
In the kernel not a chance, that doesn't even make sense.

> If there is a good technical reason to change some specific APIs we
> should consider it.
> But the "burn the spec" approach is not a rationale one.

Try do a sensible implementation, instead of burn the spec....

> As far as other transport. As people already mentioned iWARP (IETF
> RDDP).

Just skimming the DAPL docs DAPL does not yet appear to service
anything except IB.


> It is still no ready so we will start with gen2.
> But lets not loose site of what DAPL brings:
> OS independent,
> Transport independent,
> RDMA APIs!!!

At user level that is probably great, if the abstraction layer is not
too heavy.  Except that MPI is so heavy I would wonder what the point of
doing all that when MPI already provides it.  

Inside an OS independence is nonsense.  Sorry kDAPL.

......

With that said there is a fundamental issue with network packet reception
in IP.  A packet first must be DMA'd to memory to be examined and then
copied to it's user for a total of 3 copies through the memory bus to
get anywhere.  How to solve that problem while still being robust
against denial of service attacks, and the other vagaries of hostile
public networks and not going the insane path of TCP offload is a
challenge.

For dealing with very fast networks the number of memory copies
start becoming a fundamental limit on how quickly things can
go.  Currently 4x IB runs at about a 1/3 of the memory bandwidth
of a modern memory controller, when you look at it bidirectionally.
As network interfaces continue their exponential increase in
speed this will only get worse.

The various flavors of RDMA seem to be a serious stab at fixing
the memory copy problem.  How well IP/RDMA or IB/RDMA will actually
fix the general problem I don't know, but it certainly deserves
a look, and probably a serious kernel design discussion.

Beyond that from a practical user level the gen2 stack is very
much easier to use than gen1.  But gen2 still needs to be extended
so user space can do good implementations of uDAPL of MPI.  

Getting RDMA kernel support or just full featured MPI support
to user space from the Linux kernel is going to be a challenge.
To get into the mainstream kernel requires a quality work, especially
when thing escalate from a single drivers private hack to the 
next generation version of the sockets API.  Kernel maintainers
can learn and can be convinced of new things but the developers
have to be willing to do the same.  This is where running
the gauntlet, the real flame fest begins.

Good Luck,

Eric




From ebiederman at lnxi.com  Thu Feb 10 23:58:26 2005
From: ebiederman at lnxi.com (Eric W. Biederman)
Date: 11 Feb 2005 00:58:26 -0700
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108067799.7746.14.camel@localhost>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
	<20050210202704.GB29349@esmail.cup.hp.com>
	<1108067799.7746.14.camel@localhost>
Message-ID: <m3zmyba5p9.fsf@maxwell.lnxi.com>

Matt Leininger <mlleinin at hpcn.ca.sandia.gov> writes:

> On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > >   uDAPL - Oracle, MPI
> > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > 
> > Lustre will use Sandia Portals AFAIK.
> > Anyone know what Portals will use?
> > They might directly program to VAPI or something.
> > 
>   There will be a Portals over verbs.  At some point there may be a
> Portals over kDAPL to support both RDMA ethernet and IB.  


Cluster filesystems has already implemented native IB support in Lustre
against gen1. I assume that is a portals.

You might want to ask them about it some time...


Eric


From hch at lst.de  Fri Feb 11 00:14:24 2005
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 11 Feb 2005 09:14:24 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <C98692FD98048C41885E0B0FACD9DFB86EB15B@exnane01.hq.netapp.com>
References: <C98692FD98048C41885E0B0FACD9DFB86EB15B@exnane01.hq.netapp.com>
Message-ID: <20050211081424.GA23443@lst.de>

On Thu, Feb 10, 2005 at 06:49:02PM -0500, Kanevsky, Arkady wrote:
> For kDAPL:
> The iSER has been submitted to Open Ib by Voltaire already.

That's what started this discussion.

> There are application already writen to them.

We have much higher quality standards then standards comittees, so these
kernel-level "applications" will need a major rewrite anyway.

> It is in nobody's interest to splinter the user community.

We don't particularly care about it either, though.

> We need the same API on all platforms.

I don't know who "we" here is, but we as the kernel developer community
certainly disagree.



From yaronh at voltaire.com  Fri Feb 11 06:50:34 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Fri, 11 Feb 2005 16:50:34 +0200
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <35EA21F54A45CB47B879F21A91F4862F391A97@taurus.voltaire.com>

Just to add there is a Lustre NAL over kDAPL in development
And few other application specific protocols done over kDAPL I know of
All those protocols Arkady mentioned can work on both RDMA technologies
and where designed in such a way (I'm familiar with their code and
architecture).

And another grate benefit of kDAPL is the simplification of the Verbs &
Access Layer API, making the ULP's simpler to implement, where a lot of
common functionality is done by a shared library (kDAPL) 
And with a socket like connection establishment flow, etc'

If we want IB to be successful we need to find a way for software
developers to easily build implementations over it (even if not all of
the Applications are open and part of the Linux tree), forcing all Linux
RDMA developers to code to Verbs, CM, SA, ... is probably not the best
approach (the IB spec is many pages as you all know).

For all the guys worried about performance degradation, the latest Verbs
vs DAPL benchmark we did we got 100% the same BW and ONLY 200ns latency
difference.

There is agreement that the current kDAPL API and implementation are not
Linux friendly, and as mentioned before a bunch or people volunteered at
Sonoma to do the work involved in changing it, and agree to make kDAPL
API different than uDAPL and more suitable for kernel.

Yaron

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Kanevsky, Arkady
> Sent: Friday, February 11, 2005 1:49 AM
> To: Libor Michalek; Matt Leininger
> Cc: Christoph Hellwig; openib-general at openib.org; Tom Duffy
> Subject: RE: FW: [openib-general] Minutes from DAPL BOF at OpenIB
Workshop
> 
> For kDAPL:
> The iSER has been submitted to Open Ib by Voltaire already.
> NFS-RDMA is at http://sourceforge.net/projects/nfs-rdma/.
> 
> For uDAPL: Oracle, DB2 and MPI.
> I am not aware if there is an open source MPI version on uDAPL.
> 
> These are publicly known.
> 
> As far as changing the uDAPL or kDAPL APIs.
> There are application already writen to them.
> There are implementation of these APIs on other platforms besides
Linux.
> It is in nobody's interest to splinter the user community.
> We need the same API on all platforms.
> If there is a good technical reason to change some specific APIs we
> should consider it.
> But the "burn the spec" approach is not a rationale one.
> If we need to change implementation or some definitions in header
files
> it is feasible.
> 
> As far as other transport. As people already mentioned iWARP (IETF
> RDDP).
> IBM talked at the BOF about RNIC PI which is being developed as a
level
> of
> abstraction on the lower end to "discover" all the need info about
> RNIC/HCA.
> It is still no ready so we will start with gen2.
> But lets not loose site of what DAPL brings:
> OS independent,
> Transport independent,
> RDMA APIs!!!
> 
> Thanks for jumping on the code so quickly.
> Arkady
> Chair of DAT Collaborative
> 
> Arkady Kanevsky                       email: arkady at netapp.com
> Network Appliance                     phone: 781-768-5395
> 375 Totten Pond Rd.                  Fax: 781-895-1195
> Waltham, MA 02451-2010          central phone: 781-768-5300
> 
> 
> 
> > -----Original Message-----
> > From: Libor Michalek [mailto:libor at topspin.com]
> > Sent: Thursday, February 10, 2005 4:03 PM
> > To: Matt Leininger
> > Cc: Christoph Hellwig; openib-general at openib.org; Tom Duffy
> > Subject: Re: FW: [openib-general] Minutes from DAPL BOF at
> > OpenIB Workshop
> >
> >
> > On Thu, Feb 10, 2005 at 12:36:39PM -0800, Matt Leininger wrote:
> > > On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > > > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > > > >   uDAPL - Oracle, MPI
> > > > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > > >
> > > > Lustre will use Sandia Portals AFAIK.
> > > > Anyone know what Portals will use?
> > > > They might directly program to VAPI or something.
> > > >
> > >   There will be a Portals over verbs.  At some point there may be
a
> > > Portals over kDAPL to support both RDMA ethernet and IB.
> >
> >   Yup, that's one of the bigger questions, can it abstract
> > away the differences between two different RDMA technologies?
> > Having a RDMA ethernet and IB providers for kDAPL is
> > insufficient, one would need to show an actual, non-trivial,
> > protocol that works ontop of either provider with no, or
> > little, modification/ifdef'ing.
> >
> > -Libor
> >
> >
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-> general
> >
> > To
> > unsubscribe, please visit
> > http://openib.org/mailman/listinfo/openib-general
> >
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-
> general


From blist at aon.at  Fri Feb 11 07:31:39 2005
From: blist at aon.at (Bernhard Fischer)
Date: Fri, 11 Feb 2005 16:31:39 +0100
Subject: [openib-general] opensm segfaults after CL_INSUFFICIENT_MEMORY
In-Reply-To: <1108067205.4649.19.camel@localhost.localdomain>
References: <20050210194404.GB21097@aon.at>
	<1108067205.4649.19.camel@localhost.localdomain>
Message-ID: <20050211153139.GA27513@aon.at>

On Thu, Feb 10, 2005 at 03:26:45PM -0500, Hal Rosenstock wrote:
>On Thu, 2005-02-10 at 14:44, Bernhard Fischer wrote:
>> Hi,
>> 
>> I'm seeing the segfault below when i try to run opensm.

>
>> Any ideas?
>> (gdb) run
>> Starting program: /usr/local/ib/bin/opensm 
>> [Thread debugging using libthread_db enabled]
>> [New Thread -1209392032 (LWP 7991)]
>> __init: failed to create timer provider status (CL_INSUFFICIENT_MEMORY) 
>
>First malloc is failing to create the timer
>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread -1209392032 (LWP 7991)]
>> 0xb7f0b597 in memset () from /lib/tls/libc.so.6

>Initialization continues (not sure it should) and attempting to clear a

I'd say it should not but exit.

>static object fails. Even if initialization didn't continue, opensm
>would not run properly.
>
>The first problem is why the malloc fails for the timer provider. The
>second problem is pretty strange as the system callback object is local
>and should not be an address that causes a segmentation violation.
>
>Is this built with the autotools version ?

Disregard the second problem, pathological compiler settings (wrong
regparm for that system).

Please do, however, fix the path for ENOMEM.
-------------- next part --------------
diff -X ./excl -rdup ./gen2.oorig/trunk/src/userspace/management/osm/complib/cl_complib.c ./gen2/trunk/src/userspace/management/osm/complib/cl_complib.c
--- ./gen2.oorig/trunk/src/userspace/management/osm/complib/cl_complib.c	2005-02-08 16:27:20.000000000 +0100
+++ ./gen2/trunk/src/userspace/management/osm/complib/cl_complib.c	2005-02-11 13:23:08.000000000 +0100
@@ -73,6 +73,12 @@ void
 complib_exit(void);
 
 void
+complib_fini(void);
+
+void
+complib_init(void);
+
+void
 __attribute (( constructor ))
 complib_init(void)
 {
@@ -86,7 +92,7 @@ complib_init(void)
 	{
 		cl_msg_out( "__init: failed to init syshelper (%s) \n",
 									CL_STATUS_MSG( status ) );
-
+		exit(1);
 	}
 
 	/*
@@ -98,6 +104,7 @@ complib_init(void)
 	{
 		cl_msg_out( "__init: failed to create timer provider status (%s) \n",
 									CL_STATUS_MSG( status ) );
+		exit(1);
 	}
 
 	/*
@@ -108,9 +115,8 @@ complib_init(void)
 	{
 		cl_msg_out( "__init: failed to initialize syscall back (%s) \n",
 									CL_STATUS_MSG( status ) );
+		exit(1);
 	}
-
-	return;
 }
 
 void
diff -X ./excl -rdup ./gen2.oorig/trunk/src/userspace/management/osm/opensm/main.c ./gen2/trunk/src/userspace/management/osm/opensm/main.c
--- ./gen2.oorig/trunk/src/userspace/management/osm/opensm/main.c	2005-02-08 16:27:11.000000000 +0100
+++ ./gen2/trunk/src/userspace/management/osm/opensm/main.c	2005-02-11 15:45:46.000000000 +0100
@@ -74,6 +74,7 @@ osm_opensm_t osm;
 volatile int osm_exit_flag = 0;
 
 #define GUID_ARRAY_SIZE 64
+#define INVALID_GUID (0xFFFFFFFFFFFFFFFFULL) /* 1 ? */
 
 /**********************************************************************
  **********************************************************************/
@@ -101,13 +102,13 @@ show_usage(void)
           "          -d5  - increase vendor debug level.\n"
           "          -d10.. Put OpenSM in testability mode.\n"
           "          Without -d, no debug options are enabled.\n\n" );
-  printf( "-g <GUID in hex>\n"
-          "--guid <GUID in hex>\n"
+  printf( "-g<[=]GUID in hex>\n"
+          "--guid=<GUID in hex>\n"
           "          This option specifies the local port GUID value\n"
           "          with which OpenSM should bind.  OpenSM may be\n"
           "          bound to 1 port at a time.\n"
-          "          Without -g, OpenSM displays a menu of possible\n"
-          "          port GUIDs and waits for user input.\n\n" );
+          "          Without a guid, OpenSM displays a list of possible\n"
+          "          port GUIDs and exits.\n\n" );
   printf( "-h\n"
           "--help\n"
           "          Display this usage info then exit.\n\n" );
@@ -229,6 +230,36 @@ show_menu(void)
 
 /**********************************************************************
  **********************************************************************/
+static void print_all_guids(IN osm_opensm_t *p_osm);
+static void
+print_all_guids(
+  IN osm_opensm_t *p_osm )
+{
+	ib_api_status_t status;
+	uint32_t num_ports = GUID_ARRAY_SIZE;
+	ib_port_attr_t attr_array[GUID_ARRAY_SIZE];
+	int i;
+
+	/*
+		Call the transport layer for a list of local port
+		GUID values.
+	*/
+	status = osm_vendor_get_all_port_attr(p_osm->p_vendor, attr_array, &num_ports );
+	if ( status != IB_SUCCESS )
+	{
+		printf( "\nError from osm_vendor_get_all_port_attr (%x)\n", status);
+		return;
+	}
+
+	printf("\nListing GUIDs:\n");
+	for (i = 1; i < num_ports; i++) { /* excluding logical mgmnt port */
+		printf("Port %i: 0x%"PRIx64"\n", i, cl_hton64(attr_array[i].port_guid));
+	}
+}
+
+
+/**********************************************************************
+ **********************************************************************/
 ib_net64_t
 get_port_guid(
   IN osm_opensm_t *p_osm, uint64_t port_guid )
@@ -242,6 +273,7 @@ get_port_guid(
 		Call the transport layer for a list of local port
 		GUID values.
 	*/
+// "local ports" is(?) phys, shouldn't this exclude port 0 then ?
 	status = osm_vendor_get_all_port_attr(p_osm->p_vendor, attr_array, &num_ports );
 	if ( status != IB_SUCCESS )
 	{
@@ -353,11 +385,11 @@ main(
   boolean_t          run_once_flag = FALSE;
   boolean_t          mem_track = FALSE;
   int32_t		vendor_debug = 0;
-  uint32_t             next_option;
+  int32_t             next_option;
   unsigned int         exitTimeout;
   char *ignore_guids_file_name = NULL;
   uint32_t             val;
-  const char * const short_option = "i:f:d:g:l:s:t:vVhor";
+  const char * const short_option = "i:f:d:g::l:s:t:vVhor";
   /*
     In the array below, the 2nd parameter specified the number
     of arguments as follows:
@@ -368,7 +400,7 @@ main(
   const struct option long_option[] =
     {
       {  "debug", 1, NULL, 'd'},
-      {  "guid",     1, NULL, 'g'},
+      {  "guid",  2, NULL, 'g'},
       {  "ignore_guids", 1,   NULL, 'i'},
       {  "lmc",      1, NULL, 'l'},
       {  "sweep", 1, NULL, 's'},
@@ -437,8 +469,11 @@ main(
       /*
         Specifies port guid with which to bind.
       */
-      guid = cl_hton64( strtoull( optarg, NULL, 16 ));
-      printf(" Guid <0x%"PRIx64">\n", cl_hton64( guid ));
+      if (optarg) {
+	guid = cl_hton64( strtoull( optarg, NULL, 16 ));
+	printf(" Guid <0x%"PRIx64">\n", cl_hton64( guid ));
+      } else
+	guid = INVALID_GUID;
       break;
 
     case 's':
@@ -567,6 +602,7 @@ main(
 
     case 'h':
     case '?':
+    case ':':
       show_usage();
       break;
 
@@ -609,7 +645,13 @@ main(
     complib_exit();
     return( status );
   }
-  
+
+  if (cl_hton64(guid) == cl_hton64(INVALID_GUID)) {
+	print_all_guids( &osm );
+	complib_exit();
+	return( status );
+  }
+
   /*
     If the user didn't specify a GUID on the command line,
     then get a port GUID value with which to bind.

From halr at voltaire.com  Fri Feb 11 07:49:08 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Feb 2005 10:49:08 -0500
Subject: [openib-general] opensm segfaults after CL_INSUFFICIENT_MEMORY
In-Reply-To: <20050211153139.GA27513@aon.at>
References: <20050210194404.GB21097@aon.at>
	<1108067205.4649.19.camel@localhost.localdomain>
	<20050211153139.GA27513@aon.at>
Message-ID: <1108136947.7310.13.camel@localhost.localdomain>

On Fri, 2005-02-11 at 10:31, Bernhard Fischer wrote:
> On Thu, Feb 10, 2005 at 03:26:45PM -0500, Hal Rosenstock wrote:
> >Initialization continues (not sure it should) and attempting to clear a
> 
> I'd say it should not but exit.
> 
> >static object fails. Even if initialization didn't continue, opensm
> >would not run properly.
> >
> >The first problem is why the malloc fails for the timer provider. The
> >second problem is pretty strange as the system callback object is local
> >and should not be an address that causes a segmentation violation.
> >
> >Is this built with the autotools version ?
> 
> Disregard the second problem, pathological compiler settings (wrong
> regparm for that system).
> 
> Please do, however, fix the path for ENOMEM.

I presume you mean exiting on failures during initialization (of the
component library) ? 

BTW, are you now past these issues ?

-- Hal



From blist at aon.at  Fri Feb 11 08:08:34 2005
From: blist at aon.at (Bernhard Fischer)
Date: Fri, 11 Feb 2005 17:08:34 +0100
Subject: [openib-general] opensm segfaults after CL_INSUFFICIENT_MEMORY
In-Reply-To: <1108136947.7310.13.camel@localhost.localdomain>
References: <20050210194404.GB21097@aon.at>
	<1108067205.4649.19.camel@localhost.localdomain>
	<20050211153139.GA27513@aon.at>
	<1108136947.7310.13.camel@localhost.localdomain>
Message-ID: <20050211160834.GB27513@aon.at>

On Fri, Feb 11, 2005 at 10:49:08AM -0500, Hal Rosenstock wrote:
>On Fri, 2005-02-11 at 10:31, Bernhard Fischer wrote:
>> On Thu, Feb 10, 2005 at 03:26:45PM -0500, Hal Rosenstock wrote:
>> >Initialization continues (not sure it should) and attempting to clear a
>> 
>> I'd say it should not but exit.
>> 
>> >static object fails. Even if initialization didn't continue, opensm
>> >would not run properly.
>> >
>> >The first problem is why the malloc fails for the timer provider. The
>> >second problem is pretty strange as the system callback object is local
>> >and should not be an address that causes a segmentation violation.
>> >
>> >Is this built with the autotools version ?
>> 
>> Disregard the second problem, pathological compiler settings (wrong
>> regparm for that system).
>> 
>> Please do, however, fix the path for ENOMEM.
>
>I presume you mean exiting on failures during initialization (of the
>component library) ? 

Yes.
>
>BTW, are you now past these issues ?
opensm does work now.

I'm still getting ib_mthca 0000:04:00.0: CQ overrun on CQN 00000082
on the sender after having transmitted 5MB to 50MB.

I was only playing with opensm while waiting for an answer to the CQ
overrun...




From roland at topspin.com  Fri Feb 11 08:10:52 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 11 Feb 2005 08:10:52 -0800
Subject: [openib-general] opensm segfaults after CL_INSUFFICIENT_MEMORY
In-Reply-To: <20050211160834.GB27513@aon.at> (Bernhard Fischer's message of
	"Fri, 11 Feb 2005 17:08:34 +0100")
References: <20050210194404.GB21097@aon.at>
	<1108067205.4649.19.camel@localhost.localdomain>
	<20050211153139.GA27513@aon.at>
	<1108136947.7310.13.camel@localhost.localdomain>
	<20050211160834.GB27513@aon.at>
Message-ID: <52650z84c3.fsf@topspin.com>

    Bernhard> I'm still getting ib_mthca 0000:04:00.0: CQ overrun on
    Bernhard> CQN 00000082 on the sender after having transmitted 5MB
    Bernhard> to 50MB.

    Bernhard> I was only playing with opensm while waiting for an
    Bernhard> answer to the CQ overrun...

I didn't see this in your earlier email.  This almost certainly means
you need to upgrade your HCA firmware.  Tavor FW 3.3.1 and Arbel FW
4.6.1 have a bug that will cause this error message; it is fixed in FW
versions 3.3.2/4.6.2.

 - Roland


From halr at voltaire.com  Fri Feb 11 08:04:29 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Feb 2005 11:04:29 -0500
Subject: [openib-general] [PATCH] use standard path names in automake
	for managment
In-Reply-To: <1107819989.6917.16.camel@duffman>
References: <1107819989.6917.16.camel@duffman>
Message-ID: <1108137868.7310.48.camel@localhost.localdomain>

Hi Tom,

On Mon, 2005-02-07 at 18:46, Tom Duffy wrote:
> Use standard paths for autotools.  

There are many complex packages which use /usr/local/XXX rather than
just /usr/local as the "standard" path so we are not sure whether to
move this back to /usr/local. In any case, if this is to be done, we
would prefer to defer it until things are more ready for inclusion.

-- Hal





From tduffy at sun.com  Fri Feb 11 08:35:27 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 08:35:27 -0800
Subject: [openib-general] [PATCH] use standard path names in
	automake	for managment
In-Reply-To: <1108137868.7310.48.camel@localhost.localdomain>
References: <1107819989.6917.16.camel@duffman>
	<1108137868.7310.48.camel@localhost.localdomain>
Message-ID: <1108139727.28588.20.camel@duffman>

On Fri, 2005-02-11 at 11:04 -0500, Hal Rosenstock wrote:
> There are many complex packages which use /usr/local/XXX rather than
> just /usr/local as the "standard" path so we are not sure whether to
> move this back to /usr/local. In any case, if this is to be done, we
> would prefer to defer it until things are more ready for inclusion.

The point is, you can always make that happen with the appropriate
options to configure.  You can document this.

Please still consider applying the patch.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050211/6fbbadb5/attachment.sig>

From tduffy at sun.com  Fri Feb 11 08:47:08 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 08:47:08 -0800
Subject: [openib-general] [PATCH] get SDP building on sparc64
Message-ID: <1108140428.28588.22.camel@duffman>

Singed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_main.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_main.h	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
@@ -61,6 +61,8 @@
 #include <asm/byteorder.h>
 #include <asm/pgtable.h>
 #include <asm/io.h>
+#include <asm/dma.h>
+#include <asm/dma-mapping.h>
 
 #include <linux/version.h>
 #include <linux/slab.h>



From hch at lst.de  Fri Feb 11 09:05:26 2005
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 11 Feb 2005 18:05:26 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F391A97@taurus.voltaire.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A97@taurus.voltaire.com>
Message-ID: <20050211170526.GA31193@lst.de>

On Fri, Feb 11, 2005 at 04:50:34PM +0200, Yaron Haviv wrote:
> Just to add there is a Lustre NAL over kDAPL in development

As mention before we don't care about Lustre as it's not freely
available.

> And another grate benefit of kDAPL is the simplification of the Verbs &
> Access Layer API, making the ULP's simpler to implement, where a lot of
> common functionality is done by a shared library (kDAPL) 
> And with a socket like connection establishment flow, etc'

shared code is always a good thing.  But designing code to be shared
for not yet existing consumers can end up in a big mess very soon - 
as can be seen very well from the DAPL example.



From iod00d at hp.com  Fri Feb 11 09:55:26 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 11 Feb 2005 09:55:26 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050211170526.GA31193@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A97@taurus.voltaire.com>
	<20050211170526.GA31193@lst.de>
Message-ID: <20050211175526.GA574@esmail.cup.hp.com>

On Fri, Feb 11, 2005 at 06:05:26PM +0100, Christoph Hellwig wrote:
> On Fri, Feb 11, 2005 at 04:50:34PM +0200, Yaron Haviv wrote:
> > Just to add there is a Lustre NAL over kDAPL in development
> 
> As mention before we don't care about Lustre as it's not freely
> available.

Christoph,
please be careful with use of "we".

I ignore references to Lustre since I'm not working on it.
But HPC community watching this list cares alot about Lustre
and more than a few are involved in Lustre developement. I don't
need to alienate them in order to get what I think is right.

hth,
grant


From halr at voltaire.com  Fri Feb 11 09:46:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Feb 2005 12:46:00 -0500
Subject: [openib-general] [PATCH] OpenSM: Fix scope of well known MC group
	records
Message-ID: <1108143959.7310.461.camel@localhost.localdomain>

OpenSM: Scope of well known multicast group records needs to match the
scope in the MGID

Index: osm_sa.c
===================================================================
--- osm_sa.c	(revision 1745)
+++ osm_sa.c	(working copy)
@@ -531,7 +531,8 @@
   mc_rec.rate = 0x3; /* 10Gb/sec */
   mc_rec.pkt_life = OSM_DEFAULT_SUBNET_TIMEOUT;
   mc_rec.sl_flow_hop = OSM_DEFAULT_SL <<28;
-  mc_rec.scope_state = 1;
+  /* Note: scope needs to be consistent with MGID */
+  mc_rec.scope_state = 0x21;
 
   osm_sa_add_well_known_mc_record(&p_sa->mcmr_rcv, &mc_rec);
 





From libor at topspin.com  Fri Feb 11 09:57:04 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 09:57:04 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1108093196.28588.3.camel@duffman>;
	from tduffy@sun.com on Thu, Feb 10, 2005 at 07:39:56PM -0800
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>
	<1108093196.28588.3.camel@duffman>
Message-ID: <20050211095704.A20398@topspin.com>

On Thu, Feb 10, 2005 at 07:39:56PM -0800, Tom Duffy wrote:
> On Thu, 2005-02-10 at 19:09 -0800, Libor Michalek wrote:
> >   OK, I've gone with option number 3. Here is a one or two line
> > description of each file, and a TODO list. Both are checked in
> > along with the code in the infiniband/ulp/sdp directory.
> 
> Of course, you will need this patch to plumb SDP into the build.

  I was going to submit this as well, but decided to hold off since
the connection manager has yet to be plumbed into the build. Once
Sean adds the CM, then I'll apply this patch.

Thanks.

-Libor

> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/Kconfig
> ===================================================================
> --- drivers/infiniband/Kconfig	(revision 1766)
> +++ drivers/infiniband/Kconfig	(working copy)
> @@ -11,4 +11,6 @@ source "drivers/infiniband/hw/mthca/Kcon
>  
>  source "drivers/infiniband/ulp/ipoib/Kconfig"
>  
> +source "drivers/infiniband/ulp/sdp/Kconfig"
> +
>  endmenu
> Index: drivers/infiniband/Makefile
> ===================================================================
> --- drivers/infiniband/Makefile	(revision 1766)
> +++ drivers/infiniband/Makefile	(working copy)
> @@ -1,3 +1,4 @@
>  obj-$(CONFIG_INFINIBAND)		+= core/
>  obj-$(CONFIG_INFINIBAND_MTHCA)		+= hw/mthca/
>  obj-$(CONFIG_INFINIBAND_IPOIB)		+= ulp/ipoib/
> +obj-$(CONFIG_INFINIBAND_SDP)		+= ulp/sdp/


From halr at voltaire.com  Fri Feb 11 09:52:04 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Feb 2005 12:52:04 -0500
Subject: [openib-general] OpenSM/complib: Handle complib initialization
	failures better
Message-ID: <1108144024.7310.466.camel@localhost.localdomain>

OpenSM/complib: Handle complib initialization failures better (e.g.
cleanup and exit gracefully)

Index: cl_complib.c
===================================================================
--- cl_complib.c	(revision 1745)
+++ cl_complib.c	(working copy)
@@ -86,7 +86,7 @@
 	{
 		cl_msg_out( "__init: failed to init syshelper (%s) \n",
 									CL_STATUS_MSG( status ) );
-
+		exit(1);
 	}
 
 	/*
@@ -98,6 +98,8 @@
 	{
 		cl_msg_out( "__init: failed to create timer provider status (%s) \n",
 									CL_STATUS_MSG( status ) );
+		__cl_sys_callback_destroy();
+		exit(1);
 	}
 
 	/*
@@ -108,6 +110,9 @@
 	{
 		cl_msg_out( "__init: failed to initialize syscall back (%s) \n",
 									CL_STATUS_MSG( status ) );
+		__cl_timer_prov_destroy();
+		__cl_sys_callback_destroy();
+		exit(1);
 	}
 
 	return;





From hch at lst.de  Fri Feb 11 10:05:08 2005
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 11 Feb 2005 19:05:08 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050211175526.GA574@esmail.cup.hp.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A97@taurus.voltaire.com>
	<20050211170526.GA31193@lst.de>
	<20050211175526.GA574@esmail.cup.hp.com>
Message-ID: <20050211180508.GA32188@lst.de>

On Fri, Feb 11, 2005 at 09:55:26AM -0800, Grant Grundler wrote:
> I ignore references to Lustre since I'm not working on it.
> But HPC community watching this list cares alot about Lustre
> and more than a few are involved in Lustre developement. I don't
> need to alienate them in order to get what I think is right.

we is kernel developers here.  We don't add APIs for code that has
not chance to end up in the kernel tree.


From halr at voltaire.com  Fri Feb 11 09:55:42 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Feb 2005 12:55:42 -0500
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <20050211095704.A20398@topspin.com>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman> <20050210190904.C11665@topspin.com>
	<1108093196.28588.3.camel@duffman> <20050211095704.A20398@topspin.com>
Message-ID: <1108144541.7310.501.camel@localhost.localdomain>

On Fri, 2005-02-11 at 12:57, Libor Michalek wrote:
>   I was going to submit this as well, but decided to hold off since
> the connection manager has yet to be plumbed into the build. Once
> Sean adds the CM, then I'll apply this patch.

Don't we want to be a little bit careful here in terms of being able to
continue to derive patches on the pieces already submitted to kernel.org
until we decide this is ready to go forward ? (Guess that might be when
these build patches would be applied).

-- Hal



From tduffy at sun.com  Fri Feb 11 10:10:51 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 10:10:51 -0800
Subject: [openib-general] [PATCH] use C99 struct initilization syntax in SDP
Message-ID: <1108145451.28588.33.camel@duffman>

use C99, otherwise sparse complains

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_proc.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_proc.c	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_proc.c	(working copy)
@@ -81,40 +81,40 @@ static int _sdp_proc_read_parse(char *pa
  */
 static struct sdpc_proc_ent _file_entry_list[SDP_PROC_ENTRIES] = {
       {
-	      entry: NULL,
-	      type:  SDP_PROC_ENTRY_MAIN_BUFF,
-	      name:  "buffer_pool",
-	      read:  sdp_proc_dump_buff_pool,
+	      .entry = NULL,
+	      .type  = SDP_PROC_ENTRY_MAIN_BUFF,
+	      .name  = "buffer_pool",
+	      .read  = sdp_proc_dump_buff_pool,
       },
       {
-	      entry: NULL,
-	      type:  SDP_PROC_ENTRY_MAIN_CONN,
-	      name:  "conn_main",
-	      read:  sdp_proc_dump_conn_main,
+	      .entry = NULL,
+	      .type  = SDP_PROC_ENTRY_MAIN_CONN,
+	      .name  = "conn_main",
+	      .read  = sdp_proc_dump_conn_main,
       },
       {
-	      entry: NULL,
-	      type:  SDP_PROC_ENTRY_DATA_CONN,
-	      name:  "conn_data",
-	      read:  sdp_proc_dump_conn_data,
+	      .entry = NULL,
+	      .type  = SDP_PROC_ENTRY_DATA_CONN,
+	      .name  = "conn_data",
+	      .read  = sdp_proc_dump_conn_data,
       },
       {
-	      entry: NULL,
-	      type:  SDP_PROC_ENTRY_RDMA_CONN,
-	      name:  "conn_rdma",
-	      read:  sdp_proc_dump_conn_rdma,
+	      .entry = NULL,
+	      .type  = SDP_PROC_ENTRY_RDMA_CONN,
+	      .name  = "conn_rdma",
+	      .read  = sdp_proc_dump_conn_rdma,
       },
       {
-	      entry: NULL,
-	      type:  SDP_PROC_ENTRY_OPT_CONN,
-	      name:  "opt_conn",
-	      read:  sdp_proc_dump_conn_sopt,
+	      .entry = NULL,
+	      .type  = SDP_PROC_ENTRY_OPT_CONN,
+	      .name  = "opt_conn",
+	      .read  = sdp_proc_dump_conn_sopt,
       },
       {
-	      entry: NULL,
-	      type:  SDP_PROC_ENTRY_ROOT_TABLE,
-	      name:  "devices",
-	      read:  sdp_proc_dump_device,
+	      .entry = NULL,
+	      .type  = SDP_PROC_ENTRY_ROOT_TABLE,
+	      .name  = "devices",
+	      .read  = sdp_proc_dump_device,
       },
 } ;
 
Index: drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -1622,24 +1521,24 @@ static int _sdp_inet_shutdown(struct soc
 /*                                                                       */
 /* --------------------------------------------------------------------- */
 struct proto_ops _lnx_stream_ops = {
-      family:      AF_INET_SDP,
-      release:    _sdp_inet_release,
-      bind:       _sdp_inet_bind,
-      connect:    _sdp_inet_connect,
-      listen:     _sdp_inet_listen,
-      accept:     _sdp_inet_accept,
-      sendmsg:     sdp_inet_send,
-      recvmsg:     sdp_inet_recv,
-      getname:    _sdp_inet_getname,
-      poll:       _sdp_inet_poll,
-      setsockopt: _sdp_inet_setopt,
-      getsockopt: _sdp_inet_getopt,
-      shutdown:   _sdp_inet_shutdown,
-      ioctl:      _sdp_inet_ioctl,
-      sendpage:    sock_no_sendpage,
-      socketpair:  sock_no_socketpair,
-      mmap:        sock_no_mmap,
-      owner:       THIS_MODULE,
+      .family     =  AF_INET_SDP,
+      .release    = _sdp_inet_release,
+      .bind       = _sdp_inet_bind,
+      .connect    = _sdp_inet_connect,
+      .listen     = _sdp_inet_listen,
+      .accept     = _sdp_inet_accept,
+      .sendmsg    =  sdp_inet_send,
+      .recvmsg    =  sdp_inet_recv,
+      .getname    = _sdp_inet_getname,
+      .poll       = _sdp_inet_poll,
+      .setsockopt = _sdp_inet_setopt,
+      .getsockopt = _sdp_inet_getopt,
+      .shutdown   = _sdp_inet_shutdown,
+      .ioctl      = _sdp_inet_ioctl,
+      .sendpage   =  sock_no_sendpage,
+      .socketpair =  sock_no_socketpair,
+      .mmap       =  sock_no_mmap,
+      .owner      =  THIS_MODULE,
 }; /* _lnx_stream_ops */
 
 
@@ -1695,9 +1594,9 @@ static int _sdp_inet_create(struct socke
 /*                                                                       */
 /* --------------------------------------------------------------------- */
 static struct net_proto_family _sdp_proto = {
-      family:   AF_INET_SDP,
-      create:  _sdp_inet_create,
-      owner:    THIS_MODULE,
+      .family =  AF_INET_SDP,
+      .create = _sdp_inet_create,
+      .owner  =  THIS_MODULE,
 }; /* _pf_family */
 
 /* --------------------------------------------------------------------- */



From tduffy at sun.com  Fri Feb 11 10:51:00 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 10:51:00 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1108144541.7310.501.camel@localhost.localdomain>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman> <20050210190904.C11665@topspin.com>
	<1108093196.28588.3.camel@duffman> <20050211095704.A20398@topspin.com>
	<1108144541.7310.501.camel@localhost.localdomain>
Message-ID: <1108147860.28588.36.camel@duffman>

On Fri, 2005-02-11 at 12:55 -0500, Hal Rosenstock wrote:
> On Fri, 2005-02-11 at 12:57, Libor Michalek wrote:
> >   I was going to submit this as well, but decided to hold off since
> > the connection manager has yet to be plumbed into the build. Once
> > Sean adds the CM, then I'll apply this patch.
> 
> Don't we want to be a little bit careful here in terms of being able to
> continue to derive patches on the pieces already submitted to kernel.org
> until we decide this is ready to go forward ? (Guess that might be when
> these build patches would be applied).

Well, I don't think the Makefiles ore Kconfig's will be changing for
upstream submission, so as long as Roland is careful to only diff .[ch]
files that are already in lkml, it shouldn't be a big deal.  I think for
the ease of use for everybody that we plumb in cm and sdp.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050211/ea5a8fe9/attachment.sig>

From tduffy at sun.com  Fri Feb 11 10:58:18 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 10:58:18 -0800
Subject: [openib-general] [PATCH][fixed] get SDP building on sparc64
In-Reply-To: <1108140428.28588.22.camel@duffman>
References: <1108140428.28588.22.camel@duffman>
Message-ID: <1108148298.28588.41.camel@duffman>

The first patch did get SDP building on sparc64, but it would not build
on x86_64.  This is the correct fix that gets it to build on both
architectures (as well as x86_32).

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_main.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_main.h	(revision 1766)
+++ drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
@@ -79,6 +79,7 @@
 #include <linux/ip.h>
 #include <linux/pci.h>
 #include <linux/random.h>
+#include <linux/dma-mapping.h>
 
 #include <net/sock.h>
 #include <net/route.h>



From roland at topspin.com  Fri Feb 11 11:00:04 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 11 Feb 2005 11:00:04 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1108147860.28588.36.camel@duffman> (Tom Duffy's message of
	"Fri, 11 Feb 2005 10:51:00 -0800")
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman> <20050210190904.C11665@topspin.com>
	<1108093196.28588.3.camel@duffman> <20050211095704.A20398@topspin.com>
	<1108144541.7310.501.camel@localhost.localdomain>
	<1108147860.28588.36.camel@duffman>
Message-ID: <52wtte7wi3.fsf@topspin.com>

    Hal> Don't we want to be a little bit careful here in terms of
    Hal> being able to continue to derive patches on the pieces
    Hal> already submitted to kernel.org until we decide this is ready
    Hal> to go forward ? (Guess that might be when these build patches
    Hal> would be applied).

    Tom> Well, I don't think the Makefiles ore Kconfig's will be
    Tom> changing for upstream submission, so as long as Roland is
    Tom> careful to only diff .[ch] files that are already in lkml, it
    Tom> shouldn't be a big deal.  I think for the ease of use for
    Tom> everybody that we plumb in cm and sdp.

Agreed.  It's pretty easy for me to keep Kconfig/Makefile changes
straight now that we have the current stuff upstream.  I see no
problem with hooking up the CM and SDP.  (BTW it looks like SDP should
select the CM once there is a CM config option; also the SDP
dependency on INET is redundant because IPOIB alread depends on that)

 - R.


From iod00d at hp.com  Fri Feb 11 11:07:19 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 11 Feb 2005 11:07:19 -0800
Subject: [openib-general] [PATCH] get SDP building on sparc64
In-Reply-To: <1108140428.28588.22.camel@duffman>
References: <1108140428.28588.22.camel@duffman>
Message-ID: <20050211190719.GH574@esmail.cup.hp.com>

On Fri, Feb 11, 2005 at 08:47:08AM -0800, Tom Duffy wrote:
> Singed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_main.h
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_main.h	(revision 1766)
> +++ drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
> @@ -61,6 +61,8 @@
>  #include <asm/byteorder.h>
>  #include <asm/pgtable.h>
>  #include <asm/io.h>
> +#include <asm/dma.h>
> +#include <asm/dma-mapping.h>

please use linux/dma-mapping.h instead

grant


From iod00d at hp.com  Fri Feb 11 11:08:26 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 11 Feb 2005 11:08:26 -0800
Subject: [openib-general] [PATCH] get SDP building on sparc64
In-Reply-To: <20050211190719.GH574@esmail.cup.hp.com>
References: <1108140428.28588.22.camel@duffman>
	<20050211190719.GH574@esmail.cup.hp.com>
Message-ID: <20050211190826.GJ574@esmail.cup.hp.com>

On Fri, Feb 11, 2005 at 11:07:19AM -0800, Grant Grundler wrote:
> > +#include <asm/dma-mapping.h>
> 
> please use linux/dma-mapping.h instead

nm...you already noticed...

grant


From mshefty at ichips.intel.com  Fri Feb 11 11:08:55 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 11 Feb 2005 11:08:55 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <52wtte7wi3.fsf@topspin.com>
References: <20050204112115.A1592@topspin.com>	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>	<1108093196.28588.3.camel@duffman>
	<20050211095704.A20398@topspin.com>	<1108144541.7310.501.camel@localhost.localdomain>	<1108147860.28588.36.camel@duffman>
	<52wtte7wi3.fsf@topspin.com>
Message-ID: <420D02C7.3090708@ichips.intel.com>

Roland Dreier wrote:
>     Hal> Don't we want to be a little bit careful here in terms of
>     Hal> being able to continue to derive patches on the pieces
>     Hal> already submitted to kernel.org until we decide this is ready
>     Hal> to go forward ? (Guess that might be when these build patches
>     Hal> would be applied).
> 
>     Tom> Well, I don't think the Makefiles ore Kconfig's will be
>     Tom> changing for upstream submission, so as long as Roland is
>     Tom> careful to only diff .[ch] files that are already in lkml, it
>     Tom> shouldn't be a big deal.  I think for the ease of use for
>     Tom> everybody that we plumb in cm and sdp.
> 
> Agreed.  It's pretty easy for me to keep Kconfig/Makefile changes
> straight now that we have the current stuff upstream.  I see no
> problem with hooking up the CM and SDP.  (BTW it looks like SDP should
> select the CM once there is a CM config option; also the SDP
> dependency on INET is redundant because IPOIB alread depends on that)

Libor, if you want to include the changes for the CM as part of your 
SDP changes, feel free to do so.  Otherwise, I will submit a patch for 
the CM as soon as possible.

- Sean


From iod00d at hp.com  Fri Feb 11 11:29:51 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 11 Feb 2005 11:29:51 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050211180508.GA32188@lst.de>
References: <35EA21F54A45CB47B879F21A91F4862F391A97@taurus.voltaire.com>
	<20050211170526.GA31193@lst.de>
	<20050211175526.GA574@esmail.cup.hp.com>
	<20050211180508.GA32188@lst.de>
Message-ID: <20050211192951.GN574@esmail.cup.hp.com>

On Fri, Feb 11, 2005 at 07:05:08PM +0100, Christoph Hellwig wrote:
> On Fri, Feb 11, 2005 at 09:55:26AM -0800, Grant Grundler wrote:
> > I ignore references to Lustre since I'm not working on it.
> > But HPC community watching this list cares alot about Lustre
> > and more than a few are involved in Lustre developement. I don't
> > need to alienate them in order to get what I think is right.
> 
> we is kernel developers here.  We don't add APIs for code that has
> not chance to end up in the kernel tree.

The kDAPL (and whatever we make of it) isn't just for Lustre.
Several other consumers of that API were listed.
That's why I choose to ignore Lustre until someone wants to
include an older, public version (1.0.4 is available).

grant


From mshefty at ichips.intel.com  Fri Feb 11 11:32:13 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 11 Feb 2005 11:32:13 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <420D02C7.3090708@ichips.intel.com>
References: <20050204112115.A1592@topspin.com>	<1107545598.5982.34.camel@duffman>	<20050204114207.B1592@topspin.com>	<1107552392.2435.6.camel@duffman>	<20050210190904.C11665@topspin.com>	<1108093196.28588.3.camel@duffman>	<20050211095704.A20398@topspin.com>	<1108144541.7310.501.camel@localhost.localdomain>	<1108147860.28588.36.camel@duffman>	<52wtte7wi3.fsf@topspin.com>
	<420D02C7.3090708@ichips.intel.com>
Message-ID: <420D083D.6050900@ichips.intel.com>

Sean Hefty wrote:

> Roland Dreier wrote:
> 
>>     Hal> Don't we want to be a little bit careful here in terms of
>>     Hal> being able to continue to derive patches on the pieces
>>     Hal> already submitted to kernel.org until we decide this is ready
>>     Hal> to go forward ? (Guess that might be when these build patches
>>     Hal> would be applied).
>>
>>     Tom> Well, I don't think the Makefiles ore Kconfig's will be
>>     Tom> changing for upstream submission, so as long as Roland is
>>     Tom> careful to only diff .[ch] files that are already in lkml, it
>>     Tom> shouldn't be a big deal.  I think for the ease of use for
>>     Tom> everybody that we plumb in cm and sdp.
>>
>> Agreed.  It's pretty easy for me to keep Kconfig/Makefile changes
>> straight now that we have the current stuff upstream.  I see no
>> problem with hooking up the CM and SDP.  (BTW it looks like SDP should
>> select the CM once there is a CM config option; also the SDP
>> dependency on INET is redundant because IPOIB alread depends on that)
> 
> 
> Libor, if you want to include the changes for the CM as part of your SDP 
> changes, feel free to do so.  Otherwise, I will submit a patch for the 
> CM as soon as possible.

I've committed a patch to include the CM as part of the core build.

- Sean


From roland at topspin.com  Fri Feb 11 14:17:16 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 11 Feb 2005 14:17:16 -0800
Subject: [PATCH] fix CM compilation for gcc 2.95 (was Re: [openib-general]
	RFC on SDP checkin)
In-Reply-To: <420D083D.6050900@ichips.intel.com> (Sean Hefty's message of
	"Fri, 11 Feb 2005 11:32:13 -0800")
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman> <20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman> <20050210190904.C11665@topspin.com>
	<1108093196.28588.3.camel@duffman> <20050211095704.A20398@topspin.com>
	<1108144541.7310.501.camel@localhost.localdomain>
	<1108147860.28588.36.camel@duffman> <52wtte7wi3.fsf@topspin.com>
	<420D02C7.3090708@ichips.intel.com>
	<420D083D.6050900@ichips.intel.com>
Message-ID: <52oeeq7ndf.fsf_-_@topspin.com>

gcc 2.95 fails on the current tree with

    cm.c:140: field `path' has incomplete type

Unfortunately, although the C spec says that using just "[]" to
declare arrays at the end of a struct is correct, gcc 2.95 insists on
"[0]".  This patch makes that change.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1772)
+++ infiniband/core/cm.c	(working copy)
@@ -137,7 +137,7 @@ struct cm_work {
 	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
 	u32 local_id;				/* Established */
 	struct ib_cm_event cm_event;
-	struct ib_sa_path_rec path[];
+	struct ib_sa_path_rec path[0];
 };
 
 static inline void cm_deref_id(struct cm_id_private *cm_id_priv)


From mshefty at ichips.intel.com  Fri Feb 11 14:41:30 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 11 Feb 2005 14:41:30 -0800
Subject: [PATCH] fix CM compilation for gcc 2.95 (was Re: [openib-general]
	RFC on SDP checkin)
In-Reply-To: <52oeeq7ndf.fsf_-_@topspin.com>
References: <20050204112115.A1592@topspin.com>	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>	<1108093196.28588.3.camel@duffman>
	<20050211095704.A20398@topspin.com>	<1108144541.7310.501.camel@localhost.localdomain>	<1108147860.28588.36.camel@duffman>
	<52wtte7wi3.fsf@topspin.com>	<420D02C7.3090708@ichips.intel.com>	<420D083D.6050900@ichips.intel.com>
	<52oeeq7ndf.fsf_-_@topspin.com>
Message-ID: <420D349A.70805@ichips.intel.com>

Roland Dreier wrote:

> gcc 2.95 fails on the current tree with
> 
>     cm.c:140: field `path' has incomplete type
> 
> Unfortunately, although the C spec says that using just "[]" to
> declare arrays at the end of a struct is correct, gcc 2.95 insists on
> "[0]".  This patch makes that change.

Thanks!  Applied.  Btw, I'm perfectly fine if you want to apply patches
to the CM.

- Sean


From libor at topspin.com  Fri Feb 11 16:48:49 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 16:48:49 -0800
Subject: [openib-general] [PATCH] use module_param() instead of
	MODULE_PARM() in SDP
In-Reply-To: <1108095322.28588.14.camel@duffman>;
	from tduffy@sun.com on Thu, Feb 10, 2005 at 08:15:22PM -0800
References: <1108095322.28588.14.camel@duffman>
Message-ID: <20050211164849.B20777@topspin.com>

On Thu, Feb 10, 2005 at 08:15:22PM -0800, Tom Duffy wrote:
> use module_param() instead of MODULE_PARM()

 Thanks, I've applied and committed.

-Libor

> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_inet.c
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1766)
> +++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
> @@ -46,16 +46,16 @@ MODULE_AUTHOR("Libor Michalek");
>  MODULE_DESCRIPTION("InfiniBand SDP module");
>  MODULE_LICENSE("Dual BSD/GPL");
>  
> -MODULE_PARM(_proto_family, "i");
> -MODULE_PARM(_buff_min, "i");
> -MODULE_PARM(_buff_max, "i");
> -MODULE_PARM(_conn_size, "i");
> -MODULE_PARM(_recv_post_max, "i");
> -MODULE_PARM(_recv_buff_max, "i");
> -MODULE_PARM(_send_post_max, "i");
> -MODULE_PARM(_send_buff_max, "i");
> -MODULE_PARM(_send_usig_max, "i");
> -MODULE_PARM(debug_level, "i");
> +module_param(_proto_family, int, 0);
> +module_param(_buff_min, int, 0);
> +module_param(_buff_max, int, 0);
> +module_param(_conn_size, int, 0);
> +module_param(_recv_post_max, int, 0);
> +module_param(_recv_buff_max, int, 0);
> +module_param(_send_post_max, int, 0);
> +module_param(_send_buff_max, int, 0);
> +module_param(_send_usig_max, int, 0);
> +module_param(debug_level, int, 0);
>  
>  /*
>   * socket structure relevant fields:
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From libor at topspin.com  Fri Feb 11 16:49:36 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 16:49:36 -0800
Subject: [openib-general] [PATCH] Fix init and exit functions in SDP
In-Reply-To: <1108094263.28588.8.camel@duffman>;
	from tduffy@sun.com on Thu, Feb 10, 2005 at 07:57:43PM -0800
References: <1108094263.28588.8.camel@duffman>
Message-ID: <20050211164936.C20777@topspin.com>

On Thu, Feb 10, 2005 at 07:57:43PM -0800, Tom Duffy wrote:
> Signed-off-by: Tom Duffy <tduffy at sun.com>

Thanks, applied and committed.

-Libor

> Index: drivers/infiniband/ulp/sdp/sdp_inet.c
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1766)
> +++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
> @@ -1707,7 +1707,7 @@ static struct net_proto_family _sdp_prot
>  /* --------------------------------------------------------------------- */
>  /* ========================================================================= */
>  /*..sdp_init -- initialize the sdp host module */
> -int __init sdp_init(void)
> +static int __init sdp_init(void)
>  {
>  	int result = 0;
>  
> @@ -1812,7 +1812,7 @@ error_proc:
>  
>  /* ========================================================================= */
>  /*..sdp_exit -- cleanup the sdp host module */
> -void sdp_exit(void)
> +static void __exit sdp_exit(void)
>  {
>  	sdp_dbg_init("INIT: SDP module unload.");
>  	/*
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From tduffy at sun.com  Fri Feb 11 17:08:41 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 17:08:41 -0800
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <1108168999.17781.28.camel@localhost.localdomain>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
Message-ID: <1108170521.28588.61.camel@duffman>

[ Putting on OpenIB ]

On Fri, 2005-02-11 at 19:43 -0500, Hal Rosenstock wrote:
> Hey Tom,
> 
> On Fri, 2005-02-11 at 19:35, Tom Duffy wrote:
> > Solaris, you need to "ifconfig ibd0 plumb" to see it in ifconfig -a. You
> > are further than me since you see /dev/ibd0.  You might even be there...
> 
> That did it :-) Thanks.
> 
> # ifconfig ibd0 plumb
> # ifconfig -a
> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> 8232 index 1
>         inet 127.0.0.1 netmask ff000000 
> e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu
> 1500 index 2
>         inet 10.0.2.56 netmask ffffff00 broadcast 10.0.2.255
>         ether 0:30:48:27:21:2c 
> ibd0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 252 index 3
>         inet 0.0.0.0 netmask 0 
>         ipib 0:0:4:6:fe:80:0:0:0:0:0:0:0:8:f1:4:3:96:4:e 
> 
> Not sure why the MTU comes up as 252 (other than it is the 256 minimum
> minus 4 bytes overhead).

That is odd.  I don't see this when using IBSRM to manage the network:

ibd1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 3
        inet 192.168.0.78 netmask ffffff00 broadcast 192.168.0.255
        ipib 0:0:0:18:0:0:0:0:0:0:12:34:0:2:c9:1:9:76:56:31 


> It also doesn't like me changing the MTU.
> ifconfig ibd0 mtu 2044
> ifconfig: setifmtu: SIOCSLIFMTU: ibd0: Invalid argument

Huh, Nitin, any ideas?

> But when I lower the MTU on the OpenIB side, I have connectivity:
> 
> >From OpenIB -> Solaris
> ping 192.168.0.2
> PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
> 64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=8.57 ms
> 64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.118 ms
> 64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=0.127 ms
> 
> >From Solaris -> OpenIB
> ping -s 192.168.0.1
> PING 192.168.0.1: 56 data bytes
> 64 bytes from 192.168.0.1: icmp_seq=0. time=0.306 ms
> 64 bytes from 192.168.0.1: icmp_seq=1. time=0.232 ms
> 64 bytes from 192.168.0.1: icmp_seq=2. time=0.191 ms
> 64 bytes from 192.168.0.1: icmp_seq=3. time=0.180 ms

Awesome!!

> Let me know if you want the additional change to what is currently in
> the tree. (I forget whether this matters or not for Solaris). I am
> waiting to hear back from Eitan on this but am pretty sure it's correct.

Yes, I would like the patch.

Thanks,

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050211/7498e4b6/attachment.sig>

From libor at topspin.com  Fri Feb 11 17:22:12 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 17:22:12 -0800
Subject: [openib-general] [PATCH][fixed] get SDP building on sparc64
In-Reply-To: <1108148298.28588.41.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 11, 2005 at 10:58:18AM -0800
References: <1108140428.28588.22.camel@duffman>
	<1108148298.28588.41.camel@duffman>
Message-ID: <20050211172212.D20777@topspin.com>

On Fri, Feb 11, 2005 at 10:58:18AM -0800, Tom Duffy wrote:
> The first patch did get SDP building on sparc64, but it would not build
> on x86_64.  This is the correct fix that gets it to build on both
> architectures (as well as x86_32).

Thanks, checked in.

-Libor

> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_main.h
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_main.h	(revision 1766)
> +++ drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
> @@ -79,6 +79,7 @@
>  #include <linux/ip.h>
>  #include <linux/pci.h>
>  #include <linux/random.h>
> +#include <linux/dma-mapping.h>
>  
>  #include <net/sock.h>
>  #include <net/route.h>
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From halr at voltaire.com  Fri Feb 11 17:11:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Feb 2005 20:11:33 -0500
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <1108170521.28588.61.camel@duffman>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
	<1108170521.28588.61.camel@duffman>
Message-ID: <1108170692.17781.48.camel@localhost.localdomain>

On Fri, 2005-02-11 at 20:08, Tom Duffy wrote:
> [ Putting on OpenIB ]
> 
> On Fri, 2005-02-11 at 19:43 -0500, Hal Rosenstock wrote:
> > Hey Tom,
> > 
> > On Fri, 2005-02-11 at 19:35, Tom Duffy wrote:
> > > Solaris, you need to "ifconfig ibd0 plumb" to see it in ifconfig -a. You
> > > are further than me since you see /dev/ibd0.  You might even be there...
> > 
> > That did it :-) Thanks.
> > 
> > # ifconfig ibd0 plumb
> > # ifconfig -a
> > lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> > 8232 index 1
> >         inet 127.0.0.1 netmask ff000000 
> > e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu
> > 1500 index 2
> >         inet 10.0.2.56 netmask ffffff00 broadcast 10.0.2.255
> >         ether 0:30:48:27:21:2c 
> > ibd0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 252 index 3
> >         inet 0.0.0.0 netmask 0 
> >         ipib 0:0:4:6:fe:80:0:0:0:0:0:0:0:8:f1:4:3:96:4:e 
> > 
> > Not sure why the MTU comes up as 252 (other than it is the 256 minimum
> > minus 4 bytes overhead).
> 
> That is odd.  I don't see this when using IBSRM to manage the network:
> 
> ibd1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 3
>         inet 192.168.0.78 netmask ffffff00 broadcast 192.168.0.255
>         ipib 0:0:0:18:0:0:0:0:0:0:12:34:0:2:c9:1:9:76:56:31 
> 
> 
> > It also doesn't like me changing the MTU.
> > ifconfig ibd0 mtu 2044
> > ifconfig: setifmtu: SIOCSLIFMTU: ibd0: Invalid argument
> 
> Huh, Nitin, any ideas?

The log indicates that the MTU is 4 which is 2048. I also saw this in
the IB trace.

Feb 11 19:39:47 [B57AEA40] -> MCMember Record dump:
                               
MGID....................0xff12401bffff0000 : 0x0000000000000001
                               
PortGid.................0xfe80000000000000 : 0x0008f1040396040e
                                qkey....................0xB1B
                                Mlid....................0xC001
                                ScopeState..............0x21
                                Rate....................0x3
                                Mtu.....................0x4
                                TClass..................0x0
                                SLFlowLabelHopLimit.....0x0


> > But when I lower the MTU on the OpenIB side, I have connectivity:
> > 
> > >From OpenIB -> Solaris
> > ping 192.168.0.2
> > PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
> > 64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=8.57 ms
> > 64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.118 ms
> > 64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=0.127 ms
> > 
> > >From Solaris -> OpenIB
> > ping -s 192.168.0.1
> > PING 192.168.0.1: 56 data bytes
> > 64 bytes from 192.168.0.1: icmp_seq=0. time=0.306 ms
> > 64 bytes from 192.168.0.1: icmp_seq=1. time=0.232 ms
> > 64 bytes from 192.168.0.1: icmp_seq=2. time=0.191 ms
> > 64 bytes from 192.168.0.1: icmp_seq=3. time=0.180 ms
> 
> Awesome!!
> 
> > Let me know if you want the additional change to what is currently in
> > the tree. (I forget whether this matters or not for Solaris). I am
> > waiting to hear back from Eitan on this but am pretty sure it's correct.
> 
> Yes, I would like the patch.


Index: osm_sa_mcmember_record.c
===================================================================
--- osm_sa_mcmember_record.c	(revision 1745)
+++ osm_sa_mcmember_record.c	(working copy)
@@ -176,8 +176,8 @@
   p_recvd_mcmember_rec = p_ctxt->p_mcmember_rec;
   p_rcv = p_ctxt->p_rcv;
 
-  /* Why not comparing all the MGID ???? */
-  /* different scope can sneek in for the same MGID ? */
+  /* Why not compare the entire MGID ???? */
+  /* different scope can sneak in for the same MGID ? */
   /* EZ: I changed it to full compare ! */
   if (cl_memcmp(&p_mgrp->mcmember_rec.mgid,
                 &p_recvd_mcmember_rec->mgid,
@@ -732,7 +732,7 @@
 
   OSM_LOG_ENTER( p_rcv->p_log, __validate_requested_mgid );
 
-  /* 14-a : mcaut GUID must start with 0xFF */
+  /* 14-a : mcast GID must start with 0xFF */
   if (p_mcm_rec->mgid.multicast.header[0] != 0xFF)
   {
     osm_log( p_rcv->p_log, OSM_LOG_ERROR,
@@ -1009,8 +1009,8 @@
     {
 
       /*
-       * acording to the same o15-0.1.14 we must get the stored
JoinState and
-       * the request JoinState be oposite to leave - other wise just
update it
+       * according to the same o15-0.1.14 we must get the stored
JoinState and
+       * the request JoinState be opposite to leave - otherwise just
update it
        */
       port_join_state = p_mcm_port->scope_state & 0x0F;
       new_join_state = 
@@ -1419,7 +1419,7 @@
   osm_mcmr_recv_t*            const p_rcv  = p_ctxt->p_rcv;
   const ib_member_rec_t*      p_rcvd_rec   = p_ctxt->p_mcmember_rec;
   const osm_physp_t*          p_req_physp  = p_ctxt->p_req_physp;
-  ib_member_rec_t             match_rec; // since we might change
scope_state
+  ib_member_rec_t             match_rec; /* since we might change
scope_state */
   ib_net64_t                  comp_mask    = p_ctxt->comp_mask;
   osm_mcm_port_t*             p_mcm_port;
   ib_net64_t portguid = p_rcvd_rec->port_gid.unicast.interface_id;
@@ -1433,7 +1433,7 @@
           "__osm_sa_mcm_by_comp_mask_cb: "
           "Checking mlid:0x%X\n",
           cl_ntoh16(p_mgrp->mlid ));
-  /* first try to eliminate the group by MGID, MLID or P_Key */
+  /* first try to eliminate the group by MGID, MLID, or P_Key */
   if ((IB_MCR_COMPMASK_MGID & comp_mask) &&
       cl_memcmp(&p_rcvd_rec->mgid, &p_mgrp->mcmember_rec.mgid,
sizeof(ib_gid_t))) {
     goto Exit;
@@ -1490,10 +1490,10 @@
     sl_flow_hop_mask = sl_flow_hop_mask | 0x000F;
 
   if (IB_MCR_COMPMASK_FLOW & comp_mask)
-    sl_flow_hop_mask = sl_flow_hop_mask | 0x0FF0;
+    sl_flow_hop_mask = sl_flow_hop_mask | 0xFFFFFF0;
 
   if (IB_MCR_COMPMASK_HOP & comp_mask)
-    sl_flow_hop_mask = sl_flow_hop_mask | 0xF000;
+    sl_flow_hop_mask = sl_flow_hop_mask | 0xFF000000;
 
   if ((p_rcvd_rec->sl_flow_hop & sl_flow_hop_mask) !=
       (p_mgrp->mcmember_rec.sl_flow_hop & sl_flow_hop_mask)) goto Exit;



> Thanks,
> 
> -tduffy



From libor at topspin.com  Fri Feb 11 17:29:05 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 17:29:05 -0800
Subject: [openib-general] [PATCH] use C99 struct initilization syntax in
	SDP
In-Reply-To: <1108145451.28588.33.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 11, 2005 at 10:10:51AM -0800
References: <1108145451.28588.33.camel@duffman>
Message-ID: <20050211172905.E20777@topspin.com>

On Fri, Feb 11, 2005 at 10:10:51AM -0800, Tom Duffy wrote:
> use C99, otherwise sparse complains
> 
> Signed-off-by: Tom Duffy <tduffy at sun.com>

Thanks, applied and committed.

-Libor

















From tduffy at sun.com  Fri Feb 11 17:29:54 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 17:29:54 -0800
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <1108170692.17781.48.camel@localhost.localdomain>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
	<1108170521.28588.61.camel@duffman>
	<1108170692.17781.48.camel@localhost.localdomain>
Message-ID: <1108171794.28588.65.camel@duffman>

On Fri, 2005-02-11 at 20:11 -0500, Hal Rosenstock wrote:
> The log indicates that the MTU is 4 which is 2048. I also saw this in
> the IB trace.

Ok, I will ask some other Solaris IB guys as well...

> > Yes, I would like the patch.

BTW, Solaris does work now (Yippie!!!!), even without this patch (albeit
with the MTU workaround).  I just confirmed on my network.  Thanks.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050211/c1806d90/attachment.sig>

From libor at topspin.com  Fri Feb 11 17:35:32 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 17:35:32 -0800
Subject: [openib-general] RFC on SDP checkin
In-Reply-To: <1108093196.28588.3.camel@duffman>;
	from tduffy@sun.com on Thu, Feb 10, 2005 at 07:39:56PM -0800
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>
	<1108093196.28588.3.camel@duffman>
Message-ID: <20050211173532.F20777@topspin.com>

On Thu, Feb 10, 2005 at 07:39:56PM -0800, Tom Duffy wrote:
> On Thu, 2005-02-10 at 19:09 -0800, Libor Michalek wrote:
> >   OK, I've gone with option number 3. Here is a one or two line
> > description of each file, and a TODO list. Both are checked in
> > along with the code in the infiniband/ulp/sdp directory.
> 
> Of course, you will need this patch to plumb SDP into the build.

  Since Sean checked in his CM makefile change I'm checking in the
following patch, I made the change to the SDP Kconfig dependacy
based on Roland's comment as well.

-Libor

Index: infiniband/Kconfig
===================================================================
--- infiniband/Kconfig	(revision 1778)
+++ infiniband/Kconfig	(working copy)
@@ -11,4 +11,6 @@
 
 source "drivers/infiniband/ulp/ipoib/Kconfig"
 
+source "drivers/infiniband/ulp/sdp/Kconfig"
+
 endmenu
Index: infiniband/ulp/sdp/Kconfig
===================================================================
--- infiniband/ulp/sdp/Kconfig	(revision 1778)
+++ infiniband/ulp/sdp/Kconfig	(working copy)
@@ -1,6 +1,6 @@
 config INFINIBAND_SDP
 	tristate "Sockets Direct Protocol"
-	depends on INFINIBAND && INET && INFINIBAND_IPOIB
+	depends on INFINIBAND && INFINIBAND_IPOIB
 	---help---
 	  Support for Sockets Direct Protocol (SDP).  This provides
           sockets semantics over InfiniBand via address family
Index: infiniband/Makefile
===================================================================
--- infiniband/Makefile	(revision 1778)
+++ infiniband/Makefile	(working copy)
@@ -1,3 +1,4 @@
 obj-$(CONFIG_INFINIBAND)		+= core/
 obj-$(CONFIG_INFINIBAND_MTHCA)		+= hw/mthca/
 obj-$(CONFIG_INFINIBAND_IPOIB)		+= ulp/ipoib/
+obj-$(CONFIG_INFINIBAND_SDP)		+= ulp/sdp/



From tduffy at sun.com  Fri Feb 11 17:44:29 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 17:44:29 -0800
Subject: [openib-general] [PATCH][SDP] Remove sparse warning about use of 0
	instead of NULL
Message-ID: <1108172670.28588.69.camel@duffman>

Remove sparse warnings:

warning: Using plain integer as NULL pointer

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_send.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_send.c	(revision 1778)
+++ drivers/infiniband/ulp/sdp/sdp_send.c	(working copy)
@@ -32,7 +32,7 @@
 /*.._sdp_send_buff_post -- Post a buffer send on a SDP connection. */
 static int _sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	struct ib_send_wr send_param = { 0 };
+	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
 	int result;
 
@@ -322,7 +322,7 @@ error:
 static int _sdp_send_data_buff_snk(struct sdp_opt *conn,
 				   struct sdpc_buff *buff)
 {
-	struct ib_send_wr send_param = { 0 };
+	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
 	struct sdpc_advt *advt;
 	int result;
@@ -471,7 +471,7 @@ error:
 static int _sdp_send_data_iocb_snk(struct sdp_opt *conn,
 				   struct sdpc_iocb *iocb)
 {
-	struct ib_send_wr send_param = { 0 };
+	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
 	struct ib_sge sg_val;
 	struct sdpc_advt *advt;
Index: drivers/infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_recv.c	(revision 1778)
+++ drivers/infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -31,7 +31,7 @@
 /*.._sdp_post_recv_buff -- post a single buffers for data recv */
 static int _sdp_post_recv_buff(struct sdp_opt *conn)
 {
-	struct ib_recv_wr receive_param = { 0 };
+	struct ib_recv_wr receive_param = { NULL };
 	struct ib_recv_wr *bad_wr;
 	struct sdpc_buff *buff;
 	s32 result;
@@ -108,7 +108,7 @@ error:
 /*.._sdp_post_rdma_buff -- post a single buffers for rdma read on a conn */
 static int _sdp_post_rdma_buff(struct sdp_opt *conn)
 {
-	struct ib_send_wr send_param = { 0 };
+	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
 	struct sdpc_advt *advt;
 	struct sdpc_buff *buff;
@@ -237,7 +237,7 @@ done:
 /*.._sdp_post_rdma_iocb_src -- post a iocb for rdma read on a conn */
 static int _sdp_post_rdma_iocb_src(struct sdp_opt *conn)
 {
-	struct ib_send_wr send_param = { 0 };
+	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
 	struct ib_sge sg_val;
 	struct sdpc_iocb *iocb;



From libor at topspin.com  Fri Feb 11 17:55:50 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 17:55:50 -0800
Subject: [openib-general] [PATCH][Take2] Change AF_INET_SDP to 27
In-Reply-To: <1108094461.28588.10.camel@duffman>;
	from tduffy@sun.com on Thu, Feb 10, 2005 at 08:01:01PM -0800
References: <1108094010.28588.6.camel@duffman>
	<1108094461.28588.10.camel@duffman>
Message-ID: <20050211175550.G20777@topspin.com>

On Thu, Feb 10, 2005 at 08:01:01PM -0800, Tom Duffy wrote:
> Oops.  I only changed it one place.  Why is it defined in two places?
> 

  You're right, it doesn't need to be in two places. I'll delete
sdp_inet.h since it's redundant, plus make the AF_INET_SDP change.
Here's the patch.

-Libor

Index: sdp_main.h
===================================================================
--- sdp_main.h	(revision 1778)
+++ sdp_main.h	(working copy)
@@ -92,7 +92,6 @@
 #include <ib_cache.h>
 #include <ib_cm.h>
 #include "sdp_sock.h"
-#include "sdp_inet.h"
 /*
  * sdp local includes
  */
Index: sdp_inet.h
===================================================================
--- sdp_inet.h	(revision 1778)
+++ sdp_inet.h	(working copy)
@@ -1,33 +0,0 @@
-/*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_inet.h 140 2004-05-11 00:30:52Z libor $
-*/
-
-#ifndef _SDP_INET_H
-#define _SDP_INET_H
-
-/*
- * constants shared between user and kernel space.
- */
-#define AF_INET_SDP 26		/* SDP socket protocol family */
-#define AF_INET_STR "AF_INET_SDP"	/* SDP enabled environment variable */
-
-#endif /* _SDP_INET_H */
Index: sdp_sock.h
===================================================================
--- sdp_sock.h	(revision 1778)
+++ sdp_sock.h	(working copy)
@@ -28,8 +28,9 @@
  * functions (e.g bind, connect, etc.) Either AF_INET or AF_INET_SDP can
  * be used with a SDP socket.
  */
-#define AF_INET_SDP   26
-#define PF_INET_SDP   AF_INET_SDP
+#define AF_INET_SDP  27
+#define PF_INET_SDP  AF_INET_SDP
+#define AF_INET_STR "AF_INET_SDP" /* SDP enabled environment variable */
 /*
  * Socket option level for SDP specific parameters.
  */
Index: sdp_dev.h
===================================================================
--- sdp_dev.h	(revision 1778)
+++ sdp_dev.h	(working copy)
@@ -36,7 +36,6 @@
  * sdp types
  */
 #include <ib_verbs.h>
-#include "sdp_inet.h"
 #include "sdp_msgs.h"
 
 #define SDP_MSG_HDR_REQ_SIZE 0x10 /* required header size (BSDH) */


From libor at topspin.com  Fri Feb 11 17:59:51 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 17:59:51 -0800
Subject: [openib-general] [PATCH][SDP] Remove sparse warning about use of
	0 instead of NULL
In-Reply-To: <1108172670.28588.69.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 11, 2005 at 05:44:29PM -0800
References: <1108172670.28588.69.camel@duffman>
Message-ID: <20050211175951.H20777@topspin.com>

On Fri, Feb 11, 2005 at 05:44:29PM -0800, Tom Duffy wrote:
> Remove sparse warnings:
> 
> warning: Using plain integer as NULL pointer

Thanks, commited the change.

-Libor

> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_send.c
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_send.c	(revision 1778)
> +++ drivers/infiniband/ulp/sdp/sdp_send.c	(working copy)
> @@ -32,7 +32,7 @@
>  /*.._sdp_send_buff_post -- Post a buffer send on a SDP connection. */
>  static int _sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff)
>  {
> -	struct ib_send_wr send_param = { 0 };
> +	struct ib_send_wr send_param = { NULL };
>  	struct ib_send_wr *bad_wr;
>  	int result;
>  
> @@ -322,7 +322,7 @@ error:
>  static int _sdp_send_data_buff_snk(struct sdp_opt *conn,
>  				   struct sdpc_buff *buff)
>  {
> -	struct ib_send_wr send_param = { 0 };
> +	struct ib_send_wr send_param = { NULL };
>  	struct ib_send_wr *bad_wr;
>  	struct sdpc_advt *advt;
>  	int result;
> @@ -471,7 +471,7 @@ error:
>  static int _sdp_send_data_iocb_snk(struct sdp_opt *conn,
>  				   struct sdpc_iocb *iocb)
>  {
> -	struct ib_send_wr send_param = { 0 };
> +	struct ib_send_wr send_param = { NULL };
>  	struct ib_send_wr *bad_wr;
>  	struct ib_sge sg_val;
>  	struct sdpc_advt *advt;
> Index: drivers/infiniband/ulp/sdp/sdp_recv.c
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_recv.c	(revision 1778)
> +++ drivers/infiniband/ulp/sdp/sdp_recv.c	(working copy)
> @@ -31,7 +31,7 @@
>  /*.._sdp_post_recv_buff -- post a single buffers for data recv */
>  static int _sdp_post_recv_buff(struct sdp_opt *conn)
>  {
> -	struct ib_recv_wr receive_param = { 0 };
> +	struct ib_recv_wr receive_param = { NULL };
>  	struct ib_recv_wr *bad_wr;
>  	struct sdpc_buff *buff;
>  	s32 result;
> @@ -108,7 +108,7 @@ error:
>  /*.._sdp_post_rdma_buff -- post a single buffers for rdma read on a conn */
>  static int _sdp_post_rdma_buff(struct sdp_opt *conn)
>  {
> -	struct ib_send_wr send_param = { 0 };
> +	struct ib_send_wr send_param = { NULL };
>  	struct ib_send_wr *bad_wr;
>  	struct sdpc_advt *advt;
>  	struct sdpc_buff *buff;
> @@ -237,7 +237,7 @@ done:
>  /*.._sdp_post_rdma_iocb_src -- post a iocb for rdma read on a conn */
>  static int _sdp_post_rdma_iocb_src(struct sdp_opt *conn)
>  {
> -	struct ib_send_wr send_param = { 0 };
> +	struct ib_send_wr send_param = { NULL };
>  	struct ib_send_wr *bad_wr;
>  	struct ib_sge sg_val;
>  	struct sdpc_iocb *iocb;
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From tduffy at sun.com  Fri Feb 11 18:31:06 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 18:31:06 -0800
Subject: [openib-general] [PATCH][SDP] __user annotations
Message-ID: <1108175466.25868.3.camel@duffman>

Annotate __user pointers in sdp_inet.c.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1781)
+++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -1217,7 +1217,7 @@ static int _sdp_inet_ioctl(struct socket
 		}
 		else {
 
-			result = copy_to_user((void *)arg,
+			result = copy_to_user((void __user *)arg,
 					      &sk->sk_stamp,
 					      sizeof(struct timeval));
 			result = (result ? -EFAULT : result);
@@ -1231,7 +1231,7 @@ static int _sdp_inet_ioctl(struct socket
 	case SIOCDELRT:
 	case SIOCRTMSG:
 
-		result = ip_rt_ioctl(cmd, (void *)arg);
+		result = ip_rt_ioctl(cmd, (void __user *)arg);
 		break;
 		/*
 		 * Standard ARP IOCTLs
@@ -1260,7 +1260,7 @@ static int _sdp_inet_ioctl(struct socket
 	case SIOCGIFPFLAGS:
 	case SIOCSIFFLAGS:
 
-		result = devinet_ioctl(cmd, (void *)arg);
+		result = devinet_ioctl(cmd, (void __user *)arg);
 		break;
 		/*
 		 * stadard INET STREAM IOCTLs
@@ -1274,7 +1274,7 @@ static int _sdp_inet_ioctl(struct socket
 			 * TODO need to subtract/add URG (inline vs. OOB)
 			 */
 			value = conn->byte_strm;
-			result = put_user(value, (int *) arg);
+			result = put_user(value, (int __user *) arg);
 		}
 		else {
 
@@ -1290,7 +1290,7 @@ static int _sdp_inet_ioctl(struct socket
 		if (SDP_SOCK_ST_LISTEN != conn->istate) {
 
 			value = conn->send_qud;
-			result = put_user(value, (int *) arg);
+			result = put_user(value, (int __user *) arg);
 		}
 		else {
 
@@ -1316,13 +1316,13 @@ static int _sdp_inet_ioctl(struct socket
 			}
 		}
 
-		result = put_user(value, (int *) arg);
+		result = put_user(value, (int __user *) arg);
 
 		SDP_CONN_UNLOCK(conn);
 		break;
 	default:
 
-		result = dev_ioctl(cmd, (void *)arg);
+		result = dev_ioctl(cmd, (void __user *)arg);
 		break;
 	}
 
@@ -1334,7 +1334,7 @@ static int _sdp_inet_ioctl(struct socket
 static int _sdp_inet_setopt(struct socket *sock,
 			       int level, 
 			       int optname, 
-			       char *optval, 
+			       char __user *optval, 
 			       int optlen)
 {
 	struct sock *sk;
@@ -1362,7 +1362,7 @@ static int _sdp_inet_setopt(struct socke
 		return -EINVAL;
 	}
 
-	if (get_user(value, (int *)optval)) {
+	if (get_user(value, (int __user *)optval)) {
 
 		return -EFAULT;
 	}
@@ -1422,8 +1422,8 @@ static int _sdp_inet_setopt(struct socke
 static int _sdp_inet_getopt(struct socket *sock,
 			       int level,
 			       int optname,
-			       char *optval,
-			       int *optlen)
+			       char __user *optval,
+			       int __user *optlen)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;



From libor at topspin.com  Fri Feb 11 18:40:58 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 11 Feb 2005 18:40:58 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
Message-ID: <20050211184058.I20777@topspin.com>


  If no one objects, a patch to clean up compile warnings on x86_64. Most
of the warnings are a result of print format mismatches, the most common
being the need to use %Zu for size_t/sizeof.

  For the curious, I'm seeing about 720 MB/s and 17us latency on a pair
of dual 3.4 GHz 64bit Xeon PCIe systems.

-Libor

Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1781)
+++ sdp_send.c	(working copy)
@@ -2014,15 +2014,17 @@
 	 * attempt to find the IOCB for this key. we don't have an indication
 	 * whether this is a read or write.
 	 */
-	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->send_queue,
-						     _sdp_write_src_lookup,
-						     (void *)req->ki_key);
+	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
+		(&conn->send_queue,
+		 _sdp_write_src_lookup,
+		 (void *)(unsigned long)req->ki_key);
+
 	if (NULL != iocb) {
 		/*
 		 * always remove the IOCB.
 		 * If active, then place it into the correct active queue
 		 */
-		result = sdp_desc_q_remove((struct sdpc_desc *) iocb);
+		result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
 		SDP_EXPECT(!(0 > result));
 		
 		if (0 < (SDP_IOCB_F_ACTIVE & iocb->flags)) {
@@ -2073,9 +2075,11 @@
 	 * check the sink queue, not much to do, since the operation is
 	 * already in flight.
 	 */
-	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->w_snk,
-						     _sdp_write_src_lookup,
-						     (void *)req->ki_key);
+	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
+		(&conn->w_snk,
+		 _sdp_write_src_lookup,
+		 (void *)(unsigned long)req->ki_key);
+
 	if (NULL != iocb) {
 
 		iocb->flags |= SDP_IOCB_F_CANCEL;
@@ -2439,7 +2443,7 @@
 		iocb = sdp_iocb_create();
 		if (NULL == iocb) {
 			
-			sdp_dbg_warn(conn, "Failed to allocate IOCB <%d:%d>",
+			sdp_dbg_warn(conn, "Failed to allocate IOCB <%Zu:%d>",
 				     size, copied);
 			result = -ENOMEM;
 			break;
@@ -2457,7 +2461,7 @@
 		result = sdp_iocb_lock(iocb);
 		if (0 > result) {
 				
-			sdp_dbg_warn(conn, "Error <%d> locking IOCB <%d:%d>", 
+			sdp_dbg_warn(conn, "Error <%d> locking IOCB <%Zu:%d>", 
 				     result, size, copied);
         
 			(void)sdp_iocb_destroy(iocb);
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1778)
+++ sdp_conn.c	(working copy)
@@ -1085,7 +1085,8 @@
 		conn->send_cq = ib_create_cq(conn->ca,
 					     sdp_cq_event_handler,
 					     NULL,
-					     (void *)conn->hashent,
+					     (void *)
+					     (unsigned long)conn->hashent,
 					     conn->send_cq_size);
 		if (IS_ERR(conn->send_cq)) {
 
@@ -1110,7 +1111,8 @@
 		conn->recv_cq = ib_create_cq(conn->ca,
 					     sdp_cq_event_handler,
 					     NULL,
-					     (void *)conn->hashent,
+					     (void *)
+					     (unsigned long)conn->hashent,
 					     conn->recv_cq_size);
 
 		if (IS_ERR(conn->recv_cq)) {
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1778)
+++ sdp_actv.c	(working copy)
@@ -201,7 +201,7 @@
 	 */
 	if (hello_ack->bsdh.size != sizeof(struct msg_hello_ack)) {
 
-		sdp_dbg_warn(NULL, "hello ack, size mismatch. (2) <%d:%d>",
+		sdp_dbg_warn(NULL, "hello ack, size mismatch. (2) <%d:%Zu>",
 			     hello_ack->bsdh.size,
 			     sizeof(struct msg_hello_ack));
 		return -EINVAL;
@@ -518,7 +518,7 @@
 	path->packet_life = 13;
 #endif
 	conn->cm_id = ib_create_cm_id(sdp_cm_event_handler,
-				      (void *)conn->hashent);
+				      (void *)(unsigned long)conn->hashent);
 	if (NULL == conn->cm_id) {
 
 		sdp_dbg_warn(conn, "Failed to create CM handle, %d",
@@ -571,7 +571,7 @@
 	 */
 	if (sizeof(struct msg_hello) > conn->recv_size) {
 
-		sdp_dbg_warn(conn, "buffer size <%d> too small. <%d>",
+		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct msg_hello));
 		result = -ENOBUFS;
 		goto error;
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1781)
+++ sdp_recv.c	(working copy)
@@ -799,8 +799,9 @@
 	result = _sdp_read_buff_iocb(iocb, buff);
 	if (0 > result) {
 
-		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%d> to IOCB",
-			     result, iocb->len, (buff->tail - buff->data));
+		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
+			     result, iocb->len, 
+			     (unsigned)(buff->tail - buff->data));
 
 		(void)sdp_iocb_q_put_head(&conn->r_snk, iocb);
 		return result;
@@ -850,8 +851,9 @@
 	result = _sdp_read_buff_iocb(iocb, buff);
 	if (0 > result) {
 
-		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%d> to IOCB",
-			     result, iocb->len, (buff->tail - buff->data));
+		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
+			     result, iocb->len, 
+			     (unsigned)(buff->tail - buff->data));
 		return result;
 	}
 	/*
@@ -1109,9 +1111,10 @@
 	 * check the source queue, not much to do, since the operation is
 	 * already in flight.
 	 */
-	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->r_src,
-						     _sdp_read_src_lookup,
-						     (void *)req->ki_key);
+	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
+		(&conn->r_src,
+		 _sdp_read_src_lookup,
+		 (void *)(unsigned long)req->ki_key);
 	if (NULL != iocb) {
 
 		iocb->flags |= SDP_IOCB_F_CANCEL;
@@ -1623,7 +1626,7 @@
 			if (NULL == iocb) {
         
 				sdp_dbg_warn(conn, 
-					     "Failed to allocate IOCB <%d:%d>",
+					     "Error allocating IOCB <%Zu:%d>",
 					     size, copied);
 				result = -ENOMEM;
 				break;
@@ -1642,7 +1645,7 @@
 			if (0 > result) {
         
 				sdp_dbg_warn(conn, 
-					     "Error <%d> IOCB lock <%d:%d>", 
+					     "Error <%d> IOCB lock <%Zu:%d>", 
 					     result, size, copied);
 				
 				(void)sdp_iocb_destroy(iocb);
@@ -1655,7 +1658,7 @@
 			if (0 > result) {
 				
 				sdp_dbg_warn(conn, 
-					     "Error <%d> IOCB queue <%d:%d>",
+					     "Error <%d> IOCB queue <%Zu:%d>",
 					     result, size, copied);
 				
 				(void)sdp_iocb_destroy(iocb);
Index: sdp_proc.c
===================================================================
--- sdp_proc.c	(revision 1778)
+++ sdp_proc.c	(working copy)
@@ -169,7 +169,7 @@
 	 */
 	if (0 && sizeof(s32) != sizeof(char *)) {
 
-		sdp_warn("integers and pointers of a different size. <%d:%d>",
+		sdp_warn("integers and pointers of a different size <%Zu:%Zu>",
 			 sizeof(s32), sizeof(char *));
 		return -EFAULT;
 	}
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1778)
+++ sdp_pass.c	(working copy)
@@ -41,7 +41,7 @@
 	 */
 	if (sizeof(struct msg_hello_ack) > conn->recv_size) {
 
-		sdp_dbg_warn(conn, "buffer size <%d> too small. <%d>",
+		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct msg_hello_ack));
 		result = -ENOBUFS;
 		goto error;
@@ -311,7 +311,7 @@
 	 */
 	if (msg_hello->bsdh.size != sizeof(struct msg_hello)) {
 
-		sdp_dbg_warn(NULL, "hello msg size mismatch. (2) <%d:%d>",
+		sdp_dbg_warn(NULL, "hello msg size mismatch. (2) <%d:%Zu>",
 			     msg_hello->bsdh.size, sizeof(struct msg_hello));
 		return -EINVAL;
 	}
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1778)
+++ sdp_iocb.c	(working copy)
@@ -44,7 +44,7 @@
 	vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK));
 	if (NULL == vma) {
 		
-		sdp_warn("No VMA for IOCB <%lx:%d> unlock",
+		sdp_warn("No VMA for IOCB <%lx:%Zu> unlock",
 			 iocb->addr, iocb->size);
 	}
 
@@ -286,7 +286,7 @@
 	 */
 	if (result) {
 
-		sdp_dbg_err("VMA lock <%lx:%d> error <%d> <%d:%lu:%lu>",
+		sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>",
 			    iocb->addr, iocb->size, result, 
 			    iocb->page_count, iocb->mm->locked_vm, limit);
 		goto err_lock;
@@ -300,7 +300,7 @@
 		/*
 		 * sanity check.
 		 */
-		sdp_warn("No VMA for IOCB! <%lx:%d> lock",
+		sdp_warn("No VMA for IOCB! <%lx:%Zu> lock",
 			 iocb->addr, iocb->size);
 	}
 
@@ -342,7 +342,7 @@
 	result = _sdp_iocb_page_save(iocb);
 	if (result) {
 
-		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%d>", 
+		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", 
 			    result, iocb->addr, iocb->size);
 		goto err_save;
 	}
@@ -584,7 +584,7 @@
 	result = sdp_iocb_destroy(iocb);
 	if (0 > result) {
 
-		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%d>",
+		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
 	}
 
@@ -924,8 +924,8 @@
 
 		if (0 < (mask & iocb->flags) || SDP_IOCB_F_ALL == mask) {
 
-			sdp_dbg_err("IOCB <%d> cancel <%d> flag <%04x> "
-				    "size <%d:%d:%d>",
+			sdp_dbg_err("IOCB <%d> cancel <%Zu> flag <%04x> "
+				    "size <%Zu:%d:%d>",
 				    iocb->key, comp, iocb->flags, iocb->size,
 				    iocb->post, iocb->len);
 


From tduffy at sun.com  Fri Feb 11 19:47:41 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 19:47:41 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050211184058.I20777@topspin.com>
References: <20050211184058.I20777@topspin.com>
Message-ID: <1108180061.25868.7.camel@duffman>

On Fri, 2005-02-11 at 18:40 -0800, Libor Michalek wrote:
>   If no one objects, a patch to clean up compile warnings on x86_64. Most
> of the warnings are a result of print format mismatches, the most common
> being the need to use %Zu for size_t/sizeof.

Looks good.  This patch gets rid of the compile warnings on x86_64, but
I am still seeing these on sparc64:

/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c: In function `_sdp_rcvd_rdma_wr':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:237: warning: int format, different type arg (arg 10)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:237: warning: int format, different type arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c: In function `_sdp_rcvd_rdma_rd':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:306: warning: int format, different type arg (arg 10)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:306: warning: int format, different type arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c: In function `_sdp_rcvd_snk_avail':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:800: warning: long long unsigned int format, __u64 arg (arg 9)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:800: warning: long long unsigned int format, __u64 arg (arg 6)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c: In function `_sdp_rcvd_src_avail':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:966: warning: long long unsigned int format, __u64 arg (arg 10)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:966: warning: long long unsigned int format, __u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c: In function `sdp_event_recv':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c: In function `sdp_event_send':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c: In function `sdp_event_read':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:142: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:142: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c: In function `sdp_event_write':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c:66: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c:66: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c:125: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c:125: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c: In function `_sdp_send_data_buff_snk':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:396: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:396: warning: unsigned int format, different type arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:396: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:396: warning: unsigned int format, different type arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c: In function `_sdp_send_data_iocb_snk':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:533: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:533: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c: In function `sdp_inet_send':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:2273: warning: int format, different type arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c:2273: warning: int format, different type arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c: In function `_sdp_post_recv_buff':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:74: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:74: warning: unsigned int format, different type arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:74: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:74: warning: unsigned int format, different type arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c: In function `_sdp_post_rdma_buff':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:203: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:203: warning: unsigned int format, different type arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:203: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:203: warning: unsigned int format, different type arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c: In function `_sdp_post_rdma_iocb_src':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:363: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:363: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c: In function `sdp_recv_buff':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:903: warning: int format, different type arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:903: warning: int format, different type arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c: In function `sdp_inet_recv':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:1323: warning: int format, different type arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c:1323: warning: int format, different type arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c: In function `sdp_cq_event_locked':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:46: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:46: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:105: warning: long long unsigned int format, u64 arg (arg 9)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:105: warning: long long unsigned int format, u64 arg (arg 6)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c: In function `sdp_cm_req_handler':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c:381: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c:381: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c: In function `_sdp_cm_path_complete':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 5)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050211/6eae56e4/attachment.sig>

From Nitin.Hande at Sun.COM  Fri Feb 11 20:21:28 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Fri, 11 Feb 2005 20:21:28 -0800
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <1108171794.28588.65.camel@duffman>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
	<1108170521.28588.61.camel@duffman>
	<1108170692.17781.48.camel@localhost.localdomain>
	<1108171794.28588.65.camel@duffman>
Message-ID: <420D8448.9010000@sun.com>

Tom Duffy wrote:

>On Fri, 2005-02-11 at 20:11 -0500, Hal Rosenstock wrote:
>  
>
>>The log indicates that the MTU is 4 which is 2048. I also saw this in
>>the IB trace.
>>    
>>
>
>Ok, I will ask some other Solaris IB guys as well...
>
>  
>
>>>Yes, I would like the patch.
>>>      
>>>
>
>BTW, Solaris does work now (Yippie!!!!), even without this patch (albeit
>with the MTU workaround).  I just confirmed on my network.  Thanks.
>  
>
Allright, so I am not seeing this anymore on the test setup here. I will 
keep a watch and conduct some more experiments over weekend if time permits.

[root at dongon /kernel/drv/sparcv9]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 
8232 index 1
        inet 127.0.0.1 netmask ff000000
eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.6.98.60 netmask ff000000 broadcast 10.255.255.255
        ether 0:3:ba:24:48:ad
ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 6
        inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
        ipib 0:4:0:16:fe:80:0:0:0:0:0:0:0:2:c9:1:9:76:51:d1
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 
8252 index 1
        inet6 ::1/128
eri0: flags=2000841<UP,RUNNING,MULTICAST,IPv6> mtu 1500 index 2
        inet6 fe80::203:baff:fe24:48ad/10
        ether 0:3:ba:24:48:ad
[root at dongon /kernel/drv/sparcv9]# uname -a
SunOS dongon.SFBay.Sun.COM 5.10 s10_74l1 sun4u sparc SUNW,Sun-Blade-1000
[root at dongon /kernel/drv/sparcv9]# cat /etc/path_to_inst | grep ibd
"/pci at 8,600000/pci at 1/pci15b3,5a44 at 0/ibport at 1,ffff,ipib" 0 "ibd"
"/pci at 8,600000/pci at 1/pci15b3,5a44 at 0/ibport at 2,ffff,ipib" 1 "ibd"
[root at dongon /kernel/drv/sparcv9]#

Thanks
Nitin

>-tduffy
>  
>



From tduffy at sun.com  Fri Feb 11 20:31:07 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 11 Feb 2005 20:31:07 -0800
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <420D8448.9010000@sun.com>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
	<1108170521.28588.61.camel@duffman>
	<1108170692.17781.48.camel@localhost.localdomain>
	<1108171794.28588.65.camel@duffman>  <420D8448.9010000@sun.com>
Message-ID: <1108182667.25868.9.camel@duffman>

On Fri, 2005-02-11 at 20:21 -0800, Nitin Hande wrote:

> Allright, so I am not seeing this anymore on the test setup here. I will 
> keep a watch and conduct some more experiments over weekend if time permits.

Does that mean you fixed it?  Or can't reproduce it?  Can you ping or
put any traffic over the higher mtu network?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050211/f9dc6caf/attachment.sig>

From mlleinin at hpcn.ca.sandia.gov  Fri Feb 11 20:06:25 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Fri, 11 Feb 2005 20:06:25 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <m3zmyba5p9.fsf@maxwell.lnxi.com>
References: <35EA21F54A45CB47B879F21A91F4862F391A28@taurus.voltaire.com>
	<20050210134835.GA5887@lst.de> <1108056523.18794.2.camel@duffman>
	<20050210173421.GA10760@lst.de>
	<Pine.LNX.4.58.0502101035240.31745@enigma.lanl.gov>
	<1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
	<1108065958.7776.10.camel@localhost>
	<20050210202704.GB29349@esmail.cup.hp.com>
	<1108067799.7746.14.camel@localhost> <m3zmyba5p9.fsf@maxwell.lnxi.com>
Message-ID: <1108181185.28884.392.camel@pulsar>

On Fri, 2005-02-11 at 00:58 -0700, Eric W. Biederman wrote:
> Matt Leininger <mlleinin at hpcn.ca.sandia.gov> writes:
> 
> > On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > > >   uDAPL - Oracle, MPI
> > > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > > 
> > > Lustre will use Sandia Portals AFAIK.
> > > Anyone know what Portals will use?
> > > They might directly program to VAPI or something.
> > > 
> >   There will be a Portals over verbs.  At some point there may be a
> > Portals over kDAPL to support both RDMA ethernet and IB.  
> 
> 
> Cluster filesystems has already implemented native IB support in Lustre
> against gen1. I assume that is a portals.
> 
> You might want to ask them about it some time...
> 
  Already know.  Tri-labs are paying for it.  :)

   - Matt



From Nitin.Hande at Sun.COM  Fri Feb 11 21:04:33 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Fri, 11 Feb 2005 21:04:33 -0800
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <1108182667.25868.9.camel@duffman>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
	<1108170521.28588.61.camel@duffman>
	<1108170692.17781.48.camel@localhost.localdomain>
	<1108171794.28588.65.camel@duffman> <420D8448.9010000@sun.com>
	<1108182667.25868.9.camel@duffman>
Message-ID: <420D8E61.7080404@sun.com>

Tom Duffy wrote:

>On Fri, 2005-02-11 at 20:21 -0800, Nitin Hande wrote:
>
>  
>
>>Allright, so I am not seeing this anymore on the test setup here. I will 
>>keep a watch and conduct some more experiments over weekend if time permits.
>>    
>>
> 
>Does that mean you fixed it?  Or can't reproduce it?  Can you ping or
>put any traffic over the higher mtu network?
>  
>
Its not reproducible.  BTW what is peer interface for this machine ?

Nitin


>-tduffy
>  
>



From iod00d at hp.com  Fri Feb 11 21:58:27 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 11 Feb 2005 21:58:27 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050211184058.I20777@topspin.com>
References: <20050211184058.I20777@topspin.com>
Message-ID: <20050212055827.GW574@esmail.cup.hp.com>

On Fri, Feb 11, 2005 at 06:40:58PM -0800, Libor Michalek wrote:
>   If no one objects, a patch to clean up compile warnings on x86_64.

I haven't applied this patch yet - I read mail on the other side of
a firewall where my machines are. But I don't think it will fix
the compile error below.

I'm not thrilled about the number of "(void *)(unsigned long)" casts.
A small inline function to convert (int) to (void *) would be cleaner.
E.g. (not compiled):
static inline void * hash_arg(int x) { return (void *)(unsigned long) x; }



With rev 1781, on ia64 linux-2.6.11-rc3, I'm getting the expected warnings
and one error:
drivers/infiniband/ulp/sdp/sdp_iocb.c: In function `_sdp_iocb_page_save':
drivers/infiniband/ulp/sdp/sdp_iocb.c:194: error: request for member `pgd' in something not a structure or union

The offending line:
                pmd = pmd_offset(pgd, addr);

pmd_offset() is defined in include/asm/pgtable.h:

/* Find an entry in the second-level page table.. */
#define pmd_offset(dir,addr) \
        ((pmd_t *) pud_page(*(dir)) + (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1)))

#define pud_page(pud) ((unsigned long) __va(pud_val(pud) & _PFN_MASK))

and ia64 uses asm-generic for the last bit:
include/asm-generic/pgtable-nopud.h:#define pud_val(x) (pgd_val((x).pgd))

include/asm/page.h:# define pgd_val(x)  ((x).pgd)


But the offending line is getting expanded to:
            pmd = ((pmd_t *) ((unsigned long) ({ia64_va _v; _v.l = (long) (((((*(pgd)).pgd).pgd)) & (((((unsigned long)(1)) << 50) - 1) & ~0xfffUL)); _v.f.reg = -1; _v.p;})) + (((addr) >> (14 + (14 -3))) & ((1UL << (14 -3)) - 1)));

i.e. ((((pgd)).pgd).pgd)  - one too many since asm/page.h defines it as:

  typedef struct { unsigned long pgd; } pgd_t;


I'm guessing _sdp_iocb_page_save() is just broken.
The ia64 kernel builds fine.
And every other use of pmd_offset() looks like this:
	mm/rmap.c:      pmd = pmd_offset(pud, address);

ie. they are passing in pud_t * instead of pgd_t *.
Someone care to enlighten me on which usage is correct?


One more annoyance: sdp_iocb.c only includes one local file (sdp_main.h).
That's more than I can take on tonight.

And sdp_main.h violates one of the kernel include file rules:
	include asm/ headers *after* linux/ headers.

I'd like to submit a patch for the include file ordering
but can't build right now on ia64.

thanks,
grant


From mlleinin at hpcn.ca.sandia.gov  Fri Feb 11 23:48:42 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Fri, 11 Feb 2005 23:48:42 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
Message-ID: <1108194522.10158.50.camel@localhost>


  The presentations given at the 2005 OpenIB Developers Workshop in
Sonoma this week are available at www.openib.org/workshop.html.  We
still have a few more presentations to track down, but most of them are
there.  

  - Matt



From iod00d at hp.com  Fri Feb 11 23:52:38 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 11 Feb 2005 23:52:38 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050212055827.GW574@esmail.cup.hp.com>
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
Message-ID: <20050212075238.GB2982@esmail.cup.hp.com>

On Fri, Feb 11, 2005 at 09:58:27PM -0800, Grant Grundler wrote:
> I'm guessing _sdp_iocb_page_save() is just broken.
> The ia64 kernel builds fine.
> And every other use of pmd_offset() looks like this:
> 	mm/rmap.c:      pmd = pmd_offset(pud, address);

The following patch makes _sdp_iocb_page_save() look like the
code in mm/rmap.c. I have no clue if it's right or not.
But it now builds on ia64. Not tested yet.

And rmap.c uses pXX_present() macro.
I don't know if that should be used as well.

Signed-off-by: Grant Grundler <grant.grundler at hp.com>


Index: ulp/sdp/sdp_iocb.c
===================================================================
--- ulp/sdp/sdp_iocb.c	(revision 1781)
+++ ulp/sdp/sdp_iocb.c	(working copy)
@@ -139,6 +139,7 @@ static int _sdp_iocb_page_save(struct sd
 	struct page *page;
 	unsigned long pfn;
 	pgd_t *pgd;
+	pud_t *pud;
 	pmd_t *pmd;
 	pte_t *ptep;
 	pte_t  pte;
@@ -186,12 +187,17 @@ static int _sdp_iocb_page_save(struct sd
 	     0 < size;
 	     counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) {
 
-		pgd = pgd_offset_gate(iocb->mm, addr);
+		pgd = pgd_offset(iocb->mm, addr);
 		if (!pgd || pgd_none(*pgd)) {
 			break;
 		}
-		
-		pmd = pmd_offset(pgd, addr);
+
+		pud = pud_offset(pgd, addr);
+		if (!pud || pud_none(*pud)) {
+			break;
+		}
+
+		pmd = pmd_offset(pud, addr);
 		if (!pmd || pmd_none(*pmd)) {
 			break;
 		}


From halr at voltaire.com  Sat Feb 12 04:28:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Feb 2005 07:28:10 -0500
Subject: [openib-general] Re: Solaris 10 with OpenIB OpenSM
In-Reply-To: <1108171794.28588.65.camel@duffman>
References: <1108167498.17781.8.camel@localhost.localdomain>
	<1108168523.28588.53.camel@duffman>
	<1108168999.17781.28.camel@localhost.localdomain>
	<1108170521.28588.61.camel@duffman>
	<1108170692.17781.48.camel@localhost.localdomain>
	<1108171794.28588.65.camel@duffman>
Message-ID: <1108211289.17781.72.camel@localhost.localdomain>

On Fri, 2005-02-11 at 20:29, Tom Duffy wrote:
> On Fri, 2005-02-11 at 20:11 -0500, Hal Rosenstock wrote:
> > The log indicates that the MTU is 4 which is 2048. I also saw this in
> > the IB trace.

The issue is that Solaris is pendantic about the selectors. OpenIB
OpenSM was not filling in the MTUSelector (or the other selectors). The
spec says they should be set to exactly (2) on a query response. Solaris
must be checking them. This may be some ambiguity in the spec as most
other cases this can be ignored. I would think the selector is
meaningless on a response but the spec doesn't say this :-( It appears
other implementations treated it as such.

> > > Yes, I would like the patch.
> 
> BTW, Solaris does work now (Yippie!!!!), even without this patch (albeit
> with the MTU workaround).  I just confirmed on my network.  Thanks.

Here's an updated patch which is needed for Solaris. I also need to
retest OpenIB to make sure this still works but from code inspection it
does. (It still has one question outstanding so I may apply in 2 steps) 

-- Hal

Index: osm_sa_mcmember_record.c
===================================================================
--- osm_sa_mcmember_record.c	(revision 1781)
+++ osm_sa_mcmember_record.c	(working copy)
@@ -176,8 +176,8 @@
   p_recvd_mcmember_rec = p_ctxt->p_mcmember_rec;
   p_rcv = p_ctxt->p_rcv;
 
-  /* Why not comparing all the MGID ???? */
-  /* different scope can sneek in for the same MGID ? */
+  /* Why not compare the entire MGID ???? */
+  /* different scope can sneak in for the same MGID ? */
   /* EZ: I changed it to full compare ! */
   if (cl_memcmp(&p_mgrp->mcmember_rec.mgid,
                 &p_recvd_mcmember_rec->mgid,
@@ -443,6 +443,11 @@
 
   *p_resp_mcmember_rec = *p_mcmember_rec;
 
+  /* Fill in the mtu, rate, and packet lifetime selectors */
+  p_resp_mcmember_rec->mtu |= 2<<6; /* exactly */
+  p_resp_mcmember_rec->rate |=  2<<6; /* exactly */
+  p_resp_mcmember_rec->pkt_life |= 2<<6; /* exactly */
+
   status = osm_vendor_send(
     p_resp_madw->h_bind,
     p_resp_madw,
@@ -732,7 +737,7 @@
 
   OSM_LOG_ENTER( p_rcv->p_log, __validate_requested_mgid );
 
-  /* 14-a : mcaut GUID must start with 0xFF */
+  /* 14-a : mcast GID must start with 0xFF */
   if (p_mcm_rec->mgid.multicast.header[0] != 0xFF)
   {
     osm_log( p_rcv->p_log, OSM_LOG_ERROR,
@@ -1009,8 +1014,8 @@
     {
 
       /*
-       * acording to the same o15-0.1.14 we must get the stored JoinState and
-       * the request JoinState be oposite to leave - other wise just update it
+       * according to the same o15-0.1.14 we must get the stored JoinState and
+       * the request JoinState be opposite to leave - otherwise just update it
        */
       port_join_state = p_mcm_port->scope_state & 0x0F;
       new_join_state = 
@@ -1419,7 +1424,7 @@
   osm_mcmr_recv_t*            const p_rcv  = p_ctxt->p_rcv;
   const ib_member_rec_t*      p_rcvd_rec   = p_ctxt->p_mcmember_rec;
   const osm_physp_t*          p_req_physp  = p_ctxt->p_req_physp;
-  ib_member_rec_t             match_rec; // since we might change scope_state
+  ib_member_rec_t             match_rec; /* since we might change scope_state */
   ib_net64_t                  comp_mask    = p_ctxt->comp_mask;
   osm_mcm_port_t*             p_mcm_port;
   ib_net64_t portguid = p_rcvd_rec->port_gid.unicast.interface_id;
@@ -1433,7 +1438,7 @@
           "__osm_sa_mcm_by_comp_mask_cb: "
           "Checking mlid:0x%X\n",
           cl_ntoh16(p_mgrp->mlid ));
-  /* first try to eliminate the group by MGID, MLID or P_Key */
+  /* first try to eliminate the group by MGID, MLID, or P_Key */
   if ((IB_MCR_COMPMASK_MGID & comp_mask) &&
       cl_memcmp(&p_rcvd_rec->mgid, &p_mgrp->mcmember_rec.mgid, sizeof(ib_gid_t))) {
     goto Exit;
@@ -1490,10 +1495,10 @@
     sl_flow_hop_mask = sl_flow_hop_mask | 0x000F;
 
   if (IB_MCR_COMPMASK_FLOW & comp_mask)
-    sl_flow_hop_mask = sl_flow_hop_mask | 0x0FF0;
+    sl_flow_hop_mask = sl_flow_hop_mask | 0xFFFFFF0;
 
   if (IB_MCR_COMPMASK_HOP & comp_mask)
-    sl_flow_hop_mask = sl_flow_hop_mask | 0xF000;
+    sl_flow_hop_mask = sl_flow_hop_mask | 0xFF000000;
 
   if ((p_rcvd_rec->sl_flow_hop & sl_flow_hop_mask) !=
       (p_mgrp->mcmember_rec.sl_flow_hop & sl_flow_hop_mask)) goto Exit;






From tduffy at sun.com  Sat Feb 12 06:09:24 2005
From: tduffy at sun.com (Tom Duffy)
Date: Sat, 12 Feb 2005 06:09:24 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108194522.10158.50.camel@localhost>
References: <1108194522.10158.50.camel@localhost>
Message-ID: <1108217364.25868.14.camel@duffman>

On Fri, 2005-02-11 at 23:48 -0800, Matt Leininger wrote:
>   The presentations given at the 2005 OpenIB Developers Workshop in
> Sonoma this week are available at www.openib.org/workshop.html.  We
> still have a few more presentations to track down, but most of them are
> there.  

Matt,

I am getting 404 on most of the pdfs.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050212/0e993ec2/attachment.sig>

From mlleinin at hpcn.ca.sandia.gov  Sat Feb 12 11:40:31 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Sat, 12 Feb 2005 11:40:31 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108217364.25868.14.camel@duffman>
References: <1108194522.10158.50.camel@localhost>
	<1108217364.25868.14.camel@duffman>
Message-ID: <1108237232.8527.5.camel@localhost>

On Sat, 2005-02-12 at 06:09 -0800, Tom Duffy wrote:
> On Fri, 2005-02-11 at 23:48 -0800, Matt Leininger wrote:
> >   The presentations given at the 2005 OpenIB Developers Workshop in
> > Sonoma this week are available at www.openib.org/workshop.html.  We
> > still have a few more presentations to track down, but most of them are
> > there.  
> 
> Matt,
> 
> I am getting 404 on most of the pdfs.
> 
fixed.

   - Matt



From danb at voltaire.com  Sun Feb 13 00:37:12 2005
From: danb at voltaire.com (Dan Bar Dov)
Date: Sun, 13 Feb 2005 10:37:12 +0200
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>

 

> 
> > We need the same API on all platforms.
> 
> I don't know who "we" here is, but we as the kernel developer 
> community certainly disagree.

So how do you describe scsi mid layer?
Scsi mid layer is exaclty a unifying API that hides the specific hardware device drivers from the upper layer abstractions.
The kdapl model is identical to the scsi model - a single in-kernel API that hides different hardware specific implementations of RDMA.

Dan

> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
> 


From ebiederman at lnxi.com  Sun Feb 13 00:58:42 2005
From: ebiederman at lnxi.com (Eric W. Biederman)
Date: 13 Feb 2005 01:58:42 -0700
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
Message-ID: <m3fz00q1j1.fsf@maxwell.lnxi.com>

"Dan Bar Dov" <danb at voltaire.com> writes:

> > > We need the same API on all platforms.
> > 
> > I don't know who "we" here is, but we as the kernel developer 
> > community certainly disagree.
> 
> So how do you describe scsi mid layer?
> Scsi mid layer is exaclty a unifying API that hides the specific hardware device
> drivers from the upper layer abstpractions.
> 
> The kdapl model is identical to the scsi model - a single in-kernel API that
> hides different hardware specific implementations of RDMA.

I don't think I have seen any one argue that the concept of an RDMA abstraction
is bad.  I have simply seen that kDAPL is bad.

The scsi mid layer is a really bad example as it is deprecated code.  But
ignoring that.  The scsi mid layer  is not OS agnostic, and it is even
free to change between kernel versions if a better way is found.  Plus
the API is in the linux kernel coding style.

kDAPL as designed is none of those.  The operations and concept it may
be possible to salvage.  kDAPL cannot be, at least not in the
mainstream kernel.

Eric


From 3d at sinate.com  Sun Feb 13 02:57:51 2005
From: 3d at sinate.com (Sinate)
Date: Sun, 13 Feb 2005 10:57:51 +0000
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
Message-ID: <420F32AF.4050307@sinate.com>

Hi. I'm trying to find a simple Mellanox VAPI example (sends some data 
from one host to another, nothing fancy), in C or C++.

I can stare at the Mellanox VAPI reference files, but it doesn't get me 
any where if I want to build a C/C++ program to do something.

Sorry if this is the wrong place to ask.

Cheers.


From hch at lst.de  Sun Feb 13 06:26:04 2005
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 13 Feb 2005 15:26:04 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <m3fz00q1j1.fsf@maxwell.lnxi.com>
References: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
	<m3fz00q1j1.fsf@maxwell.lnxi.com>
Message-ID: <20050213142604.GB1959@lst.de>

On Sun, Feb 13, 2005 at 01:58:42AM -0700, Eric W. Biederman wrote:
> The scsi mid layer is a really bad example as it is deprecated code.

It's certainly not ;-)  Else I agree with your post.



From hch at lst.de  Sun Feb 13 06:25:24 2005
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 13 Feb 2005 15:25:24 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
Message-ID: <20050213142524.GA1959@lst.de>

On Sun, Feb 13, 2005 at 10:37:12AM +0200, Dan Bar Dov wrote:
> > I don't know who "we" here is, but we as the kernel developer 
> > community certainly disagree.
> 
> So how do you describe scsi mid layer?
> Scsi mid layer is exaclty a unifying API that hides the specific hardware device drivers from the upper layer abstractions.
> The kdapl model is identical to the scsi model - a single in-kernel API that hides different hardware specific implementations of RDMA.

It's not really analogue.  In SCSI the hardware (or at least the drivers)
all speak the same protocol (in a few different revisions), and the scsi
midlayer and upper level drivers implement most of that protocol.  The
LLDDs program the different hardware implementing this protocol.

See the SAM-2 or SAM-3 documents for details.


From hch at lst.de  Sun Feb 13 06:29:06 2005
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 13 Feb 2005 15:29:06 +0100
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050213142524.GA1959@lst.de>
References: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
	<20050213142524.GA1959@lst.de>
Message-ID: <20050213142906.GA2083@lst.de>

On Sun, Feb 13, 2005 at 03:25:24PM +0100, Christoph Hellwig wrote:
> On Sun, Feb 13, 2005 at 10:37:12AM +0200, Dan Bar Dov wrote:
> > > I don't know who "we" here is, but we as the kernel developer 
> > > community certainly disagree.
> > 
> > So how do you describe scsi mid layer?
> > Scsi mid layer is exaclty a unifying API that hides the specific hardware device drivers from the upper layer abstractions.
> > The kdapl model is identical to the scsi model - a single in-kernel API that hides different hardware specific implementations of RDMA.
> 
> It's not really analogue.  In SCSI the hardware (or at least the drivers)
> all speak the same protocol (in a few different revisions), and the scsi
> midlayer and upper level drivers implement most of that protocol.  The
> LLDDs program the different hardware implementing this protocol.
> 
> See the SAM-2 or SAM-3 documents for details.

The right analogy for kDAPL in scsi land would be the CAM standard from
T10.org that tried to specifiy in-kernel interfaces for SCSI.  While a
few Operating Systems tried to model their stack afer CAM (Digital Unix,
FreeBSD), it soon became clear that the standard isn't implementable 1:1
in a real life system, was a hidrance in implementing clean and lean
driver and had no chance following the development of new SAM
substandards.


From mhouston at graphics.stanford.edu  Sun Feb 13 11:33:02 2005
From: mhouston at graphics.stanford.edu (Mike Houston)
Date: Sun, 13 Feb 2005 11:33:02 -0800
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
In-Reply-To: <420F32AF.4050307@sinate.com>
References: <420F32AF.4050307@sinate.com>
Message-ID: <420FAB6E.4010805@graphics.stanford.edu>

This has been a running request now for quite some time.  Everyone says 
its a good idea, but it's hard to get it actually done.  (And before 
people say, "this is an open source project so why don't you do it 
yourself", keep reading)  Since the spec is slowly evolving, we really 
need the OpenIB developers or the IBTA to write a programmer's manual 
for VAPI.  Folks working on the ULPs should do the same. 

I learned (poorly) how to use VAPI by reading the original Mellanox 
perf_main code, which was a truly awful experience.  This has lead to me 
writing a VERY fragile NAL using VAPI that has performance 
characteristics which are very difficult to reason about. 

Without updated API documentation and programming guides, IB is going to 
be a hard sell to many folks unless their important apps already work 
perfectly an fast on IB somehow.  For example, what fields do I set in 
the many different structs to get good performance?  The event handlers 
are cool, but how do you actually use them?  How do I figure out if a 
cool feature is actually supported, i.e. caps bits?  We need something 
like a "HelloIB" for all of the different ways to access the IB hardware 
(*DAPL, VAPI, SDP (with all the socket extensions, etc)), along with a 
programmer's guide.  Something like if you had to teach an intro 
undergrad course on networking, but instead of using TCP, you had to 
teach using IB.  What would your slides look like?  What would the 
homework/project questions and solutions be...  i.e UD vs. RC.  
send/recv vs. RDMA (read/write).  Basic explanations of 
WQEs/SRQs/CQs/QPs/etc.

If you look at the OpenIB archive of this list, you can find an example 
or two of how to write a ping test.  But, getting good performance 
through VAPI seems to still be a dark art.  It's obviously doable since 
the OSU folks are getting pretty good performance with their MVAPICH MPI 
implementation on IB.  I can get pretty good performance point to point, 
but 1->N, N->1, M->N, and all to all continue to be a problem for me.

This is why I've made heavy use out of SDP, but to get all the 
performance of IB, especially on PCIe, it looks like we will have to 
write code pretty close to the metal, be it VAPI or kDAPL(?).

This is just my opinion as and end user, I could be wrong.

-Mike

Sinate wrote:

> Hi. I'm trying to find a simple Mellanox VAPI example (sends some data 
> from one host to another, nothing fancy), in C or C++.
>
> I can stare at the Mellanox VAPI reference files, but it doesn't get 
> me any where if I want to build a C/C++ program to do something.
>
> Sorry if this is the wrong place to ask.
>
> Cheers.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general



From paul.baxter at dsl.pipex.com  Sun Feb 13 11:54:43 2005
From: paul.baxter at dsl.pipex.com (Paul Baxter)
Date: Sun, 13 Feb 2005 19:54:43 -0000
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
References: <420F32AF.4050307@sinate.com>
	<420FAB6E.4010805@graphics.stanford.edu>
Message-ID: <000901c51205$dd1d5270$8000000a@blorp>

From: "Mike Houston" <mhouston at graphics.stanford.edu>

> This has been a running request now for quite some time.  Everyone says 
> its a good idea, but it's hard to get it actually done.  (And before 
> people say, "this is an open source project so why don't you do it 
> yourself", keep reading)  Since the spec is slowly evolving, we really 
> need the OpenIB developers or the IBTA to write a programmer's manual for 
> VAPI.  Folks working on the ULPs should do the same.

I agree entirely with the sentiment of this email.

I know to my team's cost that this is an area where even after you buy kit 
from a vendor, you still end up with sparse documentation, a few example 
templates to copy and very little in the way of tutorial material to bring 
programmers up to speed.

We have the Mindshare book and the IB specs but these aren't exactly easy 
tutorial material. Some of the code fragments here have been helpful too, 
but in general we have kept with SDP purely because other areas seem so 
daunting.

Alternatively I'm seriously thinking of offering Roland a job ;)

Paul Baxter
Despite my levity above, opinions expressed are my own, not my company's.





From 3d at sinate.com  Sun Feb 13 13:40:27 2005
From: 3d at sinate.com (Sinate)
Date: Sun, 13 Feb 2005 21:40:27 +0000
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
In-Reply-To: <420FAB6E.4010805@graphics.stanford.edu>
References: <420F32AF.4050307@sinate.com>
	<420FAB6E.4010805@graphics.stanford.edu>
Message-ID: <420FC94B.9060505@sinate.com>

Thanks Mike. I'll put your views to my boss who is asking me to write 
the VAPI app. So far I've also used SDP, with some pretty reasonable 
results. However, the customer always wants more. I've also put this 
question to Infinicon. I won't be surprised if I don't get a response.

Guess I better start searching OpenIB archive...

Cheers,

Sinate.

Mike Houston wrote:

> This has been a running request now for quite some time.  Everyone 
> says its a good idea, but it's hard to get it actually done.  (And 
> before people say, "this is an open source project so why don't you do 
> it yourself", keep reading)  Since the spec is slowly evolving, we 
> really need the OpenIB developers or the IBTA to write a programmer's 
> manual for VAPI.  Folks working on the ULPs should do the same.
> I learned (poorly) how to use VAPI by reading the original Mellanox 
> perf_main code, which was a truly awful experience.  This has lead to 
> me writing a VERY fragile NAL using VAPI that has performance 
> characteristics which are very difficult to reason about.
> Without updated API documentation and programming guides, IB is going 
> to be a hard sell to many folks unless their important apps already 
> work perfectly an fast on IB somehow.  For example, what fields do I 
> set in the many different structs to get good performance?  The event 
> handlers are cool, but how do you actually use them?  How do I figure 
> out if a cool feature is actually supported, i.e. caps bits?  We need 
> something like a "HelloIB" for all of the different ways to access the 
> IB hardware (*DAPL, VAPI, SDP (with all the socket extensions, etc)), 
> along with a programmer's guide.  Something like if you had to teach 
> an intro undergrad course on networking, but instead of using TCP, you 
> had to teach using IB.  What would your slides look like?  What would 
> the homework/project questions and solutions be...  i.e UD vs. RC.  
> send/recv vs. RDMA (read/write).  Basic explanations of 
> WQEs/SRQs/CQs/QPs/etc.
>
> If you look at the OpenIB archive of this list, you can find an 
> example or two of how to write a ping test.  But, getting good 
> performance through VAPI seems to still be a dark art.  It's obviously 
> doable since the OSU folks are getting pretty good performance with 
> their MVAPICH MPI implementation on IB.  I can get pretty good 
> performance point to point, but 1->N, N->1, M->N, and all to all 
> continue to be a problem for me.
>
> This is why I've made heavy use out of SDP, but to get all the 
> performance of IB, especially on PCIe, it looks like we will have to 
> write code pretty close to the metal, be it VAPI or kDAPL(?).
>
> This is just my opinion as and end user, I could be wrong.
>
> -Mike
>
> Sinate wrote:
>
>> Hi. I'm trying to find a simple Mellanox VAPI example (sends some 
>> data from one host to another, nothing fancy), in C or C++.
>>
>> I can stare at the Mellanox VAPI reference files, but it doesn't get 
>> me any where if I want to build a C/C++ program to do something.
>>
>> Sorry if this is the wrong place to ask.
>>
>> Cheers.
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>>
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>
>
>


From Tom.Duffy at Sun.COM  Sun Feb 13 17:32:40 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Sun, 13 Feb 2005 17:32:40 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108237232.8527.5.camel@localhost>
References: <1108194522.10158.50.camel@localhost>
	<1108217364.25868.14.camel@duffman> <1108237232.8527.5.camel@localhost>
Message-ID: <1108344760.31946.4.camel@duffman>

On Sat, 2005-02-12 at 11:40 -0800, Matt Leininger wrote: 
> fixed.

Thanks.

Although, the SRP talk link is b0rked.

And the IPoIB talk should link to 
http://www.openib.org/docs/oib_wkshp_022005/ipoib-sdp-topspin-lmichalek.pdf

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050213/0daaf140/attachment.sig>

From mlleinin at hpcn.ca.sandia.gov  Sun Feb 13 19:04:48 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Sun, 13 Feb 2005 19:04:48 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108344760.31946.4.camel@duffman>
References: <1108194522.10158.50.camel@localhost>
	<1108217364.25868.14.camel@duffman> <1108237232.8527.5.camel@localhost>
	<1108344760.31946.4.camel@duffman>
Message-ID: <1108350288.18616.36.camel@localhost>

On Sun, 2005-02-13 at 17:32 -0800, Tom Duffy wrote:
> On Sat, 2005-02-12 at 11:40 -0800, Matt Leininger wrote: 
> > fixed.
> 
> Thanks.
> 
> Although, the SRP talk link is b0rked.

   I removed the bad link.  I don't have this talk yet.  
> 
> And the IPoIB talk should link to 
> http://www.openib.org/docs/oib_wkshp_022005/ipoib-sdp-topspin-lmichalek.pdf
> 
  fixed

 Thanks,

	- Matt



From bill at LibreSolutions.com  Sun Feb 13 23:45:39 2005
From: bill at LibreSolutions.com (Bill Thompson)
Date: Sun, 13 Feb 2005 23:45:39 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <DOEHKBHAGBADBEKBCHKCMEBFDEAA.bill@thompson.name>
References: <DOEHKBHAGBADBEKBCHKCMEBFDEAA.bill@thompson.name>
Message-ID: <1108367138.3078.15.camel@wynux.LibreSolutions>

Hope everyone had a nice time in Sonoma.  Sorry I missed it... 

Got a question:

Can someone take a SWAG at when an OpenIB stack (OpenSM, OpenMPI,
uDAPL, iSCSI, iSER, IPoIB, SDP, kDAPL) will be in RHE 4?

Now that could be CVS or Licensed from one of your companies...

Thanks,

/bill

BTW - I'm a consultant and this is not my client's address.

> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> >
> > To unsubscribe, please visit
> > http://openib.org/mailman/listinfo/openib-general
> >
-- 
William D. "Bill" Thompson, CTO
Libre Business Solutions, Inc.
San Mateo, California

Type bits/keyID Date User ID
pub  1024D/EDD107C7 2004-05-09 Bill Thompson (Libre) <bill at LibreSolutions.com>
sub  2048g/89B937CD 2004-05-09

Key fingerprint = 79E3 50E4 F69D BA4F F24E  F3DC 5E72 01A2 EDD1 07C7
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050213/111148ed/attachment.sig>

From mlleinin at hpcn.ca.sandia.gov  Mon Feb 14 01:45:21 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Mon, 14 Feb 2005 01:45:21 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108367138.3078.15.camel@wynux.LibreSolutions>
References: <DOEHKBHAGBADBEKBCHKCMEBFDEAA.bill@thompson.name>
	<1108367138.3078.15.camel@wynux.LibreSolutions>
Message-ID: <1108374321.18616.94.camel@localhost>

On Sun, 2005-02-13 at 23:45 -0800, Bill Thompson wrote:
> Hope everyone had a nice time in Sonoma.  Sorry I missed it... 
> 
> Got a question:
> 
> Can someone take a SWAG at when an OpenIB stack (OpenSM, OpenMPI,
> uDAPL, iSCSI, iSER, IPoIB, SDP, kDAPL) will be in RHE 4?
> 
> Now that could be CVS or Licensed from one of your companies...

  At the workshop a RedHat spokesperson mentioned that they want an
OpenIB stack the has multi-vendor support so your "licensed from one of
your companies" stack probably won't fly with RedHat.  

  This spokesperson also mentioned that RedHat would have pieces of
OpenIB in RHEL 4.  How much of the OpenIB stack get into RHEL 4 depends
on how far along we are.  I'd guess the initial RH release would have
support for IPoIB, with other ULPs being added as updates as they
stabilize on OpenIB.  It just depends on how long it takes to stabilize
the various components.  

  Open-MPI and uDAPL work can start once we have a user-space access
layer (verbs, cm, etc.).  Roland mentioned that this would be ready for
early testing in a couple of weeks.  Sandia and Los Alamos Labs will
then start adding OpenIB support to Open-MPI.  

  Several developers have volunteered to work on uDAPL.  The DAT
collaborative is working to get a GPL/BSD version of uDAPL to OpenIB.
James and Arkady from NetApp gave a DAT talk.  They mentioned needing 1
month to get the GPL code, and another month or two to fork the code
base.  I'm not sure why 2-3 months is needed to do this.  Perhaps James
and/or Arkady can comment.  I think we need to see the GPL/BSD uDAPL
code for OpenIB in the next couple of weeks.  Better to put the code in
the open so everyone can see what changes are required.  Same goes for
kDAPL.  Was the u/kDAPL implementation for OpenIB going to live at
openib.org or stay under cvs at the sourceforge dapl project?

I think iSCSI support will come from the Linux-iSCSI project on
SourceForge.  Libor has an SDP implementation for OpenIB that iSCSI can
use, but it still needs some tuning and features added.  There is also
the issue of Microsoft IP in SDP.  This was discussed at the workshop -
I thought is was moved to a BOF, but I didn't hear how that turned
out.  

iSER requires kDAPL.  The recent kDAPL discussions on this list and the
linux-iscsi-devel list point to several issues that need to be resolved
if kDAPL is ever going to have a chance of getting in the kernel.  

  Thanks,
	
	- Matt






From itoumsn at nttdata.co.jp  Mon Feb 14 03:25:19 2005
From: itoumsn at nttdata.co.jp (Masanori ITOH)
Date: Mon, 14 Feb 2005 20:25:19 +0900 (JST)
Subject: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1108061485.18794.35.camel@duffman>
References: <1108057387.18794.10.camel@duffman> <20050210174616.GA10990@lst.de>
	<1108061485.18794.35.camel@duffman>
Message-ID: <20050214.202519.133168027.itoumsn@nttdata.co.jp>


Hi,

From: Tom Duffy <tduffy at sun.com>
Subject: Re: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Date: Thu, 10 Feb 2005 10:51:25 -0800

> On Thu, 2005-02-10 at 18:46 +0100, Christoph Hellwig wrote:
> > Maybe you should lay down the requirement first.
> 
> I'll take a crack at it.  Let me know where I am off base.
> 
> >  - why do we need an intermediate API
> 
> To get things working today with the code that people have already
> written.  To create a proof of concept.
> 
> In any event, I am not proposing this API get kernel inclusion.  Only a
> wider audience than in the sourceforge DAPL project cvs tree.
> 
> >  - what differences does it
> 
> ??
> 
> >  - what devices does it abstract
> 
> IB for now, any other RDMA capable transports later.

AFAIK, Myricom has their own working uDAPL implementation.
  http://www.myri.com/scs/
I think there is no reason why they don't consider kDAPL too.

> >  - what are the users
> 
> NFS over RDMA, maybe -- so RPC.  Anybody else know others?

There is one more publicly available kDAPL consumer, DAFS.

Among the following page, you can find a working DAFS server and client suite
called 'fdafs'. That runs in the kernel space and provides complete
UNIX file I/O semantics.

  http://sourceforge.net/projects/dafs/

Also, it's provided as an open source software, and it's license is BSD/GPL2
dual license.
Unfortunatelly, at this moment, the above fdafs (beta1.0) works on only
2.4 kernel, but we have ported that into 2.6 kernel, and I'm planning
to contribute our work to the community.

-Masanori


From itoumsn at nttdata.co.jp  Mon Feb 14 03:26:55 2005
From: itoumsn at nttdata.co.jp (Masanori ITOH)
Date: Mon, 14 Feb 2005 20:26:55 +0900 (JST)
Subject: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <20050213142906.GA2083@lst.de>
References: <D4F8F0B3820E754C887699BEF26A89405A8F38@taurus.voltaire.com>
	<20050213142524.GA1959@lst.de> <20050213142906.GA2083@lst.de>
Message-ID: <20050214.202655.47341701.itoumsn@nttdata.co.jp>

Hi,

From: Christoph Hellwig <hch at lst.de>
Subject: Re: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Date: Sun, 13 Feb 2005 15:29:06 +0100

> On Sun, Feb 13, 2005 at 03:25:24PM +0100, Christoph Hellwig wrote:
> > On Sun, Feb 13, 2005 at 10:37:12AM +0200, Dan Bar Dov wrote:
> > > > I don't know who "we" here is, but we as the kernel developer 
> > > > community certainly disagree.
> > > 
> > > So how do you describe scsi mid layer?
> > > Scsi mid layer is exaclty a unifying API that hides the specific hardware device drivers from the upper layer abstractions.
> > > The kdapl model is identical to the scsi model - a single in-kernel API that hides different hardware specific implementations of RDMA.
> > 
> > It's not really analogue.  In SCSI the hardware (or at least the drivers)
> > all speak the same protocol (in a few different revisions), and the scsi
> > midlayer and upper level drivers implement most of that protocol.  The
> > LLDDs program the different hardware implementing this protocol.
> > 
> > See the SAM-2 or SAM-3 documents for details.
> 
> The right analogy for kDAPL in scsi land would be the CAM standard from
> T10.org that tried to specifiy in-kernel interfaces for SCSI.  While a
> few Operating Systems tried to model their stack afer CAM (Digital Unix,
> FreeBSD), it soon became clear that the standard isn't implementable 1:1
> in a real life system, was a hidrance in implementing clean and lean
> driver and had no chance following the development of new SAM
> substandards.

I think there is a great difference between T10.org and DAT activity.
In case of T10.org, I guess the development style was like
spec. first and then implementations. This sometimes causes problems like
you mentioned.
But, the DAT activity is more like IETF style. Here, I mean
DAT spec. people have been brushed up their documents getting feedbacks
from implementation people. 
Thus, I believe the current kDAPL spec. can be a good start point of
the discussion.

-Masanori


From halr at voltaire.com  Mon Feb 14 05:00:26 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 14 Feb 2005 08:00:26 -0500
Subject: [openib-general] OpenIB OpenSM and Reserved Fields on Transmit
Message-ID: <1108385864.21161.994.camel@localhost.localdomain>

Hi Eitan,

With OpenSM, in general, is there an approach that is being used to
ensure the reserved fields at set to 0 on send ? I couldn't ascertain
this from the code and I do observe reserved fields set to non 0 on
send. One approach would be to clear the MAD when it is first obtained.
Another would be to clear the reserved fields in each SA record. I am
leaning towards the former approach. Is this the responsibility of the
vendor layer ?

Thanks.

-- Hal



From eitan at mellanox.co.il  Mon Feb 14 05:33:30 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Mon, 14 Feb 2005 15:33:30 +0200
Subject: [openib-general] RE: OpenIB OpenSM and Reserved Fields on Transmit
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEE13@mtlex01.yok.mtl.com>

The lowest level to catch all MAD buffer allocations is in the
osm_vendor_get() call.
It could certainly zero out the mad buffer it provides.

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Monday, February 14, 2005 3:00 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: OpenIB OpenSM and Reserved Fields on Transmit
> 
> Hi Eitan,
> 
> With OpenSM, in general, is there an approach that is being used to
> ensure the reserved fields at set to 0 on send ? I couldn't ascertain
> this from the code and I do observe reserved fields set to non 0 on
> send. One approach would be to clear the MAD when it is first obtained.
> Another would be to clear the reserved fields in each SA record. I am
> leaning towards the former approach. Is this the responsibility of the
> vendor layer ?
> 
> Thanks.
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/50654f31/attachment.html>

From halr at voltaire.com  Mon Feb 14 05:38:52 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 14 Feb 2005 08:38:52 -0500
Subject: [openib-general] RE: OpenIB OpenSM and Reserved Fields on Transmit
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEE13@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEE13@mtlex01.yok.mtl.com>
Message-ID: <1108388227.21161.1111.camel@localhost.localdomain>

On Mon, 2005-02-14 at 08:33, Eitan Zahavi wrote:
> The lowest level to catch all MAD buffer allocations is in the
> osm_vendor_get() call.
> It could certainly zero out the mad buffer it provides.

Sounds like this is not being done currently. That's what I wanted to
confirm. Thanks.

-- Hal

> 
> Eitan Zahavi
> Design Technology Director
> Mellanox Technologies LTD
> Tel:+972-4-9097208
> Fax:+972-4-9593245
> P.O. Box 586 Yokneam 20692 ISRAEL
> 
> 
> > -----Original Message-----
> > From: Hal Rosenstock [mailto:halr at voltaire.com]
> > Sent: Monday, February 14, 2005 3:00 PM
> > To: Eitan Zahavi
> > Cc: openib-general at openib.org
> > Subject: OpenIB OpenSM and Reserved Fields on Transmit
> > 
> > Hi Eitan,
> > 
> > With OpenSM, in general, is there an approach that is being used to
> > ensure the reserved fields at set to 0 on send ? I couldn't
> ascertain
> > this from the code and I do observe reserved fields set to non 0 on
> > send. One approach would be to clear the MAD when it is first
> obtained.
> > Another would be to clear the reserved fields in each SA record. I
> am
> > leaning towards the former approach. Is this the responsibility of
> the
> > vendor layer ?
> > 
> > Thanks.
> > 
> > -- Hal
> 



From tziporet at mellanox.co.il  Mon Feb 14 06:41:57 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Mon, 14 Feb 2005 16:41:57 +0200
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ co de
Message-ID: <506C3D7B14CDD411A52C00025558DED6064BEF4D@mtlex01.yok.mtl.com>

Hi,

I don't think we need an example of VAPI code since gen2 will be the real
thing.
When user level verbs will work we will port Mellanox performance test
(known as perf_main) to gen2. 
>From this test one can learn how to get the good performance out of IB.

If you have Mellanox SW you can get perf_main sources.
If you want I can send this code to you, or put it in gen1 in some place
(but Makefiles will not work as is)

Tziporet


-----Original Message-----
From: Sinate [mailto:3d at sinate.com]
Sent: Sunday, February 13, 2005 11:40 PM
To: Mike Houston
Cc: openib-general at openib.org
Subject: Re: [openib-general] Looking for a Mellanox VAPI example C/C++
code


Thanks Mike. I'll put your views to my boss who is asking me to write 
the VAPI app. So far I've also used SDP, with some pretty reasonable 
results. However, the customer always wants more. I've also put this 
question to Infinicon. I won't be surprised if I don't get a response.

Guess I better start searching OpenIB archive...

Cheers,

Sinate.

Mike Houston wrote:

> This has been a running request now for quite some time.  Everyone 
> says its a good idea, but it's hard to get it actually done.  (And 
> before people say, "this is an open source project so why don't you do 
> it yourself", keep reading)  Since the spec is slowly evolving, we 
> really need the OpenIB developers or the IBTA to write a programmer's 
> manual for VAPI.  Folks working on the ULPs should do the same.
> I learned (poorly) how to use VAPI by reading the original Mellanox 
> perf_main code, which was a truly awful experience.  This has lead to 
> me writing a VERY fragile NAL using VAPI that has performance 
> characteristics which are very difficult to reason about.
> Without updated API documentation and programming guides, IB is going 
> to be a hard sell to many folks unless their important apps already 
> work perfectly an fast on IB somehow.  For example, what fields do I 
> set in the many different structs to get good performance?  The event 
> handlers are cool, but how do you actually use them?  How do I figure 
> out if a cool feature is actually supported, i.e. caps bits?  We need 
> something like a "HelloIB" for all of the different ways to access the 
> IB hardware (*DAPL, VAPI, SDP (with all the socket extensions, etc)), 
> along with a programmer's guide.  Something like if you had to teach 
> an intro undergrad course on networking, but instead of using TCP, you 
> had to teach using IB.  What would your slides look like?  What would 
> the homework/project questions and solutions be...  i.e UD vs. RC.  
> send/recv vs. RDMA (read/write).  Basic explanations of 
> WQEs/SRQs/CQs/QPs/etc.
>
> If you look at the OpenIB archive of this list, you can find an 
> example or two of how to write a ping test.  But, getting good 
> performance through VAPI seems to still be a dark art.  It's obviously 
> doable since the OSU folks are getting pretty good performance with 
> their MVAPICH MPI implementation on IB.  I can get pretty good 
> performance point to point, but 1->N, N->1, M->N, and all to all 
> continue to be a problem for me.
>
> This is why I've made heavy use out of SDP, but to get all the 
> performance of IB, especially on PCIe, it looks like we will have to 
> write code pretty close to the metal, be it VAPI or kDAPL(?).
>
> This is just my opinion as and end user, I could be wrong.
>
> -Mike
>
> Sinate wrote:
>
>> Hi. I'm trying to find a simple Mellanox VAPI example (sends some 
>> data from one host to another, nothing fancy), in C or C++.
>>
>> I can stare at the Mellanox VAPI reference files, but it doesn't get 
>> me any where if I want to build a C/C++ program to do something.
>>
>> Sorry if this is the wrong place to ask.
>>
>> Cheers.
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>>
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>
>
>

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/9deb2a10/attachment.html>

From tziporet at mellanox.co.il  Mon Feb 14 06:43:10 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Mon, 14 Feb 2005 16:43:10 +0200
Subject: [openib-general] Re: FW: summary of my understanding on our c
	ommon work on openib.org
Message-ID: <506C3D7B14CDD411A52C00025558DED6064BEF4E@mtlex01.yok.mtl.com>

1. This attribute is not used also by gen1 code and we checked with
customers.
2. It is not useful since on RQ there any logical flow will work with
completions.

Tziporet

-----Original Message-----
From: Grant Grundler [mailto:iod00d at hp.com]
Sent: Thursday, February 10, 2005 10:25 PM
To: openib-general at openib.org
Subject: Re: [openib-general] Re: FW: summary of my understanding on our
common work on openib.org


Offline, I exchange email with Roland about this attribute.
Seems like we want to know the answer to these questions:

o is there any code outside of openib.org using this attribute?
o if so, how useful is it?

Roland is certain none of the protocols in the proprietary Topspin stack
uses this feature.

The reasons roland gave all seemed valid for removal except the fact
the product has been shipping for a while and thus it's available to
users regardless of if mellanox supports it or not.
It's reasonable to remove if no one can show a use case.

grant
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/0e6d76bb/attachment.html>

From trimmer at infiniconsys.com  Mon Feb 14 06:46:18 2005
From: trimmer at infiniconsys.com (Rimmer, Todd)
Date: Mon, 14 Feb 2005 09:46:18 -0500
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
Message-ID: <5D78D28F88822E4D8702BB9EEF1A436708A9B0@mercury.infiniconsys.com>

Sinate and Mike,
	Lets take this off line, we can provide some help.  But first we will need to better understand what you are trying to implement (eg. driver or user space app, UD or RC, etc).

Todd Rimmer
> -----Original Message-----
> From: Sinate [mailto:3d at sinate.com]
> Sent: Sunday, February 13, 2005 4:40 PM
> To: Mike Houston
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] Looking for a Mellanox VAPI 
> example C/C++
> code
> 
> 
> Thanks Mike. I'll put your views to my boss who is asking me to write 
> the VAPI app. So far I've also used SDP, with some pretty reasonable 
> results. However, the customer always wants more. I've also put this 
> question to Infinicon. I won't be surprised if I don't get a response.
> 
> Guess I better start searching OpenIB archive...
> 
> Cheers,
> 
> Sinate.
> 
> Mike Houston wrote:
> 
> > This has been a running request now for quite some time.  Everyone 
> > says its a good idea, but it's hard to get it actually done.  (And 
> > before people say, "this is an open source project so why 
> don't you do 
> > it yourself", keep reading)  Since the spec is slowly evolving, we 
> > really need the OpenIB developers or the IBTA to write a 
> programmer's 
> > manual for VAPI.  Folks working on the ULPs should do the same.
> > I learned (poorly) how to use VAPI by reading the original Mellanox 
> > perf_main code, which was a truly awful experience.  This 
> has lead to 
> > me writing a VERY fragile NAL using VAPI that has performance 
> > characteristics which are very difficult to reason about.
> > Without updated API documentation and programming guides, 
> IB is going 
> > to be a hard sell to many folks unless their important apps already 
> > work perfectly an fast on IB somehow.  For example, what 
> fields do I 
> > set in the many different structs to get good performance?  
> The event 
> > handlers are cool, but how do you actually use them?  How 
> do I figure 
> > out if a cool feature is actually supported, i.e. caps 
> bits?  We need 
> > something like a "HelloIB" for all of the different ways to 
> access the 
> > IB hardware (*DAPL, VAPI, SDP (with all the socket 
> extensions, etc)), 
> > along with a programmer's guide.  Something like if you had 
> to teach 
> > an intro undergrad course on networking, but instead of 
> using TCP, you 
> > had to teach using IB.  What would your slides look like?  
> What would 
> > the homework/project questions and solutions be...  i.e UD vs. RC.  
> > send/recv vs. RDMA (read/write).  Basic explanations of 
> > WQEs/SRQs/CQs/QPs/etc.
> >
> > If you look at the OpenIB archive of this list, you can find an 
> > example or two of how to write a ping test.  But, getting good 
> > performance through VAPI seems to still be a dark art.  
> It's obviously 
> > doable since the OSU folks are getting pretty good performance with 
> > their MVAPICH MPI implementation on IB.  I can get pretty good 
> > performance point to point, but 1->N, N->1, M->N, and all to all 
> > continue to be a problem for me.
> >
> > This is why I've made heavy use out of SDP, but to get all the 
> > performance of IB, especially on PCIe, it looks like we 
> will have to 
> > write code pretty close to the metal, be it VAPI or kDAPL(?).
> >
> > This is just my opinion as and end user, I could be wrong.
> >
> > -Mike
> >
> > Sinate wrote:
> >
> >> Hi. I'm trying to find a simple Mellanox VAPI example (sends some 
> >> data from one host to another, nothing fancy), in C or C++.
> >>
> >> I can stare at the Mellanox VAPI reference files, but it 
> doesn't get 
> >> me any where if I want to build a C/C++ program to do something.
> >>
> >> Sorry if this is the wrong place to ask.
> >>
> >> Cheers.
> >> _______________________________________________
> >> openib-general mailing list
> >> openib-general at openib.org
> >> http://openib.org/mailman/listinfo/openib-general
> >>
> >> To unsubscribe, please visit 
> >> http://openib.org/mailman/listinfo/openib-general
> >
> >
> >
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
> 


From tduffy at sun.com  Mon Feb 14 08:23:44 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 14 Feb 2005 08:23:44 -0800
Subject: [openib-general] [PATCH][SDP] Include asm/ headers *after* linux/
	headers
In-Reply-To: <20050212055827.GW574@esmail.cup.hp.com>
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
Message-ID: <1108398224.6001.2.camel@duffman>

On Fri, 2005-02-11 at 21:58 -0800, Grant Grundler wrote:
> And sdp_main.h violates one of the kernel include file rules:
> 	include asm/ headers *after* linux/ headers.

Include asm/ headers *after* linux/ headers.  This builds for me on x86,
x86_64, and sparc64.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_main.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_main.h	(revision 1783)
+++ drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
@@ -57,11 +57,6 @@
 /*
  * kernel includes
  */
-#include <asm/atomic.h>
-#include <asm/byteorder.h>
-#include <asm/pgtable.h>
-#include <asm/io.h>
-
 #include <linux/version.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
@@ -85,6 +80,12 @@
 #include <net/route.h>
 #include <net/dst.h>
 #include <net/ip.h>
+
+#include <asm/atomic.h>
+#include <asm/byteorder.h>
+#include <asm/pgtable.h>
+#include <asm/io.h>
+
 /*
  * IB includes
  */
@@ -107,7 +108,3 @@
 #include "sdp_buff_p.h"
 
 #endif /* _SDP_MAIN_H */
-
-
-
-



From robert.j.woodruff at intel.com  Mon Feb 14 09:23:12 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Mon, 14 Feb 2005 09:23:12 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
Message-ID: <1AC79F16F5C5284499BB9591B33D6F00038CA454@orsmsx408>

 
Matt wrote > Several developers have volunteered to work on uDAPL.  The
DAT
>collaborative is working to get a GPL/BSD version of uDAPL to OpenIB.
>James and Arkady from NetApp gave a DAT talk.  They mentioned needing 1
>month to get the GPL code, and another month or two to fork the code
>base.  I'm not sure why 2-3 months is needed to do this.  Perhaps James
>and/or Arkady can comment.  I think we need to see the GPL/BSD uDAPL
>code for OpenIB in the next couple of weeks.  Better to put the code in
>the open so everyone can see what changes are required.  Same goes for
>kDAPL.  Was the u/kDAPL implementation for OpenIB going to live at
>openib.org or stay under cvs at the sourceforge dapl project?

Once the user-mode support for verbs, user-mode CM, and memory
registration, 
and SA support is ready, we can start on a uDAPL port using the code 
from sourceforge under the BSD license. No need to wait for the license
to
be changed to start the port (IMO), just need it changed before it can
be submitted to openib.org. 

woody


From roland at topspin.com  Mon Feb 14 10:24:47 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 14 Feb 2005 10:24:47 -0800
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
In-Reply-To: <420FAB6E.4010805@graphics.stanford.edu> (Mike Houston's
	message of "Sun, 13 Feb 2005 11:33:02 -0800")
References: <420F32AF.4050307@sinate.com>
	<420FAB6E.4010805@graphics.stanford.edu>
Message-ID: <52y8dr0zkg.fsf@topspin.com>

A few comments about documentation:

There seem to be two distinct requests in this thread.  First, there
is a desire for a basic verbs example that can be used to get
started.  I am writing code like this as I develop the userspace verbs
library, since I need basic tests to make sure the library works.
There will be at least a couple of examples packaged and released as
part of libibverbs.

Second, there is a request for comprehensive documentation that covers
the complete verbs API.  This is a much larger undertaking.  I am
looking at ways I can make this happen, but it is difficult to figure
out how I can put enough time in to make it happen in a reasonable
amount of time.  I'm also not sure that type of documentation some
people might want is even possible -- it will never be possible to
have a cookbook that says "set field X to value Y and your app will go
fast;" the reason field X can take different values is because the
correct value depends on the particular app.  It will never be
possible to get high performance without a deep understanding of both
IB and the app being tuned.

Finally, I don't think inexperience or lack of knowledge means that
someone can't write good documentation.  One of the best ways to learn
about something is to try and write about it.  It should be easy to
get started and cover the basics, and then study and ask for help on
the hard bits.  You can probably trick the experts into doing most of
your work by asking questions and posting docs for review on
openib-general.  (Not having the time or not being interested in doing
it are valid excuses, though ;)

 - R.


From roland at topspin.com  Mon Feb 14 10:26:19 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 14 Feb 2005 10:26:19 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050212075238.GB2982@esmail.cup.hp.com> (Grant Grundler's
	message of "Fri, 11 Feb 2005 23:52:38 -0800")
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<20050212075238.GB2982@esmail.cup.hp.com>
Message-ID: <52u0of0zhw.fsf@topspin.com>

    Grant> The following patch makes _sdp_iocb_page_save() look like
    Grant> the code in mm/rmap.c. I have no clue if it's right or not.
    Grant> But it now builds on ia64. Not tested yet.

4-level page tables went in after 2.6.10, right?  So I think we should
hang onto this patch and apply it to svn after 2.6.11 is out (any day
now, presumably...)

 - R.


From mhouston at graphics.stanford.edu  Mon Feb 14 10:27:22 2005
From: mhouston at graphics.stanford.edu (Mike Houston)
Date: Mon, 14 Feb 2005 10:27:22 -0800
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ co de
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6064BEF4D@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6064BEF4D@mtlex01.yok.mtl.com>
Message-ID: <4210ED8A.8040304@graphics.stanford.edu>

Tziporet Koren wrote:

> Hi,
>
> I don't think we need an example of VAPI code since gen2 will be the 
> real thing.
> When user level verbs will work we will port Mellanox performance test 
> (known as perf_main) to gen2.
> From this test one can learn how to get the good performance out of IB.
>
NO, NO, NO.  If you expect people to learn how to use IB (well) from 
code like perf_main, I think you are very mistaken.  User level verbs 
would be fine, if the performance is there, but we would still need FULL 
documentation and a programmer's manual.  While I'll agree that if the 
ULPs can provide good performance, users might not need to code to VAPI, 
but I'll believe it when I see it.  Regardless, programming 
documentation is still required at all levels .  What are the people 
writing ULPs supposed to use?

> If you have Mellanox SW you can get perf_main sources.
> If you want I can send this code to you, or put it in gen1 in some 
> place (but Makefiles will not work as is)
>
As I said, I have the source to perf_main, that is how I learned to 
write VAPI code poorly...

-Mike

> Tziporet
>
>
> -----Original Message-----
> From: Sinate [mailto:3d at sinate.com]
> Sent: Sunday, February 13, 2005 11:40 PM
> To: Mike Houston
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] Looking for a Mellanox VAPI example C/C++
> code
>
>
> Thanks Mike. I'll put your views to my boss who is asking me to write
> the VAPI app. So far I've also used SDP, with some pretty reasonable
> results. However, the customer always wants more. I've also put this
> question to Infinicon. I won't be surprised if I don't get a response.
>
> Guess I better start searching OpenIB archive...
>
> Cheers,
>
> Sinate.
>
> Mike Houston wrote:
>
> > This has been a running request now for quite some time.  Everyone
> > says its a good idea, but it's hard to get it actually done.  (And
> > before people say, "this is an open source project so why don't you do
> > it yourself", keep reading)  Since the spec is slowly evolving, we
> > really need the OpenIB developers or the IBTA to write a programmer's
> > manual for VAPI.  Folks working on the ULPs should do the same.
> > I learned (poorly) how to use VAPI by reading the original Mellanox
> > perf_main code, which was a truly awful experience.  This has lead to
> > me writing a VERY fragile NAL using VAPI that has performance
> > characteristics which are very difficult to reason about.
> > Without updated API documentation and programming guides, IB is going
> > to be a hard sell to many folks unless their important apps already
> > work perfectly an fast on IB somehow.  For example, what fields do I
> > set in the many different structs to get good performance?  The event
> > handlers are cool, but how do you actually use them?  How do I figure
> > out if a cool feature is actually supported, i.e. caps bits?  We need
> > something like a "HelloIB" for all of the different ways to access the
> > IB hardware (*DAPL, VAPI, SDP (with all the socket extensions, etc)),
> > along with a programmer's guide.  Something like if you had to teach
> > an intro undergrad course on networking, but instead of using TCP, you
> > had to teach using IB.  What would your slides look like?  What would
> > the homework/project questions and solutions be...  i.e UD vs. RC. 
> > send/recv vs. RDMA (read/write).  Basic explanations of
> > WQEs/SRQs/CQs/QPs/etc.
> >
> > If you look at the OpenIB archive of this list, you can find an
> > example or two of how to write a ping test.  But, getting good
> > performance through VAPI seems to still be a dark art.  It's obviously
> > doable since the OSU folks are getting pretty good performance with
> > their MVAPICH MPI implementation on IB.  I can get pretty good
> > performance point to point, but 1->N, N->1, M->N, and all to all
> > continue to be a problem for me.
> >
> > This is why I've made heavy use out of SDP, but to get all the
> > performance of IB, especially on PCIe, it looks like we will have to
> > write code pretty close to the metal, be it VAPI or kDAPL(?).
> >
> > This is just my opinion as and end user, I could be wrong.
> >
> > -Mike
> >
> > Sinate wrote:
> >
> >> Hi. I'm trying to find a simple Mellanox VAPI example (sends some
> >> data from one host to another, nothing fancy), in C or C++.
> >>
> >> I can stare at the Mellanox VAPI reference files, but it doesn't get
> >> me any where if I want to build a C/C++ program to do something.
> >>
> >> Sorry if this is the wrong place to ask.
> >>
> >> Cheers.
> >> _______________________________________________
> >> openib-general mailing list
> >> openib-general at openib.org
> >> http://openib.org/mailman/listinfo/openib-general
> >>
> >> To unsubscribe, please visit
> >> http://openib.org/mailman/listinfo/openib-general
> >
> >
> >
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>



From mhouston at graphics.stanford.edu  Mon Feb 14 10:32:42 2005
From: mhouston at graphics.stanford.edu (Mike Houston)
Date: Mon, 14 Feb 2005 10:32:42 -0800
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
In-Reply-To: <52y8dr0zkg.fsf@topspin.com>
References: <420F32AF.4050307@sinate.com>	<420FAB6E.4010805@graphics.stanford.edu>
	<52y8dr0zkg.fsf@topspin.com>
Message-ID: <4210EECA.5030504@graphics.stanford.edu>

Roland Dreier wrote:

>A few comments about documentation:
>
>There seem to be two distinct requests in this thread.  First, there
>is a desire for a basic verbs example that can be used to get
>started.  I am writing code like this as I develop the userspace verbs
>library, since I need basic tests to make sure the library works.
>There will be at least a couple of examples packaged and released as
>part of libibverbs.
>  
>
This would be really helpful to many people.  I would also suggest that 
as the userspace verbs is written by you and others, to document and 
make available your testing code.  This will get users a foothold.

>Second, there is a request for comprehensive documentation that covers
>the complete verbs API.  This is a much larger undertaking.  I am
>looking at ways I can make this happen, but it is difficult to figure
>out how I can put enough time in to make it happen in a reasonable
>amount of time.  I'm also not sure that type of documentation some
>people might want is even possible -- it will never be possible to
>have a cookbook that says "set field X to value Y and your app will go
>fast;" the reason field X can take different values is because the
>correct value depends on the particular app.  It will never be
>possible to get high performance without a deep understanding of both
>IB and the app being tuned.
>  
>
I understand that this is difficult.  But describing the basics and 
providing examples would sure help.  At least explaining what the fields 
actually mean in the documentation (i.e. what the tend to effect) would 
be a huge improvement.

Of course, these are just my opinions, but I strongly believe I echo 
*many* end users.

-Mike


From roland at topspin.com  Mon Feb 14 10:44:00 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 14 Feb 2005 10:44:00 -0800
Subject: [openib-general] Looking for a Mellanox VAPI example C/C++ code
In-Reply-To: <4210EECA.5030504@graphics.stanford.edu> (Mike Houston's
	message of "Mon, 14 Feb 2005 10:32:42 -0800")
References: <420F32AF.4050307@sinate.com>
	<420FAB6E.4010805@graphics.stanford.edu> <52y8dr0zkg.fsf@topspin.com>
	<4210EECA.5030504@graphics.stanford.edu>
Message-ID: <52psz30yof.fsf@topspin.com>

    Mike> This would be really helpful to many people.  I would also
    Mike> suggest that as the userspace verbs is written by you and
    Mike> others, to document and make available your testing code.
    Mike> This will get users a foothold.

Yes, as I said the example code will be part of the libibverbs
package.  So you would have to make a special effort to get the
userspace verbs code without the examples/ directory.

 - R.


From iod00d at hp.com  Mon Feb 14 10:53:11 2005
From: iod00d at hp.com (Grant Grundler)
Date: Mon, 14 Feb 2005 10:53:11 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <52u0of0zhw.fsf@topspin.com>
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<20050212075238.GB2982@esmail.cup.hp.com>
	<52u0of0zhw.fsf@topspin.com>
Message-ID: <20050214185311.GC10881@esmail.cup.hp.com>

On Mon, Feb 14, 2005 at 10:26:19AM -0800, Roland Dreier wrote:
> 4-level page tables went in after 2.6.10, right?

Sorry - no clue.

> So I think we should
> hang onto this patch and apply it to svn after 2.6.11 is out (any day
> now, presumably...)

uhm...2.6.11-rc4 just released. It could be a bit longer.
But I'm ok with waiting since I know to build/test on 2.6.10 for now.

thanks,
grant


From libor at topspin.com  Mon Feb 14 10:57:39 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 14 Feb 2005 10:57:39 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <52u0of0zhw.fsf@topspin.com>;
	from roland@topspin.com on Mon, Feb 14, 2005 at 10:26:19AM -0800
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<20050212075238.GB2982@esmail.cup.hp.com>
	<52u0of0zhw.fsf@topspin.com>
Message-ID: <20050214105739.A28266@topspin.com>

On Mon, Feb 14, 2005 at 10:26:19AM -0800, Roland Dreier wrote:
>     Grant> The following patch makes _sdp_iocb_page_save() look like
>     Grant> the code in mm/rmap.c. I have no clue if it's right or not.
>     Grant> But it now builds on ia64. Not tested yet.
> 
> 4-level page tables went in after 2.6.10, right?  So I think we should
> hang onto this patch and apply it to svn after 2.6.11 is out (any day
> now, presumably...)

  Do you know which architecture was the impetus for a 4th page
table level?

-Libor


From roland at topspin.com  Mon Feb 14 11:02:02 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 14 Feb 2005 11:02:02 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050214105739.A28266@topspin.com> (Libor Michalek's message
	of "Mon, 14 Feb 2005 10:57:39 -0800")
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<20050212075238.GB2982@esmail.cup.hp.com> <52u0of0zhw.fsf@topspin.com>
	<20050214105739.A28266@topspin.com>
Message-ID: <52ll9r0xud.fsf@topspin.com>

    Libor>   Do you know which architecture was the impetus for a 4th
    Libor> page table level?

x86-64 I believe.  x86-64 actually always uses 4 levels in HW but
Linux had to make one of them trivial, which limited a single memory
map to 512 GB.  Apparently some workloads that mmap'ed a lot of giant
files actually ran into this limit.

 - R.


From libor at topspin.com  Mon Feb 14 11:29:44 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 14 Feb 2005 11:29:44 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050212055827.GW574@esmail.cup.hp.com>;
	from iod00d@hp.com on Fri, Feb 11, 2005 at 09:58:27PM -0800
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
Message-ID: <20050214112944.B28266@topspin.com>

On Fri, Feb 11, 2005 at 09:58:27PM -0800, Grant Grundler wrote:
> On Fri, Feb 11, 2005 at 06:40:58PM -0800, Libor Michalek wrote:
> >   If no one objects, a patch to clean up compile warnings on x86_64.
> 
> I haven't applied this patch yet - I read mail on the other side of
> a firewall where my machines are. But I don't think it will fix
> the compile error below.
> 
> I'm not thrilled about the number of "(void *)(unsigned long)" casts.
> A small inline function to convert (int) to (void *) would be cleaner.
> E.g. (not compiled):
> static inline void * hash_arg(int x) { return (void *)(unsigned long) x; }

  Good idea, in three places I got rid of the cast, and then used an inline
in two other places. Here's the updated patch.

-Libor

Index: infiniband/ulp/sdp/sdp_actv.c
===================================================================
--- infiniband/ulp/sdp/sdp_actv.c	(revision 1778)
+++ infiniband/ulp/sdp/sdp_actv.c	(working copy)
@@ -201,7 +201,7 @@
 	 */
 	if (hello_ack->bsdh.size != sizeof(struct msg_hello_ack)) {
 
-		sdp_dbg_warn(NULL, "hello ack, size mismatch. (2) <%d:%d>",
+		sdp_dbg_warn(NULL, "hello ack, size mismatch. (2) <%d:%Zu>",
 			     hello_ack->bsdh.size,
 			     sizeof(struct msg_hello_ack));
 		return -EINVAL;
@@ -518,7 +518,7 @@
 	path->packet_life = 13;
 #endif
 	conn->cm_id = ib_create_cm_id(sdp_cm_event_handler,
-				      (void *)conn->hashent);
+				      __hashent_arg(conn->hashent));
 	if (NULL == conn->cm_id) {
 
 		sdp_dbg_warn(conn, "Failed to create CM handle, %d",
@@ -571,7 +571,7 @@
 	 */
 	if (sizeof(struct msg_hello) > conn->recv_size) {
 
-		sdp_dbg_warn(conn, "buffer size <%d> too small. <%d>",
+		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct msg_hello));
 		result = -ENOBUFS;
 		goto error;
Index: infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- infiniband/ulp/sdp/sdp_conn.c	(revision 1778)
+++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -1085,7 +1085,8 @@
 		conn->send_cq = ib_create_cq(conn->ca,
 					     sdp_cq_event_handler,
 					     NULL,
-					     (void *)conn->hashent,
+					     (void *)
+					     (unsigned long)conn->hashent,
 					     conn->send_cq_size);
 		if (IS_ERR(conn->send_cq)) {
 
@@ -1110,7 +1111,8 @@
 		conn->recv_cq = ib_create_cq(conn->ca,
 					     sdp_cq_event_handler,
 					     NULL,
-					     (void *)conn->hashent,
+					     (void *)
+					     (unsigned long)conn->hashent,
 					     conn->recv_cq_size);
 
 		if (IS_ERR(conn->recv_cq)) {
Index: infiniband/ulp/sdp/sdp_conn.h
===================================================================
--- infiniband/ulp/sdp/sdp_conn.h	(revision 1778)
+++ infiniband/ulp/sdp/sdp_conn.h	(working copy)
@@ -481,6 +481,11 @@
 #define SDP_CONN_PUT(conn)  __sdp_conn_put((conn))
 #define SDP_CONN_ERROR(conn)  __sdp_conn_error((conn))
 
+static inline void *__hashent_arg(s32 hashent) 
+{
+	return (void *)(unsigned long)hashent;
+}
+
 #endif /* _SDP_CONN_H */
 
 
Index: infiniband/ulp/sdp/sdp_iocb.c
===================================================================
--- infiniband/ulp/sdp/sdp_iocb.c	(revision 1778)
+++ infiniband/ulp/sdp/sdp_iocb.c	(working copy)
@@ -44,7 +44,7 @@
 	vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK));
 	if (NULL == vma) {
 		
-		sdp_warn("No VMA for IOCB <%lx:%d> unlock",
+		sdp_warn("No VMA for IOCB <%lx:%Zu> unlock",
 			 iocb->addr, iocb->size);
 	}
 
@@ -286,7 +286,7 @@
 	 */
 	if (result) {
 
-		sdp_dbg_err("VMA lock <%lx:%d> error <%d> <%d:%lu:%lu>",
+		sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>",
 			    iocb->addr, iocb->size, result, 
 			    iocb->page_count, iocb->mm->locked_vm, limit);
 		goto err_lock;
@@ -300,7 +300,7 @@
 		/*
 		 * sanity check.
 		 */
-		sdp_warn("No VMA for IOCB! <%lx:%d> lock",
+		sdp_warn("No VMA for IOCB! <%lx:%Zu> lock",
 			 iocb->addr, iocb->size);
 	}
 
@@ -342,7 +342,7 @@
 	result = _sdp_iocb_page_save(iocb);
 	if (result) {
 
-		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%d>", 
+		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", 
 			    result, iocb->addr, iocb->size);
 		goto err_save;
 	}
@@ -584,7 +584,7 @@
 	result = sdp_iocb_destroy(iocb);
 	if (0 > result) {
 
-		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%d>",
+		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
 	}
 
@@ -924,8 +924,8 @@
 
 		if (0 < (mask & iocb->flags) || SDP_IOCB_F_ALL == mask) {
 
-			sdp_dbg_err("IOCB <%d> cancel <%d> flag <%04x> "
-				    "size <%d:%d:%d>",
+			sdp_dbg_err("IOCB <%d> cancel <%Zu> flag <%04x> "
+				    "size <%Zu:%d:%d>",
 				    iocb->key, comp, iocb->flags, iocb->size,
 				    iocb->post, iocb->len);
 
Index: infiniband/ulp/sdp/sdp_pass.c
===================================================================
--- infiniband/ulp/sdp/sdp_pass.c	(revision 1778)
+++ infiniband/ulp/sdp/sdp_pass.c	(working copy)
@@ -41,7 +41,7 @@
 	 */
 	if (sizeof(struct msg_hello_ack) > conn->recv_size) {
 
-		sdp_dbg_warn(conn, "buffer size <%d> too small. <%d>",
+		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct msg_hello_ack));
 		result = -ENOBUFS;
 		goto error;
@@ -311,7 +311,7 @@
 	 */
 	if (msg_hello->bsdh.size != sizeof(struct msg_hello)) {
 
-		sdp_dbg_warn(NULL, "hello msg size mismatch. (2) <%d:%d>",
+		sdp_dbg_warn(NULL, "hello msg size mismatch. (2) <%d:%Zu>",
 			     msg_hello->bsdh.size, sizeof(struct msg_hello));
 		return -EINVAL;
 	}
@@ -438,7 +438,7 @@
 	 * update CM context to refer to the connection.
 	 */
 	conn->cm_id          = cm_id;
-	conn->cm_id->context = (void *)(unsigned long)conn->hashent;
+	conn->cm_id->context = __hashent_arg(conn->hashent);
 	/*
 	 * associate connection with a hca/port, and allocate IB.
 	 */
Index: infiniband/ulp/sdp/sdp_proc.c
===================================================================
--- infiniband/ulp/sdp/sdp_proc.c	(revision 1778)
+++ infiniband/ulp/sdp/sdp_proc.c	(working copy)
@@ -169,7 +169,7 @@
 	 */
 	if (0 && sizeof(s32) != sizeof(char *)) {
 
-		sdp_warn("integers and pointers of a different size. <%d:%d>",
+		sdp_warn("integers and pointers of a different size <%Zu:%Zu>",
 			 sizeof(s32), sizeof(char *));
 		return -EFAULT;
 	}
Index: infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- infiniband/ulp/sdp/sdp_recv.c	(revision 1781)
+++ infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -799,8 +799,9 @@
 	result = _sdp_read_buff_iocb(iocb, buff);
 	if (0 > result) {
 
-		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%d> to IOCB",
-			     result, iocb->len, (buff->tail - buff->data));
+		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
+			     result, iocb->len, 
+			     (unsigned)(buff->tail - buff->data));
 
 		(void)sdp_iocb_q_put_head(&conn->r_snk, iocb);
 		return result;
@@ -850,8 +851,9 @@
 	result = _sdp_read_buff_iocb(iocb, buff);
 	if (0 > result) {
 
-		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%d> to IOCB",
-			     result, iocb->len, (buff->tail - buff->data));
+		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
+			     result, iocb->len, 
+			     (unsigned)(buff->tail - buff->data));
 		return result;
 	}
 	/*
@@ -1003,12 +1005,12 @@
 static int _sdp_read_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
-	u32 value = (u32)(unsigned long)arg;
+	struct kiocb *req = (struct kiocb *)arg;
 
 	SDP_CHECK_NULL(element, -EINVAL);
 
 	if (SDP_DESC_TYPE_IOCB == element->type && 
-	    iocb->key == value) {
+	    iocb->key == req->ki_key) {
 
 		return 0;
 	}
@@ -1111,7 +1113,7 @@
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->r_src,
 						     _sdp_read_src_lookup,
-						     (void *)req->ki_key);
+						     req);
 	if (NULL != iocb) {
 
 		iocb->flags |= SDP_IOCB_F_CANCEL;
@@ -1623,7 +1625,7 @@
 			if (NULL == iocb) {
         
 				sdp_dbg_warn(conn, 
-					     "Failed to allocate IOCB <%d:%d>",
+					     "Error allocating IOCB <%Zu:%d>",
 					     size, copied);
 				result = -ENOMEM;
 				break;
@@ -1642,7 +1644,7 @@
 			if (0 > result) {
         
 				sdp_dbg_warn(conn, 
-					     "Error <%d> IOCB lock <%d:%d>", 
+					     "Error <%d> IOCB lock <%Zu:%d>", 
 					     result, size, copied);
 				
 				(void)sdp_iocb_destroy(iocb);
@@ -1655,7 +1657,7 @@
 			if (0 > result) {
 				
 				sdp_dbg_warn(conn, 
-					     "Error <%d> IOCB queue <%d:%d>",
+					     "Error <%d> IOCB queue <%Zu:%d>",
 					     result, size, copied);
 				
 				(void)sdp_iocb_destroy(iocb);
Index: infiniband/ulp/sdp/sdp_send.c
===================================================================
--- infiniband/ulp/sdp/sdp_send.c	(revision 1781)
+++ infiniband/ulp/sdp/sdp_send.c	(working copy)
@@ -1964,11 +1964,11 @@
 static int _sdp_write_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
-	u32 value = (u32)(unsigned long)arg;
+	struct kiocb *req = (struct kiocb *)arg;
 
 	SDP_CHECK_NULL(element, -EINVAL);
 
-	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == value) {
+	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key) {
 
 		return 0;
 	}
@@ -2016,13 +2016,13 @@
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->send_queue,
 						     _sdp_write_src_lookup,
-						     (void *)req->ki_key);
+						     req);
 	if (NULL != iocb) {
 		/*
 		 * always remove the IOCB.
 		 * If active, then place it into the correct active queue
 		 */
-		result = sdp_desc_q_remove((struct sdpc_desc *) iocb);
+		result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
 		SDP_EXPECT(!(0 > result));
 		
 		if (0 < (SDP_IOCB_F_ACTIVE & iocb->flags)) {
@@ -2075,7 +2075,8 @@
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->w_snk,
 						     _sdp_write_src_lookup,
-						     (void *)req->ki_key);
+						     req);
+
 	if (NULL != iocb) {
 
 		iocb->flags |= SDP_IOCB_F_CANCEL;
@@ -2439,7 +2440,7 @@
 		iocb = sdp_iocb_create();
 		if (NULL == iocb) {
 			
-			sdp_dbg_warn(conn, "Failed to allocate IOCB <%d:%d>",
+			sdp_dbg_warn(conn, "Failed to allocate IOCB <%Zu:%d>",
 				     size, copied);
 			result = -ENOMEM;
 			break;
@@ -2457,7 +2458,7 @@
 		result = sdp_iocb_lock(iocb);
 		if (0 > result) {
 				
-			sdp_dbg_warn(conn, "Error <%d> locking IOCB <%d:%d>", 
+			sdp_dbg_warn(conn, "Error <%d> locking IOCB <%Zu:%d>", 
 				     result, size, copied);
         
 			(void)sdp_iocb_destroy(iocb);


From libor at topspin.com  Mon Feb 14 11:44:21 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 14 Feb 2005 11:44:21 -0800
Subject: [openib-general] [PATCH][SDP] __user annotations
In-Reply-To: <1108175466.25868.3.camel@duffman>;
	from tduffy@sun.com on Fri, Feb 11, 2005 at 06:31:06PM -0800
References: <1108175466.25868.3.camel@duffman>
Message-ID: <20050214114421.C28266@topspin.com>

On Fri, Feb 11, 2005 at 06:31:06PM -0800, Tom Duffy wrote:
> Annotate __user pointers in sdp_inet.c.
> 

  Applied and checked in. Thanks.

-Libor

> Index: drivers/infiniband/ulp/sdp/sdp_inet.c
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1781)
> +++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
> @@ -1217,7 +1217,7 @@ static int _sdp_inet_ioctl(struct socket
>  		}
>  		else {
>  
> -			result = copy_to_user((void *)arg,
> +			result = copy_to_user((void __user *)arg,
>  					      &sk->sk_stamp,
>  					      sizeof(struct timeval));
>  			result = (result ? -EFAULT : result);
> @@ -1231,7 +1231,7 @@ static int _sdp_inet_ioctl(struct socket
>  	case SIOCDELRT:
>  	case SIOCRTMSG:
>  
> -		result = ip_rt_ioctl(cmd, (void *)arg);
> +		result = ip_rt_ioctl(cmd, (void __user *)arg);
>  		break;
>  		/*
>  		 * Standard ARP IOCTLs
> @@ -1260,7 +1260,7 @@ static int _sdp_inet_ioctl(struct socket
>  	case SIOCGIFPFLAGS:
>  	case SIOCSIFFLAGS:
>  
> -		result = devinet_ioctl(cmd, (void *)arg);
> +		result = devinet_ioctl(cmd, (void __user *)arg);
>  		break;
>  		/*
>  		 * stadard INET STREAM IOCTLs
> @@ -1274,7 +1274,7 @@ static int _sdp_inet_ioctl(struct socket
>  			 * TODO need to subtract/add URG (inline vs. OOB)
>  			 */
>  			value = conn->byte_strm;
> -			result = put_user(value, (int *) arg);
> +			result = put_user(value, (int __user *) arg);
>  		}
>  		else {
>  
> @@ -1290,7 +1290,7 @@ static int _sdp_inet_ioctl(struct socket
>  		if (SDP_SOCK_ST_LISTEN != conn->istate) {
>  
>  			value = conn->send_qud;
> -			result = put_user(value, (int *) arg);
> +			result = put_user(value, (int __user *) arg);
>  		}
>  		else {
>  
> @@ -1316,13 +1316,13 @@ static int _sdp_inet_ioctl(struct socket
>  			}
>  		}
>  
> -		result = put_user(value, (int *) arg);
> +		result = put_user(value, (int __user *) arg);
>  
>  		SDP_CONN_UNLOCK(conn);
>  		break;
>  	default:
>  
> -		result = dev_ioctl(cmd, (void *)arg);
> +		result = dev_ioctl(cmd, (void __user *)arg);
>  		break;
>  	}
>  
> @@ -1334,7 +1334,7 @@ static int _sdp_inet_ioctl(struct socket
>  static int _sdp_inet_setopt(struct socket *sock,
>  			       int level, 
>  			       int optname, 
> -			       char *optval, 
> +			       char __user *optval, 
>  			       int optlen)
>  {
>  	struct sock *sk;
> @@ -1362,7 +1362,7 @@ static int _sdp_inet_setopt(struct socke
>  		return -EINVAL;
>  	}
>  
> -	if (get_user(value, (int *)optval)) {
> +	if (get_user(value, (int __user *)optval)) {
>  
>  		return -EFAULT;
>  	}
> @@ -1422,8 +1422,8 @@ static int _sdp_inet_setopt(struct socke
>  static int _sdp_inet_getopt(struct socket *sock,
>  			       int level,
>  			       int optname,
> -			       char *optval,
> -			       int *optlen)
> +			       char __user *optval,
> +			       int __user *optlen)
>  {
>  	struct sock *sk;
>  	struct sdp_opt *conn;
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From libor at topspin.com  Mon Feb 14 11:47:34 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 14 Feb 2005 11:47:34 -0800
Subject: [openib-general] Re: [PATCH][SDP] Include asm/ headers *after*
	linux/ headers
In-Reply-To: <1108398224.6001.2.camel@duffman>;
	from tduffy@sun.com on Mon, Feb 14, 2005 at 08:23:44AM -0800
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<1108398224.6001.2.camel@duffman>
Message-ID: <20050214114734.D28266@topspin.com>

On Mon, Feb 14, 2005 at 08:23:44AM -0800, Tom Duffy wrote:
> On Fri, 2005-02-11 at 21:58 -0800, Grant Grundler wrote:
> > And sdp_main.h violates one of the kernel include file rules:
> > 	include asm/ headers *after* linux/ headers.
> 
> Include asm/ headers *after* linux/ headers.  This builds for me on x86,
> x86_64, and sparc64.

Thanks, applied and checked in.

-Libor

> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_main.h
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_main.h	(revision 1783)
> +++ drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
> @@ -57,11 +57,6 @@
>  /*
>   * kernel includes
>   */
> -#include <asm/atomic.h>
> -#include <asm/byteorder.h>
> -#include <asm/pgtable.h>
> -#include <asm/io.h>
> -
>  #include <linux/version.h>
>  #include <linux/slab.h>
>  #include <linux/spinlock.h>
> @@ -85,6 +80,12 @@
>  #include <net/route.h>
>  #include <net/dst.h>
>  #include <net/ip.h>
> +
> +#include <asm/atomic.h>
> +#include <asm/byteorder.h>
> +#include <asm/pgtable.h>
> +#include <asm/io.h>
> +
>  /*
>   * IB includes
>   */
> @@ -107,7 +108,3 @@
>  #include "sdp_buff_p.h"
>  
>  #endif /* _SDP_MAIN_H */
> -
> -
> -
> -


From iod00d at hp.com  Mon Feb 14 12:36:38 2005
From: iod00d at hp.com (Grant Grundler)
Date: Mon, 14 Feb 2005 12:36:38 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050214112944.B28266@topspin.com>
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<20050214112944.B28266@topspin.com>
Message-ID: <20050214203638.GG10881@esmail.cup.hp.com>

On Mon, Feb 14, 2005 at 11:29:44AM -0800, Libor Michalek wrote:
>   Good idea, in three places I got rid of the cast, and then used an inline
> in two other places. Here's the updated patch.

cool - thanks.  Can these two also be changed?

> Index: infiniband/ulp/sdp/sdp_conn.c
> ===================================================================
> --- infiniband/ulp/sdp/sdp_conn.c	(revision 1778)
> +++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
> @@ -1085,7 +1085,8 @@
>  		conn->send_cq = ib_create_cq(conn->ca,
>  					     sdp_cq_event_handler,
>  					     NULL,
> -					     (void *)conn->hashent,
> +					     (void *)
> +					     (unsigned long)conn->hashent,
>  					     conn->send_cq_size);
>  		if (IS_ERR(conn->send_cq)) {
>  
> @@ -1110,7 +1111,8 @@
>  		conn->recv_cq = ib_create_cq(conn->ca,
>  					     sdp_cq_event_handler,
>  					     NULL,
> -					     (void *)conn->hashent,
> +					     (void *)
> +					     (unsigned long)conn->hashent,
>  					     conn->recv_cq_size);
>  
>  		if (IS_ERR(conn->recv_cq)) {

thanks,
grant


From rminnich at lanl.gov  Mon Feb 14 12:59:20 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 14 Feb 2005 13:59:20 -0700 (MST)
Subject: [openib-general] question on opensm error
Message-ID: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>


formerly working opensm starts to get these:

[1108414727:000284173][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.
[1108414727:000384171][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.
[1108414727:000484169][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.



what's a reasonable thing to look for, or should I just svn update and 
hope for the best?

thanks

ron


From libor at topspin.com  Mon Feb 14 13:07:00 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 14 Feb 2005 13:07:00 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050214203638.GG10881@esmail.cup.hp.com>;
	from iod00d@hp.com on Mon, Feb 14, 2005 at 12:36:38PM -0800
References: <20050211184058.I20777@topspin.com>
	<20050212055827.GW574@esmail.cup.hp.com>
	<20050214112944.B28266@topspin.com>
	<20050214203638.GG10881@esmail.cup.hp.com>
Message-ID: <20050214130700.E28266@topspin.com>

On Mon, Feb 14, 2005 at 12:36:38PM -0800, Grant Grundler wrote:
> On Mon, Feb 14, 2005 at 11:29:44AM -0800, Libor Michalek wrote:
> >   Good idea, in three places I got rid of the cast, and then used an inline
> > in two other places. Here's the updated patch.
> 
> cool - thanks.  Can these two also be changed?

  Good catch, I added these two as well, and applied and commited
the changes.

-Libor

> > Index: infiniband/ulp/sdp/sdp_conn.c
> > ===================================================================
> > --- infiniband/ulp/sdp/sdp_conn.c	(revision 1778)
> > +++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
> > @@ -1085,7 +1085,8 @@
> >  		conn->send_cq = ib_create_cq(conn->ca,
> >  					     sdp_cq_event_handler,
> >  					     NULL,
> > -					     (void *)conn->hashent,
> > +					     (void *)
> > +					     (unsigned long)conn->hashent,
> >  					     conn->send_cq_size);
> >  		if (IS_ERR(conn->send_cq)) {
> >  
> > @@ -1110,7 +1111,8 @@
> >  		conn->recv_cq = ib_create_cq(conn->ca,
> >  					     sdp_cq_event_handler,
> >  					     NULL,
> > -					     (void *)conn->hashent,
> > +					     (void *)
> > +					     (unsigned long)conn->hashent,
> >  					     conn->recv_cq_size);
> >  
> >  		if (IS_ERR(conn->recv_cq)) {
> 
> thanks,
> grant


From tduffy at sun.com  Mon Feb 14 13:42:36 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 14 Feb 2005 13:42:36 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <1108180061.25868.7.camel@duffman>
References: <20050211184058.I20777@topspin.com>
	<1108180061.25868.7.camel@duffman>
Message-ID: <1108417356.6001.33.camel@duffman>

On Fri, 2005-02-11 at 19:47 -0800, Tom Duffy wrote:
> On Fri, 2005-02-11 at 18:40 -0800, Libor Michalek wrote:
> >   If no one objects, a patch to clean up compile warnings on x86_64. Most
> > of the warnings are a result of print format mismatches, the most common
> > being the need to use %Zu for size_t/sizeof.
> 
> Looks good.  This patch gets rid of the compile warnings on x86_64, but
> I am still seeing these on sparc64:

Turns out you will see these on x86_64 too if you turn on SDP data
debugging.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/8970db0b/attachment.sig>

From libor at topspin.com  Mon Feb 14 14:20:34 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 14 Feb 2005 14:20:34 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <1108417356.6001.33.camel@duffman>;
	from tduffy@sun.com on Mon, Feb 14, 2005 at 01:42:36PM -0800
References: <20050211184058.I20777@topspin.com>
	<1108180061.25868.7.camel@duffman>
	<1108417356.6001.33.camel@duffman>
Message-ID: <20050214142034.F28266@topspin.com>

On Mon, Feb 14, 2005 at 01:42:36PM -0800, Tom Duffy wrote:
> On Fri, 2005-02-11 at 19:47 -0800, Tom Duffy wrote:
> > On Fri, 2005-02-11 at 18:40 -0800, Libor Michalek wrote:
> > >   If no one objects, a patch to clean up compile warnings on x86_64. Most
> > > of the warnings are a result of print format mismatches, the most common
> > > being the need to use %Zu for size_t/sizeof.
> > 
> > Looks good.  This patch gets rid of the compile warnings on x86_64, but
> > I am still seeing these on sparc64:
> 
> Turns out you will see these on x86_64 too if you turn on SDP data
> debugging.

  Here's a patch for the print format warning in the debug data path.

-Libor

Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1778)
+++ sdp_rcvd.c	(working copy)
@@ -234,7 +234,7 @@
 
 	SDP_EXPECT((0 < (SDP_IOCB_F_RDMA_W & iocb->flags)));
 
-	sdp_dbg_data(conn, "Write <%d> size <%d:%d:%d> mode <%d> active <%d>",
+	sdp_dbg_data(conn, "Write <%d> size <%d:%d:%Zu> mode <%d> active <%d>",
 		     iocb->key, rwch->size, iocb->len, iocb->size,
 		     conn->recv_mode, conn->snk_sent);
 	/*
@@ -303,7 +303,7 @@
 
 	SDP_CONN_STAT_SRC_INC(conn);
 
-	sdp_dbg_data(conn, "Read <%d> size <%d:%d:%d> mode <%d> active <%d>",
+	sdp_dbg_data(conn, "Read <%d> size <%d:%d:%Zu> mode <%d> active <%d>",
 		     iocb->key, rrch->size, iocb->len, iocb->size,
 		     conn->recv_mode, conn->src_sent);
 	/*
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1794)
+++ sdp_send.c	(working copy)
@@ -394,7 +394,8 @@
 	advt->post += (buff->tail - buff->data);
 
 	sdp_dbg_data(conn, "POST Write BUFF wrid <%llu> bytes <%u:%d>.",
-		     buff->wrid, (buff->tail - buff->data), advt->size);
+		     buff->wrid, (unsigned)(buff->tail - buff->data),
+		     advt->size);
 	/*
 	 * dequeue if needed and the queue buffer
 	 */
@@ -2267,7 +2268,7 @@
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
-	sdp_dbg_data(conn, "send state <%04x> size <%d> flags <%08x>",
+	sdp_dbg_data(conn, "send state <%04x> size <%Zu> flags <%08x>",
 		     conn->state, size, msg->msg_flags);
 	sdp_dbg_data(conn, "write IOCB <%d> addr <%p> user <%d> flag <%08lx>",
 		     req->ki_key, msg->msg_iov->iov_base,
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1794)
+++ sdp_recv.c	(working copy)
@@ -72,7 +72,7 @@
 	}
 
 	sdp_dbg_data(conn, "POST RECV BUFF wrid <%llu> of <%u> bytes.",
-		     buff->wrid, (buff->tail - buff->data));
+		     buff->wrid, (unsigned)(buff->tail - buff->data));
 	/*
 	 * post recv
 	 */
@@ -201,7 +201,7 @@
 	}
 
 	sdp_dbg_data(conn, "POST READ BUFF wrid <%llu> of <%u> bytes.",
-		     buff->wrid, (buff->tail - buff->data));
+		     buff->wrid, (unsigned)(buff->tail - buff->data));
 	/*
 	 * update send queue depth
 	 */
@@ -900,7 +900,8 @@
 	SDP_CHECK_NULL(conn, -EINVAL);
 	SDP_CHECK_NULL(buff, -EINVAL);
 
-	sdp_dbg_data(conn, "RECV BUFF, bytes <%d>", (buff->tail - buff->data));
+	sdp_dbg_data(conn, "RECV BUFF, bytes <%u>",
+		     (unsigned)(buff->tail - buff->data));
 	/*
 	 * To emulate RFC 1122 (page 88) a connection should be reset/aborted
 	 * if data is received and the receive half of the connection has been
@@ -1319,7 +1320,7 @@
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
-	sdp_dbg_data(conn, "state <%08x> size <%d> pending <%d> falgs <%08x>",
+	sdp_dbg_data(conn, "state <%08x> size <%Zu> pending <%d> falgs <%08x>",
 		     conn->state, size, conn->byte_strm, flags);
 	sdp_dbg_data(conn, "read IOCB <%d> addr <%p> users <%d> flags <%08lx>",
 		     req->ki_key, msg->msg_iov->iov_base, 


From tduffy at sun.com  Mon Feb 14 14:39:21 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 14 Feb 2005 14:39:21 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050214142034.F28266@topspin.com>
References: <20050211184058.I20777@topspin.com>
	<1108180061.25868.7.camel@duffman> <1108417356.6001.33.camel@duffman>
	<20050214142034.F28266@topspin.com>
Message-ID: <1108420762.6001.37.camel@duffman>

On Mon, 2005-02-14 at 14:20 -0800, Libor Michalek wrote:
>   Here's a patch for the print format warning in the debug data path.

OK, good.  Thanks.  Now clean on x86_64 with data debug on.  Still
seeing these on sparc64:

/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c: In function `sdp_event_recv':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c:1312: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c: In function `sdp_event_send':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:449: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c:537: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c: In function `sdp_event_read':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c:174: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c: In function `sdp_event_write':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c:125: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c:125: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c: In function `sdp_cq_event_locked':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:46: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:46: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:105: warning: long long unsigned int format, u64 arg (arg 9)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c:105: warning: long long unsigned int format, u64 arg (arg 6)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c: In function `sdp_cm_req_handler':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c:381: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c:381: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c: In function `_sdp_cm_path_complete':
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:384: warning: long long unsigned int format, u64 arg (arg 5)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 7)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 8)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 4)
/build1/tduffy/openib-work/linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c:411: warning: long long unsigned int format, u64 arg (arg 5)


-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/e45ed9ef/attachment.sig>

From xma at us.ibm.com  Mon Feb 14 15:16:32 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Mon, 14 Feb 2005 16:16:32 -0700
Subject: [openib-general] [PATCH]deadlock problem in ipoib
Message-ID: <OF93103F49.DC9390E7-ON87256FA8.007E3E33-88256FA8.007FDB0C@us.ibm.com>

Roland, 

Please review this patch.

This patch has fixed a deadlock problem: the caller calls ipoib_put_ah() 
while holding priv->lock. (In ipoib_free_ah() the same lock is 
reacquired.) This also fixes the ipoib_flush_paths() calls __patch_free() 
without holding priv->lock.

My email has problem to inline patches. So I inline this patch for review 
and an attached file for applying this patch. Sorry for the inconvenient.

diff -urN infiniband/ulp/ipoib/ipoib_ib.c 
infiniband-panic/ulp/ipoib/ipoib_ib.c
--- infiniband/ulp/ipoib/ipoib_ib.c     2005-02-09 10:45:06.000000000 
-0800
+++ infiniband-panic/ulp/ipoib/ipoib_ib.c       2005-02-14 
14:52:43.000000000 -0800
@@ -85,11 +85,8 @@
                ipoib_dbg(priv, "Freeing ah %p\n", ah->ah);
                ib_destroy_ah(ah->ah);
                kfree(ah);
-       } else {
-               spin_lock_irqsave(&priv->lock, flags);
+       } else
                list_add_tail(&ah->list, &priv->dead_ahs);
-               spin_unlock_irqrestore(&priv->lock, flags);
-       }
 }
 
 static inline int ipoib_ib_receive(struct ipoib_dev_priv *priv,
diff -urN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-panic/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-04 16:05:13.000000000 
-0800
+++ infiniband-panic/ulp/ipoib/ipoib_main.c     2005-02-11 
15:00:33.000000000 -0800
@@ -256,7 +256,9 @@
                if (path->query)
                        ib_sa_cancel_query(path->query_id, path->query);
                wait_for_completion(&path->done);
+               spin_lock_irqsave(&priv->lock, flags);
                __path_free(dev, path);
+               spin_unlock_irqrestore(&priv->lock, flags);
        }
 }
 
diff -urN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-panic/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-04 
16:05:13.000000000 -0800
+++ infiniband-panic/ulp/ipoib/ipoib_multicast.c        2005-02-14 
14:54:52.000000000 -0800
@@ -107,11 +107,11 @@
                kfree(neigh);
        }
 
-       spin_unlock_irqrestore(&priv->lock, flags);
-
        if (mcast->ah)
                ipoib_put_ah(mcast->ah);
 
+       spin_unlock_irqrestore(&priv->lock, flags);
+
        while (!skb_queue_empty(&mcast->pkt_queue)) {
                struct sk_buff *skb = skb_dequeue(&mcast->pkt_queue);


 

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/405f1800/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-lock.patch
Type: application/octet-stream
Size: 1611 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/405f1800/attachment.obj>

From xma at us.ibm.com  Mon Feb 14 15:49:30 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Mon, 14 Feb 2005 15:49:30 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108374321.18616.94.camel@localhost>
Message-ID: <OF514291A5.BC303B45-ON87256FA8.0082864F-88256FA8.0082E007@us.ibm.com>

> It just depends on how long it takes to stabilize
the various components.

Agree. Basically, the more stable, the more chance for 
components to be picked up in the distro release.

thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050214/f7214d26/attachment.html>

From robert.j.woodruff at intel.com  Mon Feb 14 16:47:59 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Mon, 14 Feb 2005 16:47:59 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>

 
Hi Arkady,

As I mentioned in the BOF, I have a person (Arlin Davis) that can help
with
developing a uDAPL provider for the openib.org verbs. 
After discussing it more
with folks here, is seems to us that perhaps for the uDAPL user-mode
library,  it be provided to openib.org under a dual BSD + LGPL library
rather than a BSD + GPL since people normally want to use LGPL for 
libraries. 

Also, I think that we can go ahead and start porting using the BSD 
license available from sourceforge today. Once the port is complete,
we can submit it to openib.org under the dual license and submit 
any changes back to the main sourceforge project under the BSD license,
or you can simply accept the changes under the BSD license,
thus for uDAPL there is no need to wait for the expanded licensing terms
of
the sourceforge project.

Once the user-mode verbs, user-mode CM, SA support is available from
openib.org, 
we can get started. If this is OK with folks, I'll have Arlin start to
take a look
at this.

Sound OK ?

woody


From roland at topspin.com  Mon Feb 14 19:12:36 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 14 Feb 2005 19:12:36 -0800
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <OF93103F49.DC9390E7-ON87256FA8.007E3E33-88256FA8.007FDB0C@us.ibm.com>
	(Shirley Ma's message of "Mon, 14 Feb 2005 16:16:32 -0700")
References: <OF93103F49.DC9390E7-ON87256FA8.007E3E33-88256FA8.007FDB0C@us.ibm.com>
Message-ID: <52k6pazfbv.fsf@topspin.com>

    Shirley> This patch has fixed a deadlock problem: the caller calls
    Shirley> ipoib_put_ah() while holding priv->lock. (In
    Shirley> ipoib_free_ah() the same lock is reacquired.) This also
    Shirley> fixes the ipoib_flush_paths() calls __patch_free()
    Shirley> without holding priv->lock.

You definitely found some bugs but I think the real fix is to make
sure that ipoib_put_ah() is called -without- any locks held.  That's
because for future devices, ib_destroy_ah() may actually sleep
(although it won't for mthca).

The __path_free in unicast_arp_send() is actually safe to call inside
of the lock because we know that we just created the path struct, and
therefore it won't have an attached ah, so ipoib_put_ah() can never be
called.  (Actually it looks like there's a bug in path_rec_create()
also -- it doesn't set path->ah to NULL)

I think the proper fix for those places calling ipoib_put_ah() inside
a lock is to move the ipoib_ah structs to a temporary list and then
put them outside of the lock.

Thanks,
  Roland


From eitan at mellanox.co.il  Tue Feb 15 02:16:24 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 15 Feb 2005 12:16:24 +0200
Subject: [openib-general] Mailing
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>

It was really great to meet everybody at the workshop. I'm glad I could
come. 
Thanks to the organizers.

I would like to bring the topic of how we can improve the effectiveness of
our mailing interchange:
The openib-general is the mailing list we all share. It covers all the
topics openib stack has to do with.
As my main interest and contribution is in the area of IB subnet management
and tools, I had to 
write email filters that allows me to focus on this area. However, these
filters are far from perfect
and many times I loose valuable mailing interchange.

I would like to propose two options to resolve that:
1. Use more mailing lists (e.g. openib-ib-manegement )
2. Use keywords in the mail Subject (e.g. Subject: [SM] I have a problem)

I would appreciate your comments and action.

Thanks

Eitan Zahavi
 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/acb7c1f6/attachment.html>

From halr at voltaire.com  Tue Feb 15 03:46:58 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 06:46:58 -0500
Subject: [openib-general] question on opensm error
In-Reply-To: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
Message-ID: <1108467920.21161.2123.camel@localhost.localdomain>

Hi Ron,

On Mon, 2005-02-14 at 15:59, Ronald G. Minnich wrote:
> formerly working opensm starts to get these:

So the OpenSM was up and running and these messages appeared in the log.
Did anything change in the subnet ?

> [1108414727:000284173][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.
> [1108414727:000384171][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.
> [1108414727:000484169][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.

These are failures of the OpenSM to send a SM Get(NodeInfo) which are
used during the periodic subnet sweeps. I think the only way this error
happens is if physical link is not present on the local link (e.g.
logical link is not in init state or beyond). 

So was a cable pulled somewhere ? 

Is this problem intermittent ? Does it come and go for no apparent
reason ? 

Are there any other messages in the log around this which might be
useful ? 

-- Hal

> 
> 
> 
> what's a reasonable thing to look for, or should I just svn update and 
> hope for the best?
> 
> thanks
> 
> ron
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Tue Feb 15 03:50:11 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 06:50:11 -0500
Subject: [openib-general] question on opensm error
In-Reply-To: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
Message-ID: <1108468017.21161.2130.camel@localhost.localdomain>

Hi Ron,

On Mon, 2005-02-14 at 15:59, Ronald G. Minnich wrote:
> formerly working opensm starts to get these:

So the OpenSM was up and running and these messages appeared in the log.
Did anything change in the subnet ?

> [1108414727:000284173][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.
> [1108414727:000384171][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.
> [1108414727:000484169][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.

These are failures of the OpenSM to send a SM Get(NodeInfo) which are
used during the periodic subnet sweeps. I think the only way this error
happens is if physical link is not present on the local link (e.g.
logical link is not in init state or beyond). 

So was a cable pulled somewhere ? 

Is this problem intermittent ? Does it come and go for no apparent
reason ? Does the subnet get out of this state or do you need to 
restart OpenSM ?

Are there any other messages in the log around this which might be
useful ? 

Thanks.

-- Hal

> 
> 
> 
> what's a reasonable thing to look for, or should I just svn update and 
> hope for the best?
> 
> thanks
> 
> ron
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Tue Feb 15 05:09:55 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 08:09:55 -0500
Subject: [openib-general] question on opensm error
In-Reply-To: <1108468017.21161.2130.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
	<1108468017.21161.2130.camel@localhost.localdomain>
Message-ID: <1108472802.21161.2279.camel@localhost.localdomain>

Hi,

A couple more things on this:

On Tue, 2005-02-15 at 06:50, Hal Rosenstock wrote:
> Hi Ron,
> 
> On Mon, 2005-02-14 at 15:59, Ronald G. Minnich wrote:
> > formerly working opensm starts to get these:
> 
> So the OpenSM was up and running and these messages appeared in the log.
> Did anything change in the subnet ?
> 
> > [1108414727:000284173][411FF970] -> umad_receiver: send completed with 
> > error(method=1 attr=11) -- dropping.
> > [1108414727:000384171][411FF970] -> umad_receiver: send completed with 
> > error(method=1 attr=11) -- dropping.
> > [1108414727:000484169][411FF970] -> umad_receiver: send completed with 
> > error(method=1 attr=11) -- dropping.
> 
> These are failures of the OpenSM to send a SM Get(NodeInfo) which are
> used during the periodic subnet sweeps. 

The SM does two kinds of sweeps - a periodic light sweep, where it
queries all switches for switchinfo to search for port changed bit, and
heavy sweep that is triggered and is not periodic. The nodeinfo is part
of the heavy sweep.

> I think the only way this error
> happens is if physical link is not present on the local link (e.g.
> logical link is not in init state or beyond). 

ibstatus/ibstat can show the local port logical and physical port state.

> So was a cable pulled somewhere ? 

Node info should be answered even if there is no logical link. Such
errors may be a result of
        a. bad cable
        b. bad HCA or firmware
        c. hung or crashed kernel.

> Is this problem intermittent ? Does it come and go for no apparent
> reason ? Does the subnet get out of this state or do you need to 
> restart OpenSM ?
> 
> Are there any other messages in the log around this which might be
> useful ? 

It might be helpful to try running ibnetdiscover -e (to show the
errors). smpquery can also be used to query the bad link/host.

> 
> Thanks.
> 
> -- Hal
> 
> > 
> > 
> > 
> > what's a reasonable thing to look for, or should I just svn update and 
> > hope for the best?
> > 
> > thanks
> > 
> > ron
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> > 
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From rminnich at lanl.gov  Tue Feb 15 05:53:12 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Tue, 15 Feb 2005 06:53:12 -0700 (MST)
Subject: [openib-general] question on opensm error
In-Reply-To: <1108472802.21161.2279.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov> 
	<1108468017.21161.2130.camel@localhost.localdomain>
	<1108472802.21161.2279.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502150647210.12960@enigma.lanl.gov>



On Tue, 15 Feb 2005, Hal Rosenstock wrote:

> ibstatus/ibstat can show the local port logical and physical port state.

bluesteel:~ # ibstat
CA 'mthca0':
        CA type: MT23108
        Number of ports: 2
        Firmware version: 3.3.2
        Hardware version: a1
        Node GUID: 0x0002c90108a03e60
        System image GUID: 0x0002c9000100d050
        Port 1:
                State: Initializing
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0002c90108a03e61
        Port 2:
                State: Down
                Rate: 2
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0002c90108a03e62


> It might be helpful to try running ibnetdiscover -e (to show the
> errors). smpquery can also be used to query the bad link/host.

no -e switch on my copy. svn update time? 

This was kind of interesting, it did find a lot of switches ...
[0][1][3][8][7][3][3][2][8][5][8] -> known remote switch 
{0002c90108d19748} portnum 0 lid 0xe4-0xe4 "MT43132 Mellanox Technologies"
[0][1][3][8][7][3][3][2][8][2] -> processing switch {0002c90108d19200} 
portnum 0 lid 0x0-0x0 "MT43132 Mellanox Technologies"

(more like this -- much more)

and some hcas
[0][1][3][8][7][3][3][2][8][2][2] -> new remote hca {0002c901081e6700} 
portnum 1 lid 0x0-0x0 "MT23108 InfiniHost Mellanox Technologies"
        [1] {0002c901081e6700}

but osm.log is about 59MB of these:
[1108475425:000915547][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.

smpquery? Have not seen that. Remember I'm trying to get this done with 
openib ONLY. Probably a bad idea :-)



here's plain ibnetdiscover

bluesteel:~ # ibnetdiscover 
warn: [4710] _do_madrpc: retry 2 (timeout 2000 ms)
warn: [4710] _do_madrpc: send failed; Invalid argument
warn: [4710] handle_port: Nodeinfo on [0][1][3][8][7][5][3][2][8][2][4] 
port 4 failed, skipping port
warn: [4710] _do_madrpc: retry 2 (timeout 2000 ms)
warn: [4710] _do_madrpc: send failed; Invalid argument
warn: [4710] handle_port: Nodeinfo on [0][1][3][8][7][2][3][4][1][1][2] 
port 2 failed, skipping port
warn: [4710] _do_madrpc: retry 2 (timeout 2000 ms)
warn: [4710] _do_madrpc: send failed; Invalid argument
warn: [4710] handle_port: Nodeinfo on [0][1][3][8][7][2][3][1][8][4][2] 
port 2 failed, skipping port

ron


From halr at voltaire.com  Tue Feb 15 06:31:46 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 09:31:46 -0500
Subject: [openib-general] question on opensm error
In-Reply-To: <Pine.LNX.4.58.0502150647210.12960@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
	<1108468017.21161.2130.camel@localhost.localdomain>
	<1108472802.21161.2279.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502150647210.12960@enigma.lanl.gov>
Message-ID: <1108477905.21161.2454.camel@localhost.localdomain>

On Tue, 2005-02-15 at 08:53, Ronald G. Minnich wrote:
> On Tue, 15 Feb 2005, Hal Rosenstock wrote:
> 
> > ibstatus/ibstat can show the local port logical and physical port state.
> 
> bluesteel:~ # ibstat
> CA 'mthca0':
>         CA type: MT23108
>         Number of ports: 2
>         Firmware version: 3.3.2
>         Hardware version: a1
>         Node GUID: 0x0002c90108a03e60
>         System image GUID: 0x0002c9000100d050
>         Port 1:
>                 State: Initializing

Hmmm. Guess it's not a local problem. With a port state of initialize,
the SM MAD (get nodeinfo) should get out of the HCA port on VL15.

>                 Rate: 10
>                 Base lid: 0
>                 LMC: 0
>                 SM lid: 0
>                 Capability mask: 0x00500a68
>                 Port GUID: 0x0002c90108a03e61
>         Port 2:
>                 State: Down
>                 Rate: 2
>                 Base lid: 0
>                 LMC: 0
>                 SM lid: 0
>                 Capability mask: 0x00500a68
>                 Port GUID: 0x0002c90108a03e62
> 
> 
> > It might be helpful to try running ibnetdiscover -e (to show the
> > errors). smpquery can also be used to query the bad link/host.
> 
> no -e switch on my copy. svn update time? 

Not sure but it is certainly in the latest. I just tried it.

> This was kind of interesting, it did find a lot of switches ...
> [0][1][3][8][7][3][3][2][8][5][8] -> known remote switch 
> {0002c90108d19748} portnum 0 lid 0xe4-0xe4 "MT43132 Mellanox Technologies"
> [0][1][3][8][7][3][3][2][8][2] -> processing switch {0002c90108d19200} 
> portnum 0 lid 0x0-0x0 "MT43132 Mellanox Technologies"
> 
> (more like this -- much more)

Just out of curiousity, what is the deepest number of hops ?

> and some hcas
> [0][1][3][8][7][3][3][2][8][2][2] -> new remote hca {0002c901081e6700} 
> portnum 1 lid 0x0-0x0 "MT23108 InfiniHost Mellanox Technologies"
>         [1] {0002c901081e6700}
> 
> but osm.log is about 59MB of these:
> [1108475425:000915547][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.
> 
> smpquery? Have not seen that. Remember I'm trying to get this done with 
> openib ONLY. Probably a bad idea :-)

This was added to diags/net last Thursday so you likely haven't sync'd
up since then.

> here's plain ibnetdiscover
> 
> bluesteel:~ # ibnetdiscover 
> warn: [4710] _do_madrpc: retry 2 (timeout 2000 ms)
> warn: [4710] _do_madrpc: send failed; Invalid argument
> warn: [4710] handle_port: Nodeinfo on [0][1][3][8][7][5][3][2][8][2][4] 
> port 4 failed, skipping port
> warn: [4710] _do_madrpc: retry 2 (timeout 2000 ms)
> warn: [4710] _do_madrpc: send failed; Invalid argument
> warn: [4710] handle_port: Nodeinfo on [0][1][3][8][7][2][3][4][1][1][2] 
> port 2 failed, skipping port
> warn: [4710] _do_madrpc: retry 2 (timeout 2000 ms)
> warn: [4710] _do_madrpc: send failed; Invalid argument
> warn: [4710] handle_port: Nodeinfo on [0][1][3][8][7][2][3][1][8][4][2] 
> port 2 failed, skipping port

It is some sort of remote problem. Not sure what is causing the failure
but it appears to be on multiple nodes concurrently.

-- Hal

> 
> ron



From roland at topspin.com  Tue Feb 15 09:02:02 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 15 Feb 2005 09:02:02 -0800
Subject: [openib-general] Mailing
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
	(Eitan Zahavi's message of "Tue, 15 Feb 2005 12:16:24 +0200")
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
Message-ID: <523bvxzrhx.fsf@topspin.com>

    Eitan> 1. Use more mailing lists (e.g. openib-ib-manegement )

I've always resisted splitting openib-general.  I still think it's a
bad idea for several reasons:

 - openib-general is still a low-traffic list.  linux-kernel works
   fine with 10 times the number of messages per day.
 - we end up wasting time telling people they need to post to a
   different list, and users end up cross-posting to all the lists
   (and developers end up subscribing to all the lists)
 - splitting the list dilutes the community.

 - R.


From rminnich at lanl.gov  Tue Feb 15 09:06:04 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Tue, 15 Feb 2005 10:06:04 -0700 (MST)
Subject: [openib-general] Mailing
In-Reply-To: <523bvxzrhx.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
	<523bvxzrhx.fsf@topspin.com>
Message-ID: <Pine.LNX.4.58.0502151005440.13209@enigma.lanl.gov>



On Tue, 15 Feb 2005, Roland Dreier wrote:

>     Eitan> 1. Use more mailing lists (e.g. openib-ib-manegement )
> 
> I've always resisted splitting openib-general.  I still think it's a
> bad idea for several reasons:

yes, splitting makes no sense for the volume of this list.

I would hate to see a split.

ron


From iod00d at hp.com  Tue Feb 15 09:47:11 2005
From: iod00d at hp.com (Grant Grundler)
Date: Tue, 15 Feb 2005 09:47:11 -0800
Subject: [openib-general] Mailing
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
Message-ID: <20050215174711.GC15050@esmail.cup.hp.com>

On Tue, Feb 15, 2005 at 12:16:24PM +0200, Eitan Zahavi wrote:
> As my main interest and contribution is in the area of IB subnet management
> and tools, I had to write email filters that allows me to focus on this area.
> However, these filters are far from perfect and many times I loose
> valuable mailing interchange.
> 
> I would like to propose two options to resolve that:
> 1. Use more mailing lists (e.g. openib-ib-manegement )
> 2. Use keywords in the mail Subject (e.g. Subject: [SM] I have a problem)

Eitan, if you can train people to do #2, that sounds good.
"Normal" keywords are "BUG" or "PATCH". It would help to
list other keywords on http://www.openib.org/contact.html
where the mailing list is described.

However this won't be perfect either. Best bet is to feed
a coworker who is on the list anyway enough chocolate so they
voluntarily forward mail you care about. 
(/me hears pavlov's bell and runs off salivating :^)

grant


From xma at us.ibm.com  Tue Feb 15 09:59:57 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Tue, 15 Feb 2005 09:59:57 -0800
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <52k6pazfbv.fsf@topspin.com>
Message-ID: <OF7C6429CA.E2714D5D-ON87256FA9.0062CBE0-88256FA9.0062DF5F@us.ibm.com>

>I think the proper fix for those places calling ipoib_put_ah() inside
a lock is to move the ipoib_ah structs to a temporary list and then
put them outside of the lock.

I will create a new patch.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/a1336608/attachment.html>

From xma at us.ibm.com  Tue Feb 15 10:08:50 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Tue, 15 Feb 2005 10:08:50 -0800
Subject: [openib-general] bugzilla question
Message-ID: <OFAC1BE5AD.3A6E08D5-ON87256FA9.00635BBB-88256FA9.0063AF85@us.ibm.com>

I am planning to use bugzilla to file bug reports. I logged in but I only 
see one bug report there. Are we really using this?

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/7600eb40/attachment.html>

From mshefty at ichips.intel.com  Tue Feb 15 11:27:37 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 15 Feb 2005 11:27:37 -0800
Subject: [openib-general] [PATCH] [CM] match received msgs against local and
	remote ID
Message-ID: <20050215112737.12ae188d.mshefty@ichips.intel.com>

This patch matches received messages against local and remote IDs (versus
local ID only).  It relocates a structure definition, and fixes a race
receiving a duplicate REQ.

These changes will be needed as part of timewait handling.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1796)
+++ infiniband/core/cm.c	(working copy)
@@ -95,6 +95,17 @@ struct cm_av {
 	u16 pkey_index;
 };
 
+struct cm_work {
+	struct work_struct work;
+	struct list_head list;
+	struct cm_port *port;
+	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
+	u32 local_id;				/* Established */
+	u32 remote_id;
+	struct ib_cm_event cm_event;
+	struct ib_sa_path_rec path[0];
+};
+
 struct cm_id_private {
 	struct ib_cm_id	id;
 
@@ -130,16 +141,6 @@ struct cm_id_private {
 	atomic_t work_count;
 };
 
-struct cm_work {
-	struct work_struct work;
-	struct list_head list;
-	struct cm_port *port;
-	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
-	u32 local_id;				/* Established */
-	struct ib_cm_event cm_event;
-	struct ib_sa_path_rec path[0];
-};
-
 static inline void cm_deref_id(struct cm_id_private *cm_id_priv)
 {
 	if (atomic_dec_and_test(&cm_id_priv->refcount))
@@ -298,15 +299,19 @@ static void cm_free_id(u32 local_id)
 	spin_unlock_irqrestore(&cm.lock, flags);
 }
 
-static struct cm_id_private * cm_acquire_id_by_local_id(u32 local_id)
+static struct cm_id_private * cm_acquire_id(u32 local_id, u32 remote_id)
 {
 	struct cm_id_private *cm_id_priv;
 	unsigned long flags;
 
 	spin_lock_irqsave(&cm.lock, flags);
 	cm_id_priv = idr_find(&cm.local_id_table, (int) local_id);
-	if (cm_id_priv)
-		atomic_inc(&cm_id_priv->refcount);
+	if (cm_id_priv) {
+		if (cm_id_priv->id.remote_id == remote_id)
+			atomic_inc(&cm_id_priv->refcount);
+		else
+			cm_id_priv = NULL;
+	}
 	spin_unlock_irqrestore(&cm.lock, flags);
 
 	return cm_id_priv;
@@ -384,6 +389,8 @@ static struct cm_id_private * cm_insert_
 	return NULL;
 }
 
+/*
+ * Call will be needed when implementing REJ handling.
 static struct cm_id_private * cm_find_id_by_remote_id(u64 remote_ca_guid,
 						      u32 remote_id)
 {
@@ -405,6 +412,7 @@ static struct cm_id_private * cm_find_id
 	}
 	return NULL;
 }
+*/
 
 static void cm_remove_remote_id(struct cm_id_private *cm_id_priv)
 {
@@ -1009,21 +1017,13 @@ out:
 static int cm_req_handler(struct cm_work *work)
 {
 	struct ib_cm_id *cm_id;
-	struct cm_id_private *cm_id_priv, *listen_cm_id_priv;
+	struct cm_id_private *cm_id_priv, *listen_cm_id_priv, *cur_cm_id_priv;
 	struct cm_req_msg *req_msg;
 	unsigned long flags;
 	int ret;
 
 	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
 
-	/* Check for duplicate REQ. */
-	spin_lock_irqsave(&cm.lock, flags);
-	cm_id_priv = cm_find_id_by_remote_id(req_msg->local_ca_guid,
-					     req_msg->local_comm_id);
-	spin_unlock_irqrestore(&cm.lock, flags);
-	if (cm_id_priv)
-		return -EINVAL;
-
 	/* Check for peer-to-peer connection. */
 	ret = cm_peer_req_handler(work);
 	if (ret != -ENOENT)
@@ -1038,20 +1038,26 @@ static int cm_req_handler(struct cm_work
 	cm_id_priv->id.remote_id = req_msg->local_comm_id;
 	cm_id_priv->passive = 1;
 
-	/* Find matching listen request. */
 	spin_lock_irqsave(&cm.lock, flags);
+	/* Check for duplicate REQ. */
+	cur_cm_id_priv = cm_insert_remote_id(cm_id_priv);
+	if (cur_cm_id_priv) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		ret = -EINVAL;
+		goto error1;
+	}
+	/* Find matching listen request. */
 	listen_cm_id_priv = cm_find_listen(req_msg->service_id);
 	if (!listen_cm_id_priv) {
 		spin_unlock_irqrestore(&cm.lock, flags);
 		/* todo: reject with no match */
 		ret = -EINVAL;
-		goto out;
+		goto error2;
 	}
 	atomic_inc(&listen_cm_id_priv->refcount);
 	atomic_inc(&cm_id_priv->refcount);
 	cm_id_priv->id.state = IB_CM_REQ_RCVD;
 	atomic_inc(&cm_id_priv->work_count);
-	cm_insert_remote_id(cm_id_priv);
 	spin_unlock_irqrestore(&cm.lock, flags);
 
 	cm_id_priv->id.cm_handler = listen_cm_id_priv->id.cm_handler;
@@ -1063,12 +1069,12 @@ static int cm_req_handler(struct cm_work
 	ret = cm_init_av(work->port->mad_agent->device, &work->path[0],
 			 &cm_id_priv->av);
 	if (ret)
-		goto bad_req;
+		goto error3;
 	if (req_msg->alt_local_lid) {
 		ret = cm_init_av(work->port->mad_agent->device, &work->path[1],
 				 &cm_id_priv->alt_av);
 		if (ret)
-			goto bad_req;
+			goto error3;
 	}
 	cm_id_priv->timeout_ms = cm_convert_to_ms(
 					cm_req_get_local_resp_timeout(req_msg));
@@ -1087,9 +1093,11 @@ static int cm_req_handler(struct cm_work
 	cm_process_work(cm_id_priv, work);
 	cm_deref_id(listen_cm_id_priv);
 	return 0;
-bad_req:
+error3:
 	cm_deref_id(listen_cm_id_priv);
-out:
+error2:
+	cm_remove_remote_id(cm_id_priv);
+error1:
 	ib_destroy_cm_id(&cm_id_priv->id);
 	return ret;
 }
@@ -1287,7 +1295,7 @@ static int cm_rep_handler(struct cm_work
 	int ret;
 
 	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(rep_msg->remote_comm_id);
+	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id, 0);
 	if (!cm_id_priv)
 		return -EINVAL;
 
@@ -1349,7 +1357,7 @@ static int cm_establish_handler(struct c
 	int ret;
 
 	/* See comment in ib_cm_establish about lookup. */
-	cm_id_priv = cm_acquire_id_by_local_id(work->local_id);
+	cm_id_priv = cm_acquire_id(work->local_id, work->remote_id);
 	if (!cm_id_priv)
 		return -EINVAL;
 
@@ -1385,7 +1393,8 @@ static int cm_rtu_handler(struct cm_work
 	int ret;
 
 	rtu_msg = (struct cm_rtu_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(rtu_msg->remote_comm_id);
+	cm_id_priv = cm_acquire_id(rtu_msg->remote_comm_id,
+				   rtu_msg->local_comm_id);
 	if (!cm_id_priv)
 		return -EINVAL;
 
@@ -1560,7 +1569,8 @@ static int cm_dreq_handler(struct cm_wor
 	int ret;
 
 	dreq_msg = (struct cm_dreq_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(dreq_msg->remote_comm_id);
+	cm_id_priv = cm_acquire_id(dreq_msg->remote_comm_id,
+				   dreq_msg->local_comm_id);
 	if (!cm_id_priv)
 		return -EINVAL;
 
@@ -1610,7 +1620,8 @@ static int cm_drep_handler(struct cm_wor
 	int ret;
 
 	drep_msg = (struct cm_drep_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(drep_msg->remote_comm_id);
+	cm_id_priv = cm_acquire_id(drep_msg->remote_comm_id,
+				   drep_msg->local_comm_id);
 	if (!cm_id_priv)
 		return -EINVAL;
 
@@ -1961,7 +1972,8 @@ static int cm_lap_handler(struct cm_work
 
 	/* todo: verify LAP request and send reject APR if invalid. */
 	lap_msg = (struct cm_lap_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(lap_msg->remote_comm_id);
+	cm_id_priv = cm_acquire_id(lap_msg->remote_comm_id,
+				   lap_msg->local_comm_id);
 	if (!cm_id_priv)
 		return -EINVAL;
 	
@@ -2071,7 +2083,8 @@ static int cm_apr_handler(struct cm_work
 	int ret;
 
 	apr_msg = (struct cm_apr_msg *)work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(apr_msg->remote_comm_id);
+	cm_id_priv = cm_acquire_id(apr_msg->remote_comm_id,
+				   apr_msg->local_comm_id);
 	if (!cm_id_priv)
 		return -EINVAL; /* Unmatched reply. */
 
@@ -2343,7 +2356,7 @@ static int cm_sidr_rep_handler(struct cm
 
 	sidr_rep_msg = (struct cm_sidr_rep_msg *)
 				work->mad_recv_wc->recv_buf.mad;
-	cm_id_priv = cm_acquire_id_by_local_id(sidr_rep_msg->request_id);
+	cm_id_priv = cm_acquire_id(sidr_rep_msg->request_id, 0);
 	if (!cm_id_priv)
 		return -EINVAL; /* Unmatched reply. */
 
@@ -2553,6 +2566,7 @@ int ib_cm_establish(struct ib_cm_id *cm_
 	 */
 	INIT_WORK(&work->work, cm_work_handler, work);
 	work->local_id = cm_id->local_id;
+	work->remote_id = cm_id->remote_id;
 	work->cm_event.event = IB_CM_USER_ESTABLISHED;
 	queue_work(cm.wq, &work->work);
 out:


From mshefty at ichips.intel.com  Tue Feb 15 10:39:37 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 15 Feb 2005 10:39:37 -0800
Subject: [openib-general] Mailing
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
Message-ID: <421241E9.5050504@ichips.intel.com>

Eitan Zahavi wrote:

> I would like to propose two options to resolve that:
> 1. Use more mailing lists (e.g. openib-ib-manegement )
> 2. Use keywords in the mail Subject (e.g. Subject: [SM] I have a problem)

I would rather not have multiple mailing lists.  I like that use of 
keywords, but I've found that threads frequently change topics.

- Sean


From roland at topspin.com  Tue Feb 15 10:43:49 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 15 Feb 2005 10:43:49 -0800
Subject: [openib-general] Mailing
In-Reply-To: <421241E9.5050504@ichips.intel.com> (Sean Hefty's message of
	"Tue, 15 Feb 2005 10:39:37 -0800")
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
	<421241E9.5050504@ichips.intel.com>
Message-ID: <52u0ody87u.fsf@topspin.com>

    Sean> I would rather not have multiple mailing lists.  I like that
    Sean> use of keywords, but I've found that threads frequently
    Sean> change topics.

Is this really a big problem anyway?  If the list gets 40 mails a day
(which is a busy day) and it takes 15 seconds (which is generous) to
glance at a mail and decide if you're interested, you're only spending
10 minutes a day looking at mail.

 - R.



From cap at nsc.liu.se  Tue Feb 15 10:48:49 2005
From: cap at nsc.liu.se (Peter =?iso-8859-15?q?Kjellstr=F6m?=)
Date: Tue, 15 Feb 2005 19:48:49 +0100
Subject: [openib-general] curious warning message (gen1, ibgd, possibly ot)
Message-ID: <200502151948.49813.cap@nsc.liu.se>

Hello folks,

I'm sorry if this is kind of off-topic for this list but I suspect that there 
are quite a few people on this list that can answer my rather simple 
question.

Every time the tavor-driver loads I see these (kinda scary) warnings:

THH kernel module initialized successfully VAPI(2): thhul_qpm.c[2140]: 
***WARNING***: AVs for special QPs should use HOST memory; the provided PD 
has its AVs in DDR memory.
(repeated four times)

Does anybody know what this means (it's not exactly english to me ;). Can it 
be safely ignored? (the systems seems to humm along nicely..)

I'm running IBGD-1.6.1 from mellanox on my Xeon Nocona cluster (pci-express). 
My HCAs are of the so called "lion cub" type and the os is centos-3.4 
(rhel3u4). Firmware is afaict 4.5.3 (fw_ver=0x400050003 as reported by 
vstat).

tia,
 Peter

--
------------------------------------------------------------
  Peter Kjellström               |
  National Supercomputer Centre  |
  Sweden                         | http://www.nsc.liu.se


From halr at voltaire.com  Tue Feb 15 11:12:18 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 14:12:18 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
Message-ID: <1108494564.25271.29.camel@localhost.localdomain>

Hi,

Unfortunately, the Solaris 10 IPoIB MTU with OpenSM is back to the
maximum size of 252 again :-( I'm not sure whether this was ever really
fixed although I do now see the packets indicating an exact MTU of 4
(2048 bytes). I'm not sure what Solaris doesn't like about the OpenSM
response to the MCMemberRecord.

-- Hal



From robert.j.woodruff at intel.com  Tue Feb 15 12:29:18 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Tue, 15 Feb 2005 12:29:18 -0800
Subject: [openib-general] Mailing
Message-ID: <1AC79F16F5C5284499BB9591B33D6F00039160EF@orsmsx408>

 Roland Wrote,

>Is this really a big problem anyway?  If the list gets 40 mails a day
>(which is a busy day) and it takes 15 seconds (which is generous) to
>glance at a mail and decide if you're interested, you're only spending
>10 minutes a day looking at mail.

> - R.

I agree with Roland, I think one list is fine.

woody


From mshefty at ichips.intel.com  Tue Feb 15 12:42:14 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 15 Feb 2005 12:42:14 -0800
Subject: [openib-general] Mailing
In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00039160EF@orsmsx408>
References: <1AC79F16F5C5284499BB9591B33D6F00039160EF@orsmsx408>
Message-ID: <42125EA6.3090409@ichips.intel.com>

Woodruff, Robert J wrote:

>  Roland Wrote,
> 
> 
>>Is this really a big problem anyway?  If the list gets 40 mails a day
>>(which is a busy day) and it takes 15 seconds (which is generous) to
>>glance at a mail and decide if you're interested, you're only spending
>>10 minutes a day looking at mail.
> 
> 
> I agree with Roland, I think one list is fine.

I also agree that one list is fine.  But the cost of 40 interruptions a 
day is substantially more than 10 minutes of downtime.  As long as 
people are clear with their subject headings, it makes it easier to 
quickly identify which e-mail responses can be deferred.


From bill at thompson.name  Tue Feb 15 13:22:45 2005
From: bill at thompson.name (Bill Thompson)
Date: Tue, 15 Feb 2005 13:22:45 -0800
Subject: [openib-general] Mailing
In-Reply-To: <42125EA6.3090409@ichips.intel.com>
Message-ID: <DOEHKBHAGBADBEKBCHKCKECDDEAA.bill@thompson.name>

> -----Original Message-----
> >  Roland Wrote,
> > 
> > 
> >>Is this really a big problem anyway?  If the list gets 40 mails a day
> >>(which is a busy day) and it takes 15 seconds (which is generous) to
> >>glance at a mail and decide if you're interested, you're only spending
> >>10 minutes a day looking at mail.
> > 
> > 
> > I agree with Roland, I think one list is fine.

As a non-digest lurker for over a month, I've both enjoyed and been
impressed with the professionalism of those on the team and their
usage of the da'list.  My two-bits worth: if interruptions are the
issue, turn on digest mode and reduce context switching...

Awaiting with baited breath the fruits of your labors,

/bill

--------------------------------------
William D. "Bill" Thompson
Libre Business Solutions, Inc.
San Mateo, CA

bill thompson name
bill LibreSolutions com

Re: Spirit

After that, its messages to Earth became sporadic,
feeble and in some cases garbled. More detective work
determined that its processor repeatedly wakes up,
attempts to load software data, finds a problem
and then presses its own reset button.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
> 



From Nitin.Hande at Sun.COM  Tue Feb 15 13:36:38 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Tue, 15 Feb 2005 13:36:38 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108494564.25271.29.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
Message-ID: <1108503397.51537.231.camel@sr1-umpk-01>

I have a hunch for whats happening here, but before I jump into any
conclusions, I am seeing some other issue between Solaris IPoIB driver
and OpenSM. After joining the Broadcast group, the PathRecord Response
coming from OpenSM signals an error with Invalid GUID. I wonder why,
Here is the mad trace:


Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x2 - SubnAdmSet()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d100000034
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f 0123456789abcdef
 0: 01 03 02 02 00 00 00 00 09 76 51 d1 00 00 00 34  .........vQ....4
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b 00 00 00 00  .....vQ.........
60: ff ff 00 00 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x81 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d100000034
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 81 00 00 00 00 09 76 51 d1 00 00 00 34  .........vQ....4
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00  ................
30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b c0 00 04 00  .....vQ.........
60: ff ff 03 12 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

OpenSM responds positively to MCMEMBERRECORD and then:

Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x12 - SubnAdmGetTable()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d10000003a
        AttributeID: 0x35 - SA_PATHRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 3a  .........vQ....:
10: 00 35 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .5..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 00 18 0c 00 00 00 00 00 00 00 00  ................
40: ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff ff ff  .. at .............
50: fe 80 00 00 00 00 00 00 00 02 c9 01 09 76 51 d1  .............vQ.
60: 00 00 00 00 00 00 00 00 00 81 00 00 00 00 00 00  ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x92 -
        Status: 0x500 -
        ClassSpecific: 0x0
        TransactionID: 0x97651d10000003a
        AttributeID: 0x35 - SA_PATHRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 92 05 00 00 00 09 76 51 d1 00 00 00 3a  .........vQ....:
10: 00 35 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .5..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 00 18 0c 00 00 00 00 00 00 00 00  ................
40: ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff ff ff  .. at .............
50: fe 80 00 00 00 00 00 00 00 02 c9 01 09 76 51 d1  .............vQ.
60: 00 00 00 00 00 00 00 00 00 81 00 00 00 00 00 00  ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[I have manually formatted the output, hope it comes clean]

Hal, Any idea ???. BTW, I havent applied your Sunday nights patch to the
OpenSM while I try to reproduce the 252 byte MTU size problem. 

thanks
Nitin






On Tue, 2005-02-15 at 11:12, Hal Rosenstock wrote:
> Hi,
> 
> Unfortunately, the Solaris 10 IPoIB MTU with OpenSM is back to the
> maximum size of 252 again :-( I'm not sure whether this was ever really
> fixed although I do now see the packets indicating an exact MTU of 4
> (2048 bytes). I'm not sure what Solaris doesn't like about the OpenSM
> response to the MCMemberRecord.
> 
> -- Hal
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Tue Feb 15 13:36:46 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 16:36:46 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108503397.51537.231.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
Message-ID: <1108503405.4667.42.camel@localhost.localdomain>

On Tue, 2005-02-15 at 16:36, Nitin Hande wrote:
> I have a hunch for whats happening here,

Glad to hear this as I don't have a clue :-)

>  but before I jump into any
> conclusions, I am seeing some other issue between Solaris IPoIB driver
> and OpenSM. After joining the Broadcast group, the PathRecord Response
> coming from OpenSM signals an error with Invalid GUID. I wonder why,

I missed this as my decode is manual being 1.0.a style. I will look at
this and my traces and get back to you later on this.

> Here is the mad trace:
> 
> 
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x2 - SubnAdmSet()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d100000034
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>      0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f 0123456789abcdef
>  0: 01 03 02 02 00 00 00 00 09 76 51 d1 00 00 00 34  .........vQ....4
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
> 50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b 00 00 00 00  .....vQ.........
> 60: ff ff 00 00 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x81 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d100000034
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 81 00 00 00 00 09 76 51 d1 00 00 00 34  .........vQ....4
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00  ................
> 30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
> 50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b c0 00 04 00  .....vQ.........
> 60: ff ff 03 12 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> OpenSM responds positively to MCMEMBERRECORD and then:
> 
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x12 - SubnAdmGetTable()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d10000003a
>         AttributeID: 0x35 - SA_PATHRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 3a  .........vQ....:
> 10: 00 35 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .5..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 00 18 0c 00 00 00 00 00 00 00 00  ................
> 40: ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff ff ff  .. at .............
> 50: fe 80 00 00 00 00 00 00 00 02 c9 01 09 76 51 d1  .............vQ.
> 60: 00 00 00 00 00 00 00 00 00 81 00 00 00 00 00 00  ................
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x92 -
>         Status: 0x500 -
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d10000003a
>         AttributeID: 0x35 - SA_PATHRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 92 05 00 00 00 09 76 51 d1 00 00 00 3a  .........vQ....:
> 10: 00 35 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .5..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 00 18 0c 00 00 00 00 00 00 00 00  ................
> 40: ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff ff ff  .. at .............
> 50: fe 80 00 00 00 00 00 00 00 02 c9 01 09 76 51 d1  .............vQ.
> 60: 00 00 00 00 00 00 00 00 00 81 00 00 00 00 00 00  ................
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> [I have manually formatted the output, hope it comes clean]

It came thru just fine.

> Hal, Any idea ???. 

Not yet but stay tuned.

> BTW, I havent applied your Sunday nights patch to the
> OpenSM while I try to reproduce the 252 byte MTU size problem. 

That's to set the selectors to exactly in the MCMemberRecord response.

> thanks
likewise

-- Hal

> Nitin
> 
> 
> 
> 
> 
> 
> On Tue, 2005-02-15 at 11:12, Hal Rosenstock wrote:
> > Hi,
> > 
> > Unfortunately, the Solaris 10 IPoIB MTU with OpenSM is back to the
> > maximum size of 252 again :-( I'm not sure whether this was ever really
> > fixed although I do now see the packets indicating an exact MTU of 4
> > (2048 bytes). I'm not sure what Solaris doesn't like about the OpenSM
> > response to the MCMemberRecord.
> > 
> > -- Hal
> > 
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> > 
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 



From halr at voltaire.com  Tue Feb 15 13:45:56 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 16:45:56 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108503397.51537.231.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
Message-ID: <1108503956.4667.50.camel@localhost.localdomain>

Hi again Nitin,

On Tue, 2005-02-15 at 16:36, Nitin Hande wrote:
> After joining the Broadcast group, the PathRecord Response
> coming from OpenSM signals an error with Invalid GUID. I wonder why,

There appear to be only 2 places in the code (I'm not saying the code is
right) where this can occur.

osm_sa_path_record.c:

if( p_sa_mad->comp_mask & IB_PR_COMPMASK_DGID )
...
      /*
        This 'error' is the client's fault (bad gid) so
        don't enter it as an error in our own log.
        Return an error response to the client.
      */
      osm_log( p_rcv->p_log, OSM_LOG_VERBOSE,
               "__osm_pr_rcv_get_end_points: "
               "No dest port with GUID = 0x%016" PRIx64 ".\n",
               cl_ntoh64( p_pr->dgid.unicast.interface_id) );

      sa_status = IB_SA_MAD_STATUS_INVALID_GID;

and a similar thing for SGID

  if( p_sa_mad->comp_mask & IB_PR_COMPMASK_SGID )
  {
...

      /*
        This 'error' is the client's fault (bad gid) so
        don't enter it as an error in our own log.
        Return an error response to the client.
      */
      osm_log( p_rcv->p_log, OSM_LOG_VERBOSE,
               "__osm_pr_rcv_get_end_points: "
               "No source port with GUID = 0x%016" PRIx64 ".\n",
               cl_ntoh64( p_pr->sgid.unicast.interface_id) );

Can you look in the osm.log to see if the source or dest GID is
implicated ? This will help me chase it down. Thanks.

-- Hal



From halr at voltaire.com  Tue Feb 15 13:52:30 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 16:52:30 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108503956.4667.50.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108503956.4667.50.camel@localhost.localdomain>
Message-ID: <1108504350.4667.57.camel@localhost.localdomain>

Hi Nitin,

On Tue, 2005-02-15 at 16:45, Hal Rosenstock wrote:
> Can you look in the osm.log to see if the source or dest GID is
> implicated ? This will help me chase it down. Thanks.

Both SGID and DGID are in the component mask but my bet is on the DGID.
OpenSM does not currently support PathRecords for MGIDs. I will working
on fixing this.

-- Hal



From Nitin.Hande at Sun.COM  Tue Feb 15 14:45:59 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Tue, 15 Feb 2005 14:45:59 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108503956.4667.50.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108503956.4667.50.camel@localhost.localdomain>
Message-ID: <1108507558.51537.243.camel@sr1-umpk-01>

On Tue, 2005-02-15 at 13:45, Hal Rosenstock wrote:
> Hi again Nitin,
> 
> On Tue, 2005-02-15 at 16:36, Nitin Hande wrote:
> > After joining the Broadcast group, the PathRecord Response
> > coming from OpenSM signals an error with Invalid GUID. I wonder why,
> 
> There appear to be only 2 places in the code (I'm not saying the code is
> right) where this can occur.
> 
> osm_sa_path_record.c:
> 
> if( p_sa_mad->comp_mask & IB_PR_COMPMASK_DGID )
> ...
>       /*
>         This 'error' is the client's fault (bad gid) so
>         don't enter it as an error in our own log.
>         Return an error response to the client.
>       */
>       osm_log( p_rcv->p_log, OSM_LOG_VERBOSE,
>                "__osm_pr_rcv_get_end_points: "
>                "No dest port with GUID = 0x%016" PRIx64 ".\n",
>                cl_ntoh64( p_pr->dgid.unicast.interface_id) );
> 
>       sa_status = IB_SA_MAD_STATUS_INVALID_GID;
> 
> and a similar thing for SGID
> 
>   if( p_sa_mad->comp_mask & IB_PR_COMPMASK_SGID )
>   {
> ...
> 
>       /*
>         This 'error' is the client's fault (bad gid) so
>         don't enter it as an error in our own log.
>         Return an error response to the client.
>       */
>       osm_log( p_rcv->p_log, OSM_LOG_VERBOSE,
>                "__osm_pr_rcv_get_end_points: "
>                "No source port with GUID = 0x%016" PRIx64 ".\n",
>                cl_ntoh64( p_pr->sgid.unicast.interface_id) );
> 
> Can you look in the osm.log to see if the source or dest GID is
> implicated ? This will help me chase it down. Thanks.
Here is the osm log, I think we may have a lead, the dest GID is wrong: 
:

Feb 15 23:29:57 [43005960] -> osm_sm_mcgrp_join: Port 0x0002c901097651d1
joining MLID 0xC001.
Feb 15 23:29:57 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
with GUID = 0x00000000ffffffff.
Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:29:58 [43005960] -> osm_report_notice: Reporting Generic
Notice type:3 num:66 from LID:0x0001
GID:0xfe80000000000000,0x0002c9010a99e031
Feb 15 23:29:58 [43005960] -> osm_sm_mcgrp_join: Port 0x0002c901097651d1
joining MLID 0xC011.
Feb 15 23:29:58 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
with GUID = 0x00000000ffffffff.
Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:29:59 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
with GUID = 0x00000000ffffffff.
Feb 15 23:30:01 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:30:01 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:30:01 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
with GUID = 0x00000000ffffffff.
Feb 15 23:30:03 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
with GUID = 0x00000000ffffffff.
Feb 15 23:30:04 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.
Feb 15 23:30:04 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
Join State != FullMember - required for create.

thanks
Nitin

> 
> -- Hal
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From jlentini at netapp.com  Tue Feb 15 15:29:46 2005
From: jlentini at netapp.com (James Lentini)
Date: Tue, 15 Feb 2005 18:29:46 -0500 (EST)
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>
References: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>
Message-ID: <Pine.LNX.4.61.0502151821270.5704@jlentini-linux.nane.netapp.com>


> After discussing it more with folks here, is seems to us that 
> perhaps for the uDAPL user-mode library, it be provided to 
> openib.org under a dual BSD + LGPL library rather than a BSD + GPL 
> since people normally want to use LGPL for libraries.

I'm in favor of what ever licensing scheme our user's prefer. What 
licensing scheme does the OpenIB user space verbs library use?


From libor at topspin.com  Tue Feb 15 16:03:05 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 15 Feb 2005 16:03:05 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <1108420762.6001.37.camel@duffman>;
	from tduffy@sun.com on Mon, Feb 14, 2005 at 02:39:21PM -0800
References: <20050211184058.I20777@topspin.com>
	<1108180061.25868.7.camel@duffman>
	<1108417356.6001.33.camel@duffman>
	<20050214142034.F28266@topspin.com>
	<1108420762.6001.37.camel@duffman>
Message-ID: <20050215160305.A3248@topspin.com>

On Mon, Feb 14, 2005 at 02:39:21PM -0800, Tom Duffy wrote:
> On Mon, 2005-02-14 at 14:20 -0800, Libor Michalek wrote:
> >   Here's a patch for the print format warning in the debug data path.
> 
> OK, good.  Thanks.  Now clean on x86_64 with data debug on.  Still
> seeing these on sparc64:

Tom,

  This patch should clear up all the warnings, can you give it
a try and let me know if it works.

Thanks.

-Libor

Index: sdp_write.c
===================================================================
--- sdp_write.c	(revision 1799)
+++ sdp_write.c	(working copy)
@@ -124,7 +124,7 @@
 
 			sdp_dbg_warn(conn, 
 				     "No IOCB on write complete <%llu:%d:%d>",
-				     comp->wr_id,
+				     (unsigned long long)comp->wr_id,
 				     sdp_desc_q_size(&conn->w_snk),
 				     sdp_desc_q_size(&conn->send_queue));
 			
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1799)
+++ sdp_rcvd.c	(working copy)
@@ -1310,7 +1310,8 @@
 	if (comp->wr_id != buff->wrid) {
 
 		sdp_dbg_warn(conn, "work request ID mismatch. <%llu:%llu>",
-			     comp->wr_id, buff->wrid);
+			     (unsigned long long)comp->wr_id,
+			     (unsigned long long)buff->wrid);
 
 		result = -ERANGE;
 		goto drop;
Index: sdp_read.c
===================================================================
--- sdp_read.c	(revision 1799)
+++ sdp_read.c	(working copy)
@@ -172,7 +172,8 @@
 		if (comp->wr_id != buff->wrid) {
 
 			sdp_dbg_warn(conn, "work request mismatch <%llu:%llu>",
-				     comp->wr_id, buff->wrid);
+				     (unsigned long long)comp->wr_id,
+				     (unsigned long long)buff->wrid);
 
 			(void)sdp_buff_pool_put(buff);
 			result = -EPROTO;
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1799)
+++ sdp_actv.c	(working copy)
@@ -382,7 +382,8 @@
 	if (id != conn->plid) {
 
 		sdp_dbg_warn(conn, "Path record ID mismatch <%016llx:%016llx>",
-			     id, conn->plid);
+			     (unsigned long long)id, 
+			     (unsigned long long)conn->plid);
 		goto done;
 	}
 
@@ -409,7 +410,9 @@
 	}
 	
 	sdp_dbg_ctrl(conn, "Path record lookup complete <%016llx:%016llx:%d>",
+		     (unsigned long long)
 		     cpu_to_be64(path->dgid.global.subnet_prefix),
+		     (unsigned long long)
 		     cpu_to_be64(path->dgid.global.interface_id),
 		     path->dlid);
 	/*
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1799)
+++ sdp_pass.c	(working copy)
@@ -380,7 +380,7 @@
 
 	sdp_dbg_ctrl(NULL, 
 		     "CM REQ. comm <%08x> SID <%016llx> ca <%s> port <%d>",
-		     cm_id->local_id, cm_id->service_id,
+		     cm_id->local_id, (unsigned long long)cm_id->service_id,
 		     event->param.req_rcvd.device->name, 
 		     event->param.req_rcvd.port);
 	/*
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1799)
+++ sdp_sent.c	(working copy)
@@ -448,8 +448,8 @@
 			 */
 			sdp_dbg_warn(conn, 
 				     "Send wrid mismatch. <%llu:%llu:%d>",
-				     comp->wr_id,
-				     buff->wrid,
+				     (unsigned long long)comp->wr_id,
+				     (unsigned long long)buff->wrid,
 				     conn->send_usig);
 			result = -EINVAL;
 			goto drop;
@@ -536,8 +536,9 @@
 
 		sdp_dbg_warn(conn, 
 			     "Send processing mismatch. <%llu:%llu:%d:%d>",
-			     comp->wr_id, current_wrid, free_count,
-			     conn->send_usig);
+			     (unsigned long long)comp->wr_id,
+			     (unsigned long long)current_wrid,
+			     free_count, conn->send_usig);
 		result = -EINVAL;
 		goto done;
 	}
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1799)
+++ sdp_event.c	(working copy)
@@ -45,7 +45,7 @@
 		 */
 		sdp_dbg_warn(conn, "Event <%d:%llu:%u:%u:%u> ignored in state",
 			     comp->status,
-			     comp->wr_id,
+			     (unsigned long long)comp->wr_id,
 			     comp->opcode,
 			     comp->byte_len,
 			     comp->imm_data);
@@ -105,7 +105,7 @@
 		sdp_dbg_warn(conn, "ABORT on error <%d> event <%u:%llu:%u:%u>",
 			     result,
 			     comp->status,
-			     comp->wr_id,
+			     (unsigned long long)comp->wr_id,
 			     comp->opcode,
 			     comp->byte_len);
 		/*




From halr at voltaire.com  Tue Feb 15 15:57:36 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Feb 2005 18:57:36 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108507558.51537.243.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108503956.4667.50.camel@localhost.localdomain>
	<1108507558.51537.243.camel@sr1-umpk-01>
Message-ID: <1108511856.4646.39.camel@localhost.localdomain>

On Tue, 2005-02-15 at 17:45, Nitin Hande wrote:
> Here is the osm log, I think we may have a lead, the dest GID is wrong: 
> :
> 
> Feb 15 23:29:57 [43005960] -> osm_sm_mcgrp_join: Port 0x0002c901097651d1
> joining MLID 0xC001.
> Feb 15 23:29:57 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> with GUID = 0x00000000ffffffff.
> Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:29:58 [43005960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:66 from LID:0x0001
> GID:0xfe80000000000000,0x0002c9010a99e031
> Feb 15 23:29:58 [43005960] -> osm_sm_mcgrp_join: Port 0x0002c901097651d1
> joining MLID 0xC011.
> Feb 15 23:29:58 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> with GUID = 0x00000000ffffffff.
> Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:29:59 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> with GUID = 0x00000000ffffffff.
> Feb 15 23:30:01 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:30:01 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:30:01 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> with GUID = 0x00000000ffffffff.
> Feb 15 23:30:03 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> with GUID = 0x00000000ffffffff.
> Feb 15 23:30:04 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.
> Feb 15 23:30:04 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> Join State != FullMember - required for create.

Are you sure ? 

It looks to me like it is trying to get a PathRecord to the broadcast
MGID as DGID:
  AttributeID: 0x35 - SA_PATHRECORD_ATTRID
...
40: ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff ff ff

I'm not sure this is necessary as aren't all the parameters returned in
the SA GetResp MCMemberRecord in response to the Set ? Anyhow it is
perfectly legal. OpenSM just doesn't support it right now.

-- Hal



From tduffy at sun.com  Tue Feb 15 16:54:10 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 15 Feb 2005 16:54:10 -0800
Subject: [openib-general] bugzilla question
In-Reply-To: <OFAC1BE5AD.3A6E08D5-ON87256FA9.00635BBB-88256FA9.0063AF85@us.ibm.com>
References: <OFAC1BE5AD.3A6E08D5-ON87256FA9.00635BBB-88256FA9.0063AF85@us.ibm.com>
Message-ID: <1108515250.13256.9.camel@duffman>

On Tue, 2005-02-15 at 10:08 -0800, Shirley Ma wrote:
> 
> I am planning to use bugzilla to file bug reports. I logged in but I
> only see one bug report there. Are we really using this? 

I have filed a couple.  I won't discourage you using bugzilla, but you
may have quicker response on the list.  Bugzilla will be great once the
project grows beyond the bounds were issues can be tracked in email.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/11d83135/attachment.sig>

From tduffy at sun.com  Tue Feb 15 16:56:34 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 15 Feb 2005 16:56:34 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>
References: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>
Message-ID: <1108515394.13256.13.camel@duffman>

On Mon, 2005-02-14 at 16:47 -0800, Woodruff, Robert J wrote: 
>  Hi Arkady,
> 
> As I mentioned in the BOF, I have a person (Arlin Davis) that can help
> with
> developing a uDAPL provider for the openib.org verbs. 
> After discussing it more
> with folks here, is seems to us that perhaps for the uDAPL user-mode
> library,  it be provided to openib.org under a dual BSD + LGPL library
> rather than a BSD + GPL since people normally want to use LGPL for 
> libraries. 

I think using an LGPL instead of GPL would require a change to the
openib.org bylaws.

Jim, is that right?

Honestly, if they are licensed BSD too, people can do whatever they
want.  BSD/GPL will be fine on these libraries, IMO.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/f4a0812d/attachment.sig>

From tduffy at sun.com  Tue Feb 15 17:00:38 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 15 Feb 2005 17:00:38 -0800
Subject: [openib-general] curious warning message (gen1, ibgd, possibly ot)
In-Reply-To: <200502151948.49813.cap@nsc.liu.se>
References: <200502151948.49813.cap@nsc.liu.se>
Message-ID: <1108515639.13256.16.camel@duffman>

On Tue, 2005-02-15 at 19:48 +0100, Peter Kjellström wrote:
> Hello folks,
> 
> I'm sorry if this is kind of off-topic for this list but I suspect that there 
> are quite a few people on this list that can answer my rather simple 
> question.
> 
> Every time the tavor-driver loads I see these (kinda scary) warnings:
> 
> THH kernel module initialized successfully VAPI(2): thhul_qpm.c[2140]: 
> ***WARNING***: AVs for special QPs should use HOST memory; the provided PD 
> has its AVs in DDR memory.
> (repeated four times)
> 
> Does anybody know what this means (it's not exactly english to me ;). Can it 
> be safely ignored? (the systems seems to humm along nicely..)

Any reason not to use gen2?

The message looks like it is saying that the driver is using the memory
on board the Arbel card instead of main system memory when it shouldn't.
Not sure what the consequences are of doing this.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/3d0439bd/attachment.sig>

From tduffy at sun.com  Tue Feb 15 17:13:06 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 15 Feb 2005 17:13:06 -0800
Subject: [openib-general] [PATCH] SDP warnings on x86_64
In-Reply-To: <20050215160305.A3248@topspin.com>
References: <20050211184058.I20777@topspin.com>
	<1108180061.25868.7.camel@duffman> <1108417356.6001.33.camel@duffman>
	<20050214142034.F28266@topspin.com> <1108420762.6001.37.camel@duffman>
	<20050215160305.A3248@topspin.com>
Message-ID: <1108516386.13256.17.camel@duffman>

On Tue, 2005-02-15 at 16:03 -0800, Libor Michalek wrote:
>   This patch should clear up all the warnings, can you give it
> a try and let me know if it works.

Perfect.  Clean yummy compiles.  Thanks.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050215/73fe2031/attachment.sig>

From jim.ryan at intel.com  Tue Feb 15 17:32:17 2005
From: jim.ryan at intel.com (Ryan, Jim)
Date: Tue, 15 Feb 2005 17:32:17 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <DC0618F6095F6745A5B480A6A53E91AC01AD642D@orsmsx405>

Tom Duffy wrote:
> On Mon, 2005-02-14 at 16:47 -0800, Woodruff, Robert J wrote:
>>  Hi Arkady,
>> 
>> As I mentioned in the BOF, I have a person (Arlin Davis) that can
>> help with developing a uDAPL provider for the openib.org verbs.
>> After discussing it more
>> with folks here, is seems to us that perhaps for the uDAPL user-mode
>> library,  it be provided to openib.org under a dual BSD + LGPL
>> library rather than a BSD + GPL since people normally want to use
>> LGPL for libraries.
> 
> I think using an LGPL instead of GPL would require a change to the
> openib.org bylaws.
> 
> Jim, is that right?
> 
> Honestly, if they are licensed BSD too, people can do whatever they
> want.  BSD/GPL will be fine on these libraries, IMO.
> 
> -tduffy

Tom, good to hear from you. 

Yes, it would require a change in our bylaws which are very clear wrt
GPL and BSD.

Please let me know if you'd like to discuss further. 

Regards, Jim
Chairman, OpenIB


From Nitin.Hande at Sun.COM  Tue Feb 15 17:46:22 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Tue, 15 Feb 2005 17:46:22 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108511856.4646.39.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108503956.4667.50.camel@localhost.localdomain>
	<1108507558.51537.243.camel@sr1-umpk-01>
	<1108511856.4646.39.camel@localhost.localdomain>
Message-ID: <1108518381.51537.256.camel@sr1-umpk-01>

On Tue, 2005-02-15 at 15:57, Hal Rosenstock wrote:
> On Tue, 2005-02-15 at 17:45, Nitin Hande wrote:
> > Here is the osm log, I think we may have a lead, the dest GID is wrong: 
> > :
> > 
> > Feb 15 23:29:57 [43005960] -> osm_sm_mcgrp_join: Port 0x0002c901097651d1
> > joining MLID 0xC001.
> > Feb 15 23:29:57 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> > with GUID = 0x00000000ffffffff.
> > Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:29:58 [43005960] -> osm_report_notice: Reporting Generic
> > Notice type:3 num:66 from LID:0x0001
> > GID:0xfe80000000000000,0x0002c9010a99e031
> > Feb 15 23:29:58 [43005960] -> osm_sm_mcgrp_join: Port 0x0002c901097651d1
> > joining MLID 0xC011.
> > Feb 15 23:29:58 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> > with GUID = 0x00000000ffffffff.
> > Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:29:58 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:29:59 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> > with GUID = 0x00000000ffffffff.
> > Feb 15 23:30:01 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:30:01 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:30:01 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> > with GUID = 0x00000000ffffffff.
> > Feb 15 23:30:03 [43005960] -> __osm_pr_rcv_get_end_points: No dest port
> > with GUID = 0x00000000ffffffff.
> > Feb 15 23:30:04 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> > Feb 15 23:30:04 [43005960] -> osm_mcmr_rcv_join_mgrp: ERR 1B10: Provided
> > Join State != FullMember - required for create.
> 
> Are you sure ? 
No, I was looking at some other location. 
> 
> It looks to me like it is trying to get a PathRecord to the broadcast
> MGID as DGID:
>   AttributeID: 0x35 - SA_PATHRECORD_ATTRID
> ...
> 40: ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff ff ff
> 
> I'm not sure this is necessary as aren't all the parameters returned in
> the SA GetResp MCMemberRecord in response to the Set ? Anyhow it is
> perfectly legal. OpenSM just doesn't support it right now.
Yes ,sure. Will wait for the patches..


Thanks
Nitin

> 
> -- Hal
> 



From mlleinin at hpcn.ca.sandia.gov  Tue Feb 15 21:51:30 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Tue, 15 Feb 2005 21:51:30 -0800
Subject: [openib-general] Mailing
In-Reply-To: <523bvxzrhx.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
	<523bvxzrhx.fsf@topspin.com>
Message-ID: <1108533090.18619.323.camel@localhost>

On Tue, 2005-02-15 at 09:02 -0800, Roland Dreier wrote:
>     Eitan> 1. Use more mailing lists (e.g. openib-ib-manegement )
> 
> I've always resisted splitting openib-general.  I still think it's a
> bad idea for several reasons:
> 
>  - openib-general is still a low-traffic list.  linux-kernel works
>    fine with 10 times the number of messages per day.
>  - we end up wasting time telling people they need to post to a
>    different list, and users end up cross-posting to all the lists
>    (and developers end up subscribing to all the lists)
>  - splitting the list dilutes the community.
> 
  I agree.  The classic example is lkml.  They seem to do fine with one
list and their traffic is much heavier than ours.  

  We could put some suggested mail list keywords on the webpage.  But
then we may end up with a mail archive where every SM bug has a subject
line "SM problem".  

  - Matt



From paul.baxter at dsl.pipex.com  Wed Feb 16 00:48:13 2005
From: paul.baxter at dsl.pipex.com (Paul Baxter)
Date: Wed, 16 Feb 2005 08:48:13 -0000
Subject: [openib-general] Licensing for user space
References: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>
	<Pine.LNX.4.61.0502151821270.5704@jlentini-linux.nane.netapp.com>
Message-ID: <002b01c51404$40a6dff0$8000000a@blorp>

>> After discussing it more with folks here, is seems to us that perhaps for 
>> the uDAPL user-mode library, it be provided to openib.org under a dual 
>> BSD + LGPL library rather than a BSD + GPL since people normally want to 
>> use LGPL for libraries.

Why not consider a more open license still, one that allows free commercial 
or non-commercial use such as the boost license ( 
http://www.boost.org/more/license_info.html ). Is there anything to lose by 
adopting an even less restrictive license?

Paul





From tziporet at mellanox.co.il  Wed Feb 16 05:50:38 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Wed, 16 Feb 2005 15:50:38 +0200
Subject: [openib-general] curious warning message (gen1, ibgd, possibl
	y ot)
Message-ID: <506C3D7B14CDD411A52C00025558DED6064BEF79@mtlex01.yok.mtl.com>

Hi,

This message indicates that UDAVs for special QPs are located in the DDR
memory and not host memory.
It is a warning since when working with host memory the performance of the
MADs is better, however everything will work just fine.
So in general you can ignore these messages.

Tziporet

-----Original Message-----
From: Peter Kjellström [mailto:cap at nsc.liu.se]
Sent: Tuesday, February 15, 2005 8:49 PM
To: openib-general at openib.org
Subject: [openib-general] curious warning message (gen1, ibgd, possibly
ot)


Hello folks,

I'm sorry if this is kind of off-topic for this list but I suspect that
there 
are quite a few people on this list that can answer my rather simple 
question.

Every time the tavor-driver loads I see these (kinda scary) warnings:

THH kernel module initialized successfully VAPI(2): thhul_qpm.c[2140]: 
***WARNING***: AVs for special QPs should use HOST memory; the provided PD 
has its AVs in DDR memory.
(repeated four times)

Does anybody know what this means (it's not exactly english to me ;). Can it

be safely ignored? (the systems seems to humm along nicely..)

I'm running IBGD-1.6.1 from mellanox on my Xeon Nocona cluster
(pci-express). 
My HCAs are of the so called "lion cub" type and the os is centos-3.4 
(rhel3u4). Firmware is afaict 4.5.3 (fw_ver=0x400050003 as reported by 
vstat).

tia,
 Peter

--
------------------------------------------------------------
  Peter Kjellström               |
  National Supercomputer Centre  |
  Sweden                         | http://www.nsc.liu.se
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050216/0436442b/attachment.html>

From halr at voltaire.com  Wed Feb 16 06:27:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 16 Feb 2005 09:27:53 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108503397.51537.231.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
Message-ID: <1108563413.4646.171.camel@localhost.localdomain>

On Tue, 2005-02-15 at 16:36, Nitin Hande wrote:
> I have a hunch for whats happening here, but before I jump into any
> conclusions, I am seeing some other issue between Solaris IPoIB driver
> and OpenSM. After joining the Broadcast group, the PathRecord Response
> coming from OpenSM signals an error with Invalid GUID. 

Is the MTU from the PathRecord used ? Is that the theory ? So these are
one and the same issue. Thanks.

-- Hal



From jlentini at netapp.com  Wed Feb 16 06:49:27 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 16 Feb 2005 09:49:27 -0500 (EST)
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <1108374321.18616.94.camel@localhost>
References: <DOEHKBHAGBADBEKBCHKCMEBFDEAA.bill@thompson.name>
	<1108367138.3078.15.camel@wynux.LibreSolutions>
	<1108374321.18616.94.camel@localhost>
Message-ID: <Pine.LNX.4.61.0502160941030.5704@jlentini-linux.nane.netapp.com>



On Mon, 14 Feb 2005, Matt Leininger wrote:

>  Several developers have volunteered to work on uDAPL.  The DAT
> collaborative is working to get a GPL/BSD version of uDAPL to OpenIB.
> James and Arkady from NetApp gave a DAT talk.  They mentioned needing 1
> month to get the GPL code, and another month or two to fork the code
> base.  I'm not sure why 2-3 months is needed to do this.  Perhaps James
> and/or Arkady can comment.

My 2-3 month estimate was based on 1 month for the DAT Collaborative 
to discuss and vote on licensing the code under the GPL and 1-2 months 
to port and test on OpenIB Gen 2.

> I think we need to see the GPL/BSD uDAPL code for OpenIB in the next 
> couple of weeks.  Better to put the code in the open so everyone can 
> see what changes are required.  Same goes for kDAPL.  Was the 
> u/kDAPL implementation for OpenIB going to live at openib.org or 
> stay under cvs at the sourceforge dapl project?

We are proposing to maintain the code in both locations. The OpenIB 
specific implementation would be on openib.org while a 
platform/transport independent implementation would continue on 
our SourceForge site.


From roland at topspin.com  Wed Feb 16 07:51:04 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 16 Feb 2005 07:51:04 -0800
Subject: [openib-general] Licensing for user space
In-Reply-To: <002b01c51404$40a6dff0$8000000a@blorp> (Paul Baxter's message
	of "Wed, 16 Feb 2005 08:48:13 -0000")
References: <1AC79F16F5C5284499BB9591B33D6F0003915808@orsmsx408>
	<Pine.LNX.4.61.0502151821270.5704@jlentini-linux.nane.netapp.com>
	<002b01c51404$40a6dff0$8000000a@blorp>
Message-ID: <52psz0y047.fsf@topspin.com>

    Paul> Why not consider a more open license still, one that allows
    Paul> free commercial or non-commercial use such as the boost
    Paul> license ( http://www.boost.org/more/license_info.html ). Is
    Paul> there anything to lose by adopting an even less restrictive
    Paul> license?

First of all, I don't think it's worth the hassle of reopening the
license debate.  GPL/BSD is enshrined in the OpenIB bylaws and I think
it would be far more trouble than it's worth to change it.

Second, as far as I can tell, the only difference between the boost
license and the BSD license option is that the boost license does not
require the copyright notice to be included in binary-only
distributions of the software.  So I'm not sure there's any practical
difference anyone would care about between the boost license and our
existing 2-clause BSD option (and as the author of a lot of the code
we're discussing, I would sort of prefer that some attribution be
required even for binary-only distributions).

 - R.


From iod00d at hp.com  Wed Feb 16 08:20:52 2005
From: iod00d at hp.com (Grant Grundler)
Date: Wed, 16 Feb 2005 08:20:52 -0800
Subject: [openib-general] Mailing
In-Reply-To: <1108533090.18619.323.camel@localhost>
References: <506C3D7B14CDD411A52C00025558DED6047EEE25@mtlex01.yok.mtl.com>
	<523bvxzrhx.fsf@topspin.com> <1108533090.18619.323.camel@localhost>
Message-ID: <20050216162052.GB19061@esmail.cup.hp.com>

On Tue, Feb 15, 2005 at 09:51:30PM -0800, Matt Leininger wrote:
>   We could put some suggested mail list keywords on the webpage.  But
> then we may end up with a mail archive where every SM bug has a subject
> line "SM problem".  

I'm ok with that. "SM BUG" are not useful keywords for google anyway.

grant


From rminnich at lanl.gov  Wed Feb 16 08:45:17 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Wed, 16 Feb 2005 09:45:17 -0700 (MST)
Subject: [openib-general] question on opensm error
In-Reply-To: <1108530606.4646.60.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov> 
	<1108468017.21161.2130.camel@localhost.localdomain> 
	<1108472802.21161.2279.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502150647210.12960@enigma.lanl.gov> 
	<1108477905.21161.2454.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502151646160.13932@enigma.lanl.gov> 
	<1108512208.4646.50.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502152022170.14550@enigma.lanl.gov>
	<1108530606.4646.60.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502160942230.16089@enigma.lanl.gov>



On Tue, 16 Feb 2005, Hal Rosenstock wrote:

> On Tue, 2005-02-15 at 22:22, Ronald G. Minnich wrote:
> > On Tue, 15 Feb 2005, Hal Rosenstock wrote:
> > 
> > > I presume your subnet has 179 HCAs ? Do you know ?
> > 
> > no errors. It's just that opensm won't run. 
> 
> Won't run or won't do anything on the subnet ?
> 
> Not sure what you mean by won't run ?

ok, just found it. 

There is a sys fail red light on the CPU on the 96-port switch that the
opensm host attaches to.

What's weird is none of the ib admin tools found anything. ibnetdiscover 
happily walked the whole subnet. The only problem was that opensm would 
not run, but the errors were unclear. So many things appeared to be 
working that it did not occur to me to walk over and look at the switch. 
Stupid of me. 

Now that I've turned that switch off I get this:
[1108572233:000155763][40BFF970] -> __osm_state_mgr_sm_port_down_msg: 


******************************************************************
************************** SM PORT DOWN **************************
******************************************************************


[1108572233:000155778][40BFF970] -> __osm_sm_state_mgr_signal_error: ERR 
3207: Invalid signal OSM_SM_SIGNAL_DISCOVER in state 
IB_SMINFO_STATE_DISCOVERING.

which I assume is its way of telling me that the switch port is down. 

ron


From Arkady.Kanevsky at netapp.com  Wed Feb 16 08:50:16 2005
From: Arkady.Kanevsky at netapp.com (Kanevsky, Arkady)
Date: Wed, 16 Feb 2005 11:50:16 -0500
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <C98692FD98048C41885E0B0FACD9DFB86EB1B6@exnane01.hq.netapp.com>

woody,
If Open IB wants to go with LGPL license it is fine with us.
We will need to take a voit on DAT Collaborative onit also.
But from what I see on reflector LGPL license is outside the current
bylaws of
Open IB and there is a discussion on it going on.

SO until this issue is resolved on Open IB we will add GPL license to
uDAPL (and kDAPL)
and will work on getting it ready for submission to Open IB.
Once the license the GPL license is approved by DAT we can move the dev
work on Open IB SF.
If Arlin is available he can start the work now.
Lets first identify what areas require changes. 

Arkady

Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance                     phone: 781-768-5395
375 Totten Pond Rd.                  Fax: 781-895-1195
Waltham, MA 02451-2010          central phone: 781-768-5300
 


> -----Original Message-----
> From: Woodruff, Robert J [mailto:robert.j.woodruff at intel.com] 
> Sent: Monday, February 14, 2005 7:48 PM
> To: Kanevsky, Arkady
> Cc: openib-general at openib.org; Davis, Arlin R; Matt Leininger
> Subject: RE: FW: [openib-general] Minutes from DAPL BOF at 
> OpenIB Workshop
> 
> 
>  
> Hi Arkady,
> 
> As I mentioned in the BOF, I have a person (Arlin Davis) that 
> can help with developing a uDAPL provider for the openib.org verbs. 
> After discussing it more
> with folks here, is seems to us that perhaps for the uDAPL 
> user-mode library,  it be provided to openib.org under a dual 
> BSD + LGPL library rather than a BSD + GPL since people 
> normally want to use LGPL for 
> libraries. 
> 
> Also, I think that we can go ahead and start porting using the BSD 
> license available from sourceforge today. Once the port is 
> complete, we can submit it to openib.org under the dual 
> license and submit 
> any changes back to the main sourceforge project under the 
> BSD license, or you can simply accept the changes under the 
> BSD license, thus for uDAPL there is no need to wait for the 
> expanded licensing terms of the sourceforge project.
> 
> Once the user-mode verbs, user-mode CM, SA support is 
> available from openib.org, 
> we can get started. If this is OK with folks, I'll have Arlin 
> start to take a look at this.
> 
> Sound OK ?
> 
> woody
> 


From halr at voltaire.com  Wed Feb 16 08:47:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 16 Feb 2005 11:47:23 -0500
Subject: [openib-general] question on opensm error
In-Reply-To: <Pine.LNX.4.58.0502160942230.16089@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502141358490.10806@enigma.lanl.gov>
	<1108468017.21161.2130.camel@localhost.localdomain>
	<1108472802.21161.2279.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502150647210.12960@enigma.lanl.gov>
	<1108477905.21161.2454.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502151646160.13932@enigma.lanl.gov>
	<1108512208.4646.50.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502152022170.14550@enigma.lanl.gov>
	<1108530606.4646.60.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502160942230.16089@enigma.lanl.gov>
Message-ID: <1108572443.4645.64.camel@localhost.localdomain>

On Wed, 2005-02-16 at 11:45, Ronald G. Minnich wrote:
> On Tue, 16 Feb 2005, Hal Rosenstock wrote:
> 
> > On Tue, 2005-02-15 at 22:22, Ronald G. Minnich wrote:
> > > On Tue, 15 Feb 2005, Hal Rosenstock wrote:
> > > 
> > > > I presume your subnet has 179 HCAs ? Do you know ?
> > > 
> > > no errors. It's just that opensm won't run. 
> > 
> > Won't run or won't do anything on the subnet ?
> > 
> > Not sure what you mean by won't run ?
> 
> ok, just found it. 
> 
> There is a sys fail red light on the CPU on the 96-port switch that the
> opensm host attaches to.
> 
> What's weird is none of the ib admin tools found anything. ibnetdiscover 
> happily walked the whole subnet. The only problem was that opensm would 
> not run, but the errors were unclear. So many things appeared to be 
> working that it did not occur to me to walk over and look at the switch. 
> Stupid of me. 

Still not 100% clear on the failure mode. I don't know what the sys fail
light on the CPU means. It may mean that things partially work. By that,
I mean the CPU might crash but the IB chips continue to function based
on their current setup. It would depend on the split of functionality
between the CPU and the IB chip firmware (which may depend on vendor).

If you were able to walk the subnet with the (SMP based) diags, the SM
port had to be at least in init (ibstat/ibstatus).

The "keys" are what was the failure mode so we can see how this can be
detected better in the future, and what caused the switch CPU to crash
in the first place.

-- Hal

> Now that I've turned that switch off I get this:
> [1108572233:000155763][40BFF970] -> __osm_state_mgr_sm_port_down_msg: 
> 
> 
> ******************************************************************
> ************************** SM PORT DOWN **************************
> ******************************************************************
> 
> 
> [1108572233:000155778][40BFF970] -> __osm_sm_state_mgr_signal_error: ERR 
> 3207: Invalid signal OSM_SM_SIGNAL_DISCOVER in state 
> IB_SMINFO_STATE_DISCOVERING.
> 
> which I assume is its way of telling me that the switch port is down. 
> 
> ron



From Nitin.Hande at Sun.COM  Wed Feb 16 13:08:16 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Wed, 16 Feb 2005 13:08:16 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108563413.4646.171.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
Message-ID: <1108588095.16278.26.camel@sr1-umpk-01>

Hal,

On Wed, 2005-02-16 at 06:27, Hal Rosenstock wrote:
> On Tue, 2005-02-15 at 16:36, Nitin Hande wrote:
> > I have a hunch for whats happening here, but before I jump into any
> > conclusions, I am seeing some other issue between Solaris IPoIB driver
> > and OpenSM. After joining the Broadcast group, the PathRecord Response
> > coming from OpenSM signals an error with Invalid GUID. 
> 
> Is the MTU from the PathRecord used ? Is that the theory ? So these are
> one and the same issue. Thanks.
No, I was more of thinking of an endian issue between IBD and the layer
beneath it during the MCMemberRecord response. The mtu is not dependant
on PathRecord Response. Thanks to Tom, we have figured out a way of
consistently reproducing this on our systems here. The way to reproduce
is (basically start everything fresh):
1. rmmod {ib_mthca, umad and ipoib}, stop opensm
2. unplumb ibd driver and modunload ibd on Solaris,
3. modprobe and restart opensm
4. plumb ibd interface.
You should see ibd setting the mtu size to 252. Some of the above steps
maybe unecessary. From the trace, it looks like OpenSM is reporting 256
bytes of MTU to ipoib for MCMemberRecord response.

Here is the trace of 256 sized MTU:

Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x12 - SubnAdmGetTable()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d100000096
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x92 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d100000096
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 01 00  ................
60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

And on other occassions where OpenSM reports the 2048 sized MTU:

Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x12 - SubnAdmGetTable()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d10000009a
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x92 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d10000009a
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 04 00  ................
60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[output formatted manually]


thanks
Nitin


> 
> -- Hal
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Wed Feb 16 13:26:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 16 Feb 2005 16:26:14 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108588095.16278.26.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
Message-ID: <1108589174.4645.138.camel@localhost.localdomain>

On Wed, 2005-02-16 at 16:08, Nitin Hande wrote:
> Hal,
> 
> On Wed, 2005-02-16 at 06:27, Hal Rosenstock wrote:
> > On Tue, 2005-02-15 at 16:36, Nitin Hande wrote:
> > > I have a hunch for whats happening here, but before I jump into any
> > > conclusions, I am seeing some other issue between Solaris IPoIB driver
> > > and OpenSM. After joining the Broadcast group, the PathRecord Response
> > > coming from OpenSM signals an error with Invalid GUID. 
> > 
> > Is the MTU from the PathRecord used ? Is that the theory ? So these are
> > one and the same issue. Thanks.
> No, I was more of thinking of an endian issue between IBD and the layer
> beneath it during the MCMemberRecord response. The mtu is not dependant
> on PathRecord Response. Thanks to Tom, we have figured out a way of
> consistently reproducing this on our systems here. The way to reproduce
> is (basically start everything fresh):
> 1. rmmod {ib_mthca, umad and ipoib}, stop opensm
> 2. unplumb ibd driver and modunload ibd on Solaris,
> 3. modprobe and restart opensm
> 4. plumb ibd interface.
> You should see ibd setting the mtu size to 252. Some of the above steps
> maybe unecessary. From the trace, it looks like OpenSM is reporting 256
> bytes of MTU to ipoib for MCMemberRecord response.
> 
> Here is the trace of 256 sized MTU:
> 
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x12 - SubnAdmGetTable()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d100000096
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x92 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d100000096
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 01 00  ................
> 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> And on other occassions where OpenSM reports the 2048 sized MTU:
> 
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x12 - SubnAdmGetTable()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d10000009a
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x92 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d10000009a
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 04 00  ................
> 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> [output formatted manually]

These both have the exactly selector issue which I fixed but I think you
haven't picked that up out of the tree. Is that a problem for Solaris or
does it ignore this in the response ?

I have a theory for how the different MTUs (0 (256) and 4 (2048)) occur
but need a little time to validate it.

-- Hal



From Nitin.Hande at Sun.COM  Wed Feb 16 14:33:39 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Wed, 16 Feb 2005 14:33:39 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108589174.4645.138.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
	<1108589174.4645.138.camel@localhost.localdomain>
Message-ID: <1108593217.16278.47.camel@sr1-umpk-01>

On Wed, 2005-02-16 at 13:26, Hal Rosenstock wrote:
> On Wed, 2005-02-16 at 16:08, Nitin Hande wrote:
> > Hal,
[snip..]
> > 
> > 
> > Here is the trace of 256 sized MTU:
> > 
> > Outgoing MAD:
> >         BaseVersion: 0x1
> >         MgmtClass: 0x3 - SubnAdm
> >         ClassVersion: 0x2
> >         R_Method: 0x12 - SubnAdmGetTable()
> >         Status: 0x0 - NO_ERROR
> >         ClassSpecific: 0x0
> >         TransactionID: 0x97651d100000096
> >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > 
> >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> >  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
> > 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 
> > Incoming MAD:
> >         BaseVersion: 0x1
> >         MgmtClass: 0x3 - SubnAdm
> >         ClassVersion: 0x2
> >         R_Method: 0x92 -
> >         Status: 0x0 - NO_ERROR
> >         ClassSpecific: 0x0
> >         TransactionID: 0x97651d100000096
> >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > 
> >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> >  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
> > 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> > 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 01 00  ................
> > 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 
> > And on other occassions where OpenSM reports the 2048 sized MTU:
> > 
> > Outgoing MAD:
> >         BaseVersion: 0x1
> >         MgmtClass: 0x3 - SubnAdm
> >         ClassVersion: 0x2
> >         R_Method: 0x12 - SubnAdmGetTable()
> >         Status: 0x0 - NO_ERROR
> >         ClassSpecific: 0x0
> >         TransactionID: 0x97651d10000009a
> >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > 
> >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> >  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
> > 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 
> > Incoming MAD:
> >         BaseVersion: 0x1
> >         MgmtClass: 0x3 - SubnAdm
> >         ClassVersion: 0x2
> >         R_Method: 0x92 -
> >         Status: 0x0 - NO_ERROR
> >         ClassSpecific: 0x0
> >         TransactionID: 0x97651d10000009a
> >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > 
> >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> >  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
> > 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> > 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 04 00  ................
> > 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > [output formatted manually]
> 
> These both have the exactly selector issue which I fixed but I think you
> haven't picked that up out of the tree. Is that a problem for Solaris or
> does it ignore this in the response ?
> 
> I have a theory for how the different MTUs (0 (256) and 4 (2048)) occur
> but need a little time to validate it.
So I patched the OpenSM with your changes and re-ran the same
experiment. There was no difference in terms of response for
MCMemberRecord issued by ibd driver: 

Here is just the response part:

Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x92 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d1000000a3
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 a3  .........vQ.....
10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 01 00  ................
60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

What did change though is the Selector in the McMemberRecord response
when ipoib driver further tries to join this broadcast group and issues
another MCMemberRecord. 



Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x2 - SubnAdmSet()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d1000000a4
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 02 00 00 00 00 09 76 51 d1 00 00 00 a4  .........vQ.....
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b 00 00 00 00  .....vQ.........
60: ff ff 00 00 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x81 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d1000000a4
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

   0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 81 00 00 00 00 09 76 51 d1 00 00 00 a4  .........vQ.....
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00  ................
30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b c0 00 84 00  .....vQ.........
60: ff ff 83 92 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Before the patch the selector was coming 04. Do you reply 84 seeing a
particular component mask and otherwise 01 ??(I think not..) 


Thanks
Nitin





> 
> -- Hal
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From mshefty at ichips.intel.com  Wed Feb 16 15:58:28 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 16 Feb 2005 15:58:28 -0800
Subject: [openib-general] [PATCH] [CM] have SIDR use its own red-black tree
	node
Message-ID: <20050216155828.31ff0db1.mshefty@ichips.intel.com>

This patch defines a new red-black tree node for SIDR to use.  The
separation is done to simplify timewait state handling.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1799)
+++ infiniband/core/cm.c	(working copy)
@@ -112,6 +112,7 @@ struct cm_id_private {
 	struct rb_node service_node;
 	struct rb_node remote_qp_node;
 	struct rb_node remote_id_node;
+	struct rb_node sidr_id_node;
 	spinlock_t lock;
 	wait_queue_head_t wait;
 	atomic_t refcount;
@@ -501,7 +502,7 @@ static struct cm_id_private * cm_insert_
 	while (*link) {
 		parent = *link;
 		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
-					  remote_id_node);
+					  sidr_id_node);
 		if (remote_id < cur_cm_id_priv->id.remote_id)
 			link = &(*link)->rb_left;
 		else if (remote_id > cur_cm_id_priv->id.remote_id)
@@ -518,8 +519,8 @@ static struct cm_id_private * cm_insert_
 				return cur_cm_id_priv;
 		}
 	}
-	rb_link_node(&cm_id_priv->remote_id_node, parent, link);
-	rb_insert_color(&cm_id_priv->remote_id_node, &cm.remote_sidr_table);
+	rb_link_node(&cm_id_priv->sidr_id_node, parent, link);
+	rb_insert_color(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
 	return NULL;
 }
 
@@ -2240,7 +2241,7 @@ static int cm_sidr_req_handler(struct cm
 	}
 	cur_cm_id_priv = cm_find_listen(sidr_req_msg->service_id);
 	if (!cur_cm_id_priv) {
-		rb_erase(&cm_id_priv->remote_id_node, &cm.remote_sidr_table);
+		rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
 		spin_unlock_irqrestore(&cm.lock, flags);
 		/* todo: reject with no match */
 		goto out; /* No match. */
@@ -2325,7 +2326,7 @@ int ib_send_cm_sidr_rep(struct ib_cm_id 
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
 	spin_lock_irqsave(&cm.lock, flags);
-	rb_erase(&cm_id_priv->remote_id_node, &cm.remote_sidr_table);
+	rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
 	spin_unlock_irqrestore(&cm.lock, flags);
 out:
 	return ret;


From mshefty at ichips.intel.com  Wed Feb 16 16:03:34 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 16 Feb 2005 16:03:34 -0800
Subject: [openib-general] [PATCH] [CM] fix race accessing CM msg during
	destruction
Message-ID: <20050216160334.6522d5aa.mshefty@ichips.intel.com>

This patch fixes a race accessing a CM message when destroying a cm_id.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1807)
+++ infiniband/core/cm.c	(working copy)
@@ -588,6 +588,7 @@ int ib_destroy_cm_id(struct ib_cm_id *cm
 	struct cm_id_private *cm_id_priv;
 	struct cm_work *work;
 	unsigned long flags;
+	u64 wr_id;
 
 	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
 retest:
@@ -602,9 +603,9 @@ retest:
 		break;
 	case IB_CM_SIDR_REQ_SENT:
 		cm_id->state = IB_CM_IDLE;
+		wr_id = (unsigned long) cm_id_priv->msg;
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
-			      (unsigned long) cm_id_priv->msg);
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
 		break;
 	case IB_CM_SIDR_REQ_RCVD:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);


From mshefty at ichips.intel.com  Wed Feb 16 16:09:55 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 16 Feb 2005 16:09:55 -0800
Subject: [openib-general] [PATCH] [CM] remove outdated peer-to-peer handling
	code
Message-ID: <20050216160955.1208ee9a.mshefty@ichips.intel.com>

This patch removes commented out and now outdated code for handling
peer-to-peer connection requests.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1808)
+++ infiniband/core/cm.c	(working copy)
@@ -985,37 +985,6 @@ static void cm_process_work(struct cm_id
 		ib_destroy_cm_id(&cm_id_priv->id);	
 }
 
-static int cm_peer_req_handler(struct cm_work *work)
-{
-	/* todo: fix peer-to-peer
-	spin_lock_irqsave(&cm.lock, flags);
-	if (cm_is_active_peer(work->port->ca_guid,
-				req_msg->local_ca_guid,
-				cur_cm_id_priv->local_qpn,
-				cm_req_get_local_qpn(req_msg))) {
-		spin_unlock_irqrestore(&cm.lock, flags);
-		ret = 1;
-		goto out;
-	}
-	atomic_inc(&cur_cm_id_priv->refcount);
-	cur_cm_id_priv->id.state = IB_CM_REQ_RCVD;
-	rb_erase(&cur_cm_id_priv->service_node, &cm.service_table);
-	cur_cm_id_priv->peer_to_peer = 0;
-	cur_cm_id_priv->remote_ca_guid = req_msg->local_ca_guid;
-	cur_cm_id_priv->id.remote_id = req_msg->local_comm_id;
-	cm_insert_remote_id(cur_cm_id_priv);
-	spin_unlock_irqrestore(&cm.lock, flags);
-
-	ib_cancel_mad(work->port->mad_agent,
-		(unsigned long) cur_cm_id_priv->msg);
-	ib_destroy_cm_id(&cm_id_priv->id);
-	cm_id_priv = cur_cm_id_priv;
-out:
-	return ret;
-	*/
-	return -ENOENT;	/* Does not match a peer-to-peer request */
-}
-
 static int cm_req_handler(struct cm_work *work)
 {
 	struct ib_cm_id *cm_id;
@@ -1026,10 +995,7 @@ static int cm_req_handler(struct cm_work
 
 	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
 
-	/* Check for peer-to-peer connection. */
-	ret = cm_peer_req_handler(work);
-	if (ret != -ENOENT)
-		return ret;
+	/* todo: Check for peer-to-peer connection. */
 
 	cm_id = ib_create_cm_id(NULL, NULL);
 	if (IS_ERR(cm_id))
@@ -1325,14 +1291,7 @@ static int cm_rep_handler(struct cm_work
 	cm_id_priv->sq_psn = cm_rep_get_starting_psn(rep_msg);
 	cm_id_priv->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg);
 
-	/* todo: handle peer_to_peer
-	if (cm_id_priv->peer_to_peer) {
-		cm_id_priv->peer_to_peer = 0;
-		spin_lock_irqsave(&cm.lock, flags);
-		rb_erase(&cm_id_priv->service_node, &cm.service_table);
-		spin_unlock_irqrestore(&cm.lock, flags);
-	}
-	*/
+	/* todo: handle peer_to_peer */
 
 	wr_id = (unsigned long) cm_id_priv->msg;
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);


From mshefty at ichips.intel.com  Wed Feb 16 16:18:28 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 16 Feb 2005 16:18:28 -0800
Subject: [openib-general] [PATCH] [CM] add check before trying to free
	received MAD
Message-ID: <20050216161828.1bee6cda.mshefty@ichips.intel.com>

This patch adds a check to see if a mad_recv_wc exists before a call is
made to free it.  This fixes a crash after a call is made to
ib_cm_establish.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1809)
+++ infiniband/core/cm.c	(working copy)
@@ -579,7 +579,8 @@ static struct cm_work * cm_dequeue_work(
 
 static void cm_free_work(struct cm_work *work)
 {
-	ib_free_recv_mad(work->mad_recv_wc);
+	if (work->mad_recv_wc)
+		ib_free_recv_mad(work->mad_recv_wc);
 	kfree(work);
 }
 
@@ -2528,6 +2529,7 @@ int ib_cm_establish(struct ib_cm_id *cm_
 	INIT_WORK(&work->work, cm_work_handler, work);
 	work->local_id = cm_id->local_id;
 	work->remote_id = cm_id->remote_id;
+	work->mad_recv_wc = NULL;
 	work->cm_event.event = IB_CM_USER_ESTABLISHED;
 	queue_work(cm.wq, &work->work);
 out:


From robert.j.woodruff at intel.com  Wed Feb 16 15:34:24 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Wed, 16 Feb 2005 15:34:24 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
Message-ID: <1AC79F16F5C5284499BB9591B33D6F000397B200@orsmsx408>



>Kanevsky, Arkady wrote, 

>If Open IB wants to go with LGPL license it is fine with us.
>We will need to take a voit on DAT Collaborative onit also.
>But from what I see on reflector LGPL license is outside the current
>bylaws of
>Open IB and there is a discussion on it going on.

Good point.

>SO until this issue is resolved on Open IB we will add GPL license to
>uDAPL (and kDAPL)
>and will work on getting it ready for submission to Open IB.
>Once the license the GPL license is approved by DAT we can move the dev
>work on Open IB SF.
>If Arlin is available he can start the work now.
>Lets first identify what areas require changes. 

I think Arlin should be able to start pretty soon. He has a couple
of other things on his plate that he is finishing up, but should
be free'd up in a week or 2. In the mean time, he can start to take
a look at what needs to be done, etc. 

I would think that initially, the first changes are to develop a
provider that calls
the new gen2 verbs APIs rather than the gen1 APIs. This should be 
a pretty straight forward port of what currently exists in your
sourceforge project. 
There is also support for shared memory that is needed for Oracle, and I
am
not sure if that exists in any of the open source versions of DAPL.

Arlin, feel free to chime in if there is anything that I missed that
would
need to be done to make uDAPL run on openib.org.

woody



From xma at us.ibm.com  Wed Feb 16 15:51:58 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Wed, 16 Feb 2005 16:51:58 -0700
Subject: [openib-general] [PATCH][IPOIB] bring interface down panic 
Message-ID: <OF61DB1A86.7CB133C1-ON87256FAA.00820897-88256FAA.008318D5@us.ibm.com>

I found a panic while testing a lock patch. 

In ipoib_mcast_dev_flush(), list_for_each_entry() should be 
list_for_each_entry_safe().

Here is the patch.

diff -urN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-down/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-04 
16:05:13.000000000 -0800
+++ infiniband-down/ulp/ipoib/ipoib_multicast.c 2005-02-16 
15:49:42.278230248 -0800
@@ -790,7 +790,7 @@
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
-       list_for_each_entry(mcast, &remove_list, list) {
+       list_for_each_entry_safe(mcast, tmcast, &remove_list, list) {
                ipoib_mcast_leave(dev, mcast);
                ipoib_mcast_free(mcast);
        }





Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050216/7fc88a56/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-down.patch
Type: application/octet-stream
Size: 507 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050216/7fc88a56/attachment.obj>

From mshefty at ichips.intel.com  Wed Feb 16 16:50:31 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 16 Feb 2005 16:50:31 -0800
Subject: [openib-general] [PATCH] [CM] add timewait support
Message-ID: <20050216165031.6fdbdb5e.mshefty@ichips.intel.com>

This patch provides initial timewait handling in the CM.

A connection will remain in the timewait state for a period of time
equal to the local_ack_timeout.  After a connection exits the
timewait state, the user is notified of a TIMEWAIT_EXIT event,
assuming that the user has not destroyed their cm_id.

If a user destroys their cm_id before a connection has exited the
timewait state, the user will not be notified, but the connection
will still remain in the timewait state until it expires.

Note that checks for stale connections are not yet implemented.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>


Index: infiniband/include/ib_cm.h
===================================================================
--- infiniband/include/ib_cm.h	(revision 1796)
+++ infiniband/include/ib_cm.h	(working copy)
@@ -76,6 +76,7 @@ enum ib_cm_event_type {
 	IB_CM_DREQ_ERROR,
 	IB_CM_DREQ_RECEIVED,
 	IB_CM_DREP_RECEIVED,
+	IB_CM_TIMEWAIT_EXIT,
 	IB_CM_MRA_RECEIVED,
 	IB_CM_REJ_RECEIVED,
 	IB_CM_LAP_ERROR,
Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1810)
+++ infiniband/core/cm.c	(working copy)
@@ -100,24 +100,32 @@ struct cm_work {
 	struct list_head list;
 	struct cm_port *port;
 	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
-	u32 local_id;				/* Established */
+	u32 local_id;				/* Established / timewait */
 	u32 remote_id;
 	struct ib_cm_event cm_event;
 	struct ib_sa_path_rec path[0];
 };
 
+struct cm_timewait_info {
+	struct cm_work work;			/* Must be first. */
+	/* struct rb_node remote_qp_node; */
+	struct rb_node remote_id_node;
+	u64 remote_ca_guid;
+	u32 remote_qpn;
+	u8 inserted_remote_id;
+};
+
 struct cm_id_private {
 	struct ib_cm_id	id;
 
 	struct rb_node service_node;
-	struct rb_node remote_qp_node;
-	struct rb_node remote_id_node;
 	struct rb_node sidr_id_node;
 	spinlock_t lock;
 	wait_queue_head_t wait;
 	atomic_t refcount;
 
 	struct cm_msg *msg;
+	struct cm_timewait_info *timewait_info;
 	/* todo: use alternate port on send failure */
 	struct cm_av av;
 	struct cm_av alt_av;
@@ -126,11 +134,9 @@ struct cm_id_private {
 	u32 remote_qpn;
 	u32 sq_psn;
 	u32 rq_psn;
-	u64 remote_ca_guid;
 	int timeout_ms;
 	enum ib_mtu path_mtu;
 	u8 max_cm_retries;
-	u8 passive;
 	u8 peer_to_peer;
 	u8 responder_resources;
 	u8 initiator_depth;
@@ -142,6 +148,8 @@ struct cm_id_private {
 	atomic_t work_count;
 };
 
+static void cm_work_handler(void *data);
+
 static inline void cm_deref_id(struct cm_id_private *cm_id_priv)
 {
 	if (atomic_dec_and_test(&cm_id_priv->refcount))
@@ -361,32 +369,33 @@ static struct cm_id_private * cm_find_li
 	return NULL;
 }
 
-static struct cm_id_private * cm_insert_remote_id(struct cm_id_private
-						  *cm_id_priv)
+static struct cm_timewait_info * cm_insert_remote_id(struct cm_timewait_info
+						     *timewait_info)
 {
 	struct rb_node **link = &cm.remote_id_table.rb_node;
 	struct rb_node *parent = NULL;
-	struct cm_id_private *cur_cm_id_priv;
-	u64 remote_ca_guid = cm_id_priv->remote_ca_guid;
-	u32 remote_id = cm_id_priv->id.remote_id;
+	struct cm_timewait_info *cur_timewait_info;
+	u64 remote_ca_guid = timewait_info->remote_ca_guid;
+	u32 remote_id = timewait_info->work.remote_id;
 
 	while (*link) {
 		parent = *link;
-		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
-					  remote_id_node);
-		if (remote_id < cur_cm_id_priv->id.remote_id)
+		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
+					     remote_id_node);
+		if (remote_id < cur_timewait_info->work.remote_id)
 			link = &(*link)->rb_left;
-		else if (remote_id > cur_cm_id_priv->id.remote_id)
+		else if (remote_id > cur_timewait_info->work.remote_id)
 			link = &(*link)->rb_right;
-		else if (remote_ca_guid < cur_cm_id_priv->remote_ca_guid)
+		else if (remote_ca_guid < cur_timewait_info->remote_ca_guid)
 			link = &(*link)->rb_left;
-		else if (remote_ca_guid > cur_cm_id_priv->remote_ca_guid)
+		else if (remote_ca_guid > cur_timewait_info->remote_ca_guid)
 			link = &(*link)->rb_right;
 		else
-			return cur_cm_id_priv;
+			return cur_timewait_info;
 	}
-	rb_link_node(&cm_id_priv->remote_id_node, parent, link);
-	rb_insert_color(&cm_id_priv->remote_id_node, &cm.remote_id_table);
+	timewait_info->inserted_remote_id = 1;
+	rb_link_node(&timewait_info->remote_id_node, parent, link);
+	rb_insert_color(&timewait_info->remote_id_node, &cm.remote_id_table);
 	return NULL;
 }
 
@@ -415,13 +424,13 @@ static struct cm_id_private * cm_find_id
 }
 */
 
-static void cm_remove_remote_id(struct cm_id_private *cm_id_priv)
+static void cm_remove_remote_id(struct cm_timewait_info *timewait_info)
 {
 	unsigned long flags;
 
-	cm_id_priv->passive = 0;
+	timewait_info->inserted_remote_id = 0;
 	spin_lock_irqsave(&cm.lock, flags);
-	rb_erase(&cm_id_priv->remote_id_node, &cm.remote_id_table);
+	rb_erase(&timewait_info->remote_id_node, &cm.remote_id_table);
 	spin_unlock_irqrestore(&cm.lock, flags);
 }
 
@@ -584,6 +593,72 @@ static void cm_free_work(struct cm_work 
 	kfree(work);
 }
 
+static inline int cm_convert_to_ms(int iba_time)
+{
+	/* approximate conversion to ms from 4.096us x 2^iba_time */
+	return 1 << max(iba_time - 8, 0);
+}
+
+static void cm_cleanup_timewait(struct cm_timewait_info *timewait_info)
+{
+	if (timewait_info->inserted_remote_id)
+		cm_remove_remote_id(timewait_info);
+
+	/* todo: stale connection handling
+	if (timewait_info->inserted_remote_qpn)
+		cm_remove_remote_qpn(timewait_info);
+	*/
+}
+
+static struct cm_timewait_info * cm_create_timewait_info(u32 local_id,
+							 u32 remote_id,
+							 u64 remote_ca_guid,
+							 u32 remote_qpn)
+{
+	struct cm_timewait_info *timewait_info;
+
+	timewait_info = kmalloc(sizeof *timewait_info, GFP_KERNEL);
+	if (!timewait_info)
+		return ERR_PTR(-ENOMEM);
+	memset(timewait_info, 0, sizeof *timewait_info);
+
+	timewait_info->work.local_id = local_id;
+	timewait_info->work.remote_id = remote_id;
+	timewait_info->remote_ca_guid = remote_ca_guid;
+	timewait_info->remote_qpn = remote_qpn;
+
+	INIT_WORK(&timewait_info->work.work, cm_work_handler,
+		  &timewait_info->work);
+	timewait_info->work.cm_event.event = IB_CM_TIMEWAIT_EXIT;
+	return timewait_info;
+}
+
+static void cm_enter_timewait(struct cm_id_private *cm_id_priv)
+{
+	int wait_time;
+
+	/*
+	 * The cm_id could be destroyed by the user before we exit timewait.
+	 * To protect against this, we search for the cm_id after exiting
+	 * timewait before notifying the user that we've exited timewait.
+	 */
+	cm_id_priv->id.state = IB_CM_TIMEWAIT;
+	wait_time = cm_convert_to_ms(cm_id_priv->local_ack_timeout);
+	queue_delayed_work(cm.wq, &cm_id_priv->timewait_info->work.work,
+			   msecs_to_jiffies(wait_time));
+	cm_id_priv->timewait_info = NULL;
+}
+
+static void cm_reset_to_idle(struct cm_id_private *cm_id_priv)
+{
+	cm_id_priv->id.state = IB_CM_IDLE;
+	if (cm_id_priv->timewait_info) {
+		cm_cleanup_timewait(cm_id_priv->timewait_info);
+		kfree(cm_id_priv->timewait_info);
+		cm_id_priv->timewait_info = NULL;
+	}
+}
+
 int ib_destroy_cm_id(struct ib_cm_id *cm_id)
 {
 	struct cm_id_private *cm_id_priv;
@@ -630,10 +705,8 @@ retest:
 		ib_send_cm_dreq(cm_id, NULL, 0);
 		goto retest;
 	case IB_CM_DREQ_SENT:
-		cm_id->state = IB_CM_TIMEWAIT;
+		cm_enter_timewait(cm_id_priv);
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-		if (cm_id_priv->passive)
-			cm_remove_remote_id(cm_id_priv);
 		break;
 	case IB_CM_DREQ_RCVD:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
@@ -681,12 +754,6 @@ int ib_cm_listen(struct ib_cm_id *cm_id,
 }
 EXPORT_SYMBOL(ib_cm_listen);
 
-static inline int cm_convert_to_ms(int iba_time)
-{
-	/* approximate conversion to ms from 4.096us x 2^iba_time */
-	return 1 << max(iba_time - 8, 0);
-}
-
 static void cm_format_mad_hdr(struct ib_mad_hdr *hdr,
 			      struct cm_id_private *cm_id_priv,
 			      enum cm_msg_attr_id attr_id,
@@ -989,7 +1056,7 @@ static void cm_process_work(struct cm_id
 static int cm_req_handler(struct cm_work *work)
 {
 	struct ib_cm_id *cm_id;
-	struct cm_id_private *cm_id_priv, *listen_cm_id_priv, *cur_cm_id_priv;
+	struct cm_id_private *cm_id_priv, *listen_cm_id_priv;
 	struct cm_req_msg *req_msg;
 	unsigned long flags;
 	int ret;
@@ -1001,20 +1068,34 @@ static int cm_req_handler(struct cm_work
 	cm_id = ib_create_cm_id(NULL, NULL);
 	if (IS_ERR(cm_id))
 		return PTR_ERR(cm_id);
-	
+
 	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
-	cm_id_priv->remote_ca_guid = req_msg->local_ca_guid;
 	cm_id_priv->id.remote_id = req_msg->local_comm_id;
-	cm_id_priv->passive = 1;
+	cm_id_priv->timewait_info = cm_create_timewait_info(
+						cm_id_priv->id.local_id,
+						cm_id_priv->id.remote_id,
+						req_msg->local_ca_guid,
+						cm_req_get_local_qpn(req_msg));
+	if (IS_ERR(cm_id_priv->timewait_info)) {
+		ret = PTR_ERR(cm_id_priv->timewait_info);
+		goto error1;
+	}
 
 	spin_lock_irqsave(&cm.lock, flags);
 	/* Check for duplicate REQ. */
-	cur_cm_id_priv = cm_insert_remote_id(cm_id_priv);
-	if (cur_cm_id_priv) {
+	if (cm_insert_remote_id(cm_id_priv->timewait_info)) {
 		spin_unlock_irqrestore(&cm.lock, flags);
 		ret = -EINVAL;
-		goto error1;
+		goto error2;
 	}
+	/* todo: Check for a stale connection.
+	if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		todo: reject as stale
+		ret = -EINVAL;
+		goto error2;
+	}
+	*/
 	/* Find matching listen request. */
 	listen_cm_id_priv = cm_find_listen(req_msg->service_id);
 	if (!listen_cm_id_priv) {
@@ -1065,7 +1146,8 @@ static int cm_req_handler(struct cm_work
 error3:
 	cm_deref_id(listen_cm_id_priv);
 error2:
-	cm_remove_remote_id(cm_id_priv);
+	cm_cleanup_timewait(cm_id_priv->timewait_info);
+	kfree(cm_id_priv->timewait_info);
 error1:
 	ib_destroy_cm_id(&cm_id_priv->id);
 	return ret;
@@ -1258,6 +1340,7 @@ static void cm_format_rep_event(struct c
 static int cm_rep_handler(struct cm_work *work)
 {
 	struct cm_id_private *cm_id_priv;
+	struct cm_timewait_info *timewait_info;
 	struct cm_rep_msg *rep_msg;
 	unsigned long flags;
 	u64 wr_id;
@@ -1268,6 +1351,14 @@ static int cm_rep_handler(struct cm_work
 	if (!cm_id_priv)
 		return -EINVAL;
 
+	timewait_info = cm_create_timewait_info(cm_id_priv->id.local_id,
+						rep_msg->local_comm_id,
+						rep_msg->local_ca_guid,
+						cm_rep_get_local_qpn(rep_msg));
+	if (IS_ERR(timewait_info)) {
+		ret = PTR_ERR(timewait_info);
+		goto error1;
+	}
 	cm_format_rep_event(work);
 
 	spin_lock_irqsave(&cm_id_priv->lock, flags);
@@ -1278,19 +1369,21 @@ static int cm_rep_handler(struct cm_work
 	case IB_CM_ESTABLISHED:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 		cm_resend_rtu(cm_id_priv);
-		goto out;
+		ret = -EINVAL;
+		goto error2;
 	default:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-		goto out;
+		ret = -EINVAL;
+		goto error2;
 	}
 	cm_id_priv->id.state = IB_CM_REP_RCVD;
 	cm_id_priv->id.remote_id = rep_msg->local_comm_id;
-	cm_id_priv->remote_ca_guid = rep_msg->local_ca_guid;
 	cm_id_priv->remote_qpn = cm_rep_get_local_qpn(rep_msg);
 	cm_id_priv->initiator_depth = rep_msg->resp_resources;
 	cm_id_priv->responder_resources = rep_msg->initiator_depth;
 	cm_id_priv->sq_psn = cm_rep_get_starting_psn(rep_msg);
 	cm_id_priv->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg);
+	cm_id_priv->timewait_info = timewait_info;
 
 	/* todo: handle peer_to_peer */
 
@@ -1306,9 +1399,11 @@ static int cm_rep_handler(struct cm_work
 	else
 		cm_deref_id(cm_id_priv);
 	return 0;
-out:
+error2:
+	kfree(timewait_info);
+error1:
 	cm_deref_id(cm_id_priv);
-	return -EINVAL;
+	return ret;
 }
 
 static int cm_establish_handler(struct cm_work *work)
@@ -1438,11 +1533,8 @@ int ib_send_cm_dreq(struct ib_cm_id *cm_
 		cm_id->state = IB_CM_DREQ_SENT;
 		cm_id_priv->msg = msg;
 	} else
-		cm_id->state = IB_CM_TIMEWAIT;
+		cm_enter_timewait(cm_id_priv);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-
-	if (ret && cm_id_priv->passive)
-		cm_remove_remote_id(cm_id_priv);
 out:
 	if (!msg_ret && ret)
 		cm_free_msg(cm_id_priv->msg);
@@ -1510,12 +1602,8 @@ int ib_send_cm_drep(struct ib_cm_id *cm_
 	ret = msg_ret ? msg_ret :
 		ib_post_send_mad(cm_id_priv->av.port->mad_agent,
 				 &msg->send_wr, &bad_send_wr);
-
-	cm_id->state = IB_CM_TIMEWAIT;
+	cm_enter_timewait(cm_id_priv);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-
-	if (cm_id_priv->passive)
-		cm_remove_remote_id(cm_id_priv);
 out:
 	if (!msg_ret && ret)
 		cm_free_msg(cm_id_priv->msg);
@@ -1595,7 +1683,7 @@ static int cm_drep_handler(struct cm_wor
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 		goto out;
 	}
-	cm_id_priv->id.state = IB_CM_TIMEWAIT;
+	cm_enter_timewait(cm_id_priv);
 
 	wr_id = (unsigned long) cm_id_priv->msg;
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
@@ -1604,8 +1692,6 @@ static int cm_drep_handler(struct cm_wor
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
 	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
-	if (cm_id_priv->passive)
-		cm_remove_remote_id(cm_id_priv);
 
 	if (ret)
 		cm_process_work(cm_id_priv, work);
@@ -1685,11 +1771,11 @@ int ib_send_cm_rej(struct ib_cm_id *cm_i
 	case IB_CM_MRA_REQ_SENT:
 	case IB_CM_REP_RCVD:
 	case IB_CM_MRA_REP_SENT:
-		cm_id->state = IB_CM_IDLE;
+		cm_reset_to_idle(cm_id_priv);
 		break;
 	case IB_CM_REP_SENT:
 	case IB_CM_MRA_REP_RCVD:
-		cm_id->state = IB_CM_TIMEWAIT;
+		cm_enter_timewait(cm_id_priv);
 		break;
 	default:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
@@ -1698,9 +1784,6 @@ int ib_send_cm_rej(struct ib_cm_id *cm_i
 	}
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	if (cm_id_priv->passive)
-		cm_remove_remote_id(cm_id_priv);
-
 	ret = msg_ret ? msg_ret :
 		ib_post_send_mad(cm_id_priv->av.port->mad_agent,
 				 &msg->send_wr, &bad_send_wr);
@@ -2083,6 +2166,43 @@ out:
 	return -EINVAL;
 }
 
+static int cm_timewait_handler(struct cm_work *work)
+{
+	struct cm_timewait_info *timewait_info;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	int ret;
+
+	timewait_info = (struct cm_timewait_info *)work;
+	cm_cleanup_timewait(timewait_info);
+
+	cm_id_priv = cm_acquire_id(timewait_info->work.local_id,
+				   timewait_info->work.remote_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_TIMEWAIT ||
+	    cm_id_priv->remote_qpn != timewait_info->remote_qpn) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_IDLE;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
 static void cm_format_sidr_req(struct cm_sidr_req_msg *sidr_req_msg,
 			       struct cm_id_private *cm_id_priv,
 			       struct ib_cm_sidr_req_param *param)
@@ -2357,15 +2477,15 @@ static void cm_process_send_error(struct
 
 	switch (cm_id_priv->id.state) {
 	case IB_CM_REQ_SENT:
-		cm_id_priv->id.state = IB_CM_IDLE;
+		cm_reset_to_idle(cm_id_priv);
 		cm_event.event = IB_CM_REQ_ERROR;
 		break;
 	case IB_CM_REP_SENT:
-		cm_id_priv->id.state = IB_CM_IDLE;
+		cm_reset_to_idle(cm_id_priv);
 		cm_event.event = IB_CM_REP_ERROR;
 		break;
 	case IB_CM_DREQ_SENT:
-		cm_id_priv->id.state = IB_CM_TIMEWAIT;
+		cm_enter_timewait(cm_id_priv);
 		cm_event.event = IB_CM_DREQ_ERROR;
 		break;
 	case IB_CM_SIDR_REQ_SENT:
@@ -2376,13 +2496,10 @@ static void cm_process_send_error(struct
 		goto discard;
 	}
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-	if (cm_id_priv->passive)
-		cm_remove_remote_id(cm_id_priv);
 	cm_event.param.send_status = wc_status;
 	cm_id_priv->id.cm_handler(&cm_id_priv->id, &cm_event);
 	cm_free_msg(msg);
 	return;
-
 discard:
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 	cm_free_msg(msg);
@@ -2478,6 +2595,9 @@ static void cm_work_handler(void *data)
 	case IB_CM_APR_RECEIVED:
 		ret = cm_apr_handler(work);
 		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		ret = cm_timewait_handler(work);
+		break;
 	default:
 		ret = -EINVAL;
 		break;


From mlleinin at hpcn.ca.sandia.gov  Tue Feb 15 22:58:19 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Tue, 15 Feb 2005 22:58:19 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <DC0618F6095F6745A5B480A6A53E91AC01AD642D@orsmsx405>
References: <DC0618F6095F6745A5B480A6A53E91AC01AD642D@orsmsx405>
Message-ID: <1108537099.7416.29.camel@localhost>

On Tue, 2005-02-15 at 17:32 -0800, Ryan, Jim wrote:
> Tom Duffy wrote:
> > On Mon, 2005-02-14 at 16:47 -0800, Woodruff, Robert J wrote:
> >>  Hi Arkady,
> >> 
> >> As I mentioned in the BOF, I have a person (Arlin Davis) that can
> >> help with developing a uDAPL provider for the openib.org verbs.
> >> After discussing it more
> >> with folks here, is seems to us that perhaps for the uDAPL user-mode
> >> library,  it be provided to openib.org under a dual BSD + LGPL
> >> library rather than a BSD + GPL since people normally want to use
> >> LGPL for libraries.
> > 
> > I think using an LGPL instead of GPL would require a change to the
> > openib.org bylaws.
> > 
> > Jim, is that right?
> > 
> > Honestly, if they are licensed BSD too, people can do whatever they
> > want.  BSD/GPL will be fine on these libraries, IMO.
> > 
> > -tduffy
> 
> Tom, good to hear from you. 
> 
> Yes, it would require a change in our bylaws which are very clear wrt
> GPL and BSD.
> 
> Please let me know if you'd like to discuss further. 
> 
  GPL/BSD sounds good to me.

	- Matt




From ardavis at ichips.intel.com  Wed Feb 16 17:40:44 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Wed, 16 Feb 2005 17:40:44 -0800
Subject: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop
In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F000397B200@orsmsx408>
References: <1AC79F16F5C5284499BB9591B33D6F000397B200@orsmsx408>
Message-ID: <20050216174044.1eb449f5.ardavis@ichips.intel.com>

On Wed, 16 Feb 2005 15:34:24 -0800
"Woodruff, Robert J" <robert.j.woodruff at intel.com> wrote:

> 
> 
> >Kanevsky, Arkady wrote, 
> 
> >If Open IB wants to go with LGPL license it is fine with us.
> >We will need to take a voit on DAT Collaborative onit also.
> >But from what I see on reflector LGPL license is outside the current
> >bylaws of
> >Open IB and there is a discussion on it going on.
> 
> Good point.
> 
> >SO until this issue is resolved on Open IB we will add GPL license to
> >uDAPL (and kDAPL)
> >and will work on getting it ready for submission to Open IB.
> >Once the license the GPL license is approved by DAT we can move the dev
> >work on Open IB SF.
> >If Arlin is available he can start the work now.
> >Lets first identify what areas require changes. 
> 
> I think Arlin should be able to start pretty soon. He has a couple
> of other things on his plate that he is finishing up, but should
> be free'd up in a week or 2. In the mean time, he can start to take
> a look at what needs to be done, etc. 

Yes, I will free up in a couple of weeks to actually start the port. 
Based on a quick look at the user mode verbs and CM the porting effort
from IB Gen1 should be relatively straight forward. Missing is user mode
SA support (with RMPP) for ATS and Path Record queries. 

Not sure what has been discussed so far on this thread but we also 
need some discussion regarding name services and the possibility of
a kernel agent to support ATS registration and deregistration during 
IPoIB configuring (events), IP to PathRecord resolution (IPoIB arp
services), and support for shared memory. SDP and other ULP's could 
certainly make make use of this agent to resolve IP to GID mappings.

-arlin
 
> 
> I would think that initially, the first changes are to develop a
> provider that calls
> the new gen2 verbs APIs rather than the gen1 APIs. This should be 
> a pretty straight forward port of what currently exists in your
> sourceforge project. 
> There is also support for shared memory that is needed for Oracle, and I
> am
> not sure if that exists in any of the open source versions of DAPL.
> 
> Arlin, feel free to chime in if there is anything that I missed that
> would
> need to be done to make uDAPL run on openib.org.
> 
> woody
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From mlleinin at hpcn.ca.sandia.gov  Wed Feb 16 16:50:38 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Wed, 16 Feb 2005 16:50:38 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
In-Reply-To: <Pine.LNX.4.61.0502160941030.5704@jlentini-linux.nane.netapp.com>
References: <DOEHKBHAGBADBEKBCHKCMEBFDEAA.bill@thompson.name>
	<1108367138.3078.15.camel@wynux.LibreSolutions>
	<1108374321.18616.94.camel@localhost>
	<Pine.LNX.4.61.0502160941030.5704@jlentini-linux.nane.netapp.com>
Message-ID: <1108601438.3325.9.camel@localhost>

On Wed, 2005-02-16 at 09:49 -0500, James Lentini wrote:
> 
> On Mon, 14 Feb 2005, Matt Leininger wrote:
> 
> >  Several developers have volunteered to work on uDAPL.  The DAT
> > collaborative is working to get a GPL/BSD version of uDAPL to OpenIB.
> > James and Arkady from NetApp gave a DAT talk.  They mentioned needing 1
> > month to get the GPL code, and another month or two to fork the code
> > base.  I'm not sure why 2-3 months is needed to do this.  Perhaps James
> > and/or Arkady can comment.
> 
> My 2-3 month estimate was based on 1 month for the DAT Collaborative 
> to discuss and vote on licensing the code under the GPL and 1-2 months 
> to port and test on OpenIB Gen 2.

   ok that sounds better.  We could follow Woody's suggestion of
starting uDAPL development on OpenIB with the BSD code and then dual
license it GPL/BSD after the DAT Collaborative gets the licensing issues
figure out.  
> 
> > I think we need to see the GPL/BSD uDAPL code for OpenIB in the next 
> > couple of weeks.  Better to put the code in the open so everyone can 
> > see what changes are required.  Same goes for kDAPL.  Was the 
> > u/kDAPL implementation for OpenIB going to live at openib.org or 
> > stay under cvs at the sourceforge dapl project?
> 
> We are proposing to maintain the code in both locations. The OpenIB 
> specific implementation would be on openib.org while a 
> platform/transport independent implementation would continue on 
> our SourceForge site.
> 
   That's reasonable.  

   Thanks,

	- Matt
   



From mshefty at ichips.intel.com  Wed Feb 16 17:25:06 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 16 Feb 2005 17:25:06 -0800
Subject: [openib-general] [RFC] CM reject handling and redirection
Message-ID: <4213F272.2000805@ichips.intel.com>

I've started looking at the handling received rejects in the CM.  Two 
return codes of note:

* code 24 - Port and CM Redirection
* code 25 - Port Redirection

For the first case, it appears that the proper way to handle it is to 
simply expose the rejection to the user.  The user would need to issue 
a second REQ using a new path record.  Does this seem reasonable to 
everyone?

For the second case, the issue becomes more complex.  The CM is 
instructed to send to the port associated with the original REQ, but 
the connection is established to a separate port.

Currently, the CM sends to the port specified by the path record, 
ib_cm_req_param::primary_path, that will be used by the connection.  A 
port redirection REJ message changes this behavior.

After the REJ message is given to the user, a subsequent call to send a 
REQ will result in the CM sending the message to the new port.  For the 
CM to be able to send the REQ to the same port as the original REQ, it 
would need to store yet more state information regarding previous 
connection request attempts.  Since it's possible that the user could 
issue an unrelated REQ after receiving the REJ.

Should the CM try to maintain this state information, or should the API 
be expanded to allow the user to specify where to send the CM messages, 
or should we just ignore this reject code for now?   The needed state 
information would probably consist of: type of last message received, 
last REJ reason, last DGID, service ID of last REQ.  An expanded API 
could let the CM destination be optional, with a default to send based 
on the current path.

It's also not clear to me what effect this reject code has with respect 
to port failover.  Thoughts?

- Sean


From beng at isilon.com  Wed Feb 16 18:37:26 2005
From: beng at isilon.com (Brian Eng)
Date: Wed, 16 Feb 2005 18:37:26 -0800
Subject: [openib-general] Incorrect endian in GUID comparison/SM master
	selection
Message-ID: <1108607846.27002.113.camel@bengbsd.isilon.com>

Hello,

Per the IBA spec, the selection of a master SM is determined by the
GUID (lowest wins).  When OpenSM does its comparison, it does not adjust
for the endian of the host.  For our x86 systems, this means the
comparison is based on the byte-swapped value of the GUID.  It looks
like the patch below is in order.  Is that correct?

--- osm_sminfo_rcv.c    3 Feb 2005 19:33:39 -0000       1.2.2.3
+++ osm_sminfo_rcv.c    16 Feb 2005 23:13:39 -0000
@@ -173,7 +173,8 @@ __osm_sminfo_rcv_remote_sm_is_higher (
     if( ib_sminfo_get_priority( p_remote_sm ) ==
         p_rcv->p_subn->opt.sm_priority )
     {
-      if( p_remote_sm->guid < p_rcv->p_subn->sm_port_guid )
+      if( cl_ntoh64( p_remote_sm->guid ) <
+          cl_ntoh64( p_rcv->p_subn->sm_port_guid ) )
       {
         return( TRUE );
       }





From roland at topspin.com  Wed Feb 16 19:50:30 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 16 Feb 2005 19:50:30 -0800
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <4213F272.2000805@ichips.intel.com> (Sean Hefty's message of
	"Wed, 16 Feb 2005 17:25:06 -0800")
References: <4213F272.2000805@ichips.intel.com>
Message-ID: <52acq3x2t5.fsf@topspin.com>

I think the simplest solution the solution most in keeping with the
design of the CM is to always expose the rejection and the contents of
the REJ to the consumer.  As you say, that makes the code 24 (port &
CM redirection) case pretty easy to handle.

Code 25 (port redirection only) raises a separate issue that I think
is orthogonal to REJ handling.  The CM spec always allows the CM to
run on a different port from the destination of a connection so if we
want to implement the whole spec we do need a way for the consumer to
specify a primary and alternate path for the connection itself as well
as a CM path.

However, I don't know of any real applications that run the CM on a
different port from the port being connected to, so I don't know that
we need to support this (but once we do, REJ code 25 is easy to handle).

As you say, separating the CM path from the connection path does raise
the question of what to do for failover and APM.  As far as I can tell
this is an area that the IB spec is silent on and I'm not sure that
anyone ever came up with a good answer.

 - R.


From eitan at mellanox.co.il  Wed Feb 16 22:04:11 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 17 Feb 2005 08:04:11 +0200
Subject: [openib-general] Incorrect endian in GUID comparison/SM maste
	r selection
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEE4D@mtlex01.yok.mtl.com>

Thanks Brian

> 
> Per the IBA spec, the selection of a master SM is determined by the
> GUID (lowest wins).  When OpenSM does its comparison, it does not adjust
> for the endian of the host.  For our x86 systems, this means the
> comparison is based on the byte-swapped value of the GUID.  It looks
> like the patch below is in order.  Is that correct?
> 
> --- osm_sminfo_rcv.c    3 Feb 2005 19:33:39 -0000       1.2.2.3
> +++ osm_sminfo_rcv.c    16 Feb 2005 23:13:39 -0000
> @@ -173,7 +173,8 @@ __osm_sminfo_rcv_remote_sm_is_higher (
>      if( ib_sminfo_get_priority( p_remote_sm ) ==
>          p_rcv->p_subn->opt.sm_priority )
>      {
> -      if( p_remote_sm->guid < p_rcv->p_subn->sm_port_guid )
> +      if( cl_ntoh64( p_remote_sm->guid ) <
> +          cl_ntoh64( p_rcv->p_subn->sm_port_guid ) )
>        {
>          return( TRUE );
>        }
> 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/9f294420/attachment.html>

From shaharf at voltaire.com  Thu Feb 17 05:04:39 2005
From: shaharf at voltaire.com (shaharf)
Date: Thu, 17 Feb 2005 15:04:39 +0200
Subject: [openib-general] question on opensm error
Message-ID: <D4F8F0B3820E754C887699BEF26A89405A945D@taurus.voltaire.com>

Hi,

> > There is a sys fail red light on the CPU on the 96-port switch that
the
> > opensm host attaches to.
> >
> > What's weird is none of the ib admin tools found anything.
ibnetdiscover
> > happily walked the whole subnet. The only problem was that opensm
would
> > not run, but the errors were unclear. So many things appeared to be
> > working that it did not occur to me to walk over and look at the
switch.
> > Stupid of me.
> 
> Still not 100% clear on the failure mode. I don't know what the sys
fail
> light on the CPU means. It may mean that things partially work. By
that,
> I mean the CPU might crash but the IB chips continue to function based
> on their current setup. It would depend on the split of functionality
> between the CPU and the IB chip firmware (which may depend on vendor).
> 
> If you were able to walk the subnet with the (SMP based) diags, the SM
> port had to be at least in init (ibstat/ibstatus).
> 
> The "keys" are what was the failure mode so we can see how this can be
> detected better in the future, and what caused the switch CPU to crash
> in the first place.
> 
> -- Hal
> 

I totally agree with Hal. The switch's CPU error is not the bug that is
in our concern. We should handle it is just as a failure of a device,
and we should be able to either overcome such failure or at least be
able to diagnose the error.
If you are able to reproduce the situation, please do it while the SM is
running with -V flag (full verbosity) and send the osm log file
(/tmp/osm.log) to the list. This will help us understand what is the
opensm problem. The output of the ibnetdiscover may help too.

Thanks,
	Shahar
	


From mst at mellanox.co.il  Thu Feb 17 06:03:59 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 17 Feb 2005 16:03:59 +0200
Subject: [openib-general] Re: RFC on SDP checkin
In-Reply-To: <20050210190904.C11665@topspin.com>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>
Message-ID: <20050217140359.GT28446@mellanox.co.il>


Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: RFC on SDP checkin
> Possible Issues
> 
>   
> [...] 
> 
>   - Memory locking for AIO requires a call to do_mlock() which is not a
>     kernel exported function, the method for calling the function is not
>     standard.

Libor, in my eyes this is the biggest issue with this gen2 sdp code.
In short, I dont think using do_mlock is a good idea.
Even if do_mlock *would* be exported, using this call to
do user page locking has other limitations:

- number of locked pages is very restricted, if its not,
  user can easily DOS the box
- fork() is broken in many ways
- user is forbidden from doing mlock/munlock

This may work well enough for things like MPI where
software may be rewritten for infiniband use, but SDP
shall provide migration path for legacy applications without
code changes.

I would like to suggest, once again, that the standard get_user_pages
call is used, instead. It is true that virtual mappings may change,
but it is easy to check that once the transaction is complete,
and perform copy if this happends.

Mellanox implemented and benchmarked this on Gen1 with good results.

If you are interested, I may post the design document for how this
works for us, or I can try and find some time to build a patch.
However, the second option wont happend till the end of
the month, and the patch is likely to be quite big.


-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Thu Feb 17 06:06:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 09:06:39 -0500
Subject: [openib-general] [RFC] Diagnostic tree structure
Message-ID: <1108649199.4645.213.camel@localhost.localdomain>

Hi,

The current userspace diagnostics tree structure has host and net
subdirectories. The distinction between the two is blurring so we would
like to flatten the tree and just have all the tools under diags. Any
objections ?

Thanks.

-- Hal



From halr at voltaire.com  Thu Feb 17 07:23:46 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 10:23:46 -0500
Subject: [openib-general] Incorrect endian in GUID comparison/SM master
	selection
In-Reply-To: <1108607846.27002.113.camel@bengbsd.isilon.com>
References: <1108607846.27002.113.camel@bengbsd.isilon.com>
Message-ID: <1108653826.4761.8.camel@localhost.localdomain>

Hi Brian,

On Wed, 2005-02-16 at 21:37, Brian Eng wrote: 
> Per the IBA spec, the selection of a master SM is determined by the
> GUID (lowest wins).  When OpenSM does its comparison, it does not adjust
> for the endian of the host.  For our x86 systems, this means the
> comparison is based on the byte-swapped value of the GUID.  It looks
> like the patch below is in order.  Is that correct?
> 
> -- osm_sminfo_rcv.c    3 Feb 2005 19:33:39 -0000       1.2.2.3
> +++ osm_sminfo_rcv.c    16 Feb 2005 23:13:39 -0000
> @@ -173,7 +173,8 @@ __osm_sminfo_rcv_remote_sm_is_higher (
>      if( ib_sminfo_get_priority( p_remote_sm ) ==
>          p_rcv->p_subn->opt.sm_priority )
>      {
> -      if( p_remote_sm->guid < p_rcv->p_subn->sm_port_guid )
> +      if( cl_ntoh64( p_remote_sm->guid ) <
> +          cl_ntoh64( p_rcv->p_subn->sm_port_guid ) )
>        {
>          return( TRUE );
>        }

Thanks. Applied. I also added a fix to a log message when displaying the
GUID.

-- Hal



From halr at voltaire.com  Thu Feb 17 07:28:08 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 10:28:08 -0500
Subject: [openib-general] Re: ib_mthca 0000:04:00.0: CQ overrun on CQN
	00000082, [patchlet] for opensm --guid
In-Reply-To: <20050210162236.GA21097@aon.at>
References: <20050210162236.GA21097@aon.at>
Message-ID: <1108654088.4761.17.camel@localhost.localdomain>

On Thu, 2005-02-10 at 11:22, Bernhard Fischer wrote: 
> small nitpick to opensm is attached. I'm not keen on implementing
> the menu stuff, just listing them is good enough for me. Please fix
> properly.

Thanks. Applied with minor format changes.

-- Hal



From mst at mellanox.co.il  Thu Feb 17 07:35:17 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 17 Feb 2005 17:35:17 +0200
Subject: [openib-general] SDP_CONN_LOCK
Message-ID: <20050217153517.GV28446@mellanox.co.il>

Hi, Libor!
Could you please explain what are the SDP_CONN_LOCK
and friends doing in SDP?

It seems they just implement exclusive access to socket,
but if so, why is a simple semaphore or mutex not used?

Thanks,

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb 17 07:46:43 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 17 Feb 2005 17:46:43 +0200
Subject: [openib-general] [PATCH] SDP headers using unsafe macros
Message-ID: <20050217154643.GW28446@mellanox.co.il>


Some SDP hdeaders defines unsafe macros, which may be
dangerous to use, e.g. inside an if or while statement.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1810)
+++ sdp_proto.h	(working copy)
@@ -585,15 +585,16 @@
 #define SDP_EXPECT(expr)
 #define SDP_CHECK_NULL(value, result)
 #else
-#define SDP_EXPECT(expr)                                             \
-{                                                                    \
-  if (!(expr)) {                                                     \
-    sdp_dbg_err("EXCEPT: Internal error check <%s> failed.", #expr); \
-  }                                                                  \
-} /* SDP_EXPECT */
+#define SDP_EXPECT(expr)                                                   \
+do {                                                                       \
+	if (!(expr)) {                                                     \
+		sdp_dbg_err("EXCEPT: Internal error check <%s> failed.",   \
+			#expr);                                            \
+	}                                                                  \
+} while (0) /* SDP_EXPECT */
 
 #define SDP_CHECK_NULL(value, result) \
-        if (NULL == (value)) return (result);
+        do { if (NULL == (value)) return (result); } while (0)
 #endif
 /* --------------------------------------------------------------------- */
 /* Inline functions.                                                     */
Index: sdp_conn.h
===================================================================
--- sdp_conn.h	(revision 1810)
+++ sdp_conn.h	(working copy)
@@ -102,7 +102,7 @@
  * transition one of the disconnect pending states to disconnect send
  */
 #define SDP_ST_PEND_2_SEND(conn) \
-        (conn)->state = ((conn)->state + 0x0100) & ~SDP_ST_MASK_DIS_PEND;
+        ((conn)->state = ((conn)->state + 0x0100) & ~SDP_ST_MASK_DIS_PEND)
 /*
  * internal connection structure
  */
@@ -130,14 +130,14 @@
  * connection handle within a socket.
  */
 #define SDP_GET_CONN(sk) \
-       *((struct sdp_opt **)&(sk)->sk_protinfo)
+       (*((struct sdp_opt **)&(sk)->sk_protinfo))
 #define SDP_SET_CONN(sk, conn) \
-       *((struct sdp_opt **)&(sk)->sk_protinfo) = (conn)
+       (*((struct sdp_opt **)&(sk)->sk_protinfo) = (conn))
 
 #define SDP_CONN_SET_ERR(conn, val) \
-        (conn)->error = (conn)->sk->sk_err = (val)
+        ((conn)->error = (conn)->sk->sk_err = (val))
 #define SDP_CONN_GET_ERR(conn) \
-        (conn)->error
+        ((conn)->error)
 /* --------------------------------------------------------------------- */
 /* state transition information recording                                */
 /* --------------------------------------------------------------------- */
@@ -153,7 +153,7 @@
 }; /* struct sdp_conn_state */
 
 #define SDP_CONN_ST_SET(conn, val) \
-{ \
+do { \
   (conn)->state = (val); \
   if (SDP_CONN_STATE_MAX > (conn)->state_rec.value) { \
     (conn)->state_rec.state[(conn)->state_rec.value] = (val); \
@@ -161,10 +161,10 @@
     (conn)->state_rec.line[(conn)->state_rec.value] = __LINE__; \
     (conn)->state_rec.value++; \
   } \
-} 
+} while(0)
 
 #define SDP_CONN_ST_INIT(conn) \
-{ \
+do { \
   (conn)->state = SDP_CONN_ST_INVALID; \
   for ((conn)->state_rec.value = 0; \
        SDP_CONN_STATE_MAX > (conn)->state_rec.value; \
@@ -174,10 +174,10 @@
     (conn)->state_rec.line[(conn)->state_rec.value] = 0; \
   } \
   (conn)->state_rec.value = 0; \
-} 
+} while(0)
 #else
-#define SDP_CONN_ST_SET(conn, val) (conn)->state = (val)
-#define SDP_CONN_ST_INIT(conn)     (conn)->state = SDP_CONN_ST_INVALID
+#define SDP_CONN_ST_SET(conn, val) ((conn)->state = (val))
+#define SDP_CONN_ST_INIT(conn)     ((conn)->state = SDP_CONN_ST_INVALID)
 #endif
 /*
  * connection lock

-- 
MST - Michael S. Tsirkin


From robert.j.woodruff at intel.com  Thu Feb 17 08:40:58 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Thu, 17 Feb 2005 08:40:58 -0800
Subject: [openib-general] 2005 OpenIB Developers Workshop presentations
Message-ID: <1AC79F16F5C5284499BB9591B33D6F000397B790@orsmsx408>

 James Wrote, 
> We are proposing to maintain the code in both locations. The OpenIB 
> specific implementation would be on openib.org while a 
> platform/transport independent implementation would continue on 
> our SourceForge site.
> 
 
I also agree this seems like the right approach. I don't think the 
actual coding (porting from gen1 to gen2 verbs) will take more than
a few weeks. I suspect most of the time will be spent debugging
issues in the stack, since this will be one of the first
users of the user-mode infrastucture. 

woody



From mshefty at ichips.intel.com  Thu Feb 17 09:51:54 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 17 Feb 2005 09:51:54 -0800
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <52acq3x2t5.fsf@topspin.com>
References: <4213F272.2000805@ichips.intel.com> <52acq3x2t5.fsf@topspin.com>
Message-ID: <4214D9BA.30301@ichips.intel.com>

Roland Dreier wrote:
> Code 25 (port redirection only) raises a separate issue that I think
> is orthogonal to REJ handling.  The CM spec always allows the CM to
> run on a different port from the destination of a connection so if we
> want to implement the whole spec we do need a way for the consumer to
> specify a primary and alternate path for the connection itself as well
> as a CM path.

It looks like one way to handle this is to add a new field to struct 
ib_cm_req_param: struct ib_sa_path_rec *cm_path.  A user could easily 
set this to the same path record referenced by the *primary_path field. 
  This would only be a small change to existing users.

I'm still not sure how to handle a failed path between the CMs.  And 
even if you could detect a failed path, recovering from it is difficult 
without violating the REQ max_cm_retries.  Is anyone planning on using APM?

- Sean


From xma at us.ibm.com  Thu Feb 17 10:38:45 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 17 Feb 2005 11:38:45 -0700
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <OF7C6429CA.E2714D5D-ON87256FA9.0062CBE0-88256FA9.0062DF5F@us.ibm.com>
Message-ID: <OFD9622615.D707806D-ON87256FAB.0065BBEC-00256FAB.0066B58D@us.ibm.com>

Hi, Roland,

I've created a new patch. Please review this one.

diff -urN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-lock/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-17 17:24:56.000000000 
+0000
+++ infiniband-lock/ulp/ipoib/ipoib_main.c      2005-02-17 
17:46:43.000000000 +0000
@@ -220,18 +220,23 @@
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tn;
        struct sk_buff *skb;
+       LIST_HEAD(ah_list);
+       struct ipoib_ah *ah, *tah;
 
        while ((skb = __skb_dequeue(&path->queue)))
                dev_kfree_skb_irq(skb);
 
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
                if (neigh->ah)
-                       ipoib_put_ah(neigh->ah);
+                       list_add_tail(&ah_list, &neigh->ah->list); 
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
 
+       list_for_each_entry_safe(ah, tah, &ah_list, list)
+               ipoib_put_ah(ah);
+
        if (path->ah)
                ipoib_put_ah(path->ah);
 
@@ -660,6 +665,7 @@
        struct ipoib_neigh *neigh = *to_ipoib_neigh(n);
        struct ipoib_dev_priv *priv = netdev_priv(n->dev);
        unsigned long flags;
+       struct ipoib_ah *ah = NULL;
 
        ipoib_dbg(priv,
                  "neigh_destructor for %06x " IPOIB_GID_FMT "\n",
@@ -670,13 +676,16 @@
 
        if (neigh) {
                if (neigh->ah)
-                       ipoib_put_ah(neigh->ah);
+                       ah = neigh->ah;
                list_del(&neigh->list);
                *to_ipoib_neigh(n) = NULL;
                kfree(neigh);
        }
 
        spin_unlock_irqrestore(&priv->lock, flags);
+ 
+       if (ah)
+               ipoib_put_ah(ah);
 }
 
 static int ipoib_neigh_setup(struct neighbour *neigh)
diff -urN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-lock/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-17 
17:24:56.000000000 +0000
+++ infiniband-lock/ulp/ipoib/ipoib_multicast.c 2005-02-17 
17:37:03.000000000 +0000
@@ -93,6 +93,8 @@
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tmp;
        unsigned long flags;
+       LIST_HEAD(ah_list);
+       struct ipoib_ah *ah, *tah;
 
        ipoib_dbg_mcast(netdev_priv(dev),
                        "deleting multicast group " IPOIB_GID_FMT "\n",
@@ -101,7 +103,8 @@
        spin_lock_irqsave(&priv->lock, flags);
 
        list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) {
-               ipoib_put_ah(neigh->ah);
+               if (neigh->ah)
+                       list_add_tail(&ah_list, &neigh->ah->list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
@@ -109,6 +112,9 @@
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
+       list_for_each_entry_safe(ah, tah, &ah_list, list)
+               ipoib_put_ah(ah);
+
        if (mcast->ah)
                ipoib_put_ah(mcast->ah);

 

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/b3f29519/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-lock.patch
Type: application/octet-stream
Size: 2507 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/b3f29519/attachment.obj>

From libor at topspin.com  Thu Feb 17 11:02:56 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 17 Feb 2005 11:02:56 -0800
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217153517.GV28446@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 17, 2005 at 05:35:17PM +0200
References: <20050217153517.GV28446@mellanox.co.il>
Message-ID: <20050217110256.A7121@topspin.com>

On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> Hi, Libor!
> Could you please explain what are the SDP_CONN_LOCK
> and friends doing in SDP?
> 
> It seems they just implement exclusive access to socket,
> but if so, why is a simple semaphore or mutex not used?

  They do implement exclusive access to the socket, but they implement
exclusive access from both process and irq context, which is why a
semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
to lock the connection, look in sdp_cq_event_handler() for it's use,
and in process context SDP_CONN_LOCK is used.

  
  The lock structure has a spinlock around a users counter, the process
context lock will increment the counter when it's using the connection,
but will not hold the spinlock for the duration. The spinlock is only
held for the duration when used in process context. In process context
the wait queue is used if the number of users is non zero, and the
wait queue is awoken when process context decrements the users counter.

  struct sdp_conn_lock {
        __u32 users;
        spinlock_t slock;
        wait_queue_head_t waitq;
  }; /* struct sdp_conn_lock */
   
   Further more if a CQ event occurs and the connection is already locked
in process context (lock.users is non-zero) the connection is marked as
having received an event and the CQ is not polled or rearmed. Instead
when the connection is unlocked using SDP_CONN_UNLOCK the CQ is drained
and rearmed. This results in a significant reduction in interrupts when
the connection is under heavy load. The fifth macro, SDP_CONN_RELOCK,
performs the equivalent of an unlock/lock in process context, which will
cause the CQ to be drained.

   The two flags SDP_CONN_F_{SEND,RECV}_CQ_PEND should not be in the
conn->flags variable, as they are currently, since that variable can be
accessed outside of the spinlock when the process context lock is held.
Instead these two flags struct sdp_conn_lock since they are only
protected when the spinlock is held.


-Libor







From xma at us.ibm.com  Thu Feb 17 11:36:10 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 17 Feb 2005 19:36:10 +0000
Subject: [openib-general] [PATCH][IPOIB] bring interface down panic
In-Reply-To: <OF61DB1A86.7CB133C1-ON87256FAA.00820897-88256FAA.008318D5@us.ibm.com>
Message-ID: <OF0D288FAD.4D5E188D-ON87256FAB.006B926C-00256FAB.006BF72D@us.ibm.com>

Sorry, it's an oops not a panic. 

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/7319f134/attachment.html>

From halr at voltaire.com  Thu Feb 17 11:59:09 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 14:59:09 -0500
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <52acq3x2t5.fsf@topspin.com>
References: <4213F272.2000805@ichips.intel.com> <52acq3x2t5.fsf@topspin.com>
Message-ID: <1108670231.4761.478.camel@localhost.localdomain>

On Wed, 2005-02-16 at 22:50, Roland Dreier wrote:
> Code 25 (port redirection only) raises a separate issue that I think
> is orthogonal to REJ handling.  The CM spec always allows the CM to
> run on a different port from the destination of a connection so if we
> want to implement the whole spec we do need a way for the consumer to
> specify a primary and alternate path for the connection itself as well
> as a CM path.

How much complexity is added to the CM to support this ? Do we really need to
(and if so what is the application which requires this) ? 

> As you say, separating the CM path from the connection path does raise
> the question of what to do for failover and APM.  As far as I can tell
> this is an area that the IB spec is silent on and I'm not sure that
> anyone ever came up with a good answer.

I'm not sure I get the "connection" between the CM path and data path.
Doesn't failover and APM apply only to the data path ? Is it purely the
detected of a failed path between CMs (which seems orthogonal to this to
me) ? That's not to say that this is not an issue.

-- Hal





From halr at voltaire.com  Thu Feb 17 12:02:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 15:02:23 -0500
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <4214D9BA.30301@ichips.intel.com>
References: <4213F272.2000805@ichips.intel.com> <52acq3x2t5.fsf@topspin.com>
	<4214D9BA.30301@ichips.intel.com>
Message-ID: <1108670348.4761.485.camel@localhost.localdomain>

On Thu, 2005-02-17 at 12:51, Sean Hefty wrote:
> It looks like one way to handle this is to add a new field to struct 
> ib_cm_req_param: struct ib_sa_path_rec *cm_path.  A user could easily 
> set this to the same path record referenced by the *primary_path field. 
>   This would only be a small change to existing users.

Sounds straightforward.

> I'm still not sure how to handle a failed path between the CMs.  And 
> even if you could detect a failed path, recovering from it is difficult 
> without violating the REQ max_cm_retries.  Is anyone planning on using APM?

Won't the CM protocol time out in this case and this error be pushed
back on the user ? Would the user then rerequest a PathRecord ?

-- Hal



From roland at topspin.com  Thu Feb 17 12:10:11 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 17 Feb 2005 12:10:11 -0800
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <1108670231.4761.478.camel@localhost.localdomain> (Hal
	Rosenstock's message of "17 Feb 2005 14:59:09 -0500")
References: <4213F272.2000805@ichips.intel.com> <52acq3x2t5.fsf@topspin.com>
	<1108670231.4761.478.camel@localhost.localdomain>
Message-ID: <521xbfvtgc.fsf@topspin.com>

    Hal> I'm not sure I get the "connection" between the CM path and
    Hal> data path.  Doesn't failover and APM apply only to the data
    Hal> path ? Is it purely the detected of a failed path between CMs
    Hal> (which seems orthogonal to this to me) ? That's not to say
    Hal> that this is not an issue.

It is orthogonal in a sense, but consider something like the
following.  I set up a connection with a primary path and a secondary
path, using the same primary path for CM communication.  Some time
later the primary path fails, so APM causes the connection to migrate
to the secondary path.  I get a "path migrated" async event and so I
want to set up a new secondary path so that my connection remains
resilient.   Where do I send the LAP?

 - R.






From mshefty at ichips.intel.com  Thu Feb 17 12:11:49 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 17 Feb 2005 12:11:49 -0800
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <1108670231.4761.478.camel@localhost.localdomain>
References: <4213F272.2000805@ichips.intel.com> <52acq3x2t5.fsf@topspin.com>
	<1108670231.4761.478.camel@localhost.localdomain>
Message-ID: <4214FA85.4030704@ichips.intel.com>

Hal Rosenstock wrote:

>>Code 25 (port redirection only) raises a separate issue that I think
>>is orthogonal to REJ handling.  The CM spec always allows the CM to
> 
> How much complexity is added to the CM to support this ? Do we really need to
> (and if so what is the application which requires this) ? 

I think it can be as simple as adding a new field, cm_path, that can be 
used.  (The user gets back a ClassPortInfo, which would need to be 
converted to a PathRecord...)

Is it needed?  I doubt it.  I'm more concerned that if it ever does 
become needed that we don't have an architecture that prevents adding it.

> I'm not sure I get the "connection" between the CM path and data path.
> Doesn't failover and APM apply only to the data path ?

It does.  However, if the CM established the connection along the 
failed path, it will not be able to send future LAP, DREQ, etc. 
messages unless it sends those along a new path.

> Is it purely the
> detected of a failed path between CMs (which seems orthogonal to this to
> me) ? That's not to say that this is not an issue.

These are somewhat separate issues.  My intention was to have the CM 
send to the destination port that matched the currently active/primary 
path, with the hope that this would result in CM "failover" as well. 
The CM has some of the framework in place to do this, but is lacking 
most of the actual implementation.

- Sean


From halr at voltaire.com  Thu Feb 17 12:23:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 15:23:33 -0500
Subject: [openib-general] [RFC] CM reject handling and redirection
In-Reply-To: <521xbfvtgc.fsf@topspin.com>
References: <4213F272.2000805@ichips.intel.com> <52acq3x2t5.fsf@topspin.com>
	<1108670231.4761.478.camel@localhost.localdomain>
	<521xbfvtgc.fsf@topspin.com>
Message-ID: <1108671813.4761.530.camel@localhost.localdomain>

On Thu, 2005-02-17 at 15:10, Roland Dreier wrote:
> It is orthogonal in a sense, but consider something like the
> following.  I set up a connection with a primary path and a secondary
> path, using the same primary path for CM communication.  Some time
> later the primary path fails, so APM causes the connection to migrate
> to the secondary path.  I get a "path migrated" async event and so I
> want to set up a new secondary path so that my connection remains
> resilient.   Where do I send the LAP?

Hadn't quite thought it through all the way. Sean's subsequent post also
filled in more.

IMO this is worth clarifying with the CM authors. We will hopefully be
enlightened. If no one objects, I am going to take the liberty to cross
post this issue although not everyone will be able to post on the IBA
SWG as you need to be a member.

-- Hal



From mshefty at ichips.intel.com  Thu Feb 17 13:26:50 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 17 Feb 2005 13:26:50 -0800
Subject: [openib-general] [PATCH] [CM] add stale connection checks on
	passive side
Message-ID: <20050217132650.41dabc42.mshefty@ichips.intel.com>

This patch adds checks for a stale connection to the passive side of
the connection protocol.  Checks on the active side are still needed.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1817)
+++ infiniband/core/cm.c	(working copy)
@@ -108,10 +108,11 @@ struct cm_work {
 
 struct cm_timewait_info {
 	struct cm_work work;			/* Must be first. */
-	/* struct rb_node remote_qp_node; */
+	struct rb_node remote_qp_node;
 	struct rb_node remote_id_node;
 	u64 remote_ca_guid;
 	u32 remote_qpn;
+	u8 inserted_remote_qp;
 	u8 inserted_remote_id;
 };
 
@@ -424,80 +425,35 @@ static struct cm_id_private * cm_find_id
 }
 */
 
-static void cm_remove_remote_id(struct cm_timewait_info *timewait_info)
-{
-	unsigned long flags;
-
-	timewait_info->inserted_remote_id = 0;
-	spin_lock_irqsave(&cm.lock, flags);
-	rb_erase(&timewait_info->remote_id_node, &cm.remote_id_table);
-	spin_unlock_irqrestore(&cm.lock, flags);
-}
-
-/*
-todo: add test for stale connections
-static struct cm_id_private * cm_insert_remote_qpn(struct cm_id_private
-						   *cm_id_priv)
+static struct cm_timewait_info * cm_insert_remote_qpn(struct cm_timewait_info
+						      *timewait_info)
 {
 	struct rb_node **link = &cm.remote_qp_table.rb_node;
 	struct rb_node *parent = NULL;
-	struct cm_id_private *cur_cm_id_priv;
-	union ib_gid *port_gid = &cm_id_priv->dgid;
-	u32 remote_qpn = cm_id_priv->remote_qpn;
+	struct cm_timewait_info *cur_timewait_info;
+	u64 remote_ca_guid = timewait_info->remote_ca_guid;
+	u32 remote_qpn = timewait_info->remote_qpn;
 
 	while (*link) {
 		parent = *link;
-		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
-					  remote_qp_node);
-		if (remote_qpn < cur_cm_id_priv->remote_qpn)
+		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
+					     remote_qp_node);
+		if (remote_qpn < cur_timewait_info->remote_qpn)
 			link = &(*link)->rb_left;
-		else if (remote_qpn > cur_cm_id_priv->remote_qpn)
+		else if (remote_qpn > cur_timewait_info->remote_qpn)
 			link = &(*link)->rb_right;
-		else {
-			int cmp;
-			cmp = memcmp(port_gid, &cur_cm_id_priv->dgid,
-				     sizeof *port_gid);
-			if (cmp < 0)
-				link = &(*link)->rb_left;
-			else if (cmp > 0)
-				link = &(*link)->rb_right;
-			else
-				return cur_cm_id_priv;
-		}
-	}
-	rb_link_node(&cm_id_priv->remote_qp_node, parent, link);
-	rb_insert_color(&cm_id_priv->remote_qp_node, &cm.remote_qp_table);
-	return NULL;
-}
-
-static struct cm_id_private * cm_find_id_by_remote_qpn(union ib_gid *port_gid,
-						       u32 remote_qpn)
-{
-	struct rb_node *node = cm.remote_qp_table.rb_node;
-	struct cm_id_private *cm_id_priv;
-
-	while (node) {
-		cm_id_priv = rb_entry(node, struct cm_id_private, remote_qp_node);
-
-		if (remote_qpn < cm_id_priv->remote_qpn)
-			node = node->rb_left;
-		else if (remote_qpn > cm_id_priv->remote_qpn)
-			node = node->rb_right;
-		else {
-			int cmp;
-			cmp = memcmp(port_gid, &cm_id_priv->dgid,
-				     sizeof *port_gid);
-			if (cmp < 0)
-				node = node->rb_left;
-			else if (cmp > 0)
-				node = node->rb_right;
-			else
-				return cm_id_priv;
-		}
+		else if (remote_ca_guid < cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_left;
+		else if (remote_ca_guid > cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_right;
+		else
+			return cur_timewait_info;
 	}
+	timewait_info->inserted_remote_qp = 1;
+	rb_link_node(&timewait_info->remote_qp_node, parent, link);
+	rb_insert_color(&timewait_info->remote_qp_node, &cm.remote_qp_table);
 	return NULL;
 }
-*/
 
 static struct cm_id_private * cm_insert_remote_sidr(struct cm_id_private
 						    *cm_id_priv)
@@ -601,13 +557,23 @@ static inline int cm_convert_to_ms(int i
 
 static void cm_cleanup_timewait(struct cm_timewait_info *timewait_info)
 {
-	if (timewait_info->inserted_remote_id)
-		cm_remove_remote_id(timewait_info);
+	unsigned long flags;
 
-	/* todo: stale connection handling
-	if (timewait_info->inserted_remote_qpn)
-		cm_remove_remote_qpn(timewait_info);
-	*/
+	if (!timewait_info->inserted_remote_id &&
+	    !timewait_info->inserted_remote_qp)
+	    return;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	if (timewait_info->inserted_remote_id) {
+		rb_erase(&timewait_info->remote_id_node, &cm.remote_id_table);
+		timewait_info->inserted_remote_id = 0;
+	}
+
+	if (timewait_info->inserted_remote_qp) {
+		rb_erase(&timewait_info->remote_qp_node, &cm.remote_qp_table);
+		timewait_info->inserted_remote_qp = 0;
+	}
+	spin_unlock_irqrestore(&cm.lock, flags);
 }
 
 static struct cm_timewait_info * cm_create_timewait_info(u32 local_id,
@@ -1088,14 +1054,13 @@ static int cm_req_handler(struct cm_work
 		ret = -EINVAL;
 		goto error2;
 	}
-	/* todo: Check for a stale connection.
+	/* Check for a stale connection. */
 	if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) {
 		spin_unlock_irqrestore(&cm.lock, flags);
-		todo: reject as stale
+		/* todo: reject as stale */
 		ret = -EINVAL;
 		goto error2;
 	}
-	*/
 	/* Find matching listen request. */
 	listen_cm_id_priv = cm_find_listen(req_msg->service_id);
 	if (!listen_cm_id_priv) {


From halr at voltaire.com  Thu Feb 17 13:12:34 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 17 Feb 2005 16:12:34 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108593217.16278.47.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
	<1108589174.4645.138.camel@localhost.localdomain>
	<1108593217.16278.47.camel@sr1-umpk-01>
Message-ID: <1108674754.4761.634.camel@localhost.localdomain>

Hi Nitin,

On Wed, 2005-02-16 at 17:33, Nitin Hande wrote: 
> On Wed, 2005-02-16 at 13:26, Hal Rosenstock wrote:
> > On Wed, 2005-02-16 at 16:08, Nitin Hande wrote:
> > > Hal,
> [snip..]
> > > 
> > > 
> > > Here is the trace of 256 sized MTU:
> > > 
> > > Outgoing MAD:
> > >         BaseVersion: 0x1
> > >         MgmtClass: 0x3 - SubnAdm
> > >         ClassVersion: 0x2
> > >         R_Method: 0x12 - SubnAdmGetTable()
> > >         Status: 0x0 - NO_ERROR
> > >         ClassSpecific: 0x0
> > >         TransactionID: 0x97651d100000096
> > >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > > 
> > >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> > >  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
> > > 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> > > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 
> > > Incoming MAD:
> > >         BaseVersion: 0x1
> > >         MgmtClass: 0x3 - SubnAdm
> > >         ClassVersion: 0x2
> > >         R_Method: 0x92 -
> > >         Status: 0x0 - NO_ERROR
> > >         ClassSpecific: 0x0
> > >         TransactionID: 0x97651d100000096
> > >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > > 
> > >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> > >  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 96  .........vQ.....
> > > 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> > > 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> > > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > > 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 01 00  ................
> > > 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 
> > > And on other occassions where OpenSM reports the 2048 sized MTU:
> > > 
> > > Outgoing MAD:
> > >         BaseVersion: 0x1
> > >         MgmtClass: 0x3 - SubnAdm
> > >         ClassVersion: 0x2
> > >         R_Method: 0x12 - SubnAdmGetTable()
> > >         Status: 0x0 - NO_ERROR
> > >         ClassSpecific: 0x0
> > >         TransactionID: 0x97651d10000009a
> > >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > > 
> > >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> > >  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
> > > 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> > > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 
> > > Incoming MAD:
> > >         BaseVersion: 0x1
> > >         MgmtClass: 0x3 - SubnAdm
> > >         ClassVersion: 0x2
> > >         R_Method: 0x92 -
> > >         Status: 0x0 - NO_ERROR
> > >         ClassSpecific: 0x0
> > >         TransactionID: 0x97651d10000009a
> > >         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> > > 
> > >     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
> > >  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 9a  .........vQ.....
> > > 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> > > 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> > > 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> > > 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> > > 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 04 00  ................
> > > 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > > [output formatted manually]
> > 
> > These both have the exactly selector issue which I fixed but I think you
> > haven't picked that up out of the tree. Is that a problem for Solaris or
> > does it ignore this in the response ?
> > 
> > I have a theory for how the different MTUs (0 (256) and 4 (2048)) occur
> > but need a little time to validate it.
> So I patched the OpenSM with your changes and re-ran the same
> experiment. There was no difference in terms of response for
> MCMemberRecord issued by ibd driver: 
> 
> Here is just the response part:
> 
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x92 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d1000000a3
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 a3  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> 20: 00 00 00 4c 00 00 00 00 00 00 00 00 00 07 00 00  ...L............
> 30: 00 00 00 00 00 00 80 81 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 0b 1b c0 00 01 00  ................
> 60: ff ff 03 12 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> What did change though is the Selector in the McMemberRecord response
> when ipoib driver further tries to join this broadcast group and issues
> another MCMemberRecord. 
> 
> 
> 
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x2 - SubnAdmSet()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d1000000a4
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 02 00 00 00 00 09 76 51 d1 00 00 00 a4  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
> 50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b 00 00 00 00  .....vQ.........
> 60: ff ff 00 00 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x81 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d1000000a4
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 81 00 00 00 00 09 76 51 d1 00 00 00 a4  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00  ................
> 30: 00 00 00 00 00 01 b0 c7 ff 12 40 1b ff ff 00 00  .......... at .....
> 40: 00 00 00 00 ff ff ff ff fe 80 00 00 00 00 00 00  ................
> 50: 00 02 c9 01 09 76 51 d1 00 00 0b 1b c0 00 84 00  .....vQ.........
> 60: ff ff 83 92 00 00 00 00 21 00 00 00 00 00 00 00  ........!.......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> Before the patch the selector was coming 04. Do you reply 84 seeing a
> particular component mask and otherwise 01 ??(I think not..) 

I agree that OpenSM responds/should respond the same regardless of the
component mask in the request.

I was unaware of OpenSM responding with MTU of 01 until now. I have a
theory as to this. Any chance I can get the osm logs from a -V run of
the above ? 

I also have a simple patch below to try which is just to test the
theory. This is off the latest version but should be easy to apply to
any version of osm_sa_mcmember_record.c.

This is separate from the support for PathRecords with multicast DGID
and/or DLID. I have the changes for this scoped out and should be able
to implement by early next week.

Thanks.

-- Hal

Index: osm_sa_mcmember_record.c
===================================================================
--- osm_sa_mcmember_record.c	(revision 1821)
+++ osm_sa_mcmember_record.c	(working copy)
@@ -1325,11 +1325,13 @@
   /* copy qkey mlid tclass pkey sl_flow_hop mtu rate pkt_life
sl_flow_hop */
   __copy_from_create_mc_rec(&mcmember_rec, &p_mgrp->mcmember_rec);
 
+#if 0
   if(p_mgrp->well_known)
   {
     p_mgrp->mcmember_rec.mtu = mtu;
     mcmember_rec.mtu = mtu;
   }
+#endif
 
   /* Release the lock as we don't need it. */
   CL_PLOCK_RELEASE( p_rcv->p_lock );





From mst at mellanox.co.il  Thu Feb 17 13:29:37 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 17 Feb 2005 23:29:37 +0200
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217110256.A7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
Message-ID: <20050217212937.GA21072@mellanox.co.il>


Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [openib-general] SDP_CONN_LOCK
> 
> On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > Hi, Libor!
> > Could you please explain what are the SDP_CONN_LOCK
> > and friends doing in SDP?
> > 
> > It seems they just implement exclusive access to socket,
> > but if so, why is a simple semaphore or mutex not used?
> 
>   They do implement exclusive access to the socket, but they implement
> exclusive access from both process and irq context, which is why a
> semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> to lock the connection, look in sdp_cq_event_handler() for it's use,
> and in process context SDP_CONN_LOCK is used.

I dont really understand how it works.
When an interrupt arrives while users != 0, it seems you are
calling scheduler().
What is sdp_conn_internal_lock doing? I understand it is to be called
from interrupt context, but how can it call scheduler() then?



-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Thu Feb 17 14:00:15 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 17 Feb 2005 14:00:15 -0800
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217212937.GA21072@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 17, 2005 at 11:29:37PM +0200
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
Message-ID: <20050217140015.B7121@topspin.com>

On Thu, Feb 17, 2005 at 11:29:37PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: [openib-general] SDP_CONN_LOCK
> > 
> > On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > > Hi, Libor!
> > > Could you please explain what are the SDP_CONN_LOCK
> > > and friends doing in SDP?
> > > 
> > > It seems they just implement exclusive access to socket,
> > > but if so, why is a simple semaphore or mutex not used?
> > 
> >   They do implement exclusive access to the socket, but they implement
> > exclusive access from both process and irq context, which is why a
> > semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> > to lock the connection, look in sdp_cq_event_handler() for it's use,
> > and in process context SDP_CONN_LOCK is used.
> 
> I dont really understand how it works.
> When an interrupt arrives while users != 0, it seems you are
> calling scheduler().
> What is sdp_conn_internal_lock doing? I understand it is to be called
> from interrupt context, but how can it call scheduler() then?

  SDP_CONN_LOCK and SDP_CONN_UNLOCK are never called from interrupt
context, only from process context. The spinlock conn->lock.slock
is called in interrupt context and process context before the 'users'
variable is changed or read. If users != 0, then the connection is in
use by process context. If the connection is in use then another lock
request in process context will sleep until users == 0, if the connection
is in use then a lock request in interrupt context will mark the
connection as having had an event and return. When a process context
lock holder releases the lock, it checks if there has been an interrupt
event, which it then processes by draining and arming the CQ, and it
wakes any process context waiting for the lock.


-Libor




From roland at topspin.com  Thu Feb 17 14:04:47 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 17 Feb 2005 14:04:47 -0800
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217140015.B7121@topspin.com> (Libor Michalek's message of
	"Thu, 17 Feb 2005 14:00:15 -0800")
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
Message-ID: <52sm3uvo5c.fsf@topspin.com>

BTW, since mthca currently calls completion handlers directly from
interrupt context (rather than BH/tasklet context), it might be worth
renaming all the SDP locking macros so they're not confusingly named
with _BH suffixes.

 - R.


From Tom.Duffy at Sun.COM  Thu Feb 17 14:08:30 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:08:30 -0800
Subject: [openib-general] [PATCH][SDP][0/33] Fix comment style in SDP
Message-ID: <11086781103458@sun.com>

These patches clean up the comment formatting in sdp.  They also update
the copyright header to match the rest of OpenIB files.

Signed-off-by: Tom Duffy <tduffy at sun.com>



From Tom.Duffy at Sun.COM  Thu Feb 17 14:08:40 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:08:40 -0800
Subject: [openib-general] [PATCH][SDP][1/33] Fix comment style in sdp_actv.c
In-Reply-To: <11086781103458@sun.com>
References: <11086781103458@sun.com>
Message-ID: <11086781203512@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_actv.c	2005-02-15 16:33:46.230002000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_actv.c	2005-02-17 10:19:08.821002000 -0800
@@ -1,35 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_post.c 649 2004-08-13 22:41:04Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Connection establishment functions                                    */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_actv_conn_establish-- process an accepted connection request. */
+/*
+ * Connection establishment functions
+ */
+
+/*
+ * _sdp_actv_conn_establish - process an accepted connection request.
+ */
 static int _sdp_actv_conn_establish(struct sdp_opt *conn)
 {
 	struct ib_qp_attr *qp_attr;
@@ -187,8 +198,9 @@ drop:
 	return result;
 } /* _sdp_actv_conn_establish */
 
-/* ========================================================================= */
-/*.._sdp_cm_hello_ack_check -- validate the hello ack header */
+/*
+ * _sdp_cm_hello_ack_check - validate the hello ack header
+ */
 static int _sdp_cm_hello_ack_check(struct msg_hello_ack *hello_ack)
 {
 	/*
@@ -244,8 +256,9 @@ static int _sdp_cm_hello_ack_check(struc
 	return 0; /* success */
 } /* _sdp_cm_hello_ack_check */
 
-/* ========================================================================= */
-/*..sdp_cm_rep_handler -- handler for active connection open completion */
+/*
+ * sdp_cm_rep_handler - handler for active connection open completion
+ */
 int sdp_cm_rep_handler(struct ib_cm_id *cm_id,
 		       struct ib_cm_event *event,
 		       struct sdp_opt *conn)
@@ -355,8 +368,9 @@ done:
 	return error;
 } /* sdp_cm_rep_handler */
 
-/* ========================================================================= */
-/*.._sdp_cm_path_complete -- path lookup complete, initiate SDP connection */
+/*
+ * _sdp_cm_path_complete - path lookup complete, initiate SDP connection
+ */
 static void _sdp_cm_path_complete(u64 id,
 				  int status,
 				  u32 dst_addr,
@@ -562,8 +576,9 @@ done:
 	return;
 } /* _sdp_cm_path_complete */
 
-/* ========================================================================= */
-/*..sdp_cm_connect -- initiate a SDP connection with a hello message. */
+/*
+ * sdp_cm_connect - initiate a SDP connection with a hello message.
+ */
 int sdp_cm_connect(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -614,4 +629,3 @@ int sdp_cm_connect(struct sdp_opt *conn)
 error:
 	return result;
 } /* sdp_cm_connect */
-



From Tom.Duffy at Sun.COM  Thu Feb 17 14:08:50 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:08:50 -0800
Subject: [openib-general] [PATCH][SDP][2/33] Fix comment style in sdp_advt.c
In-Reply-To: <11086781203512@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
Message-ID: <1108678130413@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_advt.c	2005-02-10 18:42:56.537025000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_advt.c	2005-02-17 10:22:17.245012000 -0800
@@ -1,44 +1,53 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_advt.c 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
 static kmem_cache_t *__sdp_advt_cache = NULL;
 static kmem_cache_t *__sdp_advt_table = NULL;
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* module specific functions                                             */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* public advertisment object functions for FIFO object table            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_advt_create - create an advertisment object */
+/*
+ * module specific functions
+ */
+
+/*
+ * public advertisment object functions for FIFO object table
+ */
+
+/*
+ * sdp_advt_create - create an advertisment object
+ */
 struct sdpc_advt *sdp_advt_create(void)
 {
 	struct sdpc_advt *advt;
@@ -60,8 +69,9 @@ struct sdpc_advt *sdp_advt_create(void)
 	return advt;
 } /* sdp_advt_create */
 
-/* ========================================================================= */
-/*..sdp_advt_destroy - destroy an advertisment object */
+/*
+ * sdp_advt_destroy - destroy an advertisment object
+ */
 int sdp_advt_destroy(struct sdpc_advt *advt)
 {
 	SDP_CHECK_NULL(advt, -EINVAL);
@@ -78,8 +88,9 @@ int sdp_advt_destroy(struct sdpc_advt *a
 	return 0;
 } /* sdp_advt_destroy */
 
-/* ========================================================================= */
-/*..sdp_advt_q_get - get, and remove, the object at the tables head */
+/*
+ * sdp_advt_q_get - get, and remove, the object at the tables head
+ */
 struct sdpc_advt *sdp_advt_q_get(struct sdpc_advt_q *table)
 {
 	struct sdpc_advt *advt;
@@ -116,8 +127,9 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 	return advt;
 } /* sdp_advt_q_get */
 
-/* ========================================================================= */
-/*..sdp_advt_q_look - get, without removing, the object at the head */
+/*
+ * sdp_advt_q_look - get, without removing, the object at the head
+ */
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table)
 {
 	SDP_CHECK_NULL(table, NULL);
@@ -125,8 +137,9 @@ struct sdpc_advt *sdp_advt_q_look(struct
 	return table->head;
 } /* sdp_advt_q_look */
 
-/* ========================================================================= */
-/*..sdp_advt_q_put - put the advertisment object at the tables tail */
+/*
+ * sdp_advt_q_put - put the advertisment object at the tables tail
+ */
 int sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt)
 {
 	struct sdpc_advt *next;
@@ -157,13 +170,13 @@ int sdp_advt_q_put(struct sdpc_advt_q *t
 	return 0;
 } /* sdp_advt_q_put */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* public table functions                                                */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_advt_q_create - create an advertisment table */
+/*
+ * public table functions
+ */
+
+/*
+ * sdp_advt_q_create - create an advertisment table
+ */
 struct sdpc_advt_q *sdp_advt_q_create(int * result)
 {
 	struct sdpc_advt_q *table = NULL;
@@ -184,8 +197,9 @@ struct sdpc_advt_q *sdp_advt_q_create(in
 	return table;
 } /* sdp_advt_q_create */
 
-/* ========================================================================= */
-/*..sdp_advt_q_init - initialize a new empty advertisment table */
+/*
+ * sdp_advt_q_init - initialize a new empty advertisment table
+ */
 int sdp_advt_q_init(struct sdpc_advt_q *table)
 {
 	SDP_CHECK_NULL(table, -EINVAL);
@@ -196,8 +210,9 @@ int sdp_advt_q_init(struct sdpc_advt_q *
 	return 0;
 } /* sdp_advt_q_init */
 
-/* ========================================================================= */
-/*..sdp_advt_q_clear - clear the contents of an advertisment table */
+/*
+ * sdp_advt_q_clear - clear the contents of an advertisment table
+ */
 int sdp_advt_q_clear(struct sdpc_advt_q *table)
 {
 	struct sdpc_advt *advt;
@@ -216,8 +231,9 @@ int sdp_advt_q_clear(struct sdpc_advt_q 
 	return 0;
 } /* sdp_advt_q_clear */
 
-/* ========================================================================= */
-/*..sdp_advt_q_destroy - destroy an advertisment table */
+/*
+ * sdp_advt_q_destroy - destroy an advertisment table
+ */
 int sdp_advt_q_destroy(struct sdpc_advt_q *table)
 {
 	struct sdpc_advt *advt;
@@ -240,13 +256,13 @@ int sdp_advt_q_destroy(struct sdpc_advt_
 	return 0;
 } /* sdp_advt_q_destroy */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* primary initialization/cleanup functions                              */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_main_advt_init -- initialize the advertisment caches. */
+/*
+ * primary initialization/cleanup functions
+ */
+
+/*
+ * sdp_main_advt_init - initialize the advertisment caches.
+ */
 int sdp_main_advt_init(void)
 {
 	int result;
@@ -289,8 +305,9 @@ error_advt_c:
 	return 0;
 } /* sdp_main_advt_init */
 
-/* ========================================================================= */
-/*..sdp_main_advt_cleanup -- cleanup the advertisment caches. */
+/*
+ * sdp_main_advt_cleanup - cleanup the advertisment caches.
+ */
 int sdp_main_advt_cleanup(void)
 {
 	sdp_dbg_init("Advertisment cache cleanup.");



From Tom.Duffy at Sun.COM  Thu Feb 17 14:09:00 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:09:00 -0800
Subject: [openib-general] [PATCH][SDP][3/33] Fix comment style in sdp_advt.h
In-Reply-To: <1108678130413@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com>
Message-ID: <1108678140425@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_advt.h	2005-02-10 18:42:56.640003000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_advt.h	2005-02-17 12:45:25.967016000 -0800
@@ -1,38 +1,50 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_advt.h 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_ADVT_H
 #define _SDP_ADVT_H
 
 #include "sdp_queue.h"
+
 /*
  * IOCB flags.
  */
 #define SDP_ADVT_F_READ 0x00000001 /* ADVT has an active read operation */
-/* ----------------------------------------------------------------------- */
-/* SDP read/write advertisments                                            */
-/* ----------------------------------------------------------------------- */
 
+/*
+ * SDP read/write advertisments
+ */
 struct sdpc_advt {
 	struct sdpc_advt   *next;  /* next structure in table */
 	struct sdpc_advt   *prev;  /* previous structure in table */
@@ -49,6 +61,7 @@ struct sdpc_advt {
 	u32 flag; /* advertisment flags. */
 	u64 addr; /* advertised buffer virtual address */
 }; /* struct sdpc_advt */
+
 /*
  * table for holding SDP advertisments.
  */
@@ -56,6 +69,7 @@ struct sdpc_advt_q {
 	struct sdpc_advt *head; /* double linked list of advertisments */
 	s32 size;               /* current number of advertisments in table */
 }; /* struct sdpc_advt_q */
+
 /*
  * make size a macro.
  */



From Tom.Duffy at Sun.COM  Thu Feb 17 14:09:11 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:09:11 -0800
Subject: [openib-general] [PATCH][SDP][4/33] Fix comment style in sdp_buff.c
In-Reply-To: <1108678140425@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
Message-ID: <11086781512781@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_buff.c	2005-02-11 08:02:05.883004000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_buff.c	2005-02-17 10:27:46.250000000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_buff.c 994 2004-10-14 05:38:39Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
@@ -27,8 +38,10 @@ static struct sdpc_buff_root *main_pool 
 /*
  * data buffers managment API
  */
-/* ========================================================================= */
-/*.._sdp_buff_q_get - Get a buffer from a specific pool */
+
+/*
+ * _sdp_buff_q_get - Get a buffer from a specific pool
+ */
 static __inline__ struct sdpc_buff *_sdp_buff_q_get
 (
 	struct sdpc_buff_q *pool,
@@ -84,8 +97,9 @@ static __inline__ struct sdpc_buff *_sdp
 	return buff;
 } /* _sdp_buff_q_get */
 
-/* ========================================================================= */
-/*.._sdp_buff_q_put - Place a buffer into a specific pool */
+/*
+ * _sdp_buff_q_put - Place a buffer into a specific pool
+ */
 static __inline__ int _sdp_buff_q_put(struct sdpc_buff_q *pool,
 				      struct sdpc_buff *buff,
 				      int fifo)
@@ -124,8 +138,9 @@ static __inline__ int _sdp_buff_q_put(st
 	return 0;
 } /* _sdp_buff_q_put */
 
-/* ========================================================================= */
-/*.._sdp_buff_q_look - look at a buffer from a specific pool */
+/*
+ * _sdp_buff_q_look - look at a buffer from a specific pool
+ */
 static __inline__ struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
 						     int fifo)
 {
@@ -141,8 +156,9 @@ static __inline__ struct sdpc_buff *_sdp
 	}
 } /* _sdp_buff_q_look */
 
-/* ========================================================================= */
-/*.._sdp_buff_q_remove - remove a specific buffer from a specific pool */
+/*
+ * _sdp_buff_q_remove - remove a specific buffer from a specific pool
+ */
 static __inline__ int _sdp_buff_q_remove(struct sdpc_buff_q *pool, 
 					 struct sdpc_buff *buff)
 {
@@ -186,8 +202,9 @@ static __inline__ int _sdp_buff_q_remove
 	return 0;
 } /* _sdp_buff_q_remove */
 
-/* ========================================================================= */
-/*..sdp_buff_q_init - Init a pool drawing its buffers from the main pool */
+/*
+ * sdp_buff_q_init - Init a pool drawing its buffers from the main pool
+ */
 int sdp_buff_q_init(struct sdpc_buff_q *pool, u32 size)
 {
 	int counter;
@@ -215,8 +232,9 @@ error:
 	return result;
 } /* sdp_buff_q_init */
 
-/* ========================================================================= */
-/*..sdp_buff_q_remove - remove a specific buffer from a specific pool */
+/*
+ * sdp_buff_q_remove - remove a specific buffer from a specific pool
+ */
 int sdp_buff_q_remove(struct sdpc_buff *buff)
 {
 	struct sdpc_buff_q *pool;
@@ -232,8 +250,9 @@ int sdp_buff_q_remove(struct sdpc_buff *
 	return result;
 } /* sdp_buff_q_remove */
 
-/* ========================================================================= */
-/*..sdp_buff_q_get - Get a buffer from a specific pool */
+/*
+ * sdp_buff_q_get - Get a buffer from a specific pool
+ */
 struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool)
 {
 	struct sdpc_buff *buff;
@@ -243,8 +262,9 @@ struct sdpc_buff *sdp_buff_q_get(struct 
 	return buff;
 } /* sdp_buff_q_get */
 
-/* ========================================================================= */
-/*..sdp_buff_q_get_head - Get the buffer at the front of the pool */
+/*
+ * sdp_buff_q_get_head - Get the buffer at the front of the pool
+ */
 struct sdpc_buff *sdp_buff_q_get_head(struct sdpc_buff_q *pool)
 {
 	struct sdpc_buff *buff;
@@ -254,8 +274,9 @@ struct sdpc_buff *sdp_buff_q_get_head(st
 	return buff;
 } /* sdp_buff_q_get_head */
 
-/* ========================================================================= */
-/*..sdp_buff_q_get_tail - Get the buffer at the end of the pool */
+/*
+ * sdp_buff_q_get_tail - Get the buffer at the end of the pool
+ */
 struct sdpc_buff *sdp_buff_q_get_tail(struct sdpc_buff_q *pool)
 {
 	struct sdpc_buff *buff;
@@ -265,8 +286,9 @@ struct sdpc_buff *sdp_buff_q_get_tail(st
 	return buff;
 } /* sdp_buff_q_get_tail */
 
-/* ========================================================================= */
-/*..sdp_buff_q_look_head - look at the buffer at the front of the pool */
+/*
+ * sdp_buff_q_look_head - look at the buffer at the front of the pool
+ */
 struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool)
 {
 	struct sdpc_buff *buff;
@@ -276,8 +298,9 @@ struct sdpc_buff *sdp_buff_q_look_head(s
 	return buff;
 } /* sdp_buff_q_look_head */
 
-/* ========================================================================= */
-/*..sdp_buff_q_look_tail - look at the buffer at the end of the pool */
+/*
+ * sdp_buff_q_look_tail - look at the buffer at the end of the pool
+ */
 struct sdpc_buff *sdp_buff_q_look_tail(struct sdpc_buff_q *pool)
 {
 	struct sdpc_buff *buff;
@@ -287,8 +310,9 @@ struct sdpc_buff *sdp_buff_q_look_tail(s
 	return buff;
 } /* sdp_buff_q_look_tail */
 
-/* ========================================================================= */
-/*..sdp_buff_q_fetch_head - Get the pools first buffer, if the test passes */
+/*
+ * sdp_buff_q_fetch_head - Get the pools first buffer, if the test passes
+ */
 struct sdpc_buff *sdp_buff_q_fetch_head(struct sdpc_buff_q *pool,
 					int (*test)(struct sdpc_buff *buff,
 						    void *arg),
@@ -301,8 +325,9 @@ struct sdpc_buff *sdp_buff_q_fetch_head(
 	return buff;
 } /* sdp_buff_q_fetch_head */
 
-/* ========================================================================= */
-/*..sdp_buff_q_fetch_tail - Get the pools last buffer, if the test passes */
+/*
+ * sdp_buff_q_fetch_tail - Get the pools last buffer, if the test passes
+ */
 struct sdpc_buff *sdp_buff_q_fetch_tail(struct sdpc_buff_q *pool,
 					int (*test)(struct sdpc_buff *buff,
 						    void *arg),
@@ -315,8 +340,9 @@ struct sdpc_buff *sdp_buff_q_fetch_tail(
 	return buff;
 } /* sdp_buff_q_fetch_tail */
 
-/* ========================================================================= */
-/*..sdp_buff_q_fetch - Get the first matching buffer from the pool */
+/*
+ * sdp_buff_q_fetch - Get the first matching buffer from the pool
+ */
 struct sdpc_buff *sdp_buff_q_fetch(struct sdpc_buff_q *pool,
 				   int (*test)(struct sdpc_buff *buff,
 					       void *arg),
@@ -356,8 +382,9 @@ struct sdpc_buff *sdp_buff_q_fetch(struc
 	return NULL;
 } /* sdp_buff_q_fetch_head */
 
-/* ========================================================================= */
-/*..sdp_buff_q_trav_tail - traverse buffers in pool,from the tail */
+/*
+ * sdp_buff_q_trav_tail - traverse buffers in pool,from the tail
+ */
 int sdp_buff_q_trav_tail(struct sdpc_buff_q *pool,
 			 int (*trav_func)(struct sdpc_buff *buff,
 					  void *arg),
@@ -390,8 +417,9 @@ int sdp_buff_q_trav_tail(struct sdpc_buf
 	return result;
 } /* sdp_buff_q_trav_tail */
 
-/* ========================================================================= */
-/*..sdp_buff_q_trav_head - traverse buffers in pool, from the head */
+/*
+ * sdp_buff_q_trav_head - traverse buffers in pool, from the head
+ */
 int sdp_buff_q_trav_head(struct sdpc_buff_q *pool,
 			 int (*trav_func)(struct sdpc_buff *buff,
 					  void *arg),
@@ -424,8 +452,9 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 	return result;
 } /* sdp_buff_q_trav_head */
 
-/* ========================================================================= */
-/*..sdp_buff_q_put - Place a buffer into a specific pool */
+/*
+ * sdp_buff_q_put - Place a buffer into a specific pool
+ */
 int sdp_buff_q_put(struct sdpc_buff_q *pool,
 		   struct sdpc_buff *buff)
 {
@@ -436,8 +465,9 @@ int sdp_buff_q_put(struct sdpc_buff_q *p
 	return result;
 } /* sdp_buff_q_put */
 
-/* ========================================================================= */
-/*..sdp_buff_q_put_head - Place a buffer into the head of a specific pool */
+/*
+ * sdp_buff_q_put_head - Place a buffer into the head of a specific pool
+ */
 int sdp_buff_q_put_head(struct sdpc_buff_q *pool,
 			struct sdpc_buff *buff)
 {
@@ -448,8 +478,9 @@ int sdp_buff_q_put_head(struct sdpc_buff
 	return result;
 } /* sdp_buff_q_put_head */
 
-/* ========================================================================= */
-/*..sdp_buff_q_put_tail - Place a buffer into the tail of a specific pool */
+/*
+ * sdp_buff_q_put_tail - Place a buffer into the tail of a specific pool
+ */
 int sdp_buff_q_put_tail(struct sdpc_buff_q *pool,
 			struct sdpc_buff *buff)
 {
@@ -460,8 +491,9 @@ int sdp_buff_q_put_tail(struct sdpc_buff
 	return result;
 } /* sdp_buff_q_put_tail */
 
-/* ========================================================================= */
-/*..sdp_buff_q_clear_unmap - clear the buffers out of a specific buffer pool */
+/*
+ * sdp_buff_q_clear_unmap - clear the buffers out of a specific buffer pool
+ */
 int sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
 			   struct device *dev,
 			   int direction)
@@ -491,14 +523,13 @@ int sdp_buff_q_clear_unmap(struct sdpc_b
 	return 0;
 } /* sdp_buff_q_clear */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* internal data buffer pool manager                                     */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * internal data buffer pool manager
+ */
 
-/* ========================================================================= */
-/*.._sdp_buff_pool_release -- release allocated buffers from the main pool */
+/*
+ * _sdp_buff_pool_release - release allocated buffers from the main pool
+ */
 static int _sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
 {
 	struct sdpc_buff *buff;
@@ -526,8 +557,9 @@ static int _sdp_buff_pool_release(struct
 	return 0;
 } /* _sdp_buff_pool_release */
 
-/* ========================================================================= */
-/*.._sdp_buff_pool_release_check -- check for buffer release from main pool */
+/*
+ * _sdp_buff_pool_release_check - check for buffer release from main pool
+ */
 static __inline__ int _sdp_buff_pool_release_check
 (
 	struct sdpc_buff_root *m_pool
@@ -556,8 +588,9 @@ static __inline__ int _sdp_buff_pool_rel
 	}
 } /* _sdp_buff_pool_release_check */
 
-/* ========================================================================= */
-/*.._sdp_buff_pool_alloc -- allocate more buffers for the main pool */
+/*
+ * _sdp_buff_pool_alloc - allocate more buffers for the main pool
+ */
 static int _sdp_buff_pool_alloc(struct sdpc_buff_root *m_pool)
 {
 	struct sdpc_buff *buff;
@@ -628,8 +661,9 @@ static int _sdp_buff_pool_alloc(struct s
 	return 0;
 } /* _sdp_buff_pool_alloc */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_init - Initialize the main buffer pool of memory */
+/*
+ * sdp_buff_pool_init - Initialize the main buffer pool of memory
+ */
 int sdp_buff_pool_init(int buff_min, 
 		       int buff_max,
 		       int alloc_inc,
@@ -725,8 +759,9 @@ done:
 	return result;
 } /* sdp_buff_pool_init */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_destroy - Destroy the main buffer pool and free its memory */
+/*
+ * sdp_buff_pool_destroy - Destroy the main buffer pool and free its memory
+ */
 void sdp_buff_pool_destroy(void)
 {
 	if (NULL == main_pool) {
@@ -764,8 +799,9 @@ void sdp_buff_pool_destroy(void)
 	return;
 } /* sdp_buff_pool_destroy */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_get - Get a buffer from the main buffer pool */
+/*
+ * sdp_buff_pool_get - Get a buffer from the main buffer pool
+ */
 struct sdpc_buff *sdp_buff_pool_get(void)
 {
 	struct sdpc_buff *buff;
@@ -823,8 +859,9 @@ struct sdpc_buff *sdp_buff_pool_get(void
 	return buff;
 } /* sdp_buff_pool_get */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_put - Return a buffer to the main buffer pool */
+/*
+ * sdp_buff_pool_put - Return a buffer to the main buffer pool
+ */
 int sdp_buff_pool_put(struct sdpc_buff *buff)
 {
 	unsigned long flags;
@@ -873,8 +910,9 @@ int sdp_buff_pool_put(struct sdpc_buff *
 	return 0;
 } /* sdp_buff_pool_put */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_chain_link - create chain of buffers which can be returned */
+/*
+ * sdp_buff_pool_chain_link - create chain of buffers which can be returned
+ */
 int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(main_pool, -EINVAL);
@@ -901,8 +939,9 @@ int sdp_buff_pool_chain_link(struct sdpc
 	return 0;
 } /* sdp_buff_pool_chain_link */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_chain_put - Return a buffer to the main buffer pool */
+/*
+ * sdp_buff_pool_chain_put - Return a buffer to the main buffer pool
+ */
 int sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count)
 {
 	unsigned long flags;
@@ -948,8 +987,9 @@ int sdp_buff_pool_chain_put(struct sdpc_
 	return 0;
 } /* sdp_buff_pool_chain_put */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_size - number of elements in the main buffer pool */
+/*
+ * sdp_buff_pool_size - number of elements in the main buffer pool
+ */
 int sdp_buff_pool_size(void)
 {
 	unsigned long flags;
@@ -967,8 +1007,9 @@ int sdp_buff_pool_size(void)
 	return size;
 } /* sdp_buff_pool_size */
 
-/* ========================================================================= */
-/*..sdp_buff_pool_buff_size - return the size of buffers in the main pool */
+/*
+ * sdp_buff_pool_buff_size - return the size of buffers in the main pool
+ */
 int sdp_buff_pool_buff_size(void)
 {
 	int result;
@@ -985,8 +1026,9 @@ int sdp_buff_pool_buff_size(void)
 	return result;
 } /* sdp_buff_pool_buff_size */
 
-/* ========================================================================= */
-/*..sdp_proc_dump_buff_pool -- write the buffer pool stats to a file (/proc) */
+/*
+ * sdp_proc_dump_buff_pool - write the buffer pool stats to a file (/proc)
+ */
 int sdp_proc_dump_buff_pool(char *buffer,
 			    int   max_size,
 			    off_t start_index,



From Tom.Duffy at Sun.COM  Thu Feb 17 14:09:21 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:09:21 -0800
Subject: [openib-general] [PATCH][SDP][5/33] Fix comment style in sdp_buff.h
In-Reply-To: <11086781512781@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
Message-ID: <11086781612958@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_buff.h	2005-02-11 07:59:56.184009000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_buff.h	2005-02-17 10:28:23.514011000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_buff.h 994 2004-10-14 05:38:39Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_BUFF_H
 #define _SDP_BUFF_H



From Tom.Duffy at Sun.COM  Thu Feb 17 14:09:31 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:09:31 -0800
Subject: [openib-general] [PATCH][SDP][6/33] Fix comment style in
	sdp_buff_p.h
In-Reply-To: <11086781612958@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com> <11086781612958@sun.com>
Message-ID: <11086781711886@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_buff_p.h	2005-02-10 18:42:56.714018000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_buff_p.h	2005-02-17 10:28:48.950000000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_buff_p.h 994 2004-10-14 05:38:39Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_BUFF_P_H
 #define _SDP_BUFF_P_H
@@ -63,9 +74,3 @@ struct sdpc_buff_root {
 }; /* struct sdpc_buff_root */
 
 #endif /* _SDP_BUFF_P_H */
-
-
-
-
-
-



From Tom.Duffy at Sun.COM  Thu Feb 17 14:09:41 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:09:41 -0800
Subject: [openib-general] [PATCH][SDP][7/33] Fix comment style in sdp_conn.c
In-Reply-To: <11086781711886@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
Message-ID: <11086781814000@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_conn.c	2005-02-14 12:51:45.936001000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_conn.c	2005-02-17 10:38:44.836003000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_conn.c 1032 2004-10-20 23:17:34Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
@@ -37,13 +48,13 @@ static struct ib_client sdp_client = {
 static spinlock_t psn_lock = SPIN_LOCK_UNLOCKED;
 static u32        psn_seed;
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* module specific functions                                             */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_psn_generate -- generate a PSN for connection management. */
+/*
+ * module specific functions
+ */
+
+/*
+ * sdp_psn_generate - generate a PSN for connection management.
+ */
 static u32 _sdp_psn_generate(void)
 {
         u32 psn;
@@ -62,8 +73,9 @@ static u32 _sdp_psn_generate(void)
         return psn;
 } /* sdp_psn_generate */
 
-/* ========================================================================= */
-/*..sdp_inet_accept_q_put -- put a conn into a listen conn's accept Q. */
+/*
+ * sdp_inet_accept_q_put - put a conn into a listen conn's accept Q.
+ */
 int sdp_inet_accept_q_put(struct sdp_opt *listen_conn,
 			  struct sdp_opt *accept_conn)
 {
@@ -97,8 +109,9 @@ int sdp_inet_accept_q_put(struct sdp_opt
 	return 0;
 } /* sdp_inet_accept_q_put */
 
-/* ========================================================================= */
-/*..sdp_inet_accept_q_get -- get a conn from a listen conn's accept Q. */
+/*
+ * sdp_inet_accept_q_get - get a conn from a listen conn's accept Q.
+ */
 struct sdp_opt *sdp_inet_accept_q_get(struct sdp_opt *listen_conn)
 {
 	struct sdp_opt *prev_conn;
@@ -139,8 +152,9 @@ struct sdp_opt *sdp_inet_accept_q_get(st
 	return accept_conn;
 } /* sdp_inet_accept_q_get */
 
-/* ========================================================================= */
-/*..sdp_inet_accept_q_remove -- remove a conn from a conn's accept Q. */
+/*
+ * sdp_inet_accept_q_remove - remove a conn from a conn's accept Q.
+ */
 int sdp_inet_accept_q_remove(struct sdp_opt *accept_conn)
 {
 	struct sdp_opt *next_conn;
@@ -177,8 +191,9 @@ int sdp_inet_accept_q_remove(struct sdp_
 	return 0;
 } /* sdp_inet_accept_q_remove */
 
-/* ========================================================================= */
-/*..sdp_inet_listen_start -- start listening for new connections on a socket */
+/*
+ * sdp_inet_listen_start - start listening for new connections on a socket
+ */
 int sdp_inet_listen_start(struct sdp_opt *conn)
 {
 	unsigned long flags;
@@ -214,8 +229,9 @@ int sdp_inet_listen_start(struct sdp_opt
 	return 0;
 } /* sdp_inet_listen_start */
 
-/* ========================================================================= */
-/*..sdp_inet_listen_stop -- stop listening for new connections on a socket */
+/*
+ * sdp_inet_listen_stop - stop listening for new connections on a socket
+ */
 int sdp_inet_listen_stop(struct sdp_opt *listen_conn)
 {
 	struct sdp_opt *accept_conn;
@@ -279,8 +295,9 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	return 0;
 } /* sdp_inet_listen_stop */
 
-/* ========================================================================= */
-/*..sdp_inet_listen_lookup -- lookup a connection in the listen list */
+/*
+ * sdp_inet_listen_lookup - lookup a connection in the listen list
+ */
 struct sdp_opt *sdp_inet_listen_lookup(u32 addr, u16 port)
 {
 	struct sdp_opt *conn;
@@ -307,8 +324,9 @@ struct sdp_opt *sdp_inet_listen_lookup(u
 	return conn;
 } /* sdp_inet_listen_lookup */
 
-/* ========================================================================= */
-/*..sdp_inet_port_get -- bind a socket to a port. */
+/*
+ * sdp_inet_port_get - bind a socket to a port.
+ */
 int sdp_inet_port_get(struct sdp_opt *conn, u16 port)
 {
 	struct sock *sk;
@@ -341,7 +359,8 @@ int sdp_inet_port_get(struct sdp_opt *co
 			/*
 			 * 1) same port
 			 * 2) linux force reuse is off.
-			 * 3) same bound interface, or neither has a bound interface
+			 * 3) same bound interface, or neither has a bound
+			 *    interface
 			 */
 			if (look->src_port == port &&
 			    !(1 < sk->sk_reuse) &&
@@ -356,8 +375,10 @@ int sdp_inet_port_get(struct sdp_opt *co
 				    0 == srch->sk_reuse ||
 				    SDP_SOCK_ST_LISTEN == look->istate) {
 					/*
-					 * 5) neither socket is using a specific address
-					 * 6) both sockets are trying for the same interface.
+					 * 5) neither socket is using a
+					 *    specific address
+					 * 6) both sockets are trying for the
+					 *    same interface.
 					 */
 					if (INADDR_ANY == conn->src_addr ||
 					    INADDR_ANY == look->src_addr ||
@@ -442,8 +463,9 @@ done:
 	return result;
 } /* sdp_inet_port_get */
 
-/* ========================================================================= */
-/*..sdp_inet_port_put -- unbind a socket from a port. */
+/*
+ * sdp_inet_port_put - unbind a socket from a port.
+ */
 int sdp_inet_port_put(struct sdp_opt *conn)
 {
 	unsigned long flags;
@@ -475,8 +497,9 @@ int sdp_inet_port_put(struct sdp_opt *co
 	return 0;
 } /* sdp_inet_port_put */
 
-/* ========================================================================= */
-/*..sdp_inet_port_inherit -- inherit a port from another socket (accept) */
+/*
+ * sdp_inet_port_inherit - inherit a port from another socket (accept)
+ */
 int sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child)
 {
 	int result;
@@ -516,8 +539,9 @@ done:
 	return result;
 } /* sdp_inet_port_inherit */
 
-/* ========================================================================= */
-/*..sdp_conn_table_insert -- insert a connection into the connection table */
+/*
+ * sdp_conn_table_insert - insert a connection into the connection table
+ */
 int sdp_conn_table_insert(struct sdp_opt *conn)
 {
 	u32 counter;
@@ -567,8 +591,9 @@ int sdp_conn_table_insert(struct sdp_opt
 	return result;
 } /* sdp_conn_table_insert */
 
-/* ========================================================================= */
-/*..sdp_conn_table_remove -- remove a connection from the connection table */
+/*
+ * sdp_conn_table_remove - remove a connection from the connection table
+ */
 int sdp_conn_table_remove(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -608,8 +633,9 @@ done:
 	return result;
 } /* sdp_conn_table_remove */
 
-/* ========================================================================= */
-/*..sdp_conn_table_lookup -- look up connection in the connection table */
+/*
+ * sdp_conn_table_lookup - look up connection in the connection table
+ */
 struct sdp_opt *sdp_conn_table_lookup(s32 entry)
 {
 	struct sdp_opt *conn;
@@ -641,13 +667,13 @@ done:
 	return conn;
 } /* sdp_conn_table_lookup */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* connection allocation/deallocation                                    */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_conn_destruct -- final destructor for connection. */
+/*
+ * connection allocation/deallocation
+ */
+
+/*
+ * sdp_conn_destruct - final destructor for connection.
+ */
 int sdp_conn_destruct(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -802,8 +828,9 @@ done:
 	return result;
 } /* _sdp_conn_destruct */
 
-/* ========================================================================= */
-/*..sdp_conn_internal_lock -- lock the connection (use only from macro) */
+/*
+ * sdp_conn_internal_lock - lock the connection (use only from macro)
+ */
 void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags)
 {
 	DECLARE_WAITQUEUE(wait, current);
@@ -829,8 +856,9 @@ void sdp_conn_internal_lock(struct sdp_o
 	return;
 } /* sdp_conn_internal_lock */
 
-/* ========================================================================= */
-/*..sdp_conn_internal_relock -- test the connection (use only from macro) */
+/*
+ * sdp_conn_internal_relock - test the connection (use only from macro)
+ */
 void sdp_conn_internal_relock(struct sdp_opt *conn)
 {
 	struct ib_wc entry;
@@ -909,8 +937,9 @@ void sdp_conn_internal_relock(struct sdp
 	return;
 } /* sdp_conn_internal_relock */
 
-/* ========================================================================= */
-/*..sdp_conn_cq_drain -- drain one of the the connection's CQs */
+/*
+ * sdp_conn_cq_drain - drain one of the the connection's CQs
+ */
 int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn)
 {
 	struct ib_wc entry;
@@ -977,8 +1006,9 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 return calls;
 } /* sdp_conn_cq_drain */
 
-/* ========================================================================= */
-/*..sdp_conn_internal_unlock -- lock the connection (use only from macro) */
+/*
+ * sdp_conn_internal_unlock - lock the connection (use only from macro)
+ */
 void sdp_conn_internal_unlock(struct sdp_opt *conn)
 {
 	int calls = 0;
@@ -1003,8 +1033,9 @@ void sdp_conn_internal_unlock(struct sdp
 	return;
 } /* sdp_conn_internal_unlock */
 
-/* ========================================================================= */
-/*.._sdp_conn_lock_init -- initialize connection lock */
+/*
+ * _sdp_conn_lock_init - initialize connection lock
+ */
 static int _sdp_conn_lock_init(struct sdp_opt *conn)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -1016,8 +1047,9 @@ static int _sdp_conn_lock_init(struct sd
 	return 0;
 } /* _sdp_conn_lock_init */
 
-/* ========================================================================= */
-/*..sdp_conn_alloc_ib -- allocate IB structures for a new connection. */
+/*
+ * sdp_conn_alloc_ib - allocate IB structures for a new connection.
+ */
 int sdp_conn_alloc_ib(struct sdp_opt *conn,
 		      struct ib_device *device,
 		      u8 hw_port,
@@ -1211,8 +1243,9 @@ error_attr:
 	return result;
 } /* sdp_conn_alloc_ib  */
 
-/* ========================================================================= */
-/*..sdp_conn_alloc -- allocate a new socket, and init. */
+/*
+ * sdp_conn_alloc - allocate a new socket, and init.
+ */
 struct sdp_opt *sdp_conn_alloc(int priority)
 {
 	struct sdp_opt *conn;
@@ -1420,11 +1453,9 @@ error:
 	return NULL;
 } /* sdp_conn_alloc  */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* module public functions                                               */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * module public functions
+ */
 
 #define SDP_CONN_PROC_MAIN_SIZE  183 /* output line size. */
 #define SDP_PROC_CONN_MAIN_HEAD \
@@ -1442,8 +1473,9 @@ error:
 	"%04x %08x %04x %08x%08x %08x%08x %04x %04x "         \
 	"%06x %06x %08x%08x %08x%08x %08x%08x %08x%08x %08x %08x\n"
 
-/* ========================================================================= */
-/*..sdp_proc_dump_conn_main - dump the connection table to /proc */
+/*
+ * sdp_proc_dump_conn_main - dump the connection table to /proc
+ */
 int sdp_proc_dump_conn_main(char *buffer, 
 			    int   max_size, 
 			    off_t start_index, 
@@ -1556,8 +1588,9 @@ done:
 	"%08x %04x %04x %08x %04x %04x %04x %02x %02x " \
 	"%02x %02x %02x %04x %04x %04x %02x %02x %08x %04x %04x %04x\n"
 
-/* ========================================================================= */
-/*..sdp_proc_dump_conn_data - dump the connection table to /proc */
+/*
+ * sdp_proc_dump_conn_data - dump the connection table to /proc
+ */
 int sdp_proc_dump_conn_data(char *buffer,
 			    int   max_size,
 			    off_t start_index,
@@ -1661,8 +1694,9 @@ done:
 	"%04x %02x %02x %02x %02x %02x %02x %02x " \
 	"%04x %04x %04x %04x %04x %02x %02x %02x %02x %08x %08x %08x %08x\n"
 
-/* ========================================================================= */
-/*..sdp_proc_dump_conn_rdma - dump the connection table to /proc */
+/*
+ * sdp_proc_dump_conn_rdma - dump the connection table to /proc
+ */
 int sdp_proc_dump_conn_rdma(char *buffer, 
 			    int   max_size,
 			    off_t start_index,
@@ -1748,8 +1782,9 @@ done:
 #define SDP_PROC_CONN_SOPT_FORM \
 	"%02x.%02x.%02x.%02x:%04x %02x.%02x.%02x.%02x:%04x %08x %08x %04x\n"
 
-/* ========================================================================= */
-/*..sdp_proc_dump_conn_sopt - dump the options portion of each conn to /proc */
+/*
+ * sdp_proc_dump_conn_sopt - dump the options portion of each conn to /proc
+ */
 int sdp_proc_dump_conn_sopt(char *buffer,
 			    int   max_size,
 			    off_t start_index,
@@ -1819,8 +1854,9 @@ done:
 	return offset;
 } /* sdp_proc_dump_conn_sopt */
 
-/* ========================================================================= */
-/*..sdp_proc_dump_device - dump the primary device table to /proc */
+/*
+ * sdp_proc_dump_device - dump the primary device table to /proc
+ */
 int sdp_proc_dump_device(char *buffer,
 			 int   max_size,
 			 off_t start_index,
@@ -1867,14 +1903,13 @@ int sdp_proc_dump_device(char *buffer,
 	return offset;
 } /* sdp_proc_dump_device */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* initialization/cleanup functions                                      */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * initialization/cleanup functions
+ */
 
-/* ========================================================================= */
-/*.._sdp_device_table_init -- create hca list */
+/*
+ * sdp_device_init_one - add a device to the list
+ */
 static void sdp_device_init_one(struct ib_device *device)
 {
 	struct ib_fmr_pool_param fmr_param_s;
@@ -2009,10 +2044,11 @@ error:
 		(void)ib_dealloc_pd(hca->pd);
 
 	kfree(hca);
-} /* _sdp_device_table_init */
+} /* sdp_device_init_one */
 
-/* ========================================================================= */
-/*.._sdp_device_table_cleanup -- delete hca list */
+/*
+ * sdp_device_remove_one - remove a device from the hca list
+ */
 static void sdp_device_remove_one(struct ib_device *device)
 {
 	struct sdev_hca_port *port;
@@ -2046,10 +2082,11 @@ static void sdp_device_remove_one(struct
 		(void)ib_dealloc_pd(hca->pd);
 
 	kfree(hca);
-} /* _sdp_device_table_cleanup */
+} /* sdp_device_remove_one */
 
-/* ========================================================================= */
-/*..sdp_conn_table_init -- create a sdp connection table */
+/*
+ * sdp_conn_table_init - create a sdp connection table
+ */
 int sdp_conn_table_init(int proto_family,
 			int conn_size,
 			int recv_post_max,
@@ -2188,8 +2225,9 @@ error_hca:
 	return result;
 } /* sdp_conn_table_init */
 
-/* ========================================================================= */
-/*..sdp_conn_table_clear -- destroy connection managment and tables */
+/*
+ * sdp_conn_table_clear - destroy connection managment and tables
+ */
 int sdp_conn_table_clear(void)
  {
 	sdp_dbg_init("Deleting connection tables.");



From Tom.Duffy at Sun.COM  Thu Feb 17 14:09:52 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:09:52 -0800
Subject: [openib-general] [PATCH][SDP][8/33] Fix comment style in sdp_conn.h
In-Reply-To: <11086781814000@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
Message-ID: <11086781921405@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_conn.h	2005-02-14 11:13:15.571961000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_conn.h	2005-02-17 12:44:49.618014000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_conn.h 488 2004-07-16 23:40:31Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_CONN_H
 #define _SDP_CONN_H
@@ -27,9 +38,11 @@
 #include "sdp_advt.h"
 #include "sdp_iocb.h"
 #include "sdp_dev.h"
-/* ---------------------------------------------------------------------- */
-/* SDP connection specific definitions                                    */
-/* ---------------------------------------------------------------------- */
+
+/*
+ * SDP connection specific definitions
+ */
+
 /*
  * definitions
  */
@@ -56,6 +69,7 @@ enum sdp_mode {
 	SDP_MODE_PIPE = 0x02,
 	SDP_MODE_ERROR = 0x03
 };
+
 /*
  * First two bytes are the primary state values. Third byte is a bit
  * field used for different mask operations, defined below. Fourth
@@ -89,6 +103,7 @@ enum sdp_mode {
 #define SDP_CONN_ST_ERROR_STRM  0xFD01 /* gateway error, waiting on CM */
 #define SDP_CONN_ST_CLOSED      0xFE03 /* not connected */
 #define SDP_CONN_ST_INVALID     0xFF03 /* not connected */
+
 /*
  * states masks for SDP
  */
@@ -98,11 +113,13 @@ enum sdp_mode {
 #define SDP_ST_MASK_CTRL_OK   0x0020	/* posting control for send */
 #define SDP_ST_MASK_DIS_PEND  0x0040	/* disconnect transmission pending. */
 #define SDP_ST_MASK_RCV_POST  0x0080	/* posting IB recv's is allowed. */
+
 /*
  * transition one of the disconnect pending states to disconnect send
  */
 #define SDP_ST_PEND_2_SEND(conn) \
         (conn)->state = ((conn)->state + 0x0100) & ~SDP_ST_MASK_DIS_PEND;
+
 /*
  * internal connection structure
  */
@@ -116,16 +133,19 @@ enum sdp_mode {
 #define SDP_SOCK_ST_CLOSING     0x0800
 #define SDP_SOCK_ST_LISTEN      0x0901
 #define SDP_SOCK_ST_ERROR       0xFF01
+
 /*
  * state masks.
  */
 #define SDP_ST_MASK_CLOSED 0x0001 /* socket is not referenced by the GW. */
 #define SDP_ST_MASK_SEND   0x0004 /* valid state for API send */
 #define SDP_ST_MASK_OPEN   0x0008 /* send window is valid (writeable) */
+
 /*
  * event dispatch table
  */
 #define SDP_MSG_EVENT_TABLE_SIZE 0x20
+
 /*
  * connection handle within a socket.
  */
@@ -179,6 +199,7 @@ struct sdp_conn_state {
 #define SDP_CONN_ST_SET(conn, val) (conn)->state = (val)
 #define SDP_CONN_ST_INIT(conn)     (conn)->state = SDP_CONN_ST_INVALID
 #endif
+
 /*
  * connection lock
  */
@@ -187,6 +208,7 @@ struct sdp_conn_lock {
 	spinlock_t slock;
 	wait_queue_head_t waitq;
 }; /* struct sdp_conn_lock */
+
 /*
  * SDP Connection structure.
  */
@@ -395,6 +417,7 @@ struct sdp_opt {
 #define SDP_WRAP_LT(x, y) ((signed int)((x) - (y)) < 0)
 #define SDP_WRAP_GTE(x, y) ((signed int)((x) - (y)) >= 0)
 #define SDP_WRAP_LTE(x, y) ((signed int)((x) - (y)) <= 0)
+
 /*
  * statistics.
  */
@@ -434,12 +457,10 @@ struct sdp_opt {
 #define SDP_CONN_STAT_RECV_MID_INC(conn, mid)
 
 #endif
-/* ---------------------------------------------------------------------- */
-/*                                                                        */
-/* SDP connection lock                                                    */
-/*                                                                        */
-/* ---------------------------------------------------------------------- */
 
+/*
+ * SDP connection lock
+ */
 #define SDP_CONN_LOCK_BH(conn) \
         spin_lock_irqsave(&((conn)->lock.slock), flags)
 #define SDP_CONN_UNLOCK_BH(conn) \
@@ -487,12 +508,3 @@ static inline void *__hashent_arg(s32 ha
 }
 
 #endif /* _SDP_CONN_H */
-
-
-
-
-
-
-
-
-



From Tom.Duffy at Sun.COM  Thu Feb 17 14:10:02 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:10:02 -0800
Subject: [openib-general] [PATCH][SDP][9/33] Fix comment style in sdp_dev.h
In-Reply-To: <11086781921405@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com> <11086781921405@sun.com>
Message-ID: <11086782023076@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_dev.h	2005-02-11 17:27:53.697025000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_dev.h	2005-02-17 10:40:47.692004000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_dev.h 1032 2004-10-20 23:17:34Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_DEV_H
 #define _SDP_DEV_H
@@ -100,11 +111,11 @@
 #define SDP_SEND_POST_FRACTION   0x06
 #define SDP_SEND_POST_SLOW   0x01
 #define SDP_SEND_POST_COUNT  0x0A
-/* ---------------------------------------------------------------------- */
-/*                                                                        */
-/* SDP experimental parameters.                                           */
-/*                                                                        */
-/* ---------------------------------------------------------------------- */
+
+/*
+ * SDP experimental parameters.
+ */
+
 /*
  * maximum consecutive unsignalled send events.
  * (crap, watch out for deactivated nodelay!)
@@ -124,11 +135,10 @@
  */
 #define SDP_RECV_POST_FREQ 0x08	/* rate for posting new recv buffs */
 #define SDP_RECV_POST_ACK  0x08	/* rate for posting ack windows. */
-/* ---------------------------------------------------------------------- */
-/*                                                                        */
-/* SDP root device structure.                                             */
-/*                                                                        */
-/* ---------------------------------------------------------------------- */
+
+/*
+ * SDP root device structure
+ */
 struct sdev_hca_port {
 	u8                    index; /* port ID */
 	union ib_gid          gid;   /* port GID */



From Tom.Duffy at Sun.COM  Thu Feb 17 14:10:12 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:10:12 -0800
Subject: [openib-general] [PATCH][SDP][10/33] Fix comment style in
	sdp_event.c
In-Reply-To: <11086782023076@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
Message-ID: <11086782122447@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_event.c	2005-02-15 16:33:46.254005000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_event.c	2005-02-17 10:43:35.172004000 -0800
@@ -1,35 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_event.c 836 2004-09-14 20:50:32Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Primary QP Event Handler                                              */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_cq_event_locked -- main per QP event handler */
+/*
+ * Primary QP Event Handler
+ */
+
+/*
+ * sdp_cq_event_locked - main per QP event handler
+ */
 int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_opt *conn)
 {
 	int result = 0;
@@ -126,8 +137,9 @@ done:
 	return result;
 } /* sdp_cq_event_locked */
 
-/* ========================================================================= */
-/*..sdp_cq_event_handler -- main per QP event handler, and demuxer */
+/*
+ * sdp_cq_event_handler - main per QP event handler, and demuxer
+ */
 void sdp_cq_event_handler(struct ib_cq *cq, void *arg)
 {
 	s32 hashent = (unsigned long)arg;
@@ -200,14 +212,13 @@ done:
 	return;
 } /* sdp_cq_event_handler */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Connection establishment IB/CM callback functions                     */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * Connection establishment IB/CM callback functions
+ */
 
-/* ========================================================================= */
-/*.._sdp_cm_idle -- handler for connection idle completion */
+/*
+ * _sdp_cm_idle - handler for connection idle completion
+ */
 static int _sdp_cm_idle(struct ib_cm_id *cm_id,
 			struct ib_cm_event *event,
 			struct sdp_opt *conn)
@@ -310,8 +321,9 @@ error:
 	return result;
 } /* _sdp_cm_idle */
 
-/* ========================================================================= */
-/*.._sdp_cm_established -- handler for connection established completion */
+/*
+ * _sdp_cm_established - handler for connection established completion
+ */
 static int _sdp_cm_established(struct ib_cm_id *cm_id,
 			       struct ib_cm_event *event,
 			       struct sdp_opt *conn)
@@ -432,8 +444,9 @@ done:
 	return result;
 } /* _sdp_cm_established */
 
-/* ========================================================================= */
-/*.._sdp_cm_timewait -- handler for connection Time Wait completion */
+/*
+ * _sdp_cm_timewait - handler for connection Time Wait completion
+ */
 static int _sdp_cm_timewait(struct ib_cm_id *cm_id,
 			    struct ib_cm_event *event,
 			    struct sdp_opt *conn)
@@ -518,13 +531,13 @@ error:
 	return result;
 } /* _sdp_cm_timewait */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Primary Connection Managment callback function                        */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_cm_event_handler -- handler for CM state transitions request */
+/*
+ * Primary Connection Managment callback function
+ */
+
+/*
+ * sdp_cm_event_handler - handler for CM state transitions request
+ */
 int sdp_cm_event_handler(struct ib_cm_id *cm_id,
 			 struct ib_cm_event *event)
 {



From Tom.Duffy at Sun.COM  Thu Feb 17 14:10:22 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:10:22 -0800
Subject: [openib-general] [PATCH][SDP][11/33] Fix comment style in sdp_inet.c
In-Reply-To: <11086782122447@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
Message-ID: <11086782221006@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_inet.c	2005-02-11 17:50:31.980000000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_inet.c	2005-02-17 10:51:06.250001000 -0800
@@ -1,29 +1,41 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_inet.c 994 2004-10-14 05:38:39Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
-#include "sdp_main.h"
 #include <linux/tcp.h>
 #include <asm/ioctls.h>
+#include "sdp_main.h"
+
 /*
  * list of connections waiting for an incomming connection
  */
@@ -79,13 +91,14 @@ module_param(debug_level, int, 0);
  *    unsigned short     max_ack_backlog; (accept max backlog)
  * };
  */
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Notification of significant events.                                   */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_inet_wake_send -- test, set, and notify socket of write space */
+
+/*
+ * Notification of significant events
+ */
+
+/*
+ * sdp_inet_wake_send - test, set, and notify socket of write space
+ */
 void sdp_inet_wake_send(struct sock *sk)
 {
 	struct sdp_opt *conn;
@@ -118,8 +131,9 @@ void sdp_inet_wake_send(struct sock *sk)
 	return;
 } /* sdp_inet_wake_send */
 
-/* ========================================================================= */
-/*..sdp_inet_wake_generic -- wake up a socket */
+/*
+ * sdp_inet_wake_generic - wake up a socket
+ */
 void sdp_inet_wake_generic(struct sock *sk)
 {
 	if (NULL != sk) {
@@ -138,8 +152,9 @@ void sdp_inet_wake_generic(struct sock *
 	return;
 } /* sdp_inet_wake_generic */
 
-/* ========================================================================= */
-/*..sdp_inet_wake_recv -- wake up a socket for read */
+/*
+ * sdp_inet_wake_recv - wake up a socket for read
+ */
 void sdp_inet_wake_recv(struct sock *sk, int len)
 {
 	if (NULL != sk) {
@@ -157,8 +172,9 @@ void sdp_inet_wake_recv(struct sock *sk,
 	return;
 } /* sdp_inet_wake_recv */
 
-/* ========================================================================= */
-/*..sdp_inet_wake_error -- wake up a socket for error */
+/*
+ * sdp_inet_wake_error - wake up a socket for error
+ */
 void sdp_inet_wake_error(struct sock *sk)
 {
 	if (NULL != sk) {
@@ -176,8 +192,9 @@ void sdp_inet_wake_error(struct sock *sk
 	return;
 } /* sdp_inet_wake_error */
 
-/* ========================================================================= */
-/*..sdp_inet_wake_urg -- wake up a socket for urgent data */
+/*
+ * sdp_inet_wake_urg - wake up a socket for urgent data
+ */
 void sdp_inet_wake_urg(struct sock *sk)
 {
 	/*
@@ -191,14 +208,13 @@ void sdp_inet_wake_urg(struct sock *sk)
 	return;
 } /* sdp_inet_wake_urg */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* internal socket/handle managment functions                            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * internal socket/handle managment functions
+ */
 
-/* ========================================================================= */
-/*.._sdp_inet_abort -- abort an existing connection. */
+/*
+ * _sdp_inet_abort - abort an existing connection
+ */
 static int _sdp_inet_abort(struct sdp_opt *conn)
 {
 	int result;
@@ -250,8 +266,9 @@ static int _sdp_inet_abort(struct sdp_op
 	return result;
 } /* _sdp_inet_abort */
 
-/* ========================================================================= */
-/*.._sdp_inet_disconnect -- disconnect a connection. */
+/*
+ * _sdp_inet_disconnect - disconnect a connection
+ */
 static int _sdp_inet_disconnect(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -322,13 +339,13 @@ static int _sdp_inet_disconnect(struct s
 	return result;
 } /* _sdp_inet_disconnect */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Linux SOCKET interface, module specific functions                     */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_inet_release -- release/close a socket */
+/*
+ * Linux SOCKET interface, module specific functions
+ */
+
+/*
+ * _sdp_inet_release - release/close a socket
+ */
 static int _sdp_inet_release(struct socket *sock)
 {
 	struct sdp_opt *conn;
@@ -488,11 +505,11 @@ done:
 	return 0;
 } /* _sdp_inet_release */
 
-/* ========================================================================= */
-/*.._sdp_inet_bind -- bind a socket to an address/interface */
-static int _sdp_inet_bind(struct socket *sock,
-			     struct sockaddr *uaddr,
-			     int size)
+/*
+ * _sdp_inet_bind - bind a socket to an address/interface
+ */
+static int _sdp_inet_bind(struct socket *sock, struct sockaddr *uaddr,
+			  int size)
 {
 	struct sockaddr_in *addr = (struct sockaddr_in *)uaddr;
 	struct sock *sk;
@@ -600,8 +617,9 @@ done:
 	return result;
 } /* _sdp_inet_bind */
 
-/* ========================================================================= */
-/*.._sdp_inet_connect -- connect a socket to a remote address */
+/*
+ * _sdp_inet_connect - connect a socket to a remote address
+ */
 static int _sdp_inet_connect(struct socket *sock,
 				struct sockaddr *uaddr,
 				int size,
@@ -791,8 +809,9 @@ done:
 	return result;
 } /* _sdp_inet_connect */
 
-/* ========================================================================= */
-/*.._sdp_inet_listen -- listen on a socket for incomming addresses. */
+/*
+ * _sdp_inet_listen - listen on a socket for incomming addresses
+ */
 static int _sdp_inet_listen(struct socket *sock, int backlog)
 {
 	struct sock *sk;
@@ -855,8 +874,9 @@ done:
 	return result;
 } /* _sdp_inet_listen */
 
-/* ========================================================================= */
-/*.._sdp_inet_accept -- accept a new socket from a listen socket. */
+/*
+ * _sdp_inet_accept - accept a new socket from a listen socket
+ */
 static int _sdp_inet_accept(struct socket *listen_sock,
 			       struct socket *accept_sock, 
 			       int flags)
@@ -1035,12 +1055,11 @@ listen_done:
 	return result;
 } /* _sdp_inet_accept */
 
-/* ========================================================================= */
-/*.._sdp_inet_getname -- return a sockets address information */
-static int _sdp_inet_getname(struct socket *sock,
-				struct sockaddr *uaddr,
-				int *size, 
-				int peer)
+/*
+ * _sdp_inet_getname - return a sockets address information
+ */
+static int _sdp_inet_getname(struct socket *sock, struct sockaddr *uaddr,
+			     int *size, int peer)
 {
 	struct sockaddr_in *addr = (struct sockaddr_in *)uaddr;
 	struct sock *sk;
@@ -1083,8 +1102,9 @@ static int _sdp_inet_getname(struct sock
 	return 0;
 } /* _sdp_inet_getname */
 
-/* ========================================================================= */
-/*.._sdp_inet_poll -- poll a socket for activity */
+/*
+ * _sdp_inet_poll - poll a socket for activity
+ */
 static unsigned int _sdp_inet_poll(struct file *file,
 				   struct socket *sock,
 				   poll_table *wait)
@@ -1184,11 +1204,11 @@ static unsigned int _sdp_inet_poll(struc
 	return mask;
 } /* _sdp_inet_poll */
 
-/* ========================================================================= */
-/*.._sdp_inet_ioctl -- serivce an ioctl request on a socket */
-static int _sdp_inet_ioctl(struct socket *sock,
-			      unsigned int cmd,
-			      unsigned long arg)
+/*
+ * _sdp_inet_ioctl - serivce an ioctl request on a socket
+ */
+static int _sdp_inet_ioctl(struct socket *sock, unsigned int cmd,
+			   unsigned long arg)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1329,13 +1349,11 @@ static int _sdp_inet_ioctl(struct socket
 	return result;
 } /* _sdp_inet_ioctl */
 
-/* ========================================================================= */
-/*.._sdp_inet_setopt -- set a socket option. */
-static int _sdp_inet_setopt(struct socket *sock,
-			       int level, 
-			       int optname, 
-			       char __user *optval, 
-			       int optlen)
+/*
+ * _sdp_inet_setopt - set a socket option
+ */
+static int _sdp_inet_setopt(struct socket *sock, int level, int optname, 
+			    char __user *optval, int optlen)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1417,13 +1435,11 @@ static int _sdp_inet_setopt(struct socke
 	return result;
 } /* _sdp_inet_setopt */
 
-/* ========================================================================= */
-/*.._sdp_inet_getopt -- get a socket option. */
-static int _sdp_inet_getopt(struct socket *sock,
-			       int level,
-			       int optname,
-			       char __user *optval,
-			       int __user *optlen)
+/*
+ * _sdp_inet_getopt - get a socket option
+ */
+static int _sdp_inet_getopt(struct socket *sock, int level, int optname,
+			    char __user *optval, int __user *optlen)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1503,8 +1519,9 @@ static int _sdp_inet_getopt(struct socke
 	return 0;
 } /* _sdp_inet_getopt */
 
-/* ========================================================================= */
-/*.._sdp_inet_shutdown -- shutdown a socket. */
+/*
+ * _sdp_inet_shutdown - shutdown a socket
+ */
 static int _sdp_inet_shutdown(struct socket *sock, int flag)
 {
 	int result = 0;
@@ -1616,11 +1633,9 @@ static int _sdp_inet_shutdown(struct soc
 	return result;
 } /* _sdp_inet_shutdown */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Primary socket initialization                                         */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * Primary socket initialization
+ */
 struct proto_ops _lnx_stream_ops = {
 	.family     =  AF_INET_SDP,
 	.release    = _sdp_inet_release,
@@ -1642,9 +1657,9 @@ struct proto_ops _lnx_stream_ops = {
 	.owner      =  THIS_MODULE,
 }; /* _lnx_stream_ops */
 
-
-/* ========================================================================= */
-/*.._sdp_inet_create -- create a socket */
+/*
+ * _sdp_inet_create - create a socket
+ */
 static int _sdp_inet_create(struct socket *sock, int protocol)
 {
 	struct sdp_opt *conn;
@@ -1689,24 +1704,22 @@ static int _sdp_inet_create(struct socke
 	return 0;
 } /* _sdp_inet_create */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* INET module initialization functions                                  */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * INET module initialization functions
+ */
 static struct net_proto_family _sdp_proto = {
 	.family =  AF_INET_SDP,
 	.create = _sdp_inet_create,
 	.owner  =  THIS_MODULE,
 }; /* _pf_family */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* SDP host module load/unload functions                                 */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_init -- initialize the sdp host module */
+/*
+ * SDP host module load/unload functions
+ */
+
+/*
+ * sdp_init - initialize the sdp host module
+ */
 static int __init sdp_init(void)
 {
 	int result = 0;
@@ -1810,8 +1823,9 @@ error_proc:
 	return result;		/* success */
 } /* sdp_init */
 
-/* ========================================================================= */
-/*..sdp_exit -- cleanup the sdp host module */
+/*
+ * sdp_exit - cleanup the sdp host module
+ */
 static void __exit sdp_exit(void)
 {
 	sdp_dbg_init("INIT: SDP module unload.");



From Tom.Duffy at Sun.COM  Thu Feb 17 14:10:33 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:10:33 -0800
Subject: [openib-general] [PATCH][SDP][12/33] Fix comment style in sdp_iocb.c
In-Reply-To: <11086782221006@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com> <11086782221006@sun.com>
Message-ID: <1108678233852@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_iocb.c	2005-02-14 11:13:15.582961000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_iocb.c	2005-02-17 12:21:44.892003000 -0800
@@ -1,42 +1,53 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_iocb.c 696 2004-08-28 03:10:21Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
 static kmem_cache_t *__sdp_iocb_cache = NULL;
 static kmem_cache_t *__sdp_iocb_table = NULL;
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* memory locking functions                                              */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+
+/*
+ * memory locking functions
+ */
 #include <linux/utsname.h>
 
 typedef int (*do_mlock_ptr_t)(unsigned long, size_t, int);
 static do_mlock_ptr_t _mlock_ptr = NULL;
 
-/* ========================================================================= */
-/*.._sdp_iocb_unlock -- unlock the memory for an IOCB. */
+/*
+ * _sdp_iocb_unlock - unlock the memory for an IOCB
+ */
 static int _sdp_iocb_unlock(struct sdpc_iocb *iocb)
 {
 	struct vm_area_struct *vma;
@@ -91,8 +102,9 @@ static int _sdp_iocb_unlock(struct sdpc_
 	return 0;
 } /* _sdp_iocb_unlock */
 
-/* ========================================================================= */
-/*..sdp_iocb_unlock -- unlock the memory for an IOCB. */
+/*
+ * sdp_iocb_unlock - unlock the memory for an IOCB
+ */
 int sdp_iocb_unlock(struct sdpc_iocb *iocb)
 {
 	int result;
@@ -128,8 +140,9 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 	return result;
 } /* sdp_iocb_unlock */
 
-/* ========================================================================= */
-/*.._sdp_iocb_page_save -- save page information for an IOCB. */
+/*
+ * _sdp_iocb_page_save - save page information for an IOCB
+ */
 static int _sdp_iocb_page_save(struct sdpc_iocb *iocb)
 {
 	unsigned int counter;
@@ -241,8 +254,9 @@ err_addr:
 	return result;
 } /* _sdp_iocb_page_save */
 
-/* ========================================================================= */
-/*..sdp_iocb_lock -- lock the memory for an IOCB. */
+/*
+ * sdp_iocb_lock - lock the memory for an IOCB
+ */
 int sdp_iocb_lock(struct sdpc_iocb *iocb)
 {
 	struct vm_area_struct *vma;
@@ -367,11 +381,9 @@ err_lock:
 	return result;
 } /* sdp_iocb_lock */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* IOCB memory locking init functions                                    */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * IOCB memory locking init functions
+ */
 struct kallsym_iter
 {
   loff_t pos;
@@ -382,12 +394,10 @@ struct kallsym_iter
   char name[128];
 };
 
-/* ========================================================================= */
-/*.._sdp_mem_lock_init -- initialize the userspace memory locking. */
-static int _sdp_mem_lock_init
-(
- void
-)
+/*
+ * _sdp_mem_lock_init - initialize the userspace memory locking
+ */
+static int _sdp_mem_lock_init(void)
 {
 	struct file *kallsyms;
 	struct seq_file *seq;
@@ -436,12 +446,10 @@ done:
 	return ret;
 } /* _sdp_mem_lock_init */
 
-/* ========================================================================= */
-/*.._sdp_mem_lock_cleanup -- cleanup the memory locking tables. */
-static int _sdp_mem_lock_cleanup
-(
- void
-)
+/*
+ * _sdp_mem_lock_cleanup - cleanup the memory locking tables
+ */
+static int _sdp_mem_lock_cleanup(void)
 {
 	sdp_dbg_init("Memory Locking cleanup.");
 	/*
@@ -452,13 +460,13 @@ static int _sdp_mem_lock_cleanup
 	return 0;
 } /* _sdp_mem_lock_cleanup */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* IOCB memory registration functions                                    */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_iocb_register -- register an IOCBs memory for advertisment */
+/*
+ * IOCB memory registration functions
+ */
+
+/*
+ * sdp_iocb_register - register an IOCBs memory for advertisment
+ */
 int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn)
 {
 	int result;
@@ -515,8 +523,9 @@ error:
 	return result;
 } /* sdp_iocb_register */
 
-/* ========================================================================= */
-/*..sdp_iocb_release -- unregister an IOCBs memory  */
+/*
+ * sdp_iocb_release - unregister an IOCBs memory
+ */
 int sdp_iocb_release(struct sdpc_iocb *iocb)
 {
 	int result;
@@ -540,8 +549,9 @@ int sdp_iocb_release(struct sdpc_iocb *i
 	return 0;
 } /* sdp_iocb_release */
 
-/* ========================================================================= */
-/*.._sdp_iocb_complete -- complete an IOCB for real in thread context */
+/*
+ * _sdp_iocb_complete - complete an IOCB for real in thread context
+ */
 static void _sdp_iocb_complete(void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *)arg;
@@ -591,8 +601,9 @@ static void _sdp_iocb_complete(void *arg
 	return;
 } /* _sdp_iocb_complete */
 
-/* ========================================================================= */
-/*..sdp_iocb_complete -- complete an IOCB */
+/*
+ * sdp_iocb_complete - complete an IOCB
+ */
 int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status)
 {
 	SDP_CHECK_NULL(iocb, -EINVAL);
@@ -615,13 +626,13 @@ int sdp_iocb_complete(struct sdpc_iocb *
 	return 0;
 } /* sdp_iocb_complete */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* IOCB object managment                                                 */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_iocb_q_remove - remove the object from the table */
+/*
+ * IOCB object managment
+ */
+
+/*
+ * sdp_iocb_q_remove - remove the object from the table
+ */
 int sdp_iocb_q_remove(struct sdpc_iocb *iocb)
 {
 	struct sdpc_iocb_q *table;
@@ -662,8 +673,9 @@ int sdp_iocb_q_remove(struct sdpc_iocb *
 	return 0;
 } /* sdp_iocb_q_remove */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_lookup - find an iocb based on key, without removing */
+/*
+ * sdp_iocb_q_lookup - find an iocb based on key, without removing
+ */
 struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key)
 {
 	struct sdpc_iocb *iocb = NULL;
@@ -684,8 +696,9 @@ struct sdpc_iocb *sdp_iocb_q_lookup(stru
 	return NULL;
 } /* sdp_iocb_q_lookup */
 
-/* ========================================================================= */
-/*..sdp_iocb_create - create an IOCB object */
+/*
+ * sdp_iocb_create - create an IOCB object
+ */
 struct sdpc_iocb *sdp_iocb_create(void)
 {
 	struct sdpc_iocb *iocb;
@@ -705,8 +718,9 @@ struct sdpc_iocb *sdp_iocb_create(void)
 	return iocb;
 } /* sdp_iocb_create */
 
-/* ========================================================================= */
-/*..sdp_iocb_destroy - destroy an IOCB object */
+/*
+ * sdp_iocb_destroy - destroy an IOCB object
+ */
 int sdp_iocb_destroy(struct sdpc_iocb *iocb)
 {
 	if (NULL == iocb) {
@@ -753,8 +767,9 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
 	return 0;
 } /* sdp_iocb_destroy */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_look - get, without removing, the object at the head */
+/*
+ * sdp_iocb_q_look - get, without removing, the object at the head
+ */
 struct sdpc_iocb *sdp_iocb_q_look(struct sdpc_iocb_q *table)
 {
 	SDP_CHECK_NULL(table, NULL);
@@ -762,8 +777,9 @@ struct sdpc_iocb *sdp_iocb_q_look(struct
 	return table->head;
 } /* sdp_iocb_q_look */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_get_key - find an iocb based on key, and remove it */
+/*
+ * sdp_iocb_q_get_key - find an iocb based on key, and remove it
+ */
 struct sdpc_iocb *sdp_iocb_q_get_key(struct sdpc_iocb_q *table, u32 key)
 {
 	struct sdpc_iocb *iocb;
@@ -785,8 +801,9 @@ done:
 	return iocb;
 } /* sdp_iocb_q_get_key */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_get - get, and remove, the object at the tables head */
+/*
+ * _sdp_iocb_q_get - get, and remove, the object at the tables head
+ */
 static struct sdpc_iocb *_sdp_iocb_q_get(struct sdpc_iocb_q *table, int head)
 {
 	struct sdpc_iocb *iocb;
@@ -832,8 +849,9 @@ static struct sdpc_iocb *_sdp_iocb_q_get
 	return iocb;
 } /* _sdp_iocb_q_get */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_put - put the IOCB object at the tables tail */
+/*
+ * _sdp_iocb_q_put - put the IOCB object at the tables tail
+ */
 int _sdp_iocb_q_put(struct sdpc_iocb_q *table,
 			      struct sdpc_iocb *iocb,
 			      int head)
@@ -872,36 +890,41 @@ int _sdp_iocb_q_put(struct sdpc_iocb_q *
 	return 0;
 } /* _sdp_iocb_q_put */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_get_tail - get an IOCB object from the tables tail */
+/*
+ * sdp_iocb_q_get_tail - get an IOCB object from the tables tail
+ */
 struct sdpc_iocb *sdp_iocb_q_get_tail(struct sdpc_iocb_q *table)
 {
 	return _sdp_iocb_q_get(table, 0);
 } /* sdp_iocb_q_get_tail */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_get_head - get an IOCB object from the tables head */
+/*
+ * sdp_iocb_q_get_head - get an IOCB object from the tables head
+ */
 struct sdpc_iocb *sdp_iocb_q_get_head(struct sdpc_iocb_q *table)
 {
 	return _sdp_iocb_q_get(table, 1);
 } /* sdp_iocb_q_get_head */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_put_tail - put the IOCB object at the tables tail */
+/*
+ * sdp_iocb_q_put_tail - put the IOCB object at the tables tail
+ */
 int sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb)
 {
 	return _sdp_iocb_q_put(table, iocb, 0);
 } /* sdp_iocb_q_put_tail */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_put_head - put the IOCB object at the tables head */
+/*
+ * sdp_iocb_q_put_head - put the IOCB object at the tables head
+ */
 int sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb)
 {
 	return _sdp_iocb_q_put(table, iocb, 1);
 } /* sdp_iocb_q_put_head */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_cancel -- cancel all outstanding AIOs in a queue */
+/*
+ * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
+ */
 int sdp_iocb_q_cancel(struct sdpc_iocb_q *table,
 		      u32     mask,
 		      ssize_t comp)
@@ -942,13 +965,13 @@ int sdp_iocb_q_cancel(struct sdpc_iocb_q
 	return 0;
 } /* sdp_iocb_q_cancel */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* public table functions                                                */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_iocb_q_create - create an IOCB table */
+/*
+ * public table functions
+ */
+
+/*
+ * sdp_iocb_q_create - create an IOCB table
+ */
 struct sdpc_iocb_q *sdp_iocb_q_create(int *result)
 {
 	struct sdpc_iocb_q *table = NULL;
@@ -967,8 +990,9 @@ struct sdpc_iocb_q *sdp_iocb_q_create(in
 	return table;
 } /* sdp_iocb_q_create */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_init - initialize a new empty IOCB table */
+/*
+ * sdp_iocb_q_init - initialize a new empty IOCB table
+ */
 int sdp_iocb_q_init(struct sdpc_iocb_q *table)
 {
 	SDP_CHECK_NULL(table, -EINVAL);
@@ -979,8 +1003,9 @@ int sdp_iocb_q_init(struct sdpc_iocb_q *
 	return 0;
 } /* sdp_iocb_q_init */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_clear - clear the contents of an IOCB table */
+/*
+ * sdp_iocb_q_clear - clear the contents of an IOCB table
+ */
 int sdp_iocb_q_clear(struct sdpc_iocb_q *table)
 {
 	struct sdpc_iocb *iocb;
@@ -999,8 +1024,9 @@ int sdp_iocb_q_clear(struct sdpc_iocb_q 
 	return 0;
 } /* sdp_iocb_q_clear */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_destroy - destroy an IOCB table */
+/*
+ * sdp_iocb_q_destroy - destroy an IOCB table
+ */
 int sdp_iocb_q_destroy(struct sdpc_iocb_q *table)
 {
 	int result;
@@ -1019,13 +1045,13 @@ int sdp_iocb_q_destroy(struct sdpc_iocb_
 	return 0;
 } /* sdp_iocb_q_destroy */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* primary initialization/cleanup functions                              */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_main_iocb_init -- initialize the advertisment caches. */
+/*
+ * primary initialization/cleanup functions
+ */
+
+/*
+ * sdp_main_iocb_init - initialize the advertisment caches
+ */
 int sdp_main_iocb_init(void)
 {
 	int result;
@@ -1079,8 +1105,9 @@ error_iocb_c:
 	return result;
 } /* sdp_main_iocb_init */
 
-/* ========================================================================= */
-/*..sdp_main_iocb_cleanup -- cleanup the advertisment caches. */
+/*
+ * sdp_main_iocb_cleanup - cleanup the advertisment caches
+ */
 int sdp_main_iocb_cleanup(void)
 {
 	sdp_dbg_init("IOCB cache cleanup.");



From Tom.Duffy at Sun.COM  Thu Feb 17 14:10:43 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:10:43 -0800
Subject: [openib-general] [PATCH][SDP][13/33] Fix comment style in sdp_iocb.h
In-Reply-To: <1108678233852@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
Message-ID: <11086782432384@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_iocb.h	2005-02-10 18:42:56.974015000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_iocb.h	2005-02-17 10:55:51.305015000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_iocb.h 696 2004-08-28 03:10:21Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_IOCB_H
 #define _SDP_IOCB_H
@@ -53,9 +64,10 @@
  */
 #define sdp_iocb_q_size(table) ((table)->size)
 
-/* ----------------------------------------------------------------------- */
-/* INET read/write IOCBs                                                   */
-/* ----------------------------------------------------------------------- */
+/*
+ * INET read/write IOCBs
+ */
+
 /*
  * save a kvec read/write for processing once data shows up.
  */
@@ -103,6 +115,7 @@ struct sdpc_iocb {
 	struct kiocb *req;
 	struct sock_iocb *si;
 }; /* struct sdpc_iocb */
+
 /*
  * table for IOCBs
  */
@@ -111,45 +124,31 @@ struct sdpc_iocb_q {
 	int size;               /* current number of IOCBs in table */
 }; /* struct sdpc_iocb_q */
 
-/* ----------------------------------------------------------------------- */
-/*                                                                         */
-/* Address translations                                                    */
-/*                                                                         */
-/* ----------------------------------------------------------------------- */
-/* =========================================================== */
-/*..__sdp_kmap -- map a page into kernel space */
-static __inline__ void *__sdp_kmap
-(
- struct page *page
-)
-{
-  if (in_atomic() || irqs_disabled()) {
-    
-    return kmap_atomic(page, KM_IRQ0);
-  }
-  else {
+/*
+ * Address translations
+ */
 
-    return kmap(page);
-  }
+/*
+ * __sdp_kmap - map a page into kernel space
+ */
+static __inline__ void *__sdp_kmap(struct page *page)
+{
+	if (in_atomic() || irqs_disabled())
+		return kmap_atomic(page, KM_IRQ0);
+	else
+		return kmap(page);
 } /* __sdp_kmap */
 
-/* =========================================================== */
-/*..__sdp_kunmap -- unmap a page into kernel space */
-static __inline__ void __sdp_kunmap
-(
- struct page *page
-)
+/*
+ * __sdp_kunmap - unmap a page into kernel space
+ */
+static __inline__ void __sdp_kunmap(struct page *page)
 {
-  if (in_atomic() || irqs_disabled()) {
-    
-    kunmap_atomic(page, KM_IRQ0);
-  }
-  else {
-
-    kunmap(page);
-  }
-
-  return;
+	if (in_atomic() || irqs_disabled())
+		kunmap_atomic(page, KM_IRQ0);
+	else
+		kunmap(page);
+	return;
 } /* __sdp_kunmap */
 
 #endif /* _SDP_IOCB_H */



From Tom.Duffy at Sun.COM  Thu Feb 17 14:10:53 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:10:53 -0800
Subject: [openib-general] [PATCH][SDP][14/33] Fix comment style in sdp_kvec.c
In-Reply-To: <11086782432384@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
Message-ID: <11086782531540@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_kvec.c	2005-02-10 18:42:56.228029000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_kvec.c	2005-02-17 10:58:33.716019000 -0800
@@ -1,35 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_kvec.c 159 2004-05-14 00:16:54Z libor $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* ------------------------------------------------------------------- */
-/*                                                                     */
-/* static kvec internal functions                                      */
-/*                                                                     */
-/* ------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_lookup_func -- lookup function for cancelation */
+/*
+ * static kvec internal functions
+ */
+
+/*
+ * _sdp_iocb_q_cancel_lookup_func - lookup function for cancelation
+ */
 static int _sdp_iocb_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
 {
 	SDP_CHECK_NULL(element, -EINVAL);
@@ -37,20 +48,17 @@ static int _sdp_iocb_q_cancel_lookup_fun
 	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
 } /* _sdp_iocb_q_cancel_lookup_func */
 
-/* ------------------------------------------------------------------- */
-/*                                                                     */
-/* Public write functions                                              */
-/*                                                                     */
-/* ------------------------------------------------------------------- */
-
-/* ------------------------------------------------------------------- */
-/*                                                                     */
-/* Cancel operations                                                   */
-/*                                                                     */
-/* ------------------------------------------------------------------- */
+/*
+ * Public write functions
+ */
+
+/*
+ * Cancel operations
+ */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_read_pending -- cancel all pending read AIOs */
+/*
+ * _sdp_iocb_q_cancel_read_pending - cancel all pending read AIOs
+ */
 static int _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn, ssize_t error)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -59,8 +67,9 @@ static int _sdp_iocb_q_cancel_read_pendi
 					error);
 } /* _sdp_iocb_q_cancel_read_pending */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_read_source -- cancel all pending read AIOs */
+/*
+ * _sdp_iocb_q_cancel_read_source - cancel all pending read AIOs
+ */
 static int _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
@@ -82,8 +91,9 @@ static int _sdp_iocb_q_cancel_read_sourc
 	return 0;
 } /* _sdp_iocb_q_cancel_read_source */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_read_snk -- cancel all pending read AIOs */
+/*
+ * _sdp_iocb_q_cancel_read_snk - cancel all pending read AIOs
+ */
 static int _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -91,8 +101,9 @@ static int _sdp_iocb_q_cancel_read_snk(s
 	return sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_read_snk */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_write_pending -- cancel all pending read AIOs */
+/*
+ * _sdp_iocb_q_cancel_write_pending - cancel all pending read AIOs
+ */
 static int _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
@@ -116,8 +127,9 @@ static int _sdp_iocb_q_cancel_write_pend
 	return 0;
 } /* _sdp_iocb_q_cancel_write_pending */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_write_source -- cancel all pending source AIOs */
+/*
+ * _sdp_iocb_q_cancel_write_source - cancel all pending source AIOs
+ */
 static int _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn, ssize_t error)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -125,8 +137,9 @@ static int _sdp_iocb_q_cancel_write_sour
 	return sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_write_source */
 
-/* ========================================================================= */
-/*.._sdp_iocb_q_cancel_write_snk -- cancel all pending sink AIOs */
+/*
+ * _sdp_iocb_q_cancel_write_snk - cancel all pending sink AIOs
+ */
 static int _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
@@ -148,8 +161,9 @@ static int _sdp_iocb_q_cancel_write_snk(
 	return 0;
 } /* _sdp_iocb_q_cancel_write_snk */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_cancel_all_read -- cancel all outstanding read AIOs */
+/* 
+ * sdp_iocb_q_cancel_all_read - cancel all outstanding read AIOs
+ */
 int sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
 {
 	int result;
@@ -168,8 +182,9 @@ int sdp_iocb_q_cancel_all_read(struct sd
 	return 0;
 } /* sdp_iocb_q_cancel_all_read */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_cancel_all_write -- cancel all outstanding write AIOs */
+/*
+ * sdp_iocb_q_cancel_all_write - cancel all outstanding write AIOs
+ */
 int sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
 {
 	int result;
@@ -188,8 +203,9 @@ int sdp_iocb_q_cancel_all_write(struct s
 	return 0;
 } /* sdp_iocb_q_cancel_all_write */
 
-/* ========================================================================= */
-/*..sdp_iocb_q_cancel_all -- cancel all outstanding AIOs */
+/*
+ * sdp_iocb_q_cancel_all - cancel all outstanding AIOs
+ */
 int sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
 {
 	int result;



From Tom.Duffy at Sun.COM  Thu Feb 17 14:11:03 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:11:03 -0800
Subject: [openib-general] [PATCH][SDP][15/33] Fix comment style in sdp_link.c
In-Reply-To: <11086782531540@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com> <11086782531540@sun.com>
Message-ID: <11086782631290@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_link.c	2005-02-10 18:42:56.318026000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_link.c	2005-02-17 11:03:42.673005000 -0800
@@ -1,25 +1,37 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
-  $Id: sdp_link.c,v 1.26 2004/03/29 03:13:18 roland Exp $
-*/
 #include "ipoib.h"
 #include "sdp_main.h"
 #include "sdp_link.h"
@@ -51,8 +63,9 @@ static u64 _path_lookup_id = 0;
  */
 void _sdp_link_path_lookup(void *data);
 
-/* ========================================================================= */
-/*.._sdp_link_path_complete -- generate a path record completion for user */
+/*
+ * _sdp_link_path_complete - generate a path record completion for user
+ */
 static void _sdp_link_path_complete(u64 id,
 				    int status,
 				    struct sdp_path_info *info,
@@ -82,8 +95,9 @@ static void _sdp_link_path_complete(u64 
 	return;
 } /* _sdp_link_path_complete */
 
-/* ========================================================================= */
-/*.._sdp_path_wait_add -- add a wait entry into the wait list for a path  */
+/*
+ * _sdp_path_wait_add - add a wait entry into the wait list for a path
+ */
 static void _sdp_path_wait_add(struct sdp_path_info *info,
 			       struct sdp_path_wait *wait)
 {
@@ -100,8 +114,9 @@ static void _sdp_path_wait_add(struct sd
 	return;
 } /* _sdp_path_wait_add */
 
-/* ========================================================================= */
-/*.._sdp_path_wait_destroy -- destroy an entry for a wait element */
+/*
+ * _sdp_path_wait_destroy - destroy an entry for a wait element
+ */
 static void _sdp_path_wait_destroy(struct sdp_path_wait *wait)
 {
 	/*
@@ -123,8 +138,10 @@ static void _sdp_path_wait_destroy(struc
 
 	return;
 } /* _sdp_path_wait_destroy */
-/* ========================================================================= */
-/*.._sdp_path_wait_complete -- complete an entry for a wait element */
+
+/*
+ * _sdp_path_wait_complete - complete an entry for a wait element
+ */
 static void _sdp_path_wait_complete(struct sdp_path_wait *wait,
 				   struct sdp_path_info *info,
 				   int status)
@@ -140,10 +157,10 @@ static void _sdp_path_wait_complete(stru
 	return;
 } /* _sdp_path_wait_complete */
 
-/* ========================================================================= */
-/*.._sdp_path_info_lookup -- lookup a path record entry */
-static struct sdp_path_info *_sdp_path_info_lookup(u32 dst_ip,
-						   int dev_if)
+/*
+ * _sdp_path_info_lookup - lookup a path record entry
+ */
+static struct sdp_path_info *_sdp_path_info_lookup(u32 dst_ip, int dev_if)
 {
 	struct sdp_path_info *info;
 
@@ -159,10 +176,10 @@ static struct sdp_path_info *_sdp_path_i
 	return info;
 } /* _sdp_path_info_lookup */
 
-/* ========================================================================= */
-/*.._sdp_path_info_create -- create an entry for a path record element */
-static struct sdp_path_info *_sdp_path_info_create(u32 dst_ip,
-						   int dev_if)
+/*
+ * _sdp_path_info_create - create an entry for a path record element
+ */
+static struct sdp_path_info *_sdp_path_info_create(u32 dst_ip, int dev_if)
 {
 	struct sdp_path_info *info;
 
@@ -194,8 +211,9 @@ static struct sdp_path_info *_sdp_path_i
 	return info;
 } /* _sdp_path_info_create */
 
-/* ========================================================================= */
-/*.._sdp_path_info_destroy -- destroy an entry for a path record element */
+/*
+ * _sdp_path_info_destroy - destroy an entry for a path record element
+ */
 static void _sdp_path_info_destroy(struct sdp_path_info *info, int status)
 {
 	struct sdp_path_wait *wait;
@@ -225,8 +243,9 @@ static void _sdp_path_info_destroy(struc
 	return;
 } /* _sdp_path_info_destroy */
 
-/* ========================================================================= */
-/*.._sdp_link_path_rec_done -- path record completion function. */
+/*
+ * _sdp_link_path_rec_done - path record completion function
+ */
 static void _sdp_link_path_rec_done(int status,
 				    struct ib_sa_path_rec *resp,
 				    void *context)
@@ -299,8 +318,9 @@ static void _sdp_link_path_rec_done(int 
 	return;
 } /* _sdp_link_path_rec_done */
 
-/* ========================================================================= */
-/*.._sdp_link_path_rec_get -- resolve GIDs to a path record. */
+/*
+ * _sdp_link_path_rec_get - resolve GIDs to a path record
+ */
 int _sdp_link_path_rec_get(struct sdp_path_info *info)
 {
 	int result;
@@ -341,8 +361,9 @@ int _sdp_link_path_rec_get(struct sdp_pa
 	return result;
 } /* _sdp_link_path_rec_get */
 
-/* ========================================================================= */
-/*.._sdp_link_path_lookup -- resolve an ip address to a path record */
+/*
+ * _sdp_link_path_lookup - resolve an ip address to a path record
+ */
 void _sdp_link_path_lookup(void *data)
 {
 	struct sdp_path_info *info = data;
@@ -559,11 +580,13 @@ error:
 	return;
 } /* _sdp_link_path_lookup */
 
-/* --------------------------------------------------------------------- */
-/* Public functions                                                      */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_link_path_lookup -- resolve an ip address to a path record */
+/*
+ * Public functions
+ */
+
+/*
+ * sdp_link_path_lookup - resolve an ip address to a path record
+ */
 int sdp_link_path_lookup(u32 dst_addr,      /* NBO */
 			 u32 src_addr,      /* NBO */
 			 int bound_dev_if,  /* socket option */
@@ -636,11 +659,13 @@ error:
 	return result;
 } /* sdp_link_path_lookup */
 
-/* --------------------------------------------------------------------- */
-/* timers                                                                */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_link_sweep -- periodic path information cleanup function */
+/*
+ * timers
+ */
+
+/*
+ * _sdp_link_sweep - periodic path information cleanup function
+ */
 static void _sdp_link_sweep(void *data)
 {
 	struct sdp_path_info *info;
@@ -669,11 +694,14 @@ static void _sdp_link_sweep(void *data)
 	queue_delayed_work(_link_wq, &_link_timer, SDP_LINK_SWEEP_INTERVAL);
 	return;
 } /* _sdp_link_sweep */
-/* --------------------------------------------------------------------- */
-/* Arp packet reception for completions                                  */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_link_arp_work -- handle IB REQUEST/REPLY ARP packets. */
+
+/*
+ * Arp packet reception for completions
+ */
+
+/*
+ * _sdp_link_arp_work - handle IB REQUEST/REPLY ARP packets
+ */
 static void _sdp_link_arp_work(void *data)
 {
 	struct sdp_work *work = (struct sdp_work *)data;
@@ -729,8 +757,9 @@ done:
 	return;
 } /* _sdp_link_arp_work */
 
-/* ========================================================================= */
-/*.._sdp_link_arp_recv -- receive all ARP packets. */
+/*
+ * _sdp_link_arp_recv - receive all ARP packets
+ */
 static int _sdp_link_arp_recv(struct sk_buff *skb,
 			      struct net_device *dev,
 			      struct packet_type *pt)
@@ -764,23 +793,20 @@ done:
 	kfree_skb(skb);
 	return 0;
 } /* _sdp_link_arp_recv */
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* primary initialization/cleanup functions                              */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+
+/*
+ * primary initialization/cleanup functions
+ */
 static struct packet_type _sdp_arp_type = {
 	.type           = __constant_htons(ETH_P_ARP),
 	.func           = _sdp_link_arp_recv,
 	.af_packet_priv = (void*) 1, /* understand shared skbs */
 };
 
-/* ========================================================================= */
-/*..sdp_link_addr_init -- initialize the link address retrival code. */
-int sdp_link_addr_init
-(
- void
-)
+/*
+ * sdp_link_addr_init - initialize the link address retrival code
+ */
+int sdp_link_addr_init(void)
 {
 	int result;
 
@@ -849,12 +875,10 @@ error_path:
 	return result;
 } /* sdp_link_addr_init */
 
-/* ========================================================================= */
-/*..sdp_link_addr_cleanup -- cleanup the link address retrival code. */
-int sdp_link_addr_cleanup
-(
- void
-)
+/*
+ * sdp_link_addr_cleanup - cleanup the link address retrival code
+ */
+int sdp_link_addr_cleanup(void)
 {
 	struct sdp_path_info *info;
 
@@ -890,17 +914,3 @@ int sdp_link_addr_cleanup
 
 	return 0;
 } /* sdp_link_addr_cleanup */
-
-
-
-
-
-
-
-
-
-
-
-
-
-



From Tom.Duffy at Sun.COM  Thu Feb 17 14:11:14 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:11:14 -0800
Subject: [openib-general] [PATCH][SDP][16/33] Fix comment style in sdp_link.h
In-Reply-To: <11086782631290@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
Message-ID: <11086782741398@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_link.h	2005-02-10 18:42:56.370000000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_link.h	2005-02-17 11:04:05.652004000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_link.h,v 1.14 2004/02/24 23:48:57 roland Exp $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_LINK_H
 #define _SDP_LINK_H



From Tom.Duffy at Sun.COM  Thu Feb 17 14:11:24 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:11:24 -0800
Subject: [openib-general] [PATCH][SDP][17/33] Fix comment style in sdp_main.h
In-Reply-To: <11086782741398@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
Message-ID: <1108678284314@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_main.h	2005-02-14 11:20:29.048990000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_main.h	2005-02-17 11:04:28.550001000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_main.h 509 2004-07-21 22:57:41Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_MAIN_H
 #define _SDP_MAIN_H



From Tom.Duffy at Sun.COM  Thu Feb 17 14:11:34 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:11:34 -0800
Subject: [openib-general] [PATCH][SDP][18/33] Fix comment style in sdp_msgs.h
In-Reply-To: <1108678284314@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com> <1108678284314@sun.com>
Message-ID: <11086782942946@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_msgs.h	2005-02-10 18:42:56.449031000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_msgs.h	2005-02-17 11:07:20.058017000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_msgs.h 229 2004-06-04 00:55:12Z libor $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_MSGS_H
 #define _SDP_MSGS_H
@@ -259,12 +270,14 @@ struct msg_hdr_sch {
 #define SDP_MSG_MCH_GET_MODE(mch) (((mch)->flags & 0xF0) >> 4)
 #define SDP_MSG_MCH_SET_MODE(mch, value) \
         ((mch)->flags = (((mch)->flags & 0x0F) | (value << 4)))
-/* ------------------------------------------------------------------------ */
-/* Endian Conversions                                                       */
-/* ------------------------------------------------------------------------ */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_bsdh -- SDP header endian byte swapping    */
+/*
+ * Endian Conversions
+ */
+
+/*
+ * __sdp_msg_swap_bsdh - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
 {
 	header->recv_bufs = cpu_to_be16(header->recv_bufs);
@@ -273,8 +286,9 @@ static __inline__ void __sdp_msg_swap_bs
 	header->seq_ack = cpu_to_be32(header->seq_ack);
 } /* __sdp_msg_swap_bsdh */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_hh -- SDP header endian byte swapping      */
+/*
+ * __sdp_msg_swap_hh - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_hh(struct msg_hdr_hh *header)
 {
 	header->r_rcv_size = cpu_to_be32(header->r_rcv_size);
@@ -290,15 +304,17 @@ static __inline__ void __sdp_msg_swap_hh
 	header->dst.ipv6.addr3 = cpu_to_be32(header->dst.ipv6.addr3);
 } /* __sdp_msg_swap_ */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_hah -- SDP header endian byte swapping     */
+/*
+ * __sdp_msg_swap_hah - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
 {
 	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
 } /* __sdp_msg_swap_hah */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_srcah -- SDP header endian byte swapping   */
+/*
+ * __sdp_msg_swap_srcah - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
 {
 	header->size = cpu_to_be32(header->size);
@@ -311,8 +327,9 @@ static __inline__ void __sdp_msg_swap_sr
 #endif
 } /* __sdp_msg_swap_srcah */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_snkah -- SDP header endian byte swapping   */
+/*
+ * __sdp_msg_swap_snkah - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
 {
 	header->size = cpu_to_be32(header->size);
@@ -321,43 +338,49 @@ static __inline__ void __sdp_msg_swap_sn
 	header->non_disc = cpu_to_be32(header->non_disc);
 } /* __sdp_msg_swap_snkah */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_rwch -- SDP header endian byte swapping    */
+/*
+ * __sdp_msg_swap_rwch - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_rwch */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_rrch -- SDP header endian byte swapping    */
+/*
+ * __sdp_msg_swap_rrch - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_rrch */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_mch -- SDP header endian byte swapping     */
+/*
+ * __sdp_msg_swap_mch - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
 {
 	return;
 } /* __sdp_msg_swap_ */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_crbh -- SDP header endian byte swapping    */
+/*
+ * __sdp_msg_swap_crbh - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_crbh */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_crbah -- SDP header endian byte swapping   */
+/*
+ * __sdp_msg_swap_crbah - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_crbah */
 
-/* =========================================================== */
-/*..__sdp_msg_swap_sch -- SDP header endian byte swapping     */
+/*
+ * __sdp_msg_swap_sch - SDP header endian byte swapping
+ */
 static __inline__ void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
 {
 	header->service_id = cpu_to_be64(header->service_id);
@@ -386,9 +409,10 @@ static __inline__ void __sdp_msg_swap_sc
 #define _sdp_msg_cpu_to_net_sch   __sdp_msg_swap_sch
 #define _sdp_msg_net_to_cpu_sch   __sdp_msg_swap_sch
 
-/* ------------------------------------------------------------------------ */
-/* Miscellaneous message related informtation                               */
-/* ------------------------------------------------------------------------ */
+/*
+ * Miscellaneous message related informtation
+ */
+
 /*
  * Connection messages
  */



From Tom.Duffy at Sun.COM  Thu Feb 17 14:11:44 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:11:44 -0800
Subject: [openib-general] [PATCH][SDP][19/33] Fix comment style in sdp_pass.c
In-Reply-To: <11086782942946@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
Message-ID: <1108678304705@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_pass.c	2005-02-15 16:33:46.238009000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_pass.c	2005-02-17 11:08:43.171002000 -0800
@@ -1,30 +1,42 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_post.c 649 2004-08-13 22:41:04Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* ========================================================================= */
-/*.._sdp_cm_accept -- respond to a connection attempt with an ack */
+/*
+ * _sdp_cm_accept - respond to a connection attempt with an ack
+ */
 static int _sdp_cm_accept(struct sdp_opt *conn)
 {
 	struct ib_cm_rep_param param;
@@ -176,8 +188,9 @@ error:
 	return result;
 } /* _sdp_cm_accept */
 
-/* ========================================================================= */
-/*.._sdp_cm_listen_lookup -- match a listener with an incomming conn. */
+/*
+ * _sdp_cm_listen_lookup - match a listener with an incomming conn
+ */
 static int _sdp_cm_listen_lookup(struct sdp_opt *conn)
 {
 	struct sdp_opt *listen_conn;
@@ -297,8 +310,9 @@ lookup_err:
 	return result;
 } /* _sdp_cm_listen_lookup */
 
-/* ========================================================================= */
-/*.._sdp_cm_hello_check -- validate the hello header */
+/*
+ * _sdp_cm_hello_check - validate the hello header
+ */
 static int _sdp_cm_hello_check(struct msg_hello *msg_hello)
 {
 	/*
@@ -369,10 +383,10 @@ static int _sdp_cm_hello_check(struct ms
 	return 0; /* success */
 } /* _sdp_cm_hello_check */
 
-/* ========================================================================= */
-/*..sdp_cm_req_handler -- handler for passive connection open completion */
-int sdp_cm_req_handler(struct ib_cm_id *cm_id,
-		       struct ib_cm_event *event)
+/*
+ * sdp_cm_req_handler - handler for passive connection open completion
+ */
+int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 {
 	struct msg_hello *msg_hello = (struct msg_hello *)event->private_data;
 	struct sdp_opt *conn;
@@ -483,4 +497,3 @@ error:
 done:
 	return result;
 } /* sdp_cm_req_handler */
-



From Tom.Duffy at Sun.COM  Thu Feb 17 14:11:55 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:11:55 -0800
Subject: [openib-general] [PATCH][SDP][20/33] Fix comment style in sdp_post.c
In-Reply-To: <1108678304705@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
Message-ID: <11086783152506@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_post.c	2005-02-10 18:42:57.157027000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_post.c	2005-02-17 11:11:01.646018000 -0800
@@ -1,30 +1,42 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_post.c 649 2004-08-13 22:41:04Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* ========================================================================= */
-/*..sdp_cm_listen_start --  start listening on all possible socket ports */
+/*
+ * sdp_cm_listen_start - start listening on all possible socket ports
+ */
 int sdp_cm_listen_start(struct sdev_root *dev_root)
 {
 	int result = 0;
@@ -64,8 +76,9 @@ int sdp_cm_listen_start(struct sdev_root
 	return result;
 } /* sdp_cm_listen_start */
 
-/* ========================================================================= */
-/*..sdp_cm_listen_stop --  stop listening on all possible socket ports */
+/*
+ * sdp_cm_listen_stop - stop listening on all possible socket ports
+ */
 int sdp_cm_listen_stop(struct sdev_root *dev_root)
 {
 	int result = 0;
@@ -87,8 +100,9 @@ int sdp_cm_listen_stop(struct sdev_root 
 	return result;
 } /* sdp_cm_listen_stop */
 
-/* ========================================================================= */
-/*.._sdp_cm_disconnect -- initiate a disconnect request using the CM */
+/*
+ * _sdp_cm_disconnect - initiate a disconnect request using the CM
+ */
 static void _sdp_cm_disconnect(void *arg)
 {
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
@@ -115,8 +129,9 @@ static void _sdp_cm_disconnect(void *arg
 	return;
 } /* _sdp_cm_disconnect */
 
-/* ========================================================================= */
-/*.._sdp_cm_reject -- initiate a reject request using the CM */
+/*
+ * _sdp_cm_reject - initiate a reject request using the CM
+ */
 static void _sdp_cm_reject(void *arg)
 {
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
@@ -144,8 +159,9 @@ static void _sdp_cm_reject(void *arg)
 	return;
 } /* _sdp_cm_reject */
 
-/* ========================================================================= */
-/*.._sdp_cm_confirm -- initiate a confirm request using the CM */
+/*
+ * _sdp_cm_confirm - initiate a confirm request using the CM
+ */
 static void _sdp_cm_confirm(void *arg)
 {
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
@@ -171,8 +187,9 @@ static void _sdp_cm_confirm(void *arg)
 	return;
 } /* _sdp_cm_confirm */
 
-/* ========================================================================= */
-/*.._sdp_cm_failed -- initiate a Failed request using the CM */
+/*
+ * _sdp_cm_failed - initiate a Failed request using the CM
+ */
 static void _sdp_cm_failed(void *arg)
 {
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
@@ -201,8 +218,9 @@ static void _sdp_cm_failed(void *arg)
 	return;
 } /* _sdp_cm_failed */
 
-/* ========================================================================= */
-/*.._sdp_cm_deferred_generic -- initiate a defered request using the*/
+/*
+ * _sdp_cm_deferred_generic - initiate a defered request using the CM
+ */
 static int _sdp_cm_deferred_generic(struct sdp_opt *conn,
 				    void (*defer_func)(void *arg))
 {
@@ -228,29 +246,33 @@ static int _sdp_cm_deferred_generic(stru
 	return 0;
 } /* _sdp_cm_deferred_generic */
 
-/* ========================================================================= */
-/*..sdp_cm_disconnect -- initiate a disconnect request using the CM */
+/*
+ * sdp_cm_disconnect - initiate a disconnect request using the CM
+*/
 int sdp_cm_disconnect(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_disconnect);
 } /* sdp_cm_disconnect */
 
-/* ========================================================================= */
-/*..sdp_cm_reject -- initiate a reject request using the CM */
+/*
+ * sdp_cm_reject - initiate a reject request using the CM
+ */
 int sdp_cm_reject(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_reject);
 } /* sdp_cm_reject */
 
-/* ========================================================================= */
-/*..sdp_cm_confirm -- initiate a confirm request using the CM */
+/*
+ * sdp_cm_confirm - initiate a confirm request using the CM
+ */
 int sdp_cm_confirm(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_confirm);
 } /* sdp_cm_confirm */
 
-/* ========================================================================= */
-/*..sdp_cm_failed -- initiate a failed request using the CM */
+/*
+ * sdp_cm_failed - initiate a failed request using the CM
+ */
 int sdp_cm_failed(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_failed);



From Tom.Duffy at Sun.COM  Thu Feb 17 14:12:05 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:12:05 -0800
Subject: [openib-general] [PATCH][SDP][21/33] Fix comment style in sdp_proc.c
In-Reply-To: <11086783152506@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com> <11086783152506@sun.com>
Message-ID: <11086783252833@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_proc.c	2005-02-14 11:13:15.600961000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_proc.c	2005-02-17 11:12:47.182006000 -0800
@@ -1,38 +1,49 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_proc.c 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
 static const char _dir_name_root[] = SDP_PROC_DIR_NAME;
 static struct proc_dir_entry *_dir_root = NULL;
 
-/* ------------------------------------------------------------------- */
-/*                                                                     */
-/* Generic static functions used by read/write functions               */
-/*                                                                     */
-/* ------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_proc_read_parse -- read function for the injection table */
+/*
+ * Generic static functions used by read/write functions
+ */
+
+/*
+ * _sdp_proc_read_parse - read function for the injection table
+ */
 static int _sdp_proc_read_parse(char *page,
 				char **start, 
 				off_t offset,
@@ -69,11 +80,10 @@ static int _sdp_proc_read_parse(char *pa
 	return size;
 } /* _sdp_proc_read_parse */
 
-/* ------------------------------------------------------------------- */
-/*                                                                     */
-/* Static read/write functions for each proc/framework directory entry */
-/*                                                                     */
-/* ------------------------------------------------------------------- */
+/*
+ * Static read/write functions for each proc/framework directory entry
+ */
+
 /*
  * Initialization structure, each table in the gateway framework directory
  * (anything that is not a module) should create an entry and define read
@@ -118,14 +128,13 @@ static struct sdpc_proc_ent _file_entry_
       },
 } ;
 
-/* ------------------------------------------------------------------- */
-/*                                                                     */
-/* SDP module public functions.                                        */
-/*                                                                     */
-/* ------------------------------------------------------------------- */
+/*
+ * SDP module public functions.
+ */
 
-/* ========================================================================= */
-/*..sdp_main_proc_cleanup -- cleanup the proc filesystem entries  */
+/*
+ * sdp_main_proc_cleanup - cleanup the proc filesystem entries
+ */
 int sdp_main_proc_cleanup(void)
 {
 	struct sdpc_proc_ent *sub_entry;
@@ -154,8 +163,9 @@ int sdp_main_proc_cleanup(void)
 	return 0;
 } /* sdp_main_proc_cleanup */
 
-/* ========================================================================= */
-/*..sdp_main_proc_init -- initialize the proc filesystem entries  */
+/*
+ * sdp_main_proc_init - initialize the proc filesystem entries
+ */
 int sdp_main_proc_init(void)
 {
 	struct sdpc_proc_ent *sub_entry;



From Tom.Duffy at Sun.COM  Thu Feb 17 14:12:15 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:12:15 -0800
Subject: [openib-general] [PATCH][SDP][22/33] Fix comment style in sdp_proc.h
In-Reply-To: <11086783252833@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
Message-ID: <1108678335304@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_proc.h	2005-02-10 18:42:56.751003000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_proc.h	2005-02-17 11:13:23.598020000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_proc.h 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_PROC_H
 #define _SDP_PROC_H
@@ -27,9 +38,11 @@
 #include <linux/proc_fs.h>
 
 #define SDP_PROC_DIR_NAME "sdp"
-/* --------------------------------------------------------------------- */
-/* file and directory entries                                            */
-/* --------------------------------------------------------------------- */
+
+/*
+ * file and directory entries
+ */
+
 /*
  * proc filesystem framework table/file entries
  */
@@ -55,4 +68,3 @@ struct sdpc_proc_ent {
 }; /* struct sdpc_proc_ent */
 
 #endif /* _SDP_PROC_H */
-



From Tom.Duffy at Sun.COM  Thu Feb 17 14:12:25 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:12:25 -0800
Subject: [openib-general] [PATCH][SDP][23/33] Fix comment style in
	sdp_proto.h
In-Reply-To: <1108678335304@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
Message-ID: <11086783452340@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_proto.h	2005-02-10 18:42:56.417024000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_proto.h	2005-02-17 11:17:48.805008000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_proto.h 994 2004-10-14 05:38:39Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_PROTO_H
 #define _SDP_PROTO_H
@@ -34,9 +45,9 @@
 #include "sdp_iocb.h"
 #include "sdp_queue.h"
 
-/* --------------------------------------------------------------------- */
-/* Buffer managment                                                      */
-/* --------------------------------------------------------------------- */
+/*
+ * Buffer managment
+ */
 struct sdpc_buff *sdp_buff_pool_get(void);
 
 int sdp_buff_pool_put(struct sdpc_buff *buff);
@@ -115,9 +126,9 @@ int sdp_proc_dump_buff_pool(char *buffer
 			    off_t start_index,
 			    long *end_index);
 
-/* --------------------------------------------------------------------- */
-/* Wall between userspace protocol and SDP protocol proper               */
-/* --------------------------------------------------------------------- */
+/*
+ * Wall between userspace protocol and SDP protocol proper
+ */
 int sdp_wall_send_reject(struct sdp_opt *conn);
 
 int sdp_wall_send_close(struct sdp_opt *conn);
@@ -143,9 +154,10 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 int sdp_wall_abort(struct sdp_opt *conn);
 
 int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff);
-/* --------------------------------------------------------------------- */
-/* Zcopy advertisment managment                                          */
-/* --------------------------------------------------------------------- */
+
+/*
+ * Zcopy advertisment managment
+ */
 int sdp_main_advt_init(void);
 
 int sdp_main_advt_cleanup(void);
@@ -167,9 +179,10 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table);
 
 int sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt);
-/* --------------------------------------------------------------------- */
-/* Zcopy IOCB managment                                                  */
-/* --------------------------------------------------------------------- */
+
+/*
+ * Zcopy IOCB managment
+ */
 int sdp_main_iocb_init(void);
 
 int sdp_main_iocb_cleanup(void);
@@ -213,9 +226,10 @@ int sdp_iocb_complete(struct sdpc_iocb *
 int sdp_iocb_lock(struct sdpc_iocb *iocb);
 
 int sdp_iocb_unlock(struct sdpc_iocb *iocb);
-/* --------------------------------------------------------------------- */
-/* Generic object managment                                              */
-/* --------------------------------------------------------------------- */
+
+/*
+ * Generic object managment
+ */
 int sdp_desc_q_remove(struct sdpc_desc *element);
 
 struct sdpc_desc *sdp_desc_q_get_all(struct sdpc_desc_q *table);
@@ -265,15 +279,17 @@ int sdp_desc_q_destroy(struct sdpc_desc_
 int sdp_main_desc_init(void);
 
 int sdp_main_desc_cleanup(void);
-/* --------------------------------------------------------------------- */
-/* proc entry managment                                                  */
-/* --------------------------------------------------------------------- */
+
+/*
+ * proc entry managment
+ */
 int sdp_main_proc_init(void);
 
 int sdp_main_proc_cleanup(void);
-/* --------------------------------------------------------------------- */
-/* connection table                                                      */
-/* --------------------------------------------------------------------- */
+
+/*
+ * connection table
+ */
 int sdp_conn_table_init(int proto_family,
 			int conn_size,
 			int recv_post_max,
@@ -331,9 +347,10 @@ void sdp_inet_wake_recv(struct sock *sk,
 void sdp_inet_wake_error(struct sock *sk);
 
 void sdp_inet_wake_urg(struct sock *sk);
-/* --------------------------------------------------------------------- */
-/* port/queue managment                                                  */
-/* --------------------------------------------------------------------- */
+
+/*
+ * port/queue managment
+ */
 int sdp_inet_accept_q_put(struct sdp_opt *listen_conn,
 			  struct sdp_opt *accept_conn);
 
@@ -353,24 +370,24 @@ int sdp_inet_port_put(struct sdp_opt *co
 
 int sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child);
 
-/* --------------------------------------------------------------------- */
-/* active connect functions                                              */
-/* --------------------------------------------------------------------- */
+/*
+ * active connect functions
+ */
 int sdp_cm_connect(struct sdp_opt *conn);
 
 int sdp_cm_rep_handler(struct ib_cm_id *cm_id,
 		       struct ib_cm_event *event,
 		       struct sdp_opt *conn);
 
-/* --------------------------------------------------------------------- */
-/* passive connect functions                                             */
-/* --------------------------------------------------------------------- */
+/*
+ * passive connect functions
+ */
 int sdp_cm_req_handler(struct ib_cm_id *cm_id,
 		       struct ib_cm_event *event);
 
-/* --------------------------------------------------------------------- */
-/* post functions                                                        */
-/* --------------------------------------------------------------------- */
+/*
+ * post functions
+ */
 int sdp_cm_listen_start(struct sdev_root *dev_root);
 
 int sdp_cm_listen_stop(struct sdev_root *dev_root);
@@ -414,12 +431,13 @@ int sdp_send_ctrl_snk_cancel(struct sdp_
 
 int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn);
 
-/* --------------------------------------------------------------------- */
-/* inet functions                                                        */
-/* --------------------------------------------------------------------- */
-/* --------------------------------------------------------------------- */
-/* event functions                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * inet functions
+ */
+
+/*
+ * event functions
+ */
 int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_opt *conn);
 
 void sdp_cq_event_handler(struct ib_cq *cq, void *arg);
@@ -435,9 +453,9 @@ int sdp_event_read(struct sdp_opt *conn,
 
 int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp);
 
-/* --------------------------------------------------------------------- */
-/* internal connection lock functions                                    */
-/* --------------------------------------------------------------------- */
+/*
+ * internal connection lock functions
+ */
 void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags);
 
 void sdp_conn_internal_unlock(struct sdp_opt *conn);
@@ -446,9 +464,9 @@ void sdp_conn_internal_relock(struct sdp
 
 int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn);
 
-/* --------------------------------------------------------------------- */
-/* DATA transport                                                        */
-/* --------------------------------------------------------------------- */
+/*
+ * DATA transport
+ */
 int sdp_inet_send(struct kiocb *iocb,
 		  struct socket *sock, 
 		  struct msghdr *msg,
@@ -465,9 +483,10 @@ int sdp_iocb_q_cancel_all_read(struct sd
 int sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error);
 
 int sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error);
-/* --------------------------------------------------------------------- */
-/* link address information                                              */
-/* --------------------------------------------------------------------- */
+
+/*
+ * link address information
+ */
 int sdp_link_path_lookup(u32 dst_addr,
 			 u32 src_addr,
 			 int bound_dev_if,
@@ -485,18 +504,20 @@ int sdp_link_path_lookup(u32 dst_addr,
 int sdp_link_addr_init(void);
 
 int sdp_link_addr_cleanup(void);
-/* --------------------------------------------------------------------- */
-/* Function types.                                                       */
-/* --------------------------------------------------------------------- */
+
+/*
+ * Function types
+ */
+
 /*
  * Event handling function, demultiplexed base on Message ID
  */
 typedef int (*sdp_event_cb_func)(struct sdp_opt *conn, 
 				 struct sdpc_buff *buff);
-/* --------------------------------------------------------------------- */
-/* trace macros.                                                         */
-/* --------------------------------------------------------------------- */
 
+/*
+ * trace macros
+ */
 extern int debug_level;
 #define __SDP_DEBUG_LEVEL 4
 
@@ -595,11 +616,14 @@ extern int debug_level;
 #define SDP_CHECK_NULL(value, result) \
         if (NULL == (value)) return (result);
 #endif
-/* --------------------------------------------------------------------- */
-/* Inline functions.                                                     */
-/* --------------------------------------------------------------------- */
-/* ====================================================================== */
-/*..__sdp_inet_write_space -- writable space on send side. */
+
+/*
+ * Inline functions
+ */
+
+/*
+ * __sdp_inet_write_space - writable space on send side
+ */
 static __inline__ int __sdp_inet_write_space(struct sdp_opt *conn, int urg)
 {
 	int size;
@@ -626,8 +650,9 @@ static __inline__ int __sdp_inet_write_s
 	}
 } /* __sdp_inet_write_space */
 
-/* ====================================================================== */
-/*..__sdp_inet_writable -- return non-zero if socket is writable. */
+/*
+ * __sdp_inet_writable - return non-zero if socket is writable
+ */
 static __inline__ int __sdp_inet_writable(struct sdp_opt *conn)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -643,8 +668,9 @@ static __inline__ int __sdp_inet_writabl
 	}
 } /* __sdp_inet_writable */
 
-/* ======================================================================== */
-/*..__sdp_conn_stat_dump -- dump stats to the log */
+/*
+ * __sdp_conn_stat_dump - dump stats to the log
+ */
 static __inline__ int __sdp_conn_stat_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATS_REC
@@ -668,8 +694,9 @@ static __inline__ int __sdp_conn_stat_du
 	return 0;
 } /* __sdp_conn_stat_dump */
 
-/* ======================================================================== */
-/*..__sdp_conn_state_dump -- dump state information to the log */
+/*
+ * __sdp_conn_state_dump - dump state information to the log
+ */
 static __inline__ int __sdp_conn_state_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATE_REC
@@ -700,15 +727,17 @@ static __inline__ int __sdp_conn_state_d
 	return 0;
 } /* __sdp_conn_state_dump */
 
-/* ======================================================================== */
-/*..__sdp_conn_hold -- increment reference count */
+/*
+ * __sdp_conn_hold - increment reference count
+ */
 static inline void __sdp_conn_hold(struct sdp_opt *conn)
 {
 	atomic_inc(&conn->refcnt);
 } /* __sdp_conn_hold */
 
-/* ======================================================================== */
-/*..__sdp_conn_put -- decrement reference count */
+/*
+ * __sdp_conn_put - decrement reference count
+ */
 static inline void __sdp_conn_put(struct sdp_opt *conn)
 {
 	if (atomic_dec_and_test(&conn->refcnt)) {
@@ -717,8 +746,9 @@ static inline void __sdp_conn_put(struct
 	}
 } /* __sdp_conn_hold */
 
-/* ======================================================================== */
-/*..__sdp_conn_error -- get the connections error value destructively. */
+/*
+ * __sdp_conn_error - get the connections error value destructively
+ */
 static inline int __sdp_conn_error(struct sdp_opt *conn)
 {
 	/*



From Tom.Duffy at Sun.COM  Thu Feb 17 14:12:36 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:12:36 -0800
Subject: [openib-general] [PATCH][SDP][24/33] Fix comment style in
	sdp_queue.c
In-Reply-To: <11086783452340@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com> <11086783452340@sun.com>
Message-ID: <1108678356217@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_queue.c	2005-02-10 18:42:57.120000000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_queue.c	2005-02-17 11:21:40.495015000 -0800
@@ -1,37 +1,48 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_queue.c 620 2004-08-10 03:46:43Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
 static kmem_cache_t *__sdp_desc_table = NULL;
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* module specific functions                                             */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_desc_q_get - Get an element from a specific table */
+/*
+ * module specific functions
+ */
+
+/*
+ * _sdp_desc_q_get - Get an element from a specific table
+ */
 static struct sdpc_desc *_sdp_desc_q_get(struct sdpc_desc_q *table,
 					 int fifo)
 {
@@ -76,8 +87,9 @@ static struct sdpc_desc *_sdp_desc_q_get
 	return element;
 } /* _sdp_desc_q_get */
 
-/* ========================================================================= */
-/*.._sdp_desc_q_put - Place an element into a specific table */
+/*
+ * _sdp_desc_q_put - Place an element into a specific table
+ */
 static __inline__ int _sdp_desc_q_put(struct sdpc_desc_q *table,
 				      struct sdpc_desc *element, 
 				      int fifo)
@@ -120,13 +132,13 @@ static __inline__ int _sdp_desc_q_put(st
 	return 0;
 } /* _sdp_desc_q_put */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* public advertisment object functions for FIFO object table            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_desc_q_remove - remove a specific element from a table */
+/*
+ * public advertisment object functions for FIFO object table
+ */
+
+/*
+ * sdp_desc_q_remove - remove a specific element from a table
+ */
 int sdp_desc_q_remove(struct sdpc_desc *element)
 {
 	struct sdpc_desc_q *table;
@@ -168,8 +180,9 @@ int sdp_desc_q_remove(struct sdpc_desc *
 	return 0;
 } /* sdp_desc_q_remove */
 
-/* ========================================================================= */
-/*..sdp_desc_q_lookup - search and return an element from the table */
+/*
+ * sdp_desc_q_lookup - search and return an element from the table
+ */
 struct sdpc_desc *sdp_desc_q_lookup(struct sdpc_desc_q *table, 
 				    int (*lookup)(struct sdpc_desc *element,
 						  void *arg),
@@ -193,8 +206,9 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
 	return NULL;
 } /* sdp_desc_q_lookup */
 
-/* ========================================================================= */
-/*..sdp_desc_q_get_all - Get the element at the front of the table */
+/*
+ * sdp_desc_q_get_all - Get the element at the front of the table
+ */
 struct sdpc_desc *sdp_desc_q_get_all(struct sdpc_desc_q *table)
 {
 	struct sdpc_desc *head;
@@ -209,38 +223,43 @@ struct sdpc_desc *sdp_desc_q_get_all(str
 	return head;
 } /* sdp_desc_q_get_all */
 
-/* ========================================================================= */
-/*..sdp_desc_q_get_head - Get the element at the front of the table */
+/*
+ * sdp_desc_q_get_head - Get the element at the front of the table
+ */
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table)
 {
 	return _sdp_desc_q_get(table, 1);
 } /* sdp_desc_q_get_head */
 
-/* ========================================================================= */
-/*..sdp_desc_q_get_tail - Get the element at the end of the table */
+/*
+ * sdp_desc_q_get_tail - Get the element at the end of the table
+ */
 struct sdpc_desc *sdp_desc_q_get_tail(struct sdpc_desc_q *table)
 {
 	return _sdp_desc_q_get(table, 0);
 } /* sdp_desc_q_get_tail */
 
-/* ========================================================================= */
-/*..sdp_desc_q_put_head - Place an element into the head of a table */
+/*
+ * sdp_desc_q_put_head - Place an element into the head of a table
+ */
 int sdp_desc_q_put_head(struct sdpc_desc_q *table,
 				struct sdpc_desc *element)
 {
 	return _sdp_desc_q_put(table, element, 1);
 } /* sdp_desc_q_put_head */
 
-/* ========================================================================= */
-/*..sdp_desc_q_put_tail - Place an element into the tail of a table */
+/*
+ * sdp_desc_q_put_tail - Place an element into the tail of a table
+ */
 int sdp_desc_q_put_tail(struct sdpc_desc_q *table,
 				struct sdpc_desc *element)
 {
 	return _sdp_desc_q_put(table, element, 0);
 } /* sdp_desc_q_put_tail */
 
-/* ========================================================================= */
-/*..sdp_desc_q_look_head - look at the front of the table */
+/*
+ * sdp_desc_q_look_head - look at the front of the table
+ */
 struct sdpc_desc *sdp_desc_q_look_head(struct sdpc_desc_q *table)
 {
 	SDP_CHECK_NULL(table, NULL);
@@ -248,8 +267,9 @@ struct sdpc_desc *sdp_desc_q_look_head(s
 	return table->head;
 } /* sdp_desc_q_look_head */
 
-/* ========================================================================= */
-/*..sdp_desc_q_look_tail - look at the end of the table */
+/*
+ * sdp_desc_q_look_tail - look at the end of the table
+ */
 struct sdpc_desc *sdp_desc_q_look_tail(struct sdpc_desc_q *table)
 {
 	SDP_CHECK_NULL(table, NULL);
@@ -257,8 +277,9 @@ struct sdpc_desc *sdp_desc_q_look_tail(s
 	return ((NULL == table->head) ? NULL : table->head->prev);
 } /* sdp_desc_q_look_tail */
 
-/* ========================================================================= */
-/*..sdp_desc_q_type_head - look at the type at the front of the table */
+/*
+ * sdp_desc_q_type_head - look at the type at the front of the table
+ */
 int sdp_desc_q_type_head(struct sdpc_desc_q *table)
 {
 	SDP_CHECK_NULL(table, -EINVAL);
@@ -273,8 +294,9 @@ int sdp_desc_q_type_head(struct sdpc_des
 	}
 } /* sdp_desc_q_type_head */
 
-/* ========================================================================= */
-/*..sdp_desc_q_type_tail - look at the type at the end of the table */
+/*
+ * sdp_desc_q_type_tail - look at the type at the end of the table
+ */
 int sdp_desc_q_type_tail(struct sdpc_desc_q *table)
 {
 	SDP_CHECK_NULL(table, -EINVAL);
@@ -289,8 +311,9 @@ int sdp_desc_q_type_tail(struct sdpc_des
 	}
 } /* sdp_desc_q_type_tail */
 
-/* ========================================================================= */
-/*..sdp_desc_q_look_type_head - look at a specific object */
+/*
+ * sdp_desc_q_look_type_head - look at a specific object
+ */
 struct sdpc_desc *sdp_desc_q_look_type_head(struct sdpc_desc_q *table, 
 					   enum sdp_desc_type type)
 {
@@ -306,8 +329,9 @@ struct sdpc_desc *sdp_desc_q_look_type_h
 	}
 } /* sdp_desc_q_look_type_head */
 
-/* ========================================================================= */
-/*..sdp_desc_q_look_type_tail - look at the type at the end of the table */
+/*
+ * sdp_desc_q_look_type_tail - look at the type at the end of the table
+ */
 struct sdpc_desc *sdp_desc_q_look_type_tail(struct sdpc_desc_q *table,
 					   enum sdp_desc_type type)
 {
@@ -324,8 +348,9 @@ struct sdpc_desc *sdp_desc_q_look_type_t
 	}
 } /* sdp_desc_q_look_type_tail */
 
-/* ========================================================================= */
-/*..sdp_desc_q_types_size - return the number of elements in the table */
+/*
+ * sdp_desc_q_types_size - return the number of elements in the table
+ */
 int sdp_desc_q_types_size(struct sdpc_desc_q *table,
 				 enum sdp_desc_type type)
 {
@@ -335,13 +360,13 @@ int sdp_desc_q_types_size(struct sdpc_de
 		table->count[type] : -ERANGE);
 } /* sdp_desc_q_types_size */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* public table functions                                                */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_desc_q_create - create/allocate a generic table */
+/*
+ * public table functions
+ */
+
+/*
+ * sdp_desc_q_create - create/allocate a generic table
+ */
 struct sdpc_desc_q *sdp_desc_q_create(int *result)
 {
 	struct sdpc_desc_q *table = NULL;
@@ -364,8 +389,9 @@ struct sdpc_desc_q *sdp_desc_q_create(in
 	return table;
 } /* sdp_desc_q_create */
 
-/* ========================================================================= */
-/*..sdp_desc_q_init - initialize a new empty generic table */
+/*
+ * sdp_desc_q_init - initialize a new empty generic table
+ */
 int sdp_desc_q_init(struct sdpc_desc_q *table)
 {
 	SDP_CHECK_NULL(table, -EINVAL);
@@ -378,8 +404,9 @@ int sdp_desc_q_init(struct sdpc_desc_q *
 	return 0;
 } /* sdp_desc_q_init */
 
-/* ========================================================================= */
-/*..sdp_desc_q_clear - clear the contents of a generic table */
+/*
+ * sdp_desc_q_clear - clear the contents of a generic table
+ */
 int sdp_desc_q_clear(struct sdpc_desc_q *table)
 {
 	struct sdpc_desc *element;
@@ -401,8 +428,9 @@ int sdp_desc_q_clear(struct sdpc_desc_q 
 	return 0;
 } /* sdp_desc_q_clear */
 
-/* ========================================================================= */
-/*..sdp_desc_q_destroy - destroy a generic table */
+/*
+ * sdp_desc_q_destroy - destroy a generic table
+ */
 int sdp_desc_q_destroy(struct sdpc_desc_q *table)
 {
 	int result;
@@ -421,13 +449,13 @@ int sdp_desc_q_destroy(struct sdpc_desc_
 	return 0;
 } /* sdp_desc_q_destroy */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* primary initialization/cleanup functions                              */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_main_desc_init -- initialize the generic table caches. */
+/*
+ * primary initialization/cleanup functions
+ */
+
+/*
+ * sdp_main_desc_init - initialize the generic table caches
+ */
 int sdp_main_desc_init(void)
 {
 	int result;
@@ -456,8 +484,9 @@ error_advt_t:
 	return 0;
 } /* sdp_main_desc_init */
 
-/* ========================================================================= */
-/*..sdp_main_desc_cleanup -- cleanup the generic table caches. */
+/*
+ * sdp_main_desc_cleanup - cleanup the generic table caches
+ */
 int sdp_main_desc_cleanup(void)
 {
 	sdp_dbg_init("Generic table cache cleanup.");



From Tom.Duffy at Sun.COM  Thu Feb 17 14:12:46 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:12:46 -0800
Subject: [openib-general] [PATCH][SDP][25/33] Fix comment style in
	sdp_queue.h
In-Reply-To: <1108678356217@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
Message-ID: <11086783663245@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_queue.h	2005-02-10 18:42:56.256023000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_queue.h	2005-02-17 11:22:36.106017000 -0800
@@ -1,25 +1,36 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_queue.h 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_QUEUE_H
 #define _SDP_QUEUE_H
@@ -34,9 +45,9 @@ enum sdp_desc_type {
 	SDP_DESC_TYPE_NONE
 };
 
-/* ----------------------------------------------------------------------- */
-/* SDP generic queue for multiple object types.                            */
-/* ----------------------------------------------------------------------- */
+/*
+ * SDP generic queue for multiple object types
+ */
 
 struct sdpc_desc {
 	struct sdpc_desc   *next;  /* next structure in table */
@@ -45,6 +56,7 @@ struct sdpc_desc {
 	struct sdpc_desc_q *table; /* table to which this object belongs */
 	int (*release)(struct sdpc_desc *element); /* release the object */
 }; /* struct sdpc_desc */
+
 /*
  * table for holding SDP advertisments.
  */
@@ -53,21 +65,22 @@ struct sdpc_desc_q {
 	int size;               /* current number of advertisments in table */
 	u16 count[SDP_DESC_TYPE_NONE]; /* object specific counter */
 }; /* struct sdpc_desc_q */
-/* ----------------------------------------------------------------------- */
-/*                                                                         */
-/* SDP generic queue inline functions.                                     */
-/*                                                                         */
-/* ----------------------------------------------------------------------- */
 
-/* ========================================================================= */
-/*..__sdp_desc_q_size - return the number of elements in the table */
+/*
+ * SDP generic queue inline functions
+ */
+
+/*
+ * __sdp_desc_q_size - return the number of elements in the table
+ */
 static __inline__ int __sdp_desc_q_size(struct sdpc_desc_q *table)
 {
 	return table->size;
 } /* __sdp_desc_q_size */
 
-/* ========================================================================= */
-/*..__sdp_desc_q_member - return non-zero if element is in a table */
+/*
+ * __sdp_desc_q_member - return non-zero if element is in a table
+ */
 static __inline__ int __sdp_desc_q_member(struct sdpc_desc *element)
 {
 	return ((NULL == element->table) ? 0 : 1);



From Tom.Duffy at Sun.COM  Thu Feb 17 14:12:56 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:12:56 -0800
Subject: [openib-general] [PATCH][SDP][26/33] Fix comment style in sdp_rcvd.c
In-Reply-To: <11086783663245@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
Message-ID: <11086783763927@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_rcvd.c	2005-02-15 16:33:46.210001000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_rcvd.c	2005-02-17 11:26:31.100001000 -0800
@@ -1,35 +1,43 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_rcvd.c 836 2004-09-14 20:50:32Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Specific MID handler functions. (RECV)                                */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_rcvd_disconnect --  */
+/*
+ * Specific MID handler functions. (RECV)
+ */
+
 static int _sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
@@ -119,8 +127,6 @@ error:
 	return result;
 } /* _sdp_rcvd_disconnect */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_abort --  */
 static int _sdp_rcvd_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
@@ -158,8 +164,6 @@ static int _sdp_rcvd_abort(struct sdp_op
 	return result;
 } /* _sdp_rcvd_abort */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_send_sm --  */
 static int _sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
@@ -206,8 +210,6 @@ static int _sdp_rcvd_send_sm(struct sdp_
 	return 0;
 } /* _sdp_rcvd_send_sm */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_rdma_wr --  */
 static int _sdp_rcvd_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rwch *rwch;
@@ -275,8 +277,6 @@ error:
 	return result;
 } /* _sdp_rcvd_rdma_wr */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_rdma_rd --  */
 static int _sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rrch *rrch;
@@ -367,8 +367,6 @@ error:
 	return result;
 } /* _sdp_rcvd_rdma_rd */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_mode_change --  */
 static int _sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_mch *mch;
@@ -484,8 +482,6 @@ error:
 	return result;
 } /* _sdp_rcvd_mode_change */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_src_cancel --  */
 static int _sdp_rcvd_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_advt *advt;
@@ -590,8 +586,6 @@ done:
 	return result;
 } /* _sdp_rcvd_src_cancel */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_snk_cancel --  */
 static int _sdp_rcvd_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_advt *advt;
@@ -673,8 +667,9 @@ done:
 	return result;
 } /* _sdp_rcvd_snk_cancel */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_snk_cancel_ack -- sink cancel confirmantion */
+/*
+ * _sdp_rcvd_snk_cancel_ack - sink cancel confirmantion
+ */
 static int _sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, 
 				    struct sdpc_buff *buff)
 {
@@ -717,8 +712,9 @@ done:
 	return result;
 } /* _sdp_rcvd_snk_cancel_ack */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_resize_buff_ack --  buffer size change request */
+/*
+ * _sdp_rcvd_resize_buff_ack - buffer size change request
+ */
 static int _sdp_rcvd_resize_buff_ack(struct sdp_opt *conn,
 				     struct sdpc_buff *buff)
 {
@@ -753,8 +749,6 @@ error:
 	return result;
 } /* _sdp_rcvd_resize_buff_ack */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_suspend --  */
 static int _sdp_rcvd_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_sch *sch;
@@ -770,18 +764,14 @@ static int _sdp_rcvd_suspend(struct sdp_
 	return 0;
 } /* _sdp_rcvd_suspend */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_suspend_Ack --  */
-static int _sdp_rcvd_suspend_Ack(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int _sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
 	SDP_CHECK_NULL(buff, -EINVAL);
 
 	return 0;
-} /* _sdp_rcvd_suspend_Ack */
+} /* _sdp_rcvd_suspend_ack */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_snk_avail --  */
 static int _sdp_rcvd_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_snkah *snkah;
@@ -944,8 +934,6 @@ error:
 	return result;
 } /* _sdp_rcvd_snk_avail */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_src_avail --  */
 static int _sdp_rcvd_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_srcah *srcah;
@@ -1140,8 +1128,9 @@ done:
 	return result;
 } /* _sdp_rcvd_src_avail */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_data -- SDP data message event received */
+/*
+ * _sdp_rcvd_data - SDP data message event received
+ */
 static int _sdp_rcvd_data(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int ret_val;
@@ -1196,8 +1185,9 @@ static int _sdp_rcvd_data(struct sdp_opt
 	return ret_val;
 } /* _sdp_rcvd_sata */
 
-/* ========================================================================= */
-/*.._sdp_rcvd_unsupported -- Valid messages we're not expecting */
+/*
+ * _sdp_rcvd_unsupported - Valid messages we're not expecting
+ */
 static int _sdp_rcvd_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -1241,7 +1231,7 @@ static sdp_event_cb_func recv_event_func
 	_sdp_rcvd_resize_buff_ack,  /* SDP_MID_CH_RECV_BUF      0x0B */
 	_sdp_rcvd_unsupported,      /* SDP_MID_CH_RECV_BUF_ACK  0x0C */
 	_sdp_rcvd_suspend,          /* SDP_MID_SUSPEND          0x0D */
-	_sdp_rcvd_suspend_Ack,      /* SDP_MID_SUSPEND_ACK      0x0E */
+	_sdp_rcvd_suspend_ack,      /* SDP_MID_SUSPEND_ACK      0x0E */
 	NULL,                       /* reserved                    0x0F */
 	NULL,                       /* reserved                    0xF0 */
 	NULL,                       /* reserved                    0xF1 */
@@ -1261,8 +1251,9 @@ static sdp_event_cb_func recv_event_func
 	_sdp_rcvd_data              /* SDP_MID_DATA             0xFF */
 }; /* recv_event_funcs */
 
-/* ========================================================================= */
-/*..sdp_event_recv -- recv event demultiplexing into sdp messages. */
+/*
+ * sdp_event_recv - recv event demultiplexing into sdp messages
+ */
 int sdp_event_recv(struct sdp_opt *conn, struct ib_wc *comp)
 {
 	sdp_event_cb_func dispatch_func;



From mst at mellanox.co.il  Thu Feb 17 14:13:30 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 18 Feb 2005 00:13:30 +0200
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217140015.B7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
Message-ID: <20050217221330.GA21318@mellanox.co.il>


Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [openib-general] SDP_CONN_LOCK
> 
> On Thu, Feb 17, 2005 at 11:29:37PM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > Subject: Re: [openib-general] SDP_CONN_LOCK
> > > 
> > > On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > > > Hi, Libor!
> > > > Could you please explain what are the SDP_CONN_LOCK
> > > > and friends doing in SDP?
> > > > 
> > > > It seems they just implement exclusive access to socket,
> > > > but if so, why is a simple semaphore or mutex not used?
> > > 
> > >   They do implement exclusive access to the socket, but they implement
> > > exclusive access from both process and irq context, which is why a
> > > semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> > > to lock the connection, look in sdp_cq_event_handler() for it's use,
> > > and in process context SDP_CONN_LOCK is used.
> > 
> > I dont really understand how it works.
> > When an interrupt arrives while users != 0, it seems you are
> > calling scheduler().
> > What is sdp_conn_internal_lock doing? I understand it is to be called
> > from interrupt context, but how can it call scheduler() then?
> 
>   SDP_CONN_LOCK and SDP_CONN_UNLOCK are never called from interrupt
> context, only from process context. The spinlock conn->lock.slock
> is called in interrupt context and process context before the 'users'
> variable is changed or read. If users != 0, then the connection is in
> use by process context. If the connection is in use then another lock
> request in process context will sleep until users == 0, if the connection
> is in use then a lock request in interrupt context will mark the
> connection as having had an event and return. When a process context
> lock holder releases the lock, it checks if there has been an interrupt
> event, which it then processes by draining and arming the CQ, and it
> wakes any process context waiting for the lock.
> 
> 
> -Libor
> 

Cant all this be done by a semaphore in process context, and 
down_trylock in interrupt context?
The "connection has an event" bit could be protected by a spinlock.

-- 
MST - Michael S. Tsirkin


From Tom.Duffy at Sun.COM  Thu Feb 17 14:13:07 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:13:07 -0800
Subject: [openib-general] [PATCH][SDP][27/33] Fix comment style in sdp_read.c
In-Reply-To: <11086783763927@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com> <11086783763927@sun.com>
Message-ID: <11086783872748@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_read.c	2005-02-15 16:33:46.220000000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_read.c	2005-02-17 11:30:04.047019000 -0800
@@ -1,36 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_read.c 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* RDMA read processing functions                                        */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * RDMA read processing functions
+ */
 
-/* ========================================================================= */
-/*.._sdp_event_read_advt -- RDMA read event handler for source advertisments. */
+/*
+ * _sdp_event_read_advt - RDMA read event handler for source advertisments
+ */
 static int _sdp_event_read_advt(struct sdp_opt *conn, struct ib_wc *comp)
 {
 	struct sdpc_advt *advt;
@@ -98,13 +108,13 @@ error:
 	return result;
 } /* _sdp_event_read_advt */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* RDMA read QP Event Handler                                            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_event_read -- RDMA read event handler. */
+/*
+ * RDMA read QP Event Handler
+ */
+
+/*
+ * sdp_event_read - RDMA read event handler
+ */
 int sdp_event_read(struct sdp_opt *conn, struct ib_wc *comp)
 {
 	struct sdpc_iocb *iocb;



From Tom.Duffy at Sun.COM  Thu Feb 17 14:13:17 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:13:17 -0800
Subject: [openib-general] [PATCH][SDP][28/33] Fix comment style in sdp_recv.c
In-Reply-To: <11086783872748@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
	<11086783763927@sun.com> <11086783872748@sun.com>
Message-ID: <11086783972515@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_recv.c	2005-02-14 13:58:45.877005000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_recv.c	2005-02-17 12:03:28.285005000 -0800
@@ -1,34 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
+#include "sdp_main.h"
 
-  $Id: sdp_recv.c 836 2004-09-14 20:50:32Z roland $
-*/
+/*
+ * Receive posting function(s)
+ */
 
-#include "sdp_main.h"
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Receive posting function(s)                                           */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_post_recv_buff -- post a single buffers for data recv */
+/*
+ * _sdp_post_recv_buff - post a single buffers for data recv
+ */
 static int _sdp_post_recv_buff(struct sdp_opt *conn)
 {
 	struct ib_recv_wr receive_param = { NULL };
@@ -104,8 +116,9 @@ error:
 	return result;
 } /* _sdp_post_recv_buff */
 
-/* ========================================================================= */
-/*.._sdp_post_rdma_buff -- post a single buffers for rdma read on a conn */
+/*
+ * _sdp_post_rdma_buff - post a single buffers for rdma read on a conn
+ */
 static int _sdp_post_rdma_buff(struct sdp_opt *conn)
 {
 	struct ib_send_wr send_param = { NULL };
@@ -233,8 +246,9 @@ done:
 	return result;
 } /* _sdp_post_rdma_buff */
 
-/* ========================================================================= */
-/*.._sdp_post_rdma_iocb_src -- post a iocb for rdma read on a conn */
+/*
+ * _sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn
+ */
 static int _sdp_post_rdma_iocb_src(struct sdp_opt *conn)
 {
 	struct ib_send_wr send_param = { NULL };
@@ -388,8 +402,9 @@ done:
 	return result;
 } /* _sdp_post_rdma_iocb_src */
 
-/* ========================================================================= */
-/*.._sdp_post_rdma_iocb_snk -- post a iocb for rdma read on a conn */
+/*
+ * _sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn
+ */
 static int _sdp_post_rdma_iocb_snk(struct sdp_opt *conn)
 {
 	struct sdpc_iocb *iocb;
@@ -497,8 +512,9 @@ error:
 	return result;
 } /* _sdp_post_rdma_iocb_snk */
 
-/* ========================================================================= */
-/*.._sdp_post_rdma -- post a rdma based requests for a connection */
+/*
+ * _sdp_post_rdma - post a rdma based requests for a connection
+ */
 static int _sdp_post_rdma(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -592,8 +608,9 @@ done:
 	return result;
 } /* _sdp_post_rdma */
 
-/* ========================================================================= */
-/*..sdp_recv_flush -- post a certain number of buffers on a connection */
+/*
+ * sdp_recv_flush - post a certain number of buffers on a connection
+ */
 int sdp_recv_flush(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -701,13 +718,13 @@ done:
 	return result;
 } /* sdp_recv_flush */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Receive incoming data function(s)                                     */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..__sdp_read_buff_iocb -- read a SDP buffer into an IOCB  */
+/*
+ * Receive incoming data function(s)
+ */
+
+/*
+ * __sdp_read_buff_iocb - read a SDP buffer into an IOCB
+ */
 static int _sdp_read_buff_iocb(struct sdpc_iocb *iocb,
 			       struct sdpc_buff *buff)
 {
@@ -771,8 +788,9 @@ static int _sdp_read_buff_iocb(struct sd
 	return 0;
 } /* _sdp_read_buff_iocb */
 
-/* ========================================================================= */
-/*.._sdp_recv_buff_iocb_active -- Ease AIO read pending pressure */
+/*
+ * _sdp_recv_buff_iocb_active - Ease AIO read pending pressure
+ */
 static int _sdp_recv_buff_iocb_active(struct sdp_opt *conn,
 				      struct sdpc_buff *buff)
 {
@@ -825,8 +843,9 @@ static int _sdp_recv_buff_iocb_active(st
 	return (buff->tail - buff->data);
 } /* _sdp_recv_buff_iocb_active */
 
-/* ========================================================================= */
-/*.._sdp_recv_buff_iocb_pending -- Ease AIO read pending pressure */
+/*
+ * _sdp_recv_buff_iocb_pending - Ease AIO read pending pressure
+ */
 static int _sdp_recv_buff_iocb_pending(struct sdp_opt *conn,
 				       struct sdpc_buff *buff)
 {
@@ -890,8 +909,9 @@ static int _sdp_recv_buff_iocb_pending(s
 	return (buff->tail - buff->data);
 } /* _sdp_recv_buff_iocb_pending */
 
-/* ========================================================================= */
-/*..sdp_recv_buff -- Process a new buffer based on queue type. */
+/*
+ * sdp_recv_buff - Process a new buffer based on queue type
+ */
 int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result;
@@ -996,13 +1016,13 @@ done:
 	return result;
 } /* sdp_recv_buff */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* User initiated receive data function(s)                               */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_read_src_lookup -- lookup function for cancelation */
+/*
+ * User initiated receive data function(s)
+ */
+
+/*
+ * _sdp_read_src_lookup - lookup function for cancelation
+ */
 static int _sdp_read_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
@@ -1021,8 +1041,9 @@ static int _sdp_read_src_lookup(struct s
 	}
 } /* _sdp_read_src_lookup */
 
-/* ========================================================================= */
-/*.._sdp_inet_read_cancel -- cancel an IO operation */
+/*
+ * _sdp_inet_read_cancel - cancel an IO operation
+ */
 static int _sdp_inet_read_cancel(struct kiocb *req,
 				 struct io_event *ev)
 {
@@ -1163,8 +1184,9 @@ done:
 	return result;
 } /* _sdp_inet_read_cancel */
 
-/* ========================================================================= */
-/*.._sdp_inet_recv_urg_test -- recv queue urgent data cleanup function */
+/*
+ * _sdp_inet_recv_urg_test - recv queue urgent data cleanup function
+ */
 static int _sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
 {
 	SDP_CHECK_NULL(buff, -EINVAL);
@@ -1172,8 +1194,9 @@ static int _sdp_inet_recv_urg_test(struc
 	return ((buff->tail == buff->head) ? 1 : 0);
 } /* _sdp_inet_recv_urg_test */
 
-/* ========================================================================= */
-/*.._sdp_inet_recv_urg_trav -- recv queue urg data retreival function */
+/*
+ * _sdp_inet_recv_urg_trav - recv queue urg data retreival function
+ */
 static int _sdp_inet_recv_urg_trav(struct sdpc_buff *buff, void *arg)
 {
 	u8 *value = (u8 *) arg;
@@ -1201,8 +1224,9 @@ static int _sdp_inet_recv_urg_trav(struc
 	return 0;
 } /* _sdp_inet_recv_urg_trav */
 
-/* ========================================================================= */
-/*.._sdp_inet_recv_urg -- recv urgent data from the network to user space */
+/*
+ * _sdp_inet_recv_urg - recv urgent data from the network to user space
+ */
 static int _sdp_inet_recv_urg(struct sock *sk,
 			      struct msghdr *msg,
 			      int size,
@@ -1287,8 +1311,9 @@ done:
 	return result;
 } /* _sdp_inet_recv_urg */
 
-/* ========================================================================= */
-/*..sdp_inet_recv -- recv data from the network to user space. */
+/*
+ * sdp_inet_recv - recv data from the network to user space
+ */
 int sdp_inet_recv(struct kiocb  *req,
 		  struct socket *sock,
 		  struct msghdr *msg, 



From Tom.Duffy at Sun.COM  Thu Feb 17 14:13:27 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:13:27 -0800
Subject: [openib-general] [PATCH][SDP][29/33] Fix comment style in sdp_send.c
In-Reply-To: <11086783972515@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
	<11086783763927@sun.com> <11086783872748@sun.com>
	<11086783972515@sun.com>
Message-ID: <11086784072107@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_send.c	2005-02-14 13:58:45.860001000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_send.c	2005-02-17 11:38:10.806001000 -0800
@@ -1,35 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_send.c 836 2004-09-14 20:50:32Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/*                          COMMON functions                             */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_send_buff_post -- Post a buffer send on a SDP connection. */
+/*
+ * COMMON functions
+ */
+
+/*
+ * _sdp_send_buff_post - Post a buffer send on a SDP connection
+ */
 static int _sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct ib_send_wr send_param = { NULL };
@@ -187,14 +198,13 @@ done:
 	return result;
 } /* _sdp_send_buff_post */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/*                          DATA functions                               */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * DATA functions
+ */
 
-/* ========================================================================= */
-/*.._sdp_send_data_buff_post -- Post data for buffered transmission */
+/*
+ * _sdp_send_data_buff_post - Post data for buffered transmission
+ */
 static int _sdp_send_data_buff_post(struct sdp_opt *conn,
 				    struct sdpc_buff *buff)
 {
@@ -317,8 +327,9 @@ error:
 	return result;
 } /* _sdp_send_data_buff_post */
 
-/* ========================================================================= */
-/*.._sdp_send_data_buff_snk -- Post data for buffered transmission */
+/*
+ * _sdp_send_data_buff_snk - Post data for buffered transmission
+ */
 static int _sdp_send_data_buff_snk(struct sdp_opt *conn,
 				   struct sdpc_buff *buff)
 {
@@ -467,8 +478,9 @@ error:
 	return result;
 } /* _sdp_send_data_buff_snk */
 
-/* ========================================================================= */
-/*.._sdp_send_data_iocb_snk -- process a zcopy write advert in the data path */
+/*
+ * _sdp_send_data_iocb_snk - process a zcopy write advert in the data path
+ */
 static int _sdp_send_data_iocb_snk(struct sdp_opt *conn,
 				   struct sdpc_iocb *iocb)
 {
@@ -601,8 +613,9 @@ error:
 	return result;
 } /* _sdp_send_data_iocb_snk */
 
-/* ========================================================================= */
-/*.._sdp_send_data_iocb_src -- send a zcopy read advert in the data path */
+/*
+ * _sdp_send_data_iocb_src - send a zcopy read advert in the data path
+ */
 static int _sdp_send_data_iocb_src(struct sdp_opt *conn,
 				   struct sdpc_iocb *iocb)
 {
@@ -789,8 +802,9 @@ drop:
 	return result;
 } /* _sdp_send_data_iocb_src */
 
-/* ========================================================================= */
-/*.._sdp_send_iocb_buff_write -- write part of an iocb into a SDP buffer */
+/*
+ * _sdp_send_iocb_buff_write - write part of an iocb into a SDP buffer
+ */
 static int _sdp_send_iocb_buff_write(struct sdpc_iocb *iocb,
 				     struct sdpc_buff *buff)
 
@@ -840,8 +854,9 @@ static int _sdp_send_iocb_buff_write(str
 	return 0;
 } /* _sdp_send_iocb_buff_write */
 
-/* ========================================================================= */
-/*.._sdp_send_data_iocb_buff -- write multiple SDP buffers from an iocb */
+/*
+ * _sdp_send_data_iocb_buff - write multiple SDP buffers from an iocb
+ */
 static int _sdp_send_data_iocb_buff(struct sdp_opt *conn,
 				    struct sdpc_iocb *iocb)
 {
@@ -921,8 +936,9 @@ error:
 	return result;
 } /* _sdp_send_data_iocb_buff */
 
-/* ========================================================================= */
-/*.._sdp_send_data_iocb -- Post IOCB data for  transmission */
+/*
+ * _sdp_send_data_iocb - Post IOCB data for transmission
+ */
 static int _sdp_send_data_iocb(struct sdp_opt *conn, struct sdpc_iocb *iocb)
 {
 	int result = ENOBUFS;
@@ -1037,8 +1053,9 @@ done:
 	return result;
 } /* _sdp_send_data_iocb */
 
-/* ========================================================================= */
-/*.._sdp_send_data_queue_test -- send data buffer if conditions are met */
+/*
+ * _sdp_send_data_queue_test - send data buffer if conditions are met
+ */
 static int _sdp_send_data_queue_test(struct sdp_opt *conn,
 				     struct sdpc_desc *element)
 {
@@ -1077,8 +1094,9 @@ static int _sdp_send_data_queue_test(str
 	return result;
 } /* _sdp_send_data_queue_test */
 
-/* ========================================================================= */
-/*.._sdp_send_data_queue_flush -- Flush data from send queue, to send post. */
+/*
+ * _sdp_send_data_queue_flush - Flush data from send queue, to send post
+ */
 static int _sdp_send_data_queue_flush(struct sdp_opt *conn)
 {
 	struct sdpc_desc *element;
@@ -1125,8 +1143,9 @@ static int _sdp_send_data_queue_flush(st
 	return result;
 } /* _sdp_send_data_queue_flush */
 
-/* ========================================================================= */
-/*.._sdp_send_data_queue -- send using the data queue if necessary. */
+/*
+ * _sdp_send_data_queue - send using the data queue if necessary
+ */
 static int _sdp_send_data_queue(struct sdp_opt *conn,
 				struct sdpc_desc *element)
 {
@@ -1182,8 +1201,9 @@ done:
 	return result;
 } /* _sdp_send_data_queue */
 
-/* ========================================================================= */
-/*.._sdp_send_data_buff_get -- get an appropriate write buffer for send */
+/*
+ * _sdp_send_data_buff_get - get an appropriate write buffer for send
+ */
 static __inline__ struct sdpc_buff *_sdp_send_data_buff_get(
 	struct sdp_opt *conn)
 {
@@ -1210,8 +1230,9 @@ static __inline__ struct sdpc_buff *_sdp
 	return buff;
 } /* _sdp_send_data_buff_get */
 
-/* ========================================================================= */
-/*.._sdp_send_data_buff_put -- place a buffer into the send queue */
+/*
+ * _sdp_send_data_buff_put - place a buffer into the send queue
+ */
 static __inline__ int _sdp_send_data_buff_put(struct sdp_opt *conn,
 					      struct sdpc_buff *buff,
 					      int size,
@@ -1271,13 +1292,13 @@ static __inline__ int _sdp_send_data_buf
 	return result;
 } /* _sdp_send_data_buff_put */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/*                          CONTROL functions                            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_send_ctrl_buff_test -- determine if it's OK to post a control msg */
+/*
+ * CONTROL functions
+ */
+
+/*
+ * _sdp_send_ctrl_buff_test - determine if it's OK to post a control msg
+ */
 static int _sdp_send_ctrl_buff_test(struct sdp_opt *conn,
 				    struct sdpc_buff *buff)
 {
@@ -1308,8 +1329,9 @@ error:
 	return result;
 } /* _sdp_send_ctrl_buff_test */
 
-/* ========================================================================= */
-/*.._sdp_send_ctrl_buff_flush -- Flush control buffers, to send post. */
+/*
+ * _sdp_send_ctrl_buff_flush - Flush control buffers, to send post
+ */
 static int _sdp_send_ctrl_buff_flush(struct sdp_opt *conn)
 {
 	struct sdpc_desc *element;
@@ -1350,8 +1372,9 @@ static int _sdp_send_ctrl_buff_flush(str
 	return result;
 } /* _sdp_send_ctrl_buff_flush */
 
-/* ========================================================================= */
-/*.._sdp_send_ctrl_buff_buffered -- Send a buffered control message. */
+/*
+ * _sdp_send_ctrl_buff_buffered - Send a buffered control message
+ */
 static int _sdp_send_ctrl_buff_buffered(struct sdp_opt *conn,
 					struct sdpc_buff *buff)
 {
@@ -1389,8 +1412,9 @@ error:
 	return result;
 } /* _sdp_send_ctrl_buff_buffered */
 
-/* ========================================================================= */
-/*.._sdp_send_ctrl_buff -- Create and Send a buffered control message. */
+/*
+ * _sdp_send_ctrl_buff - Create and Send a buffered control message
+ */
 static int _sdp_send_ctrl_buff(struct sdp_opt *conn,
 			       u8 mid,
 			       int se,
@@ -1456,8 +1480,9 @@ error:
 	return result;
 } /* _sdp_send_ctrl_buff */
 
-/* ========================================================================= */
-/*.._sdp_send_ctrl_disconnect -- Send a disconnect request. */
+/*
+ * _sdp_send_ctrl_disconnect - Send a disconnect request
+ */
 static int _sdp_send_ctrl_disconnect(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -1507,8 +1532,9 @@ error:
 	return result;
 } /* _sdp_send_ctrl_disconnect */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_disconnect -- potentially send a disconnect request. */
+/*
+ * sdp_send_ctrl_disconnect - potentially send a disconnect request
+ */
 int sdp_send_ctrl_disconnect(struct sdp_opt *conn)
 {
 	int result;
@@ -1545,8 +1571,9 @@ int sdp_send_ctrl_disconnect(struct sdp_
 	return result;
 } /* sdp_send_ctrl_disconnect */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_ack -- Send a gratuitous Ack. */
+/*
+ * sdp_send_ctrl_ack - Send a gratuitous Ack
+ */
 int sdp_send_ctrl_ack(struct sdp_opt *conn)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -1569,36 +1596,41 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
 	return _sdp_send_ctrl_buff(conn, SDP_MID_DATA, 0, 0);
 } /* sdp_send_ctrl_ack */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_send_sm -- Send a request for buffered mode. */
+/*
+ * sdp_send_ctrl_send_sm - Send a request for buffered mode
+ */
 int sdp_send_ctrl_send_sm(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SEND_SM, 1, 1);
 } /* sdp_send_ctrl_send_sm */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_src_cancel -- Send a source cancel */
+/*
+ * sdp_send_ctrl_src_cancel - Send a source cancel
+ */
 int sdp_send_ctrl_src_cancel(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SRC_CANCEL, 1, 1);
 } /* sdp_send_ctrl_src_cancel */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_snk_cancel -- Send a sink cancel */
+/*
+ * sdp_send_ctrl_snk_cancel - Send a sink cancel
+ */
 int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL, 1, 1);
 } /* sdp_send_ctrl_snk_cancel */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_snk_cancel_ack -- Send an ack for a sink cancel */
+/*
+ * sdp_send_ctrl_snk_cancel_ack - Send an ack for a sink cancel
+ */
 int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL_ACK, 1, 1);
 } /* sdp_send_ctrl_snk_cancel_ack */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_abort -- Send an abort message. */
+/*
+ * sdp_send_ctrl_abort - Send an abort message
+ */
 int sdp_send_ctrl_abort(struct sdp_opt *conn)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -1608,8 +1640,9 @@ int sdp_send_ctrl_abort(struct sdp_opt *
 	return _sdp_send_ctrl_buff(conn, SDP_MID_ABORT_CONN, 1, 1);
 } /* sdp_send_ctrl_abort */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_resize_buff_ack -- Send an ack for a buffer size change */
+/*
+ * sdp_send_ctrl_resize_buff_ack - Send an ack for a buffer size change
+ */
 int sdp_send_ctrl_resize_buff_ack(struct sdp_opt *conn, u32 size)
 {
 	struct msg_hdr_crbah *crbah;
@@ -1661,8 +1694,9 @@ error:
 	return result;
 } /* sdp_send_ctrl_resize_buff_ack */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_rdma_rd -- Send an rdma read completion */
+/*
+ * sdp_send_ctrl_rdma_rd - Send an rdma read completion
+ */
 int sdp_send_ctrl_rdma_rd(struct sdp_opt *conn, s32 size)
 {
 	struct msg_hdr_rrch *rrch;
@@ -1733,8 +1767,9 @@ error:
 	return result;
 } /* sdp_send_ctrl_rdma_rd */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_rdma_wr -- Send an rdma write completion */
+/*
+ * sdp_send_ctrl_rdma_wr - Send an rdma write completion
+ */
 int sdp_send_ctrl_rdma_wr(struct sdp_opt *conn, u32 size)
 {
 	struct msg_hdr_rwch *rwch;
@@ -1792,8 +1827,9 @@ error:
 	return result;
 } /* sdp_send_ctrl_rdma_wr */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_snk_avail -- Send a sink available message */
+/*
+ * sdp_send_ctrl_snk_avail - Send a sink available message
+ */
 int sdp_send_ctrl_snk_avail(struct sdp_opt *conn,
 			    u32 size,
 			    u32 rkey,
@@ -1861,8 +1897,9 @@ error:
 	return result;
 } /* sdp_send_ctrl_snk_avail */
 
-/* ========================================================================= */
-/*..sdp_send_ctrl_mode_ch -- Send a mode change command */
+/*
+ * sdp_send_ctrl_mode_ch - Send a mode change command
+ */
 int sdp_send_ctrl_mode_ch(struct sdp_opt *conn, u8 mode)
 {
 	struct msg_hdr_mch *mch;
@@ -1955,13 +1992,13 @@ error:
 	return result;
 } /* sdp_send_ctrl_mode_ch */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/*                          GENERAL functions                            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_write_src_lookup -- lookup function for cancelation */
+/*
+ * GENERAL functions
+ */
+
+/*
+ * _sdp_write_src_lookup - lookup function for cancelation
+ */
 static int _sdp_write_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
@@ -1979,8 +2016,9 @@ static int _sdp_write_src_lookup(struct 
 	}
 } /* _sdp_write_src_lookup */
 
-/* ========================================================================= */
-/*.._sdp_inet_write_cancel -- cancel an IO operation */
+/*
+ * _sdp_inet_write_cancel - cancel an IO operation
+ */
 static int _sdp_inet_write_cancel(struct kiocb *req, struct io_event *ev)
 {
 	struct sock_iocb *si = kiocb_to_siocb(req);
@@ -2137,8 +2175,9 @@ done:
 	return result;
 } /* _sdp_inet_write_cancel */
 
-/* ========================================================================= */
-/*.._sdp_send_flush_advt -- Flush passive sink advertisments */
+/*
+ * _sdp_send_flush_advt - Flush passive sink advertisments
+ */
 static int _sdp_send_flush_advt(struct sdp_opt *conn)
 {
 	struct sdpc_advt *advt;
@@ -2176,8 +2215,9 @@ static int _sdp_send_flush_advt(struct s
 	return 0;
 } /* _sdp_send_flush_advt */
 
-/* ========================================================================= */
-/*..sdp_send_flush -- Flush buffers from send queue, in to send post. */
+/*
+ * sdp_send_flush - Flush buffers from send queue, in to send post
+ */
 int sdp_send_flush(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -2240,8 +2280,9 @@ done:
 	return result;
 } /* sdp_send_flush */
 
-/* ========================================================================= */
-/*..sdp_inet_send -- send data from user space to the network. */
+/*
+ * sdp_inet_send - send data from user space to the network
+ */
 int sdp_inet_send(struct kiocb *req,
 		  struct socket *sock,
 		  struct msghdr *msg,



From Tom.Duffy at Sun.COM  Thu Feb 17 14:13:37 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:13:37 -0800
Subject: [openib-general] [PATCH][SDP][30/33] Fix comment style in sdp_sent.c
In-Reply-To: <11086784072107@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
	<11086783763927@sun.com> <11086783872748@sun.com>
	<11086783972515@sun.com> <11086784072107@sun.com>
Message-ID: <11086784171812@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_sent.c	2005-02-15 16:33:46.247005000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_sent.c	2005-02-17 11:52:21.451002000 -0800
@@ -1,35 +1,43 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_sent.c 836 2004-09-14 20:50:32Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* Specific MID handler functions. (SEND)                                */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*.._sdp_sent_disconnect --  */
+/*
+ * Specific MID handler functions. (SEND)
+ */
+
 static int _sdp_sent_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result;
@@ -115,8 +123,6 @@ error:
 	return result;
 } /* _sdp_sent_disconnect */
 
-/* ========================================================================= */
-/*.._sdp_sent_abort --  */
 static int _sdp_sent_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result;
@@ -139,8 +145,6 @@ static int _sdp_sent_abort(struct sdp_op
 	return result;
 } /* _sdp_sent_abort */
 
-/* ========================================================================= */
-/*.._sdp_sent_send_sm --  */
 static int _sdp_sent_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -149,8 +153,6 @@ static int _sdp_sent_send_sm(struct sdp_
 	return 0;
 } /* _sdp_sent_send_sm */
 
-/* ========================================================================= */
-/*.._sdp_sent_rdma_wr --  */
 static int _sdp_sent_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rwch *rwch;
@@ -166,8 +168,6 @@ static int _sdp_sent_rdma_wr(struct sdp_
 	return 0;
 } /* _sdp_sent_rdma_wr */
 
-/* ========================================================================= */
-/*.._sdp_sent_rdma_rd --  */
 static int _sdp_sent_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rrch *rrch;
@@ -183,8 +183,6 @@ static int _sdp_sent_rdma_rd(struct sdp_
 	return 0;
 } /* _sdp_sent_rdma_rd */
 
-/* ========================================================================= */
-/*.._sdp_sent_mode_change --  */
 static int _sdp_sent_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_mch *mch;
@@ -200,8 +198,6 @@ static int _sdp_sent_mode_change(struct 
 	return 0;
 } /* _sdp_sent_mode_change */
 
-/* ========================================================================= */
-/*.._sdp_sent_src_cancel --  */
 static int _sdp_sent_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -210,8 +206,6 @@ static int _sdp_sent_src_cancel(struct s
 	return 0;
 } /* _sdp_sent_src_cancel */
 
-/* ========================================================================= */
-/*.._sdp_sent_snk_cancel --  */
 static int _sdp_sent_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -220,8 +214,6 @@ static int _sdp_sent_snk_cancel(struct s
 	return 0;
 } /* _sdp_sent_snk_cancel */
 
-/* ========================================================================= */
-/*.._sdp_sent_snk_cancel_ack --  */
 static int _sdp_sent_snk_cancel_ack(struct sdp_opt *conn,
 				    struct sdpc_buff *buff)
 {
@@ -231,8 +223,6 @@ static int _sdp_sent_snk_cancel_ack(stru
 	return 0;
 } /* _sdp_sent_snk_cancel_ack */
 
-/* ========================================================================= */
-/*.._sdp_sent_resize_buff_ack --  */
 static int _sdp_sent_resize_buff_ack(struct sdp_opt *conn,
 				     struct sdpc_buff *buff)
 {
@@ -249,8 +239,6 @@ static int _sdp_sent_resize_buff_ack(str
 	return 0;
 } /* _sdp_sent_resize_buff_ack */
 
-/* ========================================================================= */
-/*.._sdp_sent_suspend --  */
 static int _sdp_sent_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_sch *sch;
@@ -266,8 +254,6 @@ static int _sdp_sent_suspend(struct sdp_
 	return 0;
 } /* _sdp_sent_suspend */
 
-/* ========================================================================= */
-/*.._sdp_sent_suspend_ack --  */
 static int _sdp_sent_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -276,8 +262,6 @@ static int _sdp_sent_suspend_ack(struct 
 	return 0;
 } /* _sdp_sent_suspend_ack */
 
-/* ========================================================================= */
-/*.._sdp_sent_snk_avail --  */
 static int _sdp_sent_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_snkah *snkah;
@@ -293,8 +277,6 @@ static int _sdp_sent_snk_avail(struct sd
 	return 0;
 } /* _sdp_sent_snk_avail */
 
-/* ========================================================================= */
-/*.._sdp_sent_src_avail --  */
 static int _sdp_sent_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_srcah *srcah;
@@ -310,8 +292,9 @@ static int _sdp_sent_src_avail(struct sd
 	return 0;
 } /* _sdp_sent_src_avail */
 
-/* ========================================================================= */
-/*.._sdp_sent_data -- SDP data message event received */
+/*
+ * _sdp_sent_data - SDP data message event received
+ */
 static int _sdp_sent_data(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
@@ -325,8 +308,9 @@ static int _sdp_sent_data(struct sdp_opt
 	return result;
 } /* _sdp_sent_data */
 
-/* ========================================================================= */
-/*.._sdp_sent_unsupported -- Valid messages we're not sending */
+/*
+ * _sdp_sent_unsupported - Valid messages we're not sending
+ */
 static int _sdp_sent_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -388,8 +372,9 @@ static sdp_event_cb_func send_event_func
 	_sdp_sent_data             /* SDP_MID_DATA             0xFF */
 }; /* send_event_funcs */
 
-/* ========================================================================= */
-/*..sdp_event_send -- send event handler. */
+/*
+ * sdp_event_send - send event handler
+ */
 int sdp_event_send(struct sdp_opt *conn, struct ib_wc *comp)
 {
 	sdp_event_cb_func dispatch_func;



From Tom.Duffy at Sun.COM  Thu Feb 17 14:13:48 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:13:48 -0800
Subject: [openib-general] [PATCH][SDP][31/33] Fix comment style in sdp_sock.h
In-Reply-To: <11086784171812@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
	<11086783763927@sun.com> <11086783872748@sun.com>
	<11086783972515@sun.com>
	<11086784072107@sun.com> <11086784171812@sun.com>
Message-ID: <11086784271466@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_sock.h	2005-02-11 17:28:16.827018000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_sock.h	2005-02-17 11:53:24.583003000 -0800
@@ -1,28 +1,40 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_sock.h 140 2004-05-11 00:30:52Z libor $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #ifndef _SDP_SOCK_H
 #define _SDP_SOCK_H
+
 /*
  * SDP socket protocol/address family for socket() function. For all other
  * functions (e.g bind, connect, etc.) Either AF_INET or AF_INET_SDP can
@@ -31,13 +43,16 @@
 #define AF_INET_SDP  27
 #define PF_INET_SDP  AF_INET_SDP
 #define AF_INET_STR "AF_INET_SDP" /* SDP enabled environment variable */
+
 /*
  * Socket option level for SDP specific parameters.
  */
 #define SOL_SDP   1025
-/* ------------------------------------------------------------------------ */
-/* Socket options which are SDP specific.                                   */
-/* ------------------------------------------------------------------------ */
+
+/*
+ * Socket options which are SDP specific
+ */
+
 /*
  * zero copy transfer thresholds. ({get,set}sockopt parameter optval is of
  *                                 type 'int')
@@ -45,6 +60,7 @@
 #define SDP_ZCOPY_THRSH_SRC  257 /* Threshold for AIO write advertisments */
 #define SDP_ZCOPY_THRSH_SNK  258 /* Threshold for AIO read advertisments */
 #define SDP_ZCOPY_THRSH      256 /* Convenience for read and write */
+
 /*
  * Default values for SDP specific socket options. (for reference)
  */



From Tom.Duffy at Sun.COM  Thu Feb 17 14:13:58 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:13:58 -0800
Subject: [openib-general] [PATCH][SDP][32/33] Fix comment style in sdp_wall.c
In-Reply-To: <11086784271466@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
	<11086783763927@sun.com> <11086783872748@sun.com>
	<11086783972515@sun.com>
	<11086784072107@sun.com> <11086784171812@sun.com>
	<11086784271466@sun.com>
Message-ID: <1108678438289@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_wall.c	2005-02-10 18:42:56.553008000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_wall.c	2005-02-17 11:56:59.912002000 -0800
@@ -1,35 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_wall.c 488 2004-07-16 23:40:31Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* SDP protocol (public functions)                                       */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_wall_send_reject -- callback to reject an active open */
+/*
+ * SDP protocol (public functions)
+ */
+
+/*
+ * sdp_wall_send_reject - callback to reject an active open
+ */
 int sdp_wall_send_reject(struct sdp_opt *conn)
 {
 	int result;
@@ -51,8 +62,9 @@ int sdp_wall_send_reject(struct sdp_opt 
 	return 0;
 } /* sdp_wall_send_reject */
 
-/* ========================================================================= */
-/*..sdp_wall_send_close -- callback to accept an active close */
+/*
+ * sdp_wall_send_close - callback to accept an active close
+ */
 int sdp_wall_send_close(struct sdp_opt *conn)
 {
 	struct sdpc_buff *buff;
@@ -134,8 +146,9 @@ error:
 	return result;
 } /* sdp_wall_send_close */
 
-/* ========================================================================= */
-/*..sdp_wall_send_closing -- callback to confirm a passive close */
+/*
+ * sdp_wall_send_closing - callback to confirm a passive close
+ */
 int sdp_wall_send_closing(struct sdp_opt *conn)
 {
 	int result;
@@ -191,8 +204,9 @@ error:
 	return result;
 } /* sdp_wall_send_closing */
 
-/* ========================================================================= */
-/*..sdp_wall_send_abort -- callback to accept an active abort */
+/*
+ * sdp_wall_send_abort - callback to accept an active abort
+ */
 int sdp_wall_send_abort(struct sdp_opt *conn)
 {
 	int result = 0;
@@ -305,14 +319,13 @@ error:
 	return 0;
 } /* sdp_wall_send_abort */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* SDP INET (public functions)                                           */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * SDP INET (public functions)
+ */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_reject -- callback to reject an active open */
+/*
+ * sdp_wall_recv_reject - callback to reject an active open
+ */
 int sdp_wall_recv_reject(struct sdp_opt *conn, int error)
 {
 	int result;
@@ -345,8 +358,9 @@ int sdp_wall_recv_reject(struct sdp_opt 
 	return 0;
 } /* sdp_wall_recv_reject */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_confirm -- callback to confirm accepeted passive open */
+/*
+ * sdp_wall_recv_confirm - callback to confirm accepeted passive open
+ */
 int sdp_wall_recv_confirm(struct sdp_opt *conn)
 {
         struct ib_qp_attr *qp_attr;
@@ -417,8 +431,9 @@ error:
 	return result;
 } /* sdp_wall_recv_confirm */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_failed - callback to notify accepted open of a failure */
+/*
+ * sdp_wall_recv_failed - callback to notify accepted open of a failure
+ */
 int sdp_wall_recv_failed(struct sdp_opt *conn, int error)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
@@ -453,8 +468,9 @@ int sdp_wall_recv_failed(struct sdp_opt 
 	return 0;
 } /* sdp_wall_recv_failed */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_close -- callback to accept an active close */
+/*
+ * sdp_wall_recv_close - callback to accept an active close
+ */
 int sdp_wall_recv_close(struct sdp_opt *conn)
 {
 	int result;
@@ -504,8 +520,9 @@ int sdp_wall_recv_close(struct sdp_opt *
 	return 0;
 } /* sdp_wall_recv_close */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_closing -- callback for a close confirmation */
+/*
+ * sdp_wall_recv_closing - callback for a close confirmation
+ */
 int sdp_wall_recv_closing(struct sdp_opt *conn)
 {
 	int result;
@@ -543,8 +560,9 @@ int sdp_wall_recv_closing(struct sdp_opt
 	return 0;
 } /* sdp_wall_recv_closing */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_abort -- abortive close notification */
+/*
+ * sdp_wall_recv_abort - abortive close notification
+ */
 int sdp_wall_recv_abort(struct sdp_opt *conn)
 {
 	int result;
@@ -591,8 +609,9 @@ int sdp_wall_recv_abort(struct sdp_opt *
 	return 0;
 } /* sdp_wall_recv_abort */
 
-/* ========================================================================= */
-/*..sdp_wall_recv_drop -- drop SDP protocol reference to socket */
+/*
+ * sdp_wall_recv_drop - drop SDP protocol reference to socket
+ */
 int sdp_wall_recv_drop(struct sdp_opt *conn)
 {
 	int result;
@@ -661,14 +680,13 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 	return 0;
 } /* sdp_wall_recv_drop */
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* SDP common (public functions)                                         */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
+/*
+ * SDP common (public functions)
+ */
 
-/* ========================================================================= */
-/*..sdp_wall_abort -- intiate socket dropping. */
+/*
+ * sdp_wall_abort - intiate socket dropping
+ */
 int sdp_wall_abort(struct sdp_opt *conn)
 {
 	int result;



From Tom.Duffy at Sun.COM  Thu Feb 17 14:14:08 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Thu, 17 Feb 2005 14:14:08 -0800
Subject: [openib-general] [PATCH][SDP][33/33] Fix comment style in
	sdp_write.c
In-Reply-To: <1108678438289@sun.com>
References: <11086781103458@sun.com> <11086781203512@sun.com>
	<1108678130413@sun.com> <1108678140425@sun.com>
	<11086781512781@sun.com>
	<11086781612958@sun.com> <11086781711886@sun.com>
	<11086781814000@sun.com>
	<11086781921405@sun.com> <11086782023076@sun.com>
	<11086782122447@sun.com>
	<11086782221006@sun.com> <1108678233852@sun.com>
	<11086782432384@sun.com>
	<11086782531540@sun.com> <11086782631290@sun.com>
	<11086782741398@sun.com>
	<1108678284314@sun.com> <11086782942946@sun.com>
	<1108678304705@sun.com>
	<11086783152506@sun.com> <11086783252833@sun.com>
	<1108678335304@sun.com>
	<11086783452340@sun.com> <1108678356217@sun.com>
	<11086783663245@sun.com>
	<11086783763927@sun.com> <11086783872748@sun.com>
	<11086783972515@sun.com>
	<11086784072107@sun.com> <11086784171812@sun.com>
	<11086784271466@sun.com> <1108678438289@sun.com>
Message-ID: <11086784483388@sun.com>


Signed-off-by: Tom Duffy <tduffy at sun.com>

--- drivers/infiniband/ulp/sdp-orig/sdp_write.c	2005-02-15 16:33:46.194005000 -0800
+++ drivers/infiniband/ulp/sdp/sdp_write.c	2005-02-17 11:57:50.886008000 -0800
@@ -1,35 +1,46 @@
 /*
-  This software is available to you under a choice of one of two
-  licenses.  You may choose to be licensed under the terms of the GNU
-  General Public License (GPL) Version 2, available at
-  <http://www.fsf.org/copyleft/gpl.html>, or the OpenIB.org BSD
-  license, available in the LICENSE.TXT file accompanying this
-  software.  These details are also available at
-  <http://openib.org/license.html>.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
-  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
-  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-  SOFTWARE.
-
-  Copyright (c) 2005 Topspin Communications.  All rights reserved.
-
-  $Id: sdp_write.c 619 2004-08-10 03:23:02Z roland $
-*/
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
 
 #include "sdp_main.h"
 
-/* --------------------------------------------------------------------- */
-/*                                                                       */
-/* RDMA read QP Event Handler                                            */
-/*                                                                       */
-/* --------------------------------------------------------------------- */
-/* ========================================================================= */
-/*..sdp_event_write -- RDMA write event handler. */
+/*
+ * RDMA read QP Event Handler
+ */
+
+/*
+ * sdp_event_write - RDMA write event handler
+ */
 int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp)
 {
 	struct sdpc_iocb *iocb;



From libor at topspin.com  Thu Feb 17 14:46:15 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 17 Feb 2005 14:46:15 -0800
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217221330.GA21318@mellanox.co.il>;
	from mst@mellanox.co.il on Fri, Feb 18, 2005 at 12:13:30AM +0200
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
	<20050217221330.GA21318@mellanox.co.il>
Message-ID: <20050217144615.C7121@topspin.com>

On Fri, Feb 18, 2005 at 12:13:30AM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > On Thu, Feb 17, 2005 at 11:29:37PM +0200, Michael S. Tsirkin wrote:
> > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > > > > Hi, Libor!
> > > > > Could you please explain what are the SDP_CONN_LOCK
> > > > > and friends doing in SDP?
> > > > > 
> > > > > It seems they just implement exclusive access to socket,
> > > > > but if so, why is a simple semaphore or mutex not used?
> > > > 
> > > >   They do implement exclusive access to the socket, but they implement
> > > > exclusive access from both process and irq context, which is why a
> > > > semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> > > > to lock the connection, look in sdp_cq_event_handler() for it's use,
> > > > and in process context SDP_CONN_LOCK is used.
> > > 
> > > I dont really understand how it works.
> > > When an interrupt arrives while users != 0, it seems you are
> > > calling scheduler().
> > > What is sdp_conn_internal_lock doing? I understand it is to be called
> > > from interrupt context, but how can it call scheduler() then?
> > 
> >   SDP_CONN_LOCK and SDP_CONN_UNLOCK are never called from interrupt
> > context, only from process context. The spinlock conn->lock.slock
> > is called in interrupt context and process context before the 'users'
> > variable is changed or read. If users != 0, then the connection is in
> > use by process context. If the connection is in use then another lock
> > request in process context will sleep until users == 0, if the connection
> > is in use then a lock request in interrupt context will mark the
> > connection as having had an event and return. When a process context
> > lock holder releases the lock, it checks if there has been an interrupt
> > event, which it then processes by draining and arming the CQ, and it
> > wakes any process context waiting for the lock.
> 
> Cant all this be done by a semaphore in process context, and 
> down_trylock in interrupt context?
> The "connection has an event" bit could be protected by a spinlock.

  The CQ draining happens inside of the spinlock. If the CQ drain
happened in process context inside of a semaphore, then there is
a race condition for the interrupt between checking the semaphore
and setting the spinlock protected "connection event bit". The
CQ drain and the "event bit" have to happen under the same lock
to avoid the race. 

  If you move the CQ drain and the "event bit" into the same spinlock
where do you put the semaphore? In process context the spin would
need to be inside the semaphore lock, but in interrupt context the
semaphore down_trylock and up are inside the spinlock. Now there would
be a race between spin_unlock() followed by up() in process context
and spin_lock() followed by down_trylock() in interrupt context.

   spin_unlock()
                  spin_lock()
                  down_trylock()
   up()


-Libor




From mst at mellanox.co.il  Thu Feb 17 15:25:39 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 18 Feb 2005 01:25:39 +0200
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217144615.C7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
	<20050217221330.GA21318@mellanox.co.il>
	<20050217144615.C7121@topspin.com>
Message-ID: <20050217232539.GB21318@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [openib-general] SDP_CONN_LOCK
> 
> On Fri, Feb 18, 2005 at 12:13:30AM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > On Thu, Feb 17, 2005 at 11:29:37PM +0200, Michael S. Tsirkin wrote:
> > > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > > On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > > > > > Hi, Libor!
> > > > > > Could you please explain what are the SDP_CONN_LOCK
> > > > > > and friends doing in SDP?
> > > > > > 
> > > > > > It seems they just implement exclusive access to socket,
> > > > > > but if so, why is a simple semaphore or mutex not used?
> > > > > 
> > > > >   They do implement exclusive access to the socket, but they implement
> > > > > exclusive access from both process and irq context, which is why a
> > > > > semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> > > > > to lock the connection, look in sdp_cq_event_handler() for it's use,
> > > > > and in process context SDP_CONN_LOCK is used.
> > > > 
> > > > I dont really understand how it works.
> > > > When an interrupt arrives while users != 0, it seems you are
> > > > calling scheduler().
> > > > What is sdp_conn_internal_lock doing? I understand it is to be called
> > > > from interrupt context, but how can it call scheduler() then?
> > > 
> > >   SDP_CONN_LOCK and SDP_CONN_UNLOCK are never called from interrupt
> > > context, only from process context. The spinlock conn->lock.slock
> > > is called in interrupt context and process context before the 'users'
> > > variable is changed or read. If users != 0, then the connection is in
> > > use by process context. If the connection is in use then another lock
> > > request in process context will sleep until users == 0, if the connection
> > > is in use then a lock request in interrupt context will mark the
> > > connection as having had an event and return. When a process context
> > > lock holder releases the lock, it checks if there has been an interrupt
> > > event, which it then processes by draining and arming the CQ, and it
> > > wakes any process context waiting for the lock.
> > 
> > Cant all this be done by a semaphore in process context, and 
> > down_trylock in interrupt context?
> > The "connection has an event" bit could be protected by a spinlock.
> 
>   The CQ draining happens inside of the spinlock. If the CQ drain
> happened in process context inside of a semaphore, then there is
> a race condition for the interrupt between checking the semaphore
> and setting the spinlock protected "connection event bit". The
> CQ drain and the "event bit" have to happen under the same lock
> to avoid the race. 
> 
>   If you move the CQ drain and the "event bit" into the same spinlock
> where do you put the semaphore? In process context the spin would
> need to be inside the semaphore lock, but in interrupt context the
> semaphore down_trylock and up are inside the spinlock. Now there would
> be a race between spin_unlock() followed by up() in process context
> and spin_lock() followed by down_trylock() in interrupt context.
> 
>    spin_unlock()
>                   spin_lock()
>                   down_trylock()
>    up()
> 
> 
> -Libor
> 

Since CQ drain is protected by spinlock - move it outside the semaphore?

process:

down
stuff
up

lock
check bit and arm
drain
unlock

interrupt:

lock
down_trylock
set bit
up
unlock

Anyway, this flow is just an optimisation - so I wander how much does it help,
actually. My understanding this is a backport from gen1, but what was
a useful optimisation for gen1 may not be for gen2.
gen2 event flow is much faster than gen1 - it is possible
all the extra locking and complications are not worth it with mthca.
I know this is what my attempts to switch to polling in ip over ib show.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb 17 15:28:00 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 18 Feb 2005 01:28:00 +0200
Subject: [openib-general] Re: SDP_CONN_LOCK
In-Reply-To: <52sm3uvo5c.fsf@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
Message-ID: <20050217232800.GC21318@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: SDP_CONN_LOCK
> 
> BTW, since mthca currently calls completion handlers directly from
> interrupt context (rather than BH/tasklet context), it might be worth
> renaming all the SDP locking macros so they're not confusingly named
> with _BH suffixes.
> 
>  - R.

I think it would be much nicer to reduce the number of macros used.

-- 
MST - Michael S. Tsirkin


From mshefty at ichips.intel.com  Thu Feb 17 15:49:31 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 17 Feb 2005 15:49:31 -0800
Subject: [openib-general] Re: SDP_CONN_LOCK
In-Reply-To: <20050217232800.GC21318@mellanox.co.il>
References: <20050217153517.GV28446@mellanox.co.il>	<20050217110256.A7121@topspin.com>	<20050217212937.GA21072@mellanox.co.il>	<20050217140015.B7121@topspin.com>
	<52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
Message-ID: <42152D8B.2040608@ichips.intel.com>

Michael S. Tsirkin wrote:

> Quoting r. Roland Dreier <roland at topspin.com>:
> 
>>Subject: Re: SDP_CONN_LOCK
>>
>>BTW, since mthca currently calls completion handlers directly from
>>interrupt context (rather than BH/tasklet context), it might be worth
>>renaming all the SDP locking macros so they're not confusingly named
>>with _BH suffixes.
>>
>> - R.
> 
> 
> I think it would be much nicer to reduce the number of macros used.

I'd have to agree with this.  The SDP locking macros are fairly complex 
and hide a lot of functionality.  E.g. SDP_CONN_RELOCK results in 
polling/rearming the CQ, same with SDP_CONN_UNLOCK.  Maybe that's just 
a naming issue though.

I think these would probably be better off as just function calls, 
rather than macros.  SDP_CONN_LOCK calls sdp_conn_internal_lock(), and 
that appears to be the only place that the function is called. 
Similarly, SDP_CONN_UNLOCK calls sdp_conn_internal_unlock().  It seems 
that you could just merge the macros into the function calls.

Unfortunately, I don't really have the time at the moment to help do 
any of this cleanup.

- Sean


From libor at topspin.com  Thu Feb 17 16:18:39 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 17 Feb 2005 16:18:39 -0800
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217232539.GB21318@mellanox.co.il>;
	from mst@mellanox.co.il on Fri, Feb 18, 2005 at 01:25:39AM +0200
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
	<20050217221330.GA21318@mellanox.co.il>
	<20050217144615.C7121@topspin.com>
	<20050217232539.GB21318@mellanox.co.il>
Message-ID: <20050217161839.D7121@topspin.com>

On Fri, Feb 18, 2005 at 01:25:39AM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > On Fri, Feb 18, 2005 at 12:13:30AM +0200, Michael S. Tsirkin wrote:
> > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > On Thu, Feb 17, 2005 at 11:29:37PM +0200, Michael S. Tsirkin wrote:
> > > > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > > > On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > > > > > > Hi, Libor!
> > > > > > > Could you please explain what are the SDP_CONN_LOCK
> > > > > > > and friends doing in SDP?
> > > > > > > 
> > > > > > > It seems they just implement exclusive access to socket,
> > > > > > > but if so, why is a simple semaphore or mutex not used?
> > > > > > 
> > > > > >   They do implement exclusive access to the socket, but they implement
> > > > > > exclusive access from both process and irq context, which is why a
> > > > > > semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> > > > > > to lock the connection, look in sdp_cq_event_handler() for it's use,
> > > > > > and in process context SDP_CONN_LOCK is used.
> > > > > 
> > > > > I dont really understand how it works.
> > > > > When an interrupt arrives while users != 0, it seems you are
> > > > > calling scheduler().
> > > > > What is sdp_conn_internal_lock doing? I understand it is to be called
> > > > > from interrupt context, but how can it call scheduler() then?
> > > > 
> > > >   SDP_CONN_LOCK and SDP_CONN_UNLOCK are never called from interrupt
> > > > context, only from process context. The spinlock conn->lock.slock
> > > > is called in interrupt context and process context before the 'users'
> > > > variable is changed or read. If users != 0, then the connection is in
> > > > use by process context. If the connection is in use then another lock
> > > > request in process context will sleep until users == 0, if the connection
> > > > is in use then a lock request in interrupt context will mark the
> > > > connection as having had an event and return. When a process context
> > > > lock holder releases the lock, it checks if there has been an interrupt
> > > > event, which it then processes by draining and arming the CQ, and it
> > > > wakes any process context waiting for the lock.
> > > 
> > > Cant all this be done by a semaphore in process context, and 
> > > down_trylock in interrupt context?
> > > The "connection has an event" bit could be protected by a spinlock.
> > 
> >   The CQ draining happens inside of the spinlock. If the CQ drain
> > happened in process context inside of a semaphore, then there is
> > a race condition for the interrupt between checking the semaphore
> > and setting the spinlock protected "connection event bit". The
> > CQ drain and the "event bit" have to happen under the same lock
> > to avoid the race. 
> > 
> >   If you move the CQ drain and the "event bit" into the same spinlock
> > where do you put the semaphore? In process context the spin would
> > need to be inside the semaphore lock, but in interrupt context the
> > semaphore down_trylock and up are inside the spinlock. Now there would
> > be a race between spin_unlock() followed by up() in process context
> > and spin_lock() followed by down_trylock() in interrupt context.
> > 
> >    spin_unlock()
> >                   spin_lock()
> >                   down_trylock()
> >    up()
> 
> Since CQ drain is protected by spinlock - move it outside the semaphore?
> 
> process:
> 
> down
> stuff
> up
> 
> lock
> check bit and arm
> drain
> unlock

  The problem with this ordering is that 'drain' in the spinlock is actually
doing the same things as 'stuff' inside the semaphore, both are modifying
connection structures, like queues and counters, and so if two processes
are contending for the locks, then they're going to be stepping on the
connection:

   process 1:             process two

   down
   stuff
   up
                          down
   lock
   check bit and arm
   drain
     stuff                stuff
   unlock                 up

> Anyway, this flow is just an optimisation - so I wander how much does 
> it help, actually.

  No, it's not just an optimization. The drain in process context is a
optimization, but having a lock protecting in both contexts is necessary.

  Without the drain in process context you would need to handle the actual
CQ event, but if the connection is locked in process context, what do
you do with the event? You either wait for the lock to be released, or
you process it to a point, and then queue it for further handling once
the lock is released. Waiting for a lock release is not practical, and 
processing to a point is what we are currently doing, just we are
processing it the bare minimum, and leaving the rest for process
context.

  Setting the event bit and processing of the CQ in process context was
actually a big simplification which not surprisingly performed much
better as well.

> My understanding this is a backport from gen1, but what was a useful
> optimisation for gen1 may not be for gen2. gen2 event flow is much 
> faster than gen1 - it is possible all the extra locking and
> complications are not worth it with mthca.

  I agree with the general sentiment. However, with mthca faster then the
gen1 code, this code should help even more, since gen2 would be able to
generate more interrupts a second, if we let it. At the time this has
added, it resulted in a 2/3 reduction in interrupts per second. 

  Next, in this portion of the code I was actually going to try using a
single CQ instead of two to see if we get an additional improvement as a
result of fewer interrupts and simpler code.


-Libor


  


From libor at topspin.com  Thu Feb 17 16:24:13 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 17 Feb 2005 16:24:13 -0800
Subject: [openib-general] Re: SDP_CONN_LOCK
In-Reply-To: <42152D8B.2040608@ichips.intel.com>;
	from mshefty@ichips.intel.com on Thu, Feb 17, 2005 at 03:49:31PM
	-0800
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
Message-ID: <20050217162413.E7121@topspin.com>

On Thu, Feb 17, 2005 at 03:49:31PM -0800, Sean Hefty wrote:
> Michael S. Tsirkin wrote:
> > Quoting r. Roland Dreier <roland at topspin.com>:
> >>
> >>BTW, since mthca currently calls completion handlers directly from
> >>interrupt context (rather than BH/tasklet context), it might be worth
> >>renaming all the SDP locking macros so they're not confusingly named
> >>with _BH suffixes.
> > 
> > I think it would be much nicer to reduce the number of macros used.
> 
> I'd have to agree with this.  The SDP locking macros are fairly complex 
> and hide a lot of functionality.  E.g. SDP_CONN_RELOCK results in 
> polling/rearming the CQ, same with SDP_CONN_UNLOCK.  Maybe that's just 
> a naming issue though.
> 
> I think these would probably be better off as just function calls, 
> rather than macros.  SDP_CONN_LOCK calls sdp_conn_internal_lock(), and 
> that appears to be the only place that the function is called. 
> Similarly, SDP_CONN_UNLOCK calls sdp_conn_internal_unlock().  It seems 
> that you could just merge the macros into the function calls.

  OK. I had used macros since in the fast path the functions wouldn't
get called, but since it sounds like it makes things less readable I
should be able to quickly try out replacing the macros with functions,
and see what if any effect it has on performance.

  And here I would have thought SDP_CONN_HOLD as a macro would have
bugged people more. :)

-Libor


From xma at us.ibm.com  Thu Feb 17 16:28:47 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 17 Feb 2005 17:28:47 -0700
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <OFD9622615.D707806D-ON87256FAB.0065BBEC-00256FAB.0066B58D@us.ibm.com>
Message-ID: <OF93427C27.4C02A68C-ON87256FAC.0002199E-00256FAC.0002EAC2@us.ibm.com>

Sorry, made a mistake in the previous patch. Here is the right one.

diff -urN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-lock/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-17 17:24:56.000000000 
+0000
+++ infiniband-lock/ulp/ipoib/ipoib_main.c      2005-02-18 
00:20:10.000000000 +0000
@@ -220,18 +220,23 @@
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tn;
        struct sk_buff *skb;
+       LIST_HEAD(ah_list);
+       struct ipoib_ah *ah, *tah;
 
        while ((skb = __skb_dequeue(&path->queue)))
                dev_kfree_skb_irq(skb);
 
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
                if (neigh->ah)
-                       ipoib_put_ah(neigh->ah);
+                       list_add_tail(&neigh->ah->list, &ah_list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
 
+       list_for_each_entry_safe(ah, tah, &ah_list, list)
+               ipoib_put_ah(ah);
+
        if (path->ah)
                ipoib_put_ah(path->ah);
 
@@ -660,6 +665,7 @@
        struct ipoib_neigh *neigh = *to_ipoib_neigh(n);
        struct ipoib_dev_priv *priv = netdev_priv(n->dev);
        unsigned long flags;
+       struct ipoib_ah *ah = NULL;
 
        ipoib_dbg(priv,
                  "neigh_destructor for %06x " IPOIB_GID_FMT "\n",
@@ -670,13 +676,16 @@
 
        if (neigh) {
                if (neigh->ah)
-                       ipoib_put_ah(neigh->ah);
+                       ah = neigh->ah;
                list_del(&neigh->list);
                *to_ipoib_neigh(n) = NULL;
                kfree(neigh);
        }
 
        spin_unlock_irqrestore(&priv->lock, flags);
+ 
+       if (ah)
+               ipoib_put_ah(ah);
 }
 
 static int ipoib_neigh_setup(struct neighbour *neigh)
diff -urN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-lock/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-17 
17:24:56.000000000 +0000
+++ infiniband-lock/ulp/ipoib/ipoib_multicast.c 2005-02-18 
00:20:34.000000000 +0000
@@ -93,6 +93,8 @@
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tmp;
        unsigned long flags;
+       LIST_HEAD(ah_list);
+       struct ipoib_ah *ah, *tah;
 
        ipoib_dbg_mcast(netdev_priv(dev),
                        "deleting multicast group " IPOIB_GID_FMT "\n",
@@ -101,7 +103,8 @@
        spin_lock_irqsave(&priv->lock, flags);
 
        list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) {
-               ipoib_put_ah(neigh->ah);
+               if (neigh->ah)
+                       list_add_tail(&neigh->ah->list, &ah_list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
@@ -109,6 +112,9 @@
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
+       list_for_each_entry_safe(ah, tah, &ah_list, list)
+               ipoib_put_ah(ah);
+
        if (mcast->ah)
                ipoib_put_ah(mcast->ah);
 





Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/26bbff88/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-lock.patch
Type: application/octet-stream
Size: 2506 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/26bbff88/attachment.obj>

From iod00d at hp.com  Thu Feb 17 16:34:51 2005
From: iod00d at hp.com (Grant Grundler)
Date: Thu, 17 Feb 2005 16:34:51 -0800
Subject: [openib-general] Re: SDP_CONN_LOCK
In-Reply-To: <20050217162413.E7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
Message-ID: <20050218003451.GA23966@esmail.cup.hp.com>

On Thu, Feb 17, 2005 at 04:24:13PM -0800, Libor Michalek wrote:
>   OK. I had used macros since in the fast path the functions wouldn't
> get called, but since it sounds like it makes things less readable I
> should be able to quickly try out replacing the macros with functions,
> and see what if any effect it has on performance.

Well, see how big the function is and how often it gets called.
One can define it as a static inline function if it doesn't
bloat the binary too much. Calls to "leaf" functions can
be pretty light weight.

FWIW, I think of static inline functions as "macros with typechecking".
Ie they can be defined in .h files instead of .c files.

grant


From libor at topspin.com  Thu Feb 17 16:41:50 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 17 Feb 2005 16:41:50 -0800
Subject: [openib-general] Re: RFC on SDP checkin
In-Reply-To: <20050217140359.GT28446@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 17, 2005 at 04:03:59PM +0200
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>
	<20050217140359.GT28446@mellanox.co.il>
Message-ID: <20050217164150.F7121@topspin.com>

On Thu, Feb 17, 2005 at 04:03:59PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Possible Issues
> >   
> > [...] 
> > 
> >   - Memory locking for AIO requires a call to do_mlock() which is not a
> >     kernel exported function, the method for calling the function is not
> >     standard.
> 
> Libor, in my eyes this is the biggest issue with this gen2 sdp code.
> In short, I dont think using do_mlock is a good idea.

  I do agree that this is one of the more, if not most, problematic area
of the code, and I like your alternative suggestion below.

> Even if do_mlock *would* be exported, using this call to
> do user page locking has other limitations:
> 
> - number of locked pages is very restricted, if its not,
>   user can easily DOS the box

  I don't think the allowed locked number of pages is an issue, it is
controlled by the user and administrator, which prevents a DOS.

> - fork() is broken in many ways

  Agreed.

> - user is forbidden from doing mlock/munlock

  If either SDP or the userspace code needed to use mlock/munlock we
would really need to add reference counting to the mlocking of VMAs,
this is something that would need ot be pushed upstream. It is a
problem that the user could step on the locking that's going on
behind the scenes.

> I would like to suggest, once again, that the standard get_user_pages
> call is used, instead. It is true that virtual mappings may change,
> but it is easy to check that once the transaction is complete,
> and perform copy if this happends.
> 
> Mellanox implemented and benchmarked this on Gen1 with good results.
> 
> If you are interested, I may post the design document for how this
> works for us, or I can try and find some time to build a patch.
> However, the second option wont happend till the end of
> the month, and the patch is likely to be quite big.

  Yes, this approach was mentioned to me by both Dror and Tziporet,
at the developers workshop, and it does make sense. It does seem
that this would be a better approach then using do_mlock. I would
think that the patch would be isolated entirely to sdp_iocb.[ch]
aside from changing some function names. First we need to get FMR
support in mthca so we can get a baseline with existing code and
something to test against.


-Libor





From mshefty at ichips.intel.com  Thu Feb 17 17:44:42 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 17 Feb 2005 17:44:42 -0800
Subject: [openib-general] [PATCH] [CM] add stale connection test to active
	side
Message-ID: <20050217174442.62c5b404.mshefty@ichips.intel.com>

This patch adds testing for a stale connection on the active side of
the connection protocol.  It also fixes a bug where a duplicate REP
would have been dropped, rather than forcing a resend of the RTU.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>


Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1818)
+++ infiniband/core/cm.c	(working copy)
@@ -1302,6 +1302,27 @@ static void cm_format_rep_event(struct c
 	work->cm_event.private_data = &rep_msg->private_data;
 }
 
+static void cm_dup_rep_handler(struct cm_rep_msg *rep_msg)
+{
+	struct cm_id_private *cm_id_priv;
+	enum ib_cm_state state;
+	unsigned long flags;
+
+	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id,
+				   rep_msg->local_comm_id);
+	if (!cm_id_priv)
+		return;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	state = cm_id_priv->id.state;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	
+	/* RTU's received in an invalid state will be dropped. */
+	if (state == IB_CM_ESTABLISHED)
+		cm_resend_rtu(cm_id_priv);
+	cm_deref_id(cm_id_priv);
+}
+
 static int cm_rep_handler(struct cm_work *work)
 {
 	struct cm_id_private *cm_id_priv;
@@ -1313,8 +1334,10 @@ static int cm_rep_handler(struct cm_work
 
 	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
 	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id, 0);
-	if (!cm_id_priv)
+	if (!cm_id_priv) {
+		cm_dup_rep_handler(rep_msg);
 		return -EINVAL;
+	}
 
 	timewait_info = cm_create_timewait_info(cm_id_priv->id.local_id,
 						rep_msg->local_comm_id,
@@ -1324,6 +1347,22 @@ static int cm_rep_handler(struct cm_work
 		ret = PTR_ERR(timewait_info);
 		goto error1;
 	}
+	spin_lock_irqsave(&cm.lock, flags);
+	/* Check for duplicate REP. */
+	if (cm_insert_remote_id(timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		ret = -EINVAL;
+		goto error2;
+	}
+	/* Check for a stale connection. */
+	if (cm_insert_remote_qpn(timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		/* todo: reject as stale */
+		ret = -EINVAL;
+		goto error2;
+	}
+	spin_unlock_irqrestore(&cm.lock, flags);
+
 	cm_format_rep_event(work);
 
 	spin_lock_irqsave(&cm_id_priv->lock, flags);
@@ -1331,11 +1370,6 @@ static int cm_rep_handler(struct cm_work
 	case IB_CM_REQ_SENT:
 	case IB_CM_MRA_REQ_RCVD:
 		break;
-	case IB_CM_ESTABLISHED:
-		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-		cm_resend_rtu(cm_id_priv);
-		ret = -EINVAL;
-		goto error2;
 	default:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 		ret = -EINVAL;
@@ -1365,6 +1399,7 @@ static int cm_rep_handler(struct cm_work
 		cm_deref_id(cm_id_priv);
 	return 0;
 error2:
+	cm_cleanup_timewait(timewait_info);
 	kfree(timewait_info);
 error1:
 	cm_deref_id(cm_id_priv);


From xma at us.ibm.com  Thu Feb 17 16:57:07 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 17 Feb 2005 17:57:07 -0700
Subject: [openib-general] [PATCH] IPOIB: initialization patch in ipoib
Message-ID: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>

Here is the patch for :

1. initialize path->ah = NULL
2. move neigh->neighbour->ops before kfree(neigh)
3. set neighbour->ops->destructor to NULL while kfree(neigh)

diff -urN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-patch/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-17 17:24:56.000000000 
+0000
+++ infiniband-patch/ulp/ipoib/ipoib_main.c     2005-02-18 
00:52:22.000000000 +0000
@@ -348,6 +348,7 @@
 
        path->dev = dev;
        path->pathrec.dlid = 0;
+       path->ah = NULL;
 
        skb_queue_head_init(&path->queue);
 
@@ -450,8 +451,8 @@
 err:
        *to_ipoib_neigh(skb->dst->neighbour) = NULL;
        list_del(&neigh->list);
-       kfree(neigh);
        neigh->neighbour->ops->destructor = NULL;
+       kfree(neigh);
 
        ++priv->stats.tx_dropped;
        dev_kfree_skb_any(skb);
@@ -673,6 +674,7 @@
                        ipoib_put_ah(neigh->ah);
                list_del(&neigh->list);
                *to_ipoib_neigh(n) = NULL;
+               n->ops->destructor = NULL;
                kfree(neigh);
        }
 



Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/a210810b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-patch.1
Type: application/octet-stream
Size: 755 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/a210810b/attachment.obj>

From xma at us.ibm.com  Thu Feb 17 17:20:51 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 17 Feb 2005 18:20:51 -0700
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list before
	kfree(neigh)
Message-ID: <OFDA9654E4.F11208C0-ON87256FAC.0006C21B-00256FAC.0007AED2@us.ibm.com>

This patch sets caller's ah to NULL when calling ipoib_put_ah(), and 
deletes neigh from the list before kfree(neigh).

Please review this patch.

diff -urN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-ah/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-17 17:24:56.000000000 
+0000
+++ infiniband-ah/ulp/ipoib/ipoib_main.c        2005-02-18 
01:15:44.000000000 +0000
@@ -225,16 +225,20 @@
                dev_kfree_skb_irq(skb);
 
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
-               if (neigh->ah)
+               if (neigh->ah) {
                        ipoib_put_ah(neigh->ah);
+                       neigh->bh = NULL;
+               }
+               list_del(&neigh->list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
 
-       if (path->ah)
+       if (path->ah) {
                ipoib_put_ah(path->ah);
-
+               path->ah = NULL;
+       }
        rb_erase(&path->rb_node, &priv->path_tree);
        list_del(&path->list);
        kfree(path);
diff -urN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-ah/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-17 
17:24:56.000000000 +0000
+++ infiniband-ah/ulp/ipoib/ipoib_multicast.c   2005-02-18 
00:36:56.000000000 +0000
@@ -101,7 +101,11 @@
        spin_lock_irqsave(&priv->lock, flags);
 
        list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) {
-               ipoib_put_ah(neigh->ah);
+               if (neigh->ah) {
+                       ipoib_put_ah(neigh->ah);
+                       neigh->bh = NULL;
+               }
+               list_del(&neigh->list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
@@ -109,8 +113,10 @@
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
-       if (mcast->ah)
+       if (mcast->ah) {
                ipoib_put_ah(mcast->ah);
+               mcast->ah = NULL;
+       }
 
        while (!skb_queue_empty(&mcast->pkt_queue)) {
                struct sk_buff *skb = skb_dequeue(&mcast->pkt_queue);





Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/b5f6fc9b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-ah.patch
Type: application/octet-stream
Size: 1643 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/b5f6fc9b/attachment.obj>

From xma at us.ibm.com  Thu Feb 17 17:40:34 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 17 Feb 2005 18:40:34 -0700
Subject: [openib-general] [PATCH]IPOIB: adding spin_lock in path_free()
Message-ID: <OFD6356A4D.92D673E6-ON87256FAC.00089C5F-00256FAC.00097D08@us.ibm.com>

Please review this patch.

diff -urN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-path/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-17 17:24:56.000000000 
+0000
+++ infiniband-path/ulp/ipoib/ipoib_main.c      2005-02-18 
01:29:52.000000000 +0000
@@ -215,15 +215,17 @@
        return 0;
 }
 
-static void __path_free(struct net_device *dev, struct ipoib_path *path)
+static void path_free(struct net_device *dev, struct ipoib_path *path)
 {
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tn;
        struct sk_buff *skb;
+       unsigned long flags;
 
        while ((skb = __skb_dequeue(&path->queue)))
                dev_kfree_skb_irq(skb);
 
+       spin_lock_irqsave(&priv->lock, flags);
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
                if (neigh->ah)
                        ipoib_put_ah(neigh->ah);
@@ -231,13 +233,16 @@
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
+       spin_unlock_irqrestore(&priv->lock, flags);
 
        if (path->ah)
                ipoib_put_ah(path->ah);
 
+       spin_lock_irqsave(&priv->lock, flags);
        rb_erase(&path->rb_node, &priv->path_tree);
        list_del(&path->list);
        kfree(path);
+       spin_unlock_irqrestore(&priv->lock, flags);
 }
 
 void ipoib_flush_paths(struct net_device *dev)
@@ -256,7 +261,7 @@
                if (path->query)
                        ib_sa_cancel_query(path->query_id, path->query);
                wait_for_completion(&path->done);
-               __path_free(dev, path);
+               path_free(dev, path);
        }
 }
 
@@ -496,8 +501,11 @@
                        skb_push(skb, sizeof *phdr);
                        __skb_queue_tail(&path->queue, skb);
 
-                       if (path_rec_start(dev, path))
-                               __path_free(dev, path);
+                       if (path_rec_start(dev, path)) {
+                               spin_unlock(&priv->lock);
+                               path_free(dev, path);
+                               return;
+                       }
                } else {
                        ++priv->stats.tx_dropped;
                        dev_kfree_skb_any(skb);




Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/3e5825fd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-path-free.patch
Type: application/octet-stream
Size: 1727 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050217/3e5825fd/attachment.obj>

From itoumsn at nttdata.co.jp  Thu Feb 17 22:40:16 2005
From: itoumsn at nttdata.co.jp (Masanori ITOH)
Date: Fri, 18 Feb 2005 15:40:16 +0900 (JST)
Subject: [openib-general] [DAPL] missing files in dapl_gamma_3.0
Message-ID: <20050218.154016.96770694.itoumsn@nttdata.co.jp>


Hi James, Eitan,

I found 'DAPL Gamma 3.0' at the DAPL sourceforge page.
After trying to build that with VERBS=openib_gen_one option, 
especially kDAPL portion, now I think that at lest two files 'dapl_srv.c'
and 'dapl_srv.h' are missing. I checked also IBGD-1.6.1 package but couldn't
find them.
Where can I get them?

Eitan,
I couldn't find kDAPL/gen1 stack among IBGD-1.6.1 which is now available
at http://www.mellanox.com.
Is there more updated IBGD than 1.6.1?

Thanks in advance,
Masanori


From mst at mellanox.co.il  Fri Feb 18 02:38:55 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 18 Feb 2005 12:38:55 +0200
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
References: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
Message-ID: <20050218103855.GB30863@mellanox.co.il>

Quoting r. Shirley Ma <xma at us.ibm.com>:
> Subject: [PATCH] IPOIB: initialization patch in ipoib
> 
> 
> Here is the patch for :
> 
> 1. initialize path->ah = NULL
> 2. move neigh->neighbour->ops before kfree(neigh)
> 3. set neighbour->ops->destructor to NULL while kfree(neigh)
> 
> diff -urN infiniband/ulp/ipoib/ipoib_main.c infiniband-patch/ulp/ipoib/
> ipoib_main.c

Please, generate patches with diff -p option.
Thanks,

-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Fri Feb 18 05:37:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 18 Feb 2005 08:37:53 -0500
Subject: [openib-general] CM Pathing Question
Message-ID: <1108733873.4645.69.camel@localhost.localdomain>

Hi,

OpenIB is in the midst of implementing the CM and the following question
about the CM has come up on openib-general mailing list and no one has
yet supplied an answer so I thought I'd go to the source. I am cross
posting although members of openib-general won't necessarily be able to
respond directly to any response.

It appears that there is the possibility of a CM path and one or more
data paths (primary/alternate) per connection. The CM path may or may
not be the same as the data path(s).

Assume that a connection with both a primary and secondary path is
setup, using the same primary path for CM communication. Some time later
the primary path fails, so APM causes the connection to migrate to 
the secondary path. A "path migrated" async event is issued in the end
node which results in the desire to set up a new secondary path so that
the connection remains resilent. Where does the LAP get sent ? 
Similarly for any other future CM messages (like DREQ, etc.). It seems
to need to get sent along the new path. 

How is the CM path expected to be maintained in the two cases (when it
is the same and when is different from a data path) in the presence of a
path failure ? Maybe those are the same case in terms of the
architecture.

Thanks for your help with this.

-- Hal



From jlentini at netapp.com  Fri Feb 18 05:51:03 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 18 Feb 2005 08:51:03 -0500 (EST)
Subject: [openib-general] [DAPL] missing files in dapl_gamma_3.0
In-Reply-To: <20050218.154016.96770694.itoumsn@nttdata.co.jp>
References: <20050218.154016.96770694.itoumsn@nttdata.co.jp>
Message-ID: <Pine.LNX.4.61.0502180845580.5704@jlentini-linux.nane.netapp.com>



Hi Masanori,

Lets move these questions to the SourceForge dapl-devel list, since 
they concern the SourceForge DAPL project.

-james

On Fri, 18 Feb 2005, Masanori ITOH wrote:

>
> Hi James, Eitan,
>
> I found 'DAPL Gamma 3.0' at the DAPL sourceforge page.
> After trying to build that with VERBS=openib_gen_one option,
> especially kDAPL portion, now I think that at lest two files 'dapl_srv.c'
> and 'dapl_srv.h' are missing. I checked also IBGD-1.6.1 package but couldn't
> find them.
> Where can I get them?
>
> Eitan,
> I couldn't find kDAPL/gen1 stack among IBGD-1.6.1 which is now available
> at http://www.mellanox.com.
> Is there more updated IBGD than 1.6.1?
>
> Thanks in advance,
> Masanori
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>


From mst at mellanox.co.il  Fri Feb 18 07:45:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 18 Feb 2005 17:45:08 +0200
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
References: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
Message-ID: <20050218154508.GB31228@mellanox.co.il>

Quoting r. Shirley Ma <xma at us.ibm.com>:
> Subject: [PATCH] IPOIB: initialization patch in ipoib
> 
> 
> Here is the patch for :
> 
> 1. initialize path->ah = NULL
> 2. move neigh->neighbour->ops before kfree(neigh)
> 3. set neighbour->ops->destructor to NULL while kfree(neigh)
> 
> diff -urN infiniband/ulp/ipoib/ipoib_main.c infiniband-patch/ulp/ipoib/
> ipoib_main.c
> --- infiniband/ulp/ipoib/ipoib_main.c        2005-02-17 17:24:56.000000000
> +0000
> +++ infiniband-patch/ulp/ipoib/ipoib_main.c        2005-02-18
> @@ -450,8 +451,8 @@
> err:
>         *to_ipoib_neigh(skb->dst->neighbour) = NULL;
>         list_del(&neigh->list);
> -        kfree(neigh);
>         neigh->neighbour->ops->destructor = NULL;
> +        kfree(neigh);
> 
>         ++priv->stats.tx_dropped;
>         dev_kfree_skb_any(skb);

If the order of these operations really matters, wont you need
some kind of memory barrier in between?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Fri Feb 18 07:46:24 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 18 Feb 2005 17:46:24 +0200
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
References: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
Message-ID: <20050218154624.GC31228@mellanox.co.il>

Quoting r. Shirley Ma <xma at us.ibm.com>:
> Subject: [PATCH] IPOIB: initialization patch in ipoib
> 
> 3. set neighbour->ops->destructor to NULL while kfree(neigh)



> diff -urN infiniband/ulp/ipoib/ipoib_main.c infiniband-patch/ulp/ipoib/
> ipoib_main.c
> --- infiniband/ulp/ipoib/ipoib_main.c        2005-02-17 17:24:56.000000000
> +0000
> +++ infiniband-patch/ulp/ipoib/ipoib_main.c        2005-02-18
> 00:52:22.000000000 +0000
> @@ -673,6 +674,7 @@
>                         ipoib_put_ah(neigh->ah);
>                 list_del(&neigh->list);
>                 *to_ipoib_neigh(n) = NULL;
> +                n->ops->destructor = NULL;
>                 kfree(neigh);
>         }
> 
> 
> 
> 
> Thanks
> Shirley Ma

Why is this necessary?

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Fri Feb 18 08:51:16 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 18 Feb 2005 08:51:16 -0800
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <20050218154508.GB31228@mellanox.co.il> (Michael S. Tsirkin's
	message of "Fri, 18 Feb 2005 17:45:08 +0200")
References: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
	<20050218154508.GB31228@mellanox.co.il>
Message-ID: <52brahvmkb.fsf@topspin.com>

    Michael> If the order of these operations really matters, wont you
    Michael> need some kind of memory barrier in between?

No, no CPU is so insane that you need a barrier between

	x->a->b = c;

and

	kfree(x);

And it is always wrong to do

	kfree(x);
	x->a->b = c;

 - Roland


From iod00d at hp.com  Fri Feb 18 08:57:05 2005
From: iod00d at hp.com (Grant Grundler)
Date: Fri, 18 Feb 2005 08:57:05 -0800
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <20050218154508.GB31228@mellanox.co.il>
References: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
	<20050218154508.GB31228@mellanox.co.il>
Message-ID: <20050218165705.GA27063@esmail.cup.hp.com>

On Fri, Feb 18, 2005 at 05:45:08PM +0200, Michael S. Tsirkin wrote:
> > -        kfree(neigh);
> >         neigh->neighbour->ops->destructor = NULL;
> > +        kfree(neigh);
> > 
> 
> If the order of these operations really matters, wont you need
> some kind of memory barrier in between?

The original code was wrong even on a strongly ordered memory machine.

grant


From xma at us.ibm.com  Fri Feb 18 09:58:37 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Fri, 18 Feb 2005 17:58:37 +0000
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <20050218154624.GC31228@mellanox.co.il>
Message-ID: <OFB2533074.126BFA29-ON87256FAC.00629FAC-00256FAC.006308E5@us.ibm.com>

> Why is this necessary?

In case the neigh_destroy() calls after ipoib neigh is freed.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050218/8b092686/attachment.html>

From xma at us.ibm.com  Fri Feb 18 10:01:15 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Fri, 18 Feb 2005 18:01:15 +0000
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <20050218103855.GB30863@mellanox.co.il>
Message-ID: <OFB35FD8FB.6FDD66D9-ON87256FAC.0062F10E-00256FAC.0063467A@us.ibm.com>

> Please, generate patches with diff -p option.

Do you want me to regenerate all the patches?

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050218/37b272b7/attachment.html>

From roland at topspin.com  Fri Feb 18 10:03:23 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 18 Feb 2005 10:03:23 -0800
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <OFB35FD8FB.6FDD66D9-ON87256FAC.0062F10E-00256FAC.0063467A@us.ibm.com>
	(Shirley Ma's message of "Fri, 18 Feb 2005 18:01:15 +0000")
References: <OFB35FD8FB.6FDD66D9-ON87256FAC.0062F10E-00256FAC.0063467A@us.ibm.com>
Message-ID: <523bvtvj84.fsf@topspin.com>

    Shirley> Do you want me to regenerate all the patches?

No need from my point of view.  All "-p" does is annotate patches with
the name of the function where each chunk comes from, which makes them
slightly easier to review.

By the way, sorry I've been so slow in applying/reviewing your
patches.  I've had a lot of stuff to catch up on, but I should be able
to devote some time to your patches no later than the beginning of
next week.

Thanks,
  Roland


From xma at us.ibm.com  Fri Feb 18 10:12:37 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Fri, 18 Feb 2005 18:12:37 +0000
Subject: [openib-general] Re: [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <523bvtvj84.fsf@topspin.com>
Message-ID: <OFE43D1BE8.57A09A33-ON87256FAC.0063B1DA-00256FAC.0064512C@us.ibm.com>

> I've had a lot of stuff to catch up on, but I should be able
to devote some time to your patches no later than the beginning of
next week.

Thanks.

Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050218/cd85a8fa/attachment.html>

From libor at topspin.com  Fri Feb 18 10:43:57 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 18 Feb 2005 10:43:57 -0800
Subject: [openib-general] [PATCH][SDP][0/33] Fix comment style in SDP
In-Reply-To: <11086781103458@sun.com>;
	from Tom.Duffy@Sun.COM on Thu, Feb 17, 2005 at 02:08:30PM -0800
References: <11086781103458@sun.com>
Message-ID: <20050218104357.A15428@topspin.com>

On Thu, Feb 17, 2005 at 02:08:30PM -0800, Tom Duffy wrote:
> These patches clean up the comment formatting in sdp.  They also update
> the copyright header to match the rest of OpenIB files.
> 
> Signed-off-by: Tom Duffy <tduffy at sun.com>

  Thanks Tom, the patches are applied and commited. Some of the function
header comments are probably unnecessary, or more accurately not useful,
but that can be fixed as we go along.

  
-Libor



From libor at topspin.com  Fri Feb 18 10:55:03 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 18 Feb 2005 10:55:03 -0800
Subject: [openib-general] [PATCH] SDP headers using unsafe macros
In-Reply-To: <20050217154643.GW28446@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 17, 2005 at 05:46:43PM +0200
References: <20050217154643.GW28446@mellanox.co.il>
Message-ID: <20050218105503.B15428@topspin.com>

On Thu, Feb 17, 2005 at 05:46:43PM +0200, Michael S. Tsirkin wrote:
> 
> Some SDP hdeaders defines unsafe macros, which may be
> dangerous to use, e.g. inside an if or while statement.

Thanks Michael, applied and commited.

-Libor

> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
> 
> Index: sdp_proto.h
> ===================================================================
> --- sdp_proto.h	(revision 1810)
> +++ sdp_proto.h	(working copy)
> @@ -585,15 +585,16 @@
>  #define SDP_EXPECT(expr)
>  #define SDP_CHECK_NULL(value, result)
>  #else
> -#define SDP_EXPECT(expr)                                             \
> -{                                                                    \
> -  if (!(expr)) {                                                     \
> -    sdp_dbg_err("EXCEPT: Internal error check <%s> failed.", #expr); \
> -  }                                                                  \
> -} /* SDP_EXPECT */
> +#define SDP_EXPECT(expr)                                                   \
> +do {                                                                       \
> +	if (!(expr)) {                                                     \
> +		sdp_dbg_err("EXCEPT: Internal error check <%s> failed.",   \
> +			#expr);                                            \
> +	}                                                                  \
> +} while (0) /* SDP_EXPECT */
>  
>  #define SDP_CHECK_NULL(value, result) \
> -        if (NULL == (value)) return (result);
> +        do { if (NULL == (value)) return (result); } while (0)
>  #endif
>  /* --------------------------------------------------------------------- */
>  /* Inline functions.                                                     */
> Index: sdp_conn.h
> ===================================================================
> --- sdp_conn.h	(revision 1810)
> +++ sdp_conn.h	(working copy)
> @@ -102,7 +102,7 @@
>   * transition one of the disconnect pending states to disconnect send
>   */
>  #define SDP_ST_PEND_2_SEND(conn) \
> -        (conn)->state = ((conn)->state + 0x0100) & ~SDP_ST_MASK_DIS_PEND;
> +        ((conn)->state = ((conn)->state + 0x0100) & ~SDP_ST_MASK_DIS_PEND)
>  /*
>   * internal connection structure
>   */
> @@ -130,14 +130,14 @@
>   * connection handle within a socket.
>   */
>  #define SDP_GET_CONN(sk) \
> -       *((struct sdp_opt **)&(sk)->sk_protinfo)
> +       (*((struct sdp_opt **)&(sk)->sk_protinfo))
>  #define SDP_SET_CONN(sk, conn) \
> -       *((struct sdp_opt **)&(sk)->sk_protinfo) = (conn)
> +       (*((struct sdp_opt **)&(sk)->sk_protinfo) = (conn))
>  
>  #define SDP_CONN_SET_ERR(conn, val) \
> -        (conn)->error = (conn)->sk->sk_err = (val)
> +        ((conn)->error = (conn)->sk->sk_err = (val))
>  #define SDP_CONN_GET_ERR(conn) \
> -        (conn)->error
> +        ((conn)->error)
>  /* --------------------------------------------------------------------- */
>  /* state transition information recording                                */
>  /* --------------------------------------------------------------------- */
> @@ -153,7 +153,7 @@
>  }; /* struct sdp_conn_state */
>  
>  #define SDP_CONN_ST_SET(conn, val) \
> -{ \
> +do { \
>    (conn)->state = (val); \
>    if (SDP_CONN_STATE_MAX > (conn)->state_rec.value) { \
>      (conn)->state_rec.state[(conn)->state_rec.value] = (val); \
> @@ -161,10 +161,10 @@
>      (conn)->state_rec.line[(conn)->state_rec.value] = __LINE__; \
>      (conn)->state_rec.value++; \
>    } \
> -} 
> +} while(0)
>  
>  #define SDP_CONN_ST_INIT(conn) \
> -{ \
> +do { \
>    (conn)->state = SDP_CONN_ST_INVALID; \
>    for ((conn)->state_rec.value = 0; \
>         SDP_CONN_STATE_MAX > (conn)->state_rec.value; \
> @@ -174,10 +174,10 @@
>      (conn)->state_rec.line[(conn)->state_rec.value] = 0; \
>    } \
>    (conn)->state_rec.value = 0; \
> -} 
> +} while(0)
>  #else
> -#define SDP_CONN_ST_SET(conn, val) (conn)->state = (val)
> -#define SDP_CONN_ST_INIT(conn)     (conn)->state = SDP_CONN_ST_INVALID
> +#define SDP_CONN_ST_SET(conn, val) ((conn)->state = (val))
> +#define SDP_CONN_ST_INIT(conn)     ((conn)->state = SDP_CONN_ST_INVALID)
>  #endif
>  /*
>   * connection lock
> 
> -- 
> MST - Michael S. Tsirkin
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From libor at topspin.com  Fri Feb 18 12:03:06 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 18 Feb 2005 12:03:06 -0800
Subject: [openib-general] [PATCH] remove unused code from SDP.
Message-ID: <20050218120306.C15428@topspin.com>


  There's code in SDP which is never used, mostly helper functions that
at the time looked like they might be useful, but never got used.

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: sdp_link.c
===================================================================
--- sdp_link.c	(revision 1831)
+++ sdp_link.c	(working copy)
@@ -321,7 +321,7 @@
 /*
  * _sdp_link_path_rec_get - resolve GIDs to a path record
  */
-int _sdp_link_path_rec_get(struct sdp_path_info *info)
+static int _sdp_link_path_rec_get(struct sdp_path_info *info)
 {
 	int result;
 
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1831)
+++ sdp_inet.c	(working copy)
@@ -1736,16 +1736,6 @@
 		goto error_proc;
 	}
 	/*
-	 * generic table
-	 */
-	result = sdp_main_desc_init();
-	if (0 > result) {
-
-		sdp_warn("INIT: Error <%d> initializing generic table",
-			 result);
-		goto error_generic;
-	}
-	/*
 	 * advertisment table
 	 */
 	result = sdp_main_advt_init();
@@ -1816,8 +1806,6 @@
 error_link:
 	(void)sdp_main_advt_cleanup();
 error_advt:
-	(void)sdp_main_desc_cleanup();
-error_generic:
 	(void)sdp_main_proc_cleanup();
 error_proc:
 	return result;		/* success */
@@ -1850,10 +1838,6 @@
 	 */
 	(void)sdp_main_advt_cleanup();
 	/*
-	 * delete generic table
-	 */
-	(void)sdp_main_desc_cleanup();
-	/*
 	 * proc tables
 	 */
 	(void)sdp_main_proc_cleanup();
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1832)
+++ sdp_proto.h	(working copy)
@@ -56,8 +56,6 @@
 
 int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff);
 
-int sdp_buff_pool_size(void);
-
 int sdp_buff_pool_buff_size(void);
 
 int sdp_buff_q_init(struct sdpc_buff_q *pool, u32 size);
@@ -81,8 +79,6 @@
 
 struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool);
 
-struct sdpc_buff *sdp_buff_q_look_tail(struct sdpc_buff_q *pool);
-
 int sdp_buff_q_put(struct sdpc_buff_q *pool, struct sdpc_buff *buff);
 
 int sdp_buff_q_put_head(struct sdpc_buff_q *pool, struct sdpc_buff *buff);
@@ -94,16 +90,6 @@
 						    void *arg),
 					void *usr_arg);
 
-struct sdpc_buff *sdp_buff_q_fetch_tail(struct sdpc_buff_q *pool,
-					int (*test)(struct sdpc_buff *buff,
-						    void *arg),
-					void *usr_arg);
-
-int sdp_buff_q_trav_tail(struct sdpc_buff_q *pool,
-			 int (*trav_func)(struct sdpc_buff *buff,
-					  void *arg),
-			 void *usr_arg);
-
 int sdp_buff_q_trav_head(struct sdpc_buff_q *pool,
 			 int (*trav_func)(struct sdpc_buff *buff,
 					  void *arg),
@@ -129,8 +115,6 @@
 /*
  * Wall between userspace protocol and SDP protocol proper
  */
-int sdp_wall_send_reject(struct sdp_opt *conn);
-
 int sdp_wall_send_close(struct sdp_opt *conn);
 
 int sdp_wall_send_closing(struct sdp_opt *conn);
@@ -162,14 +146,10 @@
 
 int sdp_main_advt_cleanup(void);
 
-struct sdpc_advt_q *sdp_advt_q_create(int *result);
-
 int sdp_advt_q_init(struct sdpc_advt_q *table);
 
 int sdp_advt_q_clear(struct sdpc_advt_q *table);
 
-int sdp_advt_q_destroy(struct sdpc_advt_q *table);
-
 struct sdpc_advt *sdp_advt_create(void);
 
 int sdp_advt_destroy(struct sdpc_advt *advt);
@@ -187,14 +167,10 @@
 
 int sdp_main_iocb_cleanup(void);
 
-struct sdpc_iocb_q *sdp_iocb_q_create(int *result);
-
 int sdp_iocb_q_init(struct sdpc_iocb_q *table);
 
 int sdp_iocb_q_clear(struct sdpc_iocb_q *table);
 
-int sdp_iocb_q_destroy(struct sdpc_iocb_q *table);
-
 struct sdpc_iocb *sdp_iocb_create(void);
 
 int sdp_iocb_destroy(struct sdpc_iocb *iocb);
@@ -209,8 +185,6 @@
 
 int sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb);
 
-struct sdpc_iocb *sdp_iocb_q_get_key(struct sdpc_iocb_q *table, u32 key);
-
 struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key);
 
 int sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
@@ -232,8 +206,6 @@
  */
 int sdp_desc_q_remove(struct sdpc_desc *element);
 
-struct sdpc_desc *sdp_desc_q_get_all(struct sdpc_desc_q *table);
-
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table);
 
 struct sdpc_desc *sdp_desc_q_get_tail(struct sdpc_desc_q *table);
@@ -246,12 +218,8 @@
 
 struct sdpc_desc *sdp_desc_q_look_head(struct sdpc_desc_q *table);
 
-struct sdpc_desc *sdp_desc_q_look_tail(struct sdpc_desc_q *table);
-
 int sdp_desc_q_type_head(struct sdpc_desc_q *table);
 
-int sdp_desc_q_type_tail(struct sdpc_desc_q *table);
-
 struct sdpc_desc *sdp_desc_q_look_type_head(struct sdpc_desc_q *table,
 					   enum sdp_desc_type type);
 
@@ -266,20 +234,10 @@
 int sdp_desc_q_types_size(struct sdpc_desc_q *table,
 			  enum sdp_desc_type type);
 
-int sdp_desc_q_size(struct sdpc_desc_q *table);
-
-struct sdpc_desc_q *sdp_desc_q_create(int *result);
-
 int sdp_desc_q_init(struct sdpc_desc_q *table);
 
 int sdp_desc_q_clear(struct sdpc_desc_q *table);
 
-int sdp_desc_q_destroy(struct sdpc_desc_q *table);
-
-int sdp_main_desc_init(void);
-
-int sdp_main_desc_cleanup(void);
-
 /*
  * proc entry managment
  */
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1831)
+++ sdp_conn.c	(working copy)
@@ -540,9 +540,9 @@
 } /* sdp_inet_port_inherit */
 
 /*
- * sdp_conn_table_insert - insert a connection into the connection table
+ * _sdp_conn_table_insert - insert a connection into the connection table
  */
-int sdp_conn_table_insert(struct sdp_opt *conn)
+static int _sdp_conn_table_insert(struct sdp_opt *conn)
 {
 	u32 counter;
 	int result = -ENOMEM;
@@ -589,7 +589,7 @@
 	 */
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return result;
-} /* sdp_conn_table_insert */
+} /* _sdp_conn_table_insert */
 
 /*
  * sdp_conn_table_remove - remove a connection from the connection table
@@ -1426,7 +1426,7 @@
 	/*
 	 * insert connection into lookup table
 	 */
-	result = sdp_conn_table_insert(conn);
+	result = _sdp_conn_table_insert(conn);
 	if (0 > result) {
 
 		sdp_dbg_warn(conn, "Error <%d> conn table insert <%d:%d>",
Index: sdp_advt.c
===================================================================
--- sdp_advt.c	(revision 1831)
+++ sdp_advt.c	(working copy)
@@ -35,7 +35,6 @@
 #include "sdp_main.h"
 
 static kmem_cache_t *__sdp_advt_cache = NULL;
-static kmem_cache_t *__sdp_advt_table = NULL;
 
 /*
  * module specific functions
@@ -171,33 +170,6 @@
 } /* sdp_advt_q_put */
 
 /*
- * public table functions
- */
-
-/*
- * sdp_advt_q_create - create an advertisment table
- */
-struct sdpc_advt_q *sdp_advt_q_create(int * result)
-{
-	struct sdpc_advt_q *table = NULL;
-
-	SDP_CHECK_NULL(result, NULL);
-
-	table = kmem_cache_alloc(__sdp_advt_table, SLAB_KERNEL);
-	if (NULL == table) {
-
-		*result = -ENOMEM;
-		return NULL;
-	}
-
-	table->head = NULL;
-	table->size = 0;
-
-	*result = 0;
-	return table;
-} /* sdp_advt_q_create */
-
-/*
  * sdp_advt_q_init - initialize a new empty advertisment table
  */
 int sdp_advt_q_init(struct sdpc_advt_q *table)
@@ -232,31 +204,6 @@
 } /* sdp_advt_q_clear */
 
 /*
- * sdp_advt_q_destroy - destroy an advertisment table
- */
-int sdp_advt_q_destroy(struct sdpc_advt_q *table)
-{
-	struct sdpc_advt *advt;
-	int result;
-
-	SDP_CHECK_NULL(table, -EINVAL);
-	/*
-	 * drain the table of any objects
-	 */
-	while (NULL != (advt = sdp_advt_q_get(table))) {
-
-		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
-	}
-	/*
-	 * return the table to the cache
-	 */
-	kmem_cache_free(__sdp_advt_table, table);
-
-	return 0;
-} /* sdp_advt_q_destroy */
-
-/*
  * primary initialization/cleanup functions
  */
 
@@ -271,7 +218,7 @@
 	/*
 	 * initialize the caches only once.
 	 */
-	if (NULL != __sdp_advt_cache || NULL != __sdp_advt_table) {
+	if (NULL != __sdp_advt_cache) {
 
 		sdp_warn("Advertisment caches already initialized.");
 		return -EINVAL;
@@ -287,20 +234,7 @@
 		goto error_advt_c;
 	}
 
-	__sdp_advt_table = kmem_cache_create("sdp_advt_q",
-					     sizeof(struct sdpc_advt_q),
-					     0, SLAB_HWCACHE_ALIGN, NULL,
-					     NULL);
-	if (NULL == __sdp_advt_table) {
-
-		result = -ENOMEM;
-		goto error_advt_t;
-	}
-
 	return 0;
-error_advt_t:
-	kmem_cache_destroy(__sdp_advt_cache);
-	__sdp_advt_cache = NULL;
 error_advt_c:
 	return 0;
 } /* sdp_main_advt_init */
@@ -315,12 +249,10 @@
 	 * cleanup the caches
 	 */
 	kmem_cache_destroy(__sdp_advt_cache);
-	kmem_cache_destroy(__sdp_advt_table);
 	/*
 	 * null out entries.
 	 */
 	__sdp_advt_cache = NULL;
-	__sdp_advt_table = NULL;
 
 	return 0;
 } /* sdp_main_advt_cleanup */
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1831)
+++ sdp_wall.c	(working copy)
@@ -39,30 +39,6 @@
  */
 
 /*
- * sdp_wall_send_reject - callback to reject an active open
- */
-int sdp_wall_send_reject(struct sdp_opt *conn)
-{
-	int result;
-
-	SDP_CHECK_NULL(conn, -EINVAL);
-
-	sdp_dbg_ctrl(conn, "Connect reject. src <%08x:%04x> dst <%08x:%04x>",
-		     conn->src_addr, conn->src_port, 
-		     conn->dst_addr, conn->dst_port);
-	/*
-	 * respond to the remote connection manager with a REQ_REJ
-	 */
-	result = sdp_cm_reject(conn);
-	if (0 > result) {
-
-		sdp_dbg_warn(conn, "Error <%d> CM connect reject", result);
-	}
-
-	return 0;
-} /* sdp_wall_send_reject */
-
-/*
  * sdp_wall_send_close - callback to accept an active close
  */
 int sdp_wall_send_close(struct sdp_opt *conn)
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1831)
+++ sdp_iocb.c	(working copy)
@@ -35,7 +35,6 @@
 #include "sdp_main.h"
 
 static kmem_cache_t *__sdp_iocb_cache = NULL;
-static kmem_cache_t *__sdp_iocb_table = NULL;
 
 /*
  * memory locking functions
@@ -778,30 +777,6 @@
 } /* sdp_iocb_q_look */
 
 /*
- * sdp_iocb_q_get_key - find an iocb based on key, and remove it
- */
-struct sdpc_iocb *sdp_iocb_q_get_key(struct sdpc_iocb_q *table, u32 key)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	iocb = sdp_iocb_q_lookup(table, key);
-	if (NULL == iocb) {
-
-		goto done;
-	}
-
-	result = sdp_iocb_q_remove(iocb);
-	if (0 > result) {
-
-		iocb = NULL;
-	}
-
-done:
-	return iocb;
-} /* sdp_iocb_q_get_key */
-
-/*
  * _sdp_iocb_q_get - get, and remove, the object at the tables head
  */
 static struct sdpc_iocb *_sdp_iocb_q_get(struct sdpc_iocb_q *table, int head)
@@ -852,9 +827,9 @@
 /*
  * _sdp_iocb_q_put - put the IOCB object at the tables tail
  */
-int _sdp_iocb_q_put(struct sdpc_iocb_q *table,
-			      struct sdpc_iocb *iocb,
-			      int head)
+static int _sdp_iocb_q_put(struct sdpc_iocb_q *table,
+			   struct sdpc_iocb *iocb,
+			   int head)
 {
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
@@ -966,31 +941,6 @@
 } /* sdp_iocb_q_cancel */
 
 /*
- * public table functions
- */
-
-/*
- * sdp_iocb_q_create - create an IOCB table
- */
-struct sdpc_iocb_q *sdp_iocb_q_create(int *result)
-{
-	struct sdpc_iocb_q *table = NULL;
-
-	table = kmem_cache_alloc(__sdp_iocb_table, SLAB_KERNEL);
-	if (NULL == table) {
-
-		*result = -ENOMEM;
-		return NULL;
-	}
-
-	table->head = NULL;
-	table->size = 0;
-
-	*result = 0;
-	return table;
-} /* sdp_iocb_q_create */
-
-/*
  * sdp_iocb_q_init - initialize a new empty IOCB table
  */
 int sdp_iocb_q_init(struct sdpc_iocb_q *table)
@@ -1025,27 +975,6 @@
 } /* sdp_iocb_q_clear */
 
 /*
- * sdp_iocb_q_destroy - destroy an IOCB table
- */
-int sdp_iocb_q_destroy(struct sdpc_iocb_q *table)
-{
-	int result;
-
-	SDP_CHECK_NULL(table, -EINVAL);
-	/*
-	 * drain the table of any objects
-	 */
-	result = sdp_iocb_q_clear(table);
-	SDP_EXPECT(!(0 > result));
-	/*
-	 * return the table to the cache
-	 */
-	kmem_cache_free(__sdp_iocb_table, table);
-
-	return 0;
-} /* sdp_iocb_q_destroy */
-
-/*
  * primary initialization/cleanup functions
  */
 
@@ -1069,8 +998,7 @@
 	/*
 	 * initialize the caches only once.
 	 */
-	if (NULL != __sdp_iocb_cache ||
-	    NULL != __sdp_iocb_table) {
+	if (NULL != __sdp_iocb_cache) {
 
 		sdp_warn("IOCB caches already initialized.");
 		return -EINVAL;
@@ -1086,20 +1014,7 @@
 		goto error_iocb_c;
 	}
 
-	__sdp_iocb_table = kmem_cache_create("sdp_iocb_q",
-					     sizeof(struct sdpc_iocb_q),
-					     0, SLAB_HWCACHE_ALIGN, NULL,
-					     NULL);
-	if (NULL == __sdp_iocb_table) {
-
-		result = -ENOMEM;
-		goto error_iocb_t;
-	}
-
 	return 0;
-error_iocb_t:
-	kmem_cache_destroy(__sdp_iocb_cache);
-	__sdp_iocb_cache = NULL;
 error_iocb_c:
 	(void)_sdp_mem_lock_cleanup();
 	return result;
@@ -1115,12 +1030,10 @@
 	 * cleanup the caches
 	 */
 	kmem_cache_destroy(__sdp_iocb_cache);
-	kmem_cache_destroy(__sdp_iocb_table);
 	/*
 	 * null out entries.
 	 */
 	__sdp_iocb_cache = NULL;
-	__sdp_iocb_table = NULL;
 	/*
 	 * cleanup memory locking
 	 */
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1831)
+++ sdp_buff.c	(working copy)
@@ -299,18 +299,6 @@
 } /* sdp_buff_q_look_head */
 
 /*
- * sdp_buff_q_look_tail - look at the buffer at the end of the pool
- */
-struct sdpc_buff *sdp_buff_q_look_tail(struct sdpc_buff_q *pool)
-{
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_look(pool, 0);
-
-	return buff;
-} /* sdp_buff_q_look_tail */
-
-/*
  * sdp_buff_q_fetch_head - Get the pools first buffer, if the test passes
  */
 struct sdpc_buff *sdp_buff_q_fetch_head(struct sdpc_buff_q *pool,
@@ -326,21 +314,6 @@
 } /* sdp_buff_q_fetch_head */
 
 /*
- * sdp_buff_q_fetch_tail - Get the pools last buffer, if the test passes
- */
-struct sdpc_buff *sdp_buff_q_fetch_tail(struct sdpc_buff_q *pool,
-					int (*test)(struct sdpc_buff *buff,
-						    void *arg),
-					void *usr_arg)
-{
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 0, test, usr_arg);
-
-	return buff;
-} /* sdp_buff_q_fetch_tail */
-
-/*
  * sdp_buff_q_fetch - Get the first matching buffer from the pool
  */
 struct sdpc_buff *sdp_buff_q_fetch(struct sdpc_buff_q *pool,
@@ -383,41 +356,6 @@
 } /* sdp_buff_q_fetch_head */
 
 /*
- * sdp_buff_q_trav_tail - traverse buffers in pool,from the tail
- */
-int sdp_buff_q_trav_tail(struct sdpc_buff_q *pool,
-			 int (*trav_func)(struct sdpc_buff *buff,
-					  void *arg),
-			 void *usr_arg)
-{
-	struct sdpc_buff *buff;
-	int result = 0;
-	int counter;
-
-	SDP_CHECK_NULL(pool, -EINVAL);
-	SDP_CHECK_NULL(trav_func, -EINVAL);
-	/*
-	 * check to see if there is anything to traverse.
-	 */
-	if (NULL != pool->head) {
-		/*
-		 * lock to prevent corruption of table
-		 */
-		for (counter = 0, buff = pool->head->prev;
-		     counter < pool->size; counter++, buff = buff->prev) {
-
-			result = trav_func(buff, usr_arg);
-			if (0 > result) {
-
-				break;
-			}
-		}
-	}
-
-	return result;
-} /* sdp_buff_q_trav_tail */
-
-/*
  * sdp_buff_q_trav_head - traverse buffers in pool, from the head
  */
 int sdp_buff_q_trav_head(struct sdpc_buff_q *pool,
@@ -988,26 +926,6 @@
 } /* sdp_buff_pool_chain_put */
 
 /*
- * sdp_buff_pool_size - number of elements in the main buffer pool
- */
-int sdp_buff_pool_size(void)
-{
-	unsigned long flags;
-	int size;
-
-	if (NULL == main_pool) {
-
-		return -1;
-	}
-
-	spin_lock_irqsave(&main_pool->lock, flags);
-	size = sdp_buff_q_size(&main_pool->pool);
-	spin_unlock_irqrestore(&main_pool->lock, flags);
-
-	return size;
-} /* sdp_buff_pool_size */
-
-/*
  * sdp_buff_pool_buff_size - return the size of buffers in the main pool
  */
 int sdp_buff_pool_buff_size(void)
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1831)
+++ sdp_queue.c	(working copy)
@@ -34,8 +34,6 @@
 
 #include "sdp_main.h"
 
-static kmem_cache_t *__sdp_desc_table = NULL;
-
 /*
  * module specific functions
  */
@@ -207,23 +205,6 @@
 } /* sdp_desc_q_lookup */
 
 /*
- * sdp_desc_q_get_all - Get the element at the front of the table
- */
-struct sdpc_desc *sdp_desc_q_get_all(struct sdpc_desc_q *table)
-{
-	struct sdpc_desc *head;
-
-	head = table->head;
-
-	table->head = NULL;
-	table->size = 0;
-
-	memset(table->count, 0, sizeof(table->count));
-
-	return head;
-} /* sdp_desc_q_get_all */
-
-/*
  * sdp_desc_q_get_head - Get the element at the front of the table
  */
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table)
@@ -268,16 +249,6 @@
 } /* sdp_desc_q_look_head */
 
 /*
- * sdp_desc_q_look_tail - look at the end of the table
- */
-struct sdpc_desc *sdp_desc_q_look_tail(struct sdpc_desc_q *table)
-{
-	SDP_CHECK_NULL(table, NULL);
-
-	return ((NULL == table->head) ? NULL : table->head->prev);
-} /* sdp_desc_q_look_tail */
-
-/*
  * sdp_desc_q_type_head - look at the type at the front of the table
  */
 int sdp_desc_q_type_head(struct sdpc_desc_q *table)
@@ -295,23 +266,6 @@
 } /* sdp_desc_q_type_head */
 
 /*
- * sdp_desc_q_type_tail - look at the type at the end of the table
- */
-int sdp_desc_q_type_tail(struct sdpc_desc_q *table)
-{
-	SDP_CHECK_NULL(table, -EINVAL);
-
-	if (NULL == table->head) {
-
-		return SDP_DESC_TYPE_NONE;
-	}
-	else {
-
-		return table->head->prev->type;
-	}
-} /* sdp_desc_q_type_tail */
-
-/*
  * sdp_desc_q_look_type_head - look at a specific object
  */
 struct sdpc_desc *sdp_desc_q_look_type_head(struct sdpc_desc_q *table, 
@@ -361,35 +315,6 @@
 } /* sdp_desc_q_types_size */
 
 /*
- * public table functions
- */
-
-/*
- * sdp_desc_q_create - create/allocate a generic table
- */
-struct sdpc_desc_q *sdp_desc_q_create(int *result)
-{
-	struct sdpc_desc_q *table = NULL;
-
-	SDP_CHECK_NULL(result, NULL);
-
-	table = kmem_cache_alloc(__sdp_desc_table, SLAB_ATOMIC);
-	if (NULL == table) {
-
-		*result = -ENOMEM;
-		return NULL;
-	}
-
-	table->head = NULL;
-	table->size = 0;
-
-	memset(table, 0, sizeof(struct sdpc_desc_q));
-
-	*result = 0;
-	return table;
-} /* sdp_desc_q_create */
-
-/*
  * sdp_desc_q_init - initialize a new empty generic table
  */
 int sdp_desc_q_init(struct sdpc_desc_q *table)
@@ -427,77 +352,3 @@
 
 	return 0;
 } /* sdp_desc_q_clear */
-
-/*
- * sdp_desc_q_destroy - destroy a generic table
- */
-int sdp_desc_q_destroy(struct sdpc_desc_q *table)
-{
-	int result;
-
-	SDP_CHECK_NULL(table, -EINVAL);
-	/*
-	 * drain the table of any objects
-	 */
-	result = sdp_desc_q_clear(table);
-	SDP_EXPECT(!(0 > result));
-	/*
-	 * return the table to the cache
-	 */
-	kmem_cache_free(__sdp_desc_table, table);
-
-	return 0;
-} /* sdp_desc_q_destroy */
-
-/*
- * primary initialization/cleanup functions
- */
-
-/*
- * sdp_main_desc_init - initialize the generic table caches
- */
-int sdp_main_desc_init(void)
-{
-	int result;
-
-	sdp_dbg_init("Generic table cache initialization.");
-	/*
-	 * initialize the caches only once.
-	 */
-	if (NULL != __sdp_desc_table) {
-
-		sdp_warn("Generic table caches already initialized.");
-		return -EINVAL;
-	}
-
-	__sdp_desc_table = kmem_cache_create("sdp_desc_q",
-					     sizeof(struct sdpc_desc_q), 0,
-					     SLAB_HWCACHE_ALIGN, NULL, NULL);
-	if (NULL == __sdp_desc_table) {
-
-		result = -ENOMEM;
-		goto error_advt_t;
-	}
-
-	return 0;
-error_advt_t:
-	return 0;
-} /* sdp_main_desc_init */
-
-/*
- * sdp_main_desc_cleanup - cleanup the generic table caches
- */
-int sdp_main_desc_cleanup(void)
-{
-	sdp_dbg_init("Generic table cache cleanup.");
-	/*
-	 * cleanup the caches
-	 */
-	kmem_cache_destroy(__sdp_desc_table);
-	/*
-	 * null out entries.
-	 */
-	__sdp_desc_table = NULL;
-
-	return 0;
-} /* sdp_main_desc_cleanup */


From roland at topspin.com  Fri Feb 18 12:23:10 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 18 Feb 2005 12:23:10 -0800
Subject: [openib-general] [PATCH] remove unused code from SDP.
In-Reply-To: <20050218120306.C15428@topspin.com> (Libor Michalek's message
	of "Fri, 18 Feb 2005 12:03:06 -0800")
References: <20050218120306.C15428@topspin.com>
Message-ID: <52u0o9ty6p.fsf@topspin.com>

My kind of patch :)  diffstat shows:

 9 files changed, 10 insertions(+), 478 deletions(-)

 - R.


From ted.kim at sun.com  Fri Feb 18 15:10:27 2005
From: ted.kim at sun.com (Ted H. Kim)
Date: Fri, 18 Feb 2005 15:10:27 -0800 (PST)
Subject: [openib-general] Re: [swg] CM Pathing Question
Message-ID: <200502182310.j1INAR0P013229@hs-uego02-02.West.Sun.COM>

Hal,

Within the SWG there were discussions on this topic a while back.
Folks came up with various suggestions. However, at the time, we 
did not choose to mandate a particular way for the CM to track 
possible changes to the CM path. So officially, the spec is silent
here.

As you may be hinting, when the CM is using a path that is the
same as the data path, it might be okay to also use the
alt path info. But that assumes of course that the new data path
is also suitable for mgt traffic (which might not be true, 
e.g. it may not be reversible path).

The most general procedure is probably to initiate new SA
queries whenever a new CM path is needed. Since the Remote CA GUID
is recorded, that can be used to locate the NodeRecord. The RID
of the NodeRecord will contain a LID, which can be fed to
a PathRecord query (with Reversible flag on).


-ted



>OpenIB is in the midst of implementing the CM and the following question
>about the CM has come up on openib-general mailing list and no one has
>yet supplied an answer so I thought I'd go to the source. I am cross
>posting although members of openib-general won't necessarily be able to
>respond directly to any response.
>
>It appears that there is the possibility of a CM path and one or more
>data paths (primary/alternate) per connection. The CM path may or may
>not be the same as the data path(s).
>
>Assume that a connection with both a primary and secondary path is
>setup, using the same primary path for CM communication. Some time later
>the primary path fails, so APM causes the connection to migrate to 
>the secondary path. A "path migrated" async event is issued in the end
>node which results in the desire to set up a new secondary path so that
>the connection remains resilent. Where does the LAP get sent ? 
>Similarly for any other future CM messages (like DREQ, etc.). It seems
>to need to get sent along the new path. 
>
>How is the CM path expected to be maintained in the two cases (when it
>is the same and when is different from a data path) in the presence of a
>path failure ? Maybe those are the same case in terms of the
>architecture.


Ted H. Kim
Sun Microsystems, Inc.                  ted.kim at sun.com
222 North Sepulveda Blvd., 10th Floor   (310) 341-1116
El Segundo, CA  90245                   (310) 341-1120 FAX



From libor at topspin.com  Fri Feb 18 15:51:44 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 18 Feb 2005 15:51:44 -0800
Subject: [openib-general] [PATCH] Re: SDP_CONN_LOCK
In-Reply-To: <42152D8B.2040608@ichips.intel.com>;
	from mshefty@ichips.intel.com on Thu, Feb 17, 2005 at 03:49:31PM
	-0800
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
Message-ID: <20050218155144.D15428@topspin.com>

On Thu, Feb 17, 2005 at 03:49:31PM -0800, Sean Hefty wrote:
> Michael S. Tsirkin wrote:
> > Quoting r. Roland Dreier <roland at topspin.com>:
> >>
> >>BTW, since mthca currently calls completion handlers directly from
> >>interrupt context (rather than BH/tasklet context), it might be worth
> >>renaming all the SDP locking macros so they're not confusingly named
> >>with _BH suffixes.
> > 
> > I think it would be much nicer to reduce the number of macros used.
> 
> I think these would probably be better off as just function calls, 
> rather than macros.  SDP_CONN_LOCK calls sdp_conn_internal_lock(), and 
> that appears to be the only place that the function is called. 
> Similarly, SDP_CONN_UNLOCK calls sdp_conn_internal_unlock().  It seems 
> that you could just merge the macros into the function calls.

  Attached is a patch that replaces some of the connection macros with
inline function. The two internal functions you mention were not removed.
When the lock has no contention, which is the common case, those functions
are never called which is why the code was split into a macro and funciton
in the first place. The BH suffixes were replaced w/ IRQ.
  
Replaced macros:

  SDP_CONN_LOCK
  SDP_CONN_UNLOCK
  SDP_CONN_RELOCK
  SDP_CONN_HOLD
  SDP_CONN_PUT
  SDP_CONN_ERROR

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1836)
+++ sdp_inet.c	(working copy)
@@ -375,7 +375,7 @@
 	 */
 	sock->sk = NULL;
 
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 	conn->shutdown = SHUTDOWN_MASK;
 
 	if (SDP_SOCK_ST_LISTEN == conn->istate) {
@@ -456,9 +456,9 @@
 			while (0 < timeout &&
 			       0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
 
-				SDP_CONN_UNLOCK(conn);
+				sdp_conn_unlock(conn);
 				timeout = schedule_timeout(timeout);
-				SDP_CONN_LOCK(conn);
+				sdp_conn_lock(conn);
 
 				if (signal_pending(current)) {
 					
@@ -499,8 +499,8 @@
 	 * finally drop socket reference. (socket API reference)
 	 */
 	sock_orphan(sk);
-	SDP_CONN_UNLOCK(conn);
-	SDP_CONN_PUT(conn);
+	sdp_conn_unlock(conn);
+	sdp_conn_put(conn);
 
 	return 0;
 } /* _sdp_inet_release */
@@ -568,7 +568,7 @@
 	/*
 	 * socket checks.
 	 */
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	if (SDP_SOCK_ST_CLOSED != conn->istate || 0 < conn->src_port) {
 
@@ -613,7 +613,7 @@
 
 	result = 0;
 done:
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return result;
 } /* _sdp_inet_bind */
 
@@ -663,7 +663,7 @@
 	/*
 	 * lock socket
 	 */
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	switch (sock->state) {
 	case SS_UNCONNECTED:
@@ -696,7 +696,7 @@
 		conn->dst_addr = ntohl(addr->sin_addr.s_addr);
 		conn->dst_port = ntohs(addr->sin_port);
 
-		SDP_CONN_HOLD(conn);	/* CM reference */
+		sdp_conn_hold(conn);	/* CM reference */
 		/*
 		 * close, allow connection completion notification.
 		 */
@@ -716,7 +716,7 @@
 			sock->state = SS_UNCONNECTED;
 			conn->istate = SDP_SOCK_ST_CLOSED;
 
-			SDP_CONN_PUT(conn);	/* CM reference */
+			sdp_conn_put(conn);	/* CM reference */
 
 			goto done;
 		}
@@ -751,9 +751,9 @@
 
 		while (0 < timeout && SDP_SOCK_ST_CONNECT == conn->istate) {
 
-			SDP_CONN_UNLOCK(conn);
+			sdp_conn_unlock(conn);
 			timeout = schedule_timeout(timeout);
-			SDP_CONN_LOCK(conn);
+			sdp_conn_lock(conn);
 
 			if (signal_pending(current)) {
 
@@ -792,7 +792,7 @@
 	case SDP_SOCK_ST_CLOSED:
 	case SDP_SOCK_ST_ERROR:
 
-		result = SDP_CONN_ERROR(conn) ? : -ECONNABORTED;
+		result = sdp_conn_error(conn) ? : -ECONNABORTED;
 		sock->state = SS_UNCONNECTED;
 		break;
 	default:
@@ -805,7 +805,7 @@
 	sdp_dbg_ctrl(conn, "CONNECT complete");
 
 done:
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return result;
 } /* _sdp_inet_connect */
 
@@ -827,7 +827,7 @@
 	sdp_dbg_ctrl(conn, "LISTEN: addr <%08x:%04x> backlog <%04x>",
 		     conn->src_addr, conn->src_port, backlog);
 
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	if (SS_UNCONNECTED != sock->state ||
 	    (SDP_SOCK_ST_CLOSED != conn->istate &&
@@ -870,7 +870,7 @@
 	result = 0;
 
 done:
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return result;
 } /* _sdp_inet_listen */
 
@@ -898,7 +898,7 @@
 	sdp_dbg_ctrl(listen_conn, "ACCEPT: addr <%08x:%04x>",
 		     listen_conn->src_addr, listen_conn->src_port);
 
-	SDP_CONN_LOCK(listen_conn);
+	sdp_conn_lock(listen_conn);
 
 	if (SDP_SOCK_ST_LISTEN != listen_conn->istate) {
 
@@ -926,9 +926,9 @@
 			       SDP_SOCK_ST_LISTEN == listen_conn->istate &&
 			       0 == listen_conn->backlog_cnt) {
 
-				SDP_CONN_UNLOCK(listen_conn);
+				sdp_conn_unlock(listen_conn);
 				timeout = schedule_timeout(timeout);
-				SDP_CONN_LOCK(listen_conn);
+				sdp_conn_lock(listen_conn);
 
 				if (signal_pending(current)) {
 
@@ -1015,9 +1015,9 @@
 				 * trying.
 				 */
 				/* AcceptQueueGet */
-				SDP_CONN_UNLOCK(accept_conn);
+				sdp_conn_unlock(accept_conn);
 				/* INET reference (AcceptQueue ref) */
-				SDP_CONN_PUT(accept_conn);
+				sdp_conn_put(accept_conn);
 
 				accept_sk = NULL;
 				accept_sock->sk = NULL;
@@ -1034,14 +1034,14 @@
 				 * lock
 				 */
 				/* AcceptQueueGet */
-				SDP_CONN_UNLOCK(accept_conn);
+				sdp_conn_unlock(accept_conn);
 			}
 		}
 	}
 
 	result = 0;
 listen_done:
-	SDP_CONN_UNLOCK(listen_conn);
+	sdp_conn_unlock(listen_conn);
 
 	sdp_dbg_ctrl(listen_conn, 
 		     "ACCEPT: complete <%d> <%08x:%04x><%08x:%04x>",
@@ -1287,7 +1287,7 @@
 		 */
 	case SIOCINQ:
 
-		SDP_CONN_LOCK(conn);
+		sdp_conn_lock(conn);
 
 		if (SDP_SOCK_ST_LISTEN != conn->istate) {
 			/*
@@ -1301,11 +1301,11 @@
 			result = -EINVAL;
 		}
 
-		SDP_CONN_UNLOCK(conn);
+		sdp_conn_unlock(conn);
 		break;
 	case SIOCOUTQ:
 
-		SDP_CONN_LOCK(conn);
+		sdp_conn_lock(conn);
 
 		if (SDP_SOCK_ST_LISTEN != conn->istate) {
 
@@ -1317,11 +1317,11 @@
 			result = -EINVAL;
 		}
 
-		SDP_CONN_UNLOCK(conn);
+		sdp_conn_unlock(conn);
 		break;
 	case SIOCATMARK:
 
-		SDP_CONN_LOCK(conn);
+		sdp_conn_lock(conn);
 
 		value = 0;
 
@@ -1338,7 +1338,7 @@
 
 		result = put_user(value, (int __user *) arg);
 
-		SDP_CONN_UNLOCK(conn);
+		sdp_conn_unlock(conn);
 		break;
 	default:
 
@@ -1385,7 +1385,7 @@
 		return -EFAULT;
 	}
 
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	switch (optname) {
 	case TCP_NODELAY:
@@ -1431,7 +1431,7 @@
 		break;
 	}
 
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return result;
 } /* _sdp_inet_setopt */
 
@@ -1473,7 +1473,7 @@
 		return -EINVAL;
 	}
 
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	switch (optname) {
 	case TCP_NODELAY:
@@ -1504,7 +1504,7 @@
 		break;
 	}
 
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 
 	if (put_user(len, optlen)) {
 
@@ -1546,7 +1546,7 @@
 		flag++;		/* match shutdown mask. */
 	}
 
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	conn->shutdown |= flag;
 
@@ -1572,7 +1572,7 @@
 		break;
 	case SDP_SOCK_ST_ERROR:
 
-		result = SDP_CONN_ERROR(conn);
+		result = sdp_conn_error(conn);
 		result = (result < 0) ? result : -ECONNABORTED;
 		break;
 	case SDP_SOCK_ST_ACCEPTED:
@@ -1629,7 +1629,7 @@
 		sdp_inet_wake_error(sock->sk);
 	}
 
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return result;
 } /* _sdp_inet_shutdown */
 
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1836)
+++ sdp_proto.h	(working copy)
@@ -44,7 +44,6 @@
 #include "sdp_advt.h"
 #include "sdp_iocb.h"
 #include "sdp_queue.h"
-
 /*
  * Buffer managment
  */
@@ -294,8 +293,6 @@
 		      u8 hw_port,
 		      u16 pkey);
 
-int sdp_conn_destruct(struct sdp_opt *conn);
-
 void sdp_inet_wake_send(struct sock *sk);
 
 void sdp_inet_wake_generic(struct sock *sk);
@@ -412,17 +409,6 @@
 int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp);
 
 /*
- * internal connection lock functions
- */
-void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags);
-
-void sdp_conn_internal_unlock(struct sdp_opt *conn);
-
-void sdp_conn_internal_relock(struct sdp_opt *conn);
-
-int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn);
-
-/*
  * DATA transport
  */
 int sdp_inet_send(struct kiocb *iocb,
@@ -686,39 +672,4 @@
 	return 0;
 } /* __sdp_conn_state_dump */
 
-/*
- * __sdp_conn_hold - increment reference count
- */
-static inline void __sdp_conn_hold(struct sdp_opt *conn)
-{
-	atomic_inc(&conn->refcnt);
-} /* __sdp_conn_hold */
-
-/*
- * __sdp_conn_put - decrement reference count
- */
-static inline void __sdp_conn_put(struct sdp_opt *conn)
-{
-	if (atomic_dec_and_test(&conn->refcnt)) {
-
-		(void)sdp_conn_destruct(conn);
-	}
-} /* __sdp_conn_hold */
-
-/*
- * __sdp_conn_error - get the connections error value destructively
- */
-static inline int __sdp_conn_error(struct sdp_opt *conn)
-{
-	/*
-	 * The connection error parameter is set and read under the connection
-	 * lock, however the linux socket error, needs to be xchg'd since the
-	 * SO_ERROR getsockopt happens outside of the connection lock.
-	 */
-	int error = xchg(&conn->sk->sk_err, 0);
-	SDP_CONN_SET_ERR(conn, 0);
-
-	return -error;
-} /* __sdp_conn_error */
-
 #endif /* _SDP_PROTO_H */
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1836)
+++ sdp_send.c	(working copy)
@@ -2044,7 +2044,7 @@
 	 * lock the socket while we operate.
 	 */
 	conn = SDP_GET_CONN(si->sock->sk);
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	sdp_dbg_ctrl(conn, "Cancel Write IOCB. <%08x:%04x> <%08x:%04x>",
 		     conn->src_addr, conn->src_port,
@@ -2168,7 +2168,7 @@
 	result = -EAGAIN;
 
 unlock:
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 done:
 
 	aio_put_req(req); /* cancel call reference */
@@ -2315,7 +2315,7 @@
 		     req->ki_key, msg->msg_iov->iov_base,
 		     req->ki_users, req->ki_flags);
 	
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 	/*
 	 * ESTABLISED and CLOSE can send, while CONNECT and ACCEPTED can
 	 * continue being processed, it'll wait below until the send window
@@ -2416,7 +2416,7 @@
 
 		if (0 != SDP_CONN_GET_ERR(conn)) {
 
-			result = (0 < copied) ? 0 : SDP_CONN_ERROR(conn);
+			result = (0 < copied) ? 0 : sdp_conn_error(conn);
 			break;
 		}
 
@@ -2464,12 +2464,12 @@
 			 */
 			clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
-			SDP_CONN_UNLOCK(conn);
+			sdp_conn_unlock(conn);
 			if (!(0 < __sdp_inet_write_space(conn, oob))) {
 
 				timeout = schedule_timeout(timeout);
 			}
-			SDP_CONN_LOCK(conn);
+			sdp_conn_lock(conn);
 
 			remove_wait_queue(sk->sk_sleep, &wait);
 			set_current_state(TASK_RUNNING);
@@ -2526,7 +2526,7 @@
 	}
 
 done:
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	result = ((0 < copied) ? copied : result);
 
 	if (-EPIPE == result && 0 == (MSG_NOSIGNAL & msg->msg_flags)) {
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1836)
+++ sdp_actv.c	(working copy)
@@ -194,7 +194,7 @@
 drop:
 	sdp_inet_wake_error(sk);
 
-	SDP_CONN_PUT(conn);	/* CM sk reference */
+	sdp_conn_put(conn);	/* CM sk reference */
 	return result;
 } /* _sdp_actv_conn_establish */
 
@@ -295,7 +295,7 @@
 			goto done;
 		}
 
-		SDP_CONN_PUT(conn);
+		sdp_conn_put(conn);
 
 		break;
 	case SDP_CONN_ST_REQ_SENT:
@@ -389,7 +389,7 @@
 	/*
 	 * lock the socket
 	 */
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 	/*
 	 * path lookup is complete
 	 */
@@ -570,8 +570,8 @@
 	}
 	/* if */
 done:
-	SDP_CONN_UNLOCK(conn);
-	SDP_CONN_PUT(conn);
+	sdp_conn_unlock(conn);
+	sdp_conn_put(conn);
 
 	return;
 } /* _sdp_cm_path_complete */
@@ -599,8 +599,8 @@
 	/*
 	 * lookup the remote address
 	 */
-	SDP_CONN_HOLD(conn);
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_hold(conn);
+	sdp_conn_unlock(conn);
 					  
 	result = sdp_link_path_lookup(htonl(conn->dst_addr),
 				      htonl(conn->src_addr),
@@ -608,7 +608,7 @@
 				      _sdp_cm_path_complete,
 				      conn,
 				      &conn->plid);
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	if (0 > result) {
 
@@ -619,7 +619,7 @@
 		/*
 		 * callback dosn't have this socket.
 		 */
-		SDP_CONN_PUT(conn);
+		sdp_conn_put(conn);
 
 		result = -EDESTADDRREQ;
 		goto error;
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1836)
+++ sdp_conn.c	(working copy)
@@ -104,7 +104,7 @@
 	/*
 	 * up ref until we release. One ref for GW and one for INET.
 	 */
-	SDP_CONN_HOLD(accept_conn);	/* INET reference */
+	sdp_conn_hold(accept_conn);	/* INET reference */
 
 	return 0;
 } /* sdp_inet_accept_q_put */
@@ -136,7 +136,7 @@
 	 */
 	accept_conn = listen_conn->accept_prev;
 
-	SDP_CONN_LOCK(accept_conn);
+	sdp_conn_lock(accept_conn);
 
 	prev_conn = accept_conn->accept_prev;
 
@@ -172,7 +172,7 @@
 	 * list. The process context lock is used, so the function may
 	 * not be called from the CQ interrupt.
 	 */
-	SDP_CONN_LOCK(accept_conn->parent);
+	sdp_conn_lock(accept_conn->parent);
 
 	next_conn = accept_conn->accept_next;
 	prev_conn = accept_conn->accept_prev;
@@ -182,7 +182,7 @@
 
 	accept_conn->parent->backlog_cnt--;
 
-	SDP_CONN_UNLOCK(accept_conn->parent);
+	sdp_conn_unlock(accept_conn->parent);
 
 	accept_conn->accept_next = NULL;
 	accept_conn->accept_prev = NULL;
@@ -284,9 +284,9 @@
 		}
 
 		/* AcceptQueueGet */
-		SDP_CONN_UNLOCK(accept_conn);
+		sdp_conn_unlock(accept_conn);
 		/* INET reference (AcceptQueuePut). */
-		SDP_CONN_PUT(accept_conn);
+		sdp_conn_put(accept_conn);
 	}
 
 	listen_conn->accept_next = NULL;
@@ -315,7 +315,7 @@
 		if (port == conn->src_port &&
 		    (INADDR_ANY == conn->src_addr || addr == conn->src_addr)) {
 
-			SDP_CONN_HOLD(conn);
+			sdp_conn_hold(conn);
 			break;
 		}
 	}
@@ -661,7 +661,7 @@
 		goto done;
 	}
 
-	SDP_CONN_HOLD(conn);
+	sdp_conn_hold(conn);
 done:
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return conn;
@@ -857,16 +857,19 @@
 } /* sdp_conn_internal_lock */
 
 /*
- * sdp_conn_internal_relock - test the connection (use only from macro)
+ * sdp_conn_relock - test the connection (use only from macro)
  */
-void sdp_conn_internal_relock(struct sdp_opt *conn)
+void sdp_conn_relock(struct sdp_opt *conn)
 {
+	unsigned long flags;
 	struct ib_wc entry;
 	int result_r;
 	int result_s;
 	int result;
 	int rearm = 1;
 
+	spin_lock_irqsave(&conn->lock.slock, flags);
+
 	while (1) {
 
 		result_r = ib_poll_cq(conn->recv_cq, 1, &entry);
@@ -934,8 +937,9 @@
 
 	conn->flags &= ~SDP_CONN_F_MASK_EVENT;
 
+	spin_unlock_irqrestore(&conn->lock.slock, flags);
 	return;
-} /* sdp_conn_internal_relock */
+} /* sdp_conn_relock */
 
 /*
  * sdp_conn_cq_drain - drain one of the the connection's CQs
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1836)
+++ sdp_recv.c	(working copy)
@@ -1070,7 +1070,7 @@
 	 * lock the socket while we operate.
 	 */
 	conn = SDP_GET_CONN(si->sock->sk);
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	sdp_dbg_ctrl(conn, "Cancel Read IOCB. <%08x:%04x> <%08x:%04x>",
 		     conn->src_addr, conn->src_port,
@@ -1178,7 +1178,7 @@
 	result = -EAGAIN;
 	
 unlock:
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 done:
 	aio_put_req(req);
 	return result;
@@ -1364,7 +1364,7 @@
 		msg->msg_flags |= MSG_PEEK;
 	}
 
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 
 	if (SDP_SOCK_ST_LISTEN == conn->istate ||
 	    SDP_SOCK_ST_CLOSED == conn->istate) {
@@ -1548,7 +1548,7 @@
 			/*
 			 * process backlog
 			 */
-			SDP_CONN_RELOCK(conn);
+			sdp_conn_relock(conn);
 
 			if (0 < sdp_buff_q_size(&conn->recv_pool)) {
 
@@ -1589,7 +1589,7 @@
 		 */
 		if (0 != SDP_CONN_GET_ERR(conn)) {
 
-			result = (0 < copied) ? 0 : SDP_CONN_ERROR(conn);
+			result = (0 < copied) ? 0 : sdp_conn_error(conn);
 			break;
 		}
 		
@@ -1625,9 +1625,9 @@
 
 			if (0 == sdp_buff_q_size(&conn->recv_pool)) {
 
-				SDP_CONN_UNLOCK(conn);
+				sdp_conn_unlock(conn);
 				timeout = schedule_timeout(timeout);
-				SDP_CONN_LOCK(conn);
+				sdp_conn_lock(conn);
 			}
 
 			clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
@@ -1725,6 +1725,6 @@
 		}
 	}
 
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return ((0 < copied) ? copied : result);
 } /* sdp_inet_recv */
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1836)
+++ sdp_wall.c	(working copy)
@@ -329,7 +329,7 @@
 
 	sdp_inet_wake_error(conn->sk);
 
-	SDP_CONN_PUT(conn);	/* CM reference */
+	sdp_conn_put(conn);	/* CM reference */
 
 	return 0;
 } /* sdp_wall_recv_reject */
@@ -402,7 +402,7 @@
 
 	sdp_inet_wake_error(conn->sk);
 	
-	SDP_CONN_PUT(conn);	/* CM sk reference */
+	sdp_conn_put(conn);	/* CM sk reference */
 	
 	return result;
 } /* sdp_wall_recv_confirm */
@@ -439,7 +439,7 @@
 		break;
 	}
 
-	SDP_CONN_PUT(conn);	/* CM reference */
+	sdp_conn_put(conn);	/* CM reference */
 
 	return 0;
 } /* sdp_wall_recv_failed */
@@ -651,7 +651,7 @@
 
 	conn->send_buf = 0;
 
-	SDP_CONN_PUT(conn);	/* CM sk reference */
+	sdp_conn_put(conn);	/* CM sk reference */
 
 	return 0;
 } /* sdp_wall_recv_drop */
Index: sdp_conn.h
===================================================================
--- sdp_conn.h	(revision 1836)
+++ sdp_conn.h	(working copy)
@@ -58,8 +58,8 @@
 #define SDP_CONN_F_SEND_CQ_PEND 0x0080 /* send CQ event is pending */
 #define SDP_CONN_F_DEAD         0xFFFF /* connection has been deleted */
 
-#define SDP_CONN_F_MASK_EVENT   (SDP_CONN_F_RECV_CQ_PEND | \
-				    SDP_CONN_F_SEND_CQ_PEND)
+#define SDP_CONN_F_MASK_EVENT   (SDP_CONN_F_RECV_CQ_PEND| \
+				 SDP_CONN_F_SEND_CQ_PEND)
 /*
  * SDP states.
  */
@@ -461,47 +461,80 @@
 /*
  * SDP connection lock
  */
-#define SDP_CONN_LOCK_BH(conn) \
+extern void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags);
+extern void sdp_conn_internal_unlock(struct sdp_opt *conn);
+extern void sdp_conn_relock(struct sdp_opt *conn);
+extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn);
+extern int sdp_conn_destruct(struct sdp_opt *conn);
+
+#define SDP_CONN_LOCK_IRQ(conn, flags) \
         spin_lock_irqsave(&((conn)->lock.slock), flags)
-#define SDP_CONN_UNLOCK_BH(conn) \
+#define SDP_CONN_UNLOCK_IRQ(conn, flags) \
         spin_unlock_irqrestore(&((conn)->lock.slock), flags)
 
-#define SDP_CONN_LOCK(conn)                             \
-do {                                                    \
-  unsigned long flags;                                  \
-  spin_lock_irqsave(&((conn)->lock.slock), flags);      \
-  if ((conn)->lock.users != 0) {                        \
-     sdp_conn_internal_lock(conn, &flags);              \
-  } /* if */                                            \
-  (conn)->lock.users = 1;                               \
-  spin_unlock_irqrestore(&((conn)->lock.slock), flags); \
-} while(0)
+static inline void sdp_conn_lock(struct sdp_opt *conn)
+{
+	unsigned long flags;
 
-#define SDP_CONN_UNLOCK(conn)                           \
-do {                                                    \
-  unsigned long flags;                                  \
-  spin_lock_irqsave(&((conn)->lock.slock), flags);      \
-  if (0 < (SDP_CONN_F_MASK_EVENT & conn->flags) &&   \
-      0 < (SDP_ST_MASK_EVENTS & conn->state)) {      \
-     sdp_conn_internal_unlock(conn);                    \
-  } /* if */                                            \
-  (conn)->lock.users = 0;                               \
-  wake_up(&((conn)->lock.waitq));                       \
-  spin_unlock_irqrestore(&((conn)->lock.slock), flags); \
-} while(0)
+	spin_lock_irqsave(&conn->lock.slock, flags);
+	if (conn->lock.users != 0) {
 
-#define SDP_CONN_RELOCK(conn)                           \
-do {                                                    \
-  unsigned long flags;                                  \
-  spin_lock_irqsave(&((conn)->lock.slock), flags);      \
-     sdp_conn_internal_relock(conn);                    \
-  spin_unlock_irqrestore(&((conn)->lock.slock), flags); \
-} while(0)
+		sdp_conn_internal_lock(conn, &flags);
+	}
 
-#define SDP_CONN_HOLD(conn) __sdp_conn_hold((conn))
-#define SDP_CONN_PUT(conn)  __sdp_conn_put((conn))
-#define SDP_CONN_ERROR(conn)  __sdp_conn_error((conn))
+	conn->lock.users = 1;
+	spin_unlock_irqrestore(&(conn->lock.slock), flags);
+}
 
+static inline void sdp_conn_unlock(struct sdp_opt *conn)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&conn->lock.slock, flags);
+	if (0 < (SDP_CONN_F_MASK_EVENT & conn->flags) &&
+	    0 < (SDP_ST_MASK_EVENTS & conn->state)) {
+
+		sdp_conn_internal_unlock(conn);
+	}
+
+	conn->lock.users = 0;
+	wake_up(&conn->lock.waitq);
+
+	spin_unlock_irqrestore(&conn->lock.slock, flags);
+}
+
+/*
+ * connection reference counting.
+ */
+static inline void sdp_conn_hold(struct sdp_opt *conn)
+{
+	atomic_inc(&conn->refcnt);
+}
+
+static inline void sdp_conn_put(struct sdp_opt *conn)
+{
+	if (atomic_dec_and_test(&conn->refcnt)) {
+
+		(void)sdp_conn_destruct(conn);
+	}
+}
+
+/*
+ * sdp_conn_error - get the connections error value destructively
+ */
+static inline int sdp_conn_error(struct sdp_opt *conn)
+{
+	/*
+	 * The connection error parameter is set and read under the connection
+	 * lock, however the linux socket error, needs to be xchg'd since the
+	 * SO_ERROR getsockopt happens outside of the connection lock.
+	 */
+	int error = xchg(&conn->sk->sk_err, 0);
+	SDP_CONN_SET_ERR(conn, 0);
+
+	return -error;
+}
+
 static inline void *__hashent_arg(s32 hashent) 
 {
 	return (void *)(unsigned long)hashent;
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1836)
+++ sdp_pass.c	(working copy)
@@ -218,7 +218,7 @@
 		goto lookup_err;
 	}
 
-	SDP_CONN_LOCK(listen_conn);
+	sdp_conn_lock(listen_conn);
 	/*
 	 * check backlog
 	 */
@@ -304,8 +304,8 @@
 
 	result = 0;
 locked_err:
-	SDP_CONN_UNLOCK(listen_conn);
-	SDP_CONN_PUT(listen_conn);	/* ListenLookup reference. */
+	sdp_conn_unlock(listen_conn);
+	sdp_conn_put(listen_conn);	/* ListenLookup reference. */
 lookup_err:
 	return result;
 } /* _sdp_cm_listen_lookup */
@@ -421,7 +421,7 @@
 	/*
 	 * Lock the new connection before modifyingg it into any tables.
 	 */
-	SDP_CONN_LOCK(conn);
+	sdp_conn_lock(conn);
 	/*
 	 * save hello parameters.
 	 */
@@ -488,12 +488,12 @@
 	/*
 	 * unlock
 	 */
-	SDP_CONN_UNLOCK(conn);
+	sdp_conn_unlock(conn);
 	return 0;
 error:
 	conn->cm_id = NULL; /* cm_id destroyed by CM on error result. */
-	SDP_CONN_UNLOCK(conn);
-	SDP_CONN_PUT(conn);	/* CM sk reference, will destory */
+	sdp_conn_unlock(conn);
+	sdp_conn_put(conn);	/* CM sk reference, will destory */
 done:
 	return result;
 } /* sdp_cm_req_handler */
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1836)
+++ sdp_event.c	(working copy)
@@ -162,7 +162,7 @@
 	 * lock the bottom half of the socket. If the connection is in use,
 	 * then queue the event, otherwise process this event.
 	 */
-	SDP_CONN_LOCK_BH(conn);
+	SDP_CONN_LOCK_IRQ(conn, flags);
 	/*
 	 * Check for event completions before CM has transitioned to
 	 * the established state. The CQ will not be polled or rearmed
@@ -206,8 +206,8 @@
 	}
 
 unlock:
-	SDP_CONN_UNLOCK_BH(conn);
-	SDP_CONN_PUT(conn);
+	SDP_CONN_UNLOCK_IRQ(conn, flags);
+	sdp_conn_put(conn);
 done:
 	return;
 } /* sdp_cq_event_handler */
@@ -553,7 +553,7 @@
 	conn = sdp_conn_table_lookup(hashent);
 	if (NULL != conn) {
 
-		SDP_CONN_LOCK(conn);
+		sdp_conn_lock(conn);
 	}
 	else {
 
@@ -608,8 +608,8 @@
 			(void)__sdp_conn_state_dump(conn);
 		}
 
-		SDP_CONN_UNLOCK(conn);
-		SDP_CONN_PUT(conn);
+		sdp_conn_unlock(conn);
+		sdp_conn_put(conn);
 	}
 
 	return result;
Index: sdp_post.c
===================================================================
--- sdp_post.c	(revision 1836)
+++ sdp_post.c	(working copy)
@@ -124,7 +124,7 @@
 		sdp_dbg_warn(NULL, "Error <%d> CM disconnect request", result);
 	}
 
-	SDP_CONN_PUT(conn);
+	sdp_conn_put(conn);
 
 	return;
 } /* _sdp_cm_disconnect */
@@ -154,7 +154,7 @@
 		sdp_dbg_warn(NULL, "Error <%d> CM reject request", result);
 	}
 
-	SDP_CONN_PUT(conn);
+	sdp_conn_put(conn);
 
 	return;
 } /* _sdp_cm_reject */
@@ -182,7 +182,7 @@
 		sdp_dbg_warn(NULL, "Error <%d> CM confirm request", result);
 	}
 
-	SDP_CONN_PUT(conn);
+	sdp_conn_put(conn);
 
 	return;
 } /* _sdp_cm_confirm */
@@ -213,7 +213,7 @@
 		sdp_dbg_warn(NULL, "Error <%d> CM failed request", result);
 	}
 
-	SDP_CONN_PUT(conn);
+	sdp_conn_put(conn);
 
 	return;
 } /* _sdp_cm_failed */
@@ -228,7 +228,7 @@
 	/*
 	 * send a potentially defered failed request.
 	 */
-	SDP_CONN_HOLD(conn);
+	sdp_conn_hold(conn);
 
 	if (in_atomic() || irqs_disabled()) {
 



From mshefty at ichips.intel.com  Fri Feb 18 16:15:04 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 18 Feb 2005 16:15:04 -0800
Subject: [openib-general] Re: [PATCH] Re: SDP_CONN_LOCK
In-Reply-To: <20050218155144.D15428@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050218155144.D15428@topspin.com>
Message-ID: <42168508.7050600@ichips.intel.com>

Libor Michalek wrote:
>   Attached is a patch that replaces some of the connection macros with
> inline function. The two internal functions you mention were not removed.
> When the lock has no contention, which is the common case, those functions
> are never called which is why the code was split into a macro and funciton
> in the first place. The BH suffixes were replaced w/ IRQ.

Thanks.  See comments below...

> +#define SDP_CONN_LOCK_IRQ(conn, flags) \
>          spin_lock_irqsave(&((conn)->lock.slock), flags)

Can we just remove this macro?

> +#define SDP_CONN_UNLOCK_IRQ(conn, flags) \
>          spin_unlock_irqrestore(&((conn)->lock.slock), flags)

And this one?

> +static inline void sdp_conn_hold(struct sdp_opt *conn)
> +{
> +	atomic_inc(&conn->refcnt);
> +}

ditto...

> +static inline void sdp_conn_put(struct sdp_opt *conn)
> +{
> +	if (atomic_dec_and_test(&conn->refcnt)) {
> +
> +		(void)sdp_conn_destruct(conn);
> +	}
> +}

This one is a little more debatable, but for 2 lines, versus 1, I'd 
remove it.

- Sean


From Tom.Duffy at Sun.COM  Fri Feb 18 16:44:22 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:44:22 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
Message-ID: <11087738623823@sun.com>

This series of patches cleans up the whitespace in SDP.  Mostly it removes
the extra line that was after every conditional, but it also removes
unneeded brackets around single line conditionals as well.

Roland should be happy about this one, even though it is mostly cheating.

 sdp_actv.c  |   47 --------
 sdp_advt.c  |   19 ---
 sdp_buff.c  |  169 +++++++------------------------
 sdp_conn.c  |  277 +++++++++------------------------------------------
 sdp_conn.h  |    7 -
 sdp_event.c |   79 ++------------
 sdp_inet.c  |  308 +++++++++-----------------------------------------------
 sdp_iocb.c  |  202 ++++++++-----------------------------
 sdp_kvec.c  |    3 
 sdp_link.c  |  126 ++++-------------------
 sdp_pass.c  |   21 ---
 sdp_post.c  |   62 ++---------
 sdp_proc.c  |   10 -
 sdp_proto.h |   30 +----
 sdp_queue.c |   75 +++----------
 sdp_rcvd.c  |  177 ++++----------------------------
 sdp_read.c  |   41 +------
 sdp_recv.c  |  217 ++++++----------------------------------
 sdp_send.c  |  325 +++++++++---------------------------------------------------
 sdp_sent.c  |   31 -----
 sdp_wall.c  |   68 ++----------
 sdp_write.c |    9 -
 22 files changed, 420 insertions(+), 1883 deletions(-)

Signed-off-by: Tom Duffy <tduffy at sun.com>



From Tom.Duffy at Sun.COM  Fri Feb 18 16:44:32 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:44:32 -0800
Subject: [openib-general] [PATCH][SDP][1/22] Whitespace cleanup in sdp_actv.c
In-Reply-To: <11087738623823@sun.com>
References: <11087738623823@sun.com>
Message-ID: <11087738723775@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_actv.c	(working copy)
@@ -61,10 +61,8 @@ static int _sdp_actv_conn_establish(stru
 	 * which it should have been.
 	 */
 	if (SDP_SOCK_ST_CONNECT == conn->istate) {
-
 		qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
 		if (!qp_attr) {
-
 			result = -ENOMEM;
 			goto error;
 		}
@@ -75,7 +73,6 @@ static int _sdp_actv_conn_establish(stru
 
 		result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
 		if (result) {
-
 			sdp_dbg_warn(conn, "Error <%d> QP attributes for RTR",
 				     result);
 			goto error;
@@ -88,7 +85,6 @@ static int _sdp_actv_conn_establish(stru
 
 		result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
 		if (result) {
-			
 			sdp_dbg_warn(conn, "Error <%d> QP modify to RTR",
 				     result);
 			goto error;
@@ -102,7 +98,6 @@ static int _sdp_actv_conn_establish(stru
 		 */
 		result = sdp_recv_flush(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> posting recv buffers.",
 				     result);
 			goto error;
@@ -114,7 +109,6 @@ static int _sdp_actv_conn_establish(stru
 
 		result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
 		if (result) {
-
 			sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS",
 				     result);
 			goto error;
@@ -122,7 +116,6 @@ static int _sdp_actv_conn_establish(stru
 
 		result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
 		if (result) {
-			
 			sdp_dbg_warn(conn, "Error <%d> QP modify to RTS",
 				     result);
 			goto error;
@@ -132,7 +125,6 @@ static int _sdp_actv_conn_establish(stru
 		 */
 		result = sdp_cm_confirm(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> CM connect confirm",
 				     result);
 			goto error;
@@ -155,7 +147,6 @@ static int _sdp_actv_conn_establish(stru
 
 		result = sdp_send_flush(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
 				     result);
 			goto error;
@@ -167,17 +158,14 @@ static int _sdp_actv_conn_establish(stru
 		sdp_inet_wake_recv(sk, 0);
 
 		kfree(qp_attr);
-	}
-	else {
+	} else {
 		/*
 		 * fail this connection
 		 */
 		result = sdp_cm_failed(conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> CM connect failed", 
 				     result);
-		}
 
 		SDP_CONN_SET_ERR(conn, EPROTO);
 		conn->istate = SDP_SOCK_ST_ERROR;
@@ -212,7 +200,6 @@ static int _sdp_cm_hello_ack_check(struc
 	 * validation and consistency checks
 	 */
 	if (hello_ack->bsdh.size != sizeof(struct msg_hello_ack)) {
-
 		sdp_dbg_warn(NULL, "hello ack, size mismatch. (2) <%d:%Zu>",
 			     hello_ack->bsdh.size,
 			     sizeof(struct msg_hello_ack));
@@ -220,21 +207,18 @@ static int _sdp_cm_hello_ack_check(struc
 	}
 
 	if (SDP_MID_HELLO_ACK != hello_ack->bsdh.mid) {
-
 		sdp_dbg_warn(NULL, "hello ack, unexpected message. <%d>",
 			     hello_ack->bsdh.mid);
 		return -EINVAL;
 	}
 
 	if (!(0 < hello_ack->hah.max_adv)) {
-
 		sdp_dbg_warn(NULL, "hello ack, bad zcopy advertisment. <%d>",
 			     hello_ack->hah.max_adv);
 		return -EINVAL;
 	}
 
 	if ((0xF0 & hello_ack->hah.version) != (0xF0 & SDP_MSG_VERSION)) {
-
 		sdp_dbg_warn(NULL, "hello ack, version mismatch. <%d:%d>",
 			     ((0xF0 & hello_ack->hah.version) >> 4),
 			     ((0xF0 & SDP_MSG_VERSION) >> 4));
@@ -267,15 +251,12 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	int result;
 	int error;
 
-	if (NULL == conn) {
+	if (NULL == conn)
 		return -EINVAL;
-	}
 	
-	if (cm_id != conn->cm_id) {
-
+	if (cm_id != conn->cm_id)
 		sdp_dbg_warn(conn, "REP comm ID mismatch. <%08x:%08x>",
 			     conn->cm_id->local_id, cm_id->local_id);
-	}
 
 	hello_ack = (struct msg_hello_ack *)event->private_data;
 
@@ -285,10 +266,8 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	 */
 	switch (conn->state) {
 	case SDP_CONN_ST_ERROR_STRM:
-
 		result = sdp_cm_reject(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> CM reject request",
 				     result);
 			error = result;
@@ -305,7 +284,6 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 		 */
 		result = _sdp_cm_hello_ack_check(hello_ack);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> hello ack check.",
 				     result);
 			error = result;
@@ -338,7 +316,6 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 
 		result = _sdp_actv_conn_establish(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> accept receive failed",
 				     result);
 			error = result;
@@ -347,7 +324,6 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "REP received in unknown connection state");
 		/*
 		 * drop CM reference
@@ -394,7 +370,6 @@ static void _sdp_cm_path_complete(u64 id
 	 * path lookup is complete
 	 */
 	if (id != conn->plid) {
-
 		sdp_dbg_warn(conn, "Path record ID mismatch <%016llx:%016llx>",
 			     (unsigned long long)id, 
 			     (unsigned long long)conn->plid);
@@ -402,7 +377,6 @@ static void _sdp_cm_path_complete(u64 id
 	}
 
 	if (SDP_CONN_ST_REQ_PATH != conn->state) {
-
 		sdp_dbg_warn(conn, "Path record completion in bad state");
 		goto done;
 	}
@@ -417,7 +391,6 @@ static void _sdp_cm_path_complete(u64 id
 	 * create address handle
 	 */
 	if (0 != status) {
-
 		sdp_dbg_warn(conn, "Path record completion error <%d>",
 			     status);
 		goto failed;
@@ -434,7 +407,6 @@ static void _sdp_cm_path_complete(u64 id
 	 */
 	result = sdp_conn_alloc_ib(conn, ca, hw_port, path->pkey);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> allocating IB connection",
 			     result);
 		goto failed;
@@ -445,7 +417,6 @@ static void _sdp_cm_path_complete(u64 id
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buff for Hello Msg.");
 		goto failed;
 	}
@@ -488,7 +459,6 @@ static void _sdp_cm_path_complete(u64 id
 	 */
 	result = sdp_buff_q_put(&conn->send_post, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> buffering hello msg.", result);
 
 		expect = sdp_buff_pool_put(buff);
@@ -500,10 +470,8 @@ static void _sdp_cm_path_complete(u64 id
 	/*
 	 * Mellanox performance bug workaround.
 	 */
-	if (IB_MTU_1024 < path->mtu) {
-
+	if (IB_MTU_1024 < path->mtu)
 		path->mtu = IB_MTU_1024;
-	}
 #endif
 	conn->path_mtu = path->mtu;
 	/*
@@ -537,7 +505,6 @@ static void _sdp_cm_path_complete(u64 id
 	conn->cm_id = ib_create_cm_id(sdp_cm_event_handler,
 				      __hashent_arg(conn->hashent));
 	if (NULL == conn->cm_id) {
-
 		sdp_dbg_warn(conn, "Failed to create CM handle, %d",
 			     (u8)(buff->tail - buff->data));
 
@@ -551,7 +518,6 @@ static void _sdp_cm_path_complete(u64 id
 	 */
 	result = ib_send_cm_req(conn->cm_id, &param);
 	if (0 != result) {
-
 		sdp_dbg_warn(conn, "Error <%d> CM connect request", result);
 		goto failed;
 	}
@@ -562,7 +528,6 @@ failed:
 
 	result = sdp_wall_recv_reject(conn, (0 - status));
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> rejecting connection", result);
 		
 		expect = sdp_wall_recv_drop(conn);
@@ -588,7 +553,6 @@ int sdp_cm_connect(struct sdp_opt *conn)
 	 * get the buffer size we'll use for this connection. (and all others)
 	 */
 	if (sizeof(struct msg_hello) > conn->recv_size) {
-
 		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct msg_hello));
 		result = -ENOBUFS;
@@ -611,7 +575,6 @@ int sdp_cm_connect(struct sdp_opt *conn)
 	SDP_CONN_LOCK(conn);
 
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> getting link <%08x:%08x> addr",
 			     result,
 			     htonl(conn->dst_addr),



From Tom.Duffy at Sun.COM  Fri Feb 18 16:44:43 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:44:43 -0800
Subject: [openib-general] [PATCH][SDP][2/22] Whitespace cleanup in sdp_advt.c
In-Reply-To: <11087738723775@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
Message-ID: <11087738832487@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_advt.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_advt.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_advt.c	(working copy)
@@ -53,7 +53,6 @@ struct sdpc_advt *sdp_advt_create(void)
 
 	advt = kmem_cache_alloc(__sdp_advt_cache, SLAB_KERNEL);
 	if (NULL != advt) {
-
 		advt->next = NULL;
 		advt->prev = NULL;
 		advt->size = 0;
@@ -75,10 +74,8 @@ int sdp_advt_destroy(struct sdpc_advt *a
 {
 	SDP_CHECK_NULL(advt, -EINVAL);
 
-	if (NULL != advt->next || NULL != advt->prev) {
-
+	if (NULL != advt->next || NULL != advt->prev)
 		return -EACCES;
-	}
 	/*
 	 * return the object to its cache
 	 */
@@ -99,17 +96,12 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 	SDP_CHECK_NULL(table, NULL);
 
 	advt = table->head;
-	if (NULL == advt) {
-
+	if (NULL == advt)
 		return NULL;
-	}
-
-	if (advt->next == advt && advt->prev == advt) {
 
+	if (advt->next == advt && advt->prev == advt)
 		table->head = NULL;
-	}
 	else {
-
 		next = advt->next;
 		prev = advt->prev;
 		next->prev = prev;
@@ -148,13 +140,11 @@ int sdp_advt_q_put(struct sdpc_advt_q *t
 	SDP_CHECK_NULL(advt, -EINVAL);
 
 	if (NULL == table->head) {
-
 		advt->next = advt;
 		advt->prev = advt;
 		table->head = advt;
 	}
 	else {
-
 		next = table->head;
 		prev = next->prev;
 
@@ -195,7 +185,6 @@ int sdp_advt_q_clear(struct sdpc_advt_q 
 	 * drain the table of any objects
 	 */
 	while (NULL != (advt = sdp_advt_q_get(table))) {
-
 		result = sdp_advt_destroy(advt);
 		SDP_EXPECT(!(0 > result));
 	}
@@ -219,7 +208,6 @@ int sdp_main_advt_init(void)
 	 * initialize the caches only once.
 	 */
 	if (NULL != __sdp_advt_cache) {
-
 		sdp_warn("Advertisment caches already initialized.");
 		return -EINVAL;
 	}
@@ -229,7 +217,6 @@ int sdp_main_advt_init(void)
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
 	if (NULL == __sdp_advt_cache) {
-
 		result = -ENOMEM;
 		goto error_advt_c;
 	}



From Tom.Duffy at Sun.COM  Fri Feb 18 16:44:53 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:44:53 -0800
Subject: [openib-general] [PATCH][SDP][3/22] Whitespace cleanup in sdp_buff.c
In-Reply-To: <11087738832487@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com>
Message-ID: <1108773893431@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_buff.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_buff.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_buff.c	(working copy)
@@ -42,41 +42,29 @@ static struct sdpc_buff_root *main_pool 
 /*
  * _sdp_buff_q_get - Get a buffer from a specific pool
  */
-static __inline__ struct sdpc_buff *_sdp_buff_q_get
-(
-	struct sdpc_buff_q *pool,
-	int fifo,
-	int (*test_func)(struct sdpc_buff *buff,
-			 void *arg),
-	void *usr_arg
-)
+static __inline__ struct sdpc_buff *_sdp_buff_q_get(struct sdpc_buff_q *pool,
+						    int fifo,
+						    int (*test_func)
+							(struct sdpc_buff *buff,
+							 void *arg),
+						    void *usr_arg)
 {
 	struct sdpc_buff *buff;
 
 	SDP_CHECK_NULL(pool, NULL);
 
-	if (NULL == pool->head) {
-
+	if (NULL == pool->head)
 		return NULL;
-	}
-
-	if (fifo) {
 
+	if (fifo)
 		buff = pool->head;
-	}
-	else {
-
+	else
 		buff = pool->head->prev;
-	}
 
 	if (NULL == test_func || 0 == test_func(buff, usr_arg)) {
-
-		if (buff->next == buff && buff->prev == buff) {
-
+		if (buff->next == buff && buff->prev == buff)
 			pool->head = NULL;
-		}
 		else {
-
 			buff->next->prev = buff->prev;
 			buff->prev->next = buff->next;
 
@@ -89,10 +77,8 @@ static __inline__ struct sdpc_buff *_sdp
 		buff->prev = NULL;
 		buff->pool = NULL;
 	}
-	else {
-
+	else
 		buff = NULL;
-	}
 
 	return buff;
 } /* _sdp_buff_q_get */
@@ -108,28 +94,22 @@ static __inline__ int _sdp_buff_q_put(st
 	SDP_CHECK_NULL(pool, -EINVAL);
 	SDP_CHECK_NULL(buff, -EINVAL);
 
-	if (NULL != buff->pool) {
-
+	if (NULL != buff->pool)
 		return -EINVAL;
-	}
 
 	if (NULL == pool->head) {
-
 		buff->next = buff;
 		buff->prev = buff;
 		pool->head = buff;
-	}
-	else {
-
+	} else {
 		buff->next = pool->head;
 		buff->prev = pool->head->prev;
 
 		buff->next->prev = buff;
 		buff->prev->next = buff;
 
-		if (fifo) {
+		if (fifo)
 			pool->head = buff;
-		}
 	}
 
 	pool->size++;
@@ -146,14 +126,10 @@ static __inline__ struct sdpc_buff *_sdp
 {
 	SDP_CHECK_NULL(pool, NULL);
 
-	if (NULL == pool->head || fifo) {
-
+	if (NULL == pool->head || fifo)
 		return pool->head;
-	}
-	else {
-
+	else
 		return pool->head->prev;
-	}
 } /* _sdp_buff_q_look */
 
 /*
@@ -171,26 +147,19 @@ static __inline__ int _sdp_buff_q_remove
 	SDP_CHECK_NULL(buff->prev, -EINVAL);
 	SDP_CHECK_NULL(buff->pool, -EINVAL);
 
-	if (pool != buff->pool) {
-
+	if (pool != buff->pool)
 		return -EINVAL;
-	}
-
-	if (buff->next == buff && buff->prev == buff) {
 
+	if (buff->next == buff && buff->prev == buff)
 		pool->head = NULL;
-	}
 	else {
-
 		next = buff->next;
 		prev = buff->prev;
 		next->prev = prev;
 		prev->next = next;
 
-		if (pool->head == buff) {
-
+		if (pool->head == buff)
 			pool->head = next;
-		}
 	}
 
 	pool->size--;
@@ -217,10 +186,8 @@ int sdp_buff_q_init(struct sdpc_buff_q *
 	pool->size = 0;
 
 	for (counter = 0; counter < size; counter++) {
-
 		result = sdp_buff_q_put(pool, sdp_buff_pool_get());
 		if (0 > result) {
-
 			result = -ENOMEM;
 			goto error;
 		}
@@ -329,28 +296,23 @@ struct sdpc_buff *sdp_buff_q_fetch(struc
 	/*
 	 * check to see if there is anything to traverse.
 	 */
-	if (NULL != pool->head) {
+	if (NULL != pool->head)
 		/*
 		 * lock to prevent corruption of table
 		 */
 		for (counter = 0, buff = pool->head;
 		     counter < pool->size; counter++, buff = buff->next) {
-
 			result = test(buff, usr_arg);
 			if (0 < result) {
-
 				result = _sdp_buff_q_remove(pool, buff);
 				SDP_EXPECT(!(0 > result));
 
 				return buff;
 			}
 
-			if (0 > result) {
-
+			if (0 > result)
 				break;
-			}
 		}
-	}
 
 	return NULL;
 } /* sdp_buff_q_fetch_head */
@@ -372,7 +334,7 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 	/*
 	 * check to see if there is anything to traverse.
 	 */
-	if (NULL != pool->head) {
+	if (NULL != pool->head)
 		/*
 		 * lock to prevent corruption of table
 		 */
@@ -380,12 +342,9 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 		     counter < pool->size; counter++, buff = buff->next) {
 
 			result = trav_func(buff, usr_arg);
-			if (0 > result) {
-
+			if (0 > result)
 				break;
-			}
 		}
-	}
 
 	return result;
 } /* sdp_buff_q_trav_head */
@@ -442,20 +401,14 @@ int sdp_buff_q_clear_unmap(struct sdpc_b
 	SDP_CHECK_NULL(pool, -EINVAL);
 
 	while (NULL != (buff = _sdp_buff_q_get(pool, 0, NULL, NULL))) {
-
-		if (dev) {
-			dma_unmap_single(dev, 
-					 buff->real,
-					 buff->tail - buff->data,
-					 direction);
-		}
+		if (dev)
+			dma_unmap_single(dev, buff->real,
+					 buff->tail - buff->data, direction);
 
 		result = sdp_buff_pool_put(buff);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_err("Error <%d> returning buffer to main",
 				    result);
-		}
 	}
 	
 	return 0;
@@ -477,12 +430,9 @@ static int _sdp_buff_pool_release(struct
 	 * Release count buffers.
 	 */
 	while (count--) {
-		
 		buff = sdp_buff_q_get(&m_pool->pool);
-		if (NULL == buff) {
-			
+		if (NULL == buff)
 			break;
-		}
 		/*
 		 * decrement global buffer count, free buffer page, and free
 		 * buffer descriptor.
@@ -520,10 +470,8 @@ static __inline__ int _sdp_buff_pool_rel
 
 		return _sdp_buff_pool_release(m_pool, count);
 	}
-	else {
-		
+	else
 		return 0;
-	}
 } /* _sdp_buff_pool_release_check */
 
 /*
@@ -549,7 +497,6 @@ static int _sdp_buff_pool_alloc(struct s
 		 */
 		buff = kmem_cache_alloc(m_pool->buff_cache, GFP_ATOMIC);
 		if (NULL == buff) {
-
 			sdp_warn("Failed to allocate buffer. <%d:%d>",
 				 total, m_pool->buff_cur);
 			break;
@@ -557,7 +504,6 @@ static int _sdp_buff_pool_alloc(struct s
 
 		buff->head = (void *)__get_free_page(GFP_ATOMIC);
 		if (NULL == buff->head) {
-
 			sdp_warn("Failed to allocate buffer page. <%d:%d>",
 				 total, m_pool->buff_cur);
 
@@ -577,7 +523,6 @@ static int _sdp_buff_pool_alloc(struct s
 
 		result = sdp_buff_q_put(&m_pool->pool, buff);
 		if (0 > result) {
-
 			sdp_warn("Failed to queue buffer. <%d>", result);
 			
 			free_page((unsigned long)buff->head);
@@ -589,7 +534,6 @@ static int _sdp_buff_pool_alloc(struct s
 	}
 
 	if (NULL == main_pool->pool.head) {
-
 		sdp_warn("Failed to allocate any buffers. <%d:%d:%d>",
 			 total, m_pool->buff_cur, m_pool->alloc_inc);
 
@@ -610,7 +554,6 @@ int sdp_buff_pool_init(int buff_min, 
 	int result;
 
 	if (NULL != main_pool) {
-
 		sdp_warn("Main pool already initialized!");
 		return -EEXIST;
 	}
@@ -629,7 +572,6 @@ int sdp_buff_pool_init(int buff_min, 
 	 */
 	main_pool = kmalloc(sizeof(struct sdpc_buff_root), GFP_KERNEL);
 	if (NULL == main_pool) {
-
 		sdp_warn("Main pool initialization failed.");
 		result = -ENOMEM;
 		goto done;
@@ -653,7 +595,6 @@ int sdp_buff_pool_init(int buff_min, 
 						  0, SLAB_HWCACHE_ALIGN,
 						  NULL, NULL);
 	if (NULL == main_pool->pool_cache) {
-
 		sdp_warn("Failed to allocate pool cache.");
 		result = -ENOMEM;
 		goto error_pool;
@@ -664,17 +605,15 @@ int sdp_buff_pool_init(int buff_min, 
 						  0, SLAB_HWCACHE_ALIGN,
 						  NULL, NULL);
 	if (NULL == main_pool->buff_cache) {
-
 		sdp_warn("Failed to allocate buffer cache.");
 		result = -ENOMEM;
 		goto error_buff;
-	} /* if */
+	}
 	/*
 	 * allocate the minimum number of buffers.
 	 */
 	result = _sdp_buff_pool_alloc(main_pool);
 	if (0 > result) {
-
 		sdp_warn("Error <%d> allocating buffers. <%d>",
 			 result, buff_min);
 		goto error_alloc;
@@ -703,7 +642,6 @@ done:
 void sdp_buff_pool_destroy(void)
 {
 	if (NULL == main_pool) {
-
 		sdp_warn("Main pool dosn't exist.");
 		return;
 	}
@@ -714,11 +652,9 @@ void sdp_buff_pool_destroy(void)
 	/*
 	 * Sanity check that the current number of buffers was released.
 	 */
-	if (main_pool->buff_cur) {
-		
+	if (main_pool->buff_cur)
 		sdp_warn("Leaking buffers during cleanup. <%d>", 
 			 main_pool->buff_cur);
-	}
 	/*
 	 * free pool cache
 	 */
@@ -753,10 +689,8 @@ struct sdpc_buff *sdp_buff_pool_get(void
 	spin_lock_irqsave(&main_pool->lock, flags);
 
 	if (NULL == main_pool->pool.head) {
-
 		result = _sdp_buff_pool_alloc(main_pool);
 		if (0 > result) {
-
 			sdp_warn("Error <%d> allocating buffers.", result);
 			spin_unlock_irqrestore(&main_pool->lock, flags);
 			return NULL;
@@ -765,12 +699,9 @@ struct sdpc_buff *sdp_buff_pool_get(void
 
 	buff = main_pool->pool.head;
 
-	if (buff->next == buff) {
-
+	if (buff->next == buff)
 		main_pool->pool.head = NULL;
-	}
 	else {
-
 		buff->next->prev = buff->prev;
 		buff->prev->next = buff->next;
 
@@ -806,15 +737,11 @@ int sdp_buff_pool_put(struct sdpc_buff *
 
 	SDP_CHECK_NULL(main_pool, -EINVAL);
 
-	if (NULL == buff || NULL != buff->pool) {
-
+	if (NULL == buff || NULL != buff->pool)
 		return -EINVAL;
-	}
-
-	if (NULL != buff->next || NULL != buff->prev) {
 
+	if (NULL != buff->next || NULL != buff->prev)
 		return -ETOOMANYREFS;
-	}
 	/*
 	 * reset pointers
 	 */
@@ -825,13 +752,10 @@ int sdp_buff_pool_put(struct sdpc_buff *
 	spin_lock_irqsave(&main_pool->lock, flags);
 
 	if (NULL == main_pool->pool.head) {
-
 		buff->next = buff;
 		buff->prev = buff;
 		main_pool->pool.head = buff;
-	}
-	else {
-
+	} else {
 		buff->next = main_pool->pool.head;
 		buff->prev = main_pool->pool.head->prev;
 
@@ -861,12 +785,9 @@ int sdp_buff_pool_chain_link(struct sdpc
 	buff->pool = &main_pool->pool;
 
 	if (NULL == head) {
-
 		buff->next = buff;
 		buff->prev = buff;
-	}
-	else {
-
+	} else {
 		buff->next = head;
 		buff->prev = head->prev;
 
@@ -893,19 +814,14 @@ int sdp_buff_pool_chain_put(struct sdpc_
 	 */
 	SDP_CHECK_NULL(main_pool, -EINVAL);
 
-	if (NULL == buff || !(0 < count)) {
-
+	if (NULL == buff || !(0 < count))
 		return -EINVAL;
-	}
 
 	spin_lock_irqsave(&main_pool->lock, flags);
 
-	if (NULL == main_pool->pool.head) {
-
+	if (NULL == main_pool->pool.head)
 		main_pool->pool.head = buff;
-	}
 	else {
-
 		prev = buff->prev;
 		next = main_pool->pool.head->next;
 
@@ -932,14 +848,10 @@ int sdp_buff_pool_buff_size(void)
 {
 	int result;
 
-	if (NULL == main_pool) {
-
+	if (NULL == main_pool)
 		result = -1;
-	}
-	else {
-
+	else
 		result = main_pool->buff_size;
-	}
 
 	return result;
 } /* sdp_buff_pool_buff_size */
@@ -966,7 +878,6 @@ int sdp_proc_dump_buff_pool(char *buffer
 	spin_lock_irqsave(&main_pool->lock, flags);
 
 	if (0 == start_index) {
-
 		offset += sprintf((buffer + offset), 
 				  "  buffer size:         %8d\n",
 				  main_pool->buff_size);



From Tom.Duffy at Sun.COM  Fri Feb 18 16:45:03 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:45:03 -0800
Subject: [openib-general] [PATCH][SDP][4/22] Whitespace cleanup in sdp_conn.c
In-Reply-To: <1108773893431@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
Message-ID: <11087739032962@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -87,10 +87,8 @@ int sdp_inet_accept_q_put(struct sdp_opt
 	if (NULL != listen_conn->parent ||
 	    NULL != accept_conn->parent ||
 	    NULL == listen_conn->accept_next ||
-	    NULL == listen_conn->accept_prev) {
-
+	    NULL == listen_conn->accept_prev)
 		return -EFAULT;
-	}
 
 	next_conn = listen_conn->accept_next;
 
@@ -123,10 +121,8 @@ struct sdp_opt *sdp_inet_accept_q_get(st
 	    NULL == listen_conn->accept_next ||
 	    NULL == listen_conn->accept_prev ||
 	    listen_conn == listen_conn->accept_next ||
-	    listen_conn == listen_conn->accept_prev) {
-
+	    listen_conn == listen_conn->accept_prev)
 		return NULL;
-	}
 	/*
 	 * Return the next connection in the listening sockets accept
 	 * queue. The new connections lock is acquired, the caller must
@@ -162,10 +158,8 @@ int sdp_inet_accept_q_remove(struct sdp_
 
 	SDP_CHECK_NULL(accept_conn, -EINVAL);
 
-	if (NULL == accept_conn->parent) {
-
+	if (NULL == accept_conn->parent)
 		return -EFAULT;
-	}
 	/*
 	 * Removes the connection from the listening sockets accept queue.
 	 * The listning connections lock must be acquired to access the
@@ -201,7 +195,6 @@ int sdp_inet_listen_start(struct sdp_opt
 	SDP_CHECK_NULL(conn, -EINVAL);
 
 	if (SDP_SOCK_ST_CLOSED != conn->istate) {
-
 		sdp_dbg_warn(conn, "Incorrect connection state to listen.");
 		return -EBADFD;
 	}
@@ -220,10 +213,8 @@ int sdp_inet_listen_start(struct sdp_opt
 	_dev_root_s.listen_list = conn;
 	conn->lstn_p_next = &_dev_root_s.listen_list;
 
-	if (NULL != conn->lstn_next) {
-
+	if (NULL != conn->lstn_next)
 		conn->lstn_next->lstn_p_next = &conn->lstn_next;
-	}
 
 	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
 	return 0;
@@ -241,7 +232,6 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	SDP_CHECK_NULL(listen_conn, -EINVAL);
 
 	if (SDP_SOCK_ST_LISTEN != listen_conn->istate) {
-
 		sdp_dbg_warn(listen_conn, "Incorrect state to stop listen.");
 		return -EBADFD;
 	}
@@ -254,9 +244,8 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	/*
 	 * remove from listening list.
 	 */
-	if (NULL != listen_conn->lstn_next) {
+	if (NULL != listen_conn->lstn_next)
 		listen_conn->lstn_next->lstn_p_next = listen_conn->lstn_p_next;
-	}
 
 	*(listen_conn->lstn_p_next) = listen_conn->lstn_next;
 
@@ -277,11 +266,9 @@ int sdp_inet_listen_stop(struct sdp_opt 
 		accept_conn->istate = SDP_SOCK_ST_CLOSED;
 
 		result = sdp_wall_send_abort(accept_conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(accept_conn, "Error <%d> during abort",
 				     result);
-		}
 
 		/* AcceptQueueGet */
 		SDP_CONN_UNLOCK(accept_conn);
@@ -310,15 +297,12 @@ struct sdp_opt *sdp_inet_listen_lookup(u
 	 * first find a listening connection
 	 */
 	for (conn = _dev_root_s.listen_list; NULL != conn;
-	     conn = conn->lstn_next) {
-
+	     conn = conn->lstn_next)
 		if (port == conn->src_port &&
 		    (INADDR_ANY == conn->src_addr || addr == conn->src_addr)) {
-
 			SDP_CONN_HOLD(conn);
 			break;
 		}
-	}
 
 	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
 	return conn;
@@ -351,10 +335,8 @@ int sdp_inet_port_get(struct sdp_opt *co
 	 * simple linked list of sockets ordered on local port number.
 	 */
 	if (0 < port) {
-
 		for (look = _dev_root_s.bind_list, port_ok = 1;
 		     NULL != look; look = look->bind_next) {
-
 			srch = look->sk;
 			/*
 			 * 1) same port
@@ -402,43 +384,32 @@ int sdp_inet_port_get(struct sdp_opt *co
 		}
 
 		if (0 == port_ok) {
-
 			result = -EADDRINUSE;
 			goto done;
 		}
-	}
-	else {
-
+	} else {
 		low_port = SDP_INET_PORT_LOW;
 		top_port = SDP_INET_PORT_HIGH;
 		rover = (0 > rover) ? low_port : rover;
 
 		for (counter = (top_port - low_port) + 1; counter > 0;
 		     counter--) {
-
 			rover++;
-			if (rover < low_port || rover > top_port) {
-
+			if (rover < low_port || rover > top_port)
 				rover = low_port;
-			}
 
 			for (look = _dev_root_s.bind_list;
 			     NULL != look && look->src_port != port;
-			     look = look->bind_next) {
-				/*
-				 * pass
-				 */
-			}
+			     look = look->bind_next)
+				do {} while(0); /* pass */
 
 			if (NULL == look) {
-
 				port = rover;
 				break;
 			}
 		}
 
 		if (0 == port) {
-
 			result = -EADDRINUSE;
 			goto done;
 		}
@@ -452,10 +423,8 @@ int sdp_inet_port_get(struct sdp_opt *co
 	_dev_root_s.bind_list = conn;
 	conn->bind_p_next = &_dev_root_s.bind_list;
 
-	if (NULL != conn->bind_next) {
-
+	if (NULL != conn->bind_next)
 		conn->bind_next->bind_p_next = &conn->bind_next;
-	}
 
 	result = 0;
 done:
@@ -472,10 +441,8 @@ int sdp_inet_port_put(struct sdp_opt *co
 
 	SDP_CHECK_NULL(conn, -EINVAL);
 
-	if (NULL == conn->bind_p_next) {
-
+	if (NULL == conn->bind_p_next)
 		return -EADDRNOTAVAIL;
-	}
 	/*
 	 * lock table
 	 */
@@ -483,9 +450,8 @@ int sdp_inet_port_put(struct sdp_opt *co
 	/*
 	 * remove from bind list.
 	 */
-	if (NULL != conn->bind_next) {
+	if (NULL != conn->bind_next)
 		conn->bind_next->bind_p_next = conn->bind_p_next;
-	}
 
 	*(conn->bind_p_next) = conn->bind_next;
 
@@ -515,7 +481,6 @@ int sdp_inet_port_inherit(struct sdp_opt
 
 	if (NULL != child->bind_p_next ||
 	    child->src_port != parent->src_port) {
-
 		sdp_dbg_warn(child, "child already bound. <%d:%d>",
 			     parent->src_port, child->src_port);
 		result = -EADDRNOTAVAIL;
@@ -528,10 +493,8 @@ int sdp_inet_port_inherit(struct sdp_opt
 	parent->bind_next = child;
 	child->bind_p_next = &parent->bind_next;
 
-	if (NULL != child->bind_next) {
-
+	if (NULL != child->bind_next)
 		child->bind_next->bind_p_next = &child->bind_next;
-	}
 
 	result = 0;
 done:
@@ -550,10 +513,8 @@ static int _sdp_conn_table_insert(struct
 
 	SDP_CHECK_NULL(conn, -EINVAL);
 
-	if (SDP_DEV_SK_INVALID != conn->hashent) {
-
+	if (SDP_DEV_SK_INVALID != conn->hashent)
 		return -ERANGE;
-	}
 	/*
 	 * lock table
 	 */
@@ -564,14 +525,10 @@ static int _sdp_conn_table_insert(struct
 	for (counter = 0;
 	     counter < _dev_root_s.sk_size;
 	     counter++, _dev_root_s.sk_rover++) {
-
-		if (!(_dev_root_s.sk_rover < _dev_root_s.sk_size)) {
-
+		if (!(_dev_root_s.sk_rover < _dev_root_s.sk_size))
 			_dev_root_s.sk_rover = 0;
-		}
 
 		if (NULL == _dev_root_s.sk_array[_dev_root_s.sk_rover]) {
-
 			_dev_root_s.sk_array[_dev_root_s.sk_rover] = conn;
 			_dev_root_s.sk_entry++;
 			conn->hashent = _dev_root_s.sk_rover;
@@ -607,14 +564,10 @@ int sdp_conn_table_remove(struct sdp_opt
 	/*
 	 * validate entry
 	 */
-	if (SDP_DEV_SK_INVALID == conn->hashent) {
-
+	if (SDP_DEV_SK_INVALID == conn->hashent)
 		goto done;
-	}
-
-	if (0 > conn->hashent ||
-	    conn != _dev_root_s.sk_array[conn->hashent]) {
 
+	if (0 > conn->hashent || conn != _dev_root_s.sk_array[conn->hashent]) {
 		result = -ERANGE;
 		goto done;
 	}
@@ -656,10 +609,8 @@ struct sdp_opt *sdp_conn_table_lookup(s3
 #endif
 
 	conn = _dev_root_s.sk_array[entry];
-	if (NULL == conn) {
-
+	if (NULL == conn)
 		goto done;
-	}
 
 	SDP_CONN_HOLD(conn);
 done:
@@ -680,7 +631,6 @@ int sdp_conn_destruct(struct sdp_opt *co
 	int dump = 0;
 
 	if (NULL == conn) {
-
 		sdp_dbg_warn(NULL, "sk destruct, no connection!");
 		result = -EINVAL;
 		goto done;
@@ -698,12 +648,10 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 * remove connection from table
 	 */
 	result = sdp_conn_table_remove(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
 			     result, _dev_root_s.sk_entry,
 			     _dev_root_s.sk_size);
-	}
 
 	result = __sdp_conn_stat_dump(conn);
 	SDP_EXPECT(!(0 > result));
@@ -747,10 +695,8 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 * If the QP owner is not the CM, then destroy.
 	 */
 	if (conn->qp) {
-
 		result = ib_destroy_qp(conn->qp);
 		if (0 > result && -EINVAL != result) {
-
 			sdp_dbg_warn(conn, "Error <%d> detroying QP", result);
 			dump++;
 		}
@@ -759,10 +705,8 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 * destroy CQs
 	 */
 	if (conn->recv_cq) {
-
 		result = ib_destroy_cq(conn->recv_cq);
 		if (0 > result && -EINVAL != result) {
-
 			sdp_dbg_warn(conn, "Error <%d> detroying recv CQ",
 				     result);
 			dump++;
@@ -770,10 +714,8 @@ int sdp_conn_destruct(struct sdp_opt *co
 	}
 
 	if (conn->send_cq) {
-
 		result = ib_destroy_cq(conn->send_cq);
 		if (0 > result && -EINVAL != result) {
-
 			sdp_dbg_warn(conn, "Error <%d> detroying send CQ",
 				     result);
 			dump++;
@@ -783,33 +725,25 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 * in case CM/IB are still tracking this connection.
 	 */
 	if (conn->cm_id) {
-
 		sdp_dbg_warn(conn, "CM local id <%d>", conn->cm_id->local_id);
 
 		result = ib_destroy_cm_id(conn->cm_id);
-		if (result) {
-
+		if (result)
 			sdp_dbg_warn(conn, "Error <%d> detroying CM ID",
 				     result);
-		}
 	}
 	/*
 	 * check consistancy
 	 */
-	if (0 > atomic_read(&conn->refcnt)) {
-
+	if (0 > atomic_read(&conn->refcnt))
 		sdp_dbg_warn(conn, "destruct low ref count <%04x>",
 			     atomic_read(&conn->refcnt));
-	}
 	/*
 	 * free the OS socket structure
 	 */
-	if (NULL == conn->sk) {
-
+	if (NULL == conn->sk)
 		sdp_dbg_warn(conn, "destruct, no socket! continuing.");
-	}
 	else {
-
 		sk_free(conn->sk);
 		conn->sk = NULL;
 	}
@@ -820,7 +754,6 @@ int sdp_conn_destruct(struct sdp_opt *co
 done:
 
 	if (0 != dump) {
-
 		result = __sdp_conn_state_dump(conn);
 		SDP_EXPECT(!(0 > result));
 	}
@@ -844,10 +777,8 @@ void sdp_conn_internal_lock(struct sdp_o
 		spin_lock_irqsave(&(conn->lock.slock), f);
 		*flags = f;
 
-		if (0 == conn->lock.users) {
-
+		if (0 == conn->lock.users)
 			break;
-		}
 	}
 
 	current->state = TASK_RUNNING;
@@ -868,68 +799,48 @@ void sdp_conn_internal_relock(struct sdp
 	int rearm = 1;
 
 	while (1) {
-
 		result_r = ib_poll_cq(conn->recv_cq, 1, &entry);
 		if (1 == result_r) {
-			
 			result = sdp_cq_event_locked(&entry, conn);
-			if (0 > result) {
-
+			if (0 > result)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> from event handler.",
 					     result);
-			}
 
 			rearm = 1;
 		}
 
 		result_s = ib_poll_cq(conn->send_cq, 1, &entry);
 		if (1 == result_s) {
-
 			result = sdp_cq_event_locked(&entry, conn);
-			if (0 > result) {
-
+			if (0 > result)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> from event handler.",
 					     result);
-			}
-
 			rearm = 1;
 		}
 
-		if (0 != result_r || 0 != result_s) {
-
+		if (0 != result_r || 0 != result_s)
 			continue;
-		}
 
 		if (0 < rearm) {
-
 			result = ib_req_notify_cq(conn->recv_cq,
 						  IB_CQ_NEXT_COMP);
-			if (result) {
-				
+			if (result)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> rearming recv CQ",
 					     result);
-			}
 
 			result = ib_req_notify_cq(conn->send_cq,
 						  IB_CQ_NEXT_COMP);
-			if (result) {
-				
+			if (result)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> rearming send CQ",
 					     result);
-			}
 			
 			rearm = 0;
-		}
-		else {
-				/*
-				 * exit CQ handler routine.
-				 */
-			break;
-		}
+		} else
+			break;  /* exit CQ handler routine */
 	}
 
 	conn->flags &= ~SDP_CONN_F_MASK_EVENT;
@@ -961,11 +872,9 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 			 * to be armed.
 			 */
 			result = sdp_cq_event_locked(&entry, conn);
-			if (0 > result) {
-
+			if (0 > result)
 				sdp_dbg_warn(conn, "Error <%d> event handler.",
 					     result);
-			}
 
 			rearm = 1;
 			calls++;
@@ -974,33 +883,18 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 		}
 
 		if (!result) {
-
 			if (0 < rearm) {
-
 				result = ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
-				if (result) {
-
+				if (result)
 					sdp_dbg_warn(conn, 
 						     "Error <%d> rearming CQ",
 						     result);
-				}
-
 				rearm = 0;
-			}
-			else {
-				/*
-				 * exit CQ handler routine.
-				 */
-				break;
-			}
-		}
-		else {
-			/*
-			 * unexpected CQ error
-			 */
+			} else
+				break; /* exit CQ handler routine */
+		} else
 			sdp_dbg_warn(conn, "Unexpected error <%d> from CQ",
 				     result);
-		}
 	}
 
 return calls;
@@ -1016,16 +910,11 @@ void sdp_conn_internal_unlock(struct sdp
 	 * poll CQs for events.
 	 */
 	if (NULL != conn) {
-
-		if (0 < (SDP_CONN_F_RECV_CQ_PEND & conn->flags)) {
-
+		if (0 < (SDP_CONN_F_RECV_CQ_PEND & conn->flags))
 			calls += sdp_conn_cq_drain(conn->recv_cq, conn);
-		}
-
-		if (0 < (SDP_CONN_F_SEND_CQ_PEND & conn->flags)) {
 
+		if (0 < (SDP_CONN_F_SEND_CQ_PEND & conn->flags))
 			calls += sdp_conn_cq_drain(conn->send_cq, conn);
-		}
 
 		conn->flags &= ~SDP_CONN_F_MASK_EVENT;
 	}
@@ -1067,35 +956,29 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	 * look up correct HCA and port
 	 */
 	hca = ib_get_client_data(device, &sdp_client);
-	if (!hca) {
+	if (!hca)
 		return -ERANGE;
-	}
 
-	for (port = hca->port_list; NULL != port; port = port->next) {
-		if (hw_port == port->index) {
+	for (port = hca->port_list; NULL != port; port = port->next)
+		if (hw_port == port->index)
 			break;
-		}
-	}
 
-	if (!port) {
+	if (!port)
 		return -ERANGE;
-	}
 	/*
 	 * allocate creation parameters
 	 */
 	qp_attr = kmalloc(sizeof(*qp_attr),  GFP_KERNEL);
 	if (NULL == qp_attr) {
-		
 		result = -ENOMEM;
 		goto error_attr;
-	} /* if */
+	}
   
 	init_attr = kmalloc(sizeof(*init_attr), GFP_KERNEL);
 	if (NULL == init_attr) {
-		
 		result = -ENOMEM;
 		goto error_param;
-	} /* if */
+	}
 	
 	memset(qp_attr,   0, sizeof(*qp_attr));
 	memset(init_attr, 0, sizeof(*init_attr));
@@ -1113,14 +996,12 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	 * allocate IB CQ's and QP
 	 */
 	if (!conn->send_cq) {
-
 		conn->send_cq = ib_create_cq(conn->ca,
 					     sdp_cq_event_handler,
 					     NULL,
 					     __hashent_arg(conn->hashent),
 					     conn->send_cq_size);
 		if (IS_ERR(conn->send_cq)) {
-
 			result = PTR_ERR(conn->send_cq);
 			sdp_dbg_warn(conn, "Error <%d> creating send CQ <%d>",
 				     result, conn->send_cq_size);
@@ -1131,7 +1012,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 
 		result = ib_req_notify_cq(conn->send_cq, IB_CQ_NEXT_COMP);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> arming send CQ.",
 				     result);
 			goto error_rcq;
@@ -1146,7 +1026,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 					     conn->recv_cq_size);
 
 		if (IS_ERR(conn->recv_cq)) {
-
 			result = PTR_ERR(conn->recv_cq);
 			sdp_dbg_warn(conn, "Error <%d> creating recv CQ <%d>",
 				     result, conn->recv_cq_size);
@@ -1157,7 +1036,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 
 		result = ib_req_notify_cq(conn->recv_cq, IB_CQ_NEXT_COMP);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> arming recv CQ.",
 				     result);
 			goto error_qp;
@@ -1165,7 +1043,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	}
 
 	if (!conn->qp) {
-
 		init_attr->cap.max_send_wr  = conn->send_cq_size;
 		init_attr->cap.max_recv_wr  = conn->recv_cq_size;
 		init_attr->cap.max_send_sge = SDP_QP_LIMIT_SG_SEND;
@@ -1181,7 +1058,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 
 		conn->qp = ib_create_qp(conn->pd, init_attr);
 		if (IS_ERR(conn->qp)) {
-
 			result = PTR_ERR(conn->qp);
 			sdp_dbg_warn(conn, "Error <%d> creating QP", result);
 
@@ -1197,7 +1073,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 					     pkey,
 					     &qp_attr->pkey_index);
 		if (result) {
-
 			sdp_dbg_warn(conn, "Error <%d> find pkey index <%04x>",
 				     result, pkey);
 			goto error_mod;
@@ -1216,7 +1091,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 		result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
 
 		if (0 != result) {
-
 			sdp_dbg_warn(conn, "Error <%d> modifying QP", result);
 			goto error_mod;
 		}
@@ -1254,7 +1128,6 @@ struct sdp_opt *sdp_conn_alloc(int prior
 
 	sk = sk_alloc(_dev_root_s.proto, priority, 1, _dev_root_s.sock_cache);
 	if (NULL == sk) {
-
 		sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>",
 			     _dev_root_s.proto, priority);
 		return NULL;
@@ -1283,7 +1156,6 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	 */
 	conn = kmem_cache_alloc(_dev_root_s.conn_cache, priority);
 	if (NULL == conn) {
-
 		sdp_dbg_warn(conn, "connection alloc error. <%d>", priority);
 		result = -ENOMEM;
 		goto error;
@@ -1496,7 +1368,6 @@ int sdp_proc_dump_conn_main(char *buffer
 	 * header should only be printed once
 	 */
 	if (0 == start_index) {
-
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_MAIN_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_MAIN_SEP);
 	}
@@ -1507,10 +1378,8 @@ int sdp_proc_dump_conn_main(char *buffer
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size)) {
-
+	if (!(start_index < _dev_root_s.sk_size))
 		goto done;
-	}
 	/*
 	 * loop across connections.
 	 */
@@ -1518,11 +1387,8 @@ int sdp_proc_dump_conn_main(char *buffer
 	     counter < _dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_MAIN_SIZE > (max_size - offset));
 	     counter++) {
-
-		if (NULL == _dev_root_s.sk_array[counter]) {
-
+		if (NULL == _dev_root_s.sk_array[counter])
 			continue;
-		}
 
 		conn = _dev_root_s.sk_array[counter];
 
@@ -1610,7 +1476,6 @@ int sdp_proc_dump_conn_data(char *buffer
 	 * header should only be printed once
 	 */
 	if (0 == start_index) {
-
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_SEP);
 	}
@@ -1621,21 +1486,16 @@ int sdp_proc_dump_conn_data(char *buffer
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size)) {
-
+	if (!(start_index < _dev_root_s.sk_size))
 		goto done;
-	}
 	/*
 	 * loop across connections.
 	 */
 	for (counter = start_index; counter < _dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_DATA_SIZE > (max_size - offset));
 	     counter++) {
-
-		if (NULL == _dev_root_s.sk_array[counter]) {
-
+		if (NULL == _dev_root_s.sk_array[counter])
 			continue;
-		}
 
 		conn = _dev_root_s.sk_array[counter];
 		sk = conn->sk;
@@ -1715,7 +1575,6 @@ int sdp_proc_dump_conn_rdma(char *buffer
 	 * header should only be printed once
 	 */
 	if (0 == start_index) {
-
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_RDMA_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_RDMA_SEP);
 	}
@@ -1726,21 +1585,16 @@ int sdp_proc_dump_conn_rdma(char *buffer
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size)) {
-
+	if (!(start_index < _dev_root_s.sk_size))
 		goto done;
-	}
 	/*
 	 * loop across connections.
 	 */
 	for (counter = start_index; counter < _dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_RDMA_SIZE > (max_size - offset));
 	     counter++) {
-
-		if (NULL == _dev_root_s.sk_array[counter]) {
-
+		if (NULL == _dev_root_s.sk_array[counter])
 			continue;
-		}
 
 		conn = _dev_root_s.sk_array[counter];
 
@@ -1803,7 +1657,6 @@ int sdp_proc_dump_conn_sopt(char *buffer
 	 * header should only be printed once
 	 */
 	if (0 == start_index) {
-
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_SOPT_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_SOPT_SEP);
 	}
@@ -1814,21 +1667,16 @@ int sdp_proc_dump_conn_sopt(char *buffer
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size)) {
-
+	if (!(start_index < _dev_root_s.sk_size))
 		goto done;
-	}
 	/*
 	 * loop across connections.
 	 */
 	for (counter = start_index; counter < _dev_root_s.sk_size &&
 		     !(SDP_SOPT_PROC_DUMP_SIZE > (max_size - offset));
 	     counter++) {
-
-		if (NULL == _dev_root_s.sk_array[counter]) {
-
+		if (NULL == _dev_root_s.sk_array[counter])
 			continue;
-		}
 
 		conn = _dev_root_s.sk_array[counter];
 
@@ -1872,7 +1720,6 @@ int sdp_proc_dump_device(char *buffer,
 	 * header should only be printed once
 	 */
 	if (0 == start_index) {
-
 		offset += sprintf((buffer + offset),
 				  "connection table maximum: <%d>\n",
 				  _dev_root_s.sk_size);
@@ -1922,8 +1769,6 @@ static void sdp_device_init_one(struct i
 	 */
 	hca = kmalloc(sizeof(struct sdev_hca), GFP_KERNEL);
 	if (NULL == hca) {
-
-		
 		sdp_warn("Error allocating HCA <%s> memory.", device->name);
 		return;
 	}
@@ -1941,7 +1786,6 @@ static void sdp_device_init_one(struct i
 	 */
 	hca->pd = ib_alloc_pd(hca->ca);
 	if (IS_ERR(hca->pd)) {
-
 		sdp_warn("Error <%ld> creating HCA <%s> protection domain.",
 			 PTR_ERR(hca->pd), device->name);
 		goto error;
@@ -1951,7 +1795,6 @@ static void sdp_device_init_one(struct i
 	 */
 	hca->mem_h = ib_get_dma_mr(hca->pd, IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(hca->mem_h)) {
-
 		sdp_warn("Error <%ld> registering HCA <%s> memory.",
 			 PTR_ERR(hca->mem_h), device->name);
 		goto error;
@@ -1977,7 +1820,6 @@ static void sdp_device_init_one(struct i
 #ifdef _FMR_SUPPORT
 	hca->fmr_pool = ib_create_fmr_pool(hca->pd, &fmr_param_s);
 	if (IS_ERR(hca->fmr_pool)) {
-
 		sdp_warn("Error <%ld> creating HCA <%s> fast memory pool",
 			 PTR_ERR(hca->fmr_pool), device->name);
 		goto error;
@@ -1989,10 +1831,8 @@ static void sdp_device_init_one(struct i
 	for (port_count = 0; 
 	     port_count < device->phys_port_cnt;
 	     port_count++) {
-
 		port = kmalloc(sizeof(struct sdev_hca_port), GFP_KERNEL);
 		if (NULL == port) {
-
 			sdp_warn("Error allocating HCA <%s> port <%d:%d>",
 				 device->name, port_count,
 				 device->phys_port_cnt);
@@ -2011,7 +1851,6 @@ static void sdp_device_init_one(struct i
 				      0,	/* index */
 				      &port->gid);
 		if (0 != result) {
-
 			sdp_warn("Error <%d> getting GID for HCA <%s:%d:%d>",
 				 result, device->name,
 				 port->index, device->phys_port_cnt);
@@ -2025,7 +1864,6 @@ static void sdp_device_init_one(struct i
 
 error:
 	while (NULL != hca->port_list) {
-			 
 		port = hca->port_list;
 		hca->port_list = port->next;
 		port->next = NULL;
@@ -2057,13 +1895,11 @@ static void sdp_device_remove_one(struct
 	hca = ib_get_client_data(device, &sdp_client);
 
 	if (NULL == hca) {
-
 		sdp_warn("Device <%s> has no HCA info.", device->name);
 		return;
 	}
 	
 	while (NULL != hca->port_list) {
-			 
 		port = hca->port_list;
 		hca->port_list = port->next;
 		port->next = NULL;
@@ -2131,7 +1967,6 @@ int sdp_conn_table_init(int proto_family
 	 */
 	result = ib_register_client(&sdp_client);
 	if (0 > result) {
-
 		sdp_warn("Error <%d> registering SDP client.", result);
 		goto error_hca;
 	}
@@ -2139,7 +1974,6 @@ int sdp_conn_table_init(int proto_family
 	 * create socket table
 	 */
 	if (!(0 < conn_size)) {
-
 		sdp_warn("Invalid connection table size. <%d>", conn_size);
 		result = -EINVAL;
 		goto error_size;
@@ -2153,7 +1987,6 @@ int sdp_conn_table_init(int proto_family
 	_dev_root_s.sk_array = (void *) __get_free_pages(GFP_KERNEL,
 						       _dev_root_s.sk_ordr);
 	if (NULL == _dev_root_s.sk_array) {
-
 		sdp_warn("Failed to create connection table. <%d:%d:%d>",
 			 byte_size, page_size, _dev_root_s.sk_ordr);
 		result = -ENOMEM;
@@ -2170,7 +2003,6 @@ int sdp_conn_table_init(int proto_family
 	 */
 	result = sdp_main_iocb_init();
 	if (0 > result) {
-
 		sdp_warn("Error <%d> initializing SDP IOCB table.", result);
 		goto error_iocb;
 	}
@@ -2180,7 +2012,6 @@ int sdp_conn_table_init(int proto_family
 						   0, SLAB_HWCACHE_ALIGN,
 						   NULL, NULL);
 	if (NULL == _dev_root_s.conn_cache) {
-
 		sdp_warn("Failed to initialize connection cache.");
 		result = -ENOMEM;
 		goto error_conn;
@@ -2191,7 +2022,6 @@ int sdp_conn_table_init(int proto_family
 						   0, SLAB_HWCACHE_ALIGN,
 						   NULL, NULL);
         if (NULL == _dev_root_s.sock_cache) {
-
 		sdp_warn("Failed to initialize sock cache.");
 		result = -ENOMEM;
 		goto error_sock;
@@ -2202,7 +2032,6 @@ int sdp_conn_table_init(int proto_family
 	 */
 	result = sdp_cm_listen_start(&_dev_root_s);
 	if (0 > result) {
-
 		sdp_warn("Error <%d> listening for connections on HCA.",
 			 result);
 		goto error_listen;



From Tom.Duffy at Sun.COM  Fri Feb 18 16:45:14 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:45:14 -0800
Subject: [openib-general] [PATCH][SDP][5/22] Missed a comment in sdp_conn.h
In-Reply-To: <11087739032962@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
Message-ID: <1108773914773@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_conn.h
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_conn.h	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_conn.h	(working copy)
@@ -158,9 +158,10 @@ enum sdp_mode {
         ((conn)->error = (conn)->sk->sk_err = (val))
 #define SDP_CONN_GET_ERR(conn) \
         ((conn)->error)
-/* --------------------------------------------------------------------- */
-/* state transition information recording                                */
-/* --------------------------------------------------------------------- */
+
+/*
+ * state transition information recording
+ */
 #ifdef _SDP_CONN_STATE_REC
 
 #define SDP_CONN_STATE_MAX 16 /* maximum state transitions recorded. */



From Tom.Duffy at Sun.COM  Fri Feb 18 16:45:24 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:45:24 -0800
Subject: [openib-general] [PATCH][SDP][6/22] Whitespace cleanup in
	sdp_event.c
In-Reply-To: <1108773914773@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com> <1108773914773@sun.com>
Message-ID: <11087739241469@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_event.c	(working copy)
@@ -68,19 +68,15 @@ int sdp_cq_event_locked(struct ib_wc *co
 	 */
 	switch (comp->opcode) {
 	case IB_WC_RECV:
-
 		result = sdp_event_recv(conn, comp);
 		break;
 	case IB_WC_SEND:
-
 		result = sdp_event_send(conn, comp);
 		break;
 	case IB_WC_RDMA_READ:
-
 		result = sdp_event_read(conn, comp);
 		break;
 	case IB_WC_RDMA_WRITE:
-
 		result = sdp_event_write(conn, comp);
 		break;
 	default:
@@ -94,12 +90,10 @@ int sdp_cq_event_locked(struct ib_wc *co
 		case IB_WC_WR_FLUSH_ERR:
 			break;
 		case IB_WC_SUCCESS:
-
 			sdp_warn("unknown IB event. <%d>", comp->opcode);
 			result = -ERANGE;
 			break;
 		default:
-
 			sdp_warn("Unhandled status <%d> unknown event <%d>",
 				 comp->status, comp->opcode);
 			result = -EIO;
@@ -112,7 +106,6 @@ int sdp_cq_event_locked(struct ib_wc *co
 	 * release socket before error processing.
 	 */
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "ABORT on error <%d> event <%u:%llu:%u:%u>",
 			     result,
 			     comp->status,
@@ -125,10 +118,8 @@ int sdp_cq_event_locked(struct ib_wc *co
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_CQ);
 
 		result = sdp_wall_abort(conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> during abort", result);
-		}
 
 		return -EFAULT;
 	}
@@ -153,7 +144,6 @@ void sdp_cq_event_handler(struct ib_cq *
 	 */
 	conn = sdp_conn_table_lookup(hashent);
 	if (NULL == conn) {
-
 		sdp_dbg_warn(conn, "Unknown connection <%d> for cq event",
 			     hashent);
 		goto done;
@@ -176,26 +166,22 @@ void sdp_cq_event_handler(struct ib_cq *
 		 */
 		if (SDP_CONN_ST_REP_SENT == conn->state ||
 		    SDP_CONN_ST_RTU_SENT == conn->state) {
-
 			result = ib_cm_establish(conn->cm_id);
 			SDP_EXPECT(!(0 > result));
 		}
-		else {
-
+		else
 			sdp_dbg_warn(conn, "Unexpected locked state.");
-		}
 
 		conn->flags |= SDP_CONN_F_MASK_EVENT;
 		goto unlock;
 	}
 
-	if (0 == conn->lock.users) {
+	if (0 == conn->lock.users)
 		/*
 		 * dispatch CQ completions.
 		 */
 		(void)sdp_conn_cq_drain(cq, conn);
-	}
-	else {
+	else
 		/*
 		 * Mark the event which was received, for the unlock code to
 		 * process at a later time.
@@ -203,7 +189,6 @@ void sdp_cq_event_handler(struct ib_cq *
 		conn->flags |= ((cq == conn->recv_cq) ?
 				SDP_CONN_F_RECV_CQ_PEND :
 				SDP_CONN_F_SEND_CQ_PEND);
-	}
 
 unlock:
 	SDP_CONN_UNLOCK_BH(conn);
@@ -226,9 +211,8 @@ static int _sdp_cm_idle(struct ib_cm_id 
 	int result = 0;
 	int expect;
 
-	if (NULL == conn) {
+	if (NULL == conn)
 		return -EINVAL;
-	}
 	/*
 	 * IDLE should only be called after some other action on the comm_id,
 	 * which means the callback argument will be a SDP conn, since the
@@ -253,10 +237,8 @@ static int _sdp_cm_idle(struct ib_cm_id 
 		 * fall through
 		 */
 	case SDP_CONN_ST_REQ_SENT:	/* active open, Hello msg sent */
-
 		result = sdp_wall_recv_reject(conn, ECONNREFUSED);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> receiving CM reject.",
 				     result);
 			goto error;
@@ -264,7 +246,6 @@ static int _sdp_cm_idle(struct ib_cm_id 
 
 		break;
 	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
-
 		result = sdp_wall_recv_failed(conn, ECONNREFUSED);
 		if (0 > result) {
 
@@ -286,7 +267,6 @@ static int _sdp_cm_idle(struct ib_cm_id 
 		break;
 	case SDP_CONN_ST_TIME_WAIT_1:
 	case SDP_CONN_ST_ESTABLISHED:
-
 		sdp_dbg_warn(conn, "Unexpected connection state");
 		/*
 		 * fall through
@@ -301,7 +281,6 @@ static int _sdp_cm_idle(struct ib_cm_id 
 
 		break;
 	default:
-
 		sdp_warn("Unknown conn state. conn <%d> state <%04x:%04x>",
 			 conn->hashent, conn->istate, conn->state);
 		result = -EINVAL;
@@ -332,9 +311,8 @@ static int _sdp_cm_established(struct ib
 	int expect;
 	struct sdpc_buff *buff;
 
-	if (NULL == conn) {
+	if (NULL == conn)
 		return -EINVAL;
-	}
 
 	sdp_dbg_ctrl(conn, "CM ESTABLISHED. commID <%08x>", cm_id->local_id);
 	/*
@@ -345,24 +323,17 @@ static int _sdp_cm_established(struct ib
 	 * check state
 	 */
 	switch (conn->state) {
-
 	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
-
 		buff = sdp_buff_q_get_head(&conn->send_post);
-		if (NULL == buff) {
-
+		if (NULL == buff)
 			sdp_dbg_warn(conn, "hello ack missing in send pool");
-		}
-		else {
-
+		else
 			(void)sdp_buff_pool_put(buff);
-		}
 
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
 
 		result = sdp_wall_recv_confirm(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> confirming conn state",
 				     result);
 			/*
@@ -374,7 +345,6 @@ static int _sdp_cm_established(struct ib
 
 		result = sdp_send_flush(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> flushing send queue.",
 				     result);
 			goto error;
@@ -382,7 +352,6 @@ static int _sdp_cm_established(struct ib
 
 		result = sdp_recv_flush(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
 				     result);
 			goto error;
@@ -402,10 +371,8 @@ static int _sdp_cm_established(struct ib
 	case SDP_CONN_ST_DIS_PEND_R:
 		/* active open, and active close, confirm */
 	case SDP_CONN_ST_DIS_PEND_2:
-
 		result = sdp_send_flush(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
 				     result);
 			goto error;
@@ -419,11 +386,9 @@ static int _sdp_cm_established(struct ib
 		 * existing state correctly.
 		 */
 		result = sdp_cm_disconnect(conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> posting CM disconnect",
 				     result);
-		}
 
 		break;
 	default:
@@ -454,10 +419,8 @@ static int _sdp_cm_timewait(struct ib_cm
 	int result = 0;
 	int expect;
 
-	if (NULL == conn) {
-
+	if (NULL == conn)
 		return -EINVAL;
-	}
 
 	sdp_dbg_ctrl(conn, "CM TIME WAIT. commID <%08x> event <%d>",
 		     cm_id->local_id, event->event);
@@ -507,14 +470,12 @@ static int _sdp_cm_timewait(struct ib_cm
 
 		result = sdp_wall_abort(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> during abort", result);
 			goto error;
 		}
 
 		break;
 	default:
-
 		sdp_warn("Unexpected conn state. conn <%d> state <%04x:%04x>",
 			 conn->hashent, conn->istate, conn->state);
 		result = -EINVAL;
@@ -551,44 +512,32 @@ int sdp_cm_event_handler(struct ib_cm_id
 	 * lookup the connection, on a REQ_RECV the sk will be empty.
 	 */
 	conn = sdp_conn_table_lookup(hashent);
-	if (NULL != conn) {
-
+	if (NULL != conn)
 		SDP_CONN_LOCK(conn);
-	}
-	else {
-
-		if (IB_CM_REQ_RCVD != cm_id->state) {
-
+	else
+		if (IB_CM_REQ_RCVD != cm_id->state)
 			sdp_dbg_warn(NULL, 
 				     "No conn <%d> CM state <%d> event <%d>",
 				     hashent, cm_id->state, event->event);
-		}
-	}
 
 	switch (cm_id->state) {
 	case IB_CM_REQ_RCVD:
-
 		result = sdp_cm_req_handler(cm_id, event);
 		break;
 	case IB_CM_REP_RCVD:
-
 		result = sdp_cm_rep_handler(cm_id, event, conn);
 		break;
 	case IB_CM_IDLE:
-
 		result = _sdp_cm_idle(cm_id, event, conn);
 		break;
 	case IB_CM_ESTABLISHED:
-
 		result = _sdp_cm_established(cm_id, event, conn);
 		break;
 	case IB_CM_DREQ_RCVD:
 	case IB_CM_TIMEWAIT:
-
 		result = _sdp_cm_timewait(cm_id, event, conn);
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Unexpected CM state <%d>", cm_id->state);
 		result = -EINVAL;
 	}
@@ -596,9 +545,7 @@ int sdp_cm_event_handler(struct ib_cm_id
 	 * if a socket was found, release the lock, and put the reference.
 	 */
 	if (NULL != conn) {
-
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, 
 				     "CM state <%d> event <%d> error <%d>",
 				     cm_id->state, event->event, result);



From Tom.Duffy at Sun.COM  Fri Feb 18 16:45:34 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:45:34 -0800
Subject: [openib-general] [PATCH][SDP][7/22] Whitespace cleanup in sdp_inet.c
In-Reply-To: <11087739241469@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
Message-ID: <11087739341677@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -103,24 +103,18 @@ void sdp_inet_wake_send(struct sock *sk)
 {
 	struct sdp_opt *conn;
 
-	if (NULL == sk || 
-	    NULL == (conn = SDP_GET_CONN(sk))) {
-
+	if (NULL == sk || NULL == (conn = SDP_GET_CONN(sk)))
 		return;
-	} 
 
 	if (NULL != sk->sk_socket &&
 	    0 < test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) &&
 	    0 < __sdp_inet_writable(conn)) {
-
 		read_lock(&sk->sk_callback_lock);
 		clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
 
 		if (NULL != sk->sk_sleep &&
-		    0 < waitqueue_active(sk->sk_sleep)) {
-
+		    0 < waitqueue_active(sk->sk_sleep))
 			wake_up_interruptible(sk->sk_sleep);
-		}
 		/*
 		 * test, clear, and notify. SOCK_ASYNC_NOSPACE
 		 */
@@ -137,14 +131,10 @@ void sdp_inet_wake_send(struct sock *sk)
 void sdp_inet_wake_generic(struct sock *sk)
 {
 	if (NULL != sk) {
-
 		read_lock(&sk->sk_callback_lock);
 
-		if (NULL != sk->sk_sleep &&
-		    waitqueue_active(sk->sk_sleep)) {
-
+		if (NULL != sk->sk_sleep && waitqueue_active(sk->sk_sleep))
 			wake_up_interruptible_all(sk->sk_sleep);
-		}
 
 		read_unlock(&sk->sk_callback_lock);
 	}
@@ -158,12 +148,9 @@ void sdp_inet_wake_generic(struct sock *
 void sdp_inet_wake_recv(struct sock *sk, int len)
 {
 	if (NULL != sk) {
-
 		read_lock(&sk->sk_callback_lock);
-		if (NULL != sk->sk_sleep) {
-			
+		if (NULL != sk->sk_sleep)
 			wake_up_interruptible(sk->sk_sleep);
-		}
 
 		sk_wake_async(sk, 1, POLL_IN);
 		read_unlock(&sk->sk_callback_lock);
@@ -178,12 +165,9 @@ void sdp_inet_wake_recv(struct sock *sk,
 void sdp_inet_wake_error(struct sock *sk)
 {
 	if (NULL != sk) {
-
 		read_lock(&sk->sk_callback_lock);
-		if (NULL != sk->sk_sleep) {
-		
+		if (NULL != sk->sk_sleep)
 			wake_up_interruptible(sk->sk_sleep);
-		}
 
 		sk_wake_async(sk, 0, POLL_ERR);
 		read_unlock(&sk->sk_callback_lock);
@@ -201,9 +185,8 @@ void sdp_inet_wake_urg(struct sock *sk)
 	 * pid for SIGURG/SIGIO has been set. On positive send signal to
 	 * process, on negative send signal to processes group.
 	 */
-	if (NULL != sk) {
+	if (NULL != sk)
 		sk_send_sigurg(sk);
-	}
 
 	return;
 } /* sdp_inet_wake_urg */
@@ -224,18 +207,15 @@ static int _sdp_inet_abort(struct sdp_op
 	conn->send_buf = 0;
 
 	switch (conn->istate) {
-
 	case SDP_SOCK_ST_CONNECT:
 	case SDP_SOCK_ST_ACCEPTING:
 	case SDP_SOCK_ST_ACCEPTED:
-
 		sdp_dbg_warn(conn, "Unexpected abort");
 
 	case SDP_SOCK_ST_ESTABLISHED:
 	case SDP_SOCK_ST_CLOSE:
 	case SDP_SOCK_ST_DISCONNECT:
 	case SDP_SOCK_ST_CLOSING:
-
 		result = sdp_wall_abort(conn);
 		if (0 > result) {
 
@@ -248,14 +228,12 @@ static int _sdp_inet_abort(struct sdp_op
 	case SDP_SOCK_ST_LISTEN:
 	case SDP_SOCK_ST_CLOSED:
 	case SDP_SOCK_ST_ERROR:
-
 		sdp_dbg_warn(conn, "Unhandled abort");
 
 		conn->istate = SDP_SOCK_ST_ERROR;
 		result = -EINVAL;
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Unknown abort state");
 
 		conn->istate = SDP_SOCK_ST_ERROR;
@@ -276,37 +254,28 @@ static int _sdp_inet_disconnect(struct s
 	SDP_CHECK_NULL(conn, -EINVAL);
 
 	switch (conn->istate) {
-
 	case SDP_SOCK_ST_CONNECT:
-
 		result = sdp_wall_abort(conn);
 		if (0 > result) {
-
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
 			conn->istate = SDP_SOCK_ST_ERROR;
 		}
-
 		break;
 	case SDP_SOCK_ST_ESTABLISHED:
 	case SDP_SOCK_ST_ACCEPTED:
-
 		conn->istate = SDP_SOCK_ST_DISCONNECT;
 		result = sdp_wall_send_close(conn);
 		if (0 > result) {
-
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
 			conn->istate = SDP_SOCK_ST_ERROR;
 		}
-
 		break;
 	case SDP_SOCK_ST_CLOSE:
-
 		conn->istate = SDP_SOCK_ST_CLOSING;
 		result = sdp_wall_send_closing(conn);
 		if (0 > result) {
-
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
 			conn->istate = SDP_SOCK_ST_ERROR;
@@ -325,12 +294,9 @@ static int _sdp_inet_disconnect(struct s
 	case SDP_SOCK_ST_LISTEN:
 	case SDP_SOCK_ST_CLOSED:
 	case SDP_SOCK_ST_ERROR:
-
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Unknown disconnect state");
-
 		conn->istate = SDP_SOCK_ST_ERROR;
 		result = -EINVAL;
 		break;
@@ -358,7 +324,6 @@ static int _sdp_inet_release(struct sock
 	SDP_CHECK_NULL(sock->file, -EINVAL);
 
 	if (NULL == sock->sk) {
-
 		sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>",
 			     sock->type, sock->state, sock->flags);
 		return 0;
@@ -383,26 +348,20 @@ static int _sdp_inet_release(struct sock
 		 * stop listening
 		 */
 		result = sdp_inet_listen_stop(conn);
-		if (0 > result) {
-			
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> while releasing listen",
 				     result);
-		}
 
 		goto done;
 	}
 	/*
 	 * get blocking nature of the socket.
 	 */
-	if (sock->file) {
-
+	if (sock->file)
 		flags = (0 < (sock->file->f_flags & O_NONBLOCK)) ? \
 			MSG_DONTWAIT : 0;
-	}
-	else {
-
+	else
 		flags = 0;
-	}
 	/*
 	 * If there is data in the receive queue, flush it,
 	 * and consider this an abort. Otherwise consider
@@ -416,11 +375,9 @@ static int _sdp_inet_release(struct sock
 		 * abort.
 		 */
 		result = _sdp_inet_abort(conn);
-		if (0 > result) {
-			
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> while aborting socket",
 				     result);
-		}
 
 		goto done;
 	}
@@ -429,7 +386,6 @@ static int _sdp_inet_release(struct sock
 	 */
 	result = _sdp_inet_disconnect(conn);
 	if (0 > result) {
-		
 		sdp_dbg_warn(conn, "Error <%d> while disconnecting socket",
 			     result);
 		goto done;
@@ -446,7 +402,6 @@ static int _sdp_inet_release(struct sock
 		 */
 		if (sock_flag(sk, SOCK_LINGER)
 		    && !(PF_EXITING & current->flags)) {
-
 			DECLARE_WAITQUEUE(wait, current);
 			timeout = sk->sk_lingertime;
 			
@@ -455,15 +410,12 @@ static int _sdp_inet_release(struct sock
 
 			while (0 < timeout &&
 			       0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
-
 				SDP_CONN_UNLOCK(conn);
 				timeout = schedule_timeout(timeout);
 				SDP_CONN_LOCK(conn);
 
-				if (signal_pending(current)) {
-					
+				if (signal_pending(current))
 					break;
-				}
 			}
 
 			set_current_state(TASK_RUNNING);
@@ -487,14 +439,9 @@ static int _sdp_inet_release(struct sock
 	} /* if (blocking) */
 
 done:
-	if (0 < (SDP_ST_MASK_CLOSED & conn->istate)) {
-		/*
-		 * pass
-		 */
-	}
-	else {
+	if (0 < (SDP_ST_MASK_CLOSED & conn->istate))
+		do {} while(0); /* pass */
 		
-	}
 	/*
 	 * finally drop socket reference. (socket API reference)
 	 */
@@ -528,16 +475,12 @@ static int _sdp_inet_bind(struct socket 
 	sdp_dbg_ctrl(conn, "BIND: family <%d> addr <%08x:%04x>",
 		     addr->sin_family, addr->sin_addr.s_addr, addr->sin_port);
 
-	if (size < sizeof(struct sockaddr_in)) {
-
+	if (size < sizeof(struct sockaddr_in))
 		return -EINVAL;
-	}
 
 	if (_proto_family != addr->sin_family &&
-	    AF_INET != addr->sin_family && AF_UNSPEC != addr->sin_family) {
-
+	    AF_INET != addr->sin_family && AF_UNSPEC != addr->sin_family)
 		return -EAFNOSUPPORT;
-	}
 	/*
 	 * Basically we're OK with INADDR_ANY or a local interface
 	 * (TODO: loopback)
@@ -551,41 +494,33 @@ static int _sdp_inet_bind(struct socket 
 		if (inet_sk(sk)->freebind == 0 &&
 		    RTN_LOCAL != addr_result &&
 		    RTN_MULTICAST != addr_result &&
-		    RTN_BROADCAST != addr_result) {
-
+		    RTN_BROADCAST != addr_result)
 			return -EADDRNOTAVAIL;
-		}
 	}
 	/*
 	 * check bind permission for low ports.
 	 */
 	bind_port = ntohs(addr->sin_port);
 	if (0 < bind_port &&
-	    bind_port < PROT_SOCK && 0 == capable(CAP_NET_BIND_SERVICE)) {
-
+	    bind_port < PROT_SOCK && 0 == capable(CAP_NET_BIND_SERVICE))
 		return -EACCES;
-	}
 	/*
 	 * socket checks.
 	 */
 	SDP_CONN_LOCK(conn);
 
 	if (SDP_SOCK_ST_CLOSED != conn->istate || 0 < conn->src_port) {
-
 		result = -EINVAL;
 		goto done;
 	}
 
 	conn->src_addr = ntohl(addr->sin_addr.s_addr);
 
-	if (RTN_MULTICAST == addr_result || RTN_BROADCAST == addr_result) {
-
+	if (RTN_MULTICAST == addr_result || RTN_BROADCAST == addr_result)
 		conn->src_addr = 0;
-	}
 
 	result = sdp_inet_port_get(conn, bind_port);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> getting port during bind",
 			     result);
 
@@ -594,15 +529,11 @@ static int _sdp_inet_bind(struct socket 
 		goto done;
 	}
 
-	if (INADDR_ANY != conn->src_addr) {
-
+	if (INADDR_ANY != conn->src_addr)
 		sk->sk_userlocks |= SOCK_BINDADDR_LOCK;
-	}
-
-	if (0 < bind_port) {
 
+	if (0 < bind_port)
 		sk->sk_userlocks |= SOCK_BINDADDR_LOCK;
-	}
 
 	inet_sk(sk)->rcv_saddr = htonl(conn->src_addr);
 	inet_sk(sk)->saddr     = htonl(conn->src_addr);
@@ -641,25 +572,19 @@ static int _sdp_inet_connect(struct sock
 	sdp_dbg_ctrl(conn, "CONNECT: family <%d> addr <%08x:%04x>",
 		     addr->sin_family, addr->sin_addr.s_addr, addr->sin_port);
 
-	if (size < sizeof(struct sockaddr_in)) {
-
+	if (size < sizeof(struct sockaddr_in))
 		return -EINVAL;
-	}
 
 	if (_proto_family != addr->sin_family &&
-	    AF_INET != addr->sin_family && AF_UNSPEC != addr->sin_family) {
-
+	    AF_INET != addr->sin_family && AF_UNSPEC != addr->sin_family)
 		return -EAFNOSUPPORT;
-	}
 
 	if (MULTICAST(addr->sin_addr.s_addr) ||
 	    BADCLASS(addr->sin_addr.s_addr) ||
 	    ZERONET(addr->sin_addr.s_addr) ||
 	    LOCAL_MCAST(addr->sin_addr.s_addr) ||
-	    INADDR_ANY == addr->sin_addr.s_addr) {
-
+	    INADDR_ANY == addr->sin_addr.s_addr)
 		return -EINVAL;
-	}
 	/*
 	 * lock socket
 	 */
@@ -667,18 +592,14 @@ static int _sdp_inet_connect(struct sock
 
 	switch (sock->state) {
 	case SS_UNCONNECTED:
-
 		if (0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
-
 			result = -EISCONN;
 			goto done;
 		}
 
 		if (0 == conn->src_port) {
-
 			result = sdp_inet_port_get(conn, 0);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, "Error <%d> getting port",
 					     result);
 				goto done;
@@ -706,7 +627,6 @@ static int _sdp_inet_connect(struct sock
 		 */
 		result = sdp_cm_connect(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> initiating connect",
 				     result);
 
@@ -727,15 +647,12 @@ static int _sdp_inet_connect(struct sock
 		result = -EINPROGRESS;
 		break;
 	case SS_CONNECTING:
-
 		result = -EALREADY;
 		break;
 	case SS_CONNECTED:
-
 		result = -EISCONN;
 		goto done;
 	default:
-
 		result = -EINVAL;
 		goto done;
 	}
@@ -785,18 +702,15 @@ static int _sdp_inet_connect(struct sock
 		break;
 	case SDP_SOCK_ST_ESTABLISHED:
 	case SDP_SOCK_ST_CLOSE:
-
 		sock->state = SS_CONNECTED;
 		result = 0;
 		break;
 	case SDP_SOCK_ST_CLOSED:
 	case SDP_SOCK_ST_ERROR:
-
 		result = SDP_CONN_ERROR(conn) ? : -ECONNABORTED;
 		sock->state = SS_UNCONNECTED;
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Unexpected state after connect. <%08x>",
 			     sock->state);
 		break;
@@ -832,26 +746,21 @@ static int _sdp_inet_listen(struct socke
 	if (SS_UNCONNECTED != sock->state ||
 	    (SDP_SOCK_ST_CLOSED != conn->istate &&
 	     SDP_SOCK_ST_LISTEN != conn->istate)) {
-
 		result = -EINVAL;
 		goto done;
 	}
 
 	if (SDP_SOCK_ST_LISTEN != conn->istate) {
-
 		result = sdp_inet_listen_start(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> starting listen",
 				     result);
 			goto done;
 		}
 
 		if (0 == conn->src_port) {
-
 			result = sdp_inet_port_get(conn, 0);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, "Error <%d> getting port",
 					     result);
 				goto done;
@@ -901,7 +810,6 @@ static int _sdp_inet_accept(struct socke
 	SDP_CONN_LOCK(listen_conn);
 
 	if (SDP_SOCK_ST_LISTEN != listen_conn->istate) {
-
 		result = -EINVAL;
 		goto listen_done;
 	}
@@ -917,7 +825,6 @@ static int _sdp_inet_accept(struct socke
 		 */
 		accept_conn = sdp_inet_accept_q_get(listen_conn);
 		if (NULL == accept_conn) {
-
 			DECLARE_WAITQUEUE(wait, current);
 			add_wait_queue(listen_sk->sk_sleep, &wait);
 			set_current_state(TASK_INTERRUPTIBLE);
@@ -925,15 +832,12 @@ static int _sdp_inet_accept(struct socke
 			while (0 < timeout &&
 			       SDP_SOCK_ST_LISTEN == listen_conn->istate &&
 			       0 == listen_conn->backlog_cnt) {
-
 				SDP_CONN_UNLOCK(listen_conn);
 				timeout = schedule_timeout(timeout);
 				SDP_CONN_LOCK(listen_conn);
 
-				if (signal_pending(current)) {
-
+				if (signal_pending(current))
 					break;
-				}
 			}
 
 			set_current_state(TASK_RUNNING);
@@ -944,30 +848,20 @@ static int _sdp_inet_accept(struct socke
 			if (0 == listen_conn->backlog_cnt) {
 				result = 0;
 
-				if (SDP_SOCK_ST_LISTEN !=
-				    listen_conn->istate) {
-
+				if (SDP_SOCK_ST_LISTEN != listen_conn->istate)
 					result = -EINVAL;
-				}
-				if (signal_pending(current)) {
-
+				if (signal_pending(current))
 					result = sock_intr_errno(timeout);
-				}
-				if (0 == timeout) {
-
+				if (0 == timeout)
 					result = -EAGAIN;
-				}
 
 				goto listen_done;
 			}
-		}
-		else {
-
+		} else {
 			accept_sk = accept_conn->sk;
 
 			switch (accept_conn->istate) {
 			case SDP_SOCK_ST_ACCEPTED:
-
 				sock_graft(accept_sk, accept_sock);
 
 				accept_conn->pid = current->pid;
@@ -979,7 +873,6 @@ static int _sdp_inet_accept(struct socke
 
 				break;
 			case SDP_SOCK_ST_ACCEPTING:
-
 				sock_graft(accept_sk, accept_sock);
 
 				accept_conn->pid = current->pid;
@@ -994,7 +887,6 @@ static int _sdp_inet_accept(struct socke
 
 				break;
 			case SDP_SOCK_ST_CLOSE:
-
 				sock_graft(accept_sk, accept_sock);
 
 				accept_conn->pid = current->pid;
@@ -1004,7 +896,6 @@ static int _sdp_inet_accept(struct socke
 
 				break;
 			default:
-
 				sdp_dbg_warn(accept_conn, "bad accept state");
 
 			case SDP_SOCK_ST_CLOSED:
@@ -1026,7 +917,7 @@ static int _sdp_inet_accept(struct socke
 				break;
 			}
 
-			if (NULL != accept_conn) {
+			if (NULL != accept_conn)
 				/*
 				 * Connections returned from the AcceptQueue
 				 * are holding their lock, before returning
@@ -1035,7 +926,6 @@ static int _sdp_inet_accept(struct socke
 				 */
 				/* AcceptQueueGet */
 				SDP_CONN_UNLOCK(accept_conn);
-			}
 		}
 	}
 
@@ -1078,21 +968,15 @@ static int _sdp_inet_getname(struct sock
 		     conn->dst_addr, conn->dst_port);
 
 	addr->sin_family = _proto_family;
-	if (0 < peer) {
-
+	if (0 < peer)
 		if (0 < htons(conn->dst_port) &&
 		    0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
 
 			addr->sin_port = htons(conn->dst_port);
 			addr->sin_addr.s_addr = htonl(conn->dst_addr);
-		}
-		else {
-
+		} else
 			return -ENOTCONN;
-		}
-	}
 	else {
-
 		addr->sin_port = htons(conn->src_port);
 		addr->sin_addr.s_addr = htonl(conn->src_addr);
 	}
@@ -1136,17 +1020,13 @@ static unsigned int _sdp_inet_poll(struc
 	 * no locking, should be safe as is.
 	 */
 	switch (conn->istate) {
-
 	case SDP_SOCK_ST_LISTEN:
-
 		mask |= (0 < conn->backlog_cnt) ? (POLLIN | POLLRDNORM) : 0;
 		break;
 	case SDP_SOCK_ST_ERROR:
-
 		mask |= POLLERR;
 		break;
 	case SDP_SOCK_ST_CLOSED:
-
 		mask |= POLLHUP;
 		break;
 	case SDP_SOCK_ST_ESTABLISHED:
@@ -1158,19 +1038,16 @@ static unsigned int _sdp_inet_poll(struc
 		 * recv EOF _and_ recv data
 		 */
 		if (!(conn->byte_strm < sk->sk_rcvlowat) ||
-		    0 < (RCV_SHUTDOWN & conn->shutdown)) {
+		    0 < (RCV_SHUTDOWN & conn->shutdown))
 
 			mask |= POLLIN | POLLRDNORM;
-		}
 		/*
 		 * send EOF _or_ send data space. 
 		 * (Some poll() Linux documentation says that POLLHUP is 
 		 *  incompatible with the POLLOUT/POLLWR flags)
 		 */
-		if (0 < (SEND_SHUTDOWN & conn->shutdown)) {
-
+		if (0 < (SEND_SHUTDOWN & conn->shutdown))
 			mask |= POLLHUP;
-		}
 		else {
 			/*
 			 * avoid race by setting flags, and only clearing
@@ -1182,7 +1059,6 @@ static unsigned int _sdp_inet_poll(struc
 			set_bit(SOCK_NOSPACE, &sock->flags);
 
 			if (0 < __sdp_inet_writable(conn)) {
-
 				mask |= POLLOUT | POLLWRNORM;
 
 				clear_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
@@ -1190,10 +1066,8 @@ static unsigned int _sdp_inet_poll(struc
 			}
 		}
 
-		if (0 < conn->rcv_urg_cnt) {
-
+		if (0 < conn->rcv_urg_cnt)
 			mask |= POLLPRI;
-		}
 	}
 
 	sdp_dbg_data(conn, "POLL: mask <%08x> flags <%08lx> <%d:%d:%d>", 
@@ -1231,12 +1105,9 @@ static int _sdp_inet_ioctl(struct socket
 		 * standard INET IOCTLs
 		 */
 	case SIOCGSTAMP:
-		if (0 == sk->sk_stamp.tv_sec) {
-
+		if (0 == sk->sk_stamp.tv_sec)
 			result = -ENOENT;
-		}
 		else {
-
 			result = copy_to_user((void __user *)arg,
 					      &sk->sk_stamp,
 					      sizeof(struct timeval));
@@ -1250,7 +1121,6 @@ static int _sdp_inet_ioctl(struct socket
 	case SIOCADDRT:
 	case SIOCDELRT:
 	case SIOCRTMSG:
-
 		result = ip_rt_ioctl(cmd, (void __user *)arg);
 		break;
 		/*
@@ -1279,14 +1149,12 @@ static int _sdp_inet_ioctl(struct socket
 	case SIOCSIFPFLAGS:
 	case SIOCGIFPFLAGS:
 	case SIOCSIFFLAGS:
-
 		result = devinet_ioctl(cmd, (void __user *)arg);
 		break;
 		/*
 		 * stadard INET STREAM IOCTLs
 		 */
 	case SIOCINQ:
-
 		SDP_CONN_LOCK(conn);
 
 		if (SDP_SOCK_ST_LISTEN != conn->istate) {
@@ -1295,45 +1163,33 @@ static int _sdp_inet_ioctl(struct socket
 			 */
 			value = conn->byte_strm;
 			result = put_user(value, (int __user *) arg);
-		}
-		else {
-
+		} else
 			result = -EINVAL;
-		}
 
 		SDP_CONN_UNLOCK(conn);
 		break;
 	case SIOCOUTQ:
-
 		SDP_CONN_LOCK(conn);
 
 		if (SDP_SOCK_ST_LISTEN != conn->istate) {
-
 			value = conn->send_qud;
 			result = put_user(value, (int __user *) arg);
-		}
-		else {
-
+		} else
 			result = -EINVAL;
-		}
 
 		SDP_CONN_UNLOCK(conn);
 		break;
 	case SIOCATMARK:
-
 		SDP_CONN_LOCK(conn);
 
 		value = 0;
 
 		if (0 < conn->rcv_urg_cnt) {
-
 			buff = sdp_buff_q_look_head(&conn->recv_pool);
 			if (NULL != buff &&
 			    0 < (SDP_BUFF_F_OOB_PRES & buff->flags) &&
-			    1 == (buff->tail - buff->data)) {
-
+			    1 == (buff->tail - buff->data))
 				value = 1;
-			}
 		}
 
 		result = put_user(value, (int __user *) arg);
@@ -1341,7 +1197,6 @@ static int _sdp_inet_ioctl(struct socket
 		SDP_CONN_UNLOCK(conn);
 		break;
 	default:
-
 		result = dev_ioctl(cmd, (void __user *)arg);
 		break;
 	}
@@ -1370,37 +1225,28 @@ static int _sdp_inet_setopt(struct socke
 	sdp_dbg_ctrl(conn, "SETSOCKOPT: level <%d> option <%d>", 
 		     level, optname);
 
-	if (SOL_TCP != level && SOL_SDP != level) {
-
+	if (SOL_TCP != level && SOL_SDP != level)
 		return 0;
-	}
-
-	if (optlen < sizeof(int)) {
 
+	if (optlen < sizeof(int))
 		return -EINVAL;
-	}
-
-	if (get_user(value, (int __user *)optval)) {
 
+	if (get_user(value, (int __user *)optval))
 		return -EFAULT;
-	}
 
 	SDP_CONN_LOCK(conn);
 
 	switch (optname) {
 	case TCP_NODELAY:
-
 		conn->nodelay = (0 == value) ? 0 : 1;
 
 		if (0 < conn->nodelay) {
-
 			result = sdp_send_flush(conn);
 			SDP_EXPECT(!(0 > result));
 		}
 
 		break;
 	case SDP_ZCOPY_THRSH:
-
 		conn->src_zthresh = value;
 		conn->snk_zthresh =
 		    ((value >
@@ -1409,11 +1255,9 @@ static int _sdp_inet_setopt(struct socke
 							SDP_MSG_HDR_SIZE));
 		break;
 	case SDP_ZCOPY_THRSH_SRC:
-
 		conn->src_zthresh = value;
 		break;
 	case SDP_ZCOPY_THRSH_SNK:
-
 		conn->snk_zthresh =
 		    ((value >
 		      (conn->recv_size -
@@ -1421,11 +1265,9 @@ static int _sdp_inet_setopt(struct socke
 							SDP_MSG_HDR_SIZE));
 		break;
 	case SDP_UNBIND:
-
 		result = sdp_inet_port_put(conn);
 		break;
 	default:
-
 		sdp_warn("SETSOCKOPT unimplemented option <%d:%d> conn <%d>.",
 			 level, optname, conn->hashent);
 		break;
@@ -1457,48 +1299,36 @@ static int _sdp_inet_getopt(struct socke
 	sdp_dbg_ctrl(conn, "GETSOCKOPT: level <%d> option <%d>",
 		     level, optname);
 
-	if (SOL_TCP != level && SOL_SDP != level) {
-
+	if (SOL_TCP != level && SOL_SDP != level)
 		return 0;
-	}
-
-	if (get_user(len, optlen)) {
 
+	if (get_user(len, optlen))
 		return -EFAULT;
-	}
 
 	len = min(len, (int)sizeof(int));
-	if (len < 0) {
-
+	if (len < 0)
 		return -EINVAL;
-	}
 
 	SDP_CONN_LOCK(conn);
 
 	switch (optname) {
 	case TCP_NODELAY:
-
 		value = (1 == conn->nodelay);
 		break;
 	case TCP_MAXSEG:
-
 		value = max(conn->send_size, (u16)1);
 		break;
 	case SDP_ZCOPY_THRSH:
-
 		value = ((conn->src_zthresh == conn->snk_zthresh) ?
 			 conn->snk_zthresh : -EPROTO);
 		break;
 	case SDP_ZCOPY_THRSH_SRC:
-
 		value = conn->src_zthresh;
 		break;
 	case SDP_ZCOPY_THRSH_SNK:
-
 		value = conn->snk_zthresh;
 		break;
 	default:
-
 		sdp_warn("GETSOCKOPT unimplemented option <%d:%d> conn <%d>",
 			 level, optname, conn->hashent);
 		break;
@@ -1506,15 +1336,11 @@ static int _sdp_inet_getopt(struct socke
 
 	SDP_CONN_UNLOCK(conn);
 
-	if (put_user(len, optlen)) {
-
+	if (put_user(len, optlen))
 		return -EFAULT;
-	}
-
-	if (copy_to_user(optval, &value, len)) {
 
+	if (copy_to_user(optval, &value, len))
 		return -EFAULT;
-	}
 
 	return 0;
 } /* _sdp_inet_getopt */
@@ -1537,14 +1363,10 @@ static int _sdp_inet_shutdown(struct soc
 	 *       1 - send shutdown
 	 *       2 - send/recv shutdown.
 	 */
-	if (0 > flag || 2 < flag) {
-
+	if (0 > flag || 2 < flag)
 		return -EINVAL;
-	}
-	else {
-
+	else
 		flag++;		/* match shutdown mask. */
-	}
 
 	SDP_CONN_LOCK(conn);
 
@@ -1552,7 +1374,6 @@ static int _sdp_inet_shutdown(struct soc
 
 	switch (conn->istate) {
 	case SDP_SOCK_ST_CLOSED:
-
 		result = -ENOTCONN;
 		break;
 	case SDP_SOCK_ST_LISTEN:
@@ -1560,41 +1381,30 @@ static int _sdp_inet_shutdown(struct soc
 		 * Send shutdown is benign.
 		 */
 		if (0 < (RCV_SHUTDOWN & flag)) {
-
 			result = sdp_inet_listen_stop(conn);
-			if (0 > result) {
-
+			if (0 > result)
 				sdp_dbg_warn(conn, "listen stop error <%d>",
 					     result);
-			}
 		}
 
 		break;
 	case SDP_SOCK_ST_ERROR:
-
 		result = SDP_CONN_ERROR(conn);
 		result = (result < 0) ? result : -ECONNABORTED;
 		break;
 	case SDP_SOCK_ST_ACCEPTED:
 	case SDP_SOCK_ST_CONNECT:
-
 		result = _sdp_inet_abort(conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> aborting connection",
 				     result);
-		}
-
 		break;
 	case SDP_SOCK_ST_ESTABLISHED:
 	case SDP_SOCK_ST_CLOSE:
-
 		result = _sdp_inet_disconnect(conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> disconnecting conn",
 				     result);
-		}
 
 		break;
 	case SDP_SOCK_ST_DISCONNECT:
@@ -1604,7 +1414,6 @@ static int _sdp_inet_shutdown(struct soc
 		 */
 		break;
 	case SDP_SOCK_ST_ACCEPTING:
-
 		sdp_dbg_warn(conn, "connection state error");
 
 		conn->istate = SDP_SOCK_ST_ERROR;
@@ -1613,7 +1422,6 @@ static int _sdp_inet_shutdown(struct soc
 
 		break;
 	default:
-
 		sdp_warn("Unknown socket state. conn <%d> state <%04x:%04x>",
 			 conn->hashent, conn->istate, conn->state);
 
@@ -1622,12 +1430,10 @@ static int _sdp_inet_shutdown(struct soc
 		result = -EFAULT;
 	}
 
-	if (0 > result) {
+	if (0 > result)
 		sdp_inet_wake_generic(sock->sk);
-	}
-	else {
+	else
 		sdp_inet_wake_error(sock->sk);
-	}
 
 	SDP_CONN_UNLOCK(conn);
 	return result;
@@ -1671,7 +1477,6 @@ static int _sdp_inet_create(struct socke
 	
 	if (SOCK_STREAM != sock->type ||
 	    (IPPROTO_IP != protocol && IPPROTO_TCP != protocol)) {
-
 		sdp_dbg_warn(NULL, "SOCKET: unsupported type/proto. <%d:%d>",
 			     sock->type, protocol);
 
@@ -1680,7 +1485,6 @@ static int _sdp_inet_create(struct socke
 
 	conn = sdp_conn_alloc(GFP_KERNEL);
 	if (NULL == conn) {
-		
 		sdp_dbg_warn(conn, "SOCKET: failed to create socekt <%d:%d>",
 			     sock->type, protocol);
 		return -ENOMEM;
@@ -1731,7 +1535,6 @@ static int __init sdp_init(void)
 	 */
 	result = sdp_main_proc_init();
 	if (0 > result) {
-
 		sdp_warn("INIT: Error <%d> creating proc entries.", result);
 		goto error_proc;
 	}
@@ -1740,7 +1543,6 @@ static int __init sdp_init(void)
 	 */
 	result = sdp_main_advt_init();
 	if (0 > result) {
-
 		sdp_warn("INIT: Error <%d> initializing advertisments",
 			 result);
 		goto error_advt;
@@ -1750,7 +1552,6 @@ static int __init sdp_init(void)
 	 */
 	result = sdp_link_addr_init();
 	if (0 > result) {
-
 		sdp_warn("INIT: Error <%d> initializing link",
 			 result);
 		goto error_link;
@@ -1763,7 +1564,6 @@ static int __init sdp_init(void)
 				    _alloc_inc, 
 				    _free_mark);
 	if (0 > result) {
-
 		sdp_warn("INIT: Error <%d> initializing buffer pool.", result);
 		goto error_buff;
 	}
@@ -1778,7 +1578,6 @@ static int __init sdp_init(void)
 				     _send_buff_max,
 				     _send_usig_max);
 	if (0 > result) {
-
 		sdp_warn("INIT: Error <%d> initializing connection table.",
 			 result);
 		goto error_conn;
@@ -1790,7 +1589,6 @@ static int __init sdp_init(void)
 
 	result = sock_register(&_sdp_proto);
 	if (result < 0) {
-
 		sdp_warn("INIT: Error <%d> registering protocol family <%d>",
 			 result, _sdp_proto.family);
 		goto error_sock;



From Tom.Duffy at Sun.COM  Fri Feb 18 16:45:44 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:45:44 -0800
Subject: [openib-general] [PATCH][SDP][8/22] Whitespace cleanup in sdp_iocb.c
In-Reply-To: <11087739341677@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
Message-ID: <11087739441194@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c	(working copy)
@@ -52,14 +52,11 @@ static int _sdp_iocb_unlock(struct sdpc_
 	struct vm_area_struct *vma;
 
 	vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK));
-	if (NULL == vma) {
-		
+	if (NULL == vma)
 		sdp_warn("No VMA for IOCB <%lx:%Zu> unlock",
 			 iocb->addr, iocb->size);
-	}
 
 	while (vma) {
-
 		sdp_dbg_data(NULL, 
 			     "unmark <%lx> <%p> <%08lx:%08lx> <%08lx> <%ld>",
 			     iocb->addr, vma, vma->vm_start, vma->vm_end,
@@ -88,14 +85,10 @@ static int _sdp_iocb_unlock(struct sdpc_
 		/*
 		 * continue if the buffer continues onto the next vma
 		 */
-		if ((iocb->addr + iocb->size) > vma->vm_end) {
-			
+		if ((iocb->addr + iocb->size) > vma->vm_end)
 			vma = vma->vm_next;
-		}
-		else {
-			
+		else
 			vma = NULL;
-		}
 	}
 
 	return 0;
@@ -112,10 +105,8 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 	/*
 	 * check if IOCB is locked.
 	 */
-	if (0 == (SDP_IOCB_F_LOCKED & iocb->flags)) {
-		
+	if (0 == (SDP_IOCB_F_LOCKED & iocb->flags))
 		return 0;
-	} /* if */
 	/*
 	 * spin lock since this could be from interrupt context.
 	 */
@@ -155,29 +146,21 @@ static int _sdp_iocb_page_save(struct sd
 	pte_t *ptep;
 	pte_t  pte;
 
-	if (!(0 < iocb->page_count) ||
-	    !(0 < iocb->size) ||
-	    0 == iocb->addr) {
-
+	if (!(0 < iocb->page_count) || !(0 < iocb->size) || 0 == iocb->addr)
 		return -EINVAL;
-	}
 	/*
 	 * create array to hold page value which are later needed to register
 	 * the buffer with the HCA
 	 */
 	iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count),
 				   GFP_KERNEL);
-	if (NULL == iocb->addr_array) {
-
+	if (NULL == iocb->addr_array)
 		goto err_addr;
-	}
 
 	iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), 
 				   GFP_KERNEL);
-	if (NULL == iocb->page_array) {
-    
+	if (NULL == iocb->page_array)
 		goto err_page;
-	}
 	/*
 	 * iocb->addr - buffer start address
 	 * iocb->size - buffer length
@@ -197,33 +180,27 @@ static int _sdp_iocb_page_save(struct sd
 	for (counter = 0;
 	     0 < size;
 	     counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) {
-
 		pgd = pgd_offset_gate(iocb->mm, addr);
-		if (!pgd || pgd_none(*pgd)) {
+		if (!pgd || pgd_none(*pgd))
 			break;
-		}
 		
 		pmd = pmd_offset(pgd, addr);
-		if (!pmd || pmd_none(*pmd)) {
+		if (!pmd || pmd_none(*pmd))
 			break;
-		}
 
 		ptep = pte_offset_map(pmd, addr);
-		if (!ptep) {
+		if (!ptep) 
 			break;
-		}
 
 		pte = *ptep;
 		pte_unmap(ptep);
 
-		if (!pte_present(pte)) {
+		if (!pte_present(pte))
 			break;
-		}
 
 		pfn = pte_pfn(pte);
-		if (!pfn_valid(pfn)) {
+		if (!pfn_valid(pfn))
 			break;
-		}
         
 		page = pfn_to_page(pfn);
 
@@ -234,7 +211,6 @@ static int _sdp_iocb_page_save(struct sd
 	spin_unlock(&iocb->mm->page_table_lock);
 	
 	if (size > 0) {
-		
 		result = -EFAULT;
 		goto err_find;
 	}
@@ -290,15 +266,12 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	 */
 	down_write(&iocb->mm->mmap_sem);
 
-	if (!((iocb->page_count + current->mm->locked_vm) > limit)) {
-          
+	if (!((iocb->page_count + current->mm->locked_vm) > limit))
 		result = (*_mlock_ptr)(addr, size, 1);
-	}
 	/*
 	 * process result
 	 */
 	if (result) {
-
 		sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>",
 			    iocb->addr, iocb->size, result, 
 			    iocb->page_count, iocb->mm->locked_vm, limit);
@@ -309,28 +282,22 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	 * them do not copy, reference counting, and saving them. 
 	 */
 	vma = find_vma(iocb->mm, addr);
-	if (NULL == vma) {
+	if (NULL == vma)
 		/*
 		 * sanity check.
 		 */
 		sdp_warn("No VMA for IOCB! <%lx:%Zu> lock",
 			 iocb->addr, iocb->size);
-	}
 
 	while (vma) {
-
 		spin_lock(&iocb->mm->page_table_lock);
 
-		if (0 == (VM_LOCKED & vma->vm_flags)) {
-
+		if (0 == (VM_LOCKED & vma->vm_flags))
 			sdp_warn("Unlocked vma! <%08lx>", vma->vm_flags);
-		}
-
-		if (PAGE_SIZE < (unsigned long)vma->vm_private_data) {
 
+		if (PAGE_SIZE < (unsigned long)vma->vm_private_data)
 			sdp_dbg_err("VMA: private daya in use! <%08lx>",
 				    (unsigned long)vma->vm_private_data);
-		}
 		
 		vma->vm_flags |= VM_DONTCOPY;
 		vma->vm_private_data++;
@@ -342,19 +309,14 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 			     iocb->addr, vma, vma->vm_start, vma->vm_end,
 			     vma->vm_flags, (long)vma->vm_private_data);
 
-		if ((addr + size) > vma->vm_end) {
-
+		if ((addr + size) > vma->vm_end)
 			vma = vma->vm_next;
-		}
-		else {
-
+		else
 			vma = NULL;
-		}
 	}
 
 	result = _sdp_iocb_page_save(iocb);
 	if (result) {
-
 		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", 
 			    result, iocb->addr, iocb->size);
 		goto err_save;
@@ -408,36 +370,26 @@ static int _sdp_mem_lock_init(void)
 	
 	kallsyms = filp_open("/proc/kallsyms", O_RDONLY, 0);
 	if (NULL == kallsyms) {
-
 		sdp_warn("Failed to open /proc/kallsyms");
 		goto done;
 	}
 
 	seq = (struct seq_file *)kallsyms->private_data;
 	if (NULL == seq) {
-
 		sdp_warn("Failed to fetch sequential file.");
 		goto err_close;
 	}
 
 	for (iter = seq->op->start(seq, &pos);
 	     iter != NULL;
-	     iter = seq->op->next(seq, iter, &pos)) {
-		
-		if (0 == strcmp(iter->name, "do_mlock")) {
-			
+	     iter = seq->op->next(seq, iter, &pos))
+		if (0 == strcmp(iter->name, "do_mlock"))
 			_mlock_ptr = (do_mlock_ptr_t)iter->value;
-		}
-	}
-
-	if (NULL == _mlock_ptr) {
 
+	if (NULL == _mlock_ptr)
 		sdp_warn("Failed to find lock pointer.");
-	}
-	else {
-		
+	else
 		ret = 0;
-	}
 
 err_close:
 	filp_close(kallsyms, NULL);
@@ -475,10 +427,8 @@ int sdp_iocb_register(struct sdpc_iocb *
 	/*
 	 * register only once.
 	 */
-	if (SDP_IOCB_F_REG & iocb->flags) {
-		
+	if (SDP_IOCB_F_REG & iocb->flags)
 		return 0;
-	}
 	/*
 	 * prime io address with physical address of first byte?
 	 */
@@ -491,18 +441,14 @@ int sdp_iocb_register(struct sdpc_iocb *
 					 iocb->page_count,
 					 &iocb->io_addr);
 	if (IS_ERR(iocb->mem)) {
-
 		result = (int)PTR_ERR(iocb->mem);
 
-		if (-EAGAIN != result) {
-
+		if (-EAGAIN != result)
 			sdp_dbg_err("Error <%d> fmr_pool_map_phys <%d:%d:%d>",
 				    result,
 				    iocb->len,
 				    iocb->page_count,
 				    iocb->page_offset);
-		}
-
 		goto error;
 	}
 	
@@ -531,17 +477,13 @@ int sdp_iocb_release(struct sdpc_iocb *i
 
 	SDP_CHECK_NULL(iocb, -EINVAL);
 
-	if (0 == (SDP_IOCB_F_REG & iocb->flags)) {
-
+	if (0 == (SDP_IOCB_F_REG & iocb->flags))
 		return 0;
-	}
 
 	result = ib_fmr_pool_unmap(iocb->mem);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>",
 			    result, iocb->key, iocb->addr);
-	}
 
 	iocb->flags &= ~(SDP_IOCB_F_REG);
 
@@ -560,20 +502,16 @@ static void _sdp_iocb_complete(void *arg
 	 * release memory
 	 */
 	result = sdp_iocb_release(iocb);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>",
 			    result, iocb->key, iocb->addr);
-	}
 	/*
 	 * unlock now, after aio_complete the mm reference will be released.
 	 */
 	result = sdp_iocb_unlock(iocb);
-	if (0 > result) {
-		
+	if (0 > result)
 		sdp_dbg_err("Error <%d> unlocking IOCB <%d memory <%ld>>",
 			    result, iocb->key, iocb->addr);
-	}
 	/*
 	 * callback to complete IOCB
 	 */
@@ -591,11 +529,9 @@ static void _sdp_iocb_complete(void *arg
 	 * delete IOCB
 	 */
 	result = sdp_iocb_destroy(iocb);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
-	}
 
 	return;
 } /* _sdp_iocb_complete */
@@ -610,17 +546,10 @@ int sdp_iocb_complete(struct sdpc_iocb *
 	iocb->status = status;
 	
 	if (in_atomic() || irqs_disabled()) {
-		
-		INIT_WORK(&iocb->completion,
-			  _sdp_iocb_complete,
-			  (void *)iocb);
-    
+		INIT_WORK(&iocb->completion, _sdp_iocb_complete, (void *)iocb);
 		schedule_work(&iocb->completion);
-	}
-	else {
-
+	} else
 		_sdp_iocb_complete(iocb);
-	}
 	
 	return 0;
 } /* sdp_iocb_complete */
@@ -646,21 +575,16 @@ int sdp_iocb_q_remove(struct sdpc_iocb *
 
 	table = iocb->table;
 
-	if (iocb->next == iocb && iocb->prev == iocb) {
-
+	if (iocb->next == iocb && iocb->prev == iocb)
 		table->head = NULL;
-	}
 	else {
-
 		next = iocb->next;
 		prev = iocb->prev;
 		next->prev = prev;
 		prev->next = next;
 
-		if (table->head == iocb) {
-
+		if (table->head == iocb)
 			table->head = next;
-		}
 	}
 
 	table->size--;
@@ -682,15 +606,10 @@ struct sdpc_iocb *sdp_iocb_q_lookup(stru
 
 	SDP_CHECK_NULL(table, NULL);
 
-	for (counter = 0, iocb = table->head;
-	     counter < table->size;
-	     counter++, iocb = iocb->next) {
-
-		if (iocb->key == key) {
-
+	for (counter = 0, iocb = table->head; counter < table->size;
+	     counter++, iocb = iocb->next)
+		if (iocb->key == key)
 			return iocb;
-		}
-	}
 
 	return NULL;
 } /* sdp_iocb_q_lookup */
@@ -704,7 +623,6 @@ struct sdpc_iocb *sdp_iocb_create(void)
 
 	iocb = kmem_cache_alloc(__sdp_iocb_cache, SLAB_KERNEL);
 	if (NULL != iocb) {
-
 		memset(iocb, 0, sizeof(struct sdpc_iocb));
 		/*
 		 * non-zero initialization
@@ -722,16 +640,11 @@ struct sdpc_iocb *sdp_iocb_create(void)
  */
 int sdp_iocb_destroy(struct sdpc_iocb *iocb)
 {
-	if (NULL == iocb) {
-		
+	if (NULL == iocb)
 		return -EINVAL;
-	}
-
-	if (NULL != iocb->next ||
-	    NULL != iocb->prev) {
 
+	if (NULL != iocb->next || NULL != iocb->prev)
 		return -EACCES;
-	}
 	/*
 	 * release iocb registered memory
 	 */
@@ -743,15 +656,11 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
 	/*
 	 * array dealloc
 	 */
-	if (NULL != iocb->page_array) {
-
+	if (NULL != iocb->page_array)
 		kfree(iocb->page_array);
-	}
 
-	if (NULL != iocb->addr_array) {
-		
+	if (NULL != iocb->addr_array)
 		kfree(iocb->addr_array);
-	}
 	/*
 	 * clear IOCB to check for usage after free...
 	 */
@@ -787,26 +696,17 @@ static struct sdpc_iocb *_sdp_iocb_q_get
 
 	SDP_CHECK_NULL(table, NULL);
 
-	if (NULL == table->head) {
-
+	if (NULL == table->head)
 		return NULL;
-	}
-
-	if (head) {
 
+	if (head)
 		iocb = table->head;
-	}
-	else {
-
+	else
 		iocb = table->head->prev;
-	}
-
-	if (iocb->next == iocb && iocb->prev == iocb) {
 
+	if (iocb->next == iocb && iocb->prev == iocb)
 		table->head = NULL;
-	}
 	else {
-
 		next = iocb->next;
 		prev = iocb->prev;
 		next->prev = prev;
@@ -838,13 +738,10 @@ static int _sdp_iocb_q_put(struct sdpc_i
 	SDP_CHECK_NULL(iocb, -EINVAL);
 
 	if (NULL == table->head) {
-
 		iocb->next = iocb;
 		iocb->prev = iocb;
 		table->head = iocb;
-	}
-	else {
-
+	} else {
 		next = table->head;
 		prev = next->prev;
 
@@ -853,9 +750,8 @@ static int _sdp_iocb_q_put(struct sdpc_i
 		iocb->next = next;
 		next->prev = iocb;
 
-		if (head) {
+		if (head)
 			table->head = iocb;
-		}
 	}
 
 	table->size++;
@@ -917,11 +813,9 @@ int sdp_iocb_q_cancel(struct sdpc_iocb_q
 	 */
 	for (counter = 0, iocb = table->head, total = table->size;
 	     counter < total; counter++) {
-
 		next = iocb->next;
 
 		if (0 < (mask & iocb->flags) || SDP_IOCB_F_ALL == mask) {
-
 			sdp_dbg_err("IOCB <%d> cancel <%Zu> flag <%04x> "
 				    "size <%Zu:%d:%d>",
 				    iocb->key, comp, iocb->flags, iocb->size,
@@ -966,7 +860,6 @@ int sdp_iocb_q_clear(struct sdpc_iocb_q 
 	 * drain the table of any objects
 	 */
 	while (NULL != (iocb = sdp_iocb_q_get_head(table))) {
-
 		result = sdp_iocb_destroy(iocb);
 		SDP_EXPECT(!(0 > result));
 	}
@@ -991,7 +884,6 @@ int sdp_main_iocb_init(void)
 	 */
 	result =  _sdp_mem_lock_init();
 	if (0 > result) {
-		
 		sdp_warn("Error <%d> initializing memory locking.", result);
 		return result;
 	} /* if */
@@ -999,7 +891,6 @@ int sdp_main_iocb_init(void)
 	 * initialize the caches only once.
 	 */
 	if (NULL != __sdp_iocb_cache) {
-
 		sdp_warn("IOCB caches already initialized.");
 		return -EINVAL;
 	}
@@ -1009,7 +900,6 @@ int sdp_main_iocb_init(void)
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
 	if (NULL == __sdp_iocb_cache) {
-
 		result = -ENOMEM;
 		goto error_iocb_c;
 	}



From Tom.Duffy at Sun.COM  Fri Feb 18 16:45:55 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:45:55 -0800
Subject: [openib-general] [PATCH][SDP][9/22] Whitespace cleanup in sdp_kvec.c
In-Reply-To: <11087739441194@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com> <11087739441194@sun.com>
Message-ID: <11087739553583@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_kvec.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_kvec.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_kvec.c	(working copy)
@@ -80,7 +80,6 @@ static int _sdp_iocb_q_cancel_read_sourc
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->r_src,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
-
 		result = sdp_iocb_q_remove(iocb);
 		SDP_EXPECT(!(0 > result));
 
@@ -116,7 +115,6 @@ static int _sdp_iocb_q_cancel_write_pend
 		(struct sdpc_iocb *) sdp_desc_q_lookup(&conn->send_queue,
 						    _sdp_iocb_q_cancel_lookup_func,
 						    NULL))) {
-
 		result = sdp_iocb_q_remove(iocb);
 		SDP_EXPECT(!(0 > result));
 
@@ -150,7 +148,6 @@ static int _sdp_iocb_q_cancel_write_snk(
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->w_snk,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
-
 		result = sdp_iocb_q_remove(iocb);
 		SDP_EXPECT(!(0 > result));
 



From Tom.Duffy at Sun.COM  Fri Feb 18 16:46:05 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:46:05 -0800
Subject: [openib-general] [PATCH][SDP][10/22] Whitespace cleanup in
	sdp_link.c
In-Reply-To: <11087739553583@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
Message-ID: <11087739651393@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_link.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_link.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_link.c	(working copy)
@@ -106,10 +106,8 @@ static void _sdp_path_wait_add(struct sd
 	info->wait_list = wait;
 	wait->pext      = &info->wait_list;
 
-	if (NULL != wait->next) {
-
+	if (NULL != wait->next)
 		wait->next->pext = &wait->next;
-	}
 
 	return;
 } /* _sdp_path_wait_add */
@@ -123,10 +121,8 @@ static void _sdp_path_wait_destroy(struc
 	 * if it's in the list, pext will not be null
 	 */
 	if (NULL != wait->pext) {
-		
-		if (NULL != wait->next) {
+		if (NULL != wait->next)
 			wait->next->pext = wait->pext;
-		}
 
 		*(wait->pext) = wait->next;
 
@@ -164,14 +160,9 @@ static struct sdp_path_info *_sdp_path_i
 {
 	struct sdp_path_info *info;
 
-	for (info = _info_list; NULL != info; info = info->next) {
-
-		if (dst_ip == info->dst &&
-		    dev_if == info->dif) {
-
+	for (info = _info_list; NULL != info; info = info->next)
+		if (dst_ip == info->dst && dev_if == info->dif)
 			break;
-		}
-	}
 
 	return info;
 } /* _sdp_path_info_lookup */
@@ -184,10 +175,8 @@ static struct sdp_path_info *_sdp_path_i
 	struct sdp_path_info *info;
 
 	info = kmem_cache_alloc(_info_cache, SLAB_KERNEL);
-	if (NULL == info) {
-
+	if (NULL == info)
 		return NULL;
-	}
 
 	memset(info, 0, sizeof(struct sdp_path_info));
 
@@ -195,10 +184,8 @@ static struct sdp_path_info *_sdp_path_i
 	_info_list = info;
 	info->pext = &_info_list;
 
-	if (NULL != info->next) {
-
+	if (NULL != info->next)
 		info->next->pext = &info->next;
-	}
 
 	info->dst = dst_ip;
 	info->dif = dev_if;
@@ -221,10 +208,8 @@ static void _sdp_path_info_destroy(struc
 	 * if it's in the list, pext will not be null
 	 */
 	if (NULL != info->pext) {
-		
-		if (NULL != info->next) {
+		if (NULL != info->next)
 			info->next->pext = info->pext;
-		}
 
 		*(info->pext) = info->next;
 
@@ -232,10 +217,8 @@ static void _sdp_path_info_destroy(struc
 		info->next = NULL;
 	}
 
-	while (NULL != (wait = info->wait_list)) {
-
+	while (NULL != (wait = info->wait_list))
 		_sdp_path_wait_complete(wait, info, status);
-	}
 
 	cancel_delayed_work(&info->timer);
 	kmem_cache_free(_info_cache, info);
@@ -271,27 +254,21 @@ static void _sdp_link_path_rec_done(int 
 
 	sweep = info->wait_list;
 	while (NULL != sweep) {
-
 		wait  = sweep;
 		sweep = sweep->next;
 		/*
 		 * on timeout increment retries.
 		 */
-		if (-ETIMEDOUT == status) {
-
+		if (-ETIMEDOUT == status)
 			wait->retry++;
-		}
-
-		if (!status || SDP_LINK_SA_RETRY < wait->retry) {
 
+		if (!status || SDP_LINK_SA_RETRY < wait->retry)
 			_sdp_path_wait_complete(wait, info, status);
-		}
 	}
 	/*
 	 * retry if anyone is waiting.
 	 */
 	if (NULL != info->wait_list) {
-
 		info->sa_time = min(info->sa_time * 2, SDP_LINK_SA_TIME_MAX);
 
 		result = ib_sa_path_rec_get(info->ca,
@@ -308,7 +285,6 @@ static void _sdp_link_path_rec_done(int 
 					    &info->query);
 
 	        if (0 > result) {
-
 			sdp_dbg_warn(NULL, "Error <%d> restarting path query",
 				     result);
 			_sdp_path_info_destroy(info, result);
@@ -329,10 +305,8 @@ static int _sdp_link_path_rec_get(struct
 		     GID_ARG(info->path.sgid),
 		     GID_ARG(info->path.dgid));
 
-	if (SDP_LINK_F_PATH & info->flags) {
-
+	if (SDP_LINK_F_PATH & info->flags)
 		return 0;
-	}
 
 	result = ib_sa_path_rec_get(info->ca,
 				    info->port,
@@ -347,13 +321,10 @@ static int _sdp_link_path_rec_get(struct
 				    info,
 				    &info->query);
         if (0 > result) {
-
 		sdp_dbg_warn(NULL, "Error <%d> starting path record query",
 			     result);
                 info->query = NULL;
-        }
-	else {
-
+        } else {
 		info->qid    = result;
 		info->flags |= SDP_LINK_F_PATH;
 	}
@@ -391,21 +362,16 @@ void _sdp_link_path_lookup(void *data)
 	/*
 	 * path request in progress?
 	 */
-	if (NULL != info->query) {
-		
+	if (NULL != info->query)
 		goto done;
-	}
 	/*
 	 * route information present, but no path query, goto re-arp.
 	 */
-	if (NULL != info->ca) {
-
+	if (NULL != info->ca)
 		goto arp;
-	}
 
 	result = ip_route_output_key(&rt, &fl);
 	if (0 > result || NULL == rt) {
-
 		sdp_dbg_warn(NULL, "Error <%d> routing <%08x:%08x> (%d)",
 			     result, info->dst, info->src, info->dif);
 		goto error;
@@ -414,7 +380,6 @@ void _sdp_link_path_lookup(void *data)
 	 * check route flags
 	 */
 	if (0 < ((RTCF_MULTICAST|RTCF_BROADCAST) & rt->rt_flags)) {
-
 		ip_rt_put(rt);
 		result = -ENETUNREACH;
 		goto error;
@@ -422,9 +387,7 @@ void _sdp_link_path_lookup(void *data)
 	/*
 	 * check that device is IPoIB
 	 */
-	if (NULL == rt->u.dst.neighbour ||
-	    NULL == rt->u.dst.neighbour->dev) {
-
+	if (NULL == rt->u.dst.neighbour || NULL == rt->u.dst.neighbour->dev) {
 		sdp_dbg_warn(NULL, "No neighbour found for <%08x:%08x>",
 			     rt->rt_src, rt->rt_dst);
 
@@ -437,7 +400,6 @@ void _sdp_link_path_lookup(void *data)
 	 */
 	if (ARPHRD_INFINIBAND != rt->u.dst.neighbour->dev->type &&
 	    0 == (IFF_LOOPBACK & rt->u.dst.neighbour->dev->flags)) {
-
 		result = -ENETUNREACH;
 		goto error;
 	}
@@ -457,21 +419,16 @@ void _sdp_link_path_lookup(void *data)
 	info->gw  = rt->rt_gateway;
 	info->src = rt->rt_src;     /* true source IP address */
 
-	if (IFF_LOOPBACK & info->dev->flags) {
-
+	if (IFF_LOOPBACK & info->dev->flags)
 		while (NULL != (info->dev = dev_get_by_index(++counter))) {
 
 			dev_put(info->dev);
 			if (ARPHRD_INFINIBAND == info->dev->type &&
-			    0 < (IFF_UP & info->dev->flags)) {
-				
+			    0 < (IFF_UP & info->dev->flags))
 				break;
-			}
 		}
-	}
 
 	if (NULL == info->dev) {
-
 		sdp_dbg_warn(NULL, "No device for IB comm <%s:%08x:%08x>",
 			     rt->u.dst.neighbour->dev->name,
 			     rt->u.dst.neighbour->dev->flags,
@@ -497,7 +454,6 @@ void _sdp_link_path_lookup(void *data)
 	 * the IB device which was found.
 	 */
 	if (IFF_LOOPBACK & rt->u.dst.neighbour->dev->flags) {
-
 		memcpy(&info->path.dgid,
 		       (info->dev->dev_addr + 4),
 		       sizeof(union ib_gid));
@@ -508,7 +464,6 @@ void _sdp_link_path_lookup(void *data)
 
 	if ((NUD_CONNECTED|NUD_DELAY|NUD_PROBE) &
 	    rt->u.dst.neighbour->nud_state) {
-
 		memcpy(&info->path.dgid,
 		       (rt->u.dst.neighbour->ha + 4),
 		       sizeof(union ib_gid));
@@ -519,7 +474,6 @@ void _sdp_link_path_lookup(void *data)
 	 * No address entry, either ARP inprogress or needs to be issued.
 	 */
 	if (NUD_INCOMPLETE & rt->u.dst.neighbour->nud_state) {
-
 		result = 0;
 		goto done;
 	}
@@ -541,19 +495,15 @@ arp:
 
 		sweep = info->wait_list;
 		while (NULL != sweep) {
-
 			wait = sweep;
 			sweep = sweep->next;
 
-			if (SDP_LINK_SA_RETRY < wait->retry++) {
-
+			if (SDP_LINK_SA_RETRY < wait->retry++)
 				_sdp_path_wait_complete(wait, info, 
 							-ETIMEDOUT);
-			}
 		}
 
 		if (NULL == info->wait_list) {
-
 			result = -ETIMEDOUT;
 			goto error;
 		}
@@ -569,7 +519,6 @@ arp:
 path:
 	result = _sdp_link_path_rec_get(info);
 	if (result) {
-
 		sdp_dbg_warn(NULL, "Error <%d> getting path record.", result);
 		goto error;
 	}
@@ -611,10 +560,8 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	 */
 	info = _sdp_path_info_lookup(dst_addr, bound_dev_if);
 	if (NULL == info) {
-
 		info = _sdp_path_info_create(dst_addr, bound_dev_if);
 		if (NULL == info) {
-
 			sdp_dbg_warn(NULL, "Failed to create path object");
 			return -ENOMEM;
 		}
@@ -626,7 +573,6 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	 * if not waiting for result, complete.
 	 */
 	if (SDP_LINK_F_VALID & info->flags) {
-
 		_sdp_link_path_complete(*id, 0, info, completion, arg);
 		return 0;
 	}
@@ -635,7 +581,6 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	 */
 	wait = kmem_cache_alloc(_wait_cache, SLAB_KERNEL);
 	if (NULL == wait) {
-
 		sdp_dbg_warn(NULL, "Failed to create path wait object");
 		result = -ENOMEM;
 		goto error;
@@ -649,10 +594,8 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	/*
 	 * initiate address lookup, if not in progress.
 	 */
-	if (!((SDP_LINK_F_ARP|SDP_LINK_F_PATH) & info->flags)) {
-
+	if (!((SDP_LINK_F_ARP|SDP_LINK_F_PATH) & info->flags))
 		_sdp_link_path_lookup(info);
-	}
 
 	return 0;
 error:
@@ -673,12 +616,10 @@ static void _sdp_link_sweep(void *data)
 
 	sweep = _info_list; 
 	while (NULL != sweep) {
-
 		info  = sweep;
 		sweep = sweep->next;
 
 		if (jiffies > (info->use + SDP_LINK_INFO_TIMEOUT)) {
-
 			sdp_dbg_ctrl(NULL, 
 				     "info delete <%d.%d.%d.%d> <%lu:%lu>",
 				     (info->dst & 0x000000ff),
@@ -721,23 +662,16 @@ static void _sdp_link_arp_work(void *dat
 	/*
 	 * find a path info structure for the source IP address.
 	 */
-	for (info = _info_list; NULL != info; info = info->next) {
-		
-		if (info->dst == arp->src_ip) {
-
+	for (info = _info_list; NULL != info; info = info->next)
+		if (info->dst == arp->src_ip)
 			break;
-		}
-	}
-
-	if (NULL == info) {
 
+	if (NULL == info)
 		goto done;
-	}
 	/*
 	 * update record info, and request new path record data.
 	 */
 	if (SDP_LINK_F_ARP & info->flags) {
-
 		cancel_delayed_work(&info->timer);
 		info->flags &= ~SDP_LINK_F_ARP;
 	}
@@ -746,7 +680,6 @@ static void _sdp_link_arp_work(void *dat
 
 	result = _sdp_link_path_rec_get(info);
 	if (result) {
-
 		sdp_dbg_warn(NULL, "Error <%d> path request on ARP.", result);
 		_sdp_path_info_destroy(info, result);
 	}
@@ -771,18 +704,14 @@ static int _sdp_link_arp_recv(struct sk_
 
 	if (ARPHRD_INFINIBAND != dev->type ||
 	    (arp_hdr->ar_op != __constant_htons(ARPOP_REPLY) &&
-	     arp_hdr->ar_op != __constant_htons(ARPOP_REQUEST))) {
-
+	     arp_hdr->ar_op != __constant_htons(ARPOP_REQUEST)))
 		goto done;
-	}
 	/*
 	 * queue IB arp packet onto work queue.
 	 */
 	work = kmalloc(sizeof(*work), GFP_ATOMIC);
-	if (NULL == work) {
-	
+	if (NULL == work)
 		goto done;
-	}
 
 	work->arg = skb;
 	INIT_WORK(&work->work, _sdp_link_arp_work, work);
@@ -829,7 +758,6 @@ int sdp_link_addr_init(void)
 					0, SLAB_HWCACHE_ALIGN,
 					NULL, NULL);
 	if (NULL == _info_cache) {
-
 		sdp_warn("Failed to allocate path info cache.");
 
 		result = -ENOMEM;
@@ -841,7 +769,6 @@ int sdp_link_addr_init(void)
 					0, SLAB_HWCACHE_ALIGN,
 					NULL, NULL);
 	if (NULL == _wait_cache) {
-
 		sdp_warn("Failed to allocate path wait cache.");
 
 		result = -ENOMEM;
@@ -850,7 +777,6 @@ int sdp_link_addr_init(void)
 
 	_link_wq = create_workqueue("sdp_wq");
 	if (NULL == _link_wq) {
-
 		sdp_warn("Failed to allocate ARP wait queue.");
 
 		result = -ENOMEM;
@@ -886,10 +812,8 @@ int sdp_link_addr_cleanup(void)
 	/*
 	 * clear objects
 	 */
-	while (NULL != (info = _info_list)) {
-
+	while (NULL != (info = _info_list))
 		_sdp_path_info_destroy(info, -EINTR);
-	}
 	/*
 	 * remove ARP packet processing.
 	 */



From Tom.Duffy at Sun.COM  Fri Feb 18 16:46:15 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:46:15 -0800
Subject: [openib-general] [PATCH][SDP][11/22] Whitespace cleanup in
	sdp_pass.c
In-Reply-To: <11087739651393@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
Message-ID: <11087739751674@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_pass.c	(working copy)
@@ -52,7 +52,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	 * build listen response headers
 	 */
 	if (sizeof(struct msg_hello_ack) > conn->recv_size) {
-
 		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct msg_hello_ack));
 		result = -ENOBUFS;
@@ -64,7 +63,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buff for Hello Ack.");
 		result = -ENOMEM;
 		goto error;
@@ -100,7 +98,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	 */
 	result = sdp_buff_q_put(&conn->send_post, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> buffering hello ack packet.",
 			     result);
 
@@ -114,7 +111,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	 */
         qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
         if (NULL == qp_attr) {
-
 		sdp_dbg_warn(conn, "Failed to allocate QP attribute.");
                 result = -ENOMEM;
 		goto error;
@@ -126,7 +122,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	
 	result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &qp_mask);
 	if (result) {
-
 		sdp_dbg_warn(conn, "Error <%d> QP attributes for RTR",
 			     result);
 		goto error;
@@ -141,7 +136,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	kfree(qp_attr);
 	
 	if (result) {
-
 		sdp_dbg_warn(conn, "Error <%d> modifying QP to RTR.", result);
 		goto error;
         }
@@ -154,7 +148,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	 */
 	result = sdp_recv_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing receive queue",
 			     result);
 		goto error;
@@ -178,7 +171,6 @@ static int _sdp_cm_accept(struct sdp_opt
 
 	result = ib_send_cm_rep(conn->cm_id, &param);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> CM accept request.", result);
 		goto error;
 	}
@@ -226,7 +218,6 @@ static int _sdp_cm_listen_lookup(struct 
 	sk = conn->sk;
 
 	if (listen_conn->backlog_cnt > listen_conn->backlog_max) {
-
 		sdp_dbg_warn(listen_conn, 
 			     "Listen backlog <%d> too big to accept new conn",
 			     listen_conn->backlog_cnt);
@@ -236,7 +227,6 @@ static int _sdp_cm_listen_lookup(struct 
 
 	result = sdp_inet_port_inherit(listen_conn, conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(listen_conn, "Error <%d> listen port inherit.",
 			     result);
 		result = -EFAULT;
@@ -283,7 +273,6 @@ static int _sdp_cm_listen_lookup(struct 
 	 */
 	result = _sdp_cm_accept(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> CM connect accept", result);
 		goto locked_err;
 	}
@@ -292,7 +281,6 @@ static int _sdp_cm_listen_lookup(struct 
 	 */
 	result = sdp_inet_accept_q_put(listen_conn, conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, 
 			     "Error <%d> adding socket to accept queue",
 			     result);
@@ -324,28 +312,24 @@ static int _sdp_cm_hello_check(struct ms
 	 * validation and consistency checks
 	 */
 	if (msg_hello->bsdh.size != sizeof(struct msg_hello)) {
-
 		sdp_dbg_warn(NULL, "hello msg size mismatch. (2) <%d:%Zu>",
 			     msg_hello->bsdh.size, sizeof(struct msg_hello));
 		return -EINVAL;
 	}
 
 	if (SDP_MID_HELLO != msg_hello->bsdh.mid) {
-
 		sdp_dbg_warn(NULL, "hello msg unexpected ID. <%d>",
 			     msg_hello->bsdh.mid);
 		return -EINVAL;
 	}
 
 	if (!(0 < msg_hello->hh.max_adv)) {
-
 		sdp_dbg_warn(NULL, "hello msg, bad zcopy count <%d>",
 			     msg_hello->hh.max_adv);
 		return -EINVAL;
 	}
 
 	if ((0xF0 & msg_hello->hh.version) != (0xF0 & SDP_MSG_VERSION)) {
-
 		sdp_dbg_warn(NULL, "hello msg, version mismatch. <%d:%d>",
 			     ((0xF0 & msg_hello->hh.version) >> 4),
 			     ((0xF0 & SDP_MSG_VERSION) >> 4));
@@ -356,7 +340,6 @@ static int _sdp_cm_hello_check(struct ms
 #else
 	if ((SDP_MSG_IPVER & 0xF0) != (msg_hello->hh.ip_ver & 0xF0)) {
 #endif
-
 		sdp_dbg_warn(NULL, "hello msg, ip version mismatch. <%d:%d>",
 			     msg_hello->hh.ip_ver, SDP_MSG_IPVER);
 		return -EINVAL;
@@ -402,7 +385,6 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 */
 	result = _sdp_cm_hello_check(msg_hello);
 	if (0 > result) {
-
 		sdp_dbg_warn(NULL, "Error <%d> validating hello msg. <%08x>",
 			     result, cm_id->local_id);
 		goto done;
@@ -412,7 +394,6 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 */
 	conn = sdp_conn_alloc(GFP_KERNEL); /* CM sk reference */
 	if (NULL == conn) {
-
 		sdp_dbg_warn(NULL, "Failed to allocate connection. <%08x>",
 			     cm_id->local_id);
 		result = -ENOMEM;
@@ -461,7 +442,6 @@ int sdp_cm_req_handler(struct ib_cm_id *
 				   event->param.req_rcvd.port,
 				   event->param.req_rcvd.primary_path->pkey);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> binding connection to HCA/port",
 			     result);
 		goto error;
@@ -480,7 +460,6 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 */
 	result = _sdp_cm_listen_lookup(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> matching listen socket queue",
 			     result);
 		goto error;



From Tom.Duffy at Sun.COM  Fri Feb 18 16:46:25 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:46:25 -0800
Subject: [openib-general] [PATCH][SDP][12/22] Whitespace cleanup in
	sdp_post.c
In-Reply-To: <11087739751674@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com> <11087739751674@sun.com>
Message-ID: <11087739852191@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_post.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_post.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_post.c	(working copy)
@@ -43,17 +43,13 @@ int sdp_cm_listen_start(struct sdev_root
 
 	SDP_CHECK_NULL(dev_root, -EINVAL);
 
-	if (NULL != dev_root->listen_id) {
-
+	if (NULL != dev_root->listen_id)
 		sdp_dbg_warn(NULL, "Already listening for connections.");
-	}
 
 	dev_root->listen_id = ib_create_cm_id(sdp_cm_event_handler,
 					      (void *)SDP_DEV_SK_INVALID);
-	if (NULL == dev_root->listen_id) {
-
+	if (NULL == dev_root->listen_id)
 		return -ENOMEM;
-	}
 	/*
 	 * start listening
 	 */
@@ -61,17 +57,13 @@ int sdp_cm_listen_start(struct sdev_root
 		              cpu_to_be64(SDP_MSG_SERVICE_ID_VALUE),
 			      cpu_to_be64(SDP_MSG_SERVICE_ID_MASK));
 	if (0 > result) {
-
 		sdp_dbg_warn(NULL, "Error <%d> listening for SDP connections",
 			     result);
 
 		(void)ib_destroy_cm_id(dev_root->listen_id);
 		dev_root->listen_id = NULL;
-	}
-	else {
-
+	} else
 		sdp_dbg_init("Started listening for SDP connection requests");
-	}
 
 	return result;
 } /* sdp_cm_listen_start */
@@ -86,14 +78,10 @@ int sdp_cm_listen_stop(struct sdev_root 
 	SDP_CHECK_NULL(dev_root, -EINVAL);
 
 	result = ib_destroy_cm_id(dev_root->listen_id);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(NULL, "Error <%d> stopping listen", result);
-	}
-	else {
-
+	else
 		sdp_dbg_init("Stopped listening for SDP connections");
-	}
 
 	dev_root->listen_id = NULL;
 
@@ -109,7 +97,6 @@ static void _sdp_cm_disconnect(void *arg
 	int result;
 	
 	if (NULL == conn) {
-
 		sdp_dbg_warn(NULL, "Error, posting disconnect for NULL conn");
 		return;
 	}
@@ -119,10 +106,8 @@ static void _sdp_cm_disconnect(void *arg
 	 * send a disconnect request using the connection manager
 	 */
 	result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(NULL, "Error <%d> CM disconnect request", result);
-	}
 
 	SDP_CONN_PUT(conn);
 
@@ -137,10 +122,8 @@ static void _sdp_cm_reject(void *arg)
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
 
-	if (NULL == conn) {
-
+	if (NULL == conn)
 		sdp_dbg_warn(NULL, "Error, posting reject for NULL conn");
-	}
 
 	sdp_dbg_ctrl(conn, "Defered reject <%08x>", conn->cm_id->local_id);
 	/*
@@ -149,10 +132,8 @@ static void _sdp_cm_reject(void *arg)
 	result = ib_send_cm_rej(conn->cm_id, 
 				IB_CM_REJ_CONSUMER_DEFINED,
 				NULL, 0, NULL, 0);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(NULL, "Error <%d> CM reject request", result);
-	}
 
 	SDP_CONN_PUT(conn);
 
@@ -167,20 +148,16 @@ static void _sdp_cm_confirm(void *arg)
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
 
-	if (NULL == conn) {
-
+	if (NULL == conn)
 		sdp_dbg_warn(NULL, "Error, posting confirm for NULL conn");
-	}
 
 	sdp_dbg_ctrl(conn, "Defered confirm <%08x>", conn->cm_id->local_id);
 	/*
 	 * send a confirm request using the connection manager
 	 */
 	result = ib_send_cm_rtu(conn->cm_id, NULL, 0);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(NULL, "Error <%d> CM confirm request", result);
-	}
 
 	SDP_CONN_PUT(conn);
 
@@ -195,10 +172,8 @@ static void _sdp_cm_failed(void *arg)
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
 
-	if (NULL == conn) {
-
+	if (NULL == conn)
 		sdp_dbg_warn(NULL, "Error, posting failed for NULL conn");
-	}
 
 	sdp_dbg_ctrl(conn, "Defered failed request <%08x>",
 		     conn->cm_id->local_id);
@@ -208,10 +183,8 @@ static void _sdp_cm_failed(void *arg)
 	result = ib_send_cm_rej(conn->cm_id, 
 				IB_CM_REJ_CONSUMER_DEFINED,
 				NULL, 0, NULL, 0);
-	if (0 > result) {
-	
+	if (0 > result)
 		sdp_dbg_warn(NULL, "Error <%d> CM failed request", result);
-	}
 
 	SDP_CONN_PUT(conn);
 
@@ -231,17 +204,10 @@ static int _sdp_cm_deferred_generic(stru
 	SDP_CONN_HOLD(conn);
 
 	if (in_atomic() || irqs_disabled()) {
-
-		INIT_WORK(&conn->cm_exec,
-			  defer_func,
-			  (void *)conn);
-    
+		INIT_WORK(&conn->cm_exec, defer_func, (void *)conn);
 		schedule_work(&conn->cm_exec);
-	}
-	else {
-
+	} else
 		defer_func(conn);
-	}
 
 	return 0;
 } /* _sdp_cm_deferred_generic */



From Tom.Duffy at Sun.COM  Fri Feb 18 16:46:36 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:46:36 -0800
Subject: [openib-general] [PATCH][SDP][13/22] Whitespace cleanup in
	sdp_proc.c
In-Reply-To: <11087739852191@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
Message-ID: <11087739963627@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proc.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proc.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proc.c	(working copy)
@@ -66,12 +66,10 @@ static int _sdp_proc_read_parse(char *pa
 
 	size = sub_entry->read(page, count, offset, &end_index);
 	if (0 < size) {
-
 		if (0 < end_index) {
 			*start = (char *)end_index;
 			*eof = 0;
-		}
-		else {
+		} else {
 			*start = NULL;
 			*eof = 1;
 		}
@@ -146,7 +144,6 @@ int sdp_main_proc_cleanup(void)
 	 */
 	for (counter = 0; counter < SDP_PROC_ENTRIES; counter++) {
 		sub_entry = &_file_entry_list[counter];
-
 		if (NULL != sub_entry->entry) {
 			remove_proc_entry(sub_entry->name, _dir_root);
 			sub_entry->entry = NULL;
@@ -178,14 +175,12 @@ int sdp_main_proc_init(void)
 	 * validate some assumptions the write parser will be making.
 	 */
 	if (0 && sizeof(s32) != sizeof(char *)) {
-
 		sdp_warn("integers and pointers of a different size <%Zu:%Zu>",
 			 sizeof(s32), sizeof(char *));
 		return -EFAULT;
 	}
 
 	if (NULL != _dir_root) {
-
 		sdp_warn("/proc already initialized!");
 		return -EINVAL;
 	}
@@ -194,7 +189,6 @@ int sdp_main_proc_init(void)
 	 */
 	_dir_root = proc_mkdir(_dir_name_root, proc_net);
 	if (NULL == _dir_root) {
-
 		sdp_warn("Failed to create <%s> proc entry.",
 			 _dir_name_root);
 		return -EINVAL;
@@ -204,7 +198,6 @@ int sdp_main_proc_init(void)
 
 	for (counter = 0; counter < SDP_PROC_ENTRIES; counter++) {
 		sub_entry = &_file_entry_list[counter];
-
 		if (sub_entry->type != counter) {
 			result = -EFAULT;
 			goto error;
@@ -214,7 +207,6 @@ int sdp_main_proc_init(void)
 						     S_IRUGO | S_IWUGO,
 						     _dir_root);
 		if (NULL == sub_entry->entry) {
-
 			sdp_warn("Failed to create <%s> framework proc entry.",
 				 sub_entry->name);
 			result = -EINVAL;



From Tom.Duffy at Sun.COM  Fri Feb 18 16:46:46 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:46:46 -0800
Subject: [openib-general] [PATCH][SDP][14/22] Whitespace cleanup in
	sdp_proto.h
In-Reply-To: <11087739963627@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
Message-ID: <11087740063185@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proto.h
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(working copy)
@@ -599,14 +599,10 @@ static __inline__ int __sdp_inet_write_s
 	if (0 < (SDP_ST_MASK_OPEN & conn->istate) &&
 	    (conn->send_max >
 	     sdp_desc_q_types_size(&conn->send_queue,
-				   SDP_DESC_TYPE_BUFF))) {
-
+				   SDP_DESC_TYPE_BUFF)))
 		return ((SDP_INET_SEND_MARK < size || 1 < urg) ? size : 0);
-	}
-	else {
-
+	else
 		return 0;
-	}
 } /* __sdp_inet_write_space */
 
 /*
@@ -616,15 +612,11 @@ static __inline__ int __sdp_inet_writabl
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
 
-	if (0 < (SDP_ST_MASK_OPEN & conn->istate)) {
-
+	if (0 < (SDP_ST_MASK_OPEN & conn->istate))
 		return (__sdp_inet_write_space(conn, 0) <
 			(conn->send_qud / 2)) ? 0 : 1;
-	}
-	else {
-
+	else
 		return 0;
-	}
 } /* __sdp_inet_writable */
 
 /*
@@ -638,17 +630,14 @@ static __inline__ int __sdp_conn_stat_du
 	sdp_dbg_init("STAT: src <%u> snk <%u>", 
 		     conn->src_serv, conn->snk_serv);
 
-	for (counter = 0; counter < 0x20; counter++) {
-
+	for (counter = 0; counter < 0x20; counter++)
 		if (0 < conn->send_mid[counter] ||
 		    0 < conn->recv_mid[counter]) {
-
 			sdp_dbg_init("STAT: MID send <%02x> <%u>", counter,
 				     conn->send_mid[counter]);
 			sdp_dbg_init("STAT: MID recv <%02x> <%u>", counter,
 				     conn->recv_mid[counter]);
 		}
-	}
 #endif
 	return 0;
 } /* __sdp_conn_stat_dump */
@@ -666,7 +655,6 @@ static __inline__ int __sdp_conn_state_d
 	sdp_dbg_init("STATE: Connection <%04x> state:", conn->hashent);
 
 	if (SDP_CONN_ST_INVALID == conn->state_rec.state[0]) {
-
 		sdp_dbg_init("STATE:   No state history. <%d>",
 			     conn->state_rec.value);
 		return 0;
@@ -674,14 +662,12 @@ static __inline__ int __sdp_conn_state_d
 
 	for (counter = 0;
 	     SDP_CONN_ST_INVALID != conn->state_rec.state[counter];
-	     counter++) {
-
+	     counter++)
 		sdp_dbg_init("STATE:   counter <%02x> state <%04x> <%s:%d>",
 			     counter,
 			     conn->state_rec.state[counter],
 			     conn->state_rec.file[counter],
 			     conn->state_rec.line[counter]);
-	}
 #endif
 	return 0;
 } /* __sdp_conn_state_dump */
@@ -699,10 +685,8 @@ static inline void __sdp_conn_hold(struc
  */
 static inline void __sdp_conn_put(struct sdp_opt *conn)
 {
-	if (atomic_dec_and_test(&conn->refcnt)) {
-
+	if (atomic_dec_and_test(&conn->refcnt))
 		(void)sdp_conn_destruct(conn);
-	}
 } /* __sdp_conn_hold */
 
 /*



From Tom.Duffy at Sun.COM  Fri Feb 18 16:46:56 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:46:56 -0800
Subject: [openib-general] [PATCH][SDP][15/22] Whitespace cleanup in
	sdp_queue.c
In-Reply-To: <11087740063185@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
Message-ID: <11087740164018@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_queue.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_queue.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_queue.c	(working copy)
@@ -48,26 +48,17 @@ static struct sdpc_desc *_sdp_desc_q_get
 
 	SDP_CHECK_NULL(table, NULL);
 
-	if (NULL == table->head) {
-
+	if (NULL == table->head)
 		return NULL;
-	}
-
-	if (fifo) {
 
+	if (fifo)
 		element = table->head;
-	}
-	else {
-
+	else
 		element = table->head->prev;
-	}
-
-	if (element->next == element && element->prev == element) {
 
+	if (element->next == element && element->prev == element)
 		table->head = NULL;
-	}
 	else {
-
 		element->next->prev = element->prev;
 		element->prev->next = element->next;
 
@@ -98,28 +89,22 @@ static __inline__ int _sdp_desc_q_put(st
 	SDP_CHECK_NULL(table, -EINVAL);
 	SDP_CHECK_NULL(element, -EINVAL);
 
-	if (NULL != element->table) {
-
+	if (NULL != element->table)
 		return -EINVAL;
-	}
 
 	if (NULL == table->head) {
-
 		element->next = element;
 		element->prev = element;
 		table->head = element;
-	}
-	else {
-
+	} else {
 		element->next = table->head;
 		element->prev = table->head->prev;
 
 		element->next->prev = element;
 		element->prev->next = element;
 
-		if (fifo) {
+		if (fifo)
 			table->head = element;
-		}
 	}
 
 	table->size++;
@@ -150,21 +135,16 @@ int sdp_desc_q_remove(struct sdpc_desc *
 
 	table = element->table;
 
-	if (element->next == element && element->prev == element) {
-
+	if (element->next == element && element->prev == element)
 		table->head = NULL;
-	}
 	else {
-
 		next = element->next;
 		prev = element->prev;
 		next->prev = prev;
 		prev->next = next;
 
-		if (table->head == element) {
-
+		if (table->head == element)
 			table->head = next;
-		}
 	}
 
 	table->size--;
@@ -193,13 +173,9 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
 	SDP_CHECK_NULL(lookup, NULL);
 
 	for (counter = 0, element = table->head;
-	     counter < table->size; counter++, element = element->next) {
-
-		if (0 == lookup(element, arg)) {
-
+	     counter < table->size; counter++, element = element->next)
+		if (0 == lookup(element, arg))
 			return element;
-		}
-	}
 
 	return NULL;
 } /* sdp_desc_q_lookup */
@@ -255,14 +231,10 @@ int sdp_desc_q_type_head(struct sdpc_des
 {
 	SDP_CHECK_NULL(table, -EINVAL);
 
-	if (NULL == table->head) {
-
+	if (NULL == table->head)
 		return SDP_DESC_TYPE_NONE;
-	}
-	else {
-
+	else
 		return table->head->type;
-	}
 } /* sdp_desc_q_type_head */
 
 /*
@@ -273,14 +245,10 @@ struct sdpc_desc *sdp_desc_q_look_type_h
 {
 	SDP_CHECK_NULL(table, NULL);
 
-	if (NULL == table->head) {
-
+	if (NULL == table->head)
 		return NULL;
-	}
-	else {
-
+	else
 		return ((type == table->head->type) ? table->head : NULL);
-	}
 } /* sdp_desc_q_look_type_head */
 
 /*
@@ -291,15 +259,11 @@ struct sdpc_desc *sdp_desc_q_look_type_t
 {
 	SDP_CHECK_NULL(table, NULL);
 
-	if (NULL == table->head) {
-
+	if (NULL == table->head)
 		return NULL;
-	}
-	else {
-
+	else
 		return ((type == table->head->prev->type) ?
 			table->head->prev : NULL);
-	}
 } /* sdp_desc_q_look_type_tail */
 
 /*
@@ -341,14 +305,11 @@ int sdp_desc_q_clear(struct sdpc_desc_q 
 	/*
 	 * drain the table of any objects
 	 */
-	while (NULL != (element = sdp_desc_q_get_head(table))) {
-
+	while (NULL != (element = sdp_desc_q_get_head(table)))
 		if (NULL != element->release) {
-
 			result = element->release(element);
 			SDP_EXPECT(!(0 > result));
 		}
-	}
 
 	return 0;
 } /* sdp_desc_q_clear */



From Tom.Duffy at Sun.COM  Fri Feb 18 16:47:06 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:47:06 -0800
Subject: [openib-general] [PATCH][SDP][16/22] Whitespace cleanup in
	sdp_rcvd.c
In-Reply-To: <11087740164018@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
Message-ID: <11087740262597@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c	(working copy)
@@ -58,7 +58,6 @@ static int _sdp_rcvd_disconnect(struct s
 		 */
 		result = sdp_wall_recv_close(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> closing connection",
 				     result);
 			goto error;
@@ -66,14 +65,10 @@ static int _sdp_rcvd_disconnect(struct s
 
 		break;
 	case SDP_CONN_ST_DIS_PEND_1:
-
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_R);
-
 		break;
 	case SDP_CONN_ST_DIS_SEND_1:
-
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_RECV_R);
-
 		break;
 	case SDP_CONN_ST_DIS_SENT_1:
 		/*
@@ -88,7 +83,6 @@ static int _sdp_rcvd_disconnect(struct s
 		 */
 		result = sdp_wall_recv_closing(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> confirming conn close",
 				     result);
 			goto error;
@@ -102,18 +96,14 @@ static int _sdp_rcvd_disconnect(struct s
 		 * do not treat it as an error
 		 */
 		if (0 > result) {
-			
 			sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
 
-			if (-EPROTO != result) {
-				
+			if (-EPROTO != result)
 				goto error;
-			}
 		}
 
 		break;
 	default:
-
 		sdp_warn("Unexpected conn state. conn <%d> state <%04x:%04x>",
 			 conn->hashent, conn->istate, conn->state);
 
@@ -146,16 +136,12 @@ static int _sdp_rcvd_abort(struct sdp_op
 	case SDP_CONN_ST_DIS_PEND_2:
 	case SDP_CONN_ST_DIS_SEND_2:
 	case SDP_CONN_ST_DIS_SENT_2:
-
 		result = sdp_wall_abort(conn);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> during abort", result);
-		}
 
 		break;
 	default:
-
 		sdp_warn("Unexpected abort. conn <%d> state <%04x:%04x>",
 			 conn->hashent, conn->istate, conn->state);
 		result = -EPROTO;
@@ -191,7 +177,6 @@ static int _sdp_rcvd_send_sm(struct sdp_
 		 * drain the active source queue
 		 */
 		while (NULL != (iocb = sdp_iocb_q_get_tail(&conn->w_src))) {
-
 			SDP_EXPECT((0 < (SDP_IOCB_F_ACTIVE & iocb->flags)));
 			SDP_EXPECT((0 < (SDP_IOCB_F_CANCEL & iocb->flags)));
 
@@ -228,7 +213,6 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_snk);
 	if (NULL == iocb) {
-
 		sdp_dbg_warn(conn, "Cannot find IOCB for Write Completion.");
 		result = -EPROTO;
 		goto error;
@@ -243,7 +227,6 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	 * update IOCB
 	 */
 	if (rwch->size > iocb->len) {
-
 		sdp_dbg_warn(conn, "IOCB and Write size mismatch. <%d:%d>",
 			     rwch->size, iocb->len);
 		result = -EPROTO;
@@ -266,7 +249,6 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 
 	result = sdp_iocb_complete(iocb, 0);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 			     result, iocb->key);
 		goto error;
@@ -295,7 +277,6 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 */
 	iocb = sdp_iocb_q_look(&conn->w_src);
 	if (NULL == iocb) {
-
 		sdp_dbg_warn(conn, "Cannot find IOCB for Read Completion.");
 		result = -EPROTO;
 		goto error;
@@ -310,7 +291,6 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 * update IOCB
 	 */
 	if (rrch->size > iocb->len) {
-		
 		sdp_dbg_warn(conn, "IOCB and Read size mismatch. <%d:%d>",
 			     rrch->size, iocb->len);
 		result = -EPROTO;
@@ -334,7 +314,6 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 * If iocb is done, deregister memory, and generate completion.
 	 */
 	if (!(0 < iocb->len)) {
-
 		iocb = sdp_iocb_q_get_head(&conn->w_src);
 		SDP_EXPECT((NULL != iocb));
 
@@ -345,7 +324,6 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 
 		result = sdp_iocb_complete(iocb, 0);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto error;
@@ -357,7 +335,6 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 */
 	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags) &&
 	    0 == sdp_iocb_q_size(&conn->w_src)) {
-
 		conn->src_cncl = 0;
 		conn->flags &= ~(SDP_CONN_F_SRC_CANCEL_L);
 	}
@@ -389,7 +366,6 @@ static int _sdp_rcvd_mode_change(struct 
 	if (((SDP_MSG_MCH_GET_MODE(mch) & 0x7) ==
 	     ((0 < (SDP_MSG_MCH_GET_MODE(mch) & 0x8)) ?
 	      conn->send_mode : conn->recv_mode))) {
-
 		sdp_dbg_warn(conn, "Mode transition <%d> is a nop. <%d:%d>",
 			     SDP_MSG_MCH_GET_MODE(mch), conn->recv_mode,
 			     conn->send_mode);
@@ -400,17 +376,13 @@ static int _sdp_rcvd_mode_change(struct 
 	 * process mode change requests based on which state we're in
 	 */
 	switch (SDP_MSG_MCH_GET_MODE(mch)) {
-
 	case SDP_MSG_MCH_BUFF_RECV:	/* source to sink */
-
 		if (SDP_MODE_COMB != conn->recv_mode) {
-
 			result = -EPROTO;
 			goto mode_error;
 		}
 
 		if (0 < conn->src_recv) {
-
 			sdp_dbg_warn(conn, "mode error <%d> src pending <%d>",
 				     SDP_MSG_MCH_GET_MODE(mch),
 				     conn->src_recv);
@@ -420,27 +392,21 @@ static int _sdp_rcvd_mode_change(struct 
 
 		break;
 	case SDP_MSG_MCH_COMB_SEND:	/* sink to source */
-
 		if (SDP_MODE_BUFF != conn->send_mode) {
-
 			result = -EPROTO;
 			goto mode_error;
 		}
 
 		break;
 	case SDP_MSG_MCH_PIPE_RECV:	/* source to sink */
-
 		if (SDP_MODE_COMB != conn->recv_mode) {
-
 			result = -EPROTO;
 			goto mode_error;
 		}
 
 		break;
 	case SDP_MSG_MCH_COMB_RECV:	/* source to sink */
-
 		if (SDP_MODE_PIPE != conn->recv_mode) {
-
 			result = -EPROTO;
 			goto mode_error;
 		}
@@ -454,7 +420,6 @@ static int _sdp_rcvd_mode_change(struct 
 		 */
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Invalid mode transition <%d> requested.",
 			     SDP_MSG_MCH_GET_MODE(mch));
 		result = -EPROTO;
@@ -463,14 +428,10 @@ static int _sdp_rcvd_mode_change(struct 
 	/*
 	 * assign new mode
 	 */
-	if (0 < (SDP_MSG_MCH_GET_MODE(mch) & 0x8)) {
-
+	if (0 < (SDP_MSG_MCH_GET_MODE(mch) & 0x8))
 		conn->send_mode = SDP_MSG_MCH_GET_MODE(mch) & 0x7;
-	}
-	else {
-
+	else
 		conn->recv_mode = SDP_MSG_MCH_GET_MODE(mch) & 0x7;
-	}
 
 	return 0;
 
@@ -498,7 +459,6 @@ static int _sdp_rcvd_src_cancel(struct s
 	 * to do.
 	 */
 	if (!(0 < conn->src_recv)) {
-
 		sdp_dbg_warn(conn, "No SrcAvail advertisments to cancel.");
 		result = 0;
 		goto done;
@@ -514,7 +474,6 @@ static int _sdp_rcvd_src_cancel(struct s
 		 * immediate completion
 		 */
 		if (0 < (SDP_ADVT_F_READ & advt->flag)) {
-
 			advt = sdp_advt_q_get(&conn->src_pend);
 			SDP_EXPECT((NULL != advt));
 
@@ -524,12 +483,9 @@ static int _sdp_rcvd_src_cancel(struct s
 			 * keep track of cancellations
 			 */
 			conn->flags |= SDP_CONN_F_SRC_CANCEL_C;
-		}
-		else {
-
+		} else {
 			result = sdp_send_ctrl_rdma_rd(conn, advt->post);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, 
 					     "Error <%d> read completion",
 					     result);
@@ -541,7 +497,6 @@ static int _sdp_rcvd_src_cancel(struct s
 	 * drop the pending advertisment queue.
 	 */
 	while (NULL != (advt = sdp_advt_q_get(&conn->src_pend))) {
-
 		conn->flags |= SDP_CONN_F_SRC_CANCEL_C;
 
 		conn->src_recv--;
@@ -562,16 +517,13 @@ static int _sdp_rcvd_src_cancel(struct s
 		 */
 		conn->flags |= SDP_CONN_F_SRC_CANCEL_R;
 		conn->advt_seq--;
-	}
-	else {
+	} else {
 		/*
 		 * If a source was dropped, generate an ack.
 		 */
 		if (0 < (SDP_CONN_F_SRC_CANCEL_C & conn->flags)) {
-
 			result = sdp_send_ctrl_send_sm(conn);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, "Error<%d> posting SendSm",
 					     result);
 				goto done;
@@ -602,7 +554,6 @@ static int _sdp_rcvd_snk_cancel(struct s
 	 * since the message was sent, and there is nothing to do.
 	 */
 	if (!(0 < conn->snk_recv)) {
-
 		sdp_dbg_warn(conn, "No SnkAvail advertisments to cancel.");
 		result = 0;
 		goto done;
@@ -620,7 +571,6 @@ static int _sdp_rcvd_snk_cancel(struct s
 		 */
 		result = sdp_send_ctrl_rdma_wr(conn, advt->post);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> write completion",
 				     result);
 			goto done;
@@ -629,18 +579,15 @@ static int _sdp_rcvd_snk_cancel(struct s
 		 * reduce cancel counter
 		 */
 		counter = -1;
-	}
-	else {
+	} else
 		/*
 		 * cancel count.
 		 */
 		counter = 0;
-	}
 	/*
 	 * drain the advertisments which have yet to be processed.
 	 */
 	while (NULL != (advt = sdp_advt_q_get(&conn->snk_pend))) {
-
 		counter++;
 		conn->snk_recv--;
 
@@ -652,10 +599,8 @@ static int _sdp_rcvd_snk_cancel(struct s
 	 * sending a completion
 	 */
 	if (0 < counter) {
-
 		result = sdp_send_ctrl_snk_cancel_ack(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> SnkCacelAck response",
 				     result);
 			goto done;
@@ -683,7 +628,6 @@ static int _sdp_rcvd_snk_cancel_ack(stru
 		     conn->snk_sent, conn->recv_mode, conn->flags);
 
 	if (0 == (SDP_CONN_F_SNK_CANCEL & conn->flags)) {
-
 		sdp_dbg_warn(conn, "Connection not in sink cancel mode <%08x>",
 			     conn->flags);
 		result = -EPROTO;
@@ -693,12 +637,10 @@ static int _sdp_rcvd_snk_cancel_ack(stru
 	 * drain and complete all active IOCBs
 	 */
 	while (NULL != (iocb = sdp_iocb_q_get_head(&conn->r_snk))) {
-
 		conn->snk_sent--;
 
 		result = sdp_iocb_complete(iocb, 0);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto done;
@@ -738,7 +680,6 @@ static int _sdp_rcvd_resize_buff_ack(str
 
 	result = sdp_send_ctrl_resize_buff_ack(conn, conn->recv_size);
 	if (0 > result) {
-		
 		sdp_dbg_warn(conn, "Error <%d> acking size change request",
 			     result);
 		goto error;
@@ -793,7 +734,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 	 * check our send mode, and make sure parameters are within reason.
 	 */
 	if (SDP_MODE_PIPE != conn->send_mode) {
-
 		sdp_dbg_warn(conn, "SinkAvail, incorrect source mode <%d>",
 			     conn->send_mode);
 		result = -EPROTO;
@@ -801,7 +741,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 	}
 
 	if (SDP_MSG_MAX_ADVS == (conn->src_recv + conn->snk_recv)) {
-
 		sdp_dbg_warn(conn, "SinkAvail, too many advertisments. <%d>",
 			     (conn->src_recv + conn->snk_recv));
 		result = -EPROTO;
@@ -809,7 +748,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 	}
 
 	if (snkah->size < conn->send_size) {
-
 		sdp_dbg_warn(conn, "SinkAvail too small. <%d:%d>",
 			     snkah->size, conn->send_size);
 		result = -EPROTO;
@@ -820,7 +758,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 	 * discard and skip to data processing.
 	 */
 	if (conn->nond_send != snkah->non_disc) {
-
 		conn->nond_send--;
 		goto consume;
 	}
@@ -829,9 +766,7 @@ static int _sdp_rcvd_snk_avail(struct sd
 	 * invalid and the queue needs to be fixed up.
 	 */
 	if (0 < conn->src_sent) {
-
 		while (NULL != (iocb = sdp_iocb_q_get_tail(&conn->w_src))) {
-
 			SDP_EXPECT((0 < (SDP_IOCB_F_ACTIVE & iocb->flags)));
 
 			iocb->flags &= ~SDP_IOCB_F_ACTIVE;
@@ -843,14 +778,11 @@ static int _sdp_rcvd_snk_avail(struct sd
 			 * completed.
 			 */
 			if (0 == (SDP_IOCB_F_CANCEL & iocb->flags)) {
-
 				result = sdp_desc_q_put_head(&conn->send_queue,
 							     (struct sdpc_desc *)
 							     iocb);
 				SDP_EXPECT(!(0 > result));
-			}
-			else {
-
+			} else {
 				result = sdp_iocb_complete(iocb, 0);
 				SDP_EXPECT(!(0 > result));
 			}
@@ -860,7 +792,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 		 * be cleared.
 		 */
 		if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags)) {
-			
 			conn->src_cncl = 0;
 			conn->flags &= ~(SDP_CONN_F_SRC_CANCEL_L);
 		}
@@ -870,7 +801,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 	 */
 	advt = sdp_advt_create();
 	if (NULL == advt) {
-		
 		sdp_dbg_warn(conn, "SrcAvail cannot be copied.");
 		result = -ENOMEM;
 		goto error;
@@ -888,7 +818,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 
 	result = sdp_advt_q_put(&conn->snk_pend, advt);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "SnkAvail cannot be saved. <%d>", result);
 		goto advt_error;
 	}
@@ -900,29 +829,21 @@ consume:
 	 * consume any data in the advertisment for the other direction.
 	 */
 	if (0 < (buff->tail - buff->data)) {
-
 		result = sdp_recv_buff(conn, buff);
-		if (0 < result) {
+		if (0 < result)
 			/*
 			 * count number of bytes buffered by the connection,
 			 * zero byte buffers or errors can be returned, the
 			 * buffer will be dispossed of by the caller.
 			 */
 			conn->byte_strm += result;
-		}
-		else {
-
-			if (0 > result) {
-
+		else
+			if (0 > result)
 				sdp_dbg_warn(conn, "Error <%d> buffer recv",
 					     result);
-			}
-		}
-	}
-	else {
-
+	} else
 		result = 0;
-	}
+	
 	/*
 	 * PostRecv will take care of consuming this advertisment, based
 	 * on result.
@@ -967,7 +888,6 @@ static int _sdp_rcvd_src_avail(struct sd
 	}
 
 	if (0 < (SDP_CONN_F_SRC_CANCEL_R & conn->flags)) {
-
 		sdp_dbg_warn(conn, "SrcAvail during SrcAvailCancel. <%d>",
 			     conn->src_recv);
 		result = -EFAULT;
@@ -979,7 +899,6 @@ static int _sdp_rcvd_src_avail(struct sd
 	 * closed. This notifies the peer that the data was not received.
 	 */
 	if (0 < (RCV_SHUTDOWN & conn->shutdown)) {
-
 		sdp_dbg_warn(conn, "SrcAvail, receive path closed <%02x>",
 			     conn->shutdown);
 		/*
@@ -998,7 +917,6 @@ static int _sdp_rcvd_src_avail(struct sd
 	 */
 	advt = sdp_advt_create();
 	if (NULL == advt) {
-
 		sdp_dbg_warn(conn, "SrcAvail cannot be copied.");
 		result = -ENOMEM;
 		goto done;
@@ -1008,9 +926,7 @@ static int _sdp_rcvd_src_avail(struct sd
 	 * path mode to determine if all is cool for the advertisment.
 	 */
 	switch (conn->recv_mode) {
-
 	case SDP_MODE_BUFF:
-
 		sdp_dbg_warn(conn, "SrcAvail in bad mode. <%d>", 
 			     conn->recv_mode);
 		result = -EPROTO;
@@ -1018,11 +934,9 @@ static int _sdp_rcvd_src_avail(struct sd
 
 		break;
 	case SDP_MODE_COMB:
-
 		if (0 < conn->src_recv ||
 		    !(0 < size) ||
 		    !(srcah->size > size)) {
-
 			sdp_dbg_warn(conn, 
 				     "SrcAvail mode <%d> mismatch. <%d:%d:%d>",
 				     conn->recv_mode, conn->src_recv,
@@ -1043,10 +957,8 @@ static int _sdp_rcvd_src_avail(struct sd
 
 		break;
 	case SDP_MODE_PIPE:
-
 		if (SDP_MSG_MAX_ADVS == (conn->src_recv + conn->snk_recv) ||
 		    0 != size) {
-
 			sdp_dbg_warn(conn,
 				     "SrcAvail mode <%d> mismatch. <%d:%d>",
 				     conn->recv_mode,
@@ -1063,7 +975,6 @@ static int _sdp_rcvd_src_avail(struct sd
 
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "SrcAvail message in unknown mode. <%d>",
 			     conn->recv_mode);
 		result = -EPROTO;
@@ -1076,26 +987,22 @@ static int _sdp_rcvd_src_avail(struct sd
 
 	result = sdp_advt_q_put(&conn->src_pend, advt);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> queuing SrcAvail", result);
 		goto advt_error;
 	}
 	/*
 	 * process any ULP data in the message
 	 */
-	if (0 == size) {
-
+	if (0 == size)
 		goto done;
-	}
 	/*
 	 * update non-discard for sink advertisment management
 	 */
 	conn->nond_recv++;
 
 	if (!(SDP_SRC_AVAIL_RECV > size)) {
-
 		result = sdp_recv_buff(conn, buff);
-		if (0 < result) {
+		if (0 < result)
 			/*
 			 * count number of bytes buffered by the
 			 * connection, zero byte buffers or errors
@@ -1103,20 +1010,12 @@ static int _sdp_rcvd_src_avail(struct sd
 			 * dispossed of by the caller.
 			 */
 			conn->byte_strm += result;
-		}
-		else {
-
-			if (0 > result) {
-				
+		else
+			if (0 > result)
 				sdp_dbg_warn(conn, "Error <%d> buffer recv",
 					     result);
-			}
-		}
-	}
-	else {
-		
+	} else
 		result = 0;
-	}
 	/*
 	 * PostRecv will take care of consuming this advertisment.
 	 */
@@ -1138,16 +1037,13 @@ static int _sdp_rcvd_data(struct sdp_opt
 	SDP_CHECK_NULL(conn, -EINVAL);
 	SDP_CHECK_NULL(buff, -EINVAL);
 
-	if (buff->tail == buff->data) {
-
+	if (buff->tail == buff->data)
 		return 0;
-	}
 	/*
 	 * If we are processing a SrcAvail, there should be no
 	 * buffered data
 	 */
 	if (0 < conn->src_recv) {
-
 		sdp_dbg_warn(conn, "Error, recv'd data with SrcAvail active.");
 		return -EPROTO;
 	}
@@ -1156,25 +1052,19 @@ static int _sdp_rcvd_data(struct sdp_opt
 	 * is a pending urgent message. If the OOB data is in this
 	 * buffer, pull it out.
 	 */
-	if (SDP_BSDH_GET_OOB_PEND(buff->bsdh_hdr)) {
-
+	if (SDP_BSDH_GET_OOB_PEND(buff->bsdh_hdr))
 		buff->flags |= SDP_BUFF_F_OOB_PEND;
-	}
-
-	if (SDP_BSDH_GET_OOB_PRES(buff->bsdh_hdr)) {
 
+	if (SDP_BSDH_GET_OOB_PRES(buff->bsdh_hdr))
 		buff->flags |= SDP_BUFF_F_OOB_PRES;
-	}
 	/*
 	 * update non-discard for sink advertisment management
 	 */
 	conn->nond_recv++;
 
 	ret_val = sdp_recv_buff(conn, buff);
-	if (0 > ret_val) {
-
+	if (0 > ret_val)
 		sdp_dbg_warn(conn, "Error <%d> processing buff recv", ret_val);
-	}
 	/*
 	 * result contains the number of bytes in the buffer which
 	 * are being kept by the connection. (zero buffered means
@@ -1267,7 +1157,6 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 * error handling
 	 */
 	if (IB_WC_SUCCESS != comp->status) {
-
 		switch (comp->status) {
 		case IB_WC_WR_FLUSH_ERR:
 			/*
@@ -1293,13 +1182,11 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 */
 	buff = sdp_buff_q_get_head(&conn->recv_post);
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "receive event, but no posted receive?!");
 		result = -EINVAL;
 		goto done;
 	}
 	if (comp->wr_id != buff->wrid) {
-
 		sdp_dbg_warn(conn, "work request ID mismatch. <%llu:%llu>",
 			     (unsigned long long)comp->wr_id,
 			     (unsigned long long)buff->wrid);
@@ -1324,7 +1211,6 @@ int sdp_event_recv(struct sdp_opt *conn,
         _sdp_msg_net_to_cpu_bsdh(buff->bsdh_hdr);
 
 	if (comp->byte_len != buff->bsdh_hdr->size) {
-
 		sdp_dbg_warn(conn, 
 			     "receive event, message size mismatch <%d:%d>",
 			     comp->byte_len, buff->bsdh_hdr->size);
@@ -1362,17 +1248,13 @@ int sdp_event_recv(struct sdp_opt *conn,
 	/*
 	 * fast path data messages
 	 */
-	if (SDP_MID_DATA == buff->bsdh_hdr->mid) {
-
+	if (SDP_MID_DATA == buff->bsdh_hdr->mid)
 		result = _sdp_rcvd_data(conn, buff);
-	}
 	else {
-
 		offset = buff->bsdh_hdr->mid & 0x1F;
 
 		if (!(offset < SDP_MSG_EVENT_TABLE_SIZE) ||
 		    NULL == recv_event_funcs[offset]) {
-
 			sdp_dbg_warn(conn, "receive event, unknown MID <%d>",
 				     buff->bsdh_hdr->mid);
 			result = -EINVAL;
@@ -1388,7 +1270,6 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 * process result.
 	 */
 	if (0 == result) {
-
 		result = sdp_buff_pool_put(buff);
 		SDP_EXPECT(!(0 > result));
 		/*
@@ -1398,37 +1279,29 @@ int sdp_event_recv(struct sdp_opt *conn,
 		 */
 		result = sdp_recv_flush(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> flushing recv queue",
 				     result);
 			goto done;
 		}
-	}
-	else {
-
+	} else
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, 
 				     "receive event, dispatch error. <%d>",
 				     result);
 
 			goto drop;
-		}
-		else {
+		} else
 			/*
 			 * If data was consumed by the protocol, signal
 			 * the user.
 			 */
 			sdp_inet_wake_recv(conn->sk, conn->byte_strm);
-		}
-	}
 	/*
 	 * It's possible that a new recv buffer advertisment opened up the 
 	 * recv window and we can flush buffered send data
 	 */
 	result = sdp_send_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue",
 			     result);
 		goto done;



From Tom.Duffy at Sun.COM  Fri Feb 18 16:47:17 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:47:17 -0800
Subject: [openib-general] [PATCH][SDP][17/22] Whitespace cleanup in
	sdp_read.c
In-Reply-To: <11087740262597@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
	<11087740262597@sun.com>
Message-ID: <11087740372545@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_read.c	(working copy)
@@ -56,12 +56,9 @@ static int _sdp_event_read_advt(struct s
 	 */
 	advt = sdp_advt_q_look(&conn->src_actv);
 	if (NULL == advt || advt->wrid != comp->wr_id) {
-	
 		advt = sdp_advt_q_look(&conn->src_pend);
-		if (NULL != advt && advt->wrid == comp->wr_id) {
-			
+		if (NULL != advt && advt->wrid == comp->wr_id)
 			advt->flag &= ~SDP_ADVT_F_READ;
-		}
 
 		goto done;
 	}
@@ -82,17 +79,14 @@ static int _sdp_event_read_advt(struct s
 	 */
 	if (0 < (SDP_CONN_F_SRC_CANCEL_R & conn->flags) &&
 	    0 == conn->src_recv) {
-
 		conn->flags &= ~SDP_CONN_F_SRC_CANCEL_R;
 		conn->advt_seq = conn->recv_seq;
 		/*
 		 * If any data was canceled, post a SendSm, also
 		 */
 		if (0 < (SDP_CONN_F_SRC_CANCEL_C & conn->flags)) {
-
 			result = sdp_send_ctrl_send_sm(conn);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, "Error <%d> posting SendSm",
 					     result);
 				goto error;
@@ -128,7 +122,6 @@ int sdp_event_read(struct sdp_opt *conn,
 	 * error handling
 	 */
 	if (IB_WC_SUCCESS != comp->status) {
-
 		switch (comp->status) {
 		case IB_WC_WR_FLUSH_ERR:
 			/*
@@ -140,7 +133,6 @@ int sdp_event_read(struct sdp_opt *conn,
 			result = 0;
 			break;
 		default:
-
 			sdp_dbg_warn(conn, "Unhandled READ status <%d>.",
 				     comp->status);
 			result = -EIO;
@@ -175,12 +167,10 @@ int sdp_event_read(struct sdp_opt *conn,
 	type = sdp_desc_q_type_head(&conn->r_src);
 	switch (type) {
 	case SDP_DESC_TYPE_BUFF:
-
 		buff = (struct sdpc_buff *) sdp_desc_q_get_head(&conn->r_src);
 		SDP_EXPECT((NULL != buff));
 
 		if (comp->wr_id != buff->wrid) {
-
 			sdp_dbg_warn(conn, "work request mismatch <%llu:%llu>",
 				     (unsigned long long)comp->wr_id,
 				     (unsigned long long)buff->wrid);
@@ -202,14 +192,10 @@ int sdp_event_read(struct sdp_opt *conn,
 			conn->byte_strm += result;
 
 			sdp_inet_wake_recv(conn->sk, conn->byte_strm);
-		}
-		else {
-
-			if (0 > result) {
-
+		} else {
+			if (0 > result)
 				sdp_dbg_warn(conn, "Error <%d> receiving buff",
 					     result);
-			}
 
 			result = sdp_buff_pool_put(buff);
 			SDP_EXPECT(!(0 > result));
@@ -217,12 +203,9 @@ int sdp_event_read(struct sdp_opt *conn,
 
 		break;
 	case SDP_DESC_TYPE_IOCB:
-
 		iocb = (struct sdpc_iocb *) sdp_desc_q_look_head(&conn->r_src);
-		if (NULL == iocb || iocb->wrid != comp->wr_id) {
-
+		if (NULL == iocb || iocb->wrid != comp->wr_id)
 			break;
-		}
 
 		iocb = (struct sdpc_iocb *)sdp_desc_q_get_head(&conn->r_src);
 		SDP_EXPECT((NULL != iocb));
@@ -234,7 +217,6 @@ int sdp_event_read(struct sdp_opt *conn,
 
 		result = sdp_iocb_complete(iocb, 0);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				      result, iocb->key);
 			goto done;
@@ -242,25 +224,19 @@ int sdp_event_read(struct sdp_opt *conn,
 
 		break;
 	case SDP_DESC_TYPE_NONE:
-
 		iocb = sdp_iocb_q_look(&conn->r_pend);
 		if (NULL == iocb) {
-
 			result = -EPROTO;
 			goto done;
 		}
 
-		if (iocb->wrid != comp->wr_id) {
-
+		if (iocb->wrid != comp->wr_id)
 			break;
-		}
 
 		iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_R);
 
-		if (conn->sk->sk_rcvlowat > iocb->post) {
-
+		if (conn->sk->sk_rcvlowat > iocb->post)
 			break;
-		}
 
 		iocb = sdp_iocb_q_get_head(&conn->r_pend);
 		SDP_EXPECT((NULL != iocb));
@@ -270,7 +246,6 @@ int sdp_event_read(struct sdp_opt *conn,
 
 		result = sdp_iocb_complete(iocb, 0);
 		if (0 > result) {
-				
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto done;
@@ -278,7 +253,6 @@ int sdp_event_read(struct sdp_opt *conn,
 
 		break;
 	default:
-
 		sdp_warn("Unknown type <%d> at head of READ SRC queue. <%d>",
 			 type, sdp_desc_q_size(&conn->r_src));
 		result = -EPROTO;
@@ -289,7 +263,6 @@ int sdp_event_read(struct sdp_opt *conn,
 	 */
 	result = _sdp_event_read_advt(conn, comp);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> handling READ advertisment",
 			     result);
 		goto done;
@@ -300,7 +273,6 @@ int sdp_event_read(struct sdp_opt *conn,
 	 */
 	result = sdp_send_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue.", result);
 		goto done;
 	}
@@ -310,7 +282,6 @@ int sdp_event_read(struct sdp_opt *conn,
 	 */
 	result = sdp_recv_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing recv queue.", result);
 		goto done;
 	}



From Tom.Duffy at Sun.COM  Fri Feb 18 16:47:27 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:47:27 -0800
Subject: [openib-general] [PATCH][SDP][18/22] Whitespace cleanup in
	sdp_recv.c
In-Reply-To: <11087740372545@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
	<11087740262597@sun.com> <11087740372545@sun.com>
Message-ID: <11087740473391@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -54,7 +54,6 @@ static int _sdp_post_recv_buff(struct sd
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "failed to allocate buff for recv queue.");
 		result = -ENOMEM;
 		goto error;
@@ -78,7 +77,6 @@ static int _sdp_post_recv_buff(struct sd
 	 */
 	result = sdp_buff_q_put_tail(&conn->recv_post, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> queuing recv buffer.", result);
 		goto drop;
 	}
@@ -101,7 +99,6 @@ static int _sdp_post_recv_buff(struct sd
 
 	result = ib_post_recv(conn->qp, &receive_param, &bad_wr);
 	if (0 != result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting receive buffer",
 			     result);
 		(void)sdp_buff_q_get_tail(&conn->recv_post);
@@ -132,7 +129,6 @@ static int _sdp_post_rdma_buff(struct sd
 	 * check queue depth
 	 */
 	if (!(conn->send_cq_size > conn->s_wq_size)) {
-
 		result = ENODEV;
 		goto done;
 	}
@@ -141,7 +137,6 @@ static int _sdp_post_rdma_buff(struct sd
 	 */
 	advt = sdp_advt_q_look(&conn->src_pend);
 	if (NULL == advt) {
-
 		result = ENODEV;
 		goto done;
 	}
@@ -150,7 +145,6 @@ static int _sdp_post_rdma_buff(struct sd
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "failed to allocate buff for rdma read.");
 		result = -ENOMEM;
 		goto error;
@@ -181,10 +175,8 @@ static int _sdp_post_rdma_buff(struct sd
 	 * active list, and match the WRID.
 	 */
 	if (!(0 < advt->size)) {
-
 		advt = sdp_advt_q_get(&conn->src_pend);
 		if (NULL == advt) {
-
 			sdp_dbg_warn(conn, "SrcAvail disappeared. <%d>",
 				     sdp_advt_q_size(&conn->src_pend));
 			result = -ENODEV;
@@ -193,7 +185,6 @@ static int _sdp_post_rdma_buff(struct sd
 
 		result = sdp_advt_q_put(&conn->src_actv, advt);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> queuing active src",
 				     result);
 
@@ -208,7 +199,6 @@ static int _sdp_post_rdma_buff(struct sd
 	 */
 	result = sdp_desc_q_put_tail(&conn->r_src, (struct sdpc_desc *) buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> queuing rdma read.", result);
 		goto drop;
 	}
@@ -229,7 +219,6 @@ static int _sdp_post_rdma_buff(struct sd
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
 	if (0 != result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting rdma read", result);
 
 		(void)sdp_desc_q_get_tail(&conn->r_src);
@@ -264,7 +253,6 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * check queue depth
 	 */
 	if (!(conn->send_cq_size > conn->s_wq_size)) {
-
 		result = ENODEV;
 		goto done;
 	}
@@ -273,7 +261,6 @@ static int _sdp_post_rdma_iocb_src(struc
 	 */
 	advt = sdp_advt_q_look(&conn->src_pend);
 	if (NULL == advt) {
-
 		result = ENODEV;
 		goto done;
 	}
@@ -285,7 +272,6 @@ static int _sdp_post_rdma_iocb_src(struc
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_pend);
 	if (NULL == iocb) {
-
 		result = ENODEV;
 		goto done;
 	}
@@ -294,7 +280,6 @@ static int _sdp_post_rdma_iocb_src(struc
 	 */
 	result = sdp_iocb_register(iocb, conn);
 	if (0 > result) {
-		
 		sdp_dbg_warn(conn, "Error <%d> registering IOCB. <%d:%d>",
 			     result, iocb->key, iocb->len);
 		goto error;
@@ -330,10 +315,8 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * advertisment for completion
 	 */
 	if (!(0 < advt->size)) {
-
 		advt = sdp_advt_q_get(&conn->src_pend);
 		if (NULL == advt) {
-
 			sdp_dbg_warn(conn, "SrcAvail disappeared. <%d>",
 				     sdp_advt_q_size(&conn->src_pend));
 			result = -ENODEV;
@@ -342,7 +325,6 @@ static int _sdp_post_rdma_iocb_src(struc
 
 		result = sdp_advt_q_put(&conn->src_actv, advt);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> queuing active src",
 				     result);
 			(void)sdp_advt_destroy(advt);
@@ -353,10 +335,8 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * if there is no more iocb space queue the it for completion
 	 */
 	if (!(0 < iocb->len)) {
-
 		iocb = sdp_iocb_q_get_head(&conn->r_pend);
 		if (NULL == iocb) {
-
 			sdp_dbg_warn(conn, "read IOCB disappeared. <%d>",
 				     sdp_iocb_q_size(&conn->r_pend));
 			result = -ENODEV;
@@ -366,7 +346,6 @@ static int _sdp_post_rdma_iocb_src(struc
 		result = sdp_desc_q_put_tail(&conn->r_src,
 					     (struct sdpc_desc *)iocb);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> queuing read IOCB",
 				     result);
 			(void)sdp_iocb_destroy(iocb);
@@ -390,7 +369,6 @@ static int _sdp_post_rdma_iocb_src(struc
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
 	if (0 != result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting rdma read", result);
 		conn->s_wq_size--;
 		goto error;
@@ -415,7 +393,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 * check if sink cancel is pending
 	 */
 	if (0 < (SDP_CONN_F_SNK_CANCEL & conn->flags)) {
-
 		result = ENODEV;
 		goto error;
 	}
@@ -424,7 +401,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_pend);
 	if (NULL == iocb) {
-
 		result = ENODEV;
 		goto error;
 	}
@@ -432,7 +408,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 * check zcopy threshold
 	 */
 	if (conn->snk_zthresh > iocb->len) {
-
 		result = ENODEV;
 		goto error;
 	}
@@ -440,7 +415,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 * check number of outstanding sink advertisments
 	 */
 	if (!(conn->r_max_adv > conn->snk_sent)) {
-
 		result = ENODEV;
 		goto error;
 	}
@@ -449,13 +423,10 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 */
 	result = sdp_iocb_register(iocb, conn);
 	if (result) {
-
 		result = (-EAGAIN == result ? EAGAIN : result);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> registering IOCB",
 				     result);
-		}
 
 		goto error;
 	}
@@ -469,7 +440,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 */
 	iocb = sdp_iocb_q_get_head(&conn->r_pend);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "read IOCB missing from pending table <%d>",
 			     sdp_iocb_q_size(&conn->r_pend));
 		goto release;
@@ -477,7 +447,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 
 	result = sdp_iocb_q_put_tail(&conn->r_snk, iocb);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> queueing active write IOCB",
 			     result);
 		goto re_q;
@@ -490,7 +459,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 					 iocb->r_key,
 					 iocb->io_addr);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> sending SnkAvail message",
 			     result);
 		goto de_q;
@@ -526,29 +494,23 @@ static int _sdp_post_rdma(struct sdp_opt
 	 * the sink advertisment, something to explore, but SrcAvail
 	 * slow start might make that unneccessart?
 	 */
-	if (0 == (SDP_ST_MASK_SEND_OK & conn->state)) {
-
+	if (0 == (SDP_ST_MASK_SEND_OK & conn->state))
 		return 0;
-	}
 	/*
 	 * loop flushing IOCB RDMAs. Read sources, otherwise post sinks.
 	 */
 	if (0 < sdp_advt_q_size(&conn->src_pend)) {
-
 		if (0 == sdp_desc_q_types_size(&conn->r_src,
-					       SDP_DESC_TYPE_BUFF)) {
-
+					       SDP_DESC_TYPE_BUFF))
 			while (0 == (result = _sdp_post_rdma_iocb_src(conn))) {
 				/*
 				 * pass, nothing to do in loop.
 				 */
 			}
-		}
 		/*
 		 * check non-zero result
 		 */
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> posting RDMA IOCB read",
 				     result);
 			goto done;
@@ -556,47 +518,37 @@ static int _sdp_post_rdma(struct sdp_opt
 		/*
 		 * loop posting RDMA reads, if there is room.
 		 */
-		if (0 == sdp_iocb_q_size(&conn->r_pend)) {
-
+		if (0 == sdp_iocb_q_size(&conn->r_pend))
 			while (0 < sdp_advt_q_size(&conn->src_pend) &&
 			       conn->recv_max >
 			       sdp_buff_q_size(&conn->recv_pool) &&
 			       conn->rwin_max > conn->byte_strm) {
-
 				result = _sdp_post_rdma_buff(conn);
-				if (result) {
+				if (result)
 					/*
 					 * No more posts allowed.
 					 */
 					break;
-				}
 			}
-		}
 		/*
 		 * check non-zero result
 		 */
 		if (0 > result) {
-				
 			sdp_dbg_warn(conn, "Error <%d> posting RDMA BUFF read",
 				     result);
 			goto done;
 		}
-	}
-	else {
-
+	} else {
 		if (0 < sdp_iocb_q_size(&conn->r_pend) &&
 		    SDP_MODE_PIPE == conn->recv_mode &&
-		    0 == sdp_advt_q_size(&conn->src_actv)) {
-			
+		    0 == sdp_advt_q_size(&conn->src_actv))
 			while (0 == (result = _sdp_post_rdma_iocb_snk(conn))) {
 				/*
 				 * pass
 				 */
 			}
-		}
 
 		if (0 > result) {
-			
 			sdp_dbg_warn(conn, "Error <%d> posting RDMA read sink",
 				     result);
 			goto done;
@@ -620,10 +572,8 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	/*
 	 * verify that the connection is in a posting state
 	 */
-	if (0 == (SDP_ST_MASK_RCV_POST & conn->state)) {
-
+	if (0 == (SDP_ST_MASK_RCV_POST & conn->state))
 		return 0;
-	}
 	/*
 	 * loop posting receive buffers onto the queue
 	 */
@@ -647,18 +597,15 @@ int sdp_recv_flush(struct sdp_opt *conn)
 		      ((s32)conn->recv_cq_size - (s32)conn->l_recv_bf));
 
 	while (0 < counter--) {
-
 		result = _sdp_post_recv_buff(conn);
-		if (result) {
+		if (result)
 			/*
 			 * No more recv buffers allowed.
 			 */
 			break;
-		}
 	}
 
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting recv buff.", result);
 		goto done;
 	}
@@ -667,10 +614,8 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 * been consumed, we can come out of sink processing.
 	 */
 	if (0 < (SDP_CONN_F_SNK_CANCEL & conn->flags) &&
-	    0 == sdp_iocb_q_size(&conn->r_snk)) {
-
+	    0 == sdp_iocb_q_size(&conn->r_snk))
 		conn->flags &= ~SDP_CONN_F_SNK_CANCEL;
-	}
 	/*
 	 * Next the connection should consume RDMA Source advertisments or
 	 * create RDMA Sink advertisments, either way setup for RDMA's for
@@ -679,7 +624,6 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 */
 	result = _sdp_post_rdma(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting RDMAs.", result);
 		goto done;
 	}
@@ -703,10 +647,8 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	     conn->l_recv_bf > conn->l_advt_bf) ||
 	    (SDP_RECV_POST_ACK < (conn->l_recv_bf - conn->l_advt_bf) &&
 	     0 == ((u32)conn->snk_recv + (u32)conn->src_recv))) {
-
 		result = sdp_send_ctrl_ack(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> posting gratuitous ACK",
 				     result);
 			goto done;
@@ -759,10 +701,8 @@ static int _sdp_read_buff_iocb(struct sd
 		 * map correct page of iocb
 		 */
 		addr = __sdp_kmap(iocb->page_array[counter]);
-		if (NULL == addr) {
-			
+		if (NULL == addr)
 			break;
-		}
 
 		copy = min((PAGE_SIZE - offset),
 			   (unsigned long)(buff->tail - buff->data));
@@ -804,7 +744,6 @@ static int _sdp_recv_buff_iocb_active(st
 	 */
 	iocb = sdp_iocb_q_get_head(&conn->r_snk);
 	if (NULL == iocb) {
-
 		sdp_dbg_warn(conn, "Empty active IOCB queue. <%d>",
 			     sdp_iocb_q_size(&conn->r_snk));
 		return -EPROTO;
@@ -816,7 +755,6 @@ static int _sdp_recv_buff_iocb_active(st
 	 */
 	result = _sdp_read_buff_iocb(iocb, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
 			     (unsigned)(buff->tail - buff->data));
@@ -834,11 +772,9 @@ static int _sdp_recv_buff_iocb_active(st
 	 * callback to complete IOCB
 	 */
 	result = sdp_iocb_complete(iocb, 0);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 			     result, iocb->key);
-	}
 
 	return (buff->tail - buff->data);
 } /* _sdp_recv_buff_iocb_active */
@@ -859,7 +795,6 @@ static int _sdp_recv_buff_iocb_pending(s
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_pend);
 	if (NULL == iocb) {
-
 		sdp_dbg_warn(conn, "Empty pending IOCB queue. <%d>",
 			     sdp_iocb_q_size(&conn->r_pend));
 		return -EPROTO;
@@ -869,7 +804,6 @@ static int _sdp_recv_buff_iocb_pending(s
 	 */
 	result = _sdp_read_buff_iocb(iocb, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
 			     (unsigned)(buff->tail - buff->data));
@@ -899,11 +833,9 @@ static int _sdp_recv_buff_iocb_pending(s
 		 * callback to complete IOCB
 		 */
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
-		}
 	}
 
 	return (buff->tail - buff->data);
@@ -928,7 +860,6 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	 * closed. This notifies the peer that the data was not received.
 	 */
 	if (0 < (RCV_SHUTDOWN & conn->shutdown)) {
-
 		sdp_dbg_warn(conn, "Receive data path closed. <%02x>",
 			     conn->shutdown);
 		/*
@@ -946,7 +877,6 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	 * oob notification.
 	 */
 	if (0 < (SDP_BUFF_F_OOB_PEND & buff->flags)) {
-
 		conn->rcv_urg_cnt++;
 		sdp_inet_wake_urg(conn->sk);
 	}
@@ -965,33 +895,24 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 		 * or a socket recv (regular or OOB) is called.
 		 */
 		if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags) &&
-		    1 == (buff->tail - buff->data)) {
-
+		    1 == (buff->tail - buff->data))
 			break;
-		}
 		/*
 		 * process either a sink available IOCB which needs to be
 		 * discarded with exactly one buffer, or process a pending
 		 * IOCB.
 		 */
-		if (0 < conn->snk_sent) {
-
+		if (0 < conn->snk_sent)
 			result = _sdp_recv_buff_iocb_active(conn, buff);
-		}
-		else {
-
+		else
 			result = _sdp_recv_buff_iocb_pending(conn, buff);
-		}
 		/*
 		 * Check result. Postitive result is data left in the buffer
 		 */
-		if (0 == result) {
-
+		if (0 == result)
 			break;
-		}
 
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, 
 				     "Error <%d> processing IOCB. <%d:%d:%d>",
 				     result, conn->snk_sent,
@@ -1006,7 +927,6 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	buffered = buff->tail - buff->data;
 
 	if (0 < buffered) {
-
 		result = sdp_buff_q_put_tail(&conn->recv_pool, buff);
 		SDP_EXPECT(!(0 > result));
 	}
@@ -1030,15 +950,10 @@ static int _sdp_read_src_lookup(struct s
 
 	SDP_CHECK_NULL(element, -EINVAL);
 
-	if (SDP_DESC_TYPE_IOCB == element->type && 
-	    iocb->key == req->ki_key) {
-
+	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key)
 		return 0;
-	}
-	else {
-
+	else
 		return -ERANGE;
-	}
 } /* _sdp_read_src_lookup */
 
 /*
@@ -1057,10 +972,7 @@ static int _sdp_inet_read_cancel(struct 
 	sdp_dbg_ctrl(NULL, "Cancel Read IOCB. user <%d> key <%d> flag <%08lx>",
 		     req->ki_users, req->ki_key, req->ki_flags);
 	
-	if (NULL == si ||
-	    NULL == si->sock ||
-	    NULL == si->sock->sk) {
-          
+	if (NULL == si || NULL == si->sock || NULL == si->sock->sk) {
 		sdp_warn("Cancel empty read IOCB. users <%d> flags <%d:%08lx>",
 			 req->ki_users, req->ki_key, req->ki_flags);
 		result = -EFAULT;
@@ -1090,7 +1002,6 @@ static int _sdp_inet_read_cancel(struct 
 		SDP_EXPECT(!(0 > result));
 
 		if (0 == (SDP_IOCB_F_ACTIVE & iocb->flags)) {
-
 			if (0 < iocb->post) {
 				/*
 				 * callback to complete IOCB, or drop reference
@@ -1101,7 +1012,6 @@ static int _sdp_inet_read_cancel(struct 
 				result = -EAGAIN;
 			}
 			else {
-
 				result = sdp_iocb_destroy(iocb);
 				SDP_EXPECT(!(0 > result));
 				/*
@@ -1116,12 +1026,10 @@ static int _sdp_inet_read_cancel(struct 
 		}
 
 		if (0 < (SDP_IOCB_F_RDMA_W & iocb->flags)) {
-
 			result = sdp_iocb_q_put_tail(&conn->r_snk, iocb);
 			SDP_EXPECT(!(0 > result));
 		}
 		else {
-
 			SDP_EXPECT((SDP_IOCB_F_RDMA_R & iocb->flags));
 
 			result = sdp_desc_q_put_tail(&conn->r_src,
@@ -1137,7 +1045,6 @@ static int _sdp_inet_read_cancel(struct 
 						     _sdp_read_src_lookup,
 						     req);
 	if (NULL != iocb) {
-
 		iocb->flags |= SDP_IOCB_F_CANCEL;
 		result = -EAGAIN;
 
@@ -1206,14 +1113,12 @@ static int _sdp_inet_recv_urg_trav(struc
 	SDP_CHECK_NULL(value, -EINVAL);
 
 	if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
-
 		SDP_EXPECT((buff->tail > buff->data));
 
 		update = *value;
 		*value = *(u8 *) (buff->tail - 1);
 
 		if (0 < update) {
-
 			buff->tail--;
 			buff->flags &= ~SDP_BUFF_F_OOB_PRES;
 		}
@@ -1241,11 +1146,9 @@ static int _sdp_inet_recv_urg(struct soc
 	SDP_CHECK_NULL(msg, -EINVAL);
 	conn = SDP_GET_CONN(sk);
 
-	if (sock_flag(sk, SOCK_URGINLINE) ||
-	    0 == conn->rcv_urg_cnt) {
-
+	if (sock_flag(sk, SOCK_URGINLINE) || 0 == conn->rcv_urg_cnt)
 		return -EINVAL;
-	}
+
 	/*
 	 * don't cosume data on PEEK, but do consume data on TRUNC
 	 */
@@ -1259,24 +1162,19 @@ static int _sdp_inet_recv_urg(struct soc
 				      _sdp_inet_recv_urg_trav,
 				      (void *)&value);
 	if (-ERANGE != result) {
-
 		result = (0 != result) ? result : -EAGAIN;
 		goto done;
 	}
 
 	msg->msg_flags |= MSG_OOB;
 	if (0 < size) {
-
 		result = memcpy_toiovec(msg->msg_iov, &value, 1);
-		if (0 != result) {
-
+		if (0 != result)
 			goto done;
-		}
 		/*
 		 * clear urgent pointer on consumption
 		 */
 		if (0 == (MSG_PEEK & flags)) {
-
 			conn->rcv_urg_cnt -= 1;
 			conn->byte_strm -= 1;
 			
@@ -1290,7 +1188,6 @@ static int _sdp_inet_recv_urg(struct soc
 						_sdp_inet_recv_urg_test,
 						(void *)0);
 			if (NULL != buff) {
-
 				result = sdp_buff_pool_put(buff);
 				SDP_EXPECT(!(0 > result));
 
@@ -1300,9 +1197,7 @@ static int _sdp_inet_recv_urg(struct soc
 
 			result = 1;
 		}
-	}
-	else {
-
+	} else {
 		msg->msg_flags |= MSG_TRUNC;
 		result = 0;
 	}
@@ -1351,15 +1246,13 @@ int sdp_inet_recv(struct kiocb  *req,
 		     req->ki_key, msg->msg_iov->iov_base, 
 		     req->ki_users, req->ki_flags);
 
-	if (0 < (MSG_TRUNC & flags)) {
-		/*
-		 * TODO: unhandled, but need to be handled.
-		 */
+	/*
+	 * TODO: unhandled, but need to be handled.
+	 */
+	if (0 < (MSG_TRUNC & flags))
 		return -EOPNOTSUPP;
-	}
 
 	if (0 < (MSG_PEEK & flags)) {
-
 		(void)sdp_buff_q_init(&peek_queue, 0);
 		msg->msg_flags |= MSG_PEEK;
 	}
@@ -1368,7 +1261,6 @@ int sdp_inet_recv(struct kiocb  *req,
 
 	if (SDP_SOCK_ST_LISTEN == conn->istate ||
 	    SDP_SOCK_ST_CLOSED == conn->istate) {
-
 		result = -ENOTCONN;
 		goto done;
 	}
@@ -1376,7 +1268,6 @@ int sdp_inet_recv(struct kiocb  *req,
 	 * process urgent data
 	 */
 	if (0 < (MSG_OOB & flags)) {
-
 		result = _sdp_inet_recv_urg(sk, msg, size, flags);
 		copied = (0 < result) ? result : 0;
 		result = (0 < result) ? 0 : result;
@@ -1396,7 +1287,6 @@ int sdp_inet_recv(struct kiocb  *req,
 		 */
 		while (copied < size &&
 		       (buff = sdp_buff_q_get_head(&conn->recv_pool))) {
-
 			length = buff->tail - buff->data;
 			update = 0;
 
@@ -1411,12 +1301,9 @@ int sdp_inet_recv(struct kiocb  *req,
 				 * or not, to ensure that the user has a
 				 * chance to read the byte.
 				 */
-				if (1 < length) {
-
+				if (1 < length)
 					length--;
-				}
 				else {
-
 					if (0 < copied) {
 						/*
 						 * update such that we pass
@@ -1427,9 +1314,7 @@ int sdp_inet_recv(struct kiocb  *req,
 						length = 0;
 						update = 0;
 						oob = 1;    /* break on oob */
-					}
-					else {
-					  
+					} else {
 					  if (sock_flag(sk, SOCK_URGINLINE)) {
 					    /*
 							 * skip this byte, but
@@ -1454,7 +1339,6 @@ int sdp_inet_recv(struct kiocb  *req,
 							buff->data,
 							copy);
 				if (0 > result) {
-
 					expect =
 					    sdp_buff_q_put_head(&conn->
 								 recv_pool,
@@ -1474,7 +1358,6 @@ int sdp_inet_recv(struct kiocb  *req,
 			copied          += copy;
 
 			if (0 < (buff->tail - buff->data)) {
-
 				expect = sdp_buff_q_put_head(&conn->recv_pool,
 							     buff);
 				SDP_EXPECT(!(0 > expect));
@@ -1490,17 +1373,12 @@ int sdp_inet_recv(struct kiocb  *req,
 			}
 
 			if (MSG_PEEK & flags) {
-
 				expect = sdp_buff_q_put_head(&peek_queue,
 							     buff);
 				SDP_EXPECT(!(0 > expect));
-			}
-			else {
-
-				if (SDP_BUFF_F_OOB_PRES & buff->flags) {
-
+			} else {
+				if (SDP_BUFF_F_OOB_PRES & buff->flags)
 					conn->rcv_urg_cnt -= 1;
-				}
 				/*
 				 * create a link of buffers which
 				 * will be returned to the free pool
@@ -1517,12 +1395,9 @@ int sdp_inet_recv(struct kiocb  *req,
 				 * buffers...
 				 */
 				if (SDP_RECV_POST_FREQ < ++ack) {
-
 					result = sdp_recv_flush(conn);
-					if (0 > result) {
-
+					if (0 > result)
 						goto done;
-					}
 
 					ack = 0;
 				}
@@ -1533,7 +1408,6 @@ int sdp_inet_recv(struct kiocb  *req,
 		 * of low water mark, or whether there is room in the buffer.
 		 */
 		if (0 < oob) {
-
 			result = 0;
 			break;
 		}
@@ -1550,10 +1424,8 @@ int sdp_inet_recv(struct kiocb  *req,
 			 */
 			SDP_CONN_RELOCK(conn);
 
-			if (0 < sdp_buff_q_size(&conn->recv_pool)) {
-
+			if (0 < sdp_buff_q_size(&conn->recv_pool))
 				continue;
-			}
 		}
 		/*
 		 * If enough data has been copied to userspace break from
@@ -1564,7 +1436,6 @@ int sdp_inet_recv(struct kiocb  *req,
 		 */
 		if (!(copied < low_water) &&
 		    0 == conn->src_recv) {
-
 #if 0 /* performance cheat. LM */
 			if (!(conn->snk_zthresh > size)) {
 
@@ -1588,26 +1459,22 @@ int sdp_inet_recv(struct kiocb  *req,
 		 * check status. POSIX 1003.1g order.
 		 */
 		if (0 != SDP_CONN_GET_ERR(conn)) {
-
 			result = (0 < copied) ? 0 : SDP_CONN_ERROR(conn);
 			break;
 		}
 		
 		if (0 < (RCV_SHUTDOWN & conn->shutdown)) {
-			
 			result = 0;
 			break;
 		}
 
 		if (SDP_SOCK_ST_ERROR == conn->istate) {
-			
 			result = -EPROTO; /* error should always be
 					     set, but just in case */
 			break;
 		}
 
 		if (0 == timeout) {
-
 			result = -EAGAIN;
 			break;
 		}
@@ -1615,7 +1482,6 @@ int sdp_inet_recv(struct kiocb  *req,
 		 * Either wait or create IOCB for defered completion.
 		 */
 		if (is_sync_kiocb(req)) {
-
 			DECLARE_WAITQUEUE(wait, current);
 
 			add_wait_queue(sk->sk_sleep, &wait);
@@ -1624,7 +1490,6 @@ int sdp_inet_recv(struct kiocb  *req,
 			set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
 
 			if (0 == sdp_buff_q_size(&conn->recv_pool)) {
-
 				SDP_CONN_UNLOCK(conn);
 				timeout = schedule_timeout(timeout);
 				SDP_CONN_LOCK(conn);
@@ -1637,19 +1502,16 @@ int sdp_inet_recv(struct kiocb  *req,
 			 * check signal pending
 			 */
 			if (signal_pending(current)) {
-
 				result = ((0 < timeout) ?
 					  sock_intr_errno(timeout) : -EAGAIN);
 				break;
 			}
-		}
-		else {
+		} else {
 			/*
 			 * create IOCB with remaining space
 			 */
 			iocb = sdp_iocb_create();
 			if (NULL == iocb) {
-        
 				sdp_dbg_warn(conn, 
 					     "Error allocating IOCB <%Zu:%d>",
 					     size, copied);
@@ -1668,7 +1530,6 @@ int sdp_inet_recv(struct kiocb  *req,
 
 			result = sdp_iocb_lock(iocb);
 			if (0 > result) {
-        
 				sdp_dbg_warn(conn, 
 					     "Error <%d> IOCB lock <%Zu:%d>", 
 					     result, size, copied);
@@ -1681,7 +1542,6 @@ int sdp_inet_recv(struct kiocb  *req,
 
 			result = sdp_iocb_q_put_tail(&conn->r_pend, iocb);
 			if (0 > result) {
-				
 				sdp_dbg_warn(conn, 
 					     "Error <%d> IOCB queue <%Zu:%d>",
 					     result, size, copied);
@@ -1703,13 +1563,10 @@ done:
 	 * acknowledge moved data
 	 */
 	if (0 < ack) {
-
 		expect = sdp_recv_flush(conn);
-		if (0 > expect) {
-
+		if (0 > expect)
 			sdp_dbg_warn(conn, "Error <%d> flushing recv queue.",
 				     expect);
-		}
 	}
 
 	(void)sdp_buff_pool_chain_put(head, free_count);
@@ -1717,9 +1574,7 @@ done:
 	 * return any peeked buffers to the recv queue, in the correct order.
 	 */
 	if (0 < (MSG_PEEK & flags)) {
-
 		while (NULL != (buff = sdp_buff_q_get_tail(&peek_queue))) {
-
 			expect = sdp_buff_q_put_head(&conn->recv_pool, buff);
 			SDP_EXPECT(!(0 > expect));
 		}



From Tom.Duffy at Sun.COM  Fri Feb 18 16:47:37 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:47:37 -0800
Subject: [openib-general] [PATCH][SDP][19/22] Whitespace cleanup in
	sdp_send.c
In-Reply-To: <11087740473391@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
	<11087740262597@sun.com>
	<11087740372545@sun.com> <11087740473391@sun.com>
Message-ID: <11087740573361@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_send.c	(working copy)
@@ -86,7 +86,6 @@ static int _sdp_send_buff_post(struct sd
 	 */
 	if (0 < (SDP_CONN_F_OOB_SEND & conn->flags) &&
 	    !(0xFFFF < conn->oob_offset)) {
-
 		SDP_BSDH_SET_OOB_PEND(buff->bsdh_hdr);
 		SDP_BUFF_F_SET_SE(buff);
 
@@ -98,13 +97,9 @@ static int _sdp_send_buff_post(struct sd
 	 * is set if this buffer has never had pending set.
 	 */
 	if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
-
-		if (0 < conn->oob_offset) {
-
+		if (0 < conn->oob_offset)
 			SDP_BSDH_SET_OOB_PEND(buff->bsdh_hdr);
-		}
 		else {
-
 			SDP_EXPECT(!(0 > conn->oob_offset));
 			conn->oob_offset = -1;
 		}
@@ -115,21 +110,16 @@ static int _sdp_send_buff_post(struct sd
 	/*
 	 * solicite event bit.
 	 */
-	if (0 < SDP_BUFF_F_GET_SE(buff)) {
-
+	if (0 < SDP_BUFF_F_GET_SE(buff))
 		send_param.send_flags |= IB_SEND_SOLICITED;
-	}
 	/*
 	 * unsignalled event
 	 */
 	if (0 < SDP_BUFF_F_GET_UNSIG(buff) &&
 	    conn->usig_max > conn->send_cons) {
-
 		conn->send_usig++;
 		conn->send_cons++;
-	}
-	else {
-
+	} else {
 		SDP_BUFF_F_CLR_UNSIG(buff);
 		send_param.send_flags |= IB_SEND_SIGNALED;
 		conn->send_cons = 0;
@@ -138,7 +128,6 @@ static int _sdp_send_buff_post(struct sd
 	 * check queue membership. (first send attempt vs. flush)
 	 */
 	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff)) {
-
 		result = sdp_desc_q_remove((struct sdpc_desc *) buff);
 		SDP_EXPECT(!(0 > result));
 	}
@@ -147,7 +136,6 @@ static int _sdp_send_buff_post(struct sd
 	 */
 	result = sdp_buff_q_put_tail(&conn->send_post, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> queueing send buffer", result);
 		goto done;
 	}
@@ -167,7 +155,6 @@ static int _sdp_send_buff_post(struct sd
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
 	if (0 != result) {
-
 		sdp_dbg_warn(conn, 
 			     "Error <%d> posting send. <%d:%d> <%d:%d:%d>",
 			     result, conn->s_wq_cur, conn->s_wq_size,
@@ -182,13 +169,11 @@ static int _sdp_send_buff_post(struct sd
 	 * source cancels require us to save the sequence number
 	 * for validation of the cancel's completion.
 	 */
-	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags)) {
-
+	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags))
 		conn->src_cseq =
 		    ((SDP_MID_SRC_CANCEL ==
 		      buff->bsdh_hdr->mid) ? buff->bsdh_hdr->seq_num : conn->
 		     src_cseq);
-	}
 
 	return 0;
 done:
@@ -221,20 +206,17 @@ static int _sdp_send_data_buff_post(stru
 	 * 3) buffer from head of queue or as parameter
 	 * 4) nodelay check.
 	 */
-	if (3 > conn->r_recv_bf || 0 < conn->src_sent) {
-
+	if (3 > conn->r_recv_bf || 0 < conn->src_sent)
 		return ENOBUFS;
-	}
 	/*
 	 * The rest of the checks can proceed if there is a signalled event
 	 * in the pipe, otherwise we could stall...
 	 */
 	if (conn->send_usig < sdp_buff_q_size(&conn->send_post) ||
 	    0 < sdp_desc_q_size(&conn->w_snk)) {
-
 		if (buff->tail < buff->end &&
 		    0 == (SDP_BUFF_F_OOB_PRES & buff->flags) &&
-		    0 == conn->nodelay) {
+		    0 == conn->nodelay)
 			/*
 			 * If the buffer is not full, and there is already
 			 * data in the SDP pipe, then hold on to the buffer
@@ -243,7 +225,6 @@ static int _sdp_send_data_buff_post(stru
 			 * flush once it's full, which ever comes first.
 			 */
 			return ENOBUFS;
-		}
 		/*
 		 * slow start to give sink advertisments a chance for
 		 * asymmetric connections. This is desirable to offload
@@ -256,13 +237,9 @@ static int _sdp_send_data_buff_post(stru
 			 * sink advertisments.
 			 */
 			if (conn->send_cq_size > conn->s_wq_cur) {
-
-				if (SDP_SEND_POST_COUNT > conn->s_wq_par) {
-
+				if (SDP_SEND_POST_COUNT > conn->s_wq_par)
 					conn->s_wq_par++;
-				}
 				else {
-
 					conn->s_wq_cur++;
 					conn->s_wq_par = 0;
 				}
@@ -306,17 +283,13 @@ static int _sdp_send_data_buff_post(stru
 		 * update sink advertisments.
 		 */
 		conn->snk_recv--;
-	}
-	else {
-
+	} else
 		conn->nond_send++;
-	}
 	/*
 	 * transmision time
 	 */
 	result = _sdp_send_buff_post(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting send data buffer",
 			     result);
 		goto error;
@@ -346,17 +319,13 @@ static int _sdp_send_data_buff_snk(struc
 	 *
 	 * 1) sufficient send resources.
 	 */
-	if (!(conn->send_cq_size > conn->s_wq_size)) {
-
+	if (!(conn->send_cq_size > conn->s_wq_size))
 		return ENOBUFS;
-	}
 	/*
 	 * confirm type
 	 */
-	if (SDP_DESC_TYPE_BUFF != buff->type) {
-
+	if (SDP_DESC_TYPE_BUFF != buff->type)
 		return -ENOBUFS;
-	}
 	/*
 	 * nodelay buffering
 	 */
@@ -377,10 +346,8 @@ static int _sdp_send_data_buff_snk(struc
 	 * get advertisment.
 	 */
 	advt = sdp_advt_q_look(&conn->snk_pend);
-	if (NULL == advt) {
-
+	if (NULL == advt)
 		return ENOBUFS;
-	}
 	/*
 	 * signalled? With no delay turned off, data transmission may be
 	 * waiting for a send completion.
@@ -411,14 +378,12 @@ static int _sdp_send_data_buff_snk(struc
 	 * dequeue if needed and the queue buffer
 	 */
 	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff)) {
-
 		result = sdp_desc_q_remove((struct sdpc_desc *) buff);
 		SDP_EXPECT(!(0 > result));
 	}
 
 	result = sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff);
         if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> queueing write buffer. <%d>",
 			     result, sdp_desc_q_size(&conn->w_snk));
 		goto error;
@@ -439,7 +404,6 @@ static int _sdp_send_data_buff_snk(struc
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
 	if (0 != result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting rdma write", result);
 
 		conn->s_wq_size--;
@@ -450,7 +414,6 @@ static int _sdp_send_data_buff_snk(struc
 	 * advertisment.
 	 */
 	if (conn->send_size > advt->size) {
-
 		advt = sdp_advt_q_get(&conn->snk_pend);
 		SDP_EXPECT((NULL != advt));
 
@@ -461,7 +424,6 @@ static int _sdp_send_data_buff_snk(struc
 
 		result = sdp_send_ctrl_rdma_wr(conn, zcopy);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> completing sink. <%d>",
 				     result, zcopy);
 			result = -ENODEV;
@@ -500,7 +462,6 @@ static int _sdp_send_data_iocb_snk(struc
 	 */
 	result = sdp_iocb_register(iocb, conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> registering IOCB. <%d:%d>",
 			     result, iocb->key, iocb->len);
 		goto error;
@@ -513,10 +474,8 @@ static int _sdp_send_data_iocb_snk(struc
 		 * get the pending sink advertisment.
 		 */
 		advt = sdp_advt_q_look(&conn->snk_pend);
-		if (NULL == advt) {
-
+		if (NULL == advt)
 			break;
-		}
 		/*
 		 * amount of data to zcopy.
 		 */
@@ -562,7 +521,6 @@ static int _sdp_send_data_iocb_snk(struc
 
 		result = ib_post_send(conn->qp, &send_param, &bad_wr);
 		if (0 != result) {
-
 			sdp_dbg_warn(conn, "Error <%d> posting rdma write",
 				     result);
 
@@ -576,14 +534,11 @@ static int _sdp_send_data_iocb_snk(struc
 		 * outstanding sink advertisment to see how the advt
 		 * size remaining is picked.)
 		 */
-		if (!(conn->send_size > advt->size)) {
-
+		if (!(conn->send_size > advt->size))
 			continue;
-		}
 
 		advt = sdp_advt_q_get(&conn->snk_pend);
 		if (NULL == advt) {
-
 			sdp_dbg_warn(conn, "sink advertisment disappeared.");
 			result = -ENODEV;
 			goto error;
@@ -596,7 +551,6 @@ static int _sdp_send_data_iocb_snk(struc
 
 		result = sdp_send_ctrl_rdma_wr(conn, zcopy);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> completing sink. <%d>",
 				     result, zcopy);
 			result = -ENODEV;
@@ -629,34 +583,22 @@ static int _sdp_send_data_iocb_src(struc
 	 * 1) local source cancel is pending
 	 * 2) sufficient send credits for buffered transmission.
 	 */
-	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags) ||
-	    3 > conn->r_recv_bf) {
-
+	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags) || 3 > conn->r_recv_bf)
 		return ENOBUFS;
-	}
 
 	switch (conn->send_mode) {
 	case SDP_MODE_PIPE:
-
-		if (!(conn->s_cur_adv > conn->src_sent)) {
-
+		if (!(conn->s_cur_adv > conn->src_sent))
 			return ENOBUFS;
-		}
 
 		if (conn->s_cur_adv < conn->r_max_adv) {
-
 			if (!(SDP_SEND_POST_FRACTION > conn->s_par_adv)) {
-
 				conn->s_cur_adv++;
 				conn->s_par_adv = 0;
 			}
-			else {
-
+			else
 				conn->s_par_adv++;
-			}
-		}
-		else {
-
+		} else {
 			conn->s_cur_adv = conn->r_max_adv;
 			conn->s_par_adv = 0;
 		}
@@ -667,15 +609,10 @@ static int _sdp_send_data_iocb_src(struc
 #endif
 		break;
 	case SDP_MODE_COMB:
-
-		if (0 < conn->src_sent) {
-
+		if (0 < conn->src_sent)
 			return ENOBUFS;
-		}
-
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Unexpected SrcAvail mode. <%d>",
 			     conn->send_mode);
 		return -EPROTO;
@@ -685,7 +622,6 @@ static int _sdp_send_data_iocb_src(struc
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Error allocating SrcAvail buffer. <%d>",
 			     iocb->key);
 		return -ENOMEM;
@@ -697,7 +633,6 @@ static int _sdp_send_data_iocb_src(struc
 	 */
 	result = sdp_iocb_register(iocb, conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> registering IOCB. <%d:%d>",
 			     result, iocb->key, iocb->len);
 		goto drop;
@@ -739,7 +674,6 @@ static int _sdp_send_data_iocb_src(struc
 		len = min(SDP_SRC_AVAIL_MIN, (int)(PAGE_SIZE - off));
 
 		if (len > iocb->len) {
-
 			sdp_dbg_warn(conn, "Data <%d:%d:%d> from IOCB <%d:%d>",
 				     len, pos, off,
 				     iocb->page_count, 
@@ -753,7 +687,6 @@ static int _sdp_send_data_iocb_src(struc
 		 */
 		addr = __sdp_kmap(iocb->page_array[pos]);
 		if (NULL == addr) {
-
 			result = -ENOMEM;
 			goto error;
 		}
@@ -782,7 +715,6 @@ static int _sdp_send_data_iocb_src(struc
 	 */
 	result = _sdp_send_buff_post(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting SrcAvail for IOCB <%d>",
 			     result, iocb->key);
 		goto release;
@@ -828,10 +760,8 @@ static int _sdp_send_iocb_buff_write(str
 		 * map correct page of iocb
 		 */
 		addr = __sdp_kmap(iocb->page_array[counter]);
-		if (NULL == addr) {
-
+		if (NULL == addr)
 			break;
-		}
 		
 		copy = min((PAGE_SIZE - offset),
 			   (unsigned long)(buff->end - buff->tail));
@@ -867,10 +797,8 @@ static int _sdp_send_data_iocb_buff(stru
 	SDP_CHECK_NULL(conn, -EINVAL);
 	SDP_CHECK_NULL(iocb, -EINVAL);
 
-	if (0 < conn->src_sent) {
-
+	if (0 < conn->src_sent)
 		return ENOBUFS;
-	}
 	/*
 	 * loop through queued buffers and copy them to the destination
 	 */
@@ -884,7 +812,6 @@ static int _sdp_send_data_iocb_buff(stru
 		 */
 		buff = sdp_buff_pool_get();
 		if (NULL == buff) {
-
 			result = -ENOMEM;
 			goto error;
 		}
@@ -906,7 +833,6 @@ static int _sdp_send_data_iocb_buff(stru
 		 */
 		result =_sdp_send_iocb_buff_write(iocb, buff);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> copy from IOCB <%d>.",
 				     result, iocb->key);
 			goto drop;
@@ -922,7 +848,6 @@ static int _sdp_send_data_iocb_buff(stru
 		 */
 		result = _sdp_send_buff_post(conn, buff);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> send queue buff post",
 				     result);
 			goto drop;
@@ -946,22 +871,17 @@ static int _sdp_send_data_iocb(struct sd
 	SDP_CHECK_NULL(conn, -EINVAL);
 	SDP_CHECK_NULL(iocb, -EINVAL);
 
-	if (!(conn->send_cq_size > conn->s_wq_size)) {
-
+	if (!(conn->send_cq_size > conn->s_wq_size))
 		goto done;
-	}
 	/*
 	 * confirm IOCB usage.
 	 */
-	if (SDP_DESC_TYPE_IOCB != iocb->type) {
-
+	if (SDP_DESC_TYPE_IOCB != iocb->type)
 		return -ENOBUFS;
-	}
 	/*
 	 * determin if we are sending Buffered, Source or Sink.
 	 */
 	if (0 < sdp_advt_q_size(&conn->snk_pend)) {
-
 		result = _sdp_send_data_iocb_snk(conn, iocb);
 		if (0 == result) {
 			/*
@@ -970,7 +890,6 @@ static int _sdp_send_data_iocb(struct sd
 			 * failure or partial processing.
 			 */
 			if (0 < sdp_desc_q_member((struct sdpc_desc *)iocb)) {
-
 				result =
 				    sdp_desc_q_remove((struct sdpc_desc *)
 							    iocb);
@@ -980,7 +899,6 @@ static int _sdp_send_data_iocb(struct sd
 			result = sdp_desc_q_put_tail(&conn->w_snk,
 						     (struct sdpc_desc *) iocb);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, 
 					     "Error <%d> queuing write IOCB.",
 					     result);
@@ -994,22 +912,18 @@ static int _sdp_send_data_iocb(struct sd
 	 * hope that a new sink advertisment will arrive, because 
 	 * sinks are more efficient.
 	 */
-	if (0 < sdp_desc_q_size(&conn->w_snk)) {
-
+	if (0 < sdp_desc_q_size(&conn->w_snk))
 		goto done;
-	}
 
 	if (conn->src_zthresh > iocb->len ||
 	    SDP_MODE_BUFF == conn->send_mode ||
 	    0 < (SDP_IOCB_F_BUFF & iocb->flags)) {
-
 		result = _sdp_send_data_iocb_buff(conn, iocb);
 		if (0 == result) {
 			/*
 			 * complete this IOCB
 			 */
 			if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb)) {
-
 				result = sdp_desc_q_remove((struct sdpc_desc *) iocb);
 				SDP_EXPECT(!(0 > result));
 			}
@@ -1018,12 +932,10 @@ static int _sdp_send_data_iocb(struct sd
 			SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
 			result = sdp_iocb_complete(iocb, 0);
-			if (0 > result) {
-
+			if (0 > result)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> completing iocb <%d>",
 					     result, iocb->key);
-			}
 		}
 
 		goto done;
@@ -1035,18 +947,15 @@ static int _sdp_send_data_iocb(struct sd
 		 * queue IOCB
 		 */
 		if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb)) {
-
 			result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
 			SDP_EXPECT(!(0 > result));
 		}
 
 		result = sdp_iocb_q_put_tail(&conn->w_src, iocb);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> queueing write <%d:%d>",
 				     result, iocb->key,
 				     sdp_iocb_q_size(&conn->w_src));
-		}
 	}
 
 done:
@@ -1069,27 +978,19 @@ static int _sdp_send_data_queue_test(str
 	 * 2) source advertisment cancel in progress.
 	 */
 	if (0 == (SDP_ST_MASK_SEND_OK & conn->state) ||
-	    0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags)) {
-
+	    0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags))
 		return ENOBUFS;
-	}
-
-	if (SDP_DESC_TYPE_IOCB == element->type) {
 
+	if (SDP_DESC_TYPE_IOCB == element->type)
 		return _sdp_send_data_iocb(conn, (struct sdpc_iocb *)element);
-	}
 
 	if (0 == sdp_advt_q_look(&conn->snk_pend) ||
-	    (SDP_BUFF_F_OOB_PRES & ((struct sdpc_buff *)element)->flags)) {
-
+	    (SDP_BUFF_F_OOB_PRES & ((struct sdpc_buff *)element)->flags))
 		result = _sdp_send_data_buff_post(conn,
 						  (struct sdpc_buff *)element);
-	}
-	else {
-
+	else
 		result = _sdp_send_data_buff_snk(conn,
 						 (struct sdpc_buff *)element);
-	}
 
 	return result;
 } /* _sdp_send_data_queue_test */
@@ -1108,22 +1009,17 @@ static int _sdp_send_data_queue_flush(st
 	 * non-zero result is generated.
 	 * (positive: no space; negative: error)
 	 */
-	while (!result &&
-	       0 < sdp_desc_q_size(&conn->send_queue)) {
-
+	while (!result && 0 < sdp_desc_q_size(&conn->send_queue)) {
 		element = sdp_desc_q_look_head(&conn->send_queue);
 		SDP_EXPECT((NULL != element));
 
 		result = _sdp_send_data_queue_test(conn, element);
-		if (!result) {
-
+		if (!result)
 			continue;
-		}
 		/*
 		 * error
 		 */
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> post data during flush",
 				     result);
 			/*
@@ -1132,7 +1028,6 @@ static int _sdp_send_data_queue_flush(st
 			 * element, and not know how to requeue it.
 			 */
 			if (0 == sdp_desc_q_member(element)) {
-
 				result = sdp_desc_q_put_head(&conn->send_queue,
 							     element);
 				SDP_EXPECT(!(0 > result));
@@ -1162,10 +1057,8 @@ static int _sdp_send_data_queue(struct s
 	 */
 	if (0 < sdp_desc_q_size(&conn->send_queue) ||
 	    0 < (result = _sdp_send_data_queue_test(conn, element))) {
-
 		result = sdp_desc_q_put_tail(&conn->send_queue, element);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> queueing data for send",
 				     result);
 			goto done;
@@ -1176,11 +1069,9 @@ static int _sdp_send_data_queue(struct s
 		if (SDP_MODE_COMB == conn->send_mode &&
 		    !(SDP_INET_SEND_MODE >
 		      sdp_desc_q_size(&conn->send_queue))) {
-
 			result = sdp_send_ctrl_mode_ch(conn,
 						       SDP_MSG_MCH_PIPE_RECV);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, 
 					     "Error <%d> posting mode change",
 					     result);
@@ -1190,7 +1081,6 @@ static int _sdp_send_data_queue(struct s
 	}
 
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> during data send posting",
 			     result);
 		goto done;
@@ -1218,10 +1108,8 @@ static __inline__ struct sdpc_buff *_sdp
 	if (NULL == buff ||
 	    buff->tail == buff->end ||
 	    0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
-
 		buff = sdp_buff_pool_get();
 		if (NULL != buff) {
-
 			buff->tail = buff->end - conn->send_size;
 			buff->data = buff->tail;
 		}
@@ -1247,13 +1135,11 @@ static __inline__ int _sdp_send_data_buf
 	 * See note on send OOB implementation in SendBuffPost.
 	 */
 	if (0 < urg) {
-
 		buff->flags |= SDP_BUFF_F_OOB_PRES;
 		/*
 		 * The OOB PEND and PRES flags need to match up as pairs.
 		 */
 		if (0 > conn->oob_offset) {
-
 			conn->oob_offset = conn->send_pipe + size;
 			conn->flags |= SDP_CONN_F_OOB_SEND;
 		}
@@ -1263,13 +1149,10 @@ static __inline__ int _sdp_send_data_buf
 	 * buffer and dosn't need to be queued now.
 	 */
 	if (0 < (SDP_BUFF_F_QUEUED & buff->flags)) {
-
 		buff->data_size += size;
 		conn->send_qud += size;
 		conn->send_pipe += size;
-	}
-	else {
-
+	} else {
 		buff->data_size = buff->tail - buff->data;
 		conn->send_qud += buff->data_size;
 		conn->send_pipe += buff->data_size;
@@ -1280,7 +1163,6 @@ static __inline__ int _sdp_send_data_buf
 		 */
 		result = _sdp_send_data_queue(conn, (struct sdpc_desc *) buff);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> buffer to SEND queue.",
 				     result);
 
@@ -1310,16 +1192,13 @@ static int _sdp_send_ctrl_buff_test(stru
 	if (0 == (SDP_ST_MASK_CTRL_OK & conn->state) ||
 	    !(conn->send_cq_size > conn->s_wq_size) ||
 	    !(0 < conn->r_recv_bf) ||
-	    (conn->l_recv_bf == conn->l_advt_bf && 1 == conn->r_recv_bf)) {
-
+	    (conn->l_recv_bf == conn->l_advt_bf && 1 == conn->r_recv_bf))
 		return ENOBUFS;
-	}
 	/*
 	 * post the control buffer
 	 */
 	result = _sdp_send_buff_post(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control send", result);
 		goto error;
 	}
@@ -1343,25 +1222,20 @@ static int _sdp_send_ctrl_buff_flush(str
 	 * result is generated. (positive: no space; negative: error)
 	 */
 	while (!result && 0 < sdp_desc_q_size(&conn->send_ctrl)) {
-		
 		element = sdp_desc_q_look_head(&conn->send_ctrl);
 		SDP_EXPECT((NULL != element));
 
 		result = _sdp_send_ctrl_buff_test(conn, 
 						  (struct sdpc_buff *)element);
-		if (!result) {
-
+		if (!result)
 			continue;
-		}
 
 		if (0 > result) {
-			
 			sdp_dbg_warn(conn,
 				     "Error <%d> failed to flush control msg",
 				     result);
 
 			if (0 == sdp_desc_q_member(element)) {
-
 				result = sdp_desc_q_put_head(&conn->send_ctrl,
 							     element);
 				SDP_EXPECT(!(0 > result));
@@ -1393,7 +1267,6 @@ static int _sdp_send_ctrl_buff_buffered(
 		result = sdp_desc_q_put_tail(&conn->send_ctrl,
 					     (struct sdpc_desc *)buff);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> queueing control buff",
 				     result);
 			goto error;
@@ -1401,7 +1274,6 @@ static int _sdp_send_ctrl_buff_buffered(
 	}
 
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> during control send posting",
 			     result);
 		goto error;
@@ -1430,7 +1302,6 @@ static int _sdp_send_ctrl_buff(struct sd
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for control");
 		result = -ENOMEM;
 		goto error;
@@ -1446,31 +1317,22 @@ static int _sdp_send_ctrl_buff(struct sd
 	/*
 	 * solicite event flag for IB sends.
 	 */
-	if (se) {
-
+	if (se)
 		SDP_BUFF_F_SET_SE(buff);
-	}
-	else {
-
+	else
 		SDP_BUFF_F_CLR_SE(buff);
-	}
 	/*
 	 * try for unsignalled?
 	 */
-	if (sig) {
-
+	if (sig)
 		SDP_BUFF_F_CLR_UNSIG(buff);
-	}
-	else {
-
+	else
 		SDP_BUFF_F_SET_UNSIG(buff);
-	}
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1495,7 +1357,6 @@ static int _sdp_send_ctrl_disconnect(str
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for disconnect");
 		result = -ENOMEM;
 		goto error;
@@ -1521,7 +1382,6 @@ static int _sdp_send_ctrl_disconnect(str
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		goto error;
@@ -1548,26 +1408,19 @@ int sdp_send_ctrl_disconnect(struct sdp_
 	 * sent once data is flushed.
 	 */
 	if (0 < (SDP_ST_MASK_DIS_PEND & conn->state)) {
-
 		if (0 == (SDP_CONN_F_DIS_HOLD & conn->flags) &&
 		    0 == sdp_desc_q_size(&conn->send_queue) &&
-		    0 == conn->src_sent) {
-
+		    0 == conn->src_sent)
 			result = _sdp_send_ctrl_disconnect(conn);
-		}
 		else {
-
 			sdp_dbg_ctrl(conn, "defer disconnect <%d:%d> <%08x>",
 				     sdp_desc_q_size(&conn->send_queue),
 				     conn->src_sent, conn->flags);
 
 			result = 0;
 		}
-	}
-	else {
-
+	} else
 		result = -EPROTO;
-	}
 	return result;
 } /* sdp_send_ctrl_disconnect */
 
@@ -1588,10 +1441,8 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
 	 */
 	if (0 < sdp_desc_q_size(&conn->send_ctrl) ||
 	    (0 < sdp_desc_q_size(&conn->send_queue) &&
-	     2 < conn->l_advt_bf)) {
-
+	     2 < conn->l_advt_bf))
 		return 0;
-	}
 
 	return _sdp_send_ctrl_buff(conn, SDP_MID_DATA, 0, 0);
 } /* sdp_send_ctrl_ack */
@@ -1656,7 +1507,6 @@ int sdp_send_ctrl_resize_buff_ack(struct
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for resize ack");
 		result = -ENOMEM;
 		goto error;
@@ -1684,7 +1534,6 @@ int sdp_send_ctrl_resize_buff_ack(struct
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1708,7 +1557,6 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	 * check size
 	 */
 	if (0 > size) {
-
 		sdp_dbg_warn(conn, "RDMA read completion <%d> too small.",
 			     size);
 		return -ERANGE;
@@ -1719,7 +1567,6 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for RDMA rd");
 		result = -ENOMEM;
 		goto error;
@@ -1757,7 +1604,6 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1783,7 +1629,6 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for RDMA wr");
 		result = -ENOMEM;
 		goto error;
@@ -1817,7 +1662,6 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1844,7 +1688,6 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	 * check mode
 	 */
 	if (SDP_MODE_PIPE != conn->recv_mode) {
-
 		result = -EPROTO;
 		goto error;
 	}
@@ -1854,7 +1697,6 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for SnkAvail");
 		result = -ENOMEM;
 		goto error;
@@ -1886,7 +1728,6 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		goto error;
@@ -1911,29 +1752,23 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	 * validate that the requested mode transition is OK.
 	 */
 	switch (mode) {
-
 	case SDP_MSG_MCH_BUFF_RECV:	/* source to sink */
-
 		conn->send_mode = ((SDP_MODE_COMB == conn->send_mode) ?
 				   SDP_MODE_BUFF : SDP_MODE_ERROR);
 		break;
 	case SDP_MSG_MCH_COMB_SEND:	/* sink to source */
-
 		conn->recv_mode = ((SDP_MODE_BUFF == conn->recv_mode) ?
 				   SDP_MODE_COMB : SDP_MODE_ERROR);
 		break;
 	case SDP_MSG_MCH_PIPE_RECV:	/* source to sink */
-
 		conn->send_mode = ((SDP_MODE_COMB == conn->send_mode) ?
 				   SDP_MODE_PIPE : SDP_MODE_ERROR);
 		break;
 	case SDP_MSG_MCH_COMB_RECV:	/* source to sink */
-
 		conn->send_mode = ((SDP_MODE_PIPE == conn->send_mode) ?
 				   SDP_MODE_COMB : SDP_MODE_ERROR);
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Invalid mode transition <%d:%d:%d>",
 			     mode, conn->send_mode, conn->recv_mode);
 		result = -EPROTO;
@@ -1942,7 +1777,6 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 
 	if (SDP_MODE_ERROR == conn->send_mode ||
 	    SDP_MODE_ERROR == conn->recv_mode) {
-
 		sdp_dbg_warn(conn, "mode transition error <%d:%d:%d>",
 			     mode, conn->send_mode, conn->recv_mode);
 		result = -EPROTO;
@@ -1954,7 +1788,6 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	 */
 	buff = sdp_buff_pool_get();
 	if (NULL == buff) {
-
 		sdp_dbg_warn(conn, "Failed to allocate buffer for ModeChange");
 		result = -ENOMEM;
 		goto error;
@@ -1982,7 +1815,6 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -2006,14 +1838,10 @@ static int _sdp_write_src_lookup(struct 
 
 	SDP_CHECK_NULL(element, -EINVAL);
 
-	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key) {
-
+	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key)
 		return 0;
-	}
-	else {
-
+	else
 		return -ERANGE;
-	}
 } /* _sdp_write_src_lookup */
 
 /*
@@ -2031,10 +1859,7 @@ static int _sdp_inet_write_cancel(struct
 	sdp_dbg_ctrl(NULL, "Cancel Write IOCB user <%d> key <%d> flag <%08lx>",
 		     req->ki_users, req->ki_key, req->ki_flags);
 	
-	if (NULL == si ||
-	    NULL == si->sock ||
-	    NULL == si->sock->sk) {
-		
+	if (NULL == si || NULL == si->sock || NULL == si->sock->sk) {
 		sdp_warn("Cancel empty write IOCB users <%d> flags <%d:%08lx>",
 			 req->ki_users, req->ki_key, req->ki_flags);
 		result = -EFAULT;
@@ -2065,36 +1890,28 @@ static int _sdp_inet_write_cancel(struct
 		SDP_EXPECT(!(0 > result));
 		
 		if (0 < (SDP_IOCB_F_ACTIVE & iocb->flags)) {
-
 			if (0 < (SDP_IOCB_F_RDMA_W & iocb->flags)) {
-				
 				result = sdp_desc_q_put_tail(&conn->w_snk,
 							     (struct sdpc_desc *)iocb);
 				SDP_EXPECT(!(0 > result));
-			}
-			else {
-
+			} else {
 				SDP_EXPECT((SDP_IOCB_F_RDMA_R & iocb->flags));
 
 				result = sdp_iocb_q_put_tail(&conn->w_src,
 							     iocb);
 				SDP_EXPECT(!(0 > result));
 			}
-		}
-		else {
+		} else {
 			/*
 			 * empty IOCBs can be deleted, while partials
 			 * needs to be compelted.
 			 */
 			if (0 < iocb->post) {
-
 				result = sdp_iocb_complete(iocb, 0);
 				SDP_EXPECT(!(0 > result));
 
 				result = -EAGAIN;
-			}
-			else {
-
+			} else {
 				result = sdp_iocb_destroy(iocb);
 				SDP_EXPECT(!(0 > result));
 				/*
@@ -2117,7 +1934,6 @@ static int _sdp_inet_write_cancel(struct
 						     req);
 
 	if (NULL != iocb) {
-
 		iocb->flags |= SDP_IOCB_F_CANCEL;
 		result = -EAGAIN;
 
@@ -2142,12 +1958,10 @@ static int _sdp_inet_write_cancel(struct
 		 */
 		if (0 == (SDP_CONN_F_SRC_CANCEL_L & conn->flags) &&
 		    0 == (SDP_IOCB_F_CANCEL & iocb->flags)) {
-
 			conn->src_cncl++;
 			iocb->flags |= SDP_IOCB_F_CANCEL;
 
 			if (conn->src_cncl == sdp_iocb_q_size(&conn->w_src)) {
-
 				result = sdp_send_ctrl_src_cancel(conn);
 				SDP_EXPECT(!(0 > result));
 
@@ -2196,7 +2010,6 @@ static int _sdp_send_flush_advt(struct s
 		 */
 		advt = sdp_advt_q_look(&conn->snk_pend);
 		if (NULL != advt && 0 < advt->post) {
-
 			advt = sdp_advt_q_get(&conn->snk_pend);
 			SDP_EXPECT((NULL != advt));
 
@@ -2230,7 +2043,6 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 */
 	result = _sdp_send_ctrl_buff_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing control", result);
 		goto done;
 	}
@@ -2239,7 +2051,6 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 */
 	result = _sdp_send_data_queue_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing data queue", result);
 		goto done;
 	}
@@ -2247,10 +2058,8 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * Sink advertisment flush.
 	 */
 	if (0 < sdp_advt_q_size(&conn->snk_pend)) {
-
 		result = _sdp_send_flush_advt(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn,
 				     "Error <%d> flushing sink advertisments",
 				     result);
@@ -2261,10 +2070,8 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * disconnect flush
 	 */
 	if (0 < (SDP_ST_MASK_DIS_PEND & conn->state)) {
-
 		result = sdp_send_ctrl_disconnect(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> flushing disconnect",
 				     result);
 			goto done;
@@ -2322,7 +2129,6 @@ int sdp_inet_send(struct kiocb *req,
 	 * is opened on sucessful connect, or error on an unsucessful attempt.
 	 */
 	if (0 < (SDP_ST_MASK_CLOSED & conn->istate)) {
-
 		result = -EPIPE;
 		goto done;
 	}
@@ -2332,11 +2138,8 @@ int sdp_inet_send(struct kiocb *req,
 	 * they are smaller then the zopy threshold, but only if there is
 	 * no buffer write space.
 	 */
-	if (!(conn->src_zthresh > size) &&
-	    !is_sync_kiocb(req)) {
-		
+	if (!(conn->src_zthresh > size) && !is_sync_kiocb(req))
 		goto skip;
-	}
 	/*
 	 * clear ASYN space bit, it'll be reset if there is no space.
 	 */
@@ -2353,10 +2156,8 @@ int sdp_inet_send(struct kiocb *req,
 		 * data to allow some space for sending.
 		 */
 		while (0 < __sdp_inet_write_space(conn, oob)) {
-
 			buff = _sdp_send_data_buff_get(conn);
 			if (NULL == buff) {
-
 				result = -ENOMEM;
 				goto done;
 			}
@@ -2370,7 +2171,6 @@ int sdp_inet_send(struct kiocb *req,
 						  msg->msg_iov,
 						  copy);
 			if (0 > result) {
-
 				(void)sdp_buff_pool_put(buff);
 				goto done;
 			}
@@ -2383,15 +2183,11 @@ int sdp_inet_send(struct kiocb *req,
 			result = _sdp_send_data_buff_put(conn, buff, copy,
 							 ((copied ==
 							 size) ? oob : 0));
-			if (0 > result) {
-
+			if (0 > result)
 				goto done;
-			}
-
-			if (copied == size) {
 
+			if (copied == size)
 				goto done;
-			}
 		}
 		/*
 		 * set no space bits since this code path is taken
@@ -2408,39 +2204,32 @@ skip: /* entry point for IOCB based tran
 		/*
 		 * onetime setup of timeout, but only if it's needed.
 		 */
-		if (0 > timeout) {
-
+		if (0 > timeout)
 			timeout = sock_sndtimeo(sk, (MSG_DONTWAIT &
 						     msg->msg_flags));
-		}
 
 		if (0 != SDP_CONN_GET_ERR(conn)) {
-
 			result = (0 < copied) ? 0 : SDP_CONN_ERROR(conn);
 			break;
 		}
 
 		if (0 < (SEND_SHUTDOWN & conn->shutdown)) {
-
 			result = -EPIPE;
 			break;
 		}
 
 		if (SDP_SOCK_ST_ERROR == conn->istate) {
-
 			result = -EPROTO; /* error should always be set, but
 					     just in case */
 			break;
 		}
 
 		if (0 == timeout) {
-
 			result = -EAGAIN;
 			break;
 		}
 
 		if (signal_pending(current)) {
-
 			result =
 			    (0 < timeout) ? sock_intr_errno(timeout) : -EAGAIN;
 			break;
@@ -2451,7 +2240,6 @@ skip: /* entry point for IOCB based tran
 		 * call.
 		 */
 		if (is_sync_kiocb(req)) {
-
 			DECLARE_WAITQUEUE(wait, current);
 
 			add_wait_queue(sk->sk_sleep, &wait);
@@ -2465,10 +2253,8 @@ skip: /* entry point for IOCB based tran
 			clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
 			SDP_CONN_UNLOCK(conn);
-			if (!(0 < __sdp_inet_write_space(conn, oob))) {
-
+			if (!(0 < __sdp_inet_write_space(conn, oob)))
 				timeout = schedule_timeout(timeout);
-			}
 			SDP_CONN_LOCK(conn);
 
 			remove_wait_queue(sk->sk_sleep, &wait);
@@ -2481,7 +2267,6 @@ skip: /* entry point for IOCB based tran
 		 */
 		iocb = sdp_iocb_create();
 		if (NULL == iocb) {
-			
 			sdp_dbg_warn(conn, "Failed to allocate IOCB <%Zu:%d>",
 				     size, copied);
 			result = -ENOMEM;
@@ -2499,7 +2284,6 @@ skip: /* entry point for IOCB based tran
 
 		result = sdp_iocb_lock(iocb);
 		if (0 > result) {
-				
 			sdp_dbg_warn(conn, "Error <%d> locking IOCB <%Zu:%d>", 
 				     result, size, copied);
         
@@ -2513,7 +2297,6 @@ skip: /* entry point for IOCB based tran
 
 		result = _sdp_send_data_queue(conn, (struct sdpc_desc *)iocb);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> queueing write IOCB",
 				     result);
 
@@ -2529,10 +2312,8 @@ done:
 	SDP_CONN_UNLOCK(conn);
 	result = ((0 < copied) ? copied : result);
 
-	if (-EPIPE == result && 0 == (MSG_NOSIGNAL & msg->msg_flags)) {
-
+	if (-EPIPE == result && 0 == (MSG_NOSIGNAL & msg->msg_flags))
 		send_sig(SIGPIPE, current, 0);
-	}
 
 	return result;
 } /* sdp_inet_send */



From Tom.Duffy at Sun.COM  Fri Feb 18 16:47:47 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:47:47 -0800
Subject: [openib-general] [PATCH][SDP][20/22] Whitespace cleanup in
	sdp_sent.c
In-Reply-To: <11087740573361@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
	<11087740262597@sun.com>
	<11087740372545@sun.com> <11087740473391@sun.com>
	<11087740573361@sun.com>
Message-ID: <11087740671739@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_sent.c	(working copy)
@@ -70,7 +70,6 @@ static int _sdp_sent_disconnect(struct s
 		 */
 		result = sdp_cm_disconnect(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> posting CM disconnect",
 				     result);
 			goto error;
@@ -89,7 +88,6 @@ static int _sdp_sent_disconnect(struct s
 		 */
 		result = sdp_wall_recv_closing(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> closing connection.",
 				     result);
 			goto error;
@@ -102,14 +100,11 @@ static int _sdp_sent_disconnect(struct s
 		 * if the remote DREQ was already received, but unprocessed, do
 		 * not treat it as an error
 		 */
-		if (0 > result) {
-			
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
-		}
 
 		break;
 	default:
-
 		sdp_warn("Disconnect sent, unexpected state. <%d> <%04x:%04d>",
 			 conn->hashent, conn->istate, conn->state);
 		result = -EFAULT;
@@ -137,10 +132,8 @@ static int _sdp_sent_abort(struct sdp_op
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
-	}
 
 	return result;
 } /* _sdp_sent_abort */
@@ -391,7 +384,6 @@ int sdp_event_send(struct sdp_opt *conn,
 	 * error handling
 	 */
 	if (IB_WC_SUCCESS != comp->status) {
-
 		switch (comp->status) {
 		case IB_WC_WR_FLUSH_ERR:
 			/*
@@ -405,7 +397,6 @@ int sdp_event_send(struct sdp_opt *conn,
 			result = 0;
 			break;
 		default:
-
 			sdp_dbg_warn(conn, "Unhandled SEND error status <%d>.",
 				     comp->status);
 			result = -EIO;
@@ -421,7 +412,6 @@ int sdp_event_send(struct sdp_opt *conn,
 		 * sanity checks
 		 */
 		if (NULL == buff->bsdh_hdr) {
-
 			sdp_dbg_warn(conn, "Send header is missing?!");
 			result = -ENODATA;
 			goto drop;
@@ -458,17 +448,13 @@ int sdp_event_send(struct sdp_opt *conn,
 		 * data fast path we collapse the next level dispatch function.
 		 * For all other buffers we go the slow path.
 		 */
-		if (SDP_MID_DATA == buff->bsdh_hdr->mid) {
-
+		if (SDP_MID_DATA == buff->bsdh_hdr->mid)
 			conn->send_qud -= buff->data_size;
-		}
 		else {
-
 			offset = buff->bsdh_hdr->mid & 0x1F;
 
 			if (!(offset < SDP_MSG_EVENT_TABLE_SIZE) ||
 			    NULL == send_event_funcs[offset]) {
-
 				sdp_dbg_warn(conn, 
 					     "Send complete unknown MID <%d>",
 					     buff->bsdh_hdr->mid);
@@ -481,7 +467,6 @@ int sdp_event_send(struct sdp_opt *conn,
 			dispatch_func = send_event_funcs[offset];
 			result = dispatch_func(conn, buff);
 			if (0 > result) {
-
 				sdp_dbg_warn(conn, "Sent dispatch error. <%d>",
 					     result);
 				goto drop;
@@ -494,10 +479,8 @@ int sdp_event_send(struct sdp_opt *conn,
 		 */
 		conn->s_wq_size--;
 
-		if (0 < SDP_BUFF_F_GET_UNSIG(buff)) {
-
+		if (0 < SDP_BUFF_F_GET_UNSIG(buff))
 			conn->send_usig--;
-		}
 		/*
 		 * create a link of buffers which will be returned to
 		 * the free pool in one group.
@@ -508,17 +491,14 @@ int sdp_event_send(struct sdp_opt *conn,
 		head = buff;
 		free_count++;
 
-		if (comp->wr_id == current_wrid) {
-
+		if (comp->wr_id == current_wrid)
 			break;
-		}
 	}
 
 	result = sdp_buff_pool_chain_put(head, free_count);
 	SDP_EXPECT(!(0 > result));
 
 	if (!(0 < free_count) || 0 > conn->send_usig) {
-
 		sdp_dbg_warn(conn, 
 			     "Send processing mismatch. <%llu:%llu:%d:%d>",
 			     (unsigned long long)comp->wr_id,
@@ -532,7 +512,6 @@ int sdp_event_send(struct sdp_opt *conn,
 	 */
 	result = sdp_send_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue.", result);
 		goto done;
 	}



From Tom.Duffy at Sun.COM  Fri Feb 18 16:47:58 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:47:58 -0800
Subject: [openib-general] [PATCH][SDP][21/22] Whitespace cleanup in
	sdp_wall.c
In-Reply-To: <11087740671739@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
	<11087740262597@sun.com>
	<11087740372545@sun.com> <11087740473391@sun.com>
	<11087740573361@sun.com> <11087740671739@sun.com>
Message-ID: <11087740783433@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_wall.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_wall.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_wall.c	(working copy)
@@ -64,12 +64,9 @@ int sdp_wall_send_close(struct sdp_opt *
 		 * clear out the sent HelloAck message
 		 */
 		buff = sdp_buff_q_get_head(&conn->send_post);
-		if (NULL == buff) {
-
+		if (NULL == buff)
 			sdp_dbg_warn(conn, "Error, hello ack missing.");
-		}
 		else {
-
 			result = sdp_buff_pool_put(buff);
 			SDP_EXPECT(!(0 > result));
 		}
@@ -92,7 +89,6 @@ int sdp_wall_send_close(struct sdp_opt *
 
 		result = sdp_send_ctrl_disconnect(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, 
 				     "Error <%d> send disconnect request",
 				     result);
@@ -114,10 +110,8 @@ error:
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
-	}
 
 	return result;
 } /* sdp_wall_send_close */
@@ -150,7 +144,6 @@ int sdp_wall_send_closing(struct sdp_opt
 
 		result = sdp_send_ctrl_disconnect(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn,
 				     "Error <%d> send disconnect request",
 				     result);
@@ -158,7 +151,6 @@ int sdp_wall_send_closing(struct sdp_opt
 		}
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Incorrect state for disconnect");
 		result = -EBADE;
 		goto error;
@@ -172,10 +164,8 @@ error:
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
-	}
 
 	return result;
 } /* sdp_wall_send_closing */
@@ -194,7 +184,6 @@ int sdp_wall_send_abort(struct sdp_opt *
 		     conn->dst_addr, conn->dst_port);
 
 	switch (conn->state) {
-
 	case SDP_CONN_ST_DIS_SENT_1:	/* IBTA v1.1 spec A4.5.3.2 */
 		/*
 		 * clear the pending control buffer.
@@ -206,7 +195,6 @@ int sdp_wall_send_abort(struct sdp_opt *
 		 */
 	case SDP_CONN_ST_DIS_SEND_1:	/* don't touch control queue, diconnect
 					   message may still be queued. */
-
 		result = sdp_desc_q_clear(&conn->send_queue);
 		SDP_EXPECT(!(0 > result));
 		/*
@@ -214,7 +202,6 @@ int sdp_wall_send_abort(struct sdp_opt *
 		 */
 		result = sdp_send_ctrl_abort(conn);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> send abort request",
 				     result);
 			goto error;
@@ -287,10 +274,8 @@ error:
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
-	}
 
 	return 0;
 } /* sdp_wall_send_abort */
@@ -321,11 +306,9 @@ int sdp_wall_recv_reject(struct sdp_opt 
 	conn->shutdown = SHUTDOWN_MASK;
 
 	result = sdp_iocb_q_cancel_all(conn, (0 - error));
-	if (0 > result) {
-		
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
 			     result);
-	}
 
 	sdp_inet_wake_error(conn->sk);
 
@@ -351,7 +334,6 @@ int sdp_wall_recv_confirm(struct sdp_opt
 
         qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
         if (!qp_attr) {
-
                 result = -ENOMEM;
 		goto error;
         }
@@ -362,7 +344,6 @@ int sdp_wall_recv_confirm(struct sdp_opt
 
 	result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
 	if (result) {
-		
 		sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS",
 			     result);
 		goto error;
@@ -370,20 +351,16 @@ int sdp_wall_recv_confirm(struct sdp_opt
 
         result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
         kfree(qp_attr);
-	if (result) {
-	
+	if (result)
 		goto error;
-	}
 
 	switch (conn->istate) {
 	case SDP_SOCK_ST_ACCEPTING:
-
 		conn->istate = SDP_SOCK_ST_ACCEPTED;
 		conn->send_buf = SDP_INET_SEND_SIZE;
 
 		break;
 	case SDP_SOCK_ST_ACCEPTED:
-
 		conn->istate = SDP_SOCK_ST_ESTABLISHED;
 		conn->send_buf = SDP_INET_SEND_SIZE;
 
@@ -391,7 +368,6 @@ int sdp_wall_recv_confirm(struct sdp_opt
 
 		break;
 	default:
-
 		result = -EPROTO;
 		goto error;
 	}
@@ -425,7 +401,6 @@ int sdp_wall_recv_failed(struct sdp_opt 
 
 	switch (conn->istate) {
 	default:
-
 		SDP_CONN_SET_ERR(conn, error);
 		conn->sk->sk_socket->state = SS_UNCONNECTED;
 		sdp_inet_wake_error(conn->sk);
@@ -433,7 +408,6 @@ int sdp_wall_recv_failed(struct sdp_opt 
 		 * fall through
 		 */
 	case SDP_SOCK_ST_ACCEPTING:
-
 		conn->istate = SDP_SOCK_ST_ERROR;
 		conn->shutdown = SHUTDOWN_MASK;
 		break;
@@ -470,7 +444,6 @@ int sdp_wall_recv_close(struct sdp_opt *
 
 		break;
 	default:
-
 		conn->istate = SDP_SOCK_ST_CLOSE;
 		conn->shutdown |= RCV_SHUTDOWN;
 		/*
@@ -478,12 +451,10 @@ int sdp_wall_recv_close(struct sdp_opt *
 		 * no more data from the peer.
 		 */
 		result = sdp_iocb_q_cancel_all_read(conn, 0);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, 
 				     "Error <%d> canceling outstanding IOCBs",
 				     result);
-		}
 		/*
 		 * async notification. POLL_HUP on full duplex close only.
 		 */
@@ -521,11 +492,9 @@ int sdp_wall_recv_closing(struct sdp_opt
 	 * from the peer.
 	 */
 	result = sdp_iocb_q_cancel_all_read(conn, 0);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
 			     result);
-	}
 
 	sdp_inet_wake_generic(conn->sk);
 	/*
@@ -574,11 +543,9 @@ int sdp_wall_recv_abort(struct sdp_opt *
 	 * cancel all outstanding IOCBs
 	 */
 	result = sdp_iocb_q_cancel_all(conn, -ECONNRESET);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
 			     result);
-	}
 
 	sdp_inet_wake_error(conn->sk);
 
@@ -610,18 +577,15 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 		 * pull the listen sockets accept queue.
 		 */
 		result = sdp_inet_accept_q_remove(conn);
-		if (0 > result) {
-			
+		if (0 > result)
 			sdp_dbg_warn(conn, 
 				     "Error <%d> removing from accept queue.",
 				     result);
-		}
 
 		conn->istate = SDP_SOCK_ST_CLOSED;
 
 		break;
 	case SDP_SOCK_ST_CLOSING:
-
 		conn->istate = SDP_SOCK_ST_CLOSED;
 		sdp_inet_wake_generic(conn->sk);
 
@@ -638,11 +602,9 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 		 * cancel all outstanding IOCBs
 		 */
 		result = sdp_iocb_q_cancel_all(conn, -ECONNRESET);
-		if (0 > result) {
-
+		if (0 > result)
 			sdp_dbg_warn(conn, "Error <%d> canceling all IOCBs",
 				     result);
-		}
 
 		sdp_inet_wake_error(conn->sk);
 
@@ -672,16 +634,12 @@ int sdp_wall_abort(struct sdp_opt *conn)
 	 * notify both halves of the wall that the connection is being aborted.
 	 */
 	result = sdp_wall_recv_abort(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> recving abort request", result);
-	}
 	/* if */
 	result = sdp_wall_send_abort(conn);
-	if (0 > result) {
-
+	if (0 > result)
 		sdp_dbg_warn(conn, "Error <%d> sending abort request", result);
-	}
 
 	return 0;
 } /* sdp_wall_abort */



From Tom.Duffy at Sun.COM  Fri Feb 18 16:48:08 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Fri, 18 Feb 2005 16:48:08 -0800
Subject: [openib-general] [PATCH][SDP][22/22] Whitespace cleanup in
	sdp_write.c
In-Reply-To: <11087740783433@sun.com>
References: <11087738623823@sun.com> <11087738723775@sun.com>
	<11087738832487@sun.com> <1108773893431@sun.com>
	<11087739032962@sun.com>
	<1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com>
	<11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com>
	<11087740063185@sun.com> <11087740164018@sun.com>
	<11087740262597@sun.com>
	<11087740372545@sun.com> <11087740473391@sun.com>
	<11087740573361@sun.com>
	<11087740671739@sun.com> <11087740783433@sun.com>
Message-ID: <1108774088665@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c
===================================================================
--- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c	(revision 1836)
+++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_write.c	(working copy)
@@ -54,7 +54,6 @@ int sdp_event_write(struct sdp_opt *conn
 	 * error handling
 	 */
 	if (IB_WC_SUCCESS != comp->status) {
-
 		switch (comp->status) {
 		case IB_WC_WR_FLUSH_ERR:
 			/*
@@ -92,7 +91,6 @@ int sdp_event_write(struct sdp_opt *conn
 	type = sdp_desc_q_type_head(&conn->w_snk);
 	switch (type) {
 	case SDP_DESC_TYPE_BUFF:
-
 		buff = (struct sdpc_buff *) sdp_desc_q_get_head(&conn->w_snk);
 		SDP_EXPECT((NULL != buff));
 
@@ -103,7 +101,6 @@ int sdp_event_write(struct sdp_opt *conn
 
 		break;
 	case SDP_DESC_TYPE_IOCB:
-
 		iocb = (struct sdpc_iocb *)sdp_desc_q_look_head(&conn->w_snk);
 		if (NULL == iocb || iocb->wrid != comp->wr_id) {
 
@@ -120,7 +117,6 @@ int sdp_event_write(struct sdp_opt *conn
 
 		result = sdp_iocb_complete(iocb, 0);
 		if (0 > result) {
-
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto error;
@@ -128,11 +124,9 @@ int sdp_event_write(struct sdp_opt *conn
 
 		break;
 	case SDP_DESC_TYPE_NONE:
-
 		iocb = (struct sdpc_iocb *)sdp_desc_q_look_type_head(&conn->send_queue,
 								     SDP_DESC_TYPE_IOCB);
 		if (NULL == iocb) {
-
 			sdp_dbg_warn(conn, 
 				     "No IOCB on write complete <%llu:%d:%d>",
 				     (unsigned long long)comp->wr_id,
@@ -154,7 +148,6 @@ int sdp_event_write(struct sdp_opt *conn
 
 		break;
 	default:
-
 		sdp_warn("Unknown type <%d> at head of WRITE SINK queue. <%d>",
 			 type, sdp_desc_q_size(&conn->w_snk));
 		result = -EPROTO;
@@ -170,7 +163,6 @@ int sdp_event_write(struct sdp_opt *conn
 	 */
 	result = sdp_send_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue.", result);
 		goto error;
 	}
@@ -180,7 +172,6 @@ int sdp_event_write(struct sdp_opt *conn
 	 */
 	result = sdp_recv_flush(conn);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> flushing recv queue.", result);
 		goto error;
 	}



From mst at mellanox.co.il  Sat Feb 19 16:05:36 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 20 Feb 2005 02:05:36 +0200
Subject: [openib-general] Re: [PATCH][SDP][14/22] Whitespace cleanup in
	sdp_proto.h
In-Reply-To: <11087740063185@sun.com>
References: <1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com> <11087739441194@sun.com>
	<11087739553583@sun.com> <11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
Message-ID: <20050220000536.GB8268@mellanox.co.il>


Quoting r. Tom Duffy <Tom.Duffy at Sun.COM>:
> Subject: [PATCH][SDP][14/22] Whitespace cleanup in sdp_proto.h
> 
> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proto.h
> ===================================================================
> --- linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(revision 1836)
> +++ linux-2.6.10-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(working copy)
> @@ -599,14 +599,10 @@ static __inline__ int __sdp_inet_write_s

I think we also should replace __inline__ with inline everywhere.

Also, I think the convention is to add __ before inline functions
if they are an implementation detail and shouldnt be called
directly.
That does not seem to be the case for __sdp_inet_write_space, or for
other stuff in sdp_proto.h, so I think these functions shall have proper names
starting with an alphanumeric.


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sat Feb 19 16:11:37 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 20 Feb 2005 02:11:37 +0200
Subject: [openib-general] Re: Re: SDP_CONN_LOCK
In-Reply-To: <20050217162413.E7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
Message-ID: <20050220001137.GC8268@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: Re: SDP_CONN_LOCK
> 
> On Thu, Feb 17, 2005 at 03:49:31PM -0800, Sean Hefty wrote:
> > Michael S. Tsirkin wrote:
> > > Quoting r. Roland Dreier <roland at topspin.com>:
> > >>
> > >>BTW, since mthca currently calls completion handlers directly from
> > >>interrupt context (rather than BH/tasklet context), it might be worth
> > >>renaming all the SDP locking macros so they're not confusingly named
> > >>with _BH suffixes.
> > > 
> > > I think it would be much nicer to reduce the number of macros used.
> > 
> > I'd have to agree with this.  The SDP locking macros are fairly complex 
> > and hide a lot of functionality.  E.g. SDP_CONN_RELOCK results in 
> > polling/rearming the CQ, same with SDP_CONN_UNLOCK.  Maybe that's just 
> > a naming issue though.
> > 
> > I think these would probably be better off as just function calls, 
> > rather than macros.  SDP_CONN_LOCK calls sdp_conn_internal_lock(), and 
> > that appears to be the only place that the function is called. 
> > Similarly, SDP_CONN_UNLOCK calls sdp_conn_internal_unlock().  It seems 
> > that you could just merge the macros into the function calls.
> 
>   OK. I had used macros since in the fast path the functions wouldn't
> get called, but since it sounds like it makes things less readable I
> should be able to quickly try out replacing the macros with functions,
> and see what if any effect it has on performance.
> 
>   And here I would have thought SDP_CONN_HOLD as a macro would have
> bugged people more. :)
> 
> -Libor
> 
Didn't get that far yet :)

I also wander whether we could get rid of most called to 
_CHECK macros, consequently turn lots of functions
which cant otherwise fail into void?
Given that these checks are disabled upon a compile switch, one wanders
how big a debugging aid they are.

Consider sdp_wall_recv_accept calling sdp_wall_failedm both checking
their parameter for NULL - why does it make sence?


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sat Feb 19 16:15:04 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 20 Feb 2005 02:15:04 +0200
Subject: [openib-general] Re: [PATCH] remove unused code from SDP.
In-Reply-To: <20050218120306.C15428@topspin.com>
References: <20050218120306.C15428@topspin.com>
Message-ID: <20050220001503.GD8268@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: [PATCH] remove unused code from SDP.
> 
> 
>   There's code in SDP which is never used, mostly helper functions that
> at the time looked like they might be useful, but never got used.
> 
> -Libor
> 
> Signed-off-by: Libor Michalek <libor at topspin.com>
> 
> Index: sdp_link.c
> ===================================================================
> --- sdp_link.c	(revision 1831)
> +++ sdp_link.c	(working copy)
> @@ -321,7 +321,7 @@
>  /*
>   * _sdp_link_path_rec_get - resolve GIDs to a path record
>   */
> -int _sdp_link_path_rec_get(struct sdp_path_info *info)
> +static int _sdp_link_path_rec_get(struct sdp_path_info *info)
>  {
>  	int result;
>  

Good stuff. 
One comment though - one can give short names to static functions:
since they are local to file they wont conflict with anything other.
So - why prepend an "_" to the name?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 20 04:45:15 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 20 Feb 2005 14:45:15 +0200
Subject: [openib-general] linked lists in sdp
Message-ID: <20050220124515.GH28446@mellanox.co.il>

Hi!
sdp_advt seems to re-implement linked list.
Why is not the standard linked list used?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 20 05:16:40 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 20 Feb 2005 15:16:40 +0200
Subject: [openib-general] Re: Re: SDP_CONN_LOCK
In-Reply-To: <20050217162413.E7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
Message-ID: <20050220131640.GI28446@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
>   And here I would have thought SDP_CONN_HOLD as a macro would have
> bugged people more. :)
> 
> -Libor
> 

Regarding conn_hold/conn_put (macro or not):
1. conn_put is sometimes called while socket is locked (sdp_cm_connect),
   sometimes not (sdp_ip2pr_path_complete). What gives?
   Is it OK to destroy the socket while its locked? Seems strange ...

2. When conn_put is called *outside* the lock, what prevents conn_hold
   from being called while conn_put is in progress, thus hold will be
   acting in parallel with destroy?


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 20 07:07:53 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 20 Feb 2005 17:07:53 +0200
Subject: [openib-general] SDP_CONN_LOCK
In-Reply-To: <20050217161839.D7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
	<20050217221330.GA21318@mellanox.co.il>
	<20050217144615.C7121@topspin.com>
	<20050217232539.GB21318@mellanox.co.il>
	<20050217161839.D7121@topspin.com>
Message-ID: <20050220150753.GB25098@mellanox.co.il>

Hi, Libor!
I am not against optimizations, I am just expressing the
general sentiment that optimizations and tuning are
hard to judge on a theoretical basis.

It is not obvious to me that draining cq on each socket unlock
is a win, even if it was a win on gen1.



Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [openib-general] SDP_CONN_LOCK
> 
> On Fri, Feb 18, 2005 at 01:25:39AM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > On Fri, Feb 18, 2005 at 12:13:30AM +0200, Michael S. Tsirkin wrote:
> > > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > > On Thu, Feb 17, 2005 at 11:29:37PM +0200, Michael S. Tsirkin wrote:
> > > > > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > > > > On Thu, Feb 17, 2005, Michael S. Tsirkin wrote:
> > > > > > > > Hi, Libor!
> > > > > > > > Could you please explain what are the SDP_CONN_LOCK
> > > > > > > > and friends doing in SDP?
> > > > > > > > 
> > > > > > > > It seems they just implement exclusive access to socket,
> > > > > > > > but if so, why is a simple semaphore or mutex not used?
> > > > > > > 
> > > > > > >   They do implement exclusive access to the socket, but they
> > > > > > >   implement exclusive access from both process and irq
> > > > > > >   context, which is why a semaphore was not used. In
> > > > > > >   interrupt context SDP_CONN_LOCK_BH is used to lock the
> > > > > > >   connection, look in sdp_cq_event_handler() for it's use,
> > > > > > >   and in process context SDP_CONN_LOCK is used.
> > > > > > 
> > > > > > I dont really understand how it works.
> > > > > > When an interrupt arrives while users != 0, it seems you are
> > > > > > calling scheduler().  What is sdp_conn_internal_lock doing?
> > > > > > I understand it is to be called from interrupt context, but
> > > > > > how can it call scheduler() then?
> > > > > 
> > > > >   SDP_CONN_LOCK and SDP_CONN_UNLOCK are never called from
> > > > >   interrupt context, only from process context. The spinlock
> > > > >   conn->lock.slock is called in interrupt context and process
> > > > >   context before the 'users' variable is changed or read. If
> > > > >   users != 0, then the connection is in use by process
> > > > >   context. If the connection is in use then another lock
> > > > >   request in process context will sleep until users == 0, if
> > > > >   the connection is in use then a lock request in interrupt
> > > > >   context will mark the connection as having had an event and
> > > > >   return. When a process context lock holder releases the
> > > > >   lock, it checks if there has been an interrupt event, which
> > > > >   it then processes by draining and arming the CQ, and it
> > > > >   wakes any process context waiting for the lock.
> > > > 
> > > > Cant all this be done by a semaphore in process context, and 
> > > > down_trylock in interrupt context?
> > > > The "connection has an event" bit could be protected by a spinlock.
> > > 
> > >   The CQ draining happens inside of the spinlock. If the CQ drain
> > > happened in process context inside of a semaphore, then there is
> > > a race condition for the interrupt between checking the semaphore
> > > and setting the spinlock protected "connection event bit". The
> > > CQ drain and the "event bit" have to happen under the same lock
> > > to avoid the race. 
> > > 
> > >   If you move the CQ drain and the "event bit" into the same spinlock
> > > where do you put the semaphore? In process context the spin would
> > > need to be inside the semaphore lock, but in interrupt context the
> > > semaphore down_trylock and up are inside the spinlock. Now there would
> > > be a race between spin_unlock() followed by up() in process context
> > > and spin_lock() followed by down_trylock() in interrupt context.
> > > 
> > >    spin_unlock()
> > >                   spin_lock()
> > >                   down_trylock()
> > >    up()
> > 
>
> [...]
>
> > Anyway, this flow is just an optimisation - so I wander how much does 
> > it help, actually.
> 
>   No, it's not just an optimization. The drain in process context is a
> optimization, but having a lock protecting in both contexts is necessary.

Thats what I mean. Cant we simply drain the CQ from the event handler.
Events overhead is much lower with gen2 than gen1, and the advantage
with CQ drain upon event is that you always actually have a completion
to handle.

This means access to sdp buffers will have to be spinlock protected,
but on the other hand you get rid of the spinlock protected "user" flag.


>   Without the drain in process context you would need to handle the actual
> CQ event, but if the connection is locked in process context, what do
> you do with the event? You either wait for the lock to be released, or
> you process it to a point, and then queue it for further handling once
> the lock is released. Waiting for a lock release is not practical, and 
> processing to a point is what we are currently doing, just we are
> processing it the bare minimum, and leaving the rest for process
> context.

Isnt this what spin_lock_irq is for? It shall protect the data
which can be accessed from both and irq and a process.

>   Setting the event bit and processing of the CQ in process context was
> actually a big simplification which not surprisingly performed much
> better as well.
> 
> > My understanding this is a backport from gen1, but what was a useful
> > optimisation for gen1 may not be for gen2. gen2 event flow is much 
> > faster than gen1 - it is possible all the extra locking and
> > complications are not worth it with mthca.
> 
>   I agree with the general sentiment. However, with mthca faster then the
> gen1 code, this code should help even more, since gen2 would be able to
> generate more interrupts a second, if we let it. At the time this has
> added, it resulted in a 2/3 reduction in interrupts per second. 

Since event are dispatched faster, the overhead of extra locking,
and the overhead of polling the cq when you dont actually know whether
it has a completion, is larger.

>   Next, in this portion of the code I was actually going to try using a
> single CQ instead of two to see if we get an additional improvement as a
> result of fewer interrupts and simpler code.
> 
> 
> -Libor

I think this is likely to help.

-- 
MST - Michael S. Tsirkin


From blist at aon.at  Sun Feb 20 08:20:23 2005
From: blist at aon.at (Bernhard Fischer)
Date: Sun, 20 Feb 2005 17:20:23 +0100
Subject: [openib-general] Re: [openib-commits] r1821
Message-ID: <20050220162023.GQ4169@aon.at>

Hi,

Minor commentary changes.

Thank you,
-------------- next part --------------
diff -X ../excl -rdup upstream/gen2/trunk/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c gen2/trunk/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c
--- upstream/gen2/trunk/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c	2005-02-19 14:33:27.000000000 +0100
+++ gen2/trunk/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c	2005-02-19 14:45:40.000000000 +0100
@@ -490,7 +490,7 @@ __validate_more_comp_fields(
       if(mtu_mgrp < mtu_required)
         return FALSE;
       break;
-    case 1: /* Less than than MTU specified */
+    case 1: /* Less than MTU specified */
       if(mtu_mgrp > mtu_required)
         return FALSE;
       break;
@@ -516,7 +516,7 @@ __validate_more_comp_fields(
       if(rate_mgrp < rate_required)
         return FALSE;
       break;
-    case 1: /* Less than than RATE specified */
+    case 1: /* Less than RATE specified */
       if(rate_mgrp > rate_required)
         return FALSE;
       break;

From halr at voltaire.com  Sun Feb 20 08:50:41 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 20 Feb 2005 11:50:41 -0500
Subject: [openib-general] Re: [openib-commits] r1821
In-Reply-To: <20050220162023.GQ4169@aon.at>
References: <20050220162023.GQ4169@aon.at>
Message-ID: <1108918241.4645.220.camel@localhost.localdomain>

On Sun, 2005-02-20 at 11:20, Bernhard Fischer wrote:
> Hi,
> 
> Minor commentary changes.

Thanks. Applied.

-- Hal

> 
> Thank you,
> 
> ______________________________________________________________________
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From mst at mellanox.co.il  Mon Feb 21 03:29:58 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 21 Feb 2005 13:29:58 +0200
Subject: [openib-general] [PATCH] initialize device_cap_flags in mthca
Message-ID: <20050221112958.GD14855@mellanox.co.il>

device_cap_flags are uninitialized in mthca.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_dev.h
===================================================================
--- hw/mthca/mthca_dev.h	(revision 1846)
+++ hw/mthca/mthca_dev.h	(working copy)
@@ -130,6 +130,7 @@ struct mthca_limits {
 	int      reserved_mcgs;
 	int      num_pds;
 	int      reserved_pds;
+	u32      device_cap_flags;
 };
 
 struct mthca_alloc {
Index: hw/mthca/mthca_main.c
===================================================================
--- hw/mthca/mthca_main.c	(revision 1846)
+++ hw/mthca/mthca_main.c	(working copy)
@@ -171,6 +171,38 @@ static int __devinit mthca_dev_lim(struc
 	mdev->limits.reserved_uars      = dev_lim->reserved_uars;
 	mdev->limits.reserved_pds       = dev_lim->reserved_pds;
 
+	/* IB_DEVICE_RESIZE_MAX_WR not supported by driver.
+	   May be doable since hardware supports it for SRQ.
+
+	   IB_DEVICE_N_NOTIFY_CQ is supported by hardware but not by driver.
+
+	   IB_DEVICE_SRQ_RESIZE is supported by hardware but SRQ is not
+	   supported by driver. */
+	mdev->limits.device_cap_flags = IB_DEVICE_CHANGE_PHY_PORT |
+		IB_DEVICE_PORT_ACTIVE_EVENT |
+		IB_DEVICE_SYS_IMAGE_GUID |
+		IB_DEVICE_RC_RNR_NAK_GEN;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_BAD_PKEY_CNTR)
+		mdev->limits.device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_BAD_QKEY_CNTR)
+		mdev->limits.device_cap_flags |= IB_DEVICE_BAD_QKEY_CNTR;
+				
+	if (dev_lim->flags & DEV_LIM_FLAG_RAW_MULTI)
+		mdev->limits.device_cap_flags |= IB_DEVICE_RAW_MULTI;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_AUTO_PATH_MIG)
+		mdev->limits.device_cap_flags |= IB_DEVICE_AUTO_PATH_MIG;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_UD_AV_PORT_ENFORCE)
+		mdev->limits.device_cap_flags |= IB_DEVICE_UD_AV_PORT_ENFORCE;
+
+	/* IB_DEVICE_RQ_SIG_TYPE not supported in native mode */
+	if (mdev->hca_type != ARBEL_NATIVE)
+		mdev->limits.device_cap_flags |= IB_DEVICE_RQ_SIG_TYPE;
+
+
 	if (dev_lim->flags & DEV_LIM_FLAG_SRQ)
 		mdev->mthca_flags |= MTHCA_FLAG_SRQ;
 
Index: hw/mthca/mthca_provider.c
===================================================================
--- hw/mthca/mthca_provider.c	(revision 1846)
+++ hw/mthca/mthca_provider.c	(working copy)
@@ -43,6 +43,8 @@ static int mthca_query_device(struct ib_
 	struct ib_smp *in_mad  = NULL;
 	struct ib_smp *out_mad = NULL;
 	int err = -ENOMEM;
+	struct mthca_dev* mdev = to_mdev(ibdev);
+
 	u8 status;
 
 	in_mad  = kmalloc(sizeof *in_mad, GFP_KERNEL);
@@ -50,7 +52,7 @@ static int mthca_query_device(struct ib_
 	if (!in_mad || !out_mad)
 		goto out;
 
-	props->fw_ver        = to_mdev(ibdev)->fw_ver;
+	props->fw_ver              = mdev->fw_ver;
 
 	memset(in_mad, 0, sizeof *in_mad);
 	in_mad->base_version       = 1;
@@ -59,7 +61,7 @@ static int mthca_query_device(struct ib_
 	in_mad->method         	   = IB_MGMT_METHOD_GET;
 	in_mad->attr_id   	   = IB_SMP_ATTR_NODE_INFO;
 
-	err = mthca_MAD_IFC(to_mdev(ibdev), 1, 1,
+	err = mthca_MAD_IFC(mdev, 1, 1,
 			    1, NULL, NULL, in_mad, out_mad,
 			    &status);
 	if (err)
@@ -69,10 +71,11 @@ static int mthca_query_device(struct ib_
 		goto out;
 	}
 
-	props->vendor_id      = be32_to_cpup((u32 *) (out_mad->data + 36)) &
+	props->device_cap_flags = mdev->limits.device_cap_flags;
+	props->vendor_id        = be32_to_cpup((u32 *) (out_mad->data + 36)) &
 		0xffffff;
-	props->vendor_part_id = be16_to_cpup((u16 *) (out_mad->data + 30));
-	props->hw_ver         = be16_to_cpup((u16 *) (out_mad->data + 32));
+	props->vendor_part_id   = be16_to_cpup((u16 *) (out_mad->data + 30));
+	props->hw_ver           = be16_to_cpup((u16 *) (out_mad->data + 32));
 	memcpy(&props->sys_image_guid, out_mad->data +  4, 8);
 	memcpy(&props->node_guid,      out_mad->data + 12, 8);
 
Index: hw/mthca/mthca_cmd.h
===================================================================
--- hw/mthca/mthca_cmd.h	(revision 1846)
+++ hw/mthca/mthca_cmd.h	(working copy)
@@ -95,7 +95,21 @@ enum {
 };
 
 enum {
-	DEV_LIM_FLAG_SRQ = 1 << 6
+	DEV_LIM_FLAG_RC                 = 1 << 0,
+	DEV_LIM_FLAG_UC                 = 1 << 1,
+	DEV_LIM_FLAG_UD                 = 1 << 2,
+	DEV_LIM_FLAG_RD                 = 1 << 3,
+	DEV_LIM_FLAG_RAW_IPV6           = 1 << 4,
+	DEV_LIM_FLAG_RAW_ETHER          = 1 << 5,
+	DEV_LIM_FLAG_SRQ                = 1 << 6,
+	DEV_LIM_FLAG_BAD_PKEY_CNTR      = 1 << 8,
+	DEV_LIM_FLAG_BAD_QKEY_CNTR      = 1 << 9,
+	DEV_LIM_FLAG_MW                 = 1 << 16,
+	DEV_LIM_FLAG_AUTO_PATH_MIG      = 1 << 17,
+	DEV_LIM_FLAG_ATOMIC             = 1 << 18,
+	DEV_LIM_FLAG_RAW_MULTI          = 1 << 19,
+	DEV_LIM_FLAG_UD_AV_PORT_ENFORCE = 1 << 20,
+	DEV_LIM_FLAG_UD_MULTI           = 1 << 21,
 };
 
 struct mthca_dev_lim {


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 21 03:31:22 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 21 Feb 2005 13:31:22 +0200
Subject: [openib-general] [PATCH] make device_cap_flags unsigned
Message-ID: <20050221113122.GE14855@mellanox.co.il>

Since device_cap_flags is a set of ORed values, make the result
unsigned. Also to be in line with other flags.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: include/ib_verbs.h
===================================================================
--- include/ib_verbs.h	(revision 1846)
+++ include/ib_verbs.h	(working copy)
@@ -93,7 +93,7 @@ struct ib_device_attr {
 	u32			hw_ver;
 	int			max_qp;
 	int			max_qp_wr;
-	int			device_cap_flags;
+	u32			device_cap_flags;
 	int			max_sge;
 	int			max_sge_rd;
 	int			max_cq;

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 21 05:32:00 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 21 Feb 2005 15:32:00 +0200
Subject: [openib-general] [PATCH] prepend sdp_ to global names
Message-ID: <20050221133200.GI14855@mellanox.co.il>

Some extern (non-static) variables did not start with sdp_ or anything
like that. Here's a fix.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1846)
+++ sdp_inet.c	(working copy)
@@ -52,7 +52,7 @@ static int _send_post_max   = SDP_CQ_SEN
 static int _send_buff_max   = SDP_BUFF_SEND_MAX;
 static int _send_usig_max   = SDP_SEND_UNSIG_MAX;
 
-int debug_level = __SDP_DEBUG_LEVEL;
+int sdp_debug_level = __SDP_DEBUG_LEVEL;
 
 MODULE_AUTHOR("Libor Michalek");
 MODULE_DESCRIPTION("InfiniBand SDP module");
@@ -67,7 +67,7 @@ module_param(_recv_buff_max, int, 0);
 module_param(_send_post_max, int, 0);
 module_param(_send_buff_max, int, 0);
 module_param(_send_usig_max, int, 0);
-module_param(debug_level, int, 0);
+module_param( sdp_debug_level, int, 0);
 
 /*
  * socket structure relevant fields:
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1846)
+++ sdp_proto.h	(working copy)
@@ -476,7 +476,7 @@ typedef int (*sdp_event_cb_func)(struct 
 /*
  * trace macros
  */
-extern int debug_level;
+extern int sdp_debug_level;
 #define __SDP_DEBUG_LEVEL 4
 
 #if defined(CONFIG_INFINIBAND_SDP_DEBUG)
@@ -497,7 +497,7 @@ extern int debug_level;
 
 #define __sdp_dbg_out(level, type, format, arg...) \
         do { \
-                if (!(level > debug_level)) { \
+                if (!(level > sdp_debug_level)) { \
                         printk("<%d>%s: " format "\n", \
                                level, type, ## arg);  \
                 } \
Index: sdp_msgs.h
===================================================================
--- sdp_msgs.h	(revision 1846)
+++ sdp_msgs.h	(working copy)
@@ -416,14 +416,14 @@ static __inline__ void __sdp_msg_swap_sc
 /*
  * Connection messages
  */
-struct msg_hello {
+struct sdp_msg_hello {
 	struct msg_hdr_bsdh bsdh;  /* base sockets direct header */
 	struct msg_hdr_hh   hh;    /* hello message header */
-} __attribute__ ((packed)); /* struct msg_hello */
+} __attribute__ ((packed)); /* struct sdp_msg_hello */
 
-struct msg_hello_ack {
+struct sdp_msg_hello_ack {
 	struct msg_hdr_bsdh bsdh;  /* base sockets direct header */
 	struct msg_hdr_hah  hah;   /* hello ack message header */
-} __attribute__ ((packed)); /* struct msg_hello_ack */
+} __attribute__ ((packed)); /* struct sdp_msg_hello_ack */
 
 #endif /* _SDP_MSGS_H */
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1846)
+++ sdp_actv.c	(working copy)
@@ -201,7 +201,7 @@ drop:
 /*
  * _sdp_cm_hello_ack_check - validate the hello ack header
  */
-static int _sdp_cm_hello_ack_check(struct msg_hello_ack *hello_ack)
+static int _sdp_cm_hello_ack_check(struct sdp_msg_hello_ack *hello_ack)
 {
 	/*
 	 * endian swap
@@ -211,11 +211,11 @@ static int _sdp_cm_hello_ack_check(struc
 	/*
 	 * validation and consistency checks
 	 */
-	if (hello_ack->bsdh.size != sizeof(struct msg_hello_ack)) {
+	if (hello_ack->bsdh.size != sizeof(struct sdp_msg_hello_ack)) {
 
 		sdp_dbg_warn(NULL, "hello ack, size mismatch. (2) <%d:%Zu>",
 			     hello_ack->bsdh.size,
-			     sizeof(struct msg_hello_ack));
+			     sizeof(struct sdp_msg_hello_ack));
 		return -EINVAL;
 	}
 
@@ -263,7 +263,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 		       struct ib_cm_event *event,
 		       struct sdp_opt *conn)
 {
-	struct msg_hello_ack *hello_ack;
+	struct sdp_msg_hello_ack *hello_ack;
 	int result;
 	int error;
 
@@ -277,7 +277,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 			     conn->cm_id->local_id, cm_id->local_id);
 	}
 
-	hello_ack = (struct msg_hello_ack *)event->private_data;
+	hello_ack = (struct sdp_msg_hello_ack *)event->private_data;
 
 	sdp_dbg_ctrl(conn, "CM REP. comm <%08x>", cm_id->local_id);
 	/*
@@ -381,7 +381,7 @@ static void _sdp_cm_path_complete(u64 id
 				  void *arg)
 {
 	struct ib_cm_req_param param;
-	struct msg_hello *hello_msg;
+	struct sdp_msg_hello *hello_msg;
 	struct sdp_opt *conn = (struct sdp_opt *) arg;
 	struct sdpc_buff *buff;
 	int result = 0;
@@ -450,10 +450,10 @@ static void _sdp_cm_path_complete(u64 id
 		goto failed;
 	}
 
-	hello_msg = (struct msg_hello *)buff->data;
-	buff->tail = buff->data + sizeof(struct msg_hello);
+	hello_msg = (struct sdp_msg_hello *)buff->data;
+	buff->tail = buff->data + sizeof(struct sdp_msg_hello);
 
-	memset(hello_msg, 0, sizeof(struct msg_hello));
+	memset(hello_msg, 0, sizeof(struct sdp_msg_hello));
 
 	conn->l_advt_bf = conn->recv_cq_size;
 	conn->l_max_adv = SDP_MSG_MAX_ADVS;
@@ -461,7 +461,7 @@ static void _sdp_cm_path_complete(u64 id
 	hello_msg->bsdh.recv_bufs = conn->l_advt_bf;
 	hello_msg->bsdh.flags     = SDP_MSG_FLAG_NON_FLAG;
 	hello_msg->bsdh.mid       = SDP_MID_HELLO;
-	hello_msg->bsdh.size      = sizeof(struct msg_hello);
+	hello_msg->bsdh.size      = sizeof(struct sdp_msg_hello);
 	hello_msg->bsdh.seq_num   = conn->send_seq;
 	hello_msg->bsdh.seq_ack   = conn->advt_seq;
 
@@ -587,10 +587,10 @@ int sdp_cm_connect(struct sdp_opt *conn)
 	/*
 	 * get the buffer size we'll use for this connection. (and all others)
 	 */
-	if (sizeof(struct msg_hello) > conn->recv_size) {
+	if (sizeof(struct sdp_msg_hello) > conn->recv_size) {
 
 		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
-			     conn->recv_size, sizeof(struct msg_hello));
+			     conn->recv_size, sizeof(struct sdp_msg_hello));
 		result = -ENOBUFS;
 		goto error;
 	}
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1846)
+++ sdp_pass.c	(working copy)
@@ -40,7 +40,7 @@
 static int _sdp_cm_accept(struct sdp_opt *conn)
 {
 	struct ib_cm_rep_param param;
-	struct msg_hello_ack *hello_ack;
+	struct sdp_msg_hello_ack *hello_ack;
 	struct sdpc_buff *buff;
 	struct ib_qp_attr *qp_attr;
 	int qp_mask = 0;
@@ -51,10 +51,10 @@ static int _sdp_cm_accept(struct sdp_opt
 	/*
 	 * build listen response headers
 	 */
-	if (sizeof(struct msg_hello_ack) > conn->recv_size) {
+	if (sizeof(struct sdp_msg_hello_ack) > conn->recv_size) {
 
 		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
-			     conn->recv_size, sizeof(struct msg_hello_ack));
+			     conn->recv_size, sizeof(struct sdp_msg_hello_ack));
 		result = -ENOBUFS;
 		goto error;
 	}
@@ -70,12 +70,12 @@ static int _sdp_cm_accept(struct sdp_opt
 		goto error;
 	}
 
-	hello_ack = (struct msg_hello_ack *)buff->data;
-	buff->tail = buff->data + sizeof(struct msg_hello_ack);
+	hello_ack = (struct sdp_msg_hello_ack *)buff->data;
+	buff->tail = buff->data + sizeof(struct sdp_msg_hello_ack);
 	/*
 	 * create the message
 	 */
-	memset(hello_ack, 0, sizeof(struct msg_hello_ack));
+	memset(hello_ack, 0, sizeof(struct sdp_msg_hello_ack));
 
 	conn->l_advt_bf = conn->recv_cq_size;
 	conn->l_max_adv = SDP_MSG_MAX_ADVS;
@@ -83,7 +83,7 @@ static int _sdp_cm_accept(struct sdp_opt
 	hello_ack->bsdh.recv_bufs = conn->l_advt_bf;
 	hello_ack->bsdh.flags     = SDP_MSG_FLAG_NON_FLAG;
 	hello_ack->bsdh.mid       = SDP_MID_HELLO_ACK;
-	hello_ack->bsdh.size      = sizeof(struct msg_hello_ack);
+	hello_ack->bsdh.size      = sizeof(struct sdp_msg_hello_ack);
 	hello_ack->bsdh.seq_num   = conn->send_seq;
 	hello_ack->bsdh.seq_ack   = conn->advt_seq;
 
@@ -313,7 +313,7 @@ lookup_err:
 /*
  * _sdp_cm_hello_check - validate the hello header
  */
-static int _sdp_cm_hello_check(struct msg_hello *msg_hello)
+static int _sdp_cm_hello_check(struct sdp_msg_hello *msg_hello)
 {
 	/*
 	 * endian swap
@@ -323,10 +323,10 @@ static int _sdp_cm_hello_check(struct ms
 	/*
 	 * validation and consistency checks
 	 */
-	if (msg_hello->bsdh.size != sizeof(struct msg_hello)) {
+	if (msg_hello->bsdh.size != sizeof(struct sdp_msg_hello)) {
 
 		sdp_dbg_warn(NULL, "hello msg size mismatch. (2) <%d:%Zu>",
-			     msg_hello->bsdh.size, sizeof(struct msg_hello));
+			     msg_hello->bsdh.size, sizeof(struct sdp_msg_hello));
 		return -EINVAL;
 	}
 
@@ -388,7 +388,7 @@ static int _sdp_cm_hello_check(struct ms
  */
 int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 {
-	struct msg_hello *msg_hello = (struct msg_hello *)event->private_data;
+	struct sdp_msg_hello *msg_hello = (struct sdp_msg_hello *)event->private_data;
 	struct sdp_opt *conn;
 	int result;
 

-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Mon Feb 21 05:53:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 21 Feb 2005 08:53:10 -0500
Subject: [openib-general] [PATCH] OpenSM: Add well known multicast groups
	with MTU and Rate component mask bits set
Message-ID: <1108993791.4646.125.camel@localhost.localdomain>

OpenSM: Add well known multicast groups with MTU and Rate component mask
bits set

Index: opensm/osm_sa.c
===================================================================
--- opensm/osm_sa.c	(revision 1846)
+++ opensm/osm_sa.c	(working copy)
@@ -560,10 +560,12 @@
 
   osm_mgrp_t * p_mgrp = NULL;
   ib_api_status_t status;
+  ib_net64_t comp_mask;
 
+  comp_mask = IB_MCR_COMPMASK_MTU | IB_MCR_COMPMASK_RATE;
   status = osm_mcmr_rcv_create_new_mgrp(
     p_mcmr,
-    0, /*  we want OpenSM to pick the Scope. No comp mask provided. */
+    comp_mask,
     p_well_know_mc_rec,
     &p_mgrp);
   if(p_mgrp)





From halr at voltaire.com  Mon Feb 21 05:56:52 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 21 Feb 2005 08:56:52 -0500
Subject: [openib-general] [PATCH] OpenSM: In osm_mcmr_query_mgrp,
	set exactly selectors in MCMemberRecord response
Message-ID: <1108994169.4646.130.camel@localhost.localdomain>

OpenSM: In osm_mcmr_query_mgrp, set exactly selectors in MCMemberRecord
response

Index: osm_sa_mcmember_record.c
===================================================================
--- osm_sa_mcmember_record.c	(revision 1844)
+++ osm_sa_mcmember_record.c	(working copy)
@@ -1696,9 +1696,6 @@
   p_resp_sa_mad->attr_offset =
     ib_get_attr_offset( sizeof(ib_member_rec_t) );
 
-  p_resp_rec = (ib_member_rec_t*)
-    ib_sa_mad_get_payload_ptr( p_resp_sa_mad );
-
 #ifndef VENDOR_RMPP_SUPPORT
   /* we support only one packet RMPP - so we will set the first and
      last flags for gettable */
@@ -1713,7 +1710,9 @@
     p_resp_sa_mad->rmpp_flags = IB_RMPP_FLAG_ACTIVE;
 #endif
 
-    /*
+  p_resp_rec = (ib_member_rec_t*)ib_sa_mad_get_payload_ptr( p_resp_sa_mad );
+
+  /*
     p819 - The PortGID, JoinState and ProxyJoin shall be zero, 
     except in the case of a trusted request. 
     Note: In the mad controller we check that the SM_Key received on
@@ -1730,6 +1729,10 @@
     if (i < num_rec)
     {
       *p_resp_rec = p_rec_item->rec;
+      /* Fill in the mtu, rate, and packet lifetime selectors */
+      p_resp_rec->mtu |= 2<<6; /* exactly */
+      p_resp_rec->rate |= 2<<6; /* exactly */
+      p_resp_rec->pkt_life |= 2<<6; /* exactly */
       if (trusted_req == FALSE)
       {
         cl_memclr(&p_resp_rec->port_gid, sizeof(ib_gid_t));





From halr at voltaire.com  Mon Feb 21 06:15:40 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 21 Feb 2005 09:15:40 -0500
Subject: [openib-general] [PATCH] OpenSM: Add multicast destination handling
	into SA PathRecord support
Message-ID: <1108995339.4646.135.camel@localhost.localdomain>

OpenSM: Add multicast destination handling into SA PathRecord support

Index: osm_sa_path_record.c
===================================================================
--- osm_sa_path_record.c	(revision 1842)
+++ osm_sa_path_record.c	(working copy)
@@ -69,6 +69,7 @@
 #include <vendor/osm_vendor_api.h>
 #include <opensm/osm_helper.h>
 #include <opensm/osm_pkey.h>
+#include <opensm/osm_multicast.h>
 
 #define OSM_PR_RCV_POOL_MIN_SIZE    64
 #define OSM_PR_RCV_POOL_GROW_SIZE      64
@@ -90,6 +91,12 @@
   boolean_t          reversible;
 } osm_path_parms_t;
 
+typedef  struct   osm_sa_pr_mcmr_search_ctxt {
+  ib_gid_t        *p_mgid;
+  osm_mgrp_t      *p_mgrp;
+  osm_pr_rcv_t    *p_rcv;
+} osm_sa_pr_mcmr_search_ctxt_t;
+
 /**********************************************************************
  **********************************************************************/
 void
@@ -1153,8 +1160,277 @@
 }
 
 /**********************************************************************
+ *********************************************************************/
+static
+void
+__search_mgrp_by_mgid(
+  IN  cl_map_item_t* const    p_map_item,
+  IN  void*                   context )
+{
+  osm_mgrp_t* p_mgrp = (osm_mgrp_t*)p_map_item;
+  osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = (osm_sa_pr_mcmr_search_ctxt_t *) context;
+  const ib_gid_t        *p_recvd_mgid;
+  osm_pr_rcv_t *p_rcv;
+  /*  uint32_t  i; */
+
+  p_recvd_mgid = p_ctxt->p_mgid;
+  p_rcv = p_ctxt->p_rcv;
+
+  /* Why not compare the entire MGID ???? */
+  /* different scope can sneak in for the same MGID ? */
+  /* EZ: I changed it to full compare ! */
+  if (cl_memcmp(&p_mgrp->mcmember_rec.mgid,
+                p_recvd_mgid,
+                sizeof(ib_gid_t)))
+    return;
+
+#if 0
+  for ( i = 0 ; i < sizeof(p_mgrp->mcmember_rec.mgid.multicast.raw_group_id); i++)
+  {
+    if (p_mgrp->mcmember_rec.mgid.multicast.raw_group_id[i] !=
+        p_recvd_mgid->mgid.multicast.raw_group_id[i])
+      return;
+  }
+#endif
+
+  if(p_ctxt->p_mgrp)
+  {
+    osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+             "__search_mgrp_by_mgid: ERR 1B03: "
+             "Multiple MCGROUP for same MGID.\n" );
+    return;
+  }
+  p_ctxt->p_mgrp = p_mgrp;
+
+}
+
+/**********************************************************************
  **********************************************************************/
+static ib_api_status_t
+__get_mgrp_by_mgid(
+  IN osm_pr_rcv_t* const p_rcv,
+  IN ib_path_rec_t* p_recvd_path_rec,
+  OUT osm_mgrp_t **pp_mgrp)
+{
+  osm_sa_pr_mcmr_search_ctxt_t mcmr_search_context;
+
+  mcmr_search_context.p_mgid = &p_recvd_path_rec->dgid;
+  mcmr_search_context.p_rcv = p_rcv;
+  mcmr_search_context.p_mgrp = NULL;
+
+  cl_qmap_apply_func( &p_rcv->p_subn->mgrp_mlid_tbl,
+                      __search_mgrp_by_mgid,
+                      &mcmr_search_context);
+
+  if(mcmr_search_context.p_mgrp == NULL)
+  {
+    return IB_NOT_FOUND;
+  }
+
+  *pp_mgrp = mcmr_search_context.p_mgrp;
+  return IB_SUCCESS;
+}
+
+/**********************************************************************
+ **********************************************************************/
+static
+osm_mgrp_t *
+__get_mgrp_by_mlid(
+  IN const osm_pr_rcv_t* const p_rcv,
+  IN ib_net16_t const mlid)
+{
+  cl_map_item_t *map_item;
+
+  map_item = cl_qmap_get(&p_rcv->p_subn->mgrp_mlid_tbl,
+                         mlid);
+  if(map_item == cl_qmap_end(&p_rcv->p_subn->mgrp_mlid_tbl))
+  {
+    return NULL;
+  }
+
+  return (osm_mgrp_t *)map_item;
+}
+
+/**********************************************************************
+ **********************************************************************/
 static void
+__osm_pr_get_mgrp(
+  IN osm_pr_rcv_t*         const p_rcv,
+  IN const osm_madw_t*     const p_madw,
+  OUT osm_mgrp_t           **pp_mgrp )
+{
+  ib_path_rec_t*           p_pr;
+  const ib_sa_mad_t*       p_sa_mad;
+  ib_net64_t               comp_mask;
+  ib_api_status_t          status;
+
+  OSM_LOG_ENTER( p_rcv->p_log, __osm_pr_get_mgrp );
+
+  p_sa_mad = osm_madw_get_sa_mad_ptr( p_madw );
+  p_pr = (ib_path_rec_t*)ib_sa_mad_get_payload_ptr( p_sa_mad );
+
+  comp_mask = p_sa_mad->comp_mask;
+
+  if( comp_mask & IB_PR_COMPMASK_DGID )
+  {
+    status = __get_mgrp_by_mgid( p_rcv, p_pr, pp_mgrp );
+    if( status != IB_SUCCESS )
+    {
+      osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+               "__osm_pr_get_mgrp: "
+               "No MC group found for PathRecord destination GID.\n" );
+      goto Exit;
+    }
+  }
+
+  if( comp_mask & IB_PR_COMPMASK_DLID )
+  {
+    if( *pp_mgrp)
+    {
+      /* check that the MLID in the MC group is */
+      /* the same as the DLID in the PathRecord */
+      if( (*pp_mgrp)->mlid != p_pr->dlid )
+      {
+	/* Note: perhaps this might be better indicated as an invalid request */
+        osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+                 "__osm_pr_get_mgrp: "
+                 "MC group MLID does not match PathRecord destination LID.\n" );
+        *pp_mgrp = NULL;
+        goto Exit;
+      }
+    }
+    else
+    {
+      *pp_mgrp = __get_mgrp_by_mlid( p_rcv, p_pr->dlid );
+      if( *pp_mgrp == NULL)
+      {
+        osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+                 "__osm_pr_get_mgrp: "
+                 "No MC group found for PathRecord destination LID.\n" );
+      }
+    }
+  }
+
+ Exit:
+  OSM_LOG_EXIT( p_rcv->p_log );
+}
+
+/**********************************************************************
+ **********************************************************************/
+static ib_api_status_t
+__osm_pr_match_mgrp_attributes(
+  IN osm_pr_rcv_t*         const p_rcv,
+  IN const osm_madw_t*     const p_madw,
+  IN const osm_mgrp_t*     const p_mgrp )
+{
+  const ib_path_rec_t*     p_pr;
+  const ib_sa_mad_t*       p_sa_mad;
+  ib_net64_t               comp_mask;
+  ib_api_status_t          status = IB_ERROR;
+  uint32_t                 flow_label;
+  uint8_t                  sl;
+  uint8_t                  hop_limit;
+
+  OSM_LOG_ENTER( p_rcv->p_log, __osm_pr_rcv_check_mcast_dest );
+
+  p_sa_mad = osm_madw_get_sa_mad_ptr( p_madw );
+  p_pr = (ib_path_rec_t*)ib_sa_mad_get_payload_ptr( p_sa_mad );
+
+  comp_mask = p_sa_mad->comp_mask;
+
+  /* If SGID and/or SLID specified, should validate as part of MC group */
+  /* Also, not checking MTU, rate, packet lifetime, and raw traffic currently */
+  if( comp_mask & IB_PR_COMPMASK_PKEY )
+  {
+    if( p_pr->pkey != p_mgrp->mcmember_rec.pkey )
+      goto Exit;
+  }
+
+  ib_member_get_sl_flow_hop( p_mgrp->mcmember_rec.sl_flow_hop,
+                             &sl, &flow_label, &hop_limit );
+
+  if( comp_mask & IB_PR_COMPMASK_SL )
+  {
+    if( ( p_pr->sl & 0xf ) != sl )
+      goto Exit;
+  }
+
+  if( comp_mask & IB_PR_COMPMASK_NUMBPATH )
+  {
+    if( ( p_pr->num_path & 0x7f ) == 0 )
+      goto Exit;
+  }
+
+  if( comp_mask & IB_PR_COMPMASK_FLOWLABEL )
+  {
+    if( ib_path_rec_flow_lbl( p_pr ) != flow_label )
+      goto Exit;
+  }
+
+  if( comp_mask & IB_PR_COMPMASK_HOPLIMIT )
+  {
+    if( ib_path_rec_hop_limit( p_pr ) != hop_limit )
+      goto Exit;
+  }
+
+  if( comp_mask & IB_PR_COMPMASK_TCLASS )
+  {
+    if( p_pr->tclass != p_mgrp->mcmember_rec.tclass )
+      goto Exit;
+  }
+
+  status = IB_SUCCESS;
+
+ Exit:
+  OSM_LOG_EXIT( p_rcv->p_log );
+  return( status );
+}
+
+/**********************************************************************
+ **********************************************************************/
+static boolean_t 
+__osm_pr_rcv_check_mcast_dest(
+  IN osm_pr_rcv_t*         const p_rcv,
+  IN const osm_madw_t*     const p_madw )
+{
+  const ib_path_rec_t*     p_pr;
+  const ib_sa_mad_t*       p_sa_mad;
+  ib_net64_t               comp_mask;
+  boolean_t                is_multicast = FALSE;
+
+  OSM_LOG_ENTER( p_rcv->p_log, __osm_pr_rcv_check_mcast_dest );
+
+  p_sa_mad = osm_madw_get_sa_mad_ptr( p_madw );
+  p_pr = (ib_path_rec_t*)ib_sa_mad_get_payload_ptr( p_sa_mad );
+
+  comp_mask = p_sa_mad->comp_mask;
+
+  if( comp_mask & IB_PR_COMPMASK_DGID )
+  {
+    is_multicast = ib_gid_is_multicast( &p_pr->dgid );
+    if( is_multicast )
+      goto Exit;
+  }
+
+  if( comp_mask & IB_PR_COMPMASK_DLID )
+  {
+    if( cl_ntoh16( p_pr->dlid ) >= IB_LID_MCAST_START &&
+        cl_ntoh16( p_pr->dlid ) <= IB_LID_MCAST_END )
+      is_multicast = TRUE;
+    else if( is_multicast )
+      osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+               "__osm_pr_rcv_check_mcast_dest: "
+               "PathRecord request indicates MGID but not MLID.\n" );
+  }
+
+ Exit:
+  OSM_LOG_EXIT( p_rcv->p_log );
+  return( is_multicast );
+}
+
+/**********************************************************************
+ **********************************************************************/
+static void
 __osm_pr_rcv_respond(
   IN osm_pr_rcv_t*         const p_rcv,
   IN const osm_madw_t*     const p_madw,
@@ -1214,7 +1490,7 @@
   if( osm_log_is_active( p_rcv->p_log, OSM_LOG_DEBUG ) )
   {
     osm_log( p_rcv->p_log, OSM_LOG_DEBUG,
-             "__osm_pr_rcv_respond:"
+             "__osm_pr_rcv_respond: "
              "Generating response with %u records.\n", num_rec );
   }
 
@@ -1329,7 +1605,7 @@
   {
     osm_log( p_rcv->p_log, OSM_LOG_ERROR,
              "osm_pr_rcv_process: "
-             "Cannot find requestor physical port. \n" );
+             "Cannot find requestor physical port.\n" );
     goto Exit;
   }
 
@@ -1359,6 +1635,14 @@
   */
   cl_plock_acquire( p_rcv->p_lock );
 
+  /* Handle multicast destinations separately */
+  if( __osm_pr_rcv_check_mcast_dest( p_rcv, p_madw ) )
+    goto McastDest;
+
+  osm_log( p_rcv->p_log, OSM_LOG_DEBUG,
+           "osm_pr_rcv_process: "
+           "Unicast destination requested.\n" );
+
   sa_status = __osm_pr_rcv_get_end_points( p_rcv, p_madw,
                                            &p_src_port, &p_dest_port );
 
@@ -1394,7 +1678,78 @@
       __osm_pr_rcv_process_world( p_rcv, p_pr, requestor_port,
                                   p_sa_mad->comp_mask, &pr_list );
   }
+  goto Unlock;
 
+ McastDest:
+  osm_log(p_rcv->p_log, OSM_LOG_DEBUG,
+             "osm_pr_rcv_process: "
+             "Multicast destination requested.\n" );
+
+  osm_mgrp_t *p_mgrp = NULL;
+  ib_api_status_t status;
+  osm_pr_item_t* p_pr_item;
+  uint32_t flow_label;
+  uint8_t  sl;
+  uint8_t  hop_limit;
+
+  /* First, get the MC info */
+  __osm_pr_get_mgrp( p_rcv, p_madw, &p_mgrp );
+
+  if ( p_mgrp )
+  {
+    /* Make sure the rest of the PathRecord matches the MC group attributes */
+    status = __osm_pr_match_mgrp_attributes( p_rcv, p_madw, p_mgrp);
+    if ( status == IB_SUCCESS )
+    {
+      p_pr_item = (osm_pr_item_t*)cl_qlock_pool_get( &p_rcv->pr_pool );
+      if( p_pr_item == NULL )
+      {
+        osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+                 "osm_pr_rcv_process: "
+                 "Unable to allocate path record for MC group.\n" );
+      }
+      else
+      {
+	/* Copy PathRecord request into response */
+        p_sa_mad = osm_madw_get_sa_mad_ptr( p_madw );
+        p_pr = (ib_path_rec_t*)ib_sa_mad_get_payload_ptr( p_sa_mad );
+        p_pr_item->path_rec = *p_pr;
+
+        /* Now, use the MC info to cruft up the PathRecord response */        
+        p_pr_item->path_rec.dgid = p_mgrp->mcmember_rec.mgid;
+        p_pr_item->path_rec.dlid = p_mgrp->mcmember_rec.mlid;
+	p_pr_item->path_rec.tclass = p_mgrp->mcmember_rec.tclass;
+	p_pr_item->path_rec.num_path = 1;
+	p_pr_item->path_rec.pkey = p_mgrp->mcmember_rec.pkey;
+
+	/* MTU, rate, and packet lifetime should be exactly */
+	p_pr_item->path_rec.mtu = (2<<6) | p_mgrp->mcmember_rec.mtu;
+        p_pr_item->path_rec.rate = (2<<6) | p_mgrp->mcmember_rec.rate;
+        p_pr_item->path_rec.pkt_life = (2<<6) | p_mgrp->mcmember_rec.pkt_life;
+
+	/* SL, Hop Limit, and Flow Label */
+        ib_member_get_sl_flow_hop( p_mgrp->mcmember_rec.sl_flow_hop, 
+                                   &sl, &flow_label, &hop_limit );
+	p_pr_item->path_rec.sl = sl;
+        p_pr_item->path_rec.hop_flow_raw = (uint32_t)(hop_limit) |
+                                           (flow_label << 8);
+
+        cl_qlist_insert_tail( &pr_list,
+                              (cl_list_item_t*)&p_pr_item->pool_item );
+
+      }
+    }
+    else
+    {
+      osm_log( p_rcv->p_log, OSM_LOG_ERROR,
+               "osm_pr_rcv_process: "
+               "MC group attributes don't match PathRecord request.\n" ); 
+    }
+  }
+
+  /* Now, (finally) respond to the PathRecord request */
+
+ Unlock:
   cl_plock_release( p_rcv->p_lock );
   __osm_pr_rcv_respond( p_rcv, p_madw, &pr_list );
 





From mst at mellanox.co.il  Mon Feb 21 06:23:27 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 21 Feb 2005 16:23:27 +0200
Subject: [openib-general] Re: [PATCH] OpenSM: Add multicast destination
	handling into SA PathRecord support
In-Reply-To: <1108995339.4646.135.camel@localhost.localdomain>
References: <1108995339.4646.135.camel@localhost.localdomain>
Message-ID: <20050221142327.GL14855@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> @@ -1153,8 +1160,277 @@
> [...]
>
> +#if 0
> +  for ( i = 0 ; i < sizeof(p_mgrp->mcmember_rec.mgid.multicast.raw_group_id); i++)
> +  {
> +    if (p_mgrp->mcmember_rec.mgid.multicast.raw_group_id[i] !=
> +        p_recvd_mgid->mgid.multicast.raw_group_id[i])
> +      return;
> +  }
> +#endif
> +

I dont claim I know the opensm code well, but is it
a good idea to add more dead code?


-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Mon Feb 21 06:29:26 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 21 Feb 2005 09:29:26 -0500
Subject: [openib-general] Re: [PATCH] OpenSM: Add multicast destination
	handling into SA PathRecord support
In-Reply-To: <20050221142327.GL14855@mellanox.co.il>
References: <1108995339.4646.135.camel@localhost.localdomain>
	<20050221142327.GL14855@mellanox.co.il>
Message-ID: <1108996166.4646.145.camel@localhost.localdomain>

On Mon, 2005-02-21 at 09:23, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > @@ -1153,8 +1160,277 @@
> > [...]
> >
> > +#if 0
> > +  for ( i = 0 ; i < sizeof(p_mgrp->mcmember_rec.mgid.multicast.raw_group_id); i++)
> > +  {
> > +    if (p_mgrp->mcmember_rec.mgid.multicast.raw_group_id[i] !=
> > +        p_recvd_mgid->mgid.multicast.raw_group_id[i])
> > +      return;
> > +  }
> > +#endif
> > +
> 
> I dont claim I know the opensm code well, but is it
> a good idea to add more dead code?

This is just a cut and paste of the same code Eitan had added elsewhere.
Not sure about the origin. I didn't resolve the question:

  /* Why not compare the entire MGID ???? */ 
  /* different scope can sneak in for the same MGID ? */
  /* EZ: I changed it to full compare ! */

I'm not sure how a different scope would "sneak" in. I've been meaning
to ask. If it can't, I'm happy to eliminate the code here and elsewhere
where it appears.

-- Hal



From mst at mellanox.co.il  Mon Feb 21 10:12:57 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 21 Feb 2005 20:12:57 +0200
Subject: [openib-general] Re: RFC on SDP checkin
In-Reply-To: <20050217164150.F7121@topspin.com>
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>
	<20050217140359.GT28446@mellanox.co.il>
	<20050217164150.F7121@topspin.com>
Message-ID: <20050221181257.GA16314@mellanox.co.il>


Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: RFC on SDP checkin
> 
> On Thu, Feb 17, 2005 at 04:03:59PM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > Possible Issues
> > >   
> > > [...] 
> > > 
> > >   - Memory locking for AIO requires a call to do_mlock() which is not a
> > >     kernel exported function, the method for calling the function is not
> > >     standard.
> > 
> > Libor, in my eyes this is the biggest issue with this gen2 sdp code.
> > In short, I dont think using do_mlock is a good idea.
> 
>   I do agree that this is one of the more, if not most, problematic area
> of the code, and I like your alternative suggestion below.

BTW, just noted that sdp_iocb.c gets current->mm directly, and keeps
this pointer cashed instead of calling get_task_mm each time.
Thus it seems a race condition may occur if the task
is exiting while the cashed value is used.

I am not fixing it since we agreed we are changing this to get_user_pages
anyway.


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 01:33:04 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 11:33:04 +0200
Subject: [openib-general] Re: SDP_CONN_LOCK
In-Reply-To: <20050217110256.A7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
Message-ID: <20050222093304.GE23837@mellanox.co.il>




Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: SDP_CONN_LOCK
> 
> On Thu, Feb 17, 2005 at 05:35:17PM +0200, Michael S. Tsirkin wrote:
> > Hi, Libor!
> > Could you please explain what are the SDP_CONN_LOCK
> > and friends doing in SDP?
> > 
> > It seems they just implement exclusive access to socket,
> > but if so, why is a simple semaphore or mutex not used?
> 
>   They do implement exclusive access to the socket, but they implement
> exclusive access from both process and irq context, which is why a
> semaphore was not used. In interrupt context SDP_CONN_LOCK_BH is used
> to lock the connection, look in sdp_cq_event_handler() for it's use,
> and in process context SDP_CONN_LOCK is used.

By the way, I think you are supposed to change the task state
with set_current_state (or _set_current_state if you are absolutely
shure).


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 01:42:21 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 11:42:21 +0200
Subject: [openib-general] [PATCH] PUT after UNLOCK
Message-ID: <20050222094220.GF23837@mellanox.co.il>

SDP_CONN_PUT was called before SDP_CONN_UNLOCK.
If this is the last reference, the connection could be removed
and an attempt to unlock would crash.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1857)
+++ sdp_inet.c	(working copy)
@@ -716,9 +716,9 @@ static int _sdp_inet_connect(struct sock
 			sock->state = SS_UNCONNECTED;
 			conn->istate = SDP_SOCK_ST_CLOSED;
 
+			SDP_CONN_UNLOCK(conn);
 			SDP_CONN_PUT(conn);	/* CM reference */
-
-			goto done;
+			return result;
 		}
 
 		inet_sk(sk)->daddr = htonl(conn->dst_addr);


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 02:14:45 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 12:14:45 +0200
Subject: [openib-general] SDP: BUG 2034 workaround
Message-ID: <20050222101445.GK23837@mellanox.co.il>


sdp_inet.c, inside _sdp_inet_listen, we have:

#if 0                           /* BUG 2034 workaround. */
        conn->backlog_max = backlog;
#else
        conn->backlog_max = 1024;
#endif

What gives? what would be the proper fix as opposed to a work-around?


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 02:34:51 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 12:34:51 +0200
Subject: [openib-general] [PATCH] small cleanup
Message-ID: <20050222103451.GM23837@mellanox.co.il>

Replace for(;;) break with a proper C loop.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1857)
+++ sdp_conn.c	(working copy)
@@ -837,19 +837,14 @@ void sdp_conn_internal_lock(struct sdp_o
 	unsigned long f = *flags;
 
 	add_wait_queue_exclusive(&(conn->lock.waitq), &wait);
-	for (;;) {
+	do {
 		current->state = TASK_UNINTERRUPTIBLE;
 		spin_unlock_irqrestore(&(conn->lock.slock), f);
 		schedule();
 		spin_lock_irqsave(&(conn->lock.slock), f);
-		*flags = f;
-
-		if (0 == conn->lock.users) {
-
-			break;
-		}
-	}
+	} while (conn->lock.users);
 
+	*flags = f;
 	current->state = TASK_RUNNING;
 	remove_wait_queue(&(conn->lock.waitq), &wait);
 

-- 
MST - Michael S. Tsirkin


From autoresponse at neopets.com  Tue Feb 22 03:06:26 2005
From: autoresponse at neopets.com (autoresponse at neopets.com)
Date: 22 Feb 2005 03:06:26 -0800
Subject: [openib-general] Re: Sex pictures
Message-ID: <13873.26@mail1.dohring.com>

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050222/f753727e/attachment.ksh>

From roland at topspin.com  Tue Feb 22 08:47:09 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 08:47:09 -0800
Subject: [openib-general] [PATCH][IPOIB] bring interface down panic
In-Reply-To: <OF61DB1A86.7CB133C1-ON87256FAA.00820897-88256FAA.008318D5@us.ibm.com>
	(Shirley Ma's message of "Wed, 16 Feb 2005 16:51:58 -0700")
References: <OF61DB1A86.7CB133C1-ON87256FAA.00820897-88256FAA.008318D5@us.ibm.com>
Message-ID: <52ll9gsfsi.fsf@topspin.com>

Shirley, can you read Documentation/SubmittingPatches, and if the
declaration in that file applies, resend your patches with a
"Signed-off-by:" line?

Thanks,
  Roland


From roland at topspin.com  Tue Feb 22 08:50:24 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 08:50:24 -0800
Subject: [openib-general] [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <20050203175419.GJ2536@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 3 Feb 2005 19:54:19 +0200")
References: <20050203175419.GJ2536@mellanox.co.il>
Message-ID: <52ekf8sfn3.fsf@topspin.com>

I finally got around to reviewing this, and I think I see a problem.
It seems that if we get, say, a send completion followed by a receive
completion, we will not update which wq is locked so we'll update the
receive queue with the send queue locked.  Is this true?

 - R.


From roland at topspin.com  Tue Feb 22 08:52:25 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 08:52:25 -0800
Subject: [openib-general] [PATCH] make device_cap_flags unsigned
In-Reply-To: <20050221113122.GE14855@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 21 Feb 2005 13:31:22 +0200")
References: <20050221113122.GE14855@mellanox.co.il>
Message-ID: <52acpwsfjq.fsf@topspin.com>

    Michael> Since device_cap_flags is a set of ORed values, make the
    Michael> result unsigned. Also to be in line with other flags.

This doesn't seem to make any practical difference, since the largest
device cap flag is 1<<15, and even if it were 1<<31 we never do any
non-bitwise arithmetic so signedness doesn't matter.  If we're going
to change the type at all, I would suggest changing it to "enum
ib_device_cap_flags," which at least is somewhat self-documenting.

 - R.


From roland at topspin.com  Tue Feb 22 10:11:41 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 10:11:41 -0800
Subject: Remove unsignaled receive support (was: [openib-general] Re: FW:
	summary of my understanding on our common work on openib.org)
In-Reply-To: <20050210101519.GG18350@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 10 Feb 2005 12:15:19 +0200")
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
Message-ID: <52oeecqxb6.fsf_-_@topspin.com>

Thanks, I've finally applied this.

 - R.


From roland at topspin.com  Tue Feb 22 10:30:40 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 10:30:40 -0800
Subject: [openib-general] Re: Remove unsignaled receive support
In-Reply-To: <52oeecqxb6.fsf_-_@topspin.com> (Roland Dreier's message of
	"Tue, 22 Feb 2005 10:11:41 -0800")
References: <506C3D7B14CDD411A52C00025558DED606FCEA44@mtlex01.yok.mtl.com>
	<20050210101519.GG18350@mellanox.co.il>
	<52oeecqxb6.fsf_-_@topspin.com>
Message-ID: <52hdk4qwfj.fsf@topspin.com>

I've also committed this trivial fix to make SDP build again.

 - R.

Index: infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- infiniband/ulp/sdp/sdp_conn.c	(revision 1857)
+++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -1174,7 +1174,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 		init_attr->send_cq        = conn->send_cq;
 		init_attr->recv_cq        = conn->recv_cq;
 		init_attr->sq_sig_type    = IB_SIGNAL_REQ_WR;
-		init_attr->rq_sig_type    = IB_SIGNAL_ALL_WR;
 		init_attr->qp_type        = IB_QPT_RC;
 		/* TODO: real handler */
 		init_attr->event_handler  = NULL;
Index: infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- infiniband/ulp/sdp/sdp_recv.c	(revision 1857)
+++ infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -97,7 +97,6 @@ static int _sdp_post_recv_buff(struct sd
 	receive_param.wr_id      = buff->wrid;
 	receive_param.sg_list    = (struct ib_sge *)&buff->real;
 	receive_param.num_sge    = 1;
-	receive_param.recv_flags = IB_RECV_SIGNALED;
 
 	result = ib_post_recv(conn->qp, &receive_param, &bad_wr);
 	if (0 != result) {


From xma at us.ibm.com  Tue Feb 22 10:31:01 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Tue, 22 Feb 2005 11:31:01 -0700
Subject: [openib-general] [PATCH][IPOIB] bring interface down panic
In-Reply-To: <52ll9gsfsi.fsf@topspin.com>
Message-ID: <OF5F398356.1168B6EB-ON87256FB0.006555E0-88256FB0.0065B3FC@us.ibm.com>

list_for_each_entry() should be list_for_each_entry_safe() in 
ipoib_mcast_dev_flush(). Here is the patch. Please review it.

Signed-off-by: Shirley Ma <xma at us.ibm.com>

diff -urpN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-down/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-04 
16:05:13.000000000 -0800
+++ infiniband-down/ulp/ipoib/ipoib_multicast.c 2005-02-16 
15:49:42.000000000 -0800
@@ -790,7 +790,7 @@ void ipoib_mcast_dev_flush(struct net_de
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
-       list_for_each_entry(mcast, &remove_list, list) {
+       list_for_each_entry_safe(mcast, tmcast, &remove_list, list) {
                ipoib_mcast_leave(dev, mcast);
                ipoib_mcast_free(mcast);
        }



Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050222/4aa8e352/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-down.patch
Type: application/octet-stream
Size: 549 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050222/4aa8e352/attachment.obj>

From roland at topspin.com  Tue Feb 22 10:36:39 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 10:36:39 -0800
Subject: [openib-general] [PATCH][IPOIB] bring interface down panic
In-Reply-To: <OF5F398356.1168B6EB-ON87256FB0.006555E0-88256FB0.0065B3FC@us.ibm.com>
	(Shirley Ma's message of "Tue, 22 Feb 2005 11:31:01 -0700")
References: <OF5F398356.1168B6EB-ON87256FB0.006555E0-88256FB0.0065B3FC@us.ibm.com>
Message-ID: <52d5usqw5k.fsf@topspin.com>

Thanks, good catch... applied.

 - R.

I really need to start testing with CONFIG_SLAB_DEBUG again...



From mst at mellanox.co.il  Tue Feb 22 10:40:55 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 20:40:55 +0200
Subject: [openib-general] [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <52ekf8sfn3.fsf@topspin.com>
References: <20050203175419.GJ2536@mellanox.co.il> <52ekf8sfn3.fsf@topspin.com>
Message-ID: <20050222184055.GA25382@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] 1/2 separate locking for send and receive q in mthca
> 
> I finally got around to reviewing this, and I think I see a problem.
> It seems that if we get, say, a send completion followed by a receive
> completion, we will not update which wq is locked so we'll update the
> receive queue with the send queue locked.  Is this true?
> 
>  - R.
> 


Ugh. True. No idea what was I thinking, or why is it working so well.
To fix this, I'd have to store the is_send flag between runs.

Instead of doing this, how about something else I would prefer: we could
avoid locking the QP on CQ poll altogether, if there is a separate
last polled index that is written only by cq poll and read by
qp post.
This index update would be protected by a cq lock.

We thus have a circular buffer along the lines described in LDD2
(http://www.xml.com/ldd/chapter/book/ch09.html#t8).

We could still get "false positives" where post sees the qp is
full since polled index (consumer) is updated by another cpu.
To solve this, if we see a full qp, we lock the cq and re-test.
This is a rare case so it should not hurt.

If all this makes sence to you I'll send a patch.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 10:50:07 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 20:50:07 +0200
Subject: [openib-general] Re: [PATCH] initialize device_cap_flags in mthca
In-Reply-To: <20050221112958.GD14855@mellanox.co.il>
References: <20050221112958.GD14855@mellanox.co.il>
Message-ID: <20050222185007.GB25382@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: [PATCH] initialize device_cap_flags in mthca
> 
> device_cap_flags are uninitialized in mthca.
> 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

This was reporting the RQ_SIG capability which is now gone.
Here's an update.

Index: hw/mthca/mthca_main.c
===================================================================
--- hw/mthca/mthca_main.c	(revision 1862)
+++ hw/mthca/mthca_main.c	(working copy)
@@ -171,6 +171,33 @@ static int __devinit mthca_dev_lim(struc
 	mdev->limits.reserved_uars      = dev_lim->reserved_uars;
 	mdev->limits.reserved_pds       = dev_lim->reserved_pds;
 
+	/* IB_DEVICE_RESIZE_MAX_WR not supported by driver.
+	   May be doable since hardware supports it for SRQ.
+
+	   IB_DEVICE_N_NOTIFY_CQ is supported by hardware but not by driver.
+
+	   IB_DEVICE_SRQ_RESIZE is supported by hardware but SRQ is not
+	   supported by driver. */
+	mdev->limits.device_cap_flags = IB_DEVICE_CHANGE_PHY_PORT |
+		IB_DEVICE_PORT_ACTIVE_EVENT |
+		IB_DEVICE_SYS_IMAGE_GUID |
+		IB_DEVICE_RC_RNR_NAK_GEN;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_BAD_PKEY_CNTR)
+		mdev->limits.device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_BAD_QKEY_CNTR)
+		mdev->limits.device_cap_flags |= IB_DEVICE_BAD_QKEY_CNTR;
+				
+	if (dev_lim->flags & DEV_LIM_FLAG_RAW_MULTI)
+		mdev->limits.device_cap_flags |= IB_DEVICE_RAW_MULTI;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_AUTO_PATH_MIG)
+		mdev->limits.device_cap_flags |= IB_DEVICE_AUTO_PATH_MIG;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_UD_AV_PORT_ENFORCE)
+		mdev->limits.device_cap_flags |= IB_DEVICE_UD_AV_PORT_ENFORCE;
+
 	if (dev_lim->flags & DEV_LIM_FLAG_SRQ)
 		mdev->mthca_flags |= MTHCA_FLAG_SRQ;
 
Index: hw/mthca/mthca_provider.c
===================================================================
--- hw/mthca/mthca_provider.c	(revision 1862)
+++ hw/mthca/mthca_provider.c	(working copy)
@@ -43,6 +43,8 @@ static int mthca_query_device(struct ib_
 	struct ib_smp *in_mad  = NULL;
 	struct ib_smp *out_mad = NULL;
 	int err = -ENOMEM;
+	struct mthca_dev* mdev = to_mdev(ibdev);
+
 	u8 status;
 
 	in_mad  = kmalloc(sizeof *in_mad, GFP_KERNEL);
@@ -50,7 +52,7 @@ static int mthca_query_device(struct ib_
 	if (!in_mad || !out_mad)
 		goto out;
 
-	props->fw_ver        = to_mdev(ibdev)->fw_ver;
+	props->fw_ver              = mdev->fw_ver;
 
 	memset(in_mad, 0, sizeof *in_mad);
 	in_mad->base_version       = 1;
@@ -59,7 +61,7 @@ static int mthca_query_device(struct ib_
 	in_mad->method         	   = IB_MGMT_METHOD_GET;
 	in_mad->attr_id   	   = IB_SMP_ATTR_NODE_INFO;
 
-	err = mthca_MAD_IFC(to_mdev(ibdev), 1, 1,
+	err = mthca_MAD_IFC(mdev, 1, 1,
 			    1, NULL, NULL, in_mad, out_mad,
 			    &status);
 	if (err)
@@ -69,10 +71,11 @@ static int mthca_query_device(struct ib_
 		goto out;
 	}
 
-	props->vendor_id      = be32_to_cpup((u32 *) (out_mad->data + 36)) &
+	props->device_cap_flags = mdev->limits.device_cap_flags;
+	props->vendor_id        = be32_to_cpup((u32 *) (out_mad->data + 36)) &
 		0xffffff;
-	props->vendor_part_id = be16_to_cpup((u16 *) (out_mad->data + 30));
-	props->hw_ver         = be16_to_cpup((u16 *) (out_mad->data + 32));
+	props->vendor_part_id   = be16_to_cpup((u16 *) (out_mad->data + 30));
+	props->hw_ver           = be16_to_cpup((u16 *) (out_mad->data + 32));
 	memcpy(&props->sys_image_guid, out_mad->data +  4, 8);
 	memcpy(&props->node_guid,      out_mad->data + 12, 8);
 
Index: hw/mthca/mthca_cmd.h
===================================================================
--- hw/mthca/mthca_cmd.h	(revision 1862)
+++ hw/mthca/mthca_cmd.h	(working copy)
@@ -95,7 +95,21 @@ enum {
 };
 
 enum {
-	DEV_LIM_FLAG_SRQ = 1 << 6
+	DEV_LIM_FLAG_RC                 = 1 << 0,
+	DEV_LIM_FLAG_UC                 = 1 << 1,
+	DEV_LIM_FLAG_UD                 = 1 << 2,
+	DEV_LIM_FLAG_RD                 = 1 << 3,
+	DEV_LIM_FLAG_RAW_IPV6           = 1 << 4,
+	DEV_LIM_FLAG_RAW_ETHER          = 1 << 5,
+	DEV_LIM_FLAG_SRQ                = 1 << 6,
+	DEV_LIM_FLAG_BAD_PKEY_CNTR      = 1 << 8,
+	DEV_LIM_FLAG_BAD_QKEY_CNTR      = 1 << 9,
+	DEV_LIM_FLAG_MW                 = 1 << 16,
+	DEV_LIM_FLAG_AUTO_PATH_MIG      = 1 << 17,
+	DEV_LIM_FLAG_ATOMIC             = 1 << 18,
+	DEV_LIM_FLAG_RAW_MULTI          = 1 << 19,
+	DEV_LIM_FLAG_UD_AV_PORT_ENFORCE = 1 << 20,
+	DEV_LIM_FLAG_UD_MULTI           = 1 << 21,
 };
 
 struct mthca_dev_lim {

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Tue Feb 22 11:08:06 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 11:08:06 -0800
Subject: [openib-general] [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <20050222184055.GA25382@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 22 Feb 2005 20:40:55 +0200")
References: <20050203175419.GJ2536@mellanox.co.il>
	<52ekf8sfn3.fsf@topspin.com> <20050222184055.GA25382@mellanox.co.il>
Message-ID: <52zmxwpg4p.fsf@topspin.com>

    Michael> Instead of doing this, how about something else I would
    Michael> prefer: we could avoid locking the QP on CQ poll
    Michael> altogether, if there is a separate last polled index that
    Michael> is written only by cq poll and read by qp post.  This
    Michael> index update would be protected by a cq lock.

Yes, that's a greate solution.  Splitting things up so that some
values are only written inside the WQ lock and some are only written
inside the CQ lock is a very clean solution.

 - R.


From mst at mellanox.co.il  Tue Feb 22 11:18:09 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 21:18:09 +0200
Subject: [openib-general] Re: [PATCH] ipoib: dont lock tx on completion
In-Reply-To: <20050208105112.GM2141@mellanox.co.il>
References: <20050208105112.GM2141@mellanox.co.il>
Message-ID: <20050222191809.GC25382@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: [PATCH] ipoib: dont lock tx on completion
> 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
> 
> By testing again, and waking the net queue if completions arrived,
> we can avoid taking the tx lock on send completion in ip over ib,
> reducing the lock contention.

OK, since this is getting complicated, lets start with keeping
a separate lock for completions.
The posting thread will take it before stopping the interface.

Roland, if it makes sence I'll implement it tomorrow.

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Tue Feb 22 11:23:59 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 11:23:59 -0800
Subject: [openib-general] ANNOUNCE: First usable version of userspace verbs
Message-ID: <52u0o4pfe8.fsf@topspin.com>

I'm happy to announce the initial availability of userspace verbs
support for brave testers.

To try this out, check out the roland-uverbs subversion branch:

    svn co https://openib.org/svn/gen2/branches/roland-uverbs

and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
userspace verbs support.

If you want to use a linux-2.6.10 kernel, you will need to apply the
new linux-2.6.10-backports.diff patch from the branch (which just
exports get_sb_pseudo()).  No patches at all are required for an
up-to-date BK or linux-2.6.11-rc4 tree.

If you use udev, add the rule

    KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"

to your configuration.  Otherwise, create the required device files:

    mknod /dev/infiniband/uverbs0 c 231 128
    mknod /dev/infiniband/uverbs1 c 231 129

and so on for as many HCAs as you have installed.

The build the userspace libraries in src/userspace/libibverbs and
src/userspace/libmthca with the usual

    ./autogen.sh && ./configure && make && sudo make install

passing whatever parameters to configure you want; you can use
--prefix to install to another location.  If you set a non-standard
prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
configure for libmthca.

Once you have the libraries built and installed, load the ib_mthca and
ib_uverbs modules. By default, libibverbs will search for driver
libraries in <prefix>/lib/infiniband; if you installed libmthca
somewhere else, set the OPENIB_DRIVER_PATH environment variable to
point to the directory with mthca.so.

To actually try things out, you can use the ibv_pingpong program
shipped as part of the libibverbs package.  For example, one one
system start the server side

    $ ibv_pingpong

and on another system start the client by passing the address of the
server (in this example I use IPv6 over IPoIB):

    $ ibv_pingpong fe80::202:c901:7fc:c711%ib0

The pingpong program has a number of options -- run ibv_pingpong -h to
see a list of the switches you can try.

The current code is stable for me, but all that means is that my tiny
selection of tests and test systems has not uncovered any of the bugs
that are undoubtedly present.  Some of the limitations I know about:

 - Only RC is implemented.  There are not even any functions to call
   to create UD address handles yet.
 - Only Tavor mode is supported -- PCI Express HCAs will not work if
   they are running mem-free firmware.
 - On x86, only CPUs with SSE will work now.  I'd be surprised if
   anyone has x86 system with an HCA that doesn't have SSE.

Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
kernels -- I don't expect any portability problems but I haven't even
built for other architectures.

In any case, please give this a spin and let me know how it looks to you.

My short- and medium-term plans are:

  1. Catch up on reviewing and applying the patche queue I'm sitting on.
  2. Land the Arbel mem-free mode support from the roland-uverbs branch
     onto the main trunk (and merge it upstream once 2.6.11 is out and
     2.6.12 opens).
  3. Implement UD support for userspace.  I should have this done before
     the end of next week.
  4. Implement mem-free support for userspace.

Thanks,
  Roland


From mshefty at ichips.intel.com  Tue Feb 22 12:16:04 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 22 Feb 2005 12:16:04 -0800
Subject: [openib-general] [PATCH] [CM]: Add initial reject handling code
Message-ID: <20050222121604.3ffb9bde.mshefty@ichips.intel.com>

This patch adds initial support for handling CM reject messages.  It
should work with the following reject codes: 1-3, 5, 8-9, 11-23,
26-29, 32-33.  (These are basically rejections by the remote consumer,
versus by the remote CM.)

I still need to determine the proper handling for the reject codes not
listed above.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1863)
+++ infiniband/core/cm.c	(working copy)
@@ -1794,9 +1794,85 @@ out:
 }
 EXPORT_SYMBOL(ib_send_cm_rej);
 
+static void cm_format_rej_event(struct cm_work *work)
+{
+	struct cm_rej_msg *rej_msg;
+	struct ib_cm_rej_event_param *param;
+
+	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.rej_rcvd;
+	param->ari = rej_msg->ari;
+	param->ari_length = cm_rej_get_reject_info_len(rej_msg);
+	param->reason = rej_msg->reason;
+	work->cm_event.private_data = &rej_msg->private_data;
+}
+
+static struct cm_id_private * cm_acquire_rejected_id(struct cm_rej_msg *rej_msg)
+{
+	u32 local_id, remote_id;
+
+	/* todo: lookup by remote CA for rejects due to timeouts */
+	local_id = rej_msg->remote_comm_id;
+	remote_id = rej_msg->local_comm_id;
+
+	return cm_acquire_id(local_id, remote_id);
+}
+
 static int cm_rej_handler(struct cm_work *work)
 {
-	/* todo: write reject handler */
+	struct cm_id_private *cm_id_priv;
+	struct cm_rej_msg *rej_msg;
+	unsigned long flags;
+	u64 wr_id = 0;
+	int ret;
+
+	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_rejected_id(rej_msg);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	cm_format_rej_event(work);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_REP_SENT:
+		wr_id = (unsigned long) cm_id_priv->msg;
+		/* fall through */
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_MRA_REP_RCVD:
+		cm_reset_to_idle(cm_id_priv);
+		break;
+	case IB_CM_DREQ_SENT:
+		wr_id = (unsigned long) cm_id_priv->msg;
+		/* fall through */
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_ESTABLISHED:
+		cm_enter_timewait(cm_id_priv);
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (wr_id)
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
 	return -EINVAL;
 }
 

 


From mst at mellanox.co.il  Tue Feb 22 11:40:01 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 21:40:01 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <52u0o4pfe8.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
Message-ID: <20050222194001.GD25382@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: ANNOUNCE: First usable version of userspace verbs
> 
> I'm happy to announce the initial availability of userspace verbs
> support for brave testers.
> 
> To try this out, check out the roland-uverbs subversion branch:
> 
>     svn co https://openib.org/svn/gen2/branches/roland-uverbs
> 
> and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
> userspace verbs support.
> 
> If you want to use a linux-2.6.10 kernel, you will need to apply the
> new linux-2.6.10-backports.diff patch from the branch (which just
> exports get_sb_pseudo()).  No patches at all are required for an
> up-to-date BK or linux-2.6.11-rc4 tree.
> 
> If you use udev, add the rule
> 
>     KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> 
> to your configuration.  Otherwise, create the required device files:
> 
>     mknod /dev/infiniband/uverbs0 c 231 128
>     mknod /dev/infiniband/uverbs1 c 231 129
> 
> and so on for as many HCAs as you have installed.
> 
> The build the userspace libraries in src/userspace/libibverbs and
> src/userspace/libmthca with the usual
> 
>     ./autogen.sh && ./configure && make && sudo make install
> 
> passing whatever parameters to configure you want; you can use
> --prefix to install to another location.  If you set a non-standard
> prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
> configure for libmthca.
> 
> Once you have the libraries built and installed, load the ib_mthca and
> ib_uverbs modules. By default, libibverbs will search for driver
> libraries in <prefix>/lib/infiniband; if you installed libmthca
> somewhere else, set the OPENIB_DRIVER_PATH environment variable to
> point to the directory with mthca.so.
> 
> To actually try things out, you can use the ibv_pingpong program
> shipped as part of the libibverbs package.  For example, one one
> system start the server side
> 
>     $ ibv_pingpong
> 
> and on another system start the client by passing the address of the
> server (in this example I use IPv6 over IPoIB):
> 
>     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> 
> The pingpong program has a number of options -- run ibv_pingpong -h to
> see a list of the switches you can try.
> 
> The current code is stable for me, but all that means is that my tiny
> selection of tests and test systems has not uncovered any of the bugs
> that are undoubtedly present.  Some of the limitations I know about:
> 
>  - Only RC is implemented.  There are not even any functions to call
>    to create UD address handles yet.
>  - Only Tavor mode is supported -- PCI Express HCAs will not work if
>    they are running mem-free firmware.
>  - On x86, only CPUs with SSE will work now.  I'd be surprised if
>    anyone has x86 system with an HCA that doesn't have SSE.
> 
> Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
> kernels -- I don't expect any portability problems but I haven't even
> built for other architectures.
> 
> In any case, please give this a spin and let me know how it looks to you.
> 
> My short- and medium-term plans are:
> 
>   1. Catch up on reviewing and applying the patche queue I'm sitting on.
>   2. Land the Arbel mem-free mode support from the roland-uverbs branch
>      onto the main trunk (and merge it upstream once 2.6.11 is out and
>      2.6.12 opens).

Would you like help with (2)?
Specifically, how should are backport patches split up?
I guess per file? Still, it seems unlikely
they can be made independent, right?

>   3. Implement UD support for userspace.  I should have this done before
>      the end of next week.
>   4. Implement mem-free support for userspace.
> 
> Thanks,
>   Roland

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Tue Feb 22 11:44:35 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 11:44:35 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050222194001.GD25382@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 22 Feb 2005 21:40:01 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050222194001.GD25382@mellanox.co.il>
Message-ID: <52psyspefw.fsf@topspin.com>

    Michael> Would you like help with (2)?  Specifically, how should
    Michael> are backport patches split up?  I guess per file? Still,
    Michael> it seems unlikely they can be made independent, right?

No thanks, I think I can handle it.  I'm planning on making the
patches independent and per-change -- for example update the EQ
handling, add support for DB allocatation, update QP/CQ allocation, etc.

 - R.


From roland at topspin.com  Tue Feb 22 11:58:44 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 11:58:44 -0800
Subject: [openib-general] Re: [PATCH] ipoib: dont lock tx on completion
In-Reply-To: <20050222191809.GC25382@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 22 Feb 2005 21:18:09 +0200")
References: <20050208105112.GM2141@mellanox.co.il>
	<20050222191809.GC25382@mellanox.co.il>
Message-ID: <52k6p0pdsb.fsf@topspin.com>

    Michael> OK, since this is getting complicated, lets start with
    Michael> keeping a separate lock for completions.  The posting
    Michael> thread will take it before stopping the interface.

    Michael> Roland, if it makes sence I'll implement it tomorrow.

Let's just hold off on this until all of Shirley's patches are
landed.  As I recall, this change doesn't fix a bug or measurably
improve performance, so I don't see any urgency.

 - R.


From libor at topspin.com  Tue Feb 22 12:20:58 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 12:20:58 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
In-Reply-To: <11087738623823@sun.com>;
	from Tom.Duffy@Sun.COM on Fri, Feb 18, 2005 at 04:44:22PM -0800
References: <11087738623823@sun.com>
Message-ID: <20050222122058.A23420@topspin.com>

On Fri, Feb 18, 2005 at 04:44:22PM -0800, Tom Duffy wrote:
> This series of patches cleans up the whitespace in SDP.  Mostly it removes
> the extra line that was after every conditional, but it also removes
> unneeded brackets around single line conditionals as well.

  Thanks Tom, I've applied and commited the patches.

-Libor


From libor at topspin.com  Tue Feb 22 12:24:17 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 12:24:17 -0800
Subject: [openib-general] Re: [PATCH][SDP][14/22] Whitespace cleanup in
	sdp_proto.h
In-Reply-To: <1108922863.15680.9.camel@duffman>;
	from tduffy@sun.com on Sun, Feb 20, 2005 at 10:07:43AM -0800
References: <11087739341677@sun.com> <11087739441194@sun.com>
	<11087739553583@sun.com> <11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
	<20050220000536.GB8268@mellanox.co.il>
	<1108922863.15680.9.camel@duffman>
Message-ID: <20050222122417.B23420@topspin.com>

On Sun, Feb 20, 2005 at 10:07:43AM -0800, Tom Duffy wrote:
> On Sun, 2005-02-20 at 02:05 +0200, Michael S. Tsirkin wrote:
> >
> > I think we also should replace __inline__ with inline everywhere.
> > 
> > Also, I think the convention is to add __ before inline functions
> > if they are an implementation detail and shouldnt be called
> > directly.
> > That does not seem to be the case for __sdp_inet_write_space, or for
> > other stuff in sdp_proto.h, so I think these functions shall have
> >  proper names starting with an alphanumeric.
> 
> Yeah, I also don't like the convention of using _ in front of static
> functions.  Those can be updated in the next set of patches ;)

   I agree, all those changes should be made to the code. There's a
number of coding conventions which need to be changed to a saner
form.

-Libor


From libor at topspin.com  Tue Feb 22 12:28:01 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 12:28:01 -0800
Subject: [openib-general] Re: Re: SDP_CONN_LOCK
In-Reply-To: <20050220001137.GC8268@mellanox.co.il>;
	from mst@mellanox.co.il on Sun, Feb 20, 2005 at 02:11:37AM +0200
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
	<20050220001137.GC8268@mellanox.co.il>
Message-ID: <20050222122801.C23420@topspin.com>

On Sun, Feb 20, 2005 at 02:11:37AM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > 
> >   And here I would have thought SDP_CONN_HOLD as a macro would have
> > bugged people more. :)
>
> Didn't get that far yet :)
> 
> I also wander whether we could get rid of most called to 
> _CHECK macros, consequently turn lots of functions
> which cant otherwise fail into void?
> Given that these checks are disabled upon a compile switch, one wanders
> how big a debugging aid they are.

  Yes, I think the _CHECK and EXPECT macros can be gotten rid of. Getting
rid of _CHECK does have the added benefit of turning a lot of functions
into void return functions, which in turn can get rid of most of the
EXPECT macros. I do agree that there are quite a few function that return
an status which can safely be turned into void functions.

-Libor



From libor at topspin.com  Tue Feb 22 12:31:47 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 12:31:47 -0800
Subject: [openib-general] Re: [PATCH] remove unused code from SDP.
In-Reply-To: <20050220001503.GD8268@mellanox.co.il>;
	from mst@mellanox.co.il on Sun, Feb 20, 2005 at 02:15:04AM +0200
References: <20050218120306.C15428@topspin.com>
	<20050220001503.GD8268@mellanox.co.il>
Message-ID: <20050222123147.D23420@topspin.com>

On Sun, Feb 20, 2005 at 02:15:04AM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> >
> > -int _sdp_link_path_rec_get(struct sdp_path_info *info)
> > +static int _sdp_link_path_rec_get(struct sdp_path_info *info)

> Good stuff. 
> One comment though - one can give short names to static functions:
> since they are local to file they wont conflict with anything other.
> So - why prepend an "_" to the name?

  It was just habit, there's no good reason for the convention. I
have no problem changing it. 

  This file in particular (sdp_link.c) in the future should probably
be broken out into a seperate module, since it provides IP address to
PathRecord resolution using IPoIB and there are other protocols/modules
which will need this functionality. 

-Libor


From mst at mellanox.co.il  Tue Feb 22 12:32:05 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 22:32:05 +0200
Subject: [openib-general] Re: Re: [PATCH][SDP][14/22] Whitespace cleanup in
	sdp_proto.h
In-Reply-To: <20050222122417.B23420@topspin.com>
References: <11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com> <11087739751674@sun.com>
	<11087739852191@sun.com> <11087739963627@sun.com>
	<11087740063185@sun.com> <20050220000536.GB8268@mellanox.co.il>
	<1108922863.15680.9.camel@duffman>
	<20050222122417.B23420@topspin.com>
Message-ID: <20050222203205.GE25382@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: Re: [PATCH][SDP][14/22] Whitespace cleanup in sdp_proto.h
> 
> On Sun, Feb 20, 2005 at 10:07:43AM -0800, Tom Duffy wrote:
> > On Sun, 2005-02-20 at 02:05 +0200, Michael S. Tsirkin wrote:
> > >
> > > I think we also should replace __inline__ with inline everywhere.
> > > 
> > > Also, I think the convention is to add __ before inline functions
> > > if they are an implementation detail and shouldnt be called
> > > directly.
> > > That does not seem to be the case for __sdp_inet_write_space, or for
> > > other stuff in sdp_proto.h, so I think these functions shall have
> > >  proper names starting with an alphanumeric.
> > 
> > Yeah, I also don't like the convention of using _ in front of static
> > functions.  Those can be updated in the next set of patches ;)
> 
>    I agree, all those changes should be made to the code. There's a
> number of coding conventions which need to be changed to a saner
> form.
> 
> -Libor
> 

Good. By the way, do you plan to keep the convention of writing
0 == foo?

I personally dislike it (and Linux expressed the same santiment at some
point), this practice unnecessary with gcc
which warns if you write if (foo = 0) accidentally, and
kernel code itself never does it, but some in-kernel drivers 
do it ... its easy to fix more or less automatically.


-- 
MST - Michael S. Tsirkin


From tduffy at sun.com  Tue Feb 22 12:57:00 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 22 Feb 2005 12:57:00 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
In-Reply-To: <20050222122058.A23420@topspin.com>
References: <11087738623823@sun.com>  <20050222122058.A23420@topspin.com>
Message-ID: <1109105820.24909.48.camel@duffman>

On Tue, 2005-02-22 at 12:20 -0800, Libor Michalek wrote:
>   Thanks Tom, I've applied and commited the patches.

Did you choose not to apply these changes, or was this a simple
oversight?

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1865)
+++ sdp_inet.c	(working copy)
@@ -410,7 +410,6 @@ static int _sdp_inet_release(struct sock
 
 			while (0 < timeout &&
 			       0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
-
 				sdp_conn_unlock(conn);
 				timeout = schedule_timeout(timeout);
 				sdp_conn_lock(conn);
@@ -522,7 +521,6 @@ static int _sdp_inet_bind(struct socket 
 
 	result = sdp_inet_port_get(conn, bind_port);
 	if (0 > result) {
-
 		sdp_dbg_warn(conn, "Error <%d> getting port during bind",
 			     result);
 
@@ -1174,7 +1172,6 @@ static int _sdp_inet_ioctl(struct socket
 		sdp_conn_lock(conn);
 
 		if (SDP_SOCK_ST_LISTEN != conn->istate) {
-
 			value = conn->send_qud;
 			result = put_user(value, (int __user *) arg);
 		} else
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1865)
+++ sdp_event.c	(working copy)
@@ -512,44 +512,32 @@ int sdp_cm_event_handler(struct ib_cm_id
 	 * lookup the connection, on a REQ_RECV the sk will be empty.
 	 */
 	conn = sdp_conn_table_lookup(hashent);
-	if (NULL != conn) {
-
+	if (NULL != conn)
 		sdp_conn_lock(conn);
-	}
-	else {
-
-		if (IB_CM_REQ_RCVD != cm_id->state) {
-
+	else
+		if (IB_CM_REQ_RCVD != cm_id->state)
 			sdp_dbg_warn(NULL, 
 				     "No conn <%d> CM state <%d> event <%d>",
 				     hashent, cm_id->state, event->event);
-		}
-	}
 
 	switch (cm_id->state) {
 	case IB_CM_REQ_RCVD:
-
 		result = sdp_cm_req_handler(cm_id, event);
 		break;
 	case IB_CM_REP_RCVD:
-
 		result = sdp_cm_rep_handler(cm_id, event, conn);
 		break;
 	case IB_CM_IDLE:
-
 		result = _sdp_cm_idle(cm_id, event, conn);
 		break;
 	case IB_CM_ESTABLISHED:
-
 		result = _sdp_cm_established(cm_id, event, conn);
 		break;
 	case IB_CM_DREQ_RCVD:
 	case IB_CM_TIMEWAIT:
-
 		result = _sdp_cm_timewait(cm_id, event, conn);
 		break;
 	default:
-
 		sdp_dbg_warn(conn, "Unexpected CM state <%d>", cm_id->state);
 		result = -EINVAL;
 	}



From tduffy at sun.com  Tue Feb 22 13:10:21 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 22 Feb 2005 13:10:21 -0800
Subject: [openib-general] [PATCH][SDP] replace __inline__ with plain inline
In-Reply-To: <20050220000536.GB8268@mellanox.co.il>
References: <1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com> <11087739441194@sun.com>
	<11087739553583@sun.com>
	<11087739651393@sun.com> <11087739751674@sun.com>
	<11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
	<20050220000536.GB8268@mellanox.co.il>
Message-ID: <1109106621.24909.51.camel@duffman>

On Sun, 2005-02-20 at 02:05 +0200, Michael S. Tsirkin wrote:
> I think we also should replace __inline__ with inline everywhere.

Index: sdp_queue.h
===================================================================
--- sdp_queue.h	(revision 1865)
+++ sdp_queue.h	(working copy)
@@ -73,7 +73,7 @@ struct sdpc_desc_q {
 /*
  * __sdp_desc_q_size - return the number of elements in the table
  */
-static __inline__ int __sdp_desc_q_size(struct sdpc_desc_q *table)
+static inline int __sdp_desc_q_size(struct sdpc_desc_q *table)
 {
 	return table->size;
 } /* __sdp_desc_q_size */
@@ -81,7 +81,7 @@ static __inline__ int __sdp_desc_q_size(
 /*
  * __sdp_desc_q_member - return non-zero if element is in a table
  */
-static __inline__ int __sdp_desc_q_member(struct sdpc_desc *element)
+static inline int __sdp_desc_q_member(struct sdpc_desc *element)
 {
 	return ((NULL == element->table) ? 0 : 1);
 } /* __sdp_desc_q_member */
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1865)
+++ sdp_proto.h	(working copy)
@@ -569,7 +569,7 @@ do {                                    
 /*
  * __sdp_inet_write_space - writable space on send side
  */
-static __inline__ int __sdp_inet_write_space(struct sdp_opt *conn, int urg)
+static inline int __sdp_inet_write_space(struct sdp_opt *conn, int urg)
 {
 	int size;
 
@@ -594,7 +594,7 @@ static __inline__ int __sdp_inet_write_s
 /*
  * __sdp_inet_writable - return non-zero if socket is writable
  */
-static __inline__ int __sdp_inet_writable(struct sdp_opt *conn)
+static inline int __sdp_inet_writable(struct sdp_opt *conn)
 {
 	SDP_CHECK_NULL(conn, -EINVAL);
 
@@ -608,7 +608,7 @@ static __inline__ int __sdp_inet_writabl
 /*
  * __sdp_conn_stat_dump - dump stats to the log
  */
-static __inline__ int __sdp_conn_stat_dump(struct sdp_opt *conn)
+static inline int __sdp_conn_stat_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATS_REC
 	int counter;
@@ -631,7 +631,7 @@ static __inline__ int __sdp_conn_stat_du
 /*
  * __sdp_conn_state_dump - dump state information to the log
  */
-static __inline__ int __sdp_conn_state_dump(struct sdp_opt *conn)
+static inline int __sdp_conn_state_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATE_REC
 	int counter;
Index: sdp_msgs.h
===================================================================
--- sdp_msgs.h	(revision 1865)
+++ sdp_msgs.h	(working copy)
@@ -278,7 +278,7 @@ struct msg_hdr_sch {
 /*
  * __sdp_msg_swap_bsdh - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
+static inline void __sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
 {
 	header->recv_bufs = cpu_to_be16(header->recv_bufs);
 	header->size = cpu_to_be32(header->size);
@@ -289,7 +289,7 @@ static __inline__ void __sdp_msg_swap_bs
 /*
  * __sdp_msg_swap_hh - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_hh(struct msg_hdr_hh *header)
+static inline void __sdp_msg_swap_hh(struct msg_hdr_hh *header)
 {
 	header->r_rcv_size = cpu_to_be32(header->r_rcv_size);
 	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
@@ -307,7 +307,7 @@ static __inline__ void __sdp_msg_swap_hh
 /*
  * __sdp_msg_swap_hah - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
+static inline void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
 {
 	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
 } /* __sdp_msg_swap_hah */
@@ -315,7 +315,7 @@ static __inline__ void __sdp_msg_swap_ha
 /*
  * __sdp_msg_swap_srcah - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
+static inline void __sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
 {
 	header->size = cpu_to_be32(header->size);
 #ifdef _SDP_MS_APRIL_ERROR_COMPAT
@@ -330,7 +330,7 @@ static __inline__ void __sdp_msg_swap_sr
 /*
  * __sdp_msg_swap_snkah - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
+static inline void __sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
 {
 	header->size = cpu_to_be32(header->size);
 	header->r_key = cpu_to_be32(header->r_key);
@@ -341,7 +341,7 @@ static __inline__ void __sdp_msg_swap_sn
 /*
  * __sdp_msg_swap_rwch - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
+static inline void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_rwch */
@@ -349,7 +349,7 @@ static __inline__ void __sdp_msg_swap_rw
 /*
  * __sdp_msg_swap_rrch - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
+static inline void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_rrch */
@@ -357,7 +357,7 @@ static __inline__ void __sdp_msg_swap_rr
 /*
  * __sdp_msg_swap_mch - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
+static inline void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
 {
 	return;
 } /* __sdp_msg_swap_ */
@@ -365,7 +365,7 @@ static __inline__ void __sdp_msg_swap_mc
 /*
  * __sdp_msg_swap_crbh - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
+static inline void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_crbh */
@@ -373,7 +373,7 @@ static __inline__ void __sdp_msg_swap_cr
 /*
  * __sdp_msg_swap_crbah - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
+static inline void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
 {
 	header->size = cpu_to_be32(header->size);
 } /* __sdp_msg_swap_crbah */
@@ -381,7 +381,7 @@ static __inline__ void __sdp_msg_swap_cr
 /*
  * __sdp_msg_swap_sch - SDP header endian byte swapping
  */
-static __inline__ void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
+static inline void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
 {
 	header->service_id = cpu_to_be64(header->service_id);
 } /* __sdp_msg_swap_sch */
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1865)
+++ sdp_send.c	(working copy)
@@ -1094,8 +1094,7 @@ done:
 /*
  * _sdp_send_data_buff_get - get an appropriate write buffer for send
  */
-static __inline__ struct sdpc_buff *_sdp_send_data_buff_get(
-	struct sdp_opt *conn)
+static inline struct sdpc_buff *_sdp_send_data_buff_get(struct sdp_opt *conn)
 {
 	struct sdpc_buff *buff;
 
@@ -1121,10 +1120,10 @@ static __inline__ struct sdpc_buff *_sdp
 /*
  * _sdp_send_data_buff_put - place a buffer into the send queue
  */
-static __inline__ int _sdp_send_data_buff_put(struct sdp_opt *conn,
-					      struct sdpc_buff *buff,
-					      int size,
-					      int urg)
+static inline int _sdp_send_data_buff_put(struct sdp_opt *conn,
+					  struct sdpc_buff *buff,
+					  int size,
+					  int urg)
 {
 	int result = 0;
 	int expect;
Index: sdp_iocb.h
===================================================================
--- sdp_iocb.h	(revision 1865)
+++ sdp_iocb.h	(working copy)
@@ -131,7 +131,7 @@ struct sdpc_iocb_q {
 /*
  * __sdp_kmap - map a page into kernel space
  */
-static __inline__ void *__sdp_kmap(struct page *page)
+static inline void *__sdp_kmap(struct page *page)
 {
 	if (in_atomic() || irqs_disabled())
 		return kmap_atomic(page, KM_IRQ0);
@@ -142,7 +142,7 @@ static __inline__ void *__sdp_kmap(struc
 /*
  * __sdp_kunmap - unmap a page into kernel space
  */
-static __inline__ void __sdp_kunmap(struct page *page)
+static inline void __sdp_kunmap(struct page *page)
 {
 	if (in_atomic() || irqs_disabled())
 		kunmap_atomic(page, KM_IRQ0);
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1865)
+++ sdp_buff.c	(working copy)
@@ -42,12 +42,12 @@ static struct sdpc_buff_root *main_pool 
 /*
  * _sdp_buff_q_get - Get a buffer from a specific pool
  */
-static __inline__ struct sdpc_buff *_sdp_buff_q_get(struct sdpc_buff_q *pool,
-						    int fifo,
-						    int (*test_func)
-							(struct sdpc_buff *buff,
-							 void *arg),
-						    void *usr_arg)
+static inline struct sdpc_buff *_sdp_buff_q_get(struct sdpc_buff_q *pool,
+						int fifo,
+						int (*test_func)
+						     (struct sdpc_buff *buff,
+						      void *arg),
+						void *usr_arg)
 {
 	struct sdpc_buff *buff;
 
@@ -86,9 +86,9 @@ static __inline__ struct sdpc_buff *_sdp
 /*
  * _sdp_buff_q_put - Place a buffer into a specific pool
  */
-static __inline__ int _sdp_buff_q_put(struct sdpc_buff_q *pool,
-				      struct sdpc_buff *buff,
-				      int fifo)
+static inline int _sdp_buff_q_put(struct sdpc_buff_q *pool,
+				  struct sdpc_buff *buff,
+				  int fifo)
 {
 	/* fifo: false == tail, true == head */
 	SDP_CHECK_NULL(pool, -EINVAL);
@@ -121,8 +121,8 @@ static __inline__ int _sdp_buff_q_put(st
 /*
  * _sdp_buff_q_look - look at a buffer from a specific pool
  */
-static __inline__ struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
-						     int fifo)
+static inline struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
+						 int fifo)
 {
 	SDP_CHECK_NULL(pool, NULL);
 
@@ -135,8 +135,8 @@ static __inline__ struct sdpc_buff *_sdp
 /*
  * _sdp_buff_q_remove - remove a specific buffer from a specific pool
  */
-static __inline__ int _sdp_buff_q_remove(struct sdpc_buff_q *pool, 
-					 struct sdpc_buff *buff)
+static inline int _sdp_buff_q_remove(struct sdpc_buff_q *pool, 
+				     struct sdpc_buff *buff)
 {
 	struct sdpc_buff *prev;
 	struct sdpc_buff *next;
@@ -448,10 +448,7 @@ static int _sdp_buff_pool_release(struct
 /*
  * _sdp_buff_pool_release_check - check for buffer release from main pool
  */
-static __inline__ int _sdp_buff_pool_release_check
-(
-	struct sdpc_buff_root *m_pool
-)
+static inline int _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
 {
 	SDP_CHECK_NULL(m_pool, -EINVAL);
 	/*
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1865)
+++ sdp_queue.c	(working copy)
@@ -79,9 +79,9 @@ static struct sdpc_desc *_sdp_desc_q_get
 /*
  * _sdp_desc_q_put - Place an element into a specific table
  */
-static __inline__ int _sdp_desc_q_put(struct sdpc_desc_q *table,
-				      struct sdpc_desc *element, 
-				      int fifo)
+static inline int _sdp_desc_q_put(struct sdpc_desc_q *table,
+				  struct sdpc_desc *element, 
+				  int fifo)
 {
 	/*
 	 * fifo: false == tail, true == head



From mst at mellanox.co.il  Tue Feb 22 13:14:16 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 23:14:16 +0200
Subject: [openib-general] Re: [PATCH] 1/2 separate locking for send and
	receive q in mthca
In-Reply-To: <52zmxwpg4p.fsf@topspin.com>
References: <20050203175419.GJ2536@mellanox.co.il> <52ekf8sfn3.fsf@topspin.com>
	<20050222184055.GA25382@mellanox.co.il>
	<52zmxwpg4p.fsf@topspin.com>
Message-ID: <20050222211416.GF25382@mellanox.co.il>


Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] 1/2 separate locking for send and receive q in mthca
> 
>     Michael> Instead of doing this, how about something else I would
>     Michael> prefer: we could avoid locking the QP on CQ poll
>     Michael> altogether, if there is a separate last polled index that
>     Michael> is written only by cq poll and read by qp post.  This
>     Michael> index update would be protected by a cq lock.
> 
> Yes, that's a greate solution.  Splitting things up so that some
> values are only written inside the WQ lock and some are only written
> inside the CQ lock is a very clean solution.
> 
>  - R.
> 

Does something along the following lines look more or less OK?
(warning: untested, dont commit, I post it just for comments).

Note also how the fact that qp locks the cq
when it overflows let us avoid a cq doorbell when we switch qps.
Which has no effect on ip over ib but is nice for ulps that have more
than one qp per cq.

Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 1862)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -147,8 +147,8 @@ struct mthca_cq {
 };
 
 struct mthca_wq {
+	spinlock_t lock;
 	int   max;
-	int   cur;
 	int   next;
 	int   last_comp;
 	void *last;
@@ -158,7 +158,6 @@ struct mthca_wq {
 
 struct mthca_qp {
 	struct ib_qp           ibqp;
-	spinlock_t             lock;
 	atomic_t               refcount;
 	u32                    qpn;
 	int                    is_direct;
Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 1862)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -405,15 +405,6 @@ static inline int mthca_poll_one(struct 
 	is_send  = is_error ? cqe->opcode & 0x01 : cqe->is_send & 0x80;
 
 	if (!*cur_qp || be32_to_cpu(cqe->my_qpn) != (*cur_qp)->qpn) {
-		if (*cur_qp) {
-			if (*freed) {
-				wmb();
-				inc_cons_index(dev, cq, *freed);
-				*freed = 0;
-			}
-			spin_unlock(&(*cur_qp)->lock);
-		}
-
 		/*
 		 * We do not have to take the QP table lock here,
 		 * because CQs will be locked while QPs are removed
@@ -428,8 +419,6 @@ static inline int mthca_poll_one(struct 
 			err = -EINVAL;
 			goto out;
 		}
-
-		spin_lock(&(*cur_qp)->lock);
 	}
 
 	entry->qp_num = (*cur_qp)->qpn;
@@ -446,11 +435,6 @@ static inline int mthca_poll_one(struct 
 		entry->wr_id = (*cur_qp)->wrid[wqe_index];
 	}
 
-	if (wq->last_comp < wqe_index)
-		wq->cur -= wqe_index - wq->last_comp;
-	else
-		wq->cur -= wq->max - wq->last_comp + wqe_index;
-
 	wq->last_comp = wqe_index;
 
 	if (0)
@@ -533,9 +517,6 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 		inc_cons_index(dev, cq, freed);
 	}
 
-	if (qp)
-		spin_unlock(&qp->lock);
-
 	spin_unlock_irqrestore(&cq->lock, flags);
 
 	return err == 0 || err == -EAGAIN ? npolled : err;
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 1862)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -552,9 +552,11 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 		else
 			cur_state = attr->cur_qp_state;
 	} else {
-		spin_lock_irq(&qp->lock);
+		spin_lock_irq(&qp->sq.lock);
+		spin_lock(&qp->rq.lock);
 		cur_state = qp->state;
-		spin_unlock_irq(&qp->lock);
+		spin_unlock(&qp->rq.lock);
+		spin_unlock_irq(&qp->sq.lock);
 	}
 
 	if (attr_mask & IB_QP_STATE) {
@@ -972,6 +974,14 @@ static int mthca_alloc_wqe_buf(struct mt
 	return err;
 }
 
+static void mthca_wq_init(struct mthca_wq* wq)
+{
+	spin_lock_init(&wq->lock);
+	wq->next      = 0;
+	wq->last_comp = wq->max - 1;
+	wq->last      = NULL;
+}
+
 static int mthca_alloc_qp_common(struct mthca_dev *dev,
 				 struct mthca_pd *pd,
 				 struct mthca_cq *send_cq,
@@ -981,20 +991,13 @@ static int mthca_alloc_qp_common(struct 
 {
 	int err;
 
-	spin_lock_init(&qp->lock);
 	atomic_set(&qp->refcount, 1);
 	qp->state    	 = IB_QPS_RESET;
 	qp->atomic_rd_en = 0;
 	qp->resp_depth   = 0;
 	qp->sq_policy    = send_policy;
-	qp->rq.cur       = 0;
-	qp->sq.cur       = 0;
-	qp->rq.next      = 0;
-	qp->sq.next      = 0;
-	qp->rq.last_comp = qp->rq.max - 1;
-	qp->sq.last_comp = qp->sq.max - 1;
-	qp->rq.last      = NULL;
-	qp->sq.last      = NULL;
+	mthca_wq_init(&qp->sq);
+	mthca_wq_init(&qp->rq);
 
 	err = mthca_alloc_wqe_buf(dev, pd, qp);
 	return err;
@@ -1240,6 +1243,24 @@ static int build_mlx_header(struct mthca
 	return 0;
 }
 
+static inline int mthca_wq_overflow(struct mthca_wq* wq, int nreq,
+	 struct ib_cq* ib_cq)
+{
+	int cur;
+	struct mthca_cq* cq;
+
+	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	if (likely(cur + nreq < wq->max))
+		return 0;
+
+	cq = to_mcq(ib_cq);
+	spin_lock(&cq->lock);
+	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	spin_unlock(&cq->lock);
+
+	return cur + nreq >= wq->max;
+}
+
 int mthca_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
 		    struct ib_send_wr **bad_wr)
 {
@@ -1267,16 +1288,18 @@ int mthca_post_send(struct ib_qp *ibqp, 
 		[IB_WR_ATOMIC_FETCH_AND_ADD] = MTHCA_OPCODE_ATOMIC_FA,
 	};
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->sq.lock, flags);
 
 	/* XXX check that state is OK to post send */
 
 	ind = qp->sq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->sq.cur + nreq >= qp->sq.max) {
-			mthca_err(dev, "SQ full (%d posted, %d max, %d nreq)\n",
-				  qp->sq.cur, qp->sq.max, nreq);
+		if (mthca_wq_overflow(&qp->sq, nreq, qp->ibqp->send_cq)) {
+			mthca_err(dev, "SQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->sq.next, qp->sq.last_comp,
+					qp->sq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1447,10 +1470,9 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->sq.cur += nreq;
 	qp->sq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->sq.lock, flags);
 	return err;
 }
 
@@ -1469,15 +1491,18 @@ int mthca_post_receive(struct ib_qp *ibq
 	void *wqe;
 	void *prev_wqe;
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->rq.lock, flags);
 
 	/* XXX check that state is OK to post receive */
 
 	ind = qp->rq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->rq.cur + nreq >= qp->rq.max) {
-			mthca_err(dev, "RQ %06x full\n", qp->qpn);
+		if (mthca_wq_overflow(&qp->rq, nreq, qp->ibqp->recv_cq)) {
+			mthca_err(dev, "RQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->rq.next, qp->rq.last_comp,
+					qp->rq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1544,10 +1569,9 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->rq.cur += nreq;
 	qp->rq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->rq.lock, flags);
 	return err;
 }
 
-- 
MST - Michael S. Tsirkin


From tduffy at sun.com  Tue Feb 22 13:22:03 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 22 Feb 2005 13:22:03 -0800
Subject: [openib-general] Re: Re: [PATCH][SDP][14/22] Whitespace
	cleanup in sdp_proto.h
In-Reply-To: <20050222203205.GE25382@mellanox.co.il>
References: <11087739441194@sun.com> <11087739553583@sun.com>
	<11087739651393@sun.com> <11087739751674@sun.com>
	<11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
	<20050220000536.GB8268@mellanox.co.il>
	<1108922863.15680.9.camel@duffman>
	<20050222122417.B23420@topspin.com>
	<20050222203205.GE25382@mellanox.co.il>
Message-ID: <1109107323.24909.55.camel@duffman>

On Tue, 2005-02-22 at 22:32 +0200, Michael S. Tsirkin wrote:
> Good. By the way, do you plan to keep the convention of writing
> 0 == foo?
> 
> I personally dislike it (and Linux expressed the same santiment at some
> point), this practice unnecessary with gcc
> which warns if you write if (foo = 0) accidentally, and
> kernel code itself never does it, but some in-kernel drivers 
> do it ... its easy to fix more or less automatically.

I don't mind that one, but I don't really care.

How about the SDP_CHECK_NULL macro?  It is used in most (but not all)
places in the SDP code.  Does turning this checking off in non-debug
builds really speed things up a lot?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050222/b3aeebfe/attachment.sig>

From roland at topspin.com  Tue Feb 22 13:24:32 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 13:24:32 -0800
Subject: [openib-general] [PATCH] Rename debug_level to ipoib_debug_level
Message-ID: <52acpwcmpb.fsf@topspin.com>

This renames the variable "debug_level" to "ipoib_debug_level" in the
IPoIB module to avoid name clashes if IPoIB is built into the kernel.
SDP has the same variable name so this fixes "allyesconfig" for now.
module_param_named() is used to prevent the user-variable parameter
name from changing.

Libor, on general principles SDP should probably make the same change
to avoid polluting the namespace with generic names.

 - R.

Index: infiniband/ulp/ipoib/ipoib_main.c
===================================================================
--- infiniband/ulp/ipoib/ipoib_main.c	(revision 1865)
+++ infiniband/ulp/ipoib/ipoib_main.c	(working copy)
@@ -51,9 +51,9 @@ MODULE_DESCRIPTION("IP-over-InfiniBand n
 MODULE_LICENSE("Dual BSD/GPL");
 
 #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG
-int debug_level;
+int ipoib_debug_level;
 
-module_param(debug_level, int, 0644);
+module_param_named(debug_level, ipoib_debug_level, int, 0644);
 MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0");
 #endif
 
Index: infiniband/ulp/ipoib/ipoib.h
===================================================================
--- infiniband/ulp/ipoib/ipoib.h	(revision 1865)
+++ infiniband/ulp/ipoib/ipoib.h	(working copy)
@@ -308,11 +308,11 @@ static inline void ipoib_unregister_debu
 
 
 #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG
-extern int debug_level;
+extern int ipoib_debug_level;
 
 #define ipoib_dbg(priv, format, arg...)			\
 	do {					        \
-		if (debug_level > 0)			\
+		if (ipoib_debug_level > 0)			\
 			ipoib_printk(KERN_DEBUG, priv, format , ## arg); \
 	} while (0)
 #define ipoib_dbg_mcast(priv, format, arg...)		\


From mst at mellanox.co.il  Tue Feb 22 13:23:14 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 23:23:14 +0200
Subject: [openib-general] Re: Re: [PATCH][SDP][14/22] Whitespace clean up
	in sdp_proto.h
In-Reply-To: <1109107323.24909.55.camel@duffman>
References: <1109107323.24909.55.camel@duffman>
Message-ID: <20050222212314.GA10729@mellanox.co.il>

Quoting r. Tom Duffy <tduffy at sun.com>:
> Subject: Re: [openib-general] Re: Re: [PATCH][SDP][14/22] Whitespace clean up in sdp_proto.h
> 
> On Tue, 2005-02-22 at 22:32 +0200, Michael S. Tsirkin wrote:
> > Good. By the way, do you plan to keep the convention of writing
> > 0 == foo?
> > 
> > I personally dislike it (and Linux expressed the same santiment at
> > some point), this practice unnecessary with gcc
> > which warns if you write if (foo = 0) accidentally, and
> > kernel code itself never does it, but some in-kernel drivers 
> > do it ... its easy to fix more or less automatically.
> 
> I don't mind that one, but I don't really care.
> 
> How about the SDP_CHECK_NULL macro?  It is used in most (but not all)
> places in the SDP code.  Does turning this checking off in non-debug
> builds really speed things up a lot?
> 
> -tduffy
> 

I dont expect it to give a noticeable speedup, but neither do your
whitespace cleanups :)

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 13:24:29 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 23:24:29 +0200
Subject: [openib-general] Re: [PATCH] Rename debug_level to ipoib_debug_level
In-Reply-To: <52acpwcmpb.fsf@topspin.com>
References: <52acpwcmpb.fsf@topspin.com>
Message-ID: <20050222212429.GB10729@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: [PATCH] Rename debug_level to ipoib_debug_level
> 
> This renames the variable "debug_level" to "ipoib_debug_level" in the
> IPoIB module to avoid name clashes if IPoIB is built into the kernel.
> SDP has the same variable name so this fixes "allyesconfig" for now.
> module_param_named() is used to prevent the user-variable parameter
> name from changing.
> 
> Libor, on general principles SDP should probably make the same change
> to avoid polluting the namespace with generic names.
> 
>  - R.
> 

Yep, that's fixed in a patch I recently posted.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Tue Feb 22 13:27:02 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 22 Feb 2005 23:27:02 +0200
Subject: [openib-general] Re: Re: Re: [PATCH][SDP][14/22] Whitespace clean
	up in sdp_proto.h
In-Reply-To: <20050222212314.GA10729@mellanox.co.il>
References: <1109107323.24909.55.camel@duffman>
	<20050222212314.GA10729@mellanox.co.il>
Message-ID: <20050222212702.GC10729@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: Re: Re: Re: [PATCH][SDP][14/22] Whitespace clean up in sdp_proto.h
> 
> Quoting r. Tom Duffy <tduffy at sun.com>:
> > Subject: Re: [openib-general] Re: Re: [PATCH][SDP][14/22] Whitespace clean up in sdp_proto.h
> > 
> > On Tue, 2005-02-22 at 22:32 +0200, Michael S. Tsirkin wrote:
> > > Good. By the way, do you plan to keep the convention of writing
> > > 0 == foo?
> > > 
> > > I personally dislike it (and Linux expressed the same santiment at
> > > some point), this practice unnecessary with gcc
> > > which warns if you write if (foo = 0) accidentally, and
> > > kernel code itself never does it, but some in-kernel drivers 
> > > do it ... its easy to fix more or less automatically.
> > 
> > I don't mind that one, but I don't really care.
> > 
> > How about the SDP_CHECK_NULL macro?  It is used in most (but not all)
> > places in the SDP code.  Does turning this checking off in non-debug
> > builds really speed things up a lot?
> > 
> > -tduffy
> > 
> 
> I dont expect it to give a noticeable speedup, but neither do your
> whitespace cleanups :)

To clarify: please dont replace CHECK macros, with if, rather
we shall remove them altogether from the code.
As Libor pointed out elsewhere, this has a nice effect of
turning many int functions into void, documenting in a cleaner
way the fact that we dont really expect them to fail.

-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Tue Feb 22 13:34:40 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 13:34:40 -0800
Subject: [openib-general] linked lists in sdp
In-Reply-To: <20050220124515.GH28446@mellanox.co.il>;
	from mst@mellanox.co.il on Sun, Feb 20, 2005 at 02:45:15PM +0200
References: <20050220124515.GH28446@mellanox.co.il>
Message-ID: <20050222133440.E23420@topspin.com>

On Sun, Feb 20, 2005 at 02:45:15PM +0200, Michael S. Tsirkin wrote:
> Hi!
> sdp_advt seems to re-implement linked list.
> Why is not the standard linked list used?

  There is some redundancy between the different lists even within
the sdp code base. There are three objects, iocb, buff, and advt
which each has it's own list. Plus a generic descriptor list which
can take a mix of the different objects, the first five fields of
each object are identical. All the lists are double linked lists,
which are accessed both at the head and tail.

  I can add it to the TODO to replace some/all of these with the
standard linked list, or at least consolidate some more.

-Libor


From tduffy at sun.com  Tue Feb 22 13:36:07 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 22 Feb 2005 13:36:07 -0800
Subject: [openib-general] Re: Re: Re: [PATCH][SDP][14/22] Whitespace
	clean	up in sdp_proto.h
In-Reply-To: <20050222212702.GC10729@mellanox.co.il>
References: <1109107323.24909.55.camel@duffman>
	<20050222212314.GA10729@mellanox.co.il>
	<20050222212702.GC10729@mellanox.co.il>
Message-ID: <1109108167.24909.58.camel@duffman>

On Tue, 2005-02-22 at 23:27 +0200, Michael S. Tsirkin wrote:
> To clarify: please dont replace CHECK macros, with if, rather
> we shall remove them altogether from the code.
> As Libor pointed out elsewhere, this has a nice effect of
> turning many int functions into void, documenting in a cleaner
> way the fact that we dont really expect them to fail.

Oh, sorry.  I seem to have missed the previous conversation about this.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050222/f618c1ab/attachment.sig>

From libor at topspin.com  Tue Feb 22 13:48:20 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 13:48:20 -0800
Subject: [openib-general] Re: RFC on SDP checkin
In-Reply-To: <20050221181257.GA16314@mellanox.co.il>;
	from mst@mellanox.co.il on Mon, Feb 21, 2005 at 08:12:57PM +0200
References: <20050204112115.A1592@topspin.com>
	<1107545598.5982.34.camel@duffman>
	<20050204114207.B1592@topspin.com>
	<1107552392.2435.6.camel@duffman>
	<20050210190904.C11665@topspin.com>
	<20050217140359.GT28446@mellanox.co.il>
	<20050217164150.F7121@topspin.com>
	<20050221181257.GA16314@mellanox.co.il>
Message-ID: <20050222134820.F23420@topspin.com>

On Mon, Feb 21, 2005 at 08:12:57PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > On Thu, Feb 17, 2005 at 04:03:59PM +0200, Michael S. Tsirkin wrote:
> > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > Possible Issues
> > > > 
> > > >  - Memory locking for AIO requires a call to do_mlock() which is not a
> > > >    kernel exported function, the method for calling the function is not
> > > >    standard.
> > > 
> > > Libor, in my eyes this is the biggest issue with this gen2 sdp code.
> > > In short, I dont think using do_mlock is a good idea.
> > 
> >   I do agree that this is one of the more, if not most, problematic area
> > of the code, and I like your alternative suggestion below.
> 
> BTW, just noted that sdp_iocb.c gets current->mm directly, and keeps
> this pointer cashed instead of calling get_task_mm each time.
> Thus it seems a race condition may occur if the task
> is exiting while the cashed value is used.

  Actually, the AIO code itself, outside of SDP, is incrementing the 
mm reference count for each submitted IO which has a one-to-one mapping
with sdpc_iocb instances. We cannot increment the reference count using
get_task_mm() otherwise the AIO code will run into problems. The 
destruction of the mm, when the reference count is decremented, triggers
the AIO cleanup code, which will not complete until all outstanding AIOs
have been completed or canceled. If we were to increment the mm reference
count using get_task_mm() then a userspace process which has outstanding
AIOs and is terminated would not attempt to destroy it's mm, which would
not cleanup the oustanding AIOs, which would then hang the process.

> I am not fixing it since we agreed we are changing this to get_user_pages
> anyway.

  OK.

-Libor

To: Tom Duffy <tduffy at sun.com>
Subject: Re: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
Message-ID: <20050222141603.G23420 at topspin.com>
References: <11087738623823 at sun.com> <20050222122058.A23420 at topspin.com>
	<1109105820.24909.48.camel at duffman>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <1109105820.24909.48.camel at duffman>;
	from tduffy at sun.com on Tue, Feb 22, 2005 at 12:57:00PM -0800
X-OriginalArrivalTime: 22 Feb 2005 22:16:03.0362 (UTC)
	FILETIME=[18B3D020:01C5192C]
X-Spam-Checker-Version: SpamAssassin 2.64 (2004-01-11) on openib.ca.sandia.gov
X-Spam-Level: 
X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.64
Cc: openib-general at openib.org
X-BeenThere: openib-general at openib.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: OpenIB General Mailing List <openib-general.openib.org>
List-Unsubscribe: <http://openib.org/mailman/listinfo/openib-general>,
	<mailto:openib-general-request at openib.org?subject=unsubscribe>
List-Archive: <http://openib.org/pipermail/openib-general>
List-Post: <mailto:openib-general at openib.org>
List-Help: <mailto:openib-general-request at openib.org?subject=help>
List-Subscribe: <http://openib.org/mailman/listinfo/openib-general>,
	<mailto:openib-general-request at openib.org?subject=subscribe>
X-List-Received-Date: Tue, 22 Feb 2005 22:18:29 -0000
Status: O

On Tue, Feb 22, 2005 at 12:57:00PM -0800, Tom Duffy wrote:
> On Tue, 2005-02-22 at 12:20 -0800, Libor Michalek wrote:
> >   Thanks Tom, I've applied and commited the patches.
> 
> Did you choose not to apply these changes, or was this a simple
> oversight?

  Yup, it was an oversight. I had to hand apply some of the patches
that failed, becasue of the connection lock name change patch I had
made. Thanks for catching it, I've checked them in now.

-Libor

> Index: sdp_inet.c
> ===================================================================
> --- sdp_inet.c	(revision 1865)
> +++ sdp_inet.c	(working copy)
> @@ -410,7 +410,6 @@ static int _sdp_inet_release(struct sock
>  
>  			while (0 < timeout &&
>  			       0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
> -
>  				sdp_conn_unlock(conn);
>  				timeout = schedule_timeout(timeout);
>  				sdp_conn_lock(conn);
> @@ -522,7 +521,6 @@ static int _sdp_inet_bind(struct socket 
>  
>  	result = sdp_inet_port_get(conn, bind_port);
>  	if (0 > result) {
> -
>  		sdp_dbg_warn(conn, "Error <%d> getting port during bind",
>  			     result);
>  
> @@ -1174,7 +1172,6 @@ static int _sdp_inet_ioctl(struct socket
>  		sdp_conn_lock(conn);
>  
>  		if (SDP_SOCK_ST_LISTEN != conn->istate) {
> -
>  			value = conn->send_qud;
>  			result = put_user(value, (int __user *) arg);
>  		} else
> Index: sdp_event.c
> ===================================================================
> --- sdp_event.c	(revision 1865)
> +++ sdp_event.c	(working copy)
> @@ -512,44 +512,32 @@ int sdp_cm_event_handler(struct ib_cm_id
>  	 * lookup the connection, on a REQ_RECV the sk will be empty.
>  	 */
>  	conn = sdp_conn_table_lookup(hashent);
> -	if (NULL != conn) {
> -
> +	if (NULL != conn)
>  		sdp_conn_lock(conn);
> -	}
> -	else {
> -
> -		if (IB_CM_REQ_RCVD != cm_id->state) {
> -
> +	else
> +		if (IB_CM_REQ_RCVD != cm_id->state)
>  			sdp_dbg_warn(NULL, 
>  				     "No conn <%d> CM state <%d> event <%d>",
>  				     hashent, cm_id->state, event->event);
> -		}
> -	}
>  
>  	switch (cm_id->state) {
>  	case IB_CM_REQ_RCVD:
> -
>  		result = sdp_cm_req_handler(cm_id, event);
>  		break;
>  	case IB_CM_REP_RCVD:
> -
>  		result = sdp_cm_rep_handler(cm_id, event, conn);
>  		break;
>  	case IB_CM_IDLE:
> -
>  		result = _sdp_cm_idle(cm_id, event, conn);
>  		break;
>  	case IB_CM_ESTABLISHED:
> -
>  		result = _sdp_cm_established(cm_id, event, conn);
>  		break;
>  	case IB_CM_DREQ_RCVD:
>  	case IB_CM_TIMEWAIT:
> -
>  		result = _sdp_cm_timewait(cm_id, event, conn);
>  		break;
>  	default:
> -
>  		sdp_dbg_warn(conn, "Unexpected CM state <%d>", cm_id->state);
>  		result = -EINVAL;
>  	}


From roland at topspin.com  Tue Feb 22 14:38:00 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 14:38:00 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
In-Reply-To: <20050222141603.G23420@topspin.com> (Libor Michalek's message
	of "Tue, 22 Feb 2005 14:16:03 -0800")
References: <11087738623823@sun.com> <20050222122058.A23420@topspin.com>
	<1109105820.24909.48.camel@duffman>
	<20050222141603.G23420@topspin.com>
Message-ID: <521xb8cjav.fsf@topspin.com>

Speaking of missing patches, I still see a lot of warnings from SDP on
64-bit archs because of "%ll" printk formats being used to print u64
values inside of sdp_debug_data statements.  Is there a patch floating
around for this or do you want me to write one?

 - R.



From libor at topspin.com  Tue Feb 22 14:52:36 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 14:52:36 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
In-Reply-To: <521xb8cjav.fsf@topspin.com>;
	from roland@topspin.com on Tue, Feb 22, 2005 at 02:38:00PM -0800
References: <11087738623823@sun.com> <20050222122058.A23420@topspin.com>
	<1109105820.24909.48.camel@duffman>
	<20050222141603.G23420@topspin.com> <521xb8cjav.fsf@topspin.com>
Message-ID: <20050222145236.H23420@topspin.com>

On Tue, Feb 22, 2005 at 02:38:00PM -0800, Roland Dreier wrote:
> Speaking of missing patches, I still see a lot of warnings from SDP on
> 64-bit archs because of "%ll" printk formats being used to print u64
> values inside of sdp_debug_data statements.  Is there a patch floating
> around for this or do you want me to write one?

  No, there's no patch for this yet, I'd be happy to take one.

-Libor


From roland at topspin.com  Tue Feb 22 15:04:10 2005
From: roland at topspin.com (Roland Dreier)
Date: Tue, 22 Feb 2005 15:04:10 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
In-Reply-To: <20050222145236.H23420@topspin.com> (Libor Michalek's message
	of "Tue, 22 Feb 2005 14:52:36 -0800")
References: <11087738623823@sun.com> <20050222122058.A23420@topspin.com>
	<1109105820.24909.48.camel@duffman>
	<20050222141603.G23420@topspin.com> <521xb8cjav.fsf@topspin.com>
	<20050222145236.H23420@topspin.com>
Message-ID: <52wtt0b3it.fsf@topspin.com>

OK, here's the patch.


Cast u64 values to unsigned long long inside sdp_dbg_data() to get rid
of compiler warnings on 64-bit archs when building with
INFINIBAND_SDP_DEBUG_DATA turned on.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: infiniband/ulp/sdp/sdp_write.c
===================================================================
--- infiniband/ulp/sdp/sdp_write.c	(revision 1867)
+++ infiniband/ulp/sdp/sdp_write.c	(working copy)
@@ -74,7 +74,7 @@ int sdp_event_write(struct sdp_opt *conn
 	}
 
 	sdp_dbg_data(conn, "Write complete <%llu> of <%u> bytes.",
-		     comp->wr_id, comp->byte_len);
+		     (unsigned long long) comp->wr_id, comp->byte_len);
 	/*
 	 * Four basic scenarios:
 	 *
Index: infiniband/ulp/sdp/sdp_rcvd.c
===================================================================
--- infiniband/ulp/sdp/sdp_rcvd.c	(revision 1867)
+++ infiniband/ulp/sdp/sdp_rcvd.c	(working copy)
@@ -729,7 +729,8 @@ static int _sdp_rcvd_snk_avail(struct sd
 	_sdp_msg_net_to_cpu_snkah(snkah);
 
 	sdp_dbg_data(conn, "SnkAvail received. <%d:%d:%016llx> mode <%d>",
-		     snkah->size, snkah->r_key, snkah->addr, conn->send_mode);
+		     snkah->size, snkah->r_key, (unsigned long long) snkah->addr,
+		     conn->send_mode);
 	/*
 	 * check our send mode, and make sure parameters are within reason.
 	 */
@@ -873,8 +874,8 @@ static int _sdp_rcvd_src_avail(struct sd
 	size = buff->tail - buff->data;
 
 	sdp_dbg_data(conn, "SrcAvail received. <%d:%d:%d:%016llx> mode <%d>",
-		     srcah->size, srcah->r_key, size, srcah->addr,
-		     conn->recv_mode);
+		     srcah->size, srcah->r_key, size,
+		     (unsigned long long) srcah->addr, conn->recv_mode);
 
 	if (0 < conn->snk_sent) {
 		/*
Index: infiniband/ulp/sdp/sdp_read.c
===================================================================
--- infiniband/ulp/sdp/sdp_read.c	(revision 1867)
+++ infiniband/ulp/sdp/sdp_read.c	(working copy)
@@ -142,7 +142,7 @@ int sdp_event_read(struct sdp_opt *conn,
 	}
 
 	sdp_dbg_data(conn, "Read complete <%llu> of <%u> bytes.",
-		     comp->wr_id, comp->byte_len);
+		     (unsigned long long) comp->wr_id, comp->byte_len);
 	/*
 	 * update queue depth
 	 */
Index: infiniband/ulp/sdp/sdp_send.c
===================================================================
--- infiniband/ulp/sdp/sdp_send.c	(revision 1867)
+++ infiniband/ulp/sdp/sdp_send.c	(working copy)
@@ -372,7 +372,7 @@ static int _sdp_send_data_buff_snk(struc
 	advt->post += (buff->tail - buff->data);
 
 	sdp_dbg_data(conn, "POST Write BUFF wrid <%llu> bytes <%u:%d>.",
-		     buff->wrid, (unsigned)(buff->tail - buff->data),
+		     (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data),
 		     advt->size);
 	/*
 	 * dequeue if needed and the queue buffer
@@ -502,9 +502,8 @@ static int _sdp_send_data_iocb_snk(struc
 		advt->addr += zcopy;
 		advt->post += zcopy;
 
-		sdp_dbg_data(conn, 
-			     "POST Write IOCB wrid <%llu> bytes <%u:%d:%d>.",
-			     iocb->wrid, zcopy, iocb->len, advt->size);
+		sdp_dbg_data(conn, "POST Write IOCB wrid <%llu> bytes <%u:%d:%d>.",
+			     (unsigned long long) iocb->wrid, zcopy, iocb->len, advt->size);
 		/*
 		 * update send queue depth
 		 */
Index: infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- infiniband/ulp/sdp/sdp_recv.c	(revision 1867)
+++ infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -82,7 +82,7 @@ static int _sdp_post_recv_buff(struct sd
 	}
 
 	sdp_dbg_data(conn, "POST RECV BUFF wrid <%llu> of <%u> bytes.",
-		     buff->wrid, (unsigned)(buff->tail - buff->data));
+		     (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data));
 	/*
 	 * post recv
 	 */
@@ -203,7 +203,7 @@ static int _sdp_post_rdma_buff(struct sd
 	}
 
 	sdp_dbg_data(conn, "POST READ BUFF wrid <%llu> of <%u> bytes.",
-		     buff->wrid, (unsigned)(buff->tail - buff->data));
+		     (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data));
 	/*
 	 * update send queue depth
 	 */
@@ -353,7 +353,7 @@ static int _sdp_post_rdma_iocb_src(struc
 	}
 
 	sdp_dbg_data(conn, "POST READ IOCB wrid <%llu> bytes <%u:%d:%d>.",
-		 iocb->wrid, zcopy, iocb->len, advt->size);
+		     (unsigned long long) iocb->wrid, zcopy, iocb->len, advt->size);
 	/*
 	 * update send queue depth
 	 */


From tduffy at sun.com  Tue Feb 22 15:09:01 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 22 Feb 2005 15:09:01 -0800
Subject: [openib-general] Re: [PATCH][SDP][14/22] Whitespace cleanup in
	sdp_proto.h
In-Reply-To: <20050220000536.GB8268@mellanox.co.il>
References: <1108773914773@sun.com> <11087739241469@sun.com>
	<11087739341677@sun.com> <11087739441194@sun.com>
	<11087739553583@sun.com>
	<11087739651393@sun.com> <11087739751674@sun.com>
	<11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
	<20050220000536.GB8268@mellanox.co.il>
Message-ID: <1109113741.24909.64.camel@duffman>

On Sun, 2005-02-20 at 02:05 +0200, Michael S. Tsirkin wrote:
> That does not seem to be the case for __sdp_inet_write_space, or for
> other stuff in sdp_proto.h, so I think these functions shall have proper names
> starting with an alphanumeric.

If functions (or structs) are static, does it make sense to prefix them
with sdp_ either?

For that matter, what about the .c and .h files?  Shouldn't the sdp_ be
removed on those as well?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050222/39d5fd64/attachment.sig>

From mshefty at ichips.intel.com  Tue Feb 22 16:17:11 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 22 Feb 2005 16:17:11 -0800
Subject: [openib-general] [PATCH] [CM] add reject handling for timeouts
Message-ID: <20050222161711.1c38fd00.mshefty@ichips.intel.com>

The following patch adds reject handling for reject code 4, timeout.
The update requires searching for the local cm_id based on the remote
CA GUID and remote communication ID.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1864)
+++ infiniband/core/cm.c	(working copy)
@@ -400,30 +400,28 @@ static struct cm_timewait_info * cm_inse
 	return NULL;
 }
 
-/*
- * Call will be needed when implementing REJ handling.
-static struct cm_id_private * cm_find_id_by_remote_id(u64 remote_ca_guid,
-						      u32 remote_id)
+static struct cm_timewait_info * cm_find_remote_id(u64 remote_ca_guid,
+						   u32 remote_id)
 {
 	struct rb_node *node = cm.remote_id_table.rb_node;
-	struct cm_id_private *cm_id_priv;
+	struct cm_timewait_info *timewait_info;
 
 	while (node) {
-		cm_id_priv = rb_entry(node, struct cm_id_private, remote_id_node);
-		if (remote_id < cm_id_priv->id.remote_id)
+		timewait_info = rb_entry(node, struct cm_timewait_info,
+					 remote_id_node);
+		if (remote_id < timewait_info->work.remote_id)
 			node = node->rb_left;
-		else if (remote_id > cm_id_priv->id.remote_id)
+		else if (remote_id > timewait_info->work.remote_id)
 			node = node->rb_right;
-		else if (remote_ca_guid < cm_id_priv->remote_ca_guid)
+		else if (remote_ca_guid < timewait_info->remote_ca_guid)
 			node = node->rb_left;
-		else if (remote_ca_guid > cm_id_priv->remote_ca_guid)
+		else if (remote_ca_guid > timewait_info->remote_ca_guid)
 			node = node->rb_right;
 		else
-			return cm_id_priv;
+			return timewait_info;
 	}
 	return NULL;
 }
-*/
 
 static struct cm_timewait_info * cm_insert_remote_qpn(struct cm_timewait_info
 						      *timewait_info)
@@ -1809,13 +1807,34 @@ static void cm_format_rej_event(struct c
 
 static struct cm_id_private * cm_acquire_rejected_id(struct cm_rej_msg *rej_msg)
 {
-	u32 local_id, remote_id;
+	struct cm_timewait_info *timewait_info;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	u32 remote_id;
 
-	/* todo: lookup by remote CA for rejects due to timeouts */
-	local_id = rej_msg->remote_comm_id;
 	remote_id = rej_msg->local_comm_id;
 
-	return cm_acquire_id(local_id, remote_id);
+	if (rej_msg->reason == IB_CM_REJ_TIMEOUT) {
+		spin_lock_irqsave(&cm.lock, flags);
+		timewait_info = cm_find_remote_id( *((u64 *) rej_msg->ari),
+						  remote_id);
+		if (!timewait_info) {
+			spin_unlock_irqrestore(&cm.lock, flags);
+			return NULL;
+		}
+		cm_id_priv = idr_find(&cm.local_id_table,
+				      (int) timewait_info->work.local_id);
+		if (cm_id_priv) {
+			if (cm_id_priv->id.remote_id == remote_id)
+				atomic_inc(&cm_id_priv->refcount);
+			else
+				cm_id_priv = NULL;
+		}
+		spin_unlock_irqrestore(&cm.lock, flags);
+	} else
+		cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, remote_id);
+
+	return cm_id_priv;
 }
 
 static int cm_rej_handler(struct cm_work *work)


From mshefty at ichips.intel.com  Tue Feb 22 16:42:10 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 22 Feb 2005 16:42:10 -0800
Subject: [openib-general] [PATCH] [CM] handle stale connection reject
Message-ID: <20050222164210.012a3a2d.mshefty@ichips.intel.com>

This patch properly places a local cm_id into the timewait state after
receiving a stale connection reject message.  It also fixes a missing
newline character at the end of the file that somehow got mangled with
my last checkin.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1868)
+++ infiniband/core/cm.c	(working copy)
@@ -1862,7 +1862,10 @@ static int cm_rej_handler(struct cm_work
 	case IB_CM_MRA_REQ_SENT:
 	case IB_CM_MRA_REQ_RCVD:
 	case IB_CM_MRA_REP_RCVD:
-		cm_reset_to_idle(cm_id_priv);
+		if (rej_msg->reason == IB_CM_REJ_STALE_CONN)
+			cm_enter_timewait(cm_id_priv);
+		else
+			cm_reset_to_idle(cm_id_priv);
 		break;
 	case IB_CM_DREQ_SENT:
 		wr_id = (unsigned long) cm_id_priv->msg;
@@ -3086,4 +3089,4 @@ static void __exit ib_cm_cleanup(void)
 
 module_init(ib_cm_init);
 module_exit(ib_cm_cleanup);
-\ No newline at end of file
+


From mshefty at ichips.intel.com  Tue Feb 22 16:17:29 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 22 Feb 2005 16:17:29 -0800
Subject: [openib-general] CM reject handling
Message-ID: <421BCB99.4040105@ichips.intel.com>

With the latest patch, most of the reject receive handling in the CM 
should be in place.  Currently, the handling simply involves passing 
the message up to the user and transitioning the cm_id to the correct 
state.  Specifically:

* code 6 - invalid comm id -- Possibly indicates a remote system 
reboot?  The user will receive the REJ failure to whatever message it 
was trying to send.

* code 7 - invalid comm instance -- Seems like it could only happen 
when sending a DREQ.  The current code should handle this as a normal 
part of receiving a REJ in the DREQ sent state.

* code 10 - stale connection -- The recipient of the REJ enters the 
timewait state.  There's nothing to handle the cm_id on the side of 
that detected the stale connection.

* code 24 - port/CM redirection -- Reject is passed to the user.

* code 25 - port redirection -- Currently just passed to the user.  I 
thought about adding in a cm_path field to the CM messages, but 
deferred this for now.

* code 30 - duplicate local comm id -- Possibly indicates a local 
system reboot?  Not sure what else can be done beyond passing rejection 
to the user.

* code 31 - unsupported class version -- I didn't look at what it would 
take to handle multiple CM versions, so I'm not sure if doing this 
would require API changes or not.

Comments?  My short-term plan is to implement sending CM issued reject 
messages for things like stale connections and invalid IDs.

- Sean


From libor at topspin.com  Tue Feb 22 16:50:41 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 16:50:41 -0800
Subject: [openib-general] [PATCH] prepend sdp_ to global names
In-Reply-To: <20050221133200.GI14855@mellanox.co.il>;
	from mst@mellanox.co.il on Mon, Feb 21, 2005 at 03:32:00PM +0200
References: <20050221133200.GI14855@mellanox.co.il>
Message-ID: <20050222165041.I23420@topspin.com>

On Mon, Feb 21, 2005 at 03:32:00PM +0200, Michael S. Tsirkin wrote:
> Some extern (non-static) variables did not start with sdp_ or anything
> like that. Here's a fix.

  Thanks, applied and commited. I did change a few lines to keep them
under 80 columns wide.

-Libor


From libor at topspin.com  Tue Feb 22 18:03:52 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 18:03:52 -0800
Subject: [openib-general] Re: Re: SDP_CONN_LOCK
In-Reply-To: <20050220131640.GI28446@mellanox.co.il>;
	from mst@mellanox.co.il on Sun, Feb 20, 2005 at 03:16:40PM +0200
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
	<20050220131640.GI28446@mellanox.co.il>
Message-ID: <20050222180352.J23420@topspin.com>

On Sun, Feb 20, 2005 at 03:16:40PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> >   And here I would have thought SDP_CONN_HOLD as a macro would have
> > bugged people more. :)
> > 
> > -Libor
> > 
> 
> Regarding conn_hold/conn_put (macro or not):
> 1. conn_put is sometimes called while socket is locked (sdp_cm_connect),
>    sometimes not (sdp_ip2pr_path_complete). What gives?
>    Is it OK to destroy the socket while its locked? Seems strange ...

  conn_put() should not be called from inside the lock as you note. The
only reason that this has not caused a problem in this instance is that
the conn_hold() in that function, and conn_put() on an error, are always
the second reference on the connection.

  There are essentially four references which are being tracked by the
connection:

  - Socket interface. 
      Increment on socket creation. Either socket() system call or when
      a passive connect request is added to a listen sockets accept queue.
      Decrement on socket release. When close() is called on either the
      socket or a listen socket, in which case the accept queue is flushed.
  - CM interface.
      Increment on REQ send or receive.
      Decrement on CM IDLE event or right before an error is returned
      from the CM event handler
  - Address resolution.
      Increment on request.
      Decrement on completion.
  - Event execution. (CQ or CM)
      Increment using sdp_conn_table_lookup()
      Decrement before leaving event handler.

> 2. When conn_put is called *outside* the lock, what prevents conn_hold
>    from being called while conn_put is in progress, thus hold will be
>    acting in parallel with destroy?

  There should only ever be contention in the case where the request
for the reference count is comming from sdp_conn_table_lookup(), but
you are right there is a small race during conn_put and before the
connection is removed from the table. 

-Libor





From libor at topspin.com  Tue Feb 22 18:59:34 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 18:59:34 -0800
Subject: [openib-general] SDP: BUG 2034 workaround
In-Reply-To: <20050222101445.GK23837@mellanox.co.il>;
	from mst@mellanox.co.il on Tue, Feb 22, 2005 at 12:14:45PM +0200
References: <20050222101445.GK23837@mellanox.co.il>
Message-ID: <20050222185934.K23420@topspin.com>

On Tue, Feb 22, 2005 at 12:14:45PM +0200, Michael S. Tsirkin wrote:
> 
> sdp_inet.c, inside _sdp_inet_listen, we have:
> 
> #if 0                           /* BUG 2034 workaround. */
>         conn->backlog_max = backlog;
> #else
>         conn->backlog_max = 1024;
> #endif
> 
> What gives? what would be the proper fix as opposed to a work-around?

  The Linux TCP listen uses two seperate values to control the listner
backlog, so I ignored the backlog parameter to get closer to the default
Linux behavior, until a really solution is devised.
 
  Basically TCP uses a two stage backlog to defend against SYN attacks.
When a SYN is received a small amount of state is kept until the full
handshake is completed, at which point a full socket is created and
queued onto the listen sockets accept queue. The second stage uses the
listen() backlog parameter to manage the accept queue. The first stage
queue size is managed using a sysctl, (net.pv4.tcp_max_syn_backlog)
which on a lot of systems defaults to 1024.

  SDP on the other hand creates the full socket on the REQ request, and
places it directly into the listen sockets accept queue. So the full
depth of the queue is governed by the listen backlog parameter. Worse
still the socket layer listen, above TCP and SDP, limits the listen()
backlog that is passed to the protocol to 128, and only recently did this
become adjustable using net.core.somaxconn. 

   Using just the backlog parameter to manage this queue, results in a
few programs that use a high connection volume to get rejected connection,
casued by a full backlog queue, even when upped to the full 128 that the
socket layer allows by default.


-Libor



From libor at topspin.com  Tue Feb 22 19:12:48 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 22 Feb 2005 19:12:48 -0800
Subject: [openib-general] Re: [PATCH][SDP] replace __inline__ with plain
	inline
In-Reply-To: <1109106621.24909.51.camel@duffman>;
	from tduffy@sun.com on Tue, Feb 22, 2005 at 01:10:21PM -0800
References: <11087739341677@sun.com> <11087739441194@sun.com>
	<11087739553583@sun.com> <11087739651393@sun.com>
	<11087739751674@sun.com> <11087739852191@sun.com>
	<11087739963627@sun.com> <11087740063185@sun.com>
	<20050220000536.GB8268@mellanox.co.il>
	<1109106621.24909.51.camel@duffman>
Message-ID: <20050222191248.L23420@topspin.com>

On Tue, Feb 22, 2005 at 01:10:21PM -0800, Tom Duffy wrote:
> On Sun, 2005-02-20 at 02:05 +0200, Michael S. Tsirkin wrote:
> > I think we also should replace __inline__ with inline everywhere.

  Thanks, I've commited the change.

-Libor

> Index: sdp_queue.h
> ===================================================================
> --- sdp_queue.h	(revision 1865)
> +++ sdp_queue.h	(working copy)
> @@ -73,7 +73,7 @@ struct sdpc_desc_q {
>  /*
>   * __sdp_desc_q_size - return the number of elements in the table
>   */
> -static __inline__ int __sdp_desc_q_size(struct sdpc_desc_q *table)
> +static inline int __sdp_desc_q_size(struct sdpc_desc_q *table)
>  {
>  	return table->size;
>  } /* __sdp_desc_q_size */
> @@ -81,7 +81,7 @@ static __inline__ int __sdp_desc_q_size(
>  /*
>   * __sdp_desc_q_member - return non-zero if element is in a table
>   */
> -static __inline__ int __sdp_desc_q_member(struct sdpc_desc *element)
> +static inline int __sdp_desc_q_member(struct sdpc_desc *element)
>  {
>  	return ((NULL == element->table) ? 0 : 1);
>  } /* __sdp_desc_q_member */
> Index: sdp_proto.h
> ===================================================================
> --- sdp_proto.h	(revision 1865)
> +++ sdp_proto.h	(working copy)
> @@ -569,7 +569,7 @@ do {                                    
>  /*
>   * __sdp_inet_write_space - writable space on send side
>   */
> -static __inline__ int __sdp_inet_write_space(struct sdp_opt *conn, int urg)
> +static inline int __sdp_inet_write_space(struct sdp_opt *conn, int urg)
>  {
>  	int size;
>  
> @@ -594,7 +594,7 @@ static __inline__ int __sdp_inet_write_s
>  /*
>   * __sdp_inet_writable - return non-zero if socket is writable
>   */
> -static __inline__ int __sdp_inet_writable(struct sdp_opt *conn)
> +static inline int __sdp_inet_writable(struct sdp_opt *conn)
>  {
>  	SDP_CHECK_NULL(conn, -EINVAL);
>  
> @@ -608,7 +608,7 @@ static __inline__ int __sdp_inet_writabl
>  /*
>   * __sdp_conn_stat_dump - dump stats to the log
>   */
> -static __inline__ int __sdp_conn_stat_dump(struct sdp_opt *conn)
> +static inline int __sdp_conn_stat_dump(struct sdp_opt *conn)
>  {
>  #ifdef _SDP_CONN_STATS_REC
>  	int counter;
> @@ -631,7 +631,7 @@ static __inline__ int __sdp_conn_stat_du
>  /*
>   * __sdp_conn_state_dump - dump state information to the log
>   */
> -static __inline__ int __sdp_conn_state_dump(struct sdp_opt *conn)
> +static inline int __sdp_conn_state_dump(struct sdp_opt *conn)
>  {
>  #ifdef _SDP_CONN_STATE_REC
>  	int counter;
> Index: sdp_msgs.h
> ===================================================================
> --- sdp_msgs.h	(revision 1865)
> +++ sdp_msgs.h	(working copy)
> @@ -278,7 +278,7 @@ struct msg_hdr_sch {
>  /*
>   * __sdp_msg_swap_bsdh - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
> +static inline void __sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
>  {
>  	header->recv_bufs = cpu_to_be16(header->recv_bufs);
>  	header->size = cpu_to_be32(header->size);
> @@ -289,7 +289,7 @@ static __inline__ void __sdp_msg_swap_bs
>  /*
>   * __sdp_msg_swap_hh - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_hh(struct msg_hdr_hh *header)
> +static inline void __sdp_msg_swap_hh(struct msg_hdr_hh *header)
>  {
>  	header->r_rcv_size = cpu_to_be32(header->r_rcv_size);
>  	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
> @@ -307,7 +307,7 @@ static __inline__ void __sdp_msg_swap_hh
>  /*
>   * __sdp_msg_swap_hah - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
> +static inline void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
>  {
>  	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
>  } /* __sdp_msg_swap_hah */
> @@ -315,7 +315,7 @@ static __inline__ void __sdp_msg_swap_ha
>  /*
>   * __sdp_msg_swap_srcah - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
> +static inline void __sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
>  {
>  	header->size = cpu_to_be32(header->size);
>  #ifdef _SDP_MS_APRIL_ERROR_COMPAT
> @@ -330,7 +330,7 @@ static __inline__ void __sdp_msg_swap_sr
>  /*
>   * __sdp_msg_swap_snkah - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
> +static inline void __sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
>  {
>  	header->size = cpu_to_be32(header->size);
>  	header->r_key = cpu_to_be32(header->r_key);
> @@ -341,7 +341,7 @@ static __inline__ void __sdp_msg_swap_sn
>  /*
>   * __sdp_msg_swap_rwch - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
> +static inline void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
>  {
>  	header->size = cpu_to_be32(header->size);
>  } /* __sdp_msg_swap_rwch */
> @@ -349,7 +349,7 @@ static __inline__ void __sdp_msg_swap_rw
>  /*
>   * __sdp_msg_swap_rrch - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
> +static inline void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
>  {
>  	header->size = cpu_to_be32(header->size);
>  } /* __sdp_msg_swap_rrch */
> @@ -357,7 +357,7 @@ static __inline__ void __sdp_msg_swap_rr
>  /*
>   * __sdp_msg_swap_mch - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
> +static inline void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
>  {
>  	return;
>  } /* __sdp_msg_swap_ */
> @@ -365,7 +365,7 @@ static __inline__ void __sdp_msg_swap_mc
>  /*
>   * __sdp_msg_swap_crbh - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
> +static inline void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
>  {
>  	header->size = cpu_to_be32(header->size);
>  } /* __sdp_msg_swap_crbh */
> @@ -373,7 +373,7 @@ static __inline__ void __sdp_msg_swap_cr
>  /*
>   * __sdp_msg_swap_crbah - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
> +static inline void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
>  {
>  	header->size = cpu_to_be32(header->size);
>  } /* __sdp_msg_swap_crbah */
> @@ -381,7 +381,7 @@ static __inline__ void __sdp_msg_swap_cr
>  /*
>   * __sdp_msg_swap_sch - SDP header endian byte swapping
>   */
> -static __inline__ void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
> +static inline void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
>  {
>  	header->service_id = cpu_to_be64(header->service_id);
>  } /* __sdp_msg_swap_sch */
> Index: sdp_send.c
> ===================================================================
> --- sdp_send.c	(revision 1865)
> +++ sdp_send.c	(working copy)
> @@ -1094,8 +1094,7 @@ done:
>  /*
>   * _sdp_send_data_buff_get - get an appropriate write buffer for send
>   */
> -static __inline__ struct sdpc_buff *_sdp_send_data_buff_get(
> -	struct sdp_opt *conn)
> +static inline struct sdpc_buff *_sdp_send_data_buff_get(struct sdp_opt *conn)
>  {
>  	struct sdpc_buff *buff;
>  
> @@ -1121,10 +1120,10 @@ static __inline__ struct sdpc_buff *_sdp
>  /*
>   * _sdp_send_data_buff_put - place a buffer into the send queue
>   */
> -static __inline__ int _sdp_send_data_buff_put(struct sdp_opt *conn,
> -					      struct sdpc_buff *buff,
> -					      int size,
> -					      int urg)
> +static inline int _sdp_send_data_buff_put(struct sdp_opt *conn,
> +					  struct sdpc_buff *buff,
> +					  int size,
> +					  int urg)
>  {
>  	int result = 0;
>  	int expect;
> Index: sdp_iocb.h
> ===================================================================
> --- sdp_iocb.h	(revision 1865)
> +++ sdp_iocb.h	(working copy)
> @@ -131,7 +131,7 @@ struct sdpc_iocb_q {
>  /*
>   * __sdp_kmap - map a page into kernel space
>   */
> -static __inline__ void *__sdp_kmap(struct page *page)
> +static inline void *__sdp_kmap(struct page *page)
>  {
>  	if (in_atomic() || irqs_disabled())
>  		return kmap_atomic(page, KM_IRQ0);
> @@ -142,7 +142,7 @@ static __inline__ void *__sdp_kmap(struc
>  /*
>   * __sdp_kunmap - unmap a page into kernel space
>   */
> -static __inline__ void __sdp_kunmap(struct page *page)
> +static inline void __sdp_kunmap(struct page *page)
>  {
>  	if (in_atomic() || irqs_disabled())
>  		kunmap_atomic(page, KM_IRQ0);
> Index: sdp_buff.c
> ===================================================================
> --- sdp_buff.c	(revision 1865)
> +++ sdp_buff.c	(working copy)
> @@ -42,12 +42,12 @@ static struct sdpc_buff_root *main_pool 
>  /*
>   * _sdp_buff_q_get - Get a buffer from a specific pool
>   */
> -static __inline__ struct sdpc_buff *_sdp_buff_q_get(struct sdpc_buff_q *pool,
> -						    int fifo,
> -						    int (*test_func)
> -							(struct sdpc_buff *buff,
> -							 void *arg),
> -						    void *usr_arg)
> +static inline struct sdpc_buff *_sdp_buff_q_get(struct sdpc_buff_q *pool,
> +						int fifo,
> +						int (*test_func)
> +						     (struct sdpc_buff *buff,
> +						      void *arg),
> +						void *usr_arg)
>  {
>  	struct sdpc_buff *buff;
>  
> @@ -86,9 +86,9 @@ static __inline__ struct sdpc_buff *_sdp
>  /*
>   * _sdp_buff_q_put - Place a buffer into a specific pool
>   */
> -static __inline__ int _sdp_buff_q_put(struct sdpc_buff_q *pool,
> -				      struct sdpc_buff *buff,
> -				      int fifo)
> +static inline int _sdp_buff_q_put(struct sdpc_buff_q *pool,
> +				  struct sdpc_buff *buff,
> +				  int fifo)
>  {
>  	/* fifo: false == tail, true == head */
>  	SDP_CHECK_NULL(pool, -EINVAL);
> @@ -121,8 +121,8 @@ static __inline__ int _sdp_buff_q_put(st
>  /*
>   * _sdp_buff_q_look - look at a buffer from a specific pool
>   */
> -static __inline__ struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
> -						     int fifo)
> +static inline struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
> +						 int fifo)
>  {
>  	SDP_CHECK_NULL(pool, NULL);
>  
> @@ -135,8 +135,8 @@ static __inline__ struct sdpc_buff *_sdp
>  /*
>   * _sdp_buff_q_remove - remove a specific buffer from a specific pool
>   */
> -static __inline__ int _sdp_buff_q_remove(struct sdpc_buff_q *pool, 
> -					 struct sdpc_buff *buff)
> +static inline int _sdp_buff_q_remove(struct sdpc_buff_q *pool, 
> +				     struct sdpc_buff *buff)
>  {
>  	struct sdpc_buff *prev;
>  	struct sdpc_buff *next;
> @@ -448,10 +448,7 @@ static int _sdp_buff_pool_release(struct
>  /*
>   * _sdp_buff_pool_release_check - check for buffer release from main pool
>   */
> -static __inline__ int _sdp_buff_pool_release_check
> -(
> -	struct sdpc_buff_root *m_pool
> -)
> +static inline int _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
>  {
>  	SDP_CHECK_NULL(m_pool, -EINVAL);
>  	/*
> Index: sdp_queue.c
> ===================================================================
> --- sdp_queue.c	(revision 1865)
> +++ sdp_queue.c	(working copy)
> @@ -79,9 +79,9 @@ static struct sdpc_desc *_sdp_desc_q_get
>  /*
>   * _sdp_desc_q_put - Place an element into a specific table
>   */
> -static __inline__ int _sdp_desc_q_put(struct sdpc_desc_q *table,
> -				      struct sdpc_desc *element, 
> -				      int fifo)
> +static inline int _sdp_desc_q_put(struct sdpc_desc_q *table,
> +				  struct sdpc_desc *element, 
> +				  int fifo)
>  {
>  	/*
>  	 * fifo: false == tail, true == head


From Nitin.Hande at Sun.COM  Tue Feb 22 19:56:25 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Tue, 22 Feb 2005 19:56:25 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1108674754.4761.634.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
	<1108589174.4645.138.camel@localhost.localdomain>
	<1108593217.16278.47.camel@sr1-umpk-01>
	<1108674754.4761.634.camel@localhost.localdomain>
Message-ID: <1109130984.16278.547.camel@sr1-umpk-01>

Hal,


On Thu, 2005-02-17 at 13:12, Hal Rosenstock wrote:
> Hi Nitin,
> 
> On Wed, 2005-02-16 at 17:33, Nitin Hande wrote: 
> > On Wed, 2005-02-16 at 13:26, Hal Rosenstock wrote:
> > > On Wed, 2005-02-16 at 16:08, Nitin Hande wrote:
> > > > Hal,
> > [snip..]
[snip...]
> > > > 
> > > > 
> > 
> > Before the patch the selector was coming 04. Do you reply 84 seeing a
> > particular component mask and otherwise 01 ??(I think not..) 
> 
> I agree that OpenSM responds/should respond the same regardless of the
> component mask in the request.
> 
> I was unaware of OpenSM responding with MTU of 01 until now. I have a
> theory as to this. Any chance I can get the osm logs from a -V run of
> the above ? 
> 
> I also have a simple patch below to try which is just to test the
> theory. This is off the latest version but should be easy to apply to
> any version of osm_sa_mcmember_record.c.
> 
> This is separate from the support for PathRecords with multicast DGID
> and/or DLID. I have the changes for this scoped out and should be able
> to implement by early next week.

So I tried the latest patches and preliminarily things seem to be
working fine. The PathRecord response is successful and so is the MTU
correct. I need to spend some more time looking at MAD and confirm it. I
could configure both interfaces and ping each other this time. Here is
some out on the solaris side:

[root at dongon ~]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
8232 index 1
        inet 127.0.0.1 netmask ff000000 
ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index
28
        inet 192.168.100.105 netmask ffffff00 broadcast 192.168.100.255
        ipib 0:2c:0:16:fe:80:0:0:0:0:0:0:0:2:c9:1:9:76:51:d1 
.....
[root at dongon ~]# ping -s 192.168.100.104
PING 192.168.100.104: 56 data bytes
64 bytes from 192.168.100.104: icmp_seq=0. time=0.590 ms
64 bytes from 192.168.100.104: icmp_seq=1. time=0.434 ms
64 bytes from 192.168.100.104: icmp_seq=2. time=0.365 ms

the other side is a openib interface runing OpenSM.

So after this test above, I try to run snoop on the solaris interface
and get the following error message from the layer below IPoIB:

Feb 22 19:50:25 dongon.SFBay.Sun.COM ibd: [ID 517869 kern.info] NOTICE:
ibd0: HCA GUID 0002c901097651d0 port 1 PKEY ffff Could not get list of
IBA multicast groups

My preliminary assumption is that OpenSm is not returning the list of
multicast groups that the ibd interface has joined. I will look at the
MAD's tomorrow and try to ascertain that.

Thanks
Nitin


 



> 
> Thanks.
> 
> -- Hal
> 
> Index: osm_sa_mcmember_record.c
> ===================================================================
> --- osm_sa_mcmember_record.c	(revision 1821)
> +++ osm_sa_mcmember_record.c	(working copy)
> @@ -1325,11 +1325,13 @@
>    /* copy qkey mlid tclass pkey sl_flow_hop mtu rate pkt_life
> sl_flow_hop */
>    __copy_from_create_mc_rec(&mcmember_rec, &p_mgrp->mcmember_rec);
>  
> +#if 0
>    if(p_mgrp->well_known)
>    {
>      p_mgrp->mcmember_rec.mtu = mtu;
>      mcmember_rec.mtu = mtu;
>    }
> +#endif
>  
>    /* Release the lock as we don't need it. */
>    CL_PLOCK_RELEASE( p_rcv->p_lock );
> 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From mst at mellanox.co.il  Wed Feb 23 01:49:25 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 11:49:25 +0200
Subject: [openib-general] Re: SDP: BUG 2034 workaround
In-Reply-To: <20050222185934.K23420@topspin.com>
References: <20050222101445.GK23837@mellanox.co.il>
	<20050222185934.K23420@topspin.com>
Message-ID: <20050223094924.GT23837@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: SDP: BUG 2034 workaround
> 
> On Tue, Feb 22, 2005 at 12:14:45PM +0200, Michael S. Tsirkin wrote:
> > 
> > sdp_inet.c, inside _sdp_inet_listen, we have:
> > 
> > #if 0                           /* BUG 2034 workaround. */
> >         conn->backlog_max = backlog;
> > #else
> >         conn->backlog_max = 1024;
> > #endif
> > 
> > What gives? what would be the proper fix as opposed to a work-around?
> 
>   The Linux TCP listen uses two seperate values to control the listner
> backlog, so I ignored the backlog parameter to get closer to the default
> Linux behavior, until a really solution is devised.
>  
>   Basically TCP uses a two stage backlog to defend against SYN attacks.
> When a SYN is received a small amount of state is kept until the full
> handshake is completed, at which point a full socket is created and
> queued onto the listen sockets accept queue. The second stage uses the
> listen() backlog parameter to manage the accept queue. The first stage
> queue size is managed using a sysctl, (net.pv4.tcp_max_syn_backlog)
> which on a lot of systems defaults to 1024.
> 
>   SDP on the other hand creates the full socket on the REQ request, and
> places it directly into the listen sockets accept queue. So the full
> depth of the queue is governed by the listen backlog parameter. Worse
> still the socket layer listen, above TCP and SDP, limits the listen()
> backlog that is passed to the protocol to 128, and only recently did this
> become adjustable using net.core.somaxconn. 
> 
>    Using just the backlog parameter to manage this queue, results in a
> few programs that use a high connection volume to get rejected connection,
> casued by a full backlog queue, even when upped to the full 128 that the
> socket layer allows by default.
> 
> 
> -Libor
> 

I see. A protocol bug.
I wander if some application could get confused if it
gets more connections it set the backlog up for. What do you think?


For now, a somewhat cleaner work-around would be:
1. Replace the /* BUG 2034 workaround. */ comment with the excellent
   explanation above.
2. Take the max value between the socket backlog and sysctl_max_syn_backlog.

Want a patch like that?


-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Wed Feb 23 02:19:57 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 23 Feb 2005 05:19:57 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1109130984.16278.547.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
	<1108589174.4645.138.camel@localhost.localdomain>
	<1108593217.16278.47.camel@sr1-umpk-01>
	<1108674754.4761.634.camel@localhost.localdomain>
	<1109130984.16278.547.camel@sr1-umpk-01>
Message-ID: <1109153997.13555.179.camel@localhost.localdomain>

On Tue, 2005-02-22 at 22:56, Nitin Hande wrote:
> So I tried the latest patches and preliminarily things seem to be
> working fine. 

Yipee.

> The PathRecord response is successful and so is the MTU
> correct. I need to spend some more time looking at MAD and confirm it. I
> could configure both interfaces and ping each other this time. Here is
> some out on the solaris side:
> 
> [root at dongon ~]# ifconfig -a
> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> 8232 index 1
>         inet 127.0.0.1 netmask ff000000 
> ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index
> 28
>         inet 192.168.100.105 netmask ffffff00 broadcast 192.168.100.255
>         ipib 0:2c:0:16:fe:80:0:0:0:0:0:0:0:2:c9:1:9:76:51:d1 
> .....
> [root at dongon ~]# ping -s 192.168.100.104
> PING 192.168.100.104: 56 data bytes
> 64 bytes from 192.168.100.104: icmp_seq=0. time=0.590 ms
> 64 bytes from 192.168.100.104: icmp_seq=1. time=0.434 ms
> 64 bytes from 192.168.100.104: icmp_seq=2. time=0.365 ms
> 
> the other side is a openib interface runing OpenSM.
> 
> So after this test above, I try to run snoop on the solaris interface
> and get the following error message from the layer below IPoIB:
> 
> Feb 22 19:50:25 dongon.SFBay.Sun.COM ibd: [ID 517869 kern.info] NOTICE:
> ibd0: HCA GUID 0002c901097651d0 port 1 PKEY ffff Could not get list of
> IBA multicast groups
> 
> My preliminary assumption is that OpenSm is not returning the list of
> multicast groups that the ibd interface has joined. I will look at the
> MAD's tomorrow and try to ascertain that.

How does S10 request this ? Remember that if it is a GetTable and
doesn't fit in a single MAD, it will be broken now. If that is the case,
we will live with this until we have real RMPP.

-- Hal



From mst at mellanox.co.il  Wed Feb 23 03:01:43 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 13:01:43 +0200
Subject: [openib-general] Re: Re: SDP_CONN_LOCK
In-Reply-To: <20050222122801.C23420@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
	<20050220001137.GC8268@mellanox.co.il>
	<20050222122801.C23420@topspin.com>
Message-ID: <20050223110143.GZ23837@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: Re: SDP_CONN_LOCK
> 
> On Sun, Feb 20, 2005 at 02:11:37AM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > 
> > >   And here I would have thought SDP_CONN_HOLD as a macro would have
> > > bugged people more. :)
> >
> > Didn't get that far yet :)
> > 
> > I also wander whether we could get rid of most called to 
> > _CHECK macros, consequently turn lots of functions
> > which cant otherwise fail into void?
> > Given that these checks are disabled upon a compile switch, one wanders
> > how big a debugging aid they are.
> 
>   Yes, I think the _CHECK and EXPECT macros can be gotten rid of. Getting
> rid of _CHECK does have the added benefit of turning a lot of functions
> into void return functions, which in turn can get rid of most of the
> EXPECT macros. I do agree that there are quite a few function that return
> an status which can safely be turned into void functions.
> 
> -Libor
> 

Here's a patch to remove CHECK_NULL's, as a first step.

Signed-off-by: Michael s. Tsirkin <mst at mellanox.co.il>


Index: sdp_kvec.c
===================================================================
--- sdp_kvec.c	(revision 1871)
+++ sdp_kvec.c	(working copy)
@@ -43,8 +43,6 @@
  */
 static int _sdp_iocb_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
 {
-	SDP_CHECK_NULL(element, -EINVAL);
-
 	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
 } /* _sdp_iocb_q_cancel_lookup_func */
 
@@ -61,8 +59,6 @@ static int _sdp_iocb_q_cancel_lookup_fun
  */
 static int _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn, ssize_t error)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	return sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL,
 					error);
 } /* _sdp_iocb_q_cancel_read_pending */
@@ -75,8 +71,6 @@ static int _sdp_iocb_q_cancel_read_sourc
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->r_src,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
@@ -95,8 +89,6 @@ static int _sdp_iocb_q_cancel_read_sourc
  */
 static int _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	return sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_read_snk */
 
@@ -108,8 +100,6 @@ static int _sdp_iocb_q_cancel_write_pend
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	while (NULL !=
 	       (iocb =
 		(struct sdpc_iocb *) sdp_desc_q_lookup(&conn->send_queue,
@@ -130,8 +120,6 @@ static int _sdp_iocb_q_cancel_write_pend
  */
 static int _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn, ssize_t error)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	return sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_write_source */
 
@@ -143,8 +131,6 @@ static int _sdp_iocb_q_cancel_write_snk(
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->w_snk,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
@@ -165,8 +151,6 @@ int sdp_iocb_q_cancel_all_read(struct sd
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	result = _sdp_iocb_q_cancel_read_pending(conn, error);
 	SDP_EXPECT(!(0 > result));
 
@@ -186,8 +170,6 @@ int sdp_iocb_q_cancel_all_write(struct s
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	result = _sdp_iocb_q_cancel_write_pending(conn, error);
 	SDP_EXPECT(!(0 > result));
 
@@ -207,8 +189,6 @@ int sdp_iocb_q_cancel_all(struct sdp_opt
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	result = sdp_iocb_q_cancel_all_read(conn, error);
 	SDP_EXPECT(!(0 > result));
 
Index: sdp_write.c
===================================================================
--- sdp_write.c	(revision 1871)
+++ sdp_write.c	(working copy)
@@ -48,8 +48,6 @@ int sdp_event_write(struct sdp_opt *conn
 	int result;
 	int type;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(comp, -EINVAL);
 	/*
 	 * error handling
 	 */
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1871)
+++ sdp_rcvd.c	(working copy)
@@ -42,9 +42,6 @@ static int _sdp_rcvd_disconnect(struct s
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Disconnect msg received.");
 
 	switch (conn->state) {
@@ -121,9 +118,6 @@ static int _sdp_rcvd_abort(struct sdp_op
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Abort msg received.");
 	/*
 	 * Connection should be in some post DisConn recveived state.
@@ -155,8 +149,6 @@ static int _sdp_rcvd_send_sm(struct sdp_
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * 1) Conn is not in source cancel mode. Send active IOCB
 	 *    using buffered mode
@@ -201,9 +193,6 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	rwch = (struct msg_hdr_rwch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rwch);
 
@@ -265,9 +254,6 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	rrch = (struct msg_hdr_rrch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rrch);
 
@@ -349,9 +335,6 @@ static int _sdp_rcvd_mode_change(struct 
 	struct msg_hdr_mch *mch;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	mch = (struct msg_hdr_mch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_mch);
 
@@ -448,9 +431,6 @@ static int _sdp_rcvd_src_cancel(struct s
 	struct sdpc_advt *advt;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Source Cancel. active <%d> pending <%d> mode <%d>",
 		     sdp_advt_q_size(&conn->src_actv),
 		     sdp_advt_q_size(&conn->src_pend), conn->send_mode);
@@ -544,9 +524,6 @@ static int _sdp_rcvd_snk_cancel(struct s
 	s32 counter;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Sink Cancel. active <%d> mode <%d>",
 		     conn->snk_recv, conn->send_mode);
 	/*
@@ -621,9 +598,6 @@ static int _sdp_rcvd_snk_cancel_ack(stru
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Sink Cancel Ack. actv <%d> mode <%d> flag <%08x>",
 		     conn->snk_sent, conn->recv_mode, conn->flags);
 
@@ -663,9 +637,6 @@ static int _sdp_rcvd_resize_buff_ack(str
 	struct msg_hdr_crbh *crbh;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	crbh = (struct msg_hdr_crbh *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_crbh);
 
@@ -694,9 +665,6 @@ static int _sdp_rcvd_suspend(struct sdp_
 {
 	struct msg_hdr_sch *sch;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sch = (struct msg_hdr_sch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_sch);
 
@@ -707,9 +675,6 @@ static int _sdp_rcvd_suspend(struct sdp_
 
 static int _sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return 0;
 } /* _sdp_rcvd_suspend_ack */
 
@@ -720,9 +685,6 @@ static int _sdp_rcvd_snk_avail(struct sd
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	snkah = (struct msg_hdr_snkah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_snkah);
 
@@ -862,9 +824,6 @@ static int _sdp_rcvd_src_avail(struct sd
 	int result;
 	s32 size;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	srcah = (struct msg_hdr_srcah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_srcah);
 
@@ -1034,9 +993,6 @@ static int _sdp_rcvd_data(struct sdp_opt
 {
 	int ret_val;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	if (buff->tail == buff->data)
 		return 0;
 	/*
@@ -1080,8 +1036,6 @@ static int _sdp_rcvd_data(struct sdp_opt
  */
 static int _sdp_rcvd_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * Since the gateway only initates RDMA's but is never a target, and
 	 * for a few other reasons, there are certain valid SDP messages
@@ -1151,8 +1105,6 @@ int sdp_event_recv(struct sdp_opt *conn,
 	u32 offset;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(comp, -EINVAL);
 	/*
 	 * error handling
 	 */
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1871)
+++ sdp_inet.c	(working copy)
@@ -202,8 +202,6 @@ static int _sdp_inet_abort(struct sdp_op
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	conn->send_buf = 0;
 
 	switch (conn->istate) {
@@ -251,8 +249,6 @@ static int _sdp_inet_disconnect(struct s
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	switch (conn->istate) {
 	case SDP_SOCK_ST_CONNECT:
 		result = sdp_wall_abort(conn);
@@ -320,9 +316,6 @@ static int _sdp_inet_release(struct sock
 	long timeout;
 	u32  flags;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->file, -EINVAL);
-
 	if (NULL == sock->sk) {
 		sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>",
 			     sock->type, sock->state, sock->flags);
@@ -465,10 +458,6 @@ static int _sdp_inet_bind(struct socket 
 	u16 bind_port;
 	int result;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(uaddr, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -562,10 +551,6 @@ static int _sdp_inet_connect(struct sock
 	long timeout;
 	int result;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(uaddr, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -732,9 +717,6 @@ static int _sdp_inet_listen(struct socke
 	struct sdp_opt *conn;
 	int result;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -797,10 +779,6 @@ static int _sdp_inet_accept(struct socke
 	int result;
 	long timeout;
 
-	SDP_CHECK_NULL(listen_sock, -EINVAL);
-	SDP_CHECK_NULL(accept_sock, -EINVAL);
-	SDP_CHECK_NULL(listen_sock->sk, -EINVAL);
-
 	listen_sk = listen_sock->sk;
 	listen_conn = SDP_GET_CONN(listen_sk);
 
@@ -955,11 +933,6 @@ static int _sdp_inet_getname(struct sock
 	struct sock *sk;
 	struct sdp_opt *conn;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(addr, -EINVAL);
-	SDP_CHECK_NULL(size, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -997,8 +970,6 @@ static unsigned int _sdp_inet_poll(struc
 	struct sdp_opt *conn;
 	unsigned int mask = 0;
 
-	SDP_CHECK_NULL(sock, 0);
-	SDP_CHECK_NULL(sock->sk, 0);
 	/*
 	 * file and/or wait can be NULL, once poll is asleep and needs to
 	 * recheck the falgs on being woken.
@@ -1090,9 +1061,6 @@ static int _sdp_inet_ioctl(struct socket
 	int result = 0;
 	int value;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -1215,10 +1183,6 @@ static int _sdp_inet_setopt(struct socke
 	int value;
 	int result = 0;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(optval, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -1286,11 +1250,6 @@ static int _sdp_inet_getopt(struct socke
 	int value;
 	int len;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(optval, -EINVAL);
-	SDP_CHECK_NULL(optlen, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
@@ -1350,8 +1309,6 @@ static int _sdp_inet_shutdown(struct soc
 	int result = 0;
 	struct sdp_opt *conn;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
 	conn = SDP_GET_CONN(sock->sk);
 
 	sdp_dbg_ctrl(conn, "SHUTDOWN: flag <%d>", flag);
@@ -1466,8 +1423,6 @@ static int _sdp_inet_create(struct socke
 {
 	struct sdp_opt *conn;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-
 	sdp_dbg_ctrl(NULL, "SOCKET: type <%d> proto <%d> state <%u:%08lx>",
 		     sock->type, protocol, sock->state, sock->flags);
 	
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1871)
+++ sdp_proto.h	(working copy)
@@ -548,7 +548,6 @@ extern int sdp_debug_level;
 
 #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_CTRL
 #define SDP_EXPECT(expr)
-#define SDP_CHECK_NULL(value, result)
 #else
 #define SDP_EXPECT(expr)                                                 \
 do {                                                                     \
@@ -557,9 +556,6 @@ do {                                    
 		            #expr);                                      \
 	}                                                                \
 } while (0) /* SDP_EXPECT */
-
-#define SDP_CHECK_NULL(value, result) \
-        do { if (NULL == (value)) return (result); } while (0)
 #endif
 
 /*
@@ -573,7 +569,6 @@ static inline int __sdp_inet_write_space
 {
 	int size;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * Allow for more space if Urgent data is being considered
 	 */
@@ -596,8 +591,6 @@ static inline int __sdp_inet_write_space
  */
 static inline int __sdp_inet_writable(struct sdp_opt *conn)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	if (0 < (SDP_ST_MASK_OPEN & conn->istate))
 		return (__sdp_inet_write_space(conn, 0) <
 			(conn->send_qud / 2)) ? 0 : 1;
@@ -636,8 +629,6 @@ static inline int __sdp_conn_state_dump(
 #ifdef _SDP_CONN_STATE_REC
 	int counter;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_init("STATE: Connection <%04x> state:", conn->hashent);
 
 	if (SDP_CONN_ST_INVALID == conn->state_rec.state[0]) {
Index: sdp_read.c
===================================================================
--- sdp_read.c	(revision 1871)
+++ sdp_read.c	(working copy)
@@ -46,8 +46,6 @@ static int _sdp_event_read_advt(struct s
 	struct sdpc_advt *advt;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(comp, -EINVAL);
 	/*
 	 * if this was the last RDMA read for an advertisment, post a notice.
 	 * Might want to post multiple RDMA read completion messages per
@@ -116,8 +114,6 @@ int sdp_event_read(struct sdp_opt *conn,
 	s32 result;
 	s32 type;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(comp, -EINVAL);
 	/*
 	 * error handling
 	 */
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1871)
+++ sdp_send.c	(working copy)
@@ -47,9 +47,6 @@ static int _sdp_send_buff_post(struct sd
 	struct ib_send_wr *bad_wr;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-	SDP_CHECK_NULL(buff->bsdh_hdr, -EINVAL);
 	/*
 	 * write header send buffer.
 	 */
@@ -196,8 +193,6 @@ static int _sdp_send_data_buff_post(stru
 	struct sdpc_advt *advt;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * check state to determine OK to send:
 	 *
@@ -312,8 +307,6 @@ static int _sdp_send_data_buff_snk(struc
 	int result;
 	int zcopy;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * check state to determine OK to send:
 	 *
@@ -453,8 +446,6 @@ static int _sdp_send_data_iocb_snk(struc
 	int result;
 	int zcopy;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(iocb, -EINVAL);
 	/*
 	 * register IOCBs physical memory, we check for previous
 	 * registration, since multiple writes may have been required
@@ -577,8 +568,6 @@ static int _sdp_send_data_iocb_src(struc
 	struct sdpc_buff *buff;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(iocb, -EINVAL);
 	/*
 	 * 1) local source cancel is pending
 	 * 2) sufficient send credits for buffered transmission.
@@ -746,8 +735,6 @@ static int _sdp_send_iocb_buff_write(str
 	unsigned int counter;
 	void  *addr;
 
-	SDP_CHECK_NULL(iocb, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * initialize counter to correct page and offset.
 	 */
@@ -794,9 +781,6 @@ static int _sdp_send_data_iocb_buff(stru
 	int result;
 	int copy;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(iocb, -EINVAL);
-
 	if (0 < conn->src_sent)
 		return ENOBUFS;
 	/*
@@ -868,9 +852,6 @@ static int _sdp_send_data_iocb(struct sd
 {
 	int result = ENOBUFS;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(iocb, -EINVAL);
-
 	if (!(conn->send_cq_size > conn->s_wq_size))
 		goto done;
 	/*
@@ -970,8 +951,6 @@ static int _sdp_send_data_queue_test(str
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(element, -EINVAL);
 	/*
 	 * Notify caller to buffer data:
 	 * 1) Invalid state for transmission
@@ -1003,7 +982,6 @@ static int _sdp_send_data_queue_flush(st
 	struct sdpc_desc *element;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * As long as there is data, try to post buffered data, until a
 	 * non-zero result is generated.
@@ -1046,8 +1024,6 @@ static int _sdp_send_data_queue(struct s
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(element, -EINVAL);
 	/*
 	 * If data is being buffered, save and return, send/recv completions
 	 * will flush the queue.
@@ -1098,7 +1074,6 @@ static inline struct sdpc_buff *_sdp_sen
 {
 	struct sdpc_buff *buff;
 
-	SDP_CHECK_NULL(conn, NULL);
 	/*
 	 * If there is no available buffer get a new one.
 	 */
@@ -1128,8 +1103,6 @@ static inline int _sdp_send_data_buff_pu
 	int result = 0;
 	int expect;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * See note on send OOB implementation in SendBuffPost.
 	 */
@@ -1185,9 +1158,6 @@ static int _sdp_send_ctrl_buff_test(stru
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	if (0 == (SDP_ST_MASK_CTRL_OK & conn->state) ||
 	    !(conn->send_cq_size > conn->s_wq_size) ||
 	    !(0 < conn->r_recv_bf) ||
@@ -1215,7 +1185,6 @@ static int _sdp_send_ctrl_buff_flush(str
 	struct sdpc_desc *element;
 	int result = 0;
 	
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * As long as there are buffers, try to post  until a non-zero
 	 * result is generated. (positive: no space; negative: error)
@@ -1253,8 +1222,6 @@ static int _sdp_send_ctrl_buff_buffered(
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
@@ -1294,7 +1261,6 @@ static int _sdp_send_ctrl_buff(struct sd
 	int result = 0;
 	struct sdpc_buff *buff;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * create the message, which contains just the bsdh header.
 	 * (don't need to worry about header space reservation)
@@ -1349,7 +1315,6 @@ static int _sdp_send_ctrl_disconnect(str
 	int result = 0;
 	struct sdpc_buff *buff;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * create the disconnect message, which contains just the bsdh header.
 	 * (don't need to worry about header space reservation)
@@ -1398,7 +1363,6 @@ int sdp_send_ctrl_disconnect(struct sdp_
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * Only create/post the message if there is no data in the data queue,
 	 * otherwise ignore the call. The flush queue will see to it, that a
@@ -1428,7 +1392,6 @@ int sdp_send_ctrl_disconnect(struct sdp_
  */
 int sdp_send_ctrl_ack(struct sdp_opt *conn)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * The gratuitous ack is not really and ack, but an update of the
 	 * number of buffers posted for receive. Important when traffic is
@@ -1483,7 +1446,6 @@ int sdp_send_ctrl_snk_cancel_ack(struct 
  */
 int sdp_send_ctrl_abort(struct sdp_opt *conn)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * send
 	 */
@@ -1499,7 +1461,6 @@ int sdp_send_ctrl_resize_buff_ack(struct
 	struct sdpc_buff *buff;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * create the message, which contains just the bsdh header.
 	 * (don't need to worry about header space reservation)
@@ -1551,7 +1512,6 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	struct sdpc_buff *buff;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * check size
 	 */
@@ -1621,7 +1581,6 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 	struct sdpc_buff *buff;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * create the message, which contains just the bsdh header.
 	 * (don't need to worry about header space reservation)
@@ -1682,7 +1641,6 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	struct sdpc_buff *buff;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * check mode
 	 */
@@ -1746,7 +1704,6 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	struct sdpc_buff *buff;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * validate that the requested mode transition is OK.
 	 */
@@ -1835,8 +1792,6 @@ static int _sdp_write_src_lookup(struct 
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
 
-	SDP_CHECK_NULL(element, -EINVAL);
-
 	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key)
 		return 0;
 	else
@@ -1853,8 +1808,6 @@ static int _sdp_inet_write_cancel(struct
 	struct sdpc_iocb *iocb;
 	int result = 0;
 
-	SDP_CHECK_NULL(req, -ERANGE);
-
 	sdp_dbg_ctrl(NULL, "Cancel Write IOCB user <%d> key <%d> flag <%08lx>",
 		     req->ki_users, req->ki_key, req->ki_flags);
 	
@@ -2034,7 +1987,6 @@ int sdp_send_flush(struct sdp_opt *conn)
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * keep posting sends as long as there is room for an SDP post.
 	 * Priority goes to control messages, and we need to follow the
@@ -2104,9 +2056,6 @@ int sdp_inet_send(struct kiocb *req,
 	int oob;
 	long timeout = -1;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(msg, -EINVAL);
 	/*
 	 * set oob flag.
 	 */
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1871)
+++ sdp_actv.c	(working copy)
@@ -48,8 +48,6 @@ static int _sdp_actv_conn_establish(stru
 	struct sock *sk;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "active etablish. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -548,7 +546,6 @@ int sdp_cm_connect(struct sdp_opt *conn)
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * get the buffer size we'll use for this connection. (and all others)
 	 */
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1871)
+++ sdp_conn.c	(working copy)
@@ -81,9 +81,6 @@ int sdp_inet_accept_q_put(struct sdp_opt
 {
 	struct sdp_opt *next_conn;
 
-	SDP_CHECK_NULL(listen_conn, -EINVAL);
-	SDP_CHECK_NULL(accept_conn, -EINVAL);
-
 	if (NULL != listen_conn->parent ||
 	    NULL != accept_conn->parent ||
 	    NULL == listen_conn->accept_next ||
@@ -115,8 +112,6 @@ struct sdp_opt *sdp_inet_accept_q_get(st
 	struct sdp_opt *prev_conn;
 	struct sdp_opt *accept_conn;
 
-	SDP_CHECK_NULL(listen_conn, NULL);
-
 	if (NULL != listen_conn->parent ||
 	    NULL == listen_conn->accept_next ||
 	    NULL == listen_conn->accept_prev ||
@@ -156,8 +151,6 @@ int sdp_inet_accept_q_remove(struct sdp_
 	struct sdp_opt *next_conn;
 	struct sdp_opt *prev_conn;
 
-	SDP_CHECK_NULL(accept_conn, -EINVAL);
-
 	if (NULL == accept_conn->parent)
 		return -EFAULT;
 	/*
@@ -192,8 +185,6 @@ int sdp_inet_listen_start(struct sdp_opt
 {
 	unsigned long flags;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	if (SDP_SOCK_ST_CLOSED != conn->istate) {
 		sdp_dbg_warn(conn, "Incorrect connection state to listen.");
 		return -EBADFD;
@@ -229,8 +220,6 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	int result;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(listen_conn, -EINVAL);
-
 	if (SDP_SOCK_ST_LISTEN != listen_conn->istate) {
 		sdp_dbg_warn(listen_conn, "Incorrect state to stop listen.");
 		return -EBADFD;
@@ -324,8 +313,6 @@ int sdp_inet_port_get(struct sdp_opt *co
 	static s32 rover = -1;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sk = conn->sk;
 	/*
 	 * lock table
@@ -439,8 +426,6 @@ int sdp_inet_port_put(struct sdp_opt *co
 {
 	unsigned long flags;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	if (NULL == conn->bind_p_next)
 		return -EADDRNOTAVAIL;
 	/*
@@ -471,9 +456,6 @@ int sdp_inet_port_inherit(struct sdp_opt
 	int result;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(child, -EINVAL);
-	SDP_CHECK_NULL(parent, -EINVAL);
-	SDP_CHECK_NULL(parent->bind_p_next, -EINVAL);
 	/*
 	 * lock table
 	 */
@@ -511,8 +493,6 @@ static int _sdp_conn_table_insert(struct
 	int result = -ENOMEM;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	if (SDP_DEV_SK_INVALID != conn->hashent)
 		return -ERANGE;
 	/*
@@ -556,7 +536,6 @@ int sdp_conn_table_remove(struct sdp_opt
 	int result = 0;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * lock table
 	 */
@@ -931,8 +910,6 @@ void sdp_conn_internal_unlock(struct sdp
  */
 static int _sdp_conn_lock_init(struct sdp_opt *conn)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	spin_lock_init(&(conn->lock.slock));
 	conn->lock.users = 0;
 	init_waitqueue_head(&(conn->lock.waitq));
@@ -955,7 +932,6 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	int                    attr_mask = 0;
 	int                    result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * look up correct HCA and port
 	 */
@@ -1363,9 +1339,6 @@ int sdp_proc_dump_conn_main(char *buffer
 	u64   d_guid;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(buffer, -EINVAL);
-	SDP_CHECK_NULL(end_index, -EINVAL);
-
 	*end_index = 0;
 	/*
 	 * header should only be printed once
@@ -1471,9 +1444,6 @@ int sdp_proc_dump_conn_data(char *buffer
 	int   offset = 0;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(buffer, -EINVAL);
-	SDP_CHECK_NULL(end_index, -EINVAL);
-
 	*end_index = 0;
 	/*
 	 * header should only be printed once
@@ -1570,9 +1540,6 @@ int sdp_proc_dump_conn_rdma(char *buffer
 	int   offset = 0;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(buffer, -EINVAL);
-	SDP_CHECK_NULL(end_index, -EINVAL);
-
 	*end_index = 0;
 	/*
 	 * header should only be printed once
@@ -1652,9 +1619,6 @@ int sdp_proc_dump_conn_sopt(char *buffer
 	int   offset = 0;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(buffer, -EINVAL);
-	SDP_CHECK_NULL(end_index, -EINVAL);
-
 	*end_index = 0;
 	/*
 	 * header should only be printed once
@@ -1715,9 +1679,6 @@ int sdp_proc_dump_device(char *buffer,
 {
 	int offset = 0;
 
-	SDP_CHECK_NULL(buffer, -EINVAL);
-	SDP_CHECK_NULL(end_index, -EINVAL);
-
 	*end_index = 0;
 	/*
 	 * header should only be printed once
Index: sdp_advt.c
===================================================================
--- sdp_advt.c	(revision 1871)
+++ sdp_advt.c	(working copy)
@@ -72,8 +72,6 @@ struct sdpc_advt *sdp_advt_create(void)
  */
 int sdp_advt_destroy(struct sdpc_advt *advt)
 {
-	SDP_CHECK_NULL(advt, -EINVAL);
-
 	if (NULL != advt->next || NULL != advt->prev)
 		return -EACCES;
 	/*
@@ -93,8 +91,6 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 	struct sdpc_advt *next;
 	struct sdpc_advt *prev;
 
-	SDP_CHECK_NULL(table, NULL);
-
 	advt = table->head;
 	if (NULL == advt)
 		return NULL;
@@ -123,8 +119,6 @@ struct sdpc_advt *sdp_advt_q_get(struct 
  */
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table)
 {
-	SDP_CHECK_NULL(table, NULL);
-
 	return table->head;
 } /* sdp_advt_q_look */
 
@@ -136,9 +130,6 @@ int sdp_advt_q_put(struct sdpc_advt_q *t
 	struct sdpc_advt *next;
 	struct sdpc_advt *prev;
 
-	SDP_CHECK_NULL(table, -EINVAL);
-	SDP_CHECK_NULL(advt, -EINVAL);
-
 	if (NULL == table->head) {
 		advt->next = advt;
 		advt->prev = advt;
@@ -164,8 +155,6 @@ int sdp_advt_q_put(struct sdpc_advt_q *t
  */
 int sdp_advt_q_init(struct sdpc_advt_q *table)
 {
-	SDP_CHECK_NULL(table, -EINVAL);
-
 	table->head = NULL;
 	table->size = 0;
 
@@ -180,7 +169,6 @@ int sdp_advt_q_clear(struct sdpc_advt_q 
 	struct sdpc_advt *advt;
 	int result;
 
-	SDP_CHECK_NULL(table, -EINVAL);
 	/*
 	 * drain the table of any objects
 	 */
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1871)
+++ sdp_recv.c	(working copy)
@@ -48,7 +48,6 @@ static int _sdp_post_recv_buff(struct sd
 	struct sdpc_buff *buff;
 	s32 result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * get a buffer
 	 */
@@ -123,7 +122,6 @@ static int _sdp_post_rdma_buff(struct sd
 	struct sdpc_buff *buff;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * check queue depth
 	 */
@@ -247,7 +245,6 @@ static int _sdp_post_rdma_iocb_src(struc
 	int result;
 	int zcopy;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * check queue depth
 	 */
@@ -387,7 +384,6 @@ static int _sdp_post_rdma_iocb_snk(struc
 	struct sdpc_iocb *iocb;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * check if sink cancel is pending
 	 */
@@ -486,7 +482,6 @@ static int _sdp_post_rdma(struct sdp_opt
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * Since RDMA Reads rely on posting to the Send WQ, stop if
 	 * we're not in an appropriate state. It's possible to queue
@@ -567,7 +562,6 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	int result = 0;
 	int counter;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * verify that the connection is in a posting state
 	 */
@@ -676,8 +670,6 @@ static int _sdp_read_buff_iocb(struct sd
 	void  *data;
 	void  *tail;
 
-	SDP_CHECK_NULL(iocb, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * OOB buffer adjustment. We basically throw away OOB data
 	 * when writing into AIO buffer. We can't split it into it's
@@ -736,8 +728,6 @@ static int _sdp_recv_buff_iocb_active(st
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * Get the IOCB, We'll fill with exactly one
 	 */
@@ -787,8 +777,6 @@ static int _sdp_recv_buff_iocb_pending(s
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * check the IOCB
 	 */
@@ -848,9 +836,6 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	int result;
 	int buffered;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_data(conn, "RECV BUFF, bytes <%u>",
 		     (unsigned)(buff->tail - buff->data));
 	/*
@@ -947,8 +932,6 @@ static int _sdp_read_src_lookup(struct s
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
 
-	SDP_CHECK_NULL(element, -EINVAL);
-
 	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key)
 		return 0;
 	else
@@ -966,8 +949,6 @@ static int _sdp_inet_read_cancel(struct 
 	struct sdpc_iocb *iocb;
 	int result = 0;
 
-	SDP_CHECK_NULL(req, -ERANGE);
-
 	sdp_dbg_ctrl(NULL, "Cancel Read IOCB. user <%d> key <%d> flag <%08lx>",
 		     req->ki_users, req->ki_key, req->ki_flags);
 	
@@ -1095,8 +1076,6 @@ done:
  */
 static int _sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
 {
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return ((buff->tail == buff->head) ? 1 : 0);
 } /* _sdp_inet_recv_urg_test */
 
@@ -1108,9 +1087,6 @@ static int _sdp_inet_recv_urg_trav(struc
 	u8 *value = (u8 *) arg;
 	u8 update;
 
-	SDP_CHECK_NULL(buff, -EINVAL);
-	SDP_CHECK_NULL(value, -EINVAL);
-
 	if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
 		SDP_EXPECT((buff->tail > buff->data));
 
@@ -1141,8 +1117,6 @@ static int _sdp_inet_recv_urg(struct soc
 	int result = 0;
 	u8 value;
 
-	SDP_CHECK_NULL(sk, -EINVAL);
-	SDP_CHECK_NULL(msg, -EINVAL);
 	conn = SDP_GET_CONN(sk);
 
 	if (sock_flag(sk, SOCK_URGINLINE) || 0 == conn->rcv_urg_cnt)
@@ -1232,10 +1206,6 @@ int sdp_inet_recv(struct kiocb  *req,
 	s8 ack = 0;
 	struct sdpc_buff_q peek_queue;
 
-	SDP_CHECK_NULL(sock, -EINVAL);
-	SDP_CHECK_NULL(sock->sk, -EINVAL);
-	SDP_CHECK_NULL(msg, -EINVAL);
-
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
 
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1871)
+++ sdp_wall.c	(working copy)
@@ -46,8 +46,6 @@ int sdp_wall_send_close(struct sdp_opt *
 	struct sdpc_buff *buff;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Close send. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -123,8 +121,6 @@ int sdp_wall_send_closing(struct sdp_opt
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Close confirm. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -177,8 +173,6 @@ int sdp_wall_send_abort(struct sdp_opt *
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Abort send. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -291,8 +285,6 @@ int sdp_wall_recv_reject(struct sdp_opt 
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Reject recv. src <%08x:%04x> dst <%08x:%04x> <%d>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port, error);
@@ -326,8 +318,6 @@ int sdp_wall_recv_confirm(struct sdp_opt
 	int attr_mask = 0;
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -388,8 +378,6 @@ error:
  */
 int sdp_wall_recv_failed(struct sdp_opt *conn, int error)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Failed recv. src <%08x:%04x> dst <%08x:%04x> <%d>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port, error);
@@ -425,8 +413,6 @@ int sdp_wall_recv_close(struct sdp_opt *
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Close recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -474,8 +460,6 @@ int sdp_wall_recv_closing(struct sdp_opt
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Closing recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -512,8 +496,6 @@ int sdp_wall_recv_abort(struct sdp_opt *
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Abort recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -559,8 +541,6 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Drop. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -629,7 +609,6 @@ int sdp_wall_abort(struct sdp_opt *conn)
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * notify both halves of the wall that the connection is being aborted.
 	 */
Index: sdp_proc.c
===================================================================
--- sdp_proc.c	(revision 1871)
+++ sdp_proc.c	(working copy)
@@ -55,8 +55,6 @@ static int _sdp_proc_read_parse(char *pa
 	long end_index = 0;
 	int  size;
 
-	SDP_CHECK_NULL(sub_entry, -EINVAL);
-
 #if 0
 	if (NULL == *start && 0 != offset) {
 		return 0; /* I'm not sure why this always gets 
@@ -138,7 +136,6 @@ int sdp_main_proc_cleanup(void)
 	struct sdpc_proc_ent *sub_entry;
 	int counter;
 
-	SDP_CHECK_NULL(_dir_root, -EINVAL);
 	/*
 	 * first clean-up the frameworks tables
 	 */
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1871)
+++ sdp_pass.c	(working copy)
@@ -47,7 +47,6 @@ static int _sdp_cm_accept(struct sdp_opt
 	int result;
 	int expect;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * build listen response headers
 	 */
@@ -191,8 +190,6 @@ static int _sdp_cm_listen_lookup(struct 
 	struct sock *sk;
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	sdp_dbg_ctrl(conn,
 		     "listen match for conn. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1871)
+++ sdp_sent.c	(working copy)
@@ -42,9 +42,6 @@ static int _sdp_sent_disconnect(struct s
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sdp_dbg_ctrl(conn, "Disconnect message sent.");
 
 	switch (conn->state) {
@@ -122,8 +119,6 @@ static int _sdp_sent_abort(struct sdp_op
 {
 	int result;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * The gateway interface should be in error state, initiate CM
 	 * disconnect.
@@ -140,9 +135,6 @@ static int _sdp_sent_abort(struct sdp_op
 
 static int _sdp_sent_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return 0;
 } /* _sdp_sent_send_sm */
 
@@ -150,9 +142,6 @@ static int _sdp_sent_rdma_wr(struct sdp_
 {
 	struct msg_hdr_rwch *rwch;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	rwch = (struct msg_hdr_rwch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rwch);
 
@@ -165,9 +154,6 @@ static int _sdp_sent_rdma_rd(struct sdp_
 {
 	struct msg_hdr_rrch *rrch;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	rrch = (struct msg_hdr_rrch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rrch);
 
@@ -180,9 +166,6 @@ static int _sdp_sent_mode_change(struct 
 {
 	struct msg_hdr_mch *mch;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	mch = (struct msg_hdr_mch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_mch);
 
@@ -193,26 +176,17 @@ static int _sdp_sent_mode_change(struct 
 
 static int _sdp_sent_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return 0;
 } /* _sdp_sent_src_cancel */
 
 static int _sdp_sent_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return 0;
 } /* _sdp_sent_snk_cancel */
 
 static int _sdp_sent_snk_cancel_ack(struct sdp_opt *conn,
 				    struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return 0;
 } /* _sdp_sent_snk_cancel_ack */
 
@@ -221,9 +195,6 @@ static int _sdp_sent_resize_buff_ack(str
 {
 	struct msg_hdr_crbah *crbah;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	crbah = (struct msg_hdr_crbah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_crbah);
 
@@ -236,9 +207,6 @@ static int _sdp_sent_suspend(struct sdp_
 {
 	struct msg_hdr_sch *sch;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	sch = (struct msg_hdr_sch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_sch);
 
@@ -249,9 +217,6 @@ static int _sdp_sent_suspend(struct sdp_
 
 static int _sdp_sent_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	return 0;
 } /* _sdp_sent_suspend_ack */
 
@@ -259,9 +224,6 @@ static int _sdp_sent_snk_avail(struct sd
 {
 	struct msg_hdr_snkah *snkah;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	snkah = (struct msg_hdr_snkah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_snkah);
 
@@ -274,9 +236,6 @@ static int _sdp_sent_src_avail(struct sd
 {
 	struct msg_hdr_srcah *srcah;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	srcah = (struct msg_hdr_srcah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_srcah);
 
@@ -292,10 +251,6 @@ static int _sdp_sent_data(struct sdp_opt
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(conn->sk, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	conn->send_qud -= buff->data_size;
 
 	return result;
@@ -306,8 +261,6 @@ static int _sdp_sent_data(struct sdp_opt
  */
 static int _sdp_sent_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
 	/*
 	 * Since the gateway only initates RDMA's but is never a target, and
 	 * for a few other reasons, there are certain valid SDP messages
@@ -378,8 +331,6 @@ int sdp_event_send(struct sdp_opt *conn,
 	struct sdpc_buff *buff;
 	struct sdpc_buff *head = NULL;
 
-	SDP_CHECK_NULL(conn, -EINVAL);
-	SDP_CHECK_NULL(comp, -EINVAL);
 	/*
 	 * error handling
 	 */
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1871)
+++ sdp_iocb.c	(working copy)
@@ -101,7 +101,6 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 {
 	int result;
 
-	SDP_CHECK_NULL(iocb, -EINVAL);
 	/*
 	 * check if IOCB is locked.
 	 */
@@ -241,7 +240,6 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	unsigned long addr;
 	size_t        size;
 	
-	SDP_CHECK_NULL(iocb, -EINVAL);
 	/*
 	 * mark IOCB as locked. We do not take a reference on the mm, AIO
 	 * handles this for us.
@@ -422,8 +420,6 @@ int sdp_iocb_register(struct sdpc_iocb *
 {
 	int result;
 
-	SDP_CHECK_NULL(iocb, -EINVAL);
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * register only once.
 	 */
@@ -475,8 +471,6 @@ int sdp_iocb_release(struct sdpc_iocb *i
 {
 	int result;
 
-	SDP_CHECK_NULL(iocb, -EINVAL);
-
 	if (0 == (SDP_IOCB_F_REG & iocb->flags))
 		return 0;
 
@@ -541,8 +535,6 @@ static void _sdp_iocb_complete(void *arg
  */
 int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status)
 {
-	SDP_CHECK_NULL(iocb, -EINVAL);
-
 	iocb->status = status;
 	
 	if (in_atomic() || irqs_disabled()) {
@@ -567,12 +559,6 @@ int sdp_iocb_q_remove(struct sdpc_iocb *
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
 
-	SDP_CHECK_NULL(iocb, -EINVAL);
-	SDP_CHECK_NULL(iocb->next, -EINVAL);
-	SDP_CHECK_NULL(iocb->prev, -EINVAL);
-	SDP_CHECK_NULL(iocb->table, -EINVAL);
-	SDP_CHECK_NULL(iocb->table->head, -EINVAL);
-
 	table = iocb->table;
 
 	if (iocb->next == iocb && iocb->prev == iocb)
@@ -604,8 +590,6 @@ struct sdpc_iocb *sdp_iocb_q_lookup(stru
 	struct sdpc_iocb *iocb = NULL;
 	int counter;
 
-	SDP_CHECK_NULL(table, NULL);
-
 	for (counter = 0, iocb = table->head; counter < table->size;
 	     counter++, iocb = iocb->next)
 		if (iocb->key == key)
@@ -680,8 +664,6 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
  */
 struct sdpc_iocb *sdp_iocb_q_look(struct sdpc_iocb_q *table)
 {
-	SDP_CHECK_NULL(table, NULL);
-
 	return table->head;
 } /* sdp_iocb_q_look */
 
@@ -694,8 +676,6 @@ static struct sdpc_iocb *_sdp_iocb_q_get
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
 
-	SDP_CHECK_NULL(table, NULL);
-
 	if (NULL == table->head)
 		return NULL;
 
@@ -734,9 +714,6 @@ static int _sdp_iocb_q_put(struct sdpc_i
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
 
-	SDP_CHECK_NULL(table, -EINVAL);
-	SDP_CHECK_NULL(iocb, -EINVAL);
-
 	if (NULL == table->head) {
 		iocb->next = iocb;
 		iocb->prev = iocb;
@@ -806,7 +783,6 @@ int sdp_iocb_q_cancel(struct sdpc_iocb_q
 	int result;
 	int total;
 
-	SDP_CHECK_NULL(table, -EINVAL);
 	/*
 	 * loop through IOCBs, completing each one with either a partial data
 	 * result, or a cancelled error.
@@ -839,8 +815,6 @@ int sdp_iocb_q_cancel(struct sdpc_iocb_q
  */
 int sdp_iocb_q_init(struct sdpc_iocb_q *table)
 {
-	SDP_CHECK_NULL(table, -EINVAL);
-
 	table->head = NULL;
 	table->size = 0;
 
@@ -855,7 +829,6 @@ int sdp_iocb_q_clear(struct sdpc_iocb_q 
 	struct sdpc_iocb *iocb;
 	int result;
 
-	SDP_CHECK_NULL(table, -EINVAL);
 	/*
 	 * drain the table of any objects
 	 */
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1871)
+++ sdp_event.c	(working copy)
@@ -45,9 +45,6 @@ int sdp_cq_event_locked(struct ib_wc *co
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(comp, -EINVAL);
-	SDP_CHECK_NULL(conn, -EINVAL);
-
 	if (0 < (SDP_ST_MASK_ERROR & conn->state)) {
 		/*
 		 * Ignore events in error state, connection is being 
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1871)
+++ sdp_buff.c	(working copy)
@@ -51,8 +51,6 @@ static inline struct sdpc_buff *_sdp_buf
 {
 	struct sdpc_buff *buff;
 
-	SDP_CHECK_NULL(pool, NULL);
-
 	if (NULL == pool->head)
 		return NULL;
 
@@ -91,9 +89,6 @@ static inline int _sdp_buff_q_put(struct
 				  int fifo)
 {
 	/* fifo: false == tail, true == head */
-	SDP_CHECK_NULL(pool, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	if (NULL != buff->pool)
 		return -EINVAL;
 
@@ -124,8 +119,6 @@ static inline int _sdp_buff_q_put(struct
 static inline struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
 						 int fifo)
 {
-	SDP_CHECK_NULL(pool, NULL);
-
 	if (NULL == pool->head || fifo)
 		return pool->head;
 	else
@@ -141,12 +134,6 @@ static inline int _sdp_buff_q_remove(str
 	struct sdpc_buff *prev;
 	struct sdpc_buff *next;
 
-	SDP_CHECK_NULL(pool, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-	SDP_CHECK_NULL(buff->next, -EINVAL);
-	SDP_CHECK_NULL(buff->prev, -EINVAL);
-	SDP_CHECK_NULL(buff->pool, -EINVAL);
-
 	if (pool != buff->pool)
 		return -EINVAL;
 
@@ -179,9 +166,6 @@ int sdp_buff_q_init(struct sdpc_buff_q *
 	int counter;
 	int result;
 
-	SDP_CHECK_NULL(pool, -EINVAL);
-	SDP_CHECK_NULL(main_pool, -EINVAL);
-
 	pool->head = NULL;
 	pool->size = 0;
 
@@ -207,9 +191,6 @@ int sdp_buff_q_remove(struct sdpc_buff *
 	struct sdpc_buff_q *pool;
 	int result;
 
-	SDP_CHECK_NULL(buff, -EINVAL);
-	SDP_CHECK_NULL(buff->pool, -EINVAL);
-
 	pool = buff->pool;
 
 	result = _sdp_buff_q_remove(pool, buff);
@@ -292,7 +273,6 @@ struct sdpc_buff *sdp_buff_q_fetch(struc
 	int result = 0;
 	int counter;
 
-	SDP_CHECK_NULL(pool, NULL);
 	/*
 	 * check to see if there is anything to traverse.
 	 */
@@ -329,8 +309,6 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 	int result = 0;
 	int counter;
 
-	SDP_CHECK_NULL(pool, -EINVAL);
-	SDP_CHECK_NULL(trav_func, -EINVAL);
 	/*
 	 * check to see if there is anything to traverse.
 	 */
@@ -398,8 +376,6 @@ int sdp_buff_q_clear_unmap(struct sdpc_b
 	int result;
 	struct sdpc_buff *buff;
 
-	SDP_CHECK_NULL(pool, -EINVAL);
-
 	while (NULL != (buff = _sdp_buff_q_get(pool, 0, NULL, NULL))) {
 		if (dev)
 			dma_unmap_single(dev, buff->real,
@@ -425,7 +401,6 @@ static int _sdp_buff_pool_release(struct
 {
 	struct sdpc_buff *buff;
 
-	SDP_CHECK_NULL(m_pool, -EINVAL);
 	/*
 	 * Release count buffers.
 	 */
@@ -450,7 +425,6 @@ static int _sdp_buff_pool_release(struct
  */
 static inline int _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
 {
-	SDP_CHECK_NULL(m_pool, -EINVAL);
 	/*
 	 * If there are more then minimum buffers outstanding, free half of
 	 * the available buffers.
@@ -480,7 +454,6 @@ static int _sdp_buff_pool_alloc(struct s
 	int total;
 	int result;
 
-	SDP_CHECK_NULL(m_pool, -EINVAL);
 	/*
 	 * Calculate the total number of buffers.
 	 */
@@ -679,7 +652,6 @@ struct sdpc_buff *sdp_buff_pool_get(void
 	int result;
 	unsigned long flags;
 
-	SDP_CHECK_NULL(main_pool, NULL);
 	/*
 	 * get buffer
 	 */
@@ -732,8 +704,6 @@ int sdp_buff_pool_put(struct sdpc_buff *
 {
 	unsigned long flags;
 
-	SDP_CHECK_NULL(main_pool, -EINVAL);
-
 	if (NULL == buff || NULL != buff->pool)
 		return -EINVAL;
 
@@ -774,9 +744,6 @@ int sdp_buff_pool_put(struct sdpc_buff *
  */
 int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff)
 {
-	SDP_CHECK_NULL(main_pool, -EINVAL);
-	SDP_CHECK_NULL(buff, -EINVAL);
-
 	buff->data = buff->head;
 	buff->tail = buff->head;
 	buff->pool = &main_pool->pool;
@@ -809,8 +776,6 @@ int sdp_buff_pool_chain_put(struct sdpc_
 	 * a number of buffers are processed in a loop, before being
 	 * returned. (e.g. send completions, recv to userspace.
 	 */
-	SDP_CHECK_NULL(main_pool, -EINVAL);
-
 	if (NULL == buff || !(0 < count))
 		return -EINVAL;
 
@@ -864,7 +829,6 @@ int sdp_proc_dump_buff_pool(char *buffer
 	unsigned long flags;
 	int offset = 0;
 
-	SDP_CHECK_NULL(buffer, -EINVAL);
 	/*
 	 * simple table read, without page boundry handling.
 	 */
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1871)
+++ sdp_queue.c	(working copy)
@@ -46,8 +46,6 @@ static struct sdpc_desc *_sdp_desc_q_get
 {
 	struct sdpc_desc *element;
 
-	SDP_CHECK_NULL(table, NULL);
-
 	if (NULL == table->head)
 		return NULL;
 
@@ -86,9 +84,6 @@ static inline int _sdp_desc_q_put(struct
 	/*
 	 * fifo: false == tail, true == head
 	 */
-	SDP_CHECK_NULL(table, -EINVAL);
-	SDP_CHECK_NULL(element, -EINVAL);
-
 	if (NULL != element->table)
 		return -EINVAL;
 
@@ -128,11 +123,6 @@ int sdp_desc_q_remove(struct sdpc_desc *
 	struct sdpc_desc *prev;
 	struct sdpc_desc *next;
 
-	SDP_CHECK_NULL(element, -EINVAL);
-	SDP_CHECK_NULL(element->table, -EINVAL);
-	SDP_CHECK_NULL(element->next, -EINVAL);
-	SDP_CHECK_NULL(element->prev, -EINVAL);
-
 	table = element->table;
 
 	if (element->next == element && element->prev == element)
@@ -169,9 +159,6 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
 	struct sdpc_desc *element;
 	int counter;
 
-	SDP_CHECK_NULL(table, NULL);
-	SDP_CHECK_NULL(lookup, NULL);
-
 	for (counter = 0, element = table->head;
 	     counter < table->size; counter++, element = element->next)
 		if (0 == lookup(element, arg))
@@ -219,8 +206,6 @@ int sdp_desc_q_put_tail(struct sdpc_desc
  */
 struct sdpc_desc *sdp_desc_q_look_head(struct sdpc_desc_q *table)
 {
-	SDP_CHECK_NULL(table, NULL);
-
 	return table->head;
 } /* sdp_desc_q_look_head */
 
@@ -229,8 +214,6 @@ struct sdpc_desc *sdp_desc_q_look_head(s
  */
 int sdp_desc_q_type_head(struct sdpc_desc_q *table)
 {
-	SDP_CHECK_NULL(table, -EINVAL);
-
 	if (NULL == table->head)
 		return SDP_DESC_TYPE_NONE;
 	else
@@ -243,8 +226,6 @@ int sdp_desc_q_type_head(struct sdpc_des
 struct sdpc_desc *sdp_desc_q_look_type_head(struct sdpc_desc_q *table, 
 					   enum sdp_desc_type type)
 {
-	SDP_CHECK_NULL(table, NULL);
-
 	if (NULL == table->head)
 		return NULL;
 	else
@@ -257,8 +238,6 @@ struct sdpc_desc *sdp_desc_q_look_type_h
 struct sdpc_desc *sdp_desc_q_look_type_tail(struct sdpc_desc_q *table,
 					   enum sdp_desc_type type)
 {
-	SDP_CHECK_NULL(table, NULL);
-
 	if (NULL == table->head)
 		return NULL;
 	else
@@ -272,8 +251,6 @@ struct sdpc_desc *sdp_desc_q_look_type_t
 int sdp_desc_q_types_size(struct sdpc_desc_q *table,
 				 enum sdp_desc_type type)
 {
-	SDP_CHECK_NULL(table, -EINVAL);
-
 	return ((SDP_DESC_TYPE_NONE > type) ?
 		table->count[type] : -ERANGE);
 } /* sdp_desc_q_types_size */
@@ -283,8 +260,6 @@ int sdp_desc_q_types_size(struct sdpc_de
  */
 int sdp_desc_q_init(struct sdpc_desc_q *table)
 {
-	SDP_CHECK_NULL(table, -EINVAL);
-
 	table->head = NULL;
 	table->size = 0;
 
@@ -301,7 +276,6 @@ int sdp_desc_q_clear(struct sdpc_desc_q 
 	struct sdpc_desc *element;
 	int result;
 
-	SDP_CHECK_NULL(table, -EINVAL);
 	/*
 	 * drain the table of any objects
 	 */
Index: sdp_post.c
===================================================================
--- sdp_post.c	(revision 1871)
+++ sdp_post.c	(working copy)
@@ -41,8 +41,6 @@ int sdp_cm_listen_start(struct sdev_root
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(dev_root, -EINVAL);
-
 	if (NULL != dev_root->listen_id)
 		sdp_dbg_warn(NULL, "Already listening for connections.");
 
@@ -75,8 +73,6 @@ int sdp_cm_listen_stop(struct sdev_root 
 {
 	int result = 0;
 
-	SDP_CHECK_NULL(dev_root, -EINVAL);
-
 	result = ib_destroy_cm_id(dev_root->listen_id);
 	if (0 > result)
 		sdp_dbg_warn(NULL, "Error <%d> stopping listen", result);
@@ -197,7 +193,6 @@ static void _sdp_cm_failed(void *arg)
 static int _sdp_cm_deferred_generic(struct sdp_opt *conn,
 				    void (*defer_func)(void *arg))
 {
-	SDP_CHECK_NULL(conn, -EINVAL);
 	/*
 	 * send a potentially defered failed request.
 	 */
-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb 23 03:02:43 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 13:02:43 +0200
Subject: [openib-general] sdp_kvec.c
Message-ID: <20050223110243.GA23837@mellanox.co.il>

Libor, shall not the code in sdp_kvec.c go into
sdp_iocb.c? I dont see kvec mentionned even once in there ...

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Wed Feb 23 05:02:05 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 15:02:05 +0200
Subject: [openib-general] [PATCH] no qp lock in poll,
	separate locks for send and receive
In-Reply-To: <52zmxwpg4p.fsf@topspin.com>
References: <20050203175419.GJ2536@mellanox.co.il> <52ekf8sfn3.fsf@topspin.com>
	<20050222184055.GA25382@mellanox.co.il>
	<52zmxwpg4p.fsf@topspin.com>
Message-ID: <20050223130205.GH23837@mellanox.co.il>



Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] 1/2 separate locking for send and receive q in mthca
> 
>     Michael> Instead of doing this, how about something else I would
>     Michael> prefer: we could avoid locking the QP on CQ poll
>     Michael> altogether, if there is a separate last polled index that
>     Michael> is written only by cq poll and read by qp post.  This
>     Michael> index update would be protected by a cq lock.
> 
> Yes, that's a greate solution.  Splitting things up so that some
> values are only written inside the WQ lock and some are only written
> inside the CQ lock is a very clean solution.
> 
>  - R.
> 

I seem to get some good speedups (5-10%) in IP over IB from the following
patch, but I think its an obvious win for any ULP.

This patch does the following:
1. Split the QP spinlock to send and receive lock.
The only place where we have to lock both is upon modify_qp,
and that is not on data path.

2. Avoid taking any QP locks when polling CQ.

This last part is achieved by getting rid of the cur field in
mthca_wq, and calculating the number of outstanding WQEs by comparing
the last_comp and next fields. next is only updated by post,
last_comp is only updated by poll.

In a rare case where an overrun is detected, a CQ is locked
and the overrun condition is re-tested, to avoid any potential
for stale last_comp value.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 1862)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -147,8 +147,8 @@ struct mthca_cq {
 };
 
 struct mthca_wq {
+	spinlock_t lock;
 	int   max;
-	int   cur;
 	int   next;
 	int   last_comp;
 	void *last;
@@ -158,7 +158,6 @@ struct mthca_wq {
 
 struct mthca_qp {
 	struct ib_qp           ibqp;
-	spinlock_t             lock;
 	atomic_t               refcount;
 	u32                    qpn;
 	int                    is_direct;
Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 1862)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -405,15 +405,6 @@ static inline int mthca_poll_one(struct 
 	is_send  = is_error ? cqe->opcode & 0x01 : cqe->is_send & 0x80;
 
 	if (!*cur_qp || be32_to_cpu(cqe->my_qpn) != (*cur_qp)->qpn) {
-		if (*cur_qp) {
-			if (*freed) {
-				wmb();
-				inc_cons_index(dev, cq, *freed);
-				*freed = 0;
-			}
-			spin_unlock(&(*cur_qp)->lock);
-		}
-
 		/*
 		 * We do not have to take the QP table lock here,
 		 * because CQs will be locked while QPs are removed
@@ -428,8 +419,6 @@ static inline int mthca_poll_one(struct 
 			err = -EINVAL;
 			goto out;
 		}
-
-		spin_lock(&(*cur_qp)->lock);
 	}
 
 	entry->qp_num = (*cur_qp)->qpn;
@@ -446,11 +435,6 @@ static inline int mthca_poll_one(struct 
 		entry->wr_id = (*cur_qp)->wrid[wqe_index];
 	}
 
-	if (wq->last_comp < wqe_index)
-		wq->cur -= wqe_index - wq->last_comp;
-	else
-		wq->cur -= wq->max - wq->last_comp + wqe_index;
-
 	wq->last_comp = wqe_index;
 
 	if (0)
@@ -533,9 +517,6 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 		inc_cons_index(dev, cq, freed);
 	}
 
-	if (qp)
-		spin_unlock(&qp->lock);
-
 	spin_unlock_irqrestore(&cq->lock, flags);
 
 	return err == 0 || err == -EAGAIN ? npolled : err;
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 1862)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -552,9 +552,11 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 		else
 			cur_state = attr->cur_qp_state;
 	} else {
-		spin_lock_irq(&qp->lock);
+		spin_lock_irq(&qp->sq.lock);
+		spin_lock(&qp->rq.lock);
 		cur_state = qp->state;
-		spin_unlock_irq(&qp->lock);
+		spin_unlock(&qp->rq.lock);
+		spin_unlock_irq(&qp->sq.lock);
 	}
 
 	if (attr_mask & IB_QP_STATE) {
@@ -972,6 +974,14 @@ static int mthca_alloc_wqe_buf(struct mt
 	return err;
 }
 
+static void mthca_wq_init(struct mthca_wq* wq)
+{
+	spin_lock_init(&wq->lock);
+	wq->next      = 0;
+	wq->last_comp = wq->max - 1;
+	wq->last      = NULL;
+}
+
 static int mthca_alloc_qp_common(struct mthca_dev *dev,
 				 struct mthca_pd *pd,
 				 struct mthca_cq *send_cq,
@@ -981,20 +991,13 @@ static int mthca_alloc_qp_common(struct 
 {
 	int err;
 
-	spin_lock_init(&qp->lock);
 	atomic_set(&qp->refcount, 1);
 	qp->state    	 = IB_QPS_RESET;
 	qp->atomic_rd_en = 0;
 	qp->resp_depth   = 0;
 	qp->sq_policy    = send_policy;
-	qp->rq.cur       = 0;
-	qp->sq.cur       = 0;
-	qp->rq.next      = 0;
-	qp->sq.next      = 0;
-	qp->rq.last_comp = qp->rq.max - 1;
-	qp->sq.last_comp = qp->sq.max - 1;
-	qp->rq.last      = NULL;
-	qp->sq.last      = NULL;
+	mthca_wq_init(&qp->sq);
+	mthca_wq_init(&qp->rq);
 
 	err = mthca_alloc_wqe_buf(dev, pd, qp);
 	return err;
@@ -1240,6 +1243,24 @@ static int build_mlx_header(struct mthca
 	return 0;
 }
 
+static inline int mthca_wq_overflow(struct mthca_wq* wq, int nreq,
+	 struct ib_cq* ib_cq)
+{
+	int cur;
+	struct mthca_cq* cq;
+
+	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	if (likely(cur + nreq < wq->max))
+		return 0;
+
+	cq = to_mcq(ib_cq);
+	spin_lock(&cq->lock);
+	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	spin_unlock(&cq->lock);
+
+	return cur + nreq >= wq->max;
+}
+
 int mthca_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
 		    struct ib_send_wr **bad_wr)
 {
@@ -1267,16 +1288,18 @@ int mthca_post_send(struct ib_qp *ibqp, 
 		[IB_WR_ATOMIC_FETCH_AND_ADD] = MTHCA_OPCODE_ATOMIC_FA,
 	};
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->sq.lock, flags);
 
 	/* XXX check that state is OK to post send */
 
 	ind = qp->sq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->sq.cur + nreq >= qp->sq.max) {
-			mthca_err(dev, "SQ full (%d posted, %d max, %d nreq)\n",
-				  qp->sq.cur, qp->sq.max, nreq);
+		if (mthca_wq_overflow(&qp->sq, nreq, qp->ibqp.send_cq)) {
+			mthca_err(dev, "SQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->sq.next, qp->sq.last_comp,
+					qp->sq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1447,10 +1470,9 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->sq.cur += nreq;
 	qp->sq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->sq.lock, flags);
 	return err;
 }
 
@@ -1469,15 +1491,18 @@ int mthca_post_receive(struct ib_qp *ibq
 	void *wqe;
 	void *prev_wqe;
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->rq.lock, flags);
 
 	/* XXX check that state is OK to post receive */
 
 	ind = qp->rq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->rq.cur + nreq >= qp->rq.max) {
-			mthca_err(dev, "RQ %06x full\n", qp->qpn);
+		if (mthca_wq_overflow(&qp->rq, nreq, qp->ibqp.recv_cq)) {
+			mthca_err(dev, "RQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->rq.next, qp->rq.last_comp,
+					qp->rq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1544,10 +1569,9 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->rq.cur += nreq;
 	qp->rq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->rq.lock, flags);
 	return err;
 }
 

-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Wed Feb 23 11:11:37 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 11:11:37 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Whitespace cleanup in SDP
In-Reply-To: <52wtt0b3it.fsf@topspin.com>;
	from roland@topspin.com on Tue, Feb 22, 2005 at 03:04:10PM -0800
References: <11087738623823@sun.com> <20050222122058.A23420@topspin.com>
	<1109105820.24909.48.camel@duffman>
	<20050222141603.G23420@topspin.com> <521xb8cjav.fsf@topspin.com>
	<20050222145236.H23420@topspin.com> <52wtt0b3it.fsf@topspin.com>
Message-ID: <20050223111137.M23420@topspin.com>

On Tue, Feb 22, 2005 at 03:04:10PM -0800, Roland Dreier wrote:
> OK, here's the patch.
> 

  Thanks, I've updated and commited the code.

-Libor

> Cast u64 values to unsigned long long inside sdp_dbg_data() to get rid
> of compiler warnings on 64-bit archs when building with
> INFINIBAND_SDP_DEBUG_DATA turned on.
> 
> Signed-off-by: Roland Dreier <roland at topspin.com>
> 
> Index: infiniband/ulp/sdp/sdp_write.c
> ===================================================================
> --- infiniband/ulp/sdp/sdp_write.c	(revision 1867)
> +++ infiniband/ulp/sdp/sdp_write.c	(working copy)
> @@ -74,7 +74,7 @@ int sdp_event_write(struct sdp_opt *conn
>  	}
>  
>  	sdp_dbg_data(conn, "Write complete <%llu> of <%u> bytes.",
> -		     comp->wr_id, comp->byte_len);
> +		     (unsigned long long) comp->wr_id, comp->byte_len);
>  	/*
>  	 * Four basic scenarios:
>  	 *
> Index: infiniband/ulp/sdp/sdp_rcvd.c
> ===================================================================
> --- infiniband/ulp/sdp/sdp_rcvd.c	(revision 1867)
> +++ infiniband/ulp/sdp/sdp_rcvd.c	(working copy)
> @@ -729,7 +729,8 @@ static int _sdp_rcvd_snk_avail(struct sd
>  	_sdp_msg_net_to_cpu_snkah(snkah);
>  
>  	sdp_dbg_data(conn, "SnkAvail received. <%d:%d:%016llx> mode <%d>",
> -		     snkah->size, snkah->r_key, snkah->addr, conn->send_mode);
> +		     snkah->size, snkah->r_key, (unsigned long long) snkah->addr,
> +		     conn->send_mode);
>  	/*
>  	 * check our send mode, and make sure parameters are within reason.
>  	 */
> @@ -873,8 +874,8 @@ static int _sdp_rcvd_src_avail(struct sd
>  	size = buff->tail - buff->data;
>  
>  	sdp_dbg_data(conn, "SrcAvail received. <%d:%d:%d:%016llx> mode <%d>",
> -		     srcah->size, srcah->r_key, size, srcah->addr,
> -		     conn->recv_mode);
> +		     srcah->size, srcah->r_key, size,
> +		     (unsigned long long) srcah->addr, conn->recv_mode);
>  
>  	if (0 < conn->snk_sent) {
>  		/*
> Index: infiniband/ulp/sdp/sdp_read.c
> ===================================================================
> --- infiniband/ulp/sdp/sdp_read.c	(revision 1867)
> +++ infiniband/ulp/sdp/sdp_read.c	(working copy)
> @@ -142,7 +142,7 @@ int sdp_event_read(struct sdp_opt *conn,
>  	}
>  
>  	sdp_dbg_data(conn, "Read complete <%llu> of <%u> bytes.",
> -		     comp->wr_id, comp->byte_len);
> +		     (unsigned long long) comp->wr_id, comp->byte_len);
>  	/*
>  	 * update queue depth
>  	 */
> Index: infiniband/ulp/sdp/sdp_send.c
> ===================================================================
> --- infiniband/ulp/sdp/sdp_send.c	(revision 1867)
> +++ infiniband/ulp/sdp/sdp_send.c	(working copy)
> @@ -372,7 +372,7 @@ static int _sdp_send_data_buff_snk(struc
>  	advt->post += (buff->tail - buff->data);
>  
>  	sdp_dbg_data(conn, "POST Write BUFF wrid <%llu> bytes <%u:%d>.",
> -		     buff->wrid, (unsigned)(buff->tail - buff->data),
> +		     (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data),
>  		     advt->size);
>  	/*
>  	 * dequeue if needed and the queue buffer
> @@ -502,9 +502,8 @@ static int _sdp_send_data_iocb_snk(struc
>  		advt->addr += zcopy;
>  		advt->post += zcopy;
>  
> -		sdp_dbg_data(conn, 
> -			     "POST Write IOCB wrid <%llu> bytes <%u:%d:%d>.",
> -			     iocb->wrid, zcopy, iocb->len, advt->size);
> +		sdp_dbg_data(conn, "POST Write IOCB wrid <%llu> bytes <%u:%d:%d>.",
> +			     (unsigned long long) iocb->wrid, zcopy, iocb->len, advt->size);
>  		/*
>  		 * update send queue depth
>  		 */
> Index: infiniband/ulp/sdp/sdp_recv.c
> ===================================================================
> --- infiniband/ulp/sdp/sdp_recv.c	(revision 1867)
> +++ infiniband/ulp/sdp/sdp_recv.c	(working copy)
> @@ -82,7 +82,7 @@ static int _sdp_post_recv_buff(struct sd
>  	}
>  
>  	sdp_dbg_data(conn, "POST RECV BUFF wrid <%llu> of <%u> bytes.",
> -		     buff->wrid, (unsigned)(buff->tail - buff->data));
> +		     (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data));
>  	/*
>  	 * post recv
>  	 */
> @@ -203,7 +203,7 @@ static int _sdp_post_rdma_buff(struct sd
>  	}
>  
>  	sdp_dbg_data(conn, "POST READ BUFF wrid <%llu> of <%u> bytes.",
> -		     buff->wrid, (unsigned)(buff->tail - buff->data));
> +		     (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data));
>  	/*
>  	 * update send queue depth
>  	 */
> @@ -353,7 +353,7 @@ static int _sdp_post_rdma_iocb_src(struc
>  	}
>  
>  	sdp_dbg_data(conn, "POST READ IOCB wrid <%llu> bytes <%u:%d:%d>.",
> -		 iocb->wrid, zcopy, iocb->len, advt->size);
> +		     (unsigned long long) iocb->wrid, zcopy, iocb->len, advt->size);
>  	/*
>  	 * update send queue depth
>  	 */


From libor at topspin.com  Wed Feb 23 11:26:24 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 11:26:24 -0800
Subject: [openib-general] Re: SDP: BUG 2034 workaround
In-Reply-To: <20050223094924.GT23837@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Feb 23, 2005 at 11:49:25AM +0200
References: <20050222101445.GK23837@mellanox.co.il>
	<20050222185934.K23420@topspin.com>
	<20050223094924.GT23837@mellanox.co.il>
Message-ID: <20050223112624.N23420@topspin.com>

On Wed, Feb 23, 2005 at 11:49:25AM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: SDP: BUG 2034 workaround
> > 
> > On Tue, Feb 22, 2005 at 12:14:45PM +0200, Michael S. Tsirkin wrote:
> > > 
> > > sdp_inet.c, inside _sdp_inet_listen, we have:
> > > 
> > > #if 0                           /* BUG 2034 workaround. */
> > >         conn->backlog_max = backlog;
> > > #else
> > >         conn->backlog_max = 1024;
> > > #endif
> > > 
> > > What gives? what would be the proper fix as opposed to a work-around?
> > 
> >   The Linux TCP listen uses two seperate values to control the listner
> > backlog, so I ignored the backlog parameter to get closer to the default
> > Linux behavior, until a really solution is devised.
> >  
> >   Basically TCP uses a two stage backlog to defend against SYN attacks.
> > When a SYN is received a small amount of state is kept until the full
> > handshake is completed, at which point a full socket is created and
> > queued onto the listen sockets accept queue. The second stage uses the
> > listen() backlog parameter to manage the accept queue. The first stage
> > queue size is managed using a sysctl, (net.pv4.tcp_max_syn_backlog)
> > which on a lot of systems defaults to 1024.
> > 
> >   SDP on the other hand creates the full socket on the REQ request, and
> > places it directly into the listen sockets accept queue. So the full
> > depth of the queue is governed by the listen backlog parameter. Worse
> > still the socket layer listen, above TCP and SDP, limits the listen()
> > backlog that is passed to the protocol to 128, and only recently did this
> > become adjustable using net.core.somaxconn. 
> > 
> >    Using just the backlog parameter to manage this queue, results in a
> > few programs that use a high connection volume to get rejected connection,
> > casued by a full backlog queue, even when upped to the full 128 that the
> > socket layer allows by default.
> 
> I see. A protocol bug.
> I wander if some application could get confused if it
> gets more connections it set the backlog up for. What do you think?

  I've not seen an application that has a hard reliance on the backlog
limit, while I have seen applications that complain when the limit is
smaller then what they're expecting. So I think as long as the behavior
is an approximation of the TCP behavior, it's better to make the queue
too big rather then too small.

> For now, a somewhat cleaner work-around would be:
> 1. Replace the /* BUG 2034 workaround. */ comment with the excellent
>    explanation above.
> 2. Take the max value between the socket backlog and sysctl_max_syn_backlog.

  That would be a better workaround, or you could use the sum of the two
values. At this point it would be overkill to use two seperate accept
queues for passive connections, one for established connections and one
for partial connections.

> Want a patch like that?

  Yes, that would be good.

Thanks.

-Libor


From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][1/6] Clean up memory region handling
In-Reply-To: <20052231131.JCngKDtSiEvKkiUy@topspin.com>
Message-ID: <20052231131.bGOfsJAboMSLQ3J1@topspin.com>

Move the request/ioremap of regions related to event handling into
mthca_eq.c.  Map the correct regions depending on whether we're in
Tavor or native mem-free mode.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: src/linux-kernel/infiniband/hw/mthca/mthca_config_reg.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_config_reg.h	2005-02-22 19:46:48.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_config_reg.h	2005-02-22 19:48:23.000000000 -0800
@@ -46,5 +46,6 @@
 #define MTHCA_MAP_ECR_SIZE     (MTHCA_ECR_SIZE + MTHCA_ECR_CLR_SIZE)
 #define MTHCA_CLR_INT_BASE     0xf00d8
 #define MTHCA_CLR_INT_SIZE     0x00008
+#define MTHCA_EQ_SET_CI_SIZE   (8 * 32)
 
 #endif /* MTHCA_CONFIG_REG_H */
Index: src/linux-kernel/infiniband/hw/mthca/mthca_dev.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_dev.h	2005-02-22 19:46:46.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_dev.h	2005-02-22 19:48:22.000000000 -0800
@@ -237,9 +237,17 @@
 	struct semaphore cap_mask_mutex;
 
 	void __iomem    *hcr;
-	void __iomem    *ecr_base;
-	void __iomem    *clr_base;
 	void __iomem    *kar;
+	void __iomem    *clr_base;
+	union {
+		struct {
+			void __iomem *ecr_base;
+		} tavor;
+		struct {
+			void __iomem *eq_arm;
+			void __iomem *eq_set_ci_base;
+		} arbel;
+	} eq_regs;
 
 	struct mthca_cmd    cmd;
 	struct mthca_limits limits;
Index: src/linux-kernel/infiniband/hw/mthca/mthca_eq.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-22 19:46:47.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-22 19:48:23.000000000 -0800
@@ -366,10 +366,10 @@
 	if (dev->eq_table.clr_mask)
 		writel(dev->eq_table.clr_mask, dev->eq_table.clr_int);
 
-	if ((ecr = readl(dev->ecr_base + 4)) != 0) {
+	if ((ecr = readl(dev->eq_regs.tavor.ecr_base + 4)) != 0) {
 		work = 1;
 
-		writel(ecr, dev->ecr_base +
+		writel(ecr, dev->eq_regs.tavor.ecr_base +
 		       MTHCA_ECR_CLR_BASE - MTHCA_ECR_BASE + 4);
 
 		for (i = 0; i < MTHCA_NUM_EQ; ++i)
@@ -578,6 +578,129 @@
 				 dev->eq_table.eq + i);
 }
 
+static int __devinit mthca_map_reg(struct mthca_dev *dev,
+				   unsigned long offset, unsigned long size,
+				   void __iomem **map)
+{
+	unsigned long base = pci_resource_start(dev->pdev, 0);
+
+	if (!request_mem_region(base + offset, size, DRV_NAME))
+		return -EBUSY;
+
+	*map = ioremap(base + offset, size);
+	if (!*map) {
+		release_mem_region(base + offset, size);
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static void mthca_unmap_reg(struct mthca_dev *dev, unsigned long offset,
+			    unsigned long size, void __iomem *map)
+{
+	unsigned long base = pci_resource_start(dev->pdev, 0);
+
+	release_mem_region(base + offset, size);
+	iounmap(map);
+}
+
+static int __devinit mthca_map_eq_regs(struct mthca_dev *dev)
+{
+	unsigned long mthca_base;
+
+	mthca_base = pci_resource_start(dev->pdev, 0);
+
+	if (dev->hca_type == ARBEL_NATIVE) {
+		/*
+		 * We assume that the EQ arm and EQ set CI registers
+		 * fall within the first BAR.  We can't trust the
+		 * values firmware gives us, since those addresses are
+		 * valid on the HCA's side of the PCI bus but not
+		 * necessarily the host side.
+		 */
+		if (mthca_map_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) &
+				  dev->fw.arbel.clr_int_base, MTHCA_CLR_INT_SIZE,
+				  &dev->clr_base)) {
+			mthca_err(dev, "Couldn't map interrupt clear register, "
+				  "aborting.\n");
+			return -ENOMEM;
+		}
+
+		/*
+		 * Add 4 because we limit ourselves to EQs 0 ... 31,
+		 * so we only need the low word of the register.
+		 */
+		if (mthca_map_reg(dev, ((pci_resource_len(dev->pdev, 0) - 1) &
+					dev->fw.arbel.eq_arm_base) + 4, 4,
+				  &dev->eq_regs.arbel.eq_arm)) {
+			mthca_err(dev, "Couldn't map interrupt clear register, "
+				  "aborting.\n");
+			mthca_unmap_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) &
+					dev->fw.arbel.clr_int_base, MTHCA_CLR_INT_SIZE,
+					dev->clr_base);
+			return -ENOMEM;
+		}
+
+		if (mthca_map_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) &
+				  dev->fw.arbel.eq_set_ci_base,
+				  MTHCA_EQ_SET_CI_SIZE,
+				  &dev->eq_regs.arbel.eq_set_ci_base)) {
+			mthca_err(dev, "Couldn't map interrupt clear register, "
+				  "aborting.\n");
+			mthca_unmap_reg(dev, ((pci_resource_len(dev->pdev, 0) - 1) &
+					      dev->fw.arbel.eq_arm_base) + 4, 4,
+					dev->eq_regs.arbel.eq_arm);
+			mthca_unmap_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) &
+					dev->fw.arbel.clr_int_base, MTHCA_CLR_INT_SIZE,
+					dev->clr_base);
+			return -ENOMEM;
+		}
+	} else {
+		if (mthca_map_reg(dev, MTHCA_CLR_INT_BASE, MTHCA_CLR_INT_SIZE,
+				  &dev->clr_base)) {
+			mthca_err(dev, "Couldn't map interrupt clear register, "
+				  "aborting.\n");
+			return -ENOMEM;
+		}
+
+		if (mthca_map_reg(dev, MTHCA_ECR_BASE,
+				  MTHCA_ECR_SIZE + MTHCA_ECR_CLR_SIZE,
+				  &dev->eq_regs.tavor.ecr_base)) {
+			mthca_err(dev, "Couldn't map ecr register, "
+				  "aborting.\n");
+			mthca_unmap_reg(dev, MTHCA_CLR_INT_BASE, MTHCA_CLR_INT_SIZE,
+					dev->clr_base);
+			return -ENOMEM;
+		}
+	}
+
+	return 0;
+
+}
+
+static void __devexit mthca_unmap_eq_regs(struct mthca_dev *dev)
+{
+	if (dev->hca_type == ARBEL_NATIVE) {
+		mthca_unmap_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) &
+				dev->fw.arbel.eq_set_ci_base,
+				MTHCA_EQ_SET_CI_SIZE,
+				dev->eq_regs.arbel.eq_set_ci_base);
+		mthca_unmap_reg(dev, ((pci_resource_len(dev->pdev, 0) - 1) &
+				      dev->fw.arbel.eq_arm_base) + 4, 4,
+				dev->eq_regs.arbel.eq_arm);
+		mthca_unmap_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) &
+				dev->fw.arbel.clr_int_base, MTHCA_CLR_INT_SIZE,
+				dev->clr_base);
+	} else {
+		mthca_unmap_reg(dev, MTHCA_ECR_BASE,
+				MTHCA_ECR_SIZE + MTHCA_ECR_CLR_SIZE,
+				dev->eq_regs.tavor.ecr_base);
+		mthca_unmap_reg(dev, MTHCA_CLR_INT_BASE, MTHCA_CLR_INT_SIZE,
+				dev->clr_base);
+	}
+}
+
 int __devinit mthca_map_eq_icm(struct mthca_dev *dev, u64 icm_virt)
 {
 	int ret;
@@ -636,6 +759,10 @@
 	if (err)
 		return err;
 
+	err = mthca_map_eq_regs(dev);
+	if (err)
+		goto err_out_free;
+
 	if (dev->mthca_flags & MTHCA_FLAG_MSI ||
 	    dev->mthca_flags & MTHCA_FLAG_MSI_X) {
 		dev->eq_table.clr_mask = 0;
@@ -653,7 +780,7 @@
 			      (dev->mthca_flags & MTHCA_FLAG_MSI_X) ? 128 : intr,
 			      &dev->eq_table.eq[MTHCA_EQ_COMP]);
 	if (err)
-		goto err_out_free;
+		goto err_out_unmap;
 
 	err = mthca_create_eq(dev, MTHCA_NUM_ASYNC_EQE,
 			      (dev->mthca_flags & MTHCA_FLAG_MSI_X) ? 129 : intr,
@@ -720,6 +847,9 @@
 err_out_comp:
 	mthca_free_eq(dev, &dev->eq_table.eq[MTHCA_EQ_COMP]);
 
+err_out_unmap:
+	mthca_unmap_eq_regs(dev);
+
 err_out_free:
 	mthca_alloc_cleanup(&dev->eq_table.alloc);
 	return err;
@@ -740,5 +870,7 @@
 	for (i = 0; i < MTHCA_NUM_EQ; ++i)
 		mthca_free_eq(dev, &dev->eq_table.eq[i]);
 
+	mthca_unmap_eq_regs(dev);
+
 	mthca_alloc_cleanup(&dev->eq_table.alloc);
 }
Index: src/linux-kernel/infiniband/hw/mthca/mthca_main.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-22 19:46:46.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-22 19:48:23.000000000 -0800
@@ -686,37 +686,18 @@
 	int err;
 
 	/*
-	 * We request our first BAR in two chunks, since the MSI-X
-	 * vector table is right in the middle.
+	 * We can't just use pci_request_regions() because the MSI-X
+	 * table is right in the middle of the first BAR.  If we did
+	 * pci_request_region and grab all of the first BAR, then
+	 * setting up MSI-X would fail, since the PCI core wants to do
+	 * request_mem_region on the MSI-X vector table.
 	 *
-	 * This is why we can't just use pci_request_regions() -- if
-	 * we did then setting up MSI-X would fail, since the PCI core
-	 * wants to do request_mem_region on the MSI-X vector table.
+	 * So just request what we need right now, and request any
+	 * other regions we need when setting up EQs.
 	 */
-	if (!request_mem_region(pci_resource_start(pdev, 0) +
-				MTHCA_HCR_BASE,
-				MTHCA_HCR_SIZE,
-				DRV_NAME)) {
-		err = -EBUSY;
-		goto err_hcr_failed;
-	}
-
-	if (!request_mem_region(pci_resource_start(pdev, 0) +
-				MTHCA_ECR_BASE,
-				MTHCA_MAP_ECR_SIZE,
-				DRV_NAME)) {
-		err = -EBUSY;
-		goto err_ecr_failed;
-	}
-
-	if (!request_mem_region(pci_resource_start(pdev, 0) +
-				MTHCA_CLR_INT_BASE,
-				MTHCA_CLR_INT_SIZE,
-				DRV_NAME)) {
-		err = -EBUSY;
-		goto err_int_failed;
-	}
-
+	if (!request_mem_region(pci_resource_start(pdev, 0) + MTHCA_HCR_BASE,
+				MTHCA_HCR_SIZE, DRV_NAME))
+		return -EBUSY;
 
 	err = pci_request_region(pdev, 2, DRV_NAME);
 	if (err)
@@ -731,24 +712,11 @@
 	return 0;
 
 err_bar4_failed:
-
 	pci_release_region(pdev, 2);
-err_bar2_failed:
-
-	release_mem_region(pci_resource_start(pdev, 0) +
-			   MTHCA_CLR_INT_BASE,
-			   MTHCA_CLR_INT_SIZE);
-err_int_failed:
-
-	release_mem_region(pci_resource_start(pdev, 0) +
-			   MTHCA_ECR_BASE,
-			   MTHCA_MAP_ECR_SIZE);
-err_ecr_failed:
 
-	release_mem_region(pci_resource_start(pdev, 0) +
-			   MTHCA_HCR_BASE,
+err_bar2_failed:
+	release_mem_region(pci_resource_start(pdev, 0) + MTHCA_HCR_BASE,
 			   MTHCA_HCR_SIZE);
-err_hcr_failed:
 
 	return err;
 }
@@ -761,16 +729,7 @@
 
 	pci_release_region(pdev, 2);
 
-	release_mem_region(pci_resource_start(pdev, 0) +
-			   MTHCA_CLR_INT_BASE,
-			   MTHCA_CLR_INT_SIZE);
-
-	release_mem_region(pci_resource_start(pdev, 0) +
-			   MTHCA_ECR_BASE,
-			   MTHCA_MAP_ECR_SIZE);
-
-	release_mem_region(pci_resource_start(pdev, 0) +
-			   MTHCA_HCR_BASE,
+	release_mem_region(pci_resource_start(pdev, 0) + MTHCA_HCR_BASE,
 			   MTHCA_HCR_SIZE);
 }
 
@@ -941,31 +900,13 @@
 		goto err_free_dev;
 	}
 
-	mdev->clr_base = ioremap(mthca_base + MTHCA_CLR_INT_BASE,
-				 MTHCA_CLR_INT_SIZE);
-	if (!mdev->clr_base) {
-		mthca_err(mdev, "Couldn't map interrupt clear register, "
-			  "aborting.\n");
-		err = -ENOMEM;
-		goto err_iounmap;
-	}
-
-	mdev->ecr_base = ioremap(mthca_base + MTHCA_ECR_BASE,
-				 MTHCA_ECR_SIZE + MTHCA_ECR_CLR_SIZE);
-	if (!mdev->ecr_base) {
-		mthca_err(mdev, "Couldn't map ecr register, "
-			  "aborting.\n");
-		err = -ENOMEM;
-		goto err_iounmap_clr;
-	}
-
 	mthca_base = pci_resource_start(pdev, 2);
 	mdev->kar = ioremap(mthca_base + PAGE_SIZE * MTHCA_KAR_PAGE, PAGE_SIZE);
 	if (!mdev->kar) {
 		mthca_err(mdev, "Couldn't map kernel access region, "
 			  "aborting.\n");
 		err = -ENOMEM;
-		goto err_iounmap_ecr;
+		goto err_iounmap;
 	}
 
 	err = mthca_tune_pci(mdev);
@@ -1014,12 +955,6 @@
 err_iounmap_kar:
 	iounmap(mdev->kar);
 
-err_iounmap_ecr:
-	iounmap(mdev->ecr_base);
-
-err_iounmap_clr:
-	iounmap(mdev->clr_base);
-
 err_iounmap:
 	iounmap(mdev->hcr);
 
@@ -1067,9 +1002,8 @@
 
 		mthca_close_hca(mdev);
 
+		iounmap(mdev->kar);
 		iounmap(mdev->hcr);
-		iounmap(mdev->ecr_base);
-		iounmap(mdev->clr_base);
 
 		if (mdev->mthca_flags & MTHCA_FLAG_MSI_X)
 			pci_disable_msix(pdev);



From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][0/6] Mem-free merge patches
Message-ID: <20052231131.JCngKDtSiEvKkiUy@topspin.com>

Here's the first batch of patches that merges mem-free support from
the roland-uverbs branch onto the trunk.  It cuts the mthca diff
between the branch and the trunk from ~3700 lines down to ~2300 lines
(of which some is userspace support).

For Michael or others who are wondering how I plan to merge mem-free
support, this should show the plan.  We need to split the changes into
manageable, independent chunks.  This makes review possible, and also
makes it possible to use the history to search for the responsible
patch if, heaven forfend, this introduces a regression.

 - R.



From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][2/6] Add support for allocating UAR
In-Reply-To: <20052231131.bGOfsJAboMSLQ3J1@topspin.com>
Message-ID: <20052231131.boGPEDrKIu7IxRyk@topspin.com>

Add support for allocating user access regions (UARs).  Use this to
allocate a region for kernel at driver init instead using hard-coded
MTHCA_KAR_PAGE index.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: src/linux-kernel/infiniband/hw/mthca/Makefile
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/Makefile	2005-02-23 11:16:38.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/Makefile	2005-02-23 11:16:42.000000000 -0800
@@ -9,4 +9,4 @@
 ib_mthca-y :=	mthca_main.o mthca_cmd.o mthca_profile.o mthca_reset.o \
 		mthca_allocator.o mthca_eq.o mthca_pd.o mthca_cq.o \
 		mthca_mr.o mthca_qp.o mthca_av.o mthca_mcg.o mthca_mad.o \
-		mthca_provider.o mthca_memfree.o
+		mthca_provider.o mthca_memfree.o mthca_uar.o
Index: src/linux-kernel/infiniband/hw/mthca/mthca_cq.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_cq.c	2005-02-23 11:16:38.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_cq.c	2005-02-23 11:16:43.000000000 -0800
@@ -666,7 +666,7 @@
 						  MTHCA_CQ_FLAG_TR);
 	cq_context->start           = cpu_to_be64(0);
 	cq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24 |
-						  MTHCA_KAR_PAGE);
+						  dev->driver_uar.index);
 	cq_context->error_eqn       = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_ASYNC].eqn);
 	cq_context->comp_eqn        = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_COMP].eqn);
 	cq_context->pd              = cpu_to_be32(dev->driver_pd.pd_num);
Index: src/linux-kernel/infiniband/hw/mthca/mthca_dev.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_dev.h	2005-02-23 11:16:41.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_dev.h	2005-02-23 11:16:47.000000000 -0800
@@ -65,7 +65,6 @@
 };
 
 enum {
-	MTHCA_KAR_PAGE  = 1,
 	MTHCA_MAX_PORTS = 2
 };
 
@@ -108,6 +107,7 @@
 	int      gid_table_len;
 	int      pkey_table_len;
 	int      local_ca_ack_delay;
+	int      num_uars;
 	int      max_sg;
 	int      num_qps;
 	int      reserved_qps;
@@ -148,6 +148,12 @@
 	} *page_list;
 };
 
+struct mthca_uar_table {
+	struct mthca_alloc alloc;
+	u64                uarc_base;
+	int                uarc_size;
+};
+
 struct mthca_pd_table {
 	struct mthca_alloc alloc;
 };
@@ -252,6 +258,7 @@
 	struct mthca_cmd    cmd;
 	struct mthca_limits limits;
 
+	struct mthca_uar_table uar_table;
 	struct mthca_pd_table  pd_table;
 	struct mthca_mr_table  mr_table;
 	struct mthca_eq_table  eq_table;
@@ -260,6 +267,7 @@
 	struct mthca_av_table  av_table;
 	struct mthca_mcg_table mcg_table;
 
+	struct mthca_uar      driver_uar;
 	struct mthca_pd       driver_pd;
 	struct mthca_mr       driver_mr;
 
@@ -318,6 +326,7 @@
 int mthca_array_init(struct mthca_array *array, int nent);
 void mthca_array_cleanup(struct mthca_array *array, int nent);
 
+int mthca_init_uar_table(struct mthca_dev *dev);
 int mthca_init_pd_table(struct mthca_dev *dev);
 int mthca_init_mr_table(struct mthca_dev *dev);
 int mthca_init_eq_table(struct mthca_dev *dev);
@@ -326,6 +335,7 @@
 int mthca_init_av_table(struct mthca_dev *dev);
 int mthca_init_mcg_table(struct mthca_dev *dev);
 
+void mthca_cleanup_uar_table(struct mthca_dev *dev);
 void mthca_cleanup_pd_table(struct mthca_dev *dev);
 void mthca_cleanup_mr_table(struct mthca_dev *dev);
 void mthca_cleanup_eq_table(struct mthca_dev *dev);
@@ -337,6 +347,9 @@
 int mthca_register_device(struct mthca_dev *dev);
 void mthca_unregister_device(struct mthca_dev *dev);
 
+int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar);
+void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar);
+
 int mthca_pd_alloc(struct mthca_dev *dev, struct mthca_pd *pd);
 void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd);
 
Index: src/linux-kernel/infiniband/hw/mthca/mthca_eq.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-23 11:16:41.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-23 11:16:43.000000000 -0800
@@ -469,7 +469,7 @@
 						  MTHCA_EQ_FLAG_TR);
 	eq_context->start           = cpu_to_be64(0);
 	eq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24 |
-						  MTHCA_KAR_PAGE);
+						  dev->driver_uar.index);
 	eq_context->pd              = cpu_to_be32(dev->driver_pd.pd_num);
 	eq_context->intr            = intr;
 	eq_context->lkey            = cpu_to_be32(eq->mr.ibmr.lkey);
Index: src/linux-kernel/infiniband/hw/mthca/mthca_main.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-23 11:16:41.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-23 11:16:42.000000000 -0800
@@ -570,13 +570,35 @@
 
 	MTHCA_INIT_DOORBELL_LOCK(&dev->doorbell_lock);
 
-	err = mthca_init_pd_table(dev);
+	err = mthca_init_uar_table(dev);
 	if (err) {
 		mthca_err(dev, "Failed to initialize "
-			  "protection domain table, aborting.\n");
+			  "user access region table, aborting.\n");
 		return err;
 	}
 
+	err = mthca_uar_alloc(dev, &dev->driver_uar);
+	if (err) {
+		mthca_err(dev, "Failed to allocate driver access region, "
+			  "aborting.\n");
+		goto err_uar_table_free;
+	}
+
+	dev->kar = ioremap(dev->driver_uar.pfn << PAGE_SHIFT, PAGE_SIZE);
+	if (!dev->kar) {
+		mthca_err(dev, "Couldn't map kernel access region, "
+			  "aborting.\n");
+		err = -ENOMEM;
+		goto err_uar_free;
+	}
+
+       err = mthca_init_pd_table(dev);
+	if (err) {
+		mthca_err(dev, "Failed to initialize "
+			  "protection domain table, aborting.\n");
+		goto err_kar_unmap;
+	}
+
 	err = mthca_init_mr_table(dev);
 	if (err) {
 		mthca_err(dev, "Failed to initialize "
@@ -677,7 +699,16 @@
 
 err_pd_table_free:
 	mthca_cleanup_pd_table(dev);
-	return err;
+
+err_kar_unmap:
+	iounmap(dev->kar);
+
+err_uar_free:
+	mthca_uar_free(dev, &dev->driver_uar);
+
+err_uar_table_free:
+	mthca_cleanup_uar_table(dev);
+       return err;
 }
 
 static int __devinit mthca_request_regions(struct pci_dev *pdev,
@@ -789,7 +820,6 @@
 	static int mthca_version_printed = 0;
 	int ddr_hidden = 0;
 	int err;
-	unsigned long mthca_base;
 	struct mthca_dev *mdev;
 
 	if (!mthca_version_printed) {
@@ -891,8 +921,7 @@
 	sema_init(&mdev->cmd.poll_sem, 1);
 	mdev->cmd.use_events = 0;
 
-	mthca_base = pci_resource_start(pdev, 0);
-	mdev->hcr = ioremap(mthca_base + MTHCA_HCR_BASE, MTHCA_HCR_SIZE);
+	mdev->hcr = ioremap(pci_resource_start(pdev, 0) + MTHCA_HCR_BASE, MTHCA_HCR_SIZE);
 	if (!mdev->hcr) {
 		mthca_err(mdev, "Couldn't map command register, "
 			  "aborting.\n");
@@ -900,22 +929,13 @@
 		goto err_free_dev;
 	}
 
-	mthca_base = pci_resource_start(pdev, 2);
-	mdev->kar = ioremap(mthca_base + PAGE_SIZE * MTHCA_KAR_PAGE, PAGE_SIZE);
-	if (!mdev->kar) {
-		mthca_err(mdev, "Couldn't map kernel access region, "
-			  "aborting.\n");
-		err = -ENOMEM;
-		goto err_iounmap;
-	}
-
 	err = mthca_tune_pci(mdev);
 	if (err)
-		goto err_iounmap_kar;
+		goto err_iounmap;
 
 	err = mthca_init_hca(mdev);
 	if (err)
-		goto err_iounmap_kar;
+		goto err_iounmap;
 
 	err = mthca_setup_hca(mdev);
 	if (err)
@@ -948,13 +968,11 @@
 
 	mthca_cleanup_mr_table(mdev);
 	mthca_cleanup_pd_table(mdev);
+	mthca_cleanup_uar_table(mdev);
 
 err_close:
 	mthca_close_hca(mdev);
 
-err_iounmap_kar:
-	iounmap(mdev->kar);
-
 err_iounmap:
 	iounmap(mdev->hcr);
 
@@ -1000,9 +1018,12 @@
 		mthca_cleanup_mr_table(mdev);
 		mthca_cleanup_pd_table(mdev);
 
+		iounmap(mdev->kar);
+		mthca_uar_free(mdev, &mdev->driver_uar);
+		mthca_cleanup_uar_table(mdev);
+
 		mthca_close_hca(mdev);
 
-		iounmap(mdev->kar);
 		iounmap(mdev->hcr);
 
 		if (mdev->mthca_flags & MTHCA_FLAG_MSI_X)
Index: src/linux-kernel/infiniband/hw/mthca/mthca_profile.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_profile.c	2005-02-23 11:16:38.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_profile.c	2005-02-23 11:16:43.000000000 -0800
@@ -236,6 +236,7 @@
 			init_hca->mtt_seg_sz     = ffs(dev_lim->mtt_seg_sz) - 7;
 			break;
 		case MTHCA_RES_UAR:
+			dev->limits.num_uars       = profile[i].num;
 			init_hca->uar_scratch_base = profile[i].start;
 			break;
 		case MTHCA_RES_UDAV:
Index: src/linux-kernel/infiniband/hw/mthca/mthca_provider.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_provider.h	2005-02-23 11:16:38.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_provider.h	2005-02-23 11:16:42.000000000 -0800
@@ -49,6 +49,11 @@
 	DECLARE_PCI_UNMAP_ADDR(mapping)
 };
 
+struct mthca_uar {
+	unsigned long pfn;
+	int           index;
+};
+
 struct mthca_mr {
 	struct ib_mr ibmr;
 	int order;
Index: src/linux-kernel/infiniband/hw/mthca/mthca_qp.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_qp.c	2005-02-23 11:16:38.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_qp.c	2005-02-23 11:16:43.000000000 -0800
@@ -625,7 +625,7 @@
 		qp_context->mtu_msgmax = cpu_to_be32((attr->path_mtu << 29) |
 						     (31 << 24));
 	}
-	qp_context->usr_page   = cpu_to_be32(MTHCA_KAR_PAGE);
+	qp_context->usr_page   = cpu_to_be32(dev->driver_uar.index);
 	qp_context->local_qpn  = cpu_to_be32(qp->qpn);
 	if (attr_mask & IB_QP_DEST_QPN) {
 		qp_context->remote_qpn = cpu_to_be32(attr->dest_qp_num);
Index: src/linux-kernel/infiniband/hw/mthca/mthca_uar.c
===================================================================
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ src/linux-kernel/infiniband/hw/mthca/mthca_uar.c	2005-02-23 11:16:43.000000000 -0800
@@ -0,0 +1,69 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id$
+ */
+
+#include "mthca_dev.h"
+
+int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar)
+{
+	uar->index = mthca_alloc(&dev->uar_table.alloc);
+	if (uar->index == -1)
+		return -ENOMEM;
+
+	uar->pfn = (pci_resource_start(dev->pdev, 2) >> PAGE_SHIFT) + uar->index;
+
+	return 0;
+}
+
+void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar)
+{
+	mthca_free(&dev->uar_table.alloc, uar->index);
+}
+
+int mthca_init_uar_table(struct mthca_dev *dev)
+{
+	int ret;
+
+	ret = mthca_alloc_init(&dev->uar_table.alloc,
+			       dev->limits.num_uars,
+			       dev->limits.num_uars - 1,
+			       dev->limits.reserved_uars);
+
+	return ret;
+}
+
+void mthca_cleanup_uar_table(struct mthca_dev *dev)
+{
+	/* XXX check if any UARs are still allocated? */
+	mthca_alloc_cleanup(&dev->uar_table.alloc);
+}



From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][3/6] Add dynamic ICM table mapping
In-Reply-To: <20052231131.boGPEDrKIu7IxRyk@topspin.com>
Message-ID: <20052231131.8awojMNRy5CeAuka@topspin.com>

Add support for mapping more memory into HCA's context to cover
context tables when new objects are allocated.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: src/linux-kernel/infiniband/hw/mthca/mthca_main.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-23 10:18:56.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-23 10:25:39.000000000 -0800
@@ -363,10 +363,9 @@
 	}
 
 	mdev->mr_table.mtt_table = mthca_alloc_icm_table(mdev, init_hca->mtt_base,
-							 mdev->limits.num_mtt_segs *
 							 init_hca->mtt_seg_sz,
-							 mdev->limits.reserved_mtts *
-							 init_hca->mtt_seg_sz, 1);
+							 mdev->limits.num_mtt_segs,
+							 mdev->limits.reserved_mtts, 1);
 	if (!mdev->mr_table.mtt_table) {
 		mthca_err(mdev, "Failed to map MTT context memory, aborting.\n");
 		err = -ENOMEM;
@@ -374,10 +373,9 @@
 	}
 
 	mdev->mr_table.mpt_table = mthca_alloc_icm_table(mdev, init_hca->mpt_base,
-							 mdev->limits.num_mpts *
 							 dev_lim->mpt_entry_sz,
-							 mdev->limits.reserved_mrws *
-							 dev_lim->mpt_entry_sz, 1);
+							 mdev->limits.num_mpts,
+							 mdev->limits.reserved_mrws, 1);
 	if (!mdev->mr_table.mpt_table) {
 		mthca_err(mdev, "Failed to map MPT context memory, aborting.\n");
 		err = -ENOMEM;
@@ -385,10 +383,9 @@
 	}
 
 	mdev->qp_table.qp_table = mthca_alloc_icm_table(mdev, init_hca->qpc_base,
-							mdev->limits.num_qps *
 							dev_lim->qpc_entry_sz,
-							mdev->limits.reserved_qps *
-							dev_lim->qpc_entry_sz, 1);
+							mdev->limits.num_qps,
+							mdev->limits.reserved_qps, 0);
 	if (!mdev->qp_table.qp_table) {
 		mthca_err(mdev, "Failed to map QP context memory, aborting.\n");
 		err = -ENOMEM;
@@ -396,10 +393,9 @@
 	}
 
 	mdev->qp_table.eqp_table = mthca_alloc_icm_table(mdev, init_hca->eqpc_base,
-							 mdev->limits.num_qps *
 							 dev_lim->eqpc_entry_sz,
-							 mdev->limits.reserved_qps *
-							 dev_lim->eqpc_entry_sz, 1);
+							 mdev->limits.num_qps,
+							 mdev->limits.reserved_qps, 0);
 	if (!mdev->qp_table.eqp_table) {
 		mthca_err(mdev, "Failed to map EQP context memory, aborting.\n");
 		err = -ENOMEM;
@@ -407,10 +403,9 @@
 	}
 
 	mdev->cq_table.table = mthca_alloc_icm_table(mdev, init_hca->cqc_base,
-						     mdev->limits.num_cqs *
 						     dev_lim->cqc_entry_sz,
-						     mdev->limits.reserved_cqs *
-						     dev_lim->cqc_entry_sz, 1);
+						     mdev->limits.num_cqs,
+						     mdev->limits.reserved_cqs, 0);
 	if (!mdev->cq_table.table) {
 		mthca_err(mdev, "Failed to map CQ context memory, aborting.\n");
 		err = -ENOMEM;
Index: src/linux-kernel/infiniband/hw/mthca/mthca_memfree.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_memfree.c	2005-02-23 10:18:56.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_memfree.c	2005-02-23 10:22:52.000000000 -0800
@@ -79,6 +79,7 @@
 	if (!icm)
 		return icm;
 
+	icm->refcount = 0;
 	INIT_LIST_HEAD(&icm->chunk_list);
 
 	cur_order = get_order(MTHCA_ICM_ALLOC_SIZE);
@@ -138,9 +139,62 @@
 	return NULL;
 }
 
+int mthca_table_get(struct mthca_dev *dev, struct mthca_icm_table *table, int obj)
+{
+	int i = (obj & (table->num_obj - 1)) * table->obj_size / MTHCA_TABLE_CHUNK_SIZE;
+	int ret = 0;
+	u8 status;
+
+	down(&table->mutex);
+
+	if (table->icm[i]) {
+		++table->icm[i]->refcount;
+		goto out;
+	}
+
+	table->icm[i] = mthca_alloc_icm(dev, MTHCA_TABLE_CHUNK_SIZE >> PAGE_SHIFT,
+					(table->lowmem ? GFP_KERNEL : GFP_HIGHUSER) |
+					__GFP_NOWARN);
+	if (!table->icm[i]) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (mthca_MAP_ICM(dev, table->icm[i], table->virt + i * MTHCA_TABLE_CHUNK_SIZE,
+			  &status) || status) {
+		mthca_free_icm(dev, table->icm[i]);
+		table->icm[i] = NULL;
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	++table->icm[i]->refcount;
+
+out:
+	up(&table->mutex);
+	return ret;
+}
+
+void mthca_table_put(struct mthca_dev *dev, struct mthca_icm_table *table, int obj)
+{
+	int i = (obj & (table->num_obj - 1)) * table->obj_size / MTHCA_TABLE_CHUNK_SIZE;
+	u8 status;
+
+	down(&table->mutex);
+
+	if (--table->icm[i]->refcount == 0) {
+		mthca_UNMAP_ICM(dev, table->virt + i * MTHCA_TABLE_CHUNK_SIZE,
+				MTHCA_TABLE_CHUNK_SIZE >> 12, &status);
+		mthca_free_icm(dev, table->icm[i]);
+		table->icm[i] = NULL;
+	}
+
+	up(&table->mutex);
+}
+
 struct mthca_icm_table *mthca_alloc_icm_table(struct mthca_dev *dev,
-					      u64 virt, unsigned size,
-					      unsigned reserved,
+					      u64 virt, int obj_size,
+					      int nobj, int reserved,
 					      int use_lowmem)
 {
 	struct mthca_icm_table *table;
@@ -148,20 +202,23 @@
 	int i;
 	u8 status;
 
-	num_icm = size / MTHCA_TABLE_CHUNK_SIZE;
+	num_icm = obj_size * nobj / MTHCA_TABLE_CHUNK_SIZE;
 
 	table = kmalloc(sizeof *table + num_icm * sizeof *table->icm, GFP_KERNEL);
 	if (!table)
 		return NULL;
 
-	table->virt    = virt;
-	table->num_icm = num_icm;
-	init_MUTEX(&table->sem);
+	table->virt     = virt;
+	table->num_icm  = num_icm;
+	table->num_obj  = nobj;
+	table->obj_size = obj_size;
+	table->lowmem   = use_lowmem;
+	init_MUTEX(&table->mutex);
 
 	for (i = 0; i < num_icm; ++i)
 		table->icm[i] = NULL;
 
-	for (i = 0; i < (reserved + MTHCA_TABLE_CHUNK_SIZE - 1) / MTHCA_TABLE_CHUNK_SIZE; ++i) {
+	for (i = 0; i * MTHCA_TABLE_CHUNK_SIZE < reserved * obj_size; ++i) {
 		table->icm[i] = mthca_alloc_icm(dev, MTHCA_TABLE_CHUNK_SIZE >> PAGE_SHIFT,
 						(use_lowmem ? GFP_KERNEL : GFP_HIGHUSER) |
 						__GFP_NOWARN);
@@ -173,6 +230,12 @@
 			table->icm[i] = NULL;
 			goto err;
 		}
+
+		/*
+		 * Add a reference to this ICM chunk so that it never
+		 * gets freed (since it contains reserved firmware objects).
+		 */
+		++table->icm[i]->refcount;
 	}
 
 	return table;
Index: src/linux-kernel/infiniband/hw/mthca/mthca_memfree.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_memfree.h	2005-02-23 10:18:56.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_memfree.h	2005-02-23 10:26:11.000000000 -0800
@@ -53,12 +53,16 @@
 
 struct mthca_icm {
 	struct list_head chunk_list;
+	int              refcount;
 };
 
 struct mthca_icm_table {
 	u64               virt;
 	int               num_icm;
-	struct semaphore  sem;
+	int               num_obj;
+	int               obj_size;
+	int               lowmem;
+	struct semaphore  mutex;
 	struct mthca_icm *icm[0];
 };
 
@@ -75,10 +79,12 @@
 void mthca_free_icm(struct mthca_dev *dev, struct mthca_icm *icm);
 
 struct mthca_icm_table *mthca_alloc_icm_table(struct mthca_dev *dev,
-					      u64 virt, unsigned size,
-					      unsigned reserved,
+					      u64 virt, int obj_size,
+					      int nobj, int reserved,
 					      int use_lowmem);
 void mthca_free_icm_table(struct mthca_dev *dev, struct mthca_icm_table *table);
+int mthca_table_get(struct mthca_dev *dev, struct mthca_icm_table *table, int obj);
+void mthca_table_put(struct mthca_dev *dev, struct mthca_icm_table *table, int obj);
 
 static inline void mthca_icm_first(struct mthca_icm *icm,
 				   struct mthca_icm_iter *iter)



From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][4/6] Add mem-free MR support
In-Reply-To: <20052231131.8awojMNRy5CeAuka@topspin.com>
Message-ID: <20052231131.rmOw9NwgmXF4dWsv@topspin.com>

Add support for mem-free mode to memory region support.  This mostly
amounts to properly munging between keys and indices.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: src/linux-kernel/infiniband/hw/mthca/mthca_mr.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_mr.c	2004-12-30 10:23:34.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_mr.c	2005-02-23 10:00:42.000000000 -0800
@@ -53,7 +53,8 @@
 	u32 window_count;
 	u32 window_count_limit;
 	u64 mtt_seg;
-	u32 reserved[3];
+	u32 mtt_sz;		/* Arbel only */
+	u32 reserved[2];
 } __attribute__((packed));
 
 #define MTHCA_MPT_FLAG_SW_OWNS       (0xfUL << 28)
@@ -121,21 +122,38 @@
 	spin_unlock(&dev->mr_table.mpt_alloc.lock);
 }
 
+static inline u32 hw_index_to_key(struct mthca_dev *dev, u32 ind)
+{
+	if (dev->hca_type == ARBEL_NATIVE)
+		return (ind >> 24) | (ind << 8);
+	else
+		return ind;
+}
+
+static inline u32 key_to_hw_index(struct mthca_dev *dev, u32 key)
+{
+	if (dev->hca_type == ARBEL_NATIVE)
+		return (key << 24) | (key >> 8);
+	else
+		return key;
+}
+
 int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd,
 			   u32 access, struct mthca_mr *mr)
 {
 	void *mailbox;
 	struct mthca_mpt_entry *mpt_entry;
+	u32 key;
 	int err;
 	u8 status;
 
 	might_sleep();
 
 	mr->order = -1;
-	mr->ibmr.lkey = mthca_alloc(&dev->mr_table.mpt_alloc);
-	if (mr->ibmr.lkey == -1)
+	key = mthca_alloc(&dev->mr_table.mpt_alloc);
+	if (key == -1)
 		return -ENOMEM;
-	mr->ibmr.rkey = mr->ibmr.lkey;
+	mr->ibmr.rkey = mr->ibmr.lkey = hw_index_to_key(dev, key);
 
 	mailbox = kmalloc(sizeof *mpt_entry + MTHCA_CMD_MAILBOX_EXTRA,
 			  GFP_KERNEL);
@@ -151,7 +169,7 @@
 				       MTHCA_MPT_FLAG_REGION      |
 				       access);
 	mpt_entry->page_size = 0;
-	mpt_entry->key       = cpu_to_be32(mr->ibmr.lkey);
+	mpt_entry->key       = cpu_to_be32(key);
 	mpt_entry->pd        = cpu_to_be32(pd);
 	mpt_entry->start     = 0;
 	mpt_entry->length    = ~0ULL;
@@ -160,7 +178,7 @@
 	       sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey));
 
 	err = mthca_SW2HW_MPT(dev, mpt_entry,
-			      mr->ibmr.lkey & (dev->limits.num_mpts - 1),
+			      key & (dev->limits.num_mpts - 1),
 			      &status);
 	if (err)
 		mthca_warn(dev, "SW2HW_MPT failed (%d)\n", err);
@@ -182,6 +200,7 @@
 	void *mailbox;
 	u64 *mtt_entry;
 	struct mthca_mpt_entry *mpt_entry;
+	u32 key;
 	int err = -ENOMEM;
 	u8 status;
 	int i;
@@ -189,10 +208,10 @@
 	might_sleep();
 	WARN_ON(buffer_size_shift >= 32);
 
-	mr->ibmr.lkey = mthca_alloc(&dev->mr_table.mpt_alloc);
-	if (mr->ibmr.lkey == -1)
+	key = mthca_alloc(&dev->mr_table.mpt_alloc);
+	if (key == -1)
 		return -ENOMEM;
-	mr->ibmr.rkey = mr->ibmr.lkey;
+	mr->ibmr.rkey = mr->ibmr.lkey = hw_index_to_key(dev, key);
 
 	for (i = dev->limits.mtt_seg_size / 8, mr->order = 0;
 	     i < list_len;
@@ -254,7 +273,7 @@
 				       access);
 
 	mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12);
-	mpt_entry->key       = cpu_to_be32(mr->ibmr.lkey);
+	mpt_entry->key       = cpu_to_be32(key);
 	mpt_entry->pd        = cpu_to_be32(pd);
 	mpt_entry->start     = cpu_to_be64(iova);
 	mpt_entry->length    = cpu_to_be64(total_size);
@@ -275,7 +294,7 @@
 	}
 
 	err = mthca_SW2HW_MPT(dev, mpt_entry,
-			      mr->ibmr.lkey & (dev->limits.num_mpts - 1),
+			      key & (dev->limits.num_mpts - 1),
 			      &status);
 	if (err)
 		mthca_warn(dev, "SW2HW_MPT failed (%d)\n", err);
@@ -307,7 +326,8 @@
 	might_sleep();
 
 	err = mthca_HW2SW_MPT(dev, NULL,
-			      mr->ibmr.lkey & (dev->limits.num_mpts - 1),
+			      key_to_hw_index(dev, mr->ibmr.lkey) &
+			      (dev->limits.num_mpts - 1),
 			      &status);
 	if (err)
 		mthca_warn(dev, "HW2SW_MPT failed (%d)\n", err);
@@ -318,7 +338,7 @@
 	if (mr->order >= 0)
 		mthca_free_mtt(dev, mr->first_seg, mr->order);
 
-	mthca_free(&dev->mr_table.mpt_alloc, mr->ibmr.lkey);
+	mthca_free(&dev->mr_table.mpt_alloc, key_to_hw_index(dev, mr->ibmr.lkey));
 }
 
 int __devinit mthca_init_mr_table(struct mthca_dev *dev)



From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][5/6] Add mem-free EQ init support
In-Reply-To: <20052231131.rmOw9NwgmXF4dWsv@topspin.com>
Message-ID: <20052231131.EvrAJez1EuWYxuVX@topspin.com>

Add code to initialize EQ context properly in mem-free mode.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: src/linux-kernel/infiniband/hw/mthca/mthca_eq.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-22 20:31:48.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-23 10:10:27.000000000 -0800
@@ -54,10 +54,10 @@
 	u32 flags;
 	u64 start;
 	u32 logsize_usrpage;
-	u32 pd;
+	u32 tavor_pd;		/* reserved for Arbel */
 	u8  reserved1[3];
 	u8  intr;
-	u32 lost_count;
+	u32 arbel_pd;		/* lost_count for Tavor */
 	u32 lkey;
 	u32 reserved2[2];
 	u32 consumer_index;
@@ -75,6 +75,7 @@
 #define MTHCA_EQ_STATE_ARMED        ( 1 <<  8)
 #define MTHCA_EQ_STATE_FIRED        ( 2 <<  8)
 #define MTHCA_EQ_STATE_ALWAYS_ARMED ( 3 <<  8)
+#define MTHCA_EQ_STATE_ARBEL        ( 8 <<  8)
 
 enum {
 	MTHCA_EVENT_TYPE_COMP       	    = 0x00,
@@ -467,10 +468,16 @@
 						  MTHCA_EQ_OWNER_HW    |
 						  MTHCA_EQ_STATE_ARMED |
 						  MTHCA_EQ_FLAG_TR);
-	eq_context->start           = cpu_to_be64(0);
-	eq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24 |
-						  dev->driver_uar.index);
-	eq_context->pd              = cpu_to_be32(dev->driver_pd.pd_num);
+	if (dev->hca_type == ARBEL_NATIVE)
+		eq_context->flags  |= cpu_to_be32(MTHCA_EQ_STATE_ARBEL);
+
+	eq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24);
+	if (dev->hca_type == ARBEL_NATIVE) {
+		eq_context->arbel_pd = cpu_to_be32(dev->driver_pd.pd_num);
+	} else {
+		eq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar.index);
+		eq_context->tavor_pd         = cpu_to_be32(dev->driver_pd.pd_num);
+	}
 	eq_context->intr            = intr;
 	eq_context->lkey            = cpu_to_be32(eq->mr.ibmr.lkey);
 



From roland at topspin.com  Wed Feb 23 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 11:31:53 -0800
Subject: [openib-general] [PATCH][6/6] Add mem-free interrupt support
In-Reply-To: <20052231131.EvrAJez1EuWYxuVX@topspin.com>
Message-ID: <20052231131.F5xbzU41Iz2Otaf2@topspin.com>

Update interrupt handling code to handle mem-free mode.  While we're
at it, improve the Tavor interrupt handling to avoid an extra MMIO
read of the event cause register.

Signed-off-by: Roland Dreier <roland at topspin.com>

Index: src/linux-kernel/infiniband/hw/mthca/mthca_dev.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_dev.h	2005-02-23 11:16:47.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_dev.h	2005-02-23 11:18:09.000000000 -0800
@@ -171,6 +171,7 @@
 	struct mthca_alloc alloc;
 	void __iomem      *clr_int;
 	u32                clr_mask;
+	u32                arm_mask;
 	struct mthca_eq    eq[MTHCA_NUM_EQ];
 	u64                icm_virt;
 	struct page       *icm_page;
Index: src/linux-kernel/infiniband/hw/mthca/mthca_eq.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-23 11:18:08.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_eq.c	2005-02-23 11:26:13.000000000 -0800
@@ -165,19 +165,46 @@
 		MTHCA_ASYNC_EVENT_MASK;
 }
 
-static inline void set_eq_ci(struct mthca_dev *dev, struct mthca_eq *eq, u32 ci)
+static inline void tavor_set_eq_ci(struct mthca_dev *dev, struct mthca_eq *eq, u32 ci)
 {
 	u32 doorbell[2];
 
 	doorbell[0] = cpu_to_be32(MTHCA_EQ_DB_SET_CI | eq->eqn);
 	doorbell[1] = cpu_to_be32(ci & (eq->nent - 1));
 
+	/*
+	 * This barrier makes sure that all updates to ownership bits
+	 * done by set_eqe_hw() hit memory before the consumer index
+	 * is updated.  set_eq_ci() allows the HCA to possibly write
+	 * more EQ entries, and we want to avoid the exceedingly
+	 * unlikely possibility of the HCA writing an entry and then
+	 * having set_eqe_hw() overwrite the owner field.
+	 */
+	wmb();
 	mthca_write64(doorbell,
 		      dev->kar + MTHCA_EQ_DOORBELL,
 		      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 }
 
-static inline void eq_req_not(struct mthca_dev *dev, int eqn)
+static inline void arbel_set_eq_ci(struct mthca_dev *dev, struct mthca_eq *eq, u32 ci)
+{
+	/* See comment in tavor_set_eq_ci() above. */
+	wmb();
+	__raw_writel(cpu_to_be32(ci), dev->eq_regs.arbel.eq_set_ci_base +
+		     eq->eqn * 8);
+	/* We still want ordering, just not swabbing, so add a barrier */
+	mb();
+}
+
+static inline void set_eq_ci(struct mthca_dev *dev, struct mthca_eq *eq, u32 ci)
+{
+	if (dev->hca_type == ARBEL_NATIVE)
+		arbel_set_eq_ci(dev, eq, ci);
+	else
+		tavor_set_eq_ci(dev, eq, ci);
+}
+
+static inline void tavor_eq_req_not(struct mthca_dev *dev, int eqn)
 {
 	u32 doorbell[2];
 
@@ -189,16 +216,23 @@
 		      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 }
 
+static inline void arbel_eq_req_not(struct mthca_dev *dev, u32 eqn_mask)
+{
+	writel(eqn_mask, dev->eq_regs.arbel.eq_arm);
+}
+
 static inline void disarm_cq(struct mthca_dev *dev, int eqn, int cqn)
 {
-	u32 doorbell[2];
+	if (dev->hca_type != ARBEL_NATIVE) {
+		u32 doorbell[2];
 
-	doorbell[0] = cpu_to_be32(MTHCA_EQ_DB_DISARM_CQ | eqn);
-	doorbell[1] = cpu_to_be32(cqn);
+		doorbell[0] = cpu_to_be32(MTHCA_EQ_DB_DISARM_CQ | eqn);
+		doorbell[1] = cpu_to_be32(cqn);
 
-	mthca_write64(doorbell,
-		      dev->kar + MTHCA_EQ_DOORBELL,
-		      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+		mthca_write64(doorbell,
+			      dev->kar + MTHCA_EQ_DOORBELL,
+			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+	}
 }
 
 static inline struct mthca_eqe *get_eqe(struct mthca_eq *eq, u32 entry)
@@ -233,7 +267,7 @@
 	ib_dispatch_event(&record);
 }
 
-static void mthca_eq_int(struct mthca_dev *dev, struct mthca_eq *eq)
+static int mthca_eq_int(struct mthca_dev *dev, struct mthca_eq *eq)
 {
 	struct mthca_eqe *eqe;
 	int disarm_cqn;
@@ -334,60 +368,93 @@
 		++eq->cons_index;
 		eqes_found = 1;
 
-		if (set_ci) {
-			wmb(); /* see comment below */
+		if (unlikely(set_ci)) {
+			/*
+			 * Conditional on hca_type is OK here because
+			 * this is a rare case, not the fast path.
+			 */
 			set_eq_ci(dev, eq, eq->cons_index);
 			set_ci = 0;
 		}
 	}
 
 	/*
-	 * This barrier makes sure that all updates to
-	 * ownership bits done by set_eqe_hw() hit memory
-	 * before the consumer index is updated.  set_eq_ci()
-	 * allows the HCA to possibly write more EQ entries,
-	 * and we want to avoid the exceedingly unlikely
-	 * possibility of the HCA writing an entry and then
-	 * having set_eqe_hw() overwrite the owner field.
+	 * Rely on caller to set consumer index so that we don't have
+	 * to test hca_type in our interrupt handling fast path.
 	 */
-	if (likely(eqes_found)) {
-		wmb();
-		set_eq_ci(dev, eq, eq->cons_index);
-	}
-	eq_req_not(dev, eq->eqn);
+	return eqes_found;
 }
 
-static irqreturn_t mthca_interrupt(int irq, void *dev_ptr, struct pt_regs *regs)
+static irqreturn_t mthca_tavor_interrupt(int irq, void *dev_ptr, struct pt_regs *regs)
 {
 	struct mthca_dev *dev = dev_ptr;
 	u32 ecr;
-	int work = 0;
 	int i;
 
 	if (dev->eq_table.clr_mask)
 		writel(dev->eq_table.clr_mask, dev->eq_table.clr_int);
 
-	if ((ecr = readl(dev->eq_regs.tavor.ecr_base + 4)) != 0) {
-		work = 1;
-
+	ecr = readl(dev->eq_regs.tavor.ecr_base + 4);
+	if (ecr) {
 		writel(ecr, dev->eq_regs.tavor.ecr_base +
 		       MTHCA_ECR_CLR_BASE - MTHCA_ECR_BASE + 4);
 
 		for (i = 0; i < MTHCA_NUM_EQ; ++i)
-			if (ecr & dev->eq_table.eq[i].ecr_mask)
-				mthca_eq_int(dev, &dev->eq_table.eq[i]);
+			if (ecr & dev->eq_table.eq[i].eqn_mask &&
+			    mthca_eq_int(dev, &dev->eq_table.eq[i])) {
+				tavor_set_eq_ci(dev, &dev->eq_table.eq[i],
+						dev->eq_table.eq[i].cons_index);
+				tavor_eq_req_not(dev, dev->eq_table.eq[i].eqn);
+	}
 	}
 
+	return IRQ_RETVAL(ecr);
+}
+
+static irqreturn_t mthca_tavor_msi_x_interrupt(int irq, void *eq_ptr,
+					 struct pt_regs *regs)
+{
+	struct mthca_eq  *eq  = eq_ptr;
+	struct mthca_dev *dev = eq->dev;
+
+	mthca_eq_int(dev, eq);
+	tavor_set_eq_ci(dev, eq, eq->cons_index);
+	tavor_eq_req_not(dev, eq->eqn);
+
+	/* MSI-X vectors always belong to us */
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t mthca_arbel_interrupt(int irq, void *dev_ptr, struct pt_regs *regs)
+{
+	struct mthca_dev *dev = dev_ptr;
+	int work = 0;
+	int i;
+
+	if (dev->eq_table.clr_mask)
+		writel(dev->eq_table.clr_mask, dev->eq_table.clr_int);
+
+	for (i = 0; i < MTHCA_NUM_EQ; ++i)
+		if (mthca_eq_int(dev, &dev->eq_table.eq[i])) {
+			work = 1;
+			arbel_set_eq_ci(dev, &dev->eq_table.eq[i],
+					dev->eq_table.eq[i].cons_index);
+		}
+
+	arbel_eq_req_not(dev, dev->eq_table.arm_mask);
+
 	return IRQ_RETVAL(work);
 }
 
-static irqreturn_t mthca_msi_x_interrupt(int irq, void *eq_ptr,
+static irqreturn_t mthca_arbel_msi_x_interrupt(int irq, void *eq_ptr,
 					 struct pt_regs *regs)
 {
 	struct mthca_eq  *eq  = eq_ptr;
 	struct mthca_dev *dev = eq->dev;
 
 	mthca_eq_int(dev, eq);
+	arbel_set_eq_ci(dev, eq, eq->cons_index);
+	arbel_eq_req_not(dev, eq->eqn_mask);
 
 	/* MSI-X vectors always belong to us */
 	return IRQ_HANDLED;
@@ -496,10 +563,10 @@
 	kfree(dma_list);
 	kfree(mailbox);
 
-	eq->ecr_mask   = swab32(1 << eq->eqn);
+	eq->eqn_mask   = swab32(1 << eq->eqn);
 	eq->cons_index = 0;
 
-	eq_req_not(dev, eq->eqn);
+	dev->eq_table.arm_mask |= eq->eqn_mask;
 
 	mthca_dbg(dev, "Allocated EQ %d with %d entries\n",
 		  eq->eqn, nent);
@@ -551,6 +618,8 @@
 		mthca_warn(dev, "HW2SW_EQ returned status 0x%02x\n",
 			   status);
 
+	dev->eq_table.arm_mask &= ~eq->eqn_mask;
+
 	if (0) {
 		mthca_dbg(dev, "Dumping EQ context %02x:\n", eq->eqn);
 		for (i = 0; i < sizeof (struct mthca_eq_context) / 4; ++i) {
@@ -780,6 +849,8 @@
 			(dev->eq_table.inta_pin < 31 ? 4 : 0);
 	}
 
+	dev->eq_table.arm_mask = 0;
+
 	intr = (dev->mthca_flags & MTHCA_FLAG_MSI) ?
 		128 : dev->eq_table.inta_pin;
 
@@ -810,15 +881,20 @@
 
 		for (i = 0; i < MTHCA_NUM_EQ; ++i) {
 			err = request_irq(dev->eq_table.eq[i].msi_x_vector,
-					  mthca_msi_x_interrupt, 0,
-					  eq_name[i], dev->eq_table.eq + i);
+					  dev->hca_type == ARBEL_NATIVE ?
+					  mthca_arbel_msi_x_interrupt :
+					  mthca_tavor_msi_x_interrupt,
+					  0, eq_name[i], dev->eq_table.eq + i);
 			if (err)
 				goto err_out_cmd;
 			dev->eq_table.eq[i].have_irq = 1;
 		}
 	} else {
-		err = request_irq(dev->pdev->irq, mthca_interrupt, SA_SHIRQ,
-				  DRV_NAME, dev);
+		err = request_irq(dev->pdev->irq,
+				  dev->hca_type == ARBEL_NATIVE ?
+				  mthca_arbel_interrupt :
+				  mthca_tavor_interrupt,
+				  SA_SHIRQ, DRV_NAME, dev);
 		if (err)
 			goto err_out_cmd;
 		dev->eq_table.have_irq = 1;
@@ -842,6 +918,12 @@
 		mthca_warn(dev, "MAP_EQ for cmd EQ %d returned status 0x%02x\n",
 			   dev->eq_table.eq[MTHCA_EQ_CMD].eqn, status);
 
+	for (i = 0; i < MTHCA_EQ_CMD; ++i)
+		if (dev->hca_type == ARBEL_NATIVE)
+			arbel_eq_req_not(dev, dev->eq_table.eq[i].eqn_mask);
+		else
+			tavor_eq_req_not(dev, dev->eq_table.eq[i].eqn);
+
 	return 0;
 
 err_out_cmd:
Index: src/linux-kernel/infiniband/hw/mthca/mthca_main.c
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-23 11:18:07.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_main.c	2005-02-23 11:18:09.000000000 -0800
@@ -608,13 +608,6 @@
 		goto err_mr_table_free;
 	}
 
-	if (dev->hca_type == ARBEL_NATIVE) {
-		mthca_warn(dev, "Sorry, native MT25208 mode support is not done, "
-			   "aborting.\n");
-		err = -ENODEV;
-		goto err_pd_free;
-	}
-
 	err = mthca_init_eq_table(dev);
 	if (err) {
 		mthca_err(dev, "Failed to initialize "
@@ -638,8 +631,16 @@
 			mthca_err(dev, "BIOS or ACPI interrupt routing problem?\n");
 
 		goto err_cmd_poll;
-	} else
-		mthca_dbg(dev, "NOP command IRQ test passed\n");
+	}
+
+	mthca_dbg(dev, "NOP command IRQ test passed\n");
+
+	if (dev->hca_type == ARBEL_NATIVE) {
+		mthca_warn(dev, "Sorry, native MT25208 mode support is not complete, "
+			   "aborting.\n");
+		err = -ENODEV;
+		goto err_cmd_poll;
+	}
 
 	err = mthca_init_cq_table(dev);
 	if (err) {
Index: src/linux-kernel/infiniband/hw/mthca/mthca_provider.h
===================================================================
--- src.orig/linux-kernel/infiniband/hw/mthca/mthca_provider.h	2005-02-23 11:16:42.000000000 -0800
+++ src/linux-kernel/infiniband/hw/mthca/mthca_provider.h	2005-02-23 11:18:09.000000000 -0800
@@ -70,7 +70,7 @@
 struct mthca_eq {
 	struct mthca_dev      *dev;
 	int                    eqn;
-	u32                    ecr_mask;
+	u32                    eqn_mask;
 	u32                    cons_index;
 	u16                    msi_x_vector;
 	u16                    msi_x_entry;



From mst at mellanox.co.il  Wed Feb 23 11:45:04 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 21:45:04 +0200
Subject: [openib-general] Re: [PATCH][6/6] Add mem-free interrupt support
In-Reply-To: <20052231131.F5xbzU41Iz2Otaf2@topspin.com>
References: <20052231131.EvrAJez1EuWYxuVX@topspin.com>
	<20052231131.F5xbzU41Iz2Otaf2@topspin.com>
Message-ID: <20050223194504.GB3470@mellanox.co.il>

Hi!
Patches look good so far.

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: [PATCH][6/6] Add mem-free interrupt support
> 
> Update interrupt handling code to handle mem-free mode.  While we're
> at it, improve the Tavor interrupt handling to avoid an extra MMIO
> read of the event cause register.
>
>
>  		for (i = 0; i < MTHCA_NUM_EQ; ++i)
> -			if (ecr & dev->eq_table.eq[i].ecr_mask)
> -				mthca_eq_int(dev, &dev->eq_table.eq[i]);
> +			if (ecr & dev->eq_table.eq[i].eqn_mask &&
> +			    mthca_eq_int(dev, &dev->eq_table.eq[i])) {
> +				tavor_set_eq_ci(dev, &dev->eq_table.eq[i],
> +						dev->eq_table.eq[i].cons_index);
> +				tavor_eq_req_not(dev, dev->eq_table.eq[i].eqn);
> +	}
>  	}

I see we are still scanning all EQs on an interrupt.
I was thinking, shall not we rather use ffs to scan only the bits that
are set?


-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Wed Feb 23 11:46:34 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 11:46:34 -0800
Subject: [openib-general] CHECK_NULL [was Re: SDP_CONN_LOCK]
In-Reply-To: <20050223110143.GZ23837@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Feb 23, 2005 at 01:01:43PM +0200
References: <20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
	<20050220001137.GC8268@mellanox.co.il>
	<20050222122801.C23420@topspin.com>
	<20050223110143.GZ23837@mellanox.co.il>
Message-ID: <20050223114634.O23420@topspin.com>

On Wed, Feb 23, 2005 at 01:01:43PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: Re: SDP_CONN_LOCK
> > 
> > On Sun, Feb 20, 2005 at 02:11:37AM +0200, Michael S. Tsirkin wrote:
> > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > 
> > > >   And here I would have thought SDP_CONN_HOLD as a macro would have
> > > > bugged people more. :)
> > >
> > > Didn't get that far yet :)
> > > 
> > > I also wander whether we could get rid of most called to 
> > > _CHECK macros, consequently turn lots of functions
> > > which cant otherwise fail into void?
> > > Given that these checks are disabled upon a compile switch, one wanders
> > > how big a debugging aid they are.
> > 
> >   Yes, I think the _CHECK and EXPECT macros can be gotten rid of. Getting
> > rid of _CHECK does have the added benefit of turning a lot of functions
> > into void return functions, which in turn can get rid of most of the
> > EXPECT macros. I do agree that there are quite a few function that return
> > an status which can safely be turned into void functions.
> > 
> > -Libor
> > 
> 
> Here's a patch to remove CHECK_NULL's, as a first step.
> 
> Signed-off-by: Michael s. Tsirkin <mst at mellanox.co.il>

  Thanks, I've applied and commited the change. I'll make a pass at turning
some of the obvious int return functions into voids.

-Libor


From mst at mellanox.co.il  Wed Feb 23 11:50:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 21:50:54 +0200
Subject: [openib-general] Re: [PATCH][0/6] Mem-free merge patches
In-Reply-To: <20052231131.JCngKDtSiEvKkiUy@topspin.com>
References: <20052231131.JCngKDtSiEvKkiUy@topspin.com>
Message-ID: <20050223195054.GC3470@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: [PATCH][0/6] Mem-free merge patches
> 
> Here's the first batch of patches that merges mem-free support from
> the roland-uverbs branch onto the trunk.  It cuts the mthca diff
> between the branch and the trunk from ~3700 lines down to ~2300 lines
> (of which some is userspace support).
> 
> For Michael or others who are wondering how I plan to merge mem-free
> support, this should show the plan.  We need to split the changes into
> manageable, independent chunks.  This makes review possible, and also
> makes it possible to use the history to search for the responsible
> patch if, heaven forfend, this introduces a regression.
> 
>  - R.

Thanks, Roland!
To make reading these patches a bit easier, could you please generate them
with diff -p?
In case you forgot how: svn diff --diff-cmd=/usr/bin/diff -x -up

Thanks a bunch,
-- 
MST - Michael S. Tsirkin


From mshefty at ichips.intel.com  Wed Feb 23 12:03:16 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 12:03:16 -0800
Subject: [openib-general] ib_cancel_mad and the CM
Message-ID: <421CE184.4090803@ichips.intel.com>

During a code review, I've discovered what appears to be an issue 
between the CM and a call to ib_cancel_mad, and I think that it's 
possible for other MAD clients to run into a similar issue.

Here's CM pseudo-code that leads to the problem:

lock cm_id
wr_id = (unsigned long) current_msg
unlock cm_id
ib_cancel_mad(agent, wr_id)

Because ib_cancel_mad might invoke a callback that acquires the cm_id 
lock, the lock cannot be held when ib_cancel_mad is invoked.

The problem comes from the use of the current_msg pointer as the wr_id. 
  If the MAD completes before ib_cancel_mad can be invoked, the memory 
could be freed, reallocated, and re-used with a second cm_id.  The call 
to ib_cancel_mad above would then cancel the wrong MAD.  The actual 
chance of this happening in the CM seems highly unlikely, but I believe 
that it is possible.

To fix this in the CM, the call to ib_cancel_mad just needs to move 
inside the cm_id lock.  Alternatively, it may be possible to change 
ib_cancel_mad to cancel MADs based on a second set of criteria.  Both 
of these would require changes to the MAD layer.

If we think that this issue is unique to the CM, then I can try to 
figure out some other way of handling this.

- Sean


From roland at topspin.com  Wed Feb 23 12:03:18 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 12:03:18 -0800
Subject: [openib-general] Re: [PATCH][0/6] Mem-free merge patches
In-Reply-To: <20050223195054.GC3470@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 23 Feb 2005 21:50:54 +0200")
References: <20052231131.JCngKDtSiEvKkiUy@topspin.com>
	<20050223195054.GC3470@mellanox.co.il>
Message-ID: <52ll9favsp.fsf@topspin.com>

    Michael> Thanks, Roland!  To make reading these patches a bit
    Michael> easier, could you please generate them with diff -p? In
    Michael> case you forgot how: svn diff --diff-cmd=/usr/bin/diff -x -up

Actually my svn diffs are generated that way.  I made this series
using quilt so I'll have to look into how to pass diff flags to quilt.
In the meantime it should be easy to regenerate the patches by
applying and then rediffing if you need the -p output.

 - R.


From mst at mellanox.co.il  Wed Feb 23 12:06:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 23 Feb 2005 22:06:08 +0200
Subject: [openib-general] Re: [PATCH][0/6] Mem-free merge patches
In-Reply-To: <52ll9favsp.fsf@topspin.com>
References: <20052231131.JCngKDtSiEvKkiUy@topspin.com>
	<20050223195054.GC3470@mellanox.co.il> <52ll9favsp.fsf@topspin.com>
Message-ID: <20050223200608.GD3470@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH][0/6] Mem-free merge patches
> 
>     Michael> Thanks, Roland!  To make reading these patches a bit
>     Michael> easier, could you please generate them with diff -p? In
>     Michael> case you forgot how: svn diff --diff-cmd=/usr/bin/diff -x -up
> 
> Actually my svn diffs are generated that way.  I made this series
> using quilt so I'll have to look into how to pass diff flags to quilt.
> In the meantime it should be easy to regenerate the patches by
> applying and then rediffing if you need the -p output.
> 
>  - R.
> 

I know, I'm just reading this now at a PC without subversion.
Whatever.

-- 
MST - Michael S. Tsirkin


From eitan at mellanox.co.il  Wed Feb 23 13:22:06 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Wed, 23 Feb 2005 23:22:06 +0200
Subject: [openib-general] IB Management Simulator
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEBA@mtlex01.yok.mtl.com>

Hi All,

Mellanox is willing to contribute its newly developed management simulator.
This simulator can be used to validate OpenSM and diagnostic tools that send
and receive SMP and PM
MADs from the fabric. 

The simulator have an interface for clients to attach to a virtual IB node
and send/receive mads to.
OpenSM has a vendor layer implementation that can talk to the simulator.

The software involved is made of two packages: 
IBDM - a generic data model describing the fabric
IBMgtSim - the simulator (an engine for dispatching mads, an SMA and PMA
implementations).

As I'm ready to upload the code into OpenIB I wonder where it should fit in
the tree?

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050223/9a37dce3/attachment.html>

From halr at voltaire.com  Wed Feb 23 13:34:44 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 23 Feb 2005 16:34:44 -0500
Subject: [openib-general] IB Management Simulator
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEBA@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEEBA@mtlex01.yok.mtl.com>
Message-ID: <1109194484.13555.1069.camel@localhost.localdomain>

On Wed, 2005-02-23 at 16:22, Eitan Zahavi wrote:
> Hi All,
> 
> Mellanox is willing to contribute its newly developed management
> simulator.

Great news.

> This simulator can be used to validate OpenSM and diagnostic tools
> that send and receive SMP and PM
> 
> MADs from the fabric. 
> 
> The simulator have an interface for clients to attach to a virtual IB
> node and send/receive mads to.
> 
> OpenSM has a vendor layer implementation that can talk to the
> simulator.
> 
> The software involved is made of two packages: 
> 
> IBDM - a generic data model describing the fabric
> 
> IBMgtSim - the simulator (an engine for dispatching mads, an SMA and
> PMA implementations).
> 
> As I'm ready to upload the code into OpenIB I wonder where it should
> fit in the tree?

How about under gen2/utils ? I presume these are userspace tools so
perhaps a userspace directory should be put under utils and then these 2
subdirectories under that. Just one man's opinion...

-- Hal

> 
> Eitan Zahavi
> 
> Design Technology Director
> 
> Mellanox Technologies LTD
> 
> Tel:+972-4-9097208
> Fax:+972-4-9593245
> 
> P.O. Box 586 Yokneam 20692 ISRAEL
> 
> 
> 
> ______________________________________________________________________
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From libor at topspin.com  Wed Feb 23 13:44:38 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 13:44:38 -0800
Subject: [openib-general] CHECK_NULL [was Re: SDP_CONN_LOCK]
In-Reply-To: <20050223114634.O23420@topspin.com>;
	from libor@topspin.com on Wed, Feb 23, 2005 at 11:46:34AM -0800
References: <20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
	<20050220001137.GC8268@mellanox.co.il>
	<20050222122801.C23420@topspin.com>
	<20050223110143.GZ23837@mellanox.co.il>
	<20050223114634.O23420@topspin.com>
Message-ID: <20050223134438.P23420@topspin.com>

On Wed, Feb 23, 2005 at 11:46:34AM -0800, Libor Michalek wrote:
> On Wed, Feb 23, 2005 at 01:01:43PM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > Subject: Re: Re: SDP_CONN_LOCK
> > > 
> > > On Sun, Feb 20, 2005 at 02:11:37AM +0200, Michael S. Tsirkin wrote:
> > > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > > 
> > > > >   And here I would have thought SDP_CONN_HOLD as a macro would have
> > > > > bugged people more. :)
> > > >
> > > > Didn't get that far yet :)
> > > > 
> > > > I also wander whether we could get rid of most called to 
> > > > _CHECK macros, consequently turn lots of functions
> > > > which cant otherwise fail into void?
> > > > Given that these checks are disabled upon a compile switch, one wanders
> > > > how big a debugging aid they are.
> > > 
> > >   Yes, I think the _CHECK and EXPECT macros can be gotten rid of. Getting
> > > rid of _CHECK does have the added benefit of turning a lot of functions
> > > into void return functions, which in turn can get rid of most of the
> > > EXPECT macros. I do agree that there are quite a few function that return
> > > an status which can safely be turned into void functions.
> > > 
> > > -Libor
> > > 
> > 
> > Here's a patch to remove CHECK_NULL's, as a first step.
> > 
> > Signed-off-by: Michael s. Tsirkin <mst at mellanox.co.il>
> 
>   Thanks, I've applied and commited the change. I'll make a pass at turning
> some of the obvious int return functions into voids.

  Here is a pass at turning some of the more obvious functions into void
returns. I didn't convert any of the functions in sdp_wall.c, since I'm
working on moving some of them around at the moment.

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: sdp_kvec.c
===================================================================
--- sdp_kvec.c	(revision 1873)
+++ sdp_kvec.c	(working copy)
@@ -35,10 +35,6 @@
 #include "sdp_main.h"
 
 /*
- * static kvec internal functions
- */
-
-/*
  * _sdp_iocb_q_cancel_lookup_func - lookup function for cancelation
  */
 static int _sdp_iocb_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
@@ -47,26 +43,19 @@
 } /* _sdp_iocb_q_cancel_lookup_func */
 
 /*
- * Public write functions
- */
-
-/*
- * Cancel operations
- */
-
-/*
  * _sdp_iocb_q_cancel_read_pending - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn,
+					    ssize_t error)
 {
-	return sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL,
-					error);
+	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
+	return;
 } /* _sdp_iocb_q_cancel_read_pending */
 
 /*
  * _sdp_iocb_q_cancel_read_source - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -74,28 +63,29 @@
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->r_src,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		result = sdp_iocb_complete(iocb, error);
 		SDP_EXPECT(!(0 > result));
 	}
 
-	return 0;
+	return;
 } /* _sdp_iocb_q_cancel_read_source */
 
 /*
  * _sdp_iocb_q_cancel_read_snk - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
 {
-	return sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
+	return;
 } /* _sdp_iocb_q_cancel_read_snk */
 
 /*
  * _sdp_iocb_q_cancel_write_pending - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn,
+					     ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -105,28 +95,29 @@
 		(struct sdpc_iocb *) sdp_desc_q_lookup(&conn->send_queue,
 						    _sdp_iocb_q_cancel_lookup_func,
 						    NULL))) {
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		result = sdp_iocb_complete(iocb, error);
 		SDP_EXPECT(!(0 > result));
 	}
 
-	return 0;
+	return;
 } /* _sdp_iocb_q_cancel_write_pending */
 
 /*
  * _sdp_iocb_q_cancel_write_source - cancel all pending source AIOs
  */
-static int _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn,
+					    ssize_t error)
 {
-	return sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
+	return;
 } /* _sdp_iocb_q_cancel_write_source */
 
 /*
  * _sdp_iocb_q_cancel_write_snk - cancel all pending sink AIOs
  */
-static int _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -134,66 +125,46 @@
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->w_snk,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		result = sdp_iocb_complete(iocb, error);
 		SDP_EXPECT(!(0 > result));
 	}
 
-	return 0;
+	return;
 } /* _sdp_iocb_q_cancel_write_snk */
 
 /* 
  * sdp_iocb_q_cancel_all_read - cancel all outstanding read AIOs
  */
-int sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
+void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
 {
-	int result;
+	_sdp_iocb_q_cancel_read_pending(conn, error);
+	_sdp_iocb_q_cancel_read_snk(conn, error);
+	_sdp_iocb_q_cancel_read_source(conn, error);
 
-	result = _sdp_iocb_q_cancel_read_pending(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_read_snk(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_read_source(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	return 0;
+	return;
 } /* sdp_iocb_q_cancel_all_read */
 
 /*
  * sdp_iocb_q_cancel_all_write - cancel all outstanding write AIOs
  */
-int sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
+void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
 {
-	int result;
+	_sdp_iocb_q_cancel_write_pending(conn, error);
+	_sdp_iocb_q_cancel_write_source(conn, error);
+	_sdp_iocb_q_cancel_write_snk(conn, error);
 
-	result = _sdp_iocb_q_cancel_write_pending(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_write_source(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_write_snk(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	return 0;
+	return;
 } /* sdp_iocb_q_cancel_all_write */
 
 /*
  * sdp_iocb_q_cancel_all - cancel all outstanding AIOs
  */
-int sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
+void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
 {
-	int result;
+	sdp_iocb_q_cancel_all_read(conn, error);
+	sdp_iocb_q_cancel_all_write(conn, error);
 
-	result = sdp_iocb_q_cancel_all_read(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = sdp_iocb_q_cancel_all_write(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	return 0;
+	return;
 } /* sdp_iocb_q_cancel_all */
Index: sdp_write.c
===================================================================
--- sdp_write.c	(revision 1873)
+++ sdp_write.c	(working copy)
@@ -57,9 +57,7 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_desc_q_clear(&conn->w_snk);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_desc_q_clear(&conn->w_snk);
 			result = 0;
 			break;
 		default:
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1873)
+++ sdp_rcvd.c	(working copy)
@@ -454,11 +454,9 @@
 		 * immediate completion
 		 */
 		if (0 < (SDP_ADVT_F_READ & advt->flag)) {
-			advt = sdp_advt_q_get(&conn->src_pend);
-			SDP_EXPECT((NULL != advt));
 
-			result = sdp_advt_q_put(&conn->src_actv, advt);
-			SDP_EXPECT(!(0 > result));
+			sdp_advt_q_put(&conn->src_actv,
+				       sdp_advt_q_get(&conn->src_pend));
 			/*
 			 * keep track of cancellations
 			 */
@@ -780,11 +778,7 @@
 	conn->s_cur_adv = 1;
 	conn->s_par_adv = 0;
 
-	result = sdp_advt_q_put(&conn->snk_pend, advt);
-	if (0 > result) {
-		sdp_dbg_warn(conn, "SnkAvail cannot be saved. <%d>", result);
-		goto advt_error;
-	}
+	sdp_advt_q_put(&conn->snk_pend, advt);
 
 consume:
 	conn->s_wq_cur = SDP_SEND_POST_SLOW;
@@ -812,9 +806,6 @@
 	 * PostRecv will take care of consuming this advertisment, based
 	 * on result.
 	 */
-	return result;
-advt_error:
-	(void)sdp_advt_destroy(advt);
 error:
 	return result;
 } /* _sdp_rcvd_snk_avail */
@@ -946,16 +937,14 @@
 	 */
 	conn->src_recv++;
 
-	result = sdp_advt_q_put(&conn->src_pend, advt);
-	if (0 > result) {
-		sdp_dbg_warn(conn, "Error <%d> queuing SrcAvail", result);
-		goto advt_error;
-	}
+	sdp_advt_q_put(&conn->src_pend, advt);
 	/*
 	 * process any ULP data in the message
 	 */
-	if (0 == size)
+	if (0 == size) {
+		result = 0;
 		goto done;
+	}
 	/*
 	 * update non-discard for sink advertisment management
 	 */
@@ -1116,11 +1105,9 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_buff_q_clear_unmap(&conn->recv_post,
-							conn->ca->dma_device,
-							PCI_DMA_FROMDEVICE);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_buff_q_clear_unmap(&conn->recv_post,
+					       conn->ca->dma_device,
+					       PCI_DMA_FROMDEVICE);
 			result = 0;
 			break;
 		default:
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1873)
+++ sdp_inet.c	(working copy)
@@ -422,8 +422,7 @@
 		 */
 		if (0 < (SDP_ST_MASK_DRAIN & conn->istate)) {
 
-			result = sdp_iocb_q_cancel_all_write(conn, -ECANCELED);
-			SDP_EXPECT(!(0 > result));
+			sdp_iocb_q_cancel_all_write(conn, -ECANCELED);
 
 			result = _sdp_inet_disconnect(conn);
 			SDP_EXPECT(!(0 > result));
@@ -1553,7 +1552,7 @@
 error_buff:
 	(void)sdp_link_addr_cleanup();
 error_link:
-	(void)sdp_main_advt_cleanup();
+	sdp_main_advt_cleanup();
 error_advt:
 	(void)sdp_main_proc_cleanup();
 error_proc:
@@ -1585,7 +1584,7 @@
 	/*
 	 * delete advertisment table
 	 */
-	(void)sdp_main_advt_cleanup();
+	sdp_main_advt_cleanup();
 	/*
 	 * proc tables
 	 */
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1873)
+++ sdp_proto.h	(working copy)
@@ -53,19 +53,20 @@
 
 int sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count);
 
-int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff);
+void sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff);
 
 int sdp_buff_pool_buff_size(void);
 
-int sdp_buff_q_init(struct sdpc_buff_q *pool, u32 size);
+void sdp_buff_q_init(struct sdpc_buff_q *pool);
 
-int sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
-			   struct device *dev,
-			   int direction);
+void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
+			    struct device *dev,
+			    int direction);
 
-static inline int sdp_buff_q_clear(struct sdpc_buff_q *pool)
+static inline void sdp_buff_q_clear(struct sdpc_buff_q *pool)
 {
-	return sdp_buff_q_clear_unmap(pool, NULL, 0);
+	sdp_buff_q_clear_unmap(pool, NULL, 0);
+	return;
 }
 
 int sdp_buff_q_remove(struct sdpc_buff *buff);
@@ -143,11 +144,11 @@
  */
 int sdp_main_advt_init(void);
 
-int sdp_main_advt_cleanup(void);
+void sdp_main_advt_cleanup(void);
 
-int sdp_advt_q_init(struct sdpc_advt_q *table);
+void sdp_advt_q_init(struct sdpc_advt_q *table);
 
-int sdp_advt_q_clear(struct sdpc_advt_q *table);
+void sdp_advt_q_clear(struct sdpc_advt_q *table);
 
 struct sdpc_advt *sdp_advt_create(void);
 
@@ -157,18 +158,18 @@
 
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table);
 
-int sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt);
+void sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt);
 
 /*
  * Zcopy IOCB managment
  */
 int sdp_main_iocb_init(void);
 
-int sdp_main_iocb_cleanup(void);
+void sdp_main_iocb_cleanup(void);
 
-int sdp_iocb_q_init(struct sdpc_iocb_q *table);
+void sdp_iocb_q_init(struct sdpc_iocb_q *table);
 
-int sdp_iocb_q_clear(struct sdpc_iocb_q *table);
+void sdp_iocb_q_clear(struct sdpc_iocb_q *table);
 
 struct sdpc_iocb *sdp_iocb_create(void);
 
@@ -186,9 +187,9 @@
 
 struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key);
 
-int sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
+void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
 
-int sdp_iocb_q_remove(struct sdpc_iocb *iocb);
+void sdp_iocb_q_remove(struct sdpc_iocb *iocb);
 
 int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn);
 
@@ -203,7 +204,7 @@
 /*
  * Generic object managment
  */
-int sdp_desc_q_remove(struct sdpc_desc *element);
+void sdp_desc_q_remove(struct sdpc_desc *element);
 
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table);
 
@@ -233,9 +234,9 @@
 int sdp_desc_q_types_size(struct sdpc_desc_q *table,
 			  enum sdp_desc_type type);
 
-int sdp_desc_q_init(struct sdpc_desc_q *table);
+void sdp_desc_q_init(struct sdpc_desc_q *table);
 
-int sdp_desc_q_clear(struct sdpc_desc_q *table);
+void sdp_desc_q_clear(struct sdpc_desc_q *table);
 
 /*
  * proc entry managment
@@ -422,11 +423,11 @@
 		  size_t size, 
 		  int    flags);
 
-int sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error);
+void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error);
 
-int sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error);
+void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error);
 
-int sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error);
+void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error);
 
 /*
  * link address information
Index: sdp_read.c
===================================================================
--- sdp_read.c	(revision 1873)
+++ sdp_read.c	(working copy)
@@ -123,9 +123,7 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_desc_q_clear(&conn->r_src);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_desc_q_clear(&conn->r_src);
 			result = 0;
 			break;
 		default:
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1873)
+++ sdp_send.c	(working copy)
@@ -124,10 +124,8 @@
 	/*
 	 * check queue membership. (first send attempt vs. flush)
 	 */
-	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff)) {
-		result = sdp_desc_q_remove((struct sdpc_desc *) buff);
-		SDP_EXPECT(!(0 > result));
-	}
+	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff))
+		sdp_desc_q_remove((struct sdpc_desc *) buff);
 	/*
 	 * save the buffer for the event handler.
 	 */
@@ -371,10 +369,8 @@
 	/*
 	 * dequeue if needed and the queue buffer
 	 */
-	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff)) {
-		result = sdp_desc_q_remove((struct sdpc_desc *) buff);
-		SDP_EXPECT(!(0 > result));
-	}
+	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff))
+		sdp_desc_q_remove((struct sdpc_desc *) buff);
 
 	result = sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff);
         if (0 > result) {
@@ -872,15 +868,11 @@
 			 * callers to determine the fate of the IOCB on
 			 * failure or partial processing.
 			 */
-			if (0 < sdp_desc_q_member((struct sdpc_desc *)iocb)) {
-				result =
-				    sdp_desc_q_remove((struct sdpc_desc *)
-							    iocb);
-				SDP_EXPECT(!(0 > result));
-			}
+			if (0 < sdp_desc_q_member((struct sdpc_desc *)iocb))
+				sdp_desc_q_remove((struct sdpc_desc *)iocb);
 
 			result = sdp_desc_q_put_tail(&conn->w_snk,
-						     (struct sdpc_desc *) iocb);
+						     (struct sdpc_desc *)iocb);
 			if (0 > result) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> queuing write IOCB.",
@@ -906,10 +898,8 @@
 			/*
 			 * complete this IOCB
 			 */
-			if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb)) {
-				result = sdp_desc_q_remove((struct sdpc_desc *) iocb);
-				SDP_EXPECT(!(0 > result));
-			}
+			if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb))
+				sdp_desc_q_remove((struct sdpc_desc *) iocb);
 
 			SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
 			SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
@@ -929,10 +919,8 @@
 		/*
 		 * queue IOCB
 		 */
-		if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb)) {
-			result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
-			SDP_EXPECT(!(0 > result));
-		}
+		if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb))
+			sdp_desc_q_remove((struct sdpc_desc *)iocb);
 
 		result = sdp_iocb_q_put_tail(&conn->w_src, iocb);
 		if (0 > result)
@@ -1840,8 +1828,7 @@
 		 * always remove the IOCB.
 		 * If active, then place it into the correct active queue
 		 */
-		result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_desc_q_remove((struct sdpc_desc *)iocb);
 		
 		if (0 < (SDP_IOCB_F_ACTIVE & iocb->flags)) {
 			if (0 < (SDP_IOCB_F_RDMA_W & iocb->flags)) {
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1873)
+++ sdp_conn.c	(working copy)
@@ -638,38 +638,38 @@
 	 * really there shouldn't be anything in these tables, but it's
 	 * really bad if we leave a dangling reference here.
 	 */
-	(void)sdp_iocb_q_cancel_all(conn, -ECANCELED);
-	(void)sdp_iocb_q_clear(&conn->r_pend);
-	(void)sdp_iocb_q_clear(&conn->r_snk);
-	(void)sdp_iocb_q_clear(&conn->w_src);
+	sdp_iocb_q_cancel_all(conn, -ECANCELED);
+	sdp_iocb_q_clear(&conn->r_pend);
+	sdp_iocb_q_clear(&conn->r_snk);
+	sdp_iocb_q_clear(&conn->w_src);
 
-	(void)sdp_desc_q_clear(&conn->r_src);
-	(void)sdp_desc_q_clear(&conn->w_snk);
+	sdp_desc_q_clear(&conn->r_src);
+	sdp_desc_q_clear(&conn->w_snk);
 	/*
 	 * clear the buffer pools
 	 */
-	(void)sdp_buff_q_clear(&conn->recv_pool);
+	sdp_buff_q_clear(&conn->recv_pool);
 
 	if (conn->ca) {
-		(void)sdp_buff_q_clear_unmap(&conn->send_post,
-					     conn->ca->dma_device,
-					     PCI_DMA_TODEVICE);
-		(void)sdp_buff_q_clear_unmap(&conn->recv_post,
-					     conn->ca->dma_device,
-					     PCI_DMA_FROMDEVICE);
+		sdp_buff_q_clear_unmap(&conn->send_post,
+				       conn->ca->dma_device,
+				       PCI_DMA_TODEVICE);
+		sdp_buff_q_clear_unmap(&conn->recv_post,
+				       conn->ca->dma_device,
+				       PCI_DMA_FROMDEVICE);
 	}
 
 	/*
 	 * clear advertisment tables
 	 */
-	(void)sdp_advt_q_clear(&conn->src_pend);
-	(void)sdp_advt_q_clear(&conn->src_actv);
-	(void)sdp_advt_q_clear(&conn->snk_pend);
+	sdp_advt_q_clear(&conn->src_pend);
+	sdp_advt_q_clear(&conn->src_actv);
+	sdp_advt_q_clear(&conn->snk_pend);
 	/*
 	 * generic table clear
 	 */
-	(void)sdp_desc_q_clear(&conn->send_ctrl);
-	(void)sdp_desc_q_clear(&conn->send_queue);
+	sdp_desc_q_clear(&conn->send_ctrl);
+	sdp_desc_q_clear(&conn->send_queue);
 	/*
 	 * If the QP owner is not the CM, then destroy.
 	 */
@@ -908,13 +908,13 @@
 /*
  * _sdp_conn_lock_init - initialize connection lock
  */
-static int _sdp_conn_lock_init(struct sdp_opt *conn)
+static void _sdp_conn_lock_init(struct sdp_opt *conn)
 {
 	spin_lock_init(&(conn->lock.slock));
 	conn->lock.users = 0;
 	init_waitqueue_head(&(conn->lock.waitq));
 
-	return 0;
+	return;
 } /* _sdp_conn_lock_init */
 
 /*
@@ -1233,47 +1233,33 @@
 	/*
 	 * generic send queue
 	 */
-	result = sdp_desc_q_init(&conn->send_queue);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_desc_q_init(&conn->send_ctrl);
-	SDP_EXPECT(!(0 > result));
+	sdp_desc_q_init(&conn->send_queue);
+	sdp_desc_q_init(&conn->send_ctrl);
 	/*
 	 * create buffer pools for posted events
 	 */
-	result = sdp_buff_q_init(&conn->recv_post, 0);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_buff_q_init(&conn->recv_pool, 0);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_buff_q_init(&conn->send_post, 0);
-	SDP_EXPECT(!(0 > result));
+	sdp_buff_q_init(&conn->recv_post);
+	sdp_buff_q_init(&conn->recv_pool);
+	sdp_buff_q_init(&conn->send_post);
 	/*
 	 * initialize zcopy advertisment tables
 	 */
-	result = sdp_advt_q_init(&conn->src_pend);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_advt_q_init(&conn->src_actv);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_advt_q_init(&conn->snk_pend);
-	SDP_EXPECT(!(0 > result));
+	sdp_advt_q_init(&conn->src_pend);
+	sdp_advt_q_init(&conn->src_actv);
+	sdp_advt_q_init(&conn->snk_pend);
 	/*
 	 * initialize zcopy iocb tables
 	 */
-	result = sdp_iocb_q_init(&conn->r_pend);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_iocb_q_init(&conn->r_snk);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_iocb_q_init(&conn->w_src);
-	SDP_EXPECT(!(0 > result));
+	sdp_iocb_q_init(&conn->r_pend);
+	sdp_iocb_q_init(&conn->r_snk);
+	sdp_iocb_q_init(&conn->w_src);
 
-	result = sdp_desc_q_init(&conn->r_src);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_desc_q_init(&conn->w_snk);
-	SDP_EXPECT(!(0 > result));
+	sdp_desc_q_init(&conn->r_src);
+	sdp_desc_q_init(&conn->w_snk);
 	/*
 	 * connection lock
 	 */
-	result = _sdp_conn_lock_init(conn);
-	SDP_EXPECT(!(0 > result));
+	_sdp_conn_lock_init(conn);
 	/*
 	 * insert connection into lookup table
 	 */
@@ -2007,7 +1993,7 @@
 error_sock:
 	kmem_cache_destroy(_dev_root_s.conn_cache);
 error_conn:
-	(void)sdp_main_iocb_cleanup();
+	sdp_main_iocb_cleanup();
 error_iocb:
 	_dev_root_s.sk_array--;
 	free_pages((unsigned long)_dev_root_s.sk_array, _dev_root_s.sk_ordr);
@@ -2057,7 +2043,7 @@
 	/*
 	 * delete IOCB table
 	 */
-	(void)sdp_main_iocb_cleanup();
+	sdp_main_iocb_cleanup();
 
 	return 0;
 } /* sdp_conn_table_clear */
Index: sdp_advt.c
===================================================================
--- sdp_advt.c	(revision 1873)
+++ sdp_advt.c	(working copy)
@@ -125,7 +125,7 @@
 /*
  * sdp_advt_q_put - put the advertisment object at the tables tail
  */
-int sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt)
+void sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt)
 {
 	struct sdpc_advt *next;
 	struct sdpc_advt *prev;
@@ -147,24 +147,24 @@
 
 	table->size++;
 
-	return 0;
+	return;
 } /* sdp_advt_q_put */
 
 /*
  * sdp_advt_q_init - initialize a new empty advertisment table
  */
-int sdp_advt_q_init(struct sdpc_advt_q *table)
+void sdp_advt_q_init(struct sdpc_advt_q *table)
 {
 	table->head = NULL;
 	table->size = 0;
 
-	return 0;
+	return;
 } /* sdp_advt_q_init */
 
 /*
  * sdp_advt_q_clear - clear the contents of an advertisment table
  */
-int sdp_advt_q_clear(struct sdpc_advt_q *table)
+void sdp_advt_q_clear(struct sdpc_advt_q *table)
 {
 	struct sdpc_advt *advt;
 	int result;
@@ -177,7 +177,7 @@
 		SDP_EXPECT(!(0 > result));
 	}
 
-	return 0;
+	return;
 } /* sdp_advt_q_clear */
 
 /*
@@ -189,8 +189,6 @@
  */
 int sdp_main_advt_init(void)
 {
-	int result;
-
 	sdp_dbg_init("Advertisment cache initialization.");
 	/*
 	 * initialize the caches only once.
@@ -204,20 +202,16 @@
 					     sizeof(struct sdpc_advt),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (NULL == __sdp_advt_cache) {
-		result = -ENOMEM;
-		goto error_advt_c;
-	}
+	if (NULL == __sdp_advt_cache)
+		return -ENOMEM;
 
 	return 0;
-error_advt_c:
-	return 0;
 } /* sdp_main_advt_init */
 
 /*
  * sdp_main_advt_cleanup - cleanup the advertisment caches.
  */
-int sdp_main_advt_cleanup(void)
+void sdp_main_advt_cleanup(void)
 {
 	sdp_dbg_init("Advertisment cache cleanup.");
 	/*
@@ -229,5 +223,5 @@
 	 */
 	__sdp_advt_cache = NULL;
 
-	return 0;
+	return;
 } /* sdp_main_advt_cleanup */
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1873)
+++ sdp_recv.c	(working copy)
@@ -172,24 +172,9 @@
 	 * If there is no more advertised space move the advertisment to the
 	 * active list, and match the WRID.
 	 */
-	if (!(0 < advt->size)) {
-		advt = sdp_advt_q_get(&conn->src_pend);
-		if (NULL == advt) {
-			sdp_dbg_warn(conn, "SrcAvail disappeared. <%d>",
-				     sdp_advt_q_size(&conn->src_pend));
-			result = -ENODEV;
-			goto drop;
-		}
-
-		result = sdp_advt_q_put(&conn->src_actv, advt);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> queuing active src",
-				     result);
-
-			(void)sdp_advt_destroy(advt);
-			goto drop;
-		}
-	}
+	if (!(0 < advt->size))
+		sdp_advt_q_put(&conn->src_actv,
+			       sdp_advt_q_get(&conn->src_pend));
 	/*
 	 * save the buffer for the event handler. Make sure it's before
 	 * actually posting the thing. Completion event can happen before
@@ -312,23 +297,9 @@
 	 * if there is no more advertised space,  queue the
 	 * advertisment for completion
 	 */
-	if (!(0 < advt->size)) {
-		advt = sdp_advt_q_get(&conn->src_pend);
-		if (NULL == advt) {
-			sdp_dbg_warn(conn, "SrcAvail disappeared. <%d>",
-				     sdp_advt_q_size(&conn->src_pend));
-			result = -ENODEV;
-			goto error;
-		}
-
-		result = sdp_advt_q_put(&conn->src_actv, advt);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> queuing active src",
-				     result);
-			(void)sdp_advt_destroy(advt);
-			goto error;
-		}
-	}
+	if (!(0 < advt->size))
+		sdp_advt_q_put(&conn->src_actv,
+			       sdp_advt_q_get(&conn->src_pend));
 	/*
 	 * if there is no more iocb space queue the it for completion
 	 */
@@ -981,8 +952,7 @@
 		 * the correct active queue. Inactive empty IOCBs can be
 		 * deleted, while inactive partials needs to be compelted.
 		 */
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		if (0 == (SDP_IOCB_F_ACTIVE & iocb->flags)) {
 			if (0 < iocb->post) {
@@ -1225,7 +1195,7 @@
 		return -EOPNOTSUPP;
 
 	if (0 < (MSG_PEEK & flags)) {
-		(void)sdp_buff_q_init(&peek_queue, 0);
+		sdp_buff_q_init(&peek_queue);
 		msg->msg_flags |= MSG_PEEK;
 	}
 
@@ -1356,8 +1326,7 @@
 				 * will be returned to the free pool
 				 * in one group.
 				 */
-				expect = sdp_buff_pool_chain_link(head, buff);
-				SDP_EXPECT(!(0 > expect));
+				sdp_buff_pool_chain_link(head, buff);
 
 				head = buff;
 				free_count++;
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1873)
+++ sdp_wall.c	(working copy)
@@ -182,15 +182,13 @@
 		/*
 		 * clear the pending control buffer.
 		 */
-		result = sdp_desc_q_clear(&conn->send_ctrl);
-		SDP_EXPECT(!(0 > result));
+		sdp_desc_q_clear(&conn->send_ctrl);
 		/*
 		 * fall through
 		 */
 	case SDP_CONN_ST_DIS_SEND_1:	/* don't touch control queue, diconnect
 					   message may still be queued. */
-		result = sdp_desc_q_clear(&conn->send_queue);
-		SDP_EXPECT(!(0 > result));
+		sdp_desc_q_clear(&conn->send_queue);
 		/*
 		 * post abort
 		 */
@@ -283,8 +281,6 @@
  */
 int sdp_wall_recv_reject(struct sdp_opt *conn, int error)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Reject recv. src <%08x:%04x> dst <%08x:%04x> <%d>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port, error);
@@ -297,10 +293,7 @@
 	conn->istate = SDP_SOCK_ST_ERROR;
 	conn->shutdown = SHUTDOWN_MASK;
 
-	result = sdp_iocb_q_cancel_all(conn, (0 - error));
-	if (0 > result)
-		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
-			     result);
+	sdp_iocb_q_cancel_all(conn, (0 - error));
 
 	sdp_inet_wake_error(conn->sk);
 
@@ -411,8 +404,6 @@
  */
 int sdp_wall_recv_close(struct sdp_opt *conn)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Close recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -436,11 +427,7 @@
 		 * cancel all outstanding read AIO's since there will be
 		 * no more data from the peer.
 		 */
-		result = sdp_iocb_q_cancel_all_read(conn, 0);
-		if (0 > result)
-			sdp_dbg_warn(conn, 
-				     "Error <%d> canceling outstanding IOCBs",
-				     result);
+		sdp_iocb_q_cancel_all_read(conn, 0);
 		/*
 		 * async notification. POLL_HUP on full duplex close only.
 		 */
@@ -458,8 +445,6 @@
  */
 int sdp_wall_recv_closing(struct sdp_opt *conn)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Closing recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -475,15 +460,11 @@
 	 * cancel all outstanding read AIO's since there will be no more data
 	 * from the peer.
 	 */
-	result = sdp_iocb_q_cancel_all_read(conn, 0);
-	if (0 > result)
-		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
-			     result);
-
-	sdp_inet_wake_generic(conn->sk);
+	sdp_iocb_q_cancel_all_read(conn, 0);
 	/*
 	 * async notification. POLL_HUP on full duplex close only.
 	 */
+	sdp_inet_wake_generic(conn->sk);
 	sk_wake_async(conn->sk, 1, POLL_HUP);
 
 	return 0;
@@ -494,8 +475,6 @@
  */
 int sdp_wall_recv_abort(struct sdp_opt *conn)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Abort recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -524,10 +503,7 @@
 	/*
 	 * cancel all outstanding IOCBs
 	 */
-	result = sdp_iocb_q_cancel_all(conn, -ECONNRESET);
-	if (0 > result)
-		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
-			     result);
+	sdp_iocb_q_cancel_all(conn, -ECONNRESET);
 
 	sdp_inet_wake_error(conn->sk);
 
@@ -581,10 +557,7 @@
 		/*
 		 * cancel all outstanding IOCBs
 		 */
-		result = sdp_iocb_q_cancel_all(conn, -ECONNRESET);
-		if (0 > result)
-			sdp_dbg_warn(conn, "Error <%d> canceling all IOCBs",
-				     result);
+		sdp_iocb_q_cancel_all(conn, -ECONNRESET);
 
 		sdp_inet_wake_error(conn->sk);
 
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1873)
+++ sdp_sent.c	(working copy)
@@ -340,11 +340,9 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_buff_q_clear_unmap(&conn->send_post,
-							conn->ca->dma_device,
-							PCI_DMA_TODEVICE);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_buff_q_clear_unmap(&conn->send_post,
+					       conn->ca->dma_device,
+					       PCI_DMA_TODEVICE);
 			result = 0;
 			break;
 		default:
@@ -436,8 +434,7 @@
 		 * create a link of buffers which will be returned to
 		 * the free pool in one group.
 		 */
-		result = sdp_buff_pool_chain_link(head, buff);
-		SDP_EXPECT(!(0 > result));
+		sdp_buff_pool_chain_link(head, buff);
 
 		head = buff;
 		free_count++;
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1873)
+++ sdp_iocb.c	(working copy)
@@ -553,7 +553,7 @@
 /*
  * sdp_iocb_q_remove - remove the object from the table
  */
-int sdp_iocb_q_remove(struct sdpc_iocb *iocb)
+void sdp_iocb_q_remove(struct sdpc_iocb *iocb)
 {
 	struct sdpc_iocb_q *table;
 	struct sdpc_iocb *next;
@@ -579,7 +579,7 @@
 	iocb->next = NULL;
 	iocb->prev = NULL;
 
-	return 0;
+	return;
 } /* sdp_iocb_q_remove */
 
 /*
@@ -773,9 +773,9 @@
 /*
  * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
  */
-int sdp_iocb_q_cancel(struct sdpc_iocb_q *table,
-		      u32     mask,
-		      ssize_t comp)
+void sdp_iocb_q_cancel(struct sdpc_iocb_q *table,
+		       u32     mask,
+		       ssize_t comp)
 {
 	struct sdpc_iocb *iocb;
 	struct sdpc_iocb *next;
@@ -797,8 +797,7 @@
 				    iocb->key, comp, iocb->flags, iocb->size,
 				    iocb->post, iocb->len);
 
-			result = sdp_iocb_q_remove(iocb);
-			SDP_EXPECT(!(0 > result));
+			sdp_iocb_q_remove(iocb);
 
 			result = sdp_iocb_complete(iocb, comp);
 			SDP_EXPECT(!(0 > result));
@@ -807,24 +806,24 @@
 		iocb = next;
 	}
 
-	return 0;
+	return;
 } /* sdp_iocb_q_cancel */
 
 /*
  * sdp_iocb_q_init - initialize a new empty IOCB table
  */
-int sdp_iocb_q_init(struct sdpc_iocb_q *table)
+void sdp_iocb_q_init(struct sdpc_iocb_q *table)
 {
 	table->head = NULL;
 	table->size = 0;
 
-	return 0;
+	return;
 } /* sdp_iocb_q_init */
 
 /*
  * sdp_iocb_q_clear - clear the contents of an IOCB table
  */
-int sdp_iocb_q_clear(struct sdpc_iocb_q *table)
+void sdp_iocb_q_clear(struct sdpc_iocb_q *table)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -837,7 +836,7 @@
 		SDP_EXPECT(!(0 > result));
 	}
 
-	return 0;
+	return;
 } /* sdp_iocb_q_clear */
 
 /*
@@ -886,7 +885,7 @@
 /*
  * sdp_main_iocb_cleanup - cleanup the advertisment caches
  */
-int sdp_main_iocb_cleanup(void)
+void sdp_main_iocb_cleanup(void)
 {
 	sdp_dbg_init("IOCB cache cleanup.");
 	/*
@@ -902,5 +901,5 @@
 	 */
 	(void)_sdp_mem_lock_cleanup();
 
-	return 0;
+	return;
 } /* sdp_main_iocb_cleanup */
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1873)
+++ sdp_event.c	(working copy)
@@ -427,9 +427,9 @@
 	 * torn down. Here is a good spot since we know that the QP has gone to
 	 * reset, and pretty much all take downs end up here.
 	 */
-	(void)sdp_buff_q_clear_unmap(&conn->recv_post,
-				     conn->ca->dma_device,
-				     PCI_DMA_FROMDEVICE);
+	sdp_buff_q_clear_unmap(&conn->recv_post,
+			       conn->ca->dma_device,
+			       PCI_DMA_FROMDEVICE);
 	/*
 	 * check state
 	 */
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1873)
+++ sdp_buff.c	(working copy)
@@ -161,26 +161,12 @@
 /*
  * sdp_buff_q_init - Init a pool drawing its buffers from the main pool
  */
-int sdp_buff_q_init(struct sdpc_buff_q *pool, u32 size)
+void sdp_buff_q_init(struct sdpc_buff_q *pool)
 {
-	int counter;
-	int result;
-
 	pool->head = NULL;
 	pool->size = 0;
 
-	for (counter = 0; counter < size; counter++) {
-		result = sdp_buff_q_put(pool, sdp_buff_pool_get());
-		if (0 > result) {
-			result = -ENOMEM;
-			goto error;
-		}
-	}
-
-	return 0;
-error:
-	(void)sdp_buff_q_clear(pool);
-	return result;
+	return;
 } /* sdp_buff_q_init */
 
 /*
@@ -189,13 +175,9 @@
 int sdp_buff_q_remove(struct sdpc_buff *buff)
 {
 	struct sdpc_buff_q *pool;
-	int result;
 
 	pool = buff->pool;
-
-	result = _sdp_buff_q_remove(pool, buff);
-
-	return result;
+	return _sdp_buff_q_remove(pool, buff);
 } /* sdp_buff_q_remove */
 
 /*
@@ -203,11 +185,7 @@
  */
 struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 1, NULL, NULL);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 1, NULL, NULL);
 } /* sdp_buff_q_get */
 
 /*
@@ -215,11 +193,7 @@
  */
 struct sdpc_buff *sdp_buff_q_get_head(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 1, NULL, NULL);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 1, NULL, NULL);
 } /* sdp_buff_q_get_head */
 
 /*
@@ -227,11 +201,7 @@
  */
 struct sdpc_buff *sdp_buff_q_get_tail(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 0, NULL, NULL);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 0, NULL, NULL);
 } /* sdp_buff_q_get_tail */
 
 /*
@@ -239,11 +209,7 @@
  */
 struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_look(pool, 1);
-
-	return buff;
+	return _sdp_buff_q_look(pool, 1);
 } /* sdp_buff_q_look_head */
 
 /*
@@ -254,11 +220,7 @@
 						    void *arg),
 					void *usr_arg)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 1, test, usr_arg);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 1, test, usr_arg);
 } /* sdp_buff_q_fetch_head */
 
 /*
@@ -333,11 +295,7 @@
 int sdp_buff_q_put(struct sdpc_buff_q *pool,
 		   struct sdpc_buff *buff)
 {
-	int result;
-
-	result = _sdp_buff_q_put(pool, buff, 1);
-
-	return result;
+	return _sdp_buff_q_put(pool, buff, 1);
 } /* sdp_buff_q_put */
 
 /*
@@ -346,11 +304,7 @@
 int sdp_buff_q_put_head(struct sdpc_buff_q *pool,
 			struct sdpc_buff *buff)
 {
-	int result;
-
-	result = _sdp_buff_q_put(pool, buff, 1);
-
-	return result;
+	return _sdp_buff_q_put(pool, buff, 1);
 } /* sdp_buff_q_put_head */
 
 /*
@@ -359,19 +313,15 @@
 int sdp_buff_q_put_tail(struct sdpc_buff_q *pool,
 			struct sdpc_buff *buff)
 {
-	int result;
-
-	result = _sdp_buff_q_put(pool, buff, 0);
-
-	return result;
+	return _sdp_buff_q_put(pool, buff, 0);
 } /* sdp_buff_q_put_tail */
 
 /*
  * sdp_buff_q_clear_unmap - clear the buffers out of a specific buffer pool
  */
-int sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
-			   struct device *dev,
-			   int direction)
+void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
+			    struct device *dev,
+			    int direction)
 {
 	int result;
 	struct sdpc_buff *buff;
@@ -387,8 +337,8 @@
 				    result);
 	}
 	
-	return 0;
-} /* sdp_buff_q_clear */
+	return;
+} /* sdp_buff_q_clear_unmap */
 
 /*
  * internal data buffer pool manager
@@ -397,7 +347,7 @@
 /*
  * _sdp_buff_pool_release - release allocated buffers from the main pool
  */
-static int _sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
+static void _sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
 {
 	struct sdpc_buff *buff;
 
@@ -417,13 +367,13 @@
 		kmem_cache_free(m_pool->buff_cache, buff);
 	}
 	
-	return 0;
+	return;
 } /* _sdp_buff_pool_release */
 
 /*
  * _sdp_buff_pool_release_check - check for buffer release from main pool
  */
-static inline int _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
+static inline void _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
 {
 	/*
 	 * If there are more then minimum buffers outstanding, free half of
@@ -439,10 +389,10 @@
 		count = min((m_pool->buff_cur - m_pool->buff_min),
 			    (m_pool->free_mark/2));
 
-		return _sdp_buff_pool_release(m_pool, count);
+		_sdp_buff_pool_release(m_pool, count);
 	}
-	else
-		return 0;
+
+	return;
 } /* _sdp_buff_pool_release_check */
 
 /*
@@ -556,10 +506,8 @@
 	main_pool->free_mark = free_mark;
 
 	spin_lock_init(&main_pool->lock);
+	sdp_buff_q_init(&main_pool->pool);
 
-	result = sdp_buff_q_init(&main_pool->pool, 0);
-	SDP_EXPECT(!(0 > result));
-
 	main_pool->pool_cache = kmem_cache_create("sdp_buff_pool",
 						  sizeof(struct sdpc_buff_q),
 						  0, SLAB_HWCACHE_ALIGN,
@@ -618,7 +566,7 @@
 	/*
 	 * Free all the buffers.
 	 */
-	(void)_sdp_buff_pool_release(main_pool, main_pool->buff_cur);
+	_sdp_buff_pool_release(main_pool, main_pool->buff_cur);
 	/*
 	 * Sanity check that the current number of buffers was released.
 	 */
@@ -732,7 +680,7 @@
 
 	main_pool->pool.size++;
 
-	(void)_sdp_buff_pool_release_check(main_pool);
+	_sdp_buff_pool_release_check(main_pool);
 
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
@@ -742,7 +690,7 @@
 /*
  * sdp_buff_pool_chain_link - create chain of buffers which can be returned
  */
-int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff)
+void sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff)
 {
 	buff->data = buff->head;
 	buff->tail = buff->head;
@@ -759,7 +707,7 @@
 		buff->prev->next = buff;
 	}
 
-	return 0;
+	return;
 } /* sdp_buff_pool_chain_link */
 
 /*
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1873)
+++ sdp_queue.c	(working copy)
@@ -117,7 +117,7 @@
 /*
  * sdp_desc_q_remove - remove a specific element from a table
  */
-int sdp_desc_q_remove(struct sdpc_desc *element)
+void sdp_desc_q_remove(struct sdpc_desc *element)
 {
 	struct sdpc_desc_q *table;
 	struct sdpc_desc *prev;
@@ -138,14 +138,13 @@
 	}
 
 	table->size--;
-	table->count[element->type] -=
-	    ((SDP_DESC_TYPE_NONE > element->type) ? 1 : 0);
-
+	table->count[element->type] -=((SDP_DESC_TYPE_NONE > element->type) ?
+				       1 : 0);
 	element->table = NULL;
 	element->next = NULL;
 	element->prev = NULL;
 
-	return 0;
+	return;
 } /* sdp_desc_q_remove */
 
 /*
@@ -258,20 +257,20 @@
 /*
  * sdp_desc_q_init - initialize a new empty generic table
  */
-int sdp_desc_q_init(struct sdpc_desc_q *table)
+void sdp_desc_q_init(struct sdpc_desc_q *table)
 {
 	table->head = NULL;
 	table->size = 0;
 
 	memset(table, 0, sizeof(struct sdpc_desc_q));
 
-	return 0;
+	return;
 } /* sdp_desc_q_init */
 
 /*
  * sdp_desc_q_clear - clear the contents of a generic table
  */
-int sdp_desc_q_clear(struct sdpc_desc_q *table)
+void sdp_desc_q_clear(struct sdpc_desc_q *table)
 {
 	struct sdpc_desc *element;
 	int result;
@@ -285,5 +284,5 @@
 			SDP_EXPECT(!(0 > result));
 		}
 
-	return 0;
+	return;
 } /* sdp_desc_q_clear */


From roland at topspin.com  Wed Feb 23 13:49:15 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 13:49:15 -0800
Subject: [openib-general] CHECK_NULL
In-Reply-To: <20050223134438.P23420@topspin.com> (Libor Michalek's message
	of "Wed, 23 Feb 2005 13:44:38 -0800")
References: <20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com> <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com> <20050217162413.E7121@topspin.com>
	<20050220001137.GC8268@mellanox.co.il>
	<20050222122801.C23420@topspin.com>
	<20050223110143.GZ23837@mellanox.co.il>
	<20050223114634.O23420@topspin.com>
	<20050223134438.P23420@topspin.com>
Message-ID: <52hdk3aqw4.fsf@topspin.com>

There's no need for a return statement at the end of a void function
as in code like:

 > -static int _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn, ssize_t error)
 > +static void _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn,
 > +					    ssize_t error)
 >  {
 > -	return sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL,
 > -					error);
 > +	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
 > +	return;
 >  } /* _sdp_iocb_q_cancel_read_pending */

 - R.


From libor at topspin.com  Wed Feb 23 13:56:29 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 13:56:29 -0800
Subject: [openib-general] CHECK_NULL
In-Reply-To: <52hdk3aqw4.fsf@topspin.com>;
	from roland@topspin.com on Wed, Feb 23, 2005 at 01:49:15PM -0800
References: <52sm3uvo5c.fsf@topspin.com>
	<20050217232800.GC21318@mellanox.co.il>
	<42152D8B.2040608@ichips.intel.com>
	<20050217162413.E7121@topspin.com>
	<20050220001137.GC8268@mellanox.co.il>
	<20050222122801.C23420@topspin.com>
	<20050223110143.GZ23837@mellanox.co.il>
	<20050223114634.O23420@topspin.com>
	<20050223134438.P23420@topspin.com> <52hdk3aqw4.fsf@topspin.com>
Message-ID: <20050223135629.Q23420@topspin.com>

On Wed, Feb 23, 2005 at 01:49:15PM -0800, Roland Dreier wrote:
> There's no need for a return statement at the end of a void function
> as in code like:
>
>  > -	return sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL,
>  > -					error);
>  > +	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
>  > +	return;

  Good point, here's the revised patch.

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: sdp_kvec.c
===================================================================
--- sdp_kvec.c	(revision 1873)
+++ sdp_kvec.c	(working copy)
@@ -35,10 +35,6 @@
 #include "sdp_main.h"
 
 /*
- * static kvec internal functions
- */
-
-/*
  * _sdp_iocb_q_cancel_lookup_func - lookup function for cancelation
  */
 static int _sdp_iocb_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
@@ -47,26 +43,18 @@
 } /* _sdp_iocb_q_cancel_lookup_func */
 
 /*
- * Public write functions
- */
-
-/*
- * Cancel operations
- */
-
-/*
  * _sdp_iocb_q_cancel_read_pending - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn,
+					    ssize_t error)
 {
-	return sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL,
-					error);
+	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_read_pending */
 
 /*
  * _sdp_iocb_q_cancel_read_source - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -74,28 +62,26 @@
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->r_src,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		result = sdp_iocb_complete(iocb, error);
 		SDP_EXPECT(!(0 > result));
 	}
-
-	return 0;
 } /* _sdp_iocb_q_cancel_read_source */
 
 /*
  * _sdp_iocb_q_cancel_read_snk - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
 {
-	return sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_read_snk */
 
 /*
  * _sdp_iocb_q_cancel_write_pending - cancel all pending read AIOs
  */
-static int _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn,
+					     ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -105,28 +91,26 @@
 		(struct sdpc_iocb *) sdp_desc_q_lookup(&conn->send_queue,
 						    _sdp_iocb_q_cancel_lookup_func,
 						    NULL))) {
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		result = sdp_iocb_complete(iocb, error);
 		SDP_EXPECT(!(0 > result));
 	}
-
-	return 0;
 } /* _sdp_iocb_q_cancel_write_pending */
 
 /*
  * _sdp_iocb_q_cancel_write_source - cancel all pending source AIOs
  */
-static int _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn,
+					    ssize_t error)
 {
-	return sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
 } /* _sdp_iocb_q_cancel_write_source */
 
 /*
  * _sdp_iocb_q_cancel_write_snk - cancel all pending sink AIOs
  */
-static int _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
+static void _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -134,66 +118,38 @@
 	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->w_snk,
 								   _sdp_iocb_q_cancel_lookup_func,
 								   NULL))) {
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		result = sdp_iocb_complete(iocb, error);
 		SDP_EXPECT(!(0 > result));
 	}
-
-	return 0;
 } /* _sdp_iocb_q_cancel_write_snk */
 
 /* 
  * sdp_iocb_q_cancel_all_read - cancel all outstanding read AIOs
  */
-int sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
+void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
 {
-	int result;
-
-	result = _sdp_iocb_q_cancel_read_pending(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_read_snk(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_read_source(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	return 0;
+	_sdp_iocb_q_cancel_read_pending(conn, error);
+	_sdp_iocb_q_cancel_read_snk(conn, error);
+	_sdp_iocb_q_cancel_read_source(conn, error);
 } /* sdp_iocb_q_cancel_all_read */
 
 /*
  * sdp_iocb_q_cancel_all_write - cancel all outstanding write AIOs
  */
-int sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
+void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
 {
-	int result;
-
-	result = _sdp_iocb_q_cancel_write_pending(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_write_source(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = _sdp_iocb_q_cancel_write_snk(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	return 0;
+	_sdp_iocb_q_cancel_write_pending(conn, error);
+	_sdp_iocb_q_cancel_write_source(conn, error);
+	_sdp_iocb_q_cancel_write_snk(conn, error);
 } /* sdp_iocb_q_cancel_all_write */
 
 /*
  * sdp_iocb_q_cancel_all - cancel all outstanding AIOs
  */
-int sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
+void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
 {
-	int result;
-
-	result = sdp_iocb_q_cancel_all_read(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	result = sdp_iocb_q_cancel_all_write(conn, error);
-	SDP_EXPECT(!(0 > result));
-
-	return 0;
+	sdp_iocb_q_cancel_all_read(conn, error);
+	sdp_iocb_q_cancel_all_write(conn, error);
 } /* sdp_iocb_q_cancel_all */
Index: sdp_write.c
===================================================================
--- sdp_write.c	(revision 1873)
+++ sdp_write.c	(working copy)
@@ -57,9 +57,7 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_desc_q_clear(&conn->w_snk);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_desc_q_clear(&conn->w_snk);
 			result = 0;
 			break;
 		default:
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1873)
+++ sdp_rcvd.c	(working copy)
@@ -454,11 +454,9 @@
 		 * immediate completion
 		 */
 		if (0 < (SDP_ADVT_F_READ & advt->flag)) {
-			advt = sdp_advt_q_get(&conn->src_pend);
-			SDP_EXPECT((NULL != advt));
 
-			result = sdp_advt_q_put(&conn->src_actv, advt);
-			SDP_EXPECT(!(0 > result));
+			sdp_advt_q_put(&conn->src_actv,
+				       sdp_advt_q_get(&conn->src_pend));
 			/*
 			 * keep track of cancellations
 			 */
@@ -780,11 +778,7 @@
 	conn->s_cur_adv = 1;
 	conn->s_par_adv = 0;
 
-	result = sdp_advt_q_put(&conn->snk_pend, advt);
-	if (0 > result) {
-		sdp_dbg_warn(conn, "SnkAvail cannot be saved. <%d>", result);
-		goto advt_error;
-	}
+	sdp_advt_q_put(&conn->snk_pend, advt);
 
 consume:
 	conn->s_wq_cur = SDP_SEND_POST_SLOW;
@@ -812,9 +806,6 @@
 	 * PostRecv will take care of consuming this advertisment, based
 	 * on result.
 	 */
-	return result;
-advt_error:
-	(void)sdp_advt_destroy(advt);
 error:
 	return result;
 } /* _sdp_rcvd_snk_avail */
@@ -946,16 +937,14 @@
 	 */
 	conn->src_recv++;
 
-	result = sdp_advt_q_put(&conn->src_pend, advt);
-	if (0 > result) {
-		sdp_dbg_warn(conn, "Error <%d> queuing SrcAvail", result);
-		goto advt_error;
-	}
+	sdp_advt_q_put(&conn->src_pend, advt);
 	/*
 	 * process any ULP data in the message
 	 */
-	if (0 == size)
+	if (0 == size) {
+		result = 0;
 		goto done;
+	}
 	/*
 	 * update non-discard for sink advertisment management
 	 */
@@ -1116,11 +1105,9 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_buff_q_clear_unmap(&conn->recv_post,
-							conn->ca->dma_device,
-							PCI_DMA_FROMDEVICE);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_buff_q_clear_unmap(&conn->recv_post,
+					       conn->ca->dma_device,
+					       PCI_DMA_FROMDEVICE);
 			result = 0;
 			break;
 		default:
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1873)
+++ sdp_inet.c	(working copy)
@@ -121,8 +121,6 @@
 		sk_wake_async(sk, 2, POLL_OUT);
 		read_unlock(&sk->sk_callback_lock);
 	}
-
-	return;
 } /* sdp_inet_wake_send */
 
 /*
@@ -138,8 +136,6 @@
 
 		read_unlock(&sk->sk_callback_lock);
 	}
-
-	return;
 } /* sdp_inet_wake_generic */
 
 /*
@@ -155,8 +151,6 @@
 		sk_wake_async(sk, 1, POLL_IN);
 		read_unlock(&sk->sk_callback_lock);
 	}
-
-	return;
 } /* sdp_inet_wake_recv */
 
 /*
@@ -172,8 +166,6 @@
 		sk_wake_async(sk, 0, POLL_ERR);
 		read_unlock(&sk->sk_callback_lock);
 	}
-
-	return;
 } /* sdp_inet_wake_error */
 
 /*
@@ -188,7 +180,6 @@
 	if (NULL != sk)
 		sk_send_sigurg(sk);
 
-	return;
 } /* sdp_inet_wake_urg */
 
 /*
@@ -422,8 +413,7 @@
 		 */
 		if (0 < (SDP_ST_MASK_DRAIN & conn->istate)) {
 
-			result = sdp_iocb_q_cancel_all_write(conn, -ECANCELED);
-			SDP_EXPECT(!(0 > result));
+			sdp_iocb_q_cancel_all_write(conn, -ECANCELED);
 
 			result = _sdp_inet_disconnect(conn);
 			SDP_EXPECT(!(0 > result));
@@ -1553,7 +1543,7 @@
 error_buff:
 	(void)sdp_link_addr_cleanup();
 error_link:
-	(void)sdp_main_advt_cleanup();
+	sdp_main_advt_cleanup();
 error_advt:
 	(void)sdp_main_proc_cleanup();
 error_proc:
@@ -1585,13 +1575,11 @@
 	/*
 	 * delete advertisment table
 	 */
-	(void)sdp_main_advt_cleanup();
+	sdp_main_advt_cleanup();
 	/*
 	 * proc tables
 	 */
 	(void)sdp_main_proc_cleanup();
-
-	return;
 } /* sdp_exit */
 
 module_init(sdp_init);
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1873)
+++ sdp_proto.h	(working copy)
@@ -53,19 +53,19 @@
 
 int sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count);
 
-int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff);
+void sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff);
 
 int sdp_buff_pool_buff_size(void);
 
-int sdp_buff_q_init(struct sdpc_buff_q *pool, u32 size);
+void sdp_buff_q_init(struct sdpc_buff_q *pool);
 
-int sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
-			   struct device *dev,
-			   int direction);
+void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
+			    struct device *dev,
+			    int direction);
 
-static inline int sdp_buff_q_clear(struct sdpc_buff_q *pool)
+static inline void sdp_buff_q_clear(struct sdpc_buff_q *pool)
 {
-	return sdp_buff_q_clear_unmap(pool, NULL, 0);
+	sdp_buff_q_clear_unmap(pool, NULL, 0);
 }
 
 int sdp_buff_q_remove(struct sdpc_buff *buff);
@@ -143,11 +143,11 @@
  */
 int sdp_main_advt_init(void);
 
-int sdp_main_advt_cleanup(void);
+void sdp_main_advt_cleanup(void);
 
-int sdp_advt_q_init(struct sdpc_advt_q *table);
+void sdp_advt_q_init(struct sdpc_advt_q *table);
 
-int sdp_advt_q_clear(struct sdpc_advt_q *table);
+void sdp_advt_q_clear(struct sdpc_advt_q *table);
 
 struct sdpc_advt *sdp_advt_create(void);
 
@@ -157,18 +157,18 @@
 
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table);
 
-int sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt);
+void sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt);
 
 /*
  * Zcopy IOCB managment
  */
 int sdp_main_iocb_init(void);
 
-int sdp_main_iocb_cleanup(void);
+void sdp_main_iocb_cleanup(void);
 
-int sdp_iocb_q_init(struct sdpc_iocb_q *table);
+void sdp_iocb_q_init(struct sdpc_iocb_q *table);
 
-int sdp_iocb_q_clear(struct sdpc_iocb_q *table);
+void sdp_iocb_q_clear(struct sdpc_iocb_q *table);
 
 struct sdpc_iocb *sdp_iocb_create(void);
 
@@ -186,9 +186,9 @@
 
 struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key);
 
-int sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
+void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
 
-int sdp_iocb_q_remove(struct sdpc_iocb *iocb);
+void sdp_iocb_q_remove(struct sdpc_iocb *iocb);
 
 int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn);
 
@@ -203,7 +203,7 @@
 /*
  * Generic object managment
  */
-int sdp_desc_q_remove(struct sdpc_desc *element);
+void sdp_desc_q_remove(struct sdpc_desc *element);
 
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table);
 
@@ -233,9 +233,9 @@
 int sdp_desc_q_types_size(struct sdpc_desc_q *table,
 			  enum sdp_desc_type type);
 
-int sdp_desc_q_init(struct sdpc_desc_q *table);
+void sdp_desc_q_init(struct sdpc_desc_q *table);
 
-int sdp_desc_q_clear(struct sdpc_desc_q *table);
+void sdp_desc_q_clear(struct sdpc_desc_q *table);
 
 /*
  * proc entry managment
@@ -422,11 +422,11 @@
 		  size_t size, 
 		  int    flags);
 
-int sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error);
+void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error);
 
-int sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error);
+void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error);
 
-int sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error);
+void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error);
 
 /*
  * link address information
Index: sdp_read.c
===================================================================
--- sdp_read.c	(revision 1873)
+++ sdp_read.c	(working copy)
@@ -123,9 +123,7 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_desc_q_clear(&conn->r_src);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_desc_q_clear(&conn->r_src);
 			result = 0;
 			break;
 		default:
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1873)
+++ sdp_send.c	(working copy)
@@ -124,10 +124,8 @@
 	/*
 	 * check queue membership. (first send attempt vs. flush)
 	 */
-	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff)) {
-		result = sdp_desc_q_remove((struct sdpc_desc *) buff);
-		SDP_EXPECT(!(0 > result));
-	}
+	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff))
+		sdp_desc_q_remove((struct sdpc_desc *) buff);
 	/*
 	 * save the buffer for the event handler.
 	 */
@@ -371,10 +369,8 @@
 	/*
 	 * dequeue if needed and the queue buffer
 	 */
-	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff)) {
-		result = sdp_desc_q_remove((struct sdpc_desc *) buff);
-		SDP_EXPECT(!(0 > result));
-	}
+	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff))
+		sdp_desc_q_remove((struct sdpc_desc *) buff);
 
 	result = sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff);
         if (0 > result) {
@@ -872,15 +868,11 @@
 			 * callers to determine the fate of the IOCB on
 			 * failure or partial processing.
 			 */
-			if (0 < sdp_desc_q_member((struct sdpc_desc *)iocb)) {
-				result =
-				    sdp_desc_q_remove((struct sdpc_desc *)
-							    iocb);
-				SDP_EXPECT(!(0 > result));
-			}
+			if (0 < sdp_desc_q_member((struct sdpc_desc *)iocb))
+				sdp_desc_q_remove((struct sdpc_desc *)iocb);
 
 			result = sdp_desc_q_put_tail(&conn->w_snk,
-						     (struct sdpc_desc *) iocb);
+						     (struct sdpc_desc *)iocb);
 			if (0 > result) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> queuing write IOCB.",
@@ -906,10 +898,8 @@
 			/*
 			 * complete this IOCB
 			 */
-			if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb)) {
-				result = sdp_desc_q_remove((struct sdpc_desc *) iocb);
-				SDP_EXPECT(!(0 > result));
-			}
+			if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb))
+				sdp_desc_q_remove((struct sdpc_desc *) iocb);
 
 			SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
 			SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
@@ -929,10 +919,8 @@
 		/*
 		 * queue IOCB
 		 */
-		if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb)) {
-			result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
-			SDP_EXPECT(!(0 > result));
-		}
+		if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb))
+			sdp_desc_q_remove((struct sdpc_desc *)iocb);
 
 		result = sdp_iocb_q_put_tail(&conn->w_src, iocb);
 		if (0 > result)
@@ -1840,8 +1828,7 @@
 		 * always remove the IOCB.
 		 * If active, then place it into the correct active queue
 		 */
-		result = sdp_desc_q_remove((struct sdpc_desc *)iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_desc_q_remove((struct sdpc_desc *)iocb);
 		
 		if (0 < (SDP_IOCB_F_ACTIVE & iocb->flags)) {
 			if (0 < (SDP_IOCB_F_RDMA_W & iocb->flags)) {
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1873)
+++ sdp_conn.c	(working copy)
@@ -638,38 +638,38 @@
 	 * really there shouldn't be anything in these tables, but it's
 	 * really bad if we leave a dangling reference here.
 	 */
-	(void)sdp_iocb_q_cancel_all(conn, -ECANCELED);
-	(void)sdp_iocb_q_clear(&conn->r_pend);
-	(void)sdp_iocb_q_clear(&conn->r_snk);
-	(void)sdp_iocb_q_clear(&conn->w_src);
+	sdp_iocb_q_cancel_all(conn, -ECANCELED);
+	sdp_iocb_q_clear(&conn->r_pend);
+	sdp_iocb_q_clear(&conn->r_snk);
+	sdp_iocb_q_clear(&conn->w_src);
 
-	(void)sdp_desc_q_clear(&conn->r_src);
-	(void)sdp_desc_q_clear(&conn->w_snk);
+	sdp_desc_q_clear(&conn->r_src);
+	sdp_desc_q_clear(&conn->w_snk);
 	/*
 	 * clear the buffer pools
 	 */
-	(void)sdp_buff_q_clear(&conn->recv_pool);
+	sdp_buff_q_clear(&conn->recv_pool);
 
 	if (conn->ca) {
-		(void)sdp_buff_q_clear_unmap(&conn->send_post,
-					     conn->ca->dma_device,
-					     PCI_DMA_TODEVICE);
-		(void)sdp_buff_q_clear_unmap(&conn->recv_post,
-					     conn->ca->dma_device,
-					     PCI_DMA_FROMDEVICE);
+		sdp_buff_q_clear_unmap(&conn->send_post,
+				       conn->ca->dma_device,
+				       PCI_DMA_TODEVICE);
+		sdp_buff_q_clear_unmap(&conn->recv_post,
+				       conn->ca->dma_device,
+				       PCI_DMA_FROMDEVICE);
 	}
 
 	/*
 	 * clear advertisment tables
 	 */
-	(void)sdp_advt_q_clear(&conn->src_pend);
-	(void)sdp_advt_q_clear(&conn->src_actv);
-	(void)sdp_advt_q_clear(&conn->snk_pend);
+	sdp_advt_q_clear(&conn->src_pend);
+	sdp_advt_q_clear(&conn->src_actv);
+	sdp_advt_q_clear(&conn->snk_pend);
 	/*
 	 * generic table clear
 	 */
-	(void)sdp_desc_q_clear(&conn->send_ctrl);
-	(void)sdp_desc_q_clear(&conn->send_queue);
+	sdp_desc_q_clear(&conn->send_ctrl);
+	sdp_desc_q_clear(&conn->send_queue);
 	/*
 	 * If the QP owner is not the CM, then destroy.
 	 */
@@ -908,13 +908,11 @@
 /*
  * _sdp_conn_lock_init - initialize connection lock
  */
-static int _sdp_conn_lock_init(struct sdp_opt *conn)
+static void _sdp_conn_lock_init(struct sdp_opt *conn)
 {
 	spin_lock_init(&(conn->lock.slock));
 	conn->lock.users = 0;
 	init_waitqueue_head(&(conn->lock.waitq));
-
-	return 0;
 } /* _sdp_conn_lock_init */
 
 /*
@@ -1233,47 +1231,33 @@
 	/*
 	 * generic send queue
 	 */
-	result = sdp_desc_q_init(&conn->send_queue);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_desc_q_init(&conn->send_ctrl);
-	SDP_EXPECT(!(0 > result));
+	sdp_desc_q_init(&conn->send_queue);
+	sdp_desc_q_init(&conn->send_ctrl);
 	/*
 	 * create buffer pools for posted events
 	 */
-	result = sdp_buff_q_init(&conn->recv_post, 0);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_buff_q_init(&conn->recv_pool, 0);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_buff_q_init(&conn->send_post, 0);
-	SDP_EXPECT(!(0 > result));
+	sdp_buff_q_init(&conn->recv_post);
+	sdp_buff_q_init(&conn->recv_pool);
+	sdp_buff_q_init(&conn->send_post);
 	/*
 	 * initialize zcopy advertisment tables
 	 */
-	result = sdp_advt_q_init(&conn->src_pend);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_advt_q_init(&conn->src_actv);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_advt_q_init(&conn->snk_pend);
-	SDP_EXPECT(!(0 > result));
+	sdp_advt_q_init(&conn->src_pend);
+	sdp_advt_q_init(&conn->src_actv);
+	sdp_advt_q_init(&conn->snk_pend);
 	/*
 	 * initialize zcopy iocb tables
 	 */
-	result = sdp_iocb_q_init(&conn->r_pend);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_iocb_q_init(&conn->r_snk);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_iocb_q_init(&conn->w_src);
-	SDP_EXPECT(!(0 > result));
+	sdp_iocb_q_init(&conn->r_pend);
+	sdp_iocb_q_init(&conn->r_snk);
+	sdp_iocb_q_init(&conn->w_src);
 
-	result = sdp_desc_q_init(&conn->r_src);
-	SDP_EXPECT(!(0 > result));
-	result = sdp_desc_q_init(&conn->w_snk);
-	SDP_EXPECT(!(0 > result));
+	sdp_desc_q_init(&conn->r_src);
+	sdp_desc_q_init(&conn->w_snk);
 	/*
 	 * connection lock
 	 */
-	result = _sdp_conn_lock_init(conn);
-	SDP_EXPECT(!(0 > result));
+	_sdp_conn_lock_init(conn);
 	/*
 	 * insert connection into lookup table
 	 */
@@ -2007,7 +1991,7 @@
 error_sock:
 	kmem_cache_destroy(_dev_root_s.conn_cache);
 error_conn:
-	(void)sdp_main_iocb_cleanup();
+	sdp_main_iocb_cleanup();
 error_iocb:
 	_dev_root_s.sk_array--;
 	free_pages((unsigned long)_dev_root_s.sk_array, _dev_root_s.sk_ordr);
@@ -2057,7 +2041,7 @@
 	/*
 	 * delete IOCB table
 	 */
-	(void)sdp_main_iocb_cleanup();
+	sdp_main_iocb_cleanup();
 
 	return 0;
 } /* sdp_conn_table_clear */
Index: sdp_advt.c
===================================================================
--- sdp_advt.c	(revision 1873)
+++ sdp_advt.c	(working copy)
@@ -125,7 +125,7 @@
 /*
  * sdp_advt_q_put - put the advertisment object at the tables tail
  */
-int sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt)
+void sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt)
 {
 	struct sdpc_advt *next;
 	struct sdpc_advt *prev;
@@ -146,25 +146,21 @@
 	}
 
 	table->size++;
-
-	return 0;
 } /* sdp_advt_q_put */
 
 /*
  * sdp_advt_q_init - initialize a new empty advertisment table
  */
-int sdp_advt_q_init(struct sdpc_advt_q *table)
+void sdp_advt_q_init(struct sdpc_advt_q *table)
 {
 	table->head = NULL;
 	table->size = 0;
-
-	return 0;
 } /* sdp_advt_q_init */
 
 /*
  * sdp_advt_q_clear - clear the contents of an advertisment table
  */
-int sdp_advt_q_clear(struct sdpc_advt_q *table)
+void sdp_advt_q_clear(struct sdpc_advt_q *table)
 {
 	struct sdpc_advt *advt;
 	int result;
@@ -176,8 +172,6 @@
 		result = sdp_advt_destroy(advt);
 		SDP_EXPECT(!(0 > result));
 	}
-
-	return 0;
 } /* sdp_advt_q_clear */
 
 /*
@@ -189,8 +183,6 @@
  */
 int sdp_main_advt_init(void)
 {
-	int result;
-
 	sdp_dbg_init("Advertisment cache initialization.");
 	/*
 	 * initialize the caches only once.
@@ -204,20 +196,16 @@
 					     sizeof(struct sdpc_advt),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (NULL == __sdp_advt_cache) {
-		result = -ENOMEM;
-		goto error_advt_c;
-	}
+	if (NULL == __sdp_advt_cache)
+		return -ENOMEM;
 
 	return 0;
-error_advt_c:
-	return 0;
 } /* sdp_main_advt_init */
 
 /*
  * sdp_main_advt_cleanup - cleanup the advertisment caches.
  */
-int sdp_main_advt_cleanup(void)
+void sdp_main_advt_cleanup(void)
 {
 	sdp_dbg_init("Advertisment cache cleanup.");
 	/*
@@ -228,6 +216,4 @@
 	 * null out entries.
 	 */
 	__sdp_advt_cache = NULL;
-
-	return 0;
 } /* sdp_main_advt_cleanup */
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1873)
+++ sdp_recv.c	(working copy)
@@ -172,24 +172,9 @@
 	 * If there is no more advertised space move the advertisment to the
 	 * active list, and match the WRID.
 	 */
-	if (!(0 < advt->size)) {
-		advt = sdp_advt_q_get(&conn->src_pend);
-		if (NULL == advt) {
-			sdp_dbg_warn(conn, "SrcAvail disappeared. <%d>",
-				     sdp_advt_q_size(&conn->src_pend));
-			result = -ENODEV;
-			goto drop;
-		}
-
-		result = sdp_advt_q_put(&conn->src_actv, advt);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> queuing active src",
-				     result);
-
-			(void)sdp_advt_destroy(advt);
-			goto drop;
-		}
-	}
+	if (!(0 < advt->size))
+		sdp_advt_q_put(&conn->src_actv,
+			       sdp_advt_q_get(&conn->src_pend));
 	/*
 	 * save the buffer for the event handler. Make sure it's before
 	 * actually posting the thing. Completion event can happen before
@@ -312,23 +297,9 @@
 	 * if there is no more advertised space,  queue the
 	 * advertisment for completion
 	 */
-	if (!(0 < advt->size)) {
-		advt = sdp_advt_q_get(&conn->src_pend);
-		if (NULL == advt) {
-			sdp_dbg_warn(conn, "SrcAvail disappeared. <%d>",
-				     sdp_advt_q_size(&conn->src_pend));
-			result = -ENODEV;
-			goto error;
-		}
-
-		result = sdp_advt_q_put(&conn->src_actv, advt);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> queuing active src",
-				     result);
-			(void)sdp_advt_destroy(advt);
-			goto error;
-		}
-	}
+	if (!(0 < advt->size))
+		sdp_advt_q_put(&conn->src_actv,
+			       sdp_advt_q_get(&conn->src_pend));
 	/*
 	 * if there is no more iocb space queue the it for completion
 	 */
@@ -981,8 +952,7 @@
 		 * the correct active queue. Inactive empty IOCBs can be
 		 * deleted, while inactive partials needs to be compelted.
 		 */
-		result = sdp_iocb_q_remove(iocb);
-		SDP_EXPECT(!(0 > result));
+		sdp_iocb_q_remove(iocb);
 
 		if (0 == (SDP_IOCB_F_ACTIVE & iocb->flags)) {
 			if (0 < iocb->post) {
@@ -1225,7 +1195,7 @@
 		return -EOPNOTSUPP;
 
 	if (0 < (MSG_PEEK & flags)) {
-		(void)sdp_buff_q_init(&peek_queue, 0);
+		sdp_buff_q_init(&peek_queue);
 		msg->msg_flags |= MSG_PEEK;
 	}
 
@@ -1356,8 +1326,7 @@
 				 * will be returned to the free pool
 				 * in one group.
 				 */
-				expect = sdp_buff_pool_chain_link(head, buff);
-				SDP_EXPECT(!(0 > expect));
+				sdp_buff_pool_chain_link(head, buff);
 
 				head = buff;
 				free_count++;
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1873)
+++ sdp_wall.c	(working copy)
@@ -182,15 +182,13 @@
 		/*
 		 * clear the pending control buffer.
 		 */
-		result = sdp_desc_q_clear(&conn->send_ctrl);
-		SDP_EXPECT(!(0 > result));
+		sdp_desc_q_clear(&conn->send_ctrl);
 		/*
 		 * fall through
 		 */
 	case SDP_CONN_ST_DIS_SEND_1:	/* don't touch control queue, diconnect
 					   message may still be queued. */
-		result = sdp_desc_q_clear(&conn->send_queue);
-		SDP_EXPECT(!(0 > result));
+		sdp_desc_q_clear(&conn->send_queue);
 		/*
 		 * post abort
 		 */
@@ -283,8 +281,6 @@
  */
 int sdp_wall_recv_reject(struct sdp_opt *conn, int error)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Reject recv. src <%08x:%04x> dst <%08x:%04x> <%d>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port, error);
@@ -297,10 +293,7 @@
 	conn->istate = SDP_SOCK_ST_ERROR;
 	conn->shutdown = SHUTDOWN_MASK;
 
-	result = sdp_iocb_q_cancel_all(conn, (0 - error));
-	if (0 > result)
-		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
-			     result);
+	sdp_iocb_q_cancel_all(conn, (0 - error));
 
 	sdp_inet_wake_error(conn->sk);
 
@@ -411,8 +404,6 @@
  */
 int sdp_wall_recv_close(struct sdp_opt *conn)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Close recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -436,11 +427,7 @@
 		 * cancel all outstanding read AIO's since there will be
 		 * no more data from the peer.
 		 */
-		result = sdp_iocb_q_cancel_all_read(conn, 0);
-		if (0 > result)
-			sdp_dbg_warn(conn, 
-				     "Error <%d> canceling outstanding IOCBs",
-				     result);
+		sdp_iocb_q_cancel_all_read(conn, 0);
 		/*
 		 * async notification. POLL_HUP on full duplex close only.
 		 */
@@ -458,8 +445,6 @@
  */
 int sdp_wall_recv_closing(struct sdp_opt *conn)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Closing recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -475,15 +460,11 @@
 	 * cancel all outstanding read AIO's since there will be no more data
 	 * from the peer.
 	 */
-	result = sdp_iocb_q_cancel_all_read(conn, 0);
-	if (0 > result)
-		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
-			     result);
-
-	sdp_inet_wake_generic(conn->sk);
+	sdp_iocb_q_cancel_all_read(conn, 0);
 	/*
 	 * async notification. POLL_HUP on full duplex close only.
 	 */
+	sdp_inet_wake_generic(conn->sk);
 	sk_wake_async(conn->sk, 1, POLL_HUP);
 
 	return 0;
@@ -494,8 +475,6 @@
  */
 int sdp_wall_recv_abort(struct sdp_opt *conn)
 {
-	int result;
-
 	sdp_dbg_ctrl(conn, "Abort recv. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
@@ -524,10 +503,7 @@
 	/*
 	 * cancel all outstanding IOCBs
 	 */
-	result = sdp_iocb_q_cancel_all(conn, -ECONNRESET);
-	if (0 > result)
-		sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs",
-			     result);
+	sdp_iocb_q_cancel_all(conn, -ECONNRESET);
 
 	sdp_inet_wake_error(conn->sk);
 
@@ -581,10 +557,7 @@
 		/*
 		 * cancel all outstanding IOCBs
 		 */
-		result = sdp_iocb_q_cancel_all(conn, -ECONNRESET);
-		if (0 > result)
-			sdp_dbg_warn(conn, "Error <%d> canceling all IOCBs",
-				     result);
+		sdp_iocb_q_cancel_all(conn, -ECONNRESET);
 
 		sdp_inet_wake_error(conn->sk);
 
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1873)
+++ sdp_sent.c	(working copy)
@@ -340,11 +340,9 @@
 			/*
 			 * clear posted buffers from error'd queue
 			 */
-			result = sdp_buff_q_clear_unmap(&conn->send_post,
-							conn->ca->dma_device,
-							PCI_DMA_TODEVICE);
-			SDP_EXPECT(!(0 > result));
-
+			sdp_buff_q_clear_unmap(&conn->send_post,
+					       conn->ca->dma_device,
+					       PCI_DMA_TODEVICE);
 			result = 0;
 			break;
 		default:
@@ -436,8 +434,7 @@
 		 * create a link of buffers which will be returned to
 		 * the free pool in one group.
 		 */
-		result = sdp_buff_pool_chain_link(head, buff);
-		SDP_EXPECT(!(0 > result));
+		sdp_buff_pool_chain_link(head, buff);
 
 		head = buff;
 		free_count++;
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1873)
+++ sdp_iocb.c	(working copy)
@@ -527,7 +527,6 @@
 		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
 
-	return;
 } /* _sdp_iocb_complete */
 
 /*
@@ -553,7 +552,7 @@
 /*
  * sdp_iocb_q_remove - remove the object from the table
  */
-int sdp_iocb_q_remove(struct sdpc_iocb *iocb)
+void sdp_iocb_q_remove(struct sdpc_iocb *iocb)
 {
 	struct sdpc_iocb_q *table;
 	struct sdpc_iocb *next;
@@ -578,8 +577,6 @@
 	iocb->table = NULL;
 	iocb->next = NULL;
 	iocb->prev = NULL;
-
-	return 0;
 } /* sdp_iocb_q_remove */
 
 /*
@@ -773,9 +770,9 @@
 /*
  * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
  */
-int sdp_iocb_q_cancel(struct sdpc_iocb_q *table,
-		      u32     mask,
-		      ssize_t comp)
+void sdp_iocb_q_cancel(struct sdpc_iocb_q *table,
+		       u32     mask,
+		       ssize_t comp)
 {
 	struct sdpc_iocb *iocb;
 	struct sdpc_iocb *next;
@@ -797,8 +794,7 @@
 				    iocb->key, comp, iocb->flags, iocb->size,
 				    iocb->post, iocb->len);
 
-			result = sdp_iocb_q_remove(iocb);
-			SDP_EXPECT(!(0 > result));
+			sdp_iocb_q_remove(iocb);
 
 			result = sdp_iocb_complete(iocb, comp);
 			SDP_EXPECT(!(0 > result));
@@ -806,25 +802,21 @@
 
 		iocb = next;
 	}
-
-	return 0;
 } /* sdp_iocb_q_cancel */
 
 /*
  * sdp_iocb_q_init - initialize a new empty IOCB table
  */
-int sdp_iocb_q_init(struct sdpc_iocb_q *table)
+void sdp_iocb_q_init(struct sdpc_iocb_q *table)
 {
 	table->head = NULL;
 	table->size = 0;
-
-	return 0;
 } /* sdp_iocb_q_init */
 
 /*
  * sdp_iocb_q_clear - clear the contents of an IOCB table
  */
-int sdp_iocb_q_clear(struct sdpc_iocb_q *table)
+void sdp_iocb_q_clear(struct sdpc_iocb_q *table)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -836,8 +828,6 @@
 		result = sdp_iocb_destroy(iocb);
 		SDP_EXPECT(!(0 > result));
 	}
-
-	return 0;
 } /* sdp_iocb_q_clear */
 
 /*
@@ -886,7 +876,7 @@
 /*
  * sdp_main_iocb_cleanup - cleanup the advertisment caches
  */
-int sdp_main_iocb_cleanup(void)
+void sdp_main_iocb_cleanup(void)
 {
 	sdp_dbg_init("IOCB cache cleanup.");
 	/*
@@ -901,6 +891,4 @@
 	 * cleanup memory locking
 	 */
 	(void)_sdp_mem_lock_cleanup();
-
-	return 0;
 } /* sdp_main_iocb_cleanup */
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1873)
+++ sdp_event.c	(working copy)
@@ -427,9 +427,9 @@
 	 * torn down. Here is a good spot since we know that the QP has gone to
 	 * reset, and pretty much all take downs end up here.
 	 */
-	(void)sdp_buff_q_clear_unmap(&conn->recv_post,
-				     conn->ca->dma_device,
-				     PCI_DMA_FROMDEVICE);
+	sdp_buff_q_clear_unmap(&conn->recv_post,
+			       conn->ca->dma_device,
+			       PCI_DMA_FROMDEVICE);
 	/*
 	 * check state
 	 */
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1873)
+++ sdp_buff.c	(working copy)
@@ -161,26 +161,10 @@
 /*
  * sdp_buff_q_init - Init a pool drawing its buffers from the main pool
  */
-int sdp_buff_q_init(struct sdpc_buff_q *pool, u32 size)
+void sdp_buff_q_init(struct sdpc_buff_q *pool)
 {
-	int counter;
-	int result;
-
 	pool->head = NULL;
 	pool->size = 0;
-
-	for (counter = 0; counter < size; counter++) {
-		result = sdp_buff_q_put(pool, sdp_buff_pool_get());
-		if (0 > result) {
-			result = -ENOMEM;
-			goto error;
-		}
-	}
-
-	return 0;
-error:
-	(void)sdp_buff_q_clear(pool);
-	return result;
 } /* sdp_buff_q_init */
 
 /*
@@ -189,13 +173,9 @@
 int sdp_buff_q_remove(struct sdpc_buff *buff)
 {
 	struct sdpc_buff_q *pool;
-	int result;
 
 	pool = buff->pool;
-
-	result = _sdp_buff_q_remove(pool, buff);
-
-	return result;
+	return _sdp_buff_q_remove(pool, buff);
 } /* sdp_buff_q_remove */
 
 /*
@@ -203,11 +183,7 @@
  */
 struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 1, NULL, NULL);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 1, NULL, NULL);
 } /* sdp_buff_q_get */
 
 /*
@@ -215,11 +191,7 @@
  */
 struct sdpc_buff *sdp_buff_q_get_head(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 1, NULL, NULL);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 1, NULL, NULL);
 } /* sdp_buff_q_get_head */
 
 /*
@@ -227,11 +199,7 @@
  */
 struct sdpc_buff *sdp_buff_q_get_tail(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 0, NULL, NULL);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 0, NULL, NULL);
 } /* sdp_buff_q_get_tail */
 
 /*
@@ -239,11 +207,7 @@
  */
 struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_look(pool, 1);
-
-	return buff;
+	return _sdp_buff_q_look(pool, 1);
 } /* sdp_buff_q_look_head */
 
 /*
@@ -254,11 +218,7 @@
 						    void *arg),
 					void *usr_arg)
 {
-	struct sdpc_buff *buff;
-
-	buff = _sdp_buff_q_get(pool, 1, test, usr_arg);
-
-	return buff;
+	return _sdp_buff_q_get(pool, 1, test, usr_arg);
 } /* sdp_buff_q_fetch_head */
 
 /*
@@ -333,11 +293,7 @@
 int sdp_buff_q_put(struct sdpc_buff_q *pool,
 		   struct sdpc_buff *buff)
 {
-	int result;
-
-	result = _sdp_buff_q_put(pool, buff, 1);
-
-	return result;
+	return _sdp_buff_q_put(pool, buff, 1);
 } /* sdp_buff_q_put */
 
 /*
@@ -346,11 +302,7 @@
 int sdp_buff_q_put_head(struct sdpc_buff_q *pool,
 			struct sdpc_buff *buff)
 {
-	int result;
-
-	result = _sdp_buff_q_put(pool, buff, 1);
-
-	return result;
+	return _sdp_buff_q_put(pool, buff, 1);
 } /* sdp_buff_q_put_head */
 
 /*
@@ -359,19 +311,15 @@
 int sdp_buff_q_put_tail(struct sdpc_buff_q *pool,
 			struct sdpc_buff *buff)
 {
-	int result;
-
-	result = _sdp_buff_q_put(pool, buff, 0);
-
-	return result;
+	return _sdp_buff_q_put(pool, buff, 0);
 } /* sdp_buff_q_put_tail */
 
 /*
  * sdp_buff_q_clear_unmap - clear the buffers out of a specific buffer pool
  */
-int sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
-			   struct device *dev,
-			   int direction)
+void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
+			    struct device *dev,
+			    int direction)
 {
 	int result;
 	struct sdpc_buff *buff;
@@ -386,9 +334,7 @@
 			sdp_dbg_err("Error <%d> returning buffer to main",
 				    result);
 	}
-	
-	return 0;
-} /* sdp_buff_q_clear */
+} /* sdp_buff_q_clear_unmap */
 
 /*
  * internal data buffer pool manager
@@ -397,7 +343,7 @@
 /*
  * _sdp_buff_pool_release - release allocated buffers from the main pool
  */
-static int _sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
+static void _sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
 {
 	struct sdpc_buff *buff;
 
@@ -416,14 +362,12 @@
 		free_page((unsigned long)buff->head);
 		kmem_cache_free(m_pool->buff_cache, buff);
 	}
-	
-	return 0;
 } /* _sdp_buff_pool_release */
 
 /*
  * _sdp_buff_pool_release_check - check for buffer release from main pool
  */
-static inline int _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
+static inline void _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
 {
 	/*
 	 * If there are more then minimum buffers outstanding, free half of
@@ -439,10 +383,8 @@
 		count = min((m_pool->buff_cur - m_pool->buff_min),
 			    (m_pool->free_mark/2));
 
-		return _sdp_buff_pool_release(m_pool, count);
+		_sdp_buff_pool_release(m_pool, count);
 	}
-	else
-		return 0;
 } /* _sdp_buff_pool_release_check */
 
 /*
@@ -556,10 +498,8 @@
 	main_pool->free_mark = free_mark;
 
 	spin_lock_init(&main_pool->lock);
+	sdp_buff_q_init(&main_pool->pool);
 
-	result = sdp_buff_q_init(&main_pool->pool, 0);
-	SDP_EXPECT(!(0 > result));
-
 	main_pool->pool_cache = kmem_cache_create("sdp_buff_pool",
 						  sizeof(struct sdpc_buff_q),
 						  0, SLAB_HWCACHE_ALIGN,
@@ -618,7 +558,7 @@
 	/*
 	 * Free all the buffers.
 	 */
-	(void)_sdp_buff_pool_release(main_pool, main_pool->buff_cur);
+	_sdp_buff_pool_release(main_pool, main_pool->buff_cur);
 	/*
 	 * Sanity check that the current number of buffers was released.
 	 */
@@ -639,8 +579,6 @@
 	 * done
 	 */
 	sdp_dbg_init("Main pool destroyed.");
-
-	return;
 } /* sdp_buff_pool_destroy */
 
 /*
@@ -732,7 +670,7 @@
 
 	main_pool->pool.size++;
 
-	(void)_sdp_buff_pool_release_check(main_pool);
+	_sdp_buff_pool_release_check(main_pool);
 
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
@@ -742,7 +680,7 @@
 /*
  * sdp_buff_pool_chain_link - create chain of buffers which can be returned
  */
-int sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff)
+void sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff)
 {
 	buff->data = buff->head;
 	buff->tail = buff->head;
@@ -758,8 +696,6 @@
 		buff->next->prev = buff;
 		buff->prev->next = buff;
 	}
-
-	return 0;
 } /* sdp_buff_pool_chain_link */
 
 /*
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1873)
+++ sdp_queue.c	(working copy)
@@ -117,7 +117,7 @@
 /*
  * sdp_desc_q_remove - remove a specific element from a table
  */
-int sdp_desc_q_remove(struct sdpc_desc *element)
+void sdp_desc_q_remove(struct sdpc_desc *element)
 {
 	struct sdpc_desc_q *table;
 	struct sdpc_desc *prev;
@@ -138,14 +138,11 @@
 	}
 
 	table->size--;
-	table->count[element->type] -=
-	    ((SDP_DESC_TYPE_NONE > element->type) ? 1 : 0);
-
+	table->count[element->type] -=((SDP_DESC_TYPE_NONE > element->type) ?
+				       1 : 0);
 	element->table = NULL;
 	element->next = NULL;
 	element->prev = NULL;
-
-	return 0;
 } /* sdp_desc_q_remove */
 
 /*
@@ -258,20 +255,18 @@
 /*
  * sdp_desc_q_init - initialize a new empty generic table
  */
-int sdp_desc_q_init(struct sdpc_desc_q *table)
+void sdp_desc_q_init(struct sdpc_desc_q *table)
 {
 	table->head = NULL;
 	table->size = 0;
 
 	memset(table, 0, sizeof(struct sdpc_desc_q));
-
-	return 0;
 } /* sdp_desc_q_init */
 
 /*
  * sdp_desc_q_clear - clear the contents of a generic table
  */
-int sdp_desc_q_clear(struct sdpc_desc_q *table)
+void sdp_desc_q_clear(struct sdpc_desc_q *table)
 {
 	struct sdpc_desc *element;
 	int result;
@@ -284,6 +279,4 @@
 			result = element->release(element);
 			SDP_EXPECT(!(0 > result));
 		}
-
-	return 0;
 } /* sdp_desc_q_clear */


From roland at topspin.com  Wed Feb 23 13:58:37 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 13:58:37 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <421CE184.4090803@ichips.intel.com> (Sean Hefty's message of
	"Wed, 23 Feb 2005 12:03:16 -0800")
References: <421CE184.4090803@ichips.intel.com>
Message-ID: <52d5uraqgi.fsf@topspin.com>

Using wr_ids to hold pointers often seems to lead to issues like this.
So it might be best to rework the CM so uses wr_id some other way.

I'll also remind you of the following Sean Hefty quote, circa
September 2004 ;)

    "But, yes, users wanting to be able to cancel MADs cannot use duplicate wr_ids."

 - R.


From Nitin.Hande at Sun.COM  Wed Feb 23 14:19:50 2005
From: Nitin.Hande at Sun.COM (Nitin Hande)
Date: Wed, 23 Feb 2005 14:19:50 -0800
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1109153997.13555.179.camel@localhost.localdomain>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
	<1108589174.4645.138.camel@localhost.localdomain>
	<1108593217.16278.47.camel@sr1-umpk-01>
	<1108674754.4761.634.camel@localhost.localdomain>
	<1109130984.16278.547.camel@sr1-umpk-01>
	<1109153997.13555.179.camel@localhost.localdomain>
Message-ID: <1109197189.16278.756.camel@sr1-umpk-01>

Hal, 

[comments below]
On Wed, 2005-02-23 at 02:19, Hal Rosenstock wrote:
> On Tue, 2005-02-22 at 22:56, Nitin Hande wrote:
> > So I tried the latest patches and preliminarily things seem to be
> > working fine. 
> 
> Yipee.
[snip..]
> 
> > 
> > So after this test above, I try to run snoop on the solaris interface
> > and get the following error message from the layer below IPoIB:
> > 
> > Feb 22 19:50:25 dongon.SFBay.Sun.COM ibd: [ID 517869 kern.info] NOTICE:
> > ibd0: HCA GUID 0002c901097651d0 port 1 PKEY ffff Could not get list of
> > IBA multicast groups
> > 
> > My preliminary assumption is that OpenSm is not returning the list of
> > multicast groups that the ibd interface has joined. I will look at the
> > MAD's tomorrow and try to ascertain that.
> 
> How does S10 request this ? Remember that if it is a GetTable and
> doesn't fit in a single MAD, it will be broken now. If that is the case,
> we will live with this until we have real RMPP.
Below is an an example of a single GetTable request and response between
Solaris and OpenSM. OpenSM is not reporting the MCgroups in case of a
single request/response.  I have also provided a MAD output between
Solaris IPoIB driver and IBSRM single GetTable request response below
this example.

Here is the MAD trace between solaris and OpenSM:
Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x12 - SubnAdmGetTable()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d1000000ec
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 ec  .........vQ.....
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00  ................
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 0b 1b 00 00 84 00  ................
60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x92 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x97651d1000000ec
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 ec  .........vQ.....
10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
20: 00 00 00 14 00 00 00 00 00 00 00 00 00 07 00 00  ................
30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00  ................
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Here is the transaction between IBSRM and Solaris IPoIB driver. 

Outgoing MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x12 - SubnAdmGetTable()
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x8fecc610000009a
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 12 00 00 00 00 08 fe cc 61 00 00 00 9a  ...........a....
10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00  ................
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 81 23 45 68 00 00 84 00  .........#Eh....
60: 80 01 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Incoming MAD:
        BaseVersion: 0x1
        MgmtClass: 0x3 - SubnAdm
        ClassVersion: 0x2
        R_Method: 0x92 -
        Status: 0x0 - NO_ERROR
        ClassSpecific: 0x0
        TransactionID: 0x8fecc610000009a
        AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
 0: 01 03 02 92 00 00 00 00 08 fe cc 61 00 00 00 9a  ...........a....
10: 00 38 00 00 00 00 00 00 01 01 73 00 00 00 00 01  .8........s.....
20: 00 00 01 40 00 00 00 00 00 00 00 00 00 07 00 00  ... at ............
30: 00 00 00 00 00 00 00 00 ff 12 40 1b 80 01 00 00  .......... at .....
40: 00 00 00 00 00 00 00 09 00 00 00 00 00 00 00 00  ................
50: 00 00 00 00 00 00 00 00 81 23 45 68 c0 04 84 00  .........#Eh....
60: 80 01 83 8d 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
70: ff 12 40 1b 80 01 00 00 00 00 00 00 00 00 00 01  .. at .............
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
90: 81 23 45 68 c0 03 84 00 80 01 83 8d 00 00 00 00  .#Eh............
a0: 20 00 00 00 00 00 00 00 ff 12 40 1b 80 01 00 00   ......... at .....
b0: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
c0: 00 00 00 00 00 00 00 00 81 23 45 68 c0 00 84 00  .........#Eh....
d0: 80 01 83 8d 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
e0: ff 12 60 1b 80 01 00 00 00 00 00 01 ff 76 5b 01  ..`..........v[.
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Thanks
Nitin

> 
> -- Hal
> 



From halr at voltaire.com  Wed Feb 23 14:17:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 23 Feb 2005 17:17:05 -0500
Subject: [openib-general] Re: ib_cancel_mad and the CM
In-Reply-To: <421CE184.4090803@ichips.intel.com>
References: <421CE184.4090803@ichips.intel.com>
Message-ID: <1109197011.13555.1209.camel@localhost.localdomain>

On Wed, 2005-02-23 at 15:03, Sean Hefty wrote:
> Alternatively, it may be possible to change 
> ib_cancel_mad to cancel MADs based on a second set of criteria.

The second set of crtieria could be an incarnation number associated
with the message, saved and later checked before cancel to make sure it
is the same. However, I think this means an additional field in struct
ib_send_wr although I only quickly glanced at this.

-- Hal





From mshefty at ichips.intel.com  Wed Feb 23 14:28:54 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 14:28:54 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <52d5uraqgi.fsf@topspin.com>
References: <421CE184.4090803@ichips.intel.com> <52d5uraqgi.fsf@topspin.com>
Message-ID: <421D03A6.3020304@ichips.intel.com>

Roland Dreier wrote:

> Using wr_ids to hold pointers often seems to lead to issues like this.
> So it might be best to rework the CM so uses wr_id some other way.
> 
> I'll also remind you of the following Sean Hefty quote, circa
> September 2004 ;)
> 
>     "But, yes, users wanting to be able to cancel MADs cannot use duplicate wr_ids."

There are never duplicated values for wr_ids outstanding at the same 
time, which is what the above quote was referring to.  To me, the 
problem seems more related to the inability to synchronize canceling a 
MAD with processing its completion.  Without that ability, it seems 
that the user needs to have some sort of increasing wr_id value.

I did look at what the sa_query did for canceling queries, but I 
_think_ it may have a similar issue.  It's just harding to hit because 
it checks both a pointer and an index.  But if a problem does exist, 
then a user could cancel a query issued by another user.

- Sean


From libor at topspin.com  Wed Feb 23 14:36:22 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 14:36:22 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <421D03A6.3020304@ichips.intel.com>;
	from mshefty@ichips.intel.com on Wed, Feb 23, 2005 at 02:28:54PM
	-0800
References: <421CE184.4090803@ichips.intel.com> <52d5uraqgi.fsf@topspin.com>
	<421D03A6.3020304@ichips.intel.com>
Message-ID: <20050223143622.R23420@topspin.com>

On Wed, Feb 23, 2005 at 02:28:54PM -0800, Sean Hefty wrote:
> There are never duplicated values for wr_ids outstanding at the same 
> time, which is what the above quote was referring to.  To me, the 
> problem seems more related to the inability to synchronize canceling a 
> MAD with processing its completion.  Without that ability, it seems 
> that the user needs to have some sort of increasing wr_id value.
> 
> I did look at what the sa_query did for canceling queries, but I 
> _think_ it may have a similar issue.  It's just harding to hit because 
> it checks both a pointer and an index.  But if a problem does exist, 
> then a user could cancel a query issued by another user.

  Are you saying that the layer which assigns the wr_id is not the
same layer that actually executes the cancel? That would seem like
a recipe for problems. 

-Libor


From mshefty at ichips.intel.com  Wed Feb 23 14:36:53 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 14:36:53 -0800
Subject: [openib-general] Re: ib_cancel_mad and the CM
In-Reply-To: <1109197011.13555.1209.camel@localhost.localdomain>
References: <421CE184.4090803@ichips.intel.com>
	<1109197011.13555.1209.camel@localhost.localdomain>
Message-ID: <421D0585.3000400@ichips.intel.com>

Hal Rosenstock wrote:

>>Alternatively, it may be possible to change 
>>ib_cancel_mad to cancel MADs based on a second set of criteria.
> 
> The second set of crtieria could be an incarnation number associated
> with the message, saved and later checked before cancel to make sure it
> is the same. However, I think this means an additional field in struct
> ib_send_wr although I only quickly glanced at this.

I think that would be the case.  Of course, the CM could use the wr_id 
differently, with the drawback that it could no longer store a pointer. 
  E.g. It could store the cm_msg in an idr, with part of the wr_id 
being the incarnation number.  What I'm trying to determine is if this 
is an issue related to just the CM implementation, or if this would be 
something that multiple users would run into.

- Sean


From mshefty at ichips.intel.com  Wed Feb 23 14:46:42 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 14:46:42 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <20050223143622.R23420@topspin.com>
References: <421CE184.4090803@ichips.intel.com> <52d5uraqgi.fsf@topspin.com>
	<421D03A6.3020304@ichips.intel.com>
	<20050223143622.R23420@topspin.com>
Message-ID: <421D07D2.3090805@ichips.intel.com>

Libor Michalek wrote:

>   Are you saying that the layer which assigns the wr_id is not the
> same layer that actually executes the cancel? That would seem like
> a recipe for problems. 

No.  For both the CM and SA query code, the layer that assigns the 
wr_id executes the cancel.  For the CM, it's the fact that the 
assignment goes to a pointer, which would allow two MADs sent one after 
the other's completion to use the same wr_id.

- Sean


From libor at topspin.com  Wed Feb 23 14:55:50 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 14:55:50 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <421D07D2.3090805@ichips.intel.com>;
	from mshefty@ichips.intel.com on Wed, Feb 23, 2005 at 02:46:42PM
	-0800
References: <421CE184.4090803@ichips.intel.com> <52d5uraqgi.fsf@topspin.com>
	<421D03A6.3020304@ichips.intel.com>
	<20050223143622.R23420@topspin.com>
	<421D07D2.3090805@ichips.intel.com>
Message-ID: <20050223145550.S23420@topspin.com>

On Wed, Feb 23, 2005 at 02:46:42PM -0800, Sean Hefty wrote:
> Libor Michalek wrote:
> 
> >   Are you saying that the layer which assigns the wr_id is not the
> > same layer that actually executes the cancel? That would seem like
> > a recipe for problems. 
> 
> No.  For both the CM and SA query code, the layer that assigns the 
> wr_id executes the cancel.  For the CM, it's the fact that the 
> assignment goes to a pointer, which would allow two MADs sent one after 
> the other's completion to use the same wr_id.

  No, what I'm saying is that having the CM assigns the wr_id, and then
having the mad code actually executes the cancel, is a problem. There is
nothing to prevent multiple mad consumers from using the same wr_id, in
fact I would assume it. If you want to use wr_id as a handle that the
mad layer can use to lookup a message, then the mad layer needs to be
where the wr_id is assigned/generated. Either that or you need some other
identifier which you can use to cancel messages.

  In my opinion using pointers in the wr_id is bad for other reasons,
this is a problem with the scope of a certain value. The assignment and
cancel execution have to occur in the same scope.

-Libor


From mshefty at ichips.intel.com  Wed Feb 23 15:07:27 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 15:07:27 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <20050223145550.S23420@topspin.com>
References: <421CE184.4090803@ichips.intel.com> <52d5uraqgi.fsf@topspin.com>
	<421D03A6.3020304@ichips.intel.com>
	<20050223143622.R23420@topspin.com>
	<421D07D2.3090805@ichips.intel.com>
	<20050223145550.S23420@topspin.com>
Message-ID: <421D0CAF.9040701@ichips.intel.com>

Libor Michalek wrote:

  >   No, what I'm saying is that having the CM assigns the wr_id, and then
> having the mad code actually executes the cancel, is a problem. There is
> nothing to prevent multiple mad consumers from using the same wr_id, in
> fact I would assume it. If you want to use wr_id as a handle that the
> mad layer can use to lookup a message, then the mad layer needs to be
> where the wr_id is assigned/generated. Either that or you need some other
> identifier which you can use to cancel messages.

The scope of the MAD is limited by the mad_agent, which is defined per 
consumer of the MAD layer, so the re-use of wr_id's between consumers 
isn't an issue.

>   In my opinion using pointers in the wr_id is bad for other reasons,
> this is a problem with the scope of a certain value. The assignment and
> cancel execution have to occur in the same scope.

Whether the MAD context is a pointer or an index into a table, the 
current problem is still the same.

- Sean


From roland at topspin.com  Wed Feb 23 15:12:47 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 15:12:47 -0800
Subject: [openib-general] [PATCH]
Message-ID: <52wtsyan0w.fsf@topspin.com>

Mem-free mode requires the driver to allocate additonal doorbell pages
for each user access region.  Add support for this in mthca_memfree.c,
and have the driver allocate a table in db_tab for kernel use.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_dev.h	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -268,9 +268,10 @@ struct mthca_dev {
 	struct mthca_av_table  av_table;
 	struct mthca_mcg_table mcg_table;
 
-	struct mthca_uar      driver_uar;
-	struct mthca_pd       driver_pd;
-	struct mthca_mr       driver_mr;
+	struct mthca_uar       driver_uar;
+	struct mthca_db_table *db_tab;
+	struct mthca_pd        driver_pd;
+	struct mthca_mr        driver_mr;
 
 	struct ib_mad_agent  *send_agent[MTHCA_MAX_PORTS][2];
 	struct ib_ah         *sm_ah[MTHCA_MAX_PORTS];
--- linux-kernel/infiniband/hw/mthca/mthca_memfree.h	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_memfree.h	(working copy)
@@ -125,4 +125,37 @@ static inline unsigned long mthca_icm_si
 	return sg_dma_len(&iter->chunk->mem[iter->page_idx]);
 }
 
+enum {
+	MTHCA_DB_REC_PER_PAGE = 4096 / 8
+};
+
+struct mthca_db_page {
+	DECLARE_BITMAP(used, MTHCA_DB_REC_PER_PAGE);
+	u64       *db_rec;
+	dma_addr_t mapping;
+};
+
+struct mthca_db_table {
+	int 	       	      npages;
+	int 	       	      max_group1;
+	int 	       	      min_group2;
+	struct mthca_db_page *page;
+	struct semaphore      mutex;
+};
+
+enum {
+	MTHCA_DB_TYPE_INVALID   = 0x0,
+	MTHCA_DB_TYPE_CQ_SET_CI = 0x1,
+	MTHCA_DB_TYPE_CQ_ARM    = 0x2,
+	MTHCA_DB_TYPE_SQ        = 0x3,
+	MTHCA_DB_TYPE_RQ        = 0x4,
+	MTHCA_DB_TYPE_SRQ       = 0x5,
+	MTHCA_DB_TYPE_GROUP_SEP = 0x7
+};
+
+int mthca_init_db_tab(struct mthca_dev *dev);
+void mthca_cleanup_db_tab(struct mthca_dev *dev);
+int mthca_alloc_db(struct mthca_dev *dev, int type, u32 qn, u32 **db);
+void mthca_free_db(struct mthca_dev *dev, int type, int db_index);
+
 #endif /* MTHCA_MEMFREE_H */
--- linux-kernel/infiniband/hw/mthca/mthca_uar.c	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_uar.c	(working copy)
@@ -33,6 +33,7 @@
  */
 
 #include "mthca_dev.h"
+#include "mthca_memfree.h"
 
 int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar)
 {
@@ -58,12 +59,20 @@ int mthca_init_uar_table(struct mthca_de
 			       dev->limits.num_uars,
 			       dev->limits.num_uars - 1,
 			       dev->limits.reserved_uars);
+	if (ret)
+		return ret;
+
+	ret = mthca_init_db_tab(dev);
+	if (ret)
+		mthca_alloc_cleanup(&dev->uar_table.alloc);
 
 	return ret;
 }
 
 void mthca_cleanup_uar_table(struct mthca_dev *dev)
 {
+	mthca_cleanup_db_tab(dev);
+
 	/* XXX check if any UARs are still allocated? */
 	mthca_alloc_cleanup(&dev->uar_table.alloc);
 }
--- linux-kernel/infiniband/hw/mthca/mthca_profile.c	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_profile.c	(working copy)
@@ -244,9 +244,11 @@ u64 mthca_make_profile(struct mthca_dev 
 			dev->av_table.num_ddr_avs = profile[i].num;
 			break;
 		case MTHCA_RES_UARC:
-			init_hca->uarc_base   = profile[i].start;
-			init_hca->log_uarc_sz = ffs(request->uarc_size) - 13;
-			init_hca->log_uar_sz  = ffs(request->num_uar) - 1;
+			dev->uar_table.uarc_size = request->uarc_size;
+			dev->uar_table.uarc_base = profile[i].start;
+			init_hca->uarc_base   	 = profile[i].start;
+			init_hca->log_uarc_sz 	 = ffs(request->uarc_size) - 13;
+			init_hca->log_uar_sz  	 = ffs(request->num_uar) - 1;
 			break;
 		default:
 			break;
--- linux-kernel/infiniband/hw/mthca/mthca_memfree.c	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_memfree.c	(working copy)
@@ -267,3 +267,199 @@ void mthca_free_icm_table(struct mthca_d
 
 	kfree(table);
 }
+
+static u64 mthca_uarc_virt(struct mthca_dev *dev, int page)
+{
+	return dev->uar_table.uarc_base +
+		dev->driver_uar.index * dev->uar_table.uarc_size +
+		page * 4096;
+}
+
+int mthca_alloc_db(struct mthca_dev *dev, int type, u32 qn, u32 **db)
+{
+	int group;
+	int start, end, dir;
+	int i, j;
+	struct mthca_db_page *page;
+	int ret = 0;
+	u8 status;
+
+	down(&dev->db_tab->mutex);
+
+	switch (type) {
+	case MTHCA_DB_TYPE_CQ_ARM:
+	case MTHCA_DB_TYPE_SQ:
+		group = 0;
+		start = 0;
+		end   = dev->db_tab->max_group1;
+		dir   = 1;
+		break;
+
+	case MTHCA_DB_TYPE_CQ_SET_CI:
+	case MTHCA_DB_TYPE_RQ:
+	case MTHCA_DB_TYPE_SRQ:
+		group = 1;
+		start = dev->db_tab->npages - 1;
+		end   = dev->db_tab->min_group2;
+		dir   = -1;
+		break;
+
+	default:
+		return -1;
+	}
+	
+	for (i = start; i != end; i += dir)
+		if (dev->db_tab->page[i].db_rec &&
+		    !bitmap_full(dev->db_tab->page[i].used,
+				 MTHCA_DB_REC_PER_PAGE)) {
+			page = dev->db_tab->page + i;
+			goto found;
+		}
+
+	if (dev->db_tab->max_group1 >= dev->db_tab->min_group2 - 1) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	page = dev->db_tab->page + end;
+	page->db_rec = dma_alloc_coherent(&dev->pdev->dev, 4096,
+					  &page->mapping, GFP_KERNEL);
+	if (!page->db_rec) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	memset(page->db_rec, 0, 4096);
+
+	ret = mthca_MAP_ICM_page(dev, page->mapping, mthca_uarc_virt(dev, i), &status);
+	if (!ret && status)
+		ret = -EINVAL;
+	if (ret) {
+		dma_free_coherent(&dev->pdev->dev, 4096,
+				  page->db_rec, page->mapping);
+		goto out;
+	}
+
+	bitmap_zero(page->used, MTHCA_DB_REC_PER_PAGE);
+	if (group == 0)
+		++dev->db_tab->max_group1;
+	else
+		--dev->db_tab->min_group2;
+
+found:
+	j = find_first_zero_bit(page->used, MTHCA_DB_REC_PER_PAGE);
+	set_bit(j, page->used);
+
+	if (group == 1)
+		j = MTHCA_DB_REC_PER_PAGE - 1 - j;
+
+	ret = i * MTHCA_DB_REC_PER_PAGE + j;
+
+	page->db_rec[j] = cpu_to_be64((qn << 8) | (type << 5));
+	
+	*db = (u32 *) &page->db_rec[j];
+
+out:
+	up(&dev->db_tab->mutex);
+
+	return ret;
+}
+
+void mthca_free_db(struct mthca_dev *dev, int type, int db_index)
+{
+	int i, j;
+	struct mthca_db_page *page;
+	u8 status;
+
+	i = db_index / MTHCA_DB_REC_PER_PAGE;
+	j = db_index % MTHCA_DB_REC_PER_PAGE;
+
+	page = dev->db_tab->page + i;
+
+	down(&dev->db_tab->mutex);
+
+	page->db_rec[j] = 0;
+	if (i >= dev->db_tab->min_group2)
+		j = MTHCA_DB_REC_PER_PAGE - 1 - j;
+	clear_bit(j, page->used);
+
+	if (bitmap_empty(page->used, MTHCA_DB_REC_PER_PAGE) &&
+	    i >= dev->db_tab->max_group1 - 1) {
+		mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, i), 1, &status);
+		
+		dma_free_coherent(&dev->pdev->dev, 4096,
+				  page->db_rec, page->mapping);
+		page->db_rec = NULL;
+
+		if (i == dev->db_tab->max_group1) {
+			--dev->db_tab->max_group1;
+			/* XXX may be able to unmap more pages now */
+		}
+		if (i == dev->db_tab->min_group2)
+			++dev->db_tab->min_group2;
+	}
+
+	up(&dev->db_tab->mutex);
+}
+
+int mthca_init_db_tab(struct mthca_dev *dev)
+{
+	int i;
+
+	if (dev->hca_type != ARBEL_NATIVE)
+		return 0;
+
+	dev->db_tab = kmalloc(sizeof *dev->db_tab, GFP_KERNEL);
+	if (!dev->db_tab)
+		return -ENOMEM;
+
+	init_MUTEX(&dev->db_tab->mutex);
+
+	dev->db_tab->npages     = dev->uar_table.uarc_size / PAGE_SIZE;
+	dev->db_tab->max_group1 = 0;
+	dev->db_tab->min_group2 = dev->db_tab->npages - 1;
+
+	dev->db_tab->page = kmalloc(dev->db_tab->npages *
+				    sizeof *dev->db_tab->page,
+				    GFP_KERNEL);
+	if (!dev->db_tab->page) {
+		kfree(dev->db_tab);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < dev->db_tab->npages; ++i)
+		dev->db_tab->page[i].db_rec = NULL;
+
+	return 0;
+}
+
+void mthca_cleanup_db_tab(struct mthca_dev *dev)
+{
+	int i;
+	u8 status;
+
+	if (dev->hca_type != ARBEL_NATIVE)
+		return;
+
+	/*
+	 * Because we don't always free our UARC pages when they
+	 * become empty to make mthca_free_db() simpler we need to
+	 * make a sweep through the doorbell pages and free any
+	 * leftover pages now.
+	 */
+	for (i = 0; i < dev->db_tab->npages; ++i) {
+		if (!dev->db_tab->page[i].db_rec)
+			continue;
+
+		if (!bitmap_empty(dev->db_tab->page[i].used, MTHCA_DB_REC_PER_PAGE))
+			mthca_warn(dev, "Kernel UARC page %d not empty\n", i);
+
+		mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, i), 1, &status);
+		
+		dma_free_coherent(&dev->pdev->dev, 4096,
+				  dev->db_tab->page[i].db_rec,
+				  dev->db_tab->page[i].mapping);
+	}
+
+	kfree(dev->db_tab->page);
+	kfree(dev->db_tab);
+}


From libor at topspin.com  Wed Feb 23 15:52:45 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 23 Feb 2005 15:52:45 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <421D0CAF.9040701@ichips.intel.com>;
	from mshefty@ichips.intel.com on Wed, Feb 23, 2005 at 03:07:27PM
	-0800
References: <421CE184.4090803@ichips.intel.com> <52d5uraqgi.fsf@topspin.com>
	<421D03A6.3020304@ichips.intel.com>
	<20050223143622.R23420@topspin.com>
	<421D07D2.3090805@ichips.intel.com>
	<20050223145550.S23420@topspin.com>
	<421D0CAF.9040701@ichips.intel.com>
Message-ID: <20050223155245.T23420@topspin.com>

On Wed, Feb 23, 2005 at 03:07:27PM -0800, Sean Hefty wrote:
> Libor Michalek wrote:
> 
>   >   No, what I'm saying is that having the CM assigns the wr_id, and then
> > having the mad code actually executes the cancel, is a problem. There is
> > nothing to prevent multiple mad consumers from using the same wr_id, in
> > fact I would assume it. If you want to use wr_id as a handle that the
> > mad layer can use to lookup a message, then the mad layer needs to be
> > where the wr_id is assigned/generated. Either that or you need some other
> > identifier which you can use to cancel messages.
> 
> The scope of the MAD is limited by the mad_agent, which is defined per 
> consumer of the MAD layer, so the re-use of wr_id's between consumers 
> isn't an issue.

  Ahh, got it, that was the source of my confusion. In which case it's
just a race condition between executing the callback and the cancel,
right? In which case the two have to occur under some form of exclusion.
  In the past I've used a spinlock protected variable to mark that the
callback is being executed, in which case the cancel fails with a
specific error code...

-Libor


From mshefty at ichips.intel.com  Wed Feb 23 16:26:37 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 16:26:37 -0800
Subject: [openib-general] Re: ib_cancel_mad and the CM
In-Reply-To: <1109197011.13555.1209.camel@localhost.localdomain>
References: <421CE184.4090803@ichips.intel.com>
	<1109197011.13555.1209.camel@localhost.localdomain>
Message-ID: <421D1F3D.8050500@ichips.intel.com>

Hal Rosenstock wrote:

>>Alternatively, it may be possible to change 
>>ib_cancel_mad to cancel MADs based on a second set of criteria.
> 
> The second set of crtieria could be an incarnation number associated
> with the message, saved and later checked before cancel to make sure it
> is the same. However, I think this means an additional field in struct
> ib_send_wr although I only quickly glanced at this.

We could include the TID as part of the cancel call.  This may not 
solve the problem in general, depending on how a client allocates TIDs, 
but it should work for the CM.

- Sean


From roland at topspin.com  Wed Feb 23 19:51:54 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 19:51:54 -0800
Subject: [openib-general] [PATCH] Factor out completion queue buffer
	alloc/free
Message-ID: <52oeeaaa3p.fsf@topspin.com>

Factor the allocation and freeing of completion queue buffers into
mthca_alloc_cq_buf() and mthca_free_cq_buf().  This makes the code
more readable and will eventually make handling userspace CQs simpler
(the kernel doesn't have to allocate a buffer at all).

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_cq.c	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_cq.c	(working copy)
@@ -557,32 +557,40 @@ void mthca_arm_cq(struct mthca_dev *dev,
 		      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 }
 
-int mthca_init_cq(struct mthca_dev *dev, int nent,
-		  struct mthca_cq *cq)
+static void mthca_free_cq_buf(struct mthca_dev *dev, struct mthca_cq *cq)
 {
-	int size = nent * MTHCA_CQ_ENTRY_SIZE;
-	dma_addr_t t;
-	void *mailbox = NULL;
-	int npages, shift;
-	u64 *dma_list = NULL;
-	struct mthca_cq_context *cq_context;
-	int err = -ENOMEM;
-	u8 status;
 	int i;
+	int size;
 
-	might_sleep();
+	if (cq->is_direct)
+		pci_free_consistent(dev->pdev,
+				    (cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE,
+				    cq->queue.direct.buf,
+				    pci_unmap_addr(&cq->queue.direct,
+						   mapping));
+	else {
+		size = (cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE;
+		for (i = 0; i < (size + PAGE_SIZE - 1) / PAGE_SIZE; ++i)
+			if (cq->queue.page_list[i].buf)
+				pci_free_consistent(dev->pdev, PAGE_SIZE,
+						    cq->queue.page_list[i].buf,
+						    pci_unmap_addr(&cq->queue.page_list[i],
+								   mapping));
 
-	mailbox = kmalloc(sizeof (struct mthca_cq_context) + MTHCA_CMD_MAILBOX_EXTRA,
-			  GFP_KERNEL);
-	if (!mailbox)
-		goto err_out;
+		kfree(cq->queue.page_list);
+	}
+}
 
-	cq_context = MAILBOX_ALIGN(mailbox);
+static int mthca_alloc_cq_buf(struct mthca_dev *dev, int size,
+			      struct mthca_cq *cq)
+{
+	int err = -ENOMEM;
+	int npages, shift;
+	u64 *dma_list = NULL;
+	dma_addr_t t;
+	int i;
 
 	if (size <= MTHCA_MAX_DIRECT_CQ_SIZE) {
-		if (0)
-			mthca_dbg(dev, "Creating direct CQ of size %d\n", size);
-
 		cq->is_direct = 1;
 		npages        = 1;
 		shift         = get_order(size) + PAGE_SHIFT;
@@ -590,7 +598,7 @@ int mthca_init_cq(struct mthca_dev *dev,
 		cq->queue.direct.buf = pci_alloc_consistent(dev->pdev,
 							    size, &t);
 		if (!cq->queue.direct.buf)
-			goto err_out;
+			return -ENOMEM;
 
 		pci_unmap_addr_set(&cq->queue.direct, mapping, t);
 
@@ -603,7 +611,7 @@ int mthca_init_cq(struct mthca_dev *dev,
 
 		dma_list = kmalloc(npages * sizeof *dma_list, GFP_KERNEL);
 		if (!dma_list)
-			goto err_out_free;
+			goto err_free;
 
 		for (i = 0; i < npages; ++i)
 			dma_list[i] = t + i * (1 << shift);
@@ -612,12 +620,9 @@ int mthca_init_cq(struct mthca_dev *dev,
 		npages        = (size + PAGE_SIZE - 1) / PAGE_SIZE;
 		shift         = PAGE_SHIFT;
 
-		if (0)
-			mthca_dbg(dev, "Creating indirect CQ with %d pages\n", npages);
-
 		dma_list = kmalloc(npages * sizeof *dma_list, GFP_KERNEL);
 		if (!dma_list)
-			goto err_out;
+			return -ENOMEM;
 
 		cq->queue.page_list = kmalloc(npages * sizeof *cq->queue.page_list,
 					      GFP_KERNEL);
@@ -631,7 +636,7 @@ int mthca_init_cq(struct mthca_dev *dev,
 			cq->queue.page_list[i].buf =
 				pci_alloc_consistent(dev->pdev, PAGE_SIZE, &t);
 			if (!cq->queue.page_list[i].buf)
-				goto err_out_free;
+				goto err_free;
 
 			dma_list[i] = t;
 			pci_unmap_addr_set(&cq->queue.page_list[i], mapping, t);
@@ -640,13 +645,6 @@ int mthca_init_cq(struct mthca_dev *dev,
 		}
 	}
 
-	for (i = 0; i < nent; ++i)
-		set_cqe_hw(get_cqe(cq, i));
-
-	cq->cqn = mthca_alloc(&dev->cq_table.alloc);
-	if (cq->cqn == -1)
-		goto err_out_free;
-
 	err = mthca_mr_alloc_phys(dev, dev->driver_pd.pd_num,
 				  dma_list, shift, npages,
 				  0, size,
@@ -654,7 +652,52 @@ int mthca_init_cq(struct mthca_dev *dev,
 				  MTHCA_MPT_FLAG_LOCAL_READ,
 				  &cq->mr);
 	if (err)
-		goto err_out_free_cq;
+		goto err_free;
+
+	kfree(dma_list);
+
+	return 0;
+
+err_free:
+	mthca_free_cq_buf(dev, cq);
+
+err_out:
+	kfree(dma_list);
+
+	return err;
+}
+
+int mthca_init_cq(struct mthca_dev *dev, int nent,
+		  struct mthca_cq *cq)
+{
+	int size = nent * MTHCA_CQ_ENTRY_SIZE;
+	void *mailbox = NULL;
+	struct mthca_cq_context *cq_context;
+	int err = -ENOMEM;
+	u8 status;
+	int i;
+
+	might_sleep();
+
+	cq->ibcq.cqe = nent - 1;
+
+	cq->cqn = mthca_alloc(&dev->cq_table.alloc);
+	if (cq->cqn == -1)
+		return -ENOMEM;
+
+	mailbox = kmalloc(sizeof (struct mthca_cq_context) + MTHCA_CMD_MAILBOX_EXTRA,
+			  GFP_KERNEL);
+	if (!mailbox)
+		goto err_out;
+
+	cq_context = MAILBOX_ALIGN(mailbox);
+
+	err = mthca_alloc_cq_buf(dev, size, cq);
+	if (err)
+		goto err_out_mailbox;
+
+	for (i = 0; i < nent; ++i)
+		set_cqe_hw(get_cqe(cq, i));
 
 	spin_lock_init(&cq->lock);
 	atomic_set(&cq->refcount, 1);
@@ -697,37 +740,20 @@ int mthca_init_cq(struct mthca_dev *dev,
 
 	cq->cons_index = 0;
 
-	kfree(dma_list);
 	kfree(mailbox);
 
 	return 0;
 
- err_out_free_mr:
+err_out_free_mr:
 	mthca_free_mr(dev, &cq->mr);
+	mthca_free_cq_buf(dev, cq);
 
- err_out_free_cq:
-	mthca_free(&dev->cq_table.alloc, cq->cqn);
-
- err_out_free:
-	if (cq->is_direct)
-		pci_free_consistent(dev->pdev, size,
-				    cq->queue.direct.buf,
-				    pci_unmap_addr(&cq->queue.direct, mapping));
-	else {
-		for (i = 0; i < npages; ++i)
-			if (cq->queue.page_list[i].buf)
-				pci_free_consistent(dev->pdev, PAGE_SIZE,
-						    cq->queue.page_list[i].buf,
-						    pci_unmap_addr(&cq->queue.page_list[i],
-								   mapping));
-
-		kfree(cq->queue.page_list);
-	}
-
- err_out:
-	kfree(dma_list);
+err_out_mailbox:
 	kfree(mailbox);
 
+err_out:
+	mthca_free(&dev->cq_table.alloc, cq->cqn);
+
 	return err;
 }
 
@@ -778,27 +804,7 @@ void mthca_free_cq(struct mthca_dev *dev
 	wait_event(cq->wait, !atomic_read(&cq->refcount));
 
 	mthca_free_mr(dev, &cq->mr);
-
-	if (cq->is_direct)
-		pci_free_consistent(dev->pdev,
-				    (cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE,
-				    cq->queue.direct.buf,
-				    pci_unmap_addr(&cq->queue.direct,
-						   mapping));
-	else {
-		int i;
-
-		for (i = 0;
-		     i < ((cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE + PAGE_SIZE - 1) /
-			     PAGE_SIZE;
-		     ++i)
-			pci_free_consistent(dev->pdev, PAGE_SIZE,
-					    cq->queue.page_list[i].buf,
-					    pci_unmap_addr(&cq->queue.page_list[i],
-							   mapping));
-
-		kfree(cq->queue.page_list);
-	}
+	mthca_free_cq_buf(dev, cq);
 
 	mthca_free(&dev->cq_table.alloc, cq->cqn);
 	kfree(mailbox);


From roland at topspin.com  Wed Feb 23 20:06:39 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 20:06:39 -0800
Subject: [openib-general] [PATCH] mem-free CQ init
Message-ID: <52k6oya9f4.fsf@topspin.com>

Update CQ initialization and cleanup to handle mem-free mode: we need
to make sure the HCA has memory mapped for the entry in the CQ context
table we will use and also allocate doorbell records.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_provider.h	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_provider.h	(working copy)
@@ -143,6 +143,14 @@ struct mthca_cq {
 	int                    cqn;
 	int                    cons_index;
 	int                    is_direct;
+
+	/* Next fields are Arbel only */
+	int                    set_ci_db_index;
+	u32                   *set_ci_db;
+	int                    arm_db_index;
+	u32                   *arm_db;
+	int                    arm_sn;
+
 	union {
 		struct mthca_buf_list direct;
 		struct mthca_buf_list *page_list;
--- linux-kernel/infiniband/hw/mthca/mthca_cq.c	(revision 1887)
+++ linux-kernel/infiniband/hw/mthca/mthca_cq.c	(working copy)
@@ -39,6 +39,7 @@
 
 #include "mthca_dev.h"
 #include "mthca_cmd.h"
+#include "mthca_memfree.h"
 
 enum {
 	MTHCA_MAX_DIRECT_CQ_SIZE = 4 * PAGE_SIZE
@@ -55,7 +56,7 @@ struct mthca_cq_context {
 	u32 flags;
 	u64 start;
 	u32 logsize_usrpage;
-	u32 error_eqn;
+	u32 error_eqn;		/* Tavor only */
 	u32 comp_eqn;
 	u32 pd;
 	u32 lkey;
@@ -64,7 +65,9 @@ struct mthca_cq_context {
 	u32 consumer_index;
 	u32 producer_index;
 	u32 cqn;
-	u32 reserved[3];
+	u32 ci_db;		/* Arbel only */
+	u32 state_db;		/* Arbel only */
+	u32 reserved;
 } __attribute__((packed));
 
 #define MTHCA_CQ_STATUS_OK          ( 0 << 28)
@@ -685,10 +688,30 @@ int mthca_init_cq(struct mthca_dev *dev,
 	if (cq->cqn == -1)
 		return -ENOMEM;
 
+	if (dev->hca_type == ARBEL_NATIVE) {
+		cq->arm_sn = 1;
+
+		err = mthca_table_get(dev, dev->cq_table.table, cq->cqn);
+		if (err)
+			goto err_out;
+
+		err = -ENOMEM;
+
+		cq->set_ci_db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_CQ_SET_CI,
+						     cq->cqn, &cq->set_ci_db);
+		if (cq->set_ci_db_index < 0)
+			goto err_out_icm;
+
+		cq->arm_db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_CQ_ARM,
+						  cq->cqn, &cq->arm_db);
+		if (cq->arm_db_index < 0)
+			goto err_out_ci;
+	}
+
 	mailbox = kmalloc(sizeof (struct mthca_cq_context) + MTHCA_CMD_MAILBOX_EXTRA,
 			  GFP_KERNEL);
 	if (!mailbox)
-		goto err_out;
+		goto err_out_mailbox;
 
 	cq_context = MAILBOX_ALIGN(mailbox);
 
@@ -716,6 +739,11 @@ int mthca_init_cq(struct mthca_dev *dev,
 	cq_context->lkey            = cpu_to_be32(cq->mr.ibmr.lkey);
 	cq_context->cqn             = cpu_to_be32(cq->cqn);
 
+	if (dev->hca_type == ARBEL_NATIVE) {
+		cq_context->ci_db    = cpu_to_be32(cq->set_ci_db_index);
+		cq_context->state_db = cpu_to_be32(cq->arm_db_index);
+	}
+
 	err = mthca_SW2HW_CQ(dev, cq_context, cq->cqn, &status);
 	if (err) {
 		mthca_warn(dev, "SW2HW_CQ failed (%d)\n", err);
@@ -751,6 +779,14 @@ err_out_free_mr:
 err_out_mailbox:
 	kfree(mailbox);
 
+	mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM, cq->arm_db_index);
+
+err_out_ci:
+	mthca_free_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, cq->set_ci_db_index);
+
+err_out_icm:
+	mthca_table_put(dev, dev->cq_table.table, cq->cqn);
+
 err_out:
 	mthca_free(&dev->cq_table.alloc, cq->cqn);
 
@@ -806,6 +842,12 @@ void mthca_free_cq(struct mthca_dev *dev
 	mthca_free_mr(dev, &cq->mr);
 	mthca_free_cq_buf(dev, cq);
 
+	if (dev->hca_type == ARBEL_NATIVE) {
+		mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM,    cq->arm_db_index);
+		mthca_free_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, cq->set_ci_db_index);
+		mthca_table_put(dev, dev->cq_table.table, cq->cqn);
+	}
+
 	mthca_free(&dev->cq_table.alloc, cq->cqn);
 	kfree(mailbox);
 }


From roland at topspin.com  Wed Feb 23 20:20:03 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 20:20:03 -0800
Subject: [openib-general] [PATCH] mem-free CQ operations
Message-ID: <52fyzma8ss.fsf@topspin.com>

Add support for CQ operations (request notification, update consumer
index) in mem-free mode.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_dev.h	(revision 1884)
+++ linux-kernel/infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -368,8 +368,8 @@ void mthca_unmap_eq_icm(struct mthca_dev
 
 int mthca_poll_cq(struct ib_cq *ibcq, int num_entries,
 		  struct ib_wc *entry);
-void mthca_arm_cq(struct mthca_dev *dev, struct mthca_cq *cq,
-		  int solicited);
+int mthca_tavor_arm_cq(struct ib_cq *cq, enum ib_cq_notify notify);
+int mthca_arbel_arm_cq(struct ib_cq *cq, enum ib_cq_notify notify);
 int mthca_init_cq(struct mthca_dev *dev, int nent,
 		  struct mthca_cq *cq);
 void mthca_free_cq(struct mthca_dev *dev,
@@ -384,7 +384,7 @@ int mthca_post_send(struct ib_qp *ibqp, 
 		    struct ib_send_wr **bad_wr);
 int mthca_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
 		       struct ib_recv_wr **bad_wr);
-int mthca_free_err_wqe(struct mthca_qp *qp, int is_send,
+int mthca_free_err_wqe(struct mthca_dev *dev, struct mthca_qp *qp, int is_send,
 		       int index, int *dbd, u32 *new_wqe);
 int mthca_alloc_qp(struct mthca_dev *dev,
 		   struct mthca_pd *pd,
--- linux-kernel/infiniband/hw/mthca/mthca_provider.c	(revision 1888)
+++ linux-kernel/infiniband/hw/mthca/mthca_provider.c	(working copy)
@@ -421,13 +421,6 @@ static int mthca_destroy_cq(struct ib_cq
 	return 0;
 }
 
-static int mthca_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify notify)
-{
-	mthca_arm_cq(to_mdev(cq->device), to_mcq(cq),
-		     notify == IB_CQ_SOLICITED);
-	return 0;
-}
-
 static inline u32 convert_access(int acc)
 {
 	return (acc & IB_ACCESS_REMOTE_ATOMIC ? MTHCA_MPT_FLAG_ATOMIC       : 0) |
@@ -625,7 +618,6 @@ int mthca_register_device(struct mthca_d
 	dev->ib_dev.create_cq            = mthca_create_cq;
 	dev->ib_dev.destroy_cq           = mthca_destroy_cq;
 	dev->ib_dev.poll_cq              = mthca_poll_cq;
-	dev->ib_dev.req_notify_cq        = mthca_req_notify_cq;
 	dev->ib_dev.get_dma_mr           = mthca_get_dma_mr;
 	dev->ib_dev.reg_phys_mr          = mthca_reg_phys_mr;
 	dev->ib_dev.dereg_mr             = mthca_dereg_mr;
@@ -633,6 +625,11 @@ int mthca_register_device(struct mthca_d
 	dev->ib_dev.detach_mcast         = mthca_multicast_detach;
 	dev->ib_dev.process_mad          = mthca_process_mad;
 
+	if (dev->hca_type == ARBEL_NATIVE)
+		dev->ib_dev.req_notify_cq = mthca_arbel_arm_cq;
+	else
+		dev->ib_dev.req_notify_cq = mthca_tavor_arm_cq;
+
 	init_MUTEX(&dev->cap_mask_mutex);
 
 	ret = ib_register_device(&dev->ib_dev);
--- linux-kernel/infiniband/hw/mthca/mthca_provider.h	(revision 1889)
+++ linux-kernel/infiniband/hw/mthca/mthca_provider.h	(working copy)
@@ -141,7 +141,7 @@ struct mthca_cq {
 	spinlock_t             lock;
 	atomic_t               refcount;
 	int                    cqn;
-	int                    cons_index;
+	u32                    cons_index;
 	int                    is_direct;
 
 	/* Next fields are Arbel only */
--- linux-kernel/infiniband/hw/mthca/mthca_cq.c	(revision 1889)
+++ linux-kernel/infiniband/hw/mthca/mthca_cq.c	(working copy)
@@ -136,11 +136,15 @@ struct mthca_err_cqe {
 #define MTHCA_CQ_ENTRY_OWNER_SW      (0 << 7)
 #define MTHCA_CQ_ENTRY_OWNER_HW      (1 << 7)
 
-#define MTHCA_CQ_DB_INC_CI       (1 << 24)
-#define MTHCA_CQ_DB_REQ_NOT      (2 << 24)
-#define MTHCA_CQ_DB_REQ_NOT_SOL  (3 << 24)
-#define MTHCA_CQ_DB_SET_CI       (4 << 24)
-#define MTHCA_CQ_DB_REQ_NOT_MULT (5 << 24)
+#define MTHCA_TAVOR_CQ_DB_INC_CI       (1 << 24)
+#define MTHCA_TAVOR_CQ_DB_REQ_NOT      (2 << 24)
+#define MTHCA_TAVOR_CQ_DB_REQ_NOT_SOL  (3 << 24)
+#define MTHCA_TAVOR_CQ_DB_SET_CI       (4 << 24)
+#define MTHCA_TAVOR_CQ_DB_REQ_NOT_MULT (5 << 24)
+
+#define MTHCA_ARBEL_CQ_DB_REQ_NOT_SOL  (1 << 24)
+#define MTHCA_ARBEL_CQ_DB_REQ_NOT      (2 << 24)
+#define MTHCA_ARBEL_CQ_DB_REQ_NOT_MULT (3 << 24)
 
 static inline struct mthca_cqe *get_cqe(struct mthca_cq *cq, int entry)
 {
@@ -159,7 +163,7 @@ static inline struct mthca_cqe *cqe_sw(s
 
 static inline struct mthca_cqe *next_cqe_sw(struct mthca_cq *cq)
 {
-	return cqe_sw(cq, cq->cons_index);
+	return cqe_sw(cq, cq->cons_index & cq->ibcq.cqe);
 }
 
 static inline void set_cqe_hw(struct mthca_cqe *cqe)
@@ -167,17 +171,26 @@ static inline void set_cqe_hw(struct mth
 	cqe->owner = MTHCA_CQ_ENTRY_OWNER_HW;
 }
 
-static inline void inc_cons_index(struct mthca_dev *dev, struct mthca_cq *cq,
-				  int nent)
+/*
+ * incr is ignored in native Arbel (mem-free) mode, so cq->cons_index
+ * should be correct before calling update_cons_index().
+ */
+static inline void update_cons_index(struct mthca_dev *dev, struct mthca_cq *cq,
+				     int incr)
 {
 	u32 doorbell[2];
 
-	doorbell[0] = cpu_to_be32(MTHCA_CQ_DB_INC_CI | cq->cqn);
-	doorbell[1] = cpu_to_be32(nent - 1);
+	if (dev->hca_type == ARBEL_NATIVE) {
+		*cq->set_ci_db = cpu_to_be32(cq->cons_index);
+		wmb();
+	} else {
+		doorbell[0] = cpu_to_be32(MTHCA_TAVOR_CQ_DB_INC_CI | cq->cqn);
+		doorbell[1] = cpu_to_be32(incr - 1);
 
-	mthca_write64(doorbell,
-		      dev->kar + MTHCA_CQ_DOORBELL,
-		      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+		mthca_write64(doorbell,
+			      dev->kar + MTHCA_CQ_DOORBELL,
+			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+	}
 }
 
 void mthca_cq_event(struct mthca_dev *dev, u32 cqn)
@@ -191,6 +204,8 @@ void mthca_cq_event(struct mthca_dev *de
 		return;
 	}
 
+	++cq->arm_sn;
+
 	cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
 }
 
@@ -247,8 +262,8 @@ void mthca_cq_clean(struct mthca_dev *de
 
 	if (nfreed) {
 		wmb();
-		inc_cons_index(dev, cq, nfreed);
-		cq->cons_index = (cq->cons_index + nfreed) & cq->ibcq.cqe;
+		cq->cons_index += nfreed;
+		update_cons_index(dev, cq, nfreed);
 	}
 
 	spin_unlock_irq(&cq->lock);
@@ -341,7 +356,7 @@ static int handle_error_cqe(struct mthca
 		break;
 	}
 
-	err = mthca_free_err_wqe(qp, is_send, wqe_index, &dbd, &new_wqe);
+	err = mthca_free_err_wqe(dev, qp, is_send, wqe_index, &dbd, &new_wqe);
 	if (err)
 		return err;
 
@@ -411,7 +426,7 @@ static inline int mthca_poll_one(struct 
 		if (*cur_qp) {
 			if (*freed) {
 				wmb();
-				inc_cons_index(dev, cq, *freed);
+				update_cons_index(dev, cq, *freed);
 				*freed = 0;
 			}
 			spin_unlock(&(*cur_qp)->lock);
@@ -505,7 +520,7 @@ static inline int mthca_poll_one(struct 
 	if (likely(free_cqe)) {
 		set_cqe_hw(cqe);
 		++(*freed);
-		cq->cons_index = (cq->cons_index + 1) & cq->ibcq.cqe;
+		++cq->cons_index;
 	}
 
 	return err;
@@ -533,7 +548,7 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 
 	if (freed) {
 		wmb();
-		inc_cons_index(dev, cq, freed);
+		update_cons_index(dev, cq, freed);
 	}
 
 	if (qp)
@@ -544,20 +559,57 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 	return err == 0 || err == -EAGAIN ? npolled : err;
 }
 
-void mthca_arm_cq(struct mthca_dev *dev, struct mthca_cq *cq,
-		  int solicited)
+int mthca_tavor_arm_cq(struct ib_cq *cq, enum ib_cq_notify notify)
 {
 	u32 doorbell[2];
 
-	doorbell[0] =  cpu_to_be32((solicited ?
-				    MTHCA_CQ_DB_REQ_NOT_SOL :
-				    MTHCA_CQ_DB_REQ_NOT)      |
-				   cq->cqn);
+	doorbell[0] = cpu_to_be32((notify == IB_CQ_SOLICITED ?
+				   MTHCA_TAVOR_CQ_DB_REQ_NOT_SOL :
+				   MTHCA_TAVOR_CQ_DB_REQ_NOT)      |
+				  to_mcq(cq)->cqn);
 	doorbell[1] = 0xffffffff;
 
 	mthca_write64(doorbell,
-		      dev->kar + MTHCA_CQ_DOORBELL,
-		      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+		      to_mdev(cq->device)->kar + MTHCA_CQ_DOORBELL,
+		      MTHCA_GET_DOORBELL_LOCK(&to_mdev(cq->device)->doorbell_lock));
+
+	return 0;
+}
+
+int mthca_arbel_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify notify)
+{
+	struct mthca_cq *cq = to_mcq(ibcq);
+	u32 doorbell[2];
+	u32 sn;
+	u32 ci;
+
+	sn = cq->arm_sn & 3;
+	ci = cpu_to_be32(cq->cons_index);
+
+	doorbell[0] = ci;
+	doorbell[1] = cpu_to_be32((cq->cqn << 8) | (2 << 5) | (sn << 3) |
+				  (notify == IB_CQ_SOLICITED ? 1 : 2));
+
+	mthca_write_db_rec(doorbell, cq->arm_db);
+
+	/*
+	 * Make sure that the doorbell record in host memory is
+	 * written before ringing the doorbell via PCI MMIO.
+	 */
+	wmb();
+
+	doorbell[0] = cpu_to_be32((sn << 28)                       |
+				  (notify == IB_CQ_SOLICITED ?
+				   MTHCA_ARBEL_CQ_DB_REQ_NOT_SOL :
+				   MTHCA_ARBEL_CQ_DB_REQ_NOT)      |
+				  cq->cqn);
+	doorbell[1] = ci;
+
+	mthca_write64(doorbell,
+		      to_mdev(ibcq->device)->kar + MTHCA_CQ_DOORBELL,
+		      MTHCA_GET_DOORBELL_LOCK(&to_mdev(ibcq->device)->doorbell_lock));
+
+	return 0;
 }
 
 static void mthca_free_cq_buf(struct mthca_dev *dev, struct mthca_cq *cq)
--- linux-kernel/infiniband/hw/mthca/mthca_qp.c	(revision 1879)
+++ linux-kernel/infiniband/hw/mthca/mthca_qp.c	(working copy)
@@ -1551,7 +1551,7 @@ out:
 	return err;
 }
 
-int mthca_free_err_wqe(struct mthca_qp *qp, int is_send,
+int mthca_free_err_wqe(struct mthca_dev *dev, struct mthca_qp *qp, int is_send,
 		       int index, int *dbd, u32 *new_wqe)
 {
 	struct mthca_next_seg *next;
@@ -1561,7 +1561,10 @@ int mthca_free_err_wqe(struct mthca_qp *
 	else
 		next = get_recv_wqe(qp, index);
 
-	*dbd = !!(next->ee_nds & cpu_to_be32(MTHCA_NEXT_DBD));
+	if (dev->hca_type == ARBEL_NATIVE)
+		*dbd = 1;
+	else
+		*dbd = !!(next->ee_nds & cpu_to_be32(MTHCA_NEXT_DBD));
 	if (next->ee_nds & cpu_to_be32(0x3f))
 		*new_wqe = (next->nda_op & cpu_to_be32(~0x3f)) |
 			(next->ee_nds & cpu_to_be32(0x3f));


From roland at topspin.com  Wed Feb 23 20:37:44 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 20:37:44 -0800
Subject: [openib-general] [PATCH] mem-free QP init
Message-ID: <52braaa7zb.fsf@topspin.com>

Update QP initialization and cleanup to handle mem-free mode.  In
mem-free mode, work queue sizes have to be rounded up to a power of 2,
we need to allocate doorbells, there must be memory mapped for the
entries in the QP and extended QP context table that we use, and the
entries of the receive queue must be initialized.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_provider.h	(revision 1890)
+++ linux-kernel/infiniband/hw/mthca/mthca_provider.h	(working copy)
@@ -167,6 +167,9 @@ struct mthca_wq {
 	void *last;
 	int   max_gs;
 	int   wqe_shift;
+
+	int   db_index;		/* Arbel only */
+	u32  *db;
 };
 
 struct mthca_qp {
--- linux-kernel/infiniband/hw/mthca/mthca_qp.c	(revision 1890)
+++ linux-kernel/infiniband/hw/mthca/mthca_qp.c	(working copy)
@@ -40,6 +40,7 @@
 
 #include "mthca_dev.h"
 #include "mthca_cmd.h"
+#include "mthca_memfree.h"
 
 enum {
 	MTHCA_MAX_DIRECT_QP_SIZE = 4 * PAGE_SIZE,
@@ -105,8 +106,11 @@ struct mthca_qp_path {
 
 struct mthca_qp_context {
 	u32 flags;
-	u32 sched_queue;
-	u32 mtu_msgmax;
+	u32 tavor_sched_queue;	/* Reserved on Arbel */
+	u8  mtu_msgmax;
+	u8  rq_size_stride;	/* Reserved on Tavor */
+	u8  sq_size_stride;	/* Reserved on Tavor */
+	u8  rlkey_arbel_sched_queue;	/* Reserved on Tavor */
 	u32 usr_page;
 	u32 local_qpn;
 	u32 remote_qpn;
@@ -121,18 +125,22 @@ struct mthca_qp_context {
 	u32 reserved2;
 	u32 next_send_psn;
 	u32 cqn_snd;
-	u32 next_snd_wqe[2];
+	u32 snd_wqe_base_l;	/* Next send WQE on Tavor */
+	u32 snd_db_index;	/* (debugging only entries) */
 	u32 last_acked_psn;
 	u32 ssn;
 	u32 params2;
 	u32 rnr_nextrecvpsn;
 	u32 ra_buff_indx;
 	u32 cqn_rcv;
-	u32 next_rcv_wqe[2];
+	u32 rcv_wqe_base_l;	/* Next recv WQE on Tavor */
+	u32 rcv_db_index;	/* (debugging only entries) */
 	u32 qkey;
 	u32 srqn;
 	u32 rmsn;
-	u32 reserved3[19];
+	u16 rq_wqe_counter;	/* reserved on Tavor */
+	u16 sq_wqe_counter;	/* reserved on Tavor */
+	u32 reserved3[18];
 } __attribute__((packed));
 
 struct mthca_qp_param {
@@ -193,7 +201,7 @@ struct mthca_next_seg {
 	u32 imm;		/* immediate data */
 };
 
-struct mthca_ud_seg {
+struct mthca_tavor_ud_seg {
 	u32 reserved1;
 	u32 lkey;
 	u64 av_addr;
@@ -203,6 +211,13 @@ struct mthca_ud_seg {
 	u32 reserved3[2];
 };
 
+struct mthca_arbel_ud_seg {
+	u32 av[8];
+	u32 dqpn;
+	u32 qkey;
+	u32 reserved[2];
+};
+
 struct mthca_bind_seg {
 	u32 flags;		/* [31] Atomic [30] rem write [29] rem read */
 	u32 reserved;
@@ -617,14 +632,24 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 			break;
 		}
 	}
-	/* leave sched_queue as 0 */
+
+	/* leave tavor_sched_queue as 0 */
+
 	if (qp->transport == MLX || qp->transport == UD)
-		qp_context->mtu_msgmax = cpu_to_be32((IB_MTU_2048 << 29) |
-						     (11 << 24));
+		qp_context->mtu_msgmax = (IB_MTU_2048 << 5) | 11;
 	else if (attr_mask & IB_QP_PATH_MTU) {
-		qp_context->mtu_msgmax = cpu_to_be32((attr->path_mtu << 29) |
-						     (31 << 24));
+		qp_context->mtu_msgmax = (attr->path_mtu << 5) | 31;
+	}
+
+	if (dev->hca_type == ARBEL_NATIVE) {
+		qp_context->rq_size_stride =
+			((ffs(qp->rq.max) - 1) << 3) | (qp->rq.wqe_shift - 4);
+		qp_context->sq_size_stride =
+			((ffs(qp->sq.max) - 1) << 3) | (qp->sq.wqe_shift - 4);
 	}
+
+	/* leave arbel_sched_queue as 0 */
+
 	qp_context->usr_page   = cpu_to_be32(dev->driver_uar.index);
 	qp_context->local_qpn  = cpu_to_be32(qp->qpn);
 	if (attr_mask & IB_QP_DEST_QPN) {
@@ -708,6 +733,11 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 		qp_context->next_send_psn = cpu_to_be32(attr->sq_psn);
 	qp_context->cqn_snd = cpu_to_be32(to_mcq(ibqp->send_cq)->cqn);
 
+	if (dev->hca_type == ARBEL_NATIVE) {
+		qp_context->snd_wqe_base_l = cpu_to_be32(qp->send_wqe_offset);
+		qp_context->snd_db_index   = cpu_to_be32(qp->sq.db_index);
+	}
+
 	if (attr_mask & IB_QP_ACCESS_FLAGS) {
 		/*
 		 * Only enable RDMA/atomics if we have responder
@@ -787,12 +817,16 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 	if (attr_mask & IB_QP_RQ_PSN)
 		qp_context->rnr_nextrecvpsn |= cpu_to_be32(attr->rq_psn);
 
-	qp_context->ra_buff_indx = dev->qp_table.rdb_base +
-		((qp->qpn & (dev->limits.num_qps - 1)) * MTHCA_RDB_ENTRY_SIZE <<
-		 dev->qp_table.rdb_shift);
+	qp_context->ra_buff_indx =
+		cpu_to_be32(dev->qp_table.rdb_base +
+			    ((qp->qpn & (dev->limits.num_qps - 1)) * MTHCA_RDB_ENTRY_SIZE <<
+			     dev->qp_table.rdb_shift));
 
 	qp_context->cqn_rcv = cpu_to_be32(to_mcq(ibqp->recv_cq)->cqn);
 
+	if (dev->hca_type == ARBEL_NATIVE)
+		qp_context->rcv_db_index   = cpu_to_be32(qp->rq.db_index);
+
 	if (attr_mask & IB_QP_QKEY) {
 		qp_context->qkey = cpu_to_be32(attr->qkey);
 		qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_Q_KEY);
@@ -860,12 +894,20 @@ static int mthca_alloc_wqe_buf(struct mt
 
 	size = sizeof (struct mthca_next_seg) +
 		qp->sq.max_gs * sizeof (struct mthca_data_seg);
-	if (qp->transport == MLX)
+	switch (qp->transport) {
+	case MLX:
 		size += 2 * sizeof (struct mthca_data_seg);
-	else if (qp->transport == UD)
-		size += sizeof (struct mthca_ud_seg);
-	else /* bind seg is as big as atomic + raddr segs */
+		break;
+	case UD:
+		if (dev->hca_type == ARBEL_NATIVE)
+			size += sizeof (struct mthca_arbel_ud_seg);
+		else
+			size += sizeof (struct mthca_tavor_ud_seg);
+		break;
+	default:
+		/* bind seg is as big as atomic + raddr segs */
 		size += sizeof (struct mthca_bind_seg);
+	}
 
 	for (qp->sq.wqe_shift = 6; 1 << qp->sq.wqe_shift < size;
 	     qp->sq.wqe_shift++)
@@ -942,7 +984,6 @@ static int mthca_alloc_wqe_buf(struct mt
 
 	err = mthca_mr_alloc_phys(dev, pd->pd_num, dma_list, shift,
 				  npages, 0, size,
-				  MTHCA_MPT_FLAG_LOCAL_WRITE |
 				  MTHCA_MPT_FLAG_LOCAL_READ,
 				  &qp->mr);
 	if (err)
@@ -972,6 +1013,60 @@ static int mthca_alloc_wqe_buf(struct mt
 	return err;
 }
 
+static int mthca_alloc_memfree(struct mthca_dev *dev,
+			       struct mthca_qp *qp)
+{
+	int ret = 0;
+
+	if (dev->hca_type == ARBEL_NATIVE) {
+		ret = mthca_table_get(dev, dev->qp_table.qp_table, qp->qpn);
+		if (ret)
+			return ret;
+
+		ret = mthca_table_get(dev, dev->qp_table.eqp_table, qp->qpn);
+		if (ret)
+			goto err_qpc;
+
+		qp->rq.db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_RQ,
+						 qp->qpn, &qp->rq.db);
+		if (qp->rq.db_index < 0) {
+			ret = -ENOMEM;
+			goto err_eqpc;
+		}
+
+		qp->sq.db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_SQ,
+						 qp->qpn, &qp->sq.db);
+		if (qp->sq.db_index < 0) {
+			ret = -ENOMEM;
+			goto err_rq_db;
+		}
+	}
+
+	return 0;
+
+err_rq_db:
+	mthca_free_db(dev, MTHCA_DB_TYPE_RQ, qp->rq.db_index);
+
+err_eqpc:
+	mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn);
+
+err_qpc:
+	mthca_table_put(dev, dev->qp_table.qp_table, qp->qpn);
+
+	return ret;
+}
+
+static void mthca_free_memfree(struct mthca_dev *dev,
+			       struct mthca_qp *qp)
+{
+	if (dev->hca_type == ARBEL_NATIVE) {
+		mthca_free_db(dev, MTHCA_DB_TYPE_SQ, qp->sq.db_index);
+		mthca_free_db(dev, MTHCA_DB_TYPE_RQ, qp->rq.db_index);
+		mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn);
+		mthca_table_put(dev, dev->qp_table.qp_table, qp->qpn);
+	}
+}
+
 static int mthca_alloc_qp_common(struct mthca_dev *dev,
 				 struct mthca_pd *pd,
 				 struct mthca_cq *send_cq,
@@ -979,7 +1074,9 @@ static int mthca_alloc_qp_common(struct 
 				 enum ib_sig_type send_policy,
 				 struct mthca_qp *qp)
 {
-	int err;
+	struct mthca_next_seg *wqe;
+	int ret;
+	int i;
 
 	spin_lock_init(&qp->lock);
 	atomic_set(&qp->refcount, 1);
@@ -996,8 +1093,51 @@ static int mthca_alloc_qp_common(struct 
 	qp->rq.last      = NULL;
 	qp->sq.last      = NULL;
 
-	err = mthca_alloc_wqe_buf(dev, pd, qp);
-	return err;
+	ret = mthca_alloc_memfree(dev, qp);
+	if (ret)
+		return ret;
+
+	ret = mthca_alloc_wqe_buf(dev, pd, qp);
+	if (ret) {
+		mthca_free_memfree(dev, qp);
+		return ret;
+	}
+
+	if (dev->hca_type == ARBEL_NATIVE) {
+		for (i = 0; i < qp->rq.max; ++i) {
+			wqe = get_recv_wqe(qp, i);
+			wqe->nda_op = cpu_to_be32(((i + 1) & (qp->rq.max - 1)) <<
+						  qp->rq.wqe_shift);
+			wqe->ee_nds = cpu_to_be32(1 << (qp->rq.wqe_shift - 4));
+		}
+
+		for (i = 0; i < qp->sq.max; ++i) {
+			wqe = get_send_wqe(qp, i);
+			wqe->nda_op = cpu_to_be32((((i + 1) & (qp->sq.max - 1)) <<
+						   qp->sq.wqe_shift) +
+						  qp->send_wqe_offset);
+		}
+	}
+
+	return 0;
+}
+
+static void mthca_align_qp_size(struct mthca_dev *dev, struct mthca_qp *qp)
+{
+	int i;
+
+	if (dev->hca_type != ARBEL_NATIVE)
+		return;
+
+	for (i = 0; 1 << i < qp->rq.max; ++i)
+		; /* nothing */
+
+	qp->rq.max = 1 << i;
+
+	for (i = 0; 1 << i < qp->sq.max; ++i)
+		; /* nothing */
+
+	qp->sq.max = 1 << i;
 }
 
 int mthca_alloc_qp(struct mthca_dev *dev,
@@ -1010,6 +1150,8 @@ int mthca_alloc_qp(struct mthca_dev *dev
 {
 	int err;
 
+	mthca_align_qp_size(dev, qp);
+
 	switch (type) {
 	case IB_QPT_RC: qp->transport = RC; break;
 	case IB_QPT_UC: qp->transport = UC; break;
@@ -1048,6 +1190,8 @@ int mthca_alloc_sqp(struct mthca_dev *de
 	int err = 0;
 	u32 mqpn = qpn * 2 + dev->qp_table.sqp_start + port - 1;
 
+	mthca_align_qp_size(dev, &sqp->qp);
+
 	sqp->header_buf_size = sqp->qp.sq.max * MTHCA_UD_HEADER_SIZE;
 	sqp->header_buf = dma_alloc_coherent(&dev->pdev->dev, sqp->header_buf_size,
 					     &sqp->header_dma, GFP_KERNEL);
@@ -1160,14 +1304,15 @@ void mthca_free_qp(struct mthca_dev *dev
 
 	kfree(qp->wrid);
 
+	mthca_free_memfree(dev, qp);
+
 	if (is_sqp(dev, qp)) {
 		atomic_dec(&(to_mpd(qp->ibqp.pd)->sqp_count));
 		dma_free_coherent(&dev->pdev->dev,
 				  to_msqp(qp)->header_buf_size,
 				  to_msqp(qp)->header_buf,
 				  to_msqp(qp)->header_dma);
-	}
-	else
+	} else
 		mthca_free(&dev->qp_table.alloc, qp->qpn);
 }
 
@@ -1350,17 +1495,17 @@ int mthca_post_send(struct ib_qp *ibqp, 
 			break;
 
 		case UD:
-			((struct mthca_ud_seg *) wqe)->lkey =
+			((struct mthca_tavor_ud_seg *) wqe)->lkey =
 				cpu_to_be32(to_mah(wr->wr.ud.ah)->key);
-			((struct mthca_ud_seg *) wqe)->av_addr =
+			((struct mthca_tavor_ud_seg *) wqe)->av_addr =
 				cpu_to_be64(to_mah(wr->wr.ud.ah)->avdma);
-			((struct mthca_ud_seg *) wqe)->dqpn =
+			((struct mthca_tavor_ud_seg *) wqe)->dqpn =
 				cpu_to_be32(wr->wr.ud.remote_qpn);
-			((struct mthca_ud_seg *) wqe)->qkey =
+			((struct mthca_tavor_ud_seg *) wqe)->qkey =
 				cpu_to_be32(wr->wr.ud.remote_qkey);
 
-			wqe += sizeof (struct mthca_ud_seg);
-			size += sizeof (struct mthca_ud_seg) / 16;
+			wqe += sizeof (struct mthca_tavor_ud_seg);
+			size += sizeof (struct mthca_tavor_ud_seg) / 16;
 			break;
 
 		case MLX:


From roland at topspin.com  Wed Feb 23 20:44:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 20:44:53 -0800
Subject: [openib-general] [PATCH] mem-free AV/AH support
Message-ID: <527jkya7ne.fsf@topspin.com>

Update address vector handling to support mem-free mode.  In mem-free
mode, the address vector (in hardware format) is copied by the driver
into each send work queue entry, so our address handle creation can
become pretty trivial: we just kmalloc() a buffer to hold the
formatted address vector.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_provider.h	(revision 1891)
+++ linux-kernel/infiniband/hw/mthca/mthca_provider.h	(working copy)
@@ -82,12 +82,18 @@ struct mthca_eq {
 
 struct mthca_av;
 
+enum mthca_ah_type {
+	MTHCA_AH_ON_HCA,
+	MTHCA_AH_PCI_POOL,
+	MTHCA_AH_KMALLOC
+};
+
 struct mthca_ah {
-	struct ib_ah     ibah;
-	int              on_hca;
-	u32              key;
-	struct mthca_av *av;
-	dma_addr_t       avdma;
+	struct ib_ah       ibah;
+	enum mthca_ah_type type;
+	u32                key;
+	struct mthca_av   *av;
+	dma_addr_t         avdma;
 };
 
 /*
--- linux-kernel/infiniband/hw/mthca/mthca_av.c	(revision 1890)
+++ linux-kernel/infiniband/hw/mthca/mthca_av.c	(working copy)
@@ -60,27 +60,34 @@ int mthca_create_ah(struct mthca_dev *de
 	u32 index = -1;
 	struct mthca_av *av = NULL;
 
-	ah->on_hca = 0;
+	ah->type = MTHCA_AH_PCI_POOL;
 
-	if (!atomic_read(&pd->sqp_count) &&
-	    !(dev->mthca_flags & MTHCA_FLAG_DDR_HIDDEN)) {
+	if (dev->hca_type == ARBEL_NATIVE) {
+		ah->av   = kmalloc(sizeof *ah->av, GFP_KERNEL);
+		if (!ah->av)
+			return -ENOMEM;
+
+		ah->type = MTHCA_AH_KMALLOC;
+		av       = ah->av;
+	} else if (!atomic_read(&pd->sqp_count) &&
+		 !(dev->mthca_flags & MTHCA_FLAG_DDR_HIDDEN)) {
 		index = mthca_alloc(&dev->av_table.alloc);
 
 		/* fall back to allocate in host memory */
 		if (index == -1)
-			goto host_alloc;
+			goto on_hca_fail;
 
 		av = kmalloc(sizeof *av, GFP_KERNEL);
 		if (!av)
-			goto host_alloc;
+			goto on_hca_fail;
 
-		ah->on_hca = 1;
+		ah->type = MTHCA_AH_ON_HCA;
 		ah->avdma  = dev->av_table.ddr_av_base +
 			index * MTHCA_AV_SIZE;
 	}
 
- host_alloc:
-	if (!ah->on_hca) {
+on_hca_fail:
+	if (ah->type == MTHCA_AH_PCI_POOL) {
 		ah->av = pci_pool_alloc(dev->av_table.pool,
 					SLAB_KERNEL, &ah->avdma);
 		if (!ah->av)
@@ -123,7 +130,7 @@ int mthca_create_ah(struct mthca_dev *de
 			       j * 4, be32_to_cpu(((u32 *) av)[j]));
 	}
 
-	if (ah->on_hca) {
+	if (ah->type == MTHCA_AH_ON_HCA) {
 		memcpy_toio(dev->av_table.av_map + index * MTHCA_AV_SIZE,
 			    av, MTHCA_AV_SIZE);
 		kfree(av);
@@ -134,12 +141,21 @@ int mthca_create_ah(struct mthca_dev *de
 
 int mthca_destroy_ah(struct mthca_dev *dev, struct mthca_ah *ah)
 {
-	if (ah->on_hca)
+	switch (ah->type) {
+	case MTHCA_AH_ON_HCA:
 		mthca_free(&dev->av_table.alloc,
  			   (ah->avdma - dev->av_table.ddr_av_base) /
 			   MTHCA_AV_SIZE);
-	else
+		break;
+
+	case MTHCA_AH_PCI_POOL:
 		pci_pool_free(dev->av_table.pool, ah->av, ah->avdma);
+		break;
+
+	case MTHCA_AH_KMALLOC:
+		kfree(ah->av);
+		break;
+	}
 
 	return 0;
 }
@@ -147,7 +163,7 @@ int mthca_destroy_ah(struct mthca_dev *d
 int mthca_read_ah(struct mthca_dev *dev, struct mthca_ah *ah,
 		  struct ib_ud_header *header)
 {
-	if (ah->on_hca)
+	if (ah->type == MTHCA_AH_ON_HCA)
 		return -EINVAL;
 
 	header->lrh.service_level   = be32_to_cpu(ah->av->sl_tclass_flowlabel) >> 28;
@@ -176,6 +192,9 @@ int __devinit mthca_init_av_table(struct
 {
 	int err;
 
+	if (dev->hca_type == ARBEL_NATIVE)
+		return 0;
+
 	err = mthca_alloc_init(&dev->av_table.alloc,
 			       dev->av_table.num_ddr_avs,
 			       dev->av_table.num_ddr_avs - 1,
@@ -212,6 +231,9 @@ int __devinit mthca_init_av_table(struct
 
 void __devexit mthca_cleanup_av_table(struct mthca_dev *dev)
 {
+	if (dev->hca_type == ARBEL_NATIVE)
+		return;
+
 	if (dev->av_table.av_map)
 		iounmap(dev->av_table.av_map);
 	pci_pool_destroy(dev->av_table.pool);


From roland at topspin.com  Wed Feb 23 20:53:45 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 20:53:45 -0800
Subject: [openib-general] [PATCH] memfree work request posting
Message-ID: <523bvma78m.fsf@topspin.com>

Implement posting send and receive work requests for mem-free mode.
Also tidy up a few things in send/receive posting for Tavor mode (fix
smp_wmb()s that should really be just wmb()s, annotate tests in the
fast path with likely()/unlikely()).

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_dev.h	(revision 1890)
+++ linux-kernel/infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -380,10 +380,14 @@ void mthca_cq_clean(struct mthca_dev *de
 void mthca_qp_event(struct mthca_dev *dev, u32 qpn,
 		    enum ib_event_type event_type);
 int mthca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask);
-int mthca_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
-		    struct ib_send_wr **bad_wr);
-int mthca_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
-		       struct ib_recv_wr **bad_wr);
+int mthca_tavor_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
+			  struct ib_send_wr **bad_wr);
+int mthca_tavor_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
+			     struct ib_recv_wr **bad_wr);
+int mthca_arbel_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
+			  struct ib_send_wr **bad_wr);
+int mthca_arbel_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
+			     struct ib_recv_wr **bad_wr);
 int mthca_free_err_wqe(struct mthca_dev *dev, struct mthca_qp *qp, int is_send,
 		       int index, int *dbd, u32 *new_wqe);
 int mthca_alloc_qp(struct mthca_dev *dev,
--- linux-kernel/infiniband/hw/mthca/mthca_provider.c	(revision 1890)
+++ linux-kernel/infiniband/hw/mthca/mthca_provider.c	(working copy)
@@ -613,8 +613,6 @@ int mthca_register_device(struct mthca_d
 	dev->ib_dev.create_qp            = mthca_create_qp;
 	dev->ib_dev.modify_qp            = mthca_modify_qp;
 	dev->ib_dev.destroy_qp           = mthca_destroy_qp;
-	dev->ib_dev.post_send            = mthca_post_send;
-	dev->ib_dev.post_recv            = mthca_post_receive;
 	dev->ib_dev.create_cq            = mthca_create_cq;
 	dev->ib_dev.destroy_cq           = mthca_destroy_cq;
 	dev->ib_dev.poll_cq              = mthca_poll_cq;
@@ -625,10 +623,15 @@ int mthca_register_device(struct mthca_d
 	dev->ib_dev.detach_mcast         = mthca_multicast_detach;
 	dev->ib_dev.process_mad          = mthca_process_mad;
 
-	if (dev->hca_type == ARBEL_NATIVE)
+	if (dev->hca_type == ARBEL_NATIVE) {
 		dev->ib_dev.req_notify_cq = mthca_arbel_arm_cq;
-	else
+		dev->ib_dev.post_send     = mthca_arbel_post_send;
+		dev->ib_dev.post_recv     = mthca_arbel_post_receive;
+	} else {
 		dev->ib_dev.req_notify_cq = mthca_tavor_arm_cq;
+		dev->ib_dev.post_send     = mthca_tavor_post_send;
+		dev->ib_dev.post_recv     = mthca_tavor_post_receive;
+	}
 
 	init_MUTEX(&dev->cap_mask_mutex);
 
--- linux-kernel/infiniband/hw/mthca/mthca_qp.c	(revision 1891)
+++ linux-kernel/infiniband/hw/mthca/mthca_qp.c	(working copy)
@@ -253,6 +253,16 @@ struct mthca_mlx_seg {
 	u16 vcrc;
 };
 
+static const u8 mthca_opcode[] = {
+	[IB_WR_SEND]                 = MTHCA_OPCODE_SEND,
+	[IB_WR_SEND_WITH_IMM]        = MTHCA_OPCODE_SEND_IMM,
+	[IB_WR_RDMA_WRITE]           = MTHCA_OPCODE_RDMA_WRITE,
+	[IB_WR_RDMA_WRITE_WITH_IMM]  = MTHCA_OPCODE_RDMA_WRITE_IMM,
+	[IB_WR_RDMA_READ]            = MTHCA_OPCODE_RDMA_READ,
+	[IB_WR_ATOMIC_CMP_AND_SWP]   = MTHCA_OPCODE_ATOMIC_CS,
+	[IB_WR_ATOMIC_FETCH_AND_ADD] = MTHCA_OPCODE_ATOMIC_FA,
+};
+
 static int is_sqp(struct mthca_dev *dev, struct mthca_qp *qp)
 {
 	return qp->qpn >= dev->qp_table.sqp_start &&
@@ -637,9 +647,8 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 
 	if (qp->transport == MLX || qp->transport == UD)
 		qp_context->mtu_msgmax = (IB_MTU_2048 << 5) | 11;
-	else if (attr_mask & IB_QP_PATH_MTU) {
+	else if (attr_mask & IB_QP_PATH_MTU)
 		qp_context->mtu_msgmax = (attr->path_mtu << 5) | 31;
-	}
 
 	if (dev->hca_type == ARBEL_NATIVE) {
 		qp_context->rq_size_stride =
@@ -1385,8 +1394,8 @@ static int build_mlx_header(struct mthca
 	return 0;
 }
 
-int mthca_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
-		    struct ib_send_wr **bad_wr)
+int mthca_tavor_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
+			  struct ib_send_wr **bad_wr)
 {
 	struct mthca_dev *dev = to_mdev(ibqp->device);
 	struct mthca_qp *qp = to_mqp(ibqp);
@@ -1402,16 +1411,6 @@ int mthca_post_send(struct ib_qp *ibqp, 
 	int ind;
 	u8 op0 = 0;
 
-	static const u8 opcode[] = {
-		[IB_WR_SEND]                 = MTHCA_OPCODE_SEND,
-		[IB_WR_SEND_WITH_IMM]        = MTHCA_OPCODE_SEND_IMM,
-		[IB_WR_RDMA_WRITE]           = MTHCA_OPCODE_RDMA_WRITE,
-		[IB_WR_RDMA_WRITE_WITH_IMM]  = MTHCA_OPCODE_RDMA_WRITE_IMM,
-		[IB_WR_RDMA_READ]            = MTHCA_OPCODE_RDMA_READ,
-		[IB_WR_ATOMIC_CMP_AND_SWP]   = MTHCA_OPCODE_ATOMIC_CS,
-		[IB_WR_ATOMIC_FETCH_AND_ADD] = MTHCA_OPCODE_ATOMIC_FA,
-	};
-
 	spin_lock_irqsave(&qp->lock, flags);
 
 	/* XXX check that state is OK to post send */
@@ -1550,7 +1549,7 @@ int mthca_post_send(struct ib_qp *ibqp, 
 
 		qp->wrid[ind + qp->rq.max] = wr->wr_id;
 
-		if (wr->opcode >= ARRAY_SIZE(opcode)) {
+		if (wr->opcode >= ARRAY_SIZE(mthca_opcode)) {
 			mthca_err(dev, "opcode invalid\n");
 			err = -EINVAL;
 			*bad_wr = wr;
@@ -1561,15 +1560,15 @@ int mthca_post_send(struct ib_qp *ibqp, 
 			((struct mthca_next_seg *) prev_wqe)->nda_op =
 				cpu_to_be32(((ind << qp->sq.wqe_shift) +
 					     qp->send_wqe_offset) |
-					    opcode[wr->opcode]);
-			smp_wmb();
+					    mthca_opcode[wr->opcode]);
+			wmb();
 			((struct mthca_next_seg *) prev_wqe)->ee_nds =
 				cpu_to_be32((size0 ? 0 : MTHCA_NEXT_DBD) | size);
 		}
 
 		if (!size0) {
 			size0 = size;
-			op0   = opcode[wr->opcode];
+			op0   = mthca_opcode[wr->opcode];
 		}
 
 		++ind;
@@ -1578,7 +1577,7 @@ int mthca_post_send(struct ib_qp *ibqp, 
 	}
 
 out:
-	if (nreq) {
+	if (likely(nreq)) {
 		u32 doorbell[2];
 
 		doorbell[0] = cpu_to_be32(((qp->sq.next << qp->sq.wqe_shift) +
@@ -1599,8 +1598,8 @@ out:
 	return err;
 }
 
-int mthca_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
-		       struct ib_recv_wr **bad_wr)
+int mthca_tavor_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
+			     struct ib_recv_wr **bad_wr)
 {
 	struct mthca_dev *dev = to_mdev(ibqp->device);
 	struct mthca_qp *qp = to_mqp(ibqp);
@@ -1621,7 +1620,7 @@ int mthca_post_receive(struct ib_qp *ibq
 	ind = qp->rq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->rq.cur + nreq >= qp->rq.max) {
+		if (unlikely(qp->rq.cur + nreq >= qp->rq.max)) {
 			mthca_err(dev, "RQ %06x full\n", qp->qpn);
 			err = -ENOMEM;
 			*bad_wr = wr;
@@ -1640,7 +1639,7 @@ int mthca_post_receive(struct ib_qp *ibq
 		wqe += sizeof (struct mthca_next_seg);
 		size = sizeof (struct mthca_next_seg) / 16;
 
-		if (wr->num_sge > qp->rq.max_gs) {
+		if (unlikely(wr->num_sge > qp->rq.max_gs)) {
 			err = -EINVAL;
 			*bad_wr = wr;
 			goto out;
@@ -1659,10 +1658,10 @@ int mthca_post_receive(struct ib_qp *ibq
 
 		qp->wrid[ind] = wr->wr_id;
 
-		if (prev_wqe) {
+		if (likely(prev_wqe)) {
 			((struct mthca_next_seg *) prev_wqe)->nda_op =
 				cpu_to_be32((ind << qp->rq.wqe_shift) | 1);
-			smp_wmb();
+			wmb();
 			((struct mthca_next_seg *) prev_wqe)->ee_nds =
 				cpu_to_be32(MTHCA_NEXT_DBD | size);
 		}
@@ -1676,7 +1675,7 @@ int mthca_post_receive(struct ib_qp *ibq
 	}
 
 out:
-	if (nreq) {
+	if (likely(nreq)) {
 		u32 doorbell[2];
 
 		doorbell[0] = cpu_to_be32((qp->rq.next << qp->rq.wqe_shift) | size0);
@@ -1696,6 +1695,247 @@ out:
 	return err;
 }
 
+int mthca_arbel_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
+			  struct ib_send_wr **bad_wr)
+{
+	struct mthca_dev *dev = to_mdev(ibqp->device);
+	struct mthca_qp *qp = to_mqp(ibqp);
+	void *wqe;
+	void *prev_wqe;
+	unsigned long flags;
+	int err = 0;
+	int nreq;
+	int i;
+	int size;
+	int size0 = 0;
+	u32 f0 = 0;
+	int ind;
+	u8 op0 = 0;
+
+	spin_lock_irqsave(&qp->lock, flags);
+
+	/* XXX check that state is OK to post send */
+
+	ind = qp->sq.next & (qp->sq.max - 1);
+
+	for (nreq = 0; wr; ++nreq, wr = wr->next) {
+		if (qp->sq.cur + nreq >= qp->sq.max) {
+			mthca_err(dev, "SQ full (%d posted, %d max, %d nreq)\n",
+				  qp->sq.cur, qp->sq.max, nreq);
+			err = -ENOMEM;
+			*bad_wr = wr;
+			goto out;
+		}
+
+		wqe = get_send_wqe(qp, ind);
+		prev_wqe = qp->sq.last;
+		qp->sq.last = wqe;
+
+		((struct mthca_next_seg *) wqe)->flags =
+			((wr->send_flags & IB_SEND_SIGNALED) ?
+			 cpu_to_be32(MTHCA_NEXT_CQ_UPDATE) : 0) |
+			((wr->send_flags & IB_SEND_SOLICITED) ?
+			 cpu_to_be32(MTHCA_NEXT_SOLICIT) : 0)   |
+			cpu_to_be32(1);
+		if (wr->opcode == IB_WR_SEND_WITH_IMM ||
+		    wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM)
+			((struct mthca_next_seg *) wqe)->flags = wr->imm_data;
+
+		wqe += sizeof (struct mthca_next_seg);
+		size = sizeof (struct mthca_next_seg) / 16;
+
+		switch (qp->transport) {
+		case UD:
+			memcpy(((struct mthca_arbel_ud_seg *) wqe)->av,
+			       to_mah(wr->wr.ud.ah)->av, MTHCA_AV_SIZE);
+			((struct mthca_arbel_ud_seg *) wqe)->dqpn =
+				cpu_to_be32(wr->wr.ud.remote_qpn);
+			((struct mthca_arbel_ud_seg *) wqe)->qkey =
+				cpu_to_be32(wr->wr.ud.remote_qkey);
+
+			wqe += sizeof (struct mthca_arbel_ud_seg);
+			size += sizeof (struct mthca_arbel_ud_seg) / 16;
+			break;
+
+		case MLX:
+			err = build_mlx_header(dev, to_msqp(qp), ind, wr,
+					       wqe - sizeof (struct mthca_next_seg),
+					       wqe);
+			if (err) {
+				*bad_wr = wr;
+				goto out;
+			}
+			wqe += sizeof (struct mthca_data_seg);
+			size += sizeof (struct mthca_data_seg) / 16;
+			break;
+		}
+
+		if (wr->num_sge > qp->sq.max_gs) {
+			mthca_err(dev, "too many gathers\n");
+			err = -EINVAL;
+			*bad_wr = wr;
+			goto out;
+		}
+
+		for (i = 0; i < wr->num_sge; ++i) {
+			((struct mthca_data_seg *) wqe)->byte_count =
+				cpu_to_be32(wr->sg_list[i].length);
+			((struct mthca_data_seg *) wqe)->lkey =
+				cpu_to_be32(wr->sg_list[i].lkey);
+			((struct mthca_data_seg *) wqe)->addr =
+				cpu_to_be64(wr->sg_list[i].addr);
+			wqe += sizeof (struct mthca_data_seg);
+			size += sizeof (struct mthca_data_seg) / 16;
+		}
+
+		/* Add one more inline data segment for ICRC */
+		if (qp->transport == MLX) {
+			((struct mthca_data_seg *) wqe)->byte_count =
+				cpu_to_be32((1 << 31) | 4);
+			((u32 *) wqe)[1] = 0;
+			wqe += sizeof (struct mthca_data_seg);
+			size += sizeof (struct mthca_data_seg) / 16;
+		}
+
+		qp->wrid[ind + qp->rq.max] = wr->wr_id;
+
+		if (wr->opcode >= ARRAY_SIZE(mthca_opcode)) {
+			mthca_err(dev, "opcode invalid\n");
+			err = -EINVAL;
+			*bad_wr = wr;
+			goto out;
+		}
+
+		if (likely(prev_wqe)) {
+			((struct mthca_next_seg *) prev_wqe)->nda_op =
+				cpu_to_be32(((ind << qp->sq.wqe_shift) +
+					     qp->send_wqe_offset) |
+					    mthca_opcode[wr->opcode]);
+			wmb();
+			((struct mthca_next_seg *) prev_wqe)->ee_nds =
+				cpu_to_be32(MTHCA_NEXT_DBD | size);
+		}
+
+		if (!size0) {
+			size0 = size;
+			op0   = mthca_opcode[wr->opcode];
+		}
+
+		++ind;
+		if (unlikely(ind >= qp->sq.max))
+			ind -= qp->sq.max;
+	}
+
+out:
+	if (likely(nreq)) {
+		u32 doorbell[2];
+
+		doorbell[0] = cpu_to_be32((nreq << 24)                  |
+					  ((qp->sq.next & 0xffff) << 8) |
+					  f0 | op0);
+		doorbell[1] = cpu_to_be32((qp->qpn << 8) | size0);
+
+		qp->sq.cur  += nreq;
+		qp->sq.next += nreq;
+
+		/*
+		 * Make sure that descriptors are written before
+		 * doorbell record.
+		 */
+		wmb();
+		*qp->sq.db = cpu_to_be32(qp->sq.next & 0xffff);
+
+		/*
+		 * Make sure doorbell record is written before we
+		 * write MMIO send doorbell.
+		 */
+		wmb();
+		mthca_write64(doorbell,
+			      dev->kar + MTHCA_SEND_DOORBELL,
+			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+	}
+
+	spin_unlock_irqrestore(&qp->lock, flags);
+	return err;
+}
+
+int mthca_arbel_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
+			     struct ib_recv_wr **bad_wr)
+{
+	struct mthca_dev *dev = to_mdev(ibqp->device);
+	struct mthca_qp *qp = to_mqp(ibqp);
+	unsigned long flags;
+	int err = 0;
+	int nreq;
+	int ind;
+	int i;
+	void *wqe;
+
+	spin_lock_irqsave(&qp->lock, flags);
+
+	/* XXX check that state is OK to post receive */
+
+	ind = qp->rq.next & (qp->rq.max - 1);
+
+	for (nreq = 0; wr; ++nreq, wr = wr->next) {
+		if (unlikely(qp->rq.cur + nreq >= qp->rq.max)) {
+			mthca_err(dev, "RQ %06x full\n", qp->qpn);
+			err = -ENOMEM;
+			*bad_wr = wr;
+			goto out;
+		}
+
+		wqe = get_recv_wqe(qp, ind);
+
+		((struct mthca_next_seg *) wqe)->flags = 0;
+
+		wqe += sizeof (struct mthca_next_seg);
+
+		if (unlikely(wr->num_sge > qp->rq.max_gs)) {
+			err = -EINVAL;
+			*bad_wr = wr;
+			goto out;
+		}
+
+		for (i = 0; i < wr->num_sge; ++i) {
+			((struct mthca_data_seg *) wqe)->byte_count =
+				cpu_to_be32(wr->sg_list[i].length);
+			((struct mthca_data_seg *) wqe)->lkey =
+				cpu_to_be32(wr->sg_list[i].lkey);
+			((struct mthca_data_seg *) wqe)->addr =
+				cpu_to_be64(wr->sg_list[i].addr);
+			wqe += sizeof (struct mthca_data_seg);
+		}
+
+		if (i < qp->rq.max_gs) {
+			((struct mthca_data_seg *) wqe)->byte_count = 0;
+			((struct mthca_data_seg *) wqe)->lkey = cpu_to_be32(0x100);
+			((struct mthca_data_seg *) wqe)->addr = 0;
+		}
+
+		qp->wrid[ind] = wr->wr_id;
+
+		++ind;
+		if (unlikely(ind >= qp->rq.max))
+			ind -= qp->rq.max;
+	}
+out:
+	if (likely(nreq)) {
+		qp->rq.cur  += nreq;
+		qp->rq.next += nreq;
+
+		/*
+		 * Make sure that descriptors are written before
+		 * doorbell record.
+		 */
+		wmb();
+		*qp->rq.db = cpu_to_be32(qp->rq.next & 0xffff);
+	}
+
+	spin_unlock_irqrestore(&qp->lock, flags);
+	return err;
+}
+
 int mthca_free_err_wqe(struct mthca_dev *dev, struct mthca_qp *qp, int is_send,
 		       int index, int *dbd, u32 *new_wqe)
 {


From roland at topspin.com  Wed Feb 23 21:02:47 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 21:02:47 -0800
Subject: [openib-general] [PATCH] enable mem-free mode
Message-ID: <52y8de8s94.fsf@topspin.com>

Tie up one last loose end by mapping enough context memory to cover
the whole multicast table during initialization, and then enable
mem-free mode.  mthca now supports enough of mem-free mode so that
IPoIB works.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-kernel/infiniband/hw/mthca/mthca_dev.h	(revision 1893)
+++ linux-kernel/infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -207,8 +207,9 @@ struct mthca_av_table {
 };
 
 struct mthca_mcg_table {
-	struct semaphore   sem;
-	struct mthca_alloc alloc;
+	struct semaphore   	sem;
+	struct mthca_alloc 	alloc;
+	struct mthca_icm_table *table;
 };
 
 struct mthca_dev {
--- linux-kernel/infiniband/hw/mthca/mthca_main.c	(revision 1890)
+++ linux-kernel/infiniband/hw/mthca/mthca_main.c	(working copy)
@@ -412,8 +412,29 @@ static int __devinit mthca_init_icm(stru
 		goto err_unmap_eqp;
 	}
 
+	/*
+	 * It's not strictly required, but for simplicity just map the
+	 * whole multicast group table now.  The table isn't very big
+	 * and it's a lot easier than trying to track ref counts.
+	 */
+	mdev->mcg_table.table = mthca_alloc_icm_table(mdev, init_hca->mc_base,
+						      MTHCA_MGM_ENTRY_SIZE,
+						      mdev->limits.num_mgms +
+						      mdev->limits.num_amgms,
+						      mdev->limits.num_mgms +
+						      mdev->limits.num_amgms,
+						      0);
+	if (!mdev->mcg_table.table) {
+		mthca_err(mdev, "Failed to map MCG context memory, aborting.\n");
+		err = -ENOMEM;
+		goto err_unmap_cq;
+	}
+
 	return 0;
 
+err_unmap_cq:
+	mthca_free_icm_table(mdev, mdev->cq_table.table);
+
 err_unmap_eqp:
 	mthca_free_icm_table(mdev, mdev->qp_table.eqp_table);
 
@@ -587,7 +608,7 @@ static int __devinit mthca_setup_hca(str
 		goto err_uar_free;
 	}
 
-       err = mthca_init_pd_table(dev);
+	err = mthca_init_pd_table(dev);
 	if (err) {
 		mthca_err(dev, "Failed to initialize "
 			  "protection domain table, aborting.\n");
@@ -635,13 +656,6 @@ static int __devinit mthca_setup_hca(str
 
 	mthca_dbg(dev, "NOP command IRQ test passed\n");
 
-	if (dev->hca_type == ARBEL_NATIVE) {
-		mthca_warn(dev, "Sorry, native MT25208 mode support is not complete, "
-			   "aborting.\n");
-		err = -ENODEV;
-		goto err_cmd_poll;
-	}
-
 	err = mthca_init_cq_table(dev);
 	if (err) {
 		mthca_err(dev, "Failed to initialize "
@@ -704,7 +718,7 @@ err_uar_free:
 
 err_uar_table_free:
 	mthca_cleanup_uar_table(dev);
-       return err;
+	return err;
 }
 
 static int __devinit mthca_request_regions(struct pci_dev *pdev,
@@ -814,6 +828,7 @@ static int __devinit mthca_init_one(stru
 				    const struct pci_device_id *id)
 {
 	static int mthca_version_printed = 0;
+	static int mthca_memfree_warned = 0;
 	int ddr_hidden = 0;
 	int err;
 	struct mthca_dev *mdev;
@@ -893,6 +908,10 @@ static int __devinit mthca_init_one(stru
 	mdev->pdev     = pdev;
 	mdev->hca_type = id->driver_data;
 
+	if (mdev->hca_type == ARBEL_NATIVE && !mthca_memfree_warned++)
+		mthca_warn(mdev, "Warning: native MT25208 mode support is incomplete.  "
+			   "Your HCA may not work properly.\n");
+
 	if (ddr_hidden)
 		mdev->mthca_flags |= MTHCA_FLAG_DDR_HIDDEN;
 


From roland at topspin.com  Wed Feb 23 21:08:02 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 21:08:02 -0800
Subject: [openib-general] mem-free support now landed on trunk
Message-ID: <52u0o28s0d.fsf@topspin.com>

I've just committed the last in a series of patches that lands support
for the mem-free mode of Mellanox PCI Express HCAs onto the trunk.  If
you have a chance, I'd appreciate it if you would update to the latest
subversion trunk and try it out.  If you have mem-free HCAs I'd like
to know if it works for you, and if you have other HCAs I'd like to
know if I've introduced any regressions.

This takes the mthca diff between the trunk and the roland-uverbs
branch down from 3700 lines to a much more manageable 600 lines (all
of which is related to userspace support).

Thanks,
  Roland



From roland at topspin.com  Wed Feb 23 21:08:52 2005
From: roland at topspin.com (Roland Dreier)
Date: Wed, 23 Feb 2005 21:08:52 -0800
Subject: [openib-general] Re: [PATCH] no qp lock in poll,
	separate locks for send and receive
In-Reply-To: <20050223130205.GH23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 23 Feb 2005 15:02:05 +0200")
References: <20050203175419.GJ2536@mellanox.co.il>
	<52ekf8sfn3.fsf@topspin.com> <20050222184055.GA25382@mellanox.co.il>
	<52zmxwpg4p.fsf@topspin.com> <20050223130205.GH23837@mellanox.co.il>
Message-ID: <52psyq8ryz.fsf@topspin.com>

This looks really good to me, and is obviously the right way to do
things.  Now that I've landed mem-free support, can you rediff against
the trunk?

Thanks,
  Roland


From eitan at mellanox.co.il  Wed Feb 23 22:19:49 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 24 Feb 2005 08:19:49 +0200
Subject: [openib-general] [Utils] IB Management Simulator
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEBB@mtlex01.yok.mtl.com>

If there are no objections, I will put the code under:

https://openib.org/svn/gen2/trunk/src/userspace/utils/IBDM
and
https://openib.org/svn/gen2/trunk/src/userspace/utils/IBMgtSim


Eitan Zahavi
> 
> How about under gen2/utils ? I presume these are userspace tools so
> perhaps a userspace directory should be put under utils and then these 2
> subdirectories under that. Just one man's opinion...
> 
> -- Hal
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/4f6c4cba/attachment.html>

From sean.hefty at intel.com  Wed Feb 23 23:44:40 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Wed, 23 Feb 2005 23:44:40 -0800
Subject: [openib-general] [Utils] IB Management Simulator
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEBB@mtlex01.yok.mtl.com>
Message-ID: <ORSMSX401cR4ZDgC81T00000074@orsmsx401.amr.corp.intel.com>

If there are no objections, I will put the code under: 

https://openib.org/svn/gen2/trunk/src/userspace/utils/IBDM 
and 
https://openib.org/svn/gen2/trunk/src/userspace/utils/IBMgtSim 

How about svn/gen2/utils instead of in the trunk?

 

- Sean 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050223/118b8bc6/attachment.html>

From mst at mellanox.co.il  Thu Feb 24 00:50:00 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 10:50:00 +0200
Subject: [openib-general] Re: mem-free support now landed on trunk
In-Reply-To: <52u0o28s0d.fsf@topspin.com>
References: <52u0o28s0d.fsf@topspin.com>
Message-ID: <20050224085000.GN23837@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: mem-free support now landed on trunk
> 
> I've just committed the last in a series of patches that lands support
> for the mem-free mode of Mellanox PCI Express HCAs onto the trunk.  If
> you have a chance, I'd appreciate it if you would update to the latest
> subversion trunk and try it out.  If you have mem-free HCAs I'd like
> to know if it works for you, and if you have other HCAs I'd like to
> know if I've introduced any regressions.
> 
> This takes the mthca diff between the trunk and the roland-uverbs
> branch down from 3700 lines to a much more manageable 600 lines (all
> of which is related to userspace support).
> 
> Thanks,
>   Roland
> 

Yay!
Thanks, Roland, this is very good stuff.
Very impressive work.
I will rediff and repost my outstanding patches soon.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb 24 00:50:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 10:50:54 +0200
Subject: [openib-general] [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
Message-ID: <20050224085054.GO23837@mellanox.co.il>

Reposting - rediff against an updated trunk.
This patch does the following:

1. Split the QP spinlock to send and receive lock.
The only place where we have to lock both is upon modify_qp,
and that is not on data path.

2. Avoid taking any QP locks when polling CQ.

This last part is achieved by getting rid of the cur field in
mthca_wq, and calculating the number of outstanding WQEs by comparing
the last_comp and next fields. next is only updated by post,
last_comp is only updated by poll.

In a rare case where an overrun is detected, a CQ is locked
and the overrun condition is re-tested, to avoid any potential
for stale last_comp values.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>


Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 1894)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -166,8 +166,8 @@ struct mthca_cq {
 };
 
 struct mthca_wq {
+	spinlock_t lock;
 	int   max;
-	int   cur;
 	int   next;
 	int   last_comp;
 	void *last;
@@ -180,7 +180,6 @@ struct mthca_wq {
 
 struct mthca_qp {
 	struct ib_qp           ibqp;
-	spinlock_t             lock;
 	atomic_t               refcount;
 	u32                    qpn;
 	int                    is_direct;
Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 1894)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -423,15 +423,6 @@ static inline int mthca_poll_one(struct 
 	is_send  = is_error ? cqe->opcode & 0x01 : cqe->is_send & 0x80;
 
 	if (!*cur_qp || be32_to_cpu(cqe->my_qpn) != (*cur_qp)->qpn) {
-		if (*cur_qp) {
-			if (*freed) {
-				wmb();
-				update_cons_index(dev, cq, *freed);
-				*freed = 0;
-			}
-			spin_unlock(&(*cur_qp)->lock);
-		}
-
 		/*
 		 * We do not have to take the QP table lock here,
 		 * because CQs will be locked while QPs are removed
@@ -446,8 +437,6 @@ static inline int mthca_poll_one(struct 
 			err = -EINVAL;
 			goto out;
 		}
-
-		spin_lock(&(*cur_qp)->lock);
 	}
 
 	entry->qp_num = (*cur_qp)->qpn;
@@ -464,11 +453,6 @@ static inline int mthca_poll_one(struct 
 		entry->wr_id = (*cur_qp)->wrid[wqe_index];
 	}
 
-	if (wq->last_comp < wqe_index)
-		wq->cur -= wqe_index - wq->last_comp;
-	else
-		wq->cur -= wq->max - wq->last_comp + wqe_index;
-
 	wq->last_comp = wqe_index;
 
 	if (0)
@@ -551,9 +535,6 @@ int mthca_poll_cq(struct ib_cq *ibcq, in
 		update_cons_index(dev, cq, freed);
 	}
 
-	if (qp)
-		spin_unlock(&qp->lock);
-
 	spin_unlock_irqrestore(&cq->lock, flags);
 
 	return err == 0 || err == -EAGAIN ? npolled : err;
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 1894)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -577,9 +577,11 @@ int mthca_modify_qp(struct ib_qp *ibqp, 
 		else
 			cur_state = attr->cur_qp_state;
 	} else {
-		spin_lock_irq(&qp->lock);
+		spin_lock_irq(&qp->sq.lock);
+		spin_lock(&qp->rq.lock);
 		cur_state = qp->state;
-		spin_unlock_irq(&qp->lock);
+		spin_unlock(&qp->rq.lock);
+		spin_unlock_irq(&qp->sq.lock);
 	}
 
 	if (attr_mask & IB_QP_STATE) {
@@ -1076,6 +1078,14 @@ static void mthca_free_memfree(struct mt
 	}
 }
 
+static void mthca_wq_init(struct mthca_wq* wq)
+{
+	spin_lock_init(&wq->lock);
+	wq->next      = 0;
+	wq->last_comp = wq->max - 1;
+	wq->last      = NULL;
+}
+
 static int mthca_alloc_qp_common(struct mthca_dev *dev,
 				 struct mthca_pd *pd,
 				 struct mthca_cq *send_cq,
@@ -1087,20 +1097,13 @@ static int mthca_alloc_qp_common(struct 
 	int ret;
 	int i;
 
-	spin_lock_init(&qp->lock);
 	atomic_set(&qp->refcount, 1);
 	qp->state    	 = IB_QPS_RESET;
 	qp->atomic_rd_en = 0;
 	qp->resp_depth   = 0;
 	qp->sq_policy    = send_policy;
-	qp->rq.cur       = 0;
-	qp->sq.cur       = 0;
-	qp->rq.next      = 0;
-	qp->sq.next      = 0;
-	qp->rq.last_comp = qp->rq.max - 1;
-	qp->sq.last_comp = qp->sq.max - 1;
-	qp->rq.last      = NULL;
-	qp->sq.last      = NULL;
+	mthca_wq_init(&qp->sq);
+	mthca_wq_init(&qp->rq);
 
 	ret = mthca_alloc_memfree(dev, qp);
 	if (ret)
@@ -1394,6 +1397,24 @@ static int build_mlx_header(struct mthca
 	return 0;
 }
 
+static inline int mthca_wq_overflow(struct mthca_wq* wq, int nreq,
+	 struct ib_cq* ib_cq)
+{
+	int cur;
+	struct mthca_cq* cq;
+
+	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	if (likely(cur + nreq < wq->max))
+		return 0;
+
+	cq = to_mcq(ib_cq);
+	spin_lock(&cq->lock);
+	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	spin_unlock(&cq->lock);
+
+	return cur + nreq >= wq->max;
+}
+
 int mthca_tavor_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
 			  struct ib_send_wr **bad_wr)
 {
@@ -1411,16 +1432,18 @@ int mthca_tavor_post_send(struct ib_qp *
 	int ind;
 	u8 op0 = 0;
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->sq.lock, flags);
 
 	/* XXX check that state is OK to post send */
 
 	ind = qp->sq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->sq.cur + nreq >= qp->sq.max) {
-			mthca_err(dev, "SQ full (%d posted, %d max, %d nreq)\n",
-				  qp->sq.cur, qp->sq.max, nreq);
+		if (mthca_wq_overflow(&qp->sq, nreq, qp->ibqp.send_cq)) {
+			mthca_err(dev, "SQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->sq.next, qp->sq.last_comp,
+					qp->sq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1591,10 +1614,9 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->sq.cur += nreq;
 	qp->sq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->sq.lock, flags);
 	return err;
 }
 
@@ -1613,15 +1635,18 @@ int mthca_tavor_post_receive(struct ib_q
 	void *wqe;
 	void *prev_wqe;
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->rq.lock, flags);
 
 	/* XXX check that state is OK to post receive */
 
 	ind = qp->rq.next;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (unlikely(qp->rq.cur + nreq >= qp->rq.max)) {
-			mthca_err(dev, "RQ %06x full\n", qp->qpn);
+		if (mthca_wq_overflow(&qp->rq, nreq, qp->ibqp.recv_cq)) {
+			mthca_err(dev, "RQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->rq.next, qp->rq.last_comp,
+					qp->rq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1688,10 +1713,9 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->rq.cur += nreq;
 	qp->rq.next = ind;
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->rq.lock, flags);
 	return err;
 }
 
@@ -1712,16 +1736,18 @@ int mthca_arbel_post_send(struct ib_qp *
 	int ind;
 	u8 op0 = 0;
 
-	spin_lock_irqsave(&qp->lock, flags);
+	spin_lock_irqsave(&qp->sq.lock, flags);
 
 	/* XXX check that state is OK to post send */
 
 	ind = qp->sq.next & (qp->sq.max - 1);
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (qp->sq.cur + nreq >= qp->sq.max) {
-			mthca_err(dev, "SQ full (%d posted, %d max, %d nreq)\n",
-				  qp->sq.cur, qp->sq.max, nreq);
+		if (mthca_wq_overflow(&qp->sq, nreq, qp->ibqp.send_cq)) {
+			mthca_err(dev, "SQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->sq.next, qp->sq.last_comp,
+					qp->sq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1835,7 +1861,6 @@ out:
 					  f0 | op0);
 		doorbell[1] = cpu_to_be32((qp->qpn << 8) | size0);
 
-		qp->sq.cur  += nreq;
 		qp->sq.next += nreq;
 
 		/*
@@ -1855,7 +1880,7 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->sq.lock, flags);
 	return err;
 }
 
@@ -1871,15 +1896,18 @@ int mthca_arbel_post_receive(struct ib_q
 	int i;
 	void *wqe;
 
-	spin_lock_irqsave(&qp->lock, flags);
+ 	spin_lock_irqsave(&qp->rq.lock, flags);
 
 	/* XXX check that state is OK to post receive */
 
 	ind = qp->rq.next & (qp->rq.max - 1);
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (unlikely(qp->rq.cur + nreq >= qp->rq.max)) {
-			mthca_err(dev, "RQ %06x full\n", qp->qpn);
+		if (mthca_wq_overflow(&qp->rq, nreq, qp->ibqp.recv_cq)) {
+			mthca_err(dev, "RQ %06x full (%d next, %d last_polled,"
+					" %d max, %d nreq)\n", qp->qpn,
+					qp->rq.next, qp->rq.last_comp,
+					qp->rq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
 			goto out;
@@ -1921,7 +1949,6 @@ int mthca_arbel_post_receive(struct ib_q
 	}
 out:
 	if (likely(nreq)) {
-		qp->rq.cur  += nreq;
 		qp->rq.next += nreq;
 
 		/*
@@ -1932,7 +1959,7 @@ out:
 		*qp->rq.db = cpu_to_be32(qp->rq.next & 0xffff);
 	}
 
-	spin_unlock_irqrestore(&qp->lock, flags);
+	spin_unlock_irqrestore(&qp->rq.lock, flags);
 	return err;
 }
 

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Thu Feb 24 02:42:18 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 12:42:18 +0200
Subject: [openib-general] [PATCH] (repost) initialize device_cap_flags in
	mthca
Message-ID: <20050224104218.GP23837@mellanox.co.il>

Reposting after memfree updates. I also moved device_cap_flags
from limits to mthca_dev, near mthca_flags.

Fixes the device_cap_flags field which was uninitialized in mthca.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>


Index: hw/mthca/mthca_dev.h
===================================================================
--- hw/mthca/mthca_dev.h	(revision 1894)
+++ hw/mthca/mthca_dev.h	(working copy)
@@ -218,6 +218,7 @@ struct mthca_dev {
 
 	int          	 hca_type;
 	unsigned long	 mthca_flags;
+	unsigned long    device_cap_flags;
 
 	u32              rev_id;
 
Index: hw/mthca/mthca_main.c
===================================================================
--- hw/mthca/mthca_main.c	(revision 1894)
+++ hw/mthca/mthca_main.c	(working copy)
@@ -171,6 +171,33 @@ static int __devinit mthca_dev_lim(struc
 	mdev->limits.reserved_uars      = dev_lim->reserved_uars;
 	mdev->limits.reserved_pds       = dev_lim->reserved_pds;
 
+	/* IB_DEVICE_RESIZE_MAX_WR not supported by driver.
+	   May be doable since hardware supports it for SRQ.
+
+	   IB_DEVICE_N_NOTIFY_CQ is supported by hardware but not by driver.
+
+	   IB_DEVICE_SRQ_RESIZE is supported by hardware but SRQ is not
+	   supported by driver. */
+	mdev->device_cap_flags = IB_DEVICE_CHANGE_PHY_PORT |
+		IB_DEVICE_PORT_ACTIVE_EVENT |
+		IB_DEVICE_SYS_IMAGE_GUID |
+		IB_DEVICE_RC_RNR_NAK_GEN;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_BAD_PKEY_CNTR)
+		mdev->device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_BAD_QKEY_CNTR)
+		mdev->device_cap_flags |= IB_DEVICE_BAD_QKEY_CNTR;
+				
+	if (dev_lim->flags & DEV_LIM_FLAG_RAW_MULTI)
+		mdev->device_cap_flags |= IB_DEVICE_RAW_MULTI;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_AUTO_PATH_MIG)
+		mdev->device_cap_flags |= IB_DEVICE_AUTO_PATH_MIG;
+
+	if (dev_lim->flags & DEV_LIM_FLAG_UD_AV_PORT_ENFORCE)
+		mdev->device_cap_flags |= IB_DEVICE_UD_AV_PORT_ENFORCE;
+
 	if (dev_lim->flags & DEV_LIM_FLAG_SRQ)
 		mdev->mthca_flags |= MTHCA_FLAG_SRQ;
 
Index: hw/mthca/mthca_provider.c
===================================================================
--- hw/mthca/mthca_provider.c	(revision 1894)
+++ hw/mthca/mthca_provider.c	(working copy)
@@ -43,6 +43,8 @@ static int mthca_query_device(struct ib_
 	struct ib_smp *in_mad  = NULL;
 	struct ib_smp *out_mad = NULL;
 	int err = -ENOMEM;
+	struct mthca_dev* mdev = to_mdev(ibdev);
+
 	u8 status;
 
 	in_mad  = kmalloc(sizeof *in_mad, GFP_KERNEL);
@@ -50,7 +52,7 @@ static int mthca_query_device(struct ib_
 	if (!in_mad || !out_mad)
 		goto out;
 
-	props->fw_ver        = to_mdev(ibdev)->fw_ver;
+	props->fw_ver              = mdev->fw_ver;
 
 	memset(in_mad, 0, sizeof *in_mad);
 	in_mad->base_version       = 1;
@@ -59,7 +61,7 @@ static int mthca_query_device(struct ib_
 	in_mad->method         	   = IB_MGMT_METHOD_GET;
 	in_mad->attr_id   	   = IB_SMP_ATTR_NODE_INFO;
 
-	err = mthca_MAD_IFC(to_mdev(ibdev), 1, 1,
+	err = mthca_MAD_IFC(mdev, 1, 1,
 			    1, NULL, NULL, in_mad, out_mad,
 			    &status);
 	if (err)
@@ -69,10 +71,11 @@ static int mthca_query_device(struct ib_
 		goto out;
 	}
 
-	props->vendor_id      = be32_to_cpup((u32 *) (out_mad->data + 36)) &
+	props->device_cap_flags = mdev->device_cap_flags;
+	props->vendor_id        = be32_to_cpup((u32 *) (out_mad->data + 36)) &
 		0xffffff;
-	props->vendor_part_id = be16_to_cpup((u16 *) (out_mad->data + 30));
-	props->hw_ver         = be16_to_cpup((u16 *) (out_mad->data + 32));
+	props->vendor_part_id   = be16_to_cpup((u16 *) (out_mad->data + 30));
+	props->hw_ver           = be16_to_cpup((u16 *) (out_mad->data + 32));
 	memcpy(&props->sys_image_guid, out_mad->data +  4, 8);
 	memcpy(&props->node_guid,      out_mad->data + 12, 8);
 
Index: hw/mthca/mthca_cmd.h
===================================================================
--- hw/mthca/mthca_cmd.h	(revision 1894)
+++ hw/mthca/mthca_cmd.h	(working copy)
@@ -95,7 +95,21 @@ enum {
 };
 
 enum {
-	DEV_LIM_FLAG_SRQ = 1 << 6
+	DEV_LIM_FLAG_RC                 = 1 << 0,
+	DEV_LIM_FLAG_UC                 = 1 << 1,
+	DEV_LIM_FLAG_UD                 = 1 << 2,
+	DEV_LIM_FLAG_RD                 = 1 << 3,
+	DEV_LIM_FLAG_RAW_IPV6           = 1 << 4,
+	DEV_LIM_FLAG_RAW_ETHER          = 1 << 5,
+	DEV_LIM_FLAG_SRQ                = 1 << 6,
+	DEV_LIM_FLAG_BAD_PKEY_CNTR      = 1 << 8,
+	DEV_LIM_FLAG_BAD_QKEY_CNTR      = 1 << 9,
+	DEV_LIM_FLAG_MW                 = 1 << 16,
+	DEV_LIM_FLAG_AUTO_PATH_MIG      = 1 << 17,
+	DEV_LIM_FLAG_ATOMIC             = 1 << 18,
+	DEV_LIM_FLAG_RAW_MULTI          = 1 << 19,
+	DEV_LIM_FLAG_UD_AV_PORT_ENFORCE = 1 << 20,
+	DEV_LIM_FLAG_UD_MULTI           = 1 << 21,
 };
 
 struct mthca_dev_lim {

-- 
MST - Michael S. Tsirkin


From bunk at stusta.de  Thu Feb 24 03:11:10 2005
From: bunk at stusta.de (Adrian Bunk)
Date: Thu, 24 Feb 2005 12:11:10 +0100
Subject: [openib-general] 2.6.11-rc4-mm1: infiniband/core/user_mad.c warning
In-Reply-To: <20050223014233.6710fd73.akpm@osdl.org>
References: <20050223014233.6710fd73.akpm@osdl.org>
Message-ID: <20050224111110.GF8651@stusta.de>

On Wed, Feb 23, 2005 at 01:42:33AM -0800, Andrew Morton wrote:
>...
> Changes since 2.6.11-rc3-mm1:
>...
>  bk-driver-core-infiniband-build-fix.patch
>...

This gives me the following compile error:

<--  snip  -->

...
  CC      drivers/infiniband/core/user_mad.o
drivers/infiniband/core/user_mad.c:587: warning: 'class_device_attr_dev' defined but not used
...

<--  snip  -->


Reverting this patch fixes the warning (but I don't know about the 
underlying issues).

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed



From shaharf at voltaire.com  Thu Feb 24 05:21:56 2005
From: shaharf at voltaire.com (shaharf)
Date: Thu, 24 Feb 2005 15:21:56 +0200
Subject: [openib-general] [Utils] IB Management Simulator
Message-ID: <D4F8F0B3820E754C887699BEF26A89405A9A56@taurus.voltaire.com>

Eventually is should be part of the management tree
(gen2/trunk/src/usermode/management), or even under the osm tree
(gen2/trunk/src/usermode/management/osm) due that fact that it requires
the opensm vendor layer to be able to function.

 

However, I guess that some porting efforts must be done to be able to
use this simulator, so in the meantime, out of the tree locations are
fine by me.

 

Eitan, does Mellanox have any intention to do the porting?

 

Shahar

 

________________________________

From: openib-general-bounces at openib.org
[mailto:openib-general-bounces at openib.org] On Behalf Of Sean Hefty
Sent: Thursday, February 24, 2005 9:45 AM
To: 'Eitan Zahavi'; openib-general at openib.org
Subject: RE: [openib-general] [Utils] IB Management Simulator

 

If there are no objections, I will put the code under: 

https://openib.org/svn/gen2/trunk/src/userspace/utils/IBDM 
and 
https://openib.org/svn/gen2/trunk/src/userspace/utils/IBMgtSim 

How about svn/gen2/utils instead of in the trunk?

 

- Sean 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/20b26f47/attachment.html>

From halr at voltaire.com  Thu Feb 24 05:32:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 24 Feb 2005 08:32:10 -0500
Subject: [openib-general] [Utils] IB Management Simulator
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405A9A56@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A89405A9A56@taurus.voltaire.com>
Message-ID: <1109251930.13555.3087.camel@localhost.localdomain>

On Thu, 2005-02-24 at 08:21, shaharf wrote:
> Eventually is should be part of the management tree
> (gen2/trunk/src/usermode/management), or even under the osm tree
> (gen2/trunk/src/usermode/management/osm) due that fact that it
> requires the opensm vendor layer to be able to function.
> 
>  
> 
> However, I guess that some porting efforts must be done to be able to
> use this simulator, so in the meantime, out of the tree locations are
> fine by me.

I think it makes sense to put it there and have the port done there. 
Once it is up to OpenIB (gen2), it can be made part of some debug build.

-- Hal

> 
>  
> 
> Eitan, does Mellanox have any intention to do the porting?
> 
>  
> 
> Shahar
> 
>  
> 
>                                    
> ______________________________________________________________________
> 
> From:openib-general-bounces at openib.org
> [mailto:openib-general-bounces at openib.org] On Behalf Of Sean Hefty
> Sent: Thursday, February 24, 2005 9:45 AM
> To: 'Eitan Zahavi'; openib-general at openib.org
> Subject: RE: [openib-general] [Utils] IB Management Simulator
> 
> 
>  
> 
> If there are no objections, I will put the code under:
> 
> https://openib.org/svn/gen2/trunk/src/userspace/utils/IBDM
> and
> https://openib.org/svn/gen2/trunk/src/userspace/utils/IBMgtSim
> 
> How about svn/gen2/utils instead of in the trunk?
> 
>  
> 
> - Sean 
> 
> 
> 
> ______________________________________________________________________
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Thu Feb 24 05:42:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 24 Feb 2005 08:42:23 -0500
Subject: [openib-general] Solaris IPoIB MTU with OpenSM
In-Reply-To: <1109197189.16278.756.camel@sr1-umpk-01>
References: <1108494564.25271.29.camel@localhost.localdomain>
	<1108503397.51537.231.camel@sr1-umpk-01>
	<1108563413.4646.171.camel@localhost.localdomain>
	<1108588095.16278.26.camel@sr1-umpk-01>
	<1108589174.4645.138.camel@localhost.localdomain>
	<1108593217.16278.47.camel@sr1-umpk-01>
	<1108674754.4761.634.camel@localhost.localdomain>
	<1109130984.16278.547.camel@sr1-umpk-01>
	<1109153997.13555.179.camel@localhost.localdomain>
	<1109197189.16278.756.camel@sr1-umpk-01>
Message-ID: <1109252542.13555.3119.camel@localhost.localdomain>

Hi Nitin,

On Wed, 2005-02-23 at 17:19, Nitin Hande wrote:
> Hal, 
> 
> [comments below]
> On Wed, 2005-02-23 at 02:19, Hal Rosenstock wrote:
> > On Tue, 2005-02-22 at 22:56, Nitin Hande wrote:
> > > So I tried the latest patches and preliminarily things seem to be
> > > working fine. 
> > 
> > Yipee.
> [snip..]
> > 
> > > 
> > > So after this test above, I try to run snoop on the solaris interface
> > > and get the following error message from the layer below IPoIB:
> > > 
> > > Feb 22 19:50:25 dongon.SFBay.Sun.COM ibd: [ID 517869 kern.info] NOTICE:
> > > ibd0: HCA GUID 0002c901097651d0 port 1 PKEY ffff Could not get list of
> > > IBA multicast groups
> > > 
> > > My preliminary assumption is that OpenSm is not returning the list of
> > > multicast groups that the ibd interface has joined. I will look at the
> > > MAD's tomorrow and try to ascertain that.
> > 
> > How does S10 request this ? Remember that if it is a GetTable and
> > doesn't fit in a single MAD, it will be broken now. If that is the case,
> > we will live with this until we have real RMPP.
> Below is an an example of a single GetTable request and response between
> Solaris and OpenSM. OpenSM is not reporting the MCgroups in case of a
> single request/response.  I have also provided a MAD output between
> Solaris IPoIB driver and IBSRM single GetTable request response below
> this example.
> 
> Here is the MAD trace between solaris and OpenSM:
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x12 - SubnAdmGetTable()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d1000000ec
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>      0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 ec  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00  ................
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 0b 1b 00 00 84 00  ................
> 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x92 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x97651d1000000ec
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>      0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 ec  .........vQ.....
> 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01  .8........w.....
> 20: 00 00 00 14 00 00 00 00 00 00 00 00 00 07 00 00  ................
> 30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00  ................
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

It is likely failing the component checking in
osm_sa_mcmember_record.c::__osm_sa_mcm_by_comp_mask_cb due to an endian
issue. Either you can debug this code or I will early next week.

The component mask in the request is 0x80b4 so the only components
checked are QKey (0xb1b), MTU (exactly 2048 (4)), PKey (0xffff), and
scope (2).

If I don't hear anything by next week, I will work on this then.

Thanks.

-- Hal

> Here is the transaction between IBSRM and Solaris IPoIB driver. 
> 
> Outgoing MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x12 - SubnAdmGetTable()
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x8fecc610000009a
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>      0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 12 00 00 00 00 08 fe cc 61 00 00 00 9a  ...........a....
> 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  .8..............
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00  ................
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 81 23 45 68 00 00 84 00  .........#Eh....
> 60: 80 01 00 00 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Incoming MAD:
>         BaseVersion: 0x1
>         MgmtClass: 0x3 - SubnAdm
>         ClassVersion: 0x2
>         R_Method: 0x92 -
>         Status: 0x0 - NO_ERROR
>         ClassSpecific: 0x0
>         TransactionID: 0x8fecc610000009a
>         AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID
> 
>      0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
>  0: 01 03 02 92 00 00 00 00 08 fe cc 61 00 00 00 9a  ...........a....
> 10: 00 38 00 00 00 00 00 00 01 01 73 00 00 00 00 01  .8........s.....
> 20: 00 00 01 40 00 00 00 00 00 00 00 00 00 07 00 00  ... at ............
> 30: 00 00 00 00 00 00 00 00 ff 12 40 1b 80 01 00 00  .......... at .....
> 40: 00 00 00 00 00 00 00 09 00 00 00 00 00 00 00 00  ................
> 50: 00 00 00 00 00 00 00 00 81 23 45 68 c0 04 84 00  .........#Eh....
> 60: 80 01 83 8d 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> 70: ff 12 40 1b 80 01 00 00 00 00 00 00 00 00 00 01  .. at .............
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 90: 81 23 45 68 c0 03 84 00 80 01 83 8d 00 00 00 00  .#Eh............
> a0: 20 00 00 00 00 00 00 00 ff 12 40 1b 80 01 00 00   ......... at .....
> b0: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
> c0: 00 00 00 00 00 00 00 00 81 23 45 68 c0 00 84 00  .........#Eh....
> d0: 80 01 83 8d 00 00 00 00 20 00 00 00 00 00 00 00  ........ .......
> e0: ff 12 60 1b 80 01 00 00 00 00 00 01 ff 76 5b 01  ..`..........v[.
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> 
> Thanks
> Nitin




From eitan at mellanox.co.il  Thu Feb 24 05:47:59 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 24 Feb 2005 15:47:59 +0200
Subject: [openib-general] [Utils] IB Management Simulator
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEC1@mtlex01.yok.mtl.com>

 
 
Shahar Wrote:
Eventually is should be part of the management tree
(gen2/trunk/src/usermode/management), or even under the osm tree
(gen2/trunk/src/usermode/management/osm) due that fact that it requires the
opensm vendor layer to be able to function.
 
[EZ] The simulator by itself does not require osm_vendor. You can write
clients on top of the messages API and integrate to them simulator. The
simulator is a TCP/IP server. 
 
However, I guess that some porting efforts must be done to be able to use
this simulator, so in the meantime, out of the tree locations are fine by
me.
 
Eitan, does Mellanox have any intention to do the porting?
[EZ] As we will publish the OpenSM code we have for gen1, the "sim" vendor
is already supported by it. To attach OpenSM to the simulator you will only
need to compile OpenSM with the "sim" type of OpenSM vendor. I do not plan
to port the "sim" vendor layer to the auto-tools environment you provided as
I'm not familiar with it. But it should not be very hard to do.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/d89aaf6c/attachment.html>

From halr at voltaire.com  Thu Feb 24 05:55:36 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 24 Feb 2005 08:55:36 -0500
Subject: [openib-general] [Utils] IB Management Simulator
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEC1@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEEC1@mtlex01.yok.mtl.com>
Message-ID: <1109253252.13555.3157.camel@localhost.localdomain>

Hi Eitan,

On Thu, 2005-02-24 at 08:47, Eitan Zahavi wrote:
> Eitan, does Mellanox have any intention to do the porting?
> 
> [EZ] As we will publish the OpenSM code we have for gen1, the "sim"
> vendor is already supported by it. To attach OpenSM to the simulator
> you will only need to compile OpenSM with the "sim" type of OpenSM
> vendor. I do not plan to port the "sim" vendor layer to the auto-tools
> environment you provided as I'm not familiar with it. But it should
> not be very hard to do.

Is there anything other than building with autotools that needs to be
done to use this with gen2 ?

Thanks.

-- Hal



From eitan at mellanox.co.il  Thu Feb 24 06:04:11 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 24 Feb 2005 16:04:11 +0200
Subject: [openib-general] [Utils] IB Management Simulator
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEC2@mtlex01.yok.mtl.com>

Hal:
> Is there anything other than building with autotools that needs to be
> done to use this with gen2 ?
> 
[EZ] Yes the code from the gen1 that is defining the "sim" vendor type
should be added, as well as changes to osm_vendor_select.h and osm_vendor.h
The make files needs to know what libraries are required to link the "sim"
vendor too.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/87aa83e5/attachment.html>

From mst at mellanox.co.il  Thu Feb 24 06:09:17 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 16:09:17 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <52u0o4pfe8.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
Message-ID: <20050224140917.GX23837@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: ANNOUNCE: First usable version of userspace verbs
> 
> I'm happy to announce the initial availability of userspace verbs
> support for brave testers.
> 
> To try this out, check out the roland-uverbs subversion branch:
> 
>     svn co https://openib.org/svn/gen2/branches/roland-uverbs
> 
> and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
> userspace verbs support.
> 
> If you want to use a linux-2.6.10 kernel, you will need to apply the
> new linux-2.6.10-backports.diff patch from the branch (which just
> exports get_sb_pseudo()).  No patches at all are required for an
> up-to-date BK or linux-2.6.11-rc4 tree.
> 
> If you use udev, add the rule
> 
>     KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> 
> to your configuration.  Otherwise, create the required device files:
> 
>     mknod /dev/infiniband/uverbs0 c 231 128
>     mknod /dev/infiniband/uverbs1 c 231 129
> 
> and so on for as many HCAs as you have installed.
> 
> The build the userspace libraries in src/userspace/libibverbs and
> src/userspace/libmthca with the usual
> 
>     ./autogen.sh && ./configure && make && sudo make install
> 
> passing whatever parameters to configure you want; you can use
> --prefix to install to another location.  If you set a non-standard
> prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
> configure for libmthca.
> 
> Once you have the libraries built and installed, load the ib_mthca and
> ib_uverbs modules. By default, libibverbs will search for driver
> libraries in <prefix>/lib/infiniband; if you installed libmthca
> somewhere else, set the OPENIB_DRIVER_PATH environment variable to
> point to the directory with mthca.so.
> 
> To actually try things out, you can use the ibv_pingpong program
> shipped as part of the libibverbs package.  For example, one one
> system start the server side
> 
>     $ ibv_pingpong
> 
> and on another system start the client by passing the address of the
> server (in this example I use IPv6 over IPoIB):
> 
>     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> 
> The pingpong program has a number of options -- run ibv_pingpong -h to
> see a list of the switches you can try.
> 
> The current code is stable for me, but all that means is that my tiny
> selection of tests and test systems has not uncovered any of the bugs
> that are undoubtedly present.  Some of the limitations I know about:
> 
>  - Only RC is implemented.  There are not even any functions to call
>    to create UD address handles yet.
>  - Only Tavor mode is supported -- PCI Express HCAs will not work if
>    they are running mem-free firmware.
>  - On x86, only CPUs with SSE will work now.  I'd be surprised if
>    anyone has x86 system with an HCA that doesn't have SSE.
> 
> Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
> kernels -- I don't expect any portability problems but I haven't even
> built for other architectures.
> 
> In any case, please give this a spin and let me know how it looks to you.
> 
> My short- and medium-term plans are:
> 
>   1. Catch up on reviewing and applying the patche queue I'm sitting on.
>   2. Land the Arbel mem-free mode support from the roland-uverbs branch
>      onto the main trunk (and merge it upstream once 2.6.11 is out and
>      2.6.12 opens).
>   3. Implement UD support for userspace.  I should have this done before
>      the end of next week.
>   4. Implement mem-free support for userspace.
> 
> Thanks,
>   Roland

One problem:
infiniband/driver.h includes infiniband/kern-abi.h, which is not
installed.

I worked around that, for now, by copyng it by hand.

-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Thu Feb 24 06:06:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 24 Feb 2005 09:06:25 -0500
Subject: [openib-general] [Utils] IB Management Simulator
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEC2@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEEC2@mtlex01.yok.mtl.com>
Message-ID: <1109253985.13555.3192.camel@localhost.localdomain>

On Thu, 2005-02-24 at 09:04, Eitan Zahavi wrote:
> Hal:
> > Is there anything other than building with autotools that needs to
> be
> > done to use this with gen2 ?
> > 
> [EZ] Yes the code from the gen1 that is defining the "sim" vendor type
> should be added, as well as changes to osm_vendor_select.h and
> osm_vendor.h

Is that the same or different for gen1 v. gen2 ? 

> The make files needs to know what libraries are required to link the
> "sim" vendor too.

OK. That's just more build stuff to be wrapped into autotools.

-- Hal



From mst at mellanox.co.il  Thu Feb 24 06:10:52 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 16:10:52 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050224140917.GX23837@mellanox.co.il>
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224140917.GX23837@mellanox.co.il>
Message-ID: <20050224141052.GY23837@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
> 
> Quoting r. Roland Dreier <roland at topspin.com>:
> > Subject: ANNOUNCE: First usable version of userspace verbs
> > 
> > I'm happy to announce the initial availability of userspace verbs
> > support for brave testers.
> > 
> > To try this out, check out the roland-uverbs subversion branch:
> > 
> >     svn co https://openib.org/svn/gen2/branches/roland-uverbs
> > 
> > and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
> > userspace verbs support.
> > 
> > If you want to use a linux-2.6.10 kernel, you will need to apply the
> > new linux-2.6.10-backports.diff patch from the branch (which just
> > exports get_sb_pseudo()).  No patches at all are required for an
> > up-to-date BK or linux-2.6.11-rc4 tree.
> > 
> > If you use udev, add the rule
> > 
> >     KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> > 
> > to your configuration.  Otherwise, create the required device files:
> > 
> >     mknod /dev/infiniband/uverbs0 c 231 128
> >     mknod /dev/infiniband/uverbs1 c 231 129
> > 
> > and so on for as many HCAs as you have installed.
> > 
> > The build the userspace libraries in src/userspace/libibverbs and
> > src/userspace/libmthca with the usual
> > 
> >     ./autogen.sh && ./configure && make && sudo make install
> > 
> > passing whatever parameters to configure you want; you can use
> > --prefix to install to another location.  If you set a non-standard
> > prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
> > configure for libmthca.
> > 
> > Once you have the libraries built and installed, load the ib_mthca and
> > ib_uverbs modules. By default, libibverbs will search for driver
> > libraries in <prefix>/lib/infiniband; if you installed libmthca
> > somewhere else, set the OPENIB_DRIVER_PATH environment variable to
> > point to the directory with mthca.so.
> > 
> > To actually try things out, you can use the ibv_pingpong program
> > shipped as part of the libibverbs package.  For example, one one
> > system start the server side
> > 
> >     $ ibv_pingpong
> > 
> > and on another system start the client by passing the address of the
> > server (in this example I use IPv6 over IPoIB):
> > 
> >     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> > 
> > The pingpong program has a number of options -- run ibv_pingpong -h to
> > see a list of the switches you can try.
> > 
> > The current code is stable for me, but all that means is that my tiny
> > selection of tests and test systems has not uncovered any of the bugs
> > that are undoubtedly present.  Some of the limitations I know about:
> > 
> >  - Only RC is implemented.  There are not even any functions to call
> >    to create UD address handles yet.
> >  - Only Tavor mode is supported -- PCI Express HCAs will not work if
> >    they are running mem-free firmware.
> >  - On x86, only CPUs with SSE will work now.  I'd be surprised if
> >    anyone has x86 system with an HCA that doesn't have SSE.
> > 
> > Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
> > kernels -- I don't expect any portability problems but I haven't even
> > built for other architectures.
> > 
> > In any case, please give this a spin and let me know how it looks to you.
> > 
> > My short- and medium-term plans are:
> > 
> >   1. Catch up on reviewing and applying the patche queue I'm sitting on.
> >   2. Land the Arbel mem-free mode support from the roland-uverbs branch
> >      onto the main trunk (and merge it upstream once 2.6.11 is out and
> >      2.6.12 opens).
> >   3. Implement UD support for userspace.  I should have this done before
> >      the end of next week.
> >   4. Implement mem-free support for userspace.
> > 
> > Thanks,
> >   Roland
> 
> One problem:
> infiniband/driver.h includes infiniband/kern-abi.h, which is not
> installed.
> 
> I worked around that, for now, by copyng it by hand.
> 
> -- 
> MST - Michael S. Tsirkin
> 

Neither is infiniband/opcode.h

-- 
MST - Michael S. Tsirkin


From eitan at mellanox.co.il  Thu Feb 24 06:10:41 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 24 Feb 2005 16:10:41 +0200
Subject: [openib-general] [Utils] IB Management Simulator
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEC3@mtlex01.yok.mtl.com>

> -- Hal
> Is that the same or different for gen1 v. gen2 ?
> 
[EZ] It is the same
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/b1976985/attachment.html>

From eitan at mellanox.co.il  Thu Feb 24 06:36:23 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 24 Feb 2005 16:36:23 +0200
Subject: [openib-general] [SM] OpenSM Gen1 Update
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEC5@mtlex01.yok.mtl.com>

After a very long time I did not update the gen1 trunk I have just uploaded 
A set of changes for OpenSM. These include many fixes. I will update the
release notes when I'll have them ready.

See: https://openib.org/svn/gen1/trunk/src/userspace/osm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/7af074b7/attachment.html>

From roland at topspin.com  Thu Feb 24 06:58:21 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 06:58:21 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050224140917.GX23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 24 Feb 2005 16:09:17 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224140917.GX23837@mellanox.co.il>
Message-ID: <52k6oy80oi.fsf@topspin.com>

    Michael> One problem: infiniband/driver.h includes
    Michael> infiniband/kern-abi.h, which is not installed.

Thanks, I've updated Makefile.am to fix this (and opcode.h).  I've
been building without using "make install", so I didn't notice this.

 - R.


From mst at mellanox.co.il  Thu Feb 24 06:58:56 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 16:58:56 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <52u0o4pfe8.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
Message-ID: <20050224145856.GZ23837@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: ANNOUNCE: First usable version of userspace verbs
> 
> I'm happy to announce the initial availability of userspace verbs
> support for brave testers.
> 
> To try this out, check out the roland-uverbs subversion branch:
> 
>     svn co https://openib.org/svn/gen2/branches/roland-uverbs
> 
> and build as usual.  Select CONFIG_INFINIBAND_USER_VERBS to build
> userspace verbs support.
> 
> If you want to use a linux-2.6.10 kernel, you will need to apply the
> new linux-2.6.10-backports.diff patch from the branch (which just
> exports get_sb_pseudo()).  No patches at all are required for an
> up-to-date BK or linux-2.6.11-rc4 tree.
> 
> If you use udev, add the rule
> 
>     KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> 
> to your configuration.  Otherwise, create the required device files:
> 
>     mknod /dev/infiniband/uverbs0 c 231 128
>     mknod /dev/infiniband/uverbs1 c 231 129
> 
> and so on for as many HCAs as you have installed.
> 
> The build the userspace libraries in src/userspace/libibverbs and
> src/userspace/libmthca with the usual
> 
>     ./autogen.sh && ./configure && make && sudo make install
> 
> passing whatever parameters to configure you want; you can use
> --prefix to install to another location.  If you set a non-standard
> prefix, it may be useful to pass a -I<path> in CPPFLAGS to the
> configure for libmthca.
> 
> Once you have the libraries built and installed, load the ib_mthca and
> ib_uverbs modules. By default, libibverbs will search for driver
> libraries in <prefix>/lib/infiniband; if you installed libmthca
> somewhere else, set the OPENIB_DRIVER_PATH environment variable to
> point to the directory with mthca.so.
> 
> To actually try things out, you can use the ibv_pingpong program
> shipped as part of the libibverbs package.  For example, one one
> system start the server side
> 
>     $ ibv_pingpong
> 
> and on another system start the client by passing the address of the
> server (in this example I use IPv6 over IPoIB):
> 
>     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> 
> The pingpong program has a number of options -- run ibv_pingpong -h to
> see a list of the switches you can try.
> 
> The current code is stable for me, but all that means is that my tiny
> selection of tests and test systems has not uncovered any of the bugs
> that are undoubtedly present.  Some of the limitations I know about:
> 
>  - Only RC is implemented.  There are not even any functions to call
>    to create UD address handles yet.
>  - Only Tavor mode is supported -- PCI Express HCAs will not work if
>    they are running mem-free firmware.
>  - On x86, only CPUs with SSE will work now.  I'd be surprised if
>    anyone has x86 system with an HCA that doesn't have SSE.
> 
> Also, I've only tried 32-bit i386 userspace running on i386 and x86_64
> kernels -- I don't expect any portability problems but I haven't even
> built for other architectures.
> 
> In any case, please give this a spin and let me know how it looks to you.

The pingpong test fails for me, on x86_64. (The executable being a 64
bit one). My kernel is 2.6.10.

So far, I traced it down to ibv_cmd_create_qp in src/cmd.c, where this command
fails:

183             if (write(pd->context->cmd_fd, cmd, cmd_size) != cmd_size)

By adding some prints I see

cmd 0xbfffef30 returned 0xffffffff expected 0x48, error=22

so a write failed with errno 22 which is EINVAL.

gdb backtrace:

#0  ibv_cmd_create_qp (pd=Variable "pd" is not available.) at src/cmd.c:190
#1  0x0000002a95cb7add in mthca_create_qp (pd=Variable "pd" is not available.
) at src/verbs.c:225
#2  0x0000002a9566ed7a in ibv_create_qp (pd=Variable "pd" is not available.
) at src/verbs.c:117
#3  0x0000000000401fcf in pp_init_ctx (ib_dev=0x509850, size=4096,
    rx_depth=500, port=1) at examples/pingpong.c:308
#4  0x00000000004027de in main (argc=1, argv=0x7fbffff378)
    at examples/pingpong.c:550


-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Thu Feb 24 06:58:59 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 06:58:59 -0800
Subject: [openib-general] [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
	(Shirley Ma's message of "Thu, 17 Feb 2005 17:57:07 -0700")
References: <OF86EC4122.90C5713E-ON87256FAC.00049BDA-00256FAC.00058281@us.ibm.com>
Message-ID: <52fyzm80ng.fsf@topspin.com>

Hi Shirley,

Can you send "Signed-off-by:" versions of the rest of your IPoIB
patches (I think there are 3 still pending).

Thanks,
  Roland


From halr at voltaire.com  Thu Feb 24 07:00:07 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 24 Feb 2005 10:00:07 -0500
Subject: [openib-general] [SM] OpenSM Gen1 Update
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEC5@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEEC5@mtlex01.yok.mtl.com>
Message-ID: <1109257206.13555.3356.camel@localhost.localdomain>

Hi Eitan,

On Thu, 2005-02-24 at 09:36, Eitan Zahavi wrote:
> After a very long time I did not update the gen1 trunk I have just
> uploaded 
> A set of changes for OpenSM. These include many fixes. I will update
> the release notes when I'll have them ready.
> 
> See: https://openib.org/svn/gen1/trunk/src/userspace/osm

How does this compare to what will be coming out as Mellanox Gold 1.7 ?
Is it the same or more or less recent ?

Thanks.

-- Hal



From mst at mellanox.co.il  Thu Feb 24 07:47:30 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 17:47:30 +0200
Subject: [openib-general] Re: enhanced libsdp
In-Reply-To: <1108101863.28588.17.camel@duffman>
References: <1108101863.28588.17.camel@duffman>
Message-ID: <20050224154730.GC23837@mellanox.co.il>


Quoting r. Tom Duffy <tduffy at sun.com>:
> Subject: enhanced libsdp
> Date: Fri, 11 Feb 2005 08:03:06 +0200
> 
> At the workshop this week, Libor mentioned to me that somebody (can't
> remember who, maybe at Mellanox?) had modified libsdp to create a
> transparent connect() and listen().  It would be *really cool* if we
> could get this into the repository.  A pointer to the work would be ok
> too.
> 
> Thanks,
> 
> -tduffy


OK, I have put it up under userspace/libsdp. Its identical to what
is being shipped with the coming gold CD, with enhancements
from gen1 being:
- transparent TCP/IP / SDP support you mention above
- a new match_both directive for .conf file
- some fixes I dont recall at the moment

I know Libor is thinking about dong something at the kernel level,
but I hope it'll be useful meanwhile.
If someone's unhappy with the directory I choose, simply go ahead and move it
somewhere else, I'm offline for the weekend.

-- 
MST - Michael S. Tsirkin


From eitan at mellanox.co.il  Thu Feb 24 07:49:39 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 24 Feb 2005 17:49:39 +0200
Subject: [openib-general] [SM] OpenSM Gen1 Update
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEC6@mtlex01.yok.mtl.com>

> 
> Hal:
> How does this compare to what will be coming out as Mellanox Gold 1.7 ?
> Is it the same or more or less recent ?
[EZ] Same. Unless we find new stuff in the QA.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/30798d61/attachment.html>

From roland at topspin.com  Thu Feb 24 08:00:47 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 08:00:47 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050224145856.GZ23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 24 Feb 2005 16:58:56 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il>
Message-ID: <52braa7xsg.fsf@topspin.com>

    Michael> The pingpong test fails for me, on x86_64. (The
    Michael> executable being a 64 bit one). My kernel is 2.6.10.

Hmm... I'll take a look today.  I'm setting up 64-bit userspace on my
x86-64 system right now.

 - R.


From parks at lanl.gov  Thu Feb 24 08:49:10 2005
From: parks at lanl.gov (Parks Fields)
Date: Thu, 24 Feb 2005 09:49:10 -0700
Subject: [openib-general] [SM] OpenSM Gen1 Update
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEC5@mtlex01.yok.mtl.c
 om>
References: <506C3D7B14CDD411A52C00025558DED6047EEEC5@mtlex01.yok.mtl.com>
Message-ID: <6.1.2.0.2.20050224094602.01ced5e8@ccn-mail.lanl.gov>

At 07:36 AM 2/24/2005, Eitan Zahavi wrote:

>After a very long time I did not update the gen1 trunk I have just uploaded
>A set of changes for OpenSM. These include many fixes. I will update the 
>release notes when I'll have them ready.
>
>See: 
><https://openib.org/svn/gen1/trunk/src/userspace/osm>https://openib.org/svn/gen1/trunk/src/userspace/osm 
>


HI,

Is there a way to build just the opensm. I pulled down just the osm but 
couldn't figure out how to get it to build.

I only want to test the OSM right now.

I couldn't find an OSM in gen2 trunk so I  am assuming yours is the only one.

  Is it the same as the one in Mellanox's IB gold 1.6.1 ????

thanks

parks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/cec8718b/attachment.html>

From mshefty at ichips.intel.com  Thu Feb 24 09:40:49 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 24 Feb 2005 09:40:49 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <421CE184.4090803@ichips.intel.com>
References: <421CE184.4090803@ichips.intel.com>
Message-ID: <421E11A1.9010402@ichips.intel.com>

Sean Hefty wrote:
> Because ib_cancel_mad might invoke a callback that acquires the cm_id 
> lock, the lock cannot be held when ib_cancel_mad is invoked.
> 
{snip}
> 
> To fix this in the CM, the call to ib_cancel_mad just needs to move 
> inside the cm_id lock.  Alternatively, it may be possible to change 
> ib_cancel_mad to cancel MADs based on a second set of criteria.  Both of 
> these would require changes to the MAD layer.

Studying the problem more, I believe that this problem exists for both 
the CM and SA query code.  Unless there is an objection, I'll submit a 
patch that will invoke a user's send callback after a MAD has been 
canceled from one of the MAD threads, rather than directly from the 
user's thread.  (Similar to how the process local MAD functionality is 
implemented.)

This will allow locking around the cancel routine, which should fix the 
problem for the CM code.  However, I don't think that locking around 
the cancel routine eliminates the issue from the SA query code, but I 
also don't see a simple fix in that case.

- Sean


From roland at topspin.com  Thu Feb 24 09:47:06 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 09:47:06 -0800
Subject: [openib-general] ib_cancel_mad and the CM
In-Reply-To: <421E11A1.9010402@ichips.intel.com> (Sean Hefty's message of
	"Thu, 24 Feb 2005 09:40:49 -0800")
References: <421CE184.4090803@ichips.intel.com>
	<421E11A1.9010402@ichips.intel.com>
Message-ID: <527jkx97fp.fsf@topspin.com>

    Sean> This will allow locking around the cancel routine, which
    Sean> should fix the problem for the CM code.  However, I don't
    Sean> think that locking around the cancel routine eliminates the
    Sean> issue from the SA query code, but I also don't see a simple
    Sean> fix in that case.

Actually I think there is a simple fix for the SA that doesn't require
any changes to the MAD layer: just use the upper part of the wr_id to
hold a sequence number.  I'll post a patch later today.

 - R.


From xma at us.ibm.com  Thu Feb 24 10:07:13 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 11:07:13 -0700
Subject: [openib-general] [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <52fyzm80ng.fsf@topspin.com>
Message-ID: <OF169398D1.0A839BC2-ON87256FB2.0062D24E-00256FB2.0063B828@us.ibm.com>

Here is the patch against the most recent bit. Please review it.

Signed-off-by: Shirley Ma <xma at us.ibm.com>

diff -urpN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-init/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-24 18:06:15.000000000 
+0000
+++ infiniband-init/ulp/ipoib/ipoib_main.c      2005-02-24 
18:07:37.000000000 +0000
@@ -348,6 +348,7 @@ static struct ipoib_path *path_rec_creat
 
        path->dev = dev;
        path->pathrec.dlid = 0;
+       path->ah = NULL;
 
        skb_queue_head_init(&path->queue);
 
@@ -450,8 +451,8 @@ static void neigh_add_path(struct sk_buf
 err:
        *to_ipoib_neigh(skb->dst->neighbour) = NULL;
        list_del(&neigh->list);
-       kfree(neigh);
        neigh->neighbour->ops->destructor = NULL;
+       kfree(neigh);
 
        ++priv->stats.tx_dropped;
        dev_kfree_skb_any(skb);
@@ -673,6 +674,7 @@ static void ipoib_neigh_destructor(struc
                        ipoib_put_ah(neigh->ah);
                list_del(&neigh->list);
                *to_ipoib_neigh(n) = NULL;
+               n->ops->destructor = NULL;
                kfree(neigh);
        }

 


Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/1ac8e927/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-init.patch
Type: application/octet-stream
Size: 877 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/1ac8e927/attachment.obj>

From xma at us.ibm.com  Thu Feb 24 10:13:04 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 11:13:04 -0700
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <OF93427C27.4C02A68C-ON87256FAC.0002199E-00256FAC.0002EA4F@LocalDomain>
Message-ID: <OF0D07FEBF.448ADD5F-ON87256FB2.0063B293-00256FB2.00644178@us.ibm.com>

Here is the patch against the most recent bit. Please review it. 

Signed-off-by: Shirley Ma <xma at us.ibm.com> 

diff -urpN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-spinlock/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-24 18:06:15.000000000 
+0000
+++ infiniband-spinlock/ulp/ipoib/ipoib_main.c  2005-02-24 
18:12:13.000000000 +0000
@@ -220,18 +220,23 @@ static void __path_free(struct net_devic
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tn;
        struct sk_buff *skb;
+       LIST_HEAD(ah_list);
+       struct ipoib_ah *ah, *tah;
 
        while ((skb = __skb_dequeue(&path->queue)))
                dev_kfree_skb_irq(skb);
 
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
                if (neigh->ah)
-                       ipoib_put_ah(neigh->ah);
+                       list_add_tail(&neigh->ah->list, &ah_list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
 
+       list_for_each_entry_safe(ah, tah, &ah_list, list)
+               ipoib_put_ah(ah);
+
        if (path->ah)
                ipoib_put_ah(path->ah);
 
@@ -660,6 +665,7 @@ static void ipoib_neigh_destructor(struc
        struct ipoib_neigh *neigh = *to_ipoib_neigh(n);
        struct ipoib_dev_priv *priv = netdev_priv(n->dev);
        unsigned long flags;
+       struct ipoib_ah *ah = NULL;
 
        ipoib_dbg(priv,
                  "neigh_destructor for %06x " IPOIB_GID_FMT "\n",
@@ -670,13 +676,16 @@ static void ipoib_neigh_destructor(struc
 
        if (neigh) {
                if (neigh->ah)
-                       ipoib_put_ah(neigh->ah);
+                       ah = neigh->ah;
                list_del(&neigh->list);
                *to_ipoib_neigh(n) = NULL;
                kfree(neigh);
        }
 
        spin_unlock_irqrestore(&priv->lock, flags);
+ 
+       if (ah)
+               ipoib_put_ah(ah);
 }
 
 static int ipoib_neigh_setup(struct neighbour *neigh)
diff -urpN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-spinlock/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-24 
18:06:15.000000000 +0000
+++ infiniband-spinlock/ulp/ipoib/ipoib_multicast.c     2005-02-24 
18:12:13.000000000 +0000
@@ -93,6 +93,8 @@ static void ipoib_mcast_free(struct ipoi
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tmp;
        unsigned long flags;
+       LIST_HEAD(ah_list);
+       struct ipoib_ah *ah, *tah;
 
        ipoib_dbg_mcast(netdev_priv(dev),
                        "deleting multicast group " IPOIB_GID_FMT "\n",
@@ -101,7 +103,8 @@ static void ipoib_mcast_free(struct ipoi
        spin_lock_irqsave(&priv->lock, flags);
 
        list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) {
-               ipoib_put_ah(neigh->ah);
+               if (neigh->ah)
+                       list_add_tail(&neigh->ah->list, &ah_list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
@@ -109,6 +112,9 @@ static void ipoib_mcast_free(struct ipoi
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
+       list_for_each_entry_safe(ah, tah, &ah_list, list)
+               ipoib_put_ah(ah);
+
        if (mcast->ah)
                ipoib_put_ah(mcast->ah);
 



Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/908b496b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-spinlock.patch
Type: application/octet-stream
Size: 2770 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/908b496b/attachment.obj>

From xma at us.ibm.com  Thu Feb 24 10:19:06 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 11:19:06 -0700
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <OFDA9654E4.F11208C0-ON87256FAC.0006C21B-00256FAC.0007AED2@us.ibm.com>
Message-ID: <OFC047A7AB.BBA30E07-ON87256FB2.00642042-00256FB2.0064CE8A@us.ibm.com>

Here is the patch against the most recent bit. Please review it. 

Signed-off-by: Shirley Ma <xma at us.ibm.com> 

diff -urpN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-ah1/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-24 18:06:15.000000000 
+0000
+++ infiniband-ah1/ulp/ipoib/ipoib_main.c       2005-02-24 
18:18:12.000000000 +0000
@@ -225,16 +225,20 @@ static void __path_free(struct net_devic
                dev_kfree_skb_irq(skb);
 
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
-               if (neigh->ah)
+               if (neigh->ah) {
                        ipoib_put_ah(neigh->ah);
+                       neigh->bh = NULL;
+               }
+               list_del(&neigh->list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
 
-       if (path->ah)
+       if (path->ah) {
                ipoib_put_ah(path->ah);
-
+               path->ah = NULL;
+       }
        rb_erase(&path->rb_node, &priv->path_tree);
        list_del(&path->list);
        kfree(path);
diff -urpN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-ah1/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-24 
18:06:15.000000000 +0000
+++ infiniband-ah1/ulp/ipoib/ipoib_multicast.c  2005-02-24 
18:18:12.000000000 +0000
@@ -101,7 +101,11 @@ static void ipoib_mcast_free(struct ipoi
        spin_lock_irqsave(&priv->lock, flags);
 
        list_for_each_entry_safe(neigh, tmp, &mcast->neigh_list, list) {
-               ipoib_put_ah(neigh->ah);
+               if (neigh->ah) {
+                       ipoib_put_ah(neigh->ah);
+                       neigh->bh = NULL;
+               }
+               list_del(&neigh->list);
                *to_ipoib_neigh(neigh->neighbour) = NULL;
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
@@ -109,8 +113,10 @@ static void ipoib_mcast_free(struct ipoi
 
        spin_unlock_irqrestore(&priv->lock, flags);
 
-       if (mcast->ah)
+       if (mcast->ah) {
                ipoib_put_ah(mcast->ah);
+               mcast->ah = NULL;
+       }
 
        while (!skb_queue_empty(&mcast->pkt_queue)) {
                struct sk_buff *skb = skb_dequeue(&mcast->pkt_queue);




Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/b6dff781/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-ah1.patch
Type: application/octet-stream
Size: 1772 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/b6dff781/attachment.obj>

From xma at us.ibm.com  Thu Feb 24 10:32:03 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 11:32:03 -0700
Subject: [openib-general] [PATCH]IPOIB: adding spin_lock in path_free()
In-Reply-To: <OFD6356A4D.92D673E6-ON87256FAC.00089C5F-00256FAC.00097D08@us.ibm.com>
Message-ID: <OF0B648F9F.BC5E7700-ON87256FB2.0064AFE1-00256FB2.0065FE1A@us.ibm.com>

Here is the patch against the most recent bit. Please review it. 

Signed-off-by: Shirley Ma <xma at us.ibm.com> 

diff -urpN infiniband/ulp/ipoib/ipoib_main.c 
infiniband-pathfree/ulp/ipoib/ipoib_main.c
--- infiniband/ulp/ipoib/ipoib_main.c   2005-02-24 18:06:15.000000000 
+0000
+++ infiniband-pathfree/ulp/ipoib/ipoib_main.c  2005-02-24 
18:28:58.000000000 +0000
@@ -215,15 +215,17 @@ static int __path_add(struct net_device 
        return 0;
 }
 
-static void __path_free(struct net_device *dev, struct ipoib_path *path)
+static void path_free(struct net_device *dev, struct ipoib_path *path)
 {
        struct ipoib_dev_priv *priv = netdev_priv(dev);
        struct ipoib_neigh *neigh, *tn;
        struct sk_buff *skb;
+       unsigned long flags;
 
        while ((skb = __skb_dequeue(&path->queue)))
                dev_kfree_skb_irq(skb);
 
+       spin_lock_irqsave(&priv->lock, flags);
        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
                if (neigh->ah)
                        ipoib_put_ah(neigh->ah);
@@ -231,13 +233,16 @@ static void __path_free(struct net_devic
                neigh->neighbour->ops->destructor = NULL;
                kfree(neigh);
        }
+       spin_unlock_irqrestore(&priv->lock, flags);
 
        if (path->ah)
                ipoib_put_ah(path->ah);
 
+       spin_lock_irqsave(&priv->lock, flags);
        rb_erase(&path->rb_node, &priv->path_tree);
        list_del(&path->list);
        kfree(path);
+       spin_unlock_irqrestore(&priv->lock, flags);
 }
 
 void ipoib_flush_paths(struct net_device *dev)
@@ -256,7 +261,7 @@ void ipoib_flush_paths(struct net_device
                if (path->query)
                        ib_sa_cancel_query(path->query_id, path->query);
                wait_for_completion(&path->done);
-               __path_free(dev, path);
+               path_free(dev, path);
        }
 }
 
@@ -496,8 +501,11 @@ static void unicast_arp_send(struct sk_b
                        skb_push(skb, sizeof *phdr);
                        __skb_queue_tail(&path->queue, skb);
 
-                       if (path_rec_start(dev, path))
-                               __path_free(dev, path);
+                       if (path_rec_start(dev, path)) {
+                               spin_unlock(&priv->lock);
+                               path_free(dev, path);
+                               return;
+                       }
                } else {
                        ++priv->stats.tx_dropped;
                        dev_kfree_skb_any(skb);




Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/69406f18/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-pathfree.patch
Type: application/octet-stream
Size: 1900 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/69406f18/attachment.obj>

From roland at topspin.com  Thu Feb 24 10:40:09 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 10:40:09 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050224145856.GZ23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 24 Feb 2005 16:58:56 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il>
Message-ID: <523bvl94za.fsf@topspin.com>

    Michael> The pingpong test fails for me, on x86_64. (The
    Michael> executable being a 64 bit one). My kernel is 2.6.10.

OK, I installed x86_64 userspace and got user verbs working.  The
first problem you ran into was because on a 32-bit arch,

    struct {
            uint64_t a;
            uint32_t b;
    };

has size 12 but on a 64-bit arch it has size 16.  Because of the way I
tested sizes of parameters passed into the kernel, it was basically
looking for userspace to pass a size of 12.  I fixed this by padding
all the structs to a multiple of 8 bytes so we don't run into this.

I also fixed a few warnings and other small glitches, and now the
pingpong test works for me from both 32- and 64-bit userspace.

Since the ABI is different, you'll have to rebuild both the kernel
modules and the userspace libraries when updating.

Thanks,
  Roland



From roland at topspin.com  Thu Feb 24 10:46:36 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 10:46:36 -0800
Subject: [openib-general] [PATCH] IPOIB: initialization patch in ipoib
In-Reply-To: <OF169398D1.0A839BC2-ON87256FB2.0062D24E-00256FB2.0063B828@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 11:07:13 -0700")
References: <OF169398D1.0A839BC2-ON87256FB2.0062D24E-00256FB2.0063B828@us.ibm.com>
Message-ID: <52y8dd7q43.fsf@topspin.com>

Thanks, applied except for this bit:

 >	@@ -673,6 +674,7 @@ static void ipoib_neigh_destructor(struc
 >	                        ipoib_put_ah(neigh->ah);
 >	                list_del(&neigh->list);
 >	                *to_ipoib_neigh(n) = NULL;
 >	+               n->ops->destructor = NULL;
 >	                kfree(neigh);
 >	        }

This doesn't seem necessary to me, because we are actually in the call
to the neighbour destructor.  So the neighbour struct will be freed
right after we return and there's no risk of another call.

Thanks,
 Roland


From libor at topspin.com  Thu Feb 24 11:11:10 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 24 Feb 2005 11:11:10 -0800
Subject: [openib-general] sdp_kvec.c
In-Reply-To: <20050223110243.GA23837@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Feb 23, 2005 at 01:02:43PM +0200
References: <20050223110243.GA23837@mellanox.co.il>
Message-ID: <20050224111110.A24393@topspin.com>

On Wed, Feb 23, 2005 at 01:02:43PM +0200, Michael S. Tsirkin wrote:
> Libor, shall not the code in sdp_kvec.c go into
> sdp_iocb.c? I dont see kvec mentionned even once in there ...

  Yes. Here's a patch to simplify the code in sdp_kvec.c and move it
to sdp_conn.c. The reason I moved it there instead of sdp_iocb.c is
that the cancel functions are for a specific connection, and cancel
all outstanding AIO requests for a given connection. 

 3 files changed, 43 insertions(+), 156 deletions(-)

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: Makefile
===================================================================
--- Makefile	(revision 1905)
+++ Makefile	(working copy)
@@ -10,7 +10,6 @@
 	sdp_buff.o \
 	sdp_rcvd.o \
 	sdp_sent.o \
-	sdp_kvec.o \
 	sdp_read.o \
 	sdp_write.o \
 	sdp_wall.o \
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1905)
+++ sdp_conn.c	(working copy)
@@ -596,7 +596,50 @@
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return conn;
 } /* sdp_conn_table_lookup */
+/* 
+ * Functions to cancel IOCB requests in a conenctions queues.
+ */
+static int _sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
+{
+	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
+} /* _sdp_iocb_q_cancel_lookup_func */
 
+static void _sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
+{
+	struct sdpc_iocb *iocb;
+
+	while (NULL != (iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
+			(table,
+			 _sdp_desc_q_cancel_lookup_func,
+			 NULL))) {
+
+		sdp_iocb_q_remove(iocb);
+		(void)sdp_iocb_complete(iocb, error);
+	}
+} /* _sdp_desc_q_cancel_iocb */
+
+void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
+{
+	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
+
+	_sdp_desc_q_cancel_iocb(&conn->r_src, error);
+} /* sdp_iocb_q_cancel_all_read */
+
+void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
+{
+	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
+
+	_sdp_desc_q_cancel_iocb(&conn->send_queue, error);
+	_sdp_desc_q_cancel_iocb(&conn->w_snk, error);
+} /* sdp_iocb_q_cancel_all_write */
+
+void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
+{
+	sdp_iocb_q_cancel_all_read(conn, error);
+	sdp_iocb_q_cancel_all_write(conn, error);
+} /* sdp_iocb_q_cancel_all */
+
 /*
  * connection allocation/deallocation
  */
Index: sdp_kvec.c
===================================================================
--- sdp_kvec.c	(revision 1905)
+++ sdp_kvec.c	(working copy)
@@ -1,155 +0,0 @@
-/*
- * Copyright (c) 2005 Topspin Communications.  All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- * $Id$
- */
-
-#include "sdp_main.h"
-
-/*
- * _sdp_iocb_q_cancel_lookup_func - lookup function for cancelation
- */
-static int _sdp_iocb_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
-{
-	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
-} /* _sdp_iocb_q_cancel_lookup_func */
-
-/*
- * _sdp_iocb_q_cancel_read_pending - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn,
-					    ssize_t error)
-{
-	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
-} /* _sdp_iocb_q_cancel_read_pending */
-
-/*
- * _sdp_iocb_q_cancel_read_source - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->r_src,
-								   _sdp_iocb_q_cancel_lookup_func,
-								   NULL))) {
-		sdp_iocb_q_remove(iocb);
-
-		result = sdp_iocb_complete(iocb, error);
-		SDP_EXPECT(!(0 > result));
-	}
-} /* _sdp_iocb_q_cancel_read_source */
-
-/*
- * _sdp_iocb_q_cancel_read_snk - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
-{
-	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
-} /* _sdp_iocb_q_cancel_read_snk */
-
-/*
- * _sdp_iocb_q_cancel_write_pending - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn,
-					     ssize_t error)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	while (NULL !=
-	       (iocb =
-		(struct sdpc_iocb *) sdp_desc_q_lookup(&conn->send_queue,
-						    _sdp_iocb_q_cancel_lookup_func,
-						    NULL))) {
-		sdp_iocb_q_remove(iocb);
-
-		result = sdp_iocb_complete(iocb, error);
-		SDP_EXPECT(!(0 > result));
-	}
-} /* _sdp_iocb_q_cancel_write_pending */
-
-/*
- * _sdp_iocb_q_cancel_write_source - cancel all pending source AIOs
- */
-static void _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn,
-					    ssize_t error)
-{
-	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
-} /* _sdp_iocb_q_cancel_write_source */
-
-/*
- * _sdp_iocb_q_cancel_write_snk - cancel all pending sink AIOs
- */
-static void _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->w_snk,
-								   _sdp_iocb_q_cancel_lookup_func,
-								   NULL))) {
-		sdp_iocb_q_remove(iocb);
-
-		result = sdp_iocb_complete(iocb, error);
-		SDP_EXPECT(!(0 > result));
-	}
-} /* _sdp_iocb_q_cancel_write_snk */
-
-/* 
- * sdp_iocb_q_cancel_all_read - cancel all outstanding read AIOs
- */
-void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
-{
-	_sdp_iocb_q_cancel_read_pending(conn, error);
-	_sdp_iocb_q_cancel_read_snk(conn, error);
-	_sdp_iocb_q_cancel_read_source(conn, error);
-} /* sdp_iocb_q_cancel_all_read */
-
-/*
- * sdp_iocb_q_cancel_all_write - cancel all outstanding write AIOs
- */
-void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
-{
-	_sdp_iocb_q_cancel_write_pending(conn, error);
-	_sdp_iocb_q_cancel_write_source(conn, error);
-	_sdp_iocb_q_cancel_write_snk(conn, error);
-} /* sdp_iocb_q_cancel_all_write */
-
-/*
- * sdp_iocb_q_cancel_all - cancel all outstanding AIOs
- */
-void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
-{
-	sdp_iocb_q_cancel_all_read(conn, error);
-	sdp_iocb_q_cancel_all_write(conn, error);
-} /* sdp_iocb_q_cancel_all */


From libor at topspin.com  Thu Feb 24 11:21:32 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 24 Feb 2005 11:21:32 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <523bvl94za.fsf@topspin.com>;
	from roland@topspin.com on Thu, Feb 24, 2005 at 10:40:09AM -0800
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il>
	<523bvl94za.fsf@topspin.com>
Message-ID: <20050224112132.B24393@topspin.com>

On Thu, Feb 24, 2005 at 10:40:09AM -0800, Roland Dreier wrote:
>     Michael> The pingpong test fails for me, on x86_64. (The
>     Michael> executable being a 64 bit one). My kernel is 2.6.10.
> 
> OK, I installed x86_64 userspace and got user verbs working.  The
> first problem you ran into was because on a 32-bit arch,
> 
>     struct {
>             uint64_t a;
>             uint32_t b;
>     };
> 
> has size 12 but on a 64-bit arch it has size 16.  Because of the way I
> tested sizes of parameters passed into the kernel, it was basically
> looking for userspace to pass a size of 12.  I fixed this by padding
> all the structs to a multiple of 8 bytes so we don't run into this.

I'm curious, why pad the structure instead of using the packed attribute?

     struct {
             uint64_t a;
             uint32_t b;
     } __attribute__ ((packed));


Thanks.

-Libor


From roland at topspin.com  Thu Feb 24 11:31:53 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 11:31:53 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of
	userspace verbs
In-Reply-To: <20050224112132.B24393@topspin.com> (Libor Michalek's message
	of "Thu, 24 Feb 2005 11:21:32 -0800")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il> <523bvl94za.fsf@topspin.com>
	<20050224112132.B24393@topspin.com>
Message-ID: <528y5d7o0m.fsf@topspin.com>

    Libor> I'm curious, why pad the structure instead of using the
    Libor> packed attribute?

I always try to avoid the packed attribute because it forces gcc to
generate some terrible code on RISC architectures, since it has to
assume the structure might be unaligned.

 - R.


From roland at topspin.com  Thu Feb 24 11:39:48 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 11:39:48 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of
	userspace verbs
In-Reply-To: <528y5d7o0m.fsf@topspin.com> (Roland Dreier's message of "Thu,
	24 Feb 2005 11:31:53 -0800")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il> <523bvl94za.fsf@topspin.com>
	<20050224112132.B24393@topspin.com> <528y5d7o0m.fsf@topspin.com>
Message-ID: <523bvl7nnf.fsf@topspin.com>

Just for kicks, I compiled this code for ia64 and sparc64:

    struct foo { int a; };
    struct bar { int b; } __attribute__((packed));

    int c(struct foo *x) { return x->a; }
    int d(struct bar *x) { return x->b; }

The ia64 assembly for d() does not look pretty:

    0000000000000000 <c>:
       0:	0a 10 00 18 00 21 	[MMI]       mov r2=r12;;
       6:	00 00 09 30 23 00 	            st8 [r2]=r32
       c:	00 00 04 00       	            nop.i 0x0
      10:	0b 70 00 04 18 10 	[MMI]       ld8 r14=[r2];;
      16:	e0 00 38 20 20 00 	            ld4 r14=[r14]
      1c:	00 00 04 00       	            nop.i 0x0;;
      20:	11 40 00 1c 00 21 	[MIB]       mov r8=r14
      26:	c0 00 08 00 42 80 	            mov r12=r2
      2c:	08 00 84 00       	            br.ret.sptk.many b0;;

    0000000000000030 <d>:
      30:	0a 10 00 18 00 21 	[MMI]       mov r2=r12;;
      36:	00 00 09 30 23 00 	            st8 [r2]=r32
      3c:	00 00 04 00       	            nop.i 0x0
      40:	0b 78 00 04 18 10 	[MMI]       ld8 r15=[r2];;
      46:	00 01 3c 00 20 c0 	            ld1 r16=[r15]
      4c:	11 78 00 84       	            adds r14=1,r15;;
      50:	0b 70 00 1c 00 10 	[MMI]       ld1 r14=[r14];;
      56:	00 00 00 02 00 c0 	            nop.m 0x0
      5c:	e1 b8 dd 53       	            shl r14=r14,8;;
      60:	0d 80 40 1c 0e 20 	[MFI]       or r16=r16,r14
      66:	00 00 00 02 00 c0 	            nop.f 0x0
      6c:	21 78 00 84       	            adds r14=2,r15;;
      70:	0b 70 00 1c 00 10 	[MMI]       ld1 r14=[r14];;
      76:	00 00 00 02 00 c0 	            nop.m 0x0
      7c:	e1 78 bd 53       	            shl r14=r14,16;;
      80:	0d 80 40 1c 0e 20 	[MFI]       or r16=r16,r14
      86:	00 00 00 02 00 c0 	            nop.f 0x0
      8c:	31 78 00 84       	            adds r14=3,r15;;
      90:	0b 70 00 1c 00 10 	[MMI]       ld1 r14=[r14];;
      96:	00 00 00 02 00 c0 	            nop.m 0x0
      9c:	e1 38 9d 53       	            shl r14=r14,24;;
      a0:	0b 70 40 1c 0e 20 	[MMI]       or r14=r16,r14;;
      a6:	00 00 00 02 00 c0 	            nop.m 0x0
      ac:	e1 f8 7c 53       	            shl r14=r14,32;;
      b0:	02 00 00 00 01 00 	[MII]       nop.m 0x0
      b6:	e0 08 3a 3e 29 00 	            shr r14=r14,32;;
      bc:	01 70 00 84       	            mov r8=r14
      c0:	1d 60 00 04 00 21 	[MFB]       mov r12=r2
      c6:	00 00 00 02 00 80 	            nop.f 0x0
      cc:	08 00 84 00       	            br.ret.sptk.many b0;;

On sparc64, d() looks similarly terrible:

    0000000000000000 <c>:
       0:	9d e3 bf 40 	save  %sp, -192, %sp
       4:	f0 77 a8 7f 	stx  %i0, [ %fp + 0x87f ]
       8:	c2 5f a8 7f 	ldx  [ %fp + 0x87f ], %g1
       c:	c2 00 40 00 	ld  [ %g1 ], %g1
      10:	83 38 60 00 	sra  %g1, 0, %g1
      14:	b0 10 00 01 	mov  %g1, %i0
      18:	81 cf e0 08 	rett  %i7 + 8
      1c:	01 00 00 00 	nop 

    0000000000000020 <d>:
      20:	9d e3 bf 40 	save  %sp, -192, %sp
      24:	f0 77 a8 7f 	stx  %i0, [ %fp + 0x87f ]
      28:	c4 5f a8 7f 	ldx  [ %fp + 0x87f ], %g2
      2c:	c2 08 80 00 	ldub  [ %g2 ], %g1
      30:	82 08 60 ff 	and  %g1, 0xff, %g1
      34:	87 28 70 18 	sllx  %g1, 0x18, %g3
      38:	c2 08 a0 01 	ldub  [ %g2 + 1 ], %g1
      3c:	82 08 60 ff 	and  %g1, 0xff, %g1
      40:	83 28 70 10 	sllx  %g1, 0x10, %g1
      44:	86 10 40 03 	or  %g1, %g3, %g3
      48:	c2 08 a0 02 	ldub  [ %g2 + 2 ], %g1
      4c:	82 08 60 ff 	and  %g1, 0xff, %g1
      50:	83 28 70 08 	sllx  %g1, 8, %g1
      54:	86 10 40 03 	or  %g1, %g3, %g3
      58:	c2 08 a0 03 	ldub  [ %g2 + 3 ], %g1
      5c:	82 08 60 ff 	and  %g1, 0xff, %g1
      60:	82 10 40 03 	or  %g1, %g3, %g1
      64:	83 28 70 20 	sllx  %g1, 0x20, %g1
      68:	83 38 70 20 	srax  %g1, 0x20, %g1
      6c:	83 38 60 00 	sra  %g1, 0, %g1
      70:	b0 10 00 01 	mov  %g1, %i0
      74:	81 cf e0 08 	rett  %i7 + 8
      78:	01 00 00 00 	nop 



From mst at mellanox.co.il  Thu Feb 24 12:03:39 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 22:03:39 +0200
Subject: [openib-general] sdp_kvec.c
In-Reply-To: <20050224111110.A24393@topspin.com>
References: <20050223110243.GA23837@mellanox.co.il>
	<20050224111110.A24393@topspin.com>
Message-ID: <20050224200339.GA15534@mellanox.co.il>


Quoting r. Libor Michalek <libor at topspin.com>:
> +/* 
> + * Functions to cancel IOCB requests in a conenctions queues.
> + */
> +static int _sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
> +{
> +	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
> +} /* _sdp_iocb_q_cancel_lookup_func */

What do you say to removing the comments after the closing }?
Its clearly unnecessary (most functions are, or shall be, short enough
to see where they end), and increases the code footprint.
For me, they make the detection of the function end harder rather than
easier, since I am always looking for a sole }.

-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Thu Feb 24 12:15:50 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 24 Feb 2005 12:15:50 -0800
Subject: [openib-general] sdp_kvec.c
In-Reply-To: <20050224200339.GA15534@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 24, 2005 at 10:03:39PM +0200
References: <20050223110243.GA23837@mellanox.co.il>
	<20050224111110.A24393@topspin.com>
	<20050224200339.GA15534@mellanox.co.il>
Message-ID: <20050224121550.C24393@topspin.com>

On Thu, Feb 24, 2005 at 10:03:39PM +0200, Michael S. Tsirkin wrote:
> 
> Quoting r. Libor Michalek <libor at topspin.com>:
> > +/* 
> > + * Functions to cancel IOCB requests in a conenctions queues.
> > + */
> > +static int _sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
> > +{
> > +	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
> > +} /* _sdp_iocb_q_cancel_lookup_func */
> 
> What do you say to removing the comments after the closing }?
> Its clearly unnecessary (most functions are, or shall be, short enough
> to see where they end), and increases the code footprint.
> For me, they make the detection of the function end harder rather than
> easier, since I am always looking for a sole }.

  OK, I'll stop adding them to new functions, and remove them as I
touch code. If I or anyone else get's ambitious we can strip them
from all the functions.

-Libor


Index: Makefile
===================================================================
--- Makefile	(revision 1905)
+++ Makefile	(working copy)
@@ -10,7 +10,6 @@
 	sdp_buff.o \
 	sdp_rcvd.o \
 	sdp_sent.o \
-	sdp_kvec.o \
 	sdp_read.o \
 	sdp_write.o \
 	sdp_wall.o \
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1905)
+++ sdp_conn.c	(working copy)
@@ -597,6 +597,50 @@
 	return conn;
 } /* sdp_conn_table_lookup */
 
+/* 
+ * Functions to cancel IOCB requests in a conenctions queues.
+ */
+static int _sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
+{
+	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
+}
+
+static void _sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
+{
+	struct sdpc_iocb *iocb;
+
+	while (NULL != (iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
+			(table,
+			 _sdp_desc_q_cancel_lookup_func,
+			 NULL))) {
+
+		sdp_iocb_q_remove(iocb);
+		(void)sdp_iocb_complete(iocb, error);
+	}
+}
+
+void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
+{
+	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
+
+	_sdp_desc_q_cancel_iocb(&conn->r_src, error);
+}
+
+void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
+{
+	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
+
+	_sdp_desc_q_cancel_iocb(&conn->send_queue, error);
+	_sdp_desc_q_cancel_iocb(&conn->w_snk, error);
+}
+
+void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
+{
+	sdp_iocb_q_cancel_all_read(conn, error);
+	sdp_iocb_q_cancel_all_write(conn, error);
+}
+
 /*
  * connection allocation/deallocation
  */
Index: sdp_kvec.c
===================================================================
--- sdp_kvec.c	(revision 1905)
+++ sdp_kvec.c	(working copy)
@@ -1,155 +0,0 @@
-/*
- * Copyright (c) 2005 Topspin Communications.  All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- * $Id$
- */
-
-#include "sdp_main.h"
-
-/*
- * _sdp_iocb_q_cancel_lookup_func - lookup function for cancelation
- */
-static int _sdp_iocb_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
-{
-	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
-} /* _sdp_iocb_q_cancel_lookup_func */
-
-/*
- * _sdp_iocb_q_cancel_read_pending - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_read_pending(struct sdp_opt *conn,
-					    ssize_t error)
-{
-	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
-} /* _sdp_iocb_q_cancel_read_pending */
-
-/*
- * _sdp_iocb_q_cancel_read_source - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_read_source(struct sdp_opt *conn, ssize_t error)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->r_src,
-								   _sdp_iocb_q_cancel_lookup_func,
-								   NULL))) {
-		sdp_iocb_q_remove(iocb);
-
-		result = sdp_iocb_complete(iocb, error);
-		SDP_EXPECT(!(0 > result));
-	}
-} /* _sdp_iocb_q_cancel_read_source */
-
-/*
- * _sdp_iocb_q_cancel_read_snk - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_read_snk(struct sdp_opt *conn, ssize_t error)
-{
-	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
-} /* _sdp_iocb_q_cancel_read_snk */
-
-/*
- * _sdp_iocb_q_cancel_write_pending - cancel all pending read AIOs
- */
-static void _sdp_iocb_q_cancel_write_pending(struct sdp_opt *conn,
-					     ssize_t error)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	while (NULL !=
-	       (iocb =
-		(struct sdpc_iocb *) sdp_desc_q_lookup(&conn->send_queue,
-						    _sdp_iocb_q_cancel_lookup_func,
-						    NULL))) {
-		sdp_iocb_q_remove(iocb);
-
-		result = sdp_iocb_complete(iocb, error);
-		SDP_EXPECT(!(0 > result));
-	}
-} /* _sdp_iocb_q_cancel_write_pending */
-
-/*
- * _sdp_iocb_q_cancel_write_source - cancel all pending source AIOs
- */
-static void _sdp_iocb_q_cancel_write_source(struct sdp_opt *conn,
-					    ssize_t error)
-{
-	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
-} /* _sdp_iocb_q_cancel_write_source */
-
-/*
- * _sdp_iocb_q_cancel_write_snk - cancel all pending sink AIOs
- */
-static void _sdp_iocb_q_cancel_write_snk(struct sdp_opt *conn, ssize_t error)
-{
-	struct sdpc_iocb *iocb;
-	int result;
-
-	while (NULL != (iocb = (struct sdpc_iocb *) sdp_desc_q_lookup(&conn->w_snk,
-								   _sdp_iocb_q_cancel_lookup_func,
-								   NULL))) {
-		sdp_iocb_q_remove(iocb);
-
-		result = sdp_iocb_complete(iocb, error);
-		SDP_EXPECT(!(0 > result));
-	}
-} /* _sdp_iocb_q_cancel_write_snk */
-
-/* 
- * sdp_iocb_q_cancel_all_read - cancel all outstanding read AIOs
- */
-void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error)
-{
-	_sdp_iocb_q_cancel_read_pending(conn, error);
-	_sdp_iocb_q_cancel_read_snk(conn, error);
-	_sdp_iocb_q_cancel_read_source(conn, error);
-} /* sdp_iocb_q_cancel_all_read */
-
-/*
- * sdp_iocb_q_cancel_all_write - cancel all outstanding write AIOs
- */
-void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
-{
-	_sdp_iocb_q_cancel_write_pending(conn, error);
-	_sdp_iocb_q_cancel_write_source(conn, error);
-	_sdp_iocb_q_cancel_write_snk(conn, error);
-} /* sdp_iocb_q_cancel_all_write */
-
-/*
- * sdp_iocb_q_cancel_all - cancel all outstanding AIOs
- */
-void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
-{
-	sdp_iocb_q_cancel_all_read(conn, error);
-	sdp_iocb_q_cancel_all_write(conn, error);
-} /* sdp_iocb_q_cancel_all */


From mst at mellanox.co.il  Thu Feb 24 12:58:34 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 22:58:34 +0200
Subject: [openib-general] sdp_kvec.c
In-Reply-To: <20050224121550.C24393@topspin.com>
References: <20050223110243.GA23837@mellanox.co.il>
	<20050224111110.A24393@topspin.com>
	<20050224200339.GA15534@mellanox.co.il>
	<20050224121550.C24393@topspin.com>
Message-ID: <20050224205834.GA16594@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> > What do you say to removing the comments after the closing }?
> > Its clearly unnecessary (most functions are, or shall be, short enough
> > to see where they end), and increases the code footprint.
> > For me, they make the detection of the function end harder rather than
> > easier, since I am always looking for a sole }.
> 
>   OK, I'll stop adding them to new functions, and remove them as I
> touch code. If I or anyone else get's ambitious we can strip them
> from all the functions.
> 
> -Libor
> 

Here's a patch.
Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_queue.h
===================================================================
--- sdp_queue.h	(revision 1906)
+++ sdp_queue.h	(working copy)
@@ -76,7 +76,7 @@ struct sdpc_desc_q {
 static inline int __sdp_desc_q_size(struct sdpc_desc_q *table)
 {
 	return table->size;
-} /* __sdp_desc_q_size */
+}
 
 /*
  * __sdp_desc_q_member - return non-zero if element is in a table
@@ -84,7 +84,7 @@ static inline int __sdp_desc_q_size(stru
 static inline int __sdp_desc_q_member(struct sdpc_desc *element)
 {
 	return ((NULL == element->table) ? 0 : 1);
-} /* __sdp_desc_q_member */
+}
 
 #define sdp_desc_q_size(x)   __sdp_desc_q_size(x)
 #define sdp_desc_q_member(x) __sdp_desc_q_member(x)
Index: sdp_write.c
===================================================================
--- sdp_write.c	(revision 1906)
+++ sdp_write.c	(working copy)
@@ -175,4 +175,4 @@ int sdp_event_write(struct sdp_opt *conn
 	return 0;
 error:
 	return result;
-} /* sdp_event_write */
+}
Index: sdp_link.c
===================================================================
--- sdp_link.c	(revision 1906)
+++ sdp_link.c	(working copy)
@@ -93,7 +93,7 @@ static void _sdp_link_path_complete(u64 
 
 	info->use = jiffies;
 	return;
-} /* _sdp_link_path_complete */
+}
 
 /*
  * _sdp_path_wait_add - add a wait entry into the wait list for a path
@@ -110,7 +110,7 @@ static void _sdp_path_wait_add(struct sd
 		wait->next->pext = &wait->next;
 
 	return;
-} /* _sdp_path_wait_add */
+}
 
 /*
  * _sdp_path_wait_destroy - destroy an entry for a wait element
@@ -133,7 +133,7 @@ static void _sdp_path_wait_destroy(struc
 	kmem_cache_free(_wait_cache, wait);
 
 	return;
-} /* _sdp_path_wait_destroy */
+}
 
 /*
  * _sdp_path_wait_complete - complete an entry for a wait element
@@ -151,7 +151,7 @@ static void _sdp_path_wait_complete(stru
 	_sdp_path_wait_destroy(wait);
 
 	return;
-} /* _sdp_path_wait_complete */
+}
 
 /*
  * _sdp_path_info_lookup - lookup a path record entry
@@ -165,7 +165,7 @@ static struct sdp_path_info *_sdp_path_i
 			break;
 
 	return info;
-} /* _sdp_path_info_lookup */
+}
 
 /*
  * _sdp_path_info_create - create an entry for a path record element
@@ -196,7 +196,7 @@ static struct sdp_path_info *_sdp_path_i
 	INIT_WORK(&info->timer, _sdp_link_path_lookup, info);
 
 	return info;
-} /* _sdp_path_info_create */
+}
 
 /*
  * _sdp_path_info_destroy - destroy an entry for a path record element
@@ -224,7 +224,7 @@ static void _sdp_path_info_destroy(struc
 	kmem_cache_free(_info_cache, info);
 
 	return;
-} /* _sdp_path_info_destroy */
+}
 
 /*
  * _sdp_link_path_rec_done - path record completion function
@@ -292,7 +292,7 @@ static void _sdp_link_path_rec_done(int 
 	}
 
 	return;
-} /* _sdp_link_path_rec_done */
+}
 
 /*
  * _sdp_link_path_rec_get - resolve GIDs to a path record
@@ -330,7 +330,7 @@ static int _sdp_link_path_rec_get(struct
 	}
 
 	return result;
-} /* _sdp_link_path_rec_get */
+}
 
 /*
  * _sdp_link_path_lookup - resolve an ip address to a path record
@@ -435,7 +435,7 @@ void _sdp_link_path_lookup(void *data)
 			     rt->rt_src);
 		result = -ENODEV;
 		goto error;
-	} /* if */
+	}
 	/*
 	 * lookup local info.
 	 */
@@ -527,7 +527,7 @@ done:
 error:
 	_sdp_path_info_destroy(info, result);
 	return;
-} /* _sdp_link_path_lookup */
+}
 
 /*
  * Public functions
@@ -600,7 +600,7 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	return 0;
 error:
 	return result;
-} /* sdp_link_path_lookup */
+}
 
 /*
  * timers
@@ -634,7 +634,7 @@ static void _sdp_link_sweep(void *data)
 
 	queue_delayed_work(_link_wq, &_link_timer, SDP_LINK_SWEEP_INTERVAL);
 	return;
-} /* _sdp_link_sweep */
+}
 
 /*
  * Arp packet reception for completions
@@ -688,7 +688,7 @@ done:
 	kfree_skb(skb);
 	kfree(work);
 	return;
-} /* _sdp_link_arp_work */
+}
 
 /*
  * _sdp_link_arp_recv - receive all ARP packets
@@ -721,7 +721,7 @@ static int _sdp_link_arp_recv(struct sk_
 done:
 	kfree_skb(skb);
 	return 0;
-} /* _sdp_link_arp_recv */
+}
 
 /*
  * primary initialization/cleanup functions
@@ -751,7 +751,7 @@ int sdp_link_addr_init(void)
 	if (0 != result) {
 		
 		goto error_async;
-	} /* if */
+	}
 #endif
 	_info_cache = kmem_cache_create("sdp_path_info",
 					sizeof(struct sdp_path_info),
@@ -799,7 +799,7 @@ error_wait:
 	kmem_cache_destroy(_info_cache);
 error_path:
 	return result;
-} /* sdp_link_addr_init */
+}
 
 /*
  * sdp_link_addr_cleanup - cleanup the link address retrival code
@@ -837,4 +837,4 @@ int sdp_link_addr_cleanup(void)
 #endif
 
 	return 0;
-} /* sdp_link_addr_cleanup */
+}
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1906)
+++ sdp_rcvd.c	(working copy)
@@ -112,7 +112,7 @@ static int _sdp_rcvd_disconnect(struct s
 	return 0;
 error:
 	return result;
-} /* _sdp_rcvd_disconnect */
+}
 
 static int _sdp_rcvd_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -142,7 +142,7 @@ static int _sdp_rcvd_abort(struct sdp_op
 	}
 
 	return result;
-} /* _sdp_rcvd_abort */
+}
 
 static int _sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -185,7 +185,7 @@ static int _sdp_rcvd_send_sm(struct sdp_
 	}
 
 	return 0;
-} /* _sdp_rcvd_send_sm */
+}
 
 static int _sdp_rcvd_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -246,7 +246,7 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	return 0;
 error:
 	return result;
-} /* _sdp_rcvd_rdma_wr */
+}
 
 static int _sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -328,7 +328,7 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	return 0;
 error:
 	return result;
-} /* _sdp_rcvd_rdma_rd */
+}
 
 static int _sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -424,7 +424,7 @@ mode_error:
 		     SDP_MSG_MCH_GET_MODE(mch));
 error:
 	return result;
-} /* _sdp_rcvd_mode_change */
+}
 
 static int _sdp_rcvd_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -514,7 +514,7 @@ static int _sdp_rcvd_src_cancel(struct s
 	return 0;
 done:
 	return result;
-} /* _sdp_rcvd_src_cancel */
+}
 
 static int _sdp_rcvd_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -585,7 +585,7 @@ static int _sdp_rcvd_snk_cancel(struct s
 	return 0;
 done:
 	return result;
-} /* _sdp_rcvd_snk_cancel */
+}
 
 /*
  * _sdp_rcvd_snk_cancel_ack - sink cancel confirmantion
@@ -624,7 +624,7 @@ static int _sdp_rcvd_snk_cancel_ack(stru
 	return 0;
 done:
 	return result;
-} /* _sdp_rcvd_snk_cancel_ack */
+}
 
 /*
  * _sdp_rcvd_resize_buff_ack - buffer size change request
@@ -657,7 +657,7 @@ static int _sdp_rcvd_resize_buff_ack(str
 	return 0;
 error:
 	return result;
-} /* _sdp_rcvd_resize_buff_ack */
+}
 
 static int _sdp_rcvd_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -669,12 +669,12 @@ static int _sdp_rcvd_suspend(struct sdp_
 	_sdp_msg_net_to_cpu_sch(sch);
 
 	return 0;
-} /* _sdp_rcvd_suspend */
+}
 
 static int _sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
-} /* _sdp_rcvd_suspend_ack */
+}
 
 static int _sdp_rcvd_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -808,7 +808,7 @@ consume:
 	 */
 error:
 	return result;
-} /* _sdp_rcvd_snk_avail */
+}
 
 static int _sdp_rcvd_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -975,7 +975,7 @@ advt_error:
 	(void)sdp_advt_destroy(advt);
 done:
 	return result;
-} /* _sdp_rcvd_src_avail */
+}
 
 /*
  * _sdp_rcvd_data - SDP data message event received
@@ -1020,7 +1020,7 @@ static int _sdp_rcvd_data(struct sdp_opt
 	conn->byte_strm += ret_val;
 	
 	return ret_val;
-} /* _sdp_rcvd_sata */
+}
 
 /*
  * _sdp_rcvd_unsupported - Valid messages we're not expecting
@@ -1036,7 +1036,7 @@ static int _sdp_rcvd_unsupported(struct 
 		     buff->bsdh_hdr->mid);
 
 	return 0;
-} /* _sdp_rcvd_unsupported */
+}
 
 /*
  * Event Dispatch table. For performance a dispatch table is used to avoid
@@ -1253,4 +1253,4 @@ drop:
 	(void)sdp_buff_pool_put(buff);
 done:
 	return result;
-} /* sdp_event_recv */
+}
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1906)
+++ sdp_inet.c	(working copy)
@@ -121,7 +121,7 @@ void sdp_inet_wake_send(struct sock *sk)
 		sk_wake_async(sk, 2, POLL_OUT);
 		read_unlock(&sk->sk_callback_lock);
 	}
-} /* sdp_inet_wake_send */
+}
 
 /*
  * sdp_inet_wake_generic - wake up a socket
@@ -136,7 +136,7 @@ void sdp_inet_wake_generic(struct sock *
 
 		read_unlock(&sk->sk_callback_lock);
 	}
-} /* sdp_inet_wake_generic */
+}
 
 /*
  * sdp_inet_wake_recv - wake up a socket for read
@@ -151,7 +151,7 @@ void sdp_inet_wake_recv(struct sock *sk,
 		sk_wake_async(sk, 1, POLL_IN);
 		read_unlock(&sk->sk_callback_lock);
 	}
-} /* sdp_inet_wake_recv */
+}
 
 /*
  * sdp_inet_wake_error - wake up a socket for error
@@ -166,7 +166,7 @@ void sdp_inet_wake_error(struct sock *sk
 		sk_wake_async(sk, 0, POLL_ERR);
 		read_unlock(&sk->sk_callback_lock);
 	}
-} /* sdp_inet_wake_error */
+}
 
 /*
  * sdp_inet_wake_urg - wake up a socket for urgent data
@@ -180,7 +180,7 @@ void sdp_inet_wake_urg(struct sock *sk)
 	if (NULL != sk)
 		sk_send_sigurg(sk);
 
-} /* sdp_inet_wake_urg */
+}
 
 /*
  * internal socket/handle managment functions
@@ -231,7 +231,7 @@ static int _sdp_inet_abort(struct sdp_op
 	}
 
 	return result;
-} /* _sdp_inet_abort */
+}
 
 /*
  * _sdp_inet_disconnect - disconnect a connection
@@ -290,7 +290,7 @@ static int _sdp_inet_disconnect(struct s
 	}
 
 	return result;
-} /* _sdp_inet_disconnect */
+}
 
 /*
  * Linux SOCKET interface, module specific functions
@@ -404,7 +404,7 @@ static int _sdp_inet_release(struct sock
 
 			set_current_state(TASK_RUNNING);
 			remove_wait_queue(sk->sk_sleep, &wait);
-		} /* if (linger) */
+		}
 #if 0
 		/*
 		 * On a blocking socket, if still draining after linger,
@@ -419,7 +419,7 @@ static int _sdp_inet_release(struct sock
 			SDP_EXPECT(!(0 > result));
 		}
 #endif
-	} /* if (blocking) */
+	}
 
 done:
 	if (0 < (SDP_ST_MASK_CLOSED & conn->istate))
@@ -433,7 +433,7 @@ done:
 	sdp_conn_put(conn);
 
 	return 0;
-} /* _sdp_inet_release */
+}
 
 /*
  * _sdp_inet_bind - bind a socket to an address/interface
@@ -525,7 +525,7 @@ static int _sdp_inet_bind(struct socket 
 done:
 	sdp_conn_unlock(conn);
 	return result;
-} /* _sdp_inet_bind */
+}
 
 /*
  * _sdp_inet_connect - connect a socket to a remote address
@@ -696,7 +696,7 @@ static int _sdp_inet_connect(struct sock
 done:
 	sdp_conn_unlock(conn);
 	return result;
-} /* _sdp_inet_connect */
+}
 
 /*
  * _sdp_inet_listen - listen on a socket for incomming addresses
@@ -753,7 +753,7 @@ static int _sdp_inet_listen(struct socke
 done:
 	sdp_conn_unlock(conn);
 	return result;
-} /* _sdp_inet_listen */
+}
 
 /*
  * _sdp_inet_accept - accept a new socket from a listen socket
@@ -911,7 +911,7 @@ listen_done:
 		     (NULL == accept_sk ? 0 : accept_conn->dst_port));
 
 	return result;
-} /* _sdp_inet_accept */
+}
 
 /*
  * _sdp_inet_getname - return a sockets address information
@@ -947,7 +947,7 @@ static int _sdp_inet_getname(struct sock
 	*size = sizeof(struct sockaddr_in);
 
 	return 0;
-} /* _sdp_inet_getname */
+}
 
 /*
  * _sdp_inet_poll - poll a socket for activity
@@ -1037,7 +1037,7 @@ static unsigned int _sdp_inet_poll(struc
 	
 
 	return mask;
-} /* _sdp_inet_poll */
+}
 
 /*
  * _sdp_inet_ioctl - serivce an ioctl request on a socket
@@ -1160,7 +1160,7 @@ static int _sdp_inet_ioctl(struct socket
 	}
 
 	return result;
-} /* _sdp_inet_ioctl */
+}
 
 /*
  * _sdp_inet_setopt - set a socket option
@@ -1227,7 +1227,7 @@ static int _sdp_inet_setopt(struct socke
 
 	sdp_conn_unlock(conn);
 	return result;
-} /* _sdp_inet_setopt */
+}
 
 /*
  * _sdp_inet_getopt - get a socket option
@@ -1289,7 +1289,7 @@ static int _sdp_inet_getopt(struct socke
 		return -EFAULT;
 
 	return 0;
-} /* _sdp_inet_getopt */
+}
 
 /*
  * _sdp_inet_shutdown - shutdown a socket
@@ -1380,7 +1380,7 @@ static int _sdp_inet_shutdown(struct soc
 
 	sdp_conn_unlock(conn);
 	return result;
-} /* _sdp_inet_shutdown */
+}
 
 /*
  * Primary socket initialization
@@ -1447,7 +1447,7 @@ static int _sdp_inet_create(struct socke
 #endif
 
 	return 0;
-} /* _sdp_inet_create */
+}
 
 /*
  * INET module initialization functions
@@ -1548,7 +1548,7 @@ error_advt:
 	(void)sdp_main_proc_cleanup();
 error_proc:
 	return result;		/* success */
-} /* sdp_init */
+}
 
 /*
  * sdp_exit - cleanup the sdp host module
@@ -1580,7 +1580,7 @@ static void __exit sdp_exit(void)
 	 * proc tables
 	 */
 	(void)sdp_main_proc_cleanup();
-} /* sdp_exit */
+}
 
 module_init(sdp_init);
 module_exit(sdp_exit);
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1906)
+++ sdp_proto.h	(working copy)
@@ -584,7 +584,7 @@ static inline int __sdp_inet_write_space
 		return ((SDP_INET_SEND_MARK < size || 1 < urg) ? size : 0);
 	else
 		return 0;
-} /* __sdp_inet_write_space */
+}
 
 /*
  * __sdp_inet_writable - return non-zero if socket is writable
@@ -596,7 +596,7 @@ static inline int __sdp_inet_writable(st
 			(conn->send_qud / 2)) ? 0 : 1;
 	else
 		return 0;
-} /* __sdp_inet_writable */
+}
 
 /*
  * __sdp_conn_stat_dump - dump stats to the log
@@ -619,7 +619,7 @@ static inline int __sdp_conn_stat_dump(s
 		}
 #endif
 	return 0;
-} /* __sdp_conn_stat_dump */
+}
 
 /*
  * __sdp_conn_state_dump - dump state information to the log
@@ -647,6 +647,6 @@ static inline int __sdp_conn_state_dump(
 			     conn->state_rec.line[counter]);
 #endif
 	return 0;
-} /* __sdp_conn_state_dump */
+}
 
 #endif /* _SDP_PROTO_H */
Index: sdp_read.c
===================================================================
--- sdp_read.c	(revision 1906)
+++ sdp_read.c	(working copy)
@@ -98,7 +98,7 @@ done:
 	return 0;
 error:
 	return result;
-} /* _sdp_event_read_advt */
+}
 
 /*
  * RDMA read QP Event Handler
@@ -282,4 +282,4 @@ int sdp_event_read(struct sdp_opt *conn,
 
 done:
 	return result;
-} /* sdp_event_read */
+}
Index: sdp_msgs.h
===================================================================
--- sdp_msgs.h	(revision 1906)
+++ sdp_msgs.h	(working copy)
@@ -284,7 +284,7 @@ static inline void __sdp_msg_swap_bsdh(s
 	header->size = cpu_to_be32(header->size);
 	header->seq_num = cpu_to_be32(header->seq_num);
 	header->seq_ack = cpu_to_be32(header->seq_ack);
-} /* __sdp_msg_swap_bsdh */
+}
 
 /*
  * __sdp_msg_swap_hh - SDP header endian byte swapping
@@ -302,7 +302,7 @@ static inline void __sdp_msg_swap_hh(str
 	header->dst.ipv6.addr1 = cpu_to_be32(header->dst.ipv6.addr1);
 	header->dst.ipv6.addr2 = cpu_to_be32(header->dst.ipv6.addr2);
 	header->dst.ipv6.addr3 = cpu_to_be32(header->dst.ipv6.addr3);
-} /* __sdp_msg_swap_ */
+}
 
 /*
  * __sdp_msg_swap_hah - SDP header endian byte swapping
@@ -310,7 +310,7 @@ static inline void __sdp_msg_swap_hh(str
 static inline void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
 {
 	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
-} /* __sdp_msg_swap_hah */
+}
 
 /*
  * __sdp_msg_swap_srcah - SDP header endian byte swapping
@@ -325,7 +325,7 @@ static inline void __sdp_msg_swap_srcah(
 	header->r_key = cpu_to_be32(header->r_key);
 	header->addr = cpu_to_be64(header->addr);
 #endif
-} /* __sdp_msg_swap_srcah */
+}
 
 /*
  * __sdp_msg_swap_snkah - SDP header endian byte swapping
@@ -336,7 +336,7 @@ static inline void __sdp_msg_swap_snkah(
 	header->r_key = cpu_to_be32(header->r_key);
 	header->addr = cpu_to_be64(header->addr);
 	header->non_disc = cpu_to_be32(header->non_disc);
-} /* __sdp_msg_swap_snkah */
+}
 
 /*
  * __sdp_msg_swap_rwch - SDP header endian byte swapping
@@ -344,7 +344,7 @@ static inline void __sdp_msg_swap_snkah(
 static inline void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
 {
 	header->size = cpu_to_be32(header->size);
-} /* __sdp_msg_swap_rwch */
+}
 
 /*
  * __sdp_msg_swap_rrch - SDP header endian byte swapping
@@ -352,7 +352,7 @@ static inline void __sdp_msg_swap_rwch(s
 static inline void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
 {
 	header->size = cpu_to_be32(header->size);
-} /* __sdp_msg_swap_rrch */
+}
 
 /*
  * __sdp_msg_swap_mch - SDP header endian byte swapping
@@ -360,7 +360,7 @@ static inline void __sdp_msg_swap_rrch(s
 static inline void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
 {
 	return;
-} /* __sdp_msg_swap_ */
+}
 
 /*
  * __sdp_msg_swap_crbh - SDP header endian byte swapping
@@ -368,7 +368,7 @@ static inline void __sdp_msg_swap_mch(st
 static inline void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
 {
 	header->size = cpu_to_be32(header->size);
-} /* __sdp_msg_swap_crbh */
+}
 
 /*
  * __sdp_msg_swap_crbah - SDP header endian byte swapping
@@ -376,7 +376,7 @@ static inline void __sdp_msg_swap_crbh(s
 static inline void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
 {
 	header->size = cpu_to_be32(header->size);
-} /* __sdp_msg_swap_crbah */
+}
 
 /*
  * __sdp_msg_swap_sch - SDP header endian byte swapping
@@ -384,7 +384,7 @@ static inline void __sdp_msg_swap_crbah(
 static inline void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
 {
 	header->service_id = cpu_to_be64(header->service_id);
-} /* __sdp_msg_swap_sch */
+}
 
 #define _sdp_msg_cpu_to_net_bsdh  __sdp_msg_swap_bsdh
 #define _sdp_msg_net_to_cpu_bsdh  __sdp_msg_swap_bsdh
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1906)
+++ sdp_send.c	(working copy)
@@ -176,7 +176,7 @@ done:
 	conn->send_seq--;
 	conn->s_wq_size--;
 	return result;
-} /* _sdp_send_buff_post */
+}
 
 /*
  * DATA functions
@@ -291,7 +291,7 @@ static int _sdp_send_data_buff_post(stru
 	return 0;
 error:
 	return result;
-} /* _sdp_send_data_buff_post */
+}
 
 /*
  * _sdp_send_data_buff_snk - Post data for buffered transmission
@@ -428,7 +428,7 @@ static int _sdp_send_data_buff_snk(struc
 	return 0;
 error:
 	return result;
-} /* _sdp_send_data_buff_snk */
+}
 
 /*
  * _sdp_send_data_iocb_snk - process a zcopy write advert in the data path
@@ -554,7 +554,7 @@ static int _sdp_send_data_iocb_snk(struc
 	return iocb->len;
 error:
 	return result;
-} /* _sdp_send_data_iocb_snk */
+}
 
 /*
  * _sdp_send_data_iocb_src - send a zcopy read advert in the data path
@@ -719,7 +719,7 @@ error:
 drop:
 	(void)sdp_buff_pool_put(buff);
 	return result;
-} /* _sdp_send_data_iocb_src */
+}
 
 /*
  * _sdp_send_iocb_buff_write - write part of an iocb into a SDP buffer
@@ -767,7 +767,7 @@ static int _sdp_send_iocb_buff_write(str
 	}
 
 	return 0;
-} /* _sdp_send_iocb_buff_write */
+}
 
 /*
  * _sdp_send_data_iocb_buff - write multiple SDP buffers from an iocb
@@ -841,7 +841,7 @@ drop:
 	(void)sdp_buff_pool_put(buff);
 error:
 	return result;
-} /* _sdp_send_data_iocb_buff */
+}
 
 /*
  * _sdp_send_data_iocb - Post IOCB data for transmission
@@ -931,7 +931,7 @@ static int _sdp_send_data_iocb(struct sd
 
 done:
 	return result;
-} /* _sdp_send_data_iocb */
+}
 
 /*
  * _sdp_send_data_queue_test - send data buffer if conditions are met
@@ -962,7 +962,7 @@ static int _sdp_send_data_queue_test(str
 						 (struct sdpc_buff *)element);
 
 	return result;
-} /* _sdp_send_data_queue_test */
+}
 
 /*
  * _sdp_send_data_queue_flush - Flush data from send queue, to send post
@@ -1004,7 +1004,7 @@ static int _sdp_send_data_queue_flush(st
 	}
 
 	return result;
-} /* _sdp_send_data_queue_flush */
+}
 
 /*
  * _sdp_send_data_queue - send using the data queue if necessary
@@ -1055,7 +1055,7 @@ static int _sdp_send_data_queue(struct s
 	return 0;
 done:
 	return result;
-} /* _sdp_send_data_queue */
+}
 
 /*
  * _sdp_send_data_buff_get - get an appropriate write buffer for send
@@ -1080,7 +1080,7 @@ static inline struct sdpc_buff *_sdp_sen
 	}
 
 	return buff;
-} /* _sdp_send_data_buff_get */
+}
 
 /*
  * _sdp_send_data_buff_put - place a buffer into the send queue
@@ -1134,7 +1134,7 @@ static inline int _sdp_send_data_buff_pu
 	}
 
 	return result;
-} /* _sdp_send_data_buff_put */
+}
 
 /*
  * CONTROL functions
@@ -1165,7 +1165,7 @@ static int _sdp_send_ctrl_buff_test(stru
 	return 0;
 error:
 	return result;
-} /* _sdp_send_ctrl_buff_test */
+}
 
 /*
  * _sdp_send_ctrl_buff_flush - Flush control buffers, to send post
@@ -1202,7 +1202,7 @@ static int _sdp_send_ctrl_buff_flush(str
 	}
 
 	return result;
-} /* _sdp_send_ctrl_buff_flush */
+}
 
 /*
  * _sdp_send_ctrl_buff_buffered - Send a buffered control message
@@ -1238,7 +1238,7 @@ static int _sdp_send_ctrl_buff_buffered(
 	return 0;
 error:
 	return result;
-} /* _sdp_send_ctrl_buff_buffered */
+}
 
 /*
  * _sdp_send_ctrl_buff - Create and Send a buffered control message
@@ -1295,7 +1295,7 @@ static int _sdp_send_ctrl_buff(struct sd
 
 error:
 	return result;
-} /* _sdp_send_ctrl_buff */
+}
 
 /*
  * _sdp_send_ctrl_disconnect - Send a disconnect request
@@ -1344,7 +1344,7 @@ static int _sdp_send_ctrl_disconnect(str
 	return 0;
 error:
 	return result;
-} /* _sdp_send_ctrl_disconnect */
+}
 
 /*
  * sdp_send_ctrl_disconnect - potentially send a disconnect request
@@ -1375,7 +1375,7 @@ int sdp_send_ctrl_disconnect(struct sdp_
 	} else
 		result = -EPROTO;
 	return result;
-} /* sdp_send_ctrl_disconnect */
+}
 
 /*
  * sdp_send_ctrl_ack - Send a gratuitous Ack
@@ -1397,7 +1397,7 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
 		return 0;
 
 	return _sdp_send_ctrl_buff(conn, SDP_MID_DATA, 0, 0);
-} /* sdp_send_ctrl_ack */
+}
 
 /*
  * sdp_send_ctrl_send_sm - Send a request for buffered mode
@@ -1405,7 +1405,7 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
 int sdp_send_ctrl_send_sm(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SEND_SM, 1, 1);
-} /* sdp_send_ctrl_send_sm */
+}
 
 /*
  * sdp_send_ctrl_src_cancel - Send a source cancel
@@ -1413,7 +1413,7 @@ int sdp_send_ctrl_send_sm(struct sdp_opt
 int sdp_send_ctrl_src_cancel(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SRC_CANCEL, 1, 1);
-} /* sdp_send_ctrl_src_cancel */
+}
 
 /*
  * sdp_send_ctrl_snk_cancel - Send a sink cancel
@@ -1421,7 +1421,7 @@ int sdp_send_ctrl_src_cancel(struct sdp_
 int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL, 1, 1);
-} /* sdp_send_ctrl_snk_cancel */
+}
 
 /*
  * sdp_send_ctrl_snk_cancel_ack - Send an ack for a sink cancel
@@ -1429,7 +1429,7 @@ int sdp_send_ctrl_snk_cancel(struct sdp_
 int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn)
 {
 	return _sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL_ACK, 1, 1);
-} /* sdp_send_ctrl_snk_cancel_ack */
+}
 
 /*
  * sdp_send_ctrl_abort - Send an abort message
@@ -1440,7 +1440,7 @@ int sdp_send_ctrl_abort(struct sdp_opt *
 	 * send
 	 */
 	return _sdp_send_ctrl_buff(conn, SDP_MID_ABORT_CONN, 1, 1);
-} /* sdp_send_ctrl_abort */
+}
 
 /*
  * sdp_send_ctrl_resize_buff_ack - Send an ack for a buffer size change
@@ -1491,7 +1491,7 @@ int sdp_send_ctrl_resize_buff_ack(struct
 
 error:
 	return result;
-} /* sdp_send_ctrl_resize_buff_ack */
+}
 
 /*
  * sdp_send_ctrl_rdma_rd - Send an rdma read completion
@@ -1560,7 +1560,7 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 
 error:
 	return result;
-} /* sdp_send_ctrl_rdma_rd */
+}
 
 /*
  * sdp_send_ctrl_rdma_wr - Send an rdma write completion
@@ -1617,7 +1617,7 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 
 error:
 	return result;
-} /* sdp_send_ctrl_rdma_wr */
+}
 
 /*
  * sdp_send_ctrl_snk_avail - Send a sink available message
@@ -1683,7 +1683,7 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	result = 0;
 error:
 	return result;
-} /* sdp_send_ctrl_snk_avail */
+}
 
 /*
  * sdp_send_ctrl_mode_ch - Send a mode change command
@@ -1768,7 +1768,7 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 
 error:
 	return result;
-} /* sdp_send_ctrl_mode_ch */
+}
 
 /*
  * GENERAL functions
@@ -1786,7 +1786,7 @@ static int _sdp_write_src_lookup(struct 
 		return 0;
 	else
 		return -ERANGE;
-} /* _sdp_write_src_lookup */
+}
 
 /*
  * _sdp_inet_write_cancel - cancel an IO operation
@@ -1928,7 +1928,7 @@ done:
 
 	aio_put_req(req); /* cancel call reference */
 	return result;
-} /* _sdp_inet_write_cancel */
+}
 
 /*
  * _sdp_send_flush_advt - Flush passive sink advertisments
@@ -1967,7 +1967,7 @@ static int _sdp_send_flush_advt(struct s
 	}
 
 	return 0;
-} /* _sdp_send_flush_advt */
+}
 
 /*
  * sdp_send_flush - Flush buffers from send queue, in to send post
@@ -2025,7 +2025,7 @@ int sdp_send_flush(struct sdp_opt *conn)
 	return 0;
 done:
 	return result;
-} /* sdp_send_flush */
+}
 
 /*
  * sdp_inet_send - send data from user space to the network
@@ -2253,4 +2253,4 @@ done:
 		send_sig(SIGPIPE, current, 0);
 
 	return result;
-} /* sdp_inet_send */
+}
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1906)
+++ sdp_actv.c	(working copy)
@@ -182,7 +182,7 @@ drop:
 
 	sdp_conn_put(conn);	/* CM sk reference */
 	return result;
-} /* _sdp_actv_conn_establish */
+}
 
 /*
  * _sdp_cm_hello_ack_check - validate the hello ack header
@@ -236,7 +236,7 @@ static int _sdp_cm_hello_ack_check(struc
 		     hello_ack->hah.l_rcv_size);
 
 	return 0; /* success */
-} /* _sdp_cm_hello_ack_check */
+}
 
 /*
  * sdp_cm_rep_handler - handler for active connection open completion
@@ -340,7 +340,7 @@ reject:
 done:
 	conn->cm_id = NULL;
 	return error;
-} /* sdp_cm_rep_handler */
+}
 
 /*
  * _sdp_cm_path_complete - path lookup complete, initiate SDP connection
@@ -537,7 +537,7 @@ done:
 	sdp_conn_put(conn);
 
 	return;
-} /* _sdp_cm_path_complete */
+}
 
 /*
  * sdp_cm_connect - initiate a SDP connection with a hello message.
@@ -588,4 +588,4 @@ int sdp_cm_connect(struct sdp_opt *conn)
 	return 0;
 error:
 	return result;
-} /* sdp_cm_connect */
+}
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1906)
+++ sdp_conn.c	(working copy)
@@ -71,7 +71,7 @@ static u32 _sdp_psn_generate(void)
         spin_unlock(&psn_lock);
 
         return psn;
-} /* sdp_psn_generate */
+}
 
 /*
  * sdp_inet_accept_q_put - put a conn into a listen conn's accept Q.
@@ -102,7 +102,7 @@ int sdp_inet_accept_q_put(struct sdp_opt
 	sdp_conn_hold(accept_conn);	/* INET reference */
 
 	return 0;
-} /* sdp_inet_accept_q_put */
+}
 
 /*
  * sdp_inet_accept_q_get - get a conn from a listen conn's accept Q.
@@ -141,7 +141,7 @@ struct sdp_opt *sdp_inet_accept_q_get(st
 	listen_conn->backlog_cnt--;
 
 	return accept_conn;
-} /* sdp_inet_accept_q_get */
+}
 
 /*
  * sdp_inet_accept_q_remove - remove a conn from a conn's accept Q.
@@ -176,7 +176,7 @@ int sdp_inet_accept_q_remove(struct sdp_
 	accept_conn->parent = NULL;
 
 	return 0;
-} /* sdp_inet_accept_q_remove */
+}
 
 /*
  * sdp_inet_listen_start - start listening for new connections on a socket
@@ -209,7 +209,7 @@ int sdp_inet_listen_start(struct sdp_opt
 
 	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
 	return 0;
-} /* sdp_inet_listen_start */
+}
 
 /*
  * sdp_inet_listen_stop - stop listening for new connections on a socket
@@ -269,7 +269,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	listen_conn->accept_prev = NULL;
 
 	return 0;
-} /* sdp_inet_listen_stop */
+}
 
 /*
  * sdp_inet_listen_lookup - lookup a connection in the listen list
@@ -295,7 +295,7 @@ struct sdp_opt *sdp_inet_listen_lookup(u
 
 	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
 	return conn;
-} /* sdp_inet_listen_lookup */
+}
 
 /*
  * sdp_inet_port_get - bind a socket to a port.
@@ -417,7 +417,7 @@ int sdp_inet_port_get(struct sdp_opt *co
 done:
 	spin_unlock_irqrestore(&_dev_root_s.bind_lock, flags);
 	return result;
-} /* sdp_inet_port_get */
+}
 
 /*
  * sdp_inet_port_put - unbind a socket from a port.
@@ -446,7 +446,7 @@ int sdp_inet_port_put(struct sdp_opt *co
 
 	spin_unlock_irqrestore(&_dev_root_s.bind_lock, flags);
 	return 0;
-} /* sdp_inet_port_put */
+}
 
 /*
  * sdp_inet_port_inherit - inherit a port from another socket (accept)
@@ -482,7 +482,7 @@ int sdp_inet_port_inherit(struct sdp_opt
 done:
 	spin_unlock_irqrestore(&_dev_root_s.bind_lock, flags);
 	return result;
-} /* sdp_inet_port_inherit */
+}
 
 /*
  * _sdp_conn_table_insert - insert a connection into the connection table
@@ -526,7 +526,7 @@ static int _sdp_conn_table_insert(struct
 	 */
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return result;
-} /* _sdp_conn_table_insert */
+}
 
 /*
  * sdp_conn_table_remove - remove a connection from the connection table
@@ -563,7 +563,7 @@ done:
 	 */
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return result;
-} /* sdp_conn_table_remove */
+}
 
 /*
  * sdp_conn_table_lookup - look up connection in the connection table
@@ -595,7 +595,7 @@ struct sdp_opt *sdp_conn_table_lookup(s3
 done:
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return conn;
-} /* sdp_conn_table_lookup */
+}
 
 /* 
  * Functions to cancel IOCB requests in a conenctions queues.
@@ -782,7 +782,7 @@ done:
 	}
 
 	return result;
-} /* _sdp_conn_destruct */
+}
 
 /*
  * sdp_conn_internal_lock - lock the connection (use only from macro)
@@ -808,7 +808,7 @@ void sdp_conn_internal_lock(struct sdp_o
 	remove_wait_queue(&(conn->lock.waitq), &wait);
 
 	return;
-} /* sdp_conn_internal_lock */
+}
 
 /*
  * sdp_conn_relock - test the connection (use only from macro)
@@ -873,7 +873,7 @@ void sdp_conn_relock(struct sdp_opt *con
 
 	spin_unlock_irqrestore(&conn->lock.slock, flags);
 	return;
-} /* sdp_conn_relock */
+}
 
 /*
  * sdp_conn_cq_drain - drain one of the the connection's CQs
@@ -925,7 +925,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 	}
 
 return calls;
-} /* sdp_conn_cq_drain */
+}
 
 /*
  * sdp_conn_internal_unlock - lock the connection (use only from macro)
@@ -947,7 +947,7 @@ void sdp_conn_internal_unlock(struct sdp
 	}
 
 	return;
-} /* sdp_conn_internal_unlock */
+}
 
 /*
  * _sdp_conn_lock_init - initialize connection lock
@@ -957,7 +957,7 @@ static void _sdp_conn_lock_init(struct s
 	spin_lock_init(&(conn->lock.slock));
 	conn->lock.users = 0;
 	init_waitqueue_head(&(conn->lock.waitq));
-} /* _sdp_conn_lock_init */
+}
 
 /*
  * sdp_conn_alloc_ib - allocate IB structures for a new connection.
@@ -1136,7 +1136,7 @@ error_param:
 	kfree(qp_attr);
 error_attr:
 	return result;
-} /* sdp_conn_alloc_ib  */
+}
 
 /*
  * sdp_conn_alloc - allocate a new socket, and init.
@@ -1330,7 +1330,7 @@ error_conn:
 error:
 	sk_free(sk);
 	return NULL;
-} /* sdp_conn_alloc  */
+}
 
 /*
  * module public functions
@@ -1439,7 +1439,7 @@ int sdp_proc_dump_conn_main(char *buffer
 done:
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return offset;
-} /* sdp_proc_dump_conn_main */
+}
 
 #define SDP_CONN_PROC_DATA_SIZE  176 /* output line size. */
 #define SDP_PROC_CONN_DATA_HEAD \
@@ -1542,7 +1542,7 @@ int sdp_proc_dump_conn_data(char *buffer
 done:
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return offset;
-} /* sdp_proc_dump_conn_data */
+}
 
 #define SDP_CONN_PROC_RDMA_SIZE   98 /* output line size. */
 #define SDP_PROC_CONN_RDMA_HEAD \
@@ -1624,7 +1624,7 @@ int sdp_proc_dump_conn_rdma(char *buffer
 done:
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return offset;
-} /* sdp_proc_dump_conn_rdma */
+}
 
 #define SDP_SOPT_PROC_DUMP_SIZE   55 /* output line size. */
 #define SDP_PROC_CONN_SOPT_HEAD \
@@ -1695,7 +1695,7 @@ int sdp_proc_dump_conn_sopt(char *buffer
 done:
 	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
 	return offset;
-} /* sdp_proc_dump_conn_sopt */
+}
 
 /*
  * sdp_proc_dump_device - dump the primary device table to /proc
@@ -1740,7 +1740,7 @@ int sdp_proc_dump_device(char *buffer,
 	}
 
 	return offset;
-} /* sdp_proc_dump_device */
+}
 
 /*
  * initialization/cleanup functions
@@ -1874,7 +1874,7 @@ error:
 		(void)ib_dealloc_pd(hca->pd);
 
 	kfree(hca);
-} /* sdp_device_init_one */
+}
 
 /*
  * sdp_device_remove_one - remove a device from the hca list
@@ -1910,7 +1910,7 @@ static void sdp_device_remove_one(struct
 		(void)ib_dealloc_pd(hca->pd);
 
 	kfree(hca);
-} /* sdp_device_remove_one */
+}
 
 /*
  * sdp_conn_table_init - create a sdp connection table
@@ -2044,7 +2044,7 @@ error_size:
 	ib_unregister_client(&sdp_client);
 error_hca:
 	return result;
-} /* sdp_conn_table_init */
+}
 
 /*
  * sdp_conn_table_clear - destroy connection managment and tables
@@ -2088,4 +2088,4 @@ int sdp_conn_table_clear(void)
 	sdp_main_iocb_cleanup();
 
 	return 0;
-} /* sdp_conn_table_clear */
+}
Index: sdp_advt.c
===================================================================
--- sdp_advt.c	(revision 1906)
+++ sdp_advt.c	(working copy)
@@ -65,7 +65,7 @@ struct sdpc_advt *sdp_advt_create(void)
 	}
 
 	return advt;
-} /* sdp_advt_create */
+}
 
 /*
  * sdp_advt_destroy - destroy an advertisment object
@@ -80,7 +80,7 @@ int sdp_advt_destroy(struct sdpc_advt *a
 	kmem_cache_free(__sdp_advt_cache, advt);
 
 	return 0;
-} /* sdp_advt_destroy */
+}
 
 /*
  * sdp_advt_q_get - get, and remove, the object at the tables head
@@ -112,7 +112,7 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 	advt->prev = NULL;
 
 	return advt;
-} /* sdp_advt_q_get */
+}
 
 /*
  * sdp_advt_q_look - get, without removing, the object at the head
@@ -120,7 +120,7 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table)
 {
 	return table->head;
-} /* sdp_advt_q_look */
+}
 
 /*
  * sdp_advt_q_put - put the advertisment object at the tables tail
@@ -146,7 +146,7 @@ void sdp_advt_q_put(struct sdpc_advt_q *
 	}
 
 	table->size++;
-} /* sdp_advt_q_put */
+}
 
 /*
  * sdp_advt_q_init - initialize a new empty advertisment table
@@ -155,7 +155,7 @@ void sdp_advt_q_init(struct sdpc_advt_q 
 {
 	table->head = NULL;
 	table->size = 0;
-} /* sdp_advt_q_init */
+}
 
 /*
  * sdp_advt_q_clear - clear the contents of an advertisment table
@@ -172,7 +172,7 @@ void sdp_advt_q_clear(struct sdpc_advt_q
 		result = sdp_advt_destroy(advt);
 		SDP_EXPECT(!(0 > result));
 	}
-} /* sdp_advt_q_clear */
+}
 
 /*
  * primary initialization/cleanup functions
@@ -200,7 +200,7 @@ int sdp_main_advt_init(void)
 		return -ENOMEM;
 
 	return 0;
-} /* sdp_main_advt_init */
+}
 
 /*
  * sdp_main_advt_cleanup - cleanup the advertisment caches.
@@ -216,4 +216,4 @@ void sdp_main_advt_cleanup(void)
 	 * null out entries.
 	 */
 	__sdp_advt_cache = NULL;
-} /* sdp_main_advt_cleanup */
+}
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1906)
+++ sdp_recv.c	(working copy)
@@ -110,7 +110,7 @@ drop:
 	conn->l_recv_bf--;
 error:
 	return result;
-} /* _sdp_post_recv_buff */
+}
 
 /*
  * _sdp_post_rdma_buff - post a single buffers for rdma read on a conn
@@ -217,7 +217,7 @@ drop:
 error:
 done:
 	return result;
-} /* _sdp_post_rdma_buff */
+}
 
 /*
  * _sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn
@@ -348,7 +348,7 @@ static int _sdp_post_rdma_iocb_src(struc
 error:
 done:
 	return result;
-} /* _sdp_post_rdma_iocb_src */
+}
 
 /*
  * _sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn
@@ -447,7 +447,7 @@ release:
 	(void)sdp_iocb_release(iocb);
 error:
 	return result;
-} /* _sdp_post_rdma_iocb_snk */
+}
 
 /*
  * _sdp_post_rdma - post a rdma based requests for a connection
@@ -526,7 +526,7 @@ static int _sdp_post_rdma(struct sdp_opt
 	result = 0;
 done:
 	return result;
-} /* _sdp_post_rdma */
+}
 
 /*
  * sdp_recv_flush - post a certain number of buffers on a connection
@@ -625,7 +625,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	result = 0;
 done:
 	return result;
-} /* sdp_recv_flush */
+}
 
 /*
  * Receive incoming data function(s)
@@ -691,7 +691,7 @@ static int _sdp_read_buff_iocb(struct sd
 	buff->tail = tail;
 
 	return 0;
-} /* _sdp_read_buff_iocb */
+}
 
 /*
  * _sdp_recv_buff_iocb_active - Ease AIO read pending pressure
@@ -740,7 +740,7 @@ static int _sdp_recv_buff_iocb_active(st
 			     result, iocb->key);
 
 	return (buff->tail - buff->data);
-} /* _sdp_recv_buff_iocb_active */
+}
 
 /*
  * _sdp_recv_buff_iocb_pending - Ease AIO read pending pressure
@@ -800,7 +800,7 @@ static int _sdp_recv_buff_iocb_pending(s
 	}
 
 	return (buff->tail - buff->data);
-} /* _sdp_recv_buff_iocb_pending */
+}
 
 /*
  * sdp_recv_buff - Process a new buffer based on queue type
@@ -892,7 +892,7 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	return buffered;
 done:
 	return result;
-} /* sdp_recv_buff */
+}
 
 /*
  * User initiated receive data function(s)
@@ -910,7 +910,7 @@ static int _sdp_read_src_lookup(struct s
 		return 0;
 	else
 		return -ERANGE;
-} /* _sdp_read_src_lookup */
+}
 
 /*
  * _sdp_inet_read_cancel - cancel an IO operation
@@ -1042,7 +1042,7 @@ unlock:
 done:
 	aio_put_req(req);
 	return result;
-} /* _sdp_inet_read_cancel */
+}
 
 /*
  * _sdp_inet_recv_urg_test - recv queue urgent data cleanup function
@@ -1050,7 +1050,7 @@ done:
 static int _sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
 {
 	return ((buff->tail == buff->head) ? 1 : 0);
-} /* _sdp_inet_recv_urg_test */
+}
 
 /*
  * _sdp_inet_recv_urg_trav - recv queue urg data retreival function
@@ -1075,7 +1075,7 @@ static int _sdp_inet_recv_urg_trav(struc
 	}
 
 	return 0;
-} /* _sdp_inet_recv_urg_trav */
+}
 
 /*
  * _sdp_inet_recv_urg - recv urgent data from the network to user space
@@ -1150,7 +1150,7 @@ static int _sdp_inet_recv_urg(struct soc
 
 done:
 	return result;
-} /* _sdp_inet_recv_urg */
+}
 
 /*
  * sdp_inet_recv - recv data from the network to user space
@@ -1268,10 +1268,10 @@ int sdp_inet_recv(struct kiocb  *req,
 							    (0 <
 							     (MSG_PEEK & flags))
 							    ? 0 : 1;
-						} /* if (OOB) */
-					} /* else (copied) */
-				} /* else (sitting on URG byte) */
-			} /* if (urgent present) */
+						}
+					}
+				}
+			}
 
 			copy = min((size_t) (size - copied), length);
 
@@ -1523,4 +1523,4 @@ done:
 
 	sdp_conn_unlock(conn);
 	return ((0 < copied) ? copied : result);
-} /* sdp_inet_recv */
+}
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1906)
+++ sdp_wall.c	(working copy)
@@ -112,7 +112,7 @@ error:
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return result;
-} /* sdp_wall_send_close */
+}
 
 /*
  * sdp_wall_send_closing - callback to confirm a passive close
@@ -164,7 +164,7 @@ error:
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return result;
-} /* sdp_wall_send_closing */
+}
 
 /*
  * sdp_wall_send_abort - callback to accept an active abort
@@ -270,7 +270,7 @@ error:
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return 0;
-} /* sdp_wall_send_abort */
+}
 
 /*
  * SDP INET (public functions)
@@ -300,7 +300,7 @@ int sdp_wall_recv_reject(struct sdp_opt 
 	sdp_conn_put(conn);	/* CM reference */
 
 	return 0;
-} /* sdp_wall_recv_reject */
+}
 
 /*
  * sdp_wall_recv_confirm - callback to confirm accepeted passive open
@@ -364,7 +364,7 @@ error:
 	sdp_conn_put(conn);	/* CM sk reference */
 	
 	return result;
-} /* sdp_wall_recv_confirm */
+}
 
 /*
  * sdp_wall_recv_failed - callback to notify accepted open of a failure
@@ -397,7 +397,7 @@ int sdp_wall_recv_failed(struct sdp_opt 
 	sdp_conn_put(conn);	/* CM reference */
 
 	return 0;
-} /* sdp_wall_recv_failed */
+}
 
 /*
  * sdp_wall_recv_close - callback to accept an active close
@@ -438,7 +438,7 @@ int sdp_wall_recv_close(struct sdp_opt *
 	}
 
 	return 0;
-} /* sdp_wall_recv_close */
+}
 
 /*
  * sdp_wall_recv_closing - callback for a close confirmation
@@ -468,7 +468,7 @@ int sdp_wall_recv_closing(struct sdp_opt
 	sk_wake_async(conn->sk, 1, POLL_HUP);
 
 	return 0;
-} /* sdp_wall_recv_closing */
+}
 
 /*
  * sdp_wall_recv_abort - abortive close notification
@@ -508,7 +508,7 @@ int sdp_wall_recv_abort(struct sdp_opt *
 	sdp_inet_wake_error(conn->sk);
 
 	return 0;
-} /* sdp_wall_recv_abort */
+}
 
 /*
  * sdp_wall_recv_drop - drop SDP protocol reference to socket
@@ -569,7 +569,7 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 	sdp_conn_put(conn);	/* CM sk reference */
 
 	return 0;
-} /* sdp_wall_recv_drop */
+}
 
 /*
  * SDP common (public functions)
@@ -594,4 +594,4 @@ int sdp_wall_abort(struct sdp_opt *conn)
 		sdp_dbg_warn(conn, "Error <%d> sending abort request", result);
 
 	return 0;
-} /* sdp_wall_abort */
+}
Index: sdp_proc.c
===================================================================
--- sdp_proc.c	(revision 1906)
+++ sdp_proc.c	(working copy)
@@ -74,7 +74,7 @@ static int _sdp_proc_read_parse(char *pa
 	}
 
 	return size;
-} /* _sdp_proc_read_parse */
+}
 
 /*
  * Static read/write functions for each proc/framework directory entry
@@ -155,7 +155,7 @@ int sdp_main_proc_cleanup(void)
 	sdp_dbg_init("/proc filesystem cleanup complete.");
 
 	return 0;
-} /* sdp_main_proc_cleanup */
+}
 
 /*
  * sdp_main_proc_init - initialize the proc filesystem entries
@@ -219,4 +219,4 @@ int sdp_main_proc_init(void)
 error:
 	(void)sdp_main_proc_cleanup();
 	return result;
-} /* sdp_main_proc_init */
+}
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1906)
+++ sdp_pass.c	(working copy)
@@ -178,7 +178,7 @@ static int _sdp_cm_accept(struct sdp_opt
 	return 0;
 error:
 	return result;
-} /* _sdp_cm_accept */
+}
 
 /*
  * _sdp_cm_listen_lookup - match a listener with an incomming conn
@@ -294,7 +294,7 @@ locked_err:
 	sdp_conn_put(listen_conn);	/* ListenLookup reference. */
 lookup_err:
 	return result;
-} /* _sdp_cm_listen_lookup */
+}
 
 /*
  * _sdp_cm_hello_check - validate the hello header
@@ -363,7 +363,7 @@ static int _sdp_cm_hello_check(struct sd
 		     msg_hello->hh.dst.ipv4.addr);
 	
 	return 0; /* success */
-} /* _sdp_cm_hello_check */
+}
 
 /*
  * sdp_cm_req_handler - handler for passive connection open completion
@@ -474,4 +474,4 @@ error:
 	sdp_conn_put(conn);	/* CM sk reference, will destory */
 done:
 	return result;
-} /* sdp_cm_req_handler */
+}
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1906)
+++ sdp_sent.c	(working copy)
@@ -113,7 +113,7 @@ static int _sdp_sent_disconnect(struct s
 	return 0;
 error:
 	return result;
-} /* _sdp_sent_disconnect */
+}
 
 static int _sdp_sent_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -131,12 +131,12 @@ static int _sdp_sent_abort(struct sdp_op
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
 
 	return result;
-} /* _sdp_sent_abort */
+}
 
 static int _sdp_sent_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
-} /* _sdp_sent_send_sm */
+}
 
 static int _sdp_sent_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -148,7 +148,7 @@ static int _sdp_sent_rdma_wr(struct sdp_
 	_sdp_msg_net_to_cpu_rwch(rwch);
 
 	return 0;
-} /* _sdp_sent_rdma_wr */
+}
 
 static int _sdp_sent_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -160,7 +160,7 @@ static int _sdp_sent_rdma_rd(struct sdp_
 	_sdp_msg_net_to_cpu_rrch(rrch);
 
 	return 0;
-} /* _sdp_sent_rdma_rd */
+}
 
 static int _sdp_sent_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -172,23 +172,23 @@ static int _sdp_sent_mode_change(struct 
 	_sdp_msg_net_to_cpu_mch(mch);
 
 	return 0;
-} /* _sdp_sent_mode_change */
+}
 
 static int _sdp_sent_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
-} /* _sdp_sent_src_cancel */
+}
 
 static int _sdp_sent_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
-} /* _sdp_sent_snk_cancel */
+}
 
 static int _sdp_sent_snk_cancel_ack(struct sdp_opt *conn,
 				    struct sdpc_buff *buff)
 {
 	return 0;
-} /* _sdp_sent_snk_cancel_ack */
+}
 
 static int _sdp_sent_resize_buff_ack(struct sdp_opt *conn,
 				     struct sdpc_buff *buff)
@@ -201,7 +201,7 @@ static int _sdp_sent_resize_buff_ack(str
 	_sdp_msg_net_to_cpu_crbah(crbah);
 
 	return 0;
-} /* _sdp_sent_resize_buff_ack */
+}
 
 static int _sdp_sent_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -213,12 +213,12 @@ static int _sdp_sent_suspend(struct sdp_
 	_sdp_msg_net_to_cpu_sch(sch);
 
 	return 0;
-} /* _sdp_sent_suspend */
+}
 
 static int _sdp_sent_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
-} /* _sdp_sent_suspend_ack */
+}
 
 static int _sdp_sent_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -230,7 +230,7 @@ static int _sdp_sent_snk_avail(struct sd
 	_sdp_msg_net_to_cpu_snkah(snkah);
 
 	return 0;
-} /* _sdp_sent_snk_avail */
+}
 
 static int _sdp_sent_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
@@ -242,7 +242,7 @@ static int _sdp_sent_src_avail(struct sd
 	_sdp_msg_net_to_cpu_srcah(srcah);
 
 	return 0;
-} /* _sdp_sent_src_avail */
+}
 
 /*
  * _sdp_sent_data - SDP data message event received
@@ -254,7 +254,7 @@ static int _sdp_sent_data(struct sdp_opt
 	conn->send_qud -= buff->data_size;
 
 	return result;
-} /* _sdp_sent_data */
+}
 
 /*
  * _sdp_sent_unsupported - Valid messages we're not sending
@@ -270,7 +270,7 @@ static int _sdp_sent_unsupported(struct 
 		     buff->bsdh_hdr->mid);
 
 	return 0;
-} /* _sdp_sent_unsupported */
+}
 
 /*
  * Event Dispatch table. For performance a dispatch table is used to avoid
@@ -470,4 +470,4 @@ drop:
 	(void)sdp_buff_pool_chain_put(head, free_count);
 done:
 	return result;
-} /* sdp_event_send */
+}
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1906)
+++ sdp_iocb.c	(working copy)
@@ -92,7 +92,7 @@ static int _sdp_iocb_unlock(struct sdpc_
 	}
 
 	return 0;
-} /* _sdp_iocb_unlock */
+}
 
 /*
  * sdp_iocb_unlock - unlock the memory for an IOCB
@@ -127,7 +127,7 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 	iocb->flags &= ~SDP_IOCB_F_LOCKED;
 
 	return result;
-} /* sdp_iocb_unlock */
+}
 
 /*
  * _sdp_iocb_page_save - save page information for an IOCB
@@ -226,7 +226,7 @@ err_page:
 err_addr:
 
 	return result;
-} /* _sdp_iocb_page_save */
+}
 
 /*
  * sdp_iocb_lock - lock the memory for an IOCB
@@ -338,7 +338,7 @@ err_lock:
 	iocb->mm = NULL;
 
 	return result;
-} /* sdp_iocb_lock */
+}
 
 /*
  * IOCB memory locking init functions
@@ -393,7 +393,7 @@ err_close:
 	filp_close(kallsyms, NULL);
 done:
 	return ret;
-} /* _sdp_mem_lock_init */
+}
 
 /*
  * _sdp_mem_lock_cleanup - cleanup the memory locking tables
@@ -407,7 +407,7 @@ static int _sdp_mem_lock_cleanup(void)
 	_mlock_ptr = NULL;
 	
 	return 0;
-} /* _sdp_mem_lock_cleanup */
+}
 
 /*
  * IOCB memory registration functions
@@ -462,7 +462,7 @@ error:
 	iocb->io_addr = 0;
 
 	return result;
-} /* sdp_iocb_register */
+}
 
 /*
  * sdp_iocb_release - unregister an IOCBs memory
@@ -482,7 +482,7 @@ int sdp_iocb_release(struct sdpc_iocb *i
 	iocb->flags &= ~(SDP_IOCB_F_REG);
 
 	return 0;
-} /* sdp_iocb_release */
+}
 
 /*
  * _sdp_iocb_complete - complete an IOCB for real in thread context
@@ -527,7 +527,7 @@ static void _sdp_iocb_complete(void *arg
 		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
 
-} /* _sdp_iocb_complete */
+}
 
 /*
  * sdp_iocb_complete - complete an IOCB
@@ -543,7 +543,7 @@ int sdp_iocb_complete(struct sdpc_iocb *
 		_sdp_iocb_complete(iocb);
 	
 	return 0;
-} /* sdp_iocb_complete */
+}
 
 /*
  * IOCB object managment
@@ -577,7 +577,7 @@ void sdp_iocb_q_remove(struct sdpc_iocb 
 	iocb->table = NULL;
 	iocb->next = NULL;
 	iocb->prev = NULL;
-} /* sdp_iocb_q_remove */
+}
 
 /*
  * sdp_iocb_q_lookup - find an iocb based on key, without removing
@@ -593,7 +593,7 @@ struct sdpc_iocb *sdp_iocb_q_lookup(stru
 			return iocb;
 
 	return NULL;
-} /* sdp_iocb_q_lookup */
+}
 
 /*
  * sdp_iocb_create - create an IOCB object
@@ -614,7 +614,7 @@ struct sdpc_iocb *sdp_iocb_create(void)
 	}
 
 	return iocb;
-} /* sdp_iocb_create */
+}
 
 /*
  * sdp_iocb_destroy - destroy an IOCB object
@@ -654,7 +654,7 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
 	kmem_cache_free(__sdp_iocb_cache, iocb);
 
 	return 0;
-} /* sdp_iocb_destroy */
+}
 
 /*
  * sdp_iocb_q_look - get, without removing, the object at the head
@@ -662,7 +662,7 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
 struct sdpc_iocb *sdp_iocb_q_look(struct sdpc_iocb_q *table)
 {
 	return table->head;
-} /* sdp_iocb_q_look */
+}
 
 /*
  * _sdp_iocb_q_get - get, and remove, the object at the tables head
@@ -699,7 +699,7 @@ static struct sdpc_iocb *_sdp_iocb_q_get
 	iocb->prev = NULL;
 
 	return iocb;
-} /* _sdp_iocb_q_get */
+}
 
 /*
  * _sdp_iocb_q_put - put the IOCB object at the tables tail
@@ -733,7 +733,7 @@ static int _sdp_iocb_q_put(struct sdpc_i
 	iocb->table = table;
 
 	return 0;
-} /* _sdp_iocb_q_put */
+}
 
 /*
  * sdp_iocb_q_get_tail - get an IOCB object from the tables tail
@@ -741,7 +741,7 @@ static int _sdp_iocb_q_put(struct sdpc_i
 struct sdpc_iocb *sdp_iocb_q_get_tail(struct sdpc_iocb_q *table)
 {
 	return _sdp_iocb_q_get(table, 0);
-} /* sdp_iocb_q_get_tail */
+}
 
 /*
  * sdp_iocb_q_get_head - get an IOCB object from the tables head
@@ -749,7 +749,7 @@ struct sdpc_iocb *sdp_iocb_q_get_tail(st
 struct sdpc_iocb *sdp_iocb_q_get_head(struct sdpc_iocb_q *table)
 {
 	return _sdp_iocb_q_get(table, 1);
-} /* sdp_iocb_q_get_head */
+}
 
 /*
  * sdp_iocb_q_put_tail - put the IOCB object at the tables tail
@@ -757,7 +757,7 @@ struct sdpc_iocb *sdp_iocb_q_get_head(st
 int sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb)
 {
 	return _sdp_iocb_q_put(table, iocb, 0);
-} /* sdp_iocb_q_put_tail */
+}
 
 /*
  * sdp_iocb_q_put_head - put the IOCB object at the tables head
@@ -765,7 +765,7 @@ int sdp_iocb_q_put_tail(struct sdpc_iocb
 int sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb)
 {
 	return _sdp_iocb_q_put(table, iocb, 1);
-} /* sdp_iocb_q_put_head */
+}
 
 /*
  * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
@@ -802,7 +802,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
 
 		iocb = next;
 	}
-} /* sdp_iocb_q_cancel */
+}
 
 /*
  * sdp_iocb_q_init - initialize a new empty IOCB table
@@ -811,7 +811,7 @@ void sdp_iocb_q_init(struct sdpc_iocb_q 
 {
 	table->head = NULL;
 	table->size = 0;
-} /* sdp_iocb_q_init */
+}
 
 /*
  * sdp_iocb_q_clear - clear the contents of an IOCB table
@@ -828,7 +828,7 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q
 		result = sdp_iocb_destroy(iocb);
 		SDP_EXPECT(!(0 > result));
 	}
-} /* sdp_iocb_q_clear */
+}
 
 /*
  * primary initialization/cleanup functions
@@ -849,7 +849,7 @@ int sdp_main_iocb_init(void)
 	if (0 > result) {
 		sdp_warn("Error <%d> initializing memory locking.", result);
 		return result;
-	} /* if */
+	}
 	/*
 	 * initialize the caches only once.
 	 */
@@ -871,7 +871,7 @@ int sdp_main_iocb_init(void)
 error_iocb_c:
 	(void)_sdp_mem_lock_cleanup();
 	return result;
-} /* sdp_main_iocb_init */
+}
 
 /*
  * sdp_main_iocb_cleanup - cleanup the advertisment caches
@@ -891,4 +891,4 @@ void sdp_main_iocb_cleanup(void)
 	 * cleanup memory locking
 	 */
 	(void)_sdp_mem_lock_cleanup();
-} /* sdp_main_iocb_cleanup */
+}
Index: sdp_iocb.h
===================================================================
--- sdp_iocb.h	(revision 1906)
+++ sdp_iocb.h	(working copy)
@@ -137,7 +137,7 @@ static inline void *__sdp_kmap(struct pa
 		return kmap_atomic(page, KM_IRQ0);
 	else
 		return kmap(page);
-} /* __sdp_kmap */
+}
 
 /*
  * __sdp_kunmap - unmap a page into kernel space
@@ -149,6 +149,6 @@ static inline void __sdp_kunmap(struct p
 	else
 		kunmap(page);
 	return;
-} /* __sdp_kunmap */
+}
 
 #endif /* _SDP_IOCB_H */
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1906)
+++ sdp_event.c	(working copy)
@@ -123,7 +123,7 @@ int sdp_cq_event_locked(struct ib_wc *co
 
 done:
 	return result;
-} /* sdp_cq_event_locked */
+}
 
 /*
  * sdp_cq_event_handler - main per QP event handler, and demuxer
@@ -192,7 +192,7 @@ unlock:
 	sdp_conn_put(conn);
 done:
 	return;
-} /* sdp_cq_event_handler */
+}
 
 /*
  * Connection establishment IB/CM callback functions
@@ -295,7 +295,7 @@ error:
 	SDP_EXPECT(!(0 > expect));
 
 	return result;
-} /* _sdp_cm_idle */
+}
 
 /*
  * _sdp_cm_established - handler for connection established completion
@@ -404,7 +404,7 @@ error:
 done:
 	conn->cm_id = NULL;
 	return result;
-} /* _sdp_cm_established */
+}
 
 /*
  * _sdp_cm_timewait - handler for connection Time Wait completion
@@ -487,7 +487,7 @@ error:
 
 	conn->cm_id = NULL;
 	return result;
-} /* _sdp_cm_timewait */
+}
 
 /*
  * Primary Connection Managment callback function
@@ -557,4 +557,4 @@ int sdp_cm_event_handler(struct ib_cm_id
 	}
 
 	return result;
-} /* sdp_cm_event_handler */
+}
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1906)
+++ sdp_buff.c	(working copy)
@@ -79,7 +79,7 @@ static inline struct sdpc_buff *_sdp_buf
 		buff = NULL;
 
 	return buff;
-} /* _sdp_buff_q_get */
+}
 
 /*
  * _sdp_buff_q_put - Place a buffer into a specific pool
@@ -111,7 +111,7 @@ static inline int _sdp_buff_q_put(struct
 	buff->pool = pool;
 
 	return 0;
-} /* _sdp_buff_q_put */
+}
 
 /*
  * _sdp_buff_q_look - look at a buffer from a specific pool
@@ -123,7 +123,7 @@ static inline struct sdpc_buff *_sdp_buf
 		return pool->head;
 	else
 		return pool->head->prev;
-} /* _sdp_buff_q_look */
+}
 
 /*
  * _sdp_buff_q_remove - remove a specific buffer from a specific pool
@@ -156,7 +156,7 @@ static inline int _sdp_buff_q_remove(str
 	buff->prev = NULL;
 
 	return 0;
-} /* _sdp_buff_q_remove */
+}
 
 /*
  * sdp_buff_q_init - Init a pool drawing its buffers from the main pool
@@ -165,7 +165,7 @@ void sdp_buff_q_init(struct sdpc_buff_q 
 {
 	pool->head = NULL;
 	pool->size = 0;
-} /* sdp_buff_q_init */
+}
 
 /*
  * sdp_buff_q_remove - remove a specific buffer from a specific pool
@@ -176,7 +176,7 @@ int sdp_buff_q_remove(struct sdpc_buff *
 
 	pool = buff->pool;
 	return _sdp_buff_q_remove(pool, buff);
-} /* sdp_buff_q_remove */
+}
 
 /*
  * sdp_buff_q_get - Get a buffer from a specific pool
@@ -184,7 +184,7 @@ int sdp_buff_q_remove(struct sdpc_buff *
 struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool)
 {
 	return _sdp_buff_q_get(pool, 1, NULL, NULL);
-} /* sdp_buff_q_get */
+}
 
 /*
  * sdp_buff_q_get_head - Get the buffer at the front of the pool
@@ -192,7 +192,7 @@ struct sdpc_buff *sdp_buff_q_get(struct 
 struct sdpc_buff *sdp_buff_q_get_head(struct sdpc_buff_q *pool)
 {
 	return _sdp_buff_q_get(pool, 1, NULL, NULL);
-} /* sdp_buff_q_get_head */
+}
 
 /*
  * sdp_buff_q_get_tail - Get the buffer at the end of the pool
@@ -200,7 +200,7 @@ struct sdpc_buff *sdp_buff_q_get_head(st
 struct sdpc_buff *sdp_buff_q_get_tail(struct sdpc_buff_q *pool)
 {
 	return _sdp_buff_q_get(pool, 0, NULL, NULL);
-} /* sdp_buff_q_get_tail */
+}
 
 /*
  * sdp_buff_q_look_head - look at the buffer at the front of the pool
@@ -208,7 +208,7 @@ struct sdpc_buff *sdp_buff_q_get_tail(st
 struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool)
 {
 	return _sdp_buff_q_look(pool, 1);
-} /* sdp_buff_q_look_head */
+}
 
 /*
  * sdp_buff_q_fetch_head - Get the pools first buffer, if the test passes
@@ -219,7 +219,7 @@ struct sdpc_buff *sdp_buff_q_fetch_head(
 					void *usr_arg)
 {
 	return _sdp_buff_q_get(pool, 1, test, usr_arg);
-} /* sdp_buff_q_fetch_head */
+}
 
 /*
  * sdp_buff_q_fetch - Get the first matching buffer from the pool
@@ -255,7 +255,7 @@ struct sdpc_buff *sdp_buff_q_fetch(struc
 		}
 
 	return NULL;
-} /* sdp_buff_q_fetch_head */
+}
 
 /*
  * sdp_buff_q_trav_head - traverse buffers in pool, from the head
@@ -285,7 +285,7 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 		}
 
 	return result;
-} /* sdp_buff_q_trav_head */
+}
 
 /*
  * sdp_buff_q_put - Place a buffer into a specific pool
@@ -294,7 +294,7 @@ int sdp_buff_q_put(struct sdpc_buff_q *p
 		   struct sdpc_buff *buff)
 {
 	return _sdp_buff_q_put(pool, buff, 1);
-} /* sdp_buff_q_put */
+}
 
 /*
  * sdp_buff_q_put_head - Place a buffer into the head of a specific pool
@@ -303,7 +303,7 @@ int sdp_buff_q_put_head(struct sdpc_buff
 			struct sdpc_buff *buff)
 {
 	return _sdp_buff_q_put(pool, buff, 1);
-} /* sdp_buff_q_put_head */
+}
 
 /*
  * sdp_buff_q_put_tail - Place a buffer into the tail of a specific pool
@@ -312,7 +312,7 @@ int sdp_buff_q_put_tail(struct sdpc_buff
 			struct sdpc_buff *buff)
 {
 	return _sdp_buff_q_put(pool, buff, 0);
-} /* sdp_buff_q_put_tail */
+}
 
 /*
  * sdp_buff_q_clear_unmap - clear the buffers out of a specific buffer pool
@@ -334,7 +334,7 @@ void sdp_buff_q_clear_unmap(struct sdpc_
 			sdp_dbg_err("Error <%d> returning buffer to main",
 				    result);
 	}
-} /* sdp_buff_q_clear_unmap */
+}
 
 /*
  * internal data buffer pool manager
@@ -362,7 +362,7 @@ static void _sdp_buff_pool_release(struc
 		free_page((unsigned long)buff->head);
 		kmem_cache_free(m_pool->buff_cache, buff);
 	}
-} /* _sdp_buff_pool_release */
+}
 
 /*
  * _sdp_buff_pool_release_check - check for buffer release from main pool
@@ -385,7 +385,7 @@ static inline void _sdp_buff_pool_releas
 
 		_sdp_buff_pool_release(m_pool, count);
 	}
-} /* _sdp_buff_pool_release_check */
+}
 
 /*
  * _sdp_buff_pool_alloc - allocate more buffers for the main pool
@@ -453,7 +453,7 @@ static int _sdp_buff_pool_alloc(struct s
 	}
 
 	return 0;
-} /* _sdp_buff_pool_alloc */
+}
 
 /*
  * sdp_buff_pool_init - Initialize the main buffer pool of memory
@@ -544,7 +544,7 @@ error_pool:
 done:
 	main_pool = NULL;
 	return result;
-} /* sdp_buff_pool_init */
+}
 
 /*
  * sdp_buff_pool_destroy - Destroy the main buffer pool and free its memory
@@ -579,7 +579,7 @@ void sdp_buff_pool_destroy(void)
 	 * done
 	 */
 	sdp_dbg_init("Main pool destroyed.");
-} /* sdp_buff_pool_destroy */
+}
 
 /*
  * sdp_buff_pool_get - Get a buffer from the main buffer pool
@@ -633,7 +633,7 @@ struct sdpc_buff *sdp_buff_pool_get(void
 	buff->wrid = 0;
 
 	return buff;
-} /* sdp_buff_pool_get */
+}
 
 /*
  * sdp_buff_pool_put - Return a buffer to the main buffer pool
@@ -675,7 +675,7 @@ int sdp_buff_pool_put(struct sdpc_buff *
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
 	return 0;
-} /* sdp_buff_pool_put */
+}
 
 /*
  * sdp_buff_pool_chain_link - create chain of buffers which can be returned
@@ -696,7 +696,7 @@ void sdp_buff_pool_chain_link(struct sdp
 		buff->next->prev = buff;
 		buff->prev->next = buff;
 	}
-} /* sdp_buff_pool_chain_link */
+}
 
 /*
  * sdp_buff_pool_chain_put - Return a buffer to the main buffer pool
@@ -737,7 +737,7 @@ int sdp_buff_pool_chain_put(struct sdpc_
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
 	return 0;
-} /* sdp_buff_pool_chain_put */
+}
 
 /*
  * sdp_buff_pool_buff_size - return the size of buffers in the main pool
@@ -752,7 +752,7 @@ int sdp_buff_pool_buff_size(void)
 		result = main_pool->buff_size;
 
 	return result;
-} /* sdp_buff_pool_buff_size */
+}
 
 /*
  * sdp_proc_dump_buff_pool - write the buffer pool stats to a file (/proc)
@@ -804,4 +804,4 @@ int sdp_proc_dump_buff_pool(char *buffer
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
 	return offset;
-} /* sdp_proc_dump_buff_pool */
+}
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1906)
+++ sdp_queue.c	(working copy)
@@ -72,7 +72,7 @@ static struct sdpc_desc *_sdp_desc_q_get
 	element->table = NULL;
 
 	return element;
-} /* _sdp_desc_q_get */
+}
 
 /*
  * _sdp_desc_q_put - Place an element into a specific table
@@ -108,7 +108,7 @@ static inline int _sdp_desc_q_put(struct
 	element->table = table;
 
 	return 0;
-} /* _sdp_desc_q_put */
+}
 
 /*
  * public advertisment object functions for FIFO object table
@@ -143,7 +143,7 @@ void sdp_desc_q_remove(struct sdpc_desc 
 	element->table = NULL;
 	element->next = NULL;
 	element->prev = NULL;
-} /* sdp_desc_q_remove */
+}
 
 /*
  * sdp_desc_q_lookup - search and return an element from the table
@@ -162,7 +162,7 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
 			return element;
 
 	return NULL;
-} /* sdp_desc_q_lookup */
+}
 
 /*
  * sdp_desc_q_get_head - Get the element at the front of the table
@@ -170,7 +170,7 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table)
 {
 	return _sdp_desc_q_get(table, 1);
-} /* sdp_desc_q_get_head */
+}
 
 /*
  * sdp_desc_q_get_tail - Get the element at the end of the table
@@ -178,7 +178,7 @@ struct sdpc_desc *sdp_desc_q_get_head(st
 struct sdpc_desc *sdp_desc_q_get_tail(struct sdpc_desc_q *table)
 {
 	return _sdp_desc_q_get(table, 0);
-} /* sdp_desc_q_get_tail */
+}
 
 /*
  * sdp_desc_q_put_head - Place an element into the head of a table
@@ -187,7 +187,7 @@ int sdp_desc_q_put_head(struct sdpc_desc
 				struct sdpc_desc *element)
 {
 	return _sdp_desc_q_put(table, element, 1);
-} /* sdp_desc_q_put_head */
+}
 
 /*
  * sdp_desc_q_put_tail - Place an element into the tail of a table
@@ -196,7 +196,7 @@ int sdp_desc_q_put_tail(struct sdpc_desc
 				struct sdpc_desc *element)
 {
 	return _sdp_desc_q_put(table, element, 0);
-} /* sdp_desc_q_put_tail */
+}
 
 /*
  * sdp_desc_q_look_head - look at the front of the table
@@ -204,7 +204,7 @@ int sdp_desc_q_put_tail(struct sdpc_desc
 struct sdpc_desc *sdp_desc_q_look_head(struct sdpc_desc_q *table)
 {
 	return table->head;
-} /* sdp_desc_q_look_head */
+}
 
 /*
  * sdp_desc_q_type_head - look at the type at the front of the table
@@ -215,7 +215,7 @@ int sdp_desc_q_type_head(struct sdpc_des
 		return SDP_DESC_TYPE_NONE;
 	else
 		return table->head->type;
-} /* sdp_desc_q_type_head */
+}
 
 /*
  * sdp_desc_q_look_type_head - look at a specific object
@@ -227,7 +227,7 @@ struct sdpc_desc *sdp_desc_q_look_type_h
 		return NULL;
 	else
 		return ((type == table->head->type) ? table->head : NULL);
-} /* sdp_desc_q_look_type_head */
+}
 
 /*
  * sdp_desc_q_look_type_tail - look at the type at the end of the table
@@ -240,7 +240,7 @@ struct sdpc_desc *sdp_desc_q_look_type_t
 	else
 		return ((type == table->head->prev->type) ?
 			table->head->prev : NULL);
-} /* sdp_desc_q_look_type_tail */
+}
 
 /*
  * sdp_desc_q_types_size - return the number of elements in the table
@@ -250,7 +250,7 @@ int sdp_desc_q_types_size(struct sdpc_de
 {
 	return ((SDP_DESC_TYPE_NONE > type) ?
 		table->count[type] : -ERANGE);
-} /* sdp_desc_q_types_size */
+}
 
 /*
  * sdp_desc_q_init - initialize a new empty generic table
@@ -261,7 +261,7 @@ void sdp_desc_q_init(struct sdpc_desc_q 
 	table->size = 0;
 
 	memset(table, 0, sizeof(struct sdpc_desc_q));
-} /* sdp_desc_q_init */
+}
 
 /*
  * sdp_desc_q_clear - clear the contents of a generic table
@@ -279,4 +279,4 @@ void sdp_desc_q_clear(struct sdpc_desc_q
 			result = element->release(element);
 			SDP_EXPECT(!(0 > result));
 		}
-} /* sdp_desc_q_clear */
+}
Index: sdp_post.c
===================================================================
--- sdp_post.c	(revision 1906)
+++ sdp_post.c	(working copy)
@@ -64,7 +64,7 @@ int sdp_cm_listen_start(struct sdev_root
 		sdp_dbg_init("Started listening for SDP connection requests");
 
 	return result;
-} /* sdp_cm_listen_start */
+}
 
 /*
  * sdp_cm_listen_stop - stop listening on all possible socket ports
@@ -82,7 +82,7 @@ int sdp_cm_listen_stop(struct sdev_root 
 	dev_root->listen_id = NULL;
 
 	return result;
-} /* sdp_cm_listen_stop */
+}
 
 /*
  * _sdp_cm_disconnect - initiate a disconnect request using the CM
@@ -108,7 +108,7 @@ static void _sdp_cm_disconnect(void *arg
 	sdp_conn_put(conn);
 
 	return;
-} /* _sdp_cm_disconnect */
+}
 
 /*
  * _sdp_cm_reject - initiate a reject request using the CM
@@ -134,7 +134,7 @@ static void _sdp_cm_reject(void *arg)
 	sdp_conn_put(conn);
 
 	return;
-} /* _sdp_cm_reject */
+}
 
 /*
  * _sdp_cm_confirm - initiate a confirm request using the CM
@@ -158,7 +158,7 @@ static void _sdp_cm_confirm(void *arg)
 	sdp_conn_put(conn);
 
 	return;
-} /* _sdp_cm_confirm */
+}
 
 /*
  * _sdp_cm_failed - initiate a Failed request using the CM
@@ -185,7 +185,7 @@ static void _sdp_cm_failed(void *arg)
 	sdp_conn_put(conn);
 
 	return;
-} /* _sdp_cm_failed */
+}
 
 /*
  * _sdp_cm_deferred_generic - initiate a defered request using the CM
@@ -205,7 +205,7 @@ static int _sdp_cm_deferred_generic(stru
 		defer_func(conn);
 
 	return 0;
-} /* _sdp_cm_deferred_generic */
+}
 
 /*
  * sdp_cm_disconnect - initiate a disconnect request using the CM
@@ -213,7 +213,7 @@ static int _sdp_cm_deferred_generic(stru
 int sdp_cm_disconnect(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_disconnect);
-} /* sdp_cm_disconnect */
+}
 
 /*
  * sdp_cm_reject - initiate a reject request using the CM
@@ -221,7 +221,7 @@ int sdp_cm_disconnect(struct sdp_opt *co
 int sdp_cm_reject(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_reject);
-} /* sdp_cm_reject */
+}
 
 /*
  * sdp_cm_confirm - initiate a confirm request using the CM
@@ -229,7 +229,7 @@ int sdp_cm_reject(struct sdp_opt *conn)
 int sdp_cm_confirm(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_confirm);
-} /* sdp_cm_confirm */
+}
 
 /*
  * sdp_cm_failed - initiate a failed request using the CM
@@ -237,4 +237,4 @@ int sdp_cm_confirm(struct sdp_opt *conn)
 int sdp_cm_failed(struct sdp_opt *conn)
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_failed);
-} /* sdp_cm_failed */
+}

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Thu Feb 24 13:00:36 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 13:00:36 -0800
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <OF0D07FEBF.448ADD5F-ON87256FB2.0063B293-00256FB2.00644178@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 11:13:04 -0700")
References: <OF0D07FEBF.448ADD5F-ON87256FB2.0063B293-00256FB2.00644178@us.ibm.com>
Message-ID: <52is4h65cb.fsf@topspin.com>

Thanks, applied except for the chunk below, since I didn't see any
reason to reorder things like this.  It doesn't move the freeing out
of a locked region or fix anything as far as I can tell.

 >	@@ -220,18 +220,23 @@ static void __path_free(struct net_devic
 >	        struct ipoib_dev_priv *priv = netdev_priv(dev);
 >	        struct ipoib_neigh *neigh, *tn;
 >	        struct sk_buff *skb;
 >	+       LIST_HEAD(ah_list);
 >	+       struct ipoib_ah *ah, *tah;
 >	 
 >	        while ((skb = __skb_dequeue(&path->queue)))
 >	                dev_kfree_skb_irq(skb);
 >	 
 >	        list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
 >	                if (neigh->ah)
 >	-                       ipoib_put_ah(neigh->ah);
 >	+                       list_add_tail(&neigh->ah->list, &ah_list);
 >	                *to_ipoib_neigh(neigh->neighbour) = NULL;
 >	                neigh->neighbour->ops->destructor = NULL;
 >	                kfree(neigh);
 >	        }
 >	 
 >	+       list_for_each_entry_safe(ah, tah, &ah_list, list)
 >	+               ipoib_put_ah(ah);
 >	+
 >	        if (path->ah)
 >	                ipoib_put_ah(path->ah);


From roland at topspin.com  Thu Feb 24 13:09:12 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 13:09:12 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <OFC047A7AB.BBA30E07-ON87256FB2.00642042-00256FB2.0064CE8A@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 11:19:06 -0700")
References: <OFC047A7AB.BBA30E07-ON87256FB2.00642042-00256FB2.0064CE8A@us.ibm.com>
Message-ID: <52ekf564xz.fsf@topspin.com>

Hmm, I don't see why we need these changes.  All of these changes seem
to be deleting items from a list right before we free the struct
containing the whole list, or setting members of a struct to NULL
right before freeing the whole struct.  So it shouldn't be necessary
or fix anything.  Am I missing a race or something?

Thanks,
  Roland


From mst at mellanox.co.il  Thu Feb 24 13:15:04 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 23:15:04 +0200
Subject: [openib-general] [PATCH] remove return at the end of void functions
Message-ID: <20050224211503.GA16942@mellanox.co.il>


Removes return at the end of void functions, and a couple of extra
empty lines too.

Applies on top of previous patch [which removed the trailing comments].

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>


diff -rup sdp_old/sdp_actv.c sdp/sdp_actv.c
--- sdp_old/sdp_actv.c	2005-02-24 22:59:38.000000000 +0200
+++ sdp/sdp_actv.c	2005-02-24 23:07:53.000000000 +0200
@@ -535,8 +535,6 @@ failed:
 done:
 	sdp_conn_unlock(conn);
 	sdp_conn_put(conn);
-
-	return;
 }
 
 /*
diff -rup sdp_old/sdp_conn.c sdp/sdp_conn.c
--- sdp_old/sdp_conn.c	2005-02-24 22:59:38.000000000 +0200
+++ sdp/sdp_conn.c	2005-02-24 23:07:53.000000000 +0200
@@ -806,8 +806,6 @@ void sdp_conn_internal_lock(struct sdp_o
 
 	current->state = TASK_RUNNING;
 	remove_wait_queue(&(conn->lock.waitq), &wait);
-
-	return;
 }
 
 /*
@@ -872,7 +870,6 @@ void sdp_conn_relock(struct sdp_opt *con
 	conn->flags &= ~SDP_CONN_F_MASK_EVENT;
 
 	spin_unlock_irqrestore(&conn->lock.slock, flags);
-	return;
 }
 
 /*
@@ -945,8 +942,6 @@ void sdp_conn_internal_unlock(struct sdp
 
 		conn->flags &= ~SDP_CONN_F_MASK_EVENT;
 	}
-
-	return;
 }
 
 /*
Only in sdp/: sdp_event.c~
diff -rup sdp_old/sdp_inet.c sdp/sdp_inet.c
--- sdp_old/sdp_inet.c	2005-02-24 22:59:38.000000000 +0200
+++ sdp/sdp_inet.c	2005-02-24 23:07:53.000000000 +0200
@@ -179,7 +179,6 @@ void sdp_inet_wake_urg(struct sock *sk)
 	 */
 	if (NULL != sk)
 		sk_send_sigurg(sk);
-
 }
 
 /*
diff -rup sdp_old/sdp_iocb.c sdp/sdp_iocb.c
--- sdp_old/sdp_iocb.c	2005-02-24 22:59:39.000000000 +0200
+++ sdp/sdp_iocb.c	2005-02-24 23:07:53.000000000 +0200
@@ -526,7 +526,6 @@ static void _sdp_iocb_complete(void *arg
 	if (0 > result)
 		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
-
 }
 
 /*
diff -rup sdp_old/sdp_iocb.h sdp/sdp_iocb.h
--- sdp_old/sdp_iocb.h	2005-02-24 22:59:39.000000000 +0200
+++ sdp/sdp_iocb.h	2005-02-24 23:07:53.000000000 +0200
@@ -148,7 +148,6 @@ static inline void __sdp_kunmap(struct p
 		kunmap_atomic(page, KM_IRQ0);
 	else
 		kunmap(page);
-	return;
 }
 
 #endif /* _SDP_IOCB_H */
diff -rup sdp_old/sdp_link.c sdp/sdp_link.c
--- sdp_old/sdp_link.c	2005-02-24 22:59:38.000000000 +0200
+++ sdp/sdp_link.c	2005-02-24 23:07:53.000000000 +0200
@@ -92,7 +92,6 @@ static void _sdp_link_path_complete(u64 
 	     arg);
 
 	info->use = jiffies;
-	return;
 }
 
 /*
@@ -108,8 +107,6 @@ static void _sdp_path_wait_add(struct sd
 
 	if (NULL != wait->next)
 		wait->next->pext = &wait->next;
-
-	return;
 }
 
 /*
@@ -131,8 +128,6 @@ static void _sdp_path_wait_destroy(struc
 	}
 
 	kmem_cache_free(_wait_cache, wait);
-
-	return;
 }
 
 /*
@@ -149,8 +144,6 @@ static void _sdp_path_wait_complete(stru
 				wait->arg);
 
 	_sdp_path_wait_destroy(wait);
-
-	return;
 }
 
 /*
@@ -222,8 +215,6 @@ static void _sdp_path_info_destroy(struc
 
 	cancel_delayed_work(&info->timer);
 	kmem_cache_free(_info_cache, info);
-
-	return;
 }
 
 /*
@@ -290,8 +281,6 @@ static void _sdp_link_path_rec_done(int 
 			_sdp_path_info_destroy(info, result);
 		}
 	}
-
-	return;
 }
 
 /*
@@ -526,7 +515,6 @@ done:
 	return;
 error:
 	_sdp_path_info_destroy(info, result);
-	return;
 }
 
 /*
@@ -633,7 +621,6 @@ static void _sdp_link_sweep(void *data)
 	}
 
 	queue_delayed_work(_link_wq, &_link_timer, SDP_LINK_SWEEP_INTERVAL);
-	return;
 }
 
 /*
@@ -687,7 +674,6 @@ static void _sdp_link_arp_work(void *dat
 done:
 	kfree_skb(skb);
 	kfree(work);
-	return;
 }
 
 /*
diff -rup sdp_old/sdp_msgs.h sdp/sdp_msgs.h
--- sdp_old/sdp_msgs.h	2005-02-24 22:59:38.000000000 +0200
+++ sdp/sdp_msgs.h	2005-02-24 23:07:53.000000000 +0200
@@ -359,7 +359,6 @@ static inline void __sdp_msg_swap_rrch(s
  */
 static inline void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
 {
-	return;
 }
 
 /*
diff -rup sdp_old/sdp_post.c sdp/sdp_post.c
--- sdp_old/sdp_post.c	2005-02-24 22:59:39.000000000 +0200
+++ sdp/sdp_post.c	2005-02-24 23:07:53.000000000 +0200
@@ -106,8 +106,6 @@ static void _sdp_cm_disconnect(void *arg
 		sdp_dbg_warn(NULL, "Error <%d> CM disconnect request", result);
 
 	sdp_conn_put(conn);
-
-	return;
 }
 
 /*
@@ -132,8 +130,6 @@ static void _sdp_cm_reject(void *arg)
 		sdp_dbg_warn(NULL, "Error <%d> CM reject request", result);
 
 	sdp_conn_put(conn);
-
-	return;
 }
 
 /*
@@ -156,8 +152,6 @@ static void _sdp_cm_confirm(void *arg)
 		sdp_dbg_warn(NULL, "Error <%d> CM confirm request", result);
 
 	sdp_conn_put(conn);
-
-	return;
 }
 
 /*
@@ -183,8 +177,6 @@ static void _sdp_cm_failed(void *arg)
 		sdp_dbg_warn(NULL, "Error <%d> CM failed request", result);
 
 	sdp_conn_put(conn);
-
-	return;
 }
 
 /*

-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Thu Feb 24 13:15:38 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 24 Feb 2005 13:15:38 -0800
Subject: [openib-general] sdp_kvec.c
In-Reply-To: <20050224205834.GA16594@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 24, 2005 at 10:58:34PM +0200
References: <20050223110243.GA23837@mellanox.co.il>
	<20050224111110.A24393@topspin.com>
	<20050224200339.GA15534@mellanox.co.il>
	<20050224121550.C24393@topspin.com>
	<20050224205834.GA16594@mellanox.co.il>
Message-ID: <20050224131538.D24393@topspin.com>

On Thu, Feb 24, 2005 at 10:58:34PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > > What do you say to removing the comments after the closing }?
> > > Its clearly unnecessary (most functions are, or shall be, short enough
> > > to see where they end), and increases the code footprint.
> > > For me, they make the detection of the function end harder rather than
> > > easier, since I am always looking for a sole }.
> > 
> >   OK, I'll stop adding them to new functions, and remove them as I
> > touch code. If I or anyone else get's ambitious we can strip them
> > from all the functions.
> 
> Here's a patch.
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

  Thanks, applied and commited.

-Libor



From jlentini at netapp.com  Thu Feb 24 13:16:28 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 24 Feb 2005 16:16:28 -0500 (EST)
Subject: [openib-general] Porting DAPL to OpenIB Gen 2
Message-ID: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>


Yesterday the DAT Collaborative voted to add the GPL license to the 
DAPL Source Forge reference implementation. We are now in a position 
to begin porting the DAPL SourceForge project to OpenIB. I could like 
to hold a conference call to help plan and divide up this work.


          Date: Monday, February 28
          Time: 11:30 AM EST
      Domestic: 888-827-8686
International: 303-928-2620
Conference ID: 1125043


James Lentini                email: jlentini at netapp.com
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300


From libor at topspin.com  Thu Feb 24 13:23:46 2005
From: libor at topspin.com (Libor Michalek)
Date: Thu, 24 Feb 2005 13:23:46 -0800
Subject: [openib-general] [PATCH] remove return at the end of void
	functions
In-Reply-To: <20050224211503.GA16942@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 24, 2005 at 11:15:04PM +0200
References: <20050224211503.GA16942@mellanox.co.il>
Message-ID: <20050224132346.E24393@topspin.com>

On Thu, Feb 24, 2005 at 11:15:04PM +0200, Michael S. Tsirkin wrote:
> 
> Removes return at the end of void functions, and a couple of extra
> empty lines too.
> 
> Applies on top of previous patch [which removed the trailing comments].
> 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

  Thanks, I've applied and commited this as well.

-Libor


From roland at topspin.com  Thu Feb 24 13:45:33 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 13:45:33 -0800
Subject: [openib-general] [PATCH]IPOIB: adding spin_lock in path_free()
In-Reply-To: <OF0B648F9F.BC5E7700-ON87256FB2.0064AFE1-00256FB2.0065FE1A@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 11:32:03 -0700")
References: <OF0B648F9F.BC5E7700-ON87256FB2.0064AFE1-00256FB2.0065FE1A@us.ibm.com>
Message-ID: <523bvl639e.fsf@topspin.com>

This patch worried me, because dropping the lock between
path_rec_start() failing and calling path_free() opens a window where
the path is in our path table with no locks held.  Also dropping the
lock halfway through path_free() made me nervous too.

I thought a little about all this and came up with the following
patch.  This moves the __path_add() out of path_rec_create() so that
it's OK to drop the lock in unicast_arp_send() before calling
path_free().  Also, it uses priv->lock to serialize the freeing of
IPoIB neighbour structs.

Do you see any problems with this?

Thanks,
  Roland

--- infiniband/ulp/ipoib/ipoib_main.c	(revision 1909)
+++ infiniband/ulp/ipoib/ipoib_main.c	(working copy)
@@ -215,16 +215,25 @@ static int __path_add(struct net_device 
 	return 0;
 }
 
-static void __path_free(struct net_device *dev, struct ipoib_path *path)
+static void path_free(struct net_device *dev, struct ipoib_path *path)
 {
 	struct ipoib_dev_priv *priv = netdev_priv(dev);
 	struct ipoib_neigh *neigh, *tn;
 	struct sk_buff *skb;
+	unsigned long flags;
 
 	while ((skb = __skb_dequeue(&path->queue)))
 		dev_kfree_skb_irq(skb);
 
+	spin_lock_irqsave(&priv->lock, flags);
+
 	list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) {
+		/*
+		 * It's safe to call ipoib_put_ah() inside priv->lock
+		 * here, because we know that path->ah will always
+		 * hold one more reference, so ipoib_put_ah() will
+		 * never do more than decrement the ref count.
+		 */
 		if (neigh->ah)
 			ipoib_put_ah(neigh->ah);
 		*to_ipoib_neigh(neigh->neighbour) = NULL;
@@ -232,11 +241,11 @@ static void __path_free(struct net_devic
 		kfree(neigh);
 	}
 
+	spin_unlock_irqrestore(&priv->lock, flags);
+
 	if (path->ah)
 		ipoib_put_ah(path->ah);
 
-	rb_erase(&path->rb_node, &priv->path_tree);
-	list_del(&path->list);
 	kfree(path);
 }
 
@@ -248,15 +257,20 @@ void ipoib_flush_paths(struct net_device
 	unsigned long flags;
 
 	spin_lock_irqsave(&priv->lock, flags);
+
 	list_splice(&priv->path_list, &remove_list);
 	INIT_LIST_HEAD(&priv->path_list);
+
+	list_for_each_entry(path, &remove_list, list)
+		rb_erase(&path->rb_node, &priv->path_tree);
+
 	spin_unlock_irqrestore(&priv->lock, flags);
 
 	list_for_each_entry_safe(path, tp, &remove_list, list) {
 		if (path->query)
 			ib_sa_cancel_query(path->query_id, path->query);
 		wait_for_completion(&path->done);
-		__path_free(dev, path);
+		path_free(dev, path);
 	}
 }
 
@@ -361,8 +375,6 @@ static struct ipoib_path *path_rec_creat
 	path->pathrec.pkey      = cpu_to_be16(priv->pkey);
 	path->pathrec.numb_path = 1;
 
-	__path_add(dev, path);
-
 	return path;
 }
 
@@ -422,6 +434,8 @@ static void neigh_add_path(struct sk_buf
 				       (union ib_gid *) (skb->dst->neighbour->ha + 4));
 		if (!path)
 			goto err;
+
+		__path_add(dev, path);
 	}
 
 	list_add_tail(&neigh->list, &path->neigh_list);
@@ -497,8 +511,12 @@ static void unicast_arp_send(struct sk_b
 			skb_push(skb, sizeof *phdr);
 			__skb_queue_tail(&path->queue, skb);
 
-			if (path_rec_start(dev, path))
-				__path_free(dev, path);
+			if (path_rec_start(dev, path)) {
+				spin_unlock(&priv->lock);
+				path_free(dev, path);
+				return;
+			} else
+				__path_add(dev, path);
 		} else {
 			++priv->stats.tx_dropped;
 			dev_kfree_skb_any(skb);
@@ -658,7 +676,7 @@ static void ipoib_set_mcast_list(struct 
 
 static void ipoib_neigh_destructor(struct neighbour *n)
 {
-	struct ipoib_neigh *neigh = *to_ipoib_neigh(n);
+	struct ipoib_neigh *neigh;
 	struct ipoib_dev_priv *priv = netdev_priv(n->dev);
 	unsigned long flags;
 	struct ipoib_ah *ah = NULL;
@@ -670,6 +688,7 @@ static void ipoib_neigh_destructor(struc
 
 	spin_lock_irqsave(&priv->lock, flags);
 
+	neigh = *to_ipoib_neigh(n);
 	if (neigh) {
 		if (neigh->ah)
 			ah = neigh->ah;



From xma at us.ibm.com  Thu Feb 24 13:47:44 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 13:47:44 -0800
Subject: [openib-general] [PATCH]deadlock problem in ipoib
In-Reply-To: <52is4h65cb.fsf@topspin.com>
Message-ID: <OF9FDC7394.F4888C4C-ON87256FB2.00764A72-88256FB2.0077BA5B@us.ibm.com>

The reason to add this piece was for the next __path_free() patch. 
Caller can't hold priv spin_lock to call __path_free() since it's has 
ipoib_put_ah() inside. I moved the spin_lock inside of __path_free(), and 
changed __path_free() to path_free().  And also the original code does't 
hold priv spin_lock calling __path_free() when bringing the interface 
down.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/5b94f8d8/attachment.html>

From mst at mellanox.co.il  Thu Feb 24 13:49:28 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 24 Feb 2005 23:49:28 +0200
Subject: [openib-general] sdp: 0 > result
In-Reply-To: <20050224132346.E24393@topspin.com>
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
Message-ID: <20050224214928.GA17236@mellanox.co.il>

OK, now what about things like these:

        if (0 > result) {
                sdp_warn("Error <%d> initializing SDP IOCB table.", result);
                goto error_iocb;
        }

may I change them all to

        if (result < 0) {

While being equivalent, we are testing the result, not 0.

Similiarly (although I feel somewhat less strongly about it)

        if (0 == result)
and
        if (NULL == conn)

are better off as

	if (!result) {
and
        if (!conn)

C is a Spartan language, and this is more brief.
Libor, I think I asked about the second one, but dont recall you
answering.
If OK to both, let me know and I'll do it on Sunday.

-- 
MST - Michael S. Tsirkin


From mshefty at ichips.intel.com  Thu Feb 24 13:55:20 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 24 Feb 2005 13:55:20 -0800
Subject: [openib-general] [PATCH] [MAD] change to MAD layer callback when
	canceling a MAD
Message-ID: <20050224135520.25d3769a.mshefty@ichips.intel.com>

This patch modifies ib_cancel_mad() to invoke a user's send callback
from a different thread context than that used by the caller.  This
allows a caller to hold a lock while calling cancel that is also
acquired from their send handler.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>


Index: infiniband/core/mad.c
===================================================================
--- infiniband/core/mad.c	(revision 1905)
+++ infiniband/core/mad.c	(working copy)
@@ -68,6 +68,7 @@ static void cancel_mads(struct ib_mad_ag
 static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
 				    struct ib_mad_send_wc *mad_send_wc);
 static void timeout_sends(void *data);
+static void cancel_sends(void *data);
 static void local_completions(void *data);
 static int solicited_mad(struct ib_mad *mad);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
@@ -341,6 +342,8 @@ struct ib_mad_agent *ib_register_mad_age
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
 		   mad_agent_priv);
+	INIT_LIST_HEAD(&mad_agent_priv->canceled_list);
+	INIT_WORK(&mad_agent_priv->canceled_work, cancel_sends, mad_agent_priv);
 	atomic_set(&mad_agent_priv->refcount, 1);
 	init_waitqueue_head(&mad_agent_priv->wait);
 
@@ -2004,12 +2007,44 @@ find_send_by_wr_id(struct ib_mad_agent_p
 	return NULL;
 }
 
+void cancel_sends(void *data)
+{
+	struct ib_mad_agent_private *mad_agent_priv;
+	struct ib_mad_send_wr_private *mad_send_wr;
+	struct ib_mad_send_wc mad_send_wc;
+	unsigned long flags;
+
+	mad_agent_priv = (struct ib_mad_agent_private *)data;
+
+	mad_send_wc.status = IB_WC_WR_FLUSH_ERR;
+	mad_send_wc.vendor_err = 0;
+
+	spin_lock_irqsave(&mad_agent_priv->lock, flags);
+	while (!list_empty(&mad_agent_priv->canceled_list)) {
+		mad_send_wr = list_entry(mad_agent_priv->canceled_list.next,
+					 struct ib_mad_send_wr_private,
+					 agent_list);
+
+		list_del(&mad_send_wr->agent_list);
+		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
+
+		mad_send_wc.wr_id = mad_send_wr->wr_id;
+		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
+						   &mad_send_wc);
+
+		kfree(mad_send_wr);
+		if (atomic_dec_and_test(&mad_agent_priv->refcount))
+			wake_up(&mad_agent_priv->wait);
+		spin_lock_irqsave(&mad_agent_priv->lock, flags);
+	}
+	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
+}
+
 void ib_cancel_mad(struct ib_mad_agent *mad_agent,
 		  u64 wr_id)
 {
 	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *mad_send_wr;
-	struct ib_mad_send_wc mad_send_wc;
 	unsigned long flags;
 
 	mad_agent_priv = container_of(mad_agent, struct ib_mad_agent_private,
@@ -2031,19 +2066,12 @@ void ib_cancel_mad(struct ib_mad_agent *
 	}
 
 	list_del(&mad_send_wr->agent_list);
+	list_add_tail(&mad_send_wr->agent_list, &mad_agent_priv->canceled_list);
 	adjust_timeout(mad_agent_priv);
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
-	mad_send_wc.status = IB_WC_WR_FLUSH_ERR;
-	mad_send_wc.vendor_err = 0;
-	mad_send_wc.wr_id = mad_send_wr->wr_id;
-	mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
-					   &mad_send_wc);
-
-	kfree(mad_send_wr);
-	if (atomic_dec_and_test(&mad_agent_priv->refcount))
-		wake_up(&mad_agent_priv->wait);
-
+	queue_work(mad_agent_priv->qp_info->port_priv->wq,
+		   &mad_agent_priv->canceled_work);
 out:
 	return;
 }
Index: infiniband/core/mad_priv.h
===================================================================
--- infiniband/core/mad_priv.h	(revision 1905)
+++ infiniband/core/mad_priv.h	(working copy)
@@ -95,6 +95,8 @@ struct ib_mad_agent_private {
 	unsigned long timeout;
 	struct list_head local_list;
 	struct work_struct local_work;
+	struct list_head canceled_list;
+	struct work_struct canceled_work;
 
 	atomic_t refcount;
 	wait_queue_head_t wait;


From mst at mellanox.co.il  Thu Feb 24 14:12:50 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 00:12:50 +0200
Subject: [openib-general] Re: sdp_kvec.c
In-Reply-To: <20050224131538.D24393@topspin.com>
References: <20050223110243.GA23837@mellanox.co.il>
	<20050224111110.A24393@topspin.com>
	<20050224200339.GA15534@mellanox.co.il>
	<20050224121550.C24393@topspin.com>
	<20050224205834.GA16594@mellanox.co.il>
	<20050224131538.D24393@topspin.com>
Message-ID: <20050224221250.GA17573@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: sdp_kvec.c
> 
> On Thu, Feb 24, 2005 at 10:58:34PM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > What do you say to removing the comments after the closing }?
> > > > Its clearly unnecessary (most functions are, or shall be, short enough
> > > > to see where they end), and increases the code footprint.
> > > > For me, they make the detection of the function end harder rather than
> > > > easier, since I am always looking for a sole }.
> > > 
> > >   OK, I'll stop adding them to new functions, and remove them as I
> > > touch code. If I or anyone else get's ambitious we can strip them
> > > from all the functions.


Here's some more - I missed the ones after the structure end.
Note how at least in one case (_sdp_prot) the closing comment said
_pf_family. Which shows that these clean-ups are a good thing.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_queue.h
===================================================================
--- sdp_queue.h	(revision 1909)
+++ sdp_queue.h	(working copy)
@@ -55,7 +55,7 @@ struct sdpc_desc {
 	u32                 type;  /* element type. (for generic queue) */
 	struct sdpc_desc_q *table; /* table to which this object belongs */
 	int (*release)(struct sdpc_desc *element); /* release the object */
-}; /* struct sdpc_desc */
+};
 
 /*
  * table for holding SDP advertisments.
@@ -64,7 +64,7 @@ struct sdpc_desc_q {
 	struct sdpc_desc *head; /* double linked list of advertisments */
 	int size;               /* current number of advertisments in table */
 	u16 count[SDP_DESC_TYPE_NONE]; /* object specific counter */
-}; /* struct sdpc_desc_q */
+};
 
 /*
  * SDP generic queue inline functions
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1909)
+++ sdp_rcvd.c	(working copy)
@@ -1084,7 +1084,7 @@ static sdp_event_cb_func recv_event_func
 	_sdp_rcvd_snk_avail,        /* SDP_MID_SNK_AVAIL        0xFD */
 	_sdp_rcvd_src_avail,        /* SDP_MID_SRC_AVAIL        0xFE */
 	_sdp_rcvd_data              /* SDP_MID_DATA             0xFF */
-}; /* recv_event_funcs */
+};
 
 /*
  * sdp_event_recv - recv event demultiplexing into sdp messages
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1909)
+++ sdp_inet.c	(working copy)
@@ -1403,7 +1403,7 @@ struct proto_ops _lnx_stream_ops = {
 	.socketpair =  sock_no_socketpair,
 	.mmap       =  sock_no_mmap,
 	.owner      =  THIS_MODULE,
-}; /* _lnx_stream_ops */
+};
 
 /*
  * _sdp_inet_create - create a socket
@@ -1455,7 +1455,7 @@ static struct net_proto_family _sdp_prot
 	.family =  AF_INET_SDP,
 	.create = _sdp_inet_create,
 	.owner  =  THIS_MODULE,
-}; /* _pf_family */
+};
 
 /*
  * SDP host module load/unload functions
Index: sdp_link.h
===================================================================
--- sdp_link.h	(revision 1909)
+++ sdp_link.h	(working copy)
@@ -64,7 +64,7 @@ struct sdp_path_info {
 	struct sdp_path_info **pext; /* previous next element in path list */
 
 	struct sdp_path_wait  *wait_list; /* waiting for completions */
-}; /* struct sdp_path_info */
+};
 
 struct sdp_path_wait {
 	u64 id;  /* request identifier */
@@ -80,12 +80,12 @@ struct sdp_path_wait {
 	int   retry;
 	struct sdp_path_wait  *next;
 	struct sdp_path_wait **pext;
-}; /* struct sdp_path_wait */
+};
 
 struct sdp_work {
 	struct work_struct work;
 	void *arg;
-}; /* struct sdp_work */
+};
 
 struct sdp_link_arp {
 	/*
Index: sdp_conn.h
===================================================================
--- sdp_conn.h	(revision 1909)
+++ sdp_conn.h	(working copy)
@@ -171,7 +171,7 @@ struct sdp_conn_state {
 	__u16 state[SDP_CONN_STATE_MAX];
 	void *file[SDP_CONN_STATE_MAX];
 	__s32 line[SDP_CONN_STATE_MAX];
-}; /* struct sdp_conn_state */
+};
 
 #define SDP_CONN_ST_SET(conn, val) \
 do { \
@@ -208,7 +208,7 @@ struct sdp_conn_lock {
 	__u32 users;
 	spinlock_t slock;
 	wait_queue_head_t waitq;
-}; /* struct sdp_conn_lock */
+};
 
 /*
  * SDP Connection structure.
@@ -412,7 +412,7 @@ struct sdp_opt {
 #ifdef _SDP_CONN_STATE_REC
 	struct sdp_conn_state state_rec;
 #endif
-}; /* struct sdp_opt */
+};
 
 #define SDP_WRAP_GT(x, y) ((signed int)((x) - (y)) > 0)
 #define SDP_WRAP_LT(x, y) ((signed int)((x) - (y)) < 0)
Index: sdp_advt.h
===================================================================
--- sdp_advt.h	(revision 1909)
+++ sdp_advt.h	(working copy)
@@ -60,7 +60,7 @@ struct sdpc_advt {
 	u64 wrid; /* work request completing this advertisment */
 	u32 flag; /* advertisment flags. */
 	u64 addr; /* advertised buffer virtual address */
-}; /* struct sdpc_advt */
+};
 
 /*
  * table for holding SDP advertisments.
@@ -68,7 +68,7 @@ struct sdpc_advt {
 struct sdpc_advt_q {
 	struct sdpc_advt *head; /* double linked list of advertisments */
 	s32 size;               /* current number of advertisments in table */
-}; /* struct sdpc_advt_q */
+};
 
 /*
  * make size a macro.
Index: sdp_buff_p.h
===================================================================
--- sdp_buff_p.h	(revision 1909)
+++ sdp_buff_p.h	(working copy)
@@ -71,6 +71,6 @@ struct sdpc_buff_root {
 
 	int alloc_inc;  /* allocation increment */
 	int free_mark;  /* start freeing unused buffers */
-}; /* struct sdpc_buff_root */
+};
 
 #endif /* _SDP_BUFF_P_H */
Index: sdp_proc.h
===================================================================
--- sdp_proc.h	(revision 1909)
+++ sdp_proc.h	(working copy)
@@ -65,6 +65,6 @@ struct sdpc_proc_ent {
 		    int   max_size,
 		    off_t start, 
 		    long *end);
-}; /* struct sdpc_proc_ent */
+};
 
 #endif /* _SDP_PROC_H */
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1909)
+++ sdp_sent.c	(working copy)
@@ -316,7 +316,7 @@ static sdp_event_cb_func send_event_func
 	_sdp_sent_snk_avail,       /* SDP_MID_SNK_AVAIL        0xFD */
 	_sdp_sent_src_avail,       /* SDP_MID_SRC_AVAIL        0xFE */
 	_sdp_sent_data             /* SDP_MID_DATA             0xFF */
-}; /* send_event_funcs */
+};
 
 /*
  * sdp_event_send - send event handler
Index: sdp_iocb.h
===================================================================
--- sdp_iocb.h	(revision 1909)
+++ sdp_iocb.h	(working copy)
@@ -114,7 +114,7 @@ struct sdpc_iocb {
 	 */
 	struct kiocb *req;
 	struct sock_iocb *si;
-}; /* struct sdpc_iocb */
+};
 
 /*
  * table for IOCBs
@@ -122,7 +122,7 @@ struct sdpc_iocb {
 struct sdpc_iocb_q {
 	struct sdpc_iocb *head; /* double linked list of IOCBs */
 	int size;               /* current number of IOCBs in table */
-}; /* struct sdpc_iocb_q */
+};
 
 /*
  * Address translations
Index: sdp_dev.h
===================================================================
--- sdp_dev.h	(revision 1909)
+++ sdp_dev.h	(working copy)
@@ -143,7 +143,7 @@ struct sdev_hca_port {
 	u8                    index; /* port ID */
 	union ib_gid          gid;   /* port GID */
 	struct sdev_hca_port *next;  /* next port in the list */
-}; /* struct sdev_hca_port */
+};
 
 struct sdev_hca {
 	struct ib_device     *ca;        /* HCA */
@@ -154,7 +154,7 @@ struct sdev_hca {
 	struct ib_fmr_pool   *fmr_pool;  /* fast memory for Zcopy */
 	struct sdev_hca_port *port_list; /* ports on this HCA */
 	struct sdev_hca      *next;      /* next HCA in the list */
-}; /* struct sdev_hca */
+};
 
 struct sdev_root {
 	u32 src_addr;
@@ -196,6 +196,6 @@ struct sdev_root {
 	 */
 	kmem_cache_t *conn_cache;
 	kmem_cache_t *sock_cache;
-}; /* struct sdev_root */
+};
 
 #endif /* _SDP_DEV_H */
Index: sdp_buff.h
===================================================================
--- sdp_buff.h	(revision 1909)
+++ sdp_buff.h	(working copy)
@@ -42,7 +42,7 @@
 struct sdpc_buff_q {
 	struct sdpc_buff *head; /* double linked list of buffers */
 	u32 size;               /* number of buffers in the pool */
-}; /* struct sdpc_buff_q */
+};
 
 struct sdpc_buff {
 	struct sdpc_buff   *next;
@@ -74,7 +74,7 @@ struct sdpc_buff {
 	u64 real; /* component of scather/gather list (address) */
 	u32 size; /* component of scather/gather list (lenght)  */
 	u32 lkey; /* component of scather/gather list (key) */
-}; /* struct sdpc_buff */
+};
 
 /*
  * buffer flag defintions
-- 
MST - Michael S. Tsirkin


From xma at us.ibm.com  Thu Feb 24 14:21:08 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 14:21:08 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <52ekf564xz.fsf@topspin.com>
Message-ID: <OF796F9F5B.B31AC28F-ON87256FB2.007A1672-88256FB2.007AC907@us.ibm.com>

I did see one race during stress test on SMP, when going through 
neigh_list, neigh->ah revisited a freed memory. After applying this patch 
the problem was gone. I need to go through my test to see where is the 
race.

thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/e649df66/attachment.html>

From roland at topspin.com  Thu Feb 24 14:28:08 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 14:28:08 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <OF796F9F5B.B31AC28F-ON87256FB2.007A1672-88256FB2.007AC907@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 14:21:08 -0800")
References: <OF796F9F5B.B31AC28F-ON87256FB2.007A1672-88256FB2.007AC907@us.ibm.com>
Message-ID: <52u0o14mpz.fsf@topspin.com>

    Shirley> I did see one race during stress test on SMP, when going
    Shirley> through neigh_list, neigh->ah revisited a freed
    Shirley> memory. After applying this patch the problem was gone. I
    Shirley> need to go through my test to see where is the race.

What stress test was this?  I'd be curious to be able to run it myself.

Thanks,
  Roland


From mshefty at ichips.intel.com  Thu Feb 24 14:45:44 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 24 Feb 2005 14:45:44 -0800
Subject: [openib-general] [PATCH] [CM] synchronize canceling a MAD with
	completion processing
Message-ID: <20050224144544.411d51ba.mshefty@ichips.intel.com>

This patch synchronizes the canceling of a MAD by the CM with the
processing of completions.  The changes have not been committed as
they depend on the MAD layer patch.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>


Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1905)
+++ infiniband/core/cm.c	(working copy)
@@ -628,7 +628,6 @@ int ib_destroy_cm_id(struct ib_cm_id *cm
 	struct cm_id_private *cm_id_priv;
 	struct cm_work *work;
 	unsigned long flags;
-	u64 wr_id;
 
 	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
 retest:
@@ -643,9 +642,9 @@ retest:
 		break;
 	case IB_CM_SIDR_REQ_SENT:
 		cm_id->state = IB_CM_IDLE;
-		wr_id = (unsigned long) cm_id_priv->msg;
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-		ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
 		break;
 	case IB_CM_SIDR_REQ_RCVD:
 		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
@@ -1327,7 +1326,6 @@ static int cm_rep_handler(struct cm_work
 	struct cm_timewait_info *timewait_info;
 	struct cm_rep_msg *rep_msg;
 	unsigned long flags;
-	u64 wr_id;
 	int ret;
 
 	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
@@ -1384,13 +1382,13 @@ static int cm_rep_handler(struct cm_work
 
 	/* todo: handle peer_to_peer */
 
-	wr_id = (unsigned long) cm_id_priv->msg;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
 	if (!ret)
 		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
 	if (ret)
 		cm_process_work(cm_id_priv, work);
 	else
@@ -1408,7 +1406,6 @@ static int cm_establish_handler(struct c
 {
 	struct cm_id_private *cm_id_priv;
 	unsigned long flags;
-	u64 wr_id;
 	int ret;
 
 	/* See comment in ib_cm_establish about lookup. */
@@ -1422,13 +1419,13 @@ static int cm_establish_handler(struct c
 		goto out;
 	}
 
-	wr_id = (unsigned long) cm_id_priv->msg;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
 	if (!ret)
 		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
 	if (ret)
 		cm_process_work(cm_id_priv, work);
 	else
@@ -1444,7 +1441,6 @@ static int cm_rtu_handler(struct cm_work
 	struct cm_id_private *cm_id_priv;
 	struct cm_rtu_msg *rtu_msg;
 	unsigned long flags;
-	u64 wr_id;
 	int ret;
 
 	rtu_msg = (struct cm_rtu_msg *)work->mad_recv_wc->recv_buf.mad;
@@ -1463,13 +1459,13 @@ static int cm_rtu_handler(struct cm_work
 	}
 	cm_id_priv->id.state = IB_CM_ESTABLISHED;
 
-	wr_id = (unsigned long) cm_id_priv->msg;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
 	if (!ret)
 		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
 	if (ret)
 		cm_process_work(cm_id_priv, work);
 	else
@@ -1664,7 +1660,6 @@ static int cm_drep_handler(struct cm_wor
 	struct cm_id_private *cm_id_priv;
 	struct cm_drep_msg *drep_msg;
 	unsigned long flags;
-	u64 wr_id;
 	int ret;
 
 	drep_msg = (struct cm_drep_msg *)work->mad_recv_wc->recv_buf.mad;
@@ -1683,14 +1678,13 @@ static int cm_drep_handler(struct cm_wor
 	}
 	cm_enter_timewait(cm_id_priv);
 
-	wr_id = (unsigned long) cm_id_priv->msg;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
 	if (!ret)
 		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
-
 	if (ret)
 		cm_process_work(cm_id_priv, work);
 	else
@@ -1842,7 +1836,6 @@ static int cm_rej_handler(struct cm_work
 	struct cm_id_private *cm_id_priv;
 	struct cm_rej_msg *rej_msg;
 	unsigned long flags;
-	u64 wr_id = 0;
 	int ret;
 
 	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
@@ -1856,7 +1849,8 @@ static int cm_rej_handler(struct cm_work
 	switch (cm_id_priv->id.state) {
 	case IB_CM_REQ_SENT:
 	case IB_CM_REP_SENT:
-		wr_id = (unsigned long) cm_id_priv->msg;
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
 		/* fall through */
 	case IB_CM_REQ_RCVD:
 	case IB_CM_MRA_REQ_SENT:
@@ -1868,7 +1862,8 @@ static int cm_rej_handler(struct cm_work
 			cm_reset_to_idle(cm_id_priv);
 		break;
 	case IB_CM_DREQ_SENT:
-		wr_id = (unsigned long) cm_id_priv->msg;
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
 		/* fall through */
 	case IB_CM_REP_RCVD:
 	case IB_CM_MRA_REP_SENT:
@@ -1886,8 +1881,6 @@ static int cm_rej_handler(struct cm_work
 		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	if (wr_id)
-		ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
 	if (ret)
 		cm_process_work(cm_id_priv, work);
 	else
@@ -2222,7 +2215,6 @@ static int cm_apr_handler(struct cm_work
 	struct cm_id_private *cm_id_priv;
 	struct cm_apr_msg *apr_msg;
 	unsigned long flags;
-	u64 wr_id;
 	int ret;
 
 	apr_msg = (struct cm_apr_msg *)work->mad_recv_wc->recv_buf.mad;
@@ -2244,7 +2236,8 @@ static int cm_apr_handler(struct cm_work
 		goto out;
 	}
 	cm_id_priv->id.lap_state = IB_CM_LAP_IDLE;
-	wr_id = (unsigned long) cm_id_priv->msg;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
 	cm_id_priv->msg = NULL;
 
 	ret = atomic_inc_and_test(&cm_id_priv->work_count);
@@ -2252,8 +2245,6 @@ static int cm_apr_handler(struct cm_work
 		list_add_tail(&work->list, &cm_id_priv->work_list);
 	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
 
-	ib_cancel_mad(cm_id_priv->av.port->mad_agent, wr_id);
-
 	if (ret)
 		cm_process_work(cm_id_priv, work);
 	else
@@ -2546,10 +2537,10 @@ static int cm_sidr_rep_handler(struct cm
 		goto out;
 	}
 	cm_id_priv->id.state = IB_CM_IDLE;
-	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
-
 	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
 		      (unsigned long) cm_id_priv->msg);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
 	cm_format_sidr_rep_event(work);
 	cm_process_work(cm_id_priv, work);
 	return 0;


From roland at topspin.com  Thu Feb 24 15:05:32 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 15:05:32 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <OF796F9F5B.B31AC28F-ON87256FB2.007A1672-88256FB2.007AC907@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 14:21:08 -0800")
References: <OF796F9F5B.B31AC28F-ON87256FB2.007A1672-88256FB2.007AC907@us.ibm.com>
Message-ID: <52k6ox4kzn.fsf@topspin.com>

    Shirley> I did see one race during stress test on SMP, when going
    Shirley> through neigh_list, neigh->ah revisited a freed
    Shirley> memory. After applying this patch the problem was gone. I
    Shirley> need to go through my test to see where is the race.

By the way, do you still have the oops stack trace for this?

The proposed patch I posted for IPoIB did fix one potential problem:
ipoib_neigh_destructor() looked at *to_ipoib_neigh(n) outside of
priv->lock, which could race against other freeing of the neigh.  So
my patch may also fix this problem.

If we apply the patch in this thread, I'd be afraid that we're just
making a race window smaller, since I don't see what it's fixing.

 - R.


From xma at us.ibm.com  Thu Feb 24 15:15:02 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 15:15:02 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <52u0o14mpz.fsf@topspin.com>
Message-ID: <OF18ADBC56.C29B563D-ON87256FB2.007EE390-88256FB2.007FB878@us.ibm.com>

> What stress test was this?

To hit this problem I just run multiple netperf TCP_STEAM on an ia64 SMP 
platform. Let me revalidate this problem against the new bits.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/b75bfc46/attachment.html>

From roland at topspin.com  Thu Feb 24 15:23:32 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 15:23:32 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <OF18ADBC56.C29B563D-ON87256FB2.007EE390-88256FB2.007FB878@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 15:15:02 -0800")
References: <OF18ADBC56.C29B563D-ON87256FB2.007EE390-88256FB2.007FB878@us.ibm.com>
Message-ID: <52fyzl4k5n.fsf@topspin.com>

    Shirley> To hit this problem I just run multiple netperf TCP_STEAM
    Shirley> on an ia64 SMP platform. Let me revalidate this problem
    Shirley> against the new bits.

Thanks.  Make sure you apply the patch I posted on top of the latest
svn tree, since I haven't committed it yet.  I still need to do some
testing at least, and I'd like to make sure it passes your tests as
well.

Thanks,
  Roland


From xma at us.ibm.com  Thu Feb 24 16:10:59 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 16:10:59 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <52fyzl4k5n.fsf@topspin.com>
Message-ID: <OF17525D1F.36E30689-ON87256FB3.0000A894-88256FB3.000101BC@us.ibm.com>

I've tested the most recent bit + patch for path_free() . IPoIB works 
happy. The kernel oops is gone.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/29d5335b/attachment.html>

From tduffy at sun.com  Thu Feb 24 17:09:27 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 24 Feb 2005 17:09:27 -0800
Subject: [openib-general] Porting DAPL to OpenIB Gen 2
In-Reply-To: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
Message-ID: <1109293767.12995.24.camel@duffman>

On Thu, 2005-02-24 at 16:16 -0500, James Lentini wrote:
> Yesterday the DAT Collaborative voted to add the GPL license to the 
> DAPL Source Forge reference implementation.

That is great news.  When is this effective?

> We are now in a position 
> to begin porting the DAPL SourceForge project to OpenIB. I could like 
> to hold a conference call to help plan and divide up this work.
> 
> 
>           Date: Monday, February 28
>           Time: 11:30 AM EST
>       Domestic: 888-827-8686
> International: 303-928-2620
> Conference ID: 1125043

The chances of getting us left-coasters in the office by 8:30AM on
Monday morning seems unlikely.  I will try, but you may not get
everybody.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/ad325e2f/attachment.sig>

From roland at topspin.com  Thu Feb 24 19:32:15 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 19:32:15 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the list
	before	kfree(neigh)
In-Reply-To: <OF17525D1F.36E30689-ON87256FB3.0000A894-88256FB3.000101BC@us.ibm.com>
	(Shirley Ma's message of "Thu, 24 Feb 2005 16:10:59 -0800")
References: <OF17525D1F.36E30689-ON87256FB3.0000A894-88256FB3.000101BC@us.ibm.com>
Message-ID: <527jkx48n4.fsf@topspin.com>

    Shirley> I've tested the most recent bit + patch for path_free()
    Shirley> . IPoIB works happy. The kernel oops is gone.

Great, thanks for testing.  I've committed this last patch.

 - Roland


From xma at us.ibm.com  Thu Feb 24 19:32:51 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Thu, 24 Feb 2005 19:32:51 -0800
Subject: [openib-general] [PATCH]IPOIB: delete neigh from the
	list	before	kfree(neigh)
In-Reply-To: <OF17525D1F.36E30689-ON87256FB3.0000A894-88256FB3.000101BC@us.ibm.com>
Message-ID: <OF5D56FF7C.6A6EDBFC-ON87256FB3.0013255C-88256FB3.00137CEE@us.ibm.com>





path->ah = NULL  & mcast->ah = NULL are needed after ipoib_put_ah() in this
patch. I will create a new one.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050224/64aec0d8/attachment.html>

From roland at topspin.com  Thu Feb 24 19:34:00 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 19:34:00 -0800
Subject: [openib-general] [PATCH] (repost) no qp lock on poll, separate
	sq/rq locks
In-Reply-To: <20050224085054.GO23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 24 Feb 2005 10:50:54 +0200")
References: <20050224085054.GO23837@mellanox.co.il>
Message-ID: <523bvl48k7.fsf@topspin.com>

thanks, applied.

 - R.


From roland at topspin.com  Thu Feb 24 19:37:58 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 19:37:58 -0800
Subject: [openib-general] [PATCH] (repost) initialize device_cap_flags
	in mthca
In-Reply-To: <20050224104218.GP23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 24 Feb 2005 12:42:18 +0200")
References: <20050224104218.GP23837@mellanox.co.il>
Message-ID: <52y8dd2tt5.fsf@topspin.com>

Thanks, applied.

 - R.


From roland at topspin.com  Thu Feb 24 20:39:43 2005
From: roland at topspin.com (Roland Dreier)
Date: Thu, 24 Feb 2005 20:39:43 -0800
Subject: [openib-general] [PATCH] (repost) no qp lock on poll, separate
	sq/rq locks
In-Reply-To: <20050224085054.GO23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 24 Feb 2005 10:50:54 +0200")
References: <20050224085054.GO23837@mellanox.co.il>
Message-ID: <52r7j52qy8.fsf@topspin.com>

Ugh, I think I missed something when I thought about this the first
time around.  It seems the test for WQ overflow assumes that all WQs
have a power-of-2 size, which we currently don't enforce for Tavor
mode.  It seems there are two possible solutions:

  Round up WQ sizes for Tavor as well.  I don't like this because it
  could potentially use a lot of extra memory.

  Or, add one more counter back into the WQ struct so we can keep
  track of both the next index to use as well as the total number of
  WQEs posted in Tavor mode (we still only need one counter in
  mem-free mode).

I implemented the second option.  Does this patch look reasonable?

 - R.

Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 1912)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -167,15 +167,16 @@ struct mthca_cq {
 
 struct mthca_wq {
 	spinlock_t lock;
-	int   max;
-	int   next;
-	int   last_comp;
-	void *last;
-	int   max_gs;
-	int   wqe_shift;
+	int        max;
+	unsigned   next_ind;
+	unsigned   head;
+	unsigned   tail;
+	void      *last;
+	int        max_gs;
+	int        wqe_shift;
 
-	int   db_index;		/* Arbel only */
-	u32  *db;
+	int        db_index;	/* Arbel only */
+	u32       *db;
 };
 
 struct mthca_qp {
Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 1912)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -453,7 +453,7 @@ static inline int mthca_poll_one(struct 
 		entry->wr_id = (*cur_qp)->wrid[wqe_index];
 	}
 
-	wq->last_comp = wqe_index;
+	++wq->tail;
 
 	if (0)
 		mthca_dbg(dev, "%s completion for QP %06x, index %d (nr %d)\n",
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 1912)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -1081,8 +1081,9 @@ static void mthca_free_memfree(struct mt
 static void mthca_wq_init(struct mthca_wq* wq)
 {
 	spin_lock_init(&wq->lock);
-	wq->next      = 0;
-	wq->last_comp = wq->max - 1;
+	wq->next_ind  = 0;
+	wq->head      = 0;
+	wq->tail      = 0;
 	wq->last      = NULL;
 }
 
@@ -1397,19 +1398,19 @@ static int build_mlx_header(struct mthca
 	return 0;
 }
 
-static inline int mthca_wq_overflow(struct mthca_wq* wq, int nreq,
-	 struct ib_cq* ib_cq)
+static inline int mthca_wq_overflow(struct mthca_wq *wq, int nreq,
+				    struct ib_cq *ib_cq)
 {
-	int cur;
-	struct mthca_cq* cq;
+	unsigned cur;
+	struct mthca_cq *cq;
 
-	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	cur = wq->head - wq->tail;
 	if (likely(cur + nreq < wq->max))
 		return 0;
 
 	cq = to_mcq(ib_cq);
 	spin_lock(&cq->lock);
-	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	cur = wq->head - wq->tail;
 	spin_unlock(&cq->lock);
 
 	return cur + nreq >= wq->max;
@@ -1436,13 +1437,13 @@ int mthca_tavor_post_send(struct ib_qp *
 
 	/* XXX check that state is OK to post send */
 
-	ind = qp->sq.next;
+	ind = qp->sq.next_ind;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
 		if (mthca_wq_overflow(&qp->sq, nreq, qp->ibqp.send_cq)) {
-			mthca_err(dev, "SQ %06x full (%d next, %d last_polled,"
+			mthca_err(dev, "SQ %06x full (%u head, %u tail,"
 					" %d max, %d nreq)\n", qp->qpn,
-					qp->sq.next, qp->sq.last_comp,
+					qp->sq.head, qp->sq.tail,
 					qp->sq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
@@ -1603,7 +1604,7 @@ out:
 	if (likely(nreq)) {
 		u32 doorbell[2];
 
-		doorbell[0] = cpu_to_be32(((qp->sq.next << qp->sq.wqe_shift) +
+		doorbell[0] = cpu_to_be32(((qp->sq.next_ind << qp->sq.wqe_shift) +
 					   qp->send_wqe_offset) | f0 | op0);
 		doorbell[1] = cpu_to_be32((qp->qpn << 8) | size0);
 
@@ -1614,7 +1615,8 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->sq.next = ind;
+	qp->sq.next_ind = ind;
+	qp->sq.head    += nreq;
 
 	spin_unlock_irqrestore(&qp->sq.lock, flags);
 	return err;
@@ -1639,13 +1641,13 @@ int mthca_tavor_post_receive(struct ib_q
 
 	/* XXX check that state is OK to post receive */
 
-	ind = qp->rq.next;
+	ind = qp->rq.next_ind;
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
 		if (mthca_wq_overflow(&qp->rq, nreq, qp->ibqp.recv_cq)) {
-			mthca_err(dev, "RQ %06x full (%d next, %d last_polled,"
+			mthca_err(dev, "RQ %06x full (%u head, %u tail,"
 					" %d max, %d nreq)\n", qp->qpn,
-					qp->rq.next, qp->rq.last_comp,
+					qp->rq.head, qp->rq.tail,
 					qp->rq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
@@ -1703,7 +1705,7 @@ out:
 	if (likely(nreq)) {
 		u32 doorbell[2];
 
-		doorbell[0] = cpu_to_be32((qp->rq.next << qp->rq.wqe_shift) | size0);
+		doorbell[0] = cpu_to_be32((qp->rq.next_ind << qp->rq.wqe_shift) | size0);
 		doorbell[1] = cpu_to_be32((qp->qpn << 8) | nreq);
 
 		wmb();
@@ -1713,7 +1715,8 @@ out:
 			      MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
 	}
 
-	qp->rq.next = ind;
+	qp->rq.next_ind = ind;
+	qp->rq.head    += nreq;
 
 	spin_unlock_irqrestore(&qp->rq.lock, flags);
 	return err;
@@ -1740,13 +1743,13 @@ int mthca_arbel_post_send(struct ib_qp *
 
 	/* XXX check that state is OK to post send */
 
-	ind = qp->sq.next & (qp->sq.max - 1);
+	ind = qp->sq.head & (qp->sq.max - 1);
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
 		if (mthca_wq_overflow(&qp->sq, nreq, qp->ibqp.send_cq)) {
-			mthca_err(dev, "SQ %06x full (%d next, %d last_polled,"
+			mthca_err(dev, "SQ %06x full (%u head, %u tail,"
 					" %d max, %d nreq)\n", qp->qpn,
-					qp->sq.next, qp->sq.last_comp,
+					qp->sq.head, qp->sq.tail,
 					qp->sq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
@@ -1857,18 +1860,18 @@ out:
 		u32 doorbell[2];
 
 		doorbell[0] = cpu_to_be32((nreq << 24)                  |
-					  ((qp->sq.next & 0xffff) << 8) |
+					  ((qp->sq.head & 0xffff) << 8) |
 					  f0 | op0);
 		doorbell[1] = cpu_to_be32((qp->qpn << 8) | size0);
 
-		qp->sq.next += nreq;
+		qp->sq.head += nreq;
 
 		/*
 		 * Make sure that descriptors are written before
 		 * doorbell record.
 		 */
 		wmb();
-		*qp->sq.db = cpu_to_be32(qp->sq.next & 0xffff);
+		*qp->sq.db = cpu_to_be32(qp->sq.head & 0xffff);
 
 		/*
 		 * Make sure doorbell record is written before we
@@ -1900,13 +1903,13 @@ int mthca_arbel_post_receive(struct ib_q
 
 	/* XXX check that state is OK to post receive */
 
-	ind = qp->rq.next & (qp->rq.max - 1);
+	ind = qp->rq.head & (qp->rq.max - 1);
 
 	for (nreq = 0; wr; ++nreq, wr = wr->next) {
 		if (mthca_wq_overflow(&qp->rq, nreq, qp->ibqp.recv_cq)) {
-			mthca_err(dev, "RQ %06x full (%d next, %d last_polled,"
+			mthca_err(dev, "RQ %06x full (%u head, %u tail,"
 					" %d max, %d nreq)\n", qp->qpn,
-					qp->rq.next, qp->rq.last_comp,
+					qp->rq.head, qp->rq.tail,
 					qp->rq.max, nreq);
 			err = -ENOMEM;
 			*bad_wr = wr;
@@ -1949,14 +1952,14 @@ int mthca_arbel_post_receive(struct ib_q
 	}
 out:
 	if (likely(nreq)) {
-		qp->rq.next += nreq;
+		qp->rq.head += nreq;
 
 		/*
 		 * Make sure that descriptors are written before
 		 * doorbell record.
 		 */
 		wmb();
-		*qp->rq.db = cpu_to_be32(qp->rq.next & 0xffff);
+		*qp->rq.db = cpu_to_be32(qp->rq.head & 0xffff);
 	}
 
 	spin_unlock_irqrestore(&qp->rq.lock, flags);



From mst at mellanox.co.il  Fri Feb 25 01:03:39 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 11:03:39 +0200
Subject: [openib-general] [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <52r7j52qy8.fsf@topspin.com>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
Message-ID: <20050225090339.GA25409@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
> Ugh, I think I missed something when I thought about this the first
> time around.  It seems the test for WQ overflow assumes that all WQs
> have a power-of-2 size, which we currently don't enforce for Tavor
> mode.

Actually the old code had this

wq->last_comp = wq->max - 1;

which led me to assumed its a power of two.

> It seems there are two possible solutions:
> 
>   Round up WQ sizes for Tavor as well.  I don't like this because it
>   could potentially use a lot of extra memory.
> 
>   Or, add one more counter back into the WQ struct so we can keep
>   track of both the next index to use as well as the total number of
>   WQEs posted in Tavor mode (we still only need one counter in
>   mem-free mode).
> 
> I implemented the second option.  Does this patch look reasonable?
> 
>  - R.
> 
> Index: hw/mthca/mthca_provider.h
> ===================================================================
> --- hw/mthca/mthca_provider.h	(revision 1912)
> +++ hw/mthca/mthca_provider.h	(working copy)
> @@ -167,15 +167,16 @@ struct mthca_cq {
>  
>  struct mthca_wq {
>  	spinlock_t lock;
> -	int   max;
> -	int   next;
> -	int   last_comp;
> -	void *last;
> -	int   max_gs;
> -	int   wqe_shift;
> +	int        max;
> +	unsigned   next_ind;
> +	unsigned   head;
> +	unsigned   tail;
> +	void      *last;
> +	int        max_gs;
> +	int        wqe_shift;
>  
> -	int   db_index;		/* Arbel only */
> -	u32  *db;
> +	int        db_index;	/* Arbel only */
> +	u32       *db;
>  };
>  
>  struct mthca_qp {
> Index: hw/mthca/mthca_cq.c
> ===================================================================
> --- hw/mthca/mthca_cq.c	(revision 1912)
> +++ hw/mthca/mthca_cq.c	(working copy)
> @@ -453,7 +453,7 @@ static inline int mthca_poll_one(struct 
>  		entry->wr_id = (*cur_qp)->wrid[wqe_index];
>  	}
>  
> -	wq->last_comp = wqe_index;
> +	++wq->tail;
>  
>  	if (0)
>  		mthca_dbg(dev, "%s completion for QP %06x, index %d (nr %d)\n",

You seem to assume that each completion is for exactly one wqe.
How does this work for selective signaling?


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Fri Feb 25 04:31:35 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 14:31:35 +0200
Subject: [openib-general] [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <52r7j52qy8.fsf@topspin.com>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
Message-ID: <20050225123135.GA25571@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
> Ugh, I think I missed something when I thought about this the first
> time around.  It seems the test for WQ overflow assumes that all WQs
> have a power-of-2 size, which we currently don't enforce for Tavor
> mode.  It seems there are two possible solutions:
> 
>   Round up WQ sizes for Tavor as well.  I don't like this because it
>   could potentially use a lot of extra memory.
> 
>   Or, add one more counter back into the WQ struct so we can keep
>   track of both the next index to use as well as the total number of
>   WQEs posted in Tavor mode (we still only need one counter in
>   mem-free mode).
> 
> I implemented the second option.  Does this patch look reasonable?
> 
>  - R.
> 

True, I forgot that qp size may not be a power of 2 for tavor.
Good catch.

But, it seems to me your patch does not handle unsignaled WQEs, since
tail is incremented by 1 on each completion.
I propose reverting, and applying the following simple patch.


diff -rup hw/mthca_old/mthca_qp.c hw/mthca/mthca_qp.c
--- hw/mthca_old/mthca_qp.c	2005-02-25 14:04:35.000000000 +0200
+++ hw/mthca/mthca_qp.c	2005-02-25 14:17:25.000000000 +0200
@@ -1403,13 +1403,15 @@ static inline int mthca_wq_overflow(stru
 	int cur;
 	struct mthca_cq* cq;
 
-	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	cur = wq->next > wq->last_comp ? wq->next - wq->last_comp - 1 :
+		wq->next + wq->max - wq->last_comp - 1;
 	if (likely(cur + nreq < wq->max))
 		return 0;
 
 	cq = to_mcq(ib_cq);
 	spin_lock(&cq->lock);
-	cur = (wq->next - wq->last_comp - 1) & (wq->max - 1);
+	cur = wq->next > wq->last_comp ? wq->next - wq->last_comp - 1 :
+		wq->next + wq->max - wq->last_comp - 1;
 	spin_unlock(&cq->lock);
 
 	return cur + nreq >= wq->max;


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Fri Feb 25 04:57:27 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 14:57:27 +0200
Subject: [openib-general] common cq
In-Reply-To: <20050217161839.D7121@topspin.com>
References: <20050217153517.GV28446@mellanox.co.il>
	<20050217110256.A7121@topspin.com>
	<20050217212937.GA21072@mellanox.co.il>
	<20050217140015.B7121@topspin.com>
	<20050217221330.GA21318@mellanox.co.il>
	<20050217144615.C7121@topspin.com>
	<20050217232539.GB21318@mellanox.co.il>
	<20050217161839.D7121@topspin.com>
Message-ID: <20050225125727.GA25679@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: SDP_CONN_LOCK
>
>   Next, in this portion of the code I was actually going to try using a
> single CQ instead of two to see if we get an additional improvement as a
> result of fewer interrupts and simpler code.
> 
> 
> -Libor

Libor, what about using a common cq for multiple sockets?
The advantages would be - reduced event rate and reduced locking.
Since each CQ also uses hardware resources, this may help scalability.

Memory costs may be minimal (or non-existant), especially if/when
resize CQ is supported.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Fri Feb 25 05:02:21 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 15:02:21 +0200
Subject: [openib-general] Re: [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <20050225123135.GA25571@mellanox.co.il>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
	<20050225123135.GA25571@mellanox.co.il>
Message-ID: <20050225130221.GB25679@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: Re: [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
> Quoting r. Roland Dreier <roland at topspin.com>:
> > Subject: Re: [openib-general] [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> > 
> > Ugh, I think I missed something when I thought about this the first
> > time around.  It seems the test for WQ overflow assumes that all WQs
> > have a power-of-2 size, which we currently don't enforce for Tavor
> > mode.  It seems there are two possible solutions:
> > 
> >   Round up WQ sizes for Tavor as well.  I don't like this because it
> >   could potentially use a lot of extra memory.
> > 
> >   Or, add one more counter back into the WQ struct so we can keep
> >   track of both the next index to use as well as the total number of
> >   WQEs posted in Tavor mode (we still only need one counter in
> >   mem-free mode).
> > 
> > I implemented the second option.  Does this patch look reasonable?
> > 
> >  - R.
> > 
> 
> True, I forgot that qp size may not be a power of 2 for tavor.
> Good catch.
> 
> But, it seems to me your patch does not handle unsignaled WQEs, since
> tail is incremented by 1 on each completion.
> I propose reverting, and applying the following simple patch.
> 

I forgot the SOB for that patch. Here it is:

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

-- 
MST - Michael S. Tsirkin


From eitan at mellanox.co.il  Fri Feb 25 05:44:32 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Fri, 25 Feb 2005 15:44:32 +0200
Subject: [openib-general] [SM] OpenSM Gen1 Update
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEC8@mtlex01.yok.mtl.com>

Parks wrote:
Is there a way to build just the opensm. I pulled down just the osm but
couldn't figure out how to get it to build.

I only want to test the OSM right now.

I couldn't find an OSM in gen2 trunk so I  am assuming yours is the only
one.
[EZ] There is an osm sub directory under the gen2 trunk userlevel management
dir

 Is it the same as the one in Mellanox's IB gold 1.6.1 ????
[EZ] No it is a pre-release of the IBGD 1.7.0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050225/370bacba/attachment.html>

From roland at topspin.com  Fri Feb 25 07:08:23 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 25 Feb 2005 07:08:23 -0800
Subject: [openib-general] [PATCH] (repost) no qp lock on poll, separate
	sq/rq locks
In-Reply-To: <20050225090339.GA25409@mellanox.co.il> (Michael S. Tsirkin's
	message of "Fri, 25 Feb 2005 11:03:39 +0200")
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com> <20050225090339.GA25409@mellanox.co.il>
Message-ID: <52is4g3cew.fsf@topspin.com>

    Michael> You seem to assume that each completion is for exactly
    Michael> one wqe.  How does this work for selective signaling?

Oh crap, back to the drawing board again...

 - R.


From mst at mellanox.co.il  Fri Feb 25 07:10:02 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 17:10:02 +0200
Subject: [openib-general] [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <52is4g3cew.fsf@topspin.com>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
	<20050225090339.GA25409@mellanox.co.il>
	<52is4g3cew.fsf@topspin.com>
Message-ID: <20050225151002.GD25679@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
>     Michael> You seem to assume that each completion is for exactly
>     Michael> one wqe.  How does this work for selective signaling?
> 
> Oh crap, back to the drawing board again...
> 
>  - R.
> 

Have you seen my patch? Should do it.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Fri Feb 25 07:12:11 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 17:12:11 +0200
Subject: [openib-general] [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <52is4g3cew.fsf@topspin.com>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
	<20050225090339.GA25409@mellanox.co.il>
	<52is4g3cew.fsf@topspin.com>
Message-ID: <20050225151211.GE25679@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
>     Michael> You seem to assume that each completion is for exactly
>     Michael> one wqe.  How does this work for selective signaling?
> 
> Oh crap, back to the drawing board again...
> 
>  - R.
> 

I also have to say I dont object to the renaming of next to next_ind,
but I think last_comp is more descriptive than tail.

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Fri Feb 25 07:13:55 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 25 Feb 2005 07:13:55 -0800
Subject: [openib-general] [PATCH] (repost) no qp lock on poll, separate
	sq/rq locks
In-Reply-To: <20050225123135.GA25571@mellanox.co.il> (Michael S. Tsirkin's
	message of "Fri, 25 Feb 2005 14:31:35 +0200")
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com> <20050225123135.GA25571@mellanox.co.il>
Message-ID: <52ekf43c5o.fsf@topspin.com>

+	cur = wq->next > wq->last_comp ? wq->next - wq->last_comp - 1 :
+		wq->next + wq->max - wq->last_comp - 1;

The problem is that there are only wq->max possible values of wq->next
and wq->last_comp (0 ... wq->max-1), but there are wq->max+1 possible
values of cur (from 0 posted up to wq->max posted).  So it's not
possible to distinguish between a full WQ and an empty one.

For example, wq->next is initialized to 0 and wq->last_comp is
initialized to wq->max - 1 and the WQ is empty.  But after posting
wq->max WQEs with no completions, wq->next will be back to 0 and
wq->last_comp will still be wq->max - 1.  So we can't tell if the WQ
is full or empty.

I think the solution is to keep track of the last completed index too,
and then we can figure out how many WQEs a completion is for.

 - R.


From mst at mellanox.co.il  Fri Feb 25 07:26:29 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 25 Feb 2005 17:26:29 +0200
Subject: [openib-general] Re: [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <52ekf43c5o.fsf@topspin.com>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
	<20050225123135.GA25571@mellanox.co.il>
	<52ekf43c5o.fsf@topspin.com>
Message-ID: <20050225152629.GF25679@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
> +	cur = wq->next > wq->last_comp ? wq->next - wq->last_comp - 1 :
> +		wq->next + wq->max - wq->last_comp - 1;
> 
> The problem is that there are only wq->max possible values of wq->next
> and wq->last_comp (0 ... wq->max-1), but there are wq->max+1 possible
> values of cur (from 0 posted up to wq->max posted).  So it's not
> possible to distinguish between a full WQ and an empty one.
> 
> For example, wq->next is initialized to 0 and wq->last_comp is
> initialized to wq->max - 1 and the WQ is empty.  But after posting
> wq->max WQEs with no completions, wq->next will be back to 0 and
> wq->last_comp will still be wq->max - 1.  So we can't tell if the WQ
> is full or empty.
> 
> I think the solution is to keep track of the last completed index too,
> and then we can figure out how many WQEs a completion is for.
> 
>  - R.
> 

Hmm. One option is to add 1 spare wqe to qp size and be done with it.

If you consider that hardware dictates this for cq, anyway
(cq is a power of 2 in size but can have 1 less cqe), maybe its
the right thing to do?

-- 
MST - Michael S. Tsirkin


From halr at voltaire.com  Fri Feb 25 07:38:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Feb 2005 10:38:25 -0500
Subject: [openib-general] [SM] OpenSM Gen1 Update
In-Reply-To: <6.1.2.0.2.20050224094602.01ced5e8@ccn-mail.lanl.gov>
References: <506C3D7B14CDD411A52C00025558DED6047EEEC5@mtlex01.yok.mtl.com>
	<6.1.2.0.2.20050224094602.01ced5e8@ccn-mail.lanl.gov>
Message-ID: <1109345035.13555.4890.camel@localhost.localdomain>

On Thu, 2005-02-24 at 11:49, Parks Fields wrote:
> I only want to test the OSM right now.
> 
> I couldn't find an OSM in gen2 trunk so I  am assuming yours is the
> only one.

The gen2 OSM is in
https://openib.org/svn/gen2/trunk/src/userspace/management/osm/

>  Is it the same as the one in Mellanox's IB gold 1.6.1 ????

Yes, this is beyond Mellanox Gold 1.6.1. It has all those changes and
more.

-- Hal



From halr at voltaire.com  Fri Feb 25 07:41:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Feb 2005 10:41:39 -0500
Subject: [openib-general] [SM] OpenSM Gen1 Update
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEEC8@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6047EEEC8@mtlex01.yok.mtl.com>
Message-ID: <1109345732.13555.4915.camel@localhost.localdomain>

On Fri, 2005-02-25 at 08:44, Eitan Zahavi wrote:
> Parks wrote:
> 
> Is there a way to build just the opensm. I pulled down just the osm
> but couldn't figure out how to get it to build.
> 
> 
> I only want to test the OSM right now.
> 
> I couldn't find an OSM in gen2 trunk so I  am assuming yours is the
> only one.
> 
> [EZ] There is an osm sub directory under the gen2 trunk userlevel
> management dir
> 
> 
>  Is it the same as the one in Mellanox's IB gold 1.6.1 ????
> 
> [EZ] No it is a pre-release of the IBGD 1.7.0

We will be incorporating the changes to this (beyond what was in 1.6.1)
into the gen2 OSM shortly. Not sure of the fixes or how critical they
would be for your deployment.

-- Hal



From roland at topspin.com  Fri Feb 25 08:59:49 2005
From: roland at topspin.com (Roland Dreier)
Date: Fri, 25 Feb 2005 08:59:49 -0800
Subject: [openib-general] [PATCH] (repost) no qp lock on poll, separate
	sq/rq locks
In-Reply-To: <52ekf43c5o.fsf@topspin.com> (Roland Dreier's message of "Fri,
	25 Feb 2005 07:13:55 -0800")
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com> <20050225123135.GA25571@mellanox.co.il>
	<52ekf43c5o.fsf@topspin.com>
Message-ID: <521xb43796.fsf@topspin.com>

    Roland> I think the solution is to keep track of the last
    Roland> completed index too, and then we can figure out how many
    Roland> WQEs a completion is for.

Something like this should do the trick I think...

--- hw/mthca/mthca_provider.h	(revision 1915)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -169,6 +169,7 @@ struct mthca_wq {
 	spinlock_t lock;
 	int        max;
 	unsigned   next_ind;
+	unsigned   last_comp;
 	unsigned   head;
 	unsigned   tail;
 	void      *last;
--- hw/mthca/mthca_cq.c	(revision 1915)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -453,7 +453,12 @@ static inline int mthca_poll_one(struct 
 		entry->wr_id = (*cur_qp)->wrid[wqe_index];
 	}
 
-	++wq->tail;
+	if (wq->last_comp < wqe_index)
+		wq->tail += wqe_index - wq->last_comp;
+	else
+		wq->tail += wqe_index + wq->max - wq->last_comp;
+
+	wq->last_comp = wqe_index;
 
 	if (0)
 		mthca_dbg(dev, "%s completion for QP %06x, index %d (nr %d)\n",
--- hw/mthca/mthca_qp.c	(revision 1915)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -1082,6 +1082,7 @@ static void mthca_wq_init(struct mthca_w
 {
 	spin_lock_init(&wq->lock);
 	wq->next_ind  = 0;
+	wq->last_comp = wq->max - 1;
 	wq->head      = 0;
 	wq->tail      = 0;
 	wq->last      = NULL;


From libor at topspin.com  Fri Feb 25 10:35:35 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 25 Feb 2005 10:35:35 -0800
Subject: [openib-general] [PATCH] PUT after UNLOCK
In-Reply-To: <20050222094220.GF23837@mellanox.co.il>;
	from mst@mellanox.co.il on Tue, Feb 22, 2005 at 11:42:21AM +0200
References: <20050222094220.GF23837@mellanox.co.il>
Message-ID: <20050225103535.A13364@topspin.com>

On Tue, Feb 22, 2005 at 11:42:21AM +0200, Michael S. Tsirkin wrote:
> SDP_CONN_PUT was called before SDP_CONN_UNLOCK.
> If this is the last reference, the connection could be removed
> and an attempt to unlock would crash.

  
  The conn_hold can be moved to after sdp_cm_connect() returns a
success result, which removes the need to call conn_put in case of
error. I'd also rather move it closer to the send_cm_req() request
which is the request that is really the reason for the increment
in reference. 

  In making that move a couple other functions then need to reflect
this change. Here's a patch which makes this update, and simplifies
the active connection sequence.

-Libor

Index: infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- infiniband/ulp/sdp/sdp_inet.c	(revision 1909)
+++ infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -590,8 +590,6 @@
 
 		conn->dst_addr = ntohl(addr->sin_addr.s_addr);
 		conn->dst_port = ntohs(addr->sin_port);
-
-		sdp_conn_hold(conn);	/* CM reference */
 		/*
 		 * close, allow connection completion notification.
 		 */
@@ -610,8 +608,6 @@
 			sock->state = SS_UNCONNECTED;
 			conn->istate = SDP_SOCK_ST_CLOSED;
 
-			sdp_conn_put(conn);	/* CM reference */
-
 			goto done;
 		}
 
Index: infiniband/ulp/sdp/sdp_proto.h
===================================================================
--- infiniband/ulp/sdp/sdp_proto.h	(revision 1908)
+++ infiniband/ulp/sdp/sdp_proto.h	(working copy)
@@ -120,8 +120,6 @@
 
 int sdp_wall_send_abort(struct sdp_opt *conn);
 
-int sdp_wall_recv_reject(struct sdp_opt *conn, int error);
-
 int sdp_wall_recv_confirm(struct sdp_opt *conn);
 
 int sdp_wall_recv_failed(struct sdp_opt *conn, int error);
@@ -334,6 +332,7 @@
 		       struct ib_cm_event *event,
 		       struct sdp_opt *conn);
 
+void sdp_cm_actv_error(struct sdp_opt *conn, int error);
 /*
  * passive connect functions
  */
@@ -351,10 +350,6 @@
 
 int sdp_cm_reject(struct sdp_opt *conn);
 
-int sdp_cm_failed(struct sdp_opt *conn);
-
-int sdp_cm_confirm(struct sdp_opt *conn);
-
 int sdp_recv_flush(struct sdp_opt *conn);
 
 int sdp_send_flush(struct sdp_opt *conn);
Index: infiniband/ulp/sdp/sdp_actv.c
===================================================================
--- infiniband/ulp/sdp/sdp_actv.c	(revision 1909)
+++ infiniband/ulp/sdp/sdp_actv.c	(working copy)
@@ -37,7 +37,72 @@
 /*
  * Connection establishment functions
  */
+void sdp_cm_actv_error(struct sdp_opt *conn, int error)
+{
+	int result;
+	/*
+	 * error value is positive error.
+	 *
+	 * Handle errors within active connections stream. 
+	 * First generate appropriate response, REJ, DREQ or nothing.
+	 * Second the socket must be notified of the error.
+	 */
+	switch (conn->state) {
+	default:
+		sdp_dbg_warn(conn, "REP error in unknown connection state");
+	case SDP_CONN_ST_REQ_PATH:
+		/*
+		 * CM message was never sent.
+		 */
+		conn->state = SDP_CONN_ST_CLOSED;
+	case SDP_CONN_ST_CLOSED:
+		break;
+	case SDP_CONN_ST_ERROR_STRM: /* socket has been destroyed. */
+		error = ECONNRESET;
+        case SDP_CONN_ST_REQ_SENT: 
+	case SDP_CONN_ST_REP_RECV:
+	case SDP_CONN_ST_RTU_SENT:
+		/*
+		 * All four states we have gotten a REP and are now in
+		 * one of these states.
+		 */
+		result = ib_send_cm_rej(conn->cm_id, 
+					IB_CM_REJ_CONSUMER_DEFINED,
+					NULL, 0, NULL, 0);
 
+		if (0 > result)
+			sdp_dbg_warn(conn, "Error <%d> sending CM REJ.",
+				     result);
+
+		conn->state = SDP_CONN_ST_CLOSED;
+		break;
+	case SDP_CONN_ST_ESTABLISHED:
+		/*
+		 * Made it all the way to esablished, need to initiate a
+		 * full disconnect.
+		 */
+		result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
+		if (0 > result)
+			sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", 
+				     result);
+		
+		conn->state = SDP_CONN_ST_TIME_WAIT_1;
+		break;
+	}
+
+	SDP_CONN_SET_ERR(conn, error);
+	conn->istate = SDP_SOCK_ST_ERROR;
+	conn->shutdown = SHUTDOWN_MASK;
+	conn->send_buf = 0;
+
+	if (conn->sk->sk_socket)
+		conn->sk->sk_socket->state = SS_UNCONNECTED;
+
+	sdp_iocb_q_cancel_all(conn, (0 - error));
+	sdp_inet_wake_error(conn->sk);
+	return;
+}
+
 /*
  * _sdp_actv_conn_establish - process an accepted connection request.
  */
@@ -53,134 +118,97 @@
 		     conn->dst_addr, conn->dst_port);
 
 	sk = conn->sk;
+
+	qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
+	if (!qp_attr)
+		return -ENOMEM;
 	/*
-	 * only reason not to confirm is if the connection state has changed
-	 * from under us, and the change wasn't followed up with a Abort(),
-	 * which it should have been.
+	 * modify QP to RTR
 	 */
-	if (SDP_SOCK_ST_CONNECT == conn->istate) {
-		qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
-		if (!qp_attr) {
-			result = -ENOMEM;
-			goto error;
-		}
-		/*
-		 * modify QP to RTR
-		 */
-		qp_attr->qp_state = IB_QPS_RTR;
+	qp_attr->qp_state = IB_QPS_RTR;
 
-		result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
-		if (result) {
-			sdp_dbg_warn(conn, "Error <%d> QP attributes for RTR",
-				     result);
-			goto error;
-		}
+	result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> QP attributes for RTR", result);
+		goto done;
+	}
 
-		qp_attr->min_rnr_timer = 0; /* IB_RNR_TIMER_122_88; */
-		qp_attr->rq_psn        = conn->rq_psn;
-		
-		attr_mask |= (IB_QP_MIN_RNR_TIMER | IB_QP_RQ_PSN);
+	qp_attr->min_rnr_timer = 0; /* IB_RNR_TIMER_122_88; */
+	qp_attr->rq_psn        = conn->rq_psn;
+	
+	attr_mask |= (IB_QP_MIN_RNR_TIMER | IB_QP_RQ_PSN);
 
-		result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
-		if (result) {
-			sdp_dbg_warn(conn, "Error <%d> QP modify to RTR",
-				     result);
-			goto error;
-		}
-		/*
-		 * finalize connection acceptance.
-		 */
-		SDP_CONN_ST_SET(conn, SDP_CONN_ST_RTU_SENT);
-		/*
-		 * post receive buffers.
-		 */
-		result = sdp_recv_flush(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> posting recv buffers.",
-				     result);
-			goto error;
-		}
-		/*
-		 * modify QP to RTS
-		 */
-		qp_attr->qp_state = IB_QPS_RTS;
+	result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> QP modify to RTR", result);
+		goto done;
+	}
+	/*
+	 * finalize connection acceptance.
+	 */
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_RTU_SENT);
+	/*
+	 * post receive buffers.
+	 */
+	result = sdp_recv_flush(conn);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> posting recv buffers.", result);
+		goto done;
+	}
+	/*
+	 * modify QP to RTS
+	 */
+	qp_attr->qp_state = IB_QPS_RTS;
 
-		result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
-		if (result) {
-			sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS",
-				     result);
-			goto error;
-		}
+	result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", result);
+		goto done;
+	}
 
-		result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
-		if (result) {
-			sdp_dbg_warn(conn, "Error <%d> QP modify to RTS",
-				     result);
-			goto error;
-		}
-		/*
-		 * respond to the remote connection manager with a RTU
-		 */
-		result = sdp_cm_confirm(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> CM connect confirm",
-				     result);
-			goto error;
-		}
-		/*
-		 * wake the accepting connection
-		 */
-		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
+	result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> QP modify to RTS", result);
+		goto done;
+	}
+	/*
+	 * respond to the remote connection manager with a RTU
+	 */
+	result = ib_send_cm_rtu(conn->cm_id, NULL, 0);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> sending CM RTU.", result);
+		goto done;
+	}
+	/*
+	 * wake the accepting connection
+	 */
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
 
-		sk->sk_socket->state = SS_CONNECTED;
-		conn->istate = SDP_SOCK_ST_ESTABLISHED;
-		conn->send_buf = SDP_INET_SEND_SIZE;
-		/*
-		 * release disconnects.
-		 */
-		conn->flags &= ~SDP_CONN_F_DIS_HOLD;
+	sk->sk_socket->state = SS_CONNECTED;
+	conn->istate = SDP_SOCK_ST_ESTABLISHED;
+	conn->send_buf = SDP_INET_SEND_SIZE;
+	/*
+	 * release disconnects.
+	 */
+	conn->flags &= ~SDP_CONN_F_DIS_HOLD;
 		
-		inet_sk(sk)->saddr     = htonl(conn->src_addr);
-		inet_sk(sk)->rcv_saddr = htonl(conn->src_addr);
+	inet_sk(sk)->saddr     = htonl(conn->src_addr);
+	inet_sk(sk)->rcv_saddr = htonl(conn->src_addr);
 
-		result = sdp_send_flush(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
-				     result);
-			goto error;
-		}
-		/*
-		 * write/read ready. (for those waiting on just one...)
-		 */
-		sdp_inet_wake_send(sk);
-		sdp_inet_wake_recv(sk, 0);
-
-		kfree(qp_attr);
-	} else {
-		/*
-		 * fail this connection
-		 */
-		result = sdp_cm_failed(conn);
-		if (0 > result)
-			sdp_dbg_warn(conn, "Error <%d> CM connect failed", 
-				     result);
-
-		SDP_CONN_SET_ERR(conn, EPROTO);
-		conn->istate = SDP_SOCK_ST_ERROR;
-
-		goto drop;
+	result = sdp_send_flush(conn);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> flushing receives.", result);
+		goto done;
 	}
-
-	return 0;
-error:
-	SDP_CONN_SET_ERR(conn, result);
-	conn->istate = SDP_SOCK_ST_ERROR;
-
+	/*
+	 * write/read ready. (for those waiting on just one...)
+	 */
+	sdp_inet_wake_send(sk);
+	sdp_inet_wake_recv(sk, 0);
+	
+	result = 0;
+done:
 	kfree(qp_attr);
-drop:
-	sdp_inet_wake_error(sk);
-
-	sdp_conn_put(conn);	/* CM sk reference */
 	return result;
 }
 
@@ -246,100 +274,76 @@
 		       struct sdp_opt *conn)
 {
 	struct sdp_msg_hello_ack *hello_ack;
-	int result;
-	int error;
+	int result = -EPROTO;
 
 	if (NULL == conn)
 		return -EINVAL;
 	
-	if (cm_id != conn->cm_id)
+	if (cm_id != conn->cm_id) {
 		sdp_dbg_warn(conn, "REP comm ID mismatch. <%08x:%08x>",
 			     conn->cm_id->local_id, cm_id->local_id);
+		return -EINVAL;
+	}
 
 	hello_ack = (struct sdp_msg_hello_ack *)event->private_data;
 
 	sdp_dbg_ctrl(conn, "CM REP. comm <%08x>", cm_id->local_id);
+
+	if (conn->state != SDP_CONN_ST_REQ_SENT)
+		goto error;
+
 	/*
-	 * lock the connection
+	 * check Hello Header Ack, to determine if we want
+	 * the connection.
 	 */
-	switch (conn->state) {
-	case SDP_CONN_ST_ERROR_STRM:
-		result = sdp_cm_reject(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> CM reject request",
-				     result);
-			error = result;
-			goto done;
-		}
+	result = _sdp_cm_hello_ack_check(hello_ack);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> hello ack check.", result);
+		goto error;
+	}
 
-		sdp_conn_put(conn);
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_REP_RECV);
+	/*
+	 * read remote information
+	 */
+	conn->send_size = hello_ack->hah.l_rcv_size;
+	conn->r_max_adv = hello_ack->hah.max_adv;
+	conn->r_recv_bf = hello_ack->bsdh.recv_bufs;
+	conn->recv_seq  = hello_ack->bsdh.seq_num;
+	conn->advt_seq  = hello_ack->bsdh.seq_num;
 
-		break;
-	case SDP_CONN_ST_REQ_SENT:
-		/*
-		 * check Hello Header Ack, to determine if we want
-		 * the connection.
-		 */
-		result = _sdp_cm_hello_ack_check(hello_ack);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> hello ack check.",
-				     result);
-			error = result;
-			goto reject;
-		}
+	conn->d_qpn  = event->param.rep_rcvd.remote_qpn;
+	/*
+	 * The maximum amount of data that can be sent to the remote
+	 * peer is the smaller of the local and remote buffer sizes, 
+	 * minus the size of the message header.
+	 */
+	conn->send_size = min((u16)sdp_buff_pool_buff_size(),
+			      (u16)conn->send_size) - SDP_MSG_HDR_SIZE;
+	/*
+	 * Pop the hello message that was sent
+	 */
+	(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
 
-		SDP_CONN_ST_SET(conn, SDP_CONN_ST_REP_RECV);
-		/*
-		 * read remote information
-		 */
-		conn->send_size = hello_ack->hah.l_rcv_size;
-		conn->r_max_adv = hello_ack->hah.max_adv;
-		conn->r_recv_bf = hello_ack->bsdh.recv_bufs;
-		conn->recv_seq  = hello_ack->bsdh.seq_num;
-		conn->advt_seq  = hello_ack->bsdh.seq_num;
+	result = _sdp_actv_conn_establish(conn);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> accept receive failed", result);
+		goto error;
+	}
 
-		conn->d_qpn  = event->param.rep_rcvd.remote_qpn;
-		/*
-		 * The maximum amount of data that can be sent to the remote
-		 * peer is the smaller of the local and remote buffer sizes, 
-		 * minus the size of the message header.
-		 */
-		conn->send_size = (min((u16)sdp_buff_pool_buff_size(),
-				       (u16)conn->send_size) - 
-				   SDP_MSG_HDR_SIZE);
-		/*
-		 * Pop the hello message that was sent
-		 */
-		(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
+	return 0;
+error:
+	sdp_cm_actv_error(conn, (0 - result));
 
-		result = _sdp_actv_conn_establish(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> accept receive failed",
-				     result);
-			error = result;
-			goto done;
-		}
+	if (conn->state == SDP_CONN_ST_CLOSED) {
 
-		break;
-	default:
-		sdp_dbg_warn(conn, "REP received in unknown connection state");
-		/*
-		 * drop CM reference
-		 */
-		result = sdp_wall_recv_drop(conn);
-		SDP_EXPECT(!(0 > result));
+		conn->cm_id = NULL;
+		sdp_conn_put(conn); /* CM reference */
 
-		error = -EPROTO;
-		goto done;
+		return -EPROTO;
 	}
 
 	return 0;
-reject:
-	result = sdp_wall_recv_reject(conn, EPROTO);
-	SDP_EXPECT(!(0 > result));
-done:
-	conn->cm_id = NULL;
-	return error;
 }
 
 /*
@@ -375,8 +379,8 @@
 	}
 
 	if (SDP_CONN_ST_REQ_PATH != conn->state) {
-		sdp_dbg_warn(conn, "Path record completion in bad state");
-		goto done;
+		status = -EPROTO;
+		goto failed;
 	}
 
 	conn->plid = 0;
@@ -393,7 +397,9 @@
 			     status);
 		goto failed;
 	}
-	
+
+	status = -ENOMEM; /* incase error path is taken */
+
 	sdp_dbg_ctrl(conn, "Path record lookup complete <%016llx:%016llx:%d>",
 		     (unsigned long long)
 		     cpu_to_be64(path->dgid.global.subnet_prefix),
@@ -462,6 +468,7 @@
 		expect = sdp_buff_pool_put(buff);
 		SDP_EXPECT(!(0 > expect));
 
+		status = -EPROTO;
 		goto failed;
 	}
 #if 1
@@ -505,36 +512,28 @@
 	if (NULL == conn->cm_id) {
 		sdp_dbg_warn(conn, "Failed to create CM handle, %d",
 			     (u8)(buff->tail - buff->data));
-
-		result = -ENOMEM;
 		goto failed;
 	}
 
-	SDP_CONN_ST_SET(conn, SDP_CONN_ST_REQ_SENT);
 	/*
 	 * initiate connection
 	 */
 	result = ib_send_cm_req(conn->cm_id, &param);
 	if (0 != result) {
 		sdp_dbg_warn(conn, "Error <%d> CM connect request", result);
+		status = result;
 		goto failed;
 	}
 
-	result = 0;
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_REQ_SENT);
+	sdp_conn_hold(conn); /* CM reference */
+
 	goto done;
 failed:
-
-	result = sdp_wall_recv_reject(conn, (0 - status));
-	if (0 > result) {
-		sdp_dbg_warn(conn, "Error <%d> rejecting connection", result);
-		
-		expect = sdp_wall_recv_drop(conn);
-		SDP_EXPECT(!(0 > expect));
-	}
-	/* if */
+	sdp_cm_actv_error(conn, (0 - status));
 done:
 	sdp_conn_unlock(conn);
-	sdp_conn_put(conn);
+	sdp_conn_put(conn); /* address resolution reference */
 }
 
 /*
@@ -542,23 +541,21 @@
  */
 int sdp_cm_connect(struct sdp_opt *conn)
 {
-	int result = 0;
-
+	int result;
 	/*
 	 * get the buffer size we'll use for this connection. (and all others)
 	 */
 	if (sizeof(struct sdp_msg_hello) > conn->recv_size) {
 		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
 			     conn->recv_size, sizeof(struct sdp_msg_hello));
-		result = -ENOBUFS;
-		goto error;
+		return -ENOBUFS;
 	}
 
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_REQ_PATH);
 	/*
 	 * lookup the remote address
 	 */
-	sdp_conn_hold(conn);
+	sdp_conn_hold(conn); /* address resolution reference */
 	sdp_conn_unlock(conn);
 					  
 	result = sdp_link_path_lookup(htonl(conn->dst_addr),
@@ -577,13 +574,10 @@
 		/*
 		 * callback dosn't have this socket.
 		 */
-		sdp_conn_put(conn);
+		sdp_conn_put(conn); /* address resolution reference */
 
-		result = -EDESTADDRREQ;
-		goto error;
+		return -EDESTADDRREQ;
 	}
 
 	return 0;
-error:
-	return result;
 }
Index: infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- infiniband/ulp/sdp/sdp_conn.c	(revision 1909)
+++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -748,7 +748,7 @@
 	 * in case CM/IB are still tracking this connection.
 	 */
 	if (conn->cm_id) {
-		sdp_dbg_warn(conn, "CM local id <%d>", conn->cm_id->local_id);
+		sdp_dbg_ctrl(conn, "CM local id <%d>", conn->cm_id->local_id);
 
 		result = ib_destroy_cm_id(conn->cm_id);
 		if (result)
Index: infiniband/ulp/sdp/sdp_wall.c
===================================================================
--- infiniband/ulp/sdp/sdp_wall.c	(revision 1908)
+++ infiniband/ulp/sdp/sdp_wall.c	(working copy)
@@ -227,11 +227,6 @@
 		 * instead of canceling allow the path completion to 
 		 * determine that the socket has moved to an error state.
 		 */
-		result = sdp_wall_recv_drop(conn);
-		SDP_EXPECT(!(0 > result));
-		/*
-		 * fall through
-		 */
 	case SDP_CONN_ST_REQ_SENT:
 	case SDP_CONN_ST_REP_SENT:
 		/*
@@ -277,32 +272,6 @@
  */
 
 /*
- * sdp_wall_recv_reject - callback to reject an active open
- */
-int sdp_wall_recv_reject(struct sdp_opt *conn, int error)
-{
-	sdp_dbg_ctrl(conn, "Reject recv. src <%08x:%04x> dst <%08x:%04x> <%d>",
-		     conn->src_addr, conn->src_port, 
-		     conn->dst_addr, conn->dst_port, error);
-	/*
-	 * the connection has been rejected, move to error, and notify anyone
-	 * waiting of the state change.
-	 */
-	SDP_CONN_SET_ERR(conn, error);
-	conn->sk->sk_socket->state = SS_UNCONNECTED;
-	conn->istate = SDP_SOCK_ST_ERROR;
-	conn->shutdown = SHUTDOWN_MASK;
-
-	sdp_iocb_q_cancel_all(conn, (0 - error));
-
-	sdp_inet_wake_error(conn->sk);
-
-	sdp_conn_put(conn);	/* CM reference */
-
-	return 0;
-}
-
-/*
  * sdp_wall_recv_confirm - callback to confirm accepeted passive open
  */
 int sdp_wall_recv_confirm(struct sdp_opt *conn)
Index: infiniband/ulp/sdp/sdp_event.c
===================================================================
--- infiniband/ulp/sdp/sdp_event.c	(revision 1908)
+++ infiniband/ulp/sdp/sdp_event.c	(working copy)
@@ -222,25 +222,12 @@
 	 * check state
 	 */
 	switch (conn->state) {
-	case SDP_CONN_ST_REQ_PATH:
-		/*
-		 * cancel address resolution
-		 */
-#if _OLD_CODE
-		result = ip2pr_path_record_cancel(conn->plid);
-		SDP_EXPECT(!(0 > result));
-#endif
-		/*
-		 * fall through
-		 */
-	case SDP_CONN_ST_REQ_SENT:	/* active open, Hello msg sent */
-		result = sdp_wall_recv_reject(conn, ECONNREFUSED);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> receiving CM reject.",
-				     result);
-			goto error;
-		}
+	case SDP_CONN_ST_REQ_SENT: /* active open, Hello msg sent */
+		conn->state = SDP_CONN_ST_CLOSED;
 
+		sdp_cm_actv_error(conn, ECONNREFUSED);
+		sdp_conn_put(conn); /* CM reference */
+
 		break;
 	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
 		result = sdp_wall_recv_failed(conn, ECONNREFUSED);
@@ -252,7 +239,6 @@
 		}
 
 		break;
-	case SDP_CONN_ST_REP_RECV:	/* active open, Hello ack msg recv'd */
 	case SDP_CONN_ST_REQ_RECV:	/* passive open, Hello msg recv'd */
 		/*
 		 * connection state is outstanding to the gateway, so the
Index: infiniband/ulp/sdp/sdp_post.c
===================================================================
--- infiniband/ulp/sdp/sdp_post.c	(revision 1909)
+++ infiniband/ulp/sdp/sdp_post.c	(working copy)
@@ -131,55 +131,7 @@
 
 	sdp_conn_put(conn);
 }
-
 /*
- * _sdp_cm_confirm - initiate a confirm request using the CM
- */
-static void _sdp_cm_confirm(void *arg)
-{
-	struct sdp_opt *conn = (struct sdp_opt *)arg;
-	int result;
-
-	if (NULL == conn)
-		sdp_dbg_warn(NULL, "Error, posting confirm for NULL conn");
-
-	sdp_dbg_ctrl(conn, "Defered confirm <%08x>", conn->cm_id->local_id);
-	/*
-	 * send a confirm request using the connection manager
-	 */
-	result = ib_send_cm_rtu(conn->cm_id, NULL, 0);
-	if (0 > result)
-		sdp_dbg_warn(NULL, "Error <%d> CM confirm request", result);
-
-	sdp_conn_put(conn);
-}
-
-/*
- * _sdp_cm_failed - initiate a Failed request using the CM
- */
-static void _sdp_cm_failed(void *arg)
-{
-	struct sdp_opt *conn = (struct sdp_opt *)arg;
-	int result;
-
-	if (NULL == conn)
-		sdp_dbg_warn(NULL, "Error, posting failed for NULL conn");
-
-	sdp_dbg_ctrl(conn, "Defered failed request <%08x>",
-		     conn->cm_id->local_id);
-	/*
-	 * send a failed request using the connection manager
-	 */
-	result = ib_send_cm_rej(conn->cm_id, 
-				IB_CM_REJ_CONSUMER_DEFINED,
-				NULL, 0, NULL, 0);
-	if (0 > result)
-		sdp_dbg_warn(NULL, "Error <%d> CM failed request", result);
-
-	sdp_conn_put(conn);
-}
-
-/*
  * _sdp_cm_deferred_generic - initiate a defered request using the CM
  */
 static int _sdp_cm_deferred_generic(struct sdp_opt *conn,
@@ -214,19 +166,3 @@
 {
 	return _sdp_cm_deferred_generic(conn, _sdp_cm_reject);
 }
-
-/*
- * sdp_cm_confirm - initiate a confirm request using the CM
- */
-int sdp_cm_confirm(struct sdp_opt *conn)
-{
-	return _sdp_cm_deferred_generic(conn, _sdp_cm_confirm);
-}
-
-/*
- * sdp_cm_failed - initiate a failed request using the CM
- */
-int sdp_cm_failed(struct sdp_opt *conn)
-{
-	return _sdp_cm_deferred_generic(conn, _sdp_cm_failed);
-}



From libor at topspin.com  Fri Feb 25 11:55:11 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 25 Feb 2005 11:55:11 -0800
Subject: [openib-general] Re: sdp_kvec.c
In-Reply-To: <20050224221250.GA17573@mellanox.co.il>;
	from mst@mellanox.co.il on Fri, Feb 25, 2005 at 12:12:50AM +0200
References: <20050223110243.GA23837@mellanox.co.il>
	<20050224111110.A24393@topspin.com>
	<20050224200339.GA15534@mellanox.co.il>
	<20050224121550.C24393@topspin.com>
	<20050224205834.GA16594@mellanox.co.il>
	<20050224131538.D24393@topspin.com>
	<20050224221250.GA17573@mellanox.co.il>
Message-ID: <20050225115511.B13364@topspin.com>

On Fri, Feb 25, 2005 at 12:12:50AM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: sdp_kvec.c
> > 
> > On Thu, Feb 24, 2005 at 10:58:34PM +0200, Michael S. Tsirkin wrote:
> > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > > What do you say to removing the comments after the closing }?
> > > > > Its clearly unnecessary (most functions are, or shall be, short enough
> > > > > to see where they end), and increases the code footprint.
> > > > > For me, they make the detection of the function end harder rather than
> > > > > easier, since I am always looking for a sole }.
> > > > 
> > > >   OK, I'll stop adding them to new functions, and remove them as I
> > > > touch code. If I or anyone else get's ambitious we can strip them
> > > > from all the functions.
> 
> 
> Here's some more - I missed the ones after the structure end.
> Note how at least in one case (_sdp_prot) the closing comment said
> _pf_family. Which shows that these clean-ups are a good thing.

Thanks, I've applied the patch and checked it in.

-Libor


From libor at topspin.com  Fri Feb 25 11:58:57 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 25 Feb 2005 11:58:57 -0800
Subject: [openib-general] Re: sdp: 0 > result
In-Reply-To: <20050224214928.GA17236@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Feb 24, 2005 at 11:49:28PM +0200
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
Message-ID: <20050225115857.C13364@topspin.com>

On Thu, Feb 24, 2005 at 11:49:28PM +0200, Michael S. Tsirkin wrote:
> OK, now what about things like these:
> 
>         if (0 > result) {
> 
> may I change them all to
> 
>         if (result < 0) {
> 
> While being equivalent, we are testing the result, not 0.
> 
> Similiarly (although I feel somewhat less strongly about it)
> 
>         if (0 == result)
> and
>         if (NULL == conn)
> 
> are better off as
> 
> 	if (!result) {
> and
>         if (!conn)
> 
> C is a Spartan language, and this is more brief.
> Libor, I think I asked about the second one, but dont recall you
> answering.
> If OK to both, let me know and I'll do it on Sunday.

  I actually feel more strongly in favour of making the second change
you propose then the first. However, I'm OK with both, so feel free
to submit a patch.

-Libor


From mshefty at ichips.intel.com  Fri Feb 25 15:04:41 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 25 Feb 2005 15:04:41 -0800
Subject: [openib-general] [PATCH] [MAD] change to MAD layer callback when
	canceling a MAD
In-Reply-To: <20050224135520.25d3769a.mshefty@ichips.intel.com>
References: <20050224135520.25d3769a.mshefty@ichips.intel.com>
Message-ID: <421FAF09.7070902@ichips.intel.com>

Sean Hefty wrote:
> This patch modifies ib_cancel_mad() to invoke a user's send callback
> from a different thread context than that used by the caller.  This
> allows a caller to hold a lock while calling cancel that is also
> acquired from their send handler.
> 
> Signed-off-by: Sean Hefty <sean.hefty at intel.com>


I've committed this change.

- Sean




From mshefty at ichips.intel.com  Fri Feb 25 15:04:52 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 25 Feb 2005 15:04:52 -0800
Subject: [openib-general] [PATCH] [CM] synchronize canceling a MAD with
	completion processing
In-Reply-To: <20050224144544.411d51ba.mshefty@ichips.intel.com>
References: <20050224144544.411d51ba.mshefty@ichips.intel.com>
Message-ID: <421FAF14.1090907@ichips.intel.com>

Sean Hefty wrote:

> This patch synchronizes the canceling of a MAD by the CM with the
> processing of completions.  The changes have not been committed as
> they depend on the MAD layer patch.

I've committed this change.

- Sean




From libor at topspin.com  Fri Feb 25 17:01:05 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 25 Feb 2005 17:01:05 -0800
Subject: [openib-general] CM REP-REJ bug
In-Reply-To: <421FAF14.1090907@ichips.intel.com>;
	from mshefty@ichips.intel.com on Fri, Feb 25, 2005 at 03:04:52PM
	-0800
References: <20050224144544.411d51ba.mshefty@ichips.intel.com>
	<421FAF14.1090907@ichips.intel.com>
Message-ID: <20050225170105.D13364@topspin.com>

Sean,

   There's a problem with REJ response to REQ. The REJ handler
uses cm_acquire_id() to find the cm_id of the REQ, but it uses 
rej_msg->local_comm_id to perform the lookup. In the REQ case the
cm_id does not have a remote_id with which to match the REJ.

  Here's a possible patch.

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1922)
+++ infiniband/core/cm.c	(working copy)
@@ -317,7 +317,8 @@
 	spin_lock_irqsave(&cm.lock, flags);
 	cm_id_priv = idr_find(&cm.local_id_table, (int) local_id);
 	if (cm_id_priv) {
-		if (cm_id_priv->id.remote_id == remote_id)
+		if (cm_id_priv->id.remote_id == 0 ||
+		    cm_id_priv->id.remote_id == remote_id)
 			atomic_inc(&cm_id_priv->refcount);
 		else
 			cm_id_priv = NULL;


From libor at topspin.com  Fri Feb 25 17:41:37 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 25 Feb 2005 17:41:37 -0800
Subject: [openib-general] [PATCH] SDP passive connect rework
Message-ID: <20050225174137.E13364@topspin.com>


  This patch simplifies the passive connection process and reworks
the connection reference counting, to better match the CM. Next step
is the disconnect path.

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1922)
+++ sdp_proto.h	(working copy)
@@ -120,17 +120,13 @@
 
 int sdp_wall_send_abort(struct sdp_opt *conn);
 
-int sdp_wall_recv_confirm(struct sdp_opt *conn);
-
-int sdp_wall_recv_failed(struct sdp_opt *conn, int error);
-
 int sdp_wall_recv_close(struct sdp_opt *conn);
 
 int sdp_wall_recv_closing(struct sdp_opt *conn);
 
 int sdp_wall_recv_abort(struct sdp_opt *conn);
 
-int sdp_wall_recv_drop(struct sdp_opt *conn);
+void sdp_wall_recv_drop(struct sdp_opt *conn);
 
 int sdp_wall_abort(struct sdp_opt *conn);
 
@@ -336,6 +332,10 @@
 /*
  * passive connect functions
  */
+void sdp_cm_pass_error(struct sdp_opt *conn, int error);
+
+int sdp_cm_pass_establish(struct sdp_opt *conn);
+
 int sdp_cm_req_handler(struct ib_cm_id *cm_id,
 		       struct ib_cm_event *event);
 
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1922)
+++ sdp_actv.c	(working copy)
@@ -104,9 +104,9 @@
 }
 
 /*
- * _sdp_actv_conn_establish - process an accepted connection request.
+ * _sdp_cm_actv_establish - process an accepted connection request.
  */
-static int _sdp_actv_conn_establish(struct sdp_opt *conn)
+static int _sdp_cm_actv_establish(struct sdp_opt *conn)
 {
 	struct ib_qp_attr *qp_attr;
 	int attr_mask = 0;
@@ -275,9 +275,6 @@
 {
 	struct sdp_msg_hello_ack *hello_ack;
 	int result = -EPROTO;
-
-	if (NULL == conn)
-		return -EINVAL;
 	
 	if (cm_id != conn->cm_id) {
 		sdp_dbg_warn(conn, "REP comm ID mismatch. <%08x:%08x>",
@@ -325,7 +322,7 @@
 	 */
 	(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
 
-	result = _sdp_actv_conn_establish(conn);
+	result = _sdp_cm_actv_establish(conn);
 	if (0 > result) {
 		sdp_dbg_warn(conn, "Error <%d> accept receive failed", result);
 		goto error;
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1922)
+++ sdp_conn.c	(working copy)
@@ -99,7 +99,7 @@
 	/*
 	 * up ref until we release. One ref for GW and one for INET.
 	 */
-	sdp_conn_hold(accept_conn);	/* INET reference */
+	sdp_conn_hold(accept_conn); /* AcceptQueue INET reference */
 
 	return 0;
 }
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1922)
+++ sdp_wall.c	(working copy)
@@ -227,6 +227,8 @@
 		 * instead of canceling allow the path completion to 
 		 * determine that the socket has moved to an error state.
 		 */
+		SDP_CONN_ST_SET(conn, SDP_CONN_ST_CLOSED);
+		break;
 	case SDP_CONN_ST_REQ_SENT:
 	case SDP_CONN_ST_REP_SENT:
 		/*
@@ -272,103 +274,6 @@
  */
 
 /*
- * sdp_wall_recv_confirm - callback to confirm accepeted passive open
- */
-int sdp_wall_recv_confirm(struct sdp_opt *conn)
-{
-        struct ib_qp_attr *qp_attr;
-	int attr_mask = 0;
-	int result = 0;
-
-	sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>",
-		     conn->src_addr, conn->src_port, 
-		     conn->dst_addr, conn->dst_port);
-
-        qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
-        if (!qp_attr) {
-                result = -ENOMEM;
-		goto error;
-        }
-
-        memset(qp_attr, 0, sizeof(*qp_attr));
-
-	qp_attr->qp_state = IB_QPS_RTS;
-
-	result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
-	if (result) {
-		sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS",
-			     result);
-		goto error;
-	}
-
-        result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
-        kfree(qp_attr);
-	if (result)
-		goto error;
-
-	switch (conn->istate) {
-	case SDP_SOCK_ST_ACCEPTING:
-		conn->istate = SDP_SOCK_ST_ACCEPTED;
-		conn->send_buf = SDP_INET_SEND_SIZE;
-
-		break;
-	case SDP_SOCK_ST_ACCEPTED:
-		conn->istate = SDP_SOCK_ST_ESTABLISHED;
-		conn->send_buf = SDP_INET_SEND_SIZE;
-
-		sdp_inet_wake_send(conn->sk);
-
-		break;
-	default:
-		result = -EPROTO;
-		goto error;
-	}
-
-	return 0;
-error:
-	conn->istate = SDP_SOCK_ST_ERROR;
-
-	sdp_inet_wake_error(conn->sk);
-	
-	sdp_conn_put(conn);	/* CM sk reference */
-	
-	return result;
-}
-
-/*
- * sdp_wall_recv_failed - callback to notify accepted open of a failure
- */
-int sdp_wall_recv_failed(struct sdp_opt *conn, int error)
-{
-	sdp_dbg_ctrl(conn, "Failed recv. src <%08x:%04x> dst <%08x:%04x> <%d>",
-		     conn->src_addr, conn->src_port, 
-		     conn->dst_addr, conn->dst_port, error);
-	/*
-	 * the connection has failed, move to error, and notify anyone
-	 * waiting of the state change.
-	 */
-	SDP_EXPECT((SDP_SOCK_ST_ACCEPTED == conn->istate));
-
-	switch (conn->istate) {
-	default:
-		SDP_CONN_SET_ERR(conn, error);
-		conn->sk->sk_socket->state = SS_UNCONNECTED;
-		sdp_inet_wake_error(conn->sk);
-		/*
-		 * fall through
-		 */
-	case SDP_SOCK_ST_ACCEPTING:
-		conn->istate = SDP_SOCK_ST_ERROR;
-		conn->shutdown = SHUTDOWN_MASK;
-		break;
-	}
-
-	sdp_conn_put(conn);	/* CM reference */
-
-	return 0;
-}
-
-/*
  * sdp_wall_recv_close - callback to accept an active close
  */
 int sdp_wall_recv_close(struct sdp_opt *conn)
@@ -482,20 +387,16 @@
 /*
  * sdp_wall_recv_drop - drop SDP protocol reference to socket
  */
-int sdp_wall_recv_drop(struct sdp_opt *conn)
+void sdp_wall_recv_drop(struct sdp_opt *conn)
 {
 	int result;
 
 	sdp_dbg_ctrl(conn, "Drop. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
-	/*
-	 * Lock Note:
-	 * This function should only be called from the process context,
-	 * because AcceptQueueRemove will take the process context connection
-	 * lock. Generally this function is only called as a result of CM
-	 * state transitions.
-	 */
+
+	conn->send_buf = 0;
+
 	switch (conn->istate) {
 	case SDP_SOCK_ST_ACCEPTING:
 		/*
@@ -533,11 +434,7 @@
 		break;
 	}
 
-	conn->send_buf = 0;
-
-	sdp_conn_put(conn);	/* CM sk reference */
-
-	return 0;
+	return;
 }
 
 /*
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1922)
+++ sdp_pass.c	(working copy)
@@ -35,8 +35,116 @@
 #include "sdp_main.h"
 
 /*
- * _sdp_cm_accept - respond to a connection attempt with an ack
+ * handle incomming passive connection error. (REJ)
  */
+void sdp_cm_pass_error(struct sdp_opt *conn, int error)
+{
+	int result;
+
+	sdp_dbg_ctrl(conn, 
+		     "passive error. src <%08x:%04x> dst <%08x:%04x> <%d>",
+		     conn->src_addr, conn->src_port, 
+		     conn->dst_addr, conn->dst_port, error);
+	/*
+	 * the connection has failed, move to error, and notify anyone
+	 * waiting of the state change. remove connection from listen
+	 * queue if possible
+	 */
+	result = sdp_inet_accept_q_remove(conn);
+	if (!result)
+		sdp_conn_put(conn); /* AcceptQueue INET reference */
+
+	SDP_CONN_SET_ERR(conn, error);
+	conn->istate = SDP_SOCK_ST_ERROR;
+	conn->shutdown = SHUTDOWN_MASK;
+	conn->send_buf = 0;
+
+	if (conn->sk->sk_socket)
+		conn->sk->sk_socket->state = SS_UNCONNECTED;
+
+	sdp_iocb_q_cancel_all(conn, (0 - error));
+	sdp_inet_wake_error(conn->sk);
+}
+
+/*
+ * handle incomming passive connection establishment. (RTU)
+ */
+int sdp_cm_pass_establish(struct sdp_opt *conn)
+{
+        struct ib_qp_attr *qp_attr;
+	int attr_mask = 0;
+	int result = 0;
+
+	sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>",
+		     conn->src_addr, conn->src_port, 
+		     conn->dst_addr, conn->dst_port);
+	/*
+	 * free hello ack message
+	 */
+	(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
+	
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
+
+        qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
+        if (!qp_attr) {
+                result = -ENOMEM;
+		goto done;
+        }
+
+        memset(qp_attr, 0, sizeof(*qp_attr));
+	qp_attr->qp_state = IB_QPS_RTS;
+
+	result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS",
+			     result);
+		goto error;
+	}
+
+        result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> modifiying QP to RTS", result);
+		goto error;
+	}
+
+	conn->send_buf = SDP_INET_SEND_SIZE;
+
+	switch (conn->istate) {
+	case SDP_SOCK_ST_ACCEPTING:
+		conn->istate = SDP_SOCK_ST_ACCEPTED;
+		break;
+	case SDP_SOCK_ST_ACCEPTED:
+		conn->istate = SDP_SOCK_ST_ESTABLISHED;
+		break;
+	}
+
+	result = sdp_send_flush(conn);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> flushing sends.", result);
+		goto error;
+	}
+
+	result = sdp_recv_flush(conn);
+	if (0 > result) {
+		sdp_dbg_warn(conn, "Error <%d> flushing receives.", result);
+		goto error;
+	}
+
+	sdp_inet_wake_send(conn->sk);
+
+        kfree(qp_attr);
+	return 0;
+error:
+        kfree(qp_attr);
+done:
+	sdp_cm_pass_error(conn, (0 - result));
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
+
+	return result;
+}
+/*
+ * Functions to handle incomming passive connection requests. (REQ)
+ */
 static int _sdp_cm_accept(struct sdp_opt *conn)
 {
 	struct ib_cm_rep_param param;
@@ -46,9 +154,9 @@
 	int qp_mask = 0;
 	int result;
 	int expect;
-
 	/*
-	 * build listen response headers
+	 * Accept connection, build listen response headers and send
+	 * a REP message to remote peer.
 	 */
 	if (sizeof(struct sdp_msg_hello_ack) > conn->recv_size) {
 		sdp_dbg_warn(conn, "buffer size <%d> too small. <%Zu>",
@@ -180,16 +288,16 @@
 	return result;
 }
 
-/*
- * _sdp_cm_listen_lookup - match a listener with an incomming conn
- */
 static int _sdp_cm_listen_lookup(struct sdp_opt *conn)
 {
 	struct sdp_opt *listen_conn;
 	struct sock *listen_sk;
 	struct sock *sk;
 	int result;
-
+	/*
+	 * match a listener with an incomming conn, and generate
+	 * accept message and state on success.
+	 */
 	sdp_dbg_ctrl(conn,
 		     "listen match for conn. src <%08x:%04x> dst <%08x:%04x>",
 		     conn->src_addr, conn->src_port, 
@@ -296,9 +404,6 @@
 	return result;
 }
 
-/*
- * _sdp_cm_hello_check - validate the hello header
- */
 static int _sdp_cm_hello_check(struct sdp_msg_hello *msg_hello)
 {
 	/*
@@ -365,9 +470,6 @@
 	return 0; /* success */
 }
 
-/*
- * sdp_cm_req_handler - handler for passive connection open completion
- */
 int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 {
 	struct sdp_msg_hello *msg_hello = event->private_data;
@@ -469,9 +571,13 @@
 	sdp_conn_unlock(conn);
 	return 0;
 error:
+	SDP_CONN_ST_SET(conn, SDP_CONN_ST_CLOSED);
 	conn->cm_id = NULL; /* cm_id destroyed by CM on error result. */
 	sdp_conn_unlock(conn);
-	sdp_conn_put(conn);	/* CM sk reference, will destory */
+	sdp_conn_put(conn); /* CM reference */
 done:
+	(void)ib_send_cm_rej(cm_id,
+			     IB_CM_REJ_CONSUMER_DEFINED, 
+			     NULL, 0, NULL, 0);
 	return result;
 }
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1922)
+++ sdp_event.c	(working copy)
@@ -198,56 +198,26 @@
  * Connection establishment IB/CM callback functions
  */
 
-/*
- * _sdp_cm_idle - handler for connection idle completion
- */
 static int _sdp_cm_idle(struct ib_cm_id *cm_id,
 			struct ib_cm_event *event,
 			struct sdp_opt *conn)
 {
-	int result = 0;
-	int expect;
-
-	if (NULL == conn)
-		return -EINVAL;
-	/*
-	 * IDLE should only be called after some other action on the comm_id,
-	 * which means the callback argument will be a SDP conn, since the
-	 * only time it is not a SDP conn is the first callback in a passive
-	 * open.
-	 */
 	sdp_dbg_ctrl(conn, "CM IDLE. commID <%08x> event <%d> status <%d>",
 		     cm_id->local_id, event->event, event->param.send_status);
 	/*
 	 * check state
 	 */
 	switch (conn->state) {
-	case SDP_CONN_ST_REQ_SENT: /* active open, Hello msg sent */
+	case SDP_CONN_ST_REQ_SENT:
 		conn->state = SDP_CONN_ST_CLOSED;
 
 		sdp_cm_actv_error(conn, ECONNREFUSED);
-		sdp_conn_put(conn); /* CM reference */
-
 		break;
-	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
-		result = sdp_wall_recv_failed(conn, ECONNREFUSED);
-		if (0 > result) {
+	case SDP_CONN_ST_REP_SENT:
+		conn->state = SDP_CONN_ST_CLOSED;
 
-			sdp_dbg_warn(conn, "Error <%d> receiving CM failed",
-				     result);
-			goto error;
-		}
-
+		sdp_cm_pass_error(conn, ECONNREFUSED);
 		break;
-	case SDP_CONN_ST_REQ_RECV:	/* passive open, Hello msg recv'd */
-		/*
-		 * connection state is outstanding to the gateway, so the
-		 * connection cannot be destroyed, until the gateway responds.
-		 * The connection is marked for destruction. No return error,
-		 * since this is the last CM callback.
-		 */
-		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_CM);
-		break;
 	case SDP_CONN_ST_TIME_WAIT_1:
 	case SDP_CONN_ST_ESTABLISHED:
 		sdp_dbg_warn(conn, "Unexpected connection state");
@@ -259,44 +229,24 @@
 		/*
 		 * Connection is finally dead. Drop the CM reference
 		 */
-		result = sdp_wall_recv_drop(conn);
-		SDP_EXPECT(!(0 > result));
-
+		sdp_wall_recv_drop(conn);
 		break;
 	default:
 		sdp_warn("Unknown conn state. conn <%d> state <%04x:%04x>",
 			 conn->hashent, conn->istate, conn->state);
-		result = -EINVAL;
-		goto error;
 		break;
 	}
 
+	sdp_conn_put(conn); /* CM reference */
 	return 0;
-error:
-	/*
-	 * failed to notify INET of SDP termination of this connection,
-	 * last attempt to drop the CM reference.
-	 */
-	expect = sdp_wall_recv_drop(conn);
-	SDP_EXPECT(!(0 > expect));
-
-	return result;
 }
 
-/*
- * _sdp_cm_established - handler for connection established completion
- */
 static int _sdp_cm_established(struct ib_cm_id *cm_id,
 			       struct ib_cm_event *event,
 			       struct sdp_opt *conn)
 {
 	int result = 0;
-	int expect;
-	struct sdpc_buff *buff;
 
-	if (NULL == conn)
-		return -EINVAL;
-
 	sdp_dbg_ctrl(conn, "CM ESTABLISHED. commID <%08x>", cm_id->local_id);
 	/*
 	 * release disconnects.
@@ -306,49 +256,27 @@
 	 * check state
 	 */
 	switch (conn->state) {
-	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
-		buff = sdp_buff_q_get_head(&conn->send_post);
-		if (NULL == buff)
-			sdp_dbg_warn(conn, "hello ack missing in send pool");
-		else
-			(void)sdp_buff_pool_put(buff);
-
-		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
-
-		result = sdp_wall_recv_confirm(conn);
+	case SDP_CONN_ST_REP_SENT:
+		result = sdp_cm_pass_establish(conn);
+		if (!result)
+			break;
+		/*
+		 * on error fall through to disconnect
+		 */
+	case SDP_CONN_ST_ERROR_STRM:
+		/*
+		 * Begin abortive disconnect.
+		 * Leave state unchanged, time_wait and idle will handle the
+		 * existing state correctly.
+		 */
+		result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
 		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> confirming conn state",
+			sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", 
 				     result);
-			/*
-			 * CM sk reference released by Confirm
-			 * error processing
-			 */
 			goto done;
 		}
 
-		result = sdp_send_flush(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> flushing send queue.",
-				     result);
-			goto error;
-		}
-
-		result = sdp_recv_flush(conn);
-		if (0 > result) {
-			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
-				     result);
-			goto error;
-		}
-
 		break;
-	case SDP_CONN_ST_RTU_SENT: /* TODO: no longer approriate. */
-		/*
-		 * Set state.
-		 */
-		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
-		/*
-		 * fall through
-		 */
 	case SDP_CONN_ST_DIS_PEND_1:
 		/* active open, and active close */
 	case SDP_CONN_ST_DIS_PEND_R:
@@ -362,18 +290,6 @@
 		}
 
 		break;
-	case SDP_CONN_ST_ERROR_STRM:
-		/*
-		 * Sockets has released reference, begin abortive disconnect.
-		 * Leave state unchanged, time_wait and idle will handle the
-		 * existing state correctly.
-		 */
-		result = sdp_cm_disconnect(conn);
-		if (0 > result)
-			sdp_dbg_warn(conn, "Error <%d> posting CM disconnect",
-				     result);
-
-		break;
 	default:
 
 		sdp_warn("Unexpected conn state. conn <%d> state <%04x:%04x>",
@@ -385,10 +301,10 @@
 
 	return 0;
 error:
-	expect = sdp_wall_recv_drop(conn);
-	SDP_EXPECT(!(0 > expect));
+	sdp_wall_recv_drop(conn);
 done:
 	conn->cm_id = NULL;
+	sdp_conn_put(conn);	/* CM reference */
 	return result;
 }
 
@@ -400,11 +316,7 @@
 			    struct sdp_opt *conn)
 {
 	int result = 0;
-	int expect;
 
-	if (NULL == conn)
-		return -EINVAL;
-
 	sdp_dbg_ctrl(conn, "CM TIME WAIT. commID <%08x> event <%d>",
 		     cm_id->local_id, event->event);
 	/*
@@ -468,22 +380,17 @@
 
 	return 0;
 error:
-	expect = sdp_wall_recv_drop(conn);
-	SDP_EXPECT(!(0 > expect));
+	sdp_wall_recv_drop(conn);
 
 	conn->cm_id = NULL;
+	sdp_conn_put(conn);	/* CM reference */
 	return result;
 }
 
 /*
  * Primary Connection Managment callback function
  */
-
-/*
- * sdp_cm_event_handler - handler for CM state transitions request
- */
-int sdp_cm_event_handler(struct ib_cm_id *cm_id,
-			 struct ib_cm_event *event)
+int sdp_cm_event_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 {
 	s32 hashent = (unsigned long)cm_id->context;
 	struct sdp_opt *conn = NULL;
@@ -498,10 +405,12 @@
 	if (NULL != conn)
 		sdp_conn_lock(conn);
 	else
-		if (IB_CM_REQ_RCVD != cm_id->state)
+		if (IB_CM_REQ_RCVD != cm_id->state) {
 			sdp_dbg_warn(NULL, 
 				     "No conn <%d> CM state <%d> event <%d>",
 				     hashent, cm_id->state, event->event);
+			return -EINVAL;
+		}
 
 	switch (cm_id->state) {
 	case IB_CM_REQ_RCVD:
@@ -544,3 +453,7 @@
 
 	return result;
 }
+
+
+
+


From mst at mellanox.co.il  Sat Feb 26 08:40:39 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sat, 26 Feb 2005 18:40:39 +0200
Subject: [openib-general] [PATCH] (repost) no qp lock on poll,
	separate sq/rq locks
In-Reply-To: <521xb43796.fsf@topspin.com>
References: <20050224085054.GO23837@mellanox.co.il>
	<52r7j52qy8.fsf@topspin.com>
	<20050225123135.GA25571@mellanox.co.il>
	<52ekf43c5o.fsf@topspin.com> <521xb43796.fsf@topspin.com>
Message-ID: <20050226164039.GA1926@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [openib-general] [PATCH] (repost) no qp lock on poll, separate sq/rq locks
> 
>     Roland> I think the solution is to keep track of the last
>     Roland> completed index too, and then we can figure out how many
>     Roland> WQEs a completion is for.
> 
> Something like this should do the trick I think...

Makes sence.

-- 
MST - Michael S. Tsirkin


From jlentini at netapp.com  Sat Feb 26 17:35:26 2005
From: jlentini at netapp.com (James Lentini)
Date: Sat, 26 Feb 2005 20:35:26 -0500 (EST)
Subject: [openib-general] Porting DAPL to OpenIB Gen 2
In-Reply-To: <1109293767.12995.24.camel@duffman>
References: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
	<1109293767.12995.24.camel@duffman>
Message-ID: <Pine.LNX.4.61.0502262032500.29114@jlentini-linux.nane.netapp.com>



On Thu, 24 Feb 2005, Tom Duffy wrote:

> On Thu, 2005-02-24 at 16:16 -0500, James Lentini wrote:
>> Yesterday the DAT Collaborative voted to add the GPL license to the
>> DAPL Source Forge reference implementation.
>
> That is great news.  When is this effective?

The official vote was taken Wednesday, Feb. 23rd.

>
>> We are now in a position
>> to begin porting the DAPL SourceForge project to OpenIB. I could like
>> to hold a conference call to help plan and divide up this work.
>>
>>
>>           Date: Monday, February 28
>>           Time: 11:30 AM EST
>>       Domestic: 888-827-8686
>> International: 303-928-2620
>> Conference ID: 1125043
>
> The chances of getting us left-coasters in the office by 8:30AM on
> Monday morning seems unlikely.  I will try, but you may not get
> everybody.

I apologize for making it so early. I tried to find a time that would 
work for both the West Coast and Israel. I expect that this will be a 
one time event. I'll take notes an forward them to this reflector for 
those who are unable to attend.


>
> -tduffy
>


From mst at mellanox.co.il  Sun Feb 27 02:44:41 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 27 Feb 2005 12:44:41 +0200
Subject: [openib-general] [PATCH] sdp conditional code cleanup
In-Reply-To: <20050225115857.C13364@topspin.com>
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
	<20050225115857.C13364@topspin.com>
Message-ID: <20050227104441.GI23837@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: sdp: 0 > result
> 
> On Thu, Feb 24, 2005 at 11:49:28PM +0200, Michael S. Tsirkin wrote:
> > OK, now what about things like these:
> > 
> >         if (0 > result) {
> > 
> > may I change them all to
> > 
> >         if (result < 0) {
> > 
> > While being equivalent, we are testing the result, not 0.
> > 
> > Similiarly (although I feel somewhat less strongly about it)
> > 
> >         if (0 == result)
> > and
> >         if (NULL == conn)
> > 
> > are better off as
> > 
> > 	if (!result) {
> > and
> >         if (!conn)
> > 
> > C is a Spartan language, and this is more brief.
> > Libor, I think I asked about the second one, but dont recall you
> > answering.
> > If OK to both, let me know and I'll do it on Sunday.
> 
>   I actually feel more strongly in favour of making the second change
> you propose then the first. However, I'm OK with both, so feel free
> to submit a patch.
> 
> -Libor
> 

Here is the patch.

I generalized the approach - 
any if (CONSTANT == variable) and if ( CONSTANT & variable) is now
    if (variable == CONSTANT) and if ( variable & CONSTANT)
    and so forth.

Further, some places had weird flag testing code like this:
if ( (variable & CONSTANT) > 0 )

I changed them all to 

if (variable & CONSTANT)

same for linux test_bit macro usage.

While I touched some tests, I also cleaned some of them up: there is no reason
for if (!(result > 0)) when it can be a shorter and more clear as
    if (result <= 0),
and in a lot of cases ((!a) ? b : c) is better off as (a ? b : c)

Two things I noticed but did not fix:

A. In some places a positive error code is returned. For example
   ENOBUF and not -ENOBUF. I assume its a bug but did not touch it.

B. SDP_EXPECT seems to be used with double ((    )) in some places.
   This is probably an artefact from when the macro itself did not
   have () around the expression.

Looks like a lot of changes, I went over them several times and
everything looks in order to me. I really hope its applied before
any other change, it almost surely will conflict with any other
patch, and it'll be a lot of work to re-diff.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>


Index: sdp_queue.h
===================================================================
--- sdp_queue.h	(revision 1922)
+++ sdp_queue.h	(working copy)
@@ -83,7 +83,7 @@ static inline int __sdp_desc_q_size(stru
  */
 static inline int __sdp_desc_q_member(struct sdpc_desc *element)
 {
-	return ((NULL == element->table) ? 0 : 1);
+	return (element->table ? 1 : 0);
 }
 
 #define sdp_desc_q_size(x)   __sdp_desc_q_size(x)
Index: sdp_write.c
===================================================================
--- sdp_write.c	(revision 1922)
+++ sdp_write.c	(working copy)
@@ -88,23 +88,23 @@ int sdp_event_write(struct sdp_opt *conn
 	switch (type) {
 	case SDP_DESC_TYPE_BUFF:
 		buff = (struct sdpc_buff *) sdp_desc_q_get_head(&conn->w_snk);
-		SDP_EXPECT((NULL != buff));
+		SDP_EXPECT((buff));
 
 		conn->send_qud -= buff->data_size;
 
 		result = sdp_buff_pool_put(buff);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 
 		break;
 	case SDP_DESC_TYPE_IOCB:
 		iocb = (struct sdpc_iocb *)sdp_desc_q_look_head(&conn->w_snk);
-		if (NULL == iocb || iocb->wrid != comp->wr_id) {
+		if (!iocb || iocb->wrid != comp->wr_id) {
 
 			break;
 		}
 
 		iocb = (struct sdpc_iocb *)sdp_desc_q_get_head(&conn->w_snk);
-		SDP_EXPECT((NULL != iocb));
+		SDP_EXPECT((iocb));
 
 		iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_W);
 
@@ -112,7 +112,7 @@ int sdp_event_write(struct sdp_opt *conn
 		SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto error;
@@ -122,7 +122,7 @@ int sdp_event_write(struct sdp_opt *conn
 	case SDP_DESC_TYPE_NONE:
 		iocb = (struct sdpc_iocb *)sdp_desc_q_look_type_head(&conn->send_queue,
 								     SDP_DESC_TYPE_IOCB);
-		if (NULL == iocb) {
+		if (!iocb) {
 			sdp_dbg_warn(conn, 
 				     "No IOCB on write complete <%llu:%d:%d>",
 				     (unsigned long long)comp->wr_id,
@@ -158,7 +158,7 @@ int sdp_event_write(struct sdp_opt *conn
 	 * of some more sends.
 	 */
 	result = sdp_send_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue.", result);
 		goto error;
 	}
@@ -167,7 +167,7 @@ int sdp_event_write(struct sdp_opt *conn
 	 * reads.
 	 */
 	result = sdp_recv_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing recv queue.", result);
 		goto error;
 	}
Index: sdp_link.c
===================================================================
--- sdp_link.c	(revision 1922)
+++ sdp_link.c	(working copy)
@@ -47,7 +47,7 @@ static struct work_struct       _link_ti
 static u64 _path_lookup_id = 0;
 
 #define _SDP_PATH_LOOKUP_ID() \
-      ((0 == ++_path_lookup_id) ? ++_path_lookup_id : _path_lookup_id)
+      ((++_path_lookup_id) ? _path_lookup_id : ++_path_lookup_id)
 
 #define GID_FMT           "%x:%x:%x:%x:%x:%x:%x:%x"
 #define GID_ARG(gid)      be16_to_cpup((__be16 *) ((gid).raw +  0)), \
@@ -105,7 +105,7 @@ static void _sdp_path_wait_add(struct sd
 	info->wait_list = wait;
 	wait->pext      = &info->wait_list;
 
-	if (NULL != wait->next)
+	if (wait->next)
 		wait->next->pext = &wait->next;
 }
 
@@ -117,8 +117,8 @@ static void _sdp_path_wait_destroy(struc
 	/*
 	 * if it's in the list, pext will not be null
 	 */
-	if (NULL != wait->pext) {
-		if (NULL != wait->next)
+	if (wait->pext) {
+		if (wait->next)
 			wait->next->pext = wait->pext;
 
 		*(wait->pext) = wait->next;
@@ -153,7 +153,7 @@ static struct sdp_path_info *_sdp_path_i
 {
 	struct sdp_path_info *info;
 
-	for (info = _info_list; NULL != info; info = info->next)
+	for (info = _info_list; info; info = info->next)
 		if (dst_ip == info->dst && dev_if == info->dif)
 			break;
 
@@ -168,7 +168,7 @@ static struct sdp_path_info *_sdp_path_i
 	struct sdp_path_info *info;
 
 	info = kmem_cache_alloc(_info_cache, SLAB_KERNEL);
-	if (NULL == info)
+	if (!info)
 		return NULL;
 
 	memset(info, 0, sizeof(struct sdp_path_info));
@@ -177,7 +177,7 @@ static struct sdp_path_info *_sdp_path_i
 	_info_list = info;
 	info->pext = &_info_list;
 
-	if (NULL != info->next)
+	if (info->next)
 		info->next->pext = &info->next;
 
 	info->dst = dst_ip;
@@ -200,8 +200,8 @@ static void _sdp_path_info_destroy(struc
 	/*
 	 * if it's in the list, pext will not be null
 	 */
-	if (NULL != info->pext) {
-		if (NULL != info->next)
+	if (info->pext) {
+		if (info->next)
 			info->next->pext = info->pext;
 
 		*(info->pext) = info->next;
@@ -210,7 +210,7 @@ static void _sdp_path_info_destroy(struc
 		info->next = NULL;
 	}
 
-	while (NULL != (wait = info->wait_list))
+	while ((wait = info->wait_list))
 		_sdp_path_wait_complete(wait, info, status);
 
 	cancel_delayed_work(&info->timer);
@@ -244,13 +244,13 @@ static void _sdp_link_path_rec_done(int 
 	}
 
 	sweep = info->wait_list;
-	while (NULL != sweep) {
+	while (sweep) {
 		wait  = sweep;
 		sweep = sweep->next;
 		/*
 		 * on timeout increment retries.
 		 */
-		if (-ETIMEDOUT == status)
+		if (status == -ETIMEDOUT)
 			wait->retry++;
 
 		if (!status || SDP_LINK_SA_RETRY < wait->retry)
@@ -259,7 +259,7 @@ static void _sdp_link_path_rec_done(int 
 	/*
 	 * retry if anyone is waiting.
 	 */
-	if (NULL != info->wait_list) {
+	if (info->wait_list) {
 		info->sa_time = min(info->sa_time * 2, SDP_LINK_SA_TIME_MAX);
 
 		result = ib_sa_path_rec_get(info->ca,
@@ -275,7 +275,7 @@ static void _sdp_link_path_rec_done(int 
 					    info,
 					    &info->query);
 
-	        if (0 > result) {
+	        if (result < 0) {
 			sdp_dbg_warn(NULL, "Error <%d> restarting path query",
 				     result);
 			_sdp_path_info_destroy(info, result);
@@ -294,7 +294,7 @@ static int _sdp_link_path_rec_get(struct
 		     GID_ARG(info->path.sgid),
 		     GID_ARG(info->path.dgid));
 
-	if (SDP_LINK_F_PATH & info->flags)
+	if (info->flags & SDP_LINK_F_PATH)
 		return 0;
 
 	result = ib_sa_path_rec_get(info->ca,
@@ -309,7 +309,7 @@ static int _sdp_link_path_rec_get(struct
 				    _sdp_link_path_rec_done,
 				    info,
 				    &info->query);
-        if (0 > result) {
+        if (result < 0) {
 		sdp_dbg_warn(NULL, "Error <%d> starting path record query",
 			     result);
                 info->query = NULL;
@@ -351,16 +351,16 @@ void _sdp_link_path_lookup(void *data)
 	/*
 	 * path request in progress?
 	 */
-	if (NULL != info->query)
+	if (info->query)
 		goto done;
 	/*
 	 * route information present, but no path query, goto re-arp.
 	 */
-	if (NULL != info->ca)
+	if (info->ca)
 		goto arp;
 
 	result = ip_route_output_key(&rt, &fl);
-	if (0 > result || NULL == rt) {
+	if (result < 0 || !rt) {
 		sdp_dbg_warn(NULL, "Error <%d> routing <%08x:%08x> (%d)",
 			     result, info->dst, info->src, info->dif);
 		goto error;
@@ -368,7 +368,7 @@ void _sdp_link_path_lookup(void *data)
 	/*
 	 * check route flags
 	 */
-	if (0 < ((RTCF_MULTICAST|RTCF_BROADCAST) & rt->rt_flags)) {
+	if ((RTCF_MULTICAST|RTCF_BROADCAST) & rt->rt_flags) {
 		ip_rt_put(rt);
 		result = -ENETUNREACH;
 		goto error;
@@ -376,7 +376,7 @@ void _sdp_link_path_lookup(void *data)
 	/*
 	 * check that device is IPoIB
 	 */
-	if (NULL == rt->u.dst.neighbour || NULL == rt->u.dst.neighbour->dev) {
+	if (!rt->u.dst.neighbour || !rt->u.dst.neighbour->dev) {
 		sdp_dbg_warn(NULL, "No neighbour found for <%08x:%08x>",
 			     rt->rt_src, rt->rt_dst);
 
@@ -388,7 +388,7 @@ void _sdp_link_path_lookup(void *data)
 	 * handling.
 	 */
 	if (ARPHRD_INFINIBAND != rt->u.dst.neighbour->dev->type &&
-	    0 == (IFF_LOOPBACK & rt->u.dst.neighbour->dev->flags)) {
+	    !(rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK)) {
 		result = -ENETUNREACH;
 		goto error;
 	}
@@ -402,22 +402,22 @@ void _sdp_link_path_lookup(void *data)
 	 * In case of loopback find a valid IB device on which to
 	 * direct the loopback traffic.
 	 */
-	info->dev = ((IFF_LOOPBACK & rt->u.dst.neighbour->dev->flags) ?
+	info->dev = ((rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) ?
 		     ip_dev_find(rt->rt_src) : rt->u.dst.neighbour->dev);
 
 	info->gw  = rt->rt_gateway;
 	info->src = rt->rt_src;     /* true source IP address */
 
-	if (IFF_LOOPBACK & info->dev->flags)
-		while (NULL != (info->dev = dev_get_by_index(++counter))) {
+	if (info->dev->flags & IFF_LOOPBACK)
+		while ((info->dev = dev_get_by_index(++counter))) {
 
 			dev_put(info->dev);
 			if (ARPHRD_INFINIBAND == info->dev->type &&
-			    0 < (IFF_UP & info->dev->flags))
+			    (info->dev->flags & IFF_UP))
 				break;
 		}
 
-	if (NULL == info->dev) {
+	if (!info->dev) {
 		sdp_dbg_warn(NULL, "No device for IB comm <%s:%08x:%08x>",
 			     rt->u.dst.neighbour->dev->name,
 			     rt->u.dst.neighbour->dev->flags,
@@ -442,7 +442,7 @@ void _sdp_link_path_lookup(void *data)
 	 * If the routing device is loopback save the device address of
 	 * the IB device which was found.
 	 */
-	if (IFF_LOOPBACK & rt->u.dst.neighbour->dev->flags) {
+	if (rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) {
 		memcpy(&info->path.dgid,
 		       (info->dev->dev_addr + 4),
 		       sizeof(union ib_gid));
@@ -478,12 +478,12 @@ arp:
 	/*
 	 * start arp timer if it's not already going.
 	 */
-	if (SDP_LINK_F_ARP & info->flags) {
+	if (info->flags & SDP_LINK_F_ARP) {
 		struct sdp_path_wait *sweep;
 		struct sdp_path_wait *wait;
 
 		sweep = info->wait_list;
-		while (NULL != sweep) {
+		while (sweep) {
 			wait = sweep;
 			sweep = sweep->next;
 
@@ -492,7 +492,7 @@ arp:
 							-ETIMEDOUT);
 		}
 
-		if (NULL == info->wait_list) {
+		if (!info->wait_list) {
 			result = -ETIMEDOUT;
 			goto error;
 		}
@@ -547,9 +547,9 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	 * lookup entry, create if not found and add to wait list.
 	 */
 	info = _sdp_path_info_lookup(dst_addr, bound_dev_if);
-	if (NULL == info) {
+	if (!info) {
 		info = _sdp_path_info_create(dst_addr, bound_dev_if);
-		if (NULL == info) {
+		if (!info) {
 			sdp_dbg_warn(NULL, "Failed to create path object");
 			return -ENOMEM;
 		}
@@ -560,7 +560,7 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	/*
 	 * if not waiting for result, complete.
 	 */
-	if (SDP_LINK_F_VALID & info->flags) {
+	if (info->flags & SDP_LINK_F_VALID) {
 		_sdp_link_path_complete(*id, 0, info, completion, arg);
 		return 0;
 	}
@@ -568,7 +568,7 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	 * add request to list of lookups.
 	 */
 	wait = kmem_cache_alloc(_wait_cache, SLAB_KERNEL);
-	if (NULL == wait) {
+	if (!wait) {
 		sdp_dbg_warn(NULL, "Failed to create path wait object");
 		result = -ENOMEM;
 		goto error;
@@ -603,7 +603,7 @@ static void _sdp_link_sweep(void *data)
 	struct sdp_path_info *sweep;
 
 	sweep = _info_list; 
-	while (NULL != sweep) {
+	while (sweep) {
 		info  = sweep;
 		sweep = sweep->next;
 
@@ -649,16 +649,16 @@ static void _sdp_link_arp_work(void *dat
 	/*
 	 * find a path info structure for the source IP address.
 	 */
-	for (info = _info_list; NULL != info; info = info->next)
+	for (info = _info_list; info; info = info->next)
 		if (info->dst == arp->src_ip)
 			break;
 
-	if (NULL == info)
+	if (!info)
 		goto done;
 	/*
 	 * update record info, and request new path record data.
 	 */
-	if (SDP_LINK_F_ARP & info->flags) {
+	if (info->flags & SDP_LINK_F_ARP) {
 		cancel_delayed_work(&info->timer);
 		info->flags &= ~SDP_LINK_F_ARP;
 	}
@@ -696,7 +696,7 @@ static int _sdp_link_arp_recv(struct sk_
 	 * queue IB arp packet onto work queue.
 	 */
 	work = kmalloc(sizeof(*work), GFP_ATOMIC);
-	if (NULL == work)
+	if (!work)
 		goto done;
 
 	work->arg = skb;
@@ -734,7 +734,7 @@ int sdp_link_addr_init(void)
 						      _tsSdpAsyncEventFunc,
 						      NULL,
 						      &_tsSdpAsyncHandle);
-	if (0 != result) {
+	if (result) {
 		
 		goto error_async;
 	}
@@ -743,7 +743,7 @@ int sdp_link_addr_init(void)
 					sizeof(struct sdp_path_info),
 					0, SLAB_HWCACHE_ALIGN,
 					NULL, NULL);
-	if (NULL == _info_cache) {
+	if (!_info_cache) {
 		sdp_warn("Failed to allocate path info cache.");
 
 		result = -ENOMEM;
@@ -754,7 +754,7 @@ int sdp_link_addr_init(void)
 					sizeof(struct sdp_path_wait),
 					0, SLAB_HWCACHE_ALIGN,
 					NULL, NULL);
-	if (NULL == _wait_cache) {
+	if (!_wait_cache) {
 		sdp_warn("Failed to allocate path wait cache.");
 
 		result = -ENOMEM;
@@ -762,7 +762,7 @@ int sdp_link_addr_init(void)
 	}
 
 	_link_wq = create_workqueue("sdp_wq");
-	if (NULL == _link_wq) {
+	if (!_link_wq) {
 		sdp_warn("Failed to allocate ARP wait queue.");
 
 		result = -ENOMEM;
@@ -798,7 +798,7 @@ int sdp_link_addr_cleanup(void)
 	/*
 	 * clear objects
 	 */
-	while (NULL != (info = _info_list))
+	while ((info = _info_list))
 		_sdp_path_info_destroy(info, -EINTR);
 	/*
 	 * remove ARP packet processing.
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1922)
+++ sdp_rcvd.c	(working copy)
@@ -54,7 +54,7 @@ static int _sdp_rcvd_disconnect(struct s
 		 * initiate disconnect to framework
 		 */
 		result = sdp_wall_recv_close(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> closing connection",
 				     result);
 			goto error;
@@ -79,7 +79,7 @@ static int _sdp_rcvd_disconnect(struct s
 		 * disconnect)
 		 */
 		result = sdp_wall_recv_closing(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> confirming conn close",
 				     result);
 			goto error;
@@ -92,10 +92,10 @@ static int _sdp_rcvd_disconnect(struct s
 		 * if the remote DREQ was already received, but unprocessed, 
 		 * do not treat it as an error
 		 */
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
 
-			if (-EPROTO != result)
+			if (result != -EPROTO)
 				goto error;
 		}
 
@@ -131,7 +131,7 @@ static int _sdp_rcvd_abort(struct sdp_op
 	case SDP_CONN_ST_DIS_SEND_2:
 	case SDP_CONN_ST_DIS_SENT_2:
 		result = sdp_wall_abort(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> during abort", result);
 
 		break;
@@ -163,19 +163,19 @@ static int _sdp_rcvd_send_sm(struct sdp_
 	sdp_dbg_ctrl(conn, "SendSM msg. active <%d> count <%d> flags <%08x>",
 		     conn->src_sent, conn->src_cncl, conn->flags);
 
-	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags) &&
+	if ((conn->flags & SDP_CONN_F_SRC_CANCEL_L) &&
 	    SDP_WRAP_GTE(buff->bsdh_hdr->seq_ack, conn->src_cseq)) {
 		/*
 		 * drain the active source queue
 		 */
-		while (NULL != (iocb = sdp_iocb_q_get_tail(&conn->w_src))) {
-			SDP_EXPECT((0 < (SDP_IOCB_F_ACTIVE & iocb->flags)));
-			SDP_EXPECT((0 < (SDP_IOCB_F_CANCEL & iocb->flags)));
+		while ((iocb = sdp_iocb_q_get_tail(&conn->w_src))) {
+			SDP_EXPECT((iocb->flags & SDP_IOCB_F_ACTIVE));
+			SDP_EXPECT((iocb->flags & SDP_IOCB_F_CANCEL));
 
 			conn->src_sent--;
 
 			result = sdp_iocb_complete(iocb, 0);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 		/*
 		 * Cancel complete, clear the state.
@@ -201,13 +201,13 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	 * lookup active IOCB read.
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_snk);
-	if (NULL == iocb) {
+	if (!iocb) {
 		sdp_dbg_warn(conn, "Cannot find IOCB for Write Completion.");
 		result = -EPROTO;
 		goto error;
 	}
 
-	SDP_EXPECT((0 < (SDP_IOCB_F_RDMA_W & iocb->flags)));
+	SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_W));
 
 	sdp_dbg_data(conn, "Write <%d> size <%d:%d:%Zu> mode <%d> active <%d>",
 		     iocb->key, rwch->size, iocb->len, iocb->size,
@@ -225,7 +225,7 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	 * Iocb is done, deregister memory, and generate completion.
 	 */
 	iocb = sdp_iocb_q_get_head(&conn->r_snk);
-	SDP_EXPECT((NULL != iocb));
+	SDP_EXPECT((iocb));
 
 	conn->snk_sent--;
 
@@ -237,7 +237,7 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
 
 	result = sdp_iocb_complete(iocb, 0);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 			     result, iocb->key);
 		goto error;
@@ -262,7 +262,7 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 * lookup IOCB read.
 	 */
 	iocb = sdp_iocb_q_look(&conn->w_src);
-	if (NULL == iocb) {
+	if (!iocb) {
 		sdp_dbg_warn(conn, "Cannot find IOCB for Read Completion.");
 		result = -EPROTO;
 		goto error;
@@ -289,19 +289,19 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 * it on the first successful RDMA to make sure we don't get a
 	 * false positive of data sent. (specification ambiguity/pain)
 	 */
-	iocb->post += (0 == iocb->post) ? (iocb->size - iocb->len) : 0;
+	iocb->post += iocb->post ? 0 : (iocb->size - iocb->len);
 	iocb->len -= rrch->size;
 	iocb->post += rrch->size;
 
 	conn->send_pipe -= rrch->size;
-	conn->oob_offset -= (0 < conn->oob_offset) ? rrch->size : 0;
+	conn->oob_offset -= (conn->oob_offset > 0) ? rrch->size : 0;
 
 	/*
 	 * If iocb is done, deregister memory, and generate completion.
 	 */
-	if (!(0 < iocb->len)) {
+	if (iocb->len <= 0) {
 		iocb = sdp_iocb_q_get_head(&conn->w_src);
-		SDP_EXPECT((NULL != iocb));
+		SDP_EXPECT((iocb));
 
 		conn->src_sent--;
 
@@ -309,7 +309,7 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 		SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto error;
@@ -319,8 +319,8 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	 * If Source Cancel was in process, and there are no more outstanding
 	 * advertisments, then it should now be cleared.
 	 */
-	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags) &&
-	    0 == sdp_iocb_q_size(&conn->w_src)) {
+	if ((conn->flags & SDP_CONN_F_SRC_CANCEL_L) &&
+	    !sdp_iocb_q_size(&conn->w_src)) {
 		conn->src_cncl = 0;
 		conn->flags &= ~(SDP_CONN_F_SRC_CANCEL_L);
 	}
@@ -347,7 +347,7 @@ static int _sdp_rcvd_mode_change(struct 
 	 * Check if the mode change is to the same mode.
 	 */
 	if (((SDP_MSG_MCH_GET_MODE(mch) & 0x7) ==
-	     ((0 < (SDP_MSG_MCH_GET_MODE(mch) & 0x8)) ?
+	     ((SDP_MSG_MCH_GET_MODE(mch) & 0x8) ?
 	      conn->send_mode : conn->recv_mode))) {
 		sdp_dbg_warn(conn, "Mode transition <%d> is a nop. <%d:%d>",
 			     SDP_MSG_MCH_GET_MODE(mch), conn->recv_mode,
@@ -360,12 +360,12 @@ static int _sdp_rcvd_mode_change(struct 
 	 */
 	switch (SDP_MSG_MCH_GET_MODE(mch)) {
 	case SDP_MSG_MCH_BUFF_RECV:	/* source to sink */
-		if (SDP_MODE_COMB != conn->recv_mode) {
+		if (conn->recv_mode != SDP_MODE_COMB) {
 			result = -EPROTO;
 			goto mode_error;
 		}
 
-		if (0 < conn->src_recv) {
+		if (conn->src_recv > 0) {
 			sdp_dbg_warn(conn, "mode error <%d> src pending <%d>",
 				     SDP_MSG_MCH_GET_MODE(mch),
 				     conn->src_recv);
@@ -375,21 +375,21 @@ static int _sdp_rcvd_mode_change(struct 
 
 		break;
 	case SDP_MSG_MCH_COMB_SEND:	/* sink to source */
-		if (SDP_MODE_BUFF != conn->send_mode) {
+		if (conn->send_mode != SDP_MODE_BUFF) {
 			result = -EPROTO;
 			goto mode_error;
 		}
 
 		break;
 	case SDP_MSG_MCH_PIPE_RECV:	/* source to sink */
-		if (SDP_MODE_COMB != conn->recv_mode) {
+		if (conn->recv_mode != SDP_MODE_COMB) {
 			result = -EPROTO;
 			goto mode_error;
 		}
 
 		break;
 	case SDP_MSG_MCH_COMB_RECV:	/* source to sink */
-		if (SDP_MODE_PIPE != conn->recv_mode) {
+		if (conn->recv_mode != SDP_MODE_PIPE) {
 			result = -EPROTO;
 			goto mode_error;
 		}
@@ -411,7 +411,7 @@ static int _sdp_rcvd_mode_change(struct 
 	/*
 	 * assign new mode
 	 */
-	if (0 < (SDP_MSG_MCH_GET_MODE(mch) & 0x8))
+	if (SDP_MSG_MCH_GET_MODE(mch) & 0x8)
 		conn->send_mode = SDP_MSG_MCH_GET_MODE(mch) & 0x7;
 	else
 		conn->recv_mode = SDP_MSG_MCH_GET_MODE(mch) & 0x7;
@@ -438,7 +438,7 @@ static int _sdp_rcvd_src_cancel(struct s
 	 * If there are no outstanding advertisments, then there is nothing
 	 * to do.
 	 */
-	if (!(0 < conn->src_recv)) {
+	if (conn->src_recv <= 0) {
 		sdp_dbg_warn(conn, "No SrcAvail advertisments to cancel.");
 		result = 0;
 		goto done;
@@ -448,12 +448,12 @@ static int _sdp_rcvd_src_cancel(struct s
 	 * if it's already processed data.
 	 */
 	advt = sdp_advt_q_look(&conn->src_pend);
-	if (NULL != advt && 0 < advt->post) {
+	if (advt && advt->post > 0) {
 		/*
 		 * If active, move to the active queue. Otherwise generate an
 		 * immediate completion
 		 */
-		if (0 < (SDP_ADVT_F_READ & advt->flag)) {
+		if (advt->flag & SDP_ADVT_F_READ) {
 
 			sdp_advt_q_put(&conn->src_actv,
 				       sdp_advt_q_get(&conn->src_pend));
@@ -463,7 +463,7 @@ static int _sdp_rcvd_src_cancel(struct s
 			conn->flags |= SDP_CONN_F_SRC_CANCEL_C;
 		} else {
 			result = sdp_send_ctrl_rdma_rd(conn, advt->post);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> read completion",
 					     result);
@@ -474,19 +474,19 @@ static int _sdp_rcvd_src_cancel(struct s
 	/*
 	 * drop the pending advertisment queue.
 	 */
-	while (NULL != (advt = sdp_advt_q_get(&conn->src_pend))) {
+	while ((advt = sdp_advt_q_get(&conn->src_pend))) {
 		conn->flags |= SDP_CONN_F_SRC_CANCEL_C;
 
 		conn->src_recv--;
 
 		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 	}
 	/*
 	 * If there are active reads, mark the connection as being in
 	 * source cancel. Otherwise
 	 */
-	if (0 < sdp_advt_q_size(&conn->src_actv)) {
+	if (sdp_advt_q_size(&conn->src_actv) > 0) {
 		/*
 		 * Set flag. Adjust sequence number ack. (spec dosn't want the
 		 * seq ack in subsequent messages updated until the cancel has
@@ -499,9 +499,9 @@ static int _sdp_rcvd_src_cancel(struct s
 		/*
 		 * If a source was dropped, generate an ack.
 		 */
-		if (0 < (SDP_CONN_F_SRC_CANCEL_C & conn->flags)) {
+		if (conn->flags & SDP_CONN_F_SRC_CANCEL_C) {
 			result = sdp_send_ctrl_send_sm(conn);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, "Error<%d> posting SendSm",
 					     result);
 				goto done;
@@ -528,7 +528,7 @@ static int _sdp_rcvd_snk_cancel(struct s
 	 * If there are no outstanding advertisments, they we've completed
 	 * since the message was sent, and there is nothing to do.
 	 */
-	if (!(0 < conn->snk_recv)) {
+	if (conn->snk_recv <= 0) {
 		sdp_dbg_warn(conn, "No SnkAvail advertisments to cancel.");
 		result = 0;
 		goto done;
@@ -540,12 +540,12 @@ static int _sdp_rcvd_snk_cancel(struct s
 	 * it's possible to handle the entire Cancel here.
 	 */
 	advt = sdp_advt_q_look(&conn->snk_pend);
-	if (NULL != advt && 0 < advt->post) {
+	if (advt && advt->post > 0) {
 		/*
 		 * Generate completion
 		 */
 		result = sdp_send_ctrl_rdma_wr(conn, advt->post);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> write completion",
 				     result);
 			goto done;
@@ -562,20 +562,20 @@ static int _sdp_rcvd_snk_cancel(struct s
 	/*
 	 * drain the advertisments which have yet to be processed.
 	 */
-	while (NULL != (advt = sdp_advt_q_get(&conn->snk_pend))) {
+	while ((advt = sdp_advt_q_get(&conn->snk_pend))) {
 		counter++;
 		conn->snk_recv--;
 
 		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 	}
 	/*
 	 * A cancel ack is sent only if we cancelled an advertisment without
 	 * sending a completion
 	 */
-	if (0 < counter) {
+	if (counter > 0) {
 		result = sdp_send_ctrl_snk_cancel_ack(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> SnkCacelAck response",
 				     result);
 			goto done;
@@ -599,7 +599,7 @@ static int _sdp_rcvd_snk_cancel_ack(stru
 	sdp_dbg_ctrl(conn, "Sink Cancel Ack. actv <%d> mode <%d> flag <%08x>",
 		     conn->snk_sent, conn->recv_mode, conn->flags);
 
-	if (0 == (SDP_CONN_F_SNK_CANCEL & conn->flags)) {
+	if (!(conn->flags & SDP_CONN_F_SNK_CANCEL)) {
 		sdp_dbg_warn(conn, "Connection not in sink cancel mode <%08x>",
 			     conn->flags);
 		result = -EPROTO;
@@ -608,11 +608,11 @@ static int _sdp_rcvd_snk_cancel_ack(stru
 	/*
 	 * drain and complete all active IOCBs
 	 */
-	while (NULL != (iocb = sdp_iocb_q_get_head(&conn->r_snk))) {
+	while ((iocb = sdp_iocb_q_get_head(&conn->r_snk))) {
 		conn->snk_sent--;
 
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto done;
@@ -648,7 +648,7 @@ static int _sdp_rcvd_resize_buff_ack(str
 		     crbh->size, conn->recv_size);
 
 	result = sdp_send_ctrl_resize_buff_ack(conn, conn->recv_size);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> acking size change request",
 			     result);
 		goto error;
@@ -695,7 +695,7 @@ static int _sdp_rcvd_snk_avail(struct sd
 	/*
 	 * check our send mode, and make sure parameters are within reason.
 	 */
-	if (SDP_MODE_PIPE != conn->send_mode) {
+	if (conn->send_mode != SDP_MODE_PIPE) {
 		sdp_dbg_warn(conn, "SinkAvail, incorrect source mode <%d>",
 			     conn->send_mode);
 		result = -EPROTO;
@@ -727,9 +727,9 @@ static int _sdp_rcvd_snk_avail(struct sd
 	 * If there are outstanding SrcAvail messages, they are now 
 	 * invalid and the queue needs to be fixed up.
 	 */
-	if (0 < conn->src_sent) {
-		while (NULL != (iocb = sdp_iocb_q_get_tail(&conn->w_src))) {
-			SDP_EXPECT((0 < (SDP_IOCB_F_ACTIVE & iocb->flags)));
+	if (conn->src_sent > 0) {
+		while ((iocb = sdp_iocb_q_get_tail(&conn->w_src))) {
+			SDP_EXPECT((iocb->flags & SDP_IOCB_F_ACTIVE));
 
 			iocb->flags &= ~SDP_IOCB_F_ACTIVE;
 			conn->src_sent--;
@@ -739,21 +739,21 @@ static int _sdp_rcvd_snk_avail(struct sd
 			 * in cancel processing they need to be
 			 * completed.
 			 */
-			if (0 == (SDP_IOCB_F_CANCEL & iocb->flags)) {
+			if (!(iocb->flags & SDP_IOCB_F_CANCEL)) {
 				result = sdp_desc_q_put_head(&conn->send_queue,
 							     (struct sdpc_desc *)
 							     iocb);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			} else {
 				result = sdp_iocb_complete(iocb, 0);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			}
 		}
 		/*
 		 * If Source Cancel was in process, it should now 
 		 * be cleared.
 		 */
-		if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags)) {
+		if (conn->flags & SDP_CONN_F_SRC_CANCEL_L) {
 			conn->src_cncl = 0;
 			conn->flags &= ~(SDP_CONN_F_SRC_CANCEL_L);
 		}
@@ -762,7 +762,7 @@ static int _sdp_rcvd_snk_avail(struct sd
 	 * create and queue new advertisment
 	 */
 	advt = sdp_advt_create();
-	if (NULL == advt) {
+	if (!advt) {
 		sdp_dbg_warn(conn, "SrcAvail cannot be copied.");
 		result = -ENOMEM;
 		goto error;
@@ -786,9 +786,9 @@ consume:
 	/*
 	 * consume any data in the advertisment for the other direction.
 	 */
-	if (0 < (buff->tail - buff->data)) {
+	if ((buff->tail - buff->data) > 0) {
 		result = sdp_recv_buff(conn, buff);
-		if (0 < result)
+		if (result > 0)
 			/*
 			 * count number of bytes buffered by the connection,
 			 * zero byte buffers or errors can be returned, the
@@ -796,7 +796,7 @@ consume:
 			 */
 			conn->byte_strm += result;
 		else
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, "Error <%d> buffer recv",
 					     result);
 	} else
@@ -828,7 +828,7 @@ static int _sdp_rcvd_src_avail(struct sd
 		     srcah->size, srcah->r_key, size,
 		     (unsigned long long) srcah->addr, conn->recv_mode);
 
-	if (0 < conn->snk_sent) {
+	if (conn->snk_sent > 0) {
 		/*
 		 * crossed SrcAvail and SnkAvail, the source message is 
 		 * discarded.
@@ -839,7 +839,7 @@ static int _sdp_rcvd_src_avail(struct sd
 		goto done;
 	}
 
-	if (0 < (SDP_CONN_F_SRC_CANCEL_R & conn->flags)) {
+	if (conn->flags & SDP_CONN_F_SRC_CANCEL_R) {
 		sdp_dbg_warn(conn, "SrcAvail during SrcAvailCancel. <%d>",
 			     conn->src_recv);
 		result = -EFAULT;
@@ -850,14 +850,14 @@ static int _sdp_rcvd_src_avail(struct sd
 	 * if data is received and the receive half of the connection has been
 	 * closed. This notifies the peer that the data was not received.
 	 */
-	if (0 < (RCV_SHUTDOWN & conn->shutdown)) {
+	if (RCV_SHUTDOWN & conn->shutdown) {
 		sdp_dbg_warn(conn, "SrcAvail, receive path closed <%02x>",
 			     conn->shutdown);
 		/*
 		 * abort connection (send reset)
 		 */
 		result = sdp_wall_abort(conn);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 		/*
 		 * drop packet
 		 */
@@ -868,7 +868,7 @@ static int _sdp_rcvd_src_avail(struct sd
 	 * save the advertisment
 	 */
 	advt = sdp_advt_create();
-	if (NULL == advt) {
+	if (!advt) {
 		sdp_dbg_warn(conn, "SrcAvail cannot be copied.");
 		result = -ENOMEM;
 		goto done;
@@ -886,8 +886,8 @@ static int _sdp_rcvd_src_avail(struct sd
 
 		break;
 	case SDP_MODE_COMB:
-		if (0 < conn->src_recv ||
-		    !(0 < size) ||
+		if (conn->src_recv > 0 ||
+		    size <= 0 ||
 		    !(srcah->size > size)) {
 			sdp_dbg_warn(conn, 
 				     "SrcAvail mode <%d> mismatch. <%d:%d:%d>",
@@ -910,7 +910,7 @@ static int _sdp_rcvd_src_avail(struct sd
 		break;
 	case SDP_MODE_PIPE:
 		if (SDP_MSG_MAX_ADVS == (conn->src_recv + conn->snk_recv) ||
-		    0 != size) {
+		    size) {
 			sdp_dbg_warn(conn,
 				     "SrcAvail mode <%d> mismatch. <%d:%d>",
 				     conn->recv_mode,
@@ -941,7 +941,7 @@ static int _sdp_rcvd_src_avail(struct sd
 	/*
 	 * process any ULP data in the message
 	 */
-	if (0 == size) {
+	if (!size) {
 		result = 0;
 		goto done;
 	}
@@ -952,7 +952,7 @@ static int _sdp_rcvd_src_avail(struct sd
 
 	if (!(SDP_SRC_AVAIL_RECV > size)) {
 		result = sdp_recv_buff(conn, buff);
-		if (0 < result)
+		if (result > 0)
 			/*
 			 * count number of bytes buffered by the
 			 * connection, zero byte buffers or errors
@@ -961,7 +961,7 @@ static int _sdp_rcvd_src_avail(struct sd
 			 */
 			conn->byte_strm += result;
 		else
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, "Error <%d> buffer recv",
 					     result);
 	} else
@@ -990,7 +990,7 @@ static int _sdp_rcvd_data(struct sdp_opt
 	 * If we are processing a SrcAvail, there should be no
 	 * buffered data
 	 */
-	if (0 < conn->src_recv) {
+	if (conn->src_recv > 0) {
 		sdp_dbg_warn(conn, "Error, recv'd data with SrcAvail active.");
 		return -EPROTO;
 	}
@@ -1010,7 +1010,7 @@ static int _sdp_rcvd_data(struct sdp_opt
 	conn->nond_recv++;
 
 	ret_val = sdp_recv_buff(conn, buff);
-	if (0 > ret_val)
+	if (ret_val < 0)
 		sdp_dbg_warn(conn, "Error <%d> processing buff recv", ret_val);
 	/*
 	 * result contains the number of bytes in the buffer which
@@ -1122,7 +1122,7 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 * get data
 	 */
 	buff = sdp_buff_q_get_head(&conn->recv_post);
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "receive event, but no posted receive?!");
 		result = -EINVAL;
 		goto done;
@@ -1167,7 +1167,7 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 * SrcAvailCancel message has been processed.
 	 */
 	conn->recv_seq = buff->bsdh_hdr->seq_num;
-	conn->advt_seq = (((SDP_CONN_F_SRC_CANCEL_R & conn->flags) > 0) ?
+	conn->advt_seq = ((conn->flags & SDP_CONN_F_SRC_CANCEL_R) ?
 			  conn->advt_seq : conn->recv_seq);
 	/*
 	 * buffers advertised minus the difference in buffer count between
@@ -1194,8 +1194,8 @@ int sdp_event_recv(struct sdp_opt *conn,
 	else {
 		offset = buff->bsdh_hdr->mid & 0x1F;
 
-		if (!(offset < SDP_MSG_EVENT_TABLE_SIZE) ||
-		    NULL == recv_event_funcs[offset]) {
+		if (offset >= SDP_MSG_EVENT_TABLE_SIZE ||
+		    !recv_event_funcs[offset]) {
 			sdp_dbg_warn(conn, "receive event, unknown MID <%d>",
 				     buff->bsdh_hdr->mid);
 			result = -EINVAL;
@@ -1210,22 +1210,22 @@ int sdp_event_recv(struct sdp_opt *conn,
 	/*
 	 * process result.
 	 */
-	if (0 == result) {
+	if (!result) {
 		result = sdp_buff_pool_put(buff);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 		/*
 		 * If this buffer was consumed, then make sure sufficient
 		 * recv buffers are posted. Also we might be able to move
 		 * data with a new RDMA SrcAvail advertisment.
 		 */
 		result = sdp_recv_flush(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> flushing recv queue",
 				     result);
 			goto done;
 		}
 	} else
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, 
 				     "receive event, dispatch error. <%d>",
 				     result);
@@ -1242,7 +1242,7 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 * recv window and we can flush buffered send data
 	 */
 	result = sdp_send_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue",
 			     result);
 		goto done;
Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1922)
+++ sdp_inet.c	(working copy)
@@ -103,17 +103,17 @@ void sdp_inet_wake_send(struct sock *sk)
 {
 	struct sdp_opt *conn;
 
-	if (NULL == sk || NULL == (conn = SDP_GET_CONN(sk)))
+	if (!sk || !(conn = SDP_GET_CONN(sk)))
 		return;
 
-	if (NULL != sk->sk_socket &&
-	    0 < test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) &&
-	    0 < __sdp_inet_writable(conn)) {
+	if (sk->sk_socket &&
+	    test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) &&
+	    __sdp_inet_writable(conn)) {
 		read_lock(&sk->sk_callback_lock);
 		clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
 
-		if (NULL != sk->sk_sleep &&
-		    0 < waitqueue_active(sk->sk_sleep))
+		if (sk->sk_sleep &&
+		    waitqueue_active(sk->sk_sleep))
 			wake_up_interruptible(sk->sk_sleep);
 		/*
 		 * test, clear, and notify. SOCK_ASYNC_NOSPACE
@@ -128,10 +128,10 @@ void sdp_inet_wake_send(struct sock *sk)
  */
 void sdp_inet_wake_generic(struct sock *sk)
 {
-	if (NULL != sk) {
+	if (sk) {
 		read_lock(&sk->sk_callback_lock);
 
-		if (NULL != sk->sk_sleep && waitqueue_active(sk->sk_sleep))
+		if (sk->sk_sleep && waitqueue_active(sk->sk_sleep))
 			wake_up_interruptible_all(sk->sk_sleep);
 
 		read_unlock(&sk->sk_callback_lock);
@@ -143,9 +143,9 @@ void sdp_inet_wake_generic(struct sock *
  */
 void sdp_inet_wake_recv(struct sock *sk, int len)
 {
-	if (NULL != sk) {
+	if (sk) {
 		read_lock(&sk->sk_callback_lock);
-		if (NULL != sk->sk_sleep)
+		if (sk->sk_sleep)
 			wake_up_interruptible(sk->sk_sleep);
 
 		sk_wake_async(sk, 1, POLL_IN);
@@ -158,9 +158,9 @@ void sdp_inet_wake_recv(struct sock *sk,
  */
 void sdp_inet_wake_error(struct sock *sk)
 {
-	if (NULL != sk) {
+	if (sk) {
 		read_lock(&sk->sk_callback_lock);
-		if (NULL != sk->sk_sleep)
+		if (sk->sk_sleep)
 			wake_up_interruptible(sk->sk_sleep);
 
 		sk_wake_async(sk, 0, POLL_ERR);
@@ -177,7 +177,7 @@ void sdp_inet_wake_urg(struct sock *sk)
 	 * pid for SIGURG/SIGIO has been set. On positive send signal to
 	 * process, on negative send signal to processes group.
 	 */
-	if (NULL != sk)
+	if (sk)
 		sk_send_sigurg(sk);
 }
 
@@ -205,7 +205,7 @@ static int _sdp_inet_abort(struct sdp_op
 	case SDP_SOCK_ST_DISCONNECT:
 	case SDP_SOCK_ST_CLOSING:
 		result = sdp_wall_abort(conn);
-		if (0 > result) {
+		if (result < 0) {
 
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
@@ -242,7 +242,7 @@ static int _sdp_inet_disconnect(struct s
 	switch (conn->istate) {
 	case SDP_SOCK_ST_CONNECT:
 		result = sdp_wall_abort(conn);
-		if (0 > result) {
+		if (result < 0) {
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
 			conn->istate = SDP_SOCK_ST_ERROR;
@@ -252,7 +252,7 @@ static int _sdp_inet_disconnect(struct s
 	case SDP_SOCK_ST_ACCEPTED:
 		conn->istate = SDP_SOCK_ST_DISCONNECT;
 		result = sdp_wall_send_close(conn);
-		if (0 > result) {
+		if (result < 0) {
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
 			conn->istate = SDP_SOCK_ST_ERROR;
@@ -261,7 +261,7 @@ static int _sdp_inet_disconnect(struct s
 	case SDP_SOCK_ST_CLOSE:
 		conn->istate = SDP_SOCK_ST_CLOSING;
 		result = sdp_wall_send_closing(conn);
-		if (0 > result) {
+		if (result < 0) {
 			result = -ECONNABORTED;
 			SDP_CONN_SET_ERR(conn, ECONNABORTED);
 			conn->istate = SDP_SOCK_ST_ERROR;
@@ -306,7 +306,7 @@ static int _sdp_inet_release(struct sock
 	long timeout;
 	u32  flags;
 
-	if (NULL == sock->sk) {
+	if (!sock->sk) {
 		sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>",
 			     sock->type, sock->state, sock->flags);
 		return 0;
@@ -326,12 +326,12 @@ static int _sdp_inet_release(struct sock
 	sdp_conn_lock(conn);
 	conn->shutdown = SHUTDOWN_MASK;
 
-	if (SDP_SOCK_ST_LISTEN == conn->istate) {
+	if (conn->istate == SDP_SOCK_ST_LISTEN) {
 		/*
 		 * stop listening
 		 */
 		result = sdp_inet_listen_stop(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> while releasing listen",
 				     result);
 
@@ -341,7 +341,7 @@ static int _sdp_inet_release(struct sock
 	 * get blocking nature of the socket.
 	 */
 	if (sock->file)
-		flags = (0 < (sock->file->f_flags & O_NONBLOCK)) ? \
+		flags = (sock->file->f_flags & O_NONBLOCK) ? \
 			MSG_DONTWAIT : 0;
 	else
 		flags = 0;
@@ -350,15 +350,15 @@ static int _sdp_inet_release(struct sock
 	 * and consider this an abort. Otherwise consider
 	 * this a gracefull close.
 	 */
-	if (0 < sdp_buff_q_size(&conn->recv_pool) ||
-	    0 < conn->src_recv ||
-	    (0 < sock_flag(sk, SOCK_LINGER) &&
-	     0 == sk->sk_lingertime)) {
+	if (sdp_buff_q_size(&conn->recv_pool) > 0 ||
+	    conn->src_recv > 0 ||
+	    (sock_flag(sk, SOCK_LINGER) &&
+	     !sk->sk_lingertime)) {
 		/*
 		 * abort.
 		 */
 		result = _sdp_inet_abort(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> while aborting socket",
 				     result);
 
@@ -368,7 +368,7 @@ static int _sdp_inet_release(struct sock
 	 * disconnect. (state dependant)
 	 */
 	result = _sdp_inet_disconnect(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> while disconnecting socket",
 			     result);
 		goto done;
@@ -377,8 +377,8 @@ static int _sdp_inet_release(struct sock
 	 * Skip lingering/canceling if 
 	 * non-blocking and not exiting.
 	 */
-	if (0 == (MSG_DONTWAIT & flags) ||
-	    0 < (PF_EXITING & current->flags)) {
+	if (!(MSG_DONTWAIT & flags) ||
+	    (PF_EXITING & current->flags)) {
 		/*
 		 * Wait if linger is set and
 		 * process is not exiting.
@@ -391,8 +391,8 @@ static int _sdp_inet_release(struct sock
 			add_wait_queue(sk->sk_sleep, &wait);
 			set_current_state(TASK_INTERRUPTIBLE);
 
-			while (0 < timeout &&
-			       0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
+			while (timeout > 0 &&
+			       !(SDP_ST_MASK_CLOSED & conn->istate)) {
 				sdp_conn_unlock(conn);
 				timeout = schedule_timeout(timeout);
 				sdp_conn_lock(conn);
@@ -410,18 +410,18 @@ static int _sdp_inet_release(struct sock
 		 * Cancel write and close again to force closing the
 		 * connection.
 		 */
-		if (0 < (SDP_ST_MASK_DRAIN & conn->istate)) {
+		if (SDP_ST_MASK_DRAIN & conn->istate) {
 
 			sdp_iocb_q_cancel_all_write(conn, -ECANCELED);
 
 			result = _sdp_inet_disconnect(conn);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 #endif
 	}
 
 done:
-	if (0 < (SDP_ST_MASK_CLOSED & conn->istate))
+	if (SDP_ST_MASK_CLOSED & conn->istate)
 		do {} while(0); /* pass */
 		
 	/*
@@ -479,15 +479,15 @@ static int _sdp_inet_bind(struct socket 
 	 * check bind permission for low ports.
 	 */
 	bind_port = ntohs(addr->sin_port);
-	if (0 < bind_port &&
-	    bind_port < PROT_SOCK && 0 == capable(CAP_NET_BIND_SERVICE))
+	if (bind_port > 0 &&
+	    bind_port < PROT_SOCK && !capable(CAP_NET_BIND_SERVICE))
 		return -EACCES;
 	/*
 	 * socket checks.
 	 */
 	sdp_conn_lock(conn);
 
-	if (SDP_SOCK_ST_CLOSED != conn->istate || 0 < conn->src_port) {
+	if (conn->istate != SDP_SOCK_ST_CLOSED || conn->src_port > 0) {
 		result = -EINVAL;
 		goto done;
 	}
@@ -498,7 +498,7 @@ static int _sdp_inet_bind(struct socket 
 		conn->src_addr = 0;
 
 	result = sdp_inet_port_get(conn, bind_port);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> getting port during bind",
 			     result);
 
@@ -510,7 +510,7 @@ static int _sdp_inet_bind(struct socket 
 	if (INADDR_ANY != conn->src_addr)
 		sk->sk_userlocks |= SOCK_BINDADDR_LOCK;
 
-	if (0 < bind_port)
+	if (bind_port > 0)
 		sk->sk_userlocks |= SOCK_BINDADDR_LOCK;
 
 	inet_sk(sk)->rcv_saddr = htonl(conn->src_addr);
@@ -566,14 +566,14 @@ static int _sdp_inet_connect(struct sock
 
 	switch (sock->state) {
 	case SS_UNCONNECTED:
-		if (0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
+		if (!(SDP_ST_MASK_CLOSED & conn->istate)) {
 			result = -EISCONN;
 			goto done;
 		}
 
-		if (0 == conn->src_port) {
+		if (!conn->src_port) {
 			result = sdp_inet_port_get(conn, 0);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, "Error <%d> getting port",
 					     result);
 				goto done;
@@ -598,7 +598,7 @@ static int _sdp_inet_connect(struct sock
 		 * post the SDP hello message
 		 */
 		result = sdp_cm_connect(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> initiating connect",
 				     result);
 
@@ -630,13 +630,13 @@ static int _sdp_inet_connect(struct sock
 	 * wait for connection to complete.
 	 */
 	timeout = sock_sndtimeo(sk, (O_NONBLOCK & flags));
-	if (0 < timeout) {
+	if (timeout > 0) {
 
 		DECLARE_WAITQUEUE(wait, current);
 		add_wait_queue(sk->sk_sleep, &wait);
 		set_current_state(TASK_INTERRUPTIBLE);
 
-		while (0 < timeout && SDP_SOCK_ST_CONNECT == conn->istate) {
+		while (timeout > 0 && conn->istate == SDP_SOCK_ST_CONNECT) {
 
 			sdp_conn_unlock(conn);
 			timeout = schedule_timeout(timeout);
@@ -651,9 +651,9 @@ static int _sdp_inet_connect(struct sock
 		set_current_state(TASK_RUNNING);
 		remove_wait_queue(sk->sk_sleep, &wait);
 
-		if (SDP_SOCK_ST_CONNECT == conn->istate) {
+		if (conn->istate == SDP_SOCK_ST_CONNECT) {
 
-			if (0 < timeout) {
+			if (timeout > 0) {
 
 				result = sock_intr_errno(timeout);
 			}
@@ -711,23 +711,23 @@ static int _sdp_inet_listen(struct socke
 	sdp_conn_lock(conn);
 
 	if (SS_UNCONNECTED != sock->state ||
-	    (SDP_SOCK_ST_CLOSED != conn->istate &&
-	     SDP_SOCK_ST_LISTEN != conn->istate)) {
+	    (conn->istate != SDP_SOCK_ST_CLOSED &&
+	     conn->istate != SDP_SOCK_ST_LISTEN)) {
 		result = -EINVAL;
 		goto done;
 	}
 
-	if (SDP_SOCK_ST_LISTEN != conn->istate) {
+	if (conn->istate != SDP_SOCK_ST_LISTEN) {
 		result = sdp_inet_listen_start(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> starting listen",
 				     result);
 			goto done;
 		}
 
-		if (0 == conn->src_port) {
+		if (!conn->src_port) {
 			result = sdp_inet_port_get(conn, 0);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, "Error <%d> getting port",
 					     result);
 				goto done;
@@ -772,7 +772,7 @@ static int _sdp_inet_accept(struct socke
 
 	sdp_conn_lock(listen_conn);
 
-	if (SDP_SOCK_ST_LISTEN != listen_conn->istate) {
+	if (listen_conn->istate != SDP_SOCK_ST_LISTEN) {
 		result = -EINVAL;
 		goto listen_done;
 	}
@@ -782,19 +782,19 @@ static int _sdp_inet_accept(struct socke
 	 * if there is no socket on the queue, wait for one. It' done in a
 	 * loop in case there is a problem with the first socket we hit.
 	 */
-	while (NULL == accept_conn) {
+	while (!accept_conn) {
 		/*
 		 * No pending socket wait.
 		 */
 		accept_conn = sdp_inet_accept_q_get(listen_conn);
-		if (NULL == accept_conn) {
+		if (!accept_conn) {
 			DECLARE_WAITQUEUE(wait, current);
 			add_wait_queue(listen_sk->sk_sleep, &wait);
 			set_current_state(TASK_INTERRUPTIBLE);
 
-			while (0 < timeout &&
-			       SDP_SOCK_ST_LISTEN == listen_conn->istate &&
-			       0 == listen_conn->backlog_cnt) {
+			while (timeout > 0 &&
+			       listen_conn->istate == SDP_SOCK_ST_LISTEN &&
+			       !listen_conn->backlog_cnt) {
 				sdp_conn_unlock(listen_conn);
 				timeout = schedule_timeout(timeout);
 				sdp_conn_lock(listen_conn);
@@ -808,14 +808,14 @@ static int _sdp_inet_accept(struct socke
 			/*
 			 * process result
 			 */
-			if (0 == listen_conn->backlog_cnt) {
+			if (!listen_conn->backlog_cnt) {
 				result = 0;
 
-				if (SDP_SOCK_ST_LISTEN != listen_conn->istate)
+				if (listen_conn->istate != SDP_SOCK_ST_LISTEN)
 					result = -EINVAL;
 				if (signal_pending(current))
 					result = sock_intr_errno(timeout);
-				if (0 == timeout)
+				if (!timeout)
 					result = -EAGAIN;
 
 				goto listen_done;
@@ -880,7 +880,7 @@ static int _sdp_inet_accept(struct socke
 				break;
 			}
 
-			if (NULL != accept_conn)
+			if (accept_conn)
 				/*
 				 * Connections returned from the AcceptQueue
 				 * are holding their lock, before returning
@@ -898,12 +898,11 @@ listen_done:
 
 	sdp_dbg_ctrl(listen_conn, 
 		     "ACCEPT: complete <%d> <%08x:%04x><%08x:%04x>",
-		     (NULL == accept_conn ?
-		      SDP_DEV_SK_INVALID : accept_conn->hashent),
-		     (NULL == accept_sk ? 0 : accept_conn->src_addr),
-		     (NULL == accept_sk ? 0 : accept_conn->src_port),
-		     (NULL == accept_sk ? 0 : accept_conn->dst_addr),
-		     (NULL == accept_sk ? 0 : accept_conn->dst_port));
+		     (accept_conn ? accept_conn->hashent : SDP_DEV_SK_INVALID),
+		     (accept_sk ? accept_conn->src_addr : 0),
+		     (accept_sk ? accept_conn->src_port : 0),
+		     (accept_sk ? accept_conn->dst_addr : 0),
+		     (accept_sk ? accept_conn->dst_port : 0));
 
 	return result;
 }
@@ -926,9 +925,9 @@ static int _sdp_inet_getname(struct sock
 		     conn->dst_addr, conn->dst_port);
 
 	addr->sin_family = _proto_family;
-	if (0 < peer)
-		if (0 < htons(conn->dst_port) &&
-		    0 == (SDP_ST_MASK_CLOSED & conn->istate)) {
+	if (peer > 0)
+		if (htons(conn->dst_port) > 0 &&
+		    !(SDP_ST_MASK_CLOSED & conn->istate)) {
 
 			addr->sin_port = htons(conn->dst_port);
 			addr->sin_addr.s_addr = htonl(conn->dst_addr);
@@ -977,7 +976,7 @@ static unsigned int _sdp_inet_poll(struc
 	 */
 	switch (conn->istate) {
 	case SDP_SOCK_ST_LISTEN:
-		mask |= (0 < conn->backlog_cnt) ? (POLLIN | POLLRDNORM) : 0;
+		mask |= (conn->backlog_cnt > 0) ? (POLLIN | POLLRDNORM) : 0;
 		break;
 	case SDP_SOCK_ST_ERROR:
 		mask |= POLLERR;
@@ -994,7 +993,7 @@ static unsigned int _sdp_inet_poll(struc
 		 * recv EOF _and_ recv data
 		 */
 		if (!(conn->byte_strm < sk->sk_rcvlowat) ||
-		    0 < (RCV_SHUTDOWN & conn->shutdown))
+		    (RCV_SHUTDOWN & conn->shutdown))
 
 			mask |= POLLIN | POLLRDNORM;
 		/*
@@ -1002,7 +1001,7 @@ static unsigned int _sdp_inet_poll(struc
 		 * (Some poll() Linux documentation says that POLLHUP is 
 		 *  incompatible with the POLLOUT/POLLWR flags)
 		 */
-		if (0 < (SEND_SHUTDOWN & conn->shutdown))
+		if (SEND_SHUTDOWN & conn->shutdown)
 			mask |= POLLHUP;
 		else {
 			/*
@@ -1014,7 +1013,7 @@ static unsigned int _sdp_inet_poll(struc
 			set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
 			set_bit(SOCK_NOSPACE, &sock->flags);
 
-			if (0 < __sdp_inet_writable(conn)) {
+			if (__sdp_inet_writable(conn)) {
 				mask |= POLLOUT | POLLWRNORM;
 
 				clear_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
@@ -1022,7 +1021,7 @@ static unsigned int _sdp_inet_poll(struc
 			}
 		}
 
-		if (0 < conn->rcv_urg_cnt)
+		if (conn->rcv_urg_cnt > 0)
 			mask |= POLLPRI;
 	}
 
@@ -1058,7 +1057,7 @@ static int _sdp_inet_ioctl(struct socket
 		 * standard INET IOCTLs
 		 */
 	case SIOCGSTAMP:
-		if (0 == sk->sk_stamp.tv_sec)
+		if (!sk->sk_stamp.tv_sec)
 			result = -ENOENT;
 		else {
 			result = copy_to_user((void __user *)arg,
@@ -1110,7 +1109,7 @@ static int _sdp_inet_ioctl(struct socket
 	case SIOCINQ:
 		sdp_conn_lock(conn);
 
-		if (SDP_SOCK_ST_LISTEN != conn->istate) {
+		if (conn->istate != SDP_SOCK_ST_LISTEN) {
 			/*
 			 * TODO need to subtract/add URG (inline vs. OOB)
 			 */
@@ -1124,7 +1123,7 @@ static int _sdp_inet_ioctl(struct socket
 	case SIOCOUTQ:
 		sdp_conn_lock(conn);
 
-		if (SDP_SOCK_ST_LISTEN != conn->istate) {
+		if (conn->istate != SDP_SOCK_ST_LISTEN) {
 			value = conn->send_qud;
 			result = put_user(value, (int __user *) arg);
 		} else
@@ -1137,10 +1136,10 @@ static int _sdp_inet_ioctl(struct socket
 
 		value = 0;
 
-		if (0 < conn->rcv_urg_cnt) {
+		if (conn->rcv_urg_cnt > 0) {
 			buff = sdp_buff_q_look_head(&conn->recv_pool);
-			if (NULL != buff &&
-			    0 < (SDP_BUFF_F_OOB_PRES & buff->flags) &&
+			if (buff &&
+			    (buff->flags & SDP_BUFF_F_OOB_PRES) &&
 			    1 == (buff->tail - buff->data))
 				value = 1;
 		}
@@ -1185,11 +1184,11 @@ static int _sdp_inet_setopt(struct socke
 
 	switch (optname) {
 	case TCP_NODELAY:
-		conn->nodelay = (0 == value) ? 0 : 1;
+		conn->nodelay = value ? 1 : 0;
 
-		if (0 < conn->nodelay) {
+		if (conn->nodelay > 0) {
 			result = sdp_send_flush(conn);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 
 		break;
@@ -1302,7 +1301,7 @@ static int _sdp_inet_shutdown(struct soc
 	 *       1 - send shutdown
 	 *       2 - send/recv shutdown.
 	 */
-	if (0 > flag || 2 < flag)
+	if (flag < 0 || flag < 2)
 		return -EINVAL;
 	else
 		flag++;		/* match shutdown mask. */
@@ -1319,9 +1318,9 @@ static int _sdp_inet_shutdown(struct soc
 		/*
 		 * Send shutdown is benign.
 		 */
-		if (0 < (RCV_SHUTDOWN & flag)) {
+		if (RCV_SHUTDOWN & flag) {
 			result = sdp_inet_listen_stop(conn);
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, "listen stop error <%d>",
 					     result);
 		}
@@ -1334,14 +1333,14 @@ static int _sdp_inet_shutdown(struct soc
 	case SDP_SOCK_ST_ACCEPTED:
 	case SDP_SOCK_ST_CONNECT:
 		result = _sdp_inet_abort(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> aborting connection",
 				     result);
 		break;
 	case SDP_SOCK_ST_ESTABLISHED:
 	case SDP_SOCK_ST_CLOSE:
 		result = _sdp_inet_disconnect(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> disconnecting conn",
 				     result);
 		break;
@@ -1368,7 +1367,7 @@ static int _sdp_inet_shutdown(struct soc
 		result = -EFAULT;
 	}
 
-	if (0 > result)
+	if (result < 0)
 		sdp_inet_wake_generic(sock->sk);
 	else
 		sdp_inet_wake_error(sock->sk);
@@ -1420,7 +1419,7 @@ static int _sdp_inet_create(struct socke
 	}
 
 	conn = sdp_conn_alloc(GFP_KERNEL);
-	if (NULL == conn) {
+	if (!conn) {
 		sdp_dbg_warn(conn, "SOCKET: failed to create socekt <%d:%d>",
 			     sock->type, protocol);
 		return -ENOMEM;
@@ -1470,7 +1469,7 @@ static int __init sdp_init(void)
 	 * proc entries
 	 */
 	result = sdp_main_proc_init();
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("INIT: Error <%d> creating proc entries.", result);
 		goto error_proc;
 	}
@@ -1478,7 +1477,7 @@ static int __init sdp_init(void)
 	 * advertisment table
 	 */
 	result = sdp_main_advt_init();
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("INIT: Error <%d> initializing advertisments",
 			 result);
 		goto error_advt;
@@ -1487,7 +1486,7 @@ static int __init sdp_init(void)
 	 * link data
 	 */
 	result = sdp_link_addr_init();
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("INIT: Error <%d> initializing link",
 			 result);
 		goto error_link;
@@ -1499,7 +1498,7 @@ static int __init sdp_init(void)
 				    _buff_max, 
 				    _alloc_inc, 
 				    _free_mark);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("INIT: Error <%d> initializing buffer pool.", result);
 		goto error_buff;
 	}
@@ -1513,7 +1512,7 @@ static int __init sdp_init(void)
 				     _send_post_max,
 				     _send_buff_max,
 				     _send_usig_max);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("INIT: Error <%d> initializing connection table.",
 			 result);
 		goto error_conn;
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1922)
+++ sdp_proto.h	(working copy)
@@ -487,7 +487,7 @@ extern int sdp_debug_level;
 #define __sdp_conn_dbg(level, type, conn, format, arg...) \
         do { \
                 struct sdp_opt *x = (conn); \
-                if (NULL != x) { \
+                if (x) { \
                         __sdp_dbg_out(level, type, \
                                       "<%d> <%04x:%04x> " format, \
                                        x->hashent, x->istate, x->state, \
@@ -572,7 +572,7 @@ static inline int __sdp_inet_write_space
 	 * write space is determined by amount of outstanding bytes of data
 	 * and number of buffers used for transmission by this connection
 	 */
-	if (0 < (SDP_ST_MASK_OPEN & conn->istate) &&
+	if ((SDP_ST_MASK_OPEN & conn->istate) &&
 	    (conn->send_max >
 	     sdp_desc_q_types_size(&conn->send_queue,
 				   SDP_DESC_TYPE_BUFF)))
@@ -586,7 +586,7 @@ static inline int __sdp_inet_write_space
  */
 static inline int __sdp_inet_writable(struct sdp_opt *conn)
 {
-	if (0 < (SDP_ST_MASK_OPEN & conn->istate))
+	if (SDP_ST_MASK_OPEN & conn->istate)
 		return (__sdp_inet_write_space(conn, 0) <
 			(conn->send_qud / 2)) ? 0 : 1;
 	else
@@ -605,8 +605,8 @@ static inline int __sdp_conn_stat_dump(s
 		     conn->src_serv, conn->snk_serv);
 
 	for (counter = 0; counter < 0x20; counter++)
-		if (0 < conn->send_mid[counter] ||
-		    0 < conn->recv_mid[counter]) {
+		if (conn->send_mid[counter] > 0 ||
+		    conn->recv_mid[counter] > 0) {
 			sdp_dbg_init("STAT: MID send <%02x> <%u>", counter,
 				     conn->send_mid[counter]);
 			sdp_dbg_init("STAT: MID recv <%02x> <%u>", counter,
@@ -626,7 +626,7 @@ static inline int __sdp_conn_state_dump(
 
 	sdp_dbg_init("STATE: Connection <%04x> state:", conn->hashent);
 
-	if (SDP_CONN_ST_INVALID == conn->state_rec.state[0]) {
+	if (conn->state_rec.state[0] == SDP_CONN_ST_INVALID) {
 		sdp_dbg_init("STATE:   No state history. <%d>",
 			     conn->state_rec.value);
 		return 0;
Index: sdp_read.c
===================================================================
--- sdp_read.c	(revision 1922)
+++ sdp_read.c	(working copy)
@@ -53,38 +53,38 @@ static int _sdp_event_read_advt(struct s
 	 * what MS does... (Either choice is correct)
 	 */
 	advt = sdp_advt_q_look(&conn->src_actv);
-	if (NULL == advt || advt->wrid != comp->wr_id) {
+	if (!advt || advt->wrid != comp->wr_id) {
 		advt = sdp_advt_q_look(&conn->src_pend);
-		if (NULL != advt && advt->wrid == comp->wr_id)
+		if (advt && advt->wrid == comp->wr_id)
 			advt->flag &= ~SDP_ADVT_F_READ;
 
 		goto done;
 	}
 
 	advt = sdp_advt_q_get(&conn->src_actv);
-	SDP_EXPECT((NULL != advt));
+	SDP_EXPECT((advt));
 
 	conn->src_recv--;
 
 	result = sdp_send_ctrl_rdma_rd(conn, advt->post);
-	SDP_EXPECT(!(0 > result));
+	SDP_EXPECT(result >= 0);
 
 	result = sdp_advt_destroy(advt);
-	SDP_EXPECT(!(0 > result));
+	SDP_EXPECT(result >= 0);
 	/*
 	 * If a SrcAvailCancel was received, and all RDMA reads
 	 * have been flushed, perform tail processing
 	 */
-	if (0 < (SDP_CONN_F_SRC_CANCEL_R & conn->flags) &&
-	    0 == conn->src_recv) {
+	if ((conn->flags & SDP_CONN_F_SRC_CANCEL_R) &&
+	    !conn->src_recv) {
 		conn->flags &= ~SDP_CONN_F_SRC_CANCEL_R;
 		conn->advt_seq = conn->recv_seq;
 		/*
 		 * If any data was canceled, post a SendSm, also
 		 */
-		if (0 < (SDP_CONN_F_SRC_CANCEL_C & conn->flags)) {
+		if (conn->flags & SDP_CONN_F_SRC_CANCEL_C) {
 			result = sdp_send_ctrl_send_sm(conn);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, "Error <%d> posting SendSm",
 					     result);
 				goto error;
@@ -162,7 +162,7 @@ int sdp_event_read(struct sdp_opt *conn,
 	switch (type) {
 	case SDP_DESC_TYPE_BUFF:
 		buff = (struct sdpc_buff *) sdp_desc_q_get_head(&conn->r_src);
-		SDP_EXPECT((NULL != buff));
+		SDP_EXPECT((buff));
 
 		if (comp->wr_id != buff->wrid) {
 			sdp_dbg_warn(conn, "work request mismatch <%llu:%llu>",
@@ -177,7 +177,7 @@ int sdp_event_read(struct sdp_opt *conn,
 		 * post data to the stream interface
 		 */
 		result = sdp_recv_buff(conn, buff);
-		if (0 < result) {
+		if (result > 0) {
 			/*
 			 * count number of bytes buffered by the connection,
 			 * zero byte buffers can be returned. If data was
@@ -187,22 +187,22 @@ int sdp_event_read(struct sdp_opt *conn,
 
 			sdp_inet_wake_recv(conn->sk, conn->byte_strm);
 		} else {
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, "Error <%d> receiving buff",
 					     result);
 
 			result = sdp_buff_pool_put(buff);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 
 		break;
 	case SDP_DESC_TYPE_IOCB:
 		iocb = (struct sdpc_iocb *) sdp_desc_q_look_head(&conn->r_src);
-		if (NULL == iocb || iocb->wrid != comp->wr_id)
+		if (!iocb || iocb->wrid != comp->wr_id)
 			break;
 
 		iocb = (struct sdpc_iocb *)sdp_desc_q_get_head(&conn->r_src);
-		SDP_EXPECT((NULL != iocb));
+		SDP_EXPECT((iocb));
 
 		iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_R);
 
@@ -210,7 +210,7 @@ int sdp_event_read(struct sdp_opt *conn,
 		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
 
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				      result, iocb->key);
 			goto done;
@@ -219,7 +219,7 @@ int sdp_event_read(struct sdp_opt *conn,
 		break;
 	case SDP_DESC_TYPE_NONE:
 		iocb = sdp_iocb_q_look(&conn->r_pend);
-		if (NULL == iocb) {
+		if (!iocb) {
 			result = -EPROTO;
 			goto done;
 		}
@@ -233,13 +233,13 @@ int sdp_event_read(struct sdp_opt *conn,
 			break;
 
 		iocb = sdp_iocb_q_get_head(&conn->r_pend);
-		SDP_EXPECT((NULL != iocb));
+		SDP_EXPECT((iocb));
 
 		SDP_CONN_STAT_READ_INC(conn, iocb->post);
 		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
 
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 			goto done;
@@ -256,7 +256,7 @@ int sdp_event_read(struct sdp_opt *conn,
 	 * The advertisment which generated this READ needs to be checked.
 	 */
 	result = _sdp_event_read_advt(conn, comp);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> handling READ advertisment",
 			     result);
 		goto done;
@@ -266,7 +266,7 @@ int sdp_event_read(struct sdp_opt *conn,
 	 * of some RDMAs
 	 */
 	result = sdp_send_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue.", result);
 		goto done;
 	}
@@ -275,7 +275,7 @@ int sdp_event_read(struct sdp_opt *conn,
 	 * reads.
 	 */
 	result = sdp_recv_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing recv queue.", result);
 		goto done;
 	}
Index: sdp_msgs.h
===================================================================
--- sdp_msgs.h	(revision 1922)
+++ sdp_msgs.h	(working copy)
@@ -68,7 +68,7 @@
 /*
  * message type masks
  */
-#define SDP_MID_CTRL(mid) ((0 < (0xF0 & mid)) ? 0 : 1)
+#define SDP_MID_CTRL(mid) ((0xF0 & mid) ? 0 : 1)
 /*
  * Base Sockets Direct Header (header for all SDP messages)
  */
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1922)
+++ sdp_send.c	(working copy)
@@ -54,7 +54,7 @@ static int _sdp_send_buff_post(struct sd
 	conn->s_wq_size++;
 	conn->l_advt_bf = conn->l_recv_bf;
 	conn->send_pipe -= buff->data_size;
-	conn->oob_offset -= (0 < conn->oob_offset) ? buff->data_size : 0;
+	conn->oob_offset -= (conn->oob_offset > 0) ? buff->data_size : 0;
 
 	buff->wrid = conn->send_wrid++;
 	buff->lkey = conn->l_key;
@@ -81,8 +81,7 @@ static int _sdp_send_buff_post(struct sd
 	 * the flag. This allows for at least one pending urgent message
 	 * to send early notification.
 	 */
-	if (0 < (SDP_CONN_F_OOB_SEND & conn->flags) &&
-	    !(0xFFFF < conn->oob_offset)) {
+	if ((conn->flags & SDP_CONN_F_OOB_SEND) && conn->oob_offset <= 0xFFFF) {
 		SDP_BSDH_SET_OOB_PEND(buff->bsdh_hdr);
 		SDP_BUFF_F_SET_SE(buff);
 
@@ -93,11 +92,11 @@ static int _sdp_send_buff_post(struct sd
 	 * and present flag is set, potentially OOB offset is cleared. pending
 	 * is set if this buffer has never had pending set.
 	 */
-	if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
-		if (0 < conn->oob_offset)
+	if (buff->flags & SDP_BUFF_F_OOB_PRES) {
+		if (conn->oob_offset > 0)
 			SDP_BSDH_SET_OOB_PEND(buff->bsdh_hdr);
 		else {
-			SDP_EXPECT(!(0 > conn->oob_offset));
+			SDP_EXPECT(conn->oob_offset >= 0);
 			conn->oob_offset = -1;
 		}
 
@@ -107,12 +106,12 @@ static int _sdp_send_buff_post(struct sd
 	/*
 	 * solicite event bit.
 	 */
-	if (0 < SDP_BUFF_F_GET_SE(buff))
+	if (SDP_BUFF_F_GET_SE(buff))
 		send_param.send_flags |= IB_SEND_SOLICITED;
 	/*
 	 * unsignalled event
 	 */
-	if (0 < SDP_BUFF_F_GET_UNSIG(buff) &&
+	if (SDP_BUFF_F_GET_UNSIG(buff) &&
 	    conn->usig_max > conn->send_cons) {
 		conn->send_usig++;
 		conn->send_cons++;
@@ -124,13 +123,13 @@ static int _sdp_send_buff_post(struct sd
 	/*
 	 * check queue membership. (first send attempt vs. flush)
 	 */
-	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff))
+	if (sdp_desc_q_member((struct sdpc_desc *) buff) > 0)
 		sdp_desc_q_remove((struct sdpc_desc *) buff);
 	/*
 	 * save the buffer for the event handler.
 	 */
 	result = sdp_buff_q_put_tail(&conn->send_post, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> queueing send buffer", result);
 		goto done;
 	}
@@ -149,7 +148,7 @@ static int _sdp_send_buff_post(struct sd
 	send_param.opcode  = IB_WR_SEND;
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
-	if (0 != result) {
+	if (result) {
 		sdp_dbg_warn(conn, 
 			     "Error <%d> posting send. <%d:%d> <%d:%d:%d>",
 			     result, conn->s_wq_cur, conn->s_wq_size,
@@ -164,11 +163,9 @@ static int _sdp_send_buff_post(struct sd
 	 * source cancels require us to save the sequence number
 	 * for validation of the cancel's completion.
 	 */
-	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags))
-		conn->src_cseq =
-		    ((SDP_MID_SRC_CANCEL ==
-		      buff->bsdh_hdr->mid) ? buff->bsdh_hdr->seq_num : conn->
-		     src_cseq);
+	if (conn->flags & SDP_CONN_F_SRC_CANCEL_L)
+		conn->src_cseq = ((buff->bsdh_hdr->mid == SDP_MID_SRC_CANCEL) ?
+				  buff->bsdh_hdr->seq_num : conn->src_cseq);
 
 	return 0;
 done:
@@ -199,17 +196,17 @@ static int _sdp_send_data_buff_post(stru
 	 * 3) buffer from head of queue or as parameter
 	 * 4) nodelay check.
 	 */
-	if (3 > conn->r_recv_bf || 0 < conn->src_sent)
+	if (conn->r_recv_bf < 3 || conn->src_sent > 0)
 		return ENOBUFS;
 	/*
 	 * The rest of the checks can proceed if there is a signalled event
 	 * in the pipe, otherwise we could stall...
 	 */
 	if (conn->send_usig < sdp_buff_q_size(&conn->send_post) ||
-	    0 < sdp_desc_q_size(&conn->w_snk)) {
+	    sdp_desc_q_size(&conn->w_snk) > 0) {
 		if (buff->tail < buff->end &&
-		    0 == (SDP_BUFF_F_OOB_PRES & buff->flags) &&
-		    0 == conn->nodelay)
+		    !(buff->flags & SDP_BUFF_F_OOB_PRES) &&
+		    !conn->nodelay)
 			/*
 			 * If the buffer is not full, and there is already
 			 * data in the SDP pipe, then hold on to the buffer
@@ -223,7 +220,7 @@ static int _sdp_send_data_buff_post(stru
 		 * asymmetric connections. This is desirable to offload
 		 * the remote host.
 		 */
-		if (!(conn->s_wq_cur > conn->s_wq_size)) {
+		if (conn->s_wq_cur <= conn->s_wq_size) {
 			/*
 			 * slow down the up take in the send data path to
 			 * give the remote side some time to post available
@@ -257,7 +254,7 @@ static int _sdp_send_data_buff_post(stru
 	 * update non-discard counter.
 	 * Make consideration for a pending sink. (can be forced by OOB)
 	 */
-	if (0 < sdp_advt_q_size(&conn->snk_pend)) {
+	if (sdp_advt_q_size(&conn->snk_pend) > 0) {
 		/*
 		 * As sink advertisment needs to be discarded. We always
 		 * complete an advertisment if there is not enough room
@@ -268,10 +265,10 @@ static int _sdp_send_data_buff_post(stru
 		 * then the buffer.)
 		 */
 		advt = sdp_advt_q_get(&conn->snk_pend);
-		SDP_EXPECT((NULL != advt));
+		SDP_EXPECT((advt));
 
 		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 		/*
 		 * update sink advertisments.
 		 */
@@ -282,7 +279,7 @@ static int _sdp_send_data_buff_post(stru
 	 * transmision time
 	 */
 	result = _sdp_send_buff_post(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting send data buffer",
 			     result);
 		goto error;
@@ -310,19 +307,19 @@ static int _sdp_send_data_buff_snk(struc
 	 *
 	 * 1) sufficient send resources.
 	 */
-	if (!(conn->send_cq_size > conn->s_wq_size))
+	if (conn->send_cq_size <= conn->s_wq_size)
 		return ENOBUFS;
 	/*
 	 * confirm type
 	 */
-	if (SDP_DESC_TYPE_BUFF != buff->type)
+	if (buff->type != SDP_DESC_TYPE_BUFF)
 		return -ENOBUFS;
 	/*
 	 * nodelay buffering
 	 */
 #if 0
 	if (buff->tail < buff->end &&
-	    0 == conn->nodelay &&
+	    !conn->nodelay &&
 	    conn->send_usig < sdp_buff_q_size(&conn->send_post)) {
 		/*
 		 * If the buffer is not full, and there is already data in the
@@ -337,7 +334,7 @@ static int _sdp_send_data_buff_snk(struc
 	 * get advertisment.
 	 */
 	advt = sdp_advt_q_look(&conn->snk_pend);
-	if (NULL == advt)
+	if (!advt)
 		return ENOBUFS;
 	/*
 	 * signalled? With no delay turned off, data transmission may be
@@ -369,11 +366,11 @@ static int _sdp_send_data_buff_snk(struc
 	/*
 	 * dequeue if needed and the queue buffer
 	 */
-	if (0 < sdp_desc_q_member((struct sdpc_desc *) buff))
+	if (sdp_desc_q_member((struct sdpc_desc *) buff) > 0)
 		sdp_desc_q_remove((struct sdpc_desc *) buff);
 
 	result = sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff);
-        if (0 > result) {
+        if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> queueing write buffer. <%d>",
 			     result, sdp_desc_q_size(&conn->w_snk));
 		goto error;
@@ -383,7 +380,7 @@ static int _sdp_send_data_buff_snk(struc
 	 */
 	conn->s_wq_size++;
 	conn->send_pipe  -= buff->data_size;
-	conn->oob_offset -= (0 < conn->oob_offset) ? buff->data_size : 0;
+	conn->oob_offset -= (conn->oob_offset > 0) ? buff->data_size : 0;
 	/*
 	 * post RDMA
 	 */
@@ -393,7 +390,7 @@ static int _sdp_send_data_buff_snk(struc
 	send_param.num_sge = 1;
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
-	if (0 != result) {
+	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> posting rdma write", result);
 
 		conn->s_wq_size--;
@@ -405,15 +402,15 @@ static int _sdp_send_data_buff_snk(struc
 	 */
 	if (conn->send_size > advt->size) {
 		advt = sdp_advt_q_get(&conn->snk_pend);
-		SDP_EXPECT((NULL != advt));
+		SDP_EXPECT((advt));
 
 		zcopy = advt->post;
 
 		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 
 		result = sdp_send_ctrl_rdma_wr(conn, zcopy);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing sink. <%d>",
 				     result, zcopy);
 			result = -ENODEV;
@@ -449,7 +446,7 @@ static int _sdp_send_data_iocb_snk(struc
 	 * to fill the advertisement
 	 */
 	result = sdp_iocb_register(iocb, conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> registering IOCB. <%d:%d>",
 			     result, iocb->key, iocb->len);
 		goto error;
@@ -457,12 +454,12 @@ static int _sdp_send_data_iocb_snk(struc
 	/*
 	 * check queue depth
 	 */
-	while (0 < iocb->len && conn->send_cq_size > conn->s_wq_size) {
+	while (iocb->len > 0 && conn->send_cq_size > conn->s_wq_size) {
 		/*
 		 * get the pending sink advertisment.
 		 */
 		advt = sdp_advt_q_look(&conn->snk_pend);
-		if (NULL == advt)
+		if (!advt)
 			break;
 		/*
 		 * amount of data to zcopy.
@@ -499,7 +496,7 @@ static int _sdp_send_data_iocb_snk(struc
 		 */
 		conn->s_wq_size++;
 		conn->send_pipe -= zcopy;
-		conn->oob_offset -= (0 < conn->oob_offset) ? zcopy : 0;
+		conn->oob_offset -= (conn->oob_offset > 0) ? zcopy : 0;
 		/*
 		 * post RDMA
 		 */
@@ -509,7 +506,7 @@ static int _sdp_send_data_iocb_snk(struc
 		send_param.num_sge = 1;
 
 		result = ib_post_send(conn->qp, &send_param, &bad_wr);
-		if (0 != result) {
+		if (result) {
 			sdp_dbg_warn(conn, "Error <%d> posting rdma write",
 				     result);
 
@@ -523,11 +520,11 @@ static int _sdp_send_data_iocb_snk(struc
 		 * outstanding sink advertisment to see how the advt
 		 * size remaining is picked.)
 		 */
-		if (!(conn->send_size > advt->size))
+		if (conn->send_size <= advt->size)
 			continue;
 
 		advt = sdp_advt_q_get(&conn->snk_pend);
-		if (NULL == advt) {
+		if (!advt) {
 			sdp_dbg_warn(conn, "sink advertisment disappeared.");
 			result = -ENODEV;
 			goto error;
@@ -536,10 +533,10 @@ static int _sdp_send_data_iocb_snk(struc
 		zcopy = advt->post;
 
 		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 
 		result = sdp_send_ctrl_rdma_wr(conn, zcopy);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> completing sink. <%d>",
 				     result, zcopy);
 			result = -ENODEV;
@@ -570,16 +567,16 @@ static int _sdp_send_data_iocb_src(struc
 	 * 1) local source cancel is pending
 	 * 2) sufficient send credits for buffered transmission.
 	 */
-	if (0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags) || 3 > conn->r_recv_bf)
+	if ((conn->flags & SDP_CONN_F_SRC_CANCEL_L) || conn->r_recv_bf < 3)
 		return ENOBUFS;
 
 	switch (conn->send_mode) {
 	case SDP_MODE_PIPE:
-		if (!(conn->s_cur_adv > conn->src_sent))
+		if (conn->s_cur_adv <= conn->src_sent)
 			return ENOBUFS;
 
 		if (conn->s_cur_adv < conn->r_max_adv) {
-			if (!(SDP_SEND_POST_FRACTION > conn->s_par_adv)) {
+			if (conn->s_par_adv >= SDP_SEND_POST_FRACTION) {
 				conn->s_cur_adv++;
 				conn->s_par_adv = 0;
 			}
@@ -596,7 +593,7 @@ static int _sdp_send_data_iocb_src(struc
 #endif
 		break;
 	case SDP_MODE_COMB:
-		if (0 < conn->src_sent)
+		if (conn->src_sent > 0)
 			return ENOBUFS;
 		break;
 	default:
@@ -608,7 +605,7 @@ static int _sdp_send_data_iocb_src(struc
 	 * get buffer
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Error allocating SrcAvail buffer. <%d>",
 			     iocb->key);
 		return -ENOMEM;
@@ -619,7 +616,7 @@ static int _sdp_send_data_iocb_src(struc
 	 * for the same IOCB.
 	 */
 	result = sdp_iocb_register(iocb, conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> registering IOCB. <%d:%d>",
 			     result, iocb->key, iocb->len);
 		goto drop;
@@ -647,7 +644,7 @@ static int _sdp_send_data_iocb_src(struc
 	SDP_BUFF_F_CLR_SE(buff);
 	SDP_BUFF_F_CLR_UNSIG(buff);
 
-	if (SDP_MODE_COMB == conn->send_mode) {
+	if (conn->send_mode == SDP_MODE_COMB) {
 		void *addr;
 		int   pos;
 		int   off;
@@ -673,7 +670,7 @@ static int _sdp_send_data_iocb_src(struc
 		 * map, copy, unmap.
 		 */
 		addr = __sdp_kmap(iocb->page_array[pos]);
-		if (NULL == addr) {
+		if (!addr) {
 			result = -ENOMEM;
 			goto error;
 		}
@@ -701,7 +698,7 @@ static int _sdp_send_data_iocb_src(struc
 	 * queue/send SrcAvail message
 	 */
 	result = _sdp_send_buff_post(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting SrcAvail for IOCB <%d>",
 			     result, iocb->key);
 		goto release;
@@ -709,10 +706,10 @@ static int _sdp_send_data_iocb_src(struc
 
 	return 0;
 release:
-	conn->nond_send -= (SDP_MODE_COMB == conn->send_mode) ? 1 : 0;
+	conn->nond_send -= (conn->send_mode == SDP_MODE_COMB) ? 1 : 0;
 	conn->src_sent--;
 
-	iocb->len += ((SDP_MODE_COMB == conn->send_mode) ?
+	iocb->len += ((conn->send_mode == SDP_MODE_COMB) ?
 		      SDP_SRC_AVAIL_MIN : 0);
 error:
 	iocb->flags &= ~(SDP_IOCB_F_RDMA_R | SDP_IOCB_F_ACTIVE);
@@ -740,12 +737,12 @@ static int _sdp_send_iocb_buff_write(str
 	offset  = (iocb->post + iocb->page_offset) & (~PAGE_MASK);
 
 	while (buff->tail < buff->end &&
-	       0 < iocb->len) {
+	       iocb->len > 0) {
 		/*
 		 * map correct page of iocb
 		 */
 		addr = __sdp_kmap(iocb->page_array[counter]);
-		if (NULL == addr)
+		if (!addr)
 			break;
 		
 		copy = min((PAGE_SIZE - offset),
@@ -779,21 +776,21 @@ static int _sdp_send_data_iocb_buff(stru
 	int result;
 	int copy;
 
-	if (0 < conn->src_sent)
+	if (conn->src_sent > 0)
 		return ENOBUFS;
 	/*
 	 * loop through queued buffers and copy them to the destination
 	 */
 	copy = min(__sdp_inet_write_space(conn, 0), iocb->len);
 
-	while (0 < copy &&
-	       2 < conn->r_recv_bf &&
+	while (copy > 0 &&
+	       conn->r_recv_bf > 2 &&
 	       conn->send_cq_size > conn->s_wq_size) {
 		/*
 		 * get a buffer for posting.
 		 */
 		buff = sdp_buff_pool_get();
-		if (NULL == buff) {
+		if (!buff) {
 			result = -ENOMEM;
 			goto error;
 		}
@@ -814,7 +811,7 @@ static int _sdp_send_data_iocb_buff(stru
 		 * TODO: need to be checking OOB here.
 		 */
 		result =_sdp_send_iocb_buff_write(iocb, buff);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> copy from IOCB <%d>.",
 				     result, iocb->key);
 			goto drop;
@@ -829,7 +826,7 @@ static int _sdp_send_data_iocb_buff(stru
 		 * since the IOCB queue took care of the increment.
 		 */
 		result = _sdp_send_buff_post(conn, buff);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> send queue buff post",
 				     result);
 			goto drop;
@@ -850,30 +847,30 @@ static int _sdp_send_data_iocb(struct sd
 {
 	int result = ENOBUFS;
 
-	if (!(conn->send_cq_size > conn->s_wq_size))
+	if (conn->send_cq_size <= conn->s_wq_size)
 		goto done;
 	/*
 	 * confirm IOCB usage.
 	 */
-	if (SDP_DESC_TYPE_IOCB != iocb->type)
+	if (iocb->type != SDP_DESC_TYPE_IOCB)
 		return -ENOBUFS;
 	/*
 	 * determin if we are sending Buffered, Source or Sink.
 	 */
-	if (0 < sdp_advt_q_size(&conn->snk_pend)) {
+	if (sdp_advt_q_size(&conn->snk_pend) > 0) {
 		result = _sdp_send_data_iocb_snk(conn, iocb);
-		if (0 == result) {
+		if (!result) {
 			/*
 			 * IOCB completely processed. Otherwise we allow the
 			 * callers to determine the fate of the IOCB on
 			 * failure or partial processing.
 			 */
-			if (0 < sdp_desc_q_member((struct sdpc_desc *)iocb))
+			if (sdp_desc_q_member((struct sdpc_desc *)iocb) > 0)
 				sdp_desc_q_remove((struct sdpc_desc *)iocb);
 
 			result = sdp_desc_q_put_tail(&conn->w_snk,
 						     (struct sdpc_desc *)iocb);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> queuing write IOCB.",
 					     result);
@@ -887,25 +884,25 @@ static int _sdp_send_data_iocb(struct sd
 	 * hope that a new sink advertisment will arrive, because 
 	 * sinks are more efficient.
 	 */
-	if (0 < sdp_desc_q_size(&conn->w_snk))
+	if (sdp_desc_q_size(&conn->w_snk) > 0)
 		goto done;
 
 	if (conn->src_zthresh > iocb->len ||
-	    SDP_MODE_BUFF == conn->send_mode ||
-	    0 < (SDP_IOCB_F_BUFF & iocb->flags)) {
+	    conn->send_mode == SDP_MODE_BUFF ||
+	    (iocb->flags & SDP_IOCB_F_BUFF)) {
 		result = _sdp_send_data_iocb_buff(conn, iocb);
-		if (0 == result) {
+		if (!result) {
 			/*
 			 * complete this IOCB
 			 */
-			if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb))
+			if (sdp_desc_q_member((struct sdpc_desc *) iocb) > 0)
 				sdp_desc_q_remove((struct sdpc_desc *) iocb);
 
 			SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
 			SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
 			result = sdp_iocb_complete(iocb, 0);
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> completing iocb <%d>",
 					     result, iocb->key);
@@ -915,15 +912,15 @@ static int _sdp_send_data_iocb(struct sd
 	}
 
 	result = _sdp_send_data_iocb_src(conn, iocb);
-	if (0 == result) {
+	if (!result) {
 		/*
 		 * queue IOCB
 		 */
-		if (0 < sdp_desc_q_member((struct sdpc_desc *) iocb))
+		if (sdp_desc_q_member((struct sdpc_desc *) iocb) > 0)
 			sdp_desc_q_remove((struct sdpc_desc *)iocb);
 
 		result = sdp_iocb_q_put_tail(&conn->w_src, iocb);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> queueing write <%d:%d>",
 				     result, iocb->key,
 				     sdp_iocb_q_size(&conn->w_src));
@@ -946,15 +943,15 @@ static int _sdp_send_data_queue_test(str
 	 * 1) Invalid state for transmission
 	 * 2) source advertisment cancel in progress.
 	 */
-	if (0 == (SDP_ST_MASK_SEND_OK & conn->state) ||
-	    0 < (SDP_CONN_F_SRC_CANCEL_L & conn->flags))
+	if (!(SDP_ST_MASK_SEND_OK & conn->state) ||
+	    (conn->flags & SDP_CONN_F_SRC_CANCEL_L))
 		return ENOBUFS;
 
-	if (SDP_DESC_TYPE_IOCB == element->type)
+	if (element->type == SDP_DESC_TYPE_IOCB)
 		return _sdp_send_data_iocb(conn, (struct sdpc_iocb *)element);
 
-	if (0 == sdp_advt_q_look(&conn->snk_pend) ||
-	    (SDP_BUFF_F_OOB_PRES & ((struct sdpc_buff *)element)->flags))
+	if (!sdp_advt_q_look(&conn->snk_pend) ||
+	    (((struct sdpc_buff *)element)->flags & SDP_BUFF_F_OOB_PRES))
 		result = _sdp_send_data_buff_post(conn,
 						  (struct sdpc_buff *)element);
 	else
@@ -977,9 +974,9 @@ static int _sdp_send_data_queue_flush(st
 	 * non-zero result is generated.
 	 * (positive: no space; negative: error)
 	 */
-	while (!result && 0 < sdp_desc_q_size(&conn->send_queue)) {
+	while (!result && sdp_desc_q_size(&conn->send_queue) > 0) {
 		element = sdp_desc_q_look_head(&conn->send_queue);
-		SDP_EXPECT((NULL != element));
+		SDP_EXPECT((element));
 
 		result = _sdp_send_data_queue_test(conn, element);
 		if (!result)
@@ -987,7 +984,7 @@ static int _sdp_send_data_queue_flush(st
 		/*
 		 * error
 		 */
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> post data during flush",
 				     result);
 			/*
@@ -995,10 +992,10 @@ static int _sdp_send_data_queue_flush(st
 			 * since called functions can dequeue the
 			 * element, and not know how to requeue it.
 			 */
-			if (0 == sdp_desc_q_member(element)) {
+			if (!sdp_desc_q_member(element)) {
 				result = sdp_desc_q_put_head(&conn->send_queue,
 							     element);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			}
 		}
 	}
@@ -1021,10 +1018,10 @@ static int _sdp_send_data_queue(struct s
 	 * requires us to buffer, a negative result is an error, a return
 	 * value of zero is a successful transmission
 	 */
-	if (0 < sdp_desc_q_size(&conn->send_queue) ||
-	    0 < (result = _sdp_send_data_queue_test(conn, element))) {
+	if (sdp_desc_q_size(&conn->send_queue) > 0 ||
+	    (result = _sdp_send_data_queue_test(conn, element)) > 0) {
 		result = sdp_desc_q_put_tail(&conn->send_queue, element);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> queueing data for send",
 				     result);
 			goto done;
@@ -1032,12 +1029,11 @@ static int _sdp_send_data_queue(struct s
 		/*
 		 * Potentially request a switch to pipelined mode.
 		 */
-		if (SDP_MODE_COMB == conn->send_mode &&
-		    !(SDP_INET_SEND_MODE >
-		      sdp_desc_q_size(&conn->send_queue))) {
+		if (conn->send_mode == SDP_MODE_COMB &&
+		    sdp_desc_q_size(&conn->send_queue) >= SDP_INET_SEND_MODE) {
 			result = sdp_send_ctrl_mode_ch(conn,
 						       SDP_MSG_MCH_PIPE_RECV);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> posting mode change",
 					     result);
@@ -1046,7 +1042,7 @@ static int _sdp_send_data_queue(struct s
 		}
 	}
 
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> during data send posting",
 			     result);
 		goto done;
@@ -1069,11 +1065,11 @@ static inline struct sdpc_buff *_sdp_sen
 	 */
 	buff = (struct sdpc_buff *)sdp_desc_q_look_type_tail(&conn->send_queue,
 							     SDP_DESC_TYPE_BUFF);
-	if (NULL == buff ||
+	if (!buff ||
 	    buff->tail == buff->end ||
-	    0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
+	    (buff->flags & SDP_BUFF_F_OOB_PRES)) {
 		buff = sdp_buff_pool_get();
-		if (NULL != buff) {
+		if (buff) {
 			buff->tail = buff->end - conn->send_size;
 			buff->data = buff->tail;
 		}
@@ -1096,12 +1092,12 @@ static inline int _sdp_send_data_buff_pu
 	/*
 	 * See note on send OOB implementation in SendBuffPost.
 	 */
-	if (0 < urg) {
+	if (urg > 0) {
 		buff->flags |= SDP_BUFF_F_OOB_PRES;
 		/*
 		 * The OOB PEND and PRES flags need to match up as pairs.
 		 */
-		if (0 > conn->oob_offset) {
+		if (conn->oob_offset < 0) {
 			conn->oob_offset = conn->send_pipe + size;
 			conn->flags |= SDP_CONN_F_OOB_SEND;
 		}
@@ -1110,7 +1106,7 @@ static inline int _sdp_send_data_buff_pu
 	 * if the buffer is already queue, then this was a fill of a partial
 	 * buffer and dosn't need to be queued now.
 	 */
-	if (0 < (SDP_BUFF_F_QUEUED & buff->flags)) {
+	if (buff->flags & SDP_BUFF_F_QUEUED) {
 		buff->data_size += size;
 		conn->send_qud += size;
 		conn->send_pipe += size;
@@ -1124,12 +1120,12 @@ static inline int _sdp_send_data_buff_pu
 		 * finally send the data buffer
 		 */
 		result = _sdp_send_data_queue(conn, (struct sdpc_desc *) buff);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> buffer to SEND queue.",
 				     result);
 
 			expect = sdp_buff_pool_put(buff);
-			SDP_EXPECT(!(0 > expect));
+			SDP_EXPECT(expect >= 0);
 		}
 	}
 
@@ -1148,16 +1144,16 @@ static int _sdp_send_ctrl_buff_test(stru
 {
 	int result = 0;
 
-	if (0 == (SDP_ST_MASK_CTRL_OK & conn->state) ||
+	if (!(SDP_ST_MASK_CTRL_OK & conn->state) ||
 	    !(conn->send_cq_size > conn->s_wq_size) ||
-	    !(0 < conn->r_recv_bf) ||
-	    (conn->l_recv_bf == conn->l_advt_bf && 1 == conn->r_recv_bf))
+	    conn->r_recv_bf <= 0 ||
+	    (conn->l_recv_bf == conn->l_advt_bf && conn->r_recv_bf == 1))
 		return ENOBUFS;
 	/*
 	 * post the control buffer
 	 */
 	result = _sdp_send_buff_post(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control send", result);
 		goto error;
 	}
@@ -1179,24 +1175,24 @@ static int _sdp_send_ctrl_buff_flush(str
 	 * As long as there are buffers, try to post  until a non-zero
 	 * result is generated. (positive: no space; negative: error)
 	 */
-	while (!result && 0 < sdp_desc_q_size(&conn->send_ctrl)) {
+	while (!result && sdp_desc_q_size(&conn->send_ctrl) > 0) {
 		element = sdp_desc_q_look_head(&conn->send_ctrl);
-		SDP_EXPECT((NULL != element));
+		SDP_EXPECT((element));
 
 		result = _sdp_send_ctrl_buff_test(conn, 
 						  (struct sdpc_buff *)element);
 		if (!result)
 			continue;
 
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn,
 				     "Error <%d> failed to flush control msg",
 				     result);
 
-			if (0 == sdp_desc_q_member(element)) {
+			if (!sdp_desc_q_member(element)) {
 				result = sdp_desc_q_put_head(&conn->send_ctrl,
 							     element);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			}
 		}
 	}
@@ -1215,21 +1211,21 @@ static int _sdp_send_ctrl_buff_buffered(
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	if (0 < sdp_desc_q_size(&conn->send_ctrl) ||
-	    0 < (result = _sdp_send_ctrl_buff_test(conn, buff))) {
+	if (sdp_desc_q_size(&conn->send_ctrl) > 0 ||
+	    (result = _sdp_send_ctrl_buff_test(conn, buff)) > 0) {
 		/*
 		 * save the buffer for later flushing into the post queue.
 		 */
 		result = sdp_desc_q_put_tail(&conn->send_ctrl,
 					     (struct sdpc_desc *)buff);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> queueing control buff",
 				     result);
 			goto error;
 		}
 	}
 
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> during control send posting",
 			     result);
 		goto error;
@@ -1256,7 +1252,7 @@ static int _sdp_send_ctrl_buff(struct sd
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for control");
 		result = -ENOMEM;
 		goto error;
@@ -1287,7 +1283,7 @@ static int _sdp_send_ctrl_buff(struct sd
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1310,7 +1306,7 @@ static int _sdp_send_ctrl_disconnect(str
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for disconnect");
 		result = -ENOMEM;
 		goto error;
@@ -1335,7 +1331,7 @@ static int _sdp_send_ctrl_disconnect(str
 	 * send
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		goto error;
@@ -1360,10 +1356,10 @@ int sdp_send_ctrl_disconnect(struct sdp_
 	 * clean. The state is now in a disconnect send, the message will be
 	 * sent once data is flushed.
 	 */
-	if (0 < (SDP_ST_MASK_DIS_PEND & conn->state)) {
-		if (0 == (SDP_CONN_F_DIS_HOLD & conn->flags) &&
-		    0 == sdp_desc_q_size(&conn->send_queue) &&
-		    0 == conn->src_sent)
+	if (SDP_ST_MASK_DIS_PEND & conn->state) {
+		if (!(conn->flags & SDP_CONN_F_DIS_HOLD) &&
+		    !sdp_desc_q_size(&conn->send_queue) &&
+		    !conn->src_sent)
 			result = _sdp_send_ctrl_disconnect(conn);
 		else {
 			sdp_dbg_ctrl(conn, "defer disconnect <%d:%d> <%08x>",
@@ -1391,9 +1387,9 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
 	 * conditions then the control queue, so there is more checking to
 	 * be done, then whether there is data in the queue.
 	 */
-	if (0 < sdp_desc_q_size(&conn->send_ctrl) ||
-	    (0 < sdp_desc_q_size(&conn->send_queue) &&
-	     2 < conn->l_advt_bf))
+	if (sdp_desc_q_size(&conn->send_ctrl) > 0 ||
+	    (sdp_desc_q_size(&conn->send_queue) > 0 &&
+	     conn->l_advt_bf > 2))
 		return 0;
 
 	return _sdp_send_ctrl_buff(conn, SDP_MID_DATA, 0, 0);
@@ -1456,7 +1452,7 @@ int sdp_send_ctrl_resize_buff_ack(struct
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for resize ack");
 		result = -ENOMEM;
 		goto error;
@@ -1483,7 +1479,7 @@ int sdp_send_ctrl_resize_buff_ack(struct
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1505,7 +1501,7 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	/*
 	 * check size
 	 */
-	if (0 > size) {
+	if (size < 0) {
 		sdp_dbg_warn(conn, "RDMA read completion <%d> too small.",
 			     size);
 		return -ERANGE;
@@ -1515,7 +1511,7 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for RDMA rd");
 		result = -ENOMEM;
 		goto error;
@@ -1552,7 +1548,7 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1576,7 +1572,7 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for RDMA wr");
 		result = -ENOMEM;
 		goto error;
@@ -1609,7 +1605,7 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1634,7 +1630,7 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	/*
 	 * check mode
 	 */
-	if (SDP_MODE_PIPE != conn->recv_mode) {
+	if (conn->recv_mode != SDP_MODE_PIPE) {
 		result = -EPROTO;
 		goto error;
 	}
@@ -1643,7 +1639,7 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for SnkAvail");
 		result = -ENOMEM;
 		goto error;
@@ -1674,7 +1670,7 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		goto error;
@@ -1699,19 +1695,19 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	 */
 	switch (mode) {
 	case SDP_MSG_MCH_BUFF_RECV:	/* source to sink */
-		conn->send_mode = ((SDP_MODE_COMB == conn->send_mode) ?
+		conn->send_mode = ((conn->send_mode == SDP_MODE_COMB) ?
 				   SDP_MODE_BUFF : SDP_MODE_ERROR);
 		break;
 	case SDP_MSG_MCH_COMB_SEND:	/* sink to source */
-		conn->recv_mode = ((SDP_MODE_BUFF == conn->recv_mode) ?
+		conn->recv_mode = ((conn->recv_mode == SDP_MODE_BUFF) ?
 				   SDP_MODE_COMB : SDP_MODE_ERROR);
 		break;
 	case SDP_MSG_MCH_PIPE_RECV:	/* source to sink */
-		conn->send_mode = ((SDP_MODE_COMB == conn->send_mode) ?
+		conn->send_mode = ((conn->send_mode == SDP_MODE_COMB) ?
 				   SDP_MODE_PIPE : SDP_MODE_ERROR);
 		break;
 	case SDP_MSG_MCH_COMB_RECV:	/* source to sink */
-		conn->send_mode = ((SDP_MODE_PIPE == conn->send_mode) ?
+		conn->send_mode = ((conn->send_mode == SDP_MODE_PIPE) ?
 				   SDP_MODE_COMB : SDP_MODE_ERROR);
 		break;
 	default:
@@ -1721,8 +1717,8 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 		goto error;
 	}
 
-	if (SDP_MODE_ERROR == conn->send_mode ||
-	    SDP_MODE_ERROR == conn->recv_mode) {
+	if (conn->send_mode == SDP_MODE_ERROR ||
+	    conn->recv_mode == SDP_MODE_ERROR) {
 		sdp_dbg_warn(conn, "mode transition error <%d:%d:%d>",
 			     mode, conn->send_mode, conn->recv_mode);
 		result = -EPROTO;
@@ -1733,7 +1729,7 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	 * (don't need to worry about header space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buffer for ModeChange");
 		result = -ENOMEM;
 		goto error;
@@ -1760,7 +1756,7 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	result = _sdp_send_ctrl_buff_buffered(conn, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
 		(void)sdp_buff_pool_put(buff);
@@ -1782,7 +1778,7 @@ static int _sdp_write_src_lookup(struct 
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
 
-	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key)
+	if (element->type == SDP_DESC_TYPE_IOCB && iocb->key == req->ki_key)
 		return 0;
 	else
 		return -ERANGE;
@@ -1801,7 +1797,7 @@ static int _sdp_inet_write_cancel(struct
 	sdp_dbg_ctrl(NULL, "Cancel Write IOCB user <%d> key <%d> flag <%08lx>",
 		     req->ki_users, req->ki_key, req->ki_flags);
 	
-	if (NULL == si || NULL == si->sock || NULL == si->sock->sk) {
+	if (!si || !si->sock || !si->sock->sk) {
 		sdp_warn("Cancel empty write IOCB users <%d> flags <%d:%08lx>",
 			 req->ki_users, req->ki_key, req->ki_flags);
 		result = -EFAULT;
@@ -1823,38 +1819,38 @@ static int _sdp_inet_write_cancel(struct
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->send_queue,
 						     _sdp_write_src_lookup,
 						     req);
-	if (NULL != iocb) {
+	if (iocb) {
 		/*
 		 * always remove the IOCB.
 		 * If active, then place it into the correct active queue
 		 */
 		sdp_desc_q_remove((struct sdpc_desc *)iocb);
 		
-		if (0 < (SDP_IOCB_F_ACTIVE & iocb->flags)) {
-			if (0 < (SDP_IOCB_F_RDMA_W & iocb->flags)) {
+		if (iocb->flags & SDP_IOCB_F_ACTIVE) {
+			if (iocb->flags & SDP_IOCB_F_RDMA_W) {
 				result = sdp_desc_q_put_tail(&conn->w_snk,
 							     (struct sdpc_desc *)iocb);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			} else {
-				SDP_EXPECT((SDP_IOCB_F_RDMA_R & iocb->flags));
+				SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_R));
 
 				result = sdp_iocb_q_put_tail(&conn->w_src,
 							     iocb);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			}
 		} else {
 			/*
 			 * empty IOCBs can be deleted, while partials
 			 * needs to be compelted.
 			 */
-			if (0 < iocb->post) {
+			if (iocb->post > 0) {
 				result = sdp_iocb_complete(iocb, 0);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 
 				result = -EAGAIN;
 			} else {
 				result = sdp_iocb_destroy(iocb);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 				/*
 				 * completion reference
 				 */
@@ -1874,7 +1870,7 @@ static int _sdp_inet_write_cancel(struct
 						     _sdp_write_src_lookup,
 						     req);
 
-	if (NULL != iocb) {
+	if (iocb) {
 		iocb->flags |= SDP_IOCB_F_CANCEL;
 		result = -EAGAIN;
 
@@ -1885,7 +1881,7 @@ static int _sdp_inet_write_cancel(struct
 	 * needs to be issued.
 	 */
 	iocb = sdp_iocb_q_lookup(&conn->w_src, req->ki_key);
-	if (NULL != iocb) {
+	if (iocb) {
 		/*
 		 * Unfortunetly there is only a course grain cancel in SDP,
 		 * so we have to cancel everything. This is OKish since it
@@ -1897,14 +1893,14 @@ static int _sdp_inet_write_cancel(struct
 		 * connection is marked as being in cancel processing so no
 		 * other writes get into the outbound pipe.
 		 */
-		if (0 == (SDP_CONN_F_SRC_CANCEL_L & conn->flags) &&
-		    0 == (SDP_IOCB_F_CANCEL & iocb->flags)) {
+		if (!(conn->flags & SDP_CONN_F_SRC_CANCEL_L) &&
+		    !(iocb->flags & SDP_IOCB_F_CANCEL)) {
 			conn->src_cncl++;
 			iocb->flags |= SDP_IOCB_F_CANCEL;
 
 			if (conn->src_cncl == sdp_iocb_q_size(&conn->w_src)) {
 				result = sdp_send_ctrl_src_cancel(conn);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 
 				conn->flags |= SDP_CONN_F_SRC_CANCEL_L;
 			}
@@ -1944,21 +1940,21 @@ static int _sdp_send_flush_advt(struct s
 	 * for transmission, and the remote host needs to be notified of
 	 * present data. (rdma ping-pong letency test...)
 	 */
-	if (0 == sdp_desc_q_size(&conn->send_queue)) {
+	if (!sdp_desc_q_size(&conn->send_queue)) {
 		/*
 		 * might be more aggressive then we want it to be. maybe
 		 * check if the active sink queue is empty as well?
 		 */
 		advt = sdp_advt_q_look(&conn->snk_pend);
-		if (NULL != advt && 0 < advt->post) {
+		if (advt && advt->post > 0) {
 			advt = sdp_advt_q_get(&conn->snk_pend);
-			SDP_EXPECT((NULL != advt));
+			SDP_EXPECT((advt));
 
 			result = sdp_send_ctrl_rdma_wr(conn, advt->post);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 
 			result = sdp_advt_destroy(advt);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 			/*
 			 * update sink advertisments.
 			 */
@@ -1982,7 +1978,7 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * send credit utilization rules.
 	 */
 	result = _sdp_send_ctrl_buff_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing control", result);
 		goto done;
 	}
@@ -1990,16 +1986,16 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * data flush
 	 */
 	result = _sdp_send_data_queue_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing data queue", result);
 		goto done;
 	}
 	/*
 	 * Sink advertisment flush.
 	 */
-	if (0 < sdp_advt_q_size(&conn->snk_pend)) {
+	if (sdp_advt_q_size(&conn->snk_pend) > 0) {
 		result = _sdp_send_flush_advt(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn,
 				     "Error <%d> flushing sink advertisments",
 				     result);
@@ -2009,9 +2005,9 @@ int sdp_send_flush(struct sdp_opt *conn)
 	/*
 	 * disconnect flush
 	 */
-	if (0 < (SDP_ST_MASK_DIS_PEND & conn->state)) {
+	if (SDP_ST_MASK_DIS_PEND & conn->state) {
 		result = sdp_send_ctrl_disconnect(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> flushing disconnect",
 				     result);
 			goto done;
@@ -2065,7 +2061,7 @@ int sdp_inet_send(struct kiocb *req,
 	 * continue being processed, it'll wait below until the send window
 	 * is opened on sucessful connect, or error on an unsucessful attempt.
 	 */
-	if (0 < (SDP_ST_MASK_CLOSED & conn->istate)) {
+	if (SDP_ST_MASK_CLOSED & conn->istate) {
 		result = -EPIPE;
 		goto done;
 	}
@@ -2092,9 +2088,9 @@ int sdp_inet_send(struct kiocb *req,
 		 * observed...) use a different threshold for urgent
 		 * data to allow some space for sending.
 		 */
-		while (0 < __sdp_inet_write_space(conn, oob)) {
+		while (__sdp_inet_write_space(conn, oob) > 0) {
 			buff = _sdp_send_data_buff_get(conn);
-			if (NULL == buff) {
+			if (!buff) {
 				result = -ENOMEM;
 				goto done;
 			}
@@ -2107,7 +2103,7 @@ int sdp_inet_send(struct kiocb *req,
 			result = memcpy_fromiovec(buff->tail, 
 						  msg->msg_iov,
 						  copy);
-			if (0 > result) {
+			if (result < 0) {
 				(void)sdp_buff_pool_put(buff);
 				goto done;
 			}
@@ -2120,7 +2116,7 @@ int sdp_inet_send(struct kiocb *req,
 			result = _sdp_send_data_buff_put(conn, buff, copy,
 							 ((copied ==
 							 size) ? oob : 0));
-			if (0 > result)
+			if (result < 0)
 				goto done;
 
 			if (copied == size)
@@ -2141,34 +2137,34 @@ skip: /* entry point for IOCB based tran
 		/*
 		 * onetime setup of timeout, but only if it's needed.
 		 */
-		if (0 > timeout)
+		if (timeout < 0)
 			timeout = sock_sndtimeo(sk, (MSG_DONTWAIT &
 						     msg->msg_flags));
 
-		if (0 != SDP_CONN_GET_ERR(conn)) {
-			result = (0 < copied) ? 0 : sdp_conn_error(conn);
+		if (SDP_CONN_GET_ERR(conn)) {
+			result = (copied > 0) ? 0 : sdp_conn_error(conn);
 			break;
 		}
 
-		if (0 < (SEND_SHUTDOWN & conn->shutdown)) {
+		if (SEND_SHUTDOWN & conn->shutdown) {
 			result = -EPIPE;
 			break;
 		}
 
-		if (SDP_SOCK_ST_ERROR == conn->istate) {
+		if (conn->istate == SDP_SOCK_ST_ERROR) {
 			result = -EPROTO; /* error should always be set, but
 					     just in case */
 			break;
 		}
 
-		if (0 == timeout) {
+		if (!timeout) {
 			result = -EAGAIN;
 			break;
 		}
 
 		if (signal_pending(current)) {
 			result =
-			    (0 < timeout) ? sock_intr_errno(timeout) : -EAGAIN;
+			    (timeout > 0) ? sock_intr_errno(timeout) : -EAGAIN;
 			break;
 		}
 		/*
@@ -2190,7 +2186,7 @@ skip: /* entry point for IOCB based tran
 			clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
 			sdp_conn_unlock(conn);
-			if (!(0 < __sdp_inet_write_space(conn, oob)))
+			if (__sdp_inet_write_space(conn, oob) <= 0)
 				timeout = schedule_timeout(timeout);
 			sdp_conn_lock(conn);
 
@@ -2203,7 +2199,7 @@ skip: /* entry point for IOCB based tran
 		 * create IOCB with remaining space
 		 */
 		iocb = sdp_iocb_create();
-		if (NULL == iocb) {
+		if (!iocb) {
 			sdp_dbg_warn(conn, "Failed to allocate IOCB <%Zu:%d>",
 				     size, copied);
 			result = -ENOMEM;
@@ -2220,7 +2216,7 @@ skip: /* entry point for IOCB based tran
 		req->ki_cancel = _sdp_inet_write_cancel;
 
 		result = sdp_iocb_lock(iocb);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> locking IOCB <%Zu:%d>", 
 				     result, size, copied);
         
@@ -2233,7 +2229,7 @@ skip: /* entry point for IOCB based tran
 		conn->send_pipe += iocb->len;
 
 		result = _sdp_send_data_queue(conn, (struct sdpc_desc *)iocb);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> queueing write IOCB",
 				     result);
 
@@ -2247,9 +2243,9 @@ skip: /* entry point for IOCB based tran
 
 done:
 	sdp_conn_unlock(conn);
-	result = ((0 < copied) ? copied : result);
+	result = ((copied > 0) ? copied : result);
 
-	if (-EPIPE == result && 0 == (MSG_NOSIGNAL & msg->msg_flags))
+	if (result == -EPIPE && !(MSG_NOSIGNAL & msg->msg_flags))
 		send_sig(SIGPIPE, current, 0);
 
 	return result;
Index: sdp_actv.c
===================================================================
--- sdp_actv.c	(revision 1922)
+++ sdp_actv.c	(working copy)
@@ -70,7 +70,7 @@ void sdp_cm_actv_error(struct sdp_opt *c
 					IB_CM_REJ_CONSUMER_DEFINED,
 					NULL, 0, NULL, 0);
 
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> sending CM REJ.",
 				     result);
 
@@ -82,7 +82,7 @@ void sdp_cm_actv_error(struct sdp_opt *c
 		 * full disconnect.
 		 */
 		result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", 
 				     result);
 		
@@ -151,7 +151,7 @@ static int _sdp_actv_conn_establish(stru
 	 * post receive buffers.
 	 */
 	result = sdp_recv_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting recv buffers.", result);
 		goto done;
 	}
@@ -175,7 +175,7 @@ static int _sdp_actv_conn_establish(stru
 	 * respond to the remote connection manager with a RTU
 	 */
 	result = ib_send_cm_rtu(conn->cm_id, NULL, 0);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> sending CM RTU.", result);
 		goto done;
 	}
@@ -196,7 +196,7 @@ static int _sdp_actv_conn_establish(stru
 	inet_sk(sk)->rcv_saddr = htonl(conn->src_addr);
 
 	result = sdp_send_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing receives.", result);
 		goto done;
 	}
@@ -238,7 +238,7 @@ static int _sdp_cm_hello_ack_check(struc
 		return -EINVAL;
 	}
 
-	if (!(0 < hello_ack->hah.max_adv)) {
+	if (hello_ack->hah.max_adv <= 0) {
 		sdp_dbg_warn(NULL, "hello ack, bad zcopy advertisment. <%d>",
 			     hello_ack->hah.max_adv);
 		return -EINVAL;
@@ -276,7 +276,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	struct sdp_msg_hello_ack *hello_ack;
 	int result = -EPROTO;
 
-	if (NULL == conn)
+	if (!conn)
 		return -EINVAL;
 	
 	if (cm_id != conn->cm_id) {
@@ -297,7 +297,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	 * the connection.
 	 */
 	result = _sdp_cm_hello_ack_check(hello_ack);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> hello ack check.", result);
 		goto error;
 	}
@@ -326,7 +326,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
 
 	result = _sdp_actv_conn_establish(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> accept receive failed", result);
 		goto error;
 	}
@@ -392,7 +392,7 @@ static void _sdp_cm_path_complete(u64 id
 	/*
 	 * create address handle
 	 */
-	if (0 != status) {
+	if (status) {
 		sdp_dbg_warn(conn, "Path record completion error <%d>",
 			     status);
 		goto failed;
@@ -410,7 +410,7 @@ static void _sdp_cm_path_complete(u64 id
 	 * allocate IB resources.
 	 */
 	result = sdp_conn_alloc_ib(conn, ca, hw_port, path->pkey);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> allocating IB connection",
 			     result);
 		goto failed;
@@ -420,7 +420,7 @@ static void _sdp_cm_path_complete(u64 id
 	 * space reservation)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buff for Hello Msg.");
 		goto failed;
 	}
@@ -462,11 +462,11 @@ static void _sdp_cm_path_complete(u64 id
 	 * save message
 	 */
 	result = sdp_buff_q_put(&conn->send_post, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> buffering hello msg.", result);
 
 		expect = sdp_buff_pool_put(buff);
-		SDP_EXPECT(!(0 > expect));
+		SDP_EXPECT(expect >= 0);
 
 		status = -EPROTO;
 		goto failed;
@@ -475,7 +475,7 @@ static void _sdp_cm_path_complete(u64 id
 	/*
 	 * Mellanox performance bug workaround.
 	 */
-	if (IB_MTU_1024 < path->mtu)
+	if (path->mtu > IB_MTU_1024)
 		path->mtu = IB_MTU_1024;
 #endif
 	conn->path_mtu = path->mtu;
@@ -509,7 +509,7 @@ static void _sdp_cm_path_complete(u64 id
 #endif
 	conn->cm_id = ib_create_cm_id(sdp_cm_event_handler,
 				      __hashent_arg(conn->hashent));
-	if (NULL == conn->cm_id) {
+	if (!conn->cm_id) {
 		sdp_dbg_warn(conn, "Failed to create CM handle, %d",
 			     (u8)(buff->tail - buff->data));
 		goto failed;
@@ -519,7 +519,7 @@ static void _sdp_cm_path_complete(u64 id
 	 * initiate connection
 	 */
 	result = ib_send_cm_req(conn->cm_id, &param);
-	if (0 != result) {
+	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> CM connect request", result);
 		status = result;
 		goto failed;
@@ -566,7 +566,7 @@ int sdp_cm_connect(struct sdp_opt *conn)
 				      &conn->plid);
 	sdp_conn_lock(conn);
 
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> getting link <%08x:%08x> addr",
 			     result,
 			     htonl(conn->dst_addr),
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1922)
+++ sdp_conn.c	(working copy)
@@ -81,10 +81,10 @@ int sdp_inet_accept_q_put(struct sdp_opt
 {
 	struct sdp_opt *next_conn;
 
-	if (NULL != listen_conn->parent ||
-	    NULL != accept_conn->parent ||
-	    NULL == listen_conn->accept_next ||
-	    NULL == listen_conn->accept_prev)
+	if (listen_conn->parent ||
+	    accept_conn->parent ||
+	    !listen_conn->accept_next ||
+	    !listen_conn->accept_prev)
 		return -EFAULT;
 
 	next_conn = listen_conn->accept_next;
@@ -112,9 +112,9 @@ struct sdp_opt *sdp_inet_accept_q_get(st
 	struct sdp_opt *prev_conn;
 	struct sdp_opt *accept_conn;
 
-	if (NULL != listen_conn->parent ||
-	    NULL == listen_conn->accept_next ||
-	    NULL == listen_conn->accept_prev ||
+	if (listen_conn->parent ||
+	    !listen_conn->accept_next ||
+	    !listen_conn->accept_prev ||
 	    listen_conn == listen_conn->accept_next ||
 	    listen_conn == listen_conn->accept_prev)
 		return NULL;
@@ -151,7 +151,7 @@ int sdp_inet_accept_q_remove(struct sdp_
 	struct sdp_opt *next_conn;
 	struct sdp_opt *prev_conn;
 
-	if (NULL == accept_conn->parent)
+	if (!accept_conn->parent)
 		return -EFAULT;
 	/*
 	 * Removes the connection from the listening sockets accept queue.
@@ -185,7 +185,7 @@ int sdp_inet_listen_start(struct sdp_opt
 {
 	unsigned long flags;
 
-	if (SDP_SOCK_ST_CLOSED != conn->istate) {
+	if (conn->istate != SDP_SOCK_ST_CLOSED) {
 		sdp_dbg_warn(conn, "Incorrect connection state to listen.");
 		return -EBADFD;
 	}
@@ -204,7 +204,7 @@ int sdp_inet_listen_start(struct sdp_opt
 	_dev_root_s.listen_list = conn;
 	conn->lstn_p_next = &_dev_root_s.listen_list;
 
-	if (NULL != conn->lstn_next)
+	if (conn->lstn_next)
 		conn->lstn_next->lstn_p_next = &conn->lstn_next;
 
 	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
@@ -220,7 +220,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	int result;
 	unsigned long flags;
 
-	if (SDP_SOCK_ST_LISTEN != listen_conn->istate) {
+	if (listen_conn->istate != SDP_SOCK_ST_LISTEN) {
 		sdp_dbg_warn(listen_conn, "Incorrect state to stop listen.");
 		return -EBADFD;
 	}
@@ -233,7 +233,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	/*
 	 * remove from listening list.
 	 */
-	if (NULL != listen_conn->lstn_next)
+	if (listen_conn->lstn_next)
 		listen_conn->lstn_next->lstn_p_next = listen_conn->lstn_p_next;
 
 	*(listen_conn->lstn_p_next) = listen_conn->lstn_next;
@@ -245,7 +245,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	/*
 	 * reject and delete all pending connections
 	 */
-	while (NULL != (accept_conn = sdp_inet_accept_q_get(listen_conn))) {
+	while ((accept_conn = sdp_inet_accept_q_get(listen_conn))) {
 		/*
 		 * The connection is going to be dropped now, mark the
 		 * state as such in case of conntension for this conn.
@@ -255,7 +255,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 		accept_conn->istate = SDP_SOCK_ST_CLOSED;
 
 		result = sdp_wall_send_abort(accept_conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(accept_conn, "Error <%d> during abort",
 				     result);
 
@@ -285,7 +285,7 @@ struct sdp_opt *sdp_inet_listen_lookup(u
 	/*
 	 * first find a listening connection
 	 */
-	for (conn = _dev_root_s.listen_list; NULL != conn;
+	for (conn = _dev_root_s.listen_list; conn;
 	     conn = conn->lstn_next)
 		if (port == conn->src_port &&
 		    (INADDR_ANY == conn->src_addr || addr == conn->src_addr)) {
@@ -321,9 +321,9 @@ int sdp_inet_port_get(struct sdp_opt *co
 	/*
 	 * simple linked list of sockets ordered on local port number.
 	 */
-	if (0 < port) {
+	if (port > 0) {
 		for (look = _dev_root_s.bind_list, port_ok = 1;
-		     NULL != look; look = look->bind_next) {
+		     look; look = look->bind_next) {
 			srch = look->sk;
 			/*
 			 * 1) same port
@@ -340,9 +340,9 @@ int sdp_inet_port_get(struct sdp_opt *co
 				 * 3) either socket has reuse turned off
 				 * 4) socket already listening on this port
 				 */
-				if (0 == sk->sk_reuse ||
-				    0 == srch->sk_reuse ||
-				    SDP_SOCK_ST_LISTEN == look->istate) {
+				if (!sk->sk_reuse ||
+				    !srch->sk_reuse ||
+				    look->istate == SDP_SOCK_ST_LISTEN) {
 					/*
 					 * 5) neither socket is using a
 					 *    specific address
@@ -370,14 +370,14 @@ int sdp_inet_port_get(struct sdp_opt *co
 			}
 		}
 
-		if (0 == port_ok) {
+		if (!port_ok) {
 			result = -EADDRINUSE;
 			goto done;
 		}
 	} else {
 		low_port = SDP_INET_PORT_LOW;
 		top_port = SDP_INET_PORT_HIGH;
-		rover = (0 > rover) ? low_port : rover;
+		rover = (rover < 0) ? low_port : rover;
 
 		for (counter = (top_port - low_port) + 1; counter > 0;
 		     counter--) {
@@ -386,17 +386,17 @@ int sdp_inet_port_get(struct sdp_opt *co
 				rover = low_port;
 
 			for (look = _dev_root_s.bind_list;
-			     NULL != look && look->src_port != port;
+			     look && look->src_port != port;
 			     look = look->bind_next)
 				do {} while(0); /* pass */
 
-			if (NULL == look) {
+			if (!look) {
 				port = rover;
 				break;
 			}
 		}
 
-		if (0 == port) {
+		if (!port) {
 			result = -EADDRINUSE;
 			goto done;
 		}
@@ -410,7 +410,7 @@ int sdp_inet_port_get(struct sdp_opt *co
 	_dev_root_s.bind_list = conn;
 	conn->bind_p_next = &_dev_root_s.bind_list;
 
-	if (NULL != conn->bind_next)
+	if (conn->bind_next)
 		conn->bind_next->bind_p_next = &conn->bind_next;
 
 	result = 0;
@@ -426,7 +426,7 @@ int sdp_inet_port_put(struct sdp_opt *co
 {
 	unsigned long flags;
 
-	if (NULL == conn->bind_p_next)
+	if (!conn->bind_p_next)
 		return -EADDRNOTAVAIL;
 	/*
 	 * lock table
@@ -435,7 +435,7 @@ int sdp_inet_port_put(struct sdp_opt *co
 	/*
 	 * remove from bind list.
 	 */
-	if (NULL != conn->bind_next)
+	if (conn->bind_next)
 		conn->bind_next->bind_p_next = conn->bind_p_next;
 
 	*(conn->bind_p_next) = conn->bind_next;
@@ -461,7 +461,7 @@ int sdp_inet_port_inherit(struct sdp_opt
 	 */
 	spin_lock_irqsave(&_dev_root_s.bind_lock, flags);
 
-	if (NULL != child->bind_p_next ||
+	if (child->bind_p_next ||
 	    child->src_port != parent->src_port) {
 		sdp_dbg_warn(child, "child already bound. <%d:%d>",
 			     parent->src_port, child->src_port);
@@ -475,7 +475,7 @@ int sdp_inet_port_inherit(struct sdp_opt
 	parent->bind_next = child;
 	child->bind_p_next = &parent->bind_next;
 
-	if (NULL != child->bind_next)
+	if (child->bind_next)
 		child->bind_next->bind_p_next = &child->bind_next;
 
 	result = 0;
@@ -508,7 +508,7 @@ static int _sdp_conn_table_insert(struct
 		if (!(_dev_root_s.sk_rover < _dev_root_s.sk_size))
 			_dev_root_s.sk_rover = 0;
 
-		if (NULL == _dev_root_s.sk_array[_dev_root_s.sk_rover]) {
+		if (!_dev_root_s.sk_array[_dev_root_s.sk_rover]) {
 			_dev_root_s.sk_array[_dev_root_s.sk_rover] = conn;
 			_dev_root_s.sk_entry++;
 			conn->hashent = _dev_root_s.sk_rover;
@@ -546,7 +546,7 @@ int sdp_conn_table_remove(struct sdp_opt
 	if (SDP_DEV_SK_INVALID == conn->hashent)
 		goto done;
 
-	if (0 > conn->hashent || conn != _dev_root_s.sk_array[conn->hashent]) {
+	if (conn->hashent < 0 || conn != _dev_root_s.sk_array[conn->hashent]) {
 		result = -ERANGE;
 		goto done;
 	}
@@ -580,7 +580,7 @@ struct sdp_opt *sdp_conn_table_lookup(s3
 	/*
 	 * validate range
 	 */
-	if (0 > entry || !(_dev_root_s.sk_size > entry)) {
+	if (entry < 0 || entry >= _dev_root_s.sk_size) {
 
 		conn = NULL;
 		goto done;
@@ -588,7 +588,7 @@ struct sdp_opt *sdp_conn_table_lookup(s3
 #endif
 
 	conn = _dev_root_s.sk_array[entry];
-	if (NULL == conn)
+	if (!conn)
 		goto done;
 
 	sdp_conn_hold(conn);
@@ -602,14 +602,14 @@ done:
  */
 static int _sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
 {
-	return ((SDP_DESC_TYPE_IOCB == element->type) ? 0 : -ERANGE);
+	return ((element->type == SDP_DESC_TYPE_IOCB) ? 0 : -ERANGE);
 }
 
 static void _sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 
-	while (NULL != (iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
+	while ((iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
 			(table,
 			 _sdp_desc_q_cancel_lookup_func,
 			 NULL))) {
@@ -653,7 +653,7 @@ int sdp_conn_destruct(struct sdp_opt *co
 	int result = 0;
 	int dump = 0;
 
-	if (NULL == conn) {
+	if (!conn) {
 		sdp_dbg_warn(NULL, "sk destruct, no connection!");
 		result = -EINVAL;
 		goto done;
@@ -671,13 +671,13 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 * remove connection from table
 	 */
 	result = sdp_conn_table_remove(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
 			     result, _dev_root_s.sk_entry,
 			     _dev_root_s.sk_size);
 
 	result = __sdp_conn_stat_dump(conn);
-	SDP_EXPECT(!(0 > result));
+	SDP_EXPECT(result >= 0);
 	/*
 	 * really there shouldn't be anything in these tables, but it's
 	 * really bad if we leave a dangling reference here.
@@ -719,7 +719,7 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 */
 	if (conn->qp) {
 		result = ib_destroy_qp(conn->qp);
-		if (0 > result && -EINVAL != result) {
+		if (result < 0 && result != -EINVAL) {
 			sdp_dbg_warn(conn, "Error <%d> detroying QP", result);
 			dump++;
 		}
@@ -729,7 +729,7 @@ int sdp_conn_destruct(struct sdp_opt *co
 	 */
 	if (conn->recv_cq) {
 		result = ib_destroy_cq(conn->recv_cq);
-		if (0 > result && -EINVAL != result) {
+		if (result < 0 && result != -EINVAL) {
 			sdp_dbg_warn(conn, "Error <%d> detroying recv CQ",
 				     result);
 			dump++;
@@ -738,7 +738,7 @@ int sdp_conn_destruct(struct sdp_opt *co
 
 	if (conn->send_cq) {
 		result = ib_destroy_cq(conn->send_cq);
-		if (0 > result && -EINVAL != result) {
+		if (result < 0 && result != -EINVAL) {
 			sdp_dbg_warn(conn, "Error <%d> detroying send CQ",
 				     result);
 			dump++;
@@ -758,13 +758,13 @@ int sdp_conn_destruct(struct sdp_opt *co
 	/*
 	 * check consistancy
 	 */
-	if (0 > atomic_read(&conn->refcnt))
+	if (atomic_read(&conn->refcnt) < 0)
 		sdp_dbg_warn(conn, "destruct low ref count <%04x>",
 			     atomic_read(&conn->refcnt));
 	/*
 	 * free the OS socket structure
 	 */
-	if (NULL == conn->sk)
+	if (!conn->sk)
 		sdp_dbg_warn(conn, "destruct, no socket! continuing.");
 	else {
 		sk_free(conn->sk);
@@ -776,9 +776,9 @@ int sdp_conn_destruct(struct sdp_opt *co
 	result = 0;
 done:
 
-	if (0 != dump) {
+	if (dump) {
 		result = __sdp_conn_state_dump(conn);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 	}
 
 	return result;
@@ -800,7 +800,7 @@ void sdp_conn_internal_lock(struct sdp_o
 		spin_lock_irqsave(&(conn->lock.slock), f);
 		*flags = f;
 
-		if (0 == conn->lock.users)
+		if (!conn->lock.users)
 			break;
 	}
 
@@ -826,7 +826,7 @@ void sdp_conn_relock(struct sdp_opt *con
 		result_r = ib_poll_cq(conn->recv_cq, 1, &entry);
 		if (1 == result_r) {
 			result = sdp_cq_event_locked(&entry, conn);
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> from event handler.",
 					     result);
@@ -837,17 +837,17 @@ void sdp_conn_relock(struct sdp_opt *con
 		result_s = ib_poll_cq(conn->send_cq, 1, &entry);
 		if (1 == result_s) {
 			result = sdp_cq_event_locked(&entry, conn);
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, 
 					     "Error <%d> from event handler.",
 					     result);
 			rearm = 1;
 		}
 
-		if (0 != result_r || 0 != result_s)
+		if (result_r || result_s)
 			continue;
 
-		if (0 < rearm) {
+		if (rearm > 0) {
 			result = ib_req_notify_cq(conn->recv_cq,
 						  IB_CQ_NEXT_COMP);
 			if (result)
@@ -896,7 +896,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 			 * to be armed.
 			 */
 			result = sdp_cq_event_locked(&entry, conn);
-			if (0 > result)
+			if (result < 0)
 				sdp_dbg_warn(conn, "Error <%d> event handler.",
 					     result);
 
@@ -907,7 +907,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 		}
 
 		if (!result) {
-			if (0 < rearm) {
+			if (rearm > 0) {
 				result = ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
 				if (result)
 					sdp_dbg_warn(conn, 
@@ -933,11 +933,11 @@ void sdp_conn_internal_unlock(struct sdp
 	/*
 	 * poll CQs for events.
 	 */
-	if (NULL != conn) {
-		if (0 < (SDP_CONN_F_RECV_CQ_PEND & conn->flags))
+	if (conn) {
+		if (conn->flags & SDP_CONN_F_RECV_CQ_PEND)
 			calls += sdp_conn_cq_drain(conn->recv_cq, conn);
 
-		if (0 < (SDP_CONN_F_SEND_CQ_PEND & conn->flags))
+		if (conn->flags & SDP_CONN_F_SEND_CQ_PEND)
 			calls += sdp_conn_cq_drain(conn->send_cq, conn);
 
 		conn->flags &= ~SDP_CONN_F_MASK_EVENT;
@@ -976,7 +976,7 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	if (!hca)
 		return -ERANGE;
 
-	for (port = hca->port_list; NULL != port; port = port->next)
+	for (port = hca->port_list; port; port = port->next)
 		if (hw_port == port->index)
 			break;
 
@@ -986,13 +986,13 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	 * allocate creation parameters
 	 */
 	qp_attr = kmalloc(sizeof(*qp_attr),  GFP_KERNEL);
-	if (NULL == qp_attr) {
+	if (!qp_attr) {
 		result = -ENOMEM;
 		goto error_attr;
 	}
   
 	init_attr = kmalloc(sizeof(*init_attr), GFP_KERNEL);
-	if (NULL == init_attr) {
+	if (!init_attr) {
 		result = -ENOMEM;
 		goto error_param;
 	}
@@ -1028,7 +1028,7 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 		conn->send_cq_size = conn->send_cq->cqe;
 
 		result = ib_req_notify_cq(conn->send_cq, IB_CQ_NEXT_COMP);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> arming send CQ.",
 				     result);
 			goto error_rcq;
@@ -1052,7 +1052,7 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 		conn->recv_cq_size = conn->recv_cq->cqe;
 
 		result = ib_req_notify_cq(conn->recv_cq, IB_CQ_NEXT_COMP);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> arming recv CQ.",
 				     result);
 			goto error_qp;
@@ -1106,7 +1106,7 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 
 		result = ib_modify_qp(conn->qp, qp_attr, attr_mask);
 
-		if (0 != result) {
+		if (result) {
 			sdp_dbg_warn(conn, "Error <%d> modifying QP", result);
 			goto error_mod;
 		}
@@ -1143,7 +1143,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	int result;
 
 	sk = sk_alloc(_dev_root_s.proto, priority, 1, _dev_root_s.sock_cache);
-	if (NULL == sk) {
+	if (!sk) {
 		sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>",
 			     _dev_root_s.proto, priority);
 		return NULL;
@@ -1171,7 +1171,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	 * create/modifies must be in that context.
 	 */
 	conn = kmem_cache_alloc(_dev_root_s.conn_cache, priority);
-	if (NULL == conn) {
+	if (!conn) {
 		sdp_dbg_warn(conn, "connection alloc error. <%d>", priority);
 		result = -ENOMEM;
 		goto error;
@@ -1301,7 +1301,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	 * insert connection into lookup table
 	 */
 	result = _sdp_conn_table_insert(conn);
-	if (0 > result) {
+	if (result < 0) {
 
 		sdp_dbg_warn(conn, "Error <%d> conn table insert <%d:%d>",
 			     result, _dev_root_s.sk_entry,
@@ -1366,7 +1366,7 @@ int sdp_proc_dump_conn_main(char *buffer
 	/*
 	 * header should only be printed once
 	 */
-	if (0 == start_index) {
+	if (!start_index) {
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_MAIN_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_MAIN_SEP);
 	}
@@ -1386,7 +1386,7 @@ int sdp_proc_dump_conn_main(char *buffer
 	     counter < _dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_MAIN_SIZE > (max_size - offset));
 	     counter++) {
-		if (NULL == _dev_root_s.sk_array[counter])
+		if (!_dev_root_s.sk_array[counter])
 			continue;
 
 		conn = _dev_root_s.sk_array[counter];
@@ -1471,7 +1471,7 @@ int sdp_proc_dump_conn_data(char *buffer
 	/*
 	 * header should only be printed once
 	 */
-	if (0 == start_index) {
+	if (!start_index) {
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_SEP);
 	}
@@ -1490,7 +1490,7 @@ int sdp_proc_dump_conn_data(char *buffer
 	for (counter = start_index; counter < _dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_DATA_SIZE > (max_size - offset));
 	     counter++) {
-		if (NULL == _dev_root_s.sk_array[counter])
+		if (!_dev_root_s.sk_array[counter])
 			continue;
 
 		conn = _dev_root_s.sk_array[counter];
@@ -1567,7 +1567,7 @@ int sdp_proc_dump_conn_rdma(char *buffer
 	/*
 	 * header should only be printed once
 	 */
-	if (0 == start_index) {
+	if (!start_index) {
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_RDMA_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_RDMA_SEP);
 	}
@@ -1586,7 +1586,7 @@ int sdp_proc_dump_conn_rdma(char *buffer
 	for (counter = start_index; counter < _dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_RDMA_SIZE > (max_size - offset));
 	     counter++) {
-		if (NULL == _dev_root_s.sk_array[counter])
+		if (!_dev_root_s.sk_array[counter])
 			continue;
 
 		conn = _dev_root_s.sk_array[counter];
@@ -1646,7 +1646,7 @@ int sdp_proc_dump_conn_sopt(char *buffer
 	/*
 	 * header should only be printed once
 	 */
-	if (0 == start_index) {
+	if (!start_index) {
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_SOPT_HEAD);
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_SOPT_SEP);
 	}
@@ -1665,7 +1665,7 @@ int sdp_proc_dump_conn_sopt(char *buffer
 	for (counter = start_index; counter < _dev_root_s.sk_size &&
 		     !(SDP_SOPT_PROC_DUMP_SIZE > (max_size - offset));
 	     counter++) {
-		if (NULL == _dev_root_s.sk_array[counter])
+		if (!_dev_root_s.sk_array[counter])
 			continue;
 
 		conn = _dev_root_s.sk_array[counter];
@@ -1706,7 +1706,7 @@ int sdp_proc_dump_device(char *buffer,
 	/*
 	 * header should only be printed once
 	 */
-	if (0 == start_index) {
+	if (!start_index) {
 		offset += sprintf((buffer + offset),
 				  "connection table maximum: <%d>\n",
 				  _dev_root_s.sk_size);
@@ -1755,7 +1755,7 @@ static void sdp_device_init_one(struct i
 	 * allocate per-HCA structure
 	 */
 	hca = kmalloc(sizeof(struct sdev_hca), GFP_KERNEL);
-	if (NULL == hca) {
+	if (!hca) {
 		sdp_warn("Error allocating HCA <%s> memory.", device->name);
 		return;
 	}
@@ -1819,7 +1819,7 @@ static void sdp_device_init_one(struct i
 	     port_count < device->phys_port_cnt;
 	     port_count++) {
 		port = kmalloc(sizeof(struct sdev_hca_port), GFP_KERNEL);
-		if (NULL == port) {
+		if (!port) {
 			sdp_warn("Error allocating HCA <%s> port <%d:%d>",
 				 device->name, port_count,
 				 device->phys_port_cnt);
@@ -1837,7 +1837,7 @@ static void sdp_device_init_one(struct i
 				      port->index, 
 				      0,	/* index */
 				      &port->gid);
-		if (0 != result) {
+		if (result) {
 			sdp_warn("Error <%d> getting GID for HCA <%s:%d:%d>",
 				 result, device->name,
 				 port->index, device->phys_port_cnt);
@@ -1850,7 +1850,7 @@ static void sdp_device_init_one(struct i
 	return;
 
 error:
-	while (NULL != hca->port_list) {
+	while (hca->port_list) {
 		port = hca->port_list;
 		hca->port_list = port->next;
 		port->next = NULL;
@@ -1881,12 +1881,12 @@ static void sdp_device_remove_one(struct
 
 	hca = ib_get_client_data(device, &sdp_client);
 
-	if (NULL == hca) {
+	if (!hca) {
 		sdp_warn("Device <%s> has no HCA info.", device->name);
 		return;
 	}
 	
-	while (NULL != hca->port_list) {
+	while (hca->port_list) {
 		port = hca->port_list;
 		hca->port_list = port->next;
 		port->next = NULL;
@@ -1953,14 +1953,14 @@ int sdp_conn_table_init(int proto_family
 	 * Get HCA/port list
 	 */
 	result = ib_register_client(&sdp_client);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("Error <%d> registering SDP client.", result);
 		goto error_hca;
 	}
 	/*
 	 * create socket table
 	 */
-	if (!(0 < conn_size)) {
+	if (conn_size <= 0) {
 		sdp_warn("Invalid connection table size. <%d>", conn_size);
 		result = -EINVAL;
 		goto error_size;
@@ -1973,7 +1973,7 @@ int sdp_conn_table_init(int proto_family
 
 	_dev_root_s.sk_array = (void *) __get_free_pages(GFP_KERNEL,
 						       _dev_root_s.sk_ordr);
-	if (NULL == _dev_root_s.sk_array) {
+	if (!_dev_root_s.sk_array) {
 		sdp_warn("Failed to create connection table. <%d:%d:%d>",
 			 byte_size, page_size, _dev_root_s.sk_ordr);
 		result = -ENOMEM;
@@ -1989,7 +1989,7 @@ int sdp_conn_table_init(int proto_family
 	 * IOCB table
 	 */
 	result = sdp_main_iocb_init();
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("Error <%d> initializing SDP IOCB table.", result);
 		goto error_iocb;
 	}
@@ -1998,7 +1998,7 @@ int sdp_conn_table_init(int proto_family
 						   sizeof(struct sdp_opt),
 						   0, SLAB_HWCACHE_ALIGN,
 						   NULL, NULL);
-	if (NULL == _dev_root_s.conn_cache) {
+	if (!_dev_root_s.conn_cache) {
 		sdp_warn("Failed to initialize connection cache.");
 		result = -ENOMEM;
 		goto error_conn;
@@ -2008,7 +2008,7 @@ int sdp_conn_table_init(int proto_family
 						   sizeof(struct inet_sock), 
 						   0, SLAB_HWCACHE_ALIGN,
 						   NULL, NULL);
-        if (NULL == _dev_root_s.sock_cache) {
+        if (!_dev_root_s.sock_cache) {
 		sdp_warn("Failed to initialize sock cache.");
 		result = -ENOMEM;
 		goto error_sock;
@@ -2018,7 +2018,7 @@ int sdp_conn_table_init(int proto_family
 	 * start listening
 	 */
 	result = sdp_cm_listen_start(&_dev_root_s);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("Error <%d> listening for connections on HCA.",
 			 result);
 		goto error_listen;
@@ -2052,7 +2052,7 @@ int sdp_conn_table_clear(void)
 	/*
 	 * drain all the connections
 	 */
-	while (NULL != (conn = _dev_root_s.conn_list)) {
+	while ((conn = _dev_root_s.conn_list)) {
 
 	}
 #endif
Index: sdp_advt.c
===================================================================
--- sdp_advt.c	(revision 1922)
+++ sdp_advt.c	(working copy)
@@ -52,7 +52,7 @@ struct sdpc_advt *sdp_advt_create(void)
 	struct sdpc_advt *advt;
 
 	advt = kmem_cache_alloc(__sdp_advt_cache, SLAB_KERNEL);
-	if (NULL != advt) {
+	if (advt) {
 		advt->next = NULL;
 		advt->prev = NULL;
 		advt->size = 0;
@@ -72,7 +72,7 @@ struct sdpc_advt *sdp_advt_create(void)
  */
 int sdp_advt_destroy(struct sdpc_advt *advt)
 {
-	if (NULL != advt->next || NULL != advt->prev)
+	if (advt->next || advt->prev)
 		return -EACCES;
 	/*
 	 * return the object to its cache
@@ -92,7 +92,7 @@ struct sdpc_advt *sdp_advt_q_get(struct 
 	struct sdpc_advt *prev;
 
 	advt = table->head;
-	if (NULL == advt)
+	if (!advt)
 		return NULL;
 
 	if (advt->next == advt && advt->prev == advt)
@@ -130,7 +130,7 @@ void sdp_advt_q_put(struct sdpc_advt_q *
 	struct sdpc_advt *next;
 	struct sdpc_advt *prev;
 
-	if (NULL == table->head) {
+	if (!table->head) {
 		advt->next = advt;
 		advt->prev = advt;
 		table->head = advt;
@@ -168,9 +168,9 @@ void sdp_advt_q_clear(struct sdpc_advt_q
 	/*
 	 * drain the table of any objects
 	 */
-	while (NULL != (advt = sdp_advt_q_get(table))) {
+	while ((advt = sdp_advt_q_get(table))) {
 		result = sdp_advt_destroy(advt);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 	}
 }
 
@@ -187,7 +187,7 @@ int sdp_main_advt_init(void)
 	/*
 	 * initialize the caches only once.
 	 */
-	if (NULL != __sdp_advt_cache) {
+	if (__sdp_advt_cache) {
 		sdp_warn("Advertisment caches already initialized.");
 		return -EINVAL;
 	}
@@ -196,7 +196,7 @@ int sdp_main_advt_init(void)
 					     sizeof(struct sdpc_advt),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (NULL == __sdp_advt_cache)
+	if (!__sdp_advt_cache)
 		return -ENOMEM;
 
 	return 0;
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1922)
+++ sdp_recv.c	(working copy)
@@ -52,7 +52,7 @@ static int _sdp_post_recv_buff(struct sd
 	 * get a buffer
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "failed to allocate buff for recv queue.");
 		result = -ENOMEM;
 		goto error;
@@ -75,7 +75,7 @@ static int _sdp_post_recv_buff(struct sd
 	 * post function returns.
 	 */
 	result = sdp_buff_q_put_tail(&conn->recv_post, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> queuing recv buffer.", result);
 		goto drop;
 	}
@@ -97,7 +97,7 @@ static int _sdp_post_recv_buff(struct sd
 	receive_param.num_sge    = 1;
 
 	result = ib_post_recv(conn->qp, &receive_param, &bad_wr);
-	if (0 != result) {
+	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> posting receive buffer",
 			     result);
 		(void)sdp_buff_q_get_tail(&conn->recv_post);
@@ -134,7 +134,7 @@ static int _sdp_post_rdma_buff(struct sd
 	 * get a reference to the first SrcAvail advertisment.
 	 */
 	advt = sdp_advt_q_look(&conn->src_pend);
-	if (NULL == advt) {
+	if (!advt) {
 		result = ENODEV;
 		goto done;
 	}
@@ -142,7 +142,7 @@ static int _sdp_post_rdma_buff(struct sd
 	 * get a buffer
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "failed to allocate buff for rdma read.");
 		result = -ENOMEM;
 		goto error;
@@ -172,7 +172,7 @@ static int _sdp_post_rdma_buff(struct sd
 	 * If there is no more advertised space move the advertisment to the
 	 * active list, and match the WRID.
 	 */
-	if (!(0 < advt->size))
+	if (advt->size <= 0)
 		sdp_advt_q_put(&conn->src_actv,
 			       sdp_advt_q_get(&conn->src_pend));
 	/*
@@ -181,7 +181,7 @@ static int _sdp_post_rdma_buff(struct sd
 	 * post function returns.
 	 */
 	result = sdp_desc_q_put_tail(&conn->r_src, (struct sdpc_desc *) buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> queuing rdma read.", result);
 		goto drop;
 	}
@@ -202,7 +202,7 @@ static int _sdp_post_rdma_buff(struct sd
 	send_param.num_sge = 1;
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
-	if (0 != result) {
+	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> posting rdma read", result);
 
 		(void)sdp_desc_q_get_tail(&conn->r_src);
@@ -243,7 +243,7 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * get a reference to the first SrcAvail advertisment.
 	 */
 	advt = sdp_advt_q_look(&conn->src_pend);
-	if (NULL == advt) {
+	if (!advt) {
 		result = ENODEV;
 		goto done;
 	}
@@ -254,7 +254,7 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * (final complete RDMA will clear it out.)
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_pend);
-	if (NULL == iocb) {
+	if (!iocb) {
 		result = ENODEV;
 		goto done;
 	}
@@ -262,7 +262,7 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * register IOCBs physical memory.
 	 */
 	result = sdp_iocb_register(iocb, conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> registering IOCB. <%d:%d>",
 			     result, iocb->key, iocb->len);
 		goto error;
@@ -297,15 +297,15 @@ static int _sdp_post_rdma_iocb_src(struc
 	 * if there is no more advertised space,  queue the
 	 * advertisment for completion
 	 */
-	if (!(0 < advt->size))
+	if (advt->size <= 0)
 		sdp_advt_q_put(&conn->src_actv,
 			       sdp_advt_q_get(&conn->src_pend));
 	/*
 	 * if there is no more iocb space queue the it for completion
 	 */
-	if (!(0 < iocb->len)) {
+	if (iocb->len <= 0) {
 		iocb = sdp_iocb_q_get_head(&conn->r_pend);
-		if (NULL == iocb) {
+		if (!iocb) {
 			sdp_dbg_warn(conn, "read IOCB disappeared. <%d>",
 				     sdp_iocb_q_size(&conn->r_pend));
 			result = -ENODEV;
@@ -314,7 +314,7 @@ static int _sdp_post_rdma_iocb_src(struc
 
 		result = sdp_desc_q_put_tail(&conn->r_src,
 					     (struct sdpc_desc *)iocb);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> queuing read IOCB",
 				     result);
 			(void)sdp_iocb_destroy(iocb);
@@ -338,7 +338,7 @@ static int _sdp_post_rdma_iocb_src(struc
 	send_param.num_sge = 1;
 
 	result = ib_post_send(conn->qp, &send_param, &bad_wr);
-	if (0 != result) {
+	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> posting rdma read", result);
 		conn->s_wq_size--;
 		goto error;
@@ -361,7 +361,7 @@ static int _sdp_post_rdma_iocb_snk(struc
 	/*
 	 * check if sink cancel is pending
 	 */
-	if (0 < (SDP_CONN_F_SNK_CANCEL & conn->flags)) {
+	if (conn->flags & SDP_CONN_F_SNK_CANCEL) {
 		result = ENODEV;
 		goto error;
 	}
@@ -369,7 +369,7 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 * get the pending iocb
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_pend);
-	if (NULL == iocb) {
+	if (!iocb) {
 		result = ENODEV;
 		goto error;
 	}
@@ -393,7 +393,7 @@ static int _sdp_post_rdma_iocb_snk(struc
 	result = sdp_iocb_register(iocb, conn);
 	if (result) {
 		result = (-EAGAIN == result ? EAGAIN : result);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> registering IOCB",
 				     result);
 
@@ -408,14 +408,14 @@ static int _sdp_post_rdma_iocb_snk(struc
 	 * queue IOCB
 	 */
 	iocb = sdp_iocb_q_get_head(&conn->r_pend);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "read IOCB missing from pending table <%d>",
 			     sdp_iocb_q_size(&conn->r_pend));
 		goto release;
 	}
 
 	result = sdp_iocb_q_put_tail(&conn->r_snk, iocb);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> queueing active write IOCB",
 			     result);
 		goto re_q;
@@ -427,7 +427,7 @@ static int _sdp_post_rdma_iocb_snk(struc
 					 iocb->len,
 					 iocb->r_key,
 					 iocb->io_addr);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> sending SnkAvail message",
 			     result);
 		goto de_q;
@@ -462,15 +462,15 @@ static int _sdp_post_rdma(struct sdp_opt
 	 * the sink advertisment, something to explore, but SrcAvail
 	 * slow start might make that unneccessart?
 	 */
-	if (0 == (SDP_ST_MASK_SEND_OK & conn->state))
+	if (!(SDP_ST_MASK_SEND_OK & conn->state))
 		return 0;
 	/*
 	 * loop flushing IOCB RDMAs. Read sources, otherwise post sinks.
 	 */
-	if (0 < sdp_advt_q_size(&conn->src_pend)) {
-		if (0 == sdp_desc_q_types_size(&conn->r_src,
+	if (sdp_advt_q_size(&conn->src_pend) > 0) {
+		if (!sdp_desc_q_types_size(&conn->r_src,
 					       SDP_DESC_TYPE_BUFF))
-			while (0 == (result = _sdp_post_rdma_iocb_src(conn))) {
+			while (!(result = _sdp_post_rdma_iocb_src(conn))) {
 				/*
 				 * pass, nothing to do in loop.
 				 */
@@ -478,7 +478,7 @@ static int _sdp_post_rdma(struct sdp_opt
 		/*
 		 * check non-zero result
 		 */
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> posting RDMA IOCB read",
 				     result);
 			goto done;
@@ -486,8 +486,8 @@ static int _sdp_post_rdma(struct sdp_opt
 		/*
 		 * loop posting RDMA reads, if there is room.
 		 */
-		if (0 == sdp_iocb_q_size(&conn->r_pend))
-			while (0 < sdp_advt_q_size(&conn->src_pend) &&
+		if (!sdp_iocb_q_size(&conn->r_pend))
+			while (sdp_advt_q_size(&conn->src_pend) > 0 &&
 			       conn->recv_max >
 			       sdp_buff_q_size(&conn->recv_pool) &&
 			       conn->rwin_max > conn->byte_strm) {
@@ -501,22 +501,22 @@ static int _sdp_post_rdma(struct sdp_opt
 		/*
 		 * check non-zero result
 		 */
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> posting RDMA BUFF read",
 				     result);
 			goto done;
 		}
 	} else {
-		if (0 < sdp_iocb_q_size(&conn->r_pend) &&
-		    SDP_MODE_PIPE == conn->recv_mode &&
-		    0 == sdp_advt_q_size(&conn->src_actv))
-			while (0 == (result = _sdp_post_rdma_iocb_snk(conn))) {
+		if (sdp_iocb_q_size(&conn->r_pend) > 0 &&
+		    conn->recv_mode == SDP_MODE_PIPE &&
+		    !sdp_advt_q_size(&conn->src_actv))
+			while (!(result = _sdp_post_rdma_iocb_snk(conn))) {
 				/*
 				 * pass
 				 */
 			}
 
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> posting RDMA read sink",
 				     result);
 			goto done;
@@ -539,7 +539,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	/*
 	 * verify that the connection is in a posting state
 	 */
-	if (0 == (SDP_ST_MASK_RCV_POST & conn->state))
+	if (!(SDP_ST_MASK_RCV_POST & conn->state))
 		return 0;
 	/*
 	 * loop posting receive buffers onto the queue
@@ -563,7 +563,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	counter = min(counter, 
 		      ((s32)conn->recv_cq_size - (s32)conn->l_recv_bf));
 
-	while (0 < counter--) {
+	while (counter-- > 0) {
 		result = _sdp_post_recv_buff(conn);
 		if (result)
 			/*
@@ -572,7 +572,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 			break;
 	}
 
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting recv buff.", result);
 		goto done;
 	}
@@ -580,8 +580,8 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 * If we are in Sink Cancel processing, and the active sink queue has
 	 * been consumed, we can come out of sink processing.
 	 */
-	if (0 < (SDP_CONN_F_SNK_CANCEL & conn->flags) &&
-	    0 == sdp_iocb_q_size(&conn->r_snk))
+	if ((conn->flags & SDP_CONN_F_SNK_CANCEL) &&
+	    !sdp_iocb_q_size(&conn->r_snk))
 		conn->flags &= ~SDP_CONN_F_SNK_CANCEL;
 	/*
 	 * Next the connection should consume RDMA Source advertisments or
@@ -590,7 +590,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 * connection peer.
 	 */
 	result = _sdp_post_rdma(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting RDMAs.", result);
 		goto done;
 	}
@@ -610,12 +610,12 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 * 4) The peer has no source or sink advertisments pending. In process
 	 *    advertisments generate completions, that's why no ack.
 	 */
-	if ((3 > conn->l_advt_bf &&
+	if ((conn->l_advt_bf < 3&&
 	     conn->l_recv_bf > conn->l_advt_bf) ||
 	    (SDP_RECV_POST_ACK < (conn->l_recv_bf - conn->l_advt_bf) &&
-	     0 == ((u32)conn->snk_recv + (u32)conn->src_recv))) {
+	     !((u32)conn->snk_recv + (u32)conn->src_recv))) {
 		result = sdp_send_ctrl_ack(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> posting gratuitous ACK",
 				     result);
 			goto done;
@@ -653,7 +653,7 @@ static int _sdp_read_buff_iocb(struct sd
 	data = buff->data;
 	tail = buff->tail;
 
-	buff->tail -= (0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) ? 1 : 0;
+	buff->tail -= (buff->flags & SDP_BUFF_F_OOB_PRES) ? 1 : 0;
 	/*
 	 * initialize counter to correct page and offset.
 	 */
@@ -661,12 +661,12 @@ static int _sdp_read_buff_iocb(struct sd
 	offset  = (iocb->post + iocb->page_offset) & (~PAGE_MASK);
 
 	while (buff->data < buff->tail &&
-	       0 < iocb->len) {
+	       iocb->len > 0) {
 		/*
 		 * map correct page of iocb
 		 */
 		addr = __sdp_kmap(iocb->page_array[counter]);
-		if (NULL == addr)
+		if (!addr)
 			break;
 
 		copy = min((PAGE_SIZE - offset),
@@ -706,18 +706,18 @@ static int _sdp_recv_buff_iocb_active(st
 	 * Get the IOCB, We'll fill with exactly one
 	 */
 	iocb = sdp_iocb_q_get_head(&conn->r_snk);
-	if (NULL == iocb) {
+	if (!iocb) {
 		sdp_dbg_warn(conn, "Empty active IOCB queue. <%d>",
 			     sdp_iocb_q_size(&conn->r_snk));
 		return -EPROTO;
 	}
 
-	SDP_EXPECT((0 < (SDP_IOCB_F_RDMA_W & iocb->flags)));
+	SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_W));
 	/*
 	 * TODO: need to be checking OOB here.
 	 */
 	result = _sdp_read_buff_iocb(iocb, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
 			     (unsigned)(buff->tail - buff->data));
@@ -735,7 +735,7 @@ static int _sdp_recv_buff_iocb_active(st
 	 * callback to complete IOCB
 	 */
 	result = sdp_iocb_complete(iocb, 0);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 			     result, iocb->key);
 
@@ -755,7 +755,7 @@ static int _sdp_recv_buff_iocb_pending(s
 	 * check the IOCB
 	 */
 	iocb = sdp_iocb_q_look(&conn->r_pend);
-	if (NULL == iocb) {
+	if (!iocb) {
 		sdp_dbg_warn(conn, "Empty pending IOCB queue. <%d>",
 			     sdp_iocb_q_size(&conn->r_pend));
 		return -EPROTO;
@@ -764,7 +764,7 @@ static int _sdp_recv_buff_iocb_pending(s
 	 * TODO: need to be checking OOB here.
 	 */
 	result = _sdp_read_buff_iocb(iocb, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
 			     (unsigned)(buff->tail - buff->data));
@@ -779,14 +779,14 @@ static int _sdp_recv_buff_iocb_pending(s
 	 *    b) the amount of data moved into the IOCB is greater then the
 	 *       socket recv low water mark.
 	 */
-	if (0 == iocb->len ||
-	    (0 == conn->src_recv &&
+	if (!iocb->len ||
+	    (!conn->src_recv &&
 	     !(conn->sk->sk_rcvlowat > iocb->post))) {
 		/*
 		 * complete IOCB
 		 */
 		iocb = sdp_iocb_q_get_head(&conn->r_pend);
-		SDP_EXPECT((NULL != iocb));
+		SDP_EXPECT((iocb));
 
 		SDP_CONN_STAT_READ_INC(conn, iocb->post);
 		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
@@ -794,7 +794,7 @@ static int _sdp_recv_buff_iocb_pending(s
 		 * callback to complete IOCB
 		 */
 		result = sdp_iocb_complete(iocb, 0);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>",
 				     result, iocb->key);
 	}
@@ -817,14 +817,14 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	 * if data is received and the receive half of the connection has been
 	 * closed. This notifies the peer that the data was not received.
 	 */
-	if (0 < (RCV_SHUTDOWN & conn->shutdown)) {
+	if (RCV_SHUTDOWN & conn->shutdown) {
 		sdp_dbg_warn(conn, "Receive data path closed. <%02x>",
 			     conn->shutdown);
 		/*
 		 * abort connection (send reset)
 		 */
 		result = sdp_wall_abort(conn);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 		/*
 		 * drop packet
 		 */
@@ -834,7 +834,7 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	/*
 	 * oob notification.
 	 */
-	if (0 < (SDP_BUFF_F_OOB_PEND & buff->flags)) {
+	if (buff->flags & SDP_BUFF_F_OOB_PEND) {
 		conn->rcv_urg_cnt++;
 		sdp_inet_wake_urg(conn->sk);
 	}
@@ -842,7 +842,7 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	 * loop while there are available IOCB's, break if there is no
 	 * more data to read
 	 */
-	while (0 < (sdp_iocb_q_size(&conn->r_pend) +
+	while ((sdp_iocb_q_size(&conn->r_pend) +
 		    sdp_iocb_q_size(&conn->r_snk))) {
 		/*
 		 * if there is OOB data in a buffer, the two functions below
@@ -852,7 +852,7 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 		 * will not be consumed until the next AIO buffer is posted,
 		 * or a socket recv (regular or OOB) is called.
 		 */
-		if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags) &&
+		if ((buff->flags & SDP_BUFF_F_OOB_PRES) &&
 		    1 == (buff->tail - buff->data))
 			break;
 		/*
@@ -860,17 +860,17 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 		 * discarded with exactly one buffer, or process a pending
 		 * IOCB.
 		 */
-		if (0 < conn->snk_sent)
+		if (conn->snk_sent > 0)
 			result = _sdp_recv_buff_iocb_active(conn, buff);
 		else
 			result = _sdp_recv_buff_iocb_pending(conn, buff);
 		/*
 		 * Check result. Postitive result is data left in the buffer
 		 */
-		if (0 == result)
+		if (!result)
 			break;
 
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, 
 				     "Error <%d> processing IOCB. <%d:%d:%d>",
 				     result, conn->snk_sent,
@@ -884,9 +884,9 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 	 */
 	buffered = buff->tail - buff->data;
 
-	if (0 < buffered) {
+	if (buffered > 0) {
 		result = sdp_buff_q_put_tail(&conn->recv_pool, buff);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 	}
 
 	return buffered;
@@ -906,7 +906,7 @@ static int _sdp_read_src_lookup(struct s
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
 
-	if (SDP_DESC_TYPE_IOCB == element->type && iocb->key == req->ki_key)
+	if (element->type == SDP_DESC_TYPE_IOCB && iocb->key == req->ki_key)
 		return 0;
 	else
 		return -ERANGE;
@@ -926,7 +926,7 @@ static int _sdp_inet_read_cancel(struct 
 	sdp_dbg_ctrl(NULL, "Cancel Read IOCB. user <%d> key <%d> flag <%08lx>",
 		     req->ki_users, req->ki_key, req->ki_flags);
 	
-	if (NULL == si || NULL == si->sock || NULL == si->sock->sk) {
+	if (!si || !si->sock || !si->sock->sk) {
 		sdp_warn("Cancel empty read IOCB. users <%d> flags <%d:%08lx>",
 			 req->ki_users, req->ki_key, req->ki_flags);
 		result = -EFAULT;
@@ -946,7 +946,7 @@ static int _sdp_inet_read_cancel(struct 
 	 * whether this is a read or write.
 	 */
 	iocb = sdp_iocb_q_lookup(&conn->r_pend, req->ki_key);
-	if (NULL != iocb) {
+	if (iocb) {
 		/*
 		 * always remove the IOCB. If active, then place it into
 		 * the correct active queue. Inactive empty IOCBs can be
@@ -954,19 +954,19 @@ static int _sdp_inet_read_cancel(struct 
 		 */
 		sdp_iocb_q_remove(iocb);
 
-		if (0 == (SDP_IOCB_F_ACTIVE & iocb->flags)) {
-			if (0 < iocb->post) {
+		if (!(iocb->flags & SDP_IOCB_F_ACTIVE)) {
+			if (iocb->post > 0) {
 				/*
 				 * callback to complete IOCB, or drop reference
 				 */
 				result = sdp_iocb_complete(iocb, 0);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 
 				result = -EAGAIN;
 			}
 			else {
 				result = sdp_iocb_destroy(iocb);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 				/*
 				 * completion reference
 				 */
@@ -978,16 +978,16 @@ static int _sdp_inet_read_cancel(struct 
 			goto unlock;
 		}
 
-		if (0 < (SDP_IOCB_F_RDMA_W & iocb->flags)) {
+		if (iocb->flags & SDP_IOCB_F_RDMA_W) {
 			result = sdp_iocb_q_put_tail(&conn->r_snk, iocb);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 		else {
-			SDP_EXPECT((SDP_IOCB_F_RDMA_R & iocb->flags));
+			SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_R));
 
 			result = sdp_desc_q_put_tail(&conn->r_src,
 						     (struct sdpc_desc *)iocb);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 	}
 	/*
@@ -997,7 +997,7 @@ static int _sdp_inet_read_cancel(struct 
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->r_src,
 						     _sdp_read_src_lookup,
 						     req);
-	if (NULL != iocb) {
+	if (iocb) {
 		iocb->flags |= SDP_IOCB_F_CANCEL;
 		result = -EAGAIN;
 
@@ -1008,17 +1008,17 @@ static int _sdp_inet_read_cancel(struct 
 	 * needs to be issued.
 	 */
 	iocb = sdp_iocb_q_lookup(&conn->r_snk, req->ki_key);
-	if (NULL != iocb) {
+	if (iocb) {
 		/*
 		 * Unfortunetly there is only a course grain cancel in SDP, so
 		 * we have to cancel everything. This is OKish since it usually
 		 * only happens at connection termination, and the remaining
 		 * source probably will get cancel requests as well.
 		 */
-		if (0 == (SDP_CONN_F_SNK_CANCEL & conn->flags)) {
+		if (!(conn->flags & SDP_CONN_F_SNK_CANCEL)) {
 			
 			result = sdp_send_ctrl_snk_cancel(conn);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 			
 			conn->flags |= SDP_CONN_F_SNK_CANCEL;
 		}
@@ -1060,13 +1060,13 @@ static int _sdp_inet_recv_urg_trav(struc
 	u8 *value = (u8 *) arg;
 	u8 update;
 
-	if (0 < (SDP_BUFF_F_OOB_PRES & buff->flags)) {
+	if (buff->flags & SDP_BUFF_F_OOB_PRES) {
 		SDP_EXPECT((buff->tail > buff->data));
 
 		update = *value;
 		*value = *(u8 *) (buff->tail - 1);
 
-		if (0 < update) {
+		if (update > 0) {
 			buff->tail--;
 			buff->flags &= ~SDP_BUFF_F_OOB_PRES;
 		}
@@ -1092,35 +1092,35 @@ static int _sdp_inet_recv_urg(struct soc
 
 	conn = SDP_GET_CONN(sk);
 
-	if (sock_flag(sk, SOCK_URGINLINE) || 0 == conn->rcv_urg_cnt)
+	if (sock_flag(sk, SOCK_URGINLINE) || !conn->rcv_urg_cnt)
 		return -EINVAL;
 
 	/*
 	 * don't cosume data on PEEK, but do consume data on TRUNC
 	 */
 #if 0
-	value = (0 < (MSG_PEEK & flags)) || (0 == size) ? 0 : 1;
+	value = (MSG_PEEK & flags) || size ? 1 : 0;
 #else
-	value = (0 < (MSG_PEEK & flags)) ? 0 : 1;
+	value = (MSG_PEEK & flags) ? 0 : 1;
 #endif
 
 	result = sdp_buff_q_trav_head(&conn->recv_pool,
 				      _sdp_inet_recv_urg_trav,
 				      (void *)&value);
-	if (-ERANGE != result) {
-		result = (0 != result) ? result : -EAGAIN;
+	if (result != -ERANGE) {
+		result = result ? result : -EAGAIN;
 		goto done;
 	}
 
 	msg->msg_flags |= MSG_OOB;
-	if (0 < size) {
+	if (size > 0) {
 		result = memcpy_toiovec(msg->msg_iov, &value, 1);
-		if (0 != result)
+		if (result)
 			goto done;
 		/*
 		 * clear urgent pointer on consumption
 		 */
-		if (0 == (MSG_PEEK & flags)) {
+		if (!(MSG_PEEK & flags)) {
 			conn->rcv_urg_cnt -= 1;
 			conn->byte_strm -= 1;
 			
@@ -1133,12 +1133,12 @@ static int _sdp_inet_recv_urg(struct soc
 			buff = sdp_buff_q_fetch(&conn->recv_pool,
 						_sdp_inet_recv_urg_test,
 						(void *)0);
-			if (NULL != buff) {
+			if (buff) {
 				result = sdp_buff_pool_put(buff);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 
 				result = sdp_recv_flush(conn);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 			}
 
 			result = 1;
@@ -1191,28 +1191,28 @@ int sdp_inet_recv(struct kiocb  *req,
 	/*
 	 * TODO: unhandled, but need to be handled.
 	 */
-	if (0 < (MSG_TRUNC & flags))
+	if (MSG_TRUNC & flags)
 		return -EOPNOTSUPP;
 
-	if (0 < (MSG_PEEK & flags)) {
+	if (MSG_PEEK & flags) {
 		sdp_buff_q_init(&peek_queue);
 		msg->msg_flags |= MSG_PEEK;
 	}
 
 	sdp_conn_lock(conn);
 
-	if (SDP_SOCK_ST_LISTEN == conn->istate ||
-	    SDP_SOCK_ST_CLOSED == conn->istate) {
+	if (conn->istate == SDP_SOCK_ST_LISTEN ||
+	    conn->istate == SDP_SOCK_ST_CLOSED) {
 		result = -ENOTCONN;
 		goto done;
 	}
 	/*
 	 * process urgent data
 	 */
-	if (0 < (MSG_OOB & flags)) {
+	if (MSG_OOB & flags) {
 		result = _sdp_inet_recv_urg(sk, msg, size, flags);
-		copied = (0 < result) ? result : 0;
-		result = (0 < result) ? 0 : result;
+		copied = (result > 0) ? result : 0;
+		result = (result > 0) ? 0 : result;
 		goto done;
 	}
 	/*
@@ -1232,7 +1232,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			length = buff->tail - buff->data;
 			update = 0;
 
-			if (SDP_BUFF_F_OOB_PRES & buff->flags) {
+			if (buff->flags & SDP_BUFF_F_OOB_PRES) {
 				/*
 				 * if data has already been read, and the
 				 * next byte is the urgent byte, reading
@@ -1246,7 +1246,7 @@ int sdp_inet_recv(struct kiocb  *req,
 				if (1 < length)
 					length--;
 				else {
-					if (0 < copied) {
+					if (copied > 0) {
 						/*
 						 * update such that we pass
 						 * through the copy phase,
@@ -1275,22 +1275,22 @@ int sdp_inet_recv(struct kiocb  *req,
 
 			copy = min((size_t) (size - copied), length);
 
-			if (0 < copy) {
+			if (copy > 0) {
 #ifndef _SDP_DATA_PATH_NULL
 				result = memcpy_toiovec(msg->msg_iov,
 							buff->data,
 							copy);
-				if (0 > result) {
+				if (result < 0) {
 					expect =
 					    sdp_buff_q_put_head(&conn->
 								 recv_pool,
 								 buff);
-					SDP_EXPECT(!(0 > expect));
+					SDP_EXPECT(expect >= 0);
 
 					goto done;
 				}
 #endif
-				update = (0 < (MSG_PEEK & flags)) ? 0 : copy;
+				update = (MSG_PEEK & flags) ? 0 : copy;
 			}
 
 			SDP_CONN_STAT_RECV_INC(conn, update);
@@ -1299,10 +1299,10 @@ int sdp_inet_recv(struct kiocb  *req,
 			buff->data      += update;
 			copied          += copy;
 
-			if (0 < (buff->tail - buff->data)) {
+			if ((buff->tail - buff->data) > 0) {
 				expect = sdp_buff_q_put_head(&conn->recv_pool,
 							     buff);
-				SDP_EXPECT(!(0 > expect));
+				SDP_EXPECT(expect >= 0);
 				/*
 				 * always break, PEEK and OOB together could
 				 * throw us into a loop without a forced
@@ -1317,9 +1317,9 @@ int sdp_inet_recv(struct kiocb  *req,
 			if (MSG_PEEK & flags) {
 				expect = sdp_buff_q_put_head(&peek_queue,
 							     buff);
-				SDP_EXPECT(!(0 > expect));
+				SDP_EXPECT(expect >= 0);
 			} else {
-				if (SDP_BUFF_F_OOB_PRES & buff->flags)
+				if (buff->flags & SDP_BUFF_F_OOB_PRES)
 					conn->rcv_urg_cnt -= 1;
 				/*
 				 * create a link of buffers which
@@ -1337,7 +1337,7 @@ int sdp_inet_recv(struct kiocb  *req,
 				 */
 				if (SDP_RECV_POST_FREQ < ++ack) {
 					result = sdp_recv_flush(conn);
-					if (0 > result)
+					if (result < 0)
 						goto done;
 
 					ack = 0;
@@ -1348,7 +1348,7 @@ int sdp_inet_recv(struct kiocb  *req,
 		 * urgent data needs to break up the data stream, regardless 
 		 * of low water mark, or whether there is room in the buffer.
 		 */
-		if (0 < oob) {
+		if (oob > 0) {
 			result = 0;
 			break;
 		}
@@ -1365,7 +1365,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			 */
 			sdp_conn_relock(conn);
 
-			if (0 < sdp_buff_q_size(&conn->recv_pool))
+			if (sdp_buff_q_size(&conn->recv_pool) > 0)
 				continue;
 		}
 		/*
@@ -1376,7 +1376,7 @@ int sdp_inet_recv(struct kiocb  *req,
 		 * data is pending and accessible.
 		 */
 		if (!(copied < low_water) &&
-		    0 == conn->src_recv) {
+		    !conn->src_recv) {
 #if 0 /* performance cheat. LM */
 			if (!(conn->snk_zthresh > size)) {
 
@@ -1384,7 +1384,7 @@ int sdp_inet_recv(struct kiocb  *req,
 
 				result = sdp_send_ctrl_snk_avail(conn,
 								 0, 0, 0);
-				if (0 > result) {
+				if (result < 0) {
 					/*
 					 * since the message did not go out,
 					 * back out the non_discard counter
@@ -1399,23 +1399,23 @@ int sdp_inet_recv(struct kiocb  *req,
 		 * check connection errors, and then wait for more data.
 		 * check status. POSIX 1003.1g order.
 		 */
-		if (0 != SDP_CONN_GET_ERR(conn)) {
-			result = (0 < copied) ? 0 : sdp_conn_error(conn);
+		if (SDP_CONN_GET_ERR(conn)) {
+			result = (copied > 0) ? 0 : sdp_conn_error(conn);
 			break;
 		}
 		
-		if (0 < (RCV_SHUTDOWN & conn->shutdown)) {
+		if (RCV_SHUTDOWN & conn->shutdown) {
 			result = 0;
 			break;
 		}
 
-		if (SDP_SOCK_ST_ERROR == conn->istate) {
+		if (conn->istate == SDP_SOCK_ST_ERROR) {
 			result = -EPROTO; /* error should always be
 					     set, but just in case */
 			break;
 		}
 
-		if (0 == timeout) {
+		if (!timeout) {
 			result = -EAGAIN;
 			break;
 		}
@@ -1430,7 +1430,7 @@ int sdp_inet_recv(struct kiocb  *req,
 
 			set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags);
 
-			if (0 == sdp_buff_q_size(&conn->recv_pool)) {
+			if (!sdp_buff_q_size(&conn->recv_pool)) {
 				sdp_conn_unlock(conn);
 				timeout = schedule_timeout(timeout);
 				sdp_conn_lock(conn);
@@ -1443,7 +1443,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			 * check signal pending
 			 */
 			if (signal_pending(current)) {
-				result = ((0 < timeout) ?
+				result = ((timeout > 0) ?
 					  sock_intr_errno(timeout) : -EAGAIN);
 				break;
 			}
@@ -1452,7 +1452,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			 * create IOCB with remaining space
 			 */
 			iocb = sdp_iocb_create();
-			if (NULL == iocb) {
+			if (!iocb) {
 				sdp_dbg_warn(conn, 
 					     "Error allocating IOCB <%Zu:%d>",
 					     size, copied);
@@ -1470,7 +1470,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			req->ki_cancel = _sdp_inet_read_cancel;
 
 			result = sdp_iocb_lock(iocb);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> IOCB lock <%Zu:%d>", 
 					     result, size, copied);
@@ -1482,7 +1482,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			SDP_CONN_STAT_RQ_INC(conn, iocb->size);
 
 			result = sdp_iocb_q_put_tail(&conn->r_pend, iocb);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, 
 					     "Error <%d> IOCB queue <%Zu:%d>",
 					     result, size, copied);
@@ -1503,9 +1503,9 @@ done:
 	/*
 	 * acknowledge moved data
 	 */
-	if (0 < ack) {
+	if (ack > 0) {
 		expect = sdp_recv_flush(conn);
-		if (0 > expect)
+		if (expect < 0)
 			sdp_dbg_warn(conn, "Error <%d> flushing recv queue.",
 				     expect);
 	}
@@ -1514,13 +1514,13 @@ done:
 	/*
 	 * return any peeked buffers to the recv queue, in the correct order.
 	 */
-	if (0 < (MSG_PEEK & flags)) {
-		while (NULL != (buff = sdp_buff_q_get_tail(&peek_queue))) {
+	if (MSG_PEEK & flags) {
+		while ((buff = sdp_buff_q_get_tail(&peek_queue))) {
 			expect = sdp_buff_q_put_head(&conn->recv_pool, buff);
-			SDP_EXPECT(!(0 > expect));
+			SDP_EXPECT(expect >= 0);
 		}
 	}
 
 	sdp_conn_unlock(conn);
-	return ((0 < copied) ? copied : result);
+	return ((copied > 0) ? copied : result);
 }
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1922)
+++ sdp_wall.c	(working copy)
@@ -62,11 +62,11 @@ int sdp_wall_send_close(struct sdp_opt *
 		 * clear out the sent HelloAck message
 		 */
 		buff = sdp_buff_q_get_head(&conn->send_post);
-		if (NULL == buff)
+		if (!buff)
 			sdp_dbg_warn(conn, "Error, hello ack missing.");
 		else {
 			result = sdp_buff_pool_put(buff);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 		/*
 		 * fall through
@@ -86,7 +86,7 @@ int sdp_wall_send_close(struct sdp_opt *
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_1);
 
 		result = sdp_send_ctrl_disconnect(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, 
 				     "Error <%d> send disconnect request",
 				     result);
@@ -108,7 +108,7 @@ error:
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return result;
@@ -139,7 +139,7 @@ int sdp_wall_send_closing(struct sdp_opt
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_2);
 
 		result = sdp_send_ctrl_disconnect(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn,
 				     "Error <%d> send disconnect request",
 				     result);
@@ -160,7 +160,7 @@ error:
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return result;
@@ -193,7 +193,7 @@ int sdp_wall_send_abort(struct sdp_opt *
 		 * post abort
 		 */
 		result = sdp_send_ctrl_abort(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> send abort request",
 				     result);
 			goto error;
@@ -261,7 +261,7 @@ error:
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return 0;
@@ -347,7 +347,7 @@ int sdp_wall_recv_failed(struct sdp_opt 
 	 * the connection has failed, move to error, and notify anyone
 	 * waiting of the state change.
 	 */
-	SDP_EXPECT((SDP_SOCK_ST_ACCEPTED == conn->istate));
+	SDP_EXPECT((conn->istate == SDP_SOCK_ST_ACCEPTED));
 
 	switch (conn->istate) {
 	default:
@@ -420,7 +420,7 @@ int sdp_wall_recv_closing(struct sdp_opt
 	/*
 	 * change state, finalize the close, and wake the closer.
 	 */
-	SDP_EXPECT((SDP_SOCK_ST_DISCONNECT == conn->istate));
+	SDP_EXPECT((conn->istate == SDP_SOCK_ST_DISCONNECT));
 
 	conn->send_buf = 0;
 	conn->istate = SDP_SOCK_ST_CLOSED;
@@ -502,7 +502,7 @@ int sdp_wall_recv_drop(struct sdp_opt *c
 		 * pull the listen sockets accept queue.
 		 */
 		result = sdp_inet_accept_q_remove(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, 
 				     "Error <%d> removing from accept queue.",
 				     result);
@@ -555,11 +555,11 @@ int sdp_wall_abort(struct sdp_opt *conn)
 	 * notify both halves of the wall that the connection is being aborted.
 	 */
 	result = sdp_wall_recv_abort(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> recving abort request", result);
 	/* if */
 	result = sdp_wall_send_abort(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> sending abort request", result);
 
 	return 0;
Index: sdp_conn.h
===================================================================
--- sdp_conn.h	(revision 1922)
+++ sdp_conn.h	(working copy)
@@ -492,8 +492,8 @@ static inline void sdp_conn_unlock(struc
 	unsigned long flags;
 
 	spin_lock_irqsave(&conn->lock.slock, flags);
-	if (0 < (SDP_CONN_F_MASK_EVENT & conn->flags) &&
-	    0 < (SDP_ST_MASK_EVENTS & conn->state)) {
+	if ((conn->flags & SDP_CONN_F_MASK_EVENT) &&
+	    (SDP_ST_MASK_EVENTS & conn->state)) {
 
 		sdp_conn_internal_unlock(conn);
 	}
Index: sdp_proc.c
===================================================================
--- sdp_proc.c	(revision 1922)
+++ sdp_proc.c	(working copy)
@@ -56,15 +56,15 @@ static int _sdp_proc_read_parse(char *pa
 	int  size;
 
 #if 0
-	if (NULL == *start && 0 != offset) {
+	if (!*start && offset) {
 		return 0; /* I'm not sure why this always gets 
 			     called twice... */
 	}
 #endif
 
 	size = sub_entry->read(page, count, offset, &end_index);
-	if (0 < size) {
-		if (0 < end_index) {
+	if (size > 0) {
+		if (end_index > 0) {
 			*start = (char *)end_index;
 			*eof = 0;
 		} else {
@@ -141,7 +141,7 @@ int sdp_main_proc_cleanup(void)
 	 */
 	for (counter = 0; counter < SDP_PROC_ENTRIES; counter++) {
 		sub_entry = &_file_entry_list[counter];
-		if (NULL != sub_entry->entry) {
+		if (sub_entry->entry) {
 			remove_proc_entry(sub_entry->name, _dir_root);
 			sub_entry->entry = NULL;
 		}
@@ -177,7 +177,7 @@ int sdp_main_proc_init(void)
 		return -EFAULT;
 	}
 
-	if (NULL != _dir_root) {
+	if (_dir_root) {
 		sdp_warn("/proc already initialized!");
 		return -EINVAL;
 	}
@@ -185,7 +185,7 @@ int sdp_main_proc_init(void)
 	 * create a gateway root, and main directories
 	 */
 	_dir_root = proc_mkdir(_dir_name_root, proc_net);
-	if (NULL == _dir_root) {
+	if (!_dir_root) {
 		sdp_warn("Failed to create <%s> proc entry.",
 			 _dir_name_root);
 		return -EINVAL;
@@ -203,7 +203,7 @@ int sdp_main_proc_init(void)
 		sub_entry->entry = create_proc_entry(sub_entry->name,
 						     S_IRUGO | S_IWUGO,
 						     _dir_root);
-		if (NULL == sub_entry->entry) {
+		if (!sub_entry->entry) {
 			sdp_warn("Failed to create <%s> framework proc entry.",
 				 sub_entry->name);
 			result = -EINVAL;
Index: sdp_pass.c
===================================================================
--- sdp_pass.c	(revision 1922)
+++ sdp_pass.c	(working copy)
@@ -62,7 +62,7 @@ static int _sdp_cm_accept(struct sdp_opt
 	 * (don't need to worry about header space reservation on sends)
 	 */
 	buff = sdp_buff_pool_get();
-	if (NULL == buff) {
+	if (!buff) {
 		sdp_dbg_warn(conn, "Failed to allocate buff for Hello Ack.");
 		result = -ENOMEM;
 		goto error;
@@ -97,12 +97,12 @@ static int _sdp_cm_accept(struct sdp_opt
 	 * save message
 	 */
 	result = sdp_buff_q_put(&conn->send_post, buff);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> buffering hello ack packet.",
 			     result);
 
 		expect = sdp_buff_pool_put(buff);
-		SDP_EXPECT(!(0 > expect));
+		SDP_EXPECT(expect >= 0);
 
 		goto error;
 	}
@@ -110,7 +110,7 @@ static int _sdp_cm_accept(struct sdp_opt
 	 * modify QP. INIT->RTR 
 	 */
         qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
-        if (NULL == qp_attr) {
+        if (!qp_attr) {
 		sdp_dbg_warn(conn, "Failed to allocate QP attribute.");
                 result = -ENOMEM;
 		goto error;
@@ -147,7 +147,7 @@ static int _sdp_cm_accept(struct sdp_opt
 	 * Post receive buffers for this connection
 	 */
 	result = sdp_recv_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing receive queue",
 			     result);
 		goto error;
@@ -170,7 +170,7 @@ static int _sdp_cm_accept(struct sdp_opt
 	param.rnr_retry_count     = SDP_CM_PARAM_RNR_RETRY;
 
 	result = ib_send_cm_rep(conn->cm_id, &param);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> CM accept request.", result);
 		goto error;
 	}
@@ -198,7 +198,7 @@ static int _sdp_cm_listen_lookup(struct 
 	 * first find a listening connection
 	 */
 	listen_conn = sdp_inet_listen_lookup(conn->src_addr, conn->src_port);
-	if (NULL == listen_conn) {
+	if (!listen_conn) {
 		/*
 		 * no connection, reject
 		 */
@@ -224,7 +224,7 @@ static int _sdp_cm_listen_lookup(struct 
 	}
 
 	result = sdp_inet_port_inherit(listen_conn, conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(listen_conn, "Error <%d> listen port inherit.",
 			     result);
 		result = -EFAULT;
@@ -270,7 +270,7 @@ static int _sdp_cm_listen_lookup(struct 
 	 * initiate a CM response message.
 	 */
 	result = _sdp_cm_accept(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> CM connect accept", result);
 		goto locked_err;
 	}
@@ -278,7 +278,7 @@ static int _sdp_cm_listen_lookup(struct 
 	 * place connection into the listen connections accept queue.
 	 */
 	result = sdp_inet_accept_q_put(listen_conn, conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, 
 			     "Error <%d> adding socket to accept queue",
 			     result);
@@ -322,7 +322,7 @@ static int _sdp_cm_hello_check(struct sd
 		return -EINVAL;
 	}
 
-	if (!(0 < msg_hello->hh.max_adv)) {
+	if (msg_hello->hh.max_adv <= 0) {
 		sdp_dbg_warn(NULL, "hello msg, bad zcopy count <%d>",
 			     msg_hello->hh.max_adv);
 		return -EINVAL;
@@ -383,7 +383,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 * check Hello Header, to determine if we want the connection.
 	 */
 	result = _sdp_cm_hello_check(msg_hello);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(NULL, "Error <%d> validating hello msg. <%08x>",
 			     result, cm_id->local_id);
 		goto done;
@@ -392,7 +392,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 * Create a connection for this request.
 	 */
 	conn = sdp_conn_alloc(GFP_KERNEL); /* CM sk reference */
-	if (NULL == conn) {
+	if (!conn) {
 		sdp_dbg_warn(NULL, "Failed to allocate connection. <%08x>",
 			     cm_id->local_id);
 		result = -ENOMEM;
@@ -440,7 +440,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 				   event->param.req_rcvd.device,
 				   event->param.req_rcvd.port,
 				   event->param.req_rcvd.primary_path->pkey);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> binding connection to HCA/port",
 			     result);
 		goto error;
@@ -458,7 +458,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 * into listeners accept queue.
 	 */
 	result = _sdp_cm_listen_lookup(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> matching listen socket queue",
 			     result);
 		goto error;
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1922)
+++ sdp_sent.c	(working copy)
@@ -66,7 +66,7 @@ static int _sdp_sent_disconnect(struct s
 		 * Begin IB/CM disconnect
 		 */
 		result = sdp_cm_disconnect(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> posting CM disconnect",
 				     result);
 			goto error;
@@ -84,7 +84,7 @@ static int _sdp_sent_disconnect(struct s
 		 * acknowledge disconnect to framework
 		 */
 		result = sdp_wall_recv_closing(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> closing connection.",
 				     result);
 			goto error;
@@ -97,7 +97,7 @@ static int _sdp_sent_disconnect(struct s
 		 * if the remote DREQ was already received, but unprocessed, do
 		 * not treat it as an error
 		 */
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
 
 		break;
@@ -123,11 +123,11 @@ static int _sdp_sent_abort(struct sdp_op
 	 * The gateway interface should be in error state, initiate CM
 	 * disconnect.
 	 */
-	SDP_EXPECT((SDP_CONN_ST_ERROR_STRM == conn->state));
+	SDP_EXPECT((conn->state == SDP_CONN_ST_ERROR_STRM));
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
 	result = sdp_cm_disconnect(conn);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
 
 	return result;
@@ -356,17 +356,17 @@ int sdp_event_send(struct sdp_opt *conn,
 	/*
 	 * get buffer.
 	 */
-	while (NULL != (buff = sdp_buff_q_get_head(&conn->send_post))) {
+	while ((buff = sdp_buff_q_get_head(&conn->send_post))) {
 		/*
 		 * sanity checks
 		 */
-		if (NULL == buff->bsdh_hdr) {
+		if (!buff->bsdh_hdr) {
 			sdp_dbg_warn(conn, "Send header is missing?!");
 			result = -ENODATA;
 			goto drop;
 		}
 		/* check WRID taking into account wrap around */
-		if (0 > (s64)(comp->wr_id - buff->wrid)) {
+		if (((s64)(comp->wr_id - buff->wrid)) < 0) {
 			/*
 			 * error
 			 */
@@ -403,7 +403,7 @@ int sdp_event_send(struct sdp_opt *conn,
 			offset = buff->bsdh_hdr->mid & 0x1F;
 
 			if (!(offset < SDP_MSG_EVENT_TABLE_SIZE) ||
-			    NULL == send_event_funcs[offset]) {
+			    !send_event_funcs[offset]) {
 				sdp_dbg_warn(conn, 
 					     "Send complete unknown MID <%d>",
 					     buff->bsdh_hdr->mid);
@@ -415,7 +415,7 @@ int sdp_event_send(struct sdp_opt *conn,
 			
 			dispatch_func = send_event_funcs[offset];
 			result = dispatch_func(conn, buff);
-			if (0 > result) {
+			if (result < 0) {
 				sdp_dbg_warn(conn, "Sent dispatch error. <%d>",
 					     result);
 				goto drop;
@@ -428,7 +428,7 @@ int sdp_event_send(struct sdp_opt *conn,
 		 */
 		conn->s_wq_size--;
 
-		if (0 < SDP_BUFF_F_GET_UNSIG(buff))
+		if (SDP_BUFF_F_GET_UNSIG(buff) > 0)
 			conn->send_usig--;
 		/*
 		 * create a link of buffers which will be returned to
@@ -444,9 +444,9 @@ int sdp_event_send(struct sdp_opt *conn,
 	}
 
 	result = sdp_buff_pool_chain_put(head, free_count);
-	SDP_EXPECT(!(0 > result));
+	SDP_EXPECT(result >= 0);
 
-	if (!(0 < free_count) || 0 > conn->send_usig) {
+	if (free_count <= 0 || conn->send_usig < 0) {
 		sdp_dbg_warn(conn, 
 			     "Send processing mismatch. <%llu:%llu:%d:%d>",
 			     (unsigned long long)comp->wr_id,
@@ -459,7 +459,7 @@ int sdp_event_send(struct sdp_opt *conn,
 	 * Flush queued send data into the post queue if there is room.
 	 */
 	result = sdp_send_flush(conn);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing send queue.", result);
 		goto done;
 	}
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1922)
+++ sdp_iocb.c	(working copy)
@@ -52,7 +52,7 @@ static int _sdp_iocb_unlock(struct sdpc_
 	struct vm_area_struct *vma;
 
 	vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK));
-	if (NULL == vma)
+	if (!vma)
 		sdp_warn("No VMA for IOCB <%lx:%Zu> unlock",
 			 iocb->addr, iocb->size);
 
@@ -104,7 +104,7 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 	/*
 	 * check if IOCB is locked.
 	 */
-	if (0 == (SDP_IOCB_F_LOCKED & iocb->flags))
+	if (!(iocb->flags & SDP_IOCB_F_LOCKED))
 		return 0;
 	/*
 	 * spin lock since this could be from interrupt context.
@@ -145,7 +145,7 @@ static int _sdp_iocb_page_save(struct sd
 	pte_t *ptep;
 	pte_t  pte;
 
-	if (!(0 < iocb->page_count) || !(0 < iocb->size) || 0 == iocb->addr)
+	if (iocb->page_count <= 0 || iocb->size <= 0 || !iocb->addr)
 		return -EINVAL;
 	/*
 	 * create array to hold page value which are later needed to register
@@ -153,12 +153,12 @@ static int _sdp_iocb_page_save(struct sd
 	 */
 	iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count),
 				   GFP_KERNEL);
-	if (NULL == iocb->addr_array)
+	if (!iocb->addr_array)
 		goto err_addr;
 
 	iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), 
 				   GFP_KERNEL);
-	if (NULL == iocb->page_array)
+	if (!iocb->page_array)
 		goto err_page;
 	/*
 	 * iocb->addr - buffer start address
@@ -177,7 +177,7 @@ static int _sdp_iocb_page_save(struct sd
 	spin_lock(&iocb->mm->page_table_lock);
 
 	for (counter = 0;
-	     0 < size;
+	     size > 0;
 	     counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) {
 		pgd = pgd_offset_gate(iocb->mm, addr);
 		if (!pgd || pgd_none(*pgd))
@@ -280,7 +280,7 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	 * them do not copy, reference counting, and saving them. 
 	 */
 	vma = find_vma(iocb->mm, addr);
-	if (NULL == vma)
+	if (!vma)
 		/*
 		 * sanity check.
 		 */
@@ -290,7 +290,7 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	while (vma) {
 		spin_lock(&iocb->mm->page_table_lock);
 
-		if (0 == (VM_LOCKED & vma->vm_flags))
+		if (!(VM_LOCKED & vma->vm_flags))
 			sdp_warn("Unlocked vma! <%08lx>", vma->vm_flags);
 
 		if (PAGE_SIZE < (unsigned long)vma->vm_private_data)
@@ -367,13 +367,13 @@ static int _sdp_mem_lock_init(void)
 	sdp_dbg_init("Memory Locking initialization.");
 	
 	kallsyms = filp_open("/proc/kallsyms", O_RDONLY, 0);
-	if (NULL == kallsyms) {
+	if (!kallsyms) {
 		sdp_warn("Failed to open /proc/kallsyms");
 		goto done;
 	}
 
 	seq = (struct seq_file *)kallsyms->private_data;
-	if (NULL == seq) {
+	if (!seq) {
 		sdp_warn("Failed to fetch sequential file.");
 		goto err_close;
 	}
@@ -381,10 +381,10 @@ static int _sdp_mem_lock_init(void)
 	for (iter = seq->op->start(seq, &pos);
 	     iter != NULL;
 	     iter = seq->op->next(seq, iter, &pos))
-		if (0 == strcmp(iter->name, "do_mlock"))
+		if (!strcmp(iter->name, "do_mlock"))
 			_mlock_ptr = (do_mlock_ptr_t)iter->value;
 
-	if (NULL == _mlock_ptr)
+	if (!_mlock_ptr)
 		sdp_warn("Failed to find lock pointer.");
 	else
 		ret = 0;
@@ -423,7 +423,7 @@ int sdp_iocb_register(struct sdpc_iocb *
 	/*
 	 * register only once.
 	 */
-	if (SDP_IOCB_F_REG & iocb->flags)
+	if (iocb->flags & SDP_IOCB_F_REG)
 		return 0;
 	/*
 	 * prime io address with physical address of first byte?
@@ -439,7 +439,7 @@ int sdp_iocb_register(struct sdpc_iocb *
 	if (IS_ERR(iocb->mem)) {
 		result = (int)PTR_ERR(iocb->mem);
 
-		if (-EAGAIN != result)
+		if (result != -EAGAIN)
 			sdp_dbg_err("Error <%d> fmr_pool_map_phys <%d:%d:%d>",
 				    result,
 				    iocb->len,
@@ -471,11 +471,11 @@ int sdp_iocb_release(struct sdpc_iocb *i
 {
 	int result;
 
-	if (0 == (SDP_IOCB_F_REG & iocb->flags))
+	if (!(iocb->flags & SDP_IOCB_F_REG))
 		return 0;
 
 	result = ib_fmr_pool_unmap(iocb->mem);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>",
 			    result, iocb->key, iocb->addr);
 
@@ -496,20 +496,20 @@ static void _sdp_iocb_complete(void *arg
 	 * release memory
 	 */
 	result = sdp_iocb_release(iocb);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>",
 			    result, iocb->key, iocb->addr);
 	/*
 	 * unlock now, after aio_complete the mm reference will be released.
 	 */
 	result = sdp_iocb_unlock(iocb);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_err("Error <%d> unlocking IOCB <%d memory <%ld>>",
 			    result, iocb->key, iocb->addr);
 	/*
 	 * callback to complete IOCB
 	 */
-	value = (0 < iocb->post) ? iocb->post : iocb->status;
+	value = (iocb->post > 0) ? iocb->post : iocb->status;
 
 	sdp_dbg_data(NULL, "IOCB complete. <%d:%d:%08lx> value <%ld>",
 		     iocb->req->ki_users, iocb->req->ki_key, 
@@ -523,7 +523,7 @@ static void _sdp_iocb_complete(void *arg
 	 * delete IOCB
 	 */
 	result = sdp_iocb_destroy(iocb);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>",
 			    result, iocb->key, iocb->status);
 }
@@ -602,7 +602,7 @@ struct sdpc_iocb *sdp_iocb_create(void)
 	struct sdpc_iocb *iocb;
 
 	iocb = kmem_cache_alloc(__sdp_iocb_cache, SLAB_KERNEL);
-	if (NULL != iocb) {
+	if (iocb) {
 		memset(iocb, 0, sizeof(struct sdpc_iocb));
 		/*
 		 * non-zero initialization
@@ -620,10 +620,10 @@ struct sdpc_iocb *sdp_iocb_create(void)
  */
 int sdp_iocb_destroy(struct sdpc_iocb *iocb)
 {
-	if (NULL == iocb)
+	if (!iocb)
 		return -EINVAL;
 
-	if (NULL != iocb->next || NULL != iocb->prev)
+	if (iocb->next || iocb->prev)
 		return -EACCES;
 	/*
 	 * release iocb registered memory
@@ -636,10 +636,10 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
 	/*
 	 * array dealloc
 	 */
-	if (NULL != iocb->page_array)
+	if (iocb->page_array)
 		kfree(iocb->page_array);
 
-	if (NULL != iocb->addr_array)
+	if (iocb->addr_array)
 		kfree(iocb->addr_array);
 	/*
 	 * clear IOCB to check for usage after free...
@@ -672,7 +672,7 @@ static struct sdpc_iocb *_sdp_iocb_q_get
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
 
-	if (NULL == table->head)
+	if (!table->head)
 		return NULL;
 
 	if (head)
@@ -710,7 +710,7 @@ static int _sdp_iocb_q_put(struct sdpc_i
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
 
-	if (NULL == table->head) {
+	if (!table->head) {
 		iocb->next = iocb;
 		iocb->prev = iocb;
 		table->head = iocb;
@@ -787,7 +787,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
 	     counter < total; counter++) {
 		next = iocb->next;
 
-		if (0 < (mask & iocb->flags) || SDP_IOCB_F_ALL == mask) {
+		if ((mask & iocb->flags) || msk == SDP_IOCB_F_ALL) {
 			sdp_dbg_err("IOCB <%d> cancel <%Zu> flag <%04x> "
 				    "size <%Zu:%d:%d>",
 				    iocb->key, comp, iocb->flags, iocb->size,
@@ -796,7 +796,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
 			sdp_iocb_q_remove(iocb);
 
 			result = sdp_iocb_complete(iocb, comp);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 
 		iocb = next;
@@ -823,9 +823,9 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q
 	/*
 	 * drain the table of any objects
 	 */
-	while (NULL != (iocb = sdp_iocb_q_get_head(table))) {
+	while ((iocb = sdp_iocb_q_get_head(table))) {
 		result = sdp_iocb_destroy(iocb);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 	}
 }
 
@@ -845,14 +845,14 @@ int sdp_main_iocb_init(void)
 	 * initialize locking code.
 	 */
 	result =  _sdp_mem_lock_init();
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("Error <%d> initializing memory locking.", result);
 		return result;
 	}
 	/*
 	 * initialize the caches only once.
 	 */
-	if (NULL != __sdp_iocb_cache) {
+	if (__sdp_iocb_cache) {
 		sdp_warn("IOCB caches already initialized.");
 		return -EINVAL;
 	}
@@ -861,7 +861,7 @@ int sdp_main_iocb_init(void)
 					     sizeof(struct sdpc_iocb),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (NULL == __sdp_iocb_cache) {
+	if (!__sdp_iocb_cache) {
 		result = -ENOMEM;
 		goto error_iocb_c;
 	}
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1922)
+++ sdp_event.c	(working copy)
@@ -45,7 +45,7 @@ int sdp_cq_event_locked(struct ib_wc *co
 {
 	int result = 0;
 
-	if (0 < (SDP_ST_MASK_ERROR & conn->state)) {
+	if (SDP_ST_MASK_ERROR & conn->state) {
 		/*
 		 * Ignore events in error state, connection is being 
 		 * terminated, connection cleanup will take care of freeing
@@ -102,7 +102,7 @@ int sdp_cq_event_locked(struct ib_wc *co
 	/*
 	 * release socket before error processing.
 	 */
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(conn, "ABORT on error <%d> event <%u:%llu:%u:%u>",
 			     result,
 			     comp->status,
@@ -115,7 +115,7 @@ int sdp_cq_event_locked(struct ib_wc *co
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_CQ);
 
 		result = sdp_wall_abort(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> during abort", result);
 
 		return -EFAULT;
@@ -140,7 +140,7 @@ void sdp_cq_event_handler(struct ib_cq *
 	 * get socket
 	 */
 	conn = sdp_conn_table_lookup(hashent);
-	if (NULL == conn) {
+	if (!conn) {
 		sdp_dbg_warn(conn, "Unknown connection <%d> for cq event",
 			     hashent);
 		goto done;
@@ -157,14 +157,14 @@ void sdp_cq_event_handler(struct ib_cq *
 	 * has been made, the act of unlocking the connection will
 	 * drain the CQ.
 	 */
-	if (0 == (SDP_ST_MASK_EVENTS & conn->state)) {
+	if (!(SDP_ST_MASK_EVENTS & conn->state)) {
 		/*
 		 * passive and active connect respectively
 		 */
-		if (SDP_CONN_ST_REP_SENT == conn->state ||
-		    SDP_CONN_ST_RTU_SENT == conn->state) {
+		if (conn->state == SDP_CONN_ST_REP_SENT ||
+		    conn->state == SDP_CONN_ST_RTU_SENT) {
 			result = ib_cm_establish(conn->cm_id);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 		else
 			sdp_dbg_warn(conn, "Unexpected locked state.");
@@ -173,7 +173,7 @@ void sdp_cq_event_handler(struct ib_cq *
 		goto unlock;
 	}
 
-	if (0 == conn->lock.users)
+	if (!conn->lock.users)
 		/*
 		 * dispatch CQ completions.
 		 */
@@ -208,7 +208,7 @@ static int _sdp_cm_idle(struct ib_cm_id 
 	int result = 0;
 	int expect;
 
-	if (NULL == conn)
+	if (!conn)
 		return -EINVAL;
 	/*
 	 * IDLE should only be called after some other action on the comm_id,
@@ -231,7 +231,7 @@ static int _sdp_cm_idle(struct ib_cm_id 
 		break;
 	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
 		result = sdp_wall_recv_failed(conn, ECONNREFUSED);
-		if (0 > result) {
+		if (result < 0) {
 
 			sdp_dbg_warn(conn, "Error <%d> receiving CM failed",
 				     result);
@@ -260,7 +260,7 @@ static int _sdp_cm_idle(struct ib_cm_id 
 		 * Connection is finally dead. Drop the CM reference
 		 */
 		result = sdp_wall_recv_drop(conn);
-		SDP_EXPECT(!(0 > result));
+		SDP_EXPECT(result >= 0);
 
 		break;
 	default:
@@ -278,7 +278,7 @@ error:
 	 * last attempt to drop the CM reference.
 	 */
 	expect = sdp_wall_recv_drop(conn);
-	SDP_EXPECT(!(0 > expect));
+	SDP_EXPECT(expect >= 0);
 
 	return result;
 }
@@ -294,7 +294,7 @@ static int _sdp_cm_established(struct ib
 	int expect;
 	struct sdpc_buff *buff;
 
-	if (NULL == conn)
+	if (!conn)
 		return -EINVAL;
 
 	sdp_dbg_ctrl(conn, "CM ESTABLISHED. commID <%08x>", cm_id->local_id);
@@ -308,7 +308,7 @@ static int _sdp_cm_established(struct ib
 	switch (conn->state) {
 	case SDP_CONN_ST_REP_SENT:	/* passive open, Hello ack msg sent */
 		buff = sdp_buff_q_get_head(&conn->send_post);
-		if (NULL == buff)
+		if (!buff)
 			sdp_dbg_warn(conn, "hello ack missing in send pool");
 		else
 			(void)sdp_buff_pool_put(buff);
@@ -316,7 +316,7 @@ static int _sdp_cm_established(struct ib
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED);
 
 		result = sdp_wall_recv_confirm(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> confirming conn state",
 				     result);
 			/*
@@ -327,14 +327,14 @@ static int _sdp_cm_established(struct ib
 		}
 
 		result = sdp_send_flush(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> flushing send queue.",
 				     result);
 			goto error;
 		}
 
 		result = sdp_recv_flush(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
 				     result);
 			goto error;
@@ -355,7 +355,7 @@ static int _sdp_cm_established(struct ib
 		/* active open, and active close, confirm */
 	case SDP_CONN_ST_DIS_PEND_2:
 		result = sdp_send_flush(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> flushing receives.",
 				     result);
 			goto error;
@@ -369,7 +369,7 @@ static int _sdp_cm_established(struct ib
 		 * existing state correctly.
 		 */
 		result = sdp_cm_disconnect(conn);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> posting CM disconnect",
 				     result);
 
@@ -386,7 +386,7 @@ static int _sdp_cm_established(struct ib
 	return 0;
 error:
 	expect = sdp_wall_recv_drop(conn);
-	SDP_EXPECT(!(0 > expect));
+	SDP_EXPECT(expect >= 0);
 done:
 	conn->cm_id = NULL;
 	return result;
@@ -402,7 +402,7 @@ static int _sdp_cm_timewait(struct ib_cm
 	int result = 0;
 	int expect;
 
-	if (NULL == conn)
+	if (!conn)
 		return -EINVAL;
 
 	sdp_dbg_ctrl(conn, "CM TIME WAIT. commID <%08x> event <%d>",
@@ -452,7 +452,7 @@ static int _sdp_cm_timewait(struct ib_cm
 		SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_1);
 
 		result = sdp_wall_abort(conn);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> during abort", result);
 			goto error;
 		}
@@ -469,7 +469,7 @@ static int _sdp_cm_timewait(struct ib_cm
 	return 0;
 error:
 	expect = sdp_wall_recv_drop(conn);
-	SDP_EXPECT(!(0 > expect));
+	SDP_EXPECT(expect >= 0);
 
 	conn->cm_id = NULL;
 	return result;
@@ -495,7 +495,7 @@ int sdp_cm_event_handler(struct ib_cm_id
 	 * lookup the connection, on a REQ_RECV the sk will be empty.
 	 */
 	conn = sdp_conn_table_lookup(hashent);
-	if (NULL != conn)
+	if (conn)
 		sdp_conn_lock(conn);
 	else
 		if (IB_CM_REQ_RCVD != cm_id->state)
@@ -527,8 +527,8 @@ int sdp_cm_event_handler(struct ib_cm_id
 	/*
 	 * if a socket was found, release the lock, and put the reference.
 	 */
-	if (NULL != conn) {
-		if (0 > result) {
+	if (conn) {
+		if (result < 0) {
 			sdp_dbg_warn(conn, 
 				     "CM state <%d> event <%d> error <%d>",
 				     cm_id->state, event->event, result);
Index: sdp_buff.c
===================================================================
--- sdp_buff.c	(revision 1922)
+++ sdp_buff.c	(working copy)
@@ -51,7 +51,7 @@ static inline struct sdpc_buff *_sdp_buf
 {
 	struct sdpc_buff *buff;
 
-	if (NULL == pool->head)
+	if (!pool->head)
 		return NULL;
 
 	if (fifo)
@@ -59,7 +59,7 @@ static inline struct sdpc_buff *_sdp_buf
 	else
 		buff = pool->head->prev;
 
-	if (NULL == test_func || 0 == test_func(buff, usr_arg)) {
+	if (!test_func || !test_func(buff, usr_arg)) {
 		if (buff->next == buff && buff->prev == buff)
 			pool->head = NULL;
 		else {
@@ -89,10 +89,10 @@ static inline int _sdp_buff_q_put(struct
 				  int fifo)
 {
 	/* fifo: false == tail, true == head */
-	if (NULL != buff->pool)
+	if (buff->pool)
 		return -EINVAL;
 
-	if (NULL == pool->head) {
+	if (!pool->head) {
 		buff->next = buff;
 		buff->prev = buff;
 		pool->head = buff;
@@ -119,7 +119,7 @@ static inline int _sdp_buff_q_put(struct
 static inline struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
 						 int fifo)
 {
-	if (NULL == pool->head || fifo)
+	if (!pool->head || fifo)
 		return pool->head;
 	else
 		return pool->head->prev;
@@ -236,21 +236,21 @@ struct sdpc_buff *sdp_buff_q_fetch(struc
 	/*
 	 * check to see if there is anything to traverse.
 	 */
-	if (NULL != pool->head)
+	if (pool->head)
 		/*
 		 * lock to prevent corruption of table
 		 */
 		for (counter = 0, buff = pool->head;
 		     counter < pool->size; counter++, buff = buff->next) {
 			result = test(buff, usr_arg);
-			if (0 < result) {
+			if (result > 0) {
 				result = _sdp_buff_q_remove(pool, buff);
-				SDP_EXPECT(!(0 > result));
+				SDP_EXPECT(result >= 0);
 
 				return buff;
 			}
 
-			if (0 > result)
+			if (result < 0)
 				break;
 		}
 
@@ -272,7 +272,7 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 	/*
 	 * check to see if there is anything to traverse.
 	 */
-	if (NULL != pool->head)
+	if (pool->head)
 		/*
 		 * lock to prevent corruption of table
 		 */
@@ -280,7 +280,7 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 		     counter < pool->size; counter++, buff = buff->next) {
 
 			result = trav_func(buff, usr_arg);
-			if (0 > result)
+			if (result < 0)
 				break;
 		}
 
@@ -324,13 +324,13 @@ void sdp_buff_q_clear_unmap(struct sdpc_
 	int result;
 	struct sdpc_buff *buff;
 
-	while (NULL != (buff = _sdp_buff_q_get(pool, 0, NULL, NULL))) {
+	while ((buff = _sdp_buff_q_get(pool, 0, NULL, NULL))) {
 		if (dev)
 			dma_unmap_single(dev, buff->real,
 					 buff->tail - buff->data, direction);
 
 		result = sdp_buff_pool_put(buff);
-		if (0 > result)
+		if (result < 0)
 			sdp_dbg_err("Error <%d> returning buffer to main",
 				    result);
 	}
@@ -352,7 +352,7 @@ static void _sdp_buff_pool_release(struc
 	 */
 	while (count--) {
 		buff = sdp_buff_q_get(&m_pool->pool);
-		if (NULL == buff)
+		if (!buff)
 			break;
 		/*
 		 * decrement global buffer count, free buffer page, and free
@@ -408,14 +408,14 @@ static int _sdp_buff_pool_alloc(struct s
 		 * the pool.
 		 */
 		buff = kmem_cache_alloc(m_pool->buff_cache, GFP_ATOMIC);
-		if (NULL == buff) {
+		if (!buff) {
 			sdp_warn("Failed to allocate buffer. <%d:%d>",
 				 total, m_pool->buff_cur);
 			break;
 		}
 
 		buff->head = (void *)__get_free_page(GFP_ATOMIC);
-		if (NULL == buff->head) {
+		if (!buff->head) {
 			sdp_warn("Failed to allocate buffer page. <%d:%d>",
 				 total, m_pool->buff_cur);
 
@@ -434,7 +434,7 @@ static int _sdp_buff_pool_alloc(struct s
 		buff->release = sdp_buff_pool_put;
 
 		result = sdp_buff_q_put(&m_pool->pool, buff);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_warn("Failed to queue buffer. <%d>", result);
 			
 			free_page((unsigned long)buff->head);
@@ -445,7 +445,7 @@ static int _sdp_buff_pool_alloc(struct s
 		m_pool->buff_cur++;
 	}
 
-	if (NULL == main_pool->pool.head) {
+	if (!main_pool->pool.head) {
 		sdp_warn("Failed to allocate any buffers. <%d:%d:%d>",
 			 total, m_pool->buff_cur, m_pool->alloc_inc);
 
@@ -465,14 +465,14 @@ int sdp_buff_pool_init(int buff_min, 
 {
 	int result;
 
-	if (NULL != main_pool) {
+	if (main_pool) {
 		sdp_warn("Main pool already initialized!");
 		return -EEXIST;
 	}
 
-	if (!(0 < buff_min) ||
-	    !(0 < alloc_inc) ||
-	    !(0 < free_mark) ||
+	if (buff_min <= 0 ||
+	    alloc_inc <= 0 ||
+	    free_mark <= 0 ||
 	    buff_max < buff_min) {
 
 		sdp_warn("Pool allocation count error. <%d:%d:%d:%d>",
@@ -483,7 +483,7 @@ int sdp_buff_pool_init(int buff_min, 
 	 * allocate the main pool structures
 	 */
 	main_pool = kmalloc(sizeof(struct sdpc_buff_root), GFP_KERNEL);
-	if (NULL == main_pool) {
+	if (!main_pool) {
 		sdp_warn("Main pool initialization failed.");
 		result = -ENOMEM;
 		goto done;
@@ -504,7 +504,7 @@ int sdp_buff_pool_init(int buff_min, 
 						  sizeof(struct sdpc_buff_q),
 						  0, SLAB_HWCACHE_ALIGN,
 						  NULL, NULL);
-	if (NULL == main_pool->pool_cache) {
+	if (!main_pool->pool_cache) {
 		sdp_warn("Failed to allocate pool cache.");
 		result = -ENOMEM;
 		goto error_pool;
@@ -514,7 +514,7 @@ int sdp_buff_pool_init(int buff_min, 
 						  sizeof(struct sdpc_buff),
 						  0, SLAB_HWCACHE_ALIGN,
 						  NULL, NULL);
-	if (NULL == main_pool->buff_cache) {
+	if (!main_pool->buff_cache) {
 		sdp_warn("Failed to allocate buffer cache.");
 		result = -ENOMEM;
 		goto error_buff;
@@ -523,7 +523,7 @@ int sdp_buff_pool_init(int buff_min, 
 	 * allocate the minimum number of buffers.
 	 */
 	result = _sdp_buff_pool_alloc(main_pool);
-	if (0 > result) {
+	if (result < 0) {
 		sdp_warn("Error <%d> allocating buffers. <%d>",
 			 result, buff_min);
 		goto error_alloc;
@@ -551,7 +551,7 @@ done:
  */
 void sdp_buff_pool_destroy(void)
 {
-	if (NULL == main_pool) {
+	if (!main_pool) {
 		sdp_warn("Main pool dosn't exist.");
 		return;
 	}
@@ -595,9 +595,9 @@ struct sdpc_buff *sdp_buff_pool_get(void
 	 */
 	spin_lock_irqsave(&main_pool->lock, flags);
 
-	if (NULL == main_pool->pool.head) {
+	if (!main_pool->pool.head) {
 		result = _sdp_buff_pool_alloc(main_pool);
-		if (0 > result) {
+		if (result < 0) {
 			sdp_warn("Error <%d> allocating buffers.", result);
 			spin_unlock_irqrestore(&main_pool->lock, flags);
 			return NULL;
@@ -642,10 +642,10 @@ int sdp_buff_pool_put(struct sdpc_buff *
 {
 	unsigned long flags;
 
-	if (NULL == buff || NULL != buff->pool)
+	if (!buff || buff->pool)
 		return -EINVAL;
 
-	if (NULL != buff->next || NULL != buff->prev)
+	if (buff->next || buff->prev)
 		return -ETOOMANYREFS;
 	/*
 	 * reset pointers
@@ -656,7 +656,7 @@ int sdp_buff_pool_put(struct sdpc_buff *
 
 	spin_lock_irqsave(&main_pool->lock, flags);
 
-	if (NULL == main_pool->pool.head) {
+	if (!main_pool->pool.head) {
 		buff->next = buff;
 		buff->prev = buff;
 		main_pool->pool.head = buff;
@@ -686,7 +686,7 @@ void sdp_buff_pool_chain_link(struct sdp
 	buff->tail = buff->head;
 	buff->pool = &main_pool->pool;
 
-	if (NULL == head) {
+	if (!head) {
 		buff->next = buff;
 		buff->prev = buff;
 	} else {
@@ -712,12 +712,12 @@ int sdp_buff_pool_chain_put(struct sdpc_
 	 * a number of buffers are processed in a loop, before being
 	 * returned. (e.g. send completions, recv to userspace.
 	 */
-	if (NULL == buff || !(0 < count))
+	if (!buff || count <= 0)
 		return -EINVAL;
 
 	spin_lock_irqsave(&main_pool->lock, flags);
 
-	if (NULL == main_pool->pool.head)
+	if (!main_pool->pool.head)
 		main_pool->pool.head = buff;
 	else {
 		prev = buff->prev;
@@ -746,7 +746,7 @@ int sdp_buff_pool_buff_size(void)
 {
 	int result;
 
-	if (NULL == main_pool)
+	if (!main_pool)
 		result = -1;
 	else
 		result = main_pool->buff_size;
@@ -774,7 +774,7 @@ int sdp_proc_dump_buff_pool(char *buffer
 	 */
 	spin_lock_irqsave(&main_pool->lock, flags);
 
-	if (0 == start_index) {
+	if (!start_index) {
 		offset += sprintf((buffer + offset), 
 				  "  buffer size:         %8d\n",
 				  main_pool->buff_size);
Index: sdp_queue.c
===================================================================
--- sdp_queue.c	(revision 1922)
+++ sdp_queue.c	(working copy)
@@ -46,7 +46,7 @@ static struct sdpc_desc *_sdp_desc_q_get
 {
 	struct sdpc_desc *element;
 
-	if (NULL == table->head)
+	if (!table->head)
 		return NULL;
 
 	if (fifo)
@@ -84,10 +84,10 @@ static inline int _sdp_desc_q_put(struct
 	/*
 	 * fifo: false == tail, true == head
 	 */
-	if (NULL != element->table)
+	if (element->table)
 		return -EINVAL;
 
-	if (NULL == table->head) {
+	if (!table->head) {
 		element->next = element;
 		element->prev = element;
 		table->head = element;
@@ -158,7 +158,7 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
 
 	for (counter = 0, element = table->head;
 	     counter < table->size; counter++, element = element->next)
-		if (0 == lookup(element, arg))
+		if (!lookup(element, arg))
 			return element;
 
 	return NULL;
@@ -211,7 +211,7 @@ struct sdpc_desc *sdp_desc_q_look_head(s
  */
 int sdp_desc_q_type_head(struct sdpc_desc_q *table)
 {
-	if (NULL == table->head)
+	if (!table->head)
 		return SDP_DESC_TYPE_NONE;
 	else
 		return table->head->type;
@@ -223,7 +223,7 @@ int sdp_desc_q_type_head(struct sdpc_des
 struct sdpc_desc *sdp_desc_q_look_type_head(struct sdpc_desc_q *table, 
 					   enum sdp_desc_type type)
 {
-	if (NULL == table->head)
+	if (!table->head)
 		return NULL;
 	else
 		return ((type == table->head->type) ? table->head : NULL);
@@ -235,7 +235,7 @@ struct sdpc_desc *sdp_desc_q_look_type_h
 struct sdpc_desc *sdp_desc_q_look_type_tail(struct sdpc_desc_q *table,
 					   enum sdp_desc_type type)
 {
-	if (NULL == table->head)
+	if (!table->head)
 		return NULL;
 	else
 		return ((type == table->head->prev->type) ?
@@ -274,9 +274,9 @@ void sdp_desc_q_clear(struct sdpc_desc_q
 	/*
 	 * drain the table of any objects
 	 */
-	while (NULL != (element = sdp_desc_q_get_head(table)))
-		if (NULL != element->release) {
+	while ((element = sdp_desc_q_get_head(table)))
+		if (element->release) {
 			result = element->release(element);
-			SDP_EXPECT(!(0 > result));
+			SDP_EXPECT(result >= 0);
 		}
 }
Index: sdp_post.c
===================================================================
--- sdp_post.c	(revision 1922)
+++ sdp_post.c	(working copy)
@@ -41,12 +41,12 @@ int sdp_cm_listen_start(struct sdev_root
 {
 	int result = 0;
 
-	if (NULL != dev_root->listen_id)
+	if (dev_root->listen_id)
 		sdp_dbg_warn(NULL, "Already listening for connections.");
 
 	dev_root->listen_id = ib_create_cm_id(sdp_cm_event_handler,
 					      (void *)SDP_DEV_SK_INVALID);
-	if (NULL == dev_root->listen_id)
+	if (!dev_root->listen_id)
 		return -ENOMEM;
 	/*
 	 * start listening
@@ -54,7 +54,7 @@ int sdp_cm_listen_start(struct sdev_root
 	result = ib_cm_listen(dev_root->listen_id,
 		              cpu_to_be64(SDP_MSG_SERVICE_ID_VALUE),
 			      cpu_to_be64(SDP_MSG_SERVICE_ID_MASK));
-	if (0 > result) {
+	if (result < 0) {
 		sdp_dbg_warn(NULL, "Error <%d> listening for SDP connections",
 			     result);
 
@@ -74,7 +74,7 @@ int sdp_cm_listen_stop(struct sdev_root 
 	int result = 0;
 
 	result = ib_destroy_cm_id(dev_root->listen_id);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(NULL, "Error <%d> stopping listen", result);
 	else
 		sdp_dbg_init("Stopped listening for SDP connections");
@@ -92,7 +92,7 @@ static void _sdp_cm_disconnect(void *arg
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
 	
-	if (NULL == conn) {
+	if (!conn) {
 		sdp_dbg_warn(NULL, "Error, posting disconnect for NULL conn");
 		return;
 	}
@@ -102,7 +102,7 @@ static void _sdp_cm_disconnect(void *arg
 	 * send a disconnect request using the connection manager
 	 */
 	result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(NULL, "Error <%d> CM disconnect request", result);
 
 	sdp_conn_put(conn);
@@ -116,7 +116,7 @@ static void _sdp_cm_reject(void *arg)
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
 
-	if (NULL == conn)
+	if (!conn)
 		sdp_dbg_warn(NULL, "Error, posting reject for NULL conn");
 
 	sdp_dbg_ctrl(conn, "Defered reject <%08x>", conn->cm_id->local_id);
@@ -126,7 +126,7 @@ static void _sdp_cm_reject(void *arg)
 	result = ib_send_cm_rej(conn->cm_id, 
 				IB_CM_REJ_CONSUMER_DEFINED,
 				NULL, 0, NULL, 0);
-	if (0 > result)
+	if (result < 0)
 		sdp_dbg_warn(NULL, "Error <%d> CM reject request", result);
 
 	sdp_conn_put(conn);


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 27 03:24:11 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 27 Feb 2005 13:24:11 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <523bvl94za.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il>
	<523bvl94za.fsf@topspin.com>
Message-ID: <20050227112411.GJ23837@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
> 
>     Michael> The pingpong test fails for me, on x86_64. (The
>     Michael> executable being a 64 bit one). My kernel is 2.6.10.
> 
> OK, I installed x86_64 userspace and got user verbs working.  The
> first problem you ran into was because on a 32-bit arch,
> 
>     struct {
>             uint64_t a;
>             uint32_t b;
>     };
> 
> has size 12 but on a 64-bit arch it has size 16.  Because of the way I
> tested sizes of parameters passed into the kernel, it was basically
> looking for userspace to pass a size of 12.  I fixed this by padding
> all the structs to a multiple of 8 bytes so we don't run into this.
> 
> I also fixed a few warnings and other small glitches, and now the
> pingpong test works for me from both 32- and 64-bit userspace.
> 
> Since the ABI is different, you'll have to rebuild both the kernel
> modules and the userspace libraries when updating.
> 
> Thanks,
>   Roland
> 

Hmm.
# /usr/local/bin/ibv_pingpong
(no diagnostic is produced)
# /usr/local/bin/ibv_pingpong -d uverbs0
IB device uverbs0 not found

swlab155:/usr/src/roland-uverbs/src/userspace/libibverbs # ls -l
/dev/infiniband/
total 124
drwxr-xr-x   2 smtp root     4096 Feb 24 15:31 .
drwxr-xr-x  36 smtp root   118784 Feb 27 12:53 ..
crw-------   1 smtp root 231,  64 Feb 24 15:31 issm0
crw-------   1 smtp root 231,  65 Feb 24 15:31 issm1
crw-------   1 smtp root 231,   0 Feb  3 19:09 umad0
crw-------   1 smtp root 231,   1 Feb  3 19:09 umad1
crw-rw-rw-   1 smtp root 231, 128 Feb 24 15:31 uverbs0

Shall uverbs and umad share the same major?

swlab155:/usr/src/roland-uverbs/src/userspace/libibverbs # cat
/proc/devices | grep 231
231 infiniband_mad
231 infiniband_verbs

somewhat surprising.

Nodes are created by udev

# tail /etc/udev/udev.rules
KERNEL="mouse*",        NAME="input/%k"
KERNEL="event*",        NAME="input/%k"
KERNEL="js*",           NAME="input/%k"
KERNEL="ts*",           NAME="input/%k"

KERNEL="umad*", NAME="infiniband/%k"
KERNEL="issm*", NAME="infiniband/%k"
KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"

Interestingly, if I remove the node:

# rm /dev/infiniband/uverbs0
# /usr/local/bin/ibv_pingpong
Couldn't get context for mthca0

shouldnt it rather report that uverbs0 is missing?

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 27 06:54:19 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 27 Feb 2005 16:54:19 +0200
Subject: [openib-general] oops with user level verbs
Message-ID: <20050227145419.GP23837@mellanox.co.il>

Hi, Roland!
On one of my machines, I am often getting this oops when running
the pingpong example.

Feb 27 15:18:40 swlab156 kernel: Unable to handle kernel paging request at 000030616368746d RIP: 
Feb 27 15:18:40 swlab156 kernel: [<000030616368746d>]
Feb 27 15:18:40 swlab156 kernel: PML4 0 
Feb 27 15:18:40 swlab156 kernel: Oops: 0000 [1] SMP 
Feb 27 15:18:40 swlab156 kernel: CPU 1 
Feb 27 15:18:40 swlab156 kernel: Modules linked in: ib_uverbs ib_umad autofs usbserial parport_pc lp parport joydev st sr_mod floppy thermal processor fan button battery ipv6 ac ib_mthca ib_mad ib_core e1000 i2c_i801 i2c_core hw_random ehci_hcd uhci_hcd usbcore evdev dm_mod ext3 jbd aic79xx ata_piix libata sd_mod scsi_mod
Feb 27 15:18:40 swlab156 kernel: Pid: 12588, comm: ibv_pingpong Not tainted 2.6.10-openib
Feb 27 15:18:40 swlab156 kernel: RIP: 0010:[<000030616368746d>] [<000030616368746d>]
Feb 27 15:18:40 swlab156 kernel: RSP: 0018:0000010170adde90  EFLAGS: 00010212
Feb 27 15:18:40 swlab156 kernel: RAX: 000001017479b800 RBX: 0000010175c04618 RCX: 0000000000000000
Feb 27 15:18:40 swlab156 kernel: RDX: 00000101736d8880 RSI: 0000010175c04618 RDI: 0000010170985480
Feb 27 15:18:40 swlab156 kernel: RBP: 000001017f705cc0 R08: 0000010170addf08 R09: 0000000000000000
Feb 27 15:18:40 swlab156 kernel: R10: 0000010175c04618 R11: 00000000000000b0 R12: 0000000000001000
Feb 27 15:18:40 swlab156 kernel: R13: 0000002a95dba000 R14: 00000000ffffffea R15: 00000100bffaf6c0
Feb 27 15:18:41 swlab156 kernel: FS:  0000002a95cb50a0(0000) GS:ffffffff8045bc00(0000) knlGS:0000000000000000
Feb 27 15:18:41 swlab156 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 27 15:18:41 swlab156 kernel: CR2: 000030616368746d CR3: 000000000b14a000 CR4: 00000000000006e0
Feb 27 15:18:41 swlab156 kernel: Process ibv_pingpong (pid: 12588, threadinfo 0000010170adc000, task 0000010174a94a40)
Feb 27 15:18:41 swlab156 kernel: Stack: ffffffffa01db022 0000000000000212 ffffffff80164cb5 0000000000000000 
Feb 27 15:18:41 swlab156 kernel:        0000000000000018 0000007fbfffee20 00000101736d8880 0000000000000000 
Feb 27 15:18:41 swlab156 kernel:        00000001bfffee20 001000fa00000000 
Feb 27 15:18:41 swlab156 kernel: Call Trace:<ffffffffa01db022>{:ib_uverbs:ib_uverbs_mmap+32} <ffffffff80164cb5>{do_mmap_pgoff+1330} 
Feb 27 15:18:41 swlab156 kernel:        <ffffffff80114584>{sys_mmap+142} <ffffffff8010e226>{system_call+126} 
Feb 27 15:18:41 swlab156 kernel:        
Feb 27 15:18:41 swlab156 kernel: 
Feb 27 15:18:41 swlab156 kernel: Code:  Bad RIP value.
Feb 27 15:18:41 swlab156 kernel: RIP [<000030616368746d>] RSP <0000010170adde90>
Feb 27 15:18:41 swlab156 kernel: CR2: 000030616368746d
Feb 27 15:18:57 swlab156 sshd[12591]: Accepted keyboard-interactive/pam for root from ::ffff:10.4.4.15 port 2846 ssh2

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 27 07:19:26 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 27 Feb 2005 17:19:26 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050227112411.GJ23837@mellanox.co.il>
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il>
	<523bvl94za.fsf@topspin.com>
	<20050227112411.GJ23837@mellanox.co.il>
Message-ID: <20050227151926.GQ23837@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
> 
> Quoting r. Roland Dreier <roland at topspin.com>:
> > Subject: Re: ANNOUNCE: First usable version of userspace verbs
> > 
> >     Michael> The pingpong test fails for me, on x86_64. (The
> >     Michael> executable being a 64 bit one). My kernel is 2.6.10.
> > 
> > OK, I installed x86_64 userspace and got user verbs working.  The
> > first problem you ran into was because on a 32-bit arch,
> > 
> >     struct {
> >             uint64_t a;
> >             uint32_t b;
> >     };
> > 
> > has size 12 but on a 64-bit arch it has size 16.  Because of the way I
> > tested sizes of parameters passed into the kernel, it was basically
> > looking for userspace to pass a size of 12.  I fixed this by padding
> > all the structs to a multiple of 8 bytes so we don't run into this.
> > 
> > I also fixed a few warnings and other small glitches, and now the
> > pingpong test works for me from both 32- and 64-bit userspace.
> > 
> > Since the ABI is different, you'll have to rebuild both the kernel
> > modules and the userspace libraries when updating.
> > 
> > Thanks,
> >   Roland
> > 
> 
> Hmm.
> # /usr/local/bin/ibv_pingpong
> (no diagnostic is produced)

This seems to be caused by SM not running. Here's a patch to make
it produce a diagnostic message.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: libibverbs/examples/pingpong.c
===================================================================
--- libibverbs/examples/pingpong.c	(revision 1922)
+++ libibverbs/examples/pingpong.c	(working copy)
@@ -560,8 +560,10 @@
 	my_dest.lid = pp_get_local_lid(ib_dev, ib_port);
 	my_dest.qpn = ctx->qp->qp_num;
 	my_dest.psn = lrand48() & 0xffffff;
-	if (!my_dest.lid)
+	if (!my_dest.lid) {
+		fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n");
 		return 1;
+	}
 
 	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
 	       my_dest.lid, my_dest.qpn, my_dest.psn);


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 27 07:53:03 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 27 Feb 2005 17:53:03 +0200
Subject: [openib-general] Re: oops with user level verbs
In-Reply-To: <20050227145419.GP23837@mellanox.co.il>
References: <20050227145419.GP23837@mellanox.co.il>
Message-ID: <20050227155303.GR23837@mellanox.co.il>


Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: oops with user level verbs
> 
> Hi, Roland!
> On one of my machines, I am often getting this oops when running
> the pingpong example.
> 
> Feb 27 15:18:40 swlab156 kernel: Unable to handle kernel paging request at 000030616368746d RIP: 
> Feb 27 15:18:40 swlab156 kernel: [<000030616368746d>]
> Feb 27 15:18:40 swlab156 kernel: PML4 0 
> Feb 27 15:18:40 swlab156 kernel: Oops: 0000 [1] SMP 
> Feb 27 15:18:40 swlab156 kernel: CPU 1 
> Feb 27 15:18:40 swlab156 kernel: Modules linked in: ib_uverbs ib_umad autofs usbserial parport_pc lp parport joydev st sr_mod floppy thermal processor fan button battery ipv6 ac ib_mthca ib_mad ib_core e1000 i2c_i801 i2c_core hw_random ehci_hcd uhci_hcd usbcore evdev dm_mod ext3 jbd aic79xx ata_piix libata sd_mod scsi_mod
> Feb 27 15:18:40 swlab156 kernel: Pid: 12588, comm: ibv_pingpong Not tainted 2.6.10-openib
> Feb 27 15:18:40 swlab156 kernel: RIP: 0010:[<000030616368746d>] [<000030616368746d>]
> Feb 27 15:18:40 swlab156 kernel: RSP: 0018:0000010170adde90  EFLAGS: 00010212
> Feb 27 15:18:40 swlab156 kernel: RAX: 000001017479b800 RBX: 0000010175c04618 RCX: 0000000000000000
> Feb 27 15:18:40 swlab156 kernel: RDX: 00000101736d8880 RSI: 0000010175c04618 RDI: 0000010170985480
> Feb 27 15:18:40 swlab156 kernel: RBP: 000001017f705cc0 R08: 0000010170addf08 R09: 0000000000000000
> Feb 27 15:18:40 swlab156 kernel: R10: 0000010175c04618 R11: 00000000000000b0 R12: 0000000000001000
> Feb 27 15:18:40 swlab156 kernel: R13: 0000002a95dba000 R14: 00000000ffffffea R15: 00000100bffaf6c0
> Feb 27 15:18:41 swlab156 kernel: FS:  0000002a95cb50a0(0000) GS:ffffffff8045bc00(0000) knlGS:0000000000000000
> Feb 27 15:18:41 swlab156 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Feb 27 15:18:41 swlab156 kernel: CR2: 000030616368746d CR3: 000000000b14a000 CR4: 00000000000006e0
> Feb 27 15:18:41 swlab156 kernel: Process ibv_pingpong (pid: 12588, threadinfo 0000010170adc000, task 0000010174a94a40)
> Feb 27 15:18:41 swlab156 kernel: Stack: ffffffffa01db022 0000000000000212 ffffffff80164cb5 0000000000000000 
> Feb 27 15:18:41 swlab156 kernel:        0000000000000018 0000007fbfffee20 00000101736d8880 0000000000000000 
> Feb 27 15:18:41 swlab156 kernel:        00000001bfffee20 001000fa00000000 
> Feb 27 15:18:41 swlab156 kernel: Call Trace:<ffffffffa01db022>{:ib_uverbs:ib_uverbs_mmap+32} <ffffffff80164cb5>{do_mmap_pgoff+1330} 
> Feb 27 15:18:41 swlab156 kernel:        <ffffffff80114584>{sys_mmap+142} <ffffffff8010e226>{system_call+126} 
> Feb 27 15:18:41 swlab156 kernel:        
> Feb 27 15:18:41 swlab156 kernel: 
> Feb 27 15:18:41 swlab156 kernel: Code:  Bad RIP value.
> Feb 27 15:18:41 swlab156 kernel: RIP [<000030616368746d>] RSP <0000010170adde90>
> Feb 27 15:18:41 swlab156 kernel: CR2: 000030616368746d
> 

Happens now every time for me, but somewhat surprisingly not on
another, identical box.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Sun Feb 27 08:02:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 27 Feb 2005 18:02:08 +0200
Subject: [openib-general] pingpong problem
In-Reply-To: <52u0o4pfe8.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
Message-ID: <20050227160208.GS23837@mellanox.co.il>

Hi!
I'm running on x86_64 64 bit user and kernel space.
the pingpong test does not seem to work with size other than the
default 4096 (I am running with external loopback):

# ibv_pingpong --size=2048 &
local address:  LID 0x0001, QPN 0x1f0404, PSN 0x65e4f6
# ibv_pingpong --size=2048 10.4.8.155
  local address:  LID 0x0001, QPN 0x1d0405, PSN 0xf53639
  remote address: LID 0x0001, QPN 0x1d0405, PSN 0xf53639
  remote address: LID 0x0001, QPN 0x1d0404, PSN 0xa7fc92
  [ 0] 001d0405
  [ 4] b5000000
  [ 8] 0f000000
  [ c] 51010000
  [10] 01d701f3
  [14] 00000000
  [18] 00000002
  [1c] fe100000
  [ 0] 001d0405
  [ 4] b5000000
  [ 8] 0f000000
  [ 0] 001d0404
  [ 4] 00f83ced
  [ 8] 001da414
  [ c] 00c03800
  [10] 128a0000
  [14] 00000000
  [18] 00007d02
  [1c] ff100000
  [ c] 51010000
  [10] 05d701f2
  [14] 00000000
  [18] 00000042
  [1c] fe100000
Failed status 1 for wr_id 1
  [ 0] 001d0404
  [ 4] a5000000
  [ 8] 0f000002
  [ c] 510100c0
  [10] 05f401f2
  [14] 00000000
  [18] 00000042
  [1c] fe100000
Failed status 9 for wr_id 2
[2]+  Exit 1                  ibv_pingpong


Works fine with the default size:

# ibv_pingpong --size=2048 &
local address:  LID 0x0001, QPN 0x1f0404, PSN 0x65e4f6

# ibv_pingpong --size=4096 10.4.8.155
  local address:  LID 0x0001, QPN 0x1f0405, PSN 0xd206a0
  remote address: LID 0x0001, QPN 0x1f0404, PSN 0x65e4f6
  remote address: LID 0x0001, QPN 0x1f0405, PSN 0xd206a0
8192000 bytes in 0.02 seconds = 2935.41 Mbit/sec
8192000 bytes in 0.02 seconds = 3183.83 Mbit/sec
1000 iters in 0.02 seconds = 20.58 usec/iter
1000 iters in 0.02 seconds = 22.33 usec/iter

thanks,

-- 
MST - Michael S. Tsirkin


From shaharf at voltaire.com  Sun Feb 27 09:53:36 2005
From: shaharf at voltaire.com (shaharf)
Date: Sun, 27 Feb 2005 19:53:36 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <D4F8F0B3820E754C887699BEF26A89405A9BFB@taurus.voltaire.com>

The following email describes few new function calls for OpenIB access
layer that enable better/simpler use of the InfiniBand fabric resources,
called ib_at (IB address translation) See the attached preliminary ib_at
h file. 

 

There are 3 main functions and few additional helper functions and
structures

1. ib_at_route_by_ip()  - Translate from Destination IP (and optionally
Source IP, IP TOS) to the IB route properties (IB device, SGID/DGID, and
recommended Partition and QoS attributes) 

 

2. ib_at_paths_by_route() - Translate from IB route properties
(SGID/DGID and optionally partition, QoS info, Multipath type) to one or
more possible path records.

 

3. ib_at_ips_by_gid() - Reverse resolution from remote IB address to IP
address (like RARP)

 

 

Ib_at will provide the following internal functionality:

1. IP address translation to GID's and adapters/ports (using IB-ARP
and/or IB-ATS) 

2. Reverse translation from GID to IP (using IB-ATS) 

3. Determine default partition (P_Key) and QoS (SL, QoS Tag) values for
use in ULP's 

4. Obtain path information (SGID/DGID to path records) 

5. Obtain multiple paths for high-availability, QoS, multi-rail, and
source based routing (using LMC) 

6. Central path caching and invalidation 

 

The set of ib_at function calls allow for a central mechanism that will
deal with all address, path and path attributes resolution allowing
ULP's to become simpler, take advantage all IB capabilities without
special knowledge of IB architecture, and reduce manual configuration.  

 

See more background material and diagrams in (from the developers'
workshop): 

 
http://www.openib.org/docs/oib_wkshp_022005/part-qos-ha-voltaire-haviv.p
df

 

ib_at will rely on the Linux IP infrastructure, and the IB SA (ATS
ServiceRecords, PathRecord, or MultiPathRecord, or QoSPathRecord) to
provide the routing/path information.

The exact implementation will be proposed after the API discussion will
be agreed on, and may be implemented in several stages (e.g. cant use SA
Multipath before RMPP is available in the GSI).

 

* note a "QoS Tag" value is defined, it is automatically generated by
ib_at_route_by_ip() or optionally specifies by the ULP, initially it
will be mapped 1:1 to IB SL and in future will be aligned to the newly
defined IB-QoS architecture. 

 

Any comments and suggestions for changes, fixes and improvements will be
welcomed.

 

Shahar

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050227/422e3481/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ib_at.h
Type: application/octet-stream
Size: 7434 bytes
Desc: ib_at.h
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050227/422e3481/attachment.obj>

From asdfsad at sadhsd.com  Sun Feb 27 12:20:45 2005
From: asdfsad at sadhsd.com (SDFSDFSD)
Date: Mon, 28 Feb 2005 04:20:45 +0800
Subject: [openib-general] (no subject)
Message-ID: <20050227202252.77C512283DD@openib.ca.sandia.gov>



From itoumsn at nttdata.co.jp  Sun Feb 27 17:27:47 2005
From: itoumsn at nttdata.co.jp (Masanori ITOH)
Date: Mon, 28 Feb 2005 10:27:47 +0900 (JST)
Subject: [openib-general] Re: [Dapl-announce] Porting DAPL to OpenIB Gen 2
In-Reply-To: <Pine.LNX.4.61.0502241035440.5704@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0502241035440.5704@jlentini-linux.nane.netapp.com>
Message-ID: <20050228.102747.68476438.itoumsn@nttdata.co.jp>

Hi all,

If possible, I'm also going to join the conf-call.
# But, in Japan it's 1:30AM of March 1, and I'm not sure. :(

Regards,
Masanori

From: James Lentini <jlentini at netapp.com>
Subject: [Dapl-announce] Porting DAPL to OpenIB Gen 2
Date: Thu, 24 Feb 2005 12:30:10 -0500 (EST)

> 
> Yesterday the DAT Collaborative voted to add the GPL license to the 
> DAPL Source Forge reference implementation. We are now in a position 
> to begin porting the DAPL SourceForge project to OpenIB. I could like 
> to hold a conference call to help plan and divide up this work.
> 
> 
>           Date: Monday, February 28
>           Time: 11:30 AM EST
>       Domestic: 888-827-8686
> International: 303-928-2620
> Conference ID: 1125043

---
Masanori ITOH  Open Source Software Development Center, NTT DATA CORPORATION
               e-mail: itoumsn at nttdata.co.jp
               phone : +81-3-3523-8122 (ext. 172-7199)


From mst at mellanox.co.il  Mon Feb 28 01:21:15 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 11:21:15 +0200
Subject: [openib-general] Re: pingpong problem
In-Reply-To: <20050227160208.GS23837@mellanox.co.il>
References: <52u0o4pfe8.fsf@topspin.com>
	<20050227160208.GS23837@mellanox.co.il>
Message-ID: <20050228092115.GT23837@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: pingpong problem
> 
> Hi!
> I'm running on x86_64 64 bit user and kernel space.
> the pingpong test does not seem to work with size other than the
> default 4096 (I am running with external loopback):

Tried again today after a reboot and could not reproduce it
anymore. The problem could have originated somewhere between the keyboard and
chair, I guess.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 28 01:35:20 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 11:35:20 +0200
Subject: [openib-general] [PATCH][uverbs] use get_task_mm
Message-ID: <20050228093520.GU23837@mellanox.co.il>

The task mm may go away while umem_get is in progress.
Semaphore access will then crash. Use get_task_mm to prevent this.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: core/uverbs_mem.c
===================================================================
--- core/uverbs_mem.c	(revision 1922)
+++ core/uverbs_mem.c	(working copy)
@@ -47,11 +47,16 @@ int ib_umem_get(struct ib_device *dev, s
 	int ret = 0;
 	int off;
 	int i;
+	struct mm_struct * mm;
 
 	page_list = (struct page **) __get_free_page(GFP_KERNEL);
 	if (!page_list)
 		return -ENOMEM;
 
+	mm = get_task_mm(current);
+	if (!mm)
+		return -EINTR;
+
 	mem->length    = size;
 	mem->offset    = (unsigned long) addr & ~PAGE_MASK;
 	mem->page_size = PAGE_SIZE;
@@ -61,10 +66,10 @@ int ib_umem_get(struct ib_device *dev, s
 	npages   = PAGE_ALIGN(size + mem->offset) >> PAGE_SHIFT;
 	cur_base = (unsigned long) addr & PAGE_MASK;
 
-	down_read(&current->mm->mmap_sem);
+	down_read(&mm->mmap_sem);
 
 	while (npages) {
-		ret = get_user_pages(current, current->mm, cur_base,
+		ret = get_user_pages(current, mm, cur_base,
 				     min_t(int, npages,
 					   PAGE_SIZE / sizeof (struct page *)),
 				     1, 0, page_list, NULL);
@@ -120,7 +125,8 @@ int ib_umem_get(struct ib_device *dev, s
 	}
 
 out:
-	up_read(&current->mm->mmap_sem);
+	up_read(&mm->mmap_sem);
+	mmput(mm);
 	free_page((unsigned long) page_list);
 
 	if (ret < 0)

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 28 01:49:08 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 11:49:08 +0200
Subject: [openib-general] [PATCH][uverbs] cleanup after error
Message-ID: <20050228094908.GV23837@mellanox.co.il>

The pd is already allocated when idr_pre_get is called. So, must deallocate
on error.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: core/uverbs_cmd.c
===================================================================
--- core/uverbs_cmd.c	(revision 1922)
+++ core/uverbs_cmd.c	(working copy)
@@ -154,7 +154,7 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uve
 retry:
 	if (!idr_pre_get(&ib_uverbs_pd_idr, GFP_KERNEL)) {
 		ret = -ENOMEM;
-		goto err;
+		goto err_pd;
 	}
 
 	down(&ib_uverbs_idr_mutex);

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 28 04:30:20 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 14:30:20 +0200
Subject: [openib-general] uverbs: reg_mr
Message-ID: <20050228123020.GA31156@mellanox.co.il>

Hi, Roland!
I'd like to suggest extending ibv_reg_mr to pass the
virtual address for RDMA, in addition to the CPU virtual address.

Currently it seems they are assumed to be the same, but there
does not appear to be an actual reason to enforce this limitation.

For example, I am now writing a simple test application for rdma,
and it would be convenient to simply rdma to virtual address 0,
and have the HCA land it at the proper address.
As it is, I have to pass the virtual address along with the RKey.

-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 28 05:12:23 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 15:12:23 +0200
Subject: [openib-general] [PATCH] [uverbs] declare qp_access with enum type
Message-ID: <20050228131223.GB31156@mellanox.co.il>

Declare qp_access_flags with the appropriate enum type.
Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: userspace/libibverbs/include/infiniband/verbs.h
===================================================================
--- userspace/libibverbs/include/infiniband/verbs.h	(revision 1922)
+++ userspace/libibverbs/include/infiniband/verbs.h	(working copy)
@@ -264,7 +264,7 @@
 	uint32_t		rq_psn;
 	uint32_t		sq_psn;
 	uint32_t		dest_qp_num;
-	int			qp_access_flags;
+	enum ibv_access_flags	qp_access_flags;
 	struct ibv_qp_cap	cap;
 	struct ibv_ah_attr	ah_attr;
 	struct ibv_ah_attr	alt_ah_attr;

-- 
MST - Michael S. Tsirkin


From jlentini at netapp.com  Mon Feb 28 07:29:47 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 28 Feb 2005 10:29:47 -0500 (EST)
Subject: [openib-general] Re: Porting DAPL to OpenIB Gen 2
In-Reply-To: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
Message-ID: <Pine.LNX.4.61.0502281025340.5704@jlentini-linux.nane.netapp.com>


Below is an agenda for the call:

+ Additional Items?

+ Enumerate Tasks:
  - uDAPL
    who: Arlin Davis at Intel

   - kDAPL
     who: Itamar Rabenstein at Mellanox
          Voltaire for CM related features

   - Linux coding style
     who: everyone

   - kDAPL API changes for Linux coding style
     who: James Lentini and Arkady Kanevsky at NetApp

   - tools (dapltest, dat conformance test, etc.)
     who: volunteers?

   - documentation (user guide, design guides, etc.)
     who: volunteers? everyone?

+ Appropriate location for DAPL source code in OpenIB tree.
   Split kernel and user space code? Split registry and provider?

+ Procedure for checkins

+ SourceForge synchronization


On Thu, 24 Feb 2005, James Lentini wrote:

>
> Yesterday the DAT Collaborative voted to add the GPL license to the DAPL 
> Source Forge reference implementation. We are now in a position to begin 
> porting the DAPL SourceForge project to OpenIB. I could like to hold a 
> conference call to help plan and divide up this work.
>
>
>         Date: Monday, February 28
>         Time: 11:30 AM EST
>     Domestic: 888-827-8686
> International: 303-928-2620
> Conference ID: 1125043
>
>
> James Lentini                email: jlentini at netapp.com
> Network Appliance            phone: 781-768-5359
> 375 Totten Pond Rd.            fax: 781-895-1195
> Waltham, MA 02451-2010        main: 781-768-5300
>
>


From mst at mellanox.co.il  Mon Feb 28 07:32:56 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 17:32:56 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <523bvl94za.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il>
	<523bvl94za.fsf@topspin.com>
Message-ID: <20050228153256.GA31510@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
>
> I also fixed a few warnings and other small glitches, and now the
> pingpong test works for me from both 32- and 64-bit userspace.

I still see these:

src/cmd.c: In function `ibv_cmd_create_qp':
src/cmd.c:187: warning: unsigned int format, pointer arg (arg 3)
src/cmd.c:187: warning: unsigned int format, different type arg (arg 5)


-- 
MST - Michael S. Tsirkin


From mst at mellanox.co.il  Mon Feb 28 07:42:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 17:42:54 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <52u0o4pfe8.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
Message-ID: <20050228154254.GB31510@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: ANNOUNCE: First usable version of userspace verbs
> 
> To actually try things out, you can use the ibv_pingpong program
> shipped as part of the libibverbs package.  For example, one one
> system start the server side
> 
>     $ ibv_pingpong
> 
> and on another system start the client by passing the address of the
> server (in this example I use IPv6 over IPoIB):
> 
>     $ ibv_pingpong fe80::202:c901:7fc:c711%ib0
> 
> The pingpong program has a number of options -- run ibv_pingpong -h to
> see a list of the switches you can try.
> 
> The current code is stable for me, but all that means is that my tiny
> selection of tests and test systems has not uncovered any of the bugs
> that are undoubtedly present.  Some of the limitations I know about:
> 

Interestingly, if I rebuild libmthca with -O3 compiler flag, the
pingpong program does not make progress.
Building libibverbs or the test itself with -O3 has no such effect.

# gcc --version
gcc (GCC) 3.3.3 (SuSE Linux)
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This is somewhat troubling, since exactly which optimisations are set
in -O2 and which in -O3 changes between gcc versions, and so,
unless it turns out to be a compiler bug, we may run into this problem even
with current -O2 in the future.

Roland - can you reproduce this problem?

-- 
MST - Michael S. Tsirkin


From roland at topspin.com  Mon Feb 28 09:13:21 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:13:21 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405A9BFB@taurus.voltaire.com>
	(shaharf@voltaire.com's
	message of "Sun, 27 Feb 2005 19:53:36 +0200")
References: <D4F8F0B3820E754C887699BEF26A89405A9BFB@taurus.voltaire.com>
Message-ID: <52d5ukvc9a.fsf@topspin.com>

This API seems overly complex and at the same time too inflexible to
me.  However, rather than getting bogged down nitpicking about APIs, I
think we have to take a few steps back.

First, let's understand the problem we're trying to solve.  Who are
the consumers of this address translation service?

Second, let's come up with the right architecture to solve the
problem.  Are we implementing a library in userspace or a kernel
module?  Do we have a single cache or do we need multiple caching
policies?  And so on...

Finally, we can design the API.

Thanks,
  Roland




From roland at topspin.com  Mon Feb 28 09:16:29 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:16:29 -0800
Subject: [openib-general] [PATCH][uverbs] cleanup after error
In-Reply-To: <20050228094908.GV23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 28 Feb 2005 11:49:08 +0200")
References: <20050228094908.GV23837@mellanox.co.il>
Message-ID: <528y58vc42.fsf@topspin.com>

Thanks, looks like I had the same bug in create CQ as well...

 - R.


From roland at topspin.com  Mon Feb 28 09:17:36 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:17:36 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050227112411.GJ23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Sun, 27 Feb 2005 13:24:11 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il> <523bvl94za.fsf@topspin.com>
	<20050227112411.GJ23837@mellanox.co.il>
Message-ID: <524qfwvc27.fsf@topspin.com>

    Michael> Shall uverbs and umad share the same major?

Yes, they should.  umad uses minors 0 ... 127 and uverbs uses minors
128 ... 160.  See <http://www.lanana.org/docs/device-list/devices-2.6+.txt>

 - R.


From roland at topspin.com  Mon Feb 28 09:18:14 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:18:14 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050228153256.GA31510@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 28 Feb 2005 17:32:56 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050224145856.GZ23837@mellanox.co.il> <523bvl94za.fsf@topspin.com>
	<20050228153256.GA31510@mellanox.co.il>
Message-ID: <52zmxotxgp.fsf@topspin.com>

    Michael> I still see these:

    Michael> src/cmd.c: In function `ibv_cmd_create_qp':
    Michael> src/cmd.c:187: warning: unsigned int format, pointer arg
    Michael> (arg 3) src/cmd.c:187: warning: unsigned int format,
    Michael> different type arg (arg 5)

Is this coming from debugging code you added?  I don't see any printf
calls at all in my version of cmd.c.

 - R.



From roland at topspin.com  Mon Feb 28 09:19:45 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:19:45 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050228154254.GB31510@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 28 Feb 2005 17:42:54 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050228154254.GB31510@mellanox.co.il>
Message-ID: <52vf8ctxe6.fsf@topspin.com>

    Michael> This is somewhat troubling, since exactly which
    Michael> optimisations are set in -O2 and which in -O3 changes
    Michael> between gcc versions, and so, unless it turns out to be a
    Michael> compiler bug, we may run into this problem even with
    Michael> current -O2 in the future.

    Michael> Roland - can you reproduce this problem?

I'll take a look.  I assume this is on x86_64?

 - R.


From roland at topspin.com  Mon Feb 28 09:20:35 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:20:35 -0800
Subject: [openib-general] Re: oops with user level verbs
In-Reply-To: <20050227155303.GR23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Sun, 27 Feb 2005 17:53:03 +0200")
References: <20050227145419.GP23837@mellanox.co.il>
	<20050227155303.GR23837@mellanox.co.il>
Message-ID: <52r7j0txcs.fsf@topspin.com>

Hmm, I can't reproduce this.  It looks like ib_uverbs_mmap() is
following a bad ib_dev->mmap() function pointer, but I can't see how
that could happen.

 - R.


From roland at topspin.com  Mon Feb 28 09:21:37 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:21:37 -0800
Subject: [openib-general] Re: pingpong problem
In-Reply-To: <20050227160208.GS23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Sun, 27 Feb 2005 18:02:08 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050227160208.GS23837@mellanox.co.il>
Message-ID: <52mztotxb2.fsf@topspin.com>

    Michael> Hi!  I'm running on x86_64 64 bit user and kernel space.
    Michael> the pingpong test does not seem to work with size other
    Michael> than the default 4096 (I am running with external
    Michael> loopback):

Looks like you had a size mismatch between the two sides.  In which
case the behavior would be exactly as you saw: the side with the
smaller size will get a local length error for its receive.

 - R.


From mshefty at ichips.intel.com  Mon Feb 28 09:23:25 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Mon, 28 Feb 2005 09:23:25 -0800
Subject: [openib-general] Re: CM REP-REJ bug
In-Reply-To: <20050225170105.D13364@topspin.com>
References: <20050224144544.411d51ba.mshefty@ichips.intel.com>
	<421FAF14.1090907@ichips.intel.com>
	<20050225170105.D13364@topspin.com>
Message-ID: <4223538D.407@ichips.intel.com>

Libor Michalek wrote:
>    There's a problem with REJ response to REQ. The REJ handler
> uses cm_acquire_id() to find the cm_id of the REQ, but it uses 
> rej_msg->local_comm_id to perform the lookup. In the REQ case the
> cm_id does not have a remote_id with which to match the REJ.

My thinking on this was that the for the sender of the REQ, the remote 
ID would be set to 0.  The sender of the REJ would set the local ID to 
0, since it wouldn't have one (unless it had previously sent an MRA). 
Does this make sense, or am I missing something else here?

- Sean


From roland at topspin.com  Mon Feb 28 09:23:56 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:23:56 -0800
Subject: [openib-general] [PATCH][uverbs] use get_task_mm
In-Reply-To: <20050228093520.GU23837@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 28 Feb 2005 11:35:20 +0200")
References: <20050228093520.GU23837@mellanox.co.il>
Message-ID: <52is4ctx77.fsf@topspin.com>

    Michael> The task mm may go away while umem_get is in progress.

I guess this patch can't hurt but can the mm really go away?  The only
time we will call ib_umem_get() is during a write system call (which
requires an open file, obviously).  Can current->mm go away before all
files are closed?

 - R.


From roland at topspin.com  Mon Feb 28 09:27:03 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:27:03 -0800
Subject: [openib-general] [PATCH][uverbs] use get_task_mm
In-Reply-To: <52is4ctx77.fsf@topspin.com> (Roland Dreier's message of "Mon,
	28 Feb 2005 09:23:56 -0800")
References: <20050228093520.GU23837@mellanox.co.il>
	<52is4ctx77.fsf@topspin.com>
Message-ID: <52ekf0tx20.fsf@topspin.com>

    Roland> I guess this patch can't hurt but can the mm really go
    Roland> away?  The only time we will call ib_umem_get() is during
    Roland> a write system call (which requires an open file,
    Roland> obviously).  Can current->mm go away before all files are
    Roland> closed?

I didn't phrase this precisely as I meant.  Can current->mm go away
during a system call for current's task?

 - R.


From roland at topspin.com  Mon Feb 28 09:41:54 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 09:41:54 -0800
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <20050228154254.GB31510@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 28 Feb 2005 17:42:54 +0200")
References: <52u0o4pfe8.fsf@topspin.com>
	<20050228154254.GB31510@mellanox.co.il>
Message-ID: <52acpotwd9.fsf@topspin.com>

    Michael> Interestingly, if I rebuild libmthca with -O3 compiler
    Michael> flag, the pingpong program does not make progress.
    Michael> Building libibverbs or the test itself with -O3 has no
    Michael> such effect.

I can't reproduce this on either i386 or x86-64 (Intel Nocona system).

    $ gcc --version
    gcc (GCC) 3.3.5 (Debian 1:3.3.5-8)
    Copyright (C) 2003 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 - R.


From xma at us.ibm.com  Mon Feb 28 09:49:34 2005
From: xma at us.ibm.com (Shirley Ma)
Date: Mon, 28 Feb 2005 10:49:34 -0700
Subject: [openib-general] [PATCH][IPOIB]mcast restart causes kernel oops
Message-ID: <OFE99E1A3C.C11BAAA2-ON87256FB6.00614982-88256FB6.0061EBB8@us.ibm.com>

list_for_each_entry() should be list_for_each_entry_safe() in 
ipoib_mcast_restart_task(). Here is the patch. Please review it.

Signed-off-by: Shirley Ma <xma at us.ibm.com>

diff -urpN infiniband/ulp/ipoib/ipoib_multicast.c 
infiniband-mcast/ulp/ipoib/ipoib_multicast.c
--- infiniband/ulp/ipoib/ipoib_multicast.c      2005-02-24 
20:20:58.000000000 -0800
+++ infiniband-mcast/ulp/ipoib/ipoib_multicast.c        2005-02-28 
09:37:16.387272016 -0800
@@ -908,7 +908,7 @@ void ipoib_mcast_restart_task(void *dev_
        spin_unlock_irqrestore(&priv->lock, flags);
 
        /* We have to cancel outside of the spinlock */
-       list_for_each_entry(mcast, &remove_list, list) {
+       list_for_each_entry_safe(mcast, &remove_list, list) {
                ipoib_mcast_leave(mcast->dev, mcast);
                ipoib_mcast_free(mcast);
        }





Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050228/3d1bb1be/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infiniband-list.patch
Type: application/octet-stream
Size: 598 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050228/3d1bb1be/attachment.obj>

From shaharf at voltaire.com  Mon Feb 28 10:00:31 2005
From: shaharf at voltaire.com (shaharf)
Date: Mon, 28 Feb 2005 20:00:31 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>

> This API seems overly complex and at the same time too inflexible to
> me.  However, rather than getting bogged down nitpicking about APIs, I
> think we have to take a few steps back.
> 

I am very open to suggestions.

> First, let's understand the problem we're trying to solve.  Who are
> the consumers of this address translation service?
> 
Any ULPs at user & kernel, and also some applications. The motivations
are:

1. Implement central address resolution functionality to enable
implementing system policies and preferences for all clients (e.g. use
ATS instead of IB-ARP)
2. Enable the (future) implementation of resolution related enhancements
such as qos mapping, several port/path fail over mechanisms (APM, port
fail over, application fail over, etc.) and source routing (enable smart
client based source ports/paths by ensuring some agreed semantics within
the lid range.

I think that we can agree that there is no use to replicate the
functionality within application and ULPs.

> Second, let's come up with the right architecture to solve the
> problem.  Are we implementing a library in userspace or a kernel
> module?  Do we have a single cache or do we need multiple caching
> policies?  And so on...
> 

My take right now is to implement a kernel based mechanism and a user
mode library to interface it.
There are other feasible solutions. I would really like you have your
suggestions and preferences.

The caching issues are non-trivial (to say the least). My approach is to
set up the minimum requirements that will enable caching in the (near)
future.
I guess that the interface will be changed and enhanced to support the
caching. It will be nice to close this issue before the implementation,
but I don't think that this is mandatory. Very frequently you can't
simply design everything before you start. The trick is to find the
correct balance.

> Finally, we can design the API.
>

I think that starting with the APIs is a valid approach that has its own
advantages and disadvantages.
 
Right now, the proposed interface does not impose anything. It is just a
direction to start with. This is good even just to place the cards on
the table and start the discussion.

> Thanks,
>   Roland
> 
Shahar


From libor at topspin.com  Mon Feb 28 10:01:50 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 10:01:50 -0800
Subject: [openib-general] Re: CM REP-REJ bug
In-Reply-To: <4223538D.407@ichips.intel.com>;
	from mshefty@ichips.intel.com on Mon, Feb 28, 2005 at 09:23:25AM
	-0800
References: <20050224144544.411d51ba.mshefty@ichips.intel.com>
	<421FAF14.1090907@ichips.intel.com>
	<20050225170105.D13364@topspin.com> <4223538D.407@ichips.intel.com>
Message-ID: <20050228100150.A21494@topspin.com>

On Mon, Feb 28, 2005 at 09:23:25AM -0800, Sean Hefty wrote:
> Libor Michalek wrote:
> >    There's a problem with REJ response to REQ. The REJ handler
> > uses cm_acquire_id() to find the cm_id of the REQ, but it uses 
> > rej_msg->local_comm_id to perform the lookup. In the REQ case the
> > cm_id does not have a remote_id with which to match the REJ.
> 
> My thinking on this was that the for the sender of the REQ, the remote 
> ID would be set to 0.  The sender of the REJ would set the local ID to 
> 0, since it wouldn't have one (unless it had previously sent an MRA). 
> Does this make sense, or am I missing something else here?

  How does the user set the local ID to zero, just reach into the
connection identifier and set local_id? This will not cause any
lookup or other problems? Seems unusual to start reaching into the
cm_id and start changing values. How about if ib_send_cm_rej()
made this change based on the CM state in which it was called?

-Libor



From mshefty at ichips.intel.com  Mon Feb 28 10:09:56 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Mon, 28 Feb 2005 10:09:56 -0800
Subject: [openib-general] Re: CM REP-REJ bug
In-Reply-To: <20050228100150.A21494@topspin.com>
References: <20050224144544.411d51ba.mshefty@ichips.intel.com>
	<421FAF14.1090907@ichips.intel.com>
	<20050225170105.D13364@topspin.com> <4223538D.407@ichips.intel.com>
	<20050228100150.A21494@topspin.com>
Message-ID: <42235E74.5040102@ichips.intel.com>

Libor Michalek wrote:

>>My thinking on this was that the for the sender of the REQ, the remote 
>>ID would be set to 0.  The sender of the REJ would set the local ID to 
>>0, since it wouldn't have one (unless it had previously sent an MRA). 
>>Does this make sense, or am I missing something else here?
> 
>   How does the user set the local ID to zero, just reach into the
> connection identifier and set local_id? This will not cause any
> lookup or other problems? Seems unusual to start reaching into the
> cm_id and start changing values. How about if ib_send_cm_rej()
> made this change based on the CM state in which it was called?

The local/remote IDs shouldn't be modified by the user.

I see the issue now.  The problem is that the sender of the REJ is 
setting the local comm ID to a non-zero value.  It needs to be set to 
zero if rejecting a REQ.  Thanks for finding this; I'll create a patch 
for it.

- Sean


From roland at topspin.com  Mon Feb 28 10:21:15 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 10:21:15 -0800
Subject: [openib-general] [PATCH][IPOIB]mcast restart causes kernel oops
In-Reply-To: <OFE99E1A3C.C11BAAA2-ON87256FB6.00614982-88256FB6.0061EBB8@us.ibm.com>
	(Shirley Ma's message of "Mon, 28 Feb 2005 10:49:34 -0700")
References: <OFE99E1A3C.C11BAAA2-ON87256FB6.00614982-88256FB6.0061EBB8@us.ibm.com>
Message-ID: <523bvgtujo.fsf@topspin.com>

Thanks, good catch.

+       list_for_each_entry_safe(mcast, &remove_list, list) {

should really be

+       list_for_each_entry_safe(mcast, tmcast, &remove_list, list) {

though, right?

When you send a patch with a title like "mcast restart causes kernel
oops" can you make it clearer if you actually saw the oops or just
found it by code review?  The best thing to do if you actually see the
oops is to include it with your patch, because then we can make sure
the patch fixes the root cause of the problem.

Also if you are fixing a reproducible oops, please verify that your
patch actually fixes the oops.  Since this patch doesn't even compile,
I know you didn't do that this time ;)

Thanks,
  Roland


From roland at topspin.com  Mon Feb 28 10:29:19 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 10:29:19 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>
	(shaharf@voltaire.com's
	message of "Mon, 28 Feb 2005 20:00:31 +0200")
References: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>
Message-ID: <52y8d8sfls.fsf@topspin.com>

    Roland> First, let's understand the problem we're trying to solve.
    Roland> Who are the consumers of this address translation service?

    shaharf> Any ULPs at user & kernel, and also some
    shaharf> applications.

I think this is too general an answer.  We should be designing based
on specific ULPs and applications.  For example, I don't see anything
particularly useful to IPoIB in this API.  Perhaps Libor can comment
on how this API works for SDP.

What application would use functions like ib_at_ips_by_gid90 or
ib_at_ips_by_subnet()?

    shaharf> My take right now is to implement a kernel based
    shaharf> mechanism and a user mode library to interface it.  There
    shaharf> are other feasible solutions. I would really like you
    shaharf> have your suggestions and preferences.

Unless there is a real kernel consumer that needs something this
elaborate, I would prefer to implement this sort of caching service as
a userspace daemon/library.  This allows for more sophisticated
implementations (eg persistent caches) and also makes debugging and
maintenance easier.

    shaharf> I think that starting with the APIs is a valid approach
    shaharf> that has its own advantages and disadvantages.
 
Sure, it's always good to have code in hand to start a discussion.
But in this case the API seems to be far ahead of its consumers, so it
ends up feeling overengineered to me.

 - R.


From mshefty at ichips.intel.com  Mon Feb 28 10:50:36 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Mon, 28 Feb 2005 10:50:36 -0800
Subject: [openib-general] [PATCH] [CM] set REJ local ID to 0 when responding
	to a REQ
Message-ID: <20050228105036.122e3b1c.mshefty@ichips.intel.com>

This patch sets the local comm ID to 0 when sending a REJ in response
to a REQ.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

Index: infiniband/core/cm.c
===================================================================
--- infiniband/core/cm.c	(revision 1927)
+++ infiniband/core/cm.c	(working copy)
@@ -1706,17 +1706,19 @@ static void cm_format_rej(struct cm_rej_
 	cm_format_mad_hdr(&rej_msg->hdr, cm_id_priv,
 			  CM_REJ_ATTR_ID, CM_MSG_SEQUENCE_REQ);
 
-	rej_msg->local_comm_id = cm_id_priv->id.local_id;
 	rej_msg->remote_comm_id = cm_id_priv->id.remote_id;
 
 	switch(cm_id_priv->id.state) {
 	case IB_CM_REQ_RCVD:
+		rej_msg->local_comm_id = 0;
 		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ);
 		break;
 	case IB_CM_REP_RCVD:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
 		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REP);
 		break;
 	default:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
 		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_OTHER);
 		break;
 	}


From libor at topspin.com  Mon Feb 28 10:55:25 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 10:55:25 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <52y8d8sfls.fsf@topspin.com>;
	from roland@topspin.com on Mon, Feb 28, 2005 at 10:29:19AM -0800
References: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>
	<52y8d8sfls.fsf@topspin.com>
Message-ID: <20050228105525.B21494@topspin.com>

On Mon, Feb 28, 2005 at 10:29:19AM -0800, Roland Dreier wrote:
>     Roland> First, let's understand the problem we're trying to solve.
>     Roland> Who are the consumers of this address translation service?
> 
>     shaharf> Any ULPs at user & kernel, and also some
>     shaharf> applications.
> 
> I think this is too general an answer.  We should be designing based
> on specific ULPs and applications.  For example, I don't see anything
> particularly useful to IPoIB in this API.  Perhaps Libor can comment
> on how this API works for SDP.

  SDP does implement a subset of the proposed functionality for
resolving IP addresses to PathRecords which can then be used in
a CM REQ request, plus some basic caching. All the code is isolated
to a single file, sdp_link.c. There's really only a single entry
point API, plus a completion function:

int sdp_link_path_lookup(u32 dst_addr,
			 u32 src_addr,
			 int bound_dev_if,
			 void (*completion)(u64 id,
					    int status,
					    u32 dst_addr,
					    u32 src_addr,
					    u8  hw_port,
					    struct ib_device *ca,
					    struct ib_sa_path_rec *path,
					    void *arg),
			 void *arg,
			 u64  *id);

  The values are based on strictly what is needed by either the Linux
routing code to resolve the address, or the IB APIs to establish the
connection. The implementation has three stages:

  - src/dst IP address -> IPoIB net_device, IB ca, IB port, IB pkey.
  - dst IP address and IPoIB net_device -> dst GID using IPoIB ARP
  - dst GID -> PathRecord using ib_sa.

  A cancel function based on the 'id' parameter would be a nice to have
but is not strictly necessary, since the lookup will eventually compelte
one way or another and any dead connection will be cleaned up at that
point. 

>     shaharf> My take right now is to implement a kernel based
>     shaharf> mechanism and a user mode library to interface it.  There
>     shaharf> are other feasible solutions. I would really like you
>     shaharf> have your suggestions and preferences.
> 
> Unless there is a real kernel consumer that needs something this
> elaborate, I would prefer to implement this sort of caching service as
> a userspace daemon/library.  This allows for more sophisticated
> implementations (eg persistent caches) and also makes debugging and
> maintenance easier.
> 
>     shaharf> I think that starting with the APIs is a valid approach
>     shaharf> that has its own advantages and disadvantages.
>  
> Sure, it's always good to have code in hand to start a discussion.
> But in this case the API seems to be far ahead of its consumers, so it
> ends up feeling overengineered to me.

  Maybe a list of near term ULPs and APIs and the functions that they
are very likely to need? All though it usually makes sens to add the
functionality to a bare framework as the ULPs and APIs are added to
avoid functionality for a ULP/API that never ends up making it to the
kernel...

  Also, I'm not too excited about mixing IP addressing with ATS.


-Libor




From yaronh at voltaire.com  Mon Feb 28 11:15:46 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Mon, 28 Feb 2005 21:15:46 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC5D@taurus.voltaire.com>

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Roland Dreier
> Sent: Monday, February 28, 2005 7:13 PM
> To: shaharf
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] IB Address Translation service
> 
> This API seems overly complex and at the same time too inflexible to
> me.  However, rather than getting bogged down nitpicking about APIs, I
> think we have to take a few steps back.

I believe the API is very flexible, but we are pretty open to here what
you think is needed in addition 

> First, let's understand the problem we're trying to solve.  Who are
> the consumers of this address translation service?

The first problem is that most ULPs use valid IP addresses for
simplicity (DAPL, iSER, NFS/RDMA, SDP, MPI, etc') and someone needs to
resolve it to an IB address and device to use IB. This should take into
account cases where there are more than one HCAs in the system.
Preferable/optionally the ULP would like to know which partition to use
if there is more than one, and leverage on the IP subnetting done by
IPoIB.

It is possible to replicate the same code you have in SDP (which is also
not complete) across all ULP's, I assume a better way is to provide it
in one central place.
There are also two proposed address resolution mechanisms, one is ARP
used by SDP, and one is ATS used by some DAPL consumers, and we believe
it is better to combine them under the same API.

The second problem relates to mapping of IB GID to one or more Path
records
This is also something needed for ALL ULP's. today each ULP provides the
minimal subset of path resolution functionality without taking into
account topics such as partitioning, QoS, source routing and
multi-pathing.
Some of these require using special SA queries (such as SA Multipath
Record query and QoSPath Query).
I don't think it make sense to put all this functionality into each ULP
as well.

Than we can also discuss, does it make sense to have each path
resolution call lead us to the sa, or does it make more sense to cache
those paths.
And if we cache, doesn't it make more sense to cache/invalidate the
routes to all ULP's rather implementing/having it in each ULP.
Also not sure how a 1000 node cluster functions without the caching.
 
And the last problem is related to reverse resolution from IB to IP
addresses that is needed for DAPL, as well as for different management
and diagnostic tools that want to know what is really that node/port
behind that GID addresses.

So how would you suggest to go about it ?
Duplicate all of that in each ULP ?
Refrain from implementing advanced routing, partitioning, QoS (we cant
really maintain all that advanced code for each ULP) ? 

Our idea is to provide those few helper functions that enable people to
make full use of IB and its features without reading all the IB spec,
and a Phd.
If you clear all the remarks from the library, you will see it is very
slim, and for my understanding includes all the relevant input and
output parameters for each of the 3 functions I mentioned.

As shahar mentioned, this is just a proposal, and if you see any thing
missing in the API, or a better way to address the requirements I just
listed, I'm happy to here.

The API doesn't define the implementation, which we can discuss once we
agree on the functionality and interfaces, and you have some valid
questions there needs to be addressed.

Yaron
 
> Second, let's come up with the right architecture to solve the
> problem.  Are we implementing a library in userspace or a kernel
> module?  Do we have a single cache or do we need multiple caching
> policies?  And so on...
> 
> Finally, we can design the API.
> 
> Thanks,
>   Roland
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-
> general


From paul.baxter at dsl.pipex.com  Mon Feb 28 11:31:47 2005
From: paul.baxter at dsl.pipex.com (Paul Baxter)
Date: Mon, 28 Feb 2005 19:31:47 -0000
Subject: [openib-general] IB Address Translation service
References: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>
	<52y8d8sfls.fsf@topspin.com>
Message-ID: <007801c51dcc$2528ec60$8000000a@blorp>

>    Roland> First, let's understand the problem we're trying to solve.
>    Roland> Who are the consumers of this address translation service?
>
>    shaharf> Any ULPs at user & kernel, and also some
>    shaharf> applications.
>
> I think this is too general an answer.  We should be designing based
> on specific ULPs and applications.  For example, I don't see anything
> particularly useful to IPoIB in this API.  Perhaps Libor can comment
> on how this API works for SDP.
>

Recently a project I've been working on has been looking at porting IB 
software to an embedded PowerPC though my question also applies to a slimmed 
down embedded Linux PC system.

If I wanted to support SDP over a link with one end being embedded, I had 
thought I'd need to port the whole of the IPoIB driver for address 
resolution (albeit I may be able to cut a few corners with the code.)

Would this proposed address translation layer interface provide a sufficient 
subset of IPoIB functionality that allowed an SDP implementation to exist 
without a larger, full-featured IPoIB implementation?

If so, it sounds like a useful decoupling (refactoring) of a more generally 
useful subset of the IPoIB driver. It wouldn't be of use for the IPoIB 
driver per se, but would be more modular and suited to other applications 
particularly slimmed down ones. (Perhaps finding a place in Linux 'on a 
bios' for network booting or slimmed down embedded processing nodes.)

Having now just read Yaron's reply, I am even more convinced that this is 
the right way to go albeit I can't comment on the API etc (Could someone 
explain the differences in using ARP and ATS. )

Paul

PS As is probably evident from the above, I didn't look into the technical 
detail of the existing code.






From roland at topspin.com  Mon Feb 28 11:40:28 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 11:40:28 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <007801c51dcc$2528ec60$8000000a@blorp> (Paul Baxter's message
	of "Mon, 28 Feb 2005 19:31:47 -0000")
References: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>
	<52y8d8sfls.fsf@topspin.com> <007801c51dcc$2528ec60$8000000a@blorp>
Message-ID: <52hdjwscb7.fsf@topspin.com>

    Paul> Would this proposed address translation layer interface
    Paul> provide a sufficient subset of IPoIB functionality that
    Paul> allowed an SDP implementation to exist without a larger,
    Paul> full-featured IPoIB implementation?

I doubt it.  Right now the IPoIB driver has very little code related
to address translation.  It relies on the kernel's ARP layer to get it
remote addresses (GID + QPN) and then calls ib_sa_path_rec() to get an
actual IB path.  The second half of this translation is not even
really needed by SDP.  If you look at sdp_link.c in Libor's SDP
implementation you can see how this can work in a few hundred lines of
code.

Essentially all of the address translation used by SDP is done by the
kernel's ARP code, with the IPoIB driver serving to send and receive
ARP packets.

Presumably this proposed address translation library would also sit on
top of the kernel's IP stack and issue ARP requests as needed, so it
would still depend on the IPoIB driver.

 - Roland


From yaronh at voltaire.com  Mon Feb 28 11:47:29 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Mon, 28 Feb 2005 21:47:29 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC63@taurus.voltaire.com>

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Paul Baxter
> Sent: Monday, February 28, 2005 9:32 PM
> To: openib-general at openib.org
> Subject: Re: [openib-general] IB Address Translation service
> 
> Having now just read Yaron's reply, I am even more convinced that this
is
> the right way to go albeit I can't comment on the API etc (Could
someone
> explain the differences in using ARP and ATS. )

Paul,

ATS (Address Resolution Service) is based on each node registering a
service record in the SM/SA with GID&P_Key=IP address.
When you want to map an IP address to IB address it issues an SA query
to the SM/SA with an IP that results in GID+P_Key values than can be
used by the ULP.

ATS is a standard defined by DAT and recently also by ICSC.

As I mentioned in the IP to GID API you can specify if to resolve based
on the IP infrastructure (like the one Libor described), or based on
ATS, or Default (first try IP/ARP, than ATS).

And as you mentioned there is value to have the same API for different
resolution mechanisms, the SDP code can be altered in future to ride
over the proposed API, so it can be used without TCP/IP.

Yaron


From libor at topspin.com  Mon Feb 28 11:48:37 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 11:48:37 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <007801c51dcc$2528ec60$8000000a@blorp>;
	from paul.baxter@dsl.pipex.com on Mon, Feb 28, 2005 at 07:31:47PM
	-0000
References: <D4F8F0B3820E754C887699BEF26A89405A9D66@taurus.voltaire.com>
	<52y8d8sfls.fsf@topspin.com> <007801c51dcc$2528ec60$8000000a@blorp>
Message-ID: <20050228114837.C21494@topspin.com>

On Mon, Feb 28, 2005 at 07:31:47PM -0000, Paul Baxter wrote:
> >    Roland> First, let's understand the problem we're trying to solve.
> >    Roland> Who are the consumers of this address translation service?
> >
> >    shaharf> Any ULPs at user & kernel, and also some
> >    shaharf> applications.
> >
> > I think this is too general an answer.  We should be designing based
> > on specific ULPs and applications.  For example, I don't see anything
> > particularly useful to IPoIB in this API.  Perhaps Libor can comment
> > on how this API works for SDP.
> >
> 
> Recently a project I've been working on has been looking at porting IB 
> software to an embedded PowerPC though my question also applies to a slimmed 
> down embedded Linux PC system.
> 
> If I wanted to support SDP over a link with one end being embedded, I had 
> thought I'd need to port the whole of the IPoIB driver for address 
> resolution (albeit I may be able to cut a few corners with the code.)
> 
> Would this proposed address translation layer interface provide a sufficient 
> subset of IPoIB functionality that allowed an SDP implementation to exist 
> without a larger, full-featured IPoIB implementation?

  No, you would still need IPoIB. In my opinion, the subset of IPoIB that
you need to support ARP is big enough, that it doesn't make sense to try 
and break out that subset.

> Having now just read Yaron's reply, I am even more convinced that this is 
> the right way to go albeit I can't comment on the API etc (Could someone 
> explain the differences in using ARP and ATS. )

  Well to start with, ARP is based on an IETF draft and ATS is from the
DAT collabrative. ATS uses a 32 bit address, but calling it an IPv4
address is a bit of a strech. Basically ATS uses the IB SM to register
node addresses and support lookup of those nodes GIDs based on the 32
bit registered address. The intention was to support IP like addressing
without an IPoIB implementation. The two are not interoperable, they 
reside in parallel, and succeed in producing much confusion. (IMO)

-Libor


From mst at mellanox.co.il  Mon Feb 28 11:55:01 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 21:55:01 +0200
Subject: [openib-general] Re: [PATCH][uverbs] use get_task_mm
In-Reply-To: <52ekf0tx20.fsf@topspin.com>
References: <20050228093520.GU23837@mellanox.co.il>
	<52is4ctx77.fsf@topspin.com> <52ekf0tx20.fsf@topspin.com>
Message-ID: <20050228195501.GA500@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: [PATCH][uverbs] use get_task_mm
> 
>     Roland> I guess this patch can't hurt but can the mm really go
>     Roland> away?  The only time we will call ib_umem_get() is during
>     Roland> a write system call (which requires an open file,
>     Roland> obviously).  Can current->mm go away before all files are
>     Roland> closed?
> 
> I didn't phrase this precisely as I meant.  Can current->mm go away
> during a system call for current's task?
> 
>  - R.

But you are right : it cant - although
the files are closed *after* the mm is destroyed, I think its safe to
use current->mm in most sys calls (well, except sys_exit and such I
guess) as long as you dont store this pointer in some data structure
which may survive after the syscall returns.

That patch seemed to help for a bit (still crashing, but in a different
place) with the crashes I reported earlier, but I guess it must have
triggered off some other problem.

-- 
MST - Michael S. Tsirkin


From yaronh at voltaire.com  Mon Feb 28 11:55:50 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Mon, 28 Feb 2005 21:55:50 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC64@taurus.voltaire.com>

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Libor Michalek
> Sent: Monday, February 28, 2005 9:49 PM
> To: Paul Baxter
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] IB Address Translation service
> 
> The two are not interoperable, they
> reside in parallel, and succeed in producing much confusion. (IMO)

One note, the two can be made interoperable, if nodes that use IPoIB
register them self in the ATS database as well (which has its merits for
reverse resolution that cannot be satisfied by IPoIB), this way the
nodes that just use ATS can locate the IPoIB ones.

That is how it works (successfully) in the Voltaire gen1 stack

Anyway the merits if the proposed API goes much beyond the use of ATS,
so I hope we don't just hang on that one.

Yaron


From mst at mellanox.co.il  Mon Feb 28 11:56:35 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 21:56:35 +0200
Subject: [openib-general] Re: ANNOUNCE: First usable version of userspace
	verbs
In-Reply-To: <52acpotwd9.fsf@topspin.com>
References: <52u0o4pfe8.fsf@topspin.com>
	<20050228154254.GB31510@mellanox.co.il>
	<52acpotwd9.fsf@topspin.com>
Message-ID: <20050228195635.GB500@mellanox.co.il>

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
> 
>     Michael> Interestingly, if I rebuild libmthca with -O3 compiler
>     Michael> flag, the pingpong program does not make progress.
>     Michael> Building libibverbs or the test itself with -O3 has no
>     Michael> such effect.
> 
> I can't reproduce this on either i386 or x86-64 (Intel Nocona system).
> 
>     $ gcc --version
>     gcc (GCC) 3.3.5 (Debian 1:3.3.5-8)
>     Copyright (C) 2003 Free Software Foundation, Inc.
>     This is free software; see the source for copying conditions.  There is NO
>     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
>  - R.
> 

I'll try upgrading gcc.

-- 
MST - Michael S. Tsirkin


From jlentini at netapp.com  Mon Feb 28 12:33:59 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 28 Feb 2005 15:33:59 -0500 (EST)
Subject: [openib-general] Re: Porting DAPL to OpenIB Gen 2
In-Reply-To: <Pine.LNX.4.61.0502281025340.5704@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
	<Pine.LNX.4.61.0502281025340.5704@jlentini-linux.nane.netapp.com>
Message-ID: <Pine.LNX.4.61.0502281532560.5704@jlentini-linux.nane.netapp.com>


Below are my meeting notes:

Porting DAPL to OpenIB
======================
Meeting Notes
Date: 2/28/2005
Time: 11:30 AM EST

Attendees:
James Lentini       jlentini at netapp.com
Dror Goldenberg     gdror at mellanox.co.il
Itamar Rabenstein   itamar at mellanox.co.il
Arkady Kanevsky     arkady at netapp.com
Hal Rosenstock      halr at voltaire.com
Aviram Gutman       aviram at mellanox.co.il
Arlin Davis         arlin.r.davis at intel.com
Tom Duffy           tduffy at sun.com
Yaron Haviv         yaronh at voltaire.com
Or Gerlitz          ogerlitz at voltaire.com
Steve Sears         sjs at netapp.com
Bob Woodruff        robert.j.woodruff at intel.com 
Tom Talpey          talpey at netapp.com

+ Enumerate Tasks:
   - uDAPL
     who: Arlin Davis at Intel

     No additional volunteers.

     We agreed to keep SourceForge structure (provider and platform
     abstraction layers) for uDAPL. Openib user space software lacks
     a CM and name resolution APIs. uDAPL will need to work around
     these deficiencies (by hard coding connections, etc.) at first.

     uDAPL will require kernel support for mapping the cookies used
     by shared registrations.

   - kDAPL
     who: Itamar Rabenstein at Mellanox
          Hal Rosenstock at Voltaire

    Initially we will work with current verbs abstraction layer. Once
    OpenIB support is functional and stable, we will begin removing
    the verbs and platform abstraction layers.

    MTHCA does not support all of DAPL required features (memory
    windows, virtual memory registration, etc.). When will these
    be supported?

    [AI] James Lentini: inquire about above

   - name resolution interface
     who: ?

     The kernel APIs are evolving. User will be done second.

   - Linux coding style
     who: everyone

     Documentation is in every linux kernel in
     "Documentation/CodingStyle" and "Documentation/SubmittingPatches"
     Tom Duffy gave James Lentini an overview of the types of changes
     at the OpenIB workshop. No apparent interest in making a single
     individual responsible for this. We expect to receive feedback
     from the community once code is in OpenIB tree.

   - kDAPL API changes for Linux coding style
     who: James Lentini and Arkady Kanevsky at NetApp

     First step is changing header files. We would like to keep
     old code from breaking while updates are being made. One option is
     to use a temporary set of typedefs while the headers are in flux.

     [AI] James Lentini: send Tom Duffy's patches to attendees

   - tools (dapltest, dat conformance test, etc.)
     who: volunteers?

     DAT Conformance test needs recode. No volunteers to do this work.

   - documentation (user guide, design guides, etc.)
     who: volunteers? everyone?

     We only touched on this briefly as time was running out. James
     would like written record of design decisions for posterity.

+ Appropriate location for DAPL source code in OpenIB tree.
   Split kernel and user space code? Split registry and provider?

   Do we want to split kernel and user space code? Majority are in
   favor of split. License issues are simpler (i.e. if the OpenIB
   community ever wanted to add the LGPL for user components this would be
   feasible). Intel's experience with past Linux projects suggest that
   this is the correct decision. Keeping them together would mean lots
   of kernel changes affect user code.

   We will keep the registry code in the same general location as the
   provider code. If other RDMA transports pop up, we will adjust this.

   Initially, we will put DAPL code on a branch and move to mainline
   when it is ready. Initial OpenIB checkin will be based on the soon
   to be created DAPL Gamma 3.1 drop (this drop will be Gamma 3.0 +
   a few fixes + GPL option).

   [AI] James Lentini: create tar of current CVS and send to Tom,
        Arlin, Itamar, and Hal. They will get started with this while
        Gamma 3.1 is being prepared.

+ Procedure for checkins

Use the same procedure for other parts of the OpenIB tree. Submitter
will send an email containing patch to openib-general at openib.org and 
cc'ing James Lenitni (jlentini at netapp.com). Submitter should indicate 
that it is for dapl by placing the text "[DAPL]" in the subject.

NOTE: We didn't discuss this on the phone, but checkins should also
have a "signed-off by" line (in accordance with OpenIB practices).

+ SourceForge synchronization

Proposal would be to have patches sent to both OpenIB and SourceForge.
The files in OpenIB will have 2 licenses and the SourceForge will have 
3 licenses.

Little enthusiasm for this on the phone. Two counter proposals
made:

1) submit code with CPL license 
2) re-license code with CPL

[AI] Tom Duffy: At the next OpenIB board meeting, ask if it is a
      violation of OpenIB bylaws to have dapl files licensed under
      a third license (CPL)? 
[AI] James Lentini: try to attend board meeting to answer questions.
[AI] James Lentini: investigate option #2 with lawyers. Note: early
      indications are that this is not feasible.



On Mon, 28 Feb 2005, James Lentini wrote:

>
> Below is an agenda for the call:
>
> + Additional Items?
>
> + Enumerate Tasks:
> - uDAPL
>   who: Arlin Davis at Intel
>
>  - kDAPL
>    who: Itamar Rabenstein at Mellanox
>         Voltaire for CM related features
>
>  - Linux coding style
>    who: everyone
>
>  - kDAPL API changes for Linux coding style
>    who: James Lentini and Arkady Kanevsky at NetApp
>
>  - tools (dapltest, dat conformance test, etc.)
>    who: volunteers?
>
>  - documentation (user guide, design guides, etc.)
>    who: volunteers? everyone?
>
> + Appropriate location for DAPL source code in OpenIB tree.
>  Split kernel and user space code? Split registry and provider?
>
> + Procedure for checkins
>
> + SourceForge synchronization
>
>
> On Thu, 24 Feb 2005, James Lentini wrote:
>
>> 
>> Yesterday the DAT Collaborative voted to add the GPL license to the DAPL 
>> Source Forge reference implementation. We are now in a position to begin 
>> porting the DAPL SourceForge project to OpenIB. I could like to hold a 
>> conference call to help plan and divide up this work.
>> 
>> 
>>         Date: Monday, February 28
>>         Time: 11:30 AM EST
>>     Domestic: 888-827-8686
>> International: 303-928-2620
>> Conference ID: 1125043
>> 
>> 
>> James Lentini                email: jlentini at netapp.com
>> Network Appliance            phone: 781-768-5359
>> 375 Totten Pond Rd.            fax: 781-895-1195
>> Waltham, MA 02451-2010        main: 781-768-5300
>> 
>> 
>


From jlentini at netapp.com  Mon Feb 28 12:42:30 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 28 Feb 2005 15:42:30 -0500 (EST)
Subject: [openib-general] MTHCA features
Message-ID: <Pine.LNX.4.61.0502281541490.5704@jlentini-linux.nane.netapp.com>


It is my understanding that the current MTHCA driver does not support
InfiniBand memory windows or memory registration using virtual 
addresses.

Is this information correct? If so, when will these features be 
supported?

Thanks,
-james

James Lentini                email: jlentini at netapp.com
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300


From jlentini at netapp.com  Mon Feb 28 12:48:35 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 28 Feb 2005 15:48:35 -0500 (EST)
Subject: [openib-general] Re: Porting DAPL to OpenIB Gen 2
In-Reply-To: <Pine.LNX.4.61.0502281532560.5704@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0502241615330.5704@jlentini-linux.nane.netapp.com>
	<Pine.LNX.4.61.0502281025340.5704@jlentini-linux.nane.netapp.com>
	<Pine.LNX.4.61.0502281532560.5704@jlentini-linux.nane.netapp.com>
Message-ID: <Pine.LNX.4.61.0502281548120.5704@jlentini-linux.nane.netapp.com>


As promised during the call today, here is a link to the patches Tom
Duffy sent:

http://sourceforge.net/mailarchive/forum.php?thread_id=6549863&forum_id=13070


On Mon, 28 Feb 2005, James Lentini wrote:

>
> Below are my meeting notes:
>
> Porting DAPL to OpenIB
> ======================
> Meeting Notes
> Date: 2/28/2005
> Time: 11:30 AM EST
>
> Attendees:
> James Lentini       jlentini at netapp.com
> Dror Goldenberg     gdror at mellanox.co.il
> Itamar Rabenstein   itamar at mellanox.co.il
> Arkady Kanevsky     arkady at netapp.com
> Hal Rosenstock      halr at voltaire.com
> Aviram Gutman       aviram at mellanox.co.il
> Arlin Davis         arlin.r.davis at intel.com
> Tom Duffy           tduffy at sun.com
> Yaron Haviv         yaronh at voltaire.com
> Or Gerlitz          ogerlitz at voltaire.com
> Steve Sears         sjs at netapp.com
> Bob Woodruff        robert.j.woodruff at intel.com Tom Talpey 
> talpey at netapp.com
>
> + Enumerate Tasks:
>  - uDAPL
>    who: Arlin Davis at Intel
>
>    No additional volunteers.
>
>    We agreed to keep SourceForge structure (provider and platform
>    abstraction layers) for uDAPL. Openib user space software lacks
>    a CM and name resolution APIs. uDAPL will need to work around
>    these deficiencies (by hard coding connections, etc.) at first.
>
>    uDAPL will require kernel support for mapping the cookies used
>    by shared registrations.
>
>  - kDAPL
>    who: Itamar Rabenstein at Mellanox
>         Hal Rosenstock at Voltaire
>
>   Initially we will work with current verbs abstraction layer. Once
>   OpenIB support is functional and stable, we will begin removing
>   the verbs and platform abstraction layers.
>
>   MTHCA does not support all of DAPL required features (memory
>   windows, virtual memory registration, etc.). When will these
>   be supported?
>
>   [AI] James Lentini: inquire about above
>
>  - name resolution interface
>    who: ?
>
>    The kernel APIs are evolving. User will be done second.
>
>  - Linux coding style
>    who: everyone
>
>    Documentation is in every linux kernel in
>    "Documentation/CodingStyle" and "Documentation/SubmittingPatches"
>    Tom Duffy gave James Lentini an overview of the types of changes
>    at the OpenIB workshop. No apparent interest in making a single
>    individual responsible for this. We expect to receive feedback
>    from the community once code is in OpenIB tree.
>
>  - kDAPL API changes for Linux coding style
>    who: James Lentini and Arkady Kanevsky at NetApp
>
>    First step is changing header files. We would like to keep
>    old code from breaking while updates are being made. One option is
>    to use a temporary set of typedefs while the headers are in flux.
>
>    [AI] James Lentini: send Tom Duffy's patches to attendees
>
>  - tools (dapltest, dat conformance test, etc.)
>    who: volunteers?
>
>    DAT Conformance test needs recode. No volunteers to do this work.
>
>  - documentation (user guide, design guides, etc.)
>    who: volunteers? everyone?
>
>    We only touched on this briefly as time was running out. James
>    would like written record of design decisions for posterity.
>
> + Appropriate location for DAPL source code in OpenIB tree.
>  Split kernel and user space code? Split registry and provider?
>
>  Do we want to split kernel and user space code? Majority are in
>  favor of split. License issues are simpler (i.e. if the OpenIB
>  community ever wanted to add the LGPL for user components this would be
>  feasible). Intel's experience with past Linux projects suggest that
>  this is the correct decision. Keeping them together would mean lots
>  of kernel changes affect user code.
>
>  We will keep the registry code in the same general location as the
>  provider code. If other RDMA transports pop up, we will adjust this.
>
>  Initially, we will put DAPL code on a branch and move to mainline
>  when it is ready. Initial OpenIB checkin will be based on the soon
>  to be created DAPL Gamma 3.1 drop (this drop will be Gamma 3.0 +
>  a few fixes + GPL option).
>
>  [AI] James Lentini: create tar of current CVS and send to Tom,
>       Arlin, Itamar, and Hal. They will get started with this while
>       Gamma 3.1 is being prepared.
>
> + Procedure for checkins
>
> Use the same procedure for other parts of the OpenIB tree. Submitter
> will send an email containing patch to openib-general at openib.org and cc'ing 
> James Lenitni (jlentini at netapp.com). Submitter should indicate that it is for 
> dapl by placing the text "[DAPL]" in the subject.
>
> NOTE: We didn't discuss this on the phone, but checkins should also
> have a "signed-off by" line (in accordance with OpenIB practices).
>
> + SourceForge synchronization
>
> Proposal would be to have patches sent to both OpenIB and SourceForge.
> The files in OpenIB will have 2 licenses and the SourceForge will have 3 
> licenses.
>
> Little enthusiasm for this on the phone. Two counter proposals
> made:
>
> 1) submit code with CPL license 2) re-license code with CPL
>
> [AI] Tom Duffy: At the next OpenIB board meeting, ask if it is a
>     violation of OpenIB bylaws to have dapl files licensed under
>     a third license (CPL)? [AI] James Lentini: try to attend board meeting 
> to answer questions.
> [AI] James Lentini: investigate option #2 with lawyers. Note: early
>     indications are that this is not feasible.
>
>
>
> On Mon, 28 Feb 2005, James Lentini wrote:
>
>> 
>> Below is an agenda for the call:
>> 
>> + Additional Items?
>> 
>> + Enumerate Tasks:
>> - uDAPL
>>   who: Arlin Davis at Intel
>> 
>>  - kDAPL
>>    who: Itamar Rabenstein at Mellanox
>>         Voltaire for CM related features
>> 
>>  - Linux coding style
>>    who: everyone
>> 
>>  - kDAPL API changes for Linux coding style
>>    who: James Lentini and Arkady Kanevsky at NetApp
>> 
>>  - tools (dapltest, dat conformance test, etc.)
>>    who: volunteers?
>> 
>>  - documentation (user guide, design guides, etc.)
>>    who: volunteers? everyone?
>> 
>> + Appropriate location for DAPL source code in OpenIB tree.
>>  Split kernel and user space code? Split registry and provider?
>> 
>> + Procedure for checkins
>> 
>> + SourceForge synchronization
>> 
>> 
>> On Thu, 24 Feb 2005, James Lentini wrote:
>> 
>>> 
>>> Yesterday the DAT Collaborative voted to add the GPL license to the DAPL 
>>> Source Forge reference implementation. We are now in a position to begin 
>>> porting the DAPL SourceForge project to OpenIB. I could like to hold a 
>>> conference call to help plan and divide up this work.
>>> 
>>> 
>>>         Date: Monday, February 28
>>>         Time: 11:30 AM EST
>>>     Domestic: 888-827-8686
>>> International: 303-928-2620
>>> Conference ID: 1125043
>>> 
>>> 
>>> James Lentini                email: jlentini at netapp.com
>>> Network Appliance            phone: 781-768-5359
>>> 375 Totten Pond Rd.            fax: 781-895-1195
>>> Waltham, MA 02451-2010        main: 781-768-5300
>>> 
>>> 
>> 
>


From roland at topspin.com  Mon Feb 28 12:54:22 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 12:54:22 -0800
Subject: [openib-general] MTHCA features
In-Reply-To: <Pine.LNX.4.61.0502281541490.5704@jlentini-linux.nane.netapp.com>
	(James
	Lentini's message of "Mon, 28 Feb 2005 15:42:30 -0500 (EST)")
References: <Pine.LNX.4.61.0502281541490.5704@jlentini-linux.nane.netapp.com>
Message-ID: <52bra4s8w1.fsf@topspin.com>

    James> It is my understanding that the current MTHCA driver does
    James> not support InfiniBand memory windows or memory
    James> registration using virtual addresses.

    James> Is this information correct? If so, when will these
    James> features be supported?

Well, memory registration is pretty complete.  By design, we only
support memory registration with physical addresses for kernel
consumers even at the verbs API level (ie there are no mthca-specific
limitations).  In the kernel, registration by virtual address is not
very useful.  For userspace verbs, only registration by virtual
address is supported for obvious reasons.

Memory windows are not implemented for mthca.  It wouldn't be a lot of
work for someone with access to Mellanox documentation to implement
them, but they're not particularly useful due to their performance
characteristics.

 - R.


From libor at topspin.com  Mon Feb 28 13:13:45 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 13:13:45 -0800
Subject: [openib-general] Re: [PATCH] sdp conditional code cleanup
In-Reply-To: <20050227104441.GI23837@mellanox.co.il>;
	from mst@mellanox.co.il on Sun, Feb 27, 2005 at 12:44:41PM +0200
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
	<20050225115857.C13364@topspin.com>
	<20050227104441.GI23837@mellanox.co.il>
Message-ID: <20050228131345.D21494@topspin.com>

On Sun, Feb 27, 2005 at 12:44:41PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > On Thu, Feb 24, 2005 at 11:49:28PM +0200, Michael S. Tsirkin wrote:
> > > OK, now what about things like these:
> > > 
> > >         if (0 > result) {
> > > 
> > > may I change them all to
> > > 
> > >         if (result < 0) {
> > > 
> > > While being equivalent, we are testing the result, not 0.
> > > 
> > > Similiarly (although I feel somewhat less strongly about it)
> > > 
> > >         if (0 == result)
> > > and
> > >         if (NULL == conn)
> > > 
> > > are better off as
> > > 
> > > 	if (!result) {
> > > and
> > >         if (!conn)
> > > 
> > > C is a Spartan language, and this is more brief.
> > > Libor, I think I asked about the second one, but dont recall you
> > > answering.
> > > If OK to both, let me know and I'll do it on Sunday.
> > 
> >   I actually feel more strongly in favour of making the second change
> > you propose then the first. However, I'm OK with both, so feel free
> > to submit a patch.
> 
> Here is the patch.
> 
> I generalized the approach - 
> any if (CONSTANT == variable) and if ( CONSTANT & variable) is now
>     if (variable == CONSTANT) and if ( variable & CONSTANT)
>     and so forth.
> 
> Further, some places had weird flag testing code like this:
> if ( (variable & CONSTANT) > 0 )
> 
> I changed them all to 
> 
> if (variable & CONSTANT)
> 
> Two things I noticed but did not fix:
> 
> A. In some places a positive error code is returned. For example
>    ENOBUF and not -ENOBUF. I assume its a bug but did not touch it.

  I'm glad you did not change this, it's not a bug. In parts of both
the send and recv data paths positive and negative returns have different
meaning. negative return is a hard error, while positive return is used
to indicate that the data path is in a flow control situation and that
the action will need to be retried later. Zero as usual means success.

> Looks like a lot of changes, I went over them several times and
> everything looks in order to me. I really hope its applied before
> any other change, it almost surely will conflict with any other
> patch, and it'll be a lot of work to re-diff.

  I found a few problem spots that I fixed up, before commiting
the code. The third problem was in an #if 0, so it wasn't an
immediate issue. Thanks.

-Libor

@@ -1302,7 +1301,7 @@ static int _sdp_inet_shutdown(struct soc
         *       1 - send shutdown
         *       2 - send/recv shutdown.
         */
-       if (0 > flag || 2 < flag)
+       if (flag < 0 || flag < 2)
                return -EINVAL;

@@ -787,7 +787,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
             counter < total; counter++) {
                next = iocb->next;
 
-               if (0 < (mask & iocb->flags) || SDP_IOCB_F_ALL == mask) {
+               if ((mask & iocb->flags) || msk == SDP_IOCB_F_ALL) {
                        sdp_dbg_err("IOCB <%d> cancel <%Zu> flag <%04x> "
                                    "size <%Zu:%d:%d>",
                                    iocb->key, comp, iocb->flags, iocb->size,

@@ -1092,35 +1092,35 @@ static int _sdp_inet_recv_urg(struct soc
 
        conn = SDP_GET_CONN(sk);
 
-       if (sock_flag(sk, SOCK_URGINLINE) || 0 == conn->rcv_urg_cnt)
+       if (sock_flag(sk, SOCK_URGINLINE) || !conn->rcv_urg_cnt)
                return -EINVAL;
 
        /*
         * don't cosume data on PEEK, but do consume data on TRUNC
         */
 #if 0
-       value = (0 < (MSG_PEEK & flags)) || (0 == size) ? 0 : 1;
+       value = (MSG_PEEK & flags) || size ? 1 : 0;
 #else
-       value = (0 < (MSG_PEEK & flags)) ? 0 : 1;
+       value = (MSG_PEEK & flags) ? 0 : 1;
 #endif



From johannes at erdfelt.com  Mon Feb 28 13:27:46 2005
From: johannes at erdfelt.com (Johannes Erdfelt)
Date: Mon, 28 Feb 2005 13:27:46 -0800
Subject: [openib-general] Re: [PATCH] sdp conditional code cleanup
In-Reply-To: <20050228131345.D21494@topspin.com>
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
	<20050225115857.C13364@topspin.com>
	<20050227104441.GI23837@mellanox.co.il>
	<20050228131345.D21494@topspin.com>
Message-ID: <20050228212746.GB18859@sventech.com>

On Mon, Feb 28, 2005, Libor Michalek <libor at topspin.com> wrote:
> @@ -1302,7 +1301,7 @@ static int _sdp_inet_shutdown(struct soc
>          *       1 - send shutdown
>          *       2 - send/recv shutdown.
>          */
> -       if (0 > flag || 2 < flag)
> +       if (flag < 0 || flag < 2)
>                 return -EINVAL;

Are you fixing or introducing a bug here? I'm guessing introducing since
the first part of that statement is superfluous now :)

JE



From mst at mellanox.co.il  Mon Feb 28 13:34:06 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 28 Feb 2005 23:34:06 +0200
Subject: [openib-general] Re: [PATCH] sdp conditional code cleanup
In-Reply-To: <20050228131345.D21494@topspin.com>
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
	<20050225115857.C13364@topspin.com>
	<20050227104441.GI23837@mellanox.co.il>
	<20050228131345.D21494@topspin.com>
Message-ID: <20050228213406.GA1032@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [PATCH] sdp conditional code cleanup
> 
> On Sun, Feb 27, 2005 at 12:44:41PM +0200, Michael S. Tsirkin wrote:
> > Quoting r. Libor Michalek <libor at topspin.com>:
> > > On Thu, Feb 24, 2005 at 11:49:28PM +0200, Michael S. Tsirkin wrote:
> > > > OK, now what about things like these:
> > > > 
> > > >         if (0 > result) {
> > > > 
> > > > may I change them all to
> > > > 
> > > >         if (result < 0) {
> > > > 
> > > > While being equivalent, we are testing the result, not 0.
> > > > 
> > > > Similiarly (although I feel somewhat less strongly about it)
> > > > 
> > > >         if (0 == result)
> > > > and
> > > >         if (NULL == conn)
> > > > 
> > > > are better off as
> > > > 
> > > > 	if (!result) {
> > > > and
> > > >         if (!conn)
> > > > 
> > > > C is a Spartan language, and this is more brief.
> > > > Libor, I think I asked about the second one, but dont recall you
> > > > answering.
> > > > If OK to both, let me know and I'll do it on Sunday.
> > > 
> > >   I actually feel more strongly in favour of making the second change
> > > you propose then the first. However, I'm OK with both, so feel free
> > > to submit a patch.
> > 
> > Here is the patch.
> > 
> > I generalized the approach - 
> > any if (CONSTANT == variable) and if ( CONSTANT & variable) is now
> >     if (variable == CONSTANT) and if ( variable & CONSTANT)
> >     and so forth.
> > 
> > Further, some places had weird flag testing code like this:
> > if ( (variable & CONSTANT) > 0 )
> > 
> > I changed them all to 
> > 
> > if (variable & CONSTANT)
> > 
> > Two things I noticed but did not fix:
> > 
> > A. In some places a positive error code is returned. For example
> >    ENOBUF and not -ENOBUF. I assume its a bug but did not touch it.
> 
>   I'm glad you did not change this, it's not a bug. In parts of both
> the send and recv data paths positive and negative returns have different
> meaning. negative return is a hard error, while positive return is used
> to indicate that the data path is in a flow control situation and that
> the action will need to be retried later. Zero as usual means success.
> 
> > Looks like a lot of changes, I went over them several times and
> > everything looks in order to me. I really hope its applied before
> > any other change, it almost surely will conflict with any other
> > patch, and it'll be a lot of work to re-diff.
> 
>   I found a few problem spots that I fixed up, before commiting
> the code. The third problem was in an #if 0, so it wasn't an
> immediate issue. Thanks.
> 
> -Libor
> 

Here is a small number of additional swaps I missed on the first pass.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: sdp_inet.c
===================================================================
--- sdp_inet.c	(revision 1929)
+++ sdp_inet.c	(working copy)
@@ -377,7 +377,7 @@ static int _sdp_inet_release(struct sock
 	 * Skip lingering/canceling if 
 	 * non-blocking and not exiting.
 	 */
-	if (!(MSG_DONTWAIT & flags) ||
+	if (!(flags & MSG_DONTWAIT) ||
 	    (PF_EXITING & current->flags)) {
 		/*
 		 * Wait if linger is set and
Index: sdp_send.c
===================================================================
--- sdp_send.c	(revision 1929)
+++ sdp_send.c	(working copy)
@@ -2044,7 +2044,7 @@ int sdp_inet_send(struct kiocb *req,
 	/*
 	 * set oob flag.
 	 */
-	oob = (MSG_OOB & msg->msg_flags);
+	oob = (msg->msg_flags & MSG_OOB);
 
 	sk = sock->sk;
 	conn = SDP_GET_CONN(sk);
@@ -2245,7 +2245,7 @@ done:
 	sdp_conn_unlock(conn);
 	result = ((copied > 0) ? copied : result);
 
-	if (result == -EPIPE && !(MSG_NOSIGNAL & msg->msg_flags))
+	if (result == -EPIPE && !(msg->msg_flags & MSG_NOSIGNAL))
 		send_sig(SIGPIPE, current, 0);
 
 	return result;
Index: sdp_recv.c
===================================================================
--- sdp_recv.c	(revision 1929)
+++ sdp_recv.c	(working copy)
@@ -1099,9 +1099,9 @@ static int _sdp_inet_recv_urg(struct soc
 	 * don't cosume data on PEEK, but do consume data on TRUNC
 	 */
 #if 0
-	value = (MSG_PEEK & flags) || !size ? 0 : 1;
+	value = (flags & MSG_PEEK) || !size ? 0 : 1;
 #else
-	value = (MSG_PEEK & flags) ? 0 : 1;
+	value = (flags & MSG_PEEK) ? 0 : 1;
 #endif
 
 	result = sdp_buff_q_trav_head(&conn->recv_pool,
@@ -1120,7 +1120,7 @@ static int _sdp_inet_recv_urg(struct soc
 		/*
 		 * clear urgent pointer on consumption
 		 */
-		if (!(MSG_PEEK & flags)) {
+		if (!(flags & MSG_PEEK)) {
 			conn->rcv_urg_cnt -= 1;
 			conn->byte_strm -= 1;
 			
@@ -1191,10 +1191,10 @@ int sdp_inet_recv(struct kiocb  *req,
 	/*
 	 * TODO: unhandled, but need to be handled.
 	 */
-	if (MSG_TRUNC & flags)
+	if (flags & MSG_TRUNC)
 		return -EOPNOTSUPP;
 
-	if (MSG_PEEK & flags) {
+	if (flags & MSG_PEEK) {
 		sdp_buff_q_init(&peek_queue);
 		msg->msg_flags |= MSG_PEEK;
 	}
@@ -1209,7 +1209,7 @@ int sdp_inet_recv(struct kiocb  *req,
 	/*
 	 * process urgent data
 	 */
-	if (MSG_OOB & flags) {
+	if (flags & MSG_OOB) {
 		result = _sdp_inet_recv_urg(sk, msg, size, flags);
 		copied = (result > 0) ? result : 0;
 		result = (result > 0) ? 0 : result;
@@ -1218,8 +1218,8 @@ int sdp_inet_recv(struct kiocb  *req,
 	/*
 	 * get socket values we'll need.
 	 */
-	timeout   = sock_rcvtimeo(sk, (MSG_DONTWAIT & flags));
-	low_water = sock_rcvlowat(sk, (MSG_WAITALL & flags), size);
+	timeout   = sock_rcvtimeo(sk, (flags & MSG_DONTWAIT));
+	low_water = sock_rcvlowat(sk, (flags & MSG_WAITALL), size);
 	/*
 	 * process data first, and then check condition, then wait
 	 */
@@ -1266,7 +1266,7 @@ int sdp_inet_recv(struct kiocb  *req,
 							length = 0;
 							update =
 							    (0 <
-							     (MSG_PEEK & flags))
+							     (flags & MSG_PEEK))
 							    ? 0 : 1;
 						}
 					}
@@ -1290,7 +1290,7 @@ int sdp_inet_recv(struct kiocb  *req,
 					goto done;
 				}
 #endif
-				update = (MSG_PEEK & flags) ? 0 : copy;
+				update = (flags & MSG_PEEK) ? 0 : copy;
 			}
 
 			SDP_CONN_STAT_RECV_INC(conn, update);
@@ -1314,7 +1314,7 @@ int sdp_inet_recv(struct kiocb  *req,
 				break;
 			}
 
-			if (MSG_PEEK & flags) {
+			if (flags & MSG_PEEK) {
 				expect = sdp_buff_q_put_head(&peek_queue,
 							     buff);
 				SDP_EXPECT(expect >= 0);
@@ -1514,7 +1514,7 @@ done:
 	/*
 	 * return any peeked buffers to the recv queue, in the correct order.
 	 */
-	if (MSG_PEEK & flags) {
+	if (flags & MSG_PEEK) {
 		while ((buff = sdp_buff_q_get_tail(&peek_queue))) {
 			expect = sdp_buff_q_put_head(&conn->recv_pool, buff);
 			SDP_EXPECT(expect >= 0);
Index: sdp_iocb.c
===================================================================
--- sdp_iocb.c	(revision 1929)
+++ sdp_iocb.c	(working copy)
@@ -787,7 +787,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
 	     counter < total; counter++) {
 		next = iocb->next;
 
-		if ((mask & iocb->flags) || mask == SDP_IOCB_F_ALL) {
+		if ((iocb->flags & mask) || mask == SDP_IOCB_F_ALL) {
 			sdp_dbg_err("IOCB <%d> cancel <%Zu> flag <%04x> "
 				    "size <%Zu:%d:%d>",
 				    iocb->key, comp, iocb->flags, iocb->size,

-- 
MST - Michael S. Tsirkin


From libor at topspin.com  Mon Feb 28 13:34:47 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 13:34:47 -0800
Subject: [openib-general] Re: [PATCH] sdp conditional code cleanup
In-Reply-To: <20050228212746.GB18859@sventech.com>;
	from johannes@erdfelt.com on Mon, Feb 28, 2005 at 01:27:46PM -0800
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
	<20050225115857.C13364@topspin.com>
	<20050227104441.GI23837@mellanox.co.il>
	<20050228131345.D21494@topspin.com>
	<20050228212746.GB18859@sventech.com>
Message-ID: <20050228133447.E21494@topspin.com>

On Mon, Feb 28, 2005 at 01:27:46PM -0800, Johannes Erdfelt wrote:
> On Mon, Feb 28, 2005, Libor Michalek <libor at topspin.com> wrote:
> > @@ -1302,7 +1301,7 @@ static int _sdp_inet_shutdown(struct soc
> >          *       1 - send shutdown
> >          *       2 - send/recv shutdown.
> >          */
> > -       if (0 > flag || 2 < flag)
> > +       if (flag < 0 || flag < 2)
> >                 return -EINVAL;
> 
> Are you fixing or introducing a bug here? I'm guessing introducing since
> the first part of that statement is superfluous now :)

  The three patches I included in my response are the patches which I
fixed, I didn't include the fixes themselves since they're obvious.
In this case what I commited was:

+       if (flag < 0 || flag > 2)


-Libor



From yaronh at voltaire.com  Mon Feb 28 13:55:54 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Mon, 28 Feb 2005 23:55:54 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC6A@taurus.voltaire.com>

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Libor Michalek
> Sent: Monday, February 28, 2005 8:55 PM
> To: Roland Dreier
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] IB Address Translation service
> 
> 
>   SDP does implement a subset of the proposed functionality for
> resolving IP addresses to PathRecords which can then be used in
> a CM REQ request, plus some basic caching. All the code is isolated
> to a single file, sdp_link.c. There's really only a single entry
> point API, plus a completion function:
> 
> int sdp_link_path_lookup(u32 dst_addr,
> 			 u32 src_addr,
> 			 int bound_dev_if,
> 			 void (*completion)(u64 id,
> 					    int status,
> 					    u32 dst_addr,
> 					    u32 src_addr,
> 					    u8  hw_port,
> 					    struct ib_device *ca,
> 					    struct ib_sa_path_rec *path,
> 					    void *arg),
> 			 void *arg,
> 			 u64  *id);
> 
>   The values are based on strictly what is needed by either the Linux
> routing code to resolve the address, or the IB APIs to establish the
> connection. The implementation has three stages:
> 
>   - src/dst IP address -> IPoIB net_device, IB ca, IB port, IB pkey.
>   - dst IP address and IPoIB net_device -> dst GID using IPoIB ARP
>   - dst GID -> PathRecord using ib_sa.
> 
>   A cancel function based on the 'id' parameter would be a nice to
have
> but is not strictly necessary, since the lookup will eventually
compelte
> one way or another and any dead connection will be cleaned up at that
> point.
> 

Libor the idea is that ib_at provides similar functionality 
Sahar looked through your SDP code prior to proposing the API
We would like to have a common API for all the ULP's that provide that
functionality, and specifically now when we implement kDAPL over OpenIB.

To summaries the differences:

The reasons we broken it to two functions (IP->GID, GID->Path) and not
have an IP->Path API (like we also used to have in our gen1 stack) are:

a. some consumers will only need the 1st part (e.g. just to know which
HCA to use)
b. some may use only the 2nd part (e.g. IPoIB, SRP)
c. you can get parameters from the first part (e.g. P_Key, and decide to
overwrite it with your own P_Key, etc')
d. the 2nd function provides more options for multipath, partitioning,
QoS
e. we can now more easily use different IP resolution mechanisms without
changing the 2nd function (ARP or ATS).  

We added source IP and TOS as optional parameters for the IP->GID, just
because IP route can be defined for Src/dst/TOS, and it's already part
of Linux.
we added multipath, IB QoS etc' because we have more than one
applications that need it today, e.g. people that want to run IPC/MPI
and NFS on the same fabric may want 2 separate VLs and SLs, some
applications need APM support, some applications need source based
routing, ..
Since there are commercial SMs that can provide all the advanced
capabilities, we want to enable the OpenIB stack to make use of it.

By default you can nest the 2 functions (call the first, and than use
its result to call the second), what you will get is that the ULP will
use the HCA/Port associated with the IPoIB subnet, will use the same
Partition as the IPoIB interface, and will use the same QoS/SL as IPoIB.
Optionally the consumer can put his own QoS, Partitioning, etc' for the
2nd function if he knows where to take it from.

An example of what you can get with the default mode:
You define few partitions in the fabric (central configuration), each
with its IP subnet and SL (from MCRecord).
And then just use different IP subnet for isolation (Partitions or VLs)
NO need for any manual/local configuration on the host side and your
IPC, Storage are now running on separate VLs and/or Partitions, what we
got is something we can explain to users and developers that haven't
read the IB 1000s page book, and don't sit regularly in IBTA meetings.

Yaron



From Tom.Duffy at Sun.COM  Mon Feb 28 13:59:59 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 13:59:59 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Remove use of leading
 underscores in static names in SDP
Message-ID: <1109627999679@sun.com>

This series of patches removes the use of a leading underscore or double
underscore before static or static inline functions and variables.  There are
also a few random whitespace fixes I noticed while going through the code.

 sdp_actv.c  |   41 +++----
 sdp_advt.c  |   16 +-
 sdp_buff.c  |   96 +++++++----------
 sdp_conn.c  |  334 ++++++++++++++++++++++++++++--------------------------------
 sdp_conn.h  |    2
 sdp_event.c |   25 +---
 sdp_inet.c  |  214 ++++++++++++++++++--------------------
 sdp_iocb.c  |  100 ++++++++---------
 sdp_iocb.h  |    8 -
 sdp_link.c  |  209 ++++++++++++++++++-------------------
 sdp_msgs.h  |   88 +++++++--------
 sdp_pass.c  |   22 +--
 sdp_post.c  |   18 +--
 sdp_proc.c  |   35 ++----
 sdp_proto.h |   38 +++---
 sdp_queue.c |   33 ++---
 sdp_queue.h |   12 +-
 sdp_rcvd.c  |  125 +++++++++++-----------
 sdp_read.c  |    6 -
 sdp_recv.c  |  101 ++++++++----------
 sdp_send.c  |  231 +++++++++++++++++++----------------------
 sdp_sent.c  |  117 ++++++++++-----------
 22 files changed, 889 insertions(+), 982 deletions(-)

Signed-off-by: Tom Duffy <tduffy at sun.com>



From Tom.Duffy at Sun.COM  Mon Feb 28 14:00:09 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:00:09 -0800
Subject: [openib-general] [PATCH][SDP][1/22] Remove use of leading
 underscores in static names in sdp_actv.c
In-Reply-To: <1109627999679@sun.com>
References: <1109627999679@sun.com>
Message-ID: <1109628009288@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_actv.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_actv.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_actv.c	(working copy)
@@ -104,9 +104,9 @@ void sdp_cm_actv_error(struct sdp_opt *c
 }
 
 /*
- * _sdp_cm_actv_establish - process an accepted connection request.
+ * sdp_cm_actv_establish - process an accepted connection request.
  */
-static int _sdp_cm_actv_establish(struct sdp_opt *conn)
+static int sdp_cm_actv_establish(struct sdp_opt *conn)
 {
 	struct ib_qp_attr *qp_attr;
 	int attr_mask = 0;
@@ -213,15 +213,15 @@ done:
 }
 
 /*
- * _sdp_cm_hello_ack_check - validate the hello ack header
+ * sdp_cm_hello_ack_check - validate the hello ack header
  */
-static int _sdp_cm_hello_ack_check(struct sdp_msg_hello_ack *hello_ack)
+static int sdp_cm_hello_ack_check(struct sdp_msg_hello_ack *hello_ack)
 {
 	/*
 	 * endian swap
 	 */
-	_sdp_msg_net_to_cpu_bsdh(&hello_ack->bsdh);
-	_sdp_msg_net_to_cpu_hah(&hello_ack->hah);
+	sdp_msg_net_to_cpu_bsdh(&hello_ack->bsdh);
+	sdp_msg_net_to_cpu_hah(&hello_ack->hah);
 	/*
 	 * validation and consistency checks
 	 */
@@ -269,8 +269,7 @@ static int _sdp_cm_hello_ack_check(struc
 /*
  * sdp_cm_rep_handler - handler for active connection open completion
  */
-int sdp_cm_rep_handler(struct ib_cm_id *cm_id,
-		       struct ib_cm_event *event,
+int sdp_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event,
 		       struct sdp_opt *conn)
 {
 	struct sdp_msg_hello_ack *hello_ack;
@@ -293,7 +292,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	 * check Hello Header Ack, to determine if we want
 	 * the connection.
 	 */
-	result = _sdp_cm_hello_ack_check(hello_ack);
+	result = sdp_cm_hello_ack_check(hello_ack);
 	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> hello ack check.", result);
 		goto error;
@@ -322,7 +321,7 @@ int sdp_cm_rep_handler(struct ib_cm_id *
 	 */
 	(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
 
-	result = _sdp_cm_actv_establish(conn);
+	result = sdp_cm_actv_establish(conn);
 	if (result) {
 		sdp_dbg_warn(conn, "Error <%d> accept receive failed", result);
 		goto error;
@@ -333,7 +332,6 @@ error:
 	sdp_cm_actv_error(conn, (0 - result));
 
 	if (conn->state == SDP_CONN_ST_CLOSED) {
-
 		conn->cm_id = NULL;
 		sdp_conn_put(conn); /* CM reference */
 
@@ -344,16 +342,11 @@ error:
 }
 
 /*
- * _sdp_cm_path_complete - path lookup complete, initiate SDP connection
+ * sdp_cm_path_complete - path lookup complete, initiate SDP connection
  */
-static void _sdp_cm_path_complete(u64 id,
-				  int status,
-				  u32 dst_addr,
-				  u32 src_addr,
-				  u8  hw_port,
-				  struct ib_device *ca,
-				  struct ib_sa_path_rec *path,
-				  void *arg)
+static void sdp_cm_path_complete(u64 id, int status, u32 dst_addr, u32 src_addr,
+				 u8 hw_port, struct ib_device *ca,
+				 struct ib_sa_path_rec *path, void *arg)
 {
 	struct ib_cm_req_param param;
 	struct sdp_msg_hello *hello_msg;
@@ -453,8 +446,8 @@ static void _sdp_cm_path_complete(u64 id
 	/*
 	 * endian swap
 	 */
-	_sdp_msg_cpu_to_net_bsdh(&hello_msg->bsdh);
-	_sdp_msg_cpu_to_net_hh(&hello_msg->hh);
+	sdp_msg_cpu_to_net_bsdh(&hello_msg->bsdh);
+	sdp_msg_cpu_to_net_hh(&hello_msg->hh);
 	/*
 	 * save message
 	 */
@@ -505,7 +498,7 @@ static void _sdp_cm_path_complete(u64 id
 	path->packet_life = 13;
 #endif
 	conn->cm_id = ib_create_cm_id(sdp_cm_event_handler,
-				      __hashent_arg(conn->hashent));
+				      hashent_arg(conn->hashent));
 	if (!conn->cm_id) {
 		sdp_dbg_warn(conn, "Failed to create CM handle, %d",
 			     (u8)(buff->tail - buff->data));
@@ -558,7 +551,7 @@ int sdp_cm_connect(struct sdp_opt *conn)
 	result = sdp_link_path_lookup(htonl(conn->dst_addr),
 				      htonl(conn->src_addr),
 				      conn->sk->sk_bound_dev_if,
-				      _sdp_cm_path_complete,
+				      sdp_cm_path_complete,
 				      conn,
 				      &conn->plid);
 	sdp_conn_lock(conn);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:00:20 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:00:20 -0800
Subject: [openib-general] [PATCH][SDP][2/22] Remove use of leading
 underscores in static names in sdp_advt.c
In-Reply-To: <1109628009288@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
Message-ID: <11096280203259@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_advt.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_advt.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_advt.c	(working copy)
@@ -34,7 +34,7 @@
 
 #include "sdp_main.h"
 
-static kmem_cache_t *__sdp_advt_cache = NULL;
+static kmem_cache_t *sdp_advt_cache = NULL;
 
 /*
  * module specific functions
@@ -51,7 +51,7 @@ struct sdpc_advt *sdp_advt_create(void)
 {
 	struct sdpc_advt *advt;
 
-	advt = kmem_cache_alloc(__sdp_advt_cache, SLAB_KERNEL);
+	advt = kmem_cache_alloc(sdp_advt_cache, SLAB_KERNEL);
 	if (advt) {
 		advt->next = NULL;
 		advt->prev = NULL;
@@ -77,7 +77,7 @@ int sdp_advt_destroy(struct sdpc_advt *a
 	/*
 	 * return the object to its cache
 	 */
-	kmem_cache_free(__sdp_advt_cache, advt);
+	kmem_cache_free(sdp_advt_cache, advt);
 
 	return 0;
 }
@@ -187,16 +187,16 @@ int sdp_main_advt_init(void)
 	/*
 	 * initialize the caches only once.
 	 */
-	if (__sdp_advt_cache) {
+	if (sdp_advt_cache) {
 		sdp_warn("Advertisment caches already initialized.");
 		return -EINVAL;
 	}
 
-	__sdp_advt_cache = kmem_cache_create("sdp_advt",
+	sdp_advt_cache = kmem_cache_create("sdp_advt",
 					     sizeof(struct sdpc_advt),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (!__sdp_advt_cache)
+	if (!sdp_advt_cache)
 		return -ENOMEM;
 
 	return 0;
@@ -211,9 +211,9 @@ void sdp_main_advt_cleanup(void)
 	/*
 	 * cleanup the caches
 	 */
-	kmem_cache_destroy(__sdp_advt_cache);
+	kmem_cache_destroy(sdp_advt_cache);
 	/*
 	 * null out entries.
 	 */
-	__sdp_advt_cache = NULL;
+	sdp_advt_cache = NULL;
 }



From Tom.Duffy at Sun.COM  Mon Feb 28 14:00:30 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:00:30 -0800
Subject: [openib-general] [PATCH][SDP][3/22] Remove use of leading
 underscores in static names in sdp_buff.c
In-Reply-To: <11096280203259@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com>
Message-ID: <11096280301502@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_buff.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_buff.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_buff.c	(working copy)
@@ -40,14 +40,14 @@ static struct sdpc_buff_root *main_pool 
  */
 
 /*
- * _sdp_buff_q_get - Get a buffer from a specific pool
+ * do_buff_q_get - Get a buffer from a specific pool
  */
-static inline struct sdpc_buff *_sdp_buff_q_get(struct sdpc_buff_q *pool,
-						int fifo,
-						int (*test_func)
+static inline struct sdpc_buff *do_buff_q_get(struct sdpc_buff_q *pool,
+					      int fifo,
+					      int (*test_func)
 						     (struct sdpc_buff *buff,
 						      void *arg),
-						void *usr_arg)
+					      void *usr_arg)
 {
 	struct sdpc_buff *buff;
 
@@ -82,11 +82,10 @@ static inline struct sdpc_buff *_sdp_buf
 }
 
 /*
- * _sdp_buff_q_put - Place a buffer into a specific pool
+ * do_buff_q_put - Place a buffer into a specific pool
  */
-static inline int _sdp_buff_q_put(struct sdpc_buff_q *pool,
-				  struct sdpc_buff *buff,
-				  int fifo)
+static inline int do_buff_q_put(struct sdpc_buff_q *pool,
+				struct sdpc_buff *buff, int fifo)
 {
 	/* fifo: false == tail, true == head */
 	if (buff->pool)
@@ -114,10 +113,10 @@ static inline int _sdp_buff_q_put(struct
 }
 
 /*
- * _sdp_buff_q_look - look at a buffer from a specific pool
+ * sdp_buff_q_look - look at a buffer from a specific pool
  */
-static inline struct sdpc_buff *_sdp_buff_q_look(struct sdpc_buff_q *pool,
-						 int fifo)
+static inline struct sdpc_buff *sdp_buff_q_look(struct sdpc_buff_q *pool,
+						int fifo)
 {
 	if (!pool->head || fifo)
 		return pool->head;
@@ -126,10 +125,10 @@ static inline struct sdpc_buff *_sdp_buf
 }
 
 /*
- * _sdp_buff_q_remove - remove a specific buffer from a specific pool
+ * do_buff_q_remove - remove a specific buffer from a specific pool
  */
-static inline int _sdp_buff_q_remove(struct sdpc_buff_q *pool, 
-				     struct sdpc_buff *buff)
+static inline int do_buff_q_remove(struct sdpc_buff_q *pool, 
+				   struct sdpc_buff *buff)
 {
 	struct sdpc_buff *prev;
 	struct sdpc_buff *next;
@@ -175,7 +174,7 @@ int sdp_buff_q_remove(struct sdpc_buff *
 	struct sdpc_buff_q *pool;
 
 	pool = buff->pool;
-	return _sdp_buff_q_remove(pool, buff);
+	return do_buff_q_remove(pool, buff);
 }
 
 /*
@@ -183,7 +182,7 @@ int sdp_buff_q_remove(struct sdpc_buff *
  */
 struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool)
 {
-	return _sdp_buff_q_get(pool, 1, NULL, NULL);
+	return do_buff_q_get(pool, 1, NULL, NULL);
 }
 
 /*
@@ -191,7 +190,7 @@ struct sdpc_buff *sdp_buff_q_get(struct 
  */
 struct sdpc_buff *sdp_buff_q_get_head(struct sdpc_buff_q *pool)
 {
-	return _sdp_buff_q_get(pool, 1, NULL, NULL);
+	return do_buff_q_get(pool, 1, NULL, NULL);
 }
 
 /*
@@ -199,7 +198,7 @@ struct sdpc_buff *sdp_buff_q_get_head(st
  */
 struct sdpc_buff *sdp_buff_q_get_tail(struct sdpc_buff_q *pool)
 {
-	return _sdp_buff_q_get(pool, 0, NULL, NULL);
+	return do_buff_q_get(pool, 0, NULL, NULL);
 }
 
 /*
@@ -207,7 +206,7 @@ struct sdpc_buff *sdp_buff_q_get_tail(st
  */
 struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool)
 {
-	return _sdp_buff_q_look(pool, 1);
+	return sdp_buff_q_look(pool, 1);
 }
 
 /*
@@ -218,7 +217,7 @@ struct sdpc_buff *sdp_buff_q_fetch_head(
 						    void *arg),
 					void *usr_arg)
 {
-	return _sdp_buff_q_get(pool, 1, test, usr_arg);
+	return do_buff_q_get(pool, 1, test, usr_arg);
 }
 
 /*
@@ -244,7 +243,7 @@ struct sdpc_buff *sdp_buff_q_fetch(struc
 		     counter < pool->size; counter++, buff = buff->next) {
 			result = test(buff, usr_arg);
 			if (result > 0) {
-				result = _sdp_buff_q_remove(pool, buff);
+				result = do_buff_q_remove(pool, buff);
 				SDP_EXPECT(result >= 0);
 
 				return buff;
@@ -290,41 +289,37 @@ int sdp_buff_q_trav_head(struct sdpc_buf
 /*
  * sdp_buff_q_put - Place a buffer into a specific pool
  */
-int sdp_buff_q_put(struct sdpc_buff_q *pool,
-		   struct sdpc_buff *buff)
+int sdp_buff_q_put(struct sdpc_buff_q *pool, struct sdpc_buff *buff)
 {
-	return _sdp_buff_q_put(pool, buff, 1);
+	return do_buff_q_put(pool, buff, 1);
 }
 
 /*
  * sdp_buff_q_put_head - Place a buffer into the head of a specific pool
  */
-int sdp_buff_q_put_head(struct sdpc_buff_q *pool,
-			struct sdpc_buff *buff)
+int sdp_buff_q_put_head(struct sdpc_buff_q *pool, struct sdpc_buff *buff)
 {
-	return _sdp_buff_q_put(pool, buff, 1);
+	return do_buff_q_put(pool, buff, 1);
 }
 
 /*
  * sdp_buff_q_put_tail - Place a buffer into the tail of a specific pool
  */
-int sdp_buff_q_put_tail(struct sdpc_buff_q *pool,
-			struct sdpc_buff *buff)
+int sdp_buff_q_put_tail(struct sdpc_buff_q *pool, struct sdpc_buff *buff)
 {
-	return _sdp_buff_q_put(pool, buff, 0);
+	return do_buff_q_put(pool, buff, 0);
 }
 
 /*
  * sdp_buff_q_clear_unmap - clear the buffers out of a specific buffer pool
  */
-void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool,
-			    struct device *dev,
+void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool, struct device *dev,
 			    int direction)
 {
 	int result;
 	struct sdpc_buff *buff;
 
-	while ((buff = _sdp_buff_q_get(pool, 0, NULL, NULL))) {
+	while ((buff = do_buff_q_get(pool, 0, NULL, NULL))) {
 		if (dev)
 			dma_unmap_single(dev, buff->real,
 					 buff->tail - buff->data, direction);
@@ -341,9 +336,9 @@ void sdp_buff_q_clear_unmap(struct sdpc_
  */
 
 /*
- * _sdp_buff_pool_release - release allocated buffers from the main pool
+ * sdp_buff_pool_release - release allocated buffers from the main pool
  */
-static void _sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
+static void sdp_buff_pool_release(struct sdpc_buff_root *m_pool, int count)
 {
 	struct sdpc_buff *buff;
 
@@ -365,9 +360,9 @@ static void _sdp_buff_pool_release(struc
 }
 
 /*
- * _sdp_buff_pool_release_check - check for buffer release from main pool
+ * sdp_buff_pool_release_check - check for buffer release from main pool
  */
-static inline void _sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
+static inline void sdp_buff_pool_release_check(struct sdpc_buff_root *m_pool)
 {
 	/*
 	 * If there are more then minimum buffers outstanding, free half of
@@ -383,14 +378,14 @@ static inline void _sdp_buff_pool_releas
 		count = min((m_pool->buff_cur - m_pool->buff_min),
 			    (m_pool->free_mark/2));
 
-		_sdp_buff_pool_release(m_pool, count);
+		sdp_buff_pool_release(m_pool, count);
 	}
 }
 
 /*
- * _sdp_buff_pool_alloc - allocate more buffers for the main pool
+ * sdp_buff_pool_alloc - allocate more buffers for the main pool
  */
-static int _sdp_buff_pool_alloc(struct sdpc_buff_root *m_pool)
+static int sdp_buff_pool_alloc(struct sdpc_buff_root *m_pool)
 {
 	struct sdpc_buff *buff;
 	int total;
@@ -458,10 +453,7 @@ static int _sdp_buff_pool_alloc(struct s
 /*
  * sdp_buff_pool_init - Initialize the main buffer pool of memory
  */
-int sdp_buff_pool_init(int buff_min, 
-		       int buff_max,
-		       int alloc_inc,
-		       int free_mark)
+int sdp_buff_pool_init(int buff_min, int buff_max, int alloc_inc, int free_mark)
 {
 	int result;
 
@@ -522,7 +514,7 @@ int sdp_buff_pool_init(int buff_min, 
 	/*
 	 * allocate the minimum number of buffers.
 	 */
-	result = _sdp_buff_pool_alloc(main_pool);
+	result = sdp_buff_pool_alloc(main_pool);
 	if (result < 0) {
 		sdp_warn("Error <%d> allocating buffers. <%d>",
 			 result, buff_min);
@@ -558,7 +550,7 @@ void sdp_buff_pool_destroy(void)
 	/*
 	 * Free all the buffers.
 	 */
-	_sdp_buff_pool_release(main_pool, main_pool->buff_cur);
+	sdp_buff_pool_release(main_pool, main_pool->buff_cur);
 	/*
 	 * Sanity check that the current number of buffers was released.
 	 */
@@ -596,7 +588,7 @@ struct sdpc_buff *sdp_buff_pool_get(void
 	spin_lock_irqsave(&main_pool->lock, flags);
 
 	if (!main_pool->pool.head) {
-		result = _sdp_buff_pool_alloc(main_pool);
+		result = sdp_buff_pool_alloc(main_pool);
 		if (result < 0) {
 			sdp_warn("Error <%d> allocating buffers.", result);
 			spin_unlock_irqrestore(&main_pool->lock, flags);
@@ -670,7 +662,7 @@ int sdp_buff_pool_put(struct sdpc_buff *
 
 	main_pool->pool.size++;
 
-	_sdp_buff_pool_release_check(main_pool);
+	sdp_buff_pool_release_check(main_pool);
 
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
@@ -732,7 +724,7 @@ int sdp_buff_pool_chain_put(struct sdpc_
 
 	main_pool->pool.size += count;
 
-	(void)_sdp_buff_pool_release_check(main_pool);
+	(void)sdp_buff_pool_release_check(main_pool);
 
 	spin_unlock_irqrestore(&main_pool->lock, flags);
 
@@ -757,9 +749,7 @@ int sdp_buff_pool_buff_size(void)
 /*
  * sdp_proc_dump_buff_pool - write the buffer pool stats to a file (/proc)
  */
-int sdp_proc_dump_buff_pool(char *buffer,
-			    int   max_size,
-			    off_t start_index,
+int sdp_proc_dump_buff_pool(char *buffer, int max_size, off_t start_index,
 			    long *end_index)
 {
 	unsigned long flags;



From Tom.Duffy at Sun.COM  Mon Feb 28 14:00:40 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:00:40 -0800
Subject: [openib-general] [PATCH][SDP][4/22] Remove use of leading
 underscores in static names in sdp_conn.c
In-Reply-To: <11096280301502@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
Message-ID: <11096280402005@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -34,7 +34,7 @@
 
 #include "sdp_main.h"
 
-static struct sdev_root _dev_root_s;
+static struct sdev_root dev_root_s;
 
 static void sdp_device_init_one(struct ib_device *device);
 static void sdp_device_remove_one(struct ib_device *device);
@@ -55,7 +55,7 @@ static u32        psn_seed;
 /*
  * sdp_psn_generate - generate a PSN for connection management.
  */
-static u32 _sdp_psn_generate(void)
+static u32 sdp_psn_generate(void)
 {
         u32 psn;
 
@@ -196,18 +196,18 @@ int sdp_inet_listen_start(struct sdp_opt
 	/*
 	 * table lock
 	 */
-	spin_lock_irqsave(&_dev_root_s.listen_lock, flags);
+	spin_lock_irqsave(&dev_root_s.listen_lock, flags);
 	/*
 	 * insert into listening list.
 	 */
-	conn->lstn_next = _dev_root_s.listen_list;
-	_dev_root_s.listen_list = conn;
-	conn->lstn_p_next = &_dev_root_s.listen_list;
+	conn->lstn_next = dev_root_s.listen_list;
+	dev_root_s.listen_list = conn;
+	conn->lstn_p_next = &dev_root_s.listen_list;
 
 	if (conn->lstn_next)
 		conn->lstn_next->lstn_p_next = &conn->lstn_next;
 
-	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.listen_lock, flags);
 	return 0;
 }
 
@@ -229,7 +229,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	/*
 	 * table lock
 	 */
-	spin_lock_irqsave(&_dev_root_s.listen_lock, flags);
+	spin_lock_irqsave(&dev_root_s.listen_lock, flags);
 	/*
 	 * remove from listening list.
 	 */
@@ -241,7 +241,7 @@ int sdp_inet_listen_stop(struct sdp_opt 
 	listen_conn->lstn_p_next = NULL;
 	listen_conn->lstn_next = NULL;
 
-	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.listen_lock, flags);
 	/*
 	 * reject and delete all pending connections
 	 */
@@ -281,19 +281,18 @@ struct sdp_opt *sdp_inet_listen_lookup(u
 	/*
 	 * table lock
 	 */
-	spin_lock_irqsave(&_dev_root_s.listen_lock, flags);
+	spin_lock_irqsave(&dev_root_s.listen_lock, flags);
 	/*
 	 * first find a listening connection
 	 */
-	for (conn = _dev_root_s.listen_list; conn;
-	     conn = conn->lstn_next)
+	for (conn = dev_root_s.listen_list; conn; conn = conn->lstn_next)
 		if (port == conn->src_port &&
 		    (INADDR_ANY == conn->src_addr || addr == conn->src_addr)) {
 			sdp_conn_hold(conn);
 			break;
 		}
 
-	spin_unlock_irqrestore(&_dev_root_s.listen_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.listen_lock, flags);
 	return conn;
 }
 
@@ -317,12 +316,12 @@ int sdp_inet_port_get(struct sdp_opt *co
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.bind_lock, flags);
+	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
 	/*
 	 * simple linked list of sockets ordered on local port number.
 	 */
 	if (port > 0) {
-		for (look = _dev_root_s.bind_list, port_ok = 1;
+		for (look = dev_root_s.bind_list, port_ok = 1;
 		     look; look = look->bind_next) {
 			srch = look->sk;
 			/*
@@ -385,7 +384,7 @@ int sdp_inet_port_get(struct sdp_opt *co
 			if (rover < low_port || rover > top_port)
 				rover = low_port;
 
-			for (look = _dev_root_s.bind_list;
+			for (look = dev_root_s.bind_list;
 			     look && look->src_port != port;
 			     look = look->bind_next)
 				do {} while(0); /* pass */
@@ -406,16 +405,16 @@ int sdp_inet_port_get(struct sdp_opt *co
 	/*
 	 * insert into listening list.
 	 */
-	conn->bind_next = _dev_root_s.bind_list;
-	_dev_root_s.bind_list = conn;
-	conn->bind_p_next = &_dev_root_s.bind_list;
+	conn->bind_next = dev_root_s.bind_list;
+	dev_root_s.bind_list = conn;
+	conn->bind_p_next = &dev_root_s.bind_list;
 
 	if (conn->bind_next)
 		conn->bind_next->bind_p_next = &conn->bind_next;
 
 	result = 0;
 done:
-	spin_unlock_irqrestore(&_dev_root_s.bind_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
 	return result;
 }
 
@@ -431,7 +430,7 @@ int sdp_inet_port_put(struct sdp_opt *co
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.bind_lock, flags);
+	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
 	/*
 	 * remove from bind list.
 	 */
@@ -444,7 +443,7 @@ int sdp_inet_port_put(struct sdp_opt *co
 	conn->bind_next = NULL;
 	conn->src_port = 0;
 
-	spin_unlock_irqrestore(&_dev_root_s.bind_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
 	return 0;
 }
 
@@ -459,7 +458,7 @@ int sdp_inet_port_inherit(struct sdp_opt
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.bind_lock, flags);
+	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
 
 	if (child->bind_p_next ||
 	    child->src_port != parent->src_port) {
@@ -480,14 +479,14 @@ int sdp_inet_port_inherit(struct sdp_opt
 
 	result = 0;
 done:
-	spin_unlock_irqrestore(&_dev_root_s.bind_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
 	return result;
 }
 
 /*
- * _sdp_conn_table_insert - insert a connection into the connection table
+ * sdp_conn_table_insert - insert a connection into the connection table
  */
-static int _sdp_conn_table_insert(struct sdp_opt *conn)
+static int sdp_conn_table_insert(struct sdp_opt *conn)
 {
 	u32 counter;
 	int result = -ENOMEM;
@@ -498,20 +497,20 @@ static int _sdp_conn_table_insert(struct
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * find an empty slot.
 	 */
 	for (counter = 0;
-	     counter < _dev_root_s.sk_size;
-	     counter++, _dev_root_s.sk_rover++) {
-		if (!(_dev_root_s.sk_rover < _dev_root_s.sk_size))
-			_dev_root_s.sk_rover = 0;
-
-		if (!_dev_root_s.sk_array[_dev_root_s.sk_rover]) {
-			_dev_root_s.sk_array[_dev_root_s.sk_rover] = conn;
-			_dev_root_s.sk_entry++;
-			conn->hashent = _dev_root_s.sk_rover;
+	     counter < dev_root_s.sk_size;
+	     counter++, dev_root_s.sk_rover++) {
+		if (!(dev_root_s.sk_rover < dev_root_s.sk_size))
+			dev_root_s.sk_rover = 0;
+
+		if (!dev_root_s.sk_array[dev_root_s.sk_rover]) {
+			dev_root_s.sk_array[dev_root_s.sk_rover] = conn;
+			dev_root_s.sk_entry++;
+			conn->hashent = dev_root_s.sk_rover;
 
 			result = 0;
 			break;
@@ -519,12 +518,12 @@ static int _sdp_conn_table_insert(struct
 	}
 
 #if 0				/* set for reproducibility from run-run. */
-	_dev_root_s.sk_rover = 0;
+	dev_root_s.sk_rover = 0;
 #endif
 	/*
 	 * unlock table
 	 */
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return result;
 }
 
@@ -539,29 +538,29 @@ int sdp_conn_table_remove(struct sdp_opt
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * validate entry
 	 */
 	if (SDP_DEV_SK_INVALID == conn->hashent)
 		goto done;
 
-	if (conn->hashent < 0 || conn != _dev_root_s.sk_array[conn->hashent]) {
+	if (conn->hashent < 0 || conn != dev_root_s.sk_array[conn->hashent]) {
 		result = -ERANGE;
 		goto done;
 	}
 	/*
 	 * drop entry
 	 */
-	_dev_root_s.sk_array[conn->hashent] = NULL;
-	_dev_root_s.sk_entry--;
+	dev_root_s.sk_array[conn->hashent] = NULL;
+	dev_root_s.sk_entry--;
 	conn->hashent = SDP_DEV_SK_INVALID;
 
 done:
 	/*
 	 * unlock table
 	 */
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return result;
 }
 
@@ -575,43 +574,42 @@ struct sdp_opt *sdp_conn_table_lookup(s3
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 #if 0
 	/*
 	 * validate range
 	 */
-	if (entry < 0 || entry >= _dev_root_s.sk_size) {
-
+	if (entry < 0 || entry >= dev_root_s.sk_size) {
 		conn = NULL;
 		goto done;
 	}
 #endif
 
-	conn = _dev_root_s.sk_array[entry];
+	conn = dev_root_s.sk_array[entry];
 	if (!conn)
 		goto done;
 
 	sdp_conn_hold(conn);
 done:
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return conn;
 }
 
 /* 
  * Functions to cancel IOCB requests in a conenctions queues.
  */
-static int _sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
+static int sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg)
 {
 	return ((element->type == SDP_DESC_TYPE_IOCB) ? 0 : -ERANGE);
 }
 
-static void _sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
+static void sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 
 	while ((iocb = (struct sdpc_iocb *)sdp_desc_q_lookup
 			(table,
-			 _sdp_desc_q_cancel_lookup_func,
+			 sdp_desc_q_cancel_lookup_func,
 			 NULL))) {
 
 		sdp_iocb_q_remove(iocb);
@@ -624,15 +622,15 @@ void sdp_iocb_q_cancel_all_read(struct s
 	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
 	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
 
-	_sdp_desc_q_cancel_iocb(&conn->r_src, error);
+	sdp_desc_q_cancel_iocb(&conn->r_src, error);
 }
 
 void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error)
 {
 	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
 
-	_sdp_desc_q_cancel_iocb(&conn->send_queue, error);
-	_sdp_desc_q_cancel_iocb(&conn->w_snk, error);
+	sdp_desc_q_cancel_iocb(&conn->send_queue, error);
+	sdp_desc_q_cancel_iocb(&conn->w_snk, error);
 }
 
 void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error)
@@ -673,10 +671,10 @@ int sdp_conn_destruct(struct sdp_opt *co
 	result = sdp_conn_table_remove(conn);
 	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
-			     result, _dev_root_s.sk_entry,
-			     _dev_root_s.sk_size);
+			     result, dev_root_s.sk_entry,
+			     dev_root_s.sk_size);
 
-	result = __sdp_conn_stat_dump(conn);
+	result = sdp_conn_stat_dump(conn);
 	SDP_EXPECT(result >= 0);
 	/*
 	 * really there shouldn't be anything in these tables, but it's
@@ -771,13 +769,13 @@ int sdp_conn_destruct(struct sdp_opt *co
 		conn->sk = NULL;
 	}
 
-	kmem_cache_free(_dev_root_s.conn_cache, conn);
+	kmem_cache_free(dev_root_s.conn_cache, conn);
 
 	result = 0;
 done:
 
 	if (dump) {
-		result = __sdp_conn_state_dump(conn);
+		result = sdp_conn_state_dump(conn);
 		SDP_EXPECT(result >= 0);
 	}
 
@@ -945,9 +943,9 @@ void sdp_conn_internal_unlock(struct sdp
 }
 
 /*
- * _sdp_conn_lock_init - initialize connection lock
+ * sdp_conn_lock_init - initialize connection lock
  */
-static void _sdp_conn_lock_init(struct sdp_opt *conn)
+static void sdp_conn_lock_init(struct sdp_opt *conn)
 {
 	spin_lock_init(&(conn->lock.slock));
 	conn->lock.users = 0;
@@ -957,10 +955,8 @@ static void _sdp_conn_lock_init(struct s
 /*
  * sdp_conn_alloc_ib - allocate IB structures for a new connection.
  */
-int sdp_conn_alloc_ib(struct sdp_opt *conn,
-		      struct ib_device *device,
-		      u8 hw_port,
-		      u16 pkey)
+int sdp_conn_alloc_ib(struct sdp_opt *conn, struct ib_device *device,
+		      u8 hw_port, u16 pkey)
 {
 	struct ib_qp_init_attr *init_attr;
 	struct ib_qp_attr     *qp_attr;
@@ -1013,10 +1009,8 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	 * allocate IB CQ's and QP
 	 */
 	if (!conn->send_cq) {
-		conn->send_cq = ib_create_cq(conn->ca,
-					     sdp_cq_event_handler,
-					     NULL,
-					     __hashent_arg(conn->hashent),
+		conn->send_cq = ib_create_cq(conn->ca, sdp_cq_event_handler,
+					     NULL, hashent_arg(conn->hashent),
 					     conn->send_cq_size);
 		if (IS_ERR(conn->send_cq)) {
 			result = PTR_ERR(conn->send_cq);
@@ -1036,10 +1030,8 @@ int sdp_conn_alloc_ib(struct sdp_opt *co
 	}
 
 	if (!conn->recv_cq) {
-		conn->recv_cq = ib_create_cq(conn->ca,
-					     sdp_cq_event_handler,
-					     NULL,
-					     __hashent_arg(conn->hashent),
+		conn->recv_cq = ib_create_cq(conn->ca, sdp_cq_event_handler,
+					     NULL, hashent_arg(conn->hashent),
 					     conn->recv_cq_size);
 
 		if (IS_ERR(conn->recv_cq)) {
@@ -1142,10 +1134,10 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	struct sock *sk;
 	int result;
 
-	sk = sk_alloc(_dev_root_s.proto, priority, 1, _dev_root_s.sock_cache);
+	sk = sk_alloc(dev_root_s.proto, priority, 1, dev_root_s.sock_cache);
 	if (!sk) {
 		sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>",
-			     _dev_root_s.proto, priority);
+			     dev_root_s.proto, priority);
 		return NULL;
 	}
 	/*
@@ -1155,7 +1147,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	/*
 	 * other non-zero sock initialization.
 	 */
-	sk->sk_family   = _dev_root_s.proto;
+	sk->sk_family   = dev_root_s.proto;
 	sk->sk_protocol = IPPROTO_TCP;
 	/*
 	 * some callbacks. higher levels of linux (e.g. setsockopt) call
@@ -1170,7 +1162,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	 * Allocate must be called from process context, since QP
 	 * create/modifies must be in that context.
 	 */
-	conn = kmem_cache_alloc(_dev_root_s.conn_cache, priority);
+	conn = kmem_cache_alloc(dev_root_s.conn_cache, priority);
 	if (!conn) {
 		sdp_dbg_warn(conn, "connection alloc error. <%d>", priority);
 		result = -ENOMEM;
@@ -1216,8 +1208,8 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	conn->nond_recv = 0;
 	conn->nond_send = 0;
 
-	conn->recv_max  = _dev_root_s.recv_buff_max;
-	conn->send_max  = _dev_root_s.send_buff_max;
+	conn->recv_max  = dev_root_s.recv_buff_max;
+	conn->send_max  = dev_root_s.send_buff_max;
 	conn->rwin_max  = SDP_INET_RECV_SIZE;
 	conn->s_wq_size = 0;
 
@@ -1247,15 +1239,15 @@ struct sdp_opt *sdp_conn_alloc(int prior
 
 	conn->send_usig = 0;
 	conn->send_cons = 0;
-	conn->usig_max  = _dev_root_s.send_usig_max;
+	conn->usig_max  = dev_root_s.send_usig_max;
 
 	conn->s_wq_par = 0;
 
 	conn->plid = 0;
 
-	conn->send_cq_size = _dev_root_s.send_post_max;
-	conn->recv_cq_size = _dev_root_s.recv_post_max;
-	conn->s_wq_cur     = _dev_root_s.send_post_max;
+	conn->send_cq_size = dev_root_s.send_post_max;
+	conn->recv_cq_size = dev_root_s.recv_post_max;
+	conn->s_wq_cur     = dev_root_s.send_post_max;
 
 	conn->send_cq  = NULL;
 	conn->recv_cq  = NULL;
@@ -1266,7 +1258,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	conn->fmr_pool = NULL;
 	conn->hw_port  = 0;
 
-	conn->rq_psn   = _sdp_psn_generate();
+	conn->rq_psn   = sdp_psn_generate();
 	/*
 	 * generic send queue
 	 */
@@ -1296,16 +1288,16 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	/*
 	 * connection lock
 	 */
-	_sdp_conn_lock_init(conn);
+	sdp_conn_lock_init(conn);
 	/*
 	 * insert connection into lookup table
 	 */
-	result = _sdp_conn_table_insert(conn);
+	result = sdp_conn_table_insert(conn);
 	if (result < 0) {
 
 		sdp_dbg_warn(conn, "Error <%d> conn table insert <%d:%d>",
-			     result, _dev_root_s.sk_entry,
-			     _dev_root_s.sk_size);
+			     result, dev_root_s.sk_entry,
+			     dev_root_s.sk_size);
 		goto error_conn;
 	}
 	/*
@@ -1321,7 +1313,7 @@ struct sdp_opt *sdp_conn_alloc(int prior
 	 */
 	return conn;
 error_conn:
-	kmem_cache_free(_dev_root_s.conn_cache, conn);
+	kmem_cache_free(dev_root_s.conn_cache, conn);
 error:
 	sk_free(sk);
 	return NULL;
@@ -1350,9 +1342,7 @@ error:
 /*
  * sdp_proc_dump_conn_main - dump the connection table to /proc
  */
-int sdp_proc_dump_conn_main(char *buffer, 
-			    int   max_size, 
-			    off_t start_index, 
+int sdp_proc_dump_conn_main(char *buffer, int max_size, off_t start_index, 
 			    long *end_index)
 {
 	struct sdp_opt *conn;
@@ -1373,23 +1363,23 @@ int sdp_proc_dump_conn_main(char *buffer
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size))
+	if (!(start_index < dev_root_s.sk_size))
 		goto done;
 	/*
 	 * loop across connections.
 	 */
 	for (counter = start_index; 
-	     counter < _dev_root_s.sk_size &&
+	     counter < dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_MAIN_SIZE > (max_size - offset));
 	     counter++) {
-		if (!_dev_root_s.sk_array[counter])
+		if (!dev_root_s.sk_array[counter])
 			continue;
 
-		conn = _dev_root_s.sk_array[counter];
+		conn = dev_root_s.sk_array[counter];
 
 		d_guid = cpu_to_be64(conn->d_gid.global.interface_id);
 		s_guid = cpu_to_be64(conn->s_gid.global.interface_id);
@@ -1432,7 +1422,7 @@ int sdp_proc_dump_conn_main(char *buffer
 	*end_index = counter - start_index;
 
 done:
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return offset;
 }
 
@@ -1456,9 +1446,7 @@ done:
 /*
  * sdp_proc_dump_conn_data - dump the connection table to /proc
  */
-int sdp_proc_dump_conn_data(char *buffer,
-			    int   max_size,
-			    off_t start_index,
+int sdp_proc_dump_conn_data(char *buffer, int max_size, off_t start_index,
 			    long *end_index)
 {
 	struct sock *sk;
@@ -1478,22 +1466,22 @@ int sdp_proc_dump_conn_data(char *buffer
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size))
+	if (!(start_index < dev_root_s.sk_size))
 		goto done;
 	/*
 	 * loop across connections.
 	 */
-	for (counter = start_index; counter < _dev_root_s.sk_size &&
+	for (counter = start_index; counter < dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_DATA_SIZE > (max_size - offset));
 	     counter++) {
-		if (!_dev_root_s.sk_array[counter])
+		if (!dev_root_s.sk_array[counter])
 			continue;
 
-		conn = _dev_root_s.sk_array[counter];
+		conn = dev_root_s.sk_array[counter];
 		sk = conn->sk;
 
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_FORM,
@@ -1535,7 +1523,7 @@ int sdp_proc_dump_conn_data(char *buffer
 
 	*end_index = counter - start_index;
 done:
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return offset;
 }
 
@@ -1553,9 +1541,7 @@ done:
 /*
  * sdp_proc_dump_conn_rdma - dump the connection table to /proc
  */
-int sdp_proc_dump_conn_rdma(char *buffer, 
-			    int   max_size,
-			    off_t start_index,
+int sdp_proc_dump_conn_rdma(char *buffer, int max_size, off_t start_index,
 			    long *end_index)
 {
 	struct sdp_opt *conn;
@@ -1574,22 +1560,22 @@ int sdp_proc_dump_conn_rdma(char *buffer
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size))
+	if (!(start_index < dev_root_s.sk_size))
 		goto done;
 	/*
 	 * loop across connections.
 	 */
-	for (counter = start_index; counter < _dev_root_s.sk_size &&
+	for (counter = start_index; counter < dev_root_s.sk_size &&
 		     !(SDP_CONN_PROC_RDMA_SIZE > (max_size - offset));
 	     counter++) {
-		if (!_dev_root_s.sk_array[counter])
+		if (!dev_root_s.sk_array[counter])
 			continue;
 
-		conn = _dev_root_s.sk_array[counter];
+		conn = dev_root_s.sk_array[counter];
 
 		offset += sprintf((buffer + offset),SDP_PROC_CONN_RDMA_FORM,
 				  conn->hashent,
@@ -1617,7 +1603,7 @@ int sdp_proc_dump_conn_rdma(char *buffer
 
 	*end_index = counter - start_index;
 done:
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return offset;
 }
 
@@ -1632,9 +1618,7 @@ done:
 /*
  * sdp_proc_dump_conn_sopt - dump the options portion of each conn to /proc
  */
-int sdp_proc_dump_conn_sopt(char *buffer,
-			    int   max_size,
-			    off_t start_index,
+int sdp_proc_dump_conn_sopt(char *buffer, int max_size, off_t start_index,
 			    long *end_index)
 {
 	struct sdp_opt *conn;
@@ -1653,22 +1637,22 @@ int sdp_proc_dump_conn_sopt(char *buffer
 	/*
 	 * lock table
 	 */
-	spin_lock_irqsave(&_dev_root_s.sock_lock, flags);
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * if the entire table has been walked, exit.
 	 */
-	if (!(start_index < _dev_root_s.sk_size))
+	if (!(start_index < dev_root_s.sk_size))
 		goto done;
 	/*
 	 * loop across connections.
 	 */
-	for (counter = start_index; counter < _dev_root_s.sk_size &&
+	for (counter = start_index; counter < dev_root_s.sk_size &&
 		     !(SDP_SOPT_PROC_DUMP_SIZE > (max_size - offset));
 	     counter++) {
-		if (!_dev_root_s.sk_array[counter])
+		if (!dev_root_s.sk_array[counter])
 			continue;
 
-		conn = _dev_root_s.sk_array[counter];
+		conn = dev_root_s.sk_array[counter];
 
 		offset += sprintf((buffer + offset), SDP_PROC_CONN_SOPT_FORM,
 				  (conn->dst_addr & 0xff),
@@ -1688,16 +1672,14 @@ int sdp_proc_dump_conn_sopt(char *buffer
 
 	*end_index = counter - start_index;
 done:
-	spin_unlock_irqrestore(&_dev_root_s.sock_lock, flags);
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return offset;
 }
 
 /*
  * sdp_proc_dump_device - dump the primary device table to /proc
  */
-int sdp_proc_dump_device(char *buffer,
-			 int   max_size,
-			 off_t start_index,
+int sdp_proc_dump_device(char *buffer, int max_size, off_t start_index,
 			 long *end_index)
 {
 	int offset = 0;
@@ -1709,29 +1691,29 @@ int sdp_proc_dump_device(char *buffer,
 	if (!start_index) {
 		offset += sprintf((buffer + offset),
 				  "connection table maximum: <%d>\n",
-				  _dev_root_s.sk_size);
+				  dev_root_s.sk_size);
 		offset += sprintf((buffer + offset),
 				  "connection table entries: <%d>\n",
-				  _dev_root_s.sk_entry);
+				  dev_root_s.sk_entry);
 		offset += sprintf((buffer + offset),
 				  "connection table   rover:  <%d>\n",
-				  _dev_root_s.sk_rover);
+				  dev_root_s.sk_rover);
 
 		offset += sprintf((buffer + offset),
 				  "max send posted:          <%d>\n",
-				  _dev_root_s.send_post_max);
+				  dev_root_s.send_post_max);
 		offset += sprintf((buffer + offset),
 				  "max send buffered:        <%d>\n",
-				  _dev_root_s.send_buff_max);
+				  dev_root_s.send_buff_max);
 		offset += sprintf((buffer + offset),
 		      "max send unsignalled:     <%d>\n",
-				  _dev_root_s.send_usig_max);
+				  dev_root_s.send_usig_max);
 		offset += sprintf((buffer + offset),
 				  "max receive posted:       <%d>\n",
-				  _dev_root_s.recv_post_max);
+				  dev_root_s.recv_post_max);
 		offset += sprintf((buffer + offset),
 				  "max receive buffered:     <%d>\n",
-				  _dev_root_s.recv_buff_max);
+				  dev_root_s.recv_buff_max);
 	}
 
 	return offset;
@@ -1910,12 +1892,8 @@ static void sdp_device_remove_one(struct
 /*
  * sdp_conn_table_init - create a sdp connection table
  */
-int sdp_conn_table_init(int proto_family,
-			int conn_size,
-			int recv_post_max,
-			int recv_buff_max,
-			int send_post_max,
-			int send_buff_max,
+int sdp_conn_table_init(int proto_family, int conn_size, int recv_post_max,
+			int recv_buff_max, int send_post_max, int send_buff_max,
 			int send_usig_max)
 {
 	int result;
@@ -1928,27 +1906,27 @@ int sdp_conn_table_init(int proto_family
 	 */
 	get_random_bytes(&psn_seed, sizeof(psn_seed));
 
-	memset(&_dev_root_s, 0, sizeof(struct sdev_root));
+	memset(&dev_root_s, 0, sizeof(struct sdev_root));
 	/*
 	 * list
 	 */
-	_dev_root_s.listen_list = NULL;
-	_dev_root_s.bind_list = NULL;
+	dev_root_s.listen_list = NULL;
+	dev_root_s.bind_list = NULL;
 
-	spin_lock_init(&_dev_root_s.sock_lock);
-	spin_lock_init(&_dev_root_s.bind_lock);
-	spin_lock_init(&_dev_root_s.listen_lock);
+	spin_lock_init(&dev_root_s.sock_lock);
+	spin_lock_init(&dev_root_s.bind_lock);
+	spin_lock_init(&dev_root_s.listen_lock);
 	/*
 	 * Initialize IB
 	 */
-	_dev_root_s.proto = proto_family;
+	dev_root_s.proto = proto_family;
 
-	_dev_root_s.recv_post_max = recv_post_max;
-	_dev_root_s.recv_buff_max = recv_buff_max;
-	_dev_root_s.send_post_max = send_post_max;
-	_dev_root_s.send_buff_max = send_buff_max;
+	dev_root_s.recv_post_max = recv_post_max;
+	dev_root_s.recv_buff_max = recv_buff_max;
+	dev_root_s.send_post_max = send_post_max;
+	dev_root_s.send_buff_max = send_buff_max;
 	
-	_dev_root_s.send_usig_max = send_usig_max;
+	dev_root_s.send_usig_max = send_usig_max;
 	/*
 	 * Get HCA/port list
 	 */
@@ -1968,21 +1946,21 @@ int sdp_conn_table_init(int proto_family
 
 	byte_size = conn_size * sizeof(struct sdp_opt *);
 	page_size = (byte_size >> 12) + ((0xfff & byte_size) > 0 ? 1 : 0);
-	for (_dev_root_s.sk_ordr = 0;
-	     (1 << _dev_root_s.sk_ordr) < page_size; _dev_root_s.sk_ordr++) ;
+	for (dev_root_s.sk_ordr = 0;
+	     (1 << dev_root_s.sk_ordr) < page_size; dev_root_s.sk_ordr++) ;
 
-	_dev_root_s.sk_array = (void *) __get_free_pages(GFP_KERNEL,
-						       _dev_root_s.sk_ordr);
-	if (!_dev_root_s.sk_array) {
+	dev_root_s.sk_array = (void *) __get_free_pages(GFP_KERNEL,
+						        dev_root_s.sk_ordr);
+	if (!dev_root_s.sk_array) {
 		sdp_warn("Failed to create connection table. <%d:%d:%d>",
-			 byte_size, page_size, _dev_root_s.sk_ordr);
+			 byte_size, page_size, dev_root_s.sk_ordr);
 		result = -ENOMEM;
 		goto error_array;
 	}
 
-	memset(_dev_root_s.sk_array, 0, byte_size);
-	_dev_root_s.sk_size = conn_size - 1; /* top is reserved for invalid */
-	_dev_root_s.sk_array++; /* bump table forward so negative -1 index
+	memset(dev_root_s.sk_array, 0, byte_size);
+	dev_root_s.sk_size = conn_size - 1; /* top is reserved for invalid */
+	dev_root_s.sk_array++; /* bump table forward so negative -1 index
 				   is always a null entry to improve invalid
 				   entry processing. */
 	/*
@@ -1994,21 +1972,21 @@ int sdp_conn_table_init(int proto_family
 		goto error_iocb;
 	}
 
-	_dev_root_s.conn_cache = kmem_cache_create("sdp_conn",
+	dev_root_s.conn_cache = kmem_cache_create("sdp_conn",
 						   sizeof(struct sdp_opt),
 						   0, SLAB_HWCACHE_ALIGN,
 						   NULL, NULL);
-	if (!_dev_root_s.conn_cache) {
+	if (!dev_root_s.conn_cache) {
 		sdp_warn("Failed to initialize connection cache.");
 		result = -ENOMEM;
 		goto error_conn;
 	}
 
-        _dev_root_s.sock_cache = kmem_cache_create("sdp_sock",
+        dev_root_s.sock_cache = kmem_cache_create("sdp_sock",
 						   sizeof(struct inet_sock), 
 						   0, SLAB_HWCACHE_ALIGN,
 						   NULL, NULL);
-        if (!_dev_root_s.sock_cache) {
+        if (!dev_root_s.sock_cache) {
 		sdp_warn("Failed to initialize sock cache.");
 		result = -ENOMEM;
 		goto error_sock;
@@ -2017,7 +1995,7 @@ int sdp_conn_table_init(int proto_family
 	/*
 	 * start listening
 	 */
-	result = sdp_cm_listen_start(&_dev_root_s);
+	result = sdp_cm_listen_start(&dev_root_s);
 	if (result < 0) {
 		sdp_warn("Error <%d> listening for connections on HCA.",
 			 result);
@@ -2026,14 +2004,14 @@ int sdp_conn_table_init(int proto_family
 
 	return 0;
 error_listen:
-	kmem_cache_destroy(_dev_root_s.sock_cache);
+	kmem_cache_destroy(dev_root_s.sock_cache);
 error_sock:
-	kmem_cache_destroy(_dev_root_s.conn_cache);
+	kmem_cache_destroy(dev_root_s.conn_cache);
 error_conn:
 	sdp_main_iocb_cleanup();
 error_iocb:
-	_dev_root_s.sk_array--;
-	free_pages((unsigned long)_dev_root_s.sk_array, _dev_root_s.sk_ordr);
+	dev_root_s.sk_array--;
+	free_pages((unsigned long)dev_root_s.sk_array, dev_root_s.sk_ordr);
 error_array:
 error_size:
 	ib_unregister_client(&sdp_client);
@@ -2052,7 +2030,7 @@ int sdp_conn_table_clear(void)
 	/*
 	 * drain all the connections
 	 */
-	while ((conn = _dev_root_s.conn_list)) {
+	while ((conn = dev_root_s.conn_list)) {
 
 	}
 #endif
@@ -2063,20 +2041,20 @@ int sdp_conn_table_clear(void)
 	/*
 	 * drop socket table
 	 */
-	_dev_root_s.sk_array--;
-	free_pages((unsigned long)_dev_root_s.sk_array, _dev_root_s.sk_ordr);
+	dev_root_s.sk_array--;
+	free_pages((unsigned long)dev_root_s.sk_array, dev_root_s.sk_ordr);
 	/*
 	 * delete conn cache
 	 */
-	kmem_cache_destroy(_dev_root_s.conn_cache);
+	kmem_cache_destroy(dev_root_s.conn_cache);
 	/*
 	 * delete sock cache
 	 */
-	kmem_cache_destroy(_dev_root_s.sock_cache);
+	kmem_cache_destroy(dev_root_s.sock_cache);
 	/*
 	 * stop listening
 	 */
-	(void)sdp_cm_listen_stop(&_dev_root_s);
+	(void)sdp_cm_listen_stop(&dev_root_s);
 	/*
 	 * delete IOCB table
 	 */



From Tom.Duffy at Sun.COM  Mon Feb 28 14:00:50 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:00:50 -0800
Subject: [openib-general] [PATCH][SDP][5/22] Remove use of leading
 underscores in static names in sdp_conn.h
In-Reply-To: <11096280402005@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
Message-ID: <11096280502716@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_conn.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_conn.h	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_conn.h	(working copy)
@@ -534,7 +534,7 @@ static inline int sdp_conn_error(struct 
 	return -error;
 }
 
-static inline void *__hashent_arg(s32 hashent) 
+static inline void *hashent_arg(s32 hashent) 
 {
 	return (void *)(unsigned long)hashent;
 }



From Tom.Duffy at Sun.COM  Mon Feb 28 14:01:01 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:01:01 -0800
Subject: [openib-general] [PATCH][SDP][6/22] Remove use of leading
 underscores in static names in sdp_event.c
In-Reply-To: <11096280502716@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com> <11096280502716@sun.com>
Message-ID: <1109628061177@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_event.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_event.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_event.c	(working copy)
@@ -198,9 +198,8 @@ done:
  * Connection establishment IB/CM callback functions
  */
 
-static int _sdp_cm_idle(struct ib_cm_id *cm_id,
-			struct ib_cm_event *event,
-			struct sdp_opt *conn)
+static int sdp_cm_idle(struct ib_cm_id *cm_id, struct ib_cm_event *event,
+		       struct sdp_opt *conn)
 {
 	sdp_dbg_ctrl(conn, "CM IDLE. commID <%08x> event <%d> status <%d>",
 		     cm_id->local_id, event->event, event->param.send_status);
@@ -241,9 +240,8 @@ static int _sdp_cm_idle(struct ib_cm_id 
 	return 0;
 }
 
-static int _sdp_cm_established(struct ib_cm_id *cm_id,
-			       struct ib_cm_event *event,
-			       struct sdp_opt *conn)
+static int sdp_cm_established(struct ib_cm_id *cm_id, struct ib_cm_event *event,
+			      struct sdp_opt *conn)
 {
 	int result = 0;
 
@@ -309,11 +307,10 @@ done:
 }
 
 /*
- * _sdp_cm_timewait - handler for connection Time Wait completion
+ * sdp_cm_timewait - handler for connection Time Wait completion
  */
-static int _sdp_cm_timewait(struct ib_cm_id *cm_id,
-			    struct ib_cm_event *event,
-			    struct sdp_opt *conn)
+static int sdp_cm_timewait(struct ib_cm_id *cm_id, struct ib_cm_event *event,
+			   struct sdp_opt *conn)
 {
 	int result = 0;
 
@@ -420,14 +417,14 @@ int sdp_cm_event_handler(struct ib_cm_id
 		result = sdp_cm_rep_handler(cm_id, event, conn);
 		break;
 	case IB_CM_IDLE:
-		result = _sdp_cm_idle(cm_id, event, conn);
+		result = sdp_cm_idle(cm_id, event, conn);
 		break;
 	case IB_CM_ESTABLISHED:
-		result = _sdp_cm_established(cm_id, event, conn);
+		result = sdp_cm_established(cm_id, event, conn);
 		break;
 	case IB_CM_DREQ_RCVD:
 	case IB_CM_TIMEWAIT:
-		result = _sdp_cm_timewait(cm_id, event, conn);
+		result = sdp_cm_timewait(cm_id, event, conn);
 		break;
 	default:
 		sdp_dbg_warn(conn, "Unexpected CM state <%d>", cm_id->state);
@@ -444,7 +441,7 @@ int sdp_cm_event_handler(struct ib_cm_id
 			/*
 			 * dump connection state if it is being recorded.
 			 */
-			(void)__sdp_conn_state_dump(conn);
+			(void)sdp_conn_state_dump(conn);
 		}
 
 		sdp_conn_unlock(conn);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:01:11 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:01:11 -0800
Subject: [openib-general] [PATCH][SDP][7/22] Remove use of leading
 underscores in static names in sdp_inet.c
In-Reply-To: <1109628061177@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com>
Message-ID: <110962807127@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -39,18 +39,18 @@
 /*
  * list of connections waiting for an incomming connection
  */
-static int _proto_family    = AF_INET_SDP;
-static int _buff_min        = SDP_BUFF_POOL_COUNT_MIN;
-static int _buff_max        = SDP_BUFF_POOL_COUNT_MAX;
-static int _alloc_inc       = SDP_BUFF_POOL_COUNT_INC;
-static int _free_mark       = SDP_BUFF_POOL_FREE_MARK;
-static int _conn_size       = SDP_DEV_SK_LIST_SIZE;
-
-static int _recv_post_max   = SDP_CQ_RECV_SIZE;
-static int _recv_buff_max   = SDP_BUFF_RECV_MAX;
-static int _send_post_max   = SDP_CQ_SEND_SIZE;
-static int _send_buff_max   = SDP_BUFF_SEND_MAX;
-static int _send_usig_max   = SDP_SEND_UNSIG_MAX;
+static int proto_family    = AF_INET_SDP;
+static int buff_min        = SDP_BUFF_POOL_COUNT_MIN;
+static int buff_max        = SDP_BUFF_POOL_COUNT_MAX;
+static int alloc_inc       = SDP_BUFF_POOL_COUNT_INC;
+static int free_mark       = SDP_BUFF_POOL_FREE_MARK;
+static int conn_size       = SDP_DEV_SK_LIST_SIZE;
+
+static int recv_post_max   = SDP_CQ_RECV_SIZE;
+static int recv_buff_max   = SDP_BUFF_RECV_MAX;
+static int send_post_max   = SDP_CQ_SEND_SIZE;
+static int send_buff_max   = SDP_BUFF_SEND_MAX;
+static int send_usig_max   = SDP_SEND_UNSIG_MAX;
 
 int sdp_debug_level = __SDP_DEBUG_LEVEL;
 
@@ -58,16 +58,16 @@ MODULE_AUTHOR("Libor Michalek");
 MODULE_DESCRIPTION("InfiniBand SDP module");
 MODULE_LICENSE("Dual BSD/GPL");
 
-module_param(_proto_family, int, 0);
-module_param(_buff_min, int, 0);
-module_param(_buff_max, int, 0);
-module_param(_conn_size, int, 0);
-module_param(_recv_post_max, int, 0);
-module_param(_recv_buff_max, int, 0);
-module_param(_send_post_max, int, 0);
-module_param(_send_buff_max, int, 0);
-module_param(_send_usig_max, int, 0);
-module_param( sdp_debug_level, int, 0);
+module_param(proto_family, int, 0);
+module_param(buff_min, int, 0);
+module_param(buff_max, int, 0);
+module_param(conn_size, int, 0);
+module_param(recv_post_max, int, 0);
+module_param(recv_buff_max, int, 0);
+module_param(send_post_max, int, 0);
+module_param(send_buff_max, int, 0);
+module_param(send_usig_max, int, 0);
+module_param(sdp_debug_level, int, 0);
 
 /*
  * socket structure relevant fields:
@@ -106,9 +106,8 @@ void sdp_inet_wake_send(struct sock *sk)
 	if (!sk || !(conn = SDP_GET_CONN(sk)))
 		return;
 
-	if (sk->sk_socket &&
-	    test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) &&
-	    __sdp_inet_writable(conn)) {
+	if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) &&
+	    sdp_inet_writable(conn)) {
 		read_lock(&sk->sk_callback_lock);
 		clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
 
@@ -186,9 +185,9 @@ void sdp_inet_wake_urg(struct sock *sk)
  */
 
 /*
- * _sdp_inet_abort - abort an existing connection
+ * sdp_inet_abort - abort an existing connection
  */
-static int _sdp_inet_abort(struct sdp_opt *conn)
+static int sdp_inet_abort(struct sdp_opt *conn)
 {
 	int result;
 
@@ -233,9 +232,9 @@ static int _sdp_inet_abort(struct sdp_op
 }
 
 /*
- * _sdp_inet_disconnect - disconnect a connection
+ * sdp_inet_disconnect - disconnect a connection
  */
-static int _sdp_inet_disconnect(struct sdp_opt *conn)
+static int sdp_inet_disconnect(struct sdp_opt *conn)
 {
 	int result = 0;
 
@@ -296,9 +295,9 @@ static int _sdp_inet_disconnect(struct s
  */
 
 /*
- * _sdp_inet_release - release/close a socket
+ * sdp_inet_release - release/close a socket
  */
-static int _sdp_inet_release(struct socket *sock)
+static int sdp_inet_release(struct socket *sock)
 {
 	struct sdp_opt *conn;
 	struct sock *sk;
@@ -357,7 +356,7 @@ static int _sdp_inet_release(struct sock
 		/*
 		 * abort.
 		 */
-		result = _sdp_inet_abort(conn);
+		result = sdp_inet_abort(conn);
 		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> while aborting socket",
 				     result);
@@ -367,7 +366,7 @@ static int _sdp_inet_release(struct sock
 	/*
 	 * disconnect. (state dependant)
 	 */
-	result = _sdp_inet_disconnect(conn);
+	result = sdp_inet_disconnect(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> while disconnecting socket",
 			     result);
@@ -414,7 +413,7 @@ static int _sdp_inet_release(struct sock
 
 			sdp_iocb_q_cancel_all_write(conn, -ECANCELED);
 
-			result = _sdp_inet_disconnect(conn);
+			result = sdp_inet_disconnect(conn);
 			SDP_EXPECT(result >= 0);
 		}
 #endif
@@ -435,10 +434,9 @@ done:
 }
 
 /*
- * _sdp_inet_bind - bind a socket to an address/interface
+ * sdp_inet_bind - bind a socket to an address/interface
  */
-static int _sdp_inet_bind(struct socket *sock, struct sockaddr *uaddr,
-			  int size)
+static int sdp_inet_bind(struct socket *sock, struct sockaddr *uaddr, int size)
 {
 	struct sockaddr_in *addr = (struct sockaddr_in *)uaddr;
 	struct sock *sk;
@@ -456,7 +454,7 @@ static int _sdp_inet_bind(struct socket 
 	if (size < sizeof(struct sockaddr_in))
 		return -EINVAL;
 
-	if (_proto_family != addr->sin_family &&
+	if (proto_family != addr->sin_family &&
 	    AF_INET != addr->sin_family && AF_UNSPEC != addr->sin_family)
 		return -EAFNOSUPPORT;
 	/*
@@ -527,12 +525,10 @@ done:
 }
 
 /*
- * _sdp_inet_connect - connect a socket to a remote address
+ * sdp_inet_connect - connect a socket to a remote address
  */
-static int _sdp_inet_connect(struct socket *sock,
-				struct sockaddr *uaddr,
-				int size,
-				int flags)
+static int sdp_inet_connect(struct socket *sock, struct sockaddr *uaddr,
+			    int size, int flags)
 {
 	struct sockaddr_in *addr = (struct sockaddr_in *)uaddr;
 	struct sock *sk;
@@ -549,7 +545,7 @@ static int _sdp_inet_connect(struct sock
 	if (size < sizeof(struct sockaddr_in))
 		return -EINVAL;
 
-	if (_proto_family != addr->sin_family &&
+	if (proto_family != addr->sin_family &&
 	    AF_INET != addr->sin_family && AF_UNSPEC != addr->sin_family)
 		return -EAFNOSUPPORT;
 
@@ -694,9 +690,9 @@ done:
 }
 
 /*
- * _sdp_inet_listen - listen on a socket for incomming addresses
+ * sdp_inet_listen - listen on a socket for incomming addresses
  */
-static int _sdp_inet_listen(struct socket *sock, int backlog)
+static int sdp_inet_listen(struct socket *sock, int backlog)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -751,11 +747,10 @@ done:
 }
 
 /*
- * _sdp_inet_accept - accept a new socket from a listen socket
+ * sdp_inet_accept - accept a new socket from a listen socket
  */
-static int _sdp_inet_accept(struct socket *listen_sock,
-			       struct socket *accept_sock, 
-			       int flags)
+static int sdp_inet_accept(struct socket *listen_sock,
+			   struct socket *accept_sock, int flags)
 {
 	struct sock *listen_sk;
 	struct sock *accept_sk = NULL;
@@ -908,10 +903,10 @@ listen_done:
 }
 
 /*
- * _sdp_inet_getname - return a sockets address information
+ * sdp_inet_getname - return a sockets address information
  */
-static int _sdp_inet_getname(struct socket *sock, struct sockaddr *uaddr,
-			     int *size, int peer)
+static int sdp_inet_getname(struct socket *sock, struct sockaddr *uaddr,
+			    int *size, int peer)
 {
 	struct sockaddr_in *addr = (struct sockaddr_in *)uaddr;
 	struct sock *sk;
@@ -924,7 +919,7 @@ static int _sdp_inet_getname(struct sock
 		     conn->src_addr, conn->src_port, 
 		     conn->dst_addr, conn->dst_port);
 
-	addr->sin_family = _proto_family;
+	addr->sin_family = proto_family;
 	if (peer > 0)
 		if (htons(conn->dst_port) > 0 &&
 		    !(SDP_ST_MASK_CLOSED & conn->istate)) {
@@ -944,11 +939,10 @@ static int _sdp_inet_getname(struct sock
 }
 
 /*
- * _sdp_inet_poll - poll a socket for activity
+ * sdp_inet_poll - poll a socket for activity
  */
-static unsigned int _sdp_inet_poll(struct file *file,
-				   struct socket *sock,
-				   poll_table *wait)
+static unsigned int sdp_inet_poll(struct file *file, struct socket *sock,
+				  poll_table *wait)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1013,7 +1007,7 @@ static unsigned int _sdp_inet_poll(struc
 			set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
 			set_bit(SOCK_NOSPACE, &sock->flags);
 
-			if (__sdp_inet_writable(conn)) {
+			if (sdp_inet_writable(conn)) {
 				mask |= POLLOUT | POLLWRNORM;
 
 				clear_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
@@ -1027,17 +1021,17 @@ static unsigned int _sdp_inet_poll(struc
 
 	sdp_dbg_data(conn, "POLL: mask <%08x> flags <%08lx> <%d:%d:%d>", 
 		     mask, sock->flags, conn->send_buf, conn->send_qud,
-		     __sdp_inet_writable(conn));
+		     sdp_inet_writable(conn));
 	
 
 	return mask;
 }
 
 /*
- * _sdp_inet_ioctl - serivce an ioctl request on a socket
+ * sdp_inet_ioctl - serivce an ioctl request on a socket
  */
-static int _sdp_inet_ioctl(struct socket *sock, unsigned int cmd,
-			   unsigned long arg)
+static int sdp_inet_ioctl(struct socket *sock, unsigned int cmd,
+			  unsigned long arg)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1157,10 +1151,10 @@ static int _sdp_inet_ioctl(struct socket
 }
 
 /*
- * _sdp_inet_setopt - set a socket option
+ * sdp_inet_setopt - set a socket option
  */
-static int _sdp_inet_setopt(struct socket *sock, int level, int optname, 
-			    char __user *optval, int optlen)
+static int sdp_inet_setopt(struct socket *sock, int level, int optname, 
+			   char __user *optval, int optlen)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1224,10 +1218,10 @@ static int _sdp_inet_setopt(struct socke
 }
 
 /*
- * _sdp_inet_getopt - get a socket option
+ * sdp_inet_getopt - get a socket option
  */
-static int _sdp_inet_getopt(struct socket *sock, int level, int optname,
-			    char __user *optval, int __user *optlen)
+static int sdp_inet_getopt(struct socket *sock, int level, int optname,
+			   char __user *optval, int __user *optlen)
 {
 	struct sock *sk;
 	struct sdp_opt *conn;
@@ -1286,9 +1280,9 @@ static int _sdp_inet_getopt(struct socke
 }
 
 /*
- * _sdp_inet_shutdown - shutdown a socket
+ * sdp_inet_shutdown - shutdown a socket
  */
-static int _sdp_inet_shutdown(struct socket *sock, int flag)
+static int sdp_inet_shutdown(struct socket *sock, int flag)
 {
 	int result = 0;
 	struct sdp_opt *conn;
@@ -1332,14 +1326,14 @@ static int _sdp_inet_shutdown(struct soc
 		break;
 	case SDP_SOCK_ST_ACCEPTED:
 	case SDP_SOCK_ST_CONNECT:
-		result = _sdp_inet_abort(conn);
+		result = sdp_inet_abort(conn);
 		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> aborting connection",
 				     result);
 		break;
 	case SDP_SOCK_ST_ESTABLISHED:
 	case SDP_SOCK_ST_CLOSE:
-		result = _sdp_inet_disconnect(conn);
+		result = sdp_inet_disconnect(conn);
 		if (result < 0)
 			sdp_dbg_warn(conn, "Error <%d> disconnecting conn",
 				     result);
@@ -1380,30 +1374,30 @@ static int _sdp_inet_shutdown(struct soc
  * Primary socket initialization
  */
 struct proto_ops _lnx_stream_ops = {
-	.family     =  AF_INET_SDP,
-	.release    = _sdp_inet_release,
-	.bind       = _sdp_inet_bind,
-	.connect    = _sdp_inet_connect,
-	.listen     = _sdp_inet_listen,
-	.accept     = _sdp_inet_accept,
-	.sendmsg    =  sdp_inet_send,
-	.recvmsg    =  sdp_inet_recv,
-	.getname    = _sdp_inet_getname,
-	.poll       = _sdp_inet_poll,
-	.setsockopt = _sdp_inet_setopt,
-	.getsockopt = _sdp_inet_getopt,
-	.shutdown   = _sdp_inet_shutdown,
-	.ioctl      = _sdp_inet_ioctl,
-	.sendpage   =  sock_no_sendpage,
-	.socketpair =  sock_no_socketpair,
-	.mmap       =  sock_no_mmap,
-	.owner      =  THIS_MODULE,
+	.family     = AF_INET_SDP,
+	.release    = sdp_inet_release,
+	.bind       = sdp_inet_bind,
+	.connect    = sdp_inet_connect,
+	.listen     = sdp_inet_listen,
+	.accept     = sdp_inet_accept,
+	.sendmsg    = sdp_inet_send,
+	.recvmsg    = sdp_inet_recv,
+	.getname    = sdp_inet_getname,
+	.poll       = sdp_inet_poll,
+	.setsockopt = sdp_inet_setopt,
+	.getsockopt = sdp_inet_getopt,
+	.shutdown   = sdp_inet_shutdown,
+	.ioctl      = sdp_inet_ioctl,
+	.sendpage   = sock_no_sendpage,
+	.socketpair = sock_no_socketpair,
+	.mmap       = sock_no_mmap,
+	.owner      = THIS_MODULE,
 };
 
 /*
- * _sdp_inet_create - create a socket
+ * sdp_inet_create - create a socket
  */
-static int _sdp_inet_create(struct socket *sock, int protocol)
+static int sdp_inet_create(struct socket *sock, int protocol)
 {
 	struct sdp_opt *conn;
 
@@ -1446,9 +1440,9 @@ static int _sdp_inet_create(struct socke
 /*
  * INET module initialization functions
  */
-static struct net_proto_family _sdp_proto = {
+static struct net_proto_family sdp_proto = {
 	.family =  AF_INET_SDP,
-	.create = _sdp_inet_create,
+	.create = sdp_inet_create,
 	.owner  =  THIS_MODULE,
 };
 
@@ -1494,10 +1488,10 @@ static int __init sdp_init(void)
 	/*
 	 * buffer memory
 	 */
-	result = sdp_buff_pool_init(_buff_min, 
-				    _buff_max, 
-				    _alloc_inc, 
-				    _free_mark);
+	result = sdp_buff_pool_init(buff_min, 
+				    buff_max, 
+				    alloc_inc, 
+				    free_mark);
 	if (result < 0) {
 		sdp_warn("INIT: Error <%d> initializing buffer pool.", result);
 		goto error_buff;
@@ -1505,13 +1499,13 @@ static int __init sdp_init(void)
 	/*
 	 * connection table
 	 */
-	result = sdp_conn_table_init(_proto_family,
-				     _conn_size, 
-				     _recv_post_max,
-				     _recv_buff_max,
-				     _send_post_max,
-				     _send_buff_max,
-				     _send_usig_max);
+	result = sdp_conn_table_init(proto_family,
+				     conn_size, 
+				     recv_post_max,
+				     recv_buff_max,
+				     send_post_max,
+				     send_buff_max,
+				     send_usig_max);
 	if (result < 0) {
 		sdp_warn("INIT: Error <%d> initializing connection table.",
 			 result);
@@ -1520,12 +1514,12 @@ static int __init sdp_init(void)
 	/*
 	 * register
 	 */
-	_sdp_proto.family = _proto_family;
+	sdp_proto.family = proto_family;
 
-	result = sock_register(&_sdp_proto);
+	result = sock_register(&sdp_proto);
 	if (result < 0) {
 		sdp_warn("INIT: Error <%d> registering protocol family <%d>",
-			 result, _sdp_proto.family);
+			 result, sdp_proto.family);
 		goto error_sock;
 	}
 
@@ -1553,7 +1547,7 @@ static void __exit sdp_exit(void)
 	/*
 	 * unregister
 	 */
-	sock_unregister(_sdp_proto.family);
+	sock_unregister(sdp_proto.family);
 	/*
 	 * connection table
 	 */



From Tom.Duffy at Sun.COM  Mon Feb 28 14:01:21 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:01:21 -0800
Subject: [openib-general] [PATCH][SDP][8/22] Remove use of leading
 underscores in static names in sdp_iocb.c
In-Reply-To: <110962807127@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
Message-ID: <11096280813891@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_iocb.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_iocb.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_iocb.c	(working copy)
@@ -34,7 +34,7 @@
 
 #include "sdp_main.h"
 
-static kmem_cache_t *__sdp_iocb_cache = NULL;
+static kmem_cache_t *sdp_iocb_cache = NULL;
 
 /*
  * memory locking functions
@@ -42,12 +42,12 @@ static kmem_cache_t *__sdp_iocb_cache = 
 #include <linux/utsname.h>
 
 typedef int (*do_mlock_ptr_t)(unsigned long, size_t, int);
-static do_mlock_ptr_t _mlock_ptr = NULL;
+static do_mlock_ptr_t mlock_ptr = NULL;
 
 /*
- * _sdp_iocb_unlock - unlock the memory for an IOCB
+ * do_iocb_unlock - unlock the memory for an IOCB
  */
-static int _sdp_iocb_unlock(struct sdpc_iocb *iocb)
+static int do_iocb_unlock(struct sdpc_iocb *iocb)
 {
 	struct vm_area_struct *vma;
 
@@ -111,7 +111,7 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 	 */
 	down_write(&iocb->mm->mmap_sem);
 	
-	result = _sdp_iocb_unlock(iocb);
+	result = do_iocb_unlock(iocb);
 
 	up_write(&iocb->mm->mmap_sem);
 
@@ -130,9 +130,9 @@ int sdp_iocb_unlock(struct sdpc_iocb *io
 }
 
 /*
- * _sdp_iocb_page_save - save page information for an IOCB
+ * sdp_iocb_page_save - save page information for an IOCB
  */
-static int _sdp_iocb_page_save(struct sdpc_iocb *iocb)
+static int sdp_iocb_page_save(struct sdpc_iocb *iocb)
 {
 	unsigned int counter;
 	unsigned long addr;
@@ -265,7 +265,7 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	down_write(&iocb->mm->mmap_sem);
 
 	if (!((iocb->page_count + current->mm->locked_vm) > limit))
-		result = (*_mlock_ptr)(addr, size, 1);
+		result = (*mlock_ptr)(addr, size, 1);
 	/*
 	 * process result
 	 */
@@ -313,7 +313,7 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 			vma = NULL;
 	}
 
-	result = _sdp_iocb_page_save(iocb);
+	result = sdp_iocb_page_save(iocb);
 	if (result) {
 		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", 
 			    result, iocb->addr, iocb->size);
@@ -326,7 +326,7 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb
 	return 0;
 err_save:
 
-	(void)_sdp_iocb_unlock(iocb);
+	(void)do_iocb_unlock(iocb);
 err_lock:
 	/*
 	 * unlock the mm and restore capabilities.
@@ -343,20 +343,19 @@ err_lock:
 /*
  * IOCB memory locking init functions
  */
-struct kallsym_iter
-{
-  loff_t pos;
-  struct module *owner;
-  unsigned long value;
-  unsigned int nameoff; /* If iterating in core kernel symbols */
-  char type;
-  char name[128];
+struct kallsym_iter {
+	loff_t pos;
+	struct module *owner;
+	unsigned long value;
+	unsigned int nameoff; /* If iterating in core kernel symbols */
+	char type;
+	char name[128];
 };
 
 /*
- * _sdp_mem_lock_init - initialize the userspace memory locking
+ * sdp_mem_lock_init - initialize the userspace memory locking
  */
-static int _sdp_mem_lock_init(void)
+static int sdp_mem_lock_init(void)
 {
 	struct file *kallsyms;
 	struct seq_file *seq;
@@ -382,9 +381,9 @@ static int _sdp_mem_lock_init(void)
 	     iter != NULL;
 	     iter = seq->op->next(seq, iter, &pos))
 		if (!strcmp(iter->name, "do_mlock"))
-			_mlock_ptr = (do_mlock_ptr_t)iter->value;
+			mlock_ptr = (do_mlock_ptr_t)iter->value;
 
-	if (!_mlock_ptr)
+	if (!mlock_ptr)
 		sdp_warn("Failed to find lock pointer.");
 	else
 		ret = 0;
@@ -396,15 +395,15 @@ done:
 }
 
 /*
- * _sdp_mem_lock_cleanup - cleanup the memory locking tables
+ * sdp_mem_lock_cleanup - cleanup the memory locking tables
  */
-static int _sdp_mem_lock_cleanup(void)
+static int sdp_mem_lock_cleanup(void)
 {
 	sdp_dbg_init("Memory Locking cleanup.");
 	/*
 	 * null out entries.
 	 */
-	_mlock_ptr = NULL;
+	mlock_ptr = NULL;
 	
 	return 0;
 }
@@ -485,9 +484,9 @@ int sdp_iocb_release(struct sdpc_iocb *i
 }
 
 /*
- * _sdp_iocb_complete - complete an IOCB for real in thread context
+ * do_iocb_complete - complete an IOCB for real in thread context
  */
-static void _sdp_iocb_complete(void *arg)
+static void do_iocb_complete(void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *)arg;
 	int result;
@@ -536,10 +535,10 @@ int sdp_iocb_complete(struct sdpc_iocb *
 	iocb->status = status;
 	
 	if (in_atomic() || irqs_disabled()) {
-		INIT_WORK(&iocb->completion, _sdp_iocb_complete, (void *)iocb);
+		INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb);
 		schedule_work(&iocb->completion);
 	} else
-		_sdp_iocb_complete(iocb);
+		do_iocb_complete(iocb);
 	
 	return 0;
 }
@@ -601,7 +600,7 @@ struct sdpc_iocb *sdp_iocb_create(void)
 {
 	struct sdpc_iocb *iocb;
 
-	iocb = kmem_cache_alloc(__sdp_iocb_cache, SLAB_KERNEL);
+	iocb = kmem_cache_alloc(sdp_iocb_cache, SLAB_KERNEL);
 	if (iocb) {
 		memset(iocb, 0, sizeof(struct sdpc_iocb));
 		/*
@@ -650,7 +649,7 @@ int sdp_iocb_destroy(struct sdpc_iocb *i
 	/*
 	 * return the object to its cache
 	 */
-	kmem_cache_free(__sdp_iocb_cache, iocb);
+	kmem_cache_free(sdp_iocb_cache, iocb);
 
 	return 0;
 }
@@ -664,9 +663,9 @@ struct sdpc_iocb *sdp_iocb_q_look(struct
 }
 
 /*
- * _sdp_iocb_q_get - get, and remove, the object at the tables head
+ * sdp_iocb_q_get - get, and remove, the object at the tables head
  */
-static struct sdpc_iocb *_sdp_iocb_q_get(struct sdpc_iocb_q *table, int head)
+static struct sdpc_iocb *sdp_iocb_q_get(struct sdpc_iocb_q *table, int head)
 {
 	struct sdpc_iocb *iocb;
 	struct sdpc_iocb *next;
@@ -701,11 +700,10 @@ static struct sdpc_iocb *_sdp_iocb_q_get
 }
 
 /*
- * _sdp_iocb_q_put - put the IOCB object at the tables tail
+ * sdp_iocb_q_put - put the IOCB object at the tables tail
  */
-static int _sdp_iocb_q_put(struct sdpc_iocb_q *table,
-			   struct sdpc_iocb *iocb,
-			   int head)
+static int sdp_iocb_q_put(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb,
+			  int head)
 {
 	struct sdpc_iocb *next;
 	struct sdpc_iocb *prev;
@@ -739,7 +737,7 @@ static int _sdp_iocb_q_put(struct sdpc_i
  */
 struct sdpc_iocb *sdp_iocb_q_get_tail(struct sdpc_iocb_q *table)
 {
-	return _sdp_iocb_q_get(table, 0);
+	return sdp_iocb_q_get(table, 0);
 }
 
 /*
@@ -747,7 +745,7 @@ struct sdpc_iocb *sdp_iocb_q_get_tail(st
  */
 struct sdpc_iocb *sdp_iocb_q_get_head(struct sdpc_iocb_q *table)
 {
-	return _sdp_iocb_q_get(table, 1);
+	return sdp_iocb_q_get(table, 1);
 }
 
 /*
@@ -755,7 +753,7 @@ struct sdpc_iocb *sdp_iocb_q_get_head(st
  */
 int sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb)
 {
-	return _sdp_iocb_q_put(table, iocb, 0);
+	return sdp_iocb_q_put(table, iocb, 0);
 }
 
 /*
@@ -763,15 +761,13 @@ int sdp_iocb_q_put_tail(struct sdpc_iocb
  */
 int sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb)
 {
-	return _sdp_iocb_q_put(table, iocb, 1);
+	return sdp_iocb_q_put(table, iocb, 1);
 }
 
 /*
  * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
  */
-void sdp_iocb_q_cancel(struct sdpc_iocb_q *table,
-		       u32     mask,
-		       ssize_t comp)
+void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp)
 {
 	struct sdpc_iocb *iocb;
 	struct sdpc_iocb *next;
@@ -844,7 +840,7 @@ int sdp_main_iocb_init(void)
 	/*
 	 * initialize locking code.
 	 */
-	result =  _sdp_mem_lock_init();
+	result =  sdp_mem_lock_init();
 	if (result < 0) {
 		sdp_warn("Error <%d> initializing memory locking.", result);
 		return result;
@@ -852,23 +848,23 @@ int sdp_main_iocb_init(void)
 	/*
 	 * initialize the caches only once.
 	 */
-	if (__sdp_iocb_cache) {
+	if (sdp_iocb_cache) {
 		sdp_warn("IOCB caches already initialized.");
 		return -EINVAL;
 	}
 
-	__sdp_iocb_cache = kmem_cache_create("sdp_iocb",
+	sdp_iocb_cache = kmem_cache_create("sdp_iocb",
 					     sizeof(struct sdpc_iocb),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (!__sdp_iocb_cache) {
+	if (!sdp_iocb_cache) {
 		result = -ENOMEM;
 		goto error_iocb_c;
 	}
 
 	return 0;
 error_iocb_c:
-	(void)_sdp_mem_lock_cleanup();
+	(void)sdp_mem_lock_cleanup();
 	return result;
 }
 
@@ -881,13 +877,13 @@ void sdp_main_iocb_cleanup(void)
 	/*
 	 * cleanup the caches
 	 */
-	kmem_cache_destroy(__sdp_iocb_cache);
+	kmem_cache_destroy(sdp_iocb_cache);
 	/*
 	 * null out entries.
 	 */
-	__sdp_iocb_cache = NULL;
+	sdp_iocb_cache = NULL;
 	/*
 	 * cleanup memory locking
 	 */
-	(void)_sdp_mem_lock_cleanup();
+	(void)sdp_mem_lock_cleanup();
 }



From Tom.Duffy at Sun.COM  Mon Feb 28 14:01:31 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:01:31 -0800
Subject: [openib-general] [PATCH][SDP][9/22] Remove use of leading
 underscores in static names in sdp_iocb.h
In-Reply-To: <11096280813891@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com>
Message-ID: <11096280911941@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_iocb.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_iocb.h	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_iocb.h	(working copy)
@@ -129,9 +129,9 @@ struct sdpc_iocb_q {
  */
 
 /*
- * __sdp_kmap - map a page into kernel space
+ * sdp_kmap - map a page into kernel space
  */
-static inline void *__sdp_kmap(struct page *page)
+static inline void *sdp_kmap(struct page *page)
 {
 	if (in_atomic() || irqs_disabled())
 		return kmap_atomic(page, KM_IRQ0);
@@ -140,9 +140,9 @@ static inline void *__sdp_kmap(struct pa
 }
 
 /*
- * __sdp_kunmap - unmap a page into kernel space
+ * sdp_kunmap - unmap a page into kernel space
  */
-static inline void __sdp_kunmap(struct page *page)
+static inline void sdp_kunmap(struct page *page)
 {
 	if (in_atomic() || irqs_disabled())
 		kunmap_atomic(page, KM_IRQ0);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:01:42 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:01:42 -0800
Subject: [openib-general] [PATCH][SDP][10/22] Remove use of leading
 underscores in static names in sdp_link.c
In-Reply-To: <11096280911941@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
Message-ID: <11096281021938@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_link.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_link.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_link.c	(working copy)
@@ -36,18 +36,18 @@
 #include "sdp_main.h"
 #include "sdp_link.h"
 
-static kmem_cache_t *_wait_cache = NULL;
-static kmem_cache_t *_info_cache = NULL;
+static kmem_cache_t *wait_cache = NULL;
+static kmem_cache_t *info_cache = NULL;
 
-static struct sdp_path_info *_info_list = NULL;
+static struct sdp_path_info *info_list = NULL;
 
-static struct workqueue_struct *_link_wq;
-static struct work_struct       _link_timer;
+static struct workqueue_struct *link_wq;
+static struct work_struct       link_timer;
 
-static u64 _path_lookup_id = 0;
+static u64 path_lookup_id = 0;
 
 #define _SDP_PATH_LOOKUP_ID() \
-      ((++_path_lookup_id) ? _path_lookup_id : ++_path_lookup_id)
+      ((++path_lookup_id) ? path_lookup_id : ++path_lookup_id)
 
 #define GID_FMT           "%x:%x:%x:%x:%x:%x:%x:%x"
 #define GID_ARG(gid)      be16_to_cpup((__be16 *) ((gid).raw +  0)), \
@@ -61,23 +61,22 @@ static u64 _path_lookup_id = 0;
 /*
  * proto
  */
-void _sdp_link_path_lookup(void *data);
+static void do_link_path_lookup(void *data);
 
 /*
- * _sdp_link_path_complete - generate a path record completion for user
+ * sdp_link_path_complete - generate a path record completion for user
  */
-static void _sdp_link_path_complete(u64 id,
-				    int status,
-				    struct sdp_path_info *info,
-				    void (*func)(u64 id,
-						 int status,
-						 u32 dst_addr,
-						 u32 src_addr,
-						 u8  hw_port,
-						 struct ib_device *ca,
-						 struct ib_sa_path_rec *path,
-						 void *arg),
-				    void *arg)
+static void sdp_link_path_complete(u64 id, int status,
+				   struct sdp_path_info *info,
+				   void (*func)(u64 id,
+						int status,
+						u32 dst_addr,
+						u32 src_addr,
+						u8  hw_port,
+						struct ib_device *ca,
+						struct ib_sa_path_rec *path,
+						void *arg),
+				   void *arg)
 {
 	/*
 	 * call completion function
@@ -95,10 +94,10 @@ static void _sdp_link_path_complete(u64 
 }
 
 /*
- * _sdp_path_wait_add - add a wait entry into the wait list for a path
+ * sdp_path_wait_add - add a wait entry into the wait list for a path
  */
-static void _sdp_path_wait_add(struct sdp_path_info *info,
-			       struct sdp_path_wait *wait)
+static void sdp_path_wait_add(struct sdp_path_info *info,
+			      struct sdp_path_wait *wait)
 {
 
 	wait->next      = info->wait_list;
@@ -110,9 +109,9 @@ static void _sdp_path_wait_add(struct sd
 }
 
 /*
- * _sdp_path_wait_destroy - destroy an entry for a wait element
+ * sdp_path_wait_destroy - destroy an entry for a wait element
  */
-static void _sdp_path_wait_destroy(struct sdp_path_wait *wait)
+static void sdp_path_wait_destroy(struct sdp_path_wait *wait)
 {
 	/*
 	 * if it's in the list, pext will not be null
@@ -127,33 +126,32 @@ static void _sdp_path_wait_destroy(struc
 		wait->next = NULL;
 	}
 
-	kmem_cache_free(_wait_cache, wait);
+	kmem_cache_free(wait_cache, wait);
 }
 
 /*
- * _sdp_path_wait_complete - complete an entry for a wait element
+ * sdp_path_wait_complete - complete an entry for a wait element
  */
-static void _sdp_path_wait_complete(struct sdp_path_wait *wait,
-				   struct sdp_path_info *info,
-				   int status)
+static void sdp_path_wait_complete(struct sdp_path_wait *wait,
+				   struct sdp_path_info *info, int status)
 {
-	_sdp_link_path_complete(wait->id, 
+	sdp_link_path_complete(wait->id, 
 				status,
 				info,
 				wait->completion,
 				wait->arg);
 
-	_sdp_path_wait_destroy(wait);
+	sdp_path_wait_destroy(wait);
 }
 
 /*
- * _sdp_path_info_lookup - lookup a path record entry
+ * sdp_path_info_lookup - lookup a path record entry
  */
-static struct sdp_path_info *_sdp_path_info_lookup(u32 dst_ip, int dev_if)
+static struct sdp_path_info *sdp_path_info_lookup(u32 dst_ip, int dev_if)
 {
 	struct sdp_path_info *info;
 
-	for (info = _info_list; info; info = info->next)
+	for (info = info_list; info; info = info->next)
 		if (dst_ip == info->dst && dev_if == info->dif)
 			break;
 
@@ -161,21 +159,21 @@ static struct sdp_path_info *_sdp_path_i
 }
 
 /*
- * _sdp_path_info_create - create an entry for a path record element
+ * sdp_path_info_create - create an entry for a path record element
  */
-static struct sdp_path_info *_sdp_path_info_create(u32 dst_ip, int dev_if)
+static struct sdp_path_info *sdp_path_info_create(u32 dst_ip, int dev_if)
 {
 	struct sdp_path_info *info;
 
-	info = kmem_cache_alloc(_info_cache, SLAB_KERNEL);
+	info = kmem_cache_alloc(info_cache, SLAB_KERNEL);
 	if (!info)
 		return NULL;
 
 	memset(info, 0, sizeof(struct sdp_path_info));
 
-	info->next = _info_list;
-	_info_list = info;
-	info->pext = &_info_list;
+	info->next = info_list;
+	info_list = info;
+	info->pext = &info_list;
 
 	if (info->next)
 		info->next->pext = &info->next;
@@ -186,15 +184,15 @@ static struct sdp_path_info *_sdp_path_i
 
 	info->sa_time  = SDP_LINK_SA_TIME_MIN;
 	info->arp_time = SDP_LINK_ARP_TIME_MIN;
-	INIT_WORK(&info->timer, _sdp_link_path_lookup, info);
+	INIT_WORK(&info->timer, do_link_path_lookup, info);
 
 	return info;
 }
 
 /*
- * _sdp_path_info_destroy - destroy an entry for a path record element
+ * sdp_path_info_destroy - destroy an entry for a path record element
  */
-static void _sdp_path_info_destroy(struct sdp_path_info *info, int status)
+static void sdp_path_info_destroy(struct sdp_path_info *info, int status)
 {
 	struct sdp_path_wait *wait;
 	/*
@@ -211,18 +209,17 @@ static void _sdp_path_info_destroy(struc
 	}
 
 	while ((wait = info->wait_list))
-		_sdp_path_wait_complete(wait, info, status);
+		sdp_path_wait_complete(wait, info, status);
 
 	cancel_delayed_work(&info->timer);
-	kmem_cache_free(_info_cache, info);
+	kmem_cache_free(info_cache, info);
 }
 
 /*
- * _sdp_link_path_rec_done - path record completion function
+ * sdp_link_path_rec_done - path record completion function
  */
-static void _sdp_link_path_rec_done(int status,
-				    struct ib_sa_path_rec *resp,
-				    void *context)
+static void sdp_link_path_rec_done(int status, struct ib_sa_path_rec *resp,
+				   void *context)
 {
 	struct sdp_path_info *info = (struct sdp_path_info *)context;
 	struct sdp_path_wait *wait;
@@ -254,7 +251,7 @@ static void _sdp_link_path_rec_done(int 
 			wait->retry++;
 
 		if (!status || SDP_LINK_SA_RETRY < wait->retry)
-			_sdp_path_wait_complete(wait, info, status);
+			sdp_path_wait_complete(wait, info, status);
 	}
 	/*
 	 * retry if anyone is waiting.
@@ -271,22 +268,22 @@ static void _sdp_link_path_rec_done(int 
 					     IB_SA_PATH_REC_NUMB_PATH),
 					    info->sa_time, 
 					    GFP_KERNEL,
-					    _sdp_link_path_rec_done,
+					    sdp_link_path_rec_done,
 					    info,
 					    &info->query);
 
 	        if (result < 0) {
 			sdp_dbg_warn(NULL, "Error <%d> restarting path query",
 				     result);
-			_sdp_path_info_destroy(info, result);
+			sdp_path_info_destroy(info, result);
 		}
 	}
 }
 
 /*
- * _sdp_link_path_rec_get - resolve GIDs to a path record
+ * sdp_link_path_rec_get - resolve GIDs to a path record
  */
-static int _sdp_link_path_rec_get(struct sdp_path_info *info)
+static int sdp_link_path_rec_get(struct sdp_path_info *info)
 {
 	int result;
 
@@ -306,7 +303,7 @@ static int _sdp_link_path_rec_get(struct
 				     IB_SA_PATH_REC_NUMB_PATH),
 				    info->sa_time,
 				    GFP_KERNEL,
-				    _sdp_link_path_rec_done,
+				    sdp_link_path_rec_done,
 				    info,
 				    &info->query);
         if (result < 0) {
@@ -322,9 +319,9 @@ static int _sdp_link_path_rec_get(struct
 }
 
 /*
- * _sdp_link_path_lookup - resolve an ip address to a path record
+ * do_link_path_lookup - resolve an ip address to a path record
  */
-void _sdp_link_path_lookup(void *data)
+static void do_link_path_lookup(void *data)
 {
 	struct sdp_path_info *info = data;
 	struct ipoib_dev_priv *priv;
@@ -488,8 +485,7 @@ arp:
 			sweep = sweep->next;
 
 			if (SDP_LINK_SA_RETRY < wait->retry++)
-				_sdp_path_wait_complete(wait, info, 
-							-ETIMEDOUT);
+				sdp_path_wait_complete(wait, info, -ETIMEDOUT);
 		}
 
 		if (!info->wait_list) {
@@ -502,11 +498,11 @@ arp:
 	}
 
 	info->flags |= SDP_LINK_F_ARP;
-	queue_delayed_work(_link_wq, &info->timer, info->arp_time);
+	queue_delayed_work(link_wq, &info->timer, info->arp_time);
 
 	return;
 path:
-	result = _sdp_link_path_rec_get(info);
+	result = sdp_link_path_rec_get(info);
 	if (result) {
 		sdp_dbg_warn(NULL, "Error <%d> getting path record.", result);
 		goto error;
@@ -514,7 +510,7 @@ path:
 done:
 	return;
 error:
-	_sdp_path_info_destroy(info, result);
+	sdp_path_info_destroy(info, result);
 }
 
 /*
@@ -546,9 +542,9 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	/*
 	 * lookup entry, create if not found and add to wait list.
 	 */
-	info = _sdp_path_info_lookup(dst_addr, bound_dev_if);
+	info = sdp_path_info_lookup(dst_addr, bound_dev_if);
 	if (!info) {
-		info = _sdp_path_info_create(dst_addr, bound_dev_if);
+		info = sdp_path_info_create(dst_addr, bound_dev_if);
 		if (!info) {
 			sdp_dbg_warn(NULL, "Failed to create path object");
 			return -ENOMEM;
@@ -561,13 +557,13 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	 * if not waiting for result, complete.
 	 */
 	if (info->flags & SDP_LINK_F_VALID) {
-		_sdp_link_path_complete(*id, 0, info, completion, arg);
+		sdp_link_path_complete(*id, 0, info, completion, arg);
 		return 0;
 	}
 	/*
 	 * add request to list of lookups.
 	 */
-	wait = kmem_cache_alloc(_wait_cache, SLAB_KERNEL);
+	wait = kmem_cache_alloc(wait_cache, SLAB_KERNEL);
 	if (!wait) {
 		sdp_dbg_warn(NULL, "Failed to create path wait object");
 		result = -ENOMEM;
@@ -578,12 +574,12 @@ int sdp_link_path_lookup(u32 dst_addr,  
 	wait->completion = completion;
 	wait->arg        = arg;
 
-	_sdp_path_wait_add(info, wait);
+	sdp_path_wait_add(info, wait);
 	/*
 	 * initiate address lookup, if not in progress.
 	 */
 	if (!((SDP_LINK_F_ARP|SDP_LINK_F_PATH) & info->flags))
-		_sdp_link_path_lookup(info);
+		do_link_path_lookup(info);
 
 	return 0;
 error:
@@ -595,14 +591,14 @@ error:
  */
 
 /*
- * _sdp_link_sweep - periodic path information cleanup function
+ * sdp_link_sweep - periodic path information cleanup function
  */
-static void _sdp_link_sweep(void *data)
+static void sdp_link_sweep(void *data)
 {
 	struct sdp_path_info *info;
 	struct sdp_path_info *sweep;
 
-	sweep = _info_list; 
+	sweep = info_list; 
 	while (sweep) {
 		info  = sweep;
 		sweep = sweep->next;
@@ -616,11 +612,11 @@ static void _sdp_link_sweep(void *data)
 				     (info->dst & 0xff000000) >> 24,
 				     jiffies, info->use);
 
-			_sdp_path_info_destroy(info, -ETIMEDOUT);
+			sdp_path_info_destroy(info, -ETIMEDOUT);
 		}
 	}
 
-	queue_delayed_work(_link_wq, &_link_timer, SDP_LINK_SWEEP_INTERVAL);
+	queue_delayed_work(link_wq, &link_timer, SDP_LINK_SWEEP_INTERVAL);
 }
 
 /*
@@ -628,9 +624,9 @@ static void _sdp_link_sweep(void *data)
  */
 
 /*
- * _sdp_link_arp_work - handle IB REQUEST/REPLY ARP packets
+ * sdp_link_arp_work - handle IB REQUEST/REPLY ARP packets
  */
-static void _sdp_link_arp_work(void *data)
+static void sdp_link_arp_work(void *data)
 {
 	struct sdp_work *work = (struct sdp_work *)data;
 	struct sk_buff *skb = (struct sk_buff *)work->arg;
@@ -649,7 +645,7 @@ static void _sdp_link_arp_work(void *dat
 	/*
 	 * find a path info structure for the source IP address.
 	 */
-	for (info = _info_list; info; info = info->next)
+	for (info = info_list; info; info = info->next)
 		if (info->dst == arp->src_ip)
 			break;
 
@@ -665,10 +661,10 @@ static void _sdp_link_arp_work(void *dat
 
 	memcpy(&info->path.dgid, &arp->src_gid, sizeof(union ib_gid));
 
-	result = _sdp_link_path_rec_get(info);
+	result = sdp_link_path_rec_get(info);
 	if (result) {
 		sdp_dbg_warn(NULL, "Error <%d> path request on ARP.", result);
-		_sdp_path_info_destroy(info, result);
+		sdp_path_info_destroy(info, result);
 	}
 
 done:
@@ -677,11 +673,10 @@ done:
 }
 
 /*
- * _sdp_link_arp_recv - receive all ARP packets
+ * sdp_link_arp_recv - receive all ARP packets
  */
-static int _sdp_link_arp_recv(struct sk_buff *skb,
-			      struct net_device *dev,
-			      struct packet_type *pt)
+static int sdp_link_arp_recv(struct sk_buff *skb, struct net_device *dev,
+			     struct packet_type *pt)
 {
 	struct sdp_work *work;
 	struct arphdr *arp_hdr;
@@ -700,8 +695,8 @@ static int _sdp_link_arp_recv(struct sk_
 		goto done;
 
 	work->arg = skb;
-	INIT_WORK(&work->work, _sdp_link_arp_work, work);
-	queue_work(_link_wq, &work->work);
+	INIT_WORK(&work->work, sdp_link_arp_work, work);
+	queue_work(link_wq, &work->work);
 
 	return 0;
 done:
@@ -712,9 +707,9 @@ done:
 /*
  * primary initialization/cleanup functions
  */
-static struct packet_type _sdp_arp_type = {
+static struct packet_type sdp_arp_type = {
 	.type           = __constant_htons(ETH_P_ARP),
-	.func           = _sdp_link_arp_recv,
+	.func           = sdp_link_arp_recv,
 	.af_packet_priv = (void*) 1, /* understand shared skbs */
 };
 
@@ -739,50 +734,50 @@ int sdp_link_addr_init(void)
 		goto error_async;
 	}
 #endif
-	_info_cache = kmem_cache_create("sdp_path_info",
+	info_cache = kmem_cache_create("sdp_path_info",
 					sizeof(struct sdp_path_info),
 					0, SLAB_HWCACHE_ALIGN,
 					NULL, NULL);
-	if (!_info_cache) {
+	if (!info_cache) {
 		sdp_warn("Failed to allocate path info cache.");
 
 		result = -ENOMEM;
 		goto error_path;
 	}
 
-	_wait_cache = kmem_cache_create("sdp_path_wait",
+	wait_cache = kmem_cache_create("sdp_path_wait",
 					sizeof(struct sdp_path_wait),
 					0, SLAB_HWCACHE_ALIGN,
 					NULL, NULL);
-	if (!_wait_cache) {
+	if (!wait_cache) {
 		sdp_warn("Failed to allocate path wait cache.");
 
 		result = -ENOMEM;
 		goto error_wait;
 	}
 
-	_link_wq = create_workqueue("sdp_wq");
-	if (!_link_wq) {
+	link_wq = create_workqueue("sdp_wq");
+	if (!link_wq) {
 		sdp_warn("Failed to allocate ARP wait queue.");
 
 		result = -ENOMEM;
 		goto error_wq;
 	}
 	
-	INIT_WORK(&_link_timer, _sdp_link_sweep, NULL);
-	queue_delayed_work(_link_wq, &_link_timer, SDP_LINK_SWEEP_INTERVAL);
+	INIT_WORK(&link_timer, sdp_link_sweep, NULL);
+	queue_delayed_work(link_wq, &link_timer, SDP_LINK_SWEEP_INTERVAL);
 	/*
 	 * install device for receiving ARP packets in parallel to the normal
 	 * Linux ARP, this will be the SDP notifier that an ARP request has
 	 * completed.
 	 */
-	dev_add_pack(&_sdp_arp_type);
+	dev_add_pack(&sdp_arp_type);
 	
 	return 0;
 error_wq:
-	kmem_cache_destroy(_wait_cache);
+	kmem_cache_destroy(wait_cache);
 error_wait:
-	kmem_cache_destroy(_info_cache);
+	kmem_cache_destroy(info_cache);
 error_path:
 	return result;
 }
@@ -798,23 +793,23 @@ int sdp_link_addr_cleanup(void)
 	/*
 	 * clear objects
 	 */
-	while ((info = _info_list))
-		_sdp_path_info_destroy(info, -EINTR);
+	while ((info = info_list))
+		sdp_path_info_destroy(info, -EINTR);
 	/*
 	 * remove ARP packet processing.
 	 */
-	dev_remove_pack(&_sdp_arp_type);
+	dev_remove_pack(&sdp_arp_type);
 	/*
 	 * destroy work queue
 	 */
-	cancel_delayed_work(&_link_timer);
-	flush_workqueue(_link_wq);
-	destroy_workqueue(_link_wq);
+	cancel_delayed_work(&link_timer);
+	flush_workqueue(link_wq);
+	destroy_workqueue(link_wq);
 	/*
 	 * destroy caches
 	 */
-	kmem_cache_destroy(_info_cache);
-	kmem_cache_destroy(_wait_cache);
+	kmem_cache_destroy(info_cache);
+	kmem_cache_destroy(wait_cache);
 #if 0
 	/*
 	 * release async event handler



From Tom.Duffy at Sun.COM  Mon Feb 28 14:01:52 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:01:52 -0800
Subject: [openib-general] [PATCH][SDP][11/22] Remove use of leading
 underscores in static names in sdp_msgs.h
In-Reply-To: <11096281021938@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
Message-ID: <11096281121662@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_msgs.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_msgs.h	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_msgs.h	(working copy)
@@ -276,9 +276,9 @@ struct msg_hdr_sch {
  */
 
 /*
- * __sdp_msg_swap_bsdh - SDP header endian byte swapping
+ * sdp_msg_swap_bsdh - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
+static inline void sdp_msg_swap_bsdh(struct msg_hdr_bsdh *header)
 {
 	header->recv_bufs = cpu_to_be16(header->recv_bufs);
 	header->size = cpu_to_be32(header->size);
@@ -287,9 +287,9 @@ static inline void __sdp_msg_swap_bsdh(s
 }
 
 /*
- * __sdp_msg_swap_hh - SDP header endian byte swapping
+ * sdp_msg_swap_hh - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_hh(struct msg_hdr_hh *header)
+static inline void sdp_msg_swap_hh(struct msg_hdr_hh *header)
 {
 	header->r_rcv_size = cpu_to_be32(header->r_rcv_size);
 	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
@@ -305,17 +305,17 @@ static inline void __sdp_msg_swap_hh(str
 }
 
 /*
- * __sdp_msg_swap_hah - SDP header endian byte swapping
+ * sdp_msg_swap_hah - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_hah(struct msg_hdr_hah *header)
+static inline void sdp_msg_swap_hah(struct msg_hdr_hah *header)
 {
 	header->l_rcv_size = cpu_to_be32(header->l_rcv_size);
 }
 
 /*
- * __sdp_msg_swap_srcah - SDP header endian byte swapping
+ * sdp_msg_swap_srcah - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
+static inline void sdp_msg_swap_srcah(struct msg_hdr_srcah *header)
 {
 	header->size = cpu_to_be32(header->size);
 #ifdef _SDP_MS_APRIL_ERROR_COMPAT
@@ -328,9 +328,9 @@ static inline void __sdp_msg_swap_srcah(
 }
 
 /*
- * __sdp_msg_swap_snkah - SDP header endian byte swapping
+ * sdp_msg_swap_snkah - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
+static inline void sdp_msg_swap_snkah(struct msg_hdr_snkah *header)
 {
 	header->size = cpu_to_be32(header->size);
 	header->r_key = cpu_to_be32(header->r_key);
@@ -339,74 +339,74 @@ static inline void __sdp_msg_swap_snkah(
 }
 
 /*
- * __sdp_msg_swap_rwch - SDP header endian byte swapping
+ * sdp_msg_swap_rwch - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
+static inline void sdp_msg_swap_rwch(struct msg_hdr_rwch *header)
 {
 	header->size = cpu_to_be32(header->size);
 }
 
 /*
- * __sdp_msg_swap_rrch - SDP header endian byte swapping
+ * sdp_msg_swap_rrch - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
+static inline void sdp_msg_swap_rrch(struct msg_hdr_rrch *header)
 {
 	header->size = cpu_to_be32(header->size);
 }
 
 /*
- * __sdp_msg_swap_mch - SDP header endian byte swapping
+ * sdp_msg_swap_mch - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_mch(struct msg_hdr_mch *header)
+static inline void sdp_msg_swap_mch(struct msg_hdr_mch *header)
 {
 }
 
 /*
- * __sdp_msg_swap_crbh - SDP header endian byte swapping
+ * sdp_msg_swap_crbh - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
+static inline void sdp_msg_swap_crbh(struct msg_hdr_crbh *header)
 {
 	header->size = cpu_to_be32(header->size);
 }
 
 /*
- * __sdp_msg_swap_crbah - SDP header endian byte swapping
+ * sdp_msg_swap_crbah - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
+static inline void sdp_msg_swap_crbah(struct msg_hdr_crbah *header)
 {
 	header->size = cpu_to_be32(header->size);
 }
 
 /*
- * __sdp_msg_swap_sch - SDP header endian byte swapping
+ * sdp_msg_swap_sch - SDP header endian byte swapping
  */
-static inline void __sdp_msg_swap_sch(struct msg_hdr_sch *header)
+static inline void sdp_msg_swap_sch(struct msg_hdr_sch *header)
 {
 	header->service_id = cpu_to_be64(header->service_id);
 }
 
-#define _sdp_msg_cpu_to_net_bsdh  __sdp_msg_swap_bsdh
-#define _sdp_msg_net_to_cpu_bsdh  __sdp_msg_swap_bsdh
-#define _sdp_msg_cpu_to_net_hh    __sdp_msg_swap_hh
-#define _sdp_msg_net_to_cpu_hh    __sdp_msg_swap_hh
-#define _sdp_msg_cpu_to_net_hah   __sdp_msg_swap_hah
-#define _sdp_msg_net_to_cpu_hah   __sdp_msg_swap_hah
-#define _sdp_msg_cpu_to_net_srcah __sdp_msg_swap_srcah
-#define _sdp_msg_net_to_cpu_srcah __sdp_msg_swap_srcah
-#define _sdp_msg_cpu_to_net_snkah __sdp_msg_swap_snkah
-#define _sdp_msg_net_to_cpu_snkah __sdp_msg_swap_snkah
-#define _sdp_msg_cpu_to_net_rwch  __sdp_msg_swap_rwch
-#define _sdp_msg_net_to_cpu_rwch  __sdp_msg_swap_rwch
-#define _sdp_msg_cpu_to_net_rrch  __sdp_msg_swap_rrch
-#define _sdp_msg_net_to_cpu_rrch  __sdp_msg_swap_rrch
-#define _sdp_msg_cpu_to_net_mch   __sdp_msg_swap_mch
-#define _sdp_msg_net_to_cpu_mch   __sdp_msg_swap_mch
-#define _sdp_msg_cpu_to_net_crbh  __sdp_msg_swap_crbh
-#define _sdp_msg_net_to_cpu_crbh  __sdp_msg_swap_crbh
-#define _sdp_msg_cpu_to_net_crbah __sdp_msg_swap_crbah
-#define _sdp_msg_net_to_cpu_crbah __sdp_msg_swap_crbah
-#define _sdp_msg_cpu_to_net_sch   __sdp_msg_swap_sch
-#define _sdp_msg_net_to_cpu_sch   __sdp_msg_swap_sch
+#define sdp_msg_cpu_to_net_bsdh  sdp_msg_swap_bsdh
+#define sdp_msg_net_to_cpu_bsdh  sdp_msg_swap_bsdh
+#define sdp_msg_cpu_to_net_hh    sdp_msg_swap_hh
+#define sdp_msg_net_to_cpu_hh    sdp_msg_swap_hh
+#define sdp_msg_cpu_to_net_hah   sdp_msg_swap_hah
+#define sdp_msg_net_to_cpu_hah   sdp_msg_swap_hah
+#define sdp_msg_cpu_to_net_srcah sdp_msg_swap_srcah
+#define sdp_msg_net_to_cpu_srcah sdp_msg_swap_srcah
+#define sdp_msg_cpu_to_net_snkah sdp_msg_swap_snkah
+#define sdp_msg_net_to_cpu_snkah sdp_msg_swap_snkah
+#define sdp_msg_cpu_to_net_rwch  sdp_msg_swap_rwch
+#define sdp_msg_net_to_cpu_rwch  sdp_msg_swap_rwch
+#define sdp_msg_cpu_to_net_rrch  sdp_msg_swap_rrch
+#define sdp_msg_net_to_cpu_rrch  sdp_msg_swap_rrch
+#define sdp_msg_cpu_to_net_mch   sdp_msg_swap_mch
+#define sdp_msg_net_to_cpu_mch   sdp_msg_swap_mch
+#define sdp_msg_cpu_to_net_crbh  sdp_msg_swap_crbh
+#define sdp_msg_net_to_cpu_crbh  sdp_msg_swap_crbh
+#define sdp_msg_cpu_to_net_crbah sdp_msg_swap_crbah
+#define sdp_msg_net_to_cpu_crbah sdp_msg_swap_crbah
+#define sdp_msg_cpu_to_net_sch   sdp_msg_swap_sch
+#define sdp_msg_net_to_cpu_sch   sdp_msg_swap_sch
 
 /*
  * Miscellaneous message related informtation



From Tom.Duffy at Sun.COM  Mon Feb 28 14:02:02 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:02:02 -0800
Subject: [openib-general] [PATCH][SDP][12/22] Remove use of leading
 underscores in static names in sdp_pass.c
In-Reply-To: <11096281121662@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com> <11096281121662@sun.com>
Message-ID: <11096281222561@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_pass.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_pass.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_pass.c	(working copy)
@@ -145,7 +145,7 @@ done:
 /*
  * Functions to handle incomming passive connection requests. (REQ)
  */
-static int _sdp_cm_accept(struct sdp_opt *conn)
+static int sdp_cm_accept(struct sdp_opt *conn)
 {
 	struct ib_cm_rep_param param;
 	struct sdp_msg_hello_ack *hello_ack;
@@ -199,8 +199,8 @@ static int _sdp_cm_accept(struct sdp_opt
 	/*
 	 * endian swap
 	 */
-	_sdp_msg_cpu_to_net_bsdh(&hello_ack->bsdh);
-	_sdp_msg_cpu_to_net_hah(&hello_ack->hah);
+	sdp_msg_cpu_to_net_bsdh(&hello_ack->bsdh);
+	sdp_msg_cpu_to_net_hah(&hello_ack->hah);
 	/*
 	 * save message
 	 */
@@ -288,7 +288,7 @@ error:
 	return result;
 }
 
-static int _sdp_cm_listen_lookup(struct sdp_opt *conn)
+static int sdp_cm_listen_lookup(struct sdp_opt *conn)
 {
 	struct sdp_opt *listen_conn;
 	struct sock *listen_sk;
@@ -377,7 +377,7 @@ static int _sdp_cm_listen_lookup(struct 
 	/*
 	 * initiate a CM response message.
 	 */
-	result = _sdp_cm_accept(conn);
+	result = sdp_cm_accept(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> CM connect accept", result);
 		goto locked_err;
@@ -404,13 +404,13 @@ lookup_err:
 	return result;
 }
 
-static int _sdp_cm_hello_check(struct sdp_msg_hello *msg_hello)
+static int sdp_cm_hello_check(struct sdp_msg_hello *msg_hello)
 {
 	/*
 	 * endian swap
 	 */
-	_sdp_msg_net_to_cpu_bsdh(&msg_hello->bsdh);
-	_sdp_msg_net_to_cpu_hh(&msg_hello->hh);
+	sdp_msg_net_to_cpu_bsdh(&msg_hello->bsdh);
+	sdp_msg_net_to_cpu_hh(&msg_hello->hh);
 	/*
 	 * validation and consistency checks
 	 */
@@ -484,7 +484,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	/*
 	 * check Hello Header, to determine if we want the connection.
 	 */
-	result = _sdp_cm_hello_check(msg_hello);
+	result = sdp_cm_hello_check(msg_hello);
 	if (result < 0) {
 		sdp_dbg_warn(NULL, "Error <%d> validating hello msg. <%08x>",
 			     result, cm_id->local_id);
@@ -534,7 +534,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 * update CM context to refer to the connection.
 	 */
 	conn->cm_id          = cm_id;
-	conn->cm_id->context = __hashent_arg(conn->hashent);
+	conn->cm_id->context = hashent_arg(conn->hashent);
 	/*
 	 * associate connection with a hca/port, and allocate IB.
 	 */
@@ -559,7 +559,7 @@ int sdp_cm_req_handler(struct ib_cm_id *
 	 * Find a matching listening socket, and insert new connection 
 	 * into listeners accept queue.
 	 */
-	result = _sdp_cm_listen_lookup(conn);
+	result = sdp_cm_listen_lookup(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> matching listen socket queue",
 			     result);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:02:12 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:02:12 -0800
Subject: [openib-general] [PATCH][SDP][13/22] Remove use of leading
 underscores in static names in sdp_post.c
In-Reply-To: <11096281222561@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
Message-ID: <11096281321386@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_post.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_post.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_post.c	(working copy)
@@ -85,9 +85,9 @@ int sdp_cm_listen_stop(struct sdev_root 
 }
 
 /*
- * _sdp_cm_disconnect - initiate a disconnect request using the CM
+ * do_cm_disconnect - initiate a disconnect request using the CM
  */
-static void _sdp_cm_disconnect(void *arg)
+static void do_cm_disconnect(void *arg)
 {
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
@@ -109,9 +109,9 @@ static void _sdp_cm_disconnect(void *arg
 }
 
 /*
- * _sdp_cm_reject - initiate a reject request using the CM
+ * do_cm_reject - initiate a reject request using the CM
  */
-static void _sdp_cm_reject(void *arg)
+static void do_cm_reject(void *arg)
 {
 	struct sdp_opt *conn = (struct sdp_opt *)arg;
 	int result;
@@ -132,10 +132,10 @@ static void _sdp_cm_reject(void *arg)
 	sdp_conn_put(conn);
 }
 /*
- * _sdp_cm_deferred_generic - initiate a defered request using the CM
+ * sdp_cm_deferred_generic - initiate a defered request using the CM
  */
-static int _sdp_cm_deferred_generic(struct sdp_opt *conn,
-				    void (*defer_func)(void *arg))
+static int sdp_cm_deferred_generic(struct sdp_opt *conn,
+				   void (*defer_func)(void *arg))
 {
 	/*
 	 * send a potentially defered failed request.
@@ -156,7 +156,7 @@ static int _sdp_cm_deferred_generic(stru
 */
 int sdp_cm_disconnect(struct sdp_opt *conn)
 {
-	return _sdp_cm_deferred_generic(conn, _sdp_cm_disconnect);
+	return sdp_cm_deferred_generic(conn, do_cm_disconnect);
 }
 
 /*
@@ -164,5 +164,5 @@ int sdp_cm_disconnect(struct sdp_opt *co
  */
 int sdp_cm_reject(struct sdp_opt *conn)
 {
-	return _sdp_cm_deferred_generic(conn, _sdp_cm_reject);
+	return sdp_cm_deferred_generic(conn, do_cm_reject);
 }



From Tom.Duffy at Sun.COM  Mon Feb 28 14:02:23 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:02:23 -0800
Subject: [openib-general] [PATCH][SDP][14/22] Remove use of leading
 underscores in static names in sdp_proc.c
In-Reply-To: <11096281321386@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
Message-ID: <11096281432614@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_proc.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_proc.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_proc.c	(working copy)
@@ -34,22 +34,18 @@
 
 #include "sdp_main.h"
 
-static const char _dir_name_root[] = SDP_PROC_DIR_NAME;
-static struct proc_dir_entry *_dir_root = NULL;
+static const char dir_name_root[] = SDP_PROC_DIR_NAME;
+static struct proc_dir_entry *dir_root = NULL;
 
 /*
  * Generic static functions used by read/write functions
  */
 
 /*
- * _sdp_proc_read_parse - read function for the injection table
+ * sdp_proc_read_parse - read function for the injection table
  */
-static int _sdp_proc_read_parse(char *page,
-				char **start, 
-				off_t offset,
-				int count,
-				int *eof,
-				void *data)
+static int sdp_proc_read_parse(char *page, char **start, off_t offset,
+			       int count, int *eof, void *data)
 {
 	struct sdpc_proc_ent *sub_entry = (struct sdpc_proc_ent *)data;
 	long end_index = 0;
@@ -142,15 +138,15 @@ int sdp_main_proc_cleanup(void)
 	for (counter = 0; counter < SDP_PROC_ENTRIES; counter++) {
 		sub_entry = &_file_entry_list[counter];
 		if (sub_entry->entry) {
-			remove_proc_entry(sub_entry->name, _dir_root);
+			remove_proc_entry(sub_entry->name, dir_root);
 			sub_entry->entry = NULL;
 		}
 	}
 	/*
 	 * remove SDP directory
 	 */
-	remove_proc_entry(_dir_name_root, proc_net);
-	_dir_root = NULL;
+	remove_proc_entry(dir_name_root, proc_net);
+	dir_root = NULL;
 
 	sdp_dbg_init("/proc filesystem cleanup complete.");
 
@@ -177,21 +173,20 @@ int sdp_main_proc_init(void)
 		return -EFAULT;
 	}
 
-	if (_dir_root) {
+	if (dir_root) {
 		sdp_warn("/proc already initialized!");
 		return -EINVAL;
 	}
 	/*
 	 * create a gateway root, and main directories
 	 */
-	_dir_root = proc_mkdir(_dir_name_root, proc_net);
-	if (!_dir_root) {
-		sdp_warn("Failed to create <%s> proc entry.",
-			 _dir_name_root);
+	dir_root = proc_mkdir(dir_name_root, proc_net);
+	if (!dir_root) {
+		sdp_warn("Failed to create <%s> proc entry.", dir_name_root);
 		return -EINVAL;
 	}
 
-	_dir_root->owner = THIS_MODULE;
+	dir_root->owner = THIS_MODULE;
 
 	for (counter = 0; counter < SDP_PROC_ENTRIES; counter++) {
 		sub_entry = &_file_entry_list[counter];
@@ -202,7 +197,7 @@ int sdp_main_proc_init(void)
 
 		sub_entry->entry = create_proc_entry(sub_entry->name,
 						     S_IRUGO | S_IWUGO,
-						     _dir_root);
+						     dir_root);
 		if (!sub_entry->entry) {
 			sdp_warn("Failed to create <%s> framework proc entry.",
 				 sub_entry->name);
@@ -210,7 +205,7 @@ int sdp_main_proc_init(void)
 			goto error;
 		}
 
-		sub_entry->entry->read_proc = _sdp_proc_read_parse;
+		sub_entry->entry->read_proc = sdp_proc_read_parse;
 		sub_entry->entry->data = sub_entry;
 		sub_entry->entry->owner = THIS_MODULE;
 	}



From Tom.Duffy at Sun.COM  Mon Feb 28 14:02:33 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:02:33 -0800
Subject: [openib-general] [PATCH][SDP][15/22] Remove use of leading
 underscores in static names in sdp_proto.h
In-Reply-To: <11096281432614@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com> <11096281432614@sun.com>
Message-ID: <11096281532803@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_proto.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_proto.h	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_proto.h	(working copy)
@@ -476,7 +476,7 @@ extern int sdp_debug_level;
 #define __SDP_DEBUG_WARN  4
 #define __SDP_DEBUG_ERROR 3
 
-#define __sdp_dbg_out(level, type, format, arg...) \
+#define sdp_dbg_out(level, type, format, arg...) \
         do { \
                 if (!(level > sdp_debug_level)) { \
                         printk("<%d>%s: " format "\n", \
@@ -484,17 +484,17 @@ extern int sdp_debug_level;
                 } \
         } while (0)
 
-#define __sdp_conn_dbg(level, type, conn, format, arg...) \
+#define sdp_conn_dbg(level, type, conn, format, arg...) \
         do { \
                 struct sdp_opt *x = (conn); \
                 if (x) { \
-                        __sdp_dbg_out(level, type, \
+                        sdp_dbg_out(level, type, \
                                       "<%d> <%04x:%04x> " format, \
                                        x->hashent, x->istate, x->state, \
                                        ## arg);                  \
                 } \
                 else {  \
-                        __sdp_dbg_out(level, type, format, ## arg); \
+                        sdp_dbg_out(level, type, format, ## arg); \
                 } \
         } while (0)
 
@@ -502,21 +502,21 @@ extern int sdp_debug_level;
 #define sdp_dbg_data(conn, format, arg...) do { } while (0)
 #else
 #define sdp_dbg_data(conn, format, arg...) \
-        __sdp_conn_dbg(__SDP_DEBUG_DATA, "DATA", conn, format, ## arg)
+        sdp_conn_dbg(__SDP_DEBUG_DATA, "DATA", conn, format, ## arg)
 #endif
 
 #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_CTRL
 #define sdp_dbg_ctrl(conn, format, arg...) do { } while (0)
 #else
 #define sdp_dbg_ctrl(conn, format, arg...) \
-        __sdp_conn_dbg(__SDP_DEBUG_CTRL, "CRTL", conn, format, ## arg)
+        sdp_conn_dbg(__SDP_DEBUG_CTRL, "CRTL", conn, format, ## arg)
 #endif
 
 #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_NOTE
 #define sdp_dbg_warn(conn, format, arg...) do { } while (0)
 #else
 #define sdp_dbg_warn(conn, format, arg...) \
-        __sdp_conn_dbg(__SDP_DEBUG_NOTE, "WARN", conn, format, ## arg)
+        sdp_conn_dbg(__SDP_DEBUG_NOTE, "WARN", conn, format, ## arg)
 #endif
 
 
@@ -524,21 +524,21 @@ extern int sdp_debug_level;
 #define sdp_dbg_init(format, arg...) do { } while (0)
 #else
 #define sdp_dbg_init(format, arg...) \
-        __sdp_dbg_out(__SDP_DEBUG_INIT, "INIT: ", format, ## arg)
+        sdp_dbg_out(__SDP_DEBUG_INIT, "INIT: ", format, ## arg)
 #endif
 
 #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_WARN
 #define sdp_dbg_err(format, arg...) do { } while (0)
 #else
 #define sdp_dbg_err(format, arg...)  \
-        __sdp_dbg_out(__SDP_DEBUG_WARN, " ERR: ", format, ## arg)
+        sdp_dbg_out(__SDP_DEBUG_WARN, " ERR: ", format, ## arg)
 #endif
 
 #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_WARN
 #define sdp_warn(format, arg...) do { } while (0)
 #else
 #define sdp_warn(format, arg...)     \
-        __sdp_dbg_out(__SDP_DEBUG_WARN, "WARN: ", format, ## arg)
+        sdp_dbg_out(__SDP_DEBUG_WARN, "WARN: ", format, ## arg)
 #endif
 
 #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_CTRL
@@ -558,9 +558,9 @@ do {                                    
  */
 
 /*
- * __sdp_inet_write_space - writable space on send side
+ * sdp_inet_write_space - writable space on send side
  */
-static inline int __sdp_inet_write_space(struct sdp_opt *conn, int urg)
+static inline int sdp_inet_write_space(struct sdp_opt *conn, int urg)
 {
 	int size;
 
@@ -582,21 +582,21 @@ static inline int __sdp_inet_write_space
 }
 
 /*
- * __sdp_inet_writable - return non-zero if socket is writable
+ * sdp_inet_writable - return non-zero if socket is writable
  */
-static inline int __sdp_inet_writable(struct sdp_opt *conn)
+static inline int sdp_inet_writable(struct sdp_opt *conn)
 {
 	if (SDP_ST_MASK_OPEN & conn->istate)
-		return (__sdp_inet_write_space(conn, 0) <
+		return (sdp_inet_write_space(conn, 0) <
 			(conn->send_qud / 2)) ? 0 : 1;
 	else
 		return 0;
 }
 
 /*
- * __sdp_conn_stat_dump - dump stats to the log
+ * sdp_conn_stat_dump - dump stats to the log
  */
-static inline int __sdp_conn_stat_dump(struct sdp_opt *conn)
+static inline int sdp_conn_stat_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATS_REC
 	int counter;
@@ -617,9 +617,9 @@ static inline int __sdp_conn_stat_dump(s
 }
 
 /*
- * __sdp_conn_state_dump - dump state information to the log
+ * sdp_conn_state_dump - dump state information to the log
  */
-static inline int __sdp_conn_state_dump(struct sdp_opt *conn)
+static inline int sdp_conn_state_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATE_REC
 	int counter;



From Tom.Duffy at Sun.COM  Mon Feb 28 14:02:43 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:02:43 -0800
Subject: [openib-general] [PATCH][SDP][16/22] Remove use of leading
 underscores in static names in sdp_queue.c
In-Reply-To: <11096281532803@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
Message-ID: <1109628163952@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_queue.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_queue.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_queue.c	(working copy)
@@ -39,10 +39,9 @@
  */
 
 /*
- * _sdp_desc_q_get - Get an element from a specific table
+ * sdp_desc_q_get - Get an element from a specific table
  */
-static struct sdpc_desc *_sdp_desc_q_get(struct sdpc_desc_q *table,
-					 int fifo)
+static struct sdpc_desc *sdp_desc_q_get(struct sdpc_desc_q *table, int fifo)
 {
 	struct sdpc_desc *element;
 
@@ -75,11 +74,10 @@ static struct sdpc_desc *_sdp_desc_q_get
 }
 
 /*
- * _sdp_desc_q_put - Place an element into a specific table
+ * sdp_desc_q_put - Place an element into a specific table
  */
-static inline int _sdp_desc_q_put(struct sdpc_desc_q *table,
-				  struct sdpc_desc *element, 
-				  int fifo)
+static inline int sdp_desc_q_put(struct sdpc_desc_q *table,
+				 struct sdpc_desc *element, int fifo)
 {
 	/*
 	 * fifo: false == tail, true == head
@@ -169,7 +167,7 @@ struct sdpc_desc *sdp_desc_q_lookup(stru
  */
 struct sdpc_desc *sdp_desc_q_get_head(struct sdpc_desc_q *table)
 {
-	return _sdp_desc_q_get(table, 1);
+	return sdp_desc_q_get(table, 1);
 }
 
 /*
@@ -177,25 +175,23 @@ struct sdpc_desc *sdp_desc_q_get_head(st
  */
 struct sdpc_desc *sdp_desc_q_get_tail(struct sdpc_desc_q *table)
 {
-	return _sdp_desc_q_get(table, 0);
+	return sdp_desc_q_get(table, 0);
 }
 
 /*
  * sdp_desc_q_put_head - Place an element into the head of a table
  */
-int sdp_desc_q_put_head(struct sdpc_desc_q *table,
-				struct sdpc_desc *element)
+int sdp_desc_q_put_head(struct sdpc_desc_q *table, struct sdpc_desc *element)
 {
-	return _sdp_desc_q_put(table, element, 1);
+	return sdp_desc_q_put(table, element, 1);
 }
 
 /*
  * sdp_desc_q_put_tail - Place an element into the tail of a table
  */
-int sdp_desc_q_put_tail(struct sdpc_desc_q *table,
-				struct sdpc_desc *element)
+int sdp_desc_q_put_tail(struct sdpc_desc_q *table, struct sdpc_desc *element)
 {
-	return _sdp_desc_q_put(table, element, 0);
+	return sdp_desc_q_put(table, element, 0);
 }
 
 /*
@@ -221,7 +217,7 @@ int sdp_desc_q_type_head(struct sdpc_des
  * sdp_desc_q_look_type_head - look at a specific object
  */
 struct sdpc_desc *sdp_desc_q_look_type_head(struct sdpc_desc_q *table, 
-					   enum sdp_desc_type type)
+					    enum sdp_desc_type type)
 {
 	if (!table->head)
 		return NULL;
@@ -233,7 +229,7 @@ struct sdpc_desc *sdp_desc_q_look_type_h
  * sdp_desc_q_look_type_tail - look at the type at the end of the table
  */
 struct sdpc_desc *sdp_desc_q_look_type_tail(struct sdpc_desc_q *table,
-					   enum sdp_desc_type type)
+					    enum sdp_desc_type type)
 {
 	if (!table->head)
 		return NULL;
@@ -245,8 +241,7 @@ struct sdpc_desc *sdp_desc_q_look_type_t
 /*
  * sdp_desc_q_types_size - return the number of elements in the table
  */
-int sdp_desc_q_types_size(struct sdpc_desc_q *table,
-				 enum sdp_desc_type type)
+int sdp_desc_q_types_size(struct sdpc_desc_q *table, enum sdp_desc_type type)
 {
 	return ((SDP_DESC_TYPE_NONE > type) ?
 		table->count[type] : -ERANGE);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:02:53 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:02:53 -0800
Subject: [openib-general] [PATCH][SDP][17/22] Remove use of leading
 underscores in static names in sdp_queue.h
In-Reply-To: <1109628163952@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com>
Message-ID: <11096281731236@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_queue.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_queue.h	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_queue.h	(working copy)
@@ -71,22 +71,22 @@ struct sdpc_desc_q {
  */
 
 /*
- * __sdp_desc_q_size - return the number of elements in the table
+ * sdp_desc_q_size - return the number of elements in the table
  */
-static inline int __sdp_desc_q_size(struct sdpc_desc_q *table)
+static inline int sdp_desc_q_size(struct sdpc_desc_q *table)
 {
 	return table->size;
 }
 
 /*
- * __sdp_desc_q_member - return non-zero if element is in a table
+ * sdp_desc_q_member - return non-zero if element is in a table
  */
-static inline int __sdp_desc_q_member(struct sdpc_desc *element)
+static inline int sdp_desc_q_member(struct sdpc_desc *element)
 {
 	return (element->table ? 1 : 0);
 }
 
-#define sdp_desc_q_size(x)   __sdp_desc_q_size(x)
-#define sdp_desc_q_member(x) __sdp_desc_q_member(x)
+#define sdp_desc_q_size(x)   sdp_desc_q_size(x)
+#define sdp_desc_q_member(x) sdp_desc_q_member(x)
 
 #endif /* _SDP_QUEUE_H */



From Tom.Duffy at Sun.COM  Mon Feb 28 14:03:04 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:03:04 -0800
Subject: [openib-general] [PATCH][SDP][18/22] Remove use of leading
 underscores in static names in sdp_rcvd.c
In-Reply-To: <11096281731236@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com> <11096281731236@sun.com>
Message-ID: <11096281843922@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_rcvd.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_rcvd.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_rcvd.c	(working copy)
@@ -38,7 +38,7 @@
  * Specific MID handler functions. (RECV)
  */
 
-static int _sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
 
@@ -114,7 +114,7 @@ error:
 	return result;
 }
 
-static int _sdp_rcvd_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
 
@@ -144,7 +144,7 @@ static int _sdp_rcvd_abort(struct sdp_op
 	return result;
 }
 
-static int _sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -187,7 +187,7 @@ static int _sdp_rcvd_send_sm(struct sdp_
 	return 0;
 }
 
-static int _sdp_rcvd_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rwch *rwch;
 	struct sdpc_iocb *iocb;
@@ -196,7 +196,7 @@ static int _sdp_rcvd_rdma_wr(struct sdp_
 	rwch = (struct msg_hdr_rwch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rwch);
 
-	_sdp_msg_net_to_cpu_rwch(rwch);
+	sdp_msg_net_to_cpu_rwch(rwch);
 	/*
 	 * lookup active IOCB read.
 	 */
@@ -248,7 +248,7 @@ error:
 	return result;
 }
 
-static int _sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rrch *rrch;
 	struct sdpc_iocb *iocb;
@@ -257,7 +257,7 @@ static int _sdp_rcvd_rdma_rd(struct sdp_
 	rrch = (struct msg_hdr_rrch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rrch);
 
-	_sdp_msg_net_to_cpu_rrch(rrch);
+	sdp_msg_net_to_cpu_rrch(rrch);
 	/*
 	 * lookup IOCB read.
 	 */
@@ -330,7 +330,7 @@ error:
 	return result;
 }
 
-static int _sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_mch *mch;
 	int result;
@@ -338,7 +338,7 @@ static int _sdp_rcvd_mode_change(struct 
 	mch = (struct msg_hdr_mch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_mch);
 
-	_sdp_msg_net_to_cpu_mch(mch);
+	sdp_msg_net_to_cpu_mch(mch);
 
 	sdp_dbg_ctrl(conn, "Mode request <%d> from current mode. <%d:%d>",
 		     SDP_MSG_MCH_GET_MODE(mch), conn->recv_mode,
@@ -426,7 +426,7 @@ error:
 	return result;
 }
 
-static int _sdp_rcvd_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_advt *advt;
 	int result;
@@ -516,7 +516,7 @@ done:
 	return result;
 }
 
-static int _sdp_rcvd_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_advt *advt;
 	s32 counter;
@@ -588,10 +588,9 @@ done:
 }
 
 /*
- * _sdp_rcvd_snk_cancel_ack - sink cancel confirmantion
+ * sdp_rcvd_snk_cancel_ack - sink cancel confirmantion
  */
-static int _sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, 
-				    struct sdpc_buff *buff)
+static int sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -627,10 +626,10 @@ done:
 }
 
 /*
- * _sdp_rcvd_resize_buff_ack - buffer size change request
+ * sdp_rcvd_resize_buff_ack - buffer size change request
  */
-static int _sdp_rcvd_resize_buff_ack(struct sdp_opt *conn,
-				     struct sdpc_buff *buff)
+static int sdp_rcvd_resize_buff_ack(struct sdp_opt *conn,
+				    struct sdpc_buff *buff)
 {
 	struct msg_hdr_crbh *crbh;
 	int result;
@@ -638,7 +637,7 @@ static int _sdp_rcvd_resize_buff_ack(str
 	crbh = (struct msg_hdr_crbh *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_crbh);
 
-	_sdp_msg_net_to_cpu_crbh(crbh);
+	sdp_msg_net_to_cpu_crbh(crbh);
 	/*
 	 * request to change our recv buffer size, we're pretty much locked
 	 * into the size we're using, once the connection is set up, so we
@@ -659,24 +658,24 @@ error:
 	return result;
 }
 
-static int _sdp_rcvd_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_sch *sch;
 
 	sch = (struct msg_hdr_sch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_sch);
 
-	_sdp_msg_net_to_cpu_sch(sch);
+	sdp_msg_net_to_cpu_sch(sch);
 
 	return 0;
 }
 
-static int _sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
 }
 
-static int _sdp_rcvd_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_snkah *snkah;
 	struct sdpc_advt *advt;
@@ -686,7 +685,7 @@ static int _sdp_rcvd_snk_avail(struct sd
 	snkah = (struct msg_hdr_snkah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_snkah);
 
-	_sdp_msg_net_to_cpu_snkah(snkah);
+	sdp_msg_net_to_cpu_snkah(snkah);
 
 	sdp_dbg_data(conn, "SnkAvail received. <%d:%d:%016llx> mode <%d>",
 		     snkah->size, snkah->r_key,
@@ -810,7 +809,7 @@ error:
 	return result;
 }
 
-static int _sdp_rcvd_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_srcah *srcah;
 	struct sdpc_advt *advt;
@@ -820,7 +819,7 @@ static int _sdp_rcvd_src_avail(struct sd
 	srcah = (struct msg_hdr_srcah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_srcah);
 
-	_sdp_msg_net_to_cpu_srcah(srcah);
+	sdp_msg_net_to_cpu_srcah(srcah);
 
 	size = buff->tail - buff->data;
 
@@ -978,9 +977,9 @@ done:
 }
 
 /*
- * _sdp_rcvd_data - SDP data message event received
+ * sdp_rcvd_data - SDP data message event received
  */
-static int _sdp_rcvd_data(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_data(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int ret_val;
 
@@ -1023,9 +1022,9 @@ static int _sdp_rcvd_data(struct sdp_opt
 }
 
 /*
- * _sdp_rcvd_unsupported - Valid messages we're not expecting
+ * sdp_rcvd_unsupported - Valid messages we're not expecting
  */
-static int _sdp_rcvd_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_rcvd_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	/*
 	 * Since the gateway only initates RDMA's but is never a target, and
@@ -1052,38 +1051,38 @@ static int _sdp_rcvd_unsupported(struct 
 #define SDP_MSG_EVENT_TABLE_SIZE 0x20
 
 static sdp_event_cb_func recv_event_funcs[SDP_MSG_EVENT_TABLE_SIZE] = {
-	NULL,                       /* SDP_MID_HELLO            0x00 */
-	NULL,                       /* SDP_MID_HELLO_ACK        0x01 */
-	_sdp_rcvd_disconnect,       /* SDP_MID_DISCONNECT       0x02 */
-	_sdp_rcvd_abort,            /* SDP_MID_ABORT_CONN       0x03 */
-	_sdp_rcvd_send_sm,          /* SDP_MID_SEND_SM          0x04 */
-	_sdp_rcvd_rdma_wr,          /* SDP_MID_RDMA_WR_COMP     0x05 */
-	_sdp_rcvd_rdma_rd,          /* SDP_MID_RDMA_RD_COMP     0x06 */
-	_sdp_rcvd_mode_change,      /* SDP_MID_MODE_CHANGE      0x07 */
-	_sdp_rcvd_src_cancel,       /* SDP_MID_SRC_CANCEL       0x08 */
-	_sdp_rcvd_snk_cancel,       /* SDP_MID_SNK_CANCEL       0x09 */
-	_sdp_rcvd_snk_cancel_ack,   /* SDP_MID_SNK_CANCEL_ACK   0x0A */
-	_sdp_rcvd_resize_buff_ack,  /* SDP_MID_CH_RECV_BUF      0x0B */
-	_sdp_rcvd_unsupported,      /* SDP_MID_CH_RECV_BUF_ACK  0x0C */
-	_sdp_rcvd_suspend,          /* SDP_MID_SUSPEND          0x0D */
-	_sdp_rcvd_suspend_ack,      /* SDP_MID_SUSPEND_ACK      0x0E */
-	NULL,                       /* reserved                    0x0F */
-	NULL,                       /* reserved                    0xF0 */
-	NULL,                       /* reserved                    0xF1 */
-	NULL,                       /* reserved                    0xF2 */
-	NULL,                       /* reserved                    0xF3 */
-	NULL,                       /* reserved                    0xF4 */
-	NULL,                       /* reserved                    0xF5 */
-	NULL,                       /* reserved                    0xF6 */
-	NULL,                       /* reserved                    0xF7 */
-	NULL,                       /* reserved                    0xF8 */
-	NULL,                       /* reserved                    0xF9 */
-	NULL,                       /* reserved                    0xFA */
-	NULL,                       /* reserved                    0xFB */
-	NULL,                       /* reserved                    0xFC */
-	_sdp_rcvd_snk_avail,        /* SDP_MID_SNK_AVAIL        0xFD */
-	_sdp_rcvd_src_avail,        /* SDP_MID_SRC_AVAIL        0xFE */
-	_sdp_rcvd_data              /* SDP_MID_DATA             0xFF */
+	NULL,                      /* SDP_MID_HELLO            0x00 */
+	NULL,                      /* SDP_MID_HELLO_ACK        0x01 */
+	sdp_rcvd_disconnect,       /* SDP_MID_DISCONNECT       0x02 */
+	sdp_rcvd_abort,            /* SDP_MID_ABORT_CONN       0x03 */
+	sdp_rcvd_send_sm,          /* SDP_MID_SEND_SM          0x04 */
+	sdp_rcvd_rdma_wr,          /* SDP_MID_RDMA_WR_COMP     0x05 */
+	sdp_rcvd_rdma_rd,          /* SDP_MID_RDMA_RD_COMP     0x06 */
+	sdp_rcvd_mode_change,      /* SDP_MID_MODE_CHANGE      0x07 */
+	sdp_rcvd_src_cancel,       /* SDP_MID_SRC_CANCEL       0x08 */
+	sdp_rcvd_snk_cancel,       /* SDP_MID_SNK_CANCEL       0x09 */
+	sdp_rcvd_snk_cancel_ack,   /* SDP_MID_SNK_CANCEL_ACK   0x0A */
+	sdp_rcvd_resize_buff_ack,  /* SDP_MID_CH_RECV_BUF      0x0B */
+	sdp_rcvd_unsupported,      /* SDP_MID_CH_RECV_BUF_ACK  0x0C */
+	sdp_rcvd_suspend,          /* SDP_MID_SUSPEND          0x0D */
+	sdp_rcvd_suspend_ack,      /* SDP_MID_SUSPEND_ACK      0x0E */
+	NULL,                      /* reserved                 0x0F */
+	NULL,                      /* reserved                 0xF0 */
+	NULL,                      /* reserved                 0xF1 */
+	NULL,                      /* reserved                 0xF2 */
+	NULL,                      /* reserved                 0xF3 */
+	NULL,                      /* reserved                 0xF4 */
+	NULL,                      /* reserved                 0xF5 */
+	NULL,                      /* reserved                 0xF6 */
+	NULL,                      /* reserved                 0xF7 */
+	NULL,                      /* reserved                 0xF8 */
+	NULL,                      /* reserved                 0xF9 */
+	NULL,                      /* reserved                 0xFA */
+	NULL,                      /* reserved                 0xFB */
+	NULL,                      /* reserved                 0xFC */
+	sdp_rcvd_snk_avail,        /* SDP_MID_SNK_AVAIL        0xFD */
+	sdp_rcvd_src_avail,        /* SDP_MID_SRC_AVAIL        0xFE */
+	sdp_rcvd_data              /* SDP_MID_DATA             0xFF */
 };
 
 /*
@@ -1149,7 +1148,7 @@ int sdp_event_recv(struct sdp_opt *conn,
 
 	buff->bsdh_hdr = (struct msg_hdr_bsdh *) buff->data;
 
-        _sdp_msg_net_to_cpu_bsdh(buff->bsdh_hdr);
+        sdp_msg_net_to_cpu_bsdh(buff->bsdh_hdr);
 
 	if (comp->byte_len != buff->bsdh_hdr->size) {
 		sdp_dbg_warn(conn, 
@@ -1190,7 +1189,7 @@ int sdp_event_recv(struct sdp_opt *conn,
 	 * fast path data messages
 	 */
 	if (SDP_MID_DATA == buff->bsdh_hdr->mid)
-		result = _sdp_rcvd_data(conn, buff);
+		result = sdp_rcvd_data(conn, buff);
 	else {
 		offset = buff->bsdh_hdr->mid & 0x1F;
 



From Tom.Duffy at Sun.COM  Mon Feb 28 14:03:14 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:03:14 -0800
Subject: [openib-general] [PATCH][SDP][19/22] Remove use of leading
 underscores in static names in sdp_read.c
In-Reply-To: <11096281843922@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com>
	<11096281731236@sun.com> <11096281843922@sun.com>
Message-ID: <11096281942265@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_read.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_read.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_read.c	(working copy)
@@ -39,9 +39,9 @@
  */
 
 /*
- * _sdp_event_read_advt - RDMA read event handler for source advertisments
+ * sdp_event_read_advt - RDMA read event handler for source advertisments
  */
-static int _sdp_event_read_advt(struct sdp_opt *conn, struct ib_wc *comp)
+static int sdp_event_read_advt(struct sdp_opt *conn, struct ib_wc *comp)
 {
 	struct sdpc_advt *advt;
 	int result;
@@ -255,7 +255,7 @@ int sdp_event_read(struct sdp_opt *conn,
 	/*
 	 * The advertisment which generated this READ needs to be checked.
 	 */
-	result = _sdp_event_read_advt(conn, comp);
+	result = sdp_event_read_advt(conn, comp);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> handling READ advertisment",
 			     result);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:03:24 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:03:24 -0800
Subject: [openib-general] [PATCH][SDP][20/22] Remove use of leading
 underscores in static names in sdp_recv.c
In-Reply-To: <11096281942265@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com>
	<11096281731236@sun.com> <11096281843922@sun.com>
	<11096281942265@sun.com>
Message-ID: <11096282043234@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_recv.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -39,9 +39,9 @@
  */
 
 /*
- * _sdp_post_recv_buff - post a single buffers for data recv
+ * sdp_post_recv_buff - post a single buffers for data recv
  */
-static int _sdp_post_recv_buff(struct sdp_opt *conn)
+static int sdp_post_recv_buff(struct sdp_opt *conn)
 {
 	struct ib_recv_wr receive_param = { NULL };
 	struct ib_recv_wr *bad_wr;
@@ -113,9 +113,9 @@ error:
 }
 
 /*
- * _sdp_post_rdma_buff - post a single buffers for rdma read on a conn
+ * sdp_post_rdma_buff - post a single buffers for rdma read on a conn
  */
-static int _sdp_post_rdma_buff(struct sdp_opt *conn)
+static int sdp_post_rdma_buff(struct sdp_opt *conn)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -220,9 +220,9 @@ done:
 }
 
 /*
- * _sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn
+ * sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn
  */
-static int _sdp_post_rdma_iocb_src(struct sdp_opt *conn)
+static int sdp_post_rdma_iocb_src(struct sdp_opt *conn)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -351,9 +351,9 @@ done:
 }
 
 /*
- * _sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn
+ * sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn
  */
-static int _sdp_post_rdma_iocb_snk(struct sdp_opt *conn)
+static int sdp_post_rdma_iocb_snk(struct sdp_opt *conn)
 {
 	struct sdpc_iocb *iocb;
 	int result = 0;
@@ -450,9 +450,9 @@ error:
 }
 
 /*
- * _sdp_post_rdma - post a rdma based requests for a connection
+ * sdp_post_rdma - post a rdma based requests for a connection
  */
-static int _sdp_post_rdma(struct sdp_opt *conn)
+static int sdp_post_rdma(struct sdp_opt *conn)
 {
 	int result = 0;
 
@@ -470,7 +470,7 @@ static int _sdp_post_rdma(struct sdp_opt
 	if (sdp_advt_q_size(&conn->src_pend) > 0) {
 		if (!sdp_desc_q_types_size(&conn->r_src,
 					       SDP_DESC_TYPE_BUFF))
-			while (!(result = _sdp_post_rdma_iocb_src(conn))) {
+			while (!(result = sdp_post_rdma_iocb_src(conn))) {
 				/*
 				 * pass, nothing to do in loop.
 				 */
@@ -491,7 +491,7 @@ static int _sdp_post_rdma(struct sdp_opt
 			       conn->recv_max >
 			       sdp_buff_q_size(&conn->recv_pool) &&
 			       conn->rwin_max > conn->byte_strm) {
-				result = _sdp_post_rdma_buff(conn);
+				result = sdp_post_rdma_buff(conn);
 				if (result)
 					/*
 					 * No more posts allowed.
@@ -510,7 +510,7 @@ static int _sdp_post_rdma(struct sdp_opt
 		if (sdp_iocb_q_size(&conn->r_pend) > 0 &&
 		    conn->recv_mode == SDP_MODE_PIPE &&
 		    !sdp_advt_q_size(&conn->src_actv))
-			while (!(result = _sdp_post_rdma_iocb_snk(conn))) {
+			while (!(result = sdp_post_rdma_iocb_snk(conn))) {
 				/*
 				 * pass
 				 */
@@ -564,7 +564,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 		      ((s32)conn->recv_cq_size - (s32)conn->l_recv_bf));
 
 	while (counter-- > 0) {
-		result = _sdp_post_recv_buff(conn);
+		result = sdp_post_recv_buff(conn);
 		if (result)
 			/*
 			 * No more recv buffers allowed.
@@ -589,7 +589,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 * data flowing from the remote connection peer to the local
 	 * connection peer.
 	 */
-	result = _sdp_post_rdma(conn);
+	result = sdp_post_rdma(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting RDMAs.", result);
 		goto done;
@@ -632,10 +632,9 @@ done:
  */
 
 /*
- * __sdp_read_buff_iocb - read a SDP buffer into an IOCB
+ * sdp_read_buff_iocb - read a SDP buffer into an IOCB
  */
-static int _sdp_read_buff_iocb(struct sdpc_iocb *iocb,
-			       struct sdpc_buff *buff)
+static int sdp_read_buff_iocb(struct sdpc_iocb *iocb, struct sdpc_buff *buff)
 {
 	unsigned long copy = 0;
 	unsigned long offset;
@@ -665,7 +664,7 @@ static int _sdp_read_buff_iocb(struct sd
 		/*
 		 * map correct page of iocb
 		 */
-		addr = __sdp_kmap(iocb->page_array[counter]);
+		addr = sdp_kmap(iocb->page_array[counter]);
 		if (!addr)
 			break;
 
@@ -683,7 +682,7 @@ static int _sdp_read_buff_iocb(struct sd
 		offset     += copy;
 		offset     &= (~PAGE_MASK);
 		
-		__sdp_kunmap(iocb->page_array[counter++]);
+		sdp_kunmap(iocb->page_array[counter++]);
 	}
 	/*
 	 * restore tail from OOB offset.
@@ -694,10 +693,10 @@ static int _sdp_read_buff_iocb(struct sd
 }
 
 /*
- * _sdp_recv_buff_iocb_active - Ease AIO read pending pressure
+ * sdp_recv_buff_iocb_active - Ease AIO read pending pressure
  */
-static int _sdp_recv_buff_iocb_active(struct sdp_opt *conn,
-				      struct sdpc_buff *buff)
+static int sdp_recv_buff_iocb_active(struct sdp_opt *conn,
+				     struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -716,7 +715,7 @@ static int _sdp_recv_buff_iocb_active(st
 	/*
 	 * TODO: need to be checking OOB here.
 	 */
-	result = _sdp_read_buff_iocb(iocb, buff);
+	result = sdp_read_buff_iocb(iocb, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
@@ -743,10 +742,10 @@ static int _sdp_recv_buff_iocb_active(st
 }
 
 /*
- * _sdp_recv_buff_iocb_pending - Ease AIO read pending pressure
+ * sdp_recv_buff_iocb_pending - Ease AIO read pending pressure
  */
-static int _sdp_recv_buff_iocb_pending(struct sdp_opt *conn,
-				       struct sdpc_buff *buff)
+static int sdp_recv_buff_iocb_pending(struct sdp_opt *conn,
+				      struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -763,7 +762,7 @@ static int _sdp_recv_buff_iocb_pending(s
 	/*
 	 * TODO: need to be checking OOB here.
 	 */
-	result = _sdp_read_buff_iocb(iocb, buff);
+	result = sdp_read_buff_iocb(iocb, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
@@ -861,9 +860,9 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 		 * IOCB.
 		 */
 		if (conn->snk_sent > 0)
-			result = _sdp_recv_buff_iocb_active(conn, buff);
+			result = sdp_recv_buff_iocb_active(conn, buff);
 		else
-			result = _sdp_recv_buff_iocb_pending(conn, buff);
+			result = sdp_recv_buff_iocb_pending(conn, buff);
 		/*
 		 * Check result. Postitive result is data left in the buffer
 		 */
@@ -899,9 +898,9 @@ done:
  */
 
 /*
- * _sdp_read_src_lookup - lookup function for cancelation
+ * sdp_read_src_lookup - lookup function for cancelation
  */
-static int _sdp_read_src_lookup(struct sdpc_desc *element, void *arg)
+static int sdp_read_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
@@ -913,10 +912,9 @@ static int _sdp_read_src_lookup(struct s
 }
 
 /*
- * _sdp_inet_read_cancel - cancel an IO operation
+ * sdp_inet_read_cancel - cancel an IO operation
  */
-static int _sdp_inet_read_cancel(struct kiocb *req,
-				 struct io_event *ev)
+static int sdp_inet_read_cancel(struct kiocb *req, struct io_event *ev)
 {
 	struct sock_iocb *si = kiocb_to_siocb(req);
 	struct sdp_opt   *conn;
@@ -995,7 +993,7 @@ static int _sdp_inet_read_cancel(struct 
 	 * already in flight.
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->r_src,
-						     _sdp_read_src_lookup,
+						     sdp_read_src_lookup,
 						     req);
 	if (iocb) {
 		iocb->flags |= SDP_IOCB_F_CANCEL;
@@ -1045,17 +1043,17 @@ done:
 }
 
 /*
- * _sdp_inet_recv_urg_test - recv queue urgent data cleanup function
+ * sdp_inet_recv_urg_test - recv queue urgent data cleanup function
  */
-static int _sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
+static int sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
 {
 	return ((buff->tail == buff->head) ? 1 : 0);
 }
 
 /*
- * _sdp_inet_recv_urg_trav - recv queue urg data retreival function
+ * sdp_inet_recv_urg_trav - recv queue urg data retreival function
  */
-static int _sdp_inet_recv_urg_trav(struct sdpc_buff *buff, void *arg)
+static int sdp_inet_recv_urg_trav(struct sdpc_buff *buff, void *arg)
 {
 	u8 *value = (u8 *) arg;
 	u8 update;
@@ -1078,12 +1076,10 @@ static int _sdp_inet_recv_urg_trav(struc
 }
 
 /*
- * _sdp_inet_recv_urg - recv urgent data from the network to user space
+ * sdp_inet_recv_urg - recv urgent data from the network to user space
  */
-static int _sdp_inet_recv_urg(struct sock *sk,
-			      struct msghdr *msg,
-			      int size,
-			      int flags)
+static int sdp_inet_recv_urg(struct sock *sk, struct msghdr *msg, int size,
+			     int flags)
 {
 	struct sdp_opt *conn;
 	struct sdpc_buff *buff;
@@ -1105,7 +1101,7 @@ static int _sdp_inet_recv_urg(struct soc
 #endif
 
 	result = sdp_buff_q_trav_head(&conn->recv_pool,
-				      _sdp_inet_recv_urg_trav,
+				      sdp_inet_recv_urg_trav,
 				      (void *)&value);
 	if (result != -ERANGE) {
 		result = result ? result : -EAGAIN;
@@ -1131,7 +1127,7 @@ static int _sdp_inet_recv_urg(struct soc
 			 * if appropriate.
 			 */
 			buff = sdp_buff_q_fetch(&conn->recv_pool,
-						_sdp_inet_recv_urg_test,
+						sdp_inet_recv_urg_test,
 						(void *)0);
 			if (buff) {
 				result = sdp_buff_pool_put(buff);
@@ -1155,11 +1151,8 @@ done:
 /*
  * sdp_inet_recv - recv data from the network to user space
  */
-int sdp_inet_recv(struct kiocb  *req,
-		  struct socket *sock,
-		  struct msghdr *msg, 
-		  size_t size, 
-		  int flags)
+int sdp_inet_recv(struct kiocb  *req, struct socket *sock, struct msghdr *msg, 
+		  size_t size, int flags)
 {
 	struct sock      *sk;
 	struct sdp_opt   *conn;
@@ -1210,7 +1203,7 @@ int sdp_inet_recv(struct kiocb  *req,
 	 * process urgent data
 	 */
 	if (MSG_OOB & flags) {
-		result = _sdp_inet_recv_urg(sk, msg, size, flags);
+		result = sdp_inet_recv_urg(sk, msg, size, flags);
 		copied = (result > 0) ? result : 0;
 		result = (result > 0) ? 0 : result;
 		goto done;
@@ -1467,7 +1460,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			iocb->key  = req->ki_key;
 			iocb->addr = (unsigned long)msg->msg_iov->iov_base;
       
-			req->ki_cancel = _sdp_inet_read_cancel;
+			req->ki_cancel = sdp_inet_read_cancel;
 
 			result = sdp_iocb_lock(iocb);
 			if (result < 0) {



From Tom.Duffy at Sun.COM  Mon Feb 28 14:03:34 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:03:34 -0800
Subject: [openib-general] [PATCH][SDP][21/22] Remove use of leading
 underscores in static names in sdp_send.c
In-Reply-To: <11096282043234@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com>
	<11096281731236@sun.com> <11096281843922@sun.com>
	<11096281942265@sun.com> <11096282043234@sun.com>
Message-ID: <11096282143619@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_send.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_send.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_send.c	(working copy)
@@ -39,9 +39,9 @@
  */
 
 /*
- * _sdp_send_buff_post - Post a buffer send on a SDP connection
+ * sdp_send_buff_post - Post a buffer send on a SDP connection
  */
-static int _sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -65,7 +65,7 @@ static int _sdp_send_buff_post(struct sd
 	/*
 	 * endian swap
 	 */
-	_sdp_msg_cpu_to_net_bsdh(buff->bsdh_hdr);
+	sdp_msg_cpu_to_net_bsdh(buff->bsdh_hdr);
 	/*
 	 * OOB processing. If there is a single OOB byte in flight then the
 	 * pending flag is set as early as possible. IF a second OOB byte
@@ -180,10 +180,9 @@ done:
  */
 
 /*
- * _sdp_send_data_buff_post - Post data for buffered transmission
+ * sdp_send_data_buff_post - Post data for buffered transmission
  */
-static int _sdp_send_data_buff_post(struct sdp_opt *conn,
-				    struct sdpc_buff *buff)
+static int sdp_send_data_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct sdpc_advt *advt;
 	int result;
@@ -278,7 +277,7 @@ static int _sdp_send_data_buff_post(stru
 	/*
 	 * transmision time
 	 */
-	result = _sdp_send_buff_post(conn, buff);
+	result = sdp_send_buff_post(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting send data buffer",
 			     result);
@@ -291,10 +290,9 @@ error:
 }
 
 /*
- * _sdp_send_data_buff_snk - Post data for buffered transmission
+ * sdp_send_data_buff_snk - Post data for buffered transmission
  */
-static int _sdp_send_data_buff_snk(struct sdp_opt *conn,
-				   struct sdpc_buff *buff)
+static int sdp_send_data_buff_snk(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -428,10 +426,9 @@ error:
 }
 
 /*
- * _sdp_send_data_iocb_snk - process a zcopy write advert in the data path
+ * sdp_send_data_iocb_snk - process a zcopy write advert in the data path
  */
-static int _sdp_send_data_iocb_snk(struct sdp_opt *conn,
-				   struct sdpc_iocb *iocb)
+static int sdp_send_data_iocb_snk(struct sdp_opt *conn, struct sdpc_iocb *iocb)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -554,10 +551,9 @@ error:
 }
 
 /*
- * _sdp_send_data_iocb_src - send a zcopy read advert in the data path
+ * sdp_send_data_iocb_src - send a zcopy read advert in the data path
  */
-static int _sdp_send_data_iocb_src(struct sdp_opt *conn,
-				   struct sdpc_iocb *iocb)
+static int sdp_send_data_iocb_src(struct sdp_opt *conn, struct sdpc_iocb *iocb)
 {
 	struct msg_hdr_srcah *src_ah;
 	struct sdpc_buff *buff;
@@ -669,7 +665,7 @@ static int _sdp_send_data_iocb_src(struc
 		/*
 		 * map, copy, unmap.
 		 */
-		addr = __sdp_kmap(iocb->page_array[pos]);
+		addr = sdp_kmap(iocb->page_array[pos]);
 		if (!addr) {
 			result = -ENOMEM;
 			goto error;
@@ -677,7 +673,7 @@ static int _sdp_send_data_iocb_src(struc
 
 		memcpy(buff->tail, (addr + off), len);
 
-		__sdp_kunmap(iocb->page_array[pos]);
+		sdp_kunmap(iocb->page_array[pos]);
 		/*
 		 * update pointers
 		 */
@@ -693,11 +689,11 @@ static int _sdp_send_data_iocb_src(struc
 	/*
 	 * endian swap of extended header
 	 */
-	_sdp_msg_cpu_to_net_srcah(src_ah);
+	sdp_msg_cpu_to_net_srcah(src_ah);
 	/*
 	 * queue/send SrcAvail message
 	 */
-	result = _sdp_send_buff_post(conn, buff);
+	result = sdp_send_buff_post(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting SrcAvail for IOCB <%d>",
 			     result, iocb->key);
@@ -719,11 +715,10 @@ drop:
 }
 
 /*
- * _sdp_send_iocb_buff_write - write part of an iocb into a SDP buffer
+ * sdp_send_iocb_buff_write - write part of an iocb into a SDP buffer
  */
-static int _sdp_send_iocb_buff_write(struct sdpc_iocb *iocb,
-				     struct sdpc_buff *buff)
-
+static int sdp_send_iocb_buff_write(struct sdpc_iocb *iocb,
+				    struct sdpc_buff *buff)
 {
 	unsigned long copy = 0;
 	unsigned long offset;
@@ -741,7 +736,7 @@ static int _sdp_send_iocb_buff_write(str
 		/*
 		 * map correct page of iocb
 		 */
-		addr = __sdp_kmap(iocb->page_array[counter]);
+		addr = sdp_kmap(iocb->page_array[counter]);
 		if (!addr)
 			break;
 		
@@ -760,17 +755,16 @@ static int _sdp_send_iocb_buff_write(str
 		offset += copy;
 		offset &= (~PAGE_MASK);
 
-		__sdp_kunmap(iocb->page_array[counter++]);
+		sdp_kunmap(iocb->page_array[counter++]);
 	}
 
 	return 0;
 }
 
 /*
- * _sdp_send_data_iocb_buff - write multiple SDP buffers from an iocb
+ * sdp_send_data_iocb_buff - write multiple SDP buffers from an iocb
  */
-static int _sdp_send_data_iocb_buff(struct sdp_opt *conn,
-				    struct sdpc_iocb *iocb)
+static int sdp_send_data_iocb_buff(struct sdp_opt *conn, struct sdpc_iocb *iocb)
 {
 	struct sdpc_buff *buff;
 	int result;
@@ -781,7 +775,7 @@ static int _sdp_send_data_iocb_buff(stru
 	/*
 	 * loop through queued buffers and copy them to the destination
 	 */
-	copy = min(__sdp_inet_write_space(conn, 0), iocb->len);
+	copy = min(sdp_inet_write_space(conn, 0), iocb->len);
 
 	while (copy > 0 &&
 	       conn->r_recv_bf > 2 &&
@@ -810,7 +804,7 @@ static int _sdp_send_data_iocb_buff(stru
 		/*
 		 * TODO: need to be checking OOB here.
 		 */
-		result =_sdp_send_iocb_buff_write(iocb, buff);
+		result =sdp_send_iocb_buff_write(iocb, buff);
 		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> copy from IOCB <%d>.",
 				     result, iocb->key);
@@ -825,7 +819,7 @@ static int _sdp_send_data_iocb_buff(stru
 		 * transmision time. An update of send_pipe is not needed,
 		 * since the IOCB queue took care of the increment.
 		 */
-		result = _sdp_send_buff_post(conn, buff);
+		result = sdp_send_buff_post(conn, buff);
 		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> send queue buff post",
 				     result);
@@ -841,9 +835,9 @@ error:
 }
 
 /*
- * _sdp_send_data_iocb - Post IOCB data for transmission
+ * sdp_send_data_iocb - Post IOCB data for transmission
  */
-static int _sdp_send_data_iocb(struct sdp_opt *conn, struct sdpc_iocb *iocb)
+static int sdp_send_data_iocb(struct sdp_opt *conn, struct sdpc_iocb *iocb)
 {
 	int result = ENOBUFS;
 
@@ -858,7 +852,7 @@ static int _sdp_send_data_iocb(struct sd
 	 * determin if we are sending Buffered, Source or Sink.
 	 */
 	if (sdp_advt_q_size(&conn->snk_pend) > 0) {
-		result = _sdp_send_data_iocb_snk(conn, iocb);
+		result = sdp_send_data_iocb_snk(conn, iocb);
 		if (!result) {
 			/*
 			 * IOCB completely processed. Otherwise we allow the
@@ -890,7 +884,7 @@ static int _sdp_send_data_iocb(struct sd
 	if (conn->src_zthresh > iocb->len ||
 	    conn->send_mode == SDP_MODE_BUFF ||
 	    (iocb->flags & SDP_IOCB_F_BUFF)) {
-		result = _sdp_send_data_iocb_buff(conn, iocb);
+		result = sdp_send_data_iocb_buff(conn, iocb);
 		if (!result) {
 			/*
 			 * complete this IOCB
@@ -911,7 +905,7 @@ static int _sdp_send_data_iocb(struct sd
 		goto done;
 	}
 
-	result = _sdp_send_data_iocb_src(conn, iocb);
+	result = sdp_send_data_iocb_src(conn, iocb);
 	if (!result) {
 		/*
 		 * queue IOCB
@@ -931,10 +925,10 @@ done:
 }
 
 /*
- * _sdp_send_data_queue_test - send data buffer if conditions are met
+ * sdp_send_data_queue_test - send data buffer if conditions are met
  */
-static int _sdp_send_data_queue_test(struct sdp_opt *conn,
-				     struct sdpc_desc *element)
+static int sdp_send_data_queue_test(struct sdp_opt *conn,
+				    struct sdpc_desc *element)
 {
 	int result;
 
@@ -948,23 +942,23 @@ static int _sdp_send_data_queue_test(str
 		return ENOBUFS;
 
 	if (element->type == SDP_DESC_TYPE_IOCB)
-		return _sdp_send_data_iocb(conn, (struct sdpc_iocb *)element);
+		return sdp_send_data_iocb(conn, (struct sdpc_iocb *)element);
 
 	if (!sdp_advt_q_look(&conn->snk_pend) ||
 	    (((struct sdpc_buff *)element)->flags & SDP_BUFF_F_OOB_PRES))
-		result = _sdp_send_data_buff_post(conn,
+		result = sdp_send_data_buff_post(conn,
 						  (struct sdpc_buff *)element);
 	else
-		result = _sdp_send_data_buff_snk(conn,
+		result = sdp_send_data_buff_snk(conn,
 						 (struct sdpc_buff *)element);
 
 	return result;
 }
 
 /*
- * _sdp_send_data_queue_flush - Flush data from send queue, to send post
+ * sdp_send_data_queue_flush - Flush data from send queue, to send post
  */
-static int _sdp_send_data_queue_flush(struct sdp_opt *conn)
+static int sdp_send_data_queue_flush(struct sdp_opt *conn)
 {
 	struct sdpc_desc *element;
 	int result = 0;
@@ -978,7 +972,7 @@ static int _sdp_send_data_queue_flush(st
 		element = sdp_desc_q_look_head(&conn->send_queue);
 		SDP_EXPECT((element));
 
-		result = _sdp_send_data_queue_test(conn, element);
+		result = sdp_send_data_queue_test(conn, element);
 		if (!result)
 			continue;
 		/*
@@ -1004,10 +998,9 @@ static int _sdp_send_data_queue_flush(st
 }
 
 /*
- * _sdp_send_data_queue - send using the data queue if necessary
+ * sdp_send_data_queue - send using the data queue if necessary
  */
-static int _sdp_send_data_queue(struct sdp_opt *conn,
-				struct sdpc_desc *element)
+static int sdp_send_data_queue(struct sdp_opt *conn, struct sdpc_desc *element)
 {
 	int result = 0;
 
@@ -1019,7 +1012,7 @@ static int _sdp_send_data_queue(struct s
 	 * value of zero is a successful transmission
 	 */
 	if (sdp_desc_q_size(&conn->send_queue) > 0 ||
-	    (result = _sdp_send_data_queue_test(conn, element)) > 0) {
+	    (result = sdp_send_data_queue_test(conn, element)) > 0) {
 		result = sdp_desc_q_put_tail(&conn->send_queue, element);
 		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> queueing data for send",
@@ -1054,9 +1047,9 @@ done:
 }
 
 /*
- * _sdp_send_data_buff_get - get an appropriate write buffer for send
+ * sdp_send_data_buff_get - get an appropriate write buffer for send
  */
-static inline struct sdpc_buff *_sdp_send_data_buff_get(struct sdp_opt *conn)
+static inline struct sdpc_buff *sdp_send_data_buff_get(struct sdp_opt *conn)
 {
 	struct sdpc_buff *buff;
 
@@ -1079,12 +1072,11 @@ static inline struct sdpc_buff *_sdp_sen
 }
 
 /*
- * _sdp_send_data_buff_put - place a buffer into the send queue
+ * sdp_send_data_buff_put - place a buffer into the send queue
  */
-static inline int _sdp_send_data_buff_put(struct sdp_opt *conn,
-					  struct sdpc_buff *buff,
-					  int size,
-					  int urg)
+static inline int sdp_send_data_buff_put(struct sdp_opt *conn,
+					 struct sdpc_buff *buff, int size,
+					 int urg)
 {
 	int result = 0;
 	int expect;
@@ -1119,7 +1111,7 @@ static inline int _sdp_send_data_buff_pu
 		/*
 		 * finally send the data buffer
 		 */
-		result = _sdp_send_data_queue(conn, (struct sdpc_desc *) buff);
+		result = sdp_send_data_queue(conn, (struct sdpc_desc *) buff);
 		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> buffer to SEND queue.",
 				     result);
@@ -1137,10 +1129,9 @@ static inline int _sdp_send_data_buff_pu
  */
 
 /*
- * _sdp_send_ctrl_buff_test - determine if it's OK to post a control msg
+ * sdp_send_ctrl_buff_test - determine if it's OK to post a control msg
  */
-static int _sdp_send_ctrl_buff_test(struct sdp_opt *conn,
-				    struct sdpc_buff *buff)
+static int sdp_send_ctrl_buff_test(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
 
@@ -1152,7 +1143,7 @@ static int _sdp_send_ctrl_buff_test(stru
 	/*
 	 * post the control buffer
 	 */
-	result = _sdp_send_buff_post(conn, buff);
+	result = sdp_send_buff_post(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control send", result);
 		goto error;
@@ -1164,9 +1155,9 @@ error:
 }
 
 /*
- * _sdp_send_ctrl_buff_flush - Flush control buffers, to send post
+ * sdp_send_ctrl_buff_flush - Flush control buffers, to send post
  */
-static int _sdp_send_ctrl_buff_flush(struct sdp_opt *conn)
+static int sdp_send_ctrl_buff_flush(struct sdp_opt *conn)
 {
 	struct sdpc_desc *element;
 	int result = 0;
@@ -1179,8 +1170,8 @@ static int _sdp_send_ctrl_buff_flush(str
 		element = sdp_desc_q_look_head(&conn->send_ctrl);
 		SDP_EXPECT((element));
 
-		result = _sdp_send_ctrl_buff_test(conn, 
-						  (struct sdpc_buff *)element);
+		result = sdp_send_ctrl_buff_test(conn, 
+						 (struct sdpc_buff *)element);
 		if (!result)
 			continue;
 
@@ -1201,10 +1192,10 @@ static int _sdp_send_ctrl_buff_flush(str
 }
 
 /*
- * _sdp_send_ctrl_buff_buffered - Send a buffered control message
+ * sdp_send_ctrl_buff_buffered - Send a buffered control message
  */
-static int _sdp_send_ctrl_buff_buffered(struct sdp_opt *conn,
-					struct sdpc_buff *buff)
+static int sdp_send_ctrl_buff_buffered(struct sdp_opt *conn,
+				       struct sdpc_buff *buff)
 {
 	int result = 0;
 
@@ -1212,7 +1203,7 @@ static int _sdp_send_ctrl_buff_buffered(
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
 	if (sdp_desc_q_size(&conn->send_ctrl) > 0 ||
-	    (result = _sdp_send_ctrl_buff_test(conn, buff)) > 0) {
+	    (result = sdp_send_ctrl_buff_test(conn, buff)) > 0) {
 		/*
 		 * save the buffer for later flushing into the post queue.
 		 */
@@ -1237,12 +1228,9 @@ error:
 }
 
 /*
- * _sdp_send_ctrl_buff - Create and Send a buffered control message
+ * sdp_send_ctrl_buff - Create and Send a buffered control message
  */
-static int _sdp_send_ctrl_buff(struct sdp_opt *conn,
-			       u8 mid,
-			       int se,
-			       int sig)
+static int sdp_send_ctrl_buff(struct sdp_opt *conn, u8 mid, int se, int sig)
 {
 	int result = 0;
 	struct sdpc_buff *buff;
@@ -1282,7 +1270,7 @@ static int _sdp_send_ctrl_buff(struct sd
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1294,9 +1282,9 @@ error:
 }
 
 /*
- * _sdp_send_ctrl_disconnect - Send a disconnect request
+ * do_send_ctrl_disconnect - Send a disconnect request
  */
-static int _sdp_send_ctrl_disconnect(struct sdp_opt *conn)
+static int do_send_ctrl_disconnect(struct sdp_opt *conn)
 {
 	int result = 0;
 	struct sdpc_buff *buff;
@@ -1330,7 +1318,7 @@ static int _sdp_send_ctrl_disconnect(str
 	/*
 	 * send
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1360,7 +1348,7 @@ int sdp_send_ctrl_disconnect(struct sdp_
 		if (!(conn->flags & SDP_CONN_F_DIS_HOLD) &&
 		    !sdp_desc_q_size(&conn->send_queue) &&
 		    !conn->src_sent)
-			result = _sdp_send_ctrl_disconnect(conn);
+			result = do_send_ctrl_disconnect(conn);
 		else {
 			sdp_dbg_ctrl(conn, "defer disconnect <%d:%d> <%08x>",
 				     sdp_desc_q_size(&conn->send_queue),
@@ -1392,7 +1380,7 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
 	     conn->l_advt_bf > 2))
 		return 0;
 
-	return _sdp_send_ctrl_buff(conn, SDP_MID_DATA, 0, 0);
+	return sdp_send_ctrl_buff(conn, SDP_MID_DATA, 0, 0);
 }
 
 /*
@@ -1400,7 +1388,7 @@ int sdp_send_ctrl_ack(struct sdp_opt *co
  */
 int sdp_send_ctrl_send_sm(struct sdp_opt *conn)
 {
-	return _sdp_send_ctrl_buff(conn, SDP_MID_SEND_SM, 1, 1);
+	return sdp_send_ctrl_buff(conn, SDP_MID_SEND_SM, 1, 1);
 }
 
 /*
@@ -1408,7 +1396,7 @@ int sdp_send_ctrl_send_sm(struct sdp_opt
  */
 int sdp_send_ctrl_src_cancel(struct sdp_opt *conn)
 {
-	return _sdp_send_ctrl_buff(conn, SDP_MID_SRC_CANCEL, 1, 1);
+	return sdp_send_ctrl_buff(conn, SDP_MID_SRC_CANCEL, 1, 1);
 }
 
 /*
@@ -1416,7 +1404,7 @@ int sdp_send_ctrl_src_cancel(struct sdp_
  */
 int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn)
 {
-	return _sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL, 1, 1);
+	return sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL, 1, 1);
 }
 
 /*
@@ -1424,7 +1412,7 @@ int sdp_send_ctrl_snk_cancel(struct sdp_
  */
 int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn)
 {
-	return _sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL_ACK, 1, 1);
+	return sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL_ACK, 1, 1);
 }
 
 /*
@@ -1435,7 +1423,7 @@ int sdp_send_ctrl_abort(struct sdp_opt *
 	/*
 	 * send
 	 */
-	return _sdp_send_ctrl_buff(conn, SDP_MID_ABORT_CONN, 1, 1);
+	return sdp_send_ctrl_buff(conn, SDP_MID_ABORT_CONN, 1, 1);
 }
 
 /*
@@ -1474,11 +1462,11 @@ int sdp_send_ctrl_resize_buff_ack(struct
 	/*
 	 * endian swap of extended header
 	 */
-	_sdp_msg_cpu_to_net_crbah(crbah);
+	sdp_msg_cpu_to_net_crbah(crbah);
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1543,11 +1531,11 @@ int sdp_send_ctrl_rdma_rd(struct sdp_opt
 	/*
 	 * endian swap of extended header
 	 */
-	_sdp_msg_cpu_to_net_rrch(rrch);
+	sdp_msg_cpu_to_net_rrch(rrch);
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1600,11 +1588,11 @@ int sdp_send_ctrl_rdma_wr(struct sdp_opt
 	/*
 	 * endian swap of extended header
 	 */
-	_sdp_msg_cpu_to_net_rwch(rwch);
+	sdp_msg_cpu_to_net_rwch(rwch);
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1618,10 +1606,7 @@ error:
 /*
  * sdp_send_ctrl_snk_avail - Send a sink available message
  */
-int sdp_send_ctrl_snk_avail(struct sdp_opt *conn,
-			    u32 size,
-			    u32 rkey,
-			    u64 addr)
+int sdp_send_ctrl_snk_avail(struct sdp_opt *conn, u32 size, u32 rkey, u64 addr)
 {
 	struct msg_hdr_snkah *snkah;
 	struct sdpc_buff *buff;
@@ -1665,11 +1650,11 @@ int sdp_send_ctrl_snk_avail(struct sdp_o
 	/*
 	 * endian swap of extended header
 	 */
-	_sdp_msg_cpu_to_net_snkah(snkah);
+	sdp_msg_cpu_to_net_snkah(snkah);
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1751,11 +1736,11 @@ int sdp_send_ctrl_mode_ch(struct sdp_opt
 	/*
 	 * endian swap of extended header
 	 */
-	_sdp_msg_cpu_to_net_mch(mch);
+	sdp_msg_cpu_to_net_mch(mch);
 	/*
 	 * Either post a send, or buffer the packet in the tx queue
 	 */
-	result = _sdp_send_ctrl_buff_buffered(conn, buff);
+	result = sdp_send_ctrl_buff_buffered(conn, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting control message",
 			     result);
@@ -1771,9 +1756,9 @@ error:
  */
 
 /*
- * _sdp_write_src_lookup - lookup function for cancelation
+ * sdp_write_src_lookup - lookup function for cancelation
  */
-static int _sdp_write_src_lookup(struct sdpc_desc *element, void *arg)
+static int sdp_write_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
@@ -1785,9 +1770,9 @@ static int _sdp_write_src_lookup(struct 
 }
 
 /*
- * _sdp_inet_write_cancel - cancel an IO operation
+ * sdp_inet_write_cancel - cancel an IO operation
  */
-static int _sdp_inet_write_cancel(struct kiocb *req, struct io_event *ev)
+static int sdp_inet_write_cancel(struct kiocb *req, struct io_event *ev)
 {
 	struct sock_iocb *si = kiocb_to_siocb(req);
 	struct sdp_opt   *conn;
@@ -1817,7 +1802,7 @@ static int _sdp_inet_write_cancel(struct
 	 * whether this is a read or write.
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->send_queue,
-						     _sdp_write_src_lookup,
+						     sdp_write_src_lookup,
 						     req);
 	if (iocb) {
 		/*
@@ -1867,7 +1852,7 @@ static int _sdp_inet_write_cancel(struct
 	 * already in flight.
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->w_snk,
-						     _sdp_write_src_lookup,
+						     sdp_write_src_lookup,
 						     req);
 
 	if (iocb) {
@@ -1927,9 +1912,9 @@ done:
 }
 
 /*
- * _sdp_send_flush_advt - Flush passive sink advertisments
+ * sdp_send_flush_advt - Flush passive sink advertisments
  */
-static int _sdp_send_flush_advt(struct sdp_opt *conn)
+static int sdp_send_flush_advt(struct sdp_opt *conn)
 {
 	struct sdpc_advt *advt;
 	int result;
@@ -1977,7 +1962,7 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * Priority goes to control messages, and we need to follow the
 	 * send credit utilization rules.
 	 */
-	result = _sdp_send_ctrl_buff_flush(conn);
+	result = sdp_send_ctrl_buff_flush(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing control", result);
 		goto done;
@@ -1985,7 +1970,7 @@ int sdp_send_flush(struct sdp_opt *conn)
 	/*
 	 * data flush
 	 */
-	result = _sdp_send_data_queue_flush(conn);
+	result = sdp_send_data_queue_flush(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> flushing data queue", result);
 		goto done;
@@ -1994,7 +1979,7 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * Sink advertisment flush.
 	 */
 	if (sdp_advt_q_size(&conn->snk_pend) > 0) {
-		result = _sdp_send_flush_advt(conn);
+		result = sdp_send_flush_advt(conn);
 		if (result < 0) {
 			sdp_dbg_warn(conn,
 				     "Error <%d> flushing sink advertisments",
@@ -2006,7 +1991,7 @@ int sdp_send_flush(struct sdp_opt *conn)
 	 * disconnect flush
 	 */
 	if (SDP_ST_MASK_DIS_PEND & conn->state) {
-		result = sdp_send_ctrl_disconnect(conn);
+		result = do_send_ctrl_disconnect(conn);
 		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> flushing disconnect",
 				     result);
@@ -2026,9 +2011,7 @@ done:
 /*
  * sdp_inet_send - send data from user space to the network
  */
-int sdp_inet_send(struct kiocb *req,
-		  struct socket *sock,
-		  struct msghdr *msg,
+int sdp_inet_send(struct kiocb *req, struct socket *sock, struct msghdr *msg,
 		  size_t size)
 {
 	struct sock      *sk;
@@ -2088,8 +2071,8 @@ int sdp_inet_send(struct kiocb *req,
 		 * observed...) use a different threshold for urgent
 		 * data to allow some space for sending.
 		 */
-		while (__sdp_inet_write_space(conn, oob) > 0) {
-			buff = _sdp_send_data_buff_get(conn);
+		while (sdp_inet_write_space(conn, oob) > 0) {
+			buff = sdp_send_data_buff_get(conn);
 			if (!buff) {
 				result = -ENOMEM;
 				goto done;
@@ -2097,7 +2080,7 @@ int sdp_inet_send(struct kiocb *req,
 
 			copy = min((size_t)(buff->end - buff->tail),
 				   (size_t)(size - copied));
-			copy = min(copy, __sdp_inet_write_space(conn, oob));
+			copy = min(copy, sdp_inet_write_space(conn, oob));
 
 #ifndef _SDP_DATA_PATH_NULL
 			result = memcpy_fromiovec(buff->tail, 
@@ -2113,9 +2096,9 @@ int sdp_inet_send(struct kiocb *req,
 
 			SDP_CONN_STAT_SEND_INC(conn, copy);
 
-			result = _sdp_send_data_buff_put(conn, buff, copy,
-							 ((copied ==
-							 size) ? oob : 0));
+			result = sdp_send_data_buff_put(conn, buff, copy,
+							((copied ==
+							size) ? oob : 0));
 			if (result < 0)
 				goto done;
 
@@ -2186,7 +2169,7 @@ skip: /* entry point for IOCB based tran
 			clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
 			sdp_conn_unlock(conn);
-			if (__sdp_inet_write_space(conn, oob) <= 0)
+			if (sdp_inet_write_space(conn, oob) <= 0)
 				timeout = schedule_timeout(timeout);
 			sdp_conn_lock(conn);
 
@@ -2213,7 +2196,7 @@ skip: /* entry point for IOCB based tran
 		iocb->key  = req->ki_key;
 		iocb->addr = (unsigned long)msg->msg_iov->iov_base;
       
-		req->ki_cancel = _sdp_inet_write_cancel;
+		req->ki_cancel = sdp_inet_write_cancel;
 
 		result = sdp_iocb_lock(iocb);
 		if (result < 0) {
@@ -2228,7 +2211,7 @@ skip: /* entry point for IOCB based tran
 			
 		conn->send_pipe += iocb->len;
 
-		result = _sdp_send_data_queue(conn, (struct sdpc_desc *)iocb);
+		result = sdp_send_data_queue(conn, (struct sdpc_desc *)iocb);
 		if (result < 0) {
 			sdp_dbg_warn(conn, "Error <%d> queueing write IOCB",
 				     result);



From Tom.Duffy at Sun.COM  Mon Feb 28 14:03:45 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 28 Feb 2005 14:03:45 -0800
Subject: [openib-general] [PATCH][SDP][22/22] Remove use of leading
 underscores in static names in sdp_sent.c
In-Reply-To: <11096282143619@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com>
	<11096281731236@sun.com> <11096281843922@sun.com>
	<11096281942265@sun.com>
	<11096282043234@sun.com> <11096282143619@sun.com>
Message-ID: <11096282251942@sun.com>

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_sent.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_sent.c	(revision 1929)
+++ drivers/infiniband/ulp/sdp/sdp_sent.c	(working copy)
@@ -38,7 +38,7 @@
  * Specific MID handler functions. (SEND)
  */
 
-static int _sdp_sent_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result;
 
@@ -115,7 +115,7 @@ error:
 	return result;
 }
 
-static int _sdp_sent_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_abort(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result;
 
@@ -133,121 +133,120 @@ static int _sdp_sent_abort(struct sdp_op
 	return result;
 }
 
-static int _sdp_sent_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
 }
 
-static int _sdp_sent_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rwch *rwch;
 
 	rwch = (struct msg_hdr_rwch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rwch);
 
-	_sdp_msg_net_to_cpu_rwch(rwch);
+	sdp_msg_net_to_cpu_rwch(rwch);
 
 	return 0;
 }
 
-static int _sdp_sent_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_rrch *rrch;
 
 	rrch = (struct msg_hdr_rrch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_rrch);
 
-	_sdp_msg_net_to_cpu_rrch(rrch);
+	sdp_msg_net_to_cpu_rrch(rrch);
 
 	return 0;
 }
 
-static int _sdp_sent_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_mch *mch;
 
 	mch = (struct msg_hdr_mch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_mch);
 
-	_sdp_msg_net_to_cpu_mch(mch);
+	sdp_msg_net_to_cpu_mch(mch);
 
 	return 0;
 }
 
-static int _sdp_sent_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
 }
 
-static int _sdp_sent_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
 }
 
-static int _sdp_sent_snk_cancel_ack(struct sdp_opt *conn,
-				    struct sdpc_buff *buff)
+static int sdp_sent_snk_cancel_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
 }
 
-static int _sdp_sent_resize_buff_ack(struct sdp_opt *conn,
-				     struct sdpc_buff *buff)
+static int sdp_sent_resize_buff_ack(struct sdp_opt *conn,
+				    struct sdpc_buff *buff)
 {
 	struct msg_hdr_crbah *crbah;
 
 	crbah = (struct msg_hdr_crbah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_crbah);
 
-	_sdp_msg_net_to_cpu_crbah(crbah);
+	sdp_msg_net_to_cpu_crbah(crbah);
 
 	return 0;
 }
 
-static int _sdp_sent_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_suspend(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_sch *sch;
 
 	sch = (struct msg_hdr_sch *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_sch);
 
-	_sdp_msg_net_to_cpu_sch(sch);
+	sdp_msg_net_to_cpu_sch(sch);
 
 	return 0;
 }
 
-static int _sdp_sent_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	return 0;
 }
 
-static int _sdp_sent_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_snkah *snkah;
 
 	snkah = (struct msg_hdr_snkah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_snkah);
 
-	_sdp_msg_net_to_cpu_snkah(snkah);
+	sdp_msg_net_to_cpu_snkah(snkah);
 
 	return 0;
 }
 
-static int _sdp_sent_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	struct msg_hdr_srcah *srcah;
 
 	srcah = (struct msg_hdr_srcah *) buff->data;
 	buff->data = buff->data + sizeof(struct msg_hdr_srcah);
 
-	_sdp_msg_net_to_cpu_srcah(srcah);
+	sdp_msg_net_to_cpu_srcah(srcah);
 
 	return 0;
 }
 
 /*
- * _sdp_sent_data - SDP data message event received
+ * sdp_sent_data - SDP data message event received
  */
-static int _sdp_sent_data(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_data(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	int result = 0;
 
@@ -257,9 +256,9 @@ static int _sdp_sent_data(struct sdp_opt
 }
 
 /*
- * _sdp_sent_unsupported - Valid messages we're not sending
+ * sdp_sent_unsupported - Valid messages we're not sending
  */
-static int _sdp_sent_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
+static int sdp_sent_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff)
 {
 	/*
 	 * Since the gateway only initates RDMA's but is never a target, and
@@ -284,38 +283,38 @@ static int _sdp_sent_unsupported(struct 
  * only those are represented in the table.
  */
 static sdp_event_cb_func send_event_funcs[SDP_MSG_EVENT_TABLE_SIZE] = {
-	NULL,                      /* SDP_MID_HELLO            0x00 */
-	NULL,                      /* SDP_MID_HELLO_ACK        0x01 */
-	_sdp_sent_disconnect,      /* SDP_MID_DISCONNECT       0x02 */
-	_sdp_sent_abort,           /* SDP_MID_ABORT_CONN       0x03 */
-	_sdp_sent_send_sm,         /* SDP_MID_SEND_SM          0x04 */
-	_sdp_sent_rdma_wr,         /* SDP_MID_RDMA_WR_COMP     0x05 */
-	_sdp_sent_rdma_rd,         /* SDP_MID_RDMA_RD_COMP     0x06 */
-	_sdp_sent_mode_change,     /* SDP_MID_MODE_CHANGE      0x07 */
-	_sdp_sent_src_cancel,      /* SDP_MID_SRC_CANCEL       0x08 */
-	_sdp_sent_snk_cancel,      /* SDP_MID_SNK_CANCEL       0x09 */
-	_sdp_sent_snk_cancel_ack,  /* SDP_MID_SNK_CANCEL_ACK   0x0A */
-	_sdp_sent_unsupported,     /* SDP_MID_CH_RECV_BUF      0x0B */
-	_sdp_sent_resize_buff_ack, /* SDP_MID_CH_RECV_BUF_ACK  0x0C */
-	_sdp_sent_suspend,         /* SDP_MID_SUSPEND          0x0D */
-	_sdp_sent_suspend_ack,     /* SDP_MID_SUSPEND_ACK      0x0E */
-	NULL,                      /* reserved                    0x0F */
-	NULL,                      /* reserved                    0xF0 */
-	NULL,                      /* reserved                    0xF1 */
-	NULL,                      /* reserved                    0xF2 */
-	NULL,                      /* reserved                    0xF3 */
-	NULL,                      /* reserved                    0xF4 */
-	NULL,                      /* reserved                    0xF5 */
-	NULL,                      /* reserved                    0xF6 */
-	NULL,                      /* reserved                    0xF7 */
-	NULL,                      /* reserved                    0xF8 */
-	NULL,                      /* reserved                    0xF9 */
-	NULL,                      /* reserved                    0xFA */
-	NULL,                      /* reserved                    0xFB */
-	NULL,                      /* reserved                    0xFC */
-	_sdp_sent_snk_avail,       /* SDP_MID_SNK_AVAIL        0xFD */
-	_sdp_sent_src_avail,       /* SDP_MID_SRC_AVAIL        0xFE */
-	_sdp_sent_data             /* SDP_MID_DATA             0xFF */
+	NULL,                     /* SDP_MID_HELLO            0x00 */
+	NULL,                     /* SDP_MID_HELLO_ACK        0x01 */
+	sdp_sent_disconnect,      /* SDP_MID_DISCONNECT       0x02 */
+	sdp_sent_abort,           /* SDP_MID_ABORT_CONN       0x03 */
+	sdp_sent_send_sm,         /* SDP_MID_SEND_SM          0x04 */
+	sdp_sent_rdma_wr,         /* SDP_MID_RDMA_WR_COMP     0x05 */
+	sdp_sent_rdma_rd,         /* SDP_MID_RDMA_RD_COMP     0x06 */
+	sdp_sent_mode_change,     /* SDP_MID_MODE_CHANGE      0x07 */
+	sdp_sent_src_cancel,      /* SDP_MID_SRC_CANCEL       0x08 */
+	sdp_sent_snk_cancel,      /* SDP_MID_SNK_CANCEL       0x09 */
+	sdp_sent_snk_cancel_ack,  /* SDP_MID_SNK_CANCEL_ACK   0x0A */
+	sdp_sent_unsupported,     /* SDP_MID_CH_RECV_BUF      0x0B */
+	sdp_sent_resize_buff_ack, /* SDP_MID_CH_RECV_BUF_ACK  0x0C */
+	sdp_sent_suspend,         /* SDP_MID_SUSPEND          0x0D */
+	sdp_sent_suspend_ack,     /* SDP_MID_SUSPEND_ACK      0x0E */
+	NULL,                     /* reserved                 0x0F */
+	NULL,                     /* reserved                 0xF0 */
+	NULL,                     /* reserved                 0xF1 */
+	NULL,                     /* reserved                 0xF2 */
+	NULL,                     /* reserved                 0xF3 */
+	NULL,                     /* reserved                 0xF4 */
+	NULL,                     /* reserved                 0xF5 */
+	NULL,                     /* reserved                 0xF6 */
+	NULL,                     /* reserved                 0xF7 */
+	NULL,                     /* reserved                 0xF8 */
+	NULL,                     /* reserved                 0xF9 */
+	NULL,                     /* reserved                 0xFA */
+	NULL,                     /* reserved                 0xFB */
+	NULL,                     /* reserved                 0xFC */
+	sdp_sent_snk_avail,       /* SDP_MID_SNK_AVAIL        0xFD */
+	sdp_sent_src_avail,       /* SDP_MID_SRC_AVAIL        0xFE */
+	sdp_sent_data             /* SDP_MID_DATA             0xFF */
 };
 
 /*



From rminnich at lanl.gov  Mon Feb 28 14:06:57 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 15:06:57 -0700 (MST)
Subject: [openib-general] Question
Message-ID: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>


If ibnetdiscover can do stuff like this: 
hcaguids=0xc074660801c90200
Hca     2 "H-0002c901086674c0"          # MT23108 InfiniHost Mellanox 
Technologies
[1]     "S-0002c90112c08b40"[2]         # lid 0 lmc 0


etc. etc.

i.e., probe all the way to the edge of the network and find things out, 
what could be going on such that opensm won't work at all? I did an svn 
update and complete rebuild friday. But opensm is still totally stuck. 

I have power cycled all switches, and indeed the whole system. I did yank 
(yet another) dead power supply on one mellanox switch, but still .. 
ibnetdiscover is happy, and opensm is not. 

opensm -r does not help. 

I'm basically baffled. What is ibnetdiscover able to do that opensm is not 
able to do?

thanks

ron


From yaronh at voltaire.com  Mon Feb 28 14:11:55 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Tue, 1 Mar 2005 00:11:55 +0200
Subject: [openib-general] Question
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>

Ron, I believe netdiscover uses direct route MADs
So it can work also when the fabric is not fully initialized 

Yaron

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Ronald G. Minnich
> Sent: Tuesday, March 01, 2005 12:07 AM
> To: openib-general at openib.org
> Subject: [openib-general] Question
> 
> 
> If ibnetdiscover can do stuff like this:
> hcaguids=0xc074660801c90200
> Hca     2 "H-0002c901086674c0"          # MT23108 InfiniHost Mellanox
> Technologies
> [1]     "S-0002c90112c08b40"[2]         # lid 0 lmc 0
> 
> 
> etc. etc.
> 
> i.e., probe all the way to the edge of the network and find things
out,
> what could be going on such that opensm won't work at all? I did an
svn
> update and complete rebuild friday. But opensm is still totally stuck.
> 
> I have power cycled all switches, and indeed the whole system. I did
yank
> (yet another) dead power supply on one mellanox switch, but still ..
> ibnetdiscover is happy, and opensm is not.
> 
> opensm -r does not help.
> 
> I'm basically baffled. What is ibnetdiscover able to do that opensm is
not
> able to do?
> 
> thanks
> 
> ron
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-
> general


From halr at voltaire.com  Mon Feb 28 14:21:06 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 17:21:06 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
Message-ID: <1109628966.13555.9910.camel@localhost.localdomain>

Hi Ron,

On Mon, 2005-02-28 at 17:06, Ronald G. Minnich wrote:
> If ibnetdiscover can do stuff like this: 
> hcaguids=0xc074660801c90200
> Hca     2 "H-0002c901086674c0"          # MT23108 InfiniHost Mellanox 
> Technologies
> [1]     "S-0002c90112c08b40"[2]         # lid 0 lmc 0
> 
> 
> etc. etc.
> 
> i.e., probe all the way to the edge of the network and find things out, 
> what could be going on such that opensm won't work at all? I did an svn 
> update and complete rebuild friday. But opensm is still totally stuck. 
> 
> I have power cycled all switches, and indeed the whole system. I did yank 
> (yet another) dead power supply on one mellanox switch, but still .. 
> ibnetdiscover is happy, and opensm is not. 
> 
> opensm -r does not help. 
> 
> I'm basically baffled. What is ibnetdiscover able to do that opensm is not 
> able to do?

What is the definition of "opensm won't work at all ?" I am presuming
that somehow the ports are not brought to active. Is that it or
something else ?

Can you provide the salient part of the logs ? How was OpenSM invoked ?
Was it with -V ?

The difference is that ibnetdiscover only performs gets whereas OpenSM
does both gets and sets.

-- Hal

> thanks
> 
> ron
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Mon Feb 28 14:24:19 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 17:24:19 -0500
Subject: [openib-general] Question
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
Message-ID: <1109628975.13555.9912.camel@localhost.localdomain>

On Mon, 2005-02-28 at 17:11, Yaron Haviv wrote:
> Ron, I believe netdiscover uses direct route MADs

So does OpenSM.

> So it can work also when the fabric is not fully initialized 

OpenSM is responsible for initializing the fabric (and needs to work
with an uninitialized fabric).

-- Hal

> Yaron
> 
> > -----Original Message-----
> > From: openib-general-bounces at openib.org [mailto:openib-general-
> > bounces at openib.org] On Behalf Of Ronald G. Minnich
> > Sent: Tuesday, March 01, 2005 12:07 AM
> > To: openib-general at openib.org
> > Subject: [openib-general] Question
> > 
> > 
> > If ibnetdiscover can do stuff like this:
> > hcaguids=0xc074660801c90200
> > Hca     2 "H-0002c901086674c0"          # MT23108 InfiniHost Mellanox
> > Technologies
> > [1]     "S-0002c90112c08b40"[2]         # lid 0 lmc 0
> > 
> > 
> > etc. etc.
> > 
> > i.e., probe all the way to the edge of the network and find things
> out,
> > what could be going on such that opensm won't work at all? I did an
> svn
> > update and complete rebuild friday. But opensm is still totally stuck.
> > 
> > I have power cycled all switches, and indeed the whole system. I did
> yank
> > (yet another) dead power supply on one mellanox switch, but still ..
> > ibnetdiscover is happy, and opensm is not.
> > 
> > opensm -r does not help.
> > 
> > I'm basically baffled. What is ibnetdiscover able to do that opensm is
> not
> > able to do?
> > 
> > thanks
> > 
> > ron
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> > 
> > To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-
> > general
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Mon Feb 28 14:27:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 17:27:33 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
Message-ID: <1109629112.13555.9915.camel@localhost.localdomain>

On Mon, 2005-02-28 at 17:06, Ronald G. Minnich wrote:
> If ibnetdiscover can do stuff like this: 
> hcaguids=0xc074660801c90200
> Hca     2 "H-0002c901086674c0"          # MT23108 InfiniHost Mellanox 
> Technologies
> [1]     "S-0002c90112c08b40"[2]         # lid 0 lmc 0
> 
> 
> etc. etc.
> 
> i.e., probe all the way to the edge of the network and find things out, 
> what could be going on such that opensm won't work at all? I did an svn 
> update and complete rebuild friday. But opensm is still totally stuck. 

In what state is the subnet stuck in ?

> I have power cycled all switches, and indeed the whole system. I did yank 
> (yet another) dead power supply on one mellanox switch, 

Was this a dead switch or did it have redundant power supplies ?

-- Hal



From tduffy at sun.com  Mon Feb 28 14:36:34 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 28 Feb 2005 14:36:34 -0800
Subject: [openib-general] [PATCHv2][SDP][20/22] Remove use of leading
	underscores in static names in sdp_recv.c
In-Reply-To: <11096282043234@sun.com>
References: <1109627999679@sun.com> <1109628009288@sun.com>
	<11096280203259@sun.com> <11096280301502@sun.com>
	<11096280402005@sun.com>
	<11096280502716@sun.com> <1109628061177@sun.com> <110962807127@sun.com>
	<11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com>
	<11096281121662@sun.com> <11096281222561@sun.com>
	<11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com>
	<11096281731236@sun.com> <11096281843922@sun.com>
	<11096281942265@sun.com> <11096282043234@sun.com>
Message-ID: <1109630194.19720.2.camel@duffman>

This is slightly updated after revision 1930 was checked in.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_recv.c	(revision 1930)
+++ drivers/infiniband/ulp/sdp/sdp_recv.c	(working copy)
@@ -39,9 +39,9 @@
  */
 
 /*
- * _sdp_post_recv_buff - post a single buffers for data recv
+ * sdp_post_recv_buff - post a single buffers for data recv
  */
-static int _sdp_post_recv_buff(struct sdp_opt *conn)
+static int sdp_post_recv_buff(struct sdp_opt *conn)
 {
 	struct ib_recv_wr receive_param = { NULL };
 	struct ib_recv_wr *bad_wr;
@@ -113,9 +113,9 @@ error:
 }
 
 /*
- * _sdp_post_rdma_buff - post a single buffers for rdma read on a conn
+ * sdp_post_rdma_buff - post a single buffers for rdma read on a conn
  */
-static int _sdp_post_rdma_buff(struct sdp_opt *conn)
+static int sdp_post_rdma_buff(struct sdp_opt *conn)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -220,9 +220,9 @@ done:
 }
 
 /*
- * _sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn
+ * sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn
  */
-static int _sdp_post_rdma_iocb_src(struct sdp_opt *conn)
+static int sdp_post_rdma_iocb_src(struct sdp_opt *conn)
 {
 	struct ib_send_wr send_param = { NULL };
 	struct ib_send_wr *bad_wr;
@@ -351,9 +351,9 @@ done:
 }
 
 /*
- * _sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn
+ * sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn
  */
-static int _sdp_post_rdma_iocb_snk(struct sdp_opt *conn)
+static int sdp_post_rdma_iocb_snk(struct sdp_opt *conn)
 {
 	struct sdpc_iocb *iocb;
 	int result = 0;
@@ -450,9 +450,9 @@ error:
 }
 
 /*
- * _sdp_post_rdma - post a rdma based requests for a connection
+ * sdp_post_rdma - post a rdma based requests for a connection
  */
-static int _sdp_post_rdma(struct sdp_opt *conn)
+static int sdp_post_rdma(struct sdp_opt *conn)
 {
 	int result = 0;
 
@@ -470,7 +470,7 @@ static int _sdp_post_rdma(struct sdp_opt
 	if (sdp_advt_q_size(&conn->src_pend) > 0) {
 		if (!sdp_desc_q_types_size(&conn->r_src,
 					       SDP_DESC_TYPE_BUFF))
-			while (!(result = _sdp_post_rdma_iocb_src(conn))) {
+			while (!(result = sdp_post_rdma_iocb_src(conn))) {
 				/*
 				 * pass, nothing to do in loop.
 				 */
@@ -491,7 +491,7 @@ static int _sdp_post_rdma(struct sdp_opt
 			       conn->recv_max >
 			       sdp_buff_q_size(&conn->recv_pool) &&
 			       conn->rwin_max > conn->byte_strm) {
-				result = _sdp_post_rdma_buff(conn);
+				result = sdp_post_rdma_buff(conn);
 				if (result)
 					/*
 					 * No more posts allowed.
@@ -510,7 +510,7 @@ static int _sdp_post_rdma(struct sdp_opt
 		if (sdp_iocb_q_size(&conn->r_pend) > 0 &&
 		    conn->recv_mode == SDP_MODE_PIPE &&
 		    !sdp_advt_q_size(&conn->src_actv))
-			while (!(result = _sdp_post_rdma_iocb_snk(conn))) {
+			while (!(result = sdp_post_rdma_iocb_snk(conn))) {
 				/*
 				 * pass
 				 */
@@ -564,7 +564,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 		      ((s32)conn->recv_cq_size - (s32)conn->l_recv_bf));
 
 	while (counter-- > 0) {
-		result = _sdp_post_recv_buff(conn);
+		result = sdp_post_recv_buff(conn);
 		if (result)
 			/*
 			 * No more recv buffers allowed.
@@ -589,7 +589,7 @@ int sdp_recv_flush(struct sdp_opt *conn)
 	 * data flowing from the remote connection peer to the local
 	 * connection peer.
 	 */
-	result = _sdp_post_rdma(conn);
+	result = sdp_post_rdma(conn);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> posting RDMAs.", result);
 		goto done;
@@ -632,10 +632,9 @@ done:
  */
 
 /*
- * __sdp_read_buff_iocb - read a SDP buffer into an IOCB
+ * sdp_read_buff_iocb - read a SDP buffer into an IOCB
  */
-static int _sdp_read_buff_iocb(struct sdpc_iocb *iocb,
-			       struct sdpc_buff *buff)
+static int sdp_read_buff_iocb(struct sdpc_iocb *iocb, struct sdpc_buff *buff)
 {
 	unsigned long copy = 0;
 	unsigned long offset;
@@ -665,7 +664,7 @@ static int _sdp_read_buff_iocb(struct sd
 		/*
 		 * map correct page of iocb
 		 */
-		addr = __sdp_kmap(iocb->page_array[counter]);
+		addr = sdp_kmap(iocb->page_array[counter]);
 		if (!addr)
 			break;
 
@@ -683,7 +682,7 @@ static int _sdp_read_buff_iocb(struct sd
 		offset     += copy;
 		offset     &= (~PAGE_MASK);
 		
-		__sdp_kunmap(iocb->page_array[counter++]);
+		sdp_kunmap(iocb->page_array[counter++]);
 	}
 	/*
 	 * restore tail from OOB offset.
@@ -694,10 +693,10 @@ static int _sdp_read_buff_iocb(struct sd
 }
 
 /*
- * _sdp_recv_buff_iocb_active - Ease AIO read pending pressure
+ * sdp_recv_buff_iocb_active - Ease AIO read pending pressure
  */
-static int _sdp_recv_buff_iocb_active(struct sdp_opt *conn,
-				      struct sdpc_buff *buff)
+static int sdp_recv_buff_iocb_active(struct sdp_opt *conn,
+				     struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -716,7 +715,7 @@ static int _sdp_recv_buff_iocb_active(st
 	/*
 	 * TODO: need to be checking OOB here.
 	 */
-	result = _sdp_read_buff_iocb(iocb, buff);
+	result = sdp_read_buff_iocb(iocb, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
@@ -743,10 +742,10 @@ static int _sdp_recv_buff_iocb_active(st
 }
 
 /*
- * _sdp_recv_buff_iocb_pending - Ease AIO read pending pressure
+ * sdp_recv_buff_iocb_pending - Ease AIO read pending pressure
  */
-static int _sdp_recv_buff_iocb_pending(struct sdp_opt *conn,
-				       struct sdpc_buff *buff)
+static int sdp_recv_buff_iocb_pending(struct sdp_opt *conn,
+				      struct sdpc_buff *buff)
 {
 	struct sdpc_iocb *iocb;
 	int result;
@@ -763,7 +762,7 @@ static int _sdp_recv_buff_iocb_pending(s
 	/*
 	 * TODO: need to be checking OOB here.
 	 */
-	result = _sdp_read_buff_iocb(iocb, buff);
+	result = sdp_read_buff_iocb(iocb, buff);
 	if (result < 0) {
 		sdp_dbg_warn(conn, "Error <%d> data copy <%d:%u> to IOCB",
 			     result, iocb->len, 
@@ -861,9 +860,9 @@ int sdp_recv_buff(struct sdp_opt *conn, 
 		 * IOCB.
 		 */
 		if (conn->snk_sent > 0)
-			result = _sdp_recv_buff_iocb_active(conn, buff);
+			result = sdp_recv_buff_iocb_active(conn, buff);
 		else
-			result = _sdp_recv_buff_iocb_pending(conn, buff);
+			result = sdp_recv_buff_iocb_pending(conn, buff);
 		/*
 		 * Check result. Postitive result is data left in the buffer
 		 */
@@ -899,9 +898,9 @@ done:
  */
 
 /*
- * _sdp_read_src_lookup - lookup function for cancelation
+ * sdp_read_src_lookup - lookup function for cancelation
  */
-static int _sdp_read_src_lookup(struct sdpc_desc *element, void *arg)
+static int sdp_read_src_lookup(struct sdpc_desc *element, void *arg)
 {
 	struct sdpc_iocb *iocb = (struct sdpc_iocb *) element;
 	struct kiocb *req = (struct kiocb *)arg;
@@ -913,10 +912,9 @@ static int _sdp_read_src_lookup(struct s
 }
 
 /*
- * _sdp_inet_read_cancel - cancel an IO operation
+ * sdp_inet_read_cancel - cancel an IO operation
  */
-static int _sdp_inet_read_cancel(struct kiocb *req,
-				 struct io_event *ev)
+static int sdp_inet_read_cancel(struct kiocb *req, struct io_event *ev)
 {
 	struct sock_iocb *si = kiocb_to_siocb(req);
 	struct sdp_opt   *conn;
@@ -995,7 +993,7 @@ static int _sdp_inet_read_cancel(struct 
 	 * already in flight.
 	 */
 	iocb = (struct sdpc_iocb *)sdp_desc_q_lookup(&conn->r_src,
-						     _sdp_read_src_lookup,
+						     sdp_read_src_lookup,
 						     req);
 	if (iocb) {
 		iocb->flags |= SDP_IOCB_F_CANCEL;
@@ -1045,17 +1043,17 @@ done:
 }
 
 /*
- * _sdp_inet_recv_urg_test - recv queue urgent data cleanup function
+ * sdp_inet_recv_urg_test - recv queue urgent data cleanup function
  */
-static int _sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
+static int sdp_inet_recv_urg_test(struct sdpc_buff *buff, void *arg)
 {
 	return ((buff->tail == buff->head) ? 1 : 0);
 }
 
 /*
- * _sdp_inet_recv_urg_trav - recv queue urg data retreival function
+ * sdp_inet_recv_urg_trav - recv queue urg data retreival function
  */
-static int _sdp_inet_recv_urg_trav(struct sdpc_buff *buff, void *arg)
+static int sdp_inet_recv_urg_trav(struct sdpc_buff *buff, void *arg)
 {
 	u8 *value = (u8 *) arg;
 	u8 update;
@@ -1078,12 +1076,10 @@ static int _sdp_inet_recv_urg_trav(struc
 }
 
 /*
- * _sdp_inet_recv_urg - recv urgent data from the network to user space
+ * sdp_inet_recv_urg - recv urgent data from the network to user space
  */
-static int _sdp_inet_recv_urg(struct sock *sk,
-			      struct msghdr *msg,
-			      int size,
-			      int flags)
+static int sdp_inet_recv_urg(struct sock *sk, struct msghdr *msg, int size,
+			     int flags)
 {
 	struct sdp_opt *conn;
 	struct sdpc_buff *buff;
@@ -1105,7 +1101,7 @@ static int _sdp_inet_recv_urg(struct soc
 #endif
 
 	result = sdp_buff_q_trav_head(&conn->recv_pool,
-				      _sdp_inet_recv_urg_trav,
+				      sdp_inet_recv_urg_trav,
 				      (void *)&value);
 	if (result != -ERANGE) {
 		result = result ? result : -EAGAIN;
@@ -1131,7 +1127,7 @@ static int _sdp_inet_recv_urg(struct soc
 			 * if appropriate.
 			 */
 			buff = sdp_buff_q_fetch(&conn->recv_pool,
-						_sdp_inet_recv_urg_test,
+						sdp_inet_recv_urg_test,
 						(void *)0);
 			if (buff) {
 				result = sdp_buff_pool_put(buff);
@@ -1155,11 +1151,8 @@ done:
 /*
  * sdp_inet_recv - recv data from the network to user space
  */
-int sdp_inet_recv(struct kiocb  *req,
-		  struct socket *sock,
-		  struct msghdr *msg, 
-		  size_t size, 
-		  int flags)
+int sdp_inet_recv(struct kiocb  *req, struct socket *sock, struct msghdr *msg, 
+		  size_t size, int flags)
 {
 	struct sock      *sk;
 	struct sdp_opt   *conn;
@@ -1210,7 +1203,7 @@ int sdp_inet_recv(struct kiocb  *req,
 	 * process urgent data
 	 */
 	if (flags & MSG_OOB) {
-		result = _sdp_inet_recv_urg(sk, msg, size, flags);
+		result = sdp_inet_recv_urg(sk, msg, size, flags);
 		copied = (result > 0) ? result : 0;
 		result = (result > 0) ? 0 : result;
 		goto done;
@@ -1467,7 +1460,7 @@ int sdp_inet_recv(struct kiocb  *req,
 			iocb->key  = req->ki_key;
 			iocb->addr = (unsigned long)msg->msg_iov->iov_base;
       
-			req->ki_cancel = _sdp_inet_read_cancel;
+			req->ki_cancel = sdp_inet_read_cancel;
 
 			result = sdp_iocb_lock(iocb);
 			if (result < 0) {



From rminnich at lanl.gov  Mon Feb 28 14:38:15 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 15:38:15 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
Message-ID: <Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov>



On Tue, 1 Mar 2005, Yaron Haviv wrote:

> Ron, I believe netdiscover uses direct route MADs
> So it can work also when the fabric is not fully initialized 

ok, that makes sense. 

So this brings up another question. ibnetdiscover is plenty fast, and 
opensm is plenty slow. What kind of messaging could I do once i have a 
network all worked out via ibnetdiscover? 

In other words, could I have a usage of infiniband that worked something 
like this;
- ibnetdiscover, save that info
- message layer uses that info and does communications over ib

also, what's the next step to finding the problem here? OK, ibnetdiscover 
works. opensm does not. What tool would you run next to look for problems?

ron


From rminnich at lanl.gov  Mon Feb 28 14:40:15 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 15:40:15 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109628966.13555.9910.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
	<1109628966.13555.9910.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov>



On Mon, 28 Feb 2005, Hal Rosenstock wrote:

> What is the definition of "opensm won't work at all ?" I am presuming
> that somehow the ports are not brought to active. Is that it or
> something else ?

[1109361464:000221781][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.

I got to 13482 of these and gave up.

> Can you provide the salient part of the logs ? How was OpenSM invoked ?
> Was it with -V ?

opensm -v

> The difference is that ibnetdiscover only performs gets whereas OpenSM
> does both gets and sets.

ah!

ron


From rminnich at lanl.gov  Mon Feb 28 14:46:16 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 15:46:16 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109628975.13555.9912.camel@localhost.localdomain>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
	<1109628975.13555.9912.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281540300.18334@enigma.lanl.gov>



On Mon, 28 Feb 2005, Hal Rosenstock wrote:

> OpenSM is responsible for initializing the fabric (and needs to work
> with an uninitialized fabric).

??

You mean, if you have an initialized fabric, opensm can't work? 

Is there useful material on the web that explains this? I keep looking for 
good technical docs on this stuff but keep finding stuff like "IB is the 
future" and "IB replaces fibrechannel" and "IB market is growing" but not 
the kind of good information I keep seeing on this list. 

ron


From rminnich at lanl.gov  Mon Feb 28 14:46:42 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 15:46:42 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109629112.13555.9915.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
	<1109629112.13555.9915.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281546250.18334@enigma.lanl.gov>



On Mon, 28 Feb 2005, Hal Rosenstock wrote:

> In what state is the subnet stuck in ?

how should I tell?

> Was this a dead switch or did it have redundant power supplies ?
redundant; it was not dead.

ron


From halr at voltaire.com  Mon Feb 28 14:48:29 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 17:48:29 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
	<Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov>
Message-ID: <1109630909.13555.9928.camel@localhost.localdomain>

Hi Ron,

On Mon, 2005-02-28 at 17:38, Ronald G. Minnich wrote:
> On Tue, 1 Mar 2005, Yaron Haviv wrote:
> 
> > Ron, I believe netdiscover uses direct route MADs
> > So it can work also when the fabric is not fully initialized 
> 
> ok, that makes sense. 

but it's incorrect.

> So this brings up another question. ibnetdiscover is plenty fast, and 
> opensm is plenty slow.

OpenSM is no slower than ibnetdiscover in terms of sending and receiving
MADs. What you are likely seeing is some set transaction which is not
completing the way it should and this is being retried ad nauseum.
OpenSM is currently bad about this (I'm not sure it ever gives up).
ibnetdiscover gives up after 3 retries (and since it is only doing gets
this is less likely to be a problem). 

>  What kind of messaging could I do once i have a 
> network all worked out via ibnetdiscover? 

Not sure what you mean here by messaging.

> In other words, could I have a usage of infiniband that worked something 
> like this;
> - ibnetdiscover, save that info
> - message layer uses that info and does communications over ib

You can save that info with ibnetdiscover. What kind of messaging would
you want to do ? RC ? UD ? The message layer could utilize this info to
preclude some of the SA queries (not all if there are sets like
multicast or services or traps) but this subverts the SM/SA (and
replicates what it does). Also, if things change, this info would need
to be updated. In the case of this email, I'm pretty sure the SA is not
the issue.

> also, what's the next step to finding the problem here? OK, ibnetdiscover 
> works. opensm does not. What tool would you run next to look for problems?

The opensm log preferably in verbose (-V) mode.

Thanks.

-- Hal



From roland at topspin.com  Mon Feb 28 14:53:46 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 14:53:46 -0800
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281540300.18334@enigma.lanl.gov> (Ronald G.
	Minnich's message of "Mon, 28 Feb 2005 15:46:16 -0700 (MST)")
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
	<1109628975.13555.9912.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281540300.18334@enigma.lanl.gov>
Message-ID: <52y8d8qosl.fsf@topspin.com>

    Ronald> You mean, if you have an initialized fabric, opensm can't work?

No, just that OpenSM has to be able to handle an uninitialized fabric,
since the SM is responsible for initializing the fabric.  However the
SM should work no matter what state the fabric is in.

    Ronald> Is there useful material on the web that explains this? I
    Ronald> keep looking for good technical docs on this stuff but
    Ronald> keep finding stuff like "IB is the future" and "IB
    Ronald> replaces fibrechannel" and "IB market is growing" but not
    Ronald> the kind of good information I keep seeing on this list.

The IB spec is probably the best reference for what the SM does.
Volume 1 Chapter 14 has a very thorough description of subnet management.

 - R.


From roland at topspin.com  Mon Feb 28 14:56:59 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 14:56:59 -0800
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov> (Ronald G.
	Minnich's message of "Mon, 28 Feb 2005 15:38:15 -0700 (MST)")
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
	<Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov>
Message-ID: <52u0nwqon8.fsf@topspin.com>

    Ronald> So this brings up another question. ibnetdiscover is
    Ronald> plenty fast, and opensm is plenty slow. What kind of
    Ronald> messaging could I do once i have a network all worked out
    Ronald> via ibnetdiscover?

Unfortunately, even if you know the topology of your network, the only
messaging you can do is to send directed route management packets.
The subnet manager is responsible for assigning LIDs to ports and
programming the switch forwarding tables.  Without that, no LID-routed
traffic can flow, and directed route traffic is very slow (every
switch in the directed route must forward the packet in firmware).

 - R.


From halr at voltaire.com  Mon Feb 28 14:52:21 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 17:52:21 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281546250.18334@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
	<1109629112.13555.9915.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281546250.18334@enigma.lanl.gov>
Message-ID: <1109631101.13555.9933.camel@localhost.localdomain>

On Mon, 2005-02-28 at 17:46, Ronald G. Minnich wrote:
> On Mon, 28 Feb 2005, Hal Rosenstock wrote:
> 
> > In what state is the subnet stuck in ?
> 
> how should I tell?

opensm logs. Also, I would be curious to see what ibstat showed about
all endport LIDs in the network. Are all the ports active that should be
(plugged into subnet) ? 

> > Was this a dead switch or did it have redundant power supplies ?
> redundant; it was not dead.

Then this should be no problem.

-- Hal

> ron



From halr at voltaire.com  Mon Feb 28 14:55:38 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 17:55:38 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281540300.18334@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
	<1109628975.13555.9912.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281540300.18334@enigma.lanl.gov>
Message-ID: <1109631141.13555.9936.camel@localhost.localdomain>

On Mon, 2005-02-28 at 17:46, Ronald G. Minnich wrote:
> On Mon, 28 Feb 2005, Hal Rosenstock wrote:
> 
> > OpenSM is responsible for initializing the fabric (and needs to work
> > with an uninitialized fabric).
> 
> ??
> 
> You mean, if you have an initialized fabric, opensm can't work? 

No, I mean that OpenSM _has_ to work on an uninitialized fabric as it is
required to bring it from uninitialized to initialized.

> Is there useful material on the web that explains this? I keep looking for 
> good technical docs on this stuff but keep finding stuff like "IB is the 
> future" and "IB replaces fibrechannel" and "IB market is growing" but not 
> the kind of good information I keep seeing on this list. 

There are some good presentations on what an SM/SA needs to do. I will
see what I can dig up and post.

-- Hal



From halr at voltaire.com  Mon Feb 28 15:00:35 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 18:00:35 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
	<1109628966.13555.9910.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov>
Message-ID: <1109631634.13555.9945.camel@localhost.localdomain>

On Mon, 2005-02-28 at 17:40, Ronald G. Minnich wrote:
> On Mon, 28 Feb 2005, Hal Rosenstock wrote:
> 
> > What is the definition of "opensm won't work at all ?" I am presuming
> > that somehow the ports are not brought to active. Is that it or
> > something else ?
> 
> [1109361464:000221781][411FF970] -> umad_receiver: send completed with 
> error(method=1 attr=11) -- dropping.
> 
> I got to 13482 of these and gave up.

That's a Get NodeInfo. So much for my theory. Wonder which node it is
going after that was not responding. What's failing is that this code is
matching the response with the request so either the request does not
make it there or the response does not make it back. This is all with
direct routed MADs.

Also, wasn't that the same failure as a while ago when one of those 96
port switches kept forwarding but didn't terminate MADs ? (Yes, I know
you recycled everything which would seem to be inconsistent with this).

> > Can you provide the salient part of the logs ? How was OpenSM invoked ?
> > Was it with -V ?
> 
> opensm -v

I'm not sure we will see anything more from the logs than the error. Is
there anything before the first error ? Can you also rerun with -V and
look for the same ?

I think I will need the output from ibnetdiscover and the failing direct
route header from the request. (I work on generating a patch for this).

-- Hal

> > The difference is that ibnetdiscover only performs gets whereas OpenSM
> > does both gets and sets.
> 
> ah!
> 
> ron



From rminnich at lanl.gov  Mon Feb 28 15:10:28 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 16:10:28 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109630909.13555.9928.camel@localhost.localdomain>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com> 
	<Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov>
	<1109630909.13555.9928.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281607460.18334@enigma.lanl.gov>



On Mon, 28 Feb 2005, Hal Rosenstock wrote:

> The opensm log preferably in verbose (-V) mode.

It's a long log :-)

Here is the point at which it goes to pieces.


[1109360115:000609200][40BFF970] -> osm_pkey_rcv_process: Got 
GetResp(PKey) block:1 port_num 1 with GUID = 0x2c90108d192e0 for parent 
node GUID = 0x2c90108d192e0, TID = 0x6a39.
[1109360115:000609563][40BFF970] -> Received a SMP on a 10 hop path:
                                Initial path = 
[0][1][3][5][5][7][3][1][3][1][2]
                                Return path  = 
[0][4][4][2][6][7][6][1][2][1][1]
[1109360115:000609574][40BFF970] -> __osm_ni_rcv_process_new: Discovered 
new Channel Adapter node,
                                GUID = 0x2c90108666d50, TID = 0x6a3a.
[1109360115:000609724][40BFF970] -> osm_pkey_rcv_process: Got 
GetResp(PKey) block:0 port_num 2 with GUID = 0x2c90108d192e0 for parent 
node GUID = 0x2c90108d192e0, TID = 0x6a3b.
[1109360115:000609887][40BFF970] -> osm_pkey_rcv_process: Got 
GetResp(PKey) block:1 port_num 2 with GUID = 0x2c90108d192e0 for parent 
node GUID = 0x2c90108d192e0, TID = 0x6a3c.
[1109360115:000610061][40BFF970] -> __osm_ni_rcv_process_existing: 
Rediscovered Switch node 0x2c90108e9c1e8
                                TID = 0x6a3d, discovered 6 times already.
[1109360115:000610226][40BFF970] -> osm_pkey_rcv_process: Got 
GetResp(PKey) block:0 port_num 3 with GUID = 0x2c90108d192e0 for parent 
node GUID = 0x2c90108d192e0, TID = 0x6a3e.
[1109360115:000610391][40BFF970] -> osm_pkey_rcv_process: Got 
GetResp(PKey) block:1 port_num 3 with GUID = 0x2c90108d192e0 for parent 
node GUID = 0x2c90108d192e0, TID = 0x6a3f.
[1109360115:000710603][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.


Form then on, that's all I see:
[1109360115:000710603][411FF970] -> umad_receiver: send completed with 
error(method=1 attr=11) -- dropping.

But to this point it's found a lot: I've got about 16710 messages in there 
with 'GUID = ' in them. 

I do get a bunch of these:
 Bad LinearFDBTop value = 0xC000 on switch 0x2c90108d19820.


ron


From rminnich at lanl.gov  Mon Feb 28 15:13:56 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 16:13:56 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109631101.13555.9933.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov> 
	<1109629112.13555.9915.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502281546250.18334@enigma.lanl.gov>
	<1109631101.13555.9933.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281613190.18334@enigma.lanl.gov>



On Mon, 28 Feb 2005, Hal Rosenstock wrote:

> opensm logs. Also, I would be curious to see what ibstat showed about
> all endport LIDs in the network. Are all the ports active that should be
> (plugged into subnet) ? 

They all look like this:

CA 'mthca0':
        CA type: MT23108
        Number of ports: 2
        Firmware version: 3.3.2
        Hardware version: a1
        Node GUID: 0x0002c90108660530
        System image GUID: 0x0002c9000100d050
        Port 1:
                State: Initializing
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0002c90108660531



On the front end, it is this:

CA 'mthca0':
        CA type: MT23108
        Number of ports: 2
        Firmware version: 3.3.2
        Hardware version: a1
        Node GUID: 0x0002c90108a03e60
        System image GUID: 0x0002c9000100d050
        Port 1:
                State: Initializing
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0002c90108a03e61
        Port 2:
                State: Down
                Rate: 2
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0002c90108a03e62


ron


From rminnich at lanl.gov  Mon Feb 28 15:16:02 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 16:16:02 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109631634.13555.9945.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov> 
	<1109628966.13555.9910.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov>
	<1109631634.13555.9945.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281614270.18334@enigma.lanl.gov>



On Mon, 28 Feb 2005, Hal Rosenstock wrote:

> Also, wasn't that the same failure as a while ago when one of those 96
> port switches kept forwarding but didn't terminate MADs ? (Yes, I know
> you recycled everything which would seem to be inconsistent with this).

yeah, something has gone south and I can't figure it out -- I lack too 
much IB clue. 

It does seem there's a gremlin in the network. We need simple ways for 
people to find such gremlins. It's also worrisome that one bad "thing" 
(whatever it is) can get opensm confused to the point that you can't bring 
up the network at all.

> I'm not sure we will see anything more from the logs than the error. Is
> there anything before the first error ? Can you also rerun with -V and
> look for the same ?

I'm doing that now.

Thanks!

ron


From libor at topspin.com  Mon Feb 28 15:17:36 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 15:17:36 -0800
Subject: [openib-general] Re: [PATCH] sdp conditional code cleanup
In-Reply-To: <20050228213406.GA1032@mellanox.co.il>;
	from mst@mellanox.co.il on Mon, Feb 28, 2005 at 11:34:06PM +0200
References: <20050224211503.GA16942@mellanox.co.il>
	<20050224132346.E24393@topspin.com>
	<20050224214928.GA17236@mellanox.co.il>
	<20050225115857.C13364@topspin.com>
	<20050227104441.GI23837@mellanox.co.il>
	<20050228131345.D21494@topspin.com>
	<20050228213406.GA1032@mellanox.co.il>
Message-ID: <20050228151736.F21494@topspin.com>

On Mon, Feb 28, 2005 at 11:34:06PM +0200, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: [PATCH] sdp conditional code cleanup
> > 
> > On Sun, Feb 27, 2005 at 12:44:41PM +0200, Michael S. Tsirkin wrote:
> > > Quoting r. Libor Michalek <libor at topspin.com>:
> > > > On Thu, Feb 24, 2005 at 11:49:28PM +0200, Michael S. Tsirkin wrote:
> > > > > OK, now what about things like these:
> > > > > 
> > > > >         if (0 > result) {
> > > > > 
> > > > > may I change them all to
> > > > > 
> > > > >         if (result < 0) {
> > > > > 
> > > > > While being equivalent, we are testing the result, not 0.
> > > > > 
> > > > > Similiarly (although I feel somewhat less strongly about it)
> > > > > 
> > > > >         if (0 == result)
> > > > > and
> > > > >         if (NULL == conn)
> > > > > 
> > > > > are better off as
> > > > > 
> > > > > 	if (!result) {
> > > > > and
> > > > >         if (!conn)
> > > > > 
> > > > > C is a Spartan language, and this is more brief.
> > > > > Libor, I think I asked about the second one, but dont recall you
> > > > > answering.
> > > > > If OK to both, let me know and I'll do it on Sunday.
> > > > 
> > > >   I actually feel more strongly in favour of making the second change
> > > > you propose then the first. However, I'm OK with both, so feel free
> > > > to submit a patch.
> > > 
> > > Here is the patch.
> > > 
> > > I generalized the approach - 
> > > any if (CONSTANT == variable) and if ( CONSTANT & variable) is now
> > >     if (variable == CONSTANT) and if ( variable & CONSTANT)
> > >     and so forth.
> > > 
> > > Further, some places had weird flag testing code like this:
> > > if ( (variable & CONSTANT) > 0 )
> > > 
> > > I changed them all to 
> > > 
> > > if (variable & CONSTANT)
> > > 
> > > Two things I noticed but did not fix:
> > > 
> > > A. In some places a positive error code is returned. For example
> > >    ENOBUF and not -ENOBUF. I assume its a bug but did not touch it.
> > 
> >   I'm glad you did not change this, it's not a bug. In parts of both
> > the send and recv data paths positive and negative returns have different
> > meaning. negative return is a hard error, while positive return is used
> > to indicate that the data path is in a flow control situation and that
> > the action will need to be retried later. Zero as usual means success.
> > 
> > > Looks like a lot of changes, I went over them several times and
> > > everything looks in order to me. I really hope its applied before
> > > any other change, it almost surely will conflict with any other
> > > patch, and it'll be a lot of work to re-diff.
> > 
> >   I found a few problem spots that I fixed up, before commiting
> > the code. The third problem was in an #if 0, so it wasn't an
> > immediate issue. Thanks.
> 
> Here is a small number of additional swaps I missed on the first pass.

  Thanks, applied and commited.

-Libor


From libor at topspin.com  Mon Feb 28 15:24:21 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 15:24:21 -0800
Subject: [openib-general] [PATCH][SDP][17/22] Remove use of leading
	underscores in static names in sdp_queue.h
In-Reply-To: <11096281731236@sun.com>;
	from Tom.Duffy@Sun.COM on Mon, Feb 28, 2005 at 02:02:53PM -0800
References: <11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com> <11096281121662@sun.com>
	<11096281222561@sun.com> <11096281321386@sun.com>
	<11096281432614@sun.com> <11096281532803@sun.com>
	<1109628163952@sun.com> <11096281731236@sun.com>
Message-ID: <20050228152421.G21494@topspin.com>

On Mon, Feb 28, 2005 at 02:02:53PM -0800, Tom Duffy wrote:
> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_queue.h
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_queue.h	(revision 1929)
> +++ drivers/infiniband/ulp/sdp/sdp_queue.h	(working copy)
> @@ -71,22 +71,22 @@ struct sdpc_desc_q {
>   */
>  
>  /*
> - * __sdp_desc_q_size - return the number of elements in the table
> + * sdp_desc_q_size - return the number of elements in the table
>   */
> -static inline int __sdp_desc_q_size(struct sdpc_desc_q *table)
> +static inline int sdp_desc_q_size(struct sdpc_desc_q *table)
>  {
>  	return table->size;
>  }
>  
>  /*
> - * __sdp_desc_q_member - return non-zero if element is in a table
> + * sdp_desc_q_member - return non-zero if element is in a table
>   */
> -static inline int __sdp_desc_q_member(struct sdpc_desc *element)
> +static inline int sdp_desc_q_member(struct sdpc_desc *element)
>  {
>  	return (element->table ? 1 : 0);
>  }
>  
> -#define sdp_desc_q_size(x)   __sdp_desc_q_size(x)
> -#define sdp_desc_q_member(x) __sdp_desc_q_member(x)
> +#define sdp_desc_q_size(x)   sdp_desc_q_size(x)
> +#define sdp_desc_q_member(x) sdp_desc_q_member(x)

  In this case wouldn't you just get rid of the two #defines? 

-Libor


From halr at voltaire.com  Mon Feb 28 15:22:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 18:22:33 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281614270.18334@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
	<1109628966.13555.9910.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov>
	<1109631634.13555.9945.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281614270.18334@enigma.lanl.gov>
Message-ID: <1109632953.13555.9951.camel@localhost.localdomain>

On Mon, 2005-02-28 at 18:16, Ronald G. Minnich wrote:
> It does seem there's a gremlin in the network. We need simple ways for 
> people to find such gremlins. It's also worrisome that one bad "thing" 
> (whatever it is) can get opensm confused to the point that you can't bring 
> up the network at all.

Some bad things are supposedly rules which are _not_ to be violated
(perhaps like LinearFDBTop one you cited). 

Others are failure modes that clearly need to be dealt with better both
on the front end (by OpenSM) and the diagnostics to find them.

-- Hal



From rminnich at lanl.gov  Mon Feb 28 15:34:31 2005
From: rminnich at lanl.gov (Ronald G. Minnich)
Date: Mon, 28 Feb 2005 16:34:31 -0700 (MST)
Subject: [openib-general] Question
In-Reply-To: <1109632953.13555.9951.camel@localhost.localdomain>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov> 
	<1109628966.13555.9910.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov> 
	<1109631634.13555.9945.camel@localhost.localdomain> 
	<Pine.LNX.4.58.0502281614270.18334@enigma.lanl.gov>
	<1109632953.13555.9951.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0502281631310.18334@enigma.lanl.gov>

ok, here you go. THis is the first one that appears to fail. 

YOu can probably guess why :-)

[1109632561:000646260][411FF970] -> __osm_sm_mad_ctrl_process_get_resp: [
[1109632561:000646268][411FF970] -> __osm_sm_mad_ctrl_update_wire_stats: [
[1109632561:000646277][411FF970] -> __osm_sm_mad_ctrl_update_wire_stats: 0 
SMPs on the wire, 322 outstanding.
[1109632561:000646285][411FF970] -> osm_vl15_poll: [
[1109632561:000646293][411FF970] -> osm_vl15_poll: Signalling poller 
thread.
[1109632561:000646305][411FF970] -> osm_vl15_poll: ]
[1109632561:000646317][411FF970] -> __osm_sm_mad_ctrl_update_wire_stats: ]
[1109632561:000646329][411FF970] -> osm_mad_pool_put: [
[1109632561:000646342][40DFF970] -> __osm_vl15_poller: Servicing p_madw = 
0x785890 (mad 0x2a96f9bb80 req 1)
[1109632561:000646360][411FF970] -> osm_mad_pool_put: Releasing p_madw = 
0x2a95f297b0, p_mad = 0x2a95f502f0.
[1109632561:000646374][411FF970] -> osm_vendor_put: [
[1109632561:000646382][411FF970] -> osm_vendor_put: Retiring UMAD 
0x2a95f502f0.
[1109632561:000646391][411FF970] -> osm_vendor_put: ]
[1109632561:000646399][411FF970] -> osm_mad_pool_put: ]
[1109632561:000646408][411FF970] -> __osm_sm_mad_ctrl_process_get_resp: 
Posting Dispatcher message OSM_MSG_MAD_PKEY.
[1109632561:000646418][411FF970] -> __osm_sm_mad_ctrl_process_get_resp: ]
[1109632561:000646430][411FF970] -> __osm_sm_mad_ctrl_rcv_callback: ]
[1109632561:000646443][40DFF970] -> SMP dump:
                                base_ver................0x1
                                mgmt_class..............0x81
                                class_ver...............0x1
                                method..................0x1 (SubnGet)
                                status..................0x0
                                hop_ptr.................0x0
                                hop_count...............0xA
                                trans_id................0x6a40
                                attr_id.................0x11 (NodeInfo)
                                resv....................0x0
                                attr_mod................0x0
                                m_key...................0x0000000000000000
                                dr_slid.................0xFFFF
                                dr_dlid.................0xFFFF

                                Initial path: 
[0][1][3][5][5][7][3][1][3][1][4]
                                Return path:  
[0][0][0][0][0][0][0][0][0][0][0]
                                Reserved:     [0][0][0][0][0][0][0]

                                00 00 00 00 00 00 00 00   00 00 00 00 00 
00 00 00

                                00 00 00 00 00 00 00 00   00 00 00 00 00 
00 00 00

                                00 00 00 00 00 00 00 00   00 00 00 00 00 
00 00 00

                                00 00 00 00 00 00 00 00   00 00 00 00 00 
00 00 00

[1109632561:000646482][40DFF970] -> osm_vendor_send: [
[1109632561:000646494][40BFF970] -> osm_pkey_rcv_process: [
[1109632561:000646508][40DFF970] -> __osm_mtl_send_callback: Completed 
Sending Request MADW:0x785890.
[1109632561:000646521][40DFF970] -> osm_vendor_send: ]
[1109632561:000646533][40BFF970] -> osm_pkey_rcv_process: Got 
GetResp(PKey) block:1 port_num 3 with GUID = 0x2c90108d192e0 for parent 
node GUID = 0x2c90108d192e0, TID = 0x6a3f.
[1109632561:000646552][40DFF970] -> __osm_vl15_poller: 1 on wire, 322 
outstanding, 0 unicasts sent, 22541 sent total.
[1109632561:000646628][40BFF970] -> P_Key table dump:
                        port_giud...........0x0002c90108d192e0
                        block_num...........0x1
                        port_num............0x3
        P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 
| 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 
0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |




Actually I'm not sure if that is the first bad one, but ... is the return 
path supposed to be all zeros? Can I take it that such a path indicates an 
error? 

Maybe not: there's a ton of them in there. 

Hal, what else can I send you?

ron




From halr at voltaire.com  Mon Feb 28 15:32:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 18:32:05 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281607460.18334@enigma.lanl.gov>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC6D@taurus.voltaire.com>
	<Pine.LNX.4.58.0502281536150.18334@enigma.lanl.gov>
	<1109630909.13555.9928.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281607460.18334@enigma.lanl.gov>
Message-ID: <1109633525.23094.6.camel@localhost.localdomain>

On Mon, 2005-02-28 at 18:10, Ronald G. Minnich wrote: 
> I do get a bunch of these:
>  Bad LinearFDBTop value = 0xC000 on switch 0x2c90108d19820.

That comes from the following in osm_sw_info_rcv.c::osm_si_rcv_process 

/*
      Hack for bad value in Mellanox switch
    */
    if( cl_ntoh16( p_si->lin_top ) > IB_LID_UCAST_END_HO )
    {
      osm_log( p_rcv->p_log, OSM_LOG_ERROR,
               "osm_si_rcv_process: ERR 3610: "
               "\n\t\t\t\tBad LinearFDBTop value = 0x%X "
               "on switch 0x%" PRIx64 "."
               "\n\t\t\t\tForcing correction to 0x%X.\n",
               cl_ntoh16( p_si->lin_top ),
               cl_ntoh64( osm_node_get_node_guid( p_node ) ),
               0 );

      p_si->lin_top = 0;
    }

where include/iba/ib_types.h:
#define IB_LID_UCAST_END_HO                                     0xBFFF

So this looks like a workaround for a bug. Not sure what any of the
other symptoms are but I'm real curious now. Can someone comment more on
this ?

At a minimum, the SMA is reporting an invalid value for
PortInfo::LinearFDBTop. I wonder if it also is incapable of forwarding
DR MADs as well. That would explain this.

OpenSM needs a better way of dealing with failures. This need has been
documented. Sounds like this needs to be a high priority item. I will
start to work on this.

Thanks.

-- Hal

> 
> ron



From tduffy at sun.com  Mon Feb 28 15:37:55 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 28 Feb 2005 15:37:55 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F3FAC63@taurus.voltaire.com>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC63@taurus.voltaire.com>
Message-ID: <1109633875.19720.11.camel@duffman>

On Mon, 2005-02-28 at 21:47 +0200, Yaron Haviv wrote:
> And as you mentioned there is value to have the same API for different
> resolution mechanisms, the SDP code can be altered in future to ride
> over the proposed API, so it can be used without TCP/IP.

I am not sure you are gaining much by having SDP use straight ATS.
Already, once the ARP table is filled with the information, it is a
local cached lookup.

Now if the service posted to the SM included the ports that the SDP node
was listening on, then that would speed up connections, but that would
no longer be ATS (as defined).

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050228/d4eb4f7d/attachment.sig>

From libor at topspin.com  Mon Feb 28 15:41:01 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 15:41:01 -0800
Subject: [openib-general] [PATCH][SDP][0/22] Remove use of leading
	underscores in static names in SDP
In-Reply-To: <1109627999679@sun.com>;
	from Tom.Duffy@Sun.COM on Mon, Feb 28, 2005 at 01:59:59PM -0800
References: <1109627999679@sun.com>
Message-ID: <20050228154101.H21494@topspin.com>

On Mon, Feb 28, 2005 at 01:59:59PM -0800, Tom Duffy wrote:
> This series of patches removes the use of a leading underscore or double
> underscore before static or static inline functions and variables.  There are
> also a few random whitespace fixes I noticed while going through the code.

Thanks Tom, applied, plus the removal of the #define's in patch 17/22,
and committed.

-Libor


From tduffy at sun.com  Mon Feb 28 15:42:28 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 28 Feb 2005 15:42:28 -0800
Subject: [openib-general] [PATCHv2][SDP][17/22] Remove use of leading
	underscores in static names in sdp_queue.h
In-Reply-To: <20050228152421.G21494@topspin.com>
References: <11096280813891@sun.com> <11096280911941@sun.com>
	<11096281021938@sun.com> <11096281121662@sun.com>
	<11096281222561@sun.com>
	<11096281321386@sun.com> <11096281432614@sun.com>
	<11096281532803@sun.com>
	<1109628163952@sun.com> <11096281731236@sun.com>
	<20050228152421.G21494@topspin.com>
Message-ID: <1109634148.19720.15.camel@duffman>

On Mon, 2005-02-28 at 15:24 -0800, Libor Michalek wrote:
> > -#define sdp_desc_q_size(x)   __sdp_desc_q_size(x)
> > -#define sdp_desc_q_member(x) __sdp_desc_q_member(x)
> > +#define sdp_desc_q_size(x)   sdp_desc_q_size(x)
> > +#define sdp_desc_q_member(x) sdp_desc_q_member(x)
> 
>   In this case wouldn't you just get rid of the two #defines? 

Duh.  Yeah.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_queue.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_queue.h	(revision 1930)
+++ drivers/infiniband/ulp/sdp/sdp_queue.h	(working copy)
@@ -71,22 +71,19 @@ struct sdpc_desc_q {
  */
 
 /*
- * __sdp_desc_q_size - return the number of elements in the table
+ * sdp_desc_q_size - return the number of elements in the table
  */
-static inline int __sdp_desc_q_size(struct sdpc_desc_q *table)
+static inline int sdp_desc_q_size(struct sdpc_desc_q *table)
 {
 	return table->size;
 }
 
 /*
- * __sdp_desc_q_member - return non-zero if element is in a table
+ * sdp_desc_q_member - return non-zero if element is in a table
  */
-static inline int __sdp_desc_q_member(struct sdpc_desc *element)
+static inline int sdp_desc_q_member(struct sdpc_desc *element)
 {
 	return (element->table ? 1 : 0);
 }
 
-#define sdp_desc_q_size(x)   __sdp_desc_q_size(x)
-#define sdp_desc_q_member(x) __sdp_desc_q_member(x)
-
 #endif /* _SDP_QUEUE_H */



From halr at voltaire.com  Mon Feb 28 15:39:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Feb 2005 18:39:23 -0500
Subject: [openib-general] Question
In-Reply-To: <Pine.LNX.4.58.0502281631310.18334@enigma.lanl.gov>
References: <Pine.LNX.4.58.0502281505150.18334@enigma.lanl.gov>
	<1109628966.13555.9910.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281539290.18334@enigma.lanl.gov>
	<1109631634.13555.9945.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281614270.18334@enigma.lanl.gov>
	<1109632953.13555.9951.camel@localhost.localdomain>
	<Pine.LNX.4.58.0502281631310.18334@enigma.lanl.gov>
Message-ID: <1109633962.23094.13.camel@localhost.localdomain>

On Mon, 2005-02-28 at 18:34, Ronald G. Minnich wrote:
> ok, here you go. THis is the first one that appears to fail. 
> 
> YOu can probably guess why :-)
> 
> [1109632561:000646260][411FF970] -> __osm_sm_mad_ctrl_process_get_resp: [
> [1109632561:000646268][411FF970] -> __osm_sm_mad_ctrl_update_wire_stats: [
> [1109632561:000646277][411FF970] -> __osm_sm_mad_ctrl_update_wire_stats: 0 
> SMPs on the wire, 322 outstanding.
> [1109632561:000646285][411FF970] -> osm_vl15_poll: [
> [1109632561:000646293][411FF970] -> osm_vl15_poll: Signalling poller 
> thread.
> [1109632561:000646305][411FF970] -> osm_vl15_poll: ]
> [1109632561:000646317][411FF970] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> [1109632561:000646329][411FF970] -> osm_mad_pool_put: [
> [1109632561:000646342][40DFF970] -> __osm_vl15_poller: Servicing p_madw = 
> 0x785890 (mad 0x2a96f9bb80 req 1)
> [1109632561:000646360][411FF970] -> osm_mad_pool_put: Releasing p_madw = 
> 0x2a95f297b0, p_mad = 0x2a95f502f0.
> [1109632561:000646374][411FF970] -> osm_vendor_put: [
> [1109632561:000646382][411FF970] -> osm_vendor_put: Retiring UMAD 
> 0x2a95f502f0.
> [1109632561:000646391][411FF970] -> osm_vendor_put: ]
> [1109632561:000646399][411FF970] -> osm_mad_pool_put: ]
> [1109632561:000646408][411FF970] -> __osm_sm_mad_ctrl_process_get_resp: 
> Posting Dispatcher message OSM_MSG_MAD_PKEY.
> [1109632561:000646418][411FF970] -> __osm_sm_mad_ctrl_process_get_resp: ]
> [1109632561:000646430][411FF970] -> __osm_sm_mad_ctrl_rcv_callback: ]
> [1109632561:000646443][40DFF970] -> SMP dump:
>                                 base_ver................0x1
>                                 mgmt_class..............0x81
>                                 class_ver...............0x1
>                                 method..................0x1 (SubnGet)
>                                 status..................0x0
>                                 hop_ptr.................0x0
>                                 hop_count...............0xA
>                                 trans_id................0x6a40
>                                 attr_id.................0x11 (NodeInfo)
>                                 resv....................0x0
>                                 attr_mod................0x0
>                                 m_key...................0x0000000000000000
>                                 dr_slid.................0xFFFF
>                                 dr_dlid.................0xFFFF
> 
>                                 Initial path: 
> [0][1][3][5][5][7][3][1][3][1][4]
>                                 Return path:  
> [0][0][0][0][0][0][0][0][0][0][0]
>                                 Reserved:     [0][0][0][0][0][0][0]
> 
>                                 00 00 00 00 00 00 00 00   00 00 00 00 00 
> 00 00 00
> 
>                                 00 00 00 00 00 00 00 00   00 00 00 00 00 
> 00 00 00
> 
>                                 00 00 00 00 00 00 00 00   00 00 00 00 00 
> 00 00 00
> 
>                                 00 00 00 00 00 00 00 00   00 00 00 00 00 
> 00 00 00
> 
> [1109632561:000646482][40DFF970] -> osm_vendor_send: [
> [1109632561:000646494][40BFF970] -> osm_pkey_rcv_process: [
> [1109632561:000646508][40DFF970] -> __osm_mtl_send_callback: Completed 
> Sending Request MADW:0x785890.
> [1109632561:000646521][40DFF970] -> osm_vendor_send: ]
> [1109632561:000646533][40BFF970] -> osm_pkey_rcv_process: Got 
> GetResp(PKey) block:1 port_num 3 with GUID = 0x2c90108d192e0 for parent 
> node GUID = 0x2c90108d192e0, TID = 0x6a3f.
> [1109632561:000646552][40DFF970] -> __osm_vl15_poller: 1 on wire, 322 
> outstanding, 0 unicasts sent, 22541 sent total.
> [1109632561:000646628][40BFF970] -> P_Key table dump:
>                         port_giud...........0x0002c90108d192e0
>                         block_num...........0x1
>                         port_num............0x3
>         P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 
> | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 
> 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> 
> 
> 
> 
> Actually I'm not sure if that is the first bad one, but ... is the return 
> path supposed to be all zeros? 

The return path is filled in on the way back (in the response) so this
is fine.

> Can I take it that such a path indicates an error? 

No.

> Maybe not: there's a ton of them in there. 

Right. It is also showing the PKey table on response.

> Hal, what else can I send you?

Are you sure ibnetdiscover gets your whole topology and nothing is
missing (like a switch) ?

-- Hal



From libor at topspin.com  Mon Feb 28 15:45:33 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 15:45:33 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <1109633875.19720.11.camel@duffman>;
	from tduffy@sun.com on Mon, Feb 28, 2005 at 03:37:55PM -0800
References: <35EA21F54A45CB47B879F21A91F4862F3FAC63@taurus.voltaire.com>
	<1109633875.19720.11.camel@duffman>
Message-ID: <20050228154533.I21494@topspin.com>

On Mon, Feb 28, 2005 at 03:37:55PM -0800, Tom Duffy wrote:
> On Mon, 2005-02-28 at 21:47 +0200, Yaron Haviv wrote:
> > And as you mentioned there is value to have the same API for different
> > resolution mechanisms, the SDP code can be altered in future to ride
> > over the proposed API, so it can be used without TCP/IP.
> 
> I am not sure you are gaining much by having SDP use straight ATS.
> Already, once the ARP table is filled with the information, it is a
> local cached lookup.
> 
> Now if the service posted to the SM included the ports that the SDP node
> was listening on, then that would speed up connections, but that would
> no longer be ATS (as defined).

  And it would no longer be SDP. As a drop in replacement for AF_INET
SOCKET_STREAM it requires IP addressing, which ATS is not. Also, caching
which ports are available on a remote node doesn't save you from notifying
the remote node that you are connection to that port, in which case
there's no savings to the caching since you need to contact the remote
node, except in the negative case, but then you have a race condition, etc...

-Libor



-Libor


From libor at topspin.com  Mon Feb 28 16:04:06 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 16:04:06 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F3FAC64@taurus.voltaire.com>;
	from yaronh@voltaire.com on Mon, Feb 28, 2005 at 09:55:50PM +0200
References: <35EA21F54A45CB47B879F21A91F4862F3FAC64@taurus.voltaire.com>
Message-ID: <20050228160406.J21494@topspin.com>

On Mon, Feb 28, 2005 at 09:55:50PM +0200, Yaron Haviv wrote:
> From: Libor Michalek
> > 
> > The two are not interoperable, they
> > reside in parallel, and succeed in producing much confusion. (IMO)
> 
> One note, the two can be made interoperable, if nodes that use IPoIB
> register them self in the ATS database as well (which has its merits for
> reverse resolution that cannot be satisfied by IPoIB), this way the
> nodes that just use ATS can locate the IPoIB ones.

  This relies on each node in a fabric keeping the information between
the two parallel methods in sync. Which leads to the question, why have
two independent methods for getting the exact same information? The
only logical answer is that there are some nodes which can only use
one of the methods. In which case the two sets of data are not identical,
because of these nodes, which succeeds in producing much confusion. Not
to mention the race conditions between keeping a centralized database (ATS)
in sync with the distributed mechanism. (ARP)

  For these reasons I cringe at hearing IP address and ATS in the
same sentence, I really wish DAT had chosen a different name for
the addresses.

  Really, we all discussed this years ago in the IETF, the merits of
using broadcast vs. centralized data store, and a solution was
developed. This is why open standards bodies are so useful.

> Anyway the merits if the proposed API goes much beyond the use of ATS,
> so I hope we don't just hang on that one.

  Agreed, there is certainly a lot more to discuss then just ATS and ARP.


-Libor


From krause at cup.hp.com  Mon Feb 28 16:07:00 2005
From: krause at cup.hp.com (Michael Krause)
Date: Mon, 28 Feb 2005 16:07:00 -0800
Subject: [openib-general] IB Address Translation service
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F3FAC63@taurus.voltaire.com
 >
References: <35EA21F54A45CB47B879F21A91F4862F3FAC63@taurus.voltaire.com>
Message-ID: <6.2.0.14.2.20050228160359.02ab0b98@esmail.cup.hp.com>

At 11:47 AM 2/28/2005, Yaron Haviv wrote:
> > -----Original Message-----
> > From: openib-general-bounces at openib.org [mailto:openib-general-
> > bounces at openib.org] On Behalf Of Paul Baxter
> > Sent: Monday, February 28, 2005 9:32 PM
> > To: openib-general at openib.org
> > Subject: Re: [openib-general] IB Address Translation service
> >
> > Having now just read Yaron's reply, I am even more convinced that this
>is
> > the right way to go albeit I can't comment on the API etc (Could
>someone
> > explain the differences in using ARP and ATS. )
>
>Paul,
>
>ATS (Address Resolution Service) is based on each node registering a
>service record in the SM/SA with GID&P_Key=IP address.
>When you want to map an IP address to IB address it issues an SA query
>to the SM/SA with an IP that results in GID+P_Key values than can be
>used by the ULP.
>
>ATS is a standard defined by DAT and recently also by ICSC.

It is not an ICSC standard.  It is a DAT specification with the ICSC simply 
acting as a proxy for DAT to manage some standard look-up values which it 
can do but I don't recall why DAT cannot.  In any case, the ICSC and DAT 
are different beasts all doing about the same damn thing at the end of the day.


>As I mentioned in the IP to GID API you can specify if to resolve based
>on the IP infrastructure (like the one Libor described), or based on
>ATS, or Default (first try IP/ARP, than ATS).
>
>And as you mentioned there is value to have the same API for different
>resolution mechanisms, the SDP code can be altered in future to ride
>over the proposed API, so it can be used without TCP/IP.

It would be a mistake to attempt to use anything by IP addresses (v4 or v6) 
from an application perspective. Mapping to IB must be application 
transparent to be viable.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050228/d46c48ce/attachment.html>

From libor at topspin.com  Mon Feb 28 17:33:07 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 17:33:07 -0800
Subject: [openib-general] [CM] destroy_cm_id
In-Reply-To: <20050228105036.122e3b1c.mshefty@ichips.intel.com>;
	from mshefty@ichips.intel.com on Mon, Feb 28, 2005 at 10:50:36AM
	-0800
References: <20050228105036.122e3b1c.mshefty@ichips.intel.com>
Message-ID: <20050228173307.K21494@topspin.com>

Sean,

  Is it ever allowed to call ib_destroy_cm_id() from a CM callback?
For some reason I thought that this was OK from only the IDLE callback,
but if I destroy from IDLE I get a hang on cm_id_priv->lock, I believe.
Should the normal mode of operation in the case be to return an error
from IDLE to ensure that cm_id gets cleaned-up?

Thanks.

-Libor


From libor at topspin.com  Mon Feb 28 17:55:34 2005
From: libor at topspin.com (Libor Michalek)
Date: Mon, 28 Feb 2005 17:55:34 -0800
Subject: [openib-general] [PATCH][SDP] CM disconnect.
Message-ID: <20050228175534.L21494@topspin.com>


  Fixup and simplify CM disconnect interface, and in the process get
rid of sdp_post.c. Plus convert a few more functions from int to void
return. 

  With this patch all the major SDP/CM code paths should be working
correctly, including connection cleanup. So if you see connections
that are not getting cleaned up, let me know.

  There's still more to be done to simplify the connection state
transitions. Specifically merge conn->istate and conn->state since
there's alot of redundant information between the two, and checking
for divergence between the two which cannot occur.

 9 files changed, 83 insertions(+), 241 deletions(-)

-Libor

Signed-off-by: Libor Michalek <libor at topspin.com>

Index: Makefile
===================================================================
--- Makefile	(revision 1931)
+++ Makefile	(working copy)
@@ -20,7 +20,6 @@
 	sdp_send.o \
 	sdp_recv.o \
 	sdp_event.o \
-	sdp_post.o \
 	sdp_pass.o \
 	sdp_actv.o \
 	sdp_link.o
Index: sdp_proto.h
===================================================================
--- sdp_proto.h	(revision 1931)
+++ sdp_proto.h	(working copy)
@@ -342,14 +342,6 @@
 /*
  * post functions
  */
-int sdp_cm_listen_start(struct sdev_root *dev_root);
-
-int sdp_cm_listen_stop(struct sdev_root *dev_root);
-
-int sdp_cm_disconnect(struct sdp_opt *conn);
-
-int sdp_cm_reject(struct sdp_opt *conn);
-
 int sdp_recv_flush(struct sdp_opt *conn);
 
 int sdp_send_flush(struct sdp_opt *conn);
@@ -619,7 +611,7 @@
 /*
  * sdp_conn_state_dump - dump state information to the log
  */
-static inline int sdp_conn_state_dump(struct sdp_opt *conn)
+static inline void sdp_conn_state_dump(struct sdp_opt *conn)
 {
 #ifdef _SDP_CONN_STATE_REC
 	int counter;
@@ -629,7 +621,7 @@
 	if (conn->state_rec.state[0] == SDP_CONN_ST_INVALID) {
 		sdp_dbg_init("STATE:   No state history. <%d>",
 			     conn->state_rec.value);
-		return 0;
+		return;
 	}
 
 	for (counter = 0;
@@ -641,7 +633,6 @@
 			     conn->state_rec.file[counter],
 			     conn->state_rec.line[counter]);
 #endif
-	return 0;
 }
 
 #endif /* _SDP_PROTO_H */
Index: sdp_rcvd.c
===================================================================
--- sdp_rcvd.c	(revision 1931)
+++ sdp_rcvd.c	(working copy)
@@ -87,13 +87,14 @@
 		/*
 		 * Begin IB/CM disconnect
 		 */
-		result = sdp_cm_disconnect(conn);
+		result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
 		/*
 		 * if the remote DREQ was already received, but unprocessed, 
 		 * do not treat it as an error
 		 */
-		if (result < 0) {
-			sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
+		if (result) {
+			sdp_dbg_warn(conn, "Error <%d> sending CM DREQ",
+				     result);
 
 			if (result != -EPROTO)
 				goto error;
Index: sdp_sent.c
===================================================================
--- sdp_sent.c	(revision 1931)
+++ sdp_sent.c	(working copy)
@@ -65,8 +65,8 @@
 		/*
 		 * Begin IB/CM disconnect
 		 */
-		result = sdp_cm_disconnect(conn);
-		if (result < 0) {
+		result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
+		if (result) {
 			sdp_dbg_warn(conn, "Error <%d> posting CM disconnect",
 				     result);
 			goto error;
@@ -92,13 +92,14 @@
 		/*
 		 * Begin IB/CM disconnect
 		 */
-		result = sdp_cm_disconnect(conn);
+		result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
 		/*
 		 * if the remote DREQ was already received, but unprocessed, do
 		 * not treat it as an error
 		 */
-		if (result < 0)
-			sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
+		if (result)
+			sdp_dbg_warn(conn, "Error <%d> sending CM DREQ",
+				     result);
 
 		break;
 	default:
@@ -126,9 +127,9 @@
 	SDP_EXPECT((conn->state == SDP_CONN_ST_ERROR_STRM));
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
-	result = sdp_cm_disconnect(conn);
-	if (result < 0)
-		sdp_dbg_warn(conn, "Error <%d> CM disconnect", result);
+	result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
+	if (result)
+		sdp_dbg_warn(conn, "Error <%d> sending CM DREQ", result);
 
 	return result;
 }
Index: sdp_wall.c
===================================================================
--- sdp_wall.c	(revision 1931)
+++ sdp_wall.c	(working copy)
@@ -43,7 +43,6 @@
  */
 int sdp_wall_send_close(struct sdp_opt *conn)
 {
-	struct sdpc_buff *buff;
 	int result;
 
 	sdp_dbg_ctrl(conn, "Close send. src <%08x:%04x> dst <%08x:%04x>",
@@ -61,13 +60,7 @@
 		/*
 		 * clear out the sent HelloAck message
 		 */
-		buff = sdp_buff_q_get_head(&conn->send_post);
-		if (!buff)
-			sdp_dbg_warn(conn, "Error, hello ack missing.");
-		else {
-			result = sdp_buff_pool_put(buff);
-			SDP_EXPECT(result >= 0);
-		}
+		(void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post));
 		/*
 		 * fall through
 		 */
@@ -105,9 +98,7 @@
 	/*
 	 * abortive close.
 	 */
-	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
-
-	result = sdp_cm_disconnect(conn);
+	result = sdp_wall_send_abort(conn);
 	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
@@ -157,9 +148,7 @@
 	/*
 	 * abortive close.
 	 */
-	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
-
-	result = sdp_cm_disconnect(conn);
+	result = sdp_wall_send_abort(conn);
 	if (result < 0)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
@@ -262,8 +251,8 @@
 	 */
 	SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM);
 
-	result = sdp_cm_disconnect(conn);
-	if (result < 0)
+	result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
+	if (result)
 		sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result);
 
 	return 0;
Index: sdp_conn.h
===================================================================
--- sdp_conn.h	(revision 1931)
+++ sdp_conn.h	(working copy)
@@ -465,8 +465,8 @@
 extern void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags);
 extern void sdp_conn_internal_unlock(struct sdp_opt *conn);
 extern void sdp_conn_relock(struct sdp_opt *conn);
+extern void sdp_conn_destruct(struct sdp_opt *conn);
 extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn);
-extern int sdp_conn_destruct(struct sdp_opt *conn);
 
 #define SDP_CONN_LOCK_IRQ(conn, flags) \
         spin_lock_irqsave(&((conn)->lock.slock), flags)
@@ -515,7 +515,7 @@
 static inline void sdp_conn_put(struct sdp_opt *conn)
 {
 	if (atomic_dec_and_test(&conn->refcnt))
-		(void)sdp_conn_destruct(conn);
+		sdp_conn_destruct(conn);
 }
 
 /*
Index: sdp_conn.c
===================================================================
--- sdp_conn.c	(revision 1931)
+++ sdp_conn.c	(working copy)
@@ -646,17 +646,11 @@
 /*
  * sdp_conn_destruct - final destructor for connection.
  */
-int sdp_conn_destruct(struct sdp_opt *conn)
+void sdp_conn_destruct(struct sdp_opt *conn)
 {
-	int result = 0;
 	int dump = 0;
+	int result;
 
-	if (!conn) {
-		sdp_dbg_warn(NULL, "sk destruct, no connection!");
-		result = -EINVAL;
-		goto done;
-	}
-
 	sdp_dbg_ctrl(conn, "DESTRUCT. <%08x:%04x> <%08x:%04x> <%u:%u>",
 		     conn->src_addr, conn->src_port,
 		     conn->dst_addr, conn->dst_port,
@@ -746,7 +740,7 @@
 	 * in case CM/IB are still tracking this connection.
 	 */
 	if (conn->cm_id) {
-		sdp_dbg_ctrl(conn, "CM local id <%d>", conn->cm_id->local_id);
+		sdp_dbg_warn(conn, "destroying CM ID.");
 
 		result = ib_destroy_cm_id(conn->cm_id);
 		if (result)
@@ -759,6 +753,9 @@
 	if (atomic_read(&conn->refcnt) < 0)
 		sdp_dbg_warn(conn, "destruct low ref count <%04x>",
 			     atomic_read(&conn->refcnt));
+
+	if (dump)
+		sdp_conn_state_dump(conn);
 	/*
 	 * free the OS socket structure
 	 */
@@ -770,16 +767,6 @@
 	}
 
 	kmem_cache_free(dev_root_s.conn_cache, conn);
-
-	result = 0;
-done:
-
-	if (dump) {
-		result = sdp_conn_state_dump(conn);
-		SDP_EXPECT(result >= 0);
-	}
-
-	return result;
 }
 
 /*
@@ -1995,15 +1982,29 @@
 	/*
 	 * start listening
 	 */
-	result = sdp_cm_listen_start(&dev_root_s);
-	if (result < 0) {
-		sdp_warn("Error <%d> listening for connections on HCA.",
-			 result);
+	dev_root_s.listen_id = ib_create_cm_id(sdp_cm_event_handler,
+					       (void *)SDP_DEV_SK_INVALID);
+	if (!dev_root_s.listen_id) {
+		sdp_warn("Failed to create listen connection identifier.");
+		result = -ENOMEM;
+		goto error_cm_id;
+	}
+
+	result = ib_cm_listen(dev_root_s.listen_id,
+		              cpu_to_be64(SDP_MSG_SERVICE_ID_VALUE),
+			      cpu_to_be64(SDP_MSG_SERVICE_ID_MASK));
+	if (result) {
+		sdp_warn("Error <%d> listening for SDP connections", result);
 		goto error_listen;
+
 	}
 
+	sdp_dbg_init("Started listening for SDP connection requests");
+
 	return 0;
 error_listen:
+	(void)ib_destroy_cm_id(dev_root_s.listen_id);
+error_cm_id:
 	kmem_cache_destroy(dev_root_s.sock_cache);
 error_sock:
 	kmem_cache_destroy(dev_root_s.conn_cache);
@@ -2044,6 +2045,10 @@
 	dev_root_s.sk_array--;
 	free_pages((unsigned long)dev_root_s.sk_array, dev_root_s.sk_ordr);
 	/*
+	 * delete IOCB table
+	 */
+	sdp_main_iocb_cleanup();
+	/*
 	 * delete conn cache
 	 */
 	kmem_cache_destroy(dev_root_s.conn_cache);
@@ -2054,11 +2059,7 @@
 	/*
 	 * stop listening
 	 */
-	(void)sdp_cm_listen_stop(&dev_root_s);
-	/*
-	 * delete IOCB table
-	 */
-	sdp_main_iocb_cleanup();
+	(void)ib_destroy_cm_id(dev_root_s.listen_id);
 
 	return 0;
 }
Index: sdp_event.c
===================================================================
--- sdp_event.c	(revision 1931)
+++ sdp_event.c	(working copy)
@@ -236,11 +236,13 @@
 		break;
 	}
 
+	conn->cm_id = NULL;
 	sdp_conn_put(conn); /* CM reference */
-	return 0;
+	return -ENOENT;     /* ensure CM cleans-up identifier. */
 }
 
-static int sdp_cm_established(struct ib_cm_id *cm_id, struct ib_cm_event *event,
+static int sdp_cm_established(struct ib_cm_id *cm_id,
+			      struct ib_cm_event *event,
 			      struct sdp_opt *conn)
 {
 	int result = 0;
@@ -306,6 +308,28 @@
 	return result;
 }
 
+static int sdp_cm_dreq_rcvd(struct ib_cm_id *cm_id, struct ib_cm_event *event,
+			    struct sdp_opt *conn)
+{
+	int result = 0;
+
+	sdp_dbg_ctrl(conn, "CM DREQ RCVD. commID <%08x> event <%d>",
+		     cm_id->local_id, event->event);
+	/*
+	 * Respond with a DREP.
+	 */
+	result = ib_send_cm_drep(conn->cm_id, NULL, 0);
+	if (result) {
+		sdp_dbg_warn(conn, "Error <%d> sending CM DREP", result);
+
+		sdp_wall_recv_drop(conn);
+
+		conn->cm_id = NULL;
+		sdp_conn_put(conn);	/* CM reference */
+	}
+
+	return result;
+}
 /*
  * sdp_cm_timewait - handler for connection Time Wait completion
  */
@@ -326,7 +350,7 @@
 			       conn->ca->dma_device,
 			       PCI_DMA_FROMDEVICE);
 	/*
-	 * check state
+	 * process state changes.
 	 */
 	switch (conn->state) {
 	case SDP_CONN_ST_ERROR_STRM:
@@ -423,6 +447,10 @@
 		result = sdp_cm_established(cm_id, event, conn);
 		break;
 	case IB_CM_DREQ_RCVD:
+		result = sdp_cm_dreq_rcvd(cm_id, event, conn);
+		if (result)
+			break;
+		/* fall through on success to handle state transition */
 	case IB_CM_TIMEWAIT:
 		result = sdp_cm_timewait(cm_id, event, conn);
 		break;
@@ -441,7 +469,7 @@
 			/*
 			 * dump connection state if it is being recorded.
 			 */
-			(void)sdp_conn_state_dump(conn);
+			sdp_conn_state_dump(conn);
 		}
 
 		sdp_conn_unlock(conn);
Index: sdp_post.c
===================================================================
--- sdp_post.c	(revision 1931)
+++ sdp_post.c	(working copy)
@@ -1,168 +0,0 @@
-/*
- * Copyright (c) 2005 Topspin Communications.  All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- * $Id$
- */
-
-#include "sdp_main.h"
-
-/*
- * sdp_cm_listen_start - start listening on all possible socket ports
- */
-int sdp_cm_listen_start(struct sdev_root *dev_root)
-{
-	int result = 0;
-
-	if (dev_root->listen_id)
-		sdp_dbg_warn(NULL, "Already listening for connections.");
-
-	dev_root->listen_id = ib_create_cm_id(sdp_cm_event_handler,
-					      (void *)SDP_DEV_SK_INVALID);
-	if (!dev_root->listen_id)
-		return -ENOMEM;
-	/*
-	 * start listening
-	 */
-	result = ib_cm_listen(dev_root->listen_id,
-		              cpu_to_be64(SDP_MSG_SERVICE_ID_VALUE),
-			      cpu_to_be64(SDP_MSG_SERVICE_ID_MASK));
-	if (result < 0) {
-		sdp_dbg_warn(NULL, "Error <%d> listening for SDP connections",
-			     result);
-
-		(void)ib_destroy_cm_id(dev_root->listen_id);
-		dev_root->listen_id = NULL;
-	} else
-		sdp_dbg_init("Started listening for SDP connection requests");
-
-	return result;
-}
-
-/*
- * sdp_cm_listen_stop - stop listening on all possible socket ports
- */
-int sdp_cm_listen_stop(struct sdev_root *dev_root)
-{
-	int result = 0;
-
-	result = ib_destroy_cm_id(dev_root->listen_id);
-	if (result < 0)
-		sdp_dbg_warn(NULL, "Error <%d> stopping listen", result);
-	else
-		sdp_dbg_init("Stopped listening for SDP connections");
-
-	dev_root->listen_id = NULL;
-
-	return result;
-}
-
-/*
- * do_cm_disconnect - initiate a disconnect request using the CM
- */
-static void do_cm_disconnect(void *arg)
-{
-	struct sdp_opt *conn = (struct sdp_opt *)arg;
-	int result;
-	
-	if (!conn) {
-		sdp_dbg_warn(NULL, "Error, posting disconnect for NULL conn");
-		return;
-	}
-
-	sdp_dbg_ctrl(conn, "Defered disconnect <%08x>", conn->cm_id->local_id);
-	/*
-	 * send a disconnect request using the connection manager
-	 */
-	result = ib_send_cm_dreq(conn->cm_id, NULL, 0);
-	if (result < 0)
-		sdp_dbg_warn(NULL, "Error <%d> CM disconnect request", result);
-
-	sdp_conn_put(conn);
-}
-
-/*
- * do_cm_reject - initiate a reject request using the CM
- */
-static void do_cm_reject(void *arg)
-{
-	struct sdp_opt *conn = (struct sdp_opt *)arg;
-	int result;
-
-	if (!conn)
-		sdp_dbg_warn(NULL, "Error, posting reject for NULL conn");
-
-	sdp_dbg_ctrl(conn, "Defered reject <%08x>", conn->cm_id->local_id);
-	/*
-	 * send a reject request using the connection manager
-	 */
-	result = ib_send_cm_rej(conn->cm_id, 
-				IB_CM_REJ_CONSUMER_DEFINED,
-				NULL, 0, NULL, 0);
-	if (result < 0)
-		sdp_dbg_warn(NULL, "Error <%d> CM reject request", result);
-
-	sdp_conn_put(conn);
-}
-/*
- * sdp_cm_deferred_generic - initiate a defered request using the CM
- */
-static int sdp_cm_deferred_generic(struct sdp_opt *conn,
-				   void (*defer_func)(void *arg))
-{
-	/*
-	 * send a potentially defered failed request.
-	 */
-	sdp_conn_hold(conn);
-
-	if (in_atomic() || irqs_disabled()) {
-		INIT_WORK(&conn->cm_exec, defer_func, (void *)conn);
-		schedule_work(&conn->cm_exec);
-	} else
-		defer_func(conn);
-
-	return 0;
-}
-
-/*
- * sdp_cm_disconnect - initiate a disconnect request using the CM
-*/
-int sdp_cm_disconnect(struct sdp_opt *conn)
-{
-	return sdp_cm_deferred_generic(conn, do_cm_disconnect);
-}
-
-/*
- * sdp_cm_reject - initiate a reject request using the CM
- */
-int sdp_cm_reject(struct sdp_opt *conn)
-{
-	return sdp_cm_deferred_generic(conn, do_cm_reject);
-}


From roland at topspin.com  Mon Feb 28 21:50:38 2005
From: roland at topspin.com (Roland Dreier)
Date: Mon, 28 Feb 2005 21:50:38 -0800
Subject: [openib-general] [PATCH] Add PCI device ID for new Mellanox HCA
Message-ID: <52fyzfrk29.fsf@topspin.com>

Hi Greg,

Here's a device ID for the new HCA that Mellanox just introduced.
Please queue this up for 2.6.12.

Thanks,
  Roland


Add PCI device ID for new Mellanox "Sinai" InfiniHost III Lx HCA.

Signed-off-by: Roland Dreier <roland at topspin.com>

--- linux-svn.orig/include/linux/pci_ids.h	2005-02-28 21:08:38.592176783 -0800
+++ linux-svn/include/linux/pci_ids.h	2005-02-28 21:10:37.198431351 -0800
@@ -1992,6 +1992,7 @@
 #define PCI_DEVICE_ID_MELLANOX_TAVOR	0x5a44
 #define PCI_DEVICE_ID_MELLANOX_ARBEL_COMPAT 0x6278
 #define PCI_DEVICE_ID_MELLANOX_ARBEL	0x6282
+#define PCI_DEVICE_ID_MELLANOX_SINAI	0x5e8c
 
 #define PCI_VENDOR_ID_PDC		0x15e9
 #define PCI_DEVICE_ID_PDC_1841		0x1841


From yaronh at voltaire.com  Mon Feb 28 21:51:58 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Tue, 1 Mar 2005 07:51:58 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC78@taurus.voltaire.com>

> -----Original Message-----
> From: Tom Duffy [mailto:tduffy at sun.com]
> Sent: Tuesday, March 01, 2005 1:38 AM
> To: Yaron Haviv
> Cc: Paul Baxter; openib-general at openib.org
> Subject: RE: [openib-general] IB Address Translation service
> 
> On Mon, 2005-02-28 at 21:47 +0200, Yaron Haviv wrote:
> > And as you mentioned there is value to have the same API for
different
> > resolution mechanisms, the SDP code can be altered in future to ride
> > over the proposed API, so it can be used without TCP/IP.
> 
> I am not sure you are gaining much by having SDP use straight ATS.
> Already, once the ARP table is filled with the information, it is a
> local cached lookup.


Tom, the value to Paul is not performance related, but an ability to
resolve an IP to GID without requiring a TCP/IP implementation, I can
think of some applications that would like thin client and still use
Valid IPs

Anyway as I mentioned the ATS support is just one (very minor) thing in
the API we proposed, can I assume you don't have comments to the main
functionality in ib_at ?

Yaron


From yaronh at voltaire.com  Mon Feb 28 22:07:28 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Tue, 1 Mar 2005 08:07:28 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC81@taurus.voltaire.com>

> -----Original Message-----
> From: Libor Michalek [mailto:libor at topspin.com]
> Sent: Tuesday, March 01, 2005 2:04 AM
> To: Yaron Haviv
> Cc: Paul Baxter; openib-general at openib.org
> Subject: Re: [openib-general] IB Address Translation service
> 
> On Mon, Feb 28, 2005 at 09:55:50PM +0200, Yaron Haviv wrote:
> > From: Libor Michalek
> > >
> > > The two are not interoperable, they
> > > reside in parallel, and succeed in producing much confusion. (IMO)
> >
> > One note, the two can be made interoperable, if nodes that use IPoIB
> > register them self in the ATS database as well (which has its merits
for
> > reverse resolution that cannot be satisfied by IPoIB), this way the
> > nodes that just use ATS can locate the IPoIB ones.
> 
>   This relies on each node in a fabric keeping the information between
> the two parallel methods in sync. Which leads to the question, why
have
> two independent methods for getting the exact same information? The
> only logical answer is that there are some nodes which can only use
> one of the methods. In which case the two sets of data are not
identical,
> because of these nodes, which succeeds in producing much confusion.
Not
> to mention the race conditions between keeping a centralized database
> (ATS)
> in sync with the distributed mechanism. (ARP)
> 
>   For these reasons I cringe at hearing IP address and ATS in the
> same sentence, I really wish DAT had chosen a different name for
> the addresses.
> 
>   Really, we all discussed this years ago in the IETF, the merits of
> using broadcast vs. centralized data store, and a solution was
> developed. This is why open standards bodies are so useful.
> 

Libor, I agree with most of your statements here, I also advocated to
use ARP based mechanisms in the DAT calls rather than ATS.
And our DAPL implementation enable ARP based resolution in addition to
ATS 
The one thing that ATS provide and is not possible with ARP is reverse
resolution GID->IP, any ideas how to achieve that without ATS ?

The protocols such as SDP and iSER pass the source IP address as part of
the CM REQ Private Data, so they don't really need the reverse
translation, DAT people have tried to make a generalized mechanism 

I assume James or Arkady should comment on the need for ATS and DAPL
reverse resolution

One other approach can be to provide ATS support to user applications
only, and eliminate the kDAPL support for those functions.
Also I kind of like Paul's application for thin IB clients.

> > Anyway the merits if the proposed API goes much beyond the use of
ATS,
> > so I hope we don't just hang on that one.
> 
>   Agreed, there is certainly a lot more to discuss then just ATS and
ARP.
> 

Any comments on my email explaining the forward resolution mechanisms
(IP->GID, GID->path) ?  (not relating to ATS)

Yaron

> 
> -Libor


From yaronh at voltaire.com  Mon Feb 28 22:18:12 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Tue, 1 Mar 2005 08:18:12 +0200
Subject: [openib-general] IB Address Translation service
Message-ID: <35EA21F54A45CB47B879F21A91F4862F3FAC85@taurus.voltaire.com>

________________________________________
From: openib-general-bounces at openib.org
[mailto:openib-general-bounces at openib.org] On Behalf Of Michael Krause
Sent: Tuesday, March 01, 2005 2:07 AM
To: openib-general at openib.org
Subject: RE: [openib-general] IB Address Translation service

At 11:47 AM 2/28/2005, Yaron Haviv wrote:

>It would be a mistake to attempt to use anything by IP addresses (v4 or
v6) from an >application perspective. Mapping to IB must be application
transparent to be viable.
>
>Mike

Mike, the all idea behind the proposed ib_at calls is to provide
semantics matching between IP and IB in a way where the applications
wont feel the difference, but will still make use of all IB
capabilities. Advanced applications can still use the advanced IB
specific functionality through the optional parameters.

ATS is just a minor option in the API (the less important one)

Yaron


From ogerlitz at voltaire.com  Mon Feb 28 22:18:29 2005
From: ogerlitz at voltaire.com (Or Gerlitz)
Date: Tue, 1 Mar 2005 08:18:29 +0200
Subject: [openib-general] MTHCA features
Message-ID: <D4F8F0B3820E754C887699BEF26A89404A6EC1@taurus.voltaire.com>

Roland > Memory windows are not implemented for mthca.  It wouldn't be a lot of
Roland > work for someone with access to Mellanox documentation to implement
Roland > them, but they're not particularly useful due to their performance characteristics.

By "performance characteristics" do you mean the extra overhead to generate another rkey
for the already registered address range (and also to create/free the mw)?

I guess an app that registers huge portions of memory in batch/startup time can also doing 
mw-ing in this time and not in the fast flow. It then have at hand pool of rkeys which it can 
use and recycle at run time.

Or.


From eitan at mellanox.co.il  Mon Feb 28 22:26:12 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 1 Mar 2005 08:26:12 +0200 
Subject: [openib-general] Question
Message-ID: <506C3D7B14CDD411A52C00025558DED6047EEEF1@mtlex01.yok.mtl.com>

Hal wrote:
> So this looks like a workaround for a bug. Not sure what any of the other
symptoms 
> are but I'm real curious now. Can someone comment more on this ?

The ERR 3610 is really just a warning. It is caused by the Anafa1 chip
responding with a LinearFDBTop 0xC000. OpenSM does know how to handle that
case and fix it.

> At a minimum, the SMA is reporting an invalid value for
PortInfo::LinearFDBTop. I 
> wonder if it also is incapable of forwarding DR MADs as well. That would
explain 
> this.
There are no issues with that switch ability to do DR mads.

EZ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050301/9b9abf9f/attachment.html>

From ebiederman at lnxi.com  Mon Feb 28 23:17:35 2005
From: ebiederman at lnxi.com (Eric W. Biederman)
Date: 01 Mar 2005 00:17:35 -0700
Subject: [openib-general] IB Address Translation service
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F3FAC5D@taurus.voltaire.com>
References: <35EA21F54A45CB47B879F21A91F4862F3FAC5D@taurus.voltaire.com>
Message-ID: <m37jkr3kds.fsf@maxwell.lnxi.com>

"Yaron Haviv" <yaronh at voltaire.com> writes:

> > -----Original Message-----
> > From: openib-general-bounces at openib.org [mailto:openib-general-
> > bounces at openib.org] On Behalf Of Roland Dreier
> > Sent: Monday, February 28, 2005 7:13 PM
> > To: shaharf
> > Cc: openib-general at openib.org
> > Subject: Re: [openib-general] IB Address Translation service
> > 
> > This API seems overly complex and at the same time too inflexible to
> > me.  However, rather than getting bogged down nitpicking about APIs, I
> > think we have to take a few steps back.
> 
> I believe the API is very flexible, but we are pretty open to here what
> you think is needed in addition 
> 
> > First, let's understand the problem we're trying to solve.  Who are
> > the consumers of this address translation service?
> 
> The first problem is that most ULPs use valid IP addresses for
> simplicity (DAPL, iSER, NFS/RDMA, SDP, MPI, etc') and someone needs to
> resolve it to an IB address and device to use IB. This should take into
> account cases where there are more than one HCAs in the system.
> Preferable/optionally the ULP would like to know which partition to use
> if there is more than one, and leverage on the IP subnetting done by
> IPoIB.

I am confused.  In any sane network the translation is:
Hostname -> address.

IP because it spans multiple networks does:
Hostname -> IP address -> hw address.

IB because it can span multiple IB networks does:
GUID+QPN -> LID + QPN.

So what is wrong with simply doing:
Hostname -> GUID
???

Then all the kernel needs to be passed GUID + QPN.

I am certain MPI does not care about IP addresses.   It is the job
of the mpi launcher to resolve where all of the pieces are.  Generally
mpirun is done over IP and it just needs to collect the native network
addresses before it leaves.

It would be brain damaged for DAPL to require IP addresses.  Not that
DAPL hasn't shown some brain damage already.

Please, please remember that IP addresses 

> It is possible to replicate the same code you have in SDP (which is also
> not complete) across all ULP's, I assume a better way is to provide it
> in one central place.

How about not even worrying about it.  It is an extra step that
introduces latency and confusion.  

You can't do GUID -> IP because there is not a requirement on 
a 1 to 1 mapping.  And in general there is no fixed IP -> GUID mapping.

What are the semantics in the upper levels when the IP -> GUID mapping
changes?  Does you connection properly follow the IP to the new GUID?

I don't see this making sense anywhere except user space.

> There are also two proposed address resolution mechanisms, one is ARP
> used by SDP, and one is ATS used by some DAPL consumers, and we believe
> it is better to combine them under the same API.

Just FYI IPv6 doesn't use arp.

> The second problem relates to mapping of IB GID to one or more Path
> records
> This is also something needed for ALL ULP's. today each ULP provides the
> minimal subset of path resolution functionality without taking into
> account topics such as partitioning, QoS, source routing and
> multi-pathing.
> Some of these require using special SA queries (such as SA Multipath
> Record query and QoSPath Query).
> I don't think it make sense to put all this functionality into each ULP
> as well.

That part is reasonable.  Although the fact it is easy to knock
OpenSM down concerns me.  However that looks to be a separate
problem.

> Than we can also discuss, does it make sense to have each path
> resolution call lead us to the sa, or does it make more sense to cache
> those paths.
> And if we cache, doesn't it make more sense to cache/invalidate the
> routes to all ULP's rather implementing/having it in each ULP.
> Also not sure how a 1000 node cluster functions without the caching.
>  
> And the last problem is related to reverse resolution from IB to IP
> addresses that is needed for DAPL, as well as for different management
> and diagnostic tools that want to know what is really that node/port
> behind that GID addresses.
> 
> So how would you suggest to go about it ?
> Duplicate all of that in each ULP ?
> Refrain from implementing advanced routing, partitioning, QoS (we cant
> really maintain all that advanced code for each ULP) ? 

One small step at a time.  Where each step is obviously correct.

One giant leap only works well for internal use.  Not for things
that are heavily used.

> Our idea is to provide those few helper functions that enable people to
> make full use of IB and its features without reading all the IB spec,
> and a Phd.
> If you clear all the remarks from the library, you will see it is very
> slim, and for my understanding includes all the relevant input and
> output parameters for each of the 3 functions I mentioned.

But an interface like that is usually provided by glibc not by the kernel.
At the mixing of levels in that proposed API is absolutely horrible.


Eric