From halr at voltaire.com  Fri Jul  1 03:27:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 01 Jul 2005 06:27:14 -0400
Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support
In-Reply-To: <1120167637.4371.630.camel@hal.voltaire.com>
References: <1120149968.4371.81.camel@hal.voltaire.com>
	<52d5q365uo.fsf@topspin.com>  <42C464E6.8040406@ichips.intel.com>
	<1120167637.4371.630.camel@hal.voltaire.com>
Message-ID: <1120213633.4371.2295.camel@hal.voltaire.com>

On Thu, 2005-06-30 at 17:40, Hal Rosenstock wrote:
> > Given that, I think that it makes more sense to add a length field to the 
> > ib_user_mad_hdr.
> 
> That's my conclusion too. ABI change :-(

Is there a need to any backward compatibility ? Specifically should the
current ABI version (4) be supported as well as the one this will be
bumped to (5) or just the new one ?

-- Hal



From mst at mellanox.co.il  Fri Jul  1 05:07:00 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 1 Jul 2005 15:07:00 +0300
Subject: [openib-general] Re: [PATCH (draft)] sdp: fix aio/sync completion
	race
In-Reply-To: <20050630181636.A29268@topspin.com>
References: <20050506175415.I26823@topspin.com>
	<20050510143232.GF2474@mellanox.co.il>
	<20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com>
	<20050513135145.GA320@mellanox.co.il>
	<20050513100149.B24123@topspin.com>
	<20050525202128.GA7463@mellanox.co.il>
	<20050525172954.A2979@topspin.com>
	<20050628201857.GA5987@mellanox.co.il>
	<20050630181636.A29268@topspin.com>
Message-ID: <20050701120700.GA10889@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [PATCH (draft)] sdp: fix aio/sync completion race
> 
> On Tue, Jun 28, 2005 at 11:18:57PM +0300, Michael S. Tsirkin wrote:
> > Libor, here's a stub at solving an old problem.
> > Most of the patch is passing sdp_opt to iocb complete and cancel functions.
> > 
> > I didnt yet test it, and I wont have the time today,
> > but maybe you could tell me whether I'm going in the right
> > direction with this.
> 
>   I'm wondering if we should prevent a new call from proceeding until
> all IOCBs have been completed, like the patch you have, or if we should
> force the creation of an IOCB when there are any IOCBs outstanding,
> even if the call can be processed synchronously.
> 
>   Remember the original problem was IOCB completes in the work queue
> completing after a call to send/recv that was processed synchronously.
> Either forcing an IOCB or waiting on the lock should prevent the
> ordering issue.

You mean using zcopy for packets below zcopy threshold?

>   I think either case if means passing the sdp_opt to iocb_complete. In
> do_iocb_complete you cannot call conn_unlock unconditionally since
> the lock.users variable is only incremented in_atomic() or irqs_disabled()
> Also, I think the conn reference needs to be incremented before
> creating the work request, and decremeneted upon completion since
> nothing prevents it from going away before the thread is scheduled.
> 

Libor, thanks for the comments.
I'll work on addressing them.

-- 
MST


From jlentini at netapp.com  Fri Jul  1 07:18:51 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 1 Jul 2005 10:18:51 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] kDAPL: remove redundant kfree checks
In-Reply-To: <1120172491.29522.28.camel@duffman>
References: <1120172491.29522.28.camel@duffman>
Message-ID: <Pine.LNX.4.61.0507011018140.29731@jlentini-linux.nane.netapp.com>



On Thu, 30 Jun 2005, Tom Duffy wrote:

tduffy> kfree() already checks for NULL.  No need to do it twice.
tduffy> 
tduffy> Signed-off-by: Tom Duffy <tduffy at sun.com>

Committed in revision 2765.


From caitlinb at siliquent.com  Fri Jul  1 08:59:06 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Fri, 1 Jul 2005 08:59:06 -0700
Subject: [openib-general] Making gen2 transport neutral
Message-ID: <8508251A6FC08A489844A94261D3693A057CA1@fiona.siliquent.com>

While waiting to see if any of the champions of the
union form appear, I'd like to ask what the enum
equivalent of the non-union struct strategy would
look like.

I'm guessing it would take something like

	enum rdma_xyz {
		/* common values */
		RDMA_XYZ_A,
		RDMA_XYZ_B,
		RDMA_XYZ_C,

		/* IB specific values */
		IB_XYZ_D,
		IB_XYZ_E,

		/* iWARP specific values */
		IWARP_XYZ_F = RDMA_XYZ_C+1,
		IWARP_XYZ_G
	}

to 

	enum rdma_xyz { /* common values */
		RDMA_XYZ_A,
		RDMA_XYZ_B,
		RDMA_XYZ_C,
		RDMA_XYZ_LIM /* must be last */
	};

	enum ib_xyz { /* extends rdma_xyz */
		IB_XYZ_D = RDMA_XYZ_LIM,
		IB_XYZ_E
	};

	enum iwarp_xzy { /* extends iwarp_xyz */
		IWARP_XYZ_F = RDMA_XYZ_LIM,
		IWARP_XYZ_G;
	};

That way the transport dependent enums could even
be in transport dependent .h files, and only the
common ones would have to be in the main header file.
That would be very friendly for adding a third
transport at some later date (I still don't think
it will happen, but it would make it easier).
		
		 

> -----Original Message-----
> From: Christoph Hellwig [mailto:hch at lst.de] 
> Sent: Thursday, June 30, 2005 9:03 AM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: Re: [openib-general] Making gen2 transport neutral
> 
> On Thu, Jun 30, 2005 at 08:52:54AM -0700, Caitlin Bestler wrote:
> > structs:
> > typically "struct ib_xyz" is transformed as follows:
> > 
> > 	struct ib_xyz {
> > 		/* Only IB specific fields remain */
> > 		/* In some cases fields have been split, because
> >              * iWARP allows two things to vary that IB had
> >              * locked together. SGE limits are the primary
> >              * example of this. iWARP can have different
> > 	 	 * limits on SGE size for each type of message
> > 		 */
> > 	};
> > 
> > 	struct iwarp_xyz {
> > 		/* equivalent iWARP specific fields */
> > 	};
> > 
> > 	struct rdma_xyz {
> > 		/* Transport neutral fields. Typically
> > 		 * a subset of what was in struct ib_xyz before
> > 		 */
> > 		union {
> > 			struct ib_xyz ib;
> > 			struct iwarp_xyz iwarp;
> > 		} xpt;
> > 	};
> 
> wrong way around, but we had that before.  It should be
> 
> struct ib_foo {
> 	struct rdma_foo common;
> 	...
> }
> 
> struct iwarp_foo {
> 	struct rdma_foo common;
> 	...
> }
> 
> see filesystem and network protocol private data for example 
> where why historically did it that union way and it didn't 
> work out at all long-term.
> 
> > I am assuming that returning a "not supported" error for irrelevant 
> > verbs is an acceptable burnden for all providers.
> 
> Even better make the methods implementing them optional and 
> let the upper layer return EOPNOTSUPP when it's not implemeneted.
> 
> 


From hch at lst.de  Fri Jul  1 09:02:51 2005
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 1 Jul 2005 18:02:51 +0200
Subject: [openib-general] Making gen2 transport neutral
In-Reply-To: <8508251A6FC08A489844A94261D3693A057CA1@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057CA1@fiona.siliquent.com>
Message-ID: <20050701160251.GA9380@lst.de>

On Fri, Jul 01, 2005 at 08:59:06AM -0700, Caitlin Bestler wrote:
> to 
> 
> 	enum rdma_xyz { /* common values */
> 		RDMA_XYZ_A,
> 		RDMA_XYZ_B,
> 		RDMA_XYZ_C,
> 		RDMA_XYZ_LIM /* must be last */
> 	};
> 
> 	enum ib_xyz { /* extends rdma_xyz */
> 		IB_XYZ_D = RDMA_XYZ_LIM,
> 		IB_XYZ_E
> 	};
> 
> 	enum iwarp_xzy { /* extends iwarp_xyz */
> 		IWARP_XYZ_F = RDMA_XYZ_LIM,
> 		IWARP_XYZ_G;
> 	};
> 
> That way the transport dependent enums could even
> be in transport dependent .h files, and only the
> common ones would have to be in the main header file.
> That would be very friendly for adding a third
> transport at some later date (I still don't think
> it will happen, but it would make it easier).

looks fine.



From caitlinb at siliquent.com  Fri Jul  1 09:14:46 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Fri, 1 Jul 2005 09:14:46 -0700
Subject: [openib-general] gen2/rnic-pi differences
Message-ID: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>

The wr_id field is the "minimal support" that I was
referring to (that and the context on callbacks).

DAPL and IT-API really need more than that to optimally
implement features such as:
	storing the ULP's 64-bit cookie.

	controlling "notification status" on a per-DTO level
	rather than just always leaving the CQ on a "next event"
	basis.

	implementing graceful disconnect (disconnect after
	the last send request completes).

	allowing the ULP to flush a request through as
	a marker.

	returning the EP or RMR pointer with the completion.

The 64 bit wr_id is fully consumed just implementing
the first option.

Without further support the Access Layer (DAT/IT-API/whatever)
must create its own parallel data structure to shadow the
work request (a DTO_COOKIE in the reference implementation).

Such a structure has been found in every DAT implementation
that I am aware of to date for the simple reason that the
verb layers have not offered much support beyond the 64-bit
work request id and a context field on callbacks.

But it is in fact very easy to solve all of these problems
at the verb layer, even if there is no hardware support.

RNIC-PI defined several verb layer features that if supported
eliminate the need for a DTO_COOKIE. If the information can
be integrated with existing verb layer structures it is a 
major improvement in efficiency, at worst case it merely
requires the verb layer to implement the same workarounds
that the Access Layer is already forced to use.

These features are:
	all verb layer objects have a consumer supplied 
	identifier (os_data) that is used to identify that
	object back to the consumer in all completions and
	callbacks. So instead of getting the QPID you get
	the EP pointer (assuming that is what you supplied).

	Three flags are identified per work request that can
	be ignored, passed through or fully implemented. They
	are Local Solicited, Consumer0 and Consumer1. If 'Local
	Solicited' is defined it means that completion of the
	work request should be treated as though it were a
	solicited event (i.e., it qualifies for 'next solicited
	event' callback notification).

with these changes there is no need for a DTO_COOKIE, which
reduces the number of memory touches, simplifies code, and
may eliminate associated locking to protect the DTO_COOKIE
resource itself.


	
	 

> -----Original Message-----
> From: James Lentini [mailto:jlentini at netapp.com] 
> Sent: Thursday, June 30, 2005 3:18 PM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: Re: [openib-general] gen2/rnic-pi differences
> 
> 
> 
> On Thu, 30 Jun 2005, Caitlin Bestler wrote:
> <snip>
> > os_data / Identification of Consumer Objects
> >
> > 	gen2 provides minimal support for identification of RDMA
> > 	resources using consumer supplied handles. A user-supplied
> > 	context is available in callbacks, but not in work completions.
> 
> Gen2 work completions store a user-supplied context in the 
> ib_wc structure's wr_id field.
> 


From ftillier at silverstorm.com  Fri Jul  1 09:37:56 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Fri, 1 Jul 2005 09:37:56 -0700
Subject: [openib-general] gen2/rnic-pi differences
In-Reply-To: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>
Message-ID: <000501c57e5b$3e287650$6501a8c0@infiniconsys.com>

> From: Caitlin Bestler [mailto:caitlinb at siliquent.com]
> Sent: Friday, July 01, 2005 9:15 AM
> 
> RNIC-PI defined several verb layer features that if supported
> eliminate the need for a DTO_COOKIE. If the information can
> be integrated with existing verb layer structures it is a
> major improvement in efficiency, at worst case it merely
> requires the verb layer to implement the same workarounds
> that the Access Layer is already forced to use.
> 
> These features are:
> 	all verb layer objects have a consumer supplied
> 	identifier (os_data) that is used to identify that
> 	object back to the consumer in all completions and
> 	callbacks. So instead of getting the QPID you get
> 	the EP pointer (assuming that is what you supplied).

This should be really easy to implement for Mellanox HCAs - the mthca driver
already has to resolve the QP structure when processing completions, and getting
the user's QP context and including it in the work completion should be a
trivial addition (for someone familiar with the code base).

> 	Three flags are identified per work request that can
> 	be ignored, passed through or fully implemented. They
> 	are Local Solicited, Consumer0 and Consumer1. If 'Local
> 	Solicited' is defined it means that completion of the
> 	work request should be treated as though it were a
> 	solicited event (i.e., it qualifies for 'next solicited
> 	event' callback notification).

Would these flags be returned in the work completion?  I don't know if I quite
understand what you're requesting here.  Do Consumer0 and Consumer1 represent
bits in a flags field?

In the Mellanox HCA implementation, the 64-bit work request ID is stored by the
driver and recovered upon a completion.  Basically, the HCA driver maintains
DTO_COOKIE-like information for each work request already.  Due to this lookup
requirement, the information stored per work request could be arbitrarily large
if so desired.  I don't know if that holds true for the PathScale HCA hardware,
though - if it doesn't it would require an additional lookup in the HCA driver
to get back to this information, in effect pushing the DTO_COOKIE concept into
the HCA driver rather than leaving it in the consumer.

- Fab



From mshefty at ichips.intel.com  Fri Jul  1 09:42:19 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 01 Jul 2005 09:42:19 -0700
Subject: [openib-general] gen2/rnic-pi differences
In-Reply-To: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>
Message-ID: <42C5726B.1070604@ichips.intel.com>

Caitlin Bestler wrote:
> The wr_id field is the "minimal support" that I was
> referring to (that and the context on callbacks).
> 
> DAPL and IT-API really need more than that to optimally
> implement features such as:
> 	storing the ULP's 64-bit cookie.
> 
> 	controlling "notification status" on a per-DTO level
> 	rather than just always leaving the CQ on a "next event"
> 	basis.
> 
> 	implementing graceful disconnect (disconnect after
> 	the last send request completes).
> 
> 	allowing the ULP to flush a request through as
> 	a marker.
> 
> 	returning the EP or RMR pointer with the completion.
> 
> The 64 bit wr_id is fully consumed just implementing
> the first option.
> 
> Without further support the Access Layer (DAT/IT-API/whatever)
> must create its own parallel data structure to shadow the
> work request (a DTO_COOKIE in the reference implementation).

The assumption being made here is that there needs to be another abstraction 
layer on top of the existing core layer.  I don't think that an optimal 
implementation would want this.

- Sean


From caitlin.bestler at gmail.com  Fri Jul  1 09:44:26 2005
From: caitlin.bestler at gmail.com (Caitlin Bestler)
Date: Fri, 1 Jul 2005 09:44:26 -0700
Subject: [openib-general] gen2/rnic-pi differences
In-Reply-To: <000501c57e5b$3e287650$6501a8c0@infiniconsys.com>
References: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>
	<000501c57e5b$3e287650$6501a8c0@infiniconsys.com>
Message-ID: <469958e0050701094493bc8ea@mail.gmail.com>

On 7/1/05, Fab Tillier <ftillier at silverstorm.com> wrote:
> > From: Caitlin Bestler [mailto:caitlinb at siliquent.com]
> > Sent: Friday, July 01, 2005 9:15 AM
> >
> > RNIC-PI defined several verb layer features that if supported
> > eliminate the need for a DTO_COOKIE. If the information can
> > be integrated with existing verb layer structures it is a
> > major improvement in efficiency, at worst case it merely
> > requires the verb layer to implement the same workarounds
> > that the Access Layer is already forced to use.
> >
> > These features are:
> >       all verb layer objects have a consumer supplied
> >       identifier (os_data) that is used to identify that
> >       object back to the consumer in all completions and
> >       callbacks. So instead of getting the QPID you get
> >       the EP pointer (assuming that is what you supplied).
> 
> This should be really easy to implement for Mellanox HCAs - the mthca driver
> already has to resolve the QP structure when processing completions, and getting
> the user's QP context and including it in the work completion should be a
> trivial addition (for someone familiar with the code base).
> 

That was the basic assumption, that the verb layer drivers
could implement these features more easily than DAT/IT-API
could because they would simply require fields in
existing data structures rather than totally separate
parallel data structures.

> >       Three flags are identified per work request that can
> >       be ignored, passed through or fully implemented. They
> >       are Local Solicited, Consumer0 and Consumer1. If 'Local
> >       Solicited' is defined it means that completion of the
> >       work request should be treated as though it were a
> >       solicited event (i.e., it qualifies for 'next solicited
> >       event' callback notification).
> 
> Would these flags be returned in the work completion?  I don't know if I quite
> understand what you're requesting here.  Do Consumer0 and Consumer1 represent
> bits in a flags field?
> 

Yes. The three flags end up being pass-throughs to the
work completion, although the Local Solicited bit can be
merged with the Remote Solicited bit without harm (you
don't really need to know which end set the bit).

> In the Mellanox HCA implementation, the 64-bit work request ID is stored by the
> driver and recovered upon a completion.  Basically, the HCA driver maintains
> DTO_COOKIE-like information for each work request already.  Due to this lookup
> requirement, the information stored per work request could be arbitrarily large
> if so desired.  I don't know if that holds true for the PathScale HCA hardware,
> though - if it doesn't it would require an additional lookup in the HCA driver
> to get back to this information, in effect pushing the DTO_COOKIE concept into
> the HCA driver rather than leaving it in the consumer.
> 

Exactly. Any implementation that has access to the original send
queue when reaping the work completion can easily extend the
64-bit work request ID with additional data.

I suspect that this is a very common implementation because
there isn't much purpose in actually sending the 64-bit ID out
over the system bus just to have it be shipped back in the
completion. Why pay for storage on the card to hold the 64-bit
wr id for every pending work request? And why pay to ship
it across the system bus twice?


From caitlin.bestler at gmail.com  Fri Jul  1 09:53:06 2005
From: caitlin.bestler at gmail.com (Caitlin Bestler)
Date: Fri, 1 Jul 2005 09:53:06 -0700
Subject: [openib-general] gen2/rnic-pi differences
In-Reply-To: <42C5726B.1070604@ichips.intel.com>
References: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>
	<42C5726B.1070604@ichips.intel.com>
Message-ID: <469958e005070109534d6c8407@mail.gmail.com>

On 7/1/05, Sean Hefty <mshefty at ichips.intel.com> wrote:
> Caitlin Bestler wrote:

> 
> The assumption being made here is that there needs to be another abstraction
> layer on top of the existing core layer.  I don't think that an optimal
> implementation would want this.
> 

I believe that any application using the core layer directly would have
the same need to identify *it's* context efficiently when processing
a work completion.

The QP ID does not truly meet that requirement. It is an arbitrary
integer selected by another software package. You have to add
your own reverse indexing or use the work request ID to fully
identify your context, which means that it cannot also be
promised to a higher layer.

So I think this problem is generic to *any* middleware consumer,
whether it is DAPL, IT-API, MPI, the communications layer of
a database, RPC, etc.

The Work Request ID by itself will typically leave any middleware
consumer forced to create a "DTO Coookie" type solution.


And while I believe that we should *allow* consumers to use the
core layer directly, I do not believe we should *encourage* it.
Middleware layers such as DAPL, IT-API or an MPI messaging
system are enable the application to focus on application issues
rather than on wire issues. That's why kDAPL has been used,
and why we should be concerned with enabling efficient middleware
even if we allow consumers to bypass it.


From sato at make-love.cx  Fri Jul  1 09:55:14 2005
From: sato at make-love.cx (make-love  )
Date: Sat, 02 Jul 2005 01:55:14 +0900
Subject: [openib-general] =?iso-2022-jp?b?GyRCOkZFWSQ0QXc/LiQ1JDsbKEI=?=
	=?iso-2022-jp?b?GyRCJEZEOiQtJF4kOSEjGyhC?=
Message-ID: <20050701232750.828.MAIL>

先日は私共の手違いによりメールが届いていない方が多数いらっしゃったそうですので
この場を借りてお詫び申し上げます。

今後はこのようなことがないよう全力で努めさせて頂く所存ですのでこれからも
よろしくお願い致します。

今回の企画は『身元の確かな出会いに縁のない女性会員様と
誠実で秘密厳守を守れる男性会員様』が出会うという素晴らしい企画でございます。

先日会員様を募集したところ、8000名以上の女性会員の方が集いました。
しかし残念なことに男性会員様は700名という形で終りました。
そこで今回は男性会員様を大募集致します。

Make-up Loveでは女性会員様の会費で運営されております。
お会いする場合にはくれぐれも失礼のないようお願い致します。
また男性会員様におきまして登録料・紹介料は一切頂いておりません。

当クラブには様々な女性会員様が在籍しております。
学生、ＯＬ、フリーター、ニート、セレブ、淫乱マダム、ＳＭ、３Ｐ、合コン
などなど夢のようです。


日本全国に女性会員様が多数在籍しておりますので、
まずは貴方様の地域に該当している女性会員様をご紹介させて頂きたいと思います。

簡単なシートですので、まずはお気軽にお書き下さいませ。


Ｑ：ＮＯ,１
　　貴方様のお会いするにあたってのご希望の地域(都道府県等)


Ｑ：ＮＯ,２
 　　当クラブをご利用するにあたってのご利用タイプ
【タイプ1】真面目に一般女性とお付き合い　　（真面目な交際・結婚前提） 
【タイプ2】女性スポンサーとお付き合い　　　（事業資金などサポートして欲しい） 
【タイプ3】割り切った大人のお付き合い　　　（愛人・セフレ・人妻） 
【タイプ4】多趣味にそして過激なお付き合い　（3P、乱交・ＳＭ） 
【タイプ特別】40歳以上同士の新しい交際



以上のご質問にお答え頂きご返信頂ければ、当クラブの各地域担当女性から
貴方様のご希望に沿った女性をご案内させて頂きます。

メール会報に登録した覚えがないという方や同様のメールが届いていると
いう場合もあるかと思いますがご了承下さい。

当クラブ「Make-up Love」の会報メールがご不要な場合にはそのまま破棄してください。
尚、「配信不要」とお送りいただければ会報メールは二度と配信致しません。

ここまで読んで「嘘くさい」「配信不要」と思った方々。
少々お考え下さい。
ここで削除、配信不要とすることは簡単です。
しかし、こんな機会は一生に一度あるかどうかと思われます。

やはりお金の心配をなさっている方もいるのではないでしょうか？
皆様の喜ぶの声を聞かせて頂ければ、それだけで充分でございます。

最後になりますが、「配信不要」と書こうと思った方も
どうか地域、ご利用のタイプをご明記ください。
直ちに女性会員の方々をご紹介させて頂きます。

ご連絡お待ちしております。

 『club　Make-up Love【メイク・ラブ】』代表　藁科美智子




From halr at voltaire.com  Fri Jul  1 09:58:50 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 01 Jul 2005 12:58:50 -0400
Subject: [openib-general] Service Record support in SA query
Message-ID: <1120237130.4371.3616.camel@hal.voltaire.com>

Hi Roland,

Any more comments on the Service Record support in SA query patch or is
this OK to commit ?

Thanks.

-- Hal



From mshefty at ichips.intel.com  Fri Jul  1 10:22:20 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 01 Jul 2005 10:22:20 -0700
Subject: [openib-general] gen2/rnic-pi differences
In-Reply-To: <469958e005070109534d6c8407@mail.gmail.com>
References: <8508251A6FC08A489844A94261D3693A057CA2@fiona.siliquent.com>	
	<42C5726B.1070604@ichips.intel.com>
	<469958e005070109534d6c8407@mail.gmail.com>
Message-ID: <42C57BCC.3050005@ichips.intel.com>

Caitlin Bestler wrote:
> And while I believe that we should *allow* consumers to use the
> core layer directly, I do not believe we should *encourage* it.
> Middleware layers such as DAPL, IT-API or an MPI messaging
> system are enable the application to focus on application issues
> rather than on wire issues. That's why kDAPL has been used,
> and why we should be concerned with enabling efficient middleware
> even if we allow consumers to bypass it.

The core layer should be designed to encourage, not discourage, its use, 
otherwise it has the wrong abstraction.

- Sean


From halr at voltaire.com  Fri Jul  1 10:24:37 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 01 Jul 2005 13:24:37 -0400
Subject: [openib-general] [oops] recent opensm crash
In-Reply-To: <1120169407.29522.17.camel@duffman>
References: <1120169407.29522.17.camel@duffman>
Message-ID: <1120238676.4371.3718.camel@hal.voltaire.com>

On Thu, 2005-06-30 at 18:10, Tom Duffy wrote:
> #0  stack_dump () at src/stack.c:72
> 72              if (!__builtin_frame_address(2))
> (gdb) bt
> #0  stack_dump () at src/stack.c:72
> #1  0x00002aaaaacbd1a6 in handler (x=11) at src/stack.c:151
> #2  <signal handler called>
> #3  __osm_sm_mad_ctrl_send_err_cb (bind_context=0x550dd8, p_madw=0x567820)
>     at osm_sm_mad_ctrl.c:832
> #4  0x00002aaaaaaaeeed in osm_vendor_send (h_bind=0x586920, p_madw=0x567820,
>     resp_expected=1) at osm_vendor_ibumad.c:889

I found one problem associated with this and just checked in a patch.
I'm not sure whether there is another one behind this or not. Any
reliable way to recreate this ?

-- Hal

> #5  0x000000000042ef72 in __osm_vl15_poller (p_ptr=0x552620) at osm_madw.h:933
> #6  0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x0) at cl_thread.c:61
> #7  0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0
> #8  0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> I was bringing up and down an node when this happened.
> 
> Attached are the last 500 lines from osm.log.
> 
> -tduffy
> 
> ______________________________________________________________________
> 
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x1 (SubnGet)
> 				status..................0x0
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6a
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x50000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][0]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: [
> Jun 30 14:59:00 [43005960] -> PortInfo dump:
> 				port number.............0x8
> 				node_guid...............0x000617000000000d
> 				port_guid...............0x000617000000000d
> 				m_key...................0x0000000000000000
> 				subnet_prefix...........0x0000000000000000
> 				base_lid................0x0
> 				master_sm_base_lid......0x0
> 				capability_mask.........0x0
> 				diag_code...............0x0
> 				m_key_lease_period......0x0
> 				local_port_num..........0x7
> 				link_width_enabled......0x3
> 				link_width_supported....0x3
> 				link_width_active.......0x0
> 				link_speed_supported....0x1
> 				port_state..............DOWN
> 				state_info2.............0x22
> 				m_key_protect_bits......0x0
> 				lmc.....................0x0
> 				link_speed..............0x11
> 				mtu_smsl................0x10
> 				vl_cap..................0x40
> 				vl_high_limit...........0x0
> 				vl_arb_high_cap.........0x20
> 				vl_arb_low_cap..........0x20
> 				mtu_cap.................0x5
> 				vl_stall_life...........0x8
> 				vl_enforce..............0x10
> 				m_key_violations........0x0
> 				p_key_violations........0x0
> 				q_key_violations........0x0
> 				guid_cap................0x0
> 				subnet_timeout..........0x0
> 				resp_time_value.........0x0
> 				error_threshold.........0xFF
> Jun 30 14:59:00 [43005960] -> Capabilities Mask:
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: [
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: [
> Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43806960] -> __osm_mtl_send_callback: Completed Sending Request MADW: 0x5b0ac0.
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: ]
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591a50.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: 1 on wire, 11 outstanding, 10 unicasts sent, 88641 sent total.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_NODE_INFO.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567428, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x592620, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x560268, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x591a50, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x567410, p_mad = 0x592654, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x560250, p_mad = 0x591a84, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88641 QP0 MADs received.
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x80000, TID = 0x16c6d.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x567410 (mad 0x592654 req 1)
> Jun 30 14:59:00 [44808960] -> SMP dump:
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x81 (SubnGetResp)
> 				status..................0x8000
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6a
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x50000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][7]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 1 MADs on wire, 12 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x5930e4.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x5930b0.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x567820 (mad 0x591bc4 req 1)
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43806960] -> SMP dump:
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x1 (SubnGet)
> 				status..................0x0
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6b
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x60000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][0]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: ]
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ]
> Jun 30 14:59:00 [43005960] -> osm_pi_rcv_process: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0ad8, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x5930b0, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0ac0, p_mad = 0x5930e4, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88642 QP0 MADs received.
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: Send failed -5 (Success).
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: [
> Jun 30 14:59:00 [44808960] -> SMP dump:
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x81 (SubnGetResp)
> 				status..................0x8000
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6b
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x60000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][7]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x567820, p_mad = 0x591bc4.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591b90.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c62.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0370, p_mad = 0x5b1244.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b1210.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 11 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c63.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x0
> 	P_Key Table:  0XFFFF | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c63.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55df60, p_mad = 0x5b0c74.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b0c40.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 10 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c64.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x1
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c64.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0920, p_mad = 0x5916c4.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591690.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 9 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c65.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x2
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c65.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55add0, p_mad = 0x591d04.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591cd0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [
> Jun 30 14:59:00 [43005960] -> NodeInfo dump:
> 				base_version............0x1
> 				class_version...........0x1
> 				node_type...............Channel Adapter
> 				num_ports...............0x2
> 				sys_guid................0x0002c9000100d050
> 				node_guid...............0x0002c901097624c0
> 				port_guid...............0x0002c901097624c1
> 				partition_cap...........0x40
> 				device_id...............0x5A44
> 				revision................0xA1
> 				port_num................0x1
> 				vendor_id...............0x2C9
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c901097624c0
> 				TID = 0x16c66, discovered 0 times already.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x55ade8, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b1210, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x55add0, p_mad = 0x5b1244, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6e.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x55add0 (mad 0x5b1244 req 1)
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 9 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c66.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b09f0, p_mad = 0x592fa4.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592f70.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 3 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c67.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x3
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c67.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x561f90, p_mad = 0x592e64.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592e30.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 7 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 4 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c68.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x4
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c68.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0850, p_mad = 0x591804.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5917d0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [
> Jun 30 14:59:00 [43005960] -> NodeInfo dump:
> 				base_version............0x1
> 				class_version...........0x1
> 				node_type...............Channel Adapter
> 				num_ports...............0x2
> 				sys_guid................0x0002c90109765633
> 				node_guid...............0x0002c90109765630
> 				port_guid...............0x0002c90109765631
> 				partition_cap...........0x20
> 				device_id...............0x5A44
> 				revision................0xA1
> 				port_num................0x1
> 				vendor_id...............0x2C9
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c90109765630
> 				TID = 0x16c69, discovered 0 times already.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0868, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b0c40, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0850, p_mad = 0x5b0c74, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6f.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5b0850 (mad 0x5b0c74 req 1)
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 7 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c69.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b02a0, p_mad = 0x592144.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592110.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 5 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6a.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x5
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6a.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x560250, p_mad = 0x592514.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5924e0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 5 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 6 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6b.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x6
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6b.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x591a84.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591a50.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 4 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_SUCCESS).
> 
> ______________________________________________________________________
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Fri Jul  1 11:31:26 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 01 Jul 2005 14:31:26 -0400
Subject: [openib-general] [PATCH v3] user_mad: Add receive side RMPP support
Message-ID: <1120242686.4371.3804.camel@hal.voltaire.com>

user_mad: Add receive side RMPP support
This change involves another ABI_VERSION change.

[NOTE: Please do not commit. This needs to be coordinated with a 
change to userspace.]

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: infiniband/include/ib_user_mad.h
===================================================================
--- infiniband/include/ib_user_mad.h	(revision 2757)
+++ infiniband/include/ib_user_mad.h	(working copy)
@@ -43,7 +43,7 @@
  * Increment this value if any changes that break userspace ABI
  * compatibility are made.
  */
-#define IB_USER_MAD_ABI_VERSION	4
+#define IB_USER_MAD_ABI_VERSION	5
 
 /*
  * Make sure that all structs defined in this file remain laid out so
@@ -78,6 +78,7 @@
 	__u32	status;
 	__u32	timeout_ms;
 	__u32	retries;
+	__u32	length;
 	__u32	qpn;
 	__u32   qkey;
 	__u16	lid;
Index: infiniband/core/user_mad.c
===================================================================
--- infiniband/core/user_mad.c	(revision 2760)
+++ infiniband/core/user_mad.c	(working copy)
@@ -176,7 +176,7 @@
 	if (mad_recv_wc->wc->status != IB_WC_SUCCESS)
 		goto out;
 
-	length = 256;	/* until RMPP is supported */
+	length = mad_recv_wc->mad_len; 
 	packet = kmalloc(sizeof *packet + length, GFP_KERNEL);
 	if (!packet)
 		goto out;
@@ -184,8 +184,10 @@
 	memset(packet, 0, sizeof *packet + length);
 	packet->length = length;
 
-	memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, length);
+	ib_coalesce_recv_mad(mad_recv_wc, packet->mad.data);
+
 	packet->mad.hdr.status    = 0;
+	packet->mad.hdr.length    = length + sizeof (struct ib_user_mad);
 	packet->mad.hdr.qpn 	  = cpu_to_be32(mad_recv_wc->wc->src_qp);
 	packet->mad.hdr.lid 	  = cpu_to_be16(mad_recv_wc->wc->slid);
 	packet->mad.hdr.sl  	  = mad_recv_wc->wc->sl;
@@ -214,7 +216,7 @@
 	struct ib_umad_packet *packet;
 	ssize_t ret;
 
-	if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ 
+	if (count < sizeof (struct ib_user_mad) + sizeof (struct ib_mad))
 		return -EINVAL;
 
 	spin_lock_irq(&file->recv_lock);
@@ -237,9 +239,14 @@
 
 	spin_unlock_irq(&file->recv_lock);
 
-	if (count < packet->length + sizeof (struct ib_user_mad))
-		ret = -EINVAL;
-	else if (copy_to_user(buf, &packet->mad,
+	if (count < packet->length + sizeof (struct ib_user_mad)) {
+		/* Return length needed (and first RMPP segment) if too small */
+		if (copy_to_user(buf, &packet->mad,
+				 sizeof (struct ib_user_mad) + sizeof (struct ib_mad)))
+			ret = -EFAULT;
+		else
+			ret = -ENOSPC;
+	} else if (copy_to_user(buf, &packet->mad,
 			      packet->length + sizeof (struct ib_user_mad)))
 		ret = -EFAULT;
 	else
Index: docs/user_mad.txt
===================================================================
--- docs/user_mad.txt	(revision 2757)
+++ docs/user_mad.txt	(working copy)
@@ -28,16 +28,38 @@
 
 Receiving MADs
 
-  MADs are received using read().  The receive side does not currently
-  support RMPP so the buffer passed to read() must be at least one
+  MADs are received using read().  The receive side now supports
+  RMPP. The buffer passed to read() must be at least one 
   struct ib_user_mad + 256 bytes. For example:
 
+  If the buffer passed is not large enough to hold the received
+  MAD (RMPP), the errno is set to ENOSPC and the length of the
+  buffer needed is set in mad.length.
+
+  Example for normal MAD (non RMPP) reads:
 	struct ib_user_mad *mad;
 	mad = malloc(sizeof *mad + 256);
 	ret = read(fd, mad, sizeof *mad + 256);
-	if (ret != sizeof mad + 256)
+	if (ret != sizeof mad + 256) {
 		perror("read");
+		free(mad);
+	}
 
+  Example for RMPP reads:
+	struct ib_user_mad *mad;
+	mad = malloc(sizeof *mad + 256);
+	ret = read(fd, mad, sizeof *mad + 256);
+	if (ret == -ENOSPC)) {
+		length = mad.length;
+		free(mad);
+		mad = malloc(sizeof *mad + length);
+		ret = read(fd, mad, sizeof *mad + length);
+	}
+	if (ret < 0) {
+		perror("read");
+		free(mad);
+	}
+
   In addition to the actual MAD contents, the other struct ib_user_mad
   fields will be filled in with information on the received MAD.  For
   example, the remote LID will be in mad.lid.





From mshefty at ichips.intel.com  Fri Jul  1 14:41:11 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 01 Jul 2005 14:41:11 -0700
Subject: [openib-general] sending RMPP non-data messages
Message-ID: <42C5B877.5040901@ichips.intel.com>

Looking at the kernel RMPP implementation, I think that we want to ensure 
that an application doesn't send a non-DATA RMPP MAD.  The RMPP code would 
interpret such a MAD as being internally generated when it wasn't.

- Sean


From jlentini at netapp.com  Fri Jul  1 14:46:13 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 1 Jul 2005 17:46:13 -0400 (EDT)
Subject: [openib-general] comments on DAT registry in OpenIB
In-Reply-To: <20050630104845.GA11393@lst.de>
References: <C98692FD98048C41885E0B0FACD9DFB8E1225D@exnane01.hq.netapp.com>
	<20050630104845.GA11393@lst.de>
Message-ID: <Pine.LNX.4.61.0507011734160.31881@jlentini-linux.nane.netapp.com>



On Thu, 30 Jun 2005, Christoph Hellwig wrote:

> Could you please stop that comitte crap?
>
> James, what do you think about doing an s/DAT/RDMA/ and s/dat/rdma/
> on the code so we can stop this endless mess?

If we explain the changes we're making, there should be no need to 
do that.

> In the end it won't look like dat anyway, and the sooner why make 
> that absolutely clear that less idiocy like this is going to happen.


From jlentini at netapp.com  Fri Jul  1 14:50:13 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 1 Jul 2005 17:50:13 -0400 (EDT)
Subject: [openib-general] comments on DAT registry in OpenIB
In-Reply-To: <52slyzakdv.fsf@topspin.com>
References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408>
	<52slyzakdv.fsf@topspin.com>
Message-ID: <Pine.LNX.4.61.0507011748370.31881@jlentini-linux.nane.netapp.com>



On Thu, 30 Jun 2005, Roland Dreier wrote:

>    Robert> I think that your suggestion to s/DAT/RDMA makes sense,
>    Robert> since this code is quickly becoming "the" RDMA transport
>    Robert> independent interface for Linux, rather than trying to
>    Robert> RNIC-PI unionize the IB core layer to make it support both
>    Robert> IB and iWarp.
>
> I disagree.  It doesn't make sense to me for us to add an abstraction
> layer on top of another abstraction layer -- let's just fix the first
> abstraction layer.
>
> If we follow the approach of changing the name of DAT to RDMA and then
> putting it in the kernel, we end up with a stack that looks like:
>
>    upper layer protocol <-> RDMA midlayer <-> IB RDMA provider <-> IB midlayer <-> IB low-level driver

I'd note that the RDMA midlayer above is very thin (plus or minus 1000 
lines for headers and source files).

>
> Let's just evolve the IB midlayer so the picture can be more sensible:
>
>    upper layer protocol <-> RDMA midlayer <-> IB low-level driver
>
> - R.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>


From halr at voltaire.com  Fri Jul  1 15:00:59 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 01 Jul 2005 18:00:59 -0400
Subject: [openib-general] Re: sending RMPP non-data messages
In-Reply-To: <42C5B877.5040901@ichips.intel.com>
References: <42C5B877.5040901@ichips.intel.com>
Message-ID: <1120255259.4389.7.camel@hal.voltaire.com>

On Fri, 2005-07-01 at 17:41, Sean Hefty wrote:
> Looking at the kernel RMPP implementation, I think that we want to ensure 
> that an application doesn't send a non-DATA RMPP MAD.  The RMPP code would 
> interpret such a MAD as being internally generated when it wasn't.

Right; sounds like a good safety check to me. There should be no need
for an application to generate anything other than DATA. ACKs, etc. are
all hidden and handled rather nicely by your implementation. If there
were some reason to terminate an in progress RMPP send, would that be
handled by a cancel operation ?

-- Hal



From rolandd at cisco.com  Fri Jul  1 16:36:18 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 01 Jul 2005 16:36:18 -0700
Subject: [openib-general] Service Record support in SA query
In-Reply-To: <1120237130.4371.3616.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "01 Jul 2005 12:58:50 -0400")
References: <1120237130.4371.3616.camel@hal.voltaire.com>
Message-ID: <52wtoayu8d.fsf@topspin.com>

    Hal> Hi Roland, Any more comments on the Service Record support in
    Hal> SA query patch or is this OK to commit ?

I haven't really had a chance to read it.  I'll get to it by Tuesday.

 - R.


From halr at voltaire.com  Fri Jul  1 16:42:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 01 Jul 2005 19:42:14 -0400
Subject: [openib-general] [PATCH] [udapl dtest]: Fix broken statement
Message-ID: <1120261202.4389.32.camel@hal.voltaire.com>

[udapl dtest]: Fix broken statement

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: dtest.c
===================================================================
--- dtest.c     (revision 2769)
+++ dtest.c     (working copy)
@@ -298,8 +298,7 @@
        ep_attr.qos                         = 0;
        ep_attr.recv_completion_flags       = 0;
        ep_attr.max_recv_dtos               = MSG_BUF_COUNT + (burst*3);
-       ep_attr.max_request_dtos            = MSG_BUF_COUNT + (burst*3) 
- MAX_RDMA_RD;
+       ep_attr.max_request_dtos            = MSG_BUF_COUNT + (burst*3) + MAX_RDMA_RD; 
        ep_attr.max_recv_iov                = 1;
        ep_attr.max_request_iov             = 1;
        ep_attr.max_rdma_read_in            = MAX_RDMA_RD;




From libor at topspin.com  Fri Jul  1 17:25:32 2005
From: libor at topspin.com (Libor Michalek)
Date: Fri, 1 Jul 2005 17:25:32 -0700
Subject: [openib-general] Re: uCM create connection ID
In-Reply-To: <42C4A3F2.2020000@ichips.intel.com>;
	from ardavis@ichips.intel.com on Thu, Jun 30, 2005 at 07:01:22PM
	-0700
References: <42C1BA15.7060205@ichips.intel.com>
	<20050629111038.G26240@topspin.com>
	<42C2F991.60005@ichips.intel.com>
	<20050629181054.J26240@topspin.com>
	<42C4A3F2.2020000@ichips.intel.com>
Message-ID: <20050701172532.A11410@topspin.com>

On Thu, Jun 30, 2005 at 07:01:22PM -0700, Arlin Davis wrote:
> Libor Michalek wrote:
> 
> >  The listen id is in the req rcvd event. (event->param.req_rcvd.listen_id)
> >Do you mean that it is not being set correctly?
> >  
> >
> Ok, I didn't look deep enough. It is set correctly and the polling seems 
> to be working.
> 
> The uDAPL code is now connecting properly but I am having difficulty 
> setting the QP states properly without the ib_cm_init_qp_attr() call. 
> Any chance of providing this call in uCM?

  To recreate that call, I was going to expand the ib_cm_attr_id() call
that's currently there, to retreive the available connection information, 
and then provide a ib_cm_init_qp_attr() like wrapper around it.

-Libor


From halr at voltaire.com  Sat Jul  2 04:20:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 02 Jul 2005 07:20:39 -0400
Subject: [openib-general] mstflint doesn't build on x86_64
Message-ID: <1120303238.4389.754.camel@hal.voltaire.com>

Hi,

mstflint doesn't build on x86_64. Here is the version of g++:
g++ -v
Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-languages=c,c++,objc,java,f77 --enable-java-awt=gtk --host=x86_64-redhat-linux
Thread model: posix
gcc version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)

Thanks.

-- Hal

g++ -O2 -g -I. -fno-exceptions -Wall flint.cpp -o mstflint
flint.cpp:598: error: expected `0' before ""
flint.cpp:598: error: invalid initializer for virtual method `virtual bool Flash::read(u_int32_t, u_int32_t*)'
flint.cpp:598: error: expected `;' before ""
flint.cpp:615: error: expected `0' before ""
flint.cpp:615: error: invalid initializer for virtual method `virtual bool Flash::wait_ready(const char*)'
flint.cpp:615: error: expected `;' before ""
flint.cpp:673: error: expected `0' before ""
flint.cpp:673: error: invalid initializer for virtual method `virtual bool Flash::init_gpios()'
flint.cpp:673: error: expected `;' before ""
flint.cpp:675: error: expected `0' before ""
flint.cpp:675: error: invalid initializer for virtual method `virtual bool Flash::get_cmd_set()'
flint.cpp:675: error: expected `;' before ""
flint.cpp:680: error: expected `0' before ""
flint.cpp:680: error: invalid initializer for virtual method `virtual bool Flash::write_internal(u_int32_t, u_int8_t)'
flint.cpp:680: error: expected `;' before ""
flint.cpp:695: error: expected `0' before ""
flint.cpp:695: error: invalid initializer for virtual method `virtual bool Flash::CmdSet::write(u_int32_t, void*, int, bool, bool)'
flint.cpp:695: error: expected `;' before ""
flint.cpp:698: error: expected `0' before ""
flint.cpp:698: error: invalid initializer for virtual method `virtual bool Flash::CmdSet::erase_sector(u_int32_t)'
flint.cpp:698: error: expected `;' before ""
flint.cpp:700: error: expected `0' before ""
flint.cpp:700: error: invalid initializer for virtual method `virtual bool Flash::CmdSet::reset()'
flint.cpp:700: error: expected `;' before ""
flint.cpp:766: error: expected `0' before ""
flint.cpp:766: error: invalid initializer for virtual method `virtual bool Flash::set_bank_int(u_int32_t)'
flint.cpp:766: error: expected `;' before ""
flint.cpp: In member function `virtual bool Flash::erase_sector(u_int32_t)':
flint.cpp:620: error: 'class Flash::CmdSet' has no member named 'erase_sector'
flint.cpp: In member function `virtual bool Flash::open(const char*, bool)':
flint.cpp:1264: error: `init_gpios' undeclared (first use this function)
flint.cpp:1264: error: (Each undeclared identifier is reported only once for each function it appears in.)
flint.cpp:1268: error: `get_cmd_set' undeclared (first use this function)
flint.cpp:1272: error: 'class Flash::CmdSet' has no member named 'reset'
flint.cpp: In member function `virtual bool Flash::read(u_int32_t, void*, int, bool)':
flint.cpp:1344: error: no matching function for call to `Flash::read(u_int32_t&, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp: In member function `virtual bool Flash::write(u_int32_t, void*, int, bool, bool)':
flint.cpp:1388: error: 'class Flash::CmdSet' has no member named 'write'
flint.cpp: In member function `virtual bool Flash::write(u_int32_t, u_int32_t)':
flint.cpp:1413: error: no matching function for call to `Flash::read(u_int32_t&, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp: In member function `bool Flash::set_bank(u_int32_t)':
flint.cpp:1432: error: `set_bank_int' undeclared (first use this function)
flint.cpp: In function `bool repair(Flash&, int, int, bool)':
flint.cpp:3146: error: no matching function for call to `Flash::read(unsigned int, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp:3154: error: no matching function for call to `Flash::read(unsigned int, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp: In function `bool FailSafe_burn_internal(Flash&, void*, int, bool)':
flint.cpp:3321: error: no matching function for call to `Flash::read(unsigned int, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp: In function `bool FailSafe_burn(Flash&, void*, int, bool, bool)':
flint.cpp:3378: error: no matching function for call to `Flash::read(int, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp:3440: error: no matching function for call to `Flash::read(unsigned int, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp:3541: error: no matching function for call to `Flash::read(unsigned int, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
flint.cpp: In function `void TerminationHandler(int)':
flint.cpp:4686: error: 'class Flash' has no member named 'wait_ready'
flint.cpp: In function `int main(int, char**)':
flint.cpp:5153: error: no matching function for call to `Flash::read(u_int32_t&, u_int32_t*)'
flint.cpp:1324: note: candidates are: virtual bool Flash::read(u_int32_t, void*, int, bool)
make: *** [mstflint] Error 1




From johannes at erdfelt.com  Sat Jul  2 09:37:12 2005
From: johannes at erdfelt.com (Johannes Erdfelt)
Date: Sat, 2 Jul 2005 09:37:12 -0700
Subject: [openib-general] mstflint doesn't build on x86_64
In-Reply-To: <1120303238.4389.754.camel@hal.voltaire.com>
References: <1120303238.4389.754.camel@hal.voltaire.com>
Message-ID: <20050702163712.GA1388@sventech.com>

On Sat, Jul 02, 2005, Hal Rosenstock <halr at voltaire.com> wrote:
> mstflint doesn't build on x86_64. Here is the version of g++:
> g++ -v
> Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.2/specs
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-languages=c,c++,objc,java,f77 --enable-java-awt=gtk --host=x86_64-redhat-linux
> Thread model: posix
> gcc version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)

It's not an x86-64 problem, it's a C++ problem. mstflint uses NULL where
it should be using 0 for the function protoytypes in the class
definitions.

Older gcc versions were a bit more lenient, but it has tightened up to
meet the C++ specs recently.

I just recently ran into this problem and I have a patch, but it's at
the office. If someone doesn't get around to it before Tuesday, I'll
send it in.

JE



From eitan at mellanox.co.il  Sat Jul  2 12:38:51 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Sat, 2 Jul 2005 22:38:51 +0300 
Subject: [openib-general] IBDM - IB DataModel build issues fixed
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>

Hi Hal,

(IBDM is a flexible data model for describing arbitrary IB systems and then
instantiating them in a topology. Then based on this topology an in-memory
data model is built and various checking, simulation and analysis procedures
are provided.)

Could not find your last mail with the issue.
But I was able to update the IBDM code at:
https://openib.org/svn/gen2/utils/src/linux-user/ibdm
<https://openib.org/svn/gen2/utils/src/linux-user/ibdm> 

I was able to checkout and build it with no issue 
But only after installing:
autoconf 2.59
automake 1.9.3

EZ
Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050702/3dca819e/attachment.html>

From mst at mellanox.co.il  Sat Jul  2 14:23:28 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 3 Jul 2005 00:23:28 +0300
Subject: [openib-general] Re: mstflint doesn't build on x86_64
In-Reply-To: <1120303238.4389.754.camel@hal.voltaire.com>
References: <1120303238.4389.754.camel@hal.voltaire.com>
Message-ID: <20050702212328.GA21801@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: mstflint doesn't build on x86_64
> 
> Hi,
> 
> mstflint doesn't build on x86_64. Here is the version of g++:
> g++ -v
> Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.2/specs
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-languages=c,c++,objc,java,f77 --enable-java-awt=gtk --host=x86_64-redhat-linux
> Thread model: posix
> gcc version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)
> 
> Thanks.
> 
> -- Hal

Probably more to do with gcc version than the architecture.

Does the following patch help?

Index: flint.cpp
===================================================================
--- flint.cpp	(revision 2744)
+++ flint.cpp	(working copy)
@@ -580,7 +580,7 @@ class Flash : public FBase {
 public:
     Flash(u_int32_t log2_bank_size) :  
     _mf(0), 
-    _cmd_set(NULL), 
+    _cmd_set(0), 
     _curr_bank(0xffffffff),
     _log2_bank_size(log2_bank_size) 
     {}
@@ -595,7 +595,7 @@ public:
     virtual void close         ();
 
     virtual bool read          (u_int32_t addr, 
-                                u_int32_t *data)            = NULL;
+                                u_int32_t *data)            = 0;
 
     virtual bool read          (u_int32_t addr, 
                                 void*     data, 
@@ -612,7 +612,7 @@ public:
 	get_size               ()                           {return _cfi_data.device_size ? _cfi_data.device_size : (u_int32_t)MAX_FLASH;}
 
 
-    virtual bool wait_ready    (const char* msg = NULL)     = NULL;
+    virtual bool wait_ready    (const char* msg = 0)     = 0;
 
 
     // Write and Erase functions are performed by the Command Set
@@ -670,14 +670,14 @@ protected:
     virtual bool lock           (bool retry=true);
     virtual bool unlock         ();
 
-    virtual bool init_gpios    ()                           = NULL;
+    virtual bool init_gpios    ()                           = 0;
 
-    virtual bool get_cmd_set   ()                           = NULL;
+    virtual bool get_cmd_set   ()                           = 0;
 
     bool set_bank      (u_int32_t addr);
 
     virtual bool write_internal(u_int32_t addr, 
-                                u_int8_t  data)             = NULL;
+                                u_int8_t  data)             = 0;
 
     bool         write_internal(u_int32_t addr, 
                                 u_int8_t* data, 
@@ -692,12 +692,12 @@ protected:
                                     void*     data, 
                                     int       cnt,
                                     bool      noerase  = false, 
-                                    bool      noverify = false)  = NULL;
+                                    bool      noverify = false)  = 0;
 
-        //virtual bool unlock_bypass (bool      unlock)            = NULL;
-        virtual bool erase_sector  (u_int32_t addr)              = NULL;
+        //virtual bool unlock_bypass (bool      unlock)            = 0;
+        virtual bool erase_sector  (u_int32_t addr)              = 0;
 
-        virtual bool reset         ()                            = NULL;
+        virtual bool reset         ()                            = 0;
 
     protected:
 
@@ -763,7 +763,7 @@ protected:
     bool print_cfi_info ( const cfi_query *q );
 
 
-    virtual bool set_bank_int  (u_int32_t bank)              = NULL;
+    virtual bool set_bank_int  (u_int32_t bank)              = 0;
     u_int32_t    bank_mask     ()                 {return((1 << _log2_bank_size) -1 );}
 
     mfile        *_mf;
@@ -829,7 +829,7 @@ public:
                                 int       len, 
                                 bool      verbose=false) {return Flash::read(addr, data, len, verbose);}
 
-    virtual bool wait_ready    (const char* msg = NULL);
+    virtual bool wait_ready    (const char* msg = 0);
 
     bool                        unlock_bypass (bool unlock);
 
@@ -4100,7 +4100,7 @@ bool patchVSD(FImage& f, char *vsd1, cha
         strncpy(&vsd[0], vsd1, VSD_OFFS);
 
 
-    if (new_psid == NULL) {
+    if (new_psid == 0) {
         // New psid is not explicitly given - take it from image
         memcpy(image_psid, (char*)ps->psid, sizeof(ps->psid));
 
@@ -4383,13 +4383,13 @@ Flash* get_serial_flash(mfile* mf) {
                flash_type_str[flash_type]);
     }
 
-    return NULL;
+    return 0;
 
 }
 
 
 Flash* get_flash(const char* device) {
-    Flash* f = NULL;
+    Flash* f = 0;
 
     //
     // Check device ID. Allocate flash accordingly
@@ -4400,7 +4400,7 @@ Flash* get_flash(const char* device) {
     mfile* mf = mopen(device);
     if (!mf) {
         printf("*** ERROR *** Can't open %s: %s\n", device,  strerror(errno));
-        return NULL;
+        return 0;
     }
 
     if (mread4(mf, 0xf0014, &dev_id) != 4) return false;
@@ -4658,7 +4658,7 @@ void usage(const char *sname, bool full 
 // Signal handlers 
 //
 
-Flash* g_flash = NULL;
+Flash* g_flash = 0;
 
 int g_signals_for_termination[] = {
     SIGINT,
@@ -4679,7 +4679,7 @@ void TerminationHandler (int signum)
 
     signal (signum, SIG_DFL);
 
-    if (g_flash != NULL) {
+    if (g_flash != 0) {
         report("\n Received signal %d. Cleaning up ...", signum);
         fflush(stdout);
         sleep(1); // let erase sector end
@@ -4756,7 +4756,7 @@ int main(int ac, char *av[])
                 auto_ptr<Flash>       tmp( get_flash(device));
 				f = tmp;
 				
-                if (f.get() == NULL) {
+                if (f.get() == 0) {
                     printf("*** ERROR *** Can't get flash type using device %s\n", device);
                     rc =  1; goto done; 
                 }

-- 
MST


From panda at cse.ohio-state.edu  Sat Jul  2 21:42:48 2005
From: panda at cse.ohio-state.edu (Dhabaleswar Panda)
Date: Sun, 3 Jul 2005 00:42:48 -0400 (EDT)
Subject: [openib-general] Announcing the release of MVAPICH2 0.6.5 with
	uDAPL support
Message-ID: <200507030442.j634gmfF002860@xi.cse.ohio-state.edu>

The MVAPICH (MPI over InfiniBand) team at the Ohio State University is
pleased to announce the release of MVAPICH2 0.6.5 for multiple
platforms (EM64T, G5, IA-32, IA-64, and Opteron) and network
interfaces (PCI-X and PCI-Express-including the new mem-free cards).

In addition to the optimized design over the VAPI interface for
InfiniBand, MVAPICH2 0.6.5 also supports optimized design over the
uDAPL interface so that users can take advantage of MPI-2
functionalities available in MVAPICH2 on networks supporting uDAPL
provider library.  The uDAPL interface of MVAPICH2 0.6.5 has been
tested with InfiniBand (IBGD uDAPL), Myrinet (DAPL-GM beta), and
Ammasso GigE (Ammasso uDAPL). Through the uDAPL interface, it provides
portability across different networks while delivering high
performance.

MVAPICH2 0.6.5 is being distributed as a single integrated package
(with the latest MPICH2 1.0.1 and MVICH). It can be downloaded with a
`single click' and installed. It is available under BSD license.

MVAPICH/MVAPICH2 software is being used by more than 230 organizations
world-wide (in 27 countries) to extract the potential of InfiniBand
networking technology for designing high-end computing systems and
servers. It is also being distributed by many IBA vendors in their
software distributions.

This new release has the following features:

      - MPI-2 functionalities (one-sided, collectives, datatype)
      - all MPI-1 functionalities
      - optimized one-sided operations (Get, Put, and Accumulate)
      - support for active and passive synchronization
      - optimized two-sided operations
      - uDAPL support (tested for InfiniBand, Myrinet, and Ammasso GigE) 
      - scalable job start-up
      - optimized and tuned for the above platforms and different 
        network interfaces (PCI-X and PCI-Express)
      - single code base for all of the above platforms
      - memory efficient scaling modes for medium and large clusters

Other features of this release include:

- Excellent performance: For two-sided operations, MVAPICH2 0.6.5
  with VAPI interface delivers 5.0 microsec latency (with the switch), 
  up to 965 MB/sec unidirectional bandwidth, and up to 1725 MB/sec
  bidirectional bandwidth on EM64T system with PCI-Express. For
  one-sided Put operation, MVAPICH2 0.6.5 delivers 6.98 microsec
  latency (with the switch), and up to 972 MB/sec unidirectional
  bandwidth on the above platform.

- With uDAPL interface, MVAPICH2 0.6.5 delivers latency for small 
  messages very close (within 1.0 microsec) to that of the uDAPL
  library. It delivers bandwidth very close to that provided by the
  VAPI interface.

- Detailed performance numbers for two-sided and all one-sided
  operations on various platforms and interconnects using VAPI and uDAPL
  interfaces are available on the project's web page.

- A set of benchmarks to evaluate one-sided operations (Put, Get, and 
  Accumulate)

- An enhanced and detailed `User and Tuning Guide' to assist users: 

       - to install this package on different platforms 
            with both interfaces (VAPI and uDAPL) and different options

       - to vary different parameters of the MPI installation to 
            extract maximum performance and achieve scalability, 
            especially on large-scale systems.

You are welcome to download the MVAPICH2 0.6.5 package and access
relevant information from the following URL:

http://nowlab.cis.ohio-state.edu/projects/mpi-iba/

Our upcoming release will include a high performance design 
of MVAPICH2 with MPICH2 1.0.2 and uDAPL support.

All feedbacks, including bug reports and hints for performance tuning,
are welcome. Please send an e-mail to mvapich-help at cse.ohio-state.edu.

Thanks, 

MVAPICH Team at OSU/NBCL 

----------

PS: If you would like to be removed from this mailing list, please end
an e-mail to mvapich_request at cse.ohio-state.edu.



From rep.nop at aon.at  Sun Jul  3 15:42:56 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 00:42:56 +0200
Subject: [openib-general] Re: [openib-commits] r2776 -
	gen2/utils/src/linux-user/ibdm/datamodel
In-Reply-To: <20050702192533.51E6922834D@openib.ca.sandia.gov>
References: <20050702192533.51E6922834D@openib.ca.sandia.gov>
Message-ID: <20050703224256.GC5997@aon.at>

On Sat, Jul 02, 2005 at 12:25:33PM -0700, eitan at openib.org wrote:
>Modified: gen2/utils/src/linux-user/ibdm/datamodel/Fabric.cpp

Eitan,

- fix a few typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
-------------- next part --------------
Index: utils/src/linux-user/ibdm/datamodel/ibdm.i
===================================================================
--- utils/src/linux-user/ibdm/datamodel/ibdm.i	(revision 2778)
+++ utils/src/linux-user/ibdm/datamodel/ibdm.i	(working copy)
@@ -780,7 +780,7 @@ int FabricUtilsVerboseLevel;
 
   IBDM exposes some of its internal objects. The objects 
   identifiers returned by the various function calls are formatted
-  acording to the following rules:
+  according to the following rules:
   Fabric: fabric:<idx>
   System: system:<fab idx>:<sys name>
   SysPort: sysport:<fab idx>:<sys name>:<port name>
Index: utils/src/linux-user/ibdm/datamodel/Fabric.cpp
===================================================================
--- utils/src/linux-user/ibdm/datamodel/Fabric.cpp	(revision 2778)
+++ utils/src/linux-user/ibdm/datamodel/Fabric.cpp	(working copy)
@@ -34,7 +34,7 @@
 
 /*
 IB Fabric Data Model
-This file hodls implementation of the data model classes and methods 
+This file holds implementation of the data model classes and methods
 
 */
 
@@ -119,8 +119,8 @@ IBPort::connect (IBPort *p_otherPort,
     if (p_remotePort != p_otherPort) {
       cout << "-W- Disconnecting: "
            << p_remotePort->getName() << " previously connected to:"
-           << p_remotePort->getName() 
-           << " whil econnecting:" << p_otherPort->getName() << endl;
+           << p_remotePort->getName()
+           << " while connecting:" << p_otherPort->getName() << endl;
       // the other side should be cleaned only if points here
       if (p_remotePort->p_remotePort == this) {
         p_remotePort->p_remotePort = NULL;
@@ -580,7 +580,7 @@ IBSystem::getSysPort(string name) {
 // constructor:
 IBSystem::IBSystem(string n, class IBFabric *p_fab, string t) {
   if (p_fab->getSystem(n)) {
-    cerr << "Can' deal with double allocation of same system!" << endl;
+    cerr << "Can't deal with double allocation of same system!" << endl;
     abort();
   }
   name = n;
@@ -679,7 +679,7 @@ IBSystem::removeBoard (string boardName)
 
   // Warn if no match:
   if (matchedNodes.empty()) {
-    cout << "-W-(RemoveBoard) Fail to find any node in:"
+    cout << "-W- removeBoard : Fail to find any node in:"
          << sysNodePrefix << " while removing:" << boardName << endl;
     return 1;
   }
@@ -1636,7 +1636,7 @@ IBFabric::parseFdbFile(string fn) {
     //cout << "-W- Ignoring line:" << sLine << endl;
   }
 
-  cout << "-I- Defined " << fdbLines << " fdb entires for:"
+  cout << "-I- Defined " << fdbLines << " fdb entries for:"
        << switches << " switches" << endl;
   f.close();
   return anyErr;
@@ -1715,7 +1715,7 @@ IBFabric::parseMCFdbFile(string fn) {
     //cout << "-W- Ignoring line:" << sLine << endl;
   }
 
-  cout << "-I- Defined " << fdbLines << " Multicast Fdb entires for:"
+  cout << "-I- Defined " << fdbLines << " Multicast Fdb entries for:"
        << switches << " switches" << endl;
   f.close();
   return anyErr;
Index: utils/src/linux-user/IBMgtSim/src/sma.cpp
===================================================================
--- utils/src/linux-user/IBMgtSim/src/sma.cpp	(revision 2778)
+++ utils/src/linux-user/IBMgtSim/src/sma.cpp	(working copy)
@@ -904,7 +904,7 @@ int IBMSSma::setIBPortBaseLid(
   unsigned int        portLidIndex;
   MSG_ENTER_FUNC;
 
-  MSGREG(inf0, 'I', "Seting base_lid for node:$ port:$ to $", "setIBPortBaseLid");
+  MSGREG(inf0, 'I', "Setting base_lid for node:$ port:$ to $", "setIBPortBaseLid");
   pNode = pSimNode->getIBNode();
   pPort = pNode->getPort(portNum);
   if (! pPort) 
@@ -925,7 +925,7 @@ int IBMSSma::setIBPortBaseLid(
   /* make sure the vector of port by lid has enough entries */
   if (pNode->p_fabric->PortByLid.size() <= base_lid)
   {
-    /* we add 20 entires each time */
+    /* we add 20 entries each time */
     pNode->p_fabric->PortByLid.resize(base_lid+20);
     for ( portLidIndex = pNode->p_fabric->PortByLid.size();
           portLidIndex < base_lid + 20;


From rep.nop at aon.at  Sun Jul  3 15:51:04 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 00:51:04 +0200
Subject: [openib-general] [PATCH][osm] commentary typos
Message-ID: <20050703225104.GD5997@aon.at>

Hi Hal,

- commentary typos

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
-------------- next part --------------
Index: trunk/src/userspace/management/osm/opensm/osm_state_mgr.c
===================================================================
--- trunk/src/userspace/management/osm/opensm/osm_state_mgr.c	(revision 2778)
+++ trunk/src/userspace/management/osm/opensm/osm_state_mgr.c	(working copy)
@@ -518,7 +518,7 @@ __osm_state_mgr_reset_node_count(
   {
     osm_log( p_mgr->p_log, OSM_LOG_DEBUG,
              "__osm_state_mgr_reset_node_count: "
-             "Reseting discovery count for node 0x%" PRIx64 ".\n",
+             "Resetting discovery count for node 0x%" PRIx64 ".\n",
              cl_ntoh64( osm_node_get_node_guid( p_node ) ));
   }
 
@@ -539,7 +539,7 @@ __osm_state_mgr_reset_port_count(
   {
     osm_log( p_mgr->p_log, OSM_LOG_DEBUG,
              "__osm_state_mgr_reset_port_count: "
-             "Reseting discovery count for port 0x%" PRIx64 ".\n",
+             "Resetting discovery count for port 0x%" PRIx64 ".\n",
              cl_ntoh64( osm_port_get_guid( p_port ) ));
   }
 
@@ -560,7 +560,7 @@ __osm_state_mgr_reset_switch_count(
   {
     osm_log( p_mgr->p_log, OSM_LOG_DEBUG,
              "__osm_state_mgr_reset_switch_count: "
-             "Reseting discovery count for switch 0x%" PRIx64 ".\n",
+             "Resetting discovery count for switch 0x%" PRIx64 ".\n",
              cl_ntoh64( osm_node_get_node_guid( p_sw->p_node ) ));
   }
 
Index: trunk/src/userspace/management/osm/opensm/osm_drop_mgr.c
===================================================================
--- trunk/src/userspace/management/osm/opensm/osm_drop_mgr.c	(revision 2778)
+++ trunk/src/userspace/management/osm/opensm/osm_drop_mgr.c	(working copy)
@@ -271,7 +271,7 @@ __osm_drop_mgr_remove_port(
             osm_port_discovery_count_reset( p_remote_port );
             osm_log( p_mgr->p_log, OSM_LOG_DEBUG,
                      "__osm_drop_mgr_remove_port: "
-                     "reseting discovery count of node: "
+                     "resetting discovery count of node: "
                      "0x%016" PRIx64 " port num:%u.\n",
                      cl_ntoh64( osm_node_get_node_guid( p_remote_node ) ),
                      remote_port_num );
Index: trunk/src/userspace/management/osm/include/complib/cl_timer.h
===================================================================
--- trunk/src/userspace/management/osm/include/complib/cl_timer.h	(revision 2778)
+++ trunk/src/userspace/management/osm/include/complib/cl_timer.h	(working copy)
@@ -331,7 +331,7 @@ cl_timer_trim(
 *	remaining time when the timer is set.
 *
 *	If the new interval time is less than the remaining time, cl_timer_trim
-*	implicitly stops the timer before reseting it.
+*	implicitly stops the timer before resetting it.
 *
 *	If the timer is reset, it is guaranteed to expire no sooner than the
 *	new interval, but may take longer to expire.
Index: trunk/src/userspace/management/osm/complib/cl_event.c
===================================================================
--- trunk/src/userspace/management/osm/complib/cl_event.c	(revision 2778)
+++ trunk/src/userspace/management/osm/complib/cl_event.c	(working copy)
@@ -102,7 +102,7 @@ cl_event_signal(
 
 	cl_spinlock_acquire( &p_event->spinlock );
 	p_event->signaled = TRUE;
-	/* Wake up one or all depending on whether the event is auto-reseting. */
+	/* Wake up one or all depending on whether the event is auto-resetting. */
 	if( p_event->manual_reset )
 		pthread_cond_broadcast( &p_event->condvar );
 	else

From eitan at mellanox.co.il  Mon Jul  4 00:20:16 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Mon, 4 Jul 2005 10:20:16 +0300 
Subject: [openib-general] RE: [openib-commits] r2776 -
	gen2/utils/src/linux-user/ibdm/datam odel
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3040A@mtlex01.yok.mtl.com>

Thanks. Committed as 2779.

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Bernhard Fischer [mailto:rep.nop at aon.at]
> Sent: Monday, July 04, 2005 1:43 AM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: Re: [openib-commits] r2776 -
gen2/utils/src/linux-user/ibdm/datamodel
> 
> On Sat, Jul 02, 2005 at 12:25:33PM -0700, eitan at openib.org wrote:
> >Modified: gen2/utils/src/linux-user/ibdm/datamodel/Fabric.cpp
> 
> Eitan,
> 
> - fix a few typos.
> 
> Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050704/f58e4060/attachment.html>

From tomduffy at gmail.com  Mon Jul  4 09:59:18 2005
From: tomduffy at gmail.com (Tom Duffy)
Date: Mon, 4 Jul 2005 12:59:18 -0400
Subject: [openib-general] [oops] recent opensm crash
In-Reply-To: <1120238676.4371.3718.camel@hal.voltaire.com>
References: <1120169407.29522.17.camel@duffman>
	<1120238676.4371.3718.camel@hal.voltaire.com>
Message-ID: <9d3b7de705070409594e2bf25b@mail.gmail.com>

On 01 Jul 2005 13:24:37 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> I found one problem associated with this and just checked in a patch.
> I'm not sure whether there is another one behind this or not. Any
> reliable way to recreate this ?

No, it happened after being up for weeks one time when I was bringing
an interface down and back up again.  I will report if I get another
crash later.

Thanks,

-tduffy


From mst at mellanox.co.il  Mon Jul  4 11:40:38 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 4 Jul 2005 21:40:38 +0300
Subject: [openib-general] Re: [PATCH] updates to copyright
In-Reply-To: <ORSMSX401GryWtIIZS200000015@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401GryWtIIZS200000015@orsmsx401.amr.corp.intel.com>
Message-ID: <20050704184038.GA16650@mellanox.co.il>

The following patch adds copyright statements for Mellanox to files that
I've touched and that are missing them.

---

Add Mellanox copyright to files modified by Michael S. Tsirkin.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: infiniband/core/uverbs_mem.c
===================================================================
--- infiniband/core/uverbs_mem.c	(revision 2780)
+++ infiniband/core/uverbs_mem.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: infiniband/core/uverbs_main.c
===================================================================
--- infiniband/core/uverbs_main.c	(revision 2780)
+++ infiniband/core/uverbs_main.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: infiniband/core/uverbs.h
===================================================================
--- infiniband/core/uverbs.h	(revision 2780)
+++ infiniband/core/uverbs.h	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU

-- 
MST


From rep.nop at aon.at  Mon Jul  4 11:51:10 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 20:51:10 +0200
Subject: [openib-general] RE: [openib-commits] r2776 -
	gen2/utils/src/linux-user/ibdm/datam odel
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3040A@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C3040A@mtlex01.yok.mtl.com>
Message-ID: <20050704185110.GE5997@aon.at>

On Mon, Jul 04, 2005 at 10:20:16AM +0300, Eitan Zahavi wrote:
>Thanks. Committed as 2779.

I'm sorry but i overlooked another occurance of acor; attached.
Would you be so kind and do these in ibdm and IBMgtSim? TIA.
s/teh/the/g
s/eb/be/g
s/fo/of/g
this may or may not work for you:
egrep -ri "([[:space:]](fo|teh|eb)[[:space:]])" .


- fix a few typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
-------------- next part --------------
Index: utils/src/linux-user/IBMgtSim/src/ibdm.i
===================================================================
--- utils/src/linux-user/IBMgtSim/src/ibdm.i	(revision 2780)
+++ utils/src/linux-user/IBMgtSim/src/ibdm.i	(working copy)
@@ -732,7 +732,7 @@ int FabricUtilsVerboseLevel;
 
   IBDM exposes some of its internal objects. The objects 
   identifiers returned by the various function calls are formatted
-  acording to the following rules:
+  according to the following rules:
   Fabric: fabric:<idx>
   System: system:<fab idx>:<sys name>
   SysPort: sysport:<fab idx>:<sys name>:<port name>
Index: utils/src/linux-user/IBMgtSim/utils/RunSimTest
===================================================================
--- utils/src/linux-user/IBMgtSim/utils/RunSimTest	(revision 2780)
+++ utils/src/linux-user/IBMgtSim/utils/RunSimTest	(working copy)
@@ -112,18 +112,18 @@ proc Help {} {
 # OpenSM log file analyzer 
 # 
 # Continuously monitor the OpenSM log file and generate a log of all the 
-# events reported in teh log file.
+# events reported in the log file.
 # 
 # On any event - it scans through the list of callbacks to be invoked 
-# and calls them acordingly. The list of callabcks is in osmLogCallbacks(logFile)
+# and calls them accordingly. The list of callbacks is in osmLogCallbacks(logFile)
 #
-# The log fo all events is accoumulated in the global list:
+# The log of all events is accoumulated in the global list:
 # osmEventLog(logFile)
 # 
 # The format of the event log list entry is:
 # <eventType> <time> <full line>
 #
-# <eventType> = SubnetUp, Standby, ERR, ErrorsInInitialization, ReportNoice, ....
+# <eventType> = SubnetUp, Standby, ERR, ErrorsInInitialization, ReportNotice, ....
 # <time> = the local time in tracker.
 # <full line> = the complete log file line.
 #

From rep.nop at aon.at  Mon Jul  4 11:56:38 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 20:56:38 +0200
Subject: [openib-general] RE: [openib-commits] r2776 -
	gen2/utils/src/linux-user/ibdm/datam odel
In-Reply-To: <20050704185110.GE5997@aon.at>
References: <506C3D7B14CDD411A52C00025558DED607C3040A@mtlex01.yok.mtl.com>
	<20050704185110.GE5997@aon.at>
Message-ID: <20050704185638.GF5997@aon.at>

On Mon, Jul 04, 2005 at 08:51:10PM +0200, Bernhard Fischer wrote:
>On Mon, Jul 04, 2005 at 10:20:16AM +0300, Eitan Zahavi wrote:
>>Thanks. Committed as 2779.
>
>I'm sorry but i overlooked another occurance of acor; attached.
>Would you be so kind and do these in ibdm and IBMgtSim? TIA.
>s/teh/the/g
>s/eb/be/g
>s/fo/of/g
>this may or may not work for you:
>egrep -ri "([[:space:]](fo|teh|eb)[[:space:]])" .

>Index: utils/src/linux-user/IBMgtSim/utils/RunSimTest
>===================================================================
>--- utils/src/linux-user/IBMgtSim/utils/RunSimTest	(revision 2780)
>+++ utils/src/linux-user/IBMgtSim/utils/RunSimTest	(working copy)
>@@ -112,18 +112,18 @@ proc Help {} {

>-# The log fo all events is accoumulated in the global list:
>+# The log of all events is accoumulated in the global list:

s/accoumulate/accumulate/


From rep.nop at aon.at  Mon Jul  4 13:44:10 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 22:44:10 +0200
Subject: [Openib-windows] Re: [openib-general] Re: IB Diagnositic Tools
In-Reply-To: <1119964919.4848.2387.camel@hal.voltaire.com>
References: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com>
	<1119895456.4848.353.camel@hal.voltaire.com>
	<20050627211346.GA17011@aon.at>
	<1119907826.4848.486.camel@hal.voltaire.com>
	<1119964919.4848.2387.camel@hal.voltaire.com>
Message-ID: <20050704204410.GG5997@aon.at>

On Tue, Jun 28, 2005 at 09:22:00AM -0400, Hal Rosenstock wrote:
>On Mon, 2005-06-27 at 17:30, Hal Rosenstock wrote: 
>> On Mon, 2005-06-27 at 17:13, Bernhard Fischer wrote:
>> > Hal,
>> > 
>> > This is kinda offtopic, 
>> 
>> Yes, this is different topic(s).
>> 
>> > but (iirc) i once stumbled over the issue of
>> > "port" vs. "mgmt port" [back then i had access to two 2-port cards]
>> > where you may have said something along the lines of
>> > \"There is clearly a bug for multi HCAs in osm_vendor_get_all_port_attr
>> > which is in the vendor layer. This needs to be fixed and is our problem.
>> > So I am close to being able to commit what I now have for this and fix
>> > this later (as there are other multi HCA issues).\"
>> > Just curious.. did somebody already have a chance to touch those or not?
>> 
>> I don't think it has been fixed :-( I will need to refresh myself on
>> this again.
>
>Just tried the diags and OpenSM on a multi HCA machine and they appear
>to run fine. I will need to dig out from my notes as that comment came
>from back on 2/16.

Please note that that specific occurance was *not* triggered in a strict
\"multi HCA\" setup but with two-port HCA(s e.g. opensm'd) back-to_back.

Back then, i saw (can't reproduce it now since i do not have access to
an appropriate setup; can't reproduce since even if i would have access
to such a setup i most likely wouldn't have root access) a listing akin
to :
phy-port 1
phy-port 1
phy-port 2
e-o-f

Please mail me the output of
opensm -g
[with opensm as primary SM -- but i guess it does literally everywhere,
be it hosts or switches] with the attached patch.

PS: Hal, if the moved question is true, please point me to a document
and section where this is defined. I may be a pita, but i'd like to
know that answer. TIA and .....

..sorry for the noise
-- 
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2780.osm-local-port.oorig/trunk/src/userspace/management/osm/opensm/main.c gen2.2780.osm-local-port/trunk/src/userspace/management/osm/opensm/main.c
--- gen2.2780.osm-local-port.oorig/trunk/src/userspace/management/osm/opensm/main.c	2005-04-26 20:58:16.000000000 +0200
+++ gen2.2780.osm-local-port/trunk/src/userspace/management/osm/opensm/main.c	2005-07-04 22:27:47.000000000 +0200
@@ -252,7 +252,8 @@ print_all_guids(
   }
 
   printf("\nListing GUIDs:\n");
-  for (i = 1; i < num_ports; i++) { /* excluding logical mgmt port */
+/* "local ports" is(?) phys, shouldn't this exclude port 0 then ? */
+  for (i = 0; i < num_ports; i++) { /* excluding logical mgmt port */
     printf("Port %i: 0x%"PRIx64"\n", i, cl_hton64(attr_array[i].port_guid));
   }
 }
@@ -272,7 +273,7 @@ get_port_guid(
 		Call the transport layer for a list of local port
 		GUID values.
 	*/
-/* "local ports" is(?) phys, shouldn't this exclude port 0 then ? */
+
 	status = osm_vendor_get_all_port_attr(p_osm->p_vendor, attr_array, &num_ports );
 	if ( status != IB_SUCCESS )
 	{

From eitan at mellanox.co.il  Mon Jul  4 13:54:18 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Mon, 4 Jul 2005 23:54:18 +0300 
Subject: [Openib-windows] Re: [openib-general] Re: IB Diagnositic Tool
	s
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30415@mtlex01.yok.mtl.com>

Hi Ben

The patch try to start scanning the ports from i=1 rather then i=0.

status = osm_vendor_get_all_port_attr(p_osm->p_vendor, attr_array,
&num_ports );

But this function to return the port attributes of all available ports does
not use the "port number" as index into the returned array of port
attributes. It simply starts with i=0 and adds on to it. The port numbers
are available in the attributes struct.

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Bernhard Fischer [mailto:rep.nop at aon.at]
> Sent: Monday, July 04, 2005 11:44 PM
> To: Hal Rosenstock
> Cc: Bernhard Fischer; Eitan Zahavi; openib-windows at openib.org; 'openib-
> general at openib.org'
> Subject: Re: [Openib-windows] Re: [openib-general] Re: IB Diagnositic
Tools
> 
> On Tue, Jun 28, 2005 at 09:22:00AM -0400, Hal Rosenstock wrote:
> >On Mon, 2005-06-27 at 17:30, Hal Rosenstock wrote:
> >> On Mon, 2005-06-27 at 17:13, Bernhard Fischer wrote:
> >> > Hal,
> >> >
> >> > This is kinda offtopic,
> >>
> >> Yes, this is different topic(s).
> >>
> >> > but (iirc) i once stumbled over the issue of
> >> > "port" vs. "mgmt port" [back then i had access to two 2-port cards]
> >> > where you may have said something along the lines of
> >> > \"There is clearly a bug for multi HCAs in
osm_vendor_get_all_port_attr
> >> > which is in the vendor layer. This needs to be fixed and is our
problem.
> >> > So I am close to being able to commit what I now have for this and
fix
> >> > this later (as there are other multi HCA issues).\"
> >> > Just curious.. did somebody already have a chance to touch those or
not?
> >>
> >> I don't think it has been fixed :-( I will need to refresh myself on
> >> this again.
> >
> >Just tried the diags and OpenSM on a multi HCA machine and they appear
> >to run fine. I will need to dig out from my notes as that comment came
> >from back on 2/16.
> 
> Please note that that specific occurance was *not* triggered in a strict
> \"multi HCA\" setup but with two-port HCA(s e.g. opensm'd) back-to_back.
> 
> Back then, i saw (can't reproduce it now since i do not have access to
> an appropriate setup; can't reproduce since even if i would have access
> to such a setup i most likely wouldn't have root access) a listing akin
> to :
> phy-port 1
> phy-port 1
> phy-port 2
> e-o-f
> 
> Please mail me the output of
> opensm -g
> [with opensm as primary SM -- but i guess it does literally everywhere,
> be it hosts or switches] with the attached patch.
> 
> PS: Hal, if the moved question is true, please point me to a document
> and section where this is defined. I may be a pita, but i'd like to
> know that answer. TIA and .....
> 
> ..sorry for the noise
> --
> Bernhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050704/b1d7d3c8/attachment.html>

From rep.nop at aon.at  Mon Jul  4 14:30:06 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 23:30:06 +0200
Subject: [openib-general] [PATCH][osm] commentary typos
Message-ID: <20050704213006.GH5997@aon.at>

Hi,

- commentary typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

No obj-code changes.

thank you,
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/include/opensm/osm_port.h gen2.2780/trunk/src/userspace/management/osm/include/opensm/osm_port.h
--- gen2.2780.oorig/trunk/src/userspace/management/osm/include/opensm/osm_port.h	2005-05-10 18:25:28.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/include/opensm/osm_port.h	2005-07-04 21:09:50.000000000 +0200
@@ -391,7 +391,7 @@ osm_link_is_healthy(
 *
 * DESCRIPTION
 *	Sets the port health flag. TRUE means the port is healthy and 
-*  should eb used for packet routing. FALSE means it should be avoided.
+*  should be used for packet routing. FALSE means it should be avoided.
 *
 * SYNOPSIS
 */
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/include/opensm/osm_switch.h gen2.2780/trunk/src/userspace/management/osm/include/opensm/osm_switch.h
--- gen2.2780.oorig/trunk/src/userspace/management/osm/include/opensm/osm_switch.h	2005-04-26 20:58:23.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/include/opensm/osm_switch.h	2005-07-04 21:09:10.000000000 +0200
@@ -1128,12 +1128,12 @@ osm_switch_recommend_path(
 *     [in] The maximum allowed number of target lids routed through 
 *     a specific port of the switch. If the port already assigned 
 *     (in the lfdb) this number of target lids - it will not be used
-*     even if it has teh smallest hops count to teh target lid.
+*     even if it has the smallest hops count to the target lid.
 *
 *  ui_ucast_fdb_assign_func_defined
 *     [in] If TRUE - this means that there is a ui ucast_fdb_assign table
 *     function defined (in pfn_ui_ucast_fdb_assign in subnet opts). This
-*     Means that all current entries in the fdbs will be used.
+*     means that all current entries in the fdbs will be used.
 *     If FALSE - such function isn't defined. Do the minimum hop checks 
 *     before accepting the current fdbs.
 *
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/include/vendor/osm_vendor_ts.h gen2.2780/trunk/src/userspace/management/osm/include/vendor/osm_vendor_ts.h
--- gen2.2780.oorig/trunk/src/userspace/management/osm/include/vendor/osm_vendor_ts.h	2005-04-26 20:58:29.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/include/vendor/osm_vendor_ts.h	2005-07-04 21:07:07.000000000 +0200
@@ -115,7 +115,7 @@ typedef struct _osm_ts_bind_info
  *		the file handle to be used for sending the MADs
  *
  * hca_hndl
- *     Handle to the HCA provided by teh underlying VAPI
+ *     Handle to the HCA provided by the underlying VAPI
  *
  *	p_vend
  *		Pointer to the vendor object.
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_anafa.c gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_anafa.c
--- gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_anafa.c	2005-04-26 20:58:32.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_anafa.c	2005-07-04 21:05:23.000000000 +0200
@@ -515,7 +515,7 @@ osm_vendor_send (IN osm_bind_handle_t h_
 			/* Only the requestor expects the error callback */
 			p_bo->send_err_cb (p_bo->cb_context, p_madw);
 		} else {
-			/* put back teh mad - it is useless ... */
+			/* put back the mad - it is useless ... */
 			osm_mad_pool_put (p_bo->p_osm_pool, p_madw);
 		}
 	}
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx.c gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx.c
--- gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx.c	2005-04-26 20:58:32.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx.c	2005-07-04 21:12:12.000000000 +0200
@@ -533,7 +533,7 @@ osm_vendor_send(
     }
     else
     {
-      /* put back teh mad - it is useless ... */
+      /* put back the mad - it is useless ... */
       osm_mad_pool_put(p_bo->p_osm_pool, p_madw );
     }
   }
@@ -542,7 +542,7 @@ osm_vendor_send(
 
   osmv_txn_unlock(p_bo);
 
-  /* we need the ref to happen after the unlock to prevent distruct of the unlock spinlock */
+  /* we need the ref to happen after the unlock to prevent destruct of the unlock spinlock */
   cl_atomic_dec(&p_bo->ref_count);
 
   if (TRUE == p_bo->is_closing)
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ibmgt.c gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ibmgt.c
--- gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ibmgt.c	2005-04-26 20:58:32.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ibmgt.c	2005-07-04 20:59:24.000000000 +0200
@@ -235,7 +235,7 @@ osmv_transport_init(IN osm_bind_info_t *
       }
 
     }
-    /* insert to list of smi's - for raising callbaks later on */
+    /* insert to list of smi's - for raising callbacks later on */
     p_obj = cl_zalloc(sizeof(cl_list_obj_t));
     cl_qlist_set_obj(p_obj,p_bo);
     cl_qlist_insert_tail(p_tpot_info->p_smi_list,&p_obj->list_item);
@@ -281,7 +281,7 @@ osmv_transport_init(IN osm_bind_info_t *
       }
       cl_qlist_init(p_tpot_info->gsi_mgmt_lists[p_info->mad_class]);
     }
-    /* insert to list of smi's - for raising callbaks later on */
+    /* insert to list of smi's - for raising callbacks later on */
     p_obj = cl_zalloc(sizeof(cl_list_obj_t));
     cl_qlist_set_obj(p_obj,p_bo);
     cl_qlist_insert_tail(p_tpot_info->gsi_mgmt_lists[p_info->mad_class],&p_obj->list_item);
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c
--- gen2.2780.oorig/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c	2005-05-26 11:47:54.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c	2005-07-04 21:13:02.000000000 +0200
@@ -362,7 +362,7 @@ osmv_transport_mad_send(IN const osm_bin
 }
 
 /* 
-   register a new mad type to teh opened device file 
+   register a new mad type to the opened device file 
    and send a mad through - the main idea is to make 
    the filter catch it such that the read unblocks
 */
@@ -375,7 +375,7 @@ __osm_transport_gen_dummy_mad(osmv_bind_
   struct ib_get_port_info_ioctl port_data;
   int ts_ioctl_ret;
   
-  /* preapre teh mad fields following the stored filter on the bind */
+  /* prepare the mad fields following the stored filter on the bind */
   cl_memclr(&ts_mad, sizeof(ts_mad));
   ts_mad.format_version = 1;
   ts_mad.mgmt_class     = p_mgr->filter.mgmt_class;
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c gen2.2780/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c
--- gen2.2780.oorig/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c	2005-06-30 20:15:14.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c	2005-07-04 21:14:09.000000000 +0200
@@ -553,7 +553,7 @@ __osm_trap_rcv_process_request(
     {
       osm_log( p_rcv->p_log, OSM_LOG_VERBOSE,
                "__osm_trap_rcv_process_request: "
-               "Continuouslly received this trap %u times. Ignoring it.\n",
+               "Continuously received this trap %u times. Ignoring it.\n",
                num_received);
       goto Exit;
     }


From rep.nop at aon.at  Mon Jul  4 14:30:15 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 23:30:15 +0200
Subject: [openib-general] [PATCH][dat] commentary typos
Message-ID: <20050704213014.GI5997@aon.at>

Hi,

- commentary typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

No obj-code changes.

thank you,
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/linux-kernel/dat-provider/dapl_sp.c gen2.2780/users/jlentini/linux-kernel/dat-provider/dapl_sp.c
--- gen2.2780.oorig/users/jlentini/linux-kernel/dat-provider/dapl_sp.c	2005-06-27 23:07:04.000000000 +0200
+++ gen2.2780/users/jlentini/linux-kernel/dat-provider/dapl_sp.c	2005-07-04 21:29:16.000000000 +0200
@@ -330,9 +330,9 @@ u32 dapl_rsp_query(struct dat_sp *dat_rs
 }
 
 /*
- * Create a persistent Public Service Point that can recieve multiple
+ * Create a persistent Public Service Point that can receive multiple
  * requests for connections and generate multiple connection request
- * instances that wil be delivered to the specified Event Dispatcher
+ * instances that will be delivered to the specified Event Dispatcher
  * in a notification event. Differs from dapl_psp_create() in that
  * the conn_qual is selected by the implementation and returned to
  * the user.
@@ -435,9 +435,9 @@ bail:
 }
 
 /*
- * Create a persistent Public Service Point that can recieve multiple
+ * Create a persistent Public Service Point that can receive multiple
  * requests for connections and generate multiple connection request
- * instances that wil be delivered to the specified Event Dispatcher
+ * instances that will be delivered to the specified Event Dispatcher
  * in a notification event.
  */
 u32 dapl_psp_create(struct dat_ia *dat_ia, DAT_CONN_QUAL conn_qual,
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/linux-kernel/dat-provider/dapl_srq.c gen2.2780/users/jlentini/linux-kernel/dat-provider/dapl_srq.c
--- gen2.2780.oorig/users/jlentini/linux-kernel/dat-provider/dapl_srq.c	2005-06-27 23:07:04.000000000 +0200
+++ gen2.2780/users/jlentini/linux-kernel/dat-provider/dapl_srq.c	2005-07-04 21:22:07.000000000 +0200
@@ -207,7 +207,7 @@ u32 dapl_srq_free(struct dat_srq *srq)
 	if (atomic_read(&srq_ptr->srq_ref_count) != 0) {
 		/*
 		 * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does
-		 * not exist. Have filed the following as an eratta.
+		 * not exist. Have filed the following as an errata.
 		 */
 		status = DAT_ERROR(DAT_INVALID_STATE,
 				   DAT_INVALID_STATE_SRQ_IN_USE);
@@ -338,7 +338,7 @@ u32 dapl_srq_query(struct dat_srq *srq, 
 /*
  * dapl_srq_resize
  *
- * Modify the size fo the event queue of a Shared Recieve Queue
+ * Modify the size of the event queue of a Shared Receive Queue
  *
  * Input:
  * 	srq
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_client.c gen2.2780/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_client.c
--- gen2.2780.oorig/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_client.c	2005-06-27 23:07:03.000000000 +0200
+++ gen2.2780/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_client.c	2005-07-04 21:29:42.000000000 +0200
@@ -409,7 +409,7 @@ DT_Performance_Test_Client_Exchange (
 			test_ptr->ep_context.ep,
 			DT_PERF_SYNC_BUFF_SIZE,
 			dto_cookie,
-			"Recieve Sync_Msg") )
+			"Received Sync_Msg") )
     {
 	return FALSE;
     }
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_server.c gen2.2780/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_server.c
--- gen2.2780.oorig/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_server.c	2005-06-27 23:07:03.000000000 +0200
+++ gen2.2780/users/jlentini/linux-kernel/test/dapltest/test/dapl_performance_server.c	2005-07-04 21:29:27.000000000 +0200
@@ -384,7 +384,7 @@ DT_Performance_Test_Server_Exchange (
 			test_ptr->ep_context.ep,
 			DT_PERF_SYNC_BUFF_SIZE,
 			dto_cookie,
-			"Recieve Sync_Msg") )
+			"Received Sync_Msg") )
     {
 	return FALSE;
     }
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_evd_resize.c gen2.2780/users/jlentini/userspace/dapl/common/dapl_evd_resize.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_evd_resize.c	2005-04-26 20:57:43.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/common/dapl_evd_resize.c	2005-07-04 21:04:18.000000000 +0200
@@ -47,7 +47,7 @@
  *
  * DAPL Requirements Version xxx, 6.3.2.5
  *
- * Modify the size fo the event queue of an Event Dispatcher
+ * Modify the size of the event queue of an Event Dispatcher
  *
  * Input:
  * 	evd_handle
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_psp_create_any.c gen2.2780/users/jlentini/userspace/dapl/common/dapl_psp_create_any.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_psp_create_any.c	2005-04-26 20:57:43.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/common/dapl_psp_create_any.c	2005-07-04 21:34:14.000000000 +0200
@@ -46,9 +46,9 @@
  *
  * uDAPL: User Direct Access Program Library Version 1.1, 6.4.3.3
  *
- * Create a persistent Public Service Point that can recieve multiple
+ * Create a persistent Public Service Point that can receive multiple
  * requests for connections and generate multiple connection request
- * instances that wil be delivered to the specified Event Dispatcher
+ * instances that will be delivered to the specified Event Dispatcher
  * in a notification event. Differs from dapl_psp_create() in that
  * the conn_qual is selected by the implementation and returned to
  * the user.
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_psp_create.c gen2.2780/users/jlentini/userspace/dapl/common/dapl_psp_create.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_psp_create.c	2005-04-26 20:57:43.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/common/dapl_psp_create.c	2005-07-04 21:34:31.000000000 +0200
@@ -46,9 +46,9 @@
  *
  * uDAPL: User Direct Access Program Library Version 1.1, 6.4.1.1
  *
- * Create a persistent Public Service Point that can recieve multiple
+ * Create a persistent Public Service Point that can receive multiple
  * requests for connections and generate multiple connection request
- * instances that wil be delivered to the specified Event Dispatcher
+ * instances that will be delivered to the specified Event Dispatcher
  * in a notification event.
  *
  * Input:
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_srq_free.c gen2.2780/users/jlentini/userspace/dapl/common/dapl_srq_free.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_srq_free.c	2005-04-26 20:57:43.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/common/dapl_srq_free.c	2005-07-04 21:20:35.000000000 +0200
@@ -88,7 +88,7 @@ dapl_srq_free (
     {
 	/*
 	 * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does
-	 * not exist. Have filed the following as an eratta.
+	 * not exist. Have filed the following as an errata.
 	 */
 	dat_status = DAT_ERROR (DAT_INVALID_STATE,DAT_INVALID_STATE_SRQ_IN_USE);
 	goto bail;
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_srq_resize.c gen2.2780/users/jlentini/userspace/dapl/common/dapl_srq_resize.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/common/dapl_srq_resize.c	2005-04-26 20:57:43.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/common/dapl_srq_resize.c	2005-07-04 21:04:06.000000000 +0200
@@ -46,7 +46,7 @@
  *
  * DAPL Requirements Version 1.2, 6.5.7
  *
- * Modify the size fo the event queue of a Shared Recieve Queue
+ * Modify the size of the event queue of a Shared Receive Queue
  *
  * Input:
  * 	srq_handle
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/udapl/dapl_evd_disable.c gen2.2780/users/jlentini/userspace/dapl/udapl/dapl_evd_disable.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/udapl/dapl_evd_disable.c	2005-04-26 20:57:38.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/udapl/dapl_evd_disable.c	2005-07-04 21:04:32.000000000 +0200
@@ -44,7 +44,7 @@
  *
  * DAPL Requirements Version xxx, 6.3.2.5
  *
- * Modify the size fo the event queue of an Event Dispatcher
+ * Modify the size of the event queue of an Event Dispatcher
  *
  * Input:
  * 	evd_handle
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/dapl/udapl/dapl_evd_enable.c gen2.2780/users/jlentini/userspace/dapl/udapl/dapl_evd_enable.c
--- gen2.2780.oorig/users/jlentini/userspace/dapl/udapl/dapl_evd_enable.c	2005-04-26 20:57:38.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/dapl/udapl/dapl_evd_enable.c	2005-07-04 21:04:41.000000000 +0200
@@ -45,7 +45,7 @@
  *
  * DAPL Requirements Version xxx, 6.3.2.5
  *
- * Modify the size fo the event queue of an Event Dispatcher
+ * Modify the size of the event queue of an Event Dispatcher
  *
  * Input:
  * 	evd_handle
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/test/dapltest/test/dapl_performance_client.c gen2.2780/users/jlentini/userspace/test/dapltest/test/dapl_performance_client.c
--- gen2.2780.oorig/users/jlentini/userspace/test/dapltest/test/dapl_performance_client.c	2005-06-29 23:12:53.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/test/dapltest/test/dapl_performance_client.c	2005-07-04 21:27:22.000000000 +0200
@@ -440,7 +440,7 @@ DT_Performance_Test_Client_Exchange (
 			test_ptr->ep_context.ep_handle,
 			DT_PERF_SYNC_BUFF_SIZE,
 			dto_cookie,
-			"Recieve Sync_Msg") )
+			"Received Sync_Msg") )
     {
 	return false;
     }
diff -X excl -rduNp gen2.2780.oorig/users/jlentini/userspace/test/dapltest/test/dapl_performance_server.c gen2.2780/users/jlentini/userspace/test/dapltest/test/dapl_performance_server.c
--- gen2.2780.oorig/users/jlentini/userspace/test/dapltest/test/dapl_performance_server.c	2005-06-29 23:12:53.000000000 +0200
+++ gen2.2780/users/jlentini/userspace/test/dapltest/test/dapl_performance_server.c	2005-07-04 21:27:08.000000000 +0200
@@ -399,7 +399,7 @@ DT_Performance_Test_Server_Exchange (
 			test_ptr->ep_context.ep_handle,
 			DT_PERF_SYNC_BUFF_SIZE,
 			dto_cookie,
-			"Recieve Sync_Msg") )
+			"Received Sync_Msg") )
     {
 	return false;
     }


From rep.nop at aon.at  Mon Jul  4 14:30:28 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 23:30:28 +0200
Subject: [openib-general] [PATCH][mad] commentary typos
Message-ID: <20050704213028.GJ5997@aon.at>

To whom it may concern,

- commentary typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

No obj-code changes.

thank you,
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2780.oorig/trunk/src/userspace/management/libibmad/src/fields.c gen2.2780/trunk/src/userspace/management/libibmad/src/fields.c
--- gen2.2780.oorig/trunk/src/userspace/management/libibmad/src/fields.c	2005-06-05 14:16:19.000000000 +0200
+++ gen2.2780/trunk/src/userspace/management/libibmad/src/fields.c	2005-07-04 21:19:48.000000000 +0200
@@ -44,9 +44,9 @@
 #include <mad.h>
 
 /*
- * BITSOFFS and BE_OFFS are required due the fact that the bit offsets are incosistently
+ * BITSOFFS and BE_OFFS are required due the fact that the bit offsets are inconsistently
  * encoded in the IB spec - IB headers are encoded such that the bit offsets
- * are in big endian convension (BE_OFFS), while the SMI/GSI queries data fields bit
+ * are in big endian convention (BE_OFFS), while the SMI/GSI queries data fields bit
  * offsets are specified using real bit offset (?!)
  * The following macros normalize everything to big endian offsets.
  */
@@ -202,7 +202,7 @@ ib_field_t ib_mad_f [] = {
 	[IB_LINEAR_FORW_TBL_F]		{0, 512, "LinearForwTbl", mad_dump_array},
 
 	/*
-	 * NodeDescription fileds:
+	 * NodeDescription fields:
 	 */
 	[IB_NODE_DESC_F]		{0, 64*8, "NodeDesc", mad_dump_string},
 


From rep.nop at aon.at  Mon Jul  4 14:30:39 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 23:30:39 +0200
Subject: [openib-general] [PATCH][iser] commentary typos
Message-ID: <20050704213039.GK5997@aon.at>

To whom it may concern,

- commentary typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

No obj-code changes.

thank you,
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2780.oorig/ulps/iser/datamover/iser_conn.c gen2.2780/ulps/iser/datamover/iser_conn.c
--- gen2.2780.oorig/ulps/iser/datamover/iser_conn.c	2005-04-26 20:57:58.000000000 +0200
+++ gen2.2780/ulps/iser/datamover/iser_conn.c	2005-07-04 21:30:11.000000000 +0200
@@ -276,7 +276,7 @@ iser_connection_accept(iser_conn_request
 			iser_cr_reject(p_iser_conn);
 			ret_val = ISER_FAILURE;
 			IERROR
-			    ("post recieve pool == NULL, connection rejected\n");
+			    ("post receive pool == NULL, connection rejected\n");
 			goto iser_connection_accept_exit;
 		}
 		/* During the login phase data for sent PDUs is drawn from the same buffer pool as recv */
@@ -1186,7 +1186,7 @@ iser_post_receive_control(struct iser_co
 	ITRACE(ISER_TRACE_CONN, "Added data buffer 0x%p to DTO as entry: %d\n",
 	       p_regd_buf, i);
 
-	ITRACE(ISER_TRACE_CONN, "Call post recieve with the DTO\n");
+	ITRACE(ISER_TRACE_CONN, "Call post receive with the DTO\n");
 	if (iser_post_recv(p_recv_dto) != 0) {
 		IERROR("Failed to post receive DTO\n");
 		ret_val = -1;


From rep.nop at aon.at  Mon Jul  4 14:30:48 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 23:30:48 +0200
Subject: [openib-general] [PATCH][IBMgtSim] commentary typos
Message-ID: <20050704213048.GL5997@aon.at>

Hi,

- commentary typos.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

No obj-code changes.

PS: Eitan; oh and yes, so far i didn't even try to build one of
ibdm nor IBMgtSim, fwiw.. I'll most likely get back to you on those.
thank you,
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2780.oorig/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp gen2.2780/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp
--- gen2.2780.oorig/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp	2005-04-26 20:57:17.000000000 +0200
+++ gen2.2780/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp	2005-07-04 21:12:07.000000000 +0200
@@ -69,7 +69,7 @@ IBMSDispatcher::IBMSDispatcher(
   MSG_EXIT_FUNC;  
 }
 
-/* distructor */
+/* destructor */
 IBMSDispatcher::~IBMSDispatcher()
 {
   MSG_ENTER_FUNC;  
diff -X excl -rduNp gen2.2780.oorig/utils/src/linux-user/IBMgtSim/src/sma.cpp gen2.2780/utils/src/linux-user/IBMgtSim/src/sma.cpp
--- gen2.2780.oorig/utils/src/linux-user/IBMgtSim/src/sma.cpp	2005-07-04 20:19:54.000000000 +0200
+++ gen2.2780/utils/src/linux-user/IBMgtSim/src/sma.cpp	2005-07-04 21:30:50.000000000 +0200
@@ -58,7 +58,7 @@ ibms_dump_mad(
 {
   if (dir == RCV)
   {
-    MSGREG(inf100, 'M', "\n    Recieved Mad Dump\n$", "ibms_dump_mad");
+    MSGREG(inf100, 'M', "\n    Received Mad Dump\n$", "ibms_dump_mad");
     MSGSND(inf100, ibms_get_mad_header_str(madMsg.header));
   }
   else
@@ -1423,7 +1423,7 @@ int IBMSSma::processMad(uint8_t inPort, 
 
   if (madMsg.header.method == IB_MAD_METHOD_TRAP_REPRESS)
   {
-    MSGREG(inf0, 'I', "--- Recieved Trap Repress ---", "processMad");
+    MSGREG(inf0, 'I', "--- Received Trap Repress ---", "processMad");
     MSGSND(inf0);
 
     MSG_EXIT_FUNC;
diff -X excl -rduNp gen2.2780.oorig/utils/src/linux-user/IBMgtSim/src/sma.h gen2.2780/utils/src/linux-user/IBMgtSim/src/sma.h
--- gen2.2780.oorig/utils/src/linux-user/IBMgtSim/src/sma.h	2005-04-26 20:57:17.000000000 +0200
+++ gen2.2780/utils/src/linux-user/IBMgtSim/src/sma.h	2005-07-04 21:30:32.000000000 +0200
@@ -63,7 +63,7 @@
 *
 * DESCRIPTION
 *	upper level function of printing all mad contant in
-*   recieving or sending of Mad
+*   receiving or sending of Mad
 *
 * SYNOPSIS
 */

From rep.nop at aon.at  Mon Jul  4 14:48:20 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 4 Jul 2005 23:48:20 +0200
Subject: [Openib-windows] Re: [openib-general] Re: IB Diagnositic Tool s
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30415@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30415@mtlex01.yok.mtl.com>
Message-ID: <20050704214820.GM5997@aon.at>

On Mon, Jul 04, 2005 at 11:54:18PM +0300, Eitan Zahavi wrote:
>Hi Ben
>
>The patch try to start scanning the ports from i=1 rather then i=0.

Hm? My patch started to scan from 0 and not 1;

The patch i sent to Hal in February kindly started at 1 (i.e. current
opensm), but i wonder what port 0 in this case does denotes.

Please clarify.

>
>status = osm_vendor_get_all_port_attr(p_osm->p_vendor, attr_array,
>&num_ports );
>
>But this function to return the port attributes of all available ports does
>not use the "port number" as index into the returned array of port
>attributes. It simply starts with i=0 and adds on to it. The port numbers
>are available in the attributes struct.
>
>Eitan Zahavi
>Design Technology Director
>Mellanox Technologies LTD
>Tel:+972-4-9097208
>Fax:+972-4-9593245
>P.O. Box 586 Yokneam 20692 ISRAEL
Bernhard


From surs at cse.ohio-state.edu  Mon Jul  4 15:19:05 2005
From: surs at cse.ohio-state.edu (Sayantan Sur)
Date: Mon, 4 Jul 2005 18:19:05 -0400
Subject: [openib-general] compilation problem (no member devt)
Message-ID: <20050704221903.GA16913@cse.ohio-state.edu>


Hi,

I am facing a compile problem with the following error:

drivers/infiniband/core/uverbs_main.c: In function `ib_uverbs_add_one':
drivers/infiniband/core/uverbs_main.c:578: error: structure has no
member named `devt'
make[3]: *** [drivers/infiniband/core/uverbs_main.o] Error 1
make[2]: *** [drivers/infiniband/core] Error 2
make[1]: *** [drivers/infiniband] Error 2
make: *** [drivers] Error 2

The kernel version I am using is 2.6.11.7. I symlinked the
drivers/infiniband directory to trunk/src/linux-kernel/infiniband/. The
Gen2 tree I am using is 2780 (for both user & kernel components).  The
platform used is IA32.

Can somebody let me know how to get around this problem?

Thanks,
Sayantan.

-- 
http://www.cse.ohio-state.edu/~surs


From mst at mellanox.co.il  Mon Jul  4 16:04:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 02:04:12 +0300
Subject: [openib-general] Re: compilation problem (no member devt)
In-Reply-To: <20050704221903.GA16913@cse.ohio-state.edu>
References: <20050704221903.GA16913@cse.ohio-state.edu>
Message-ID: <20050704230412.GA17422@mellanox.co.il>

Quoting r. Sayantan Sur <surs at cse.ohio-state.edu>:
> Subject: compilation problem (no member devt)
> 
> 
> Hi,
> 
> I am facing a compile problem with the following error:
> 
> drivers/infiniband/core/uverbs_main.c: In function `ib_uverbs_add_one':
> drivers/infiniband/core/uverbs_main.c:578: error: structure has no
> member named `devt'
> make[3]: *** [drivers/infiniband/core/uverbs_main.o] Error 1
> make[2]: *** [drivers/infiniband/core] Error 2
> make[1]: *** [drivers/infiniband] Error 2
> make: *** [drivers] Error 2
> 
> The kernel version I am using is 2.6.11.7. I symlinked the
> drivers/infiniband directory to trunk/src/linux-kernel/infiniband/. The
> Gen2 tree I am using is 2780 (for both user & kernel components).  The
> platform used is IA32.
> 
> Can somebody let me know how to get around this problem?
> 
> Thanks,
> Sayantan.

main trunk only supports 2.6.12 and up.
Try this patch:

Index: infiniband/core/uverbs_main.c
===================================================================
--- infiniband/core/uverbs_main.c	(revision 2744)
+++ infiniband/core/uverbs_main.c	(revision 2743)
@@ -509,6 +509,15 @@ static struct ib_client uverbs_client = 
 	.remove = ib_uverbs_remove_one
 };
 
+static ssize_t show_dev(struct class_device *class_dev, char *buf)
+{
+	struct ib_uverbs_device *dev =
+		container_of(class_dev, struct ib_uverbs_device, class_dev);
+
+	return print_dev_t(buf, dev->dev.dev);
+}
+static CLASS_DEVICE_ATTR(dev, S_IRUGO, show_dev, NULL);
+
 static ssize_t show_ibdev(struct class_device *class_dev, char *buf)
 {
 	struct ib_uverbs_device *dev =
@@ -575,11 +584,12 @@ static void ib_uverbs_add_one(struct ib_
 
 	uverbs_dev->class_dev.class = &uverbs_class;
 	uverbs_dev->class_dev.dev   = device->dma_device;
-	uverbs_dev->class_dev.devt  = uverbs_dev->dev.dev;
 	snprintf(uverbs_dev->class_dev.class_id, BUS_ID_SIZE, "uverbs%d", uverbs_dev->devnum);
 	if (class_device_register(&uverbs_dev->class_dev))
 		goto err_cdev;
 
+	if (class_device_create_file(&uverbs_dev->class_dev, &class_device_attr_dev))
+		goto err_class;
 	if (class_device_create_file(&uverbs_dev->class_dev, &class_device_attr_ibdev))
 		goto err_class;
 
-- 
MST


From ianjiang91 at hotmail.com  Mon Jul  4 18:07:49 2005
From: ianjiang91 at hotmail.com (Ian Jiang)
Date: Tue, 05 Jul 2005 09:07:49 +0800
Subject: [openib-general] [iser]about the target
Message-ID: <BAY22-F2C066127BE3FC92439269CAE60@phx.gbl>

Hi!
I am new to the iSER.
On "https://openib.org/tiki/tiki-index.php?page=iSER", it is said that iSER 
currently contains initiator only (no target). Will the target come out 
later? How did they test the iSER initiator without a iSER target?
Could you give some explaination?
Thanks!



Ian Jiang
ianjiang91 at hotmail.com
----
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62564394(office)

_________________________________________________________________
与联机的朋友进行交流，请使用 MSN Messenger:  http://messenger.msn.com/cn  



From rolandd at cisco.com  Mon Jul  4 19:09:54 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 04 Jul 2005 19:09:54 -0700
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <20050704221903.GA16913@cse.ohio-state.edu> (Sayantan Sur's
	message of "Mon, 4 Jul 2005 18:19:05 -0400")
References: <20050704221903.GA16913@cse.ohio-state.edu>
Message-ID: <52pstyvw99.fsf@topspin.com>

    Sayantan> I am facing a compile problem with the following error:

    Sayantan> drivers/infiniband/core/uverbs_main.c: In function
    Sayantan> `ib_uverbs_add_one':
    Sayantan> drivers/infiniband/core/uverbs_main.c:578: error:
    Sayantan> structure has no member named `devt'

Sorry, I checked in a change that requires a 2.6.13-rc1 kernel.  I've
reverted the change, so the tree should compile against 2.6.12 again.

 - R.


From rolandd at cisco.com  Mon Jul  4 19:11:21 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 04 Jul 2005 19:11:21 -0700
Subject: [openib-general] Re: [PATCH] updates to copyright
In-Reply-To: <20050704184038.GA16650@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 4 Jul 2005 21:40:38 +0300")
References: <ORSMSX401GryWtIIZS200000015@orsmsx401.amr.corp.intel.com>
	<20050704184038.GA16650@mellanox.co.il>
Message-ID: <52ll4mvw6u.fsf@topspin.com>

Thanks, applied.

 - R.


From mst at mellanox.co.il  Tue Jul  5 00:24:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 10:24:33 +0300
Subject: [openib-general] documentation under src dir
Message-ID: <20050705072433.GI14130@mellanox.co.il>

Hal, Shahar, could you please move the following files
outside the src directory?

./trunk/src/userspace/management/osm/doc/OpenSM_RN.pdf
./trunk/src/userspace/management/osm/doc/OpenSM_UM.pdf

These are not sources, so they dont belong in src.
I suggest removing OpenSM_RN.pdf and moving OpenSM_UM.pdf to the docs
section on the web site.

Thanks,

-- 
MST


From eitan at mellanox.co.il  Tue Jul  5 04:23:57 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 14:23:57 +0300 
Subject: [openib-general] RE: [openib-commits] r2776 - gen2/utils/src/
	linux-user/ibdm/datam odel
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3041C@mtlex01.yok.mtl.com>

Hi Ber,

Thanks.
I have used an ex script to fix the following typos on all the code tree's I
own.
In the future - it is much better from my perspective to simply get the list
of words I need to check for.

eb -> be
teh -> the
fo -> of
accoumulated -> accumulated

I hope you will be able to simply send me the list of words. It will make my
life easier.

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Bernhard Fischer [mailto:rep.nop at aon.at]
> Sent: Monday, July 04, 2005 9:57 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] RE: [openib-commits] r2776 -
gen2/utils/src/linux-
> user/ibdm/datam odel
> 
> On Mon, Jul 04, 2005 at 08:51:10PM +0200, Bernhard Fischer wrote:
> >On Mon, Jul 04, 2005 at 10:20:16AM +0300, Eitan Zahavi wrote:
> >>Thanks. Committed as 2779.
> >
> >I'm sorry but i overlooked another occurance of acor; attached.
> >Would you be so kind and do these in ibdm and IBMgtSim? TIA.
> >s/teh/the/g
> >s/eb/be/g
> >s/fo/of/g
> >this may or may not work for you:
> >egrep -ri "([[:space:]](fo|teh|eb)[[:space:]])" .
> 
> >Index: utils/src/linux-user/IBMgtSim/utils/RunSimTest
> >===================================================================
> >--- utils/src/linux-user/IBMgtSim/utils/RunSimTest	(revision 2780)
> >+++ utils/src/linux-user/IBMgtSim/utils/RunSimTest	(working copy)
> >@@ -112,18 +112,18 @@ proc Help {} {
> 
> >-# The log fo all events is accoumulated in the global list:
> >+# The log of all events is accoumulated in the global list:
> 
> s/accoumulate/accumulate/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/51900cc5/attachment.html>

From mst at mellanox.co.il  Tue Jul  5 04:34:45 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 14:34:45 +0300
Subject: [openib-general] Re: backport patch for 2.6.11
In-Reply-To: <20050628123932.GK25304@mellanox.co.il>
References: <20050628123932.GK25304@mellanox.co.il>
Message-ID: <20050705113445.GL14130@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: backport patch for 2.6.11
> 
> Hi!
> I have uploaded the patches to make svn trunk build on 2.6.11.
> They can be found in
> 
> https://openib.org/svn/gen2/branches/backport-to-2.6.11
> 
> Enjoy!

SDP patch updated to revision 2784.

-- 
MST


From halr at voltaire.com  Tue Jul  5 05:08:57 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 08:08:57 -0400
Subject: [openib-general] Re: mstflint doesn't build on x86_64
In-Reply-To: <20050702212328.GA21801@mellanox.co.il>
References: <1120303238.4389.754.camel@hal.voltaire.com>
	<20050702212328.GA21801@mellanox.co.il>
Message-ID: <1120565336.4388.7094.camel@hal.voltaire.com>

On Sat, 2005-07-02 at 17:23, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > Subject: mstflint doesn't build on x86_64
> > 
> > Hi,
> > 
> > mstflint doesn't build on x86_64. Here is the version of g++:
> > g++ -v
> > Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.2/specs
> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-languages=c,c++,objc,java,f77 --enable-java-awt=gtk --host=x86_64-redhat-linux
> > Thread model: posix
> > gcc version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)
> > 
> > Thanks.
> > 
> > -- Hal
> 
> Probably more to do with gcc version than the architecture.
> 
> Does the following patch help?

Yes. Thanks.

-- Hal




From halr at voltaire.com  Tue Jul  5 05:16:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 08:16:33 -0400
Subject: [openib-general] Re: [PATCH][osm] commentary typos
In-Reply-To: <20050703225104.GD5997@aon.at>
References: <20050703225104.GD5997@aon.at>
Message-ID: <1120565793.4388.7138.camel@hal.voltaire.com>

On Sun, 2005-07-03 at 18:51, Bernhard Fischer wrote:
> - commentary typos
> 
> Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

Thanks. Applied.

-- Hal



From halr at voltaire.com  Tue Jul  5 05:27:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 08:27:33 -0400
Subject: [openib-general] [PATCH][osm] commentary typos
In-Reply-To: <20050704213006.GH5997@aon.at>
References: <20050704213006.GH5997@aon.at>
Message-ID: <1120566453.4388.7215.camel@hal.voltaire.com>

On Mon, 2005-07-04 at 17:30, Bernhard Fischer wrote:
> Hi,
> 
> - commentary typos.
> 
> Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
> 
> No obj-code changes.
> 
> thank you,
> Bernhard
> 
> ______________________________________________________________________
Thanks. Applied.

Note that some of these changes are to gen1 files (not used for gen2):
M    osm/include/vendor/osm_vendor_ts.h
M    osm/libvendor/osm_vendor_mlx_ts.c
M    osm/libvendor/osm_vendor_mlx_ibmgt.c
M    osm/libvendor/osm_vendor_mlx_anafa.c

-- Hal



From halr at voltaire.com  Tue Jul  5 05:33:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 08:33:05 -0400
Subject: [openib-general] [PATCH][mad] commentary typos
In-Reply-To: <20050704213028.GJ5997@aon.at>
References: <20050704213028.GJ5997@aon.at>
Message-ID: <1120566784.4388.7265.camel@hal.voltaire.com>

On Mon, 2005-07-04 at 17:30, Bernhard Fischer wrote:
> To whom it may concern,
> 
> - commentary typos.
> 
> Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
> 
> No obj-code changes.
> 
> thank you,
> Bernhard
> 
> ______________________________________________________________________
Thanks. Applied.

Note the library is libibmad so as not to be confused with mad.

-- Hal



From halr at voltaire.com  Tue Jul  5 05:43:32 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 08:43:32 -0400
Subject: [openib-general] [PATCH][iser] commentary typos
In-Reply-To: <20050704213039.GK5997@aon.at>
References: <20050704213039.GK5997@aon.at>
Message-ID: <1120567412.4388.7346.camel@hal.voltaire.com>

On Mon, 2005-07-04 at 17:30, Bernhard Fischer wrote:
> To whom it may concern,
> 
> - commentary typos.
> 
> Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
> 
> No obj-code changes.
> 
> thank you,
> Bernhard

Thanks. Applied.

-- Hal



From mst at mellanox.co.il  Tue Jul  5 05:53:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 15:53:12 +0300
Subject: [openib-general] [PATCH] mthca: updates to copyright
In-Reply-To: <20050704184038.GA16650@mellanox.co.il>
References: <ORSMSX401GryWtIIZS200000015@orsmsx401.amr.corp.intel.com>
	<20050704184038.GA16650@mellanox.co.il>
Message-ID: <20050705125312.GR14130@mellanox.co.il>

The following patch adds copyright statements for Mellanox to files that
I've modified in mthca.

---

Add Mellanox copyright to files modified by Michael S. Tsirkin.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: hw/mthca/mthca_dev.h
===================================================================
--- hw/mthca/mthca_dev.h	(revision 2784)
+++ hw/mthca/mthca_dev.h	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_main.c
===================================================================
--- hw/mthca/mthca_main.c	(revision 2784)
+++ hw/mthca/mthca_main.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_memfree.h
===================================================================
--- hw/mthca/mthca_memfree.h	(revision 2784)
+++ hw/mthca/mthca_memfree.h	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_provider.c
===================================================================
--- hw/mthca/mthca_provider.c	(revision 2784)
+++ hw/mthca/mthca_provider.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Cisco Systems. All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_provider.h
===================================================================
--- hw/mthca/mthca_provider.h	(revision 2784)
+++ hw/mthca/mthca_provider.h	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_profile.c
===================================================================
--- hw/mthca/mthca_profile.c	(revision 2784)
+++ hw/mthca/mthca_profile.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_mad.c
===================================================================
--- hw/mthca/mthca_mad.c	(revision 2784)
+++ hw/mthca/mthca_mad.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_cmd.c
===================================================================
--- hw/mthca/mthca_cmd.c	(revision 2784)
+++ hw/mthca/mthca_cmd.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_pd.c
===================================================================
--- hw/mthca/mthca_pd.c	(revision 2784)
+++ hw/mthca/mthca_pd.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_cq.c
===================================================================
--- hw/mthca/mthca_cq.c	(revision 2784)
+++ hw/mthca/mthca_cq.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Cisco Systems, Inc. All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_profile.h
===================================================================
--- hw/mthca/mthca_profile.h	(revision 2784)
+++ hw/mthca/mthca_profile.h	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_eq.c
===================================================================
--- hw/mthca/mthca_eq.c	(revision 2784)
+++ hw/mthca/mthca_eq.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_config_reg.h
===================================================================
--- hw/mthca/mthca_config_reg.h	(revision 2784)
+++ hw/mthca/mthca_config_reg.h	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_doorbell.h
===================================================================
--- hw/mthca/mthca_doorbell.h	(revision 2784)
+++ hw/mthca/mthca_doorbell.h	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_cmd.h
===================================================================
--- hw/mthca/mthca_cmd.h	(revision 2784)
+++ hw/mthca/mthca_cmd.h	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_mr.c
===================================================================
--- hw/mthca/mthca_mr.c	(revision 2784)
+++ hw/mthca/mthca_mr.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 2784)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems. All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: hw/mthca/mthca_memfree.c
===================================================================
--- hw/mthca/mthca_memfree.c	(revision 2784)
+++ hw/mthca/mthca_memfree.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU

-- 
MST


From halr at voltaire.com  Tue Jul  5 05:53:28 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 08:53:28 -0400
Subject: [openib-general] Some Missing Userspace Verbs
Message-ID: <1120568008.4388.7430.camel@hal.voltaire.com>

Hi Roland,

In glancing at the user verbs, the following appear to be missing:

peek_cq
modify_ah

less important:
query_ah
query_qp

Thanks.

-- Hal



From mst at mellanox.co.il  Tue Jul  5 06:13:22 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 16:13:22 +0300
Subject: [openib-general] Re: Some Missing Userspace Verbs
In-Reply-To: <1120568008.4388.7430.camel@hal.voltaire.com>
References: <1120568008.4388.7430.camel@hal.voltaire.com>
Message-ID: <20050705131322.GS14130@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: Some Missing Userspace Verbs
> 
> Hi Roland,
> 
> In glancing at the user verbs, the following appear to be missing:
> 
> peek_cq
> modify_ah

Why are they useful?

-- 
MST


From halr at voltaire.com  Tue Jul  5 06:13:27 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 09:13:27 -0400
Subject: [openib-general] Re: Some Missing Userspace Verbs
In-Reply-To: <20050705131322.GS14130@mellanox.co.il>
References: <1120568008.4388.7430.camel@hal.voltaire.com>
	<20050705131322.GS14130@mellanox.co.il>
Message-ID: <1120569206.4388.7630.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 09:13, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > Subject: Some Missing Userspace Verbs
> > 
> > Hi Roland,
> > 
> > In glancing at the user verbs, the following appear to be missing:
> > 
> > peek_cq
> > modify_ah
> 
> Why are they useful?

I would presume for the same reasons they are useful in kernel space.

-- Hal



From halr at voltaire.com  Tue Jul  5 06:20:52 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 09:20:52 -0400
Subject: [openib-general] Re: IBDM - IB DataModel build issues fixed
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>
Message-ID: <1120569652.4388.7727.camel@hal.voltaire.com>

On Sat, 2005-07-02 at 15:38, Eitan Zahavi wrote:
> Could not find your last mail with the issue.
> 
> But I was able to update the IBDM code at:
> 
> https://openib.org/svn/gen2/utils/src/linux-user/ibdm
> 
> I was able to checkout and build it with no issue
> 
> But only after installing:
> 
> autoconf 2.59
> 
> automake 1.9.3

This now builds for me too. Thanks.

Rather than installing the headers in /usr/local/include/ibdm for
OpenIB, these should likely be /usr/local/include/infiniband/ibdm.

Similarly for the libraries and executables.

Some other nits:

It appears the following are the executables:
ibdmchk, ibdmtr, topotest, ibnlparse
but the README references the following:
fabric_sim, fabric_tr, and osm_check.

NEWS indicates rev 1.1 was 12/4/03. What version is this ? Also, this
installs to .../ibadm/ibdm1.0/

-- Hal



From halr at voltaire.com  Tue Jul  5 06:25:43 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 09:25:43 -0400
Subject: [openib-general] Re: IBDM - IB DataModel build issues fixed
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>
Message-ID: <1120569732.4388.7740.camel@hal.voltaire.com>

On Sat, 2005-07-02 at 15:38, Eitan Zahavi wrote:
> Could not find your last mail with the issue.
> 
> But I was able to update the IBDM code at:
> 
> https://openib.org/svn/gen2/utils/src/linux-user/ibdm
> 
> I was able to checkout and build it with no issue
> 
> But only after installing:
> 
> autoconf 2.59
> 
> automake 1.9.3

This now builds for me too. Thanks.

Rather than installing the headers in /usr/local/include/ibdm for
OpenIB, these should likely be /usr/local/include/infiniband/ibdm.

Similarly for the libraries and executables.

Some other nits:

There are +x permissions on a lot of the files which don't need them.

It appears the following are the executables:
ibdmchk, ibdmtr, topotest, ibnlparse
but the README references the following:
fabric_sim, fabric_tr, and osm_check.

NEWS indicates rev 1.1 was 12/4/03. What version is this ? Also, this
installs to .../ibadm/ibdm1.0/

-- Hal



From halr at voltaire.com  Tue Jul  5 06:28:56 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 09:28:56 -0400
Subject: [openib-general] IBMgtSim building
Message-ID: <1120569942.4388.7777.camel@hal.voltaire.com>

Hi again Eitan,

When building IBMgtSim, I get the following error:
checking for /usr/include/ibdm/Fabric.h... no
configure: error:  could not find /usr/include/ibdm/Fabric.h.

ibdm put this in /usr/local/include/ibdm (although this may be moved to
an OpenIB subdirectory).

-- Hal



From mst at mellanox.co.il  Tue Jul  5 06:38:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 16:38:49 +0300
Subject: [openib-general] Re: Some Missing Userspace Verbs
In-Reply-To: <1120569206.4388.7630.camel@hal.voltaire.com>
References: <1120568008.4388.7430.camel@hal.voltaire.com>
	<20050705131322.GS14130@mellanox.co.il>
	<1120569206.4388.7630.camel@hal.voltaire.com>
Message-ID: <20050705133848.GT14130@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > In glancing at the user verbs, the following appear to be missing:
> > > 
> > > peek_cq
> > > modify_ah
> > 
> > Why are they useful?
> 
> I would presume for the same reasons they are useful in kernel space.

Maybe they arent.
These arent implemented in mthca.

-- 
MST


From halr at voltaire.com  Tue Jul  5 06:39:47 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 09:39:47 -0400
Subject: [openib-general] Some IBMgtSim Questions and Nits
Message-ID: <1120570787.4388.7934.camel@hal.voltaire.com>

Hi again Eitan,

In INSTALL, it states:
Software Dependencies
=====================
IBMgtSim depends on: tcl8.3/8.4, OpenSM 1.7.0 and ibdm 1.0

What are the specific dependencies on OpenSM 1.7.0 ?
Should the swig dependency listed ?

Also, can you update the instructions (README, INSTALL, etc.) for OpenIB
?

README states:
If there there is no IBADM installed

What is IBADM ? Is this a typo (IBDM) or is it something else ?

Some more nits with IBMgtSim:

Should IBMgtSim/src/.autosave be removed from the tree ?

Also there are a number of files in this subdirectory (and underneath
it) which have +x permissions and don't need this.

Thanks.

-- Hal



From halr at voltaire.com  Tue Jul  5 06:43:13 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 09:43:13 -0400
Subject: [openib-general] Re: Some Missing Userspace Verbs
In-Reply-To: <20050705133848.GT14130@mellanox.co.il>
References: <1120568008.4388.7430.camel@hal.voltaire.com>
	<20050705131322.GS14130@mellanox.co.il>
	<1120569206.4388.7630.camel@hal.voltaire.com>
	<20050705133848.GT14130@mellanox.co.il>
Message-ID: <1120570895.4388.7957.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 09:38, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > > In glancing at the user verbs, the following appear to be missing:
> > > > 
> > > > peek_cq
> > > > modify_ah
> > > 
> > > Why are they useful?
> > 
> > I would presume for the same reasons they are useful in kernel space.
> 
> Maybe they arent.
> These arent implemented in mthca.

Even if so, shouldn't they still be exposed in user verbs as this is
supposed to be IB hardware independent (and return some error for mthca
if they are not useful) ?

-- Hal



From halr at voltaire.com  Tue Jul  5 07:03:16 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 10:03:16 -0400
Subject: [openib-general] Re: IBDM - IB DataModel build issues fixed
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30401@mtlex01.yok.mtl.com>
Message-ID: <1120572026.4388.8124.camel@hal.voltaire.com>

Hi Eitan,

On Sat, 2005-07-02 at 15:38, Eitan Zahavi wrote:
> (IBDM is a flexible data model for describing arbitrary IB systems and
> then instantiating them in a topology. Then based on this topology an
> in-memory data model is built and various checking, simulation and
> analysis procedures are provided.)

I need some more time with the data model. 

Why does the TOPSYSTEM need to be identified ? Is it just a way of not
having it as the first file in the system definition ?

The IBNL document mentions IBADM. What is IBADM ?

It also mentions IB netlist and topology file. What is the IB netlist ?
Where does the topology file come from ? Is there a topology file which
uses system names (and requires underlying system definition files (to
be predefined)) ?

Also, can you list the (current and anticipated) consumers of this data
model ? I presume the IBMgtSim is one. I also see: ibnlparse, ibdmchk,
ibdmtr, and topotest. Are there others ? Is there any description of
these ?

Thanks.

-- Hal



From eitan at mellanox.co.il  Tue Jul  5 07:35:54 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 17:35:54 +0300 
Subject: [openib-general] RE: IBMgtSim building
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30420@mtlex01.yok.mtl.com>

IBMgtSim configure takes --with-ibdm flag that should point to the
pre-installed IBDM prefix (which is /usr by default).

The README file in the IBMgtSim dir say:
"
2. configure:
   ./configure --with-osm=<opensm install path> --prefix=<where ever you
want> 

   Optionally you can add:
   --enable-debug --enable-maintainer-mode 

   If there there is no IBADM installed on the machine you will need to
provide:
   --with-ibdm=<where ever you did install it>

   NOTE: the pointer to the osm install path can be to any osm (not just the
one 
   supporting the simulator. e.g. /usr
   
   Example:
    ./configure --enable-debug --enable-maintainer-mode \
        --with-osm=/home/eitan/SW/osm/linuxuser/bin/x86/2.6.5/debug \
        --prefix=/home/eitan/SW/cvsroot/IBMgtSim/usr \
        --with-ibdm=/home/eitan/SW/ibadm_build/usr
"

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Tuesday, July 05, 2005 4:29 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: IBMgtSim building
> 
> Hi again Eitan,
> 
> When building IBMgtSim, I get the following error:
> checking for /usr/include/ibdm/Fabric.h... no
> configure: error:  could not find /usr/include/ibdm/Fabric.h.
> 
> ibdm put this in /usr/local/include/ibdm (although this may be moved to
> an OpenIB subdirectory).
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/9bb57ab6/attachment.html>

From halr at voltaire.com  Tue Jul  5 07:42:13 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 10:42:13 -0400
Subject: [openib-general] RE: IBMgtSim building
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30420@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30420@mtlex01.yok.mtl.com>
Message-ID: <1120574533.4388.8414.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 10:35, Eitan Zahavi wrote:
> IBMgtSim configure takes --with-ibdm flag that should point to the
> pre-installed IBDM prefix (which is /usr by default).

Shouldn't the default for IBMgtSim be the same as where ibdm put things
(by default) ?

-- Hal



From eitan at mellanox.co.il  Tue Jul  5 08:02:36 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 18:02:36 +0300 
Subject: [openib-general] RE: IBDM - IB DataModel build issues fixed
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30421@mtlex01.yok.mtl.com>

Hi Hal,

> Why does the TOPSYSTEM need to be identified ? Is it just a way of not
> having it as the first file in the system definition ?
> 
> The IBNL document mentions IBADM. What is IBADM ?
[EZ] I'm sorry about that. It is really IBDM. I attach here the updated
document. Also removed any IBADM remedies from the install path.

> 
> It also mentions IB netlist and topology file. What is the IB netlist ?
[EZ] IBNL is IB NetList - it is the internal system connectivity file. The
term "net list" is very common in the CAD world for describing connectivity
information.

> Where does the topology file come from ? 
[EZ] The user is supposed to provide that. Normally when large clusters are
built the same information is used for labeling the cables.

Is there a topology file which
> uses system names (and requires underlying system definition files (to
> be predefined)) ?
[EZ] I am not sure what you mean by "Is there"? 
Yes the topology file will use systems that are defined by IBNL and list
their inter connections. I already provided an example:

MT2400 mySwitch
   P1 -> Cougar H-1 P2
   P2 -> Cougar H-2 P1 
   
Is a simple topology file describing two Cougars (but you might use other
HCA names of course) to a 24 port switch system named MT2400.

> 
> Also, can you list the (current and anticipated) consumers of this data
> model ? 
[EZ] Good question:
ibnlparse - a simple parser and checker for IBNL files only. 

I presume the IBMgtSim is one. I also see: ibnlparse, ibdmchk,
> ibdmtr, and topotest. Are there others ? Is there any description of
> these ?
> 
> Thanks.
> 
> -- Hal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/f418f1b9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IBNL Specification.doc
Type: application/msword
Size: 96768 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/f418f1b9/attachment.doc>

From eitan at mellanox.co.il  Tue Jul  5 08:04:55 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 18:04:55 +0300 
Subject: [openib-general] RE: IBMgtSim building
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30422@mtlex01.yok.mtl.com>

I hope it is

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Tuesday, July 05, 2005 5:42 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: RE: IBMgtSim building
> 
> On Tue, 2005-07-05 at 10:35, Eitan Zahavi wrote:
> > IBMgtSim configure takes --with-ibdm flag that should point to the
> > pre-installed IBDM prefix (which is /usr by default).
> 
> Shouldn't the default for IBMgtSim be the same as where ibdm put things
> (by default) ?
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/1c898c3e/attachment.html>

From eitan at mellanox.co.il  Tue Jul  5 08:10:05 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 18:10:05 +0300 
Subject: [openib-general] RE: IBDM - IB DataModel build issues fixed - full
	version
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30423@mtlex01.yok.mtl.com>

Hi Hal,

> Why does the TOPSYSTEM need to be identified ? Is it just a way of not
> having it as the first file in the system definition ?
> 
> The IBNL document mentions IBADM. What is IBADM ?
[EZ] I'm sorry about that. It is really IBDM. I attach here the updated
document. Also removed any IBADM remedies from the install path.

> 
> It also mentions IB netlist and topology file. What is the IB netlist ?
[EZ] IBNL is IB NetList - it is the internal system connectivity file. The
term "net list" is very common in the CAD world for describing connectivity
information.

> Where does the topology file come from ? 
[EZ] The user is supposed to provide that. Normally when large clusters are
built the same information is used for labeling the cables.

Is there a topology file which
> uses system names (and requires underlying system definition files (to
> be predefined)) ?
[EZ] I am not sure what you mean by "Is there"? 
Yes the topology file will use systems that are defined by IBNL and list
their inter connections. I already provided an example:

MT2400 mySwitch
   P1 -> Cougar H-1 P2
   P2 -> Cougar H-2 P1 
   
Is a simple topology file describing two Cougars (but you might use other
HCA names of course) to a 24 port switch system named MT2400.

> 
> Also, can you list the (current and anticipated) consumers of this data
> model ? 
[EZ] Good question:
ibnlparse - a simple parser and checker for IBNL files only. 
ibdmtr - traces a directed route through the topology (both system and node
level)
ibdmchk - both "design" and "verification" tool for topologies. For "design"
mode it takes topology file as the fabric definition. For "verification"
mode it takes OpenSM dump files:subnet.lst, osm.fdbs, osm.mcfdbs
topotest - check if the given topology matches the discovered one
(discovered topology is given as opensm subnet.lst format).

I presume the IBMgtSim is one. I also see: ibnlparse, ibdmchk,
> ibdmtr, and topotest. Are there others ? Is there any description of
> these ?
Each one has -h (maybe ibnltest doe not - but it is an IBNL a developers
tool...)
> 
> Thanks.
> 
> -- Hal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/e57acdc6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IBNL Specification.doc
Type: application/msword
Size: 96768 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/e57acdc6/attachment.doc>

From eitan at mellanox.co.il  Tue Jul  5 08:22:13 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 18:22:13 +0300 
Subject: [openib-general] RE: Some IBMgtSim Questions and Nits
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30424@mtlex01.yok.mtl.com>

Hi Hal,

> 
> In INSTALL, it states:
> Software Dependencies
> =====================
> IBMgtSim depends on: tcl8.3/8.4, OpenSM 1.7.0 and ibdm 1.0
> 
> What are the specific dependencies on OpenSM 1.7.0 ?
[EZ] Actually it should say 1.7.0 and after. 
The dependency is on the subnet.lst format and osm.mcfdbs dump file. But
this is mostly a "run time" dependency for ibdmchk ...

> Should the swig dependency listed ?
[EZ] Well, I include in the release the code auto-generated by swig. So a
user that does not need to re-generate (did not change the TCL API) does not
need swig.
> 
> Also, can you update the instructions (README, INSTALL, etc.) for OpenIB
> ?
> 
> README states:
> If there there is no IBADM installed
> 
> What is IBADM ? Is this a typo (IBDM) or is it something else ?
[EZ] Should say IBDM. I will fix that.
> 
> Some more nits with IBMgtSim:
> 
> Should IBMgtSim/src/.autosave be removed from the tree ?
[EZ] Yes. Thanks - I just did that.
> 
> Also there are a number of files in this subdirectory (and underneath
> it) which have +x permissions and don't need this.
[EZ] How can I modify the file permissions after they are committed to svn ?
> 
> Thanks.
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/f260e7e9/attachment.html>

From halr at voltaire.com  Tue Jul  5 08:20:08 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 11:20:08 -0400
Subject: [openib-general] ibdm building
Message-ID: <1120576807.4388.8704.camel@hal.voltaire.com>

Hi Eitan,

With your latest ibdm changes:
D  src/linux-user/ibdm/ibnl/Makefile.in
U  src/linux-user/ibdm/src/test_topo_match.cpp
D  src/linux-user/ibdm/src/Makefile.in
U  src/linux-user/ibdm/datamodel/Makefile.am
D  src/linux-user/ibdm/datamodel/Makefile.in
U  src/linux-user/ibdm/Clusters/Makefile.am
D  src/linux-user/ibdm/Clusters/Makefile.in
D  src/linux-user/ibdm/config/share/libtool/libltdl/Makefile.in
D  src/linux-user/ibdm/replace/Makefile.in
D  src/linux-user/ibdm/Makefile.in

I now get the following when building ibdm:

./configure
...
configure: creating ./config.status
config.status: creating Makefile
config.status: error: cannot find input file: Makefile.in

-- Hal





From eitan at mellanox.co.il  Tue Jul  5 08:32:27 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 18:32:27 +0300 
Subject: [openib-general] RE: ibdm building
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30428@mtlex01.yok.mtl.com>

Makefile.in is normally only distributed at release time.
The code source tree does not normally hold Makefile.in as they are
generated by autoconf. 
You need to run ./config/bootstrap to regenerate them...

EZ

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Tuesday, July 05, 2005 6:20 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: ibdm building
> 
> Hi Eitan,
> 
> With your latest ibdm changes:
> D  src/linux-user/ibdm/ibnl/Makefile.in
> U  src/linux-user/ibdm/src/test_topo_match.cpp
> D  src/linux-user/ibdm/src/Makefile.in
> U  src/linux-user/ibdm/datamodel/Makefile.am
> D  src/linux-user/ibdm/datamodel/Makefile.in
> U  src/linux-user/ibdm/Clusters/Makefile.am
> D  src/linux-user/ibdm/Clusters/Makefile.in
> D  src/linux-user/ibdm/config/share/libtool/libltdl/Makefile.in
> D  src/linux-user/ibdm/replace/Makefile.in
> D  src/linux-user/ibdm/Makefile.in
> 
> I now get the following when building ibdm:
> 
> ./configure
> ...
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: error: cannot find input file: Makefile.in
> 
> -- Hal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/40e2e728/attachment.html>

From eitan at mellanox.co.il  Tue Jul  5 08:32:40 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 18:32:40 +0300 
Subject: [openib-general] RE: IBDM - IB DataModel build issues fixed
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30429@mtlex01.yok.mtl.com>

> It appears the following are the executables:
> ibdmchk, ibdmtr, topotest, ibnlparse
[EZ] Correct
> but the README references the following:
> fabric_sim, fabric_tr, and osm_check.
[EZ] Fixed
> 
> NEWS indicates rev 1.1 was 12/4/03. What version is this ? Also, this
> installs to .../ibadm/ibdm1.0/
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/4f4c08c0/attachment.html>

From rolandd at cisco.com  Tue Jul  5 08:49:01 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 08:49:01 -0700
Subject: [openib-general] [PATCH] mthca: updates to copyright
In-Reply-To: <20050705125312.GR14130@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 5 Jul 2005 15:53:12 +0300")
References: <ORSMSX401GryWtIIZS200000015@orsmsx401.amr.corp.intel.com>
	<20050704184038.GA16650@mellanox.co.il>
	<20050705125312.GR14130@mellanox.co.il>
Message-ID: <52pstxuuc2.fsf@topspin.com>

Thanks, applied.

 - R.


From krause at cup.hp.com  Tue Jul  5 09:35:32 2005
From: krause at cup.hp.com (Michael Krause)
Date: Tue, 05 Jul 2005 09:35:32 -0700
Subject: [openib-general] Re: IP addressing on InfiniBand networks
	(Caitlin Bestler)
In-Reply-To: <52r7ej68ga.fsf@topspin.com>
References: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com>
	<52r7ej68ga.fsf@topspin.com>
Message-ID: <6.2.0.14.2.20050705093014.0291a9d0@esmail.cup.hp.com>

At 10:49 AM 6/30/2005, Roland Dreier wrote:
>     Michael> Being the person who led the addressing definition for
>     Michael> IB, I can state quite clearly that GID are NOT IPv6
>     Michael> addresses.  They were intentionally defined to have a
>     Michael> similar look-n-feel since they were derived in large part
>     Michael> from Future I/O which had them as real IPv6 addresses.
>     Michael> But again, they are NOT IPv6 addresses.
>
>The IBA spec seems to have a different idea.  In fact chapter 4 says:
>
>     "A GID is a valid 128-bit IPv6 address (per RFC 2373)...."

I wrote the original spec here.  The text was supposed to be updated to 
clarify that the rest of the sentence, i.e. with additional rules, etc. 
thus making it not a real IPv6 address from the IETF's perspective but 
something quite close.  The intention was to allow one to manage the fabric 
by having mapping functions from traditional IP management applications to 
IB GID to minimize the amount of work to enable IB within a solution.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/2a1f88db/attachment.html>

From krause at cup.hp.com  Tue Jul  5 09:28:11 2005
From: krause at cup.hp.com (Michael Krause)
Date: Tue, 05 Jul 2005 09:28:11 -0700
Subject: [openib-general] [iser]about the target
In-Reply-To: <BAY22-F2C066127BE3FC92439269CAE60@phx.gbl>
References: <BAY22-F2C066127BE3FC92439269CAE60@phx.gbl>
Message-ID: <6.2.0.14.2.20050705092609.0291ada8@esmail.cup.hp.com>

At 06:07 PM 7/4/2005, Ian Jiang wrote:
>Hi!
>I am new to the iSER.
>On "https://openib.org/tiki/tiki-index.php?page=iSER", it is said that 
>iSER currently contains initiator only (no target). Will the target come 
>out later? How did they test the iSER initiator without a iSER target?
>Could you give some explaination?

 From a practical perspective, there are very few iSCSI targets shipping 
today.  Most people had envisioned iSER over IB to a gateway Ethernet 
device since native IB storage is also quite rare in terms of real 
product.  For many of us, our push for iSER over IB was to replace SRP 
which has a deficient ecosystem thus not really used beyond some basic 
Fibre Channel gateway cards.

Mike

>Thanks!
>
>
>
>Ian Jiang
>ianjiang91 at hotmail.com
>----
>Computer Architecture Laboratory
>Institute of Computing Technology
>Chinese Academy of Sciences
>Beijing,P.R.China
>Zip code: 100080
>Tel: +86-10-62564394(office)
>
>_________________________________________________________________
>ÓëÁª»úµÄÅóÓѽøÐн»Á÷£¬ÇëʹÓà MSN Messenger:  http://messenger.msn.com/cn
>
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/0d7f815a/attachment.html>

From halr at voltaire.com  Tue Jul  5 10:16:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 13:16:31 -0400
Subject: [openib-general] IBMgtSim building
Message-ID: <1120583522.4388.9488.camel@hal.voltaire.com>

Hi again Eitan,

Should ./config/bootstrap be run in IBMgtSim ?

This is what I see:
./config/bootstrap
/home/hal/openib/utils/src/linux-user/IBMgtSim/config/libtoolize: line 1: cd: /home/eitan/SW/autotools/share/libtool: No such file or directory
libtoolize: cannot list files in `/home/eitan/SW/autotools/share/libtool'
automake: src/Makefile.am: object `client.$(OBJEXT)' created both with libtool and without
automake: src/Makefile.am: object `msgmgr.$(OBJEXT)' created both with libtool and without
automake: src/Makefile.am: object `simmsg.$(OBJEXT)' created both with libtool and without
automake: src/Makefile.am: object `tcpcomm.$(OBJEXT)' created both with libtool and without

-- Hal



From halr at voltaire.com  Tue Jul  5 11:08:41 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 14:08:41 -0400
Subject: [openib-general] OpenIB Releases
Message-ID: <1120586921.4388.9880.camel@hal.voltaire.com>

Hi,

Some people have been asking for a snapshot for testing and other
purposes. It is getting to be time for an initial OpenIB release.
Below is a a strawman to start the dialog on this.

Each component should be released separately (at least initially)
indicating any dependencies. These releases are made by the maintainer.
An initial list of components (and their maintainers) is below.

2 digit release numbering should be sufficient. The initial release will
be numbered 1.0. Similar to the kernel, these releases will go through
release candidate phases (rc1, rc2, ...) until the maintainer deems it
final (1.0-rc1, 1.0-rc2, ..., final).

Should these releases be copies of the component tree or in the form of
a tarball ?

Once all the components are released, testing of running multiple
concurrent ULPs should occur. At some point after that, the OpenIB
release would be declared final. Final would be reserved for this.

[One note is that in terms of svn, labeling is essentially the same as a
branch with perhaps an additional rule: that no development (change)
occur on that branch.]

Initial List of components for inclusion (some in different states of
"bakedness")

Kernel
	mthca (Roland)
	core (Roland, Sean, Hal)
	ULPs
		IPoIB (Roland)
		SDP (Libor)
		kDAPL (James)

User Space
	libibverbs & libmthca (Roland)
	libibcm (Libor)
	management (OpenSM & diagnostics) (Hal)
	libsdp (Michael Tsirkin)
	tvflash (Roland)
	mstflint (Michael)
	perftest (Michael)
	examples/aio (Libor)
	uDAPL (James)
	MPI ?

Key to all this is testers picking these releases up and testing and
reporting back to the list.

Comments ?

-- Hal



From eitan at mellanox.co.il  Tue Jul  5 11:43:24 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 21:43:24 +0300 
Subject: [openib-general] RE: IBMgtSim building
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3042A@mtlex01.yok.mtl.com>


> 
> Should ./config/bootstrap be run in IBMgtSim ?
[EZ] Yes
> 
> This is what I see:
> ./config/bootstrap
> /home/hal/openib/utils/src/linux-user/IBMgtSim/config/libtoolize: line 1:
cd:
> /home/eitan/SW/autotools/share/libtool: No such file or directory
[EZ] Was using a local installation of libtool.
I have checked in the fix.
Please make sure libtool 1.4.2 or later is installed on the machine.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/86e2a4ae/attachment.html>

From halr at voltaire.com  Tue Jul  5 11:43:35 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 14:43:35 -0400
Subject: [openib-general] RE: ibdm building
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30428@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30428@mtlex01.yok.mtl.com>
Message-ID: <1120589014.4388.10086.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 11:32, Eitan Zahavi wrote:
> Makefile.in is normally only distributed at release time.
> The code source tree does not normally hold Makefile.in as they are
> generated by autoconf. 
> You need to run ./config/bootstrap to regenerate them...

What about:
/gen2/utils/src/linux-user/IBMgtSim 
  * Makefile.in
  * src/Makefile.in 
  * utils/Makefile.in

-- Hal

> 
> EZ
> 
> Eitan Zahavi
> Design Technology Director
> Mellanox Technologies LTD
> Tel:+972-4-9097208
> Fax:+972-4-9593245
> P.O. Box 586 Yokneam 20692 ISRAEL
> 
> 
> > -----Original Message-----
> > From: Hal Rosenstock [mailto:halr at voltaire.com]
> > Sent: Tuesday, July 05, 2005 6:20 PM
> > To: Eitan Zahavi
> > Cc: openib-general at openib.org
> > Subject: ibdm building
> > 
> > Hi Eitan,
> > 
> > With your latest ibdm changes:
> > D  src/linux-user/ibdm/ibnl/Makefile.in
> > U  src/linux-user/ibdm/src/test_topo_match.cpp
> > D  src/linux-user/ibdm/src/Makefile.in
> > U  src/linux-user/ibdm/datamodel/Makefile.am
> > D  src/linux-user/ibdm/datamodel/Makefile.in
> > U  src/linux-user/ibdm/Clusters/Makefile.am
> > D  src/linux-user/ibdm/Clusters/Makefile.in
> > D  src/linux-user/ibdm/config/share/libtool/libltdl/Makefile.in
> > D  src/linux-user/ibdm/replace/Makefile.in
> > D  src/linux-user/ibdm/Makefile.in
> > 
> > I now get the following when building ibdm:
> > 
> > ./configure
> > ...
> > configure: creating ./config.status
> > config.status: creating Makefile
> > config.status: error: cannot find input file: Makefile.in
> > 
> > -- Hal
> 



From rolandd at cisco.com  Tue Jul  5 12:05:31 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 12:05:31 -0700
Subject: [openib-general] OpenIB Releases
In-Reply-To: <1120586921.4388.9880.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "05 Jul 2005 14:08:41 -0400")
References: <1120586921.4388.9880.camel@hal.voltaire.com>
Message-ID: <524qb9ul8k.fsf@topspin.com>

    Hal> Comments ?

I'm not sure it makes sense to release kernel components.  We can
probably let Andrew/Linus handle the kernel release process.

I'd prefer to deprecate tvflash and let mstflint be the main Mellanox
flash burning tool.

 - R.


From eitan at mellanox.co.il  Tue Jul  5 12:13:07 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 5 Jul 2005 22:13:07 +0300 
Subject: [openib-general] RE: ibdm building
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3042B@mtlex01.yok.mtl.com>


> What about:
> /gen2/utils/src/linux-user/IBMgtSim
>   * Makefile.in
>   * src/Makefile.in
>   * utils/Makefile.in
[EZ] All gone now

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050705/dd3748bc/attachment.html>

From halr at voltaire.com  Tue Jul  5 12:09:47 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 15:09:47 -0400
Subject: [openib-general] OpenIB Releases
In-Reply-To: <524qb9ul8k.fsf@topspin.com>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com>
Message-ID: <1120590587.4388.10100.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 15:05, Roland Dreier wrote:
>     Hal> Comments ?
> 
> I'm not sure it makes sense to release kernel components.  We can
> probably let Andrew/Linus handle the kernel release process.

But the kernel is behind the OpenIB trunk.

> I'd prefer to deprecate tvflash and let mstflint be the main Mellanox
> flash burning tool.

-- Hal



From mst at mellanox.co.il  Tue Jul  5 12:19:10 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 22:19:10 +0300
Subject: [openib-general] Re: OpenIB Releases
In-Reply-To: <524qb9ul8k.fsf@topspin.com>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com>
Message-ID: <20050705191910.GA28064@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: OpenIB Releases
> 
>     Hal> Comments ?
> 
> I'm not sure it makes sense to release kernel components.  We can
> probably let Andrew/Linus handle the kernel release process.
> 
> I'd prefer to deprecate tvflash and let mstflint be the main Mellanox
> flash burning tool.
> 
>  - R.

Are there features in tvflash that mstflint doesnt have?

-- 
MST


From rolandd at cisco.com  Tue Jul  5 12:20:58 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 12:20:58 -0700
Subject: [openib-general] Re: [PATCH 11/16] IB uverbs: add mthca mmap support
In-Reply-To: <20050628170553.00a14a29.akpm@osdl.org> (Andrew Morton's
	message of "Tue, 28 Jun 2005 17:05:53 -0700")
References: <2005628163.o84QGfsM7oMSy0oU@cisco.com>
	<2005628163.gtJFW6uLUrGQteys@cisco.com>
	<20050628170553.00a14a29.akpm@osdl.org>
Message-ID: <52mzp1oy91.fsf@topspin.com>

    Andrew> What's the thinking behind the VM_DONTCOPY there?

As I said before, I don't think the thinking behind VM_DONTCOPY was
correct thinking.  Let's take it out.

I've now answered all your questions on this patchset (or at least
written something in response to all your questions ;).  What's your
feeling on merging?  If more info is required, just let me know.  I'll
also be at the kernel summit in a couple of weeks if you want to go
over it in person.

 - R.


There's no need to set VM_DONTCOPY when mmap()ing the hardware
doorbell page into userspace.

Signed-off-by: Roland Dreier <rolandd at cisco.com>


--- linux-export.orig/drivers/infiniband/hw/mthca/mthca_provider.c	2005-06-28 15:20:28.000000000 -0700
+++ linux-export/drivers/infiniband/hw/mthca/mthca_provider.c	2005-07-05 12:14:20.838664330 -0700
@@ -347,7 +347,6 @@
 		return -EINVAL;
 
 	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
-	vma->vm_flags    |= VM_DONTCOPY;
 
 	if (remap_pfn_range(vma, vma->vm_start,
 			    to_mucontext(context)->uar.pfn,


From rep.nop at aon.at  Tue Jul  5 12:22:43 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Tue, 5 Jul 2005 21:22:43 +0200
Subject: [openib-general] OpenIB Releases
In-Reply-To: <1120586921.4388.9880.camel@hal.voltaire.com>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
Message-ID: <20050705192243.GP5997@aon.at>

On Tue, Jul 05, 2005 at 02:08:41PM -0400, Hal Rosenstock wrote:
>Hi,
[]
>User Space
[]
>	MPI ?
I started to port LAM/MPI to gen2's verbs but due to lack of HW and
time it is still not completely finished (RTR->RTS state handling
has a bug).

I did choose LAM because it's available publically and right now, as
opposed to other eventually forthcoming solutions.

I, personally, would be very happy if there finally was a working
MPI implementation perusing gen2 (which can be obtained without filling
in some odd HTML form).

>Key to all this is testers picking these releases up and testing and
>reporting back to the list.
>
>Comments ?
>
>-- Hal
Bernhard


From johannes at erdfelt.com  Tue Jul  5 12:25:18 2005
From: johannes at erdfelt.com (Johannes Erdfelt)
Date: Tue, 5 Jul 2005 12:25:18 -0700
Subject: [openib-general] Re: OpenIB Releases
In-Reply-To: <20050705191910.GA28064@mellanox.co.il>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com>
	<20050705191910.GA28064@mellanox.co.il>
Message-ID: <20050705192518.GO13776@sventech.com>

On Tue, Jul 05, 2005, Michael S. Tsirkin <mst at mellanox.co.il> wrote:
> Quoting r. Roland Dreier <rolandd at cisco.com>:
> > Subject: Re: OpenIB Releases
> > 
> >     Hal> Comments ?
> > 
> > I'm not sure it makes sense to release kernel components.  We can
> > probably let Andrew/Linus handle the kernel release process.
> > 
> > I'd prefer to deprecate tvflash and let mstflint be the main Mellanox
> > flash burning tool.
> 
> Are there features in tvflash that mstflint doesnt have?

They are all Topspin specific, but mostly for handling things like
automatically upgrading firmware and configuring the boot ROM extension.

JE



From rolandd at cisco.com  Tue Jul  5 12:28:19 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 12:28:19 -0700
Subject: [openib-general] Re: OpenIB Releases
In-Reply-To: <20050705191910.GA28064@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 5 Jul 2005 22:19:10 +0300")
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com> <20050705191910.GA28064@mellanox.co.il>
Message-ID: <52d5pxoxws.fsf@topspin.com>

    Michael> Are there features in tvflash that mstflint doesnt have?

tvflash handles Topspin formatted vendor specific data.  I'm not sure
how important that is.

 - R.


From rolandd at cisco.com  Tue Jul  5 12:34:17 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 12:34:17 -0700
Subject: [openib-general] [PATCH] sa_query: Add service record support
In-Reply-To: <1120135681.4848.5971.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "30 Jun 2005 08:48:02 -0400")
References: <1120135681.4848.5971.camel@hal.voltaire.com>
Message-ID: <528y0loxmu.fsf@topspin.com>

    > +#define IB_DEFAULT_SERVICE_LEASE 	0xFFFFFFFF

This doesn't seem to be used anywhere?

    > +	/* u16 	resv;	*/

Can we just delete this comment?

    > +	query->sa_query.callback              = ib_sa_service_rec_callback;

This should probably handle a consumer passing in a NULL callback like
this other query functions do.

 - R.


From halr at voltaire.com  Tue Jul  5 12:28:16 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 15:28:16 -0400
Subject: [openib-general] OpenIB Releases
In-Reply-To: <20050705192243.GP5997@aon.at>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<20050705192243.GP5997@aon.at>
Message-ID: <1120591696.4388.10126.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 15:22, Bernhard Fischer wrote:
> On Tue, Jul 05, 2005 at 02:08:41PM -0400, Hal Rosenstock wrote:
> >Hi,
> []
> >User Space
> []
> >	MPI ?
> I started to port LAM/MPI to gen2's verbs but due to lack of HW and
> time it is still not completely finished

How far along is it ? Does it compile ? Does it link ?

> RTR->RTS state handling has a bug).

Can you elaborate on this bug ?

-- Hal

> I did choose LAM because it's available publically and right now, as
> opposed to other eventually forthcoming solutions.
> 
> I, personally, would be very happy if there finally was a working
> MPI implementation perusing gen2 (which can be obtained without filling
> in some odd HTML form).
> 
> >Key to all this is testers picking these releases up and testing and
> >reporting back to the list.
> >
> >Comments ?
> >
> >-- Hal
> Bernhard



From panda at cse.ohio-state.edu  Tue Jul  5 12:49:12 2005
From: panda at cse.ohio-state.edu (Dhabaleswar Panda)
Date: Tue, 5 Jul 2005 15:49:12 -0400 (EDT)
Subject: [openib-general] OpenIB Releases
In-Reply-To: <1120586921.4388.9880.camel@hal.voltaire.com> from "Hal
	Rosenstock" at Jul 05, 2005 02:08:41 PM
Message-ID: <200507051949.j65JnCQf027336@xi.cse.ohio-state.edu>

> User Space
> 	MPI ?

OSU is working on an initial port of MVAPICH on Gen2. A working
version will be released in the near future.

DK



From halr at voltaire.com  Tue Jul  5 13:14:07 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 16:14:07 -0400
Subject: [openib-general] [PATCH] sa_query: Add service record support
In-Reply-To: <528y0loxmu.fsf@topspin.com>
References: <1120135681.4848.5971.camel@hal.voltaire.com>
	<528y0loxmu.fsf@topspin.com>
Message-ID: <1120594447.4388.10242.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 15:34, Roland Dreier wrote:
>     > +#define IB_DEFAULT_SERVICE_LEASE 	0xFFFFFFFF
> 
> This doesn't seem to be used anywhere?

Right. It is a convenient define for a consumer using Service Records. I
would prefer to leave it in there despite it not being used.

>     > +	/* u16 	resv;	*/
> 
> Can we just delete this comment?

It does map to the actual SR layout and path record indicates where the
reserved fields are.

Is changing it to /* reserved */ acceptable like in the ib_sa_path_rec
struct declaration ?

>     > +	query->sa_query.callback              = ib_sa_service_rec_callback;
> 
> This should probably handle a consumer passing in a NULL callback like
> this other query functions do.

Good catch. I'll update and reissue the patch. Thanks for reviewing it.

BTW, don't forget that the user_mad patch is also pending. Thanks.

-- Hal



From halr at voltaire.com  Tue Jul  5 13:19:32 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 16:19:32 -0400
Subject: [openib-general] [PATCH v3] sa_query: Add service record support
Message-ID: <1120594769.4388.10253.camel@hal.voltaire.com>

sa_query: Add service record support

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: include/ib_sa.h
===================================================================
--- include/ib_sa.h	(revision 2771)
+++ include/ib_sa.h	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -193,6 +194,61 @@
 	int          proxy_join;
 };
 
+/* Service Record Component Mask Sec 15.2.5.14 Ver 1.1	*/
+#define IB_SR_COMPMASK_SID				IB_SA_COMP_MASK(0)
+#define IB_SR_COMPMASK_SGID				IB_SA_COMP_MASK(1)
+#define IB_SR_COMPMASK_SPKEY				IB_SA_COMP_MASK(2)
+#define IB_SR_COMPMASK_RES1				IB_SA_COMP_MASK(3)
+#define IB_SR_COMPMASK_SLEASE				IB_SA_COMP_MASK(4)
+#define IB_SR_COMPMASK_SKEY				IB_SA_COMP_MASK(5)
+#define IB_SR_COMPMASK_SNAME				IB_SA_COMP_MASK(6)
+#define IB_SR_COMPMASK_SDATA8_0				IB_SA_COMP_MASK(7)
+#define IB_SR_COMPMASK_SDATA8_1				IB_SA_COMP_MASK(8)
+#define IB_SR_COMPMASK_SDATA8_2				IB_SA_COMP_MASK(9)
+#define IB_SR_COMPMASK_SDATA8_3				IB_SA_COMP_MASK(10)
+#define IB_SR_COMPMASK_SDATA8_4				IB_SA_COMP_MASK(11)
+#define IB_SR_COMPMASK_SDATA8_5				IB_SA_COMP_MASK(12)
+#define IB_SR_COMPMASK_SDATA8_6				IB_SA_COMP_MASK(13)
+#define IB_SR_COMPMASK_SDATA8_7				IB_SA_COMP_MASK(14)
+#define IB_SR_COMPMASK_SDATA8_8				IB_SA_COMP_MASK(15)
+#define IB_SR_COMPMASK_SDATA8_9				IB_SA_COMP_MASK(16)
+#define IB_SR_COMPMASK_SDATA8_10			IB_SA_COMP_MASK(17)
+#define IB_SR_COMPMASK_SDATA8_11			IB_SA_COMP_MASK(18)
+#define IB_SR_COMPMASK_SDATA8_12			IB_SA_COMP_MASK(19)
+#define IB_SR_COMPMASK_SDATA8_13			IB_SA_COMP_MASK(20)
+#define IB_SR_COMPMASK_SDATA8_14			IB_SA_COMP_MASK(21)
+#define IB_SR_COMPMASK_SDATA8_15			IB_SA_COMP_MASK(22)
+#define IB_SR_COMPMASK_SDATA16_0			IB_SA_COMP_MASK(23)
+#define IB_SR_COMPMASK_SDATA16_1			IB_SA_COMP_MASK(24)
+#define IB_SR_COMPMASK_SDATA16_2			IB_SA_COMP_MASK(25)
+#define IB_SR_COMPMASK_SDATA16_3			IB_SA_COMP_MASK(26)
+#define IB_SR_COMPMASK_SDATA16_4			IB_SA_COMP_MASK(27)
+#define IB_SR_COMPMASK_SDATA16_5			IB_SA_COMP_MASK(28)
+#define IB_SR_COMPMASK_SDATA16_6			IB_SA_COMP_MASK(29)
+#define IB_SR_COMPMASK_SDATA16_7			IB_SA_COMP_MASK(30)
+#define IB_SR_COMPMASK_SDATA32_0			IB_SA_COMP_MASK(31)
+#define IB_SR_COMPMASK_SDATA32_1			IB_SA_COMP_MASK(32)
+#define IB_SR_COMPMASK_SDATA32_2			IB_SA_COMP_MASK(33)
+#define IB_SR_COMPMASK_SDATA32_3			IB_SA_COMP_MASK(34)
+#define IB_SR_COMPMASK_SDATA64_0			IB_SA_COMP_MASK(35)
+#define IB_SR_COMPMASK_SDATA64_1			IB_SA_COMP_MASK(36)
+
+#define IB_DEFAULT_SERVICE_LEASE 	0xFFFFFFFF
+
+struct ib_sa_service_rec {
+	u64		id;
+	union ib_gid	gid;		
+	u16 		pkey;
+	/* reserved */
+	u32		lease;
+	u8		key[16];
+	u8		name[64];
+	u8		data8[16];
+	u16		data16[8];
+	u32		data32[4];
+	u64		data64[2];
+};
+
 struct ib_sa_query;
 
 void ib_sa_cancel_query(int id, struct ib_sa_query *query);
@@ -218,6 +274,17 @@
 			     void *context,
 			     struct ib_sa_query **query);
 
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num,
+			 u8 method,
+			 struct ib_sa_service_rec *rec,
+			 ib_sa_comp_mask comp_mask,
+			 int timeout_ms, int gfp_mask,
+			 void (*callback)(int status,
+					  struct ib_sa_service_rec *resp,
+					  void *context),
+			 void *context,
+			 struct ib_sa_query **sa_query);
+
 /**
  * ib_sa_mcmember_rec_set - Start an MCMember set query
  * @device:device to send query on
Index: core/sa_query.c
===================================================================
--- core/sa_query.c	(revision 2771)
+++ core/sa_query.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -79,6 +80,12 @@
 	int                 id;
 };
 
+struct ib_sa_service_query {
+	void (*callback)(int, struct ib_sa_service_rec *, void *);
+	void *context;
+	struct ib_sa_query sa_query;
+};
+
 struct ib_sa_path_query {
 	void (*callback)(int, struct ib_sa_path_rec *, void *);
 	void *context;
@@ -320,6 +327,54 @@
 	  .size_bits    = 23 },
 };
 
+#define SERVICE_REC_FIELD(field) \
+	.struct_offset_bytes = offsetof(struct ib_sa_service_rec, field),	\
+	.struct_size_bytes   = sizeof ((struct ib_sa_service_rec *) 0)->field,	\
+	.field_name          = "sa_service_rec:" #field
+
+static const struct ib_field service_rec_table[] = {
+	{ SERVICE_REC_FIELD(id),
+	  .offset_words = 0,
+	  .offset_bits  = 0,
+	  .size_bits    = 64 },
+	{ SERVICE_REC_FIELD(gid),
+	  .offset_words = 2,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(pkey),
+	  .offset_words = 6,
+	  .offset_bits  = 0,
+	  .size_bits    = 16 },
+	{ SERVICE_REC_FIELD(lease),
+	  .offset_words = 7,
+	  .offset_bits  = 0,
+	  .size_bits    = 32 },
+	{ SERVICE_REC_FIELD(key),
+	  .offset_words = 8,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(name),
+	  .offset_words = 12,
+	  .offset_bits  = 0,
+	  .size_bits    = 64*8 },
+	{ SERVICE_REC_FIELD(data8),
+	  .offset_words = 28,
+	  .offset_bits  = 0,
+	  .size_bits    = 16*8 },
+	{ SERVICE_REC_FIELD(data16),
+	  .offset_words = 32,
+	  .offset_bits  = 0,
+	  .size_bits    = 8*16 },
+	{ SERVICE_REC_FIELD(data32),
+	  .offset_words = 36,
+	  .offset_bits  = 0,
+	  .size_bits    = 4*32 },
+	{ SERVICE_REC_FIELD(data64),
+	  .offset_words = 40,
+	  .offset_bits  = 0,
+	  .size_bits    = 2*64 },
+};
+
 static void free_sm_ah(struct kref *kref)
 {
 	struct ib_sa_sm_ah *sm_ah = container_of(kref, struct ib_sa_sm_ah, ref);
@@ -443,7 +498,6 @@
 				 .remote_qpn  = 1,
 				 .remote_qkey = IB_QP1_QKEY,
 				 .timeout_ms  = timeout_ms,
-				 .retries     = 0 
 			 }
 		 }
 	};
@@ -596,6 +650,114 @@
 }
 EXPORT_SYMBOL(ib_sa_path_rec_get);
 
+static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query,
+				    int status,
+				    struct ib_sa_mad *mad)
+{
+	struct ib_sa_service_query *query =
+		container_of(sa_query, struct ib_sa_service_query, sa_query);
+
+	if (mad) {
+		struct ib_sa_service_rec rec;
+
+		ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table),
+			  mad->data, &rec);
+		query->callback(status, &rec, query->context);
+	} else
+		query->callback(status, NULL, query->context);
+}
+
+static void ib_sa_service_rec_release(struct ib_sa_query *sa_query)
+{
+	kfree(sa_query->mad);
+	kfree(container_of(sa_query, struct ib_sa_service_query, sa_query));
+}
+
+/**
+ * ib_sa_service_rec_query - Start Service Record operation
+ * @device:device to send request on
+ * @port_num: port number to send request on
+ * @method:SA method - should be get, set, or delete
+ * @rec:Service Record to send in request
+ * @comp_mask:component mask to send in request
+ * @timeout_ms:time to wait for response
+ * @gfp_mask:GFP mask to use for internal allocations
+ * @callback:function called when request completes, times out or is
+ * canceled
+ * @context:opaque user context passed to callback
+ * @sa_query:request context, used to cancel request
+ *
+ * Send a Service Record set/get/delete to the SA to register,
+ * unregister or query a service record.
+ * The callback function will be called when the request completes (or
+ * fails); status is 0 for a successful response, -EINTR if the query
+ * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error
+ * occurred sending the query.  The resp parameter of the callback is
+ * only valid if status is 0.
+ *
+ * If the return value of ib_sa_service_rec_query() is negative, it is an
+ * error code.  Otherwise it is a request ID that can be used to cancel
+ * the query.
+ */
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
+			    struct ib_sa_service_rec *rec,
+			    ib_sa_comp_mask comp_mask,
+			    int timeout_ms, int gfp_mask,
+			    void (*callback)(int status,
+					     struct ib_sa_service_rec *resp,
+					     void *context),
+			    void *context,
+			    struct ib_sa_query **sa_query)
+{
+	struct ib_sa_service_query *query;
+	struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client);
+	struct ib_sa_port   *port   = &sa_dev->port[port_num - sa_dev->start_port];
+	struct ib_mad_agent *agent  = port->agent;
+	int ret;
+
+	if (method != IB_MGMT_METHOD_GET &&
+	    method != IB_MGMT_METHOD_SET &&
+	    method != IB_SA_METHOD_DELETE)
+		return -EINVAL;
+
+	query = kmalloc(sizeof *query, gfp_mask);
+	if (!query)
+		return -ENOMEM;
+	query->sa_query.mad = kmalloc(sizeof *query->sa_query.mad, gfp_mask);
+	if (!query->sa_query.mad) {
+		kfree(query);
+		return -ENOMEM;
+	}
+
+	query->callback = callback;
+	query->context  = context;
+
+	init_mad(query->sa_query.mad, agent);
+
+	query->sa_query.callback              = callback ? ib_sa_service_rec_callback : NULL;
+	query->sa_query.release               = ib_sa_service_rec_release;
+	query->sa_query.port                  = port;
+	query->sa_query.mad->mad_hdr.method   = method;
+	query->sa_query.mad->mad_hdr.attr_id  =
+				cpu_to_be16(IB_SA_ATTR_SERVICE_REC);
+	query->sa_query.mad->sa_hdr.comp_mask = comp_mask;
+
+	ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table),
+		rec, query->sa_query.mad->data);
+
+	*sa_query = &query->sa_query;
+
+	ret = send_mad(&query->sa_query, timeout_ms);
+	if (ret < 0) {
+		*sa_query = NULL;
+		kfree(query->sa_query.mad);
+		kfree(query);
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(ib_sa_service_rec_query);
+
 static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query,
 					int status,
 					struct ib_sa_mad *mad)





From rolandd at cisco.com  Tue Jul  5 13:33:54 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 13:33:54 -0700
Subject: [openib-general] [PATCH] sa_query: Add service record support
In-Reply-To: <1120594447.4388.10242.camel@hal.voltaire.com> (Hal
	Rosenstock's message of "05 Jul 2005 16:14:07 -0400")
References: <1120135681.4848.5971.camel@hal.voltaire.com>
	<528y0loxmu.fsf@topspin.com>
	<1120594447.4388.10242.camel@hal.voltaire.com>
Message-ID: <52d5pxngb1.fsf@topspin.com>

    Hal> Right. It is a convenient define for a consumer using Service
    Hal> Records. I would prefer to leave it in there despite it not
    Hal> being used.

OK, that's fine.

    Hal> It does map to the actual SR layout and path record indicates
    Hal> where the reserved fields are.

    Hal> Is changing it to /* reserved */ acceptable like in the
    Hal> ib_sa_path_rec struct declaration ?

Actually, I didn't read the patch carefully enough.  Yes, please
follow the style used in the rest of the file: name the component mask
entries as they are named in the spec, rather than using your own
abbreviations, so you have something like

#define IB_SA_SERVICE_REC_SERVICE_ID			IB_SA_COMP_MASK( 0)
#define IB_SA_SERVICE_REC_SERVICE_GID			IB_SA_COMP_MASK( 1)
#define IB_SA_SERVICE_REC_SERVICE_PKEY			IB_SA_COMP_MASK( 2)
/* reserved:								 3 */
#define IB_SA_SERVICE_REC_SERVICE_LEASE			IB_SA_COMP_MASK( 4)

and so on.  And there's no need to define a component mask for reserved rows.

Then you can put 

	/* reserved */

in the structure definition to match the component mask.

    Hal> BTW, don't forget that the user_mad patch is also
    Hal> pending. Thanks.

Thanks for the reminder.  I'll give it a final review today.

 - R.


From mst at mellanox.co.il  Tue Jul  5 13:53:51 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 23:53:51 +0300
Subject: [openib-general] Re: [PATCH 11/16] IB uverbs: add mthca mmap support
In-Reply-To: <52mzp1oy91.fsf@topspin.com>
References: <2005628163.o84QGfsM7oMSy0oU@cisco.com>
	<2005628163.gtJFW6uLUrGQteys@cisco.com>
	<20050628170553.00a14a29.akpm@osdl.org>
	<52mzp1oy91.fsf@topspin.com>
Message-ID: <20050705205351.GB28064@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [PATCH 11/16] IB uverbs: add mthca mmap support
> 
>     Andrew> What's the thinking behind the VM_DONTCOPY there?
> 
> As I said before, I don't think the thinking behind VM_DONTCOPY was
> correct thinking.  Let's take it out.

Roland, I think VM_DONTCOPY is needed here.

If a process forks, we must prevent the child from accessing
the parent's hardware page. Otherwise the child can corrupt
the parent's queues since the hardware wont be able to distinguish
between parent and child.

Does this make sense?

-- 
MST


From mst at mellanox.co.il  Tue Jul  5 13:57:35 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 5 Jul 2005 23:57:35 +0300
Subject: [openib-general] Re: Re: OpenIB Releases
In-Reply-To: <20050705192518.GO13776@sventech.com>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com>
	<20050705191910.GA28064@mellanox.co.il>
	<20050705192518.GO13776@sventech.com>
Message-ID: <20050705205735.GC28064@mellanox.co.il>

Quoting r. Johannes Erdfelt <johannes at erdfelt.com>:
> > Are there features in tvflash that mstflint doesnt have?
> 
> They are all Topspin specific, but mostly for handling things like
> automatically upgrading firmware and configuring the boot ROM extension.

How do you mean?

-- 
MST


From rep.nop at aon.at  Tue Jul  5 14:02:39 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Tue, 5 Jul 2005 23:02:39 +0200
Subject: [openib-general] OpenIB Releases
In-Reply-To: <1120591696.4388.10126.camel@hal.voltaire.com>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<20050705192243.GP5997@aon.at>
	<1120591696.4388.10126.camel@hal.voltaire.com>
Message-ID: <20050705210239.GQ5997@aon.at>

On Tue, Jul 05, 2005 at 03:28:16PM -0400, Hal Rosenstock wrote:
>On Tue, 2005-07-05 at 15:22, Bernhard Fischer wrote:
>> On Tue, Jul 05, 2005 at 02:08:41PM -0400, Hal Rosenstock wrote:
>> >Hi,
>> []
>> >User Space
>> []
>> >	MPI ?
>> I started to port LAM/MPI to gen2's verbs but due to lack of HW and
>> time it is still not completely finished
>
>How far along is it ? Does it compile ? Does it link ?

It compiles and links, fwiw.
>
>> RTR->RTS state handling has a bug).
>
>Can you elaborate on this bug ?

It's a pure bug in my port, not openib. IIRC it overflows when setting
wr_id, but i didn't track it down.

>> I, personally, would be very happy if there finally was a working
>> MPI implementation perusing gen2 (which can be obtained without filling
>> in some odd HTML form).



From johannes at erdfelt.com  Tue Jul  5 14:03:38 2005
From: johannes at erdfelt.com (Johannes Erdfelt)
Date: Tue, 5 Jul 2005 14:03:38 -0700
Subject: [openib-general] Re: Re: OpenIB Releases
In-Reply-To: <20050705205735.GC28064@mellanox.co.il>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com>
	<20050705191910.GA28064@mellanox.co.il>
	<20050705192518.GO13776@sventech.com>
	<20050705205735.GC28064@mellanox.co.il>
Message-ID: <20050705210338.GQ13776@sventech.com>

On Tue, Jul 05, 2005, Michael S. Tsirkin <mst at mellanox.co.il> wrote:
> Quoting r. Johannes Erdfelt <johannes at erdfelt.com>:
> > > Are there features in tvflash that mstflint doesnt have?
> > 
> > They are all Topspin specific, but mostly for handling things like
> > automatically upgrading firmware and configuring the boot ROM extension.
> 
> How do you mean?

The VSD area is highly customized for our purposes. We store information
about the firmware image programmed so we can automatically upgrade the
firmware when upgrading the IB software on the system.

We also use the VSD to store configuration options for our boot ROM
extension.

Other than the VSD and some minor supporting features, there isn't
anything special about tvflash versus mstflint.

JE



From halr at voltaire.com  Tue Jul  5 14:02:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 17:02:05 -0400
Subject: [openib-general] [PATCH] sa_query: Add service record support
In-Reply-To: <52d5pxngb1.fsf@topspin.com>
References: <1120135681.4848.5971.camel@hal.voltaire.com>
	<528y0loxmu.fsf@topspin.com>
	<1120594447.4388.10242.camel@hal.voltaire.com>
	<52d5pxngb1.fsf@topspin.com>
Message-ID: <1120597324.4388.10361.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 16:33, Roland Dreier wrote:
>     Hal> Right. It is a convenient define for a consumer using Service
>     Hal> Records. I would prefer to leave it in there despite it not
>     Hal> being used.
> 
> OK, that's fine.
> 
>     Hal> It does map to the actual SR layout and path record indicates
>     Hal> where the reserved fields are.
> 
>     Hal> Is changing it to /* reserved */ acceptable like in the
>     Hal> ib_sa_path_rec struct declaration ?
> 
> Actually, I didn't read the patch carefully enough.  Yes, please
> follow the style used in the rest of the file: name the component mask
> entries as they are named in the spec, rather than using your own
> abbreviations, so you have something like
> 
> #define IB_SA_SERVICE_REC_SERVICE_ID			IB_SA_COMP_MASK( 0)
> #define IB_SA_SERVICE_REC_SERVICE_GID			IB_SA_COMP_MASK( 1)
> #define IB_SA_SERVICE_REC_SERVICE_PKEY			IB_SA_COMP_MASK( 2)
> /* reserved:								 3 */
> #define IB_SA_SERVICE_REC_SERVICE_LEASE			IB_SA_COMP_MASK( 4)
> 
> and so on.  And there's no need to define a component mask for reserved rows.

I think there is a legitimate use for this: when one wants to set a lot
of component mask bits and wants to turn off the reserved bit (rather
than or'ing a long list together).

Please verify that the style is acceptable in the next version of the
patch.

> Then you can put 
> 
> 	/* reserved */
> 
> in the structure definition to match the component mask.

OK.

-- Hal



From mst at mellanox.co.il  Tue Jul  5 14:13:17 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 00:13:17 +0300
Subject: [openib-general] Re: Re: OpenIB Releases
In-Reply-To: <20050705210338.GQ13776@sventech.com>
References: <1120586921.4388.9880.camel@hal.voltaire.com>
	<524qb9ul8k.fsf@topspin.com>
	<20050705191910.GA28064@mellanox.co.il>
	<20050705192518.GO13776@sventech.com>
	<20050705205735.GC28064@mellanox.co.il>
	<20050705210338.GQ13776@sventech.com>
Message-ID: <20050705211317.GB29098@mellanox.co.il>

Quoting r. Johannes Erdfelt <johannes at erdfelt.com>:
> The VSD area is highly customized for our purposes. We store information
> about the firmware image programmed so we can automatically upgrade the
> firmware when upgrading the IB software on the system.
> 
> We also use the VSD to store configuration options for our boot ROM
> extension.
> 
> Other than the VSD and some minor supporting features, there isn't
> anything special about tvflash versus mstflint.
> 
> JE
> 

OK, I guess you can even do this on top of mstflint today, if you want
to.

-- 
MST


From halr at voltaire.com  Tue Jul  5 14:13:35 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 05 Jul 2005 17:13:35 -0400
Subject: [openib-general] [PATCH v4] sa_query: Add service record support
Message-ID: <1120598014.4388.10396.camel@hal.voltaire.com>

sa_query: Add service record support

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: include/ib_sa.h
===================================================================
--- include/ib_sa.h	(revision 2771)
+++ include/ib_sa.h	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -193,6 +194,61 @@
 	int          proxy_join;
 };
 
+/* Service Record Component Mask Sec 15.2.5.14 Ver 1.1	*/
+#define IB_SA_SERVICE_REC_SERVICE_ID			IB_SA_COMP_MASK( 0)
+#define IB_SA_SERVICE_REC_SERVICE_GID			IB_SA_COMP_MASK( 1)
+#define IB_SA_SERVICE_REC_SERVICE_PKEY			IB_SA_COMP_MASK( 2)
+/* reserved:								 3 */
+#define IB_SA_SERVICE_REC_SERVICE_LEASE			IB_SA_COMP_MASK( 4)
+#define IB_SA_SERVICE_REC_SERVICE_KEY			IB_SA_COMP_MASK( 5)
+#define IB_SA_SERVICE_REC_SERVICE_NAME			IB_SA_COMP_MASK( 6)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_0		IB_SA_COMP_MASK( 7)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_1		IB_SA_COMP_MASK( 8)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_2		IB_SA_COMP_MASK( 9)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_3		IB_SA_COMP_MASK(10)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_4		IB_SA_COMP_MASK(11)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_5		IB_SA_COMP_MASK(12)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_6		IB_SA_COMP_MASK(13)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_7		IB_SA_COMP_MASK(14)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_8		IB_SA_COMP_MASK(15)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_9		IB_SA_COMP_MASK(16)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_10		IB_SA_COMP_MASK(17)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_11		IB_SA_COMP_MASK(18)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_12		IB_SA_COMP_MASK(19)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_13		IB_SA_COMP_MASK(20)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_14		IB_SA_COMP_MASK(21)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_15		IB_SA_COMP_MASK(22)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_0		IB_SA_COMP_MASK(23)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_1		IB_SA_COMP_MASK(24)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_2		IB_SA_COMP_MASK(25)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_3		IB_SA_COMP_MASK(26)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_4		IB_SA_COMP_MASK(27)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_5		IB_SA_COMP_MASK(28)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_6		IB_SA_COMP_MASK(29)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_7		IB_SA_COMP_MASK(30)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_0		IB_SA_COMP_MASK(31)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_1		IB_SA_COMP_MASK(32)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_2		IB_SA_COMP_MASK(33)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_3		IB_SA_COMP_MASK(34)
+#define IB_SA_SERVICE_REC_SERVICE_DATA64_0		IB_SA_COMP_MASK(35)
+#define IB_SA_SERVICE_REC_SERVICE_DATA64_1		IB_SA_COMP_MASK(36)
+
+#define IB_DEFAULT_SERVICE_LEASE 	0xFFFFFFFF
+
+struct ib_sa_service_rec {
+	u64		id;
+	union ib_gid	gid;		
+	u16 		pkey;
+	/* reserved */
+	u32		lease;
+	u8		key[16];
+	u8		name[64];
+	u8		data8[16];
+	u16		data16[8];
+	u32		data32[4];
+	u64		data64[2];
+};
+
 struct ib_sa_query;
 
 void ib_sa_cancel_query(int id, struct ib_sa_query *query);
@@ -218,6 +274,17 @@
 			     void *context,
 			     struct ib_sa_query **query);
 
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num,
+			 u8 method,
+			 struct ib_sa_service_rec *rec,
+			 ib_sa_comp_mask comp_mask,
+			 int timeout_ms, int gfp_mask,
+			 void (*callback)(int status,
+					  struct ib_sa_service_rec *resp,
+					  void *context),
+			 void *context,
+			 struct ib_sa_query **sa_query);
+
 /**
  * ib_sa_mcmember_rec_set - Start an MCMember set query
  * @device:device to send query on
Index: core/sa_query.c
===================================================================
--- core/sa_query.c	(revision 2771)
+++ core/sa_query.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -79,6 +80,12 @@
 	int                 id;
 };
 
+struct ib_sa_service_query {
+	void (*callback)(int, struct ib_sa_service_rec *, void *);
+	void *context;
+	struct ib_sa_query sa_query;
+};
+
 struct ib_sa_path_query {
 	void (*callback)(int, struct ib_sa_path_rec *, void *);
 	void *context;
@@ -320,6 +327,54 @@
 	  .size_bits    = 23 },
 };
 
+#define SERVICE_REC_FIELD(field) \
+	.struct_offset_bytes = offsetof(struct ib_sa_service_rec, field),	\
+	.struct_size_bytes   = sizeof ((struct ib_sa_service_rec *) 0)->field,	\
+	.field_name          = "sa_service_rec:" #field
+
+static const struct ib_field service_rec_table[] = {
+	{ SERVICE_REC_FIELD(id),
+	  .offset_words = 0,
+	  .offset_bits  = 0,
+	  .size_bits    = 64 },
+	{ SERVICE_REC_FIELD(gid),
+	  .offset_words = 2,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(pkey),
+	  .offset_words = 6,
+	  .offset_bits  = 0,
+	  .size_bits    = 16 },
+	{ SERVICE_REC_FIELD(lease),
+	  .offset_words = 7,
+	  .offset_bits  = 0,
+	  .size_bits    = 32 },
+	{ SERVICE_REC_FIELD(key),
+	  .offset_words = 8,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(name),
+	  .offset_words = 12,
+	  .offset_bits  = 0,
+	  .size_bits    = 64*8 },
+	{ SERVICE_REC_FIELD(data8),
+	  .offset_words = 28,
+	  .offset_bits  = 0,
+	  .size_bits    = 16*8 },
+	{ SERVICE_REC_FIELD(data16),
+	  .offset_words = 32,
+	  .offset_bits  = 0,
+	  .size_bits    = 8*16 },
+	{ SERVICE_REC_FIELD(data32),
+	  .offset_words = 36,
+	  .offset_bits  = 0,
+	  .size_bits    = 4*32 },
+	{ SERVICE_REC_FIELD(data64),
+	  .offset_words = 40,
+	  .offset_bits  = 0,
+	  .size_bits    = 2*64 },
+};
+
 static void free_sm_ah(struct kref *kref)
 {
 	struct ib_sa_sm_ah *sm_ah = container_of(kref, struct ib_sa_sm_ah, ref);
@@ -443,7 +498,6 @@
 				 .remote_qpn  = 1,
 				 .remote_qkey = IB_QP1_QKEY,
 				 .timeout_ms  = timeout_ms,
-				 .retries     = 0 
 			 }
 		 }
 	};
@@ -596,6 +650,114 @@
 }
 EXPORT_SYMBOL(ib_sa_path_rec_get);
 
+static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query,
+				    int status,
+				    struct ib_sa_mad *mad)
+{
+	struct ib_sa_service_query *query =
+		container_of(sa_query, struct ib_sa_service_query, sa_query);
+
+	if (mad) {
+		struct ib_sa_service_rec rec;
+
+		ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table),
+			  mad->data, &rec);
+		query->callback(status, &rec, query->context);
+	} else
+		query->callback(status, NULL, query->context);
+}
+
+static void ib_sa_service_rec_release(struct ib_sa_query *sa_query)
+{
+	kfree(sa_query->mad);
+	kfree(container_of(sa_query, struct ib_sa_service_query, sa_query));
+}
+
+/**
+ * ib_sa_service_rec_query - Start Service Record operation
+ * @device:device to send request on
+ * @port_num: port number to send request on
+ * @method:SA method - should be get, set, or delete
+ * @rec:Service Record to send in request
+ * @comp_mask:component mask to send in request
+ * @timeout_ms:time to wait for response
+ * @gfp_mask:GFP mask to use for internal allocations
+ * @callback:function called when request completes, times out or is
+ * canceled
+ * @context:opaque user context passed to callback
+ * @sa_query:request context, used to cancel request
+ *
+ * Send a Service Record set/get/delete to the SA to register,
+ * unregister or query a service record.
+ * The callback function will be called when the request completes (or
+ * fails); status is 0 for a successful response, -EINTR if the query
+ * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error
+ * occurred sending the query.  The resp parameter of the callback is
+ * only valid if status is 0.
+ *
+ * If the return value of ib_sa_service_rec_query() is negative, it is an
+ * error code.  Otherwise it is a request ID that can be used to cancel
+ * the query.
+ */
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
+			    struct ib_sa_service_rec *rec,
+			    ib_sa_comp_mask comp_mask,
+			    int timeout_ms, int gfp_mask,
+			    void (*callback)(int status,
+					     struct ib_sa_service_rec *resp,
+					     void *context),
+			    void *context,
+			    struct ib_sa_query **sa_query)
+{
+	struct ib_sa_service_query *query;
+	struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client);
+	struct ib_sa_port   *port   = &sa_dev->port[port_num - sa_dev->start_port];
+	struct ib_mad_agent *agent  = port->agent;
+	int ret;
+
+	if (method != IB_MGMT_METHOD_GET &&
+	    method != IB_MGMT_METHOD_SET &&
+	    method != IB_SA_METHOD_DELETE)
+		return -EINVAL;
+
+	query = kmalloc(sizeof *query, gfp_mask);
+	if (!query)
+		return -ENOMEM;
+	query->sa_query.mad = kmalloc(sizeof *query->sa_query.mad, gfp_mask);
+	if (!query->sa_query.mad) {
+		kfree(query);
+		return -ENOMEM;
+	}
+
+	query->callback = callback;
+	query->context  = context;
+
+	init_mad(query->sa_query.mad, agent);
+
+	query->sa_query.callback              = callback ? ib_sa_service_rec_callback : NULL;
+	query->sa_query.release               = ib_sa_service_rec_release;
+	query->sa_query.port                  = port;
+	query->sa_query.mad->mad_hdr.method   = method;
+	query->sa_query.mad->mad_hdr.attr_id  =
+				cpu_to_be16(IB_SA_ATTR_SERVICE_REC);
+	query->sa_query.mad->sa_hdr.comp_mask = comp_mask;
+
+	ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table),
+		rec, query->sa_query.mad->data);
+
+	*sa_query = &query->sa_query;
+
+	ret = send_mad(&query->sa_query, timeout_ms);
+	if (ret < 0) {
+		*sa_query = NULL;
+		kfree(query->sa_query.mad);
+		kfree(query);
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(ib_sa_service_rec_query);
+
 static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query,
 					int status,
 					struct ib_sa_mad *mad)





From viswak at yahoo.com  Tue Jul  5 14:24:05 2005
From: viswak at yahoo.com (viswanath krishnamurthy)
Date: Tue, 5 Jul 2005 14:24:05 -0700 (PDT)
Subject: [openib-general] useraccess_cm sample client/server (gen1)
Message-ID: <20050705212405.16959.qmail@web33201.mail.mud.yahoo.com>

Is there a sample code (examples) to use the gen1
stack user level CM API (ioctls) ? Any pointers is
appreciated.

Thanks,
Vish


		
____________________________________________________ 
Yahoo! Sports 
Rekindle the Rivalries. Sign up for Fantasy Football 
http://football.fantasysports.yahoo.com


From rolandd at cisco.com  Tue Jul  5 15:07:28 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 05 Jul 2005 15:07:28 -0700
Subject: [openib-general] Re: [PATCH 11/16] IB uverbs: add mthca mmap support
In-Reply-To: <20050705205351.GB28064@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 5 Jul 2005 23:53:51 +0300")
References: <2005628163.o84QGfsM7oMSy0oU@cisco.com>
	<2005628163.gtJFW6uLUrGQteys@cisco.com>
	<20050628170553.00a14a29.akpm@osdl.org> <52mzp1oy91.fsf@topspin.com>
	<20050705205351.GB28064@mellanox.co.il>
Message-ID: <52d5pwnbz3.fsf@topspin.com>

    Michael> Roland, I think VM_DONTCOPY is needed here.

    Michael> If a process forks, we must prevent the child from
    Michael> accessing the parent's hardware page. Otherwise the child
    Michael> can corrupt the parent's queues since the hardware wont
    Michael> be able to distinguish between parent and child.

    Michael> Does this make sense?

This is true, but there are a number of pieces that are required
before fork will work for processes using userspace verbs.  One of the
ingredients that's missing is adding something like PROT_DONTCOPY for
mprotect().  Once that's in place, an app can use that on the
doorbell page before forking.

I don't consider this attack by children of a process very serious,
since a process can always fork, munmap the doorbell page in the child
process, and then fork the untrusted child into yet another child.

 - R.


From yaronh at voltaire.com  Tue Jul  5 23:43:36 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Wed, 6 Jul 2005 09:43:36 +0300
Subject: [openib-general] [iser]about the target
Message-ID: <35EA21F54A45CB47B879F21A91F4862F60DE7B@taurus.voltaire.com>

Ian,

 

An iSER target is basically an iSCSI target with an add-on of an iSER transport

As you may know Voltaire contributed the iSER Initiator and also has a full Target implementation that was tested with it

 

There are few Target solutions that are possible:

There is at least one major storage vendor working on a target that will be available later on

Voltaire also provides a software package that can turn a server to an iSCSI/iSER target (not open source)

As well as gateway solutions from iSER to FC and GbE 

In addition UNH has started some work on enabling iSER on their Open Source iSCSI target 

 

Logically iSER target is a (much faster) iSCSI Target 

So just like in iSCSI you can bridge iSER to FC just like Cisco MDS Bridge from iSCSI to FC 

There are standards that define the exact mapping between the FC naming and iSCSI naming 

 

Yaron

 

________________________________

From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of Michael Krause
Sent: Tuesday, July 05, 2005 7:28 PM
To: Ian Jiang; openib-general at openib.org
Subject: Re: [openib-general] [iser]about the target

 

At 06:07 PM 7/4/2005, Ian Jiang wrote:



Hi!
I am new to the iSER.
On " https://openib.org/tiki/tiki-index.php?page=iSER <https://openib.org/tiki/tiki-index.php?page=iSER> ", it is said that iSER currently contains initiator only (no target). Will the target come out later? How did they test the iSER initiator without a iSER target?
Could you give some explaination?


>From a practical perspective, there are very few iSCSI targets shipping today.  Most people had envisioned iSER over IB to a gateway Ethernet device since native IB storage is also quite rare in terms of real product.  For many of us, our push for iSER over IB was to replace SRP which has a deficient ecosystem thus not really used beyond some basic Fibre Channel gateway cards.  

Mike




Thanks!



Ian Jiang
ianjiang91 at hotmail.com
----
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62564394(office)

_________________________________________________________________
ÓëÁª»úµÄÅóÓѽøÐн»Á÷£¬ÇëʹÓà MSN Messenger:  http://messenger.msn.com/cn <http://messenger.msn.com/cn%A0> 

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050706/41e0001b/attachment.html>

From glebn at voltaire.com  Wed Jul  6 05:17:11 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 6 Jul 2005 15:17:11 +0300
Subject: [openib-general] [PATCH] signal driven io support for ib_uverbs
Message-ID: <20050706121711.GG12680@minantech.com>

Hello, 

Included patch adds O_ASYNC support for uverbs event handling.

Signed-off-by: Gleb Natapov <glebn at voltaire.com>

Index: trunk/src/linux-kernel/infiniband/core/uverbs_main.c
===================================================================
--- trunk/src/linux-kernel/infiniband/core/uverbs_main.c	(revision 2810)
+++ trunk/src/linux-kernel/infiniband/core/uverbs_main.c	(working copy)
@@ -258,10 +258,28 @@
 	spin_unlock_irq(&file->lock);
 }
 
+static int ib_uverbs_event_async (int fd, struct file *filp, int on)
+{
+	int ret;
+	struct ib_uverbs_event_file *file = filp->private_data;
+
+	if (!file)
+		return -EBADFD;
+
+	if ((ret = fasync_helper(fd, filp, on, &file->fasync)) < 0)
+		return ret;
+
+	if (on)
+		ret = f_setown(filp, current->pid, 0);
+	
+	return ret;
+}
+
 static int ib_uverbs_event_close(struct inode *inode, struct file *filp)
 {
 	struct ib_uverbs_event_file *file = filp->private_data;
 
+	ib_uverbs_event_async (-1, filp, 0);
 	ib_uverbs_event_release(file);
 	kref_put(&file->uverbs_file->ref, ib_uverbs_release_file);
 
@@ -277,7 +295,8 @@
 	 */
 	.read 	 = ib_uverbs_event_read,
 	.poll    = ib_uverbs_event_poll,
-	.release = ib_uverbs_event_close
+	.release = ib_uverbs_event_close,
+	.fasync  = ib_uverbs_event_async
 };
 
 void ib_uverbs_comp_handler(struct ib_cq *cq, void *cq_context)
@@ -317,6 +336,7 @@
 	spin_unlock_irqrestore(&file->async_file.lock, flags);
 
 	wake_up_interruptible(&file->async_file.poll_wait);
+	kill_fasync (&file->async_file.fasync, SIGIO, POLL_IN);
 }
 
 void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr)
@@ -351,6 +371,7 @@
 	INIT_LIST_HEAD(&file->event_list);
 	init_waitqueue_head(&file->poll_wait);
 	file->uverbs_file = uverbs_file;
+	file->fasync = NULL;
 
 	file->fd = get_unused_fd();
 	if (file->fd < 0)
Index: trunk/src/linux-kernel/infiniband/core/uverbs.h
===================================================================
--- trunk/src/linux-kernel/infiniband/core/uverbs.h	(revision 2810)
+++ trunk/src/linux-kernel/infiniband/core/uverbs.h	(working copy)
@@ -62,6 +62,7 @@
 	int					fd;
 	int					is_async;
 	wait_queue_head_t			poll_wait;
+	struct fasync_struct			*fasync;
 	struct list_head			event_list;
 };
 
--
			Gleb.


From halr at voltaire.com  Wed Jul  6 05:40:20 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 06 Jul 2005 08:40:20 -0400
Subject: [openib-general] RE: Some IBMgtSim Questions and Nits
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30424@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30424@mtlex01.yok.mtl.com>
Message-ID: <1120653380.16533.1114.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 11:22, Eitan Zahavi wrote:
> Hi Hal,
> 
> > 
> > In INSTALL, it states:
> > Software Dependencies
> > =====================
> > IBMgtSim depends on: tcl8.3/8.4, OpenSM 1.7.0 and ibdm 1.0
> > 
> > What are the specific dependencies on OpenSM 1.7.0 ?
> [EZ] Actually it should say 1.7.0 and after. 
> The dependency is on the subnet.lst format and osm.mcfdbs dump file.

Did either of these formats change at 1.7.0 (from 1.6.1) ? (You also
mentioned osm.fdbs somewhere else; did that format change ?)

> > Also there are a number of files in this subdirectory (and
> underneath
> > it) which have +x permissions and don't need this.
> [EZ] How can I modify the file permissions after they are committed to
> svn ?

svn propset

-- Hal



From trimmer at silverstorm.com  Wed Jul  6 06:14:55 2005
From: trimmer at silverstorm.com (Rimmer, Todd)
Date: Wed, 6 Jul 2005 09:14:55 -0400
Subject: [openib-general] [iser]about the target
Message-ID: <5D78D28F88822E4D8702BB9EEF1A43670A072B@mercury.infiniconsys.com>

I would like to clarify the comment on SRP.  There are companies presently shipping and demonstrating SRP native IB storage.  For example:
    Engenio (formerly LSI)
    Raytheon
    Data Direct
    Mellanox
 
SRP was designed for highly optimized storage access across an RDMA capable transport, and hence is capable of very high performance.
 
Longer term the storage vendors anticipate that iSCSI will be the focus for long haul (remote backup, etc) type solutions, while IB Native Storage and FC will be the focus for data center high performance storage solutions.
 
Todd R.

-----Original Message-----
From: Michael Krause [mailto:krause at cup.hp.com]
Sent: Tuesday, July 05, 2005 12:28 PM
To: Ian Jiang; openib-general at openib.org
Subject: Re: [openib-general] [iser]about the target


At 06:07 PM 7/4/2005, Ian Jiang wrote:


Hi!
I am new to the iSER.
On "  <https://openib.org/tiki/tiki-index.php?page=iSER> https://openib.org/tiki/tiki-index.php?page=iSER", it is said that iSER currently contains initiator only (no target). Will the target come out later? How did they test the iSER initiator without a iSER target?
Could you give some explaination?


>From a practical perspective, there are very few iSCSI targets shipping today.  Most people had envisioned iSER over IB to a gateway Ethernet device since native IB storage is also quite rare in terms of real product.  For many of us, our push for iSER over IB was to replace SRP which has a deficient ecosystem thus not really used beyond some basic Fibre Channel gateway cards.  

Mike



Thanks!



Ian Jiang
ianjiang91 at hotmail.com
----
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62564394(office)

_________________________________________________________________
ÓëÁª»úµÄÅóÓѽøÐн»Á÷£¬ÇëʹÓà MSN Messenger:  http://messenger.msn.com/cn  <http://messenger.msn.com/cn%A0> 

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050706/0fa952d5/attachment.html>

From wxg at ict.ac.cn  Wed Jul  6 06:26:00 2005
From: wxg at ict.ac.cn (=?gb2312?B?zfXO/bnz?=)
Date: Wed, 6 Jul 2005 21:26:00 +0800
Subject: [openib-general] how to choose appropriate ULPs for application
Message-ID: <20050706133304.4B2E72283D4@openib.ca.sandia.gov>

Hello！

    We plan to run lustre on InfiniBand. But now lustre does not support IB.
I know noting about the IB stack. So I think the IB Upper Layer Protocol(SDP, DAP) could be used to transport data for lustre. 
	But there are too many ULPs over IB. I know little about them. As far as I konw, iSER/SRP is for SCSI command transportation. So, it seems both SDP and uDAP are OK for my plan. I cant tell the difference between them. Which one is better, SDP or DAP? I don't konw how to choose?

	Any suggestion is appreciated!

　　

　　　　　　　　致
礼！

Sincerely yours
Wang Xigui 王锡贵
2005-07-06
--------------------
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62564394(office)
Email: wxg at ict.ac.cn 				
　　　　　　　　　　　　　　
　　　　　　　　　　　　　　
　　　　　　　　　　　　　　　　　

From eitan at mellanox.co.il  Wed Jul  6 06:29:25 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Wed, 6 Jul 2005 16:29:25 +0300 
Subject: [openib-general] RE: Some IBMgtSim Questions and Nits
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3043B@mtlex01.yok.mtl.com>

> 
> Did either of these formats change at 1.7.0 (from 1.6.1) ? (You also
> mentioned osm.fdbs somewhere else; did that format change ?)
[EZ] Not sure about that. I have to lookup the old code tree.
> 
> > > Also there are a number of files in this subdirectory (and
> > underneath
> > > it) which have +x permissions and don't need this.
> > [EZ] How can I modify the file permissions after they are committed to
> > svn ?
> 
> svn propset
[EZ] OK I will give it a try 
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050706/2f24d8bc/attachment.html>

From eitan at mellanox.co.il  Wed Jul  6 06:35:06 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Wed, 6 Jul 2005 16:35:06 +0300 
Subject: [openib-general] RE: IBDM - IB DataModel build issues fixed - full
	version
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3043C@mtlex01.yok.mtl.com>

> > > Why does the TOPSYSTEM need to be identified ? Is it just a way of
> > not
> > > having it as the first file in the system definition ?
[EZ] Actually the definition is required to be "declare before use ..."
> 
> Can you elaborate a little on TOPSYSTEM ?
[EZ] TOPSYSTEM defines the section in the netlist where the "top level" of
the system being described in the IBNL file is defined. The IBNL can define
a system that is built out of arbitrary levels of hierarchy (of
boards/subsystems etc). The top system is different as it allows definition
of modifiable attributes for its sub systems:
topsystem:
    TOPSYSTEM { gIsTopSystem = 1; } 
    system_names { ibnlMakeSystem(gSysNames); } 
    NL sub_inst_attributes 
    insts 
  ;

system:
    SYSTEM { gIsTopSystem = 0; } 
    system_names { ibnlMakeSystem(gSysNames); } NL insts 
  ;

> >
> > MT2400 mySwitch
> >    P1 -> Cougar H-1 P2
> >    P2 -> Cougar H-2 P1
> 
> So this is the physical connectivity between external systems (and IBNL
> is the internal physical connectivity of a system). Correct ?
[EZ] Yes
> 
> Is a simple topology file describing two Cougars (but you might use
> other HCA names of course) to a 24 port switch system named MT2400.
[EZ] Yes you can define whatever systems you like in the IBNL files.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050706/0b83b9fb/attachment.html>

From clemc at ammasso.com  Wed Jul  6 06:43:11 2005
From: clemc at ammasso.com (Clem Cole)
Date: Wed, 6 Jul 2005 09:43:11 -0400
Subject: [openib-general] how to choose appropriate ULPs for application
Message-ID: <8E9D028761D8264D910612167E8457E8742D7B@mail2.ammasso.com>

The quick answer here is none of SDP, iSER/SRP or uDAPL for different
reasons on each case

kDAPL is the kernel API that is designed to handle a kernel application
such as NFS, Lustre or the like.  See:
	http://www.datcollaborative.org/kdapl.html

There is are already implementations of NFS/RDMA available:
	http://www.sourceforge.net/projects/nfs-rdma

This should give you some hints on how to a distrubuted file system
uses the kDAPL interface. 



-----Original Message-----
From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org]On Behalf Of wxg at ict.ac.cn
Sent: Wednesday, July 06, 2005 9:26 AM
To: openib-general
Subject: [openib-general] how to choose appropriate ULPs for application


Hello！

    We plan to run lustre on InfiniBand. But now lustre does not support IB.
I know noting about the IB stack. So I think the IB Upper Layer Protocol(SDP, DAP) could be used to transport data for lustre. 
	But there are too many ULPs over IB. I know little about them. As far as I konw, iSER/SRP is for SCSI command transportation. So, it seems both SDP and uDAP are OK for my plan. I cant tell the difference between them. Which one is better, SDP or DAP? I don't konw how to choose?

	Any suggestion is appreciated!

　　

　　　　　　　　致
礼！

Sincerely yours
Wang Xigui 王锡贵
2005-07-06
--------------------
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62564394(office)
Email: wxg at ict.ac.cn 				
　　　　　　　　　　　　　　
　　　　　　　　　　　　　　
　　　　　　　　　　　　　　　　　


From mst at mellanox.co.il  Wed Jul  6 06:56:38 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 16:56:38 +0300
Subject: [openib-general] Re: how to choose appropriate ULPs for application
In-Reply-To: <20050706133304.4B2E72283D4@openib.ca.sandia.gov>
References: <20050706133304.4B2E72283D4@openib.ca.sandia.gov>
Message-ID: <20050706135638.GM14130@mellanox.co.il>

Quoting ?????? <wxg at ict.ac.cn>:
> Subject: how to choose appropriate ULPs for application
>
>     We plan to run lustre on InfiniBand. But now lustre does not support IB.

If it currently works on top of sockets, SDP is the fastest way
to get it running on InfiniBand, with good performance.
You basically have to change the protocol family, and you are done.

-- 
MST


From wxg at ict.ac.cn  Wed Jul  6 07:09:25 2005
From: wxg at ict.ac.cn (wangxigui)
Date: Wed, 6 Jul 2005 22:09:25 +0800 (CST)
Subject: [openib-general] how to choose appropriate ULPs for application
In-Reply-To: <8E9D028761D8264D910612167E8457E8742D7B@mail2.ammasso.com>
References: <8E9D028761D8264D910612167E8457E8742D7B@mail2.ammasso.com>
Message-ID: <2322.159.226.40.170.1120658965.squirrel@webmail.ict.ac.cn>

Thank you very much!
I'll read the materials that you supply.

     I have not described my plan clearly. I don't konw how to run lustre
on IB. Neither do I konw the lustre running in kernel space or user
space. Now the lustre NAL does not supports IB vapi. If I use IPoIB,
it works. But it doen not take take advantage of RDMA. Supporse
lustre NAL invoke SDP or uDAP, RDMA is enable.

    Invocation of Verb API to realize lustre NAL is my target. The SDP or
uDAP is first step.

Clem Cole said:
> The quick answer here is none of SDP, iSER/SRP or uDAPL for different
> reasons on each case
>
> kDAPL is the kernel API that is designed to handle a kernel application
> such as NFS, Lustre or the like.  See:
> 	http://www.datcollaborative.org/kdapl.html
>
> There is are already implementations of NFS/RDMA available:
> 	http://www.sourceforge.net/projects/nfs-rdma
>
> This should give you some hints on how to a distrubuted file system
> uses the kDAPL interface.
>
>
>
> -----Original Message-----
> From: openib-general-bounces at openib.org
> [mailto:openib-general-bounces at openib.org]On Behalf Of wxg at ict.ac.cn
> Sent: Wednesday, July 06, 2005 9:26 AM
> To: openib-general
> Subject: [openib-general] how to choose appropriate ULPs for application
>
>
> Hello！
>
>     We plan to run lustre on InfiniBand. But now lustre does not support
> IB.
> I know noting about the IB stack. So I think the IB Upper Layer
> Protocol(SDP, DAP) could be used to transport data for lustre.
> 	But there are too many ULPs over IB. I know little about them. As far as
> I konw, iSER/SRP is for SCSI command transportation. So, it seems both
> SDP and uDAP are OK for my plan. I cant tell the difference between them.
> Which one is better, SDP or DAP? I don't konw how to choose?
>
> 	Any suggestion is appreciated!
>
> 　　
>
> 　　　　　　　　致
> 礼！
>
> Sincerely yours
> Wang Xigui 王锡贵
> 2005-07-06
> --------------------
> Computer Architecture Laboratory
> Institute of Computing Technology
> Chinese Academy of Sciences
> Beijing,P.R.China
> Zip code: 100080
> Tel: +86-10-62564394(office)
> Email: wxg at ict.ac.cn
> 　　　　　　　　　　　　　　
> 　　　　　　　　　　　　　　
> 　　　　　　　　　　　　　　　　　
>


-- 
Sincerely yours
Wang Xigui 王锡贵
2005-07-06
--------------------
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62565533-9314(office)
Email: wxg at ict.ac.cn

-- 
Sincerely yours
Wang Xigui 王锡贵
2005-07-06
--------------------
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62565533-9314(office)
Email: wxg at ict.ac.cn



From caitlin.bestler at gmail.com  Wed Jul  6 08:31:48 2005
From: caitlin.bestler at gmail.com (Caitlin Bestler)
Date: Wed, 6 Jul 2005 08:31:48 -0700
Subject: [openib-general] how to choose appropriate ULPs for application
In-Reply-To: <20050706133304.4B2E72283D4@openib.ca.sandia.gov>
References: <20050706133304.4B2E72283D4@openib.ca.sandia.gov>
Message-ID: <469958e005070608313026674e@mail.gmail.com>

I would also recommend studying DAFS, and in particular
the differences between DAFS and NFS over RDMA.

NFS over RDMA leaves most of the existing file system
logic intact and basically optimizes the exchange of
individual messages over RDMA.

Using SDP would be a somewhat similar strategy,
with slightly less optimizations. The key benefit being
minimal disruption of existing file system layer code.

DAFS would be an example of fully optimizing the
peer-to-peer communication to optimize for RDMA.
If your application lacks an existing intermediate
layer, such as RPC, that type of approach may
be required to get the full benefit of RDMA. Such
an approach would suggest the use of kDAPL.

But I would recommend against use of kDAPL if
you are only replacing the transport of messages
locked to socket semantics. If you are going to
keep the socket semantics then definitely use SDP.


On 7/6/05, 王锡贵 <wxg at ict.ac.cn> wrote:
> Hello！
> 
>     We plan to run lustre on InfiniBand. But now lustre does not support IB.
> I know noting about the IB stack. So I think the IB Upper Layer Protocol(SDP, DAP) could be used to transport data for lustre.
>         But there are too many ULPs over IB. I know little about them. As far as I konw, iSER/SRP is for SCSI command transportation. So, it seems both SDP and uDAP are OK for my plan. I cant tell the difference between them. Which one is better, SDP or DAP? I don't konw how to choose?
> 
>         Any suggestion is appreciated!
> 
> 
> 
> 致
> 礼！
> 
> Sincerely yours
> Wang Xigui 王锡贵
> 2005-07-06
> --------------------
> Computer Architecture Laboratory
> Institute of Computing Technology
> Chinese Academy of Sciences
> Beijing,P.R.China
> Zip code: 100080
> Tel: +86-10-62564394(office)
> Email: wxg at ict.ac.cn
> 
> 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 
>

From David.Brean at Sun.COM  Tue Jul  5 13:08:49 2005
From: David.Brean at Sun.COM (David M. Brean)
Date: Tue, 05 Jul 2005 16:08:49 -0400
Subject: [openib-general] Re: IP addressing on InfiniBand networks	(Caitlin
	Bestler)
In-Reply-To: <6.2.0.14.2.20050705093014.0291a9d0@esmail.cup.hp.com>
References: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com>
	<52r7ej68ga.fsf@topspin.com>
	<6.2.0.14.2.20050705093014.0291a9d0@esmail.cup.hp.com>
Message-ID: <42CAE8D1.1040904@sun.com>

The IB spec says "A GID is a valid 128-bit IPv6 address (per RFC 2373) 
with additional properties / restrictions defined within IBA to 
facilitate efficient discovery, communication, and routing.", so I don't 
think it needs to say much more.  [Perhaps the IB specification should 
refer to the more recent RFC 3513 that obsoletes RFC 2373...]  One of 
those IBA restrictions result from being limited to assigning one 64-bit 
value for the upper 64-bits of the GID, called the GID prefix, per IB 
subnet.  If GIDs are used to represent IPv6 addresses on an IB subnet, 
then only one IP subnet prefix can be mapped onto an IB subnet.  This 
limitation goes along with the additional work of integrating GID 
management into the IP management applications for assigning IP addresses.

Restrictions on IP address assignment are removed when using the GID as 
specified by the IETF IPoIB as a link layer address and IB partitions 
can be used to create multiple IP links per IB subnet.  Also, the IP 
management application integration effort is significantly smaller.

Therefore, I recommend that a solution for mapping between IP and GIDs 
not rely on GIDs being used as the IP addresses.

-David

Michael Krause wrote:

> At 10:49 AM 6/30/2005, Roland Dreier wrote:
>
>>     Michael> Being the person who led the addressing definition for
>>     Michael> IB, I can state quite clearly that GID are NOT IPv6
>>     Michael> addresses.  They were intentionally defined to have a
>>     Michael> similar look-n-feel since they were derived in large part
>>     Michael> from Future I/O which had them as real IPv6 addresses.
>>     Michael> But again, they are NOT IPv6 addresses.
>>
>> The IBA spec seems to have a different idea.  In fact chapter 4 says:
>>
>>     "A GID is a valid 128-bit IPv6 address (per RFC 2373)...."
>
>
> I wrote the original spec here.  The text was supposed to be updated 
> to clarify that the rest of the sentence, i.e. with additional rules, 
> etc. thus making it not a real IPv6 address from the IETF's 
> perspective but something quite close.  The intention was to allow one 
> to manage the fabric by having mapping functions from traditional IP 
> management applications to IB GID to minimize the amount of work to 
> enable IB within a solution. 
>



From rep.nop at aon.at  Wed Jul  6 09:02:22 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Wed, 6 Jul 2005 18:02:22 +0200
Subject: [openib-general] [PATCH] ibdm nits and a few questions
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3041C@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C3041C@mtlex01.yok.mtl.com>
Message-ID: <20050706160222.GA13817@aon.at>

Hi Eitan and Hal,
On Tue, Jul 05, 2005 at 02:23:57PM +0300, Eitan Zahavi wrote:
>Hi Ber,
>
>Thanks.
>I have used an ex script to fix the following typos on all the code tree's I
>own.
>In the future - it is much better from my perspective to simply get the list
>of words I need to check for.

>I hope you will be able to simply send me the list of words. It will make my
>life easier.

Sure.

missm -> mism
optential -> potential
finaly -> finally
adn -> and
simplicty -> simplicity
consistant -> consistent
the[[:space:]]*the[[:space:]] -> the 
(i.e. double 'the' but not 'the theory')
incomming -> incoming
seperat -> separat
unseccessful -> unsuccessful
fucntions -> functions
acording -> according
(acordingly -> accordingly)
callabcks -> callbacks
distructor -> destructor
Distractor -> Destructor
reciev -> receiv
(recieving -> receiving)
(Recieved -> Received)


What about removing all the generated scripts (configure et al) and
auto-tools helpers (aclocal.m4 etc and most of config/*) from the
repository?
Note that the other userspace IB libraries do not have these in the
repository but each do provide a toplevel ./autogen.sh -- as opposed
to ibdm's ./config/bootstrap.


There are a number of other unused variables.

Is ibdm supposed to be compilable with pure ISO c++ compilers, btw?
There is at least one place which uses a variable sized array which AFAIK
is not valid ISO c++.


Attached patch does:

IBSystem::guid: As default nowadays is public, move guid into the
        private section explicitely.
configure.in: fix indentation of --enable-debug help text.
IBFabric::parseFdbFile(): initialize p_node
IBFabric::parseMCFdbFile(): ditto
IBFabric::IBLinkWidth: remove comma at end of enumerator list
IBFabric::IBLinkSpeed: ditto
IBFabric:: remove 'IBFabric::' qualifier from member function
	parseMCFdbFile()
Fabric.h: remove extra ';' from member functions
datamodel/ibdmsh_wrap.cpp::ibdmGetObjPtrByTclName(): avoid using
	uninitialized name
datamodel/ibdm_wrap.cpp::ibdmGetObjPtrByTclName(): ditto
datamodel/SubnMgt.cpp::SubnMgtCalcMinHopTables(): make sure worstHopLid
	is initialized
datamodel/SubnMgt.cpp::SubnReportNonUpDownCa2CaPaths(): make sure
	p_prevNode is initialized
datamodel/TopoMatch.cpp::TopoMatchNodeByAdjacentMatched(): make sure
	succeedMatch is initialized
datamodel/TopoMatch.cpp::TopoMatchSpecNodesByAdjacentNode(): add missing
	return numMatched;
datamodel/TraceRoute.cpp::TraceDRPathRoute(): remove check which is
	always false due to limited datatype
src/osm_check.cpp::main(): remove unused variable 'status'

IBMgtSim doesn't compile for me as iba/ib_types.h isn't installed, so
the above is only compile-tested.
Hal, will you take care of installing the missing headers?

Thank you,
Bernhard
-------------- next part --------------
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/configure.in gen2.2810/utils/src/linux-user/ibdm/configure.in
--- gen2.2810.oorig/utils/src/linux-user/ibdm/configure.in	2005-03-24 17:55:47.000000000 +0100
+++ gen2.2810/utils/src/linux-user/ibdm/configure.in	2005-07-06 13:42:52.000000000 +0200
@@ -73,7 +73,7 @@ dnl AC_SUBST(LTLIBOBJS)
 
 dnl Define an input config option to control debug compile
 AC_ARG_ENABLE(debug,
-[  --enable-debug    Turn on debugging],
+[  --enable-debug          Turn on debugging],
 [case "${enableval}" in
   yes) debug=true ;;
   no)  debug=false ;;
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/Fabric.cpp gen2.2810/utils/src/linux-user/ibdm/datamodel/Fabric.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/Fabric.cpp	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/Fabric.cpp	2005-07-06 14:31:01.000000000 +0200
@@ -1593,7 +1593,7 @@ IBFabric::parseFdbFile(string fn) {
   
   cout << "-I- Parsing OpenSM FDBs file:" << fn.c_str() << endl;
   
-  IBNode *p_node;
+  IBNode *p_node = NULL;
   int anyErr = 0;
 
   while (f.good()) {
@@ -1665,7 +1665,7 @@ IBFabric::parseMCFdbFile(string fn) {
   
   cout << "-I- Parsing OpenSM Multicast FDBs file:" << fn.c_str() << endl;
   
-  IBNode *p_node;
+  IBNode *p_node = NULL;
   int anyErr = 0;
 
   while (f.good()) {
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/Fabric.h gen2.2810/utils/src/linux-user/ibdm/datamodel/Fabric.h
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/Fabric.h	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/Fabric.h	2005-07-06 14:16:07.000000000 +0200
@@ -132,7 +132,7 @@ typedef enum {IB_UNKNOWN_LINK_WIDTH = 0,
               IB_LINK_WIDTH_1X = 1,
               IB_LINK_WIDTH_4X = 2,
               IB_LINK_WIDTH_8X = 4,
-              IB_LINK_WIDTH_12X =8,
+              IB_LINK_WIDTH_12X =8
 } IBLinkWidth;
 
 static inline IBLinkWidth char2width( const char *w) 
@@ -143,7 +143,7 @@ static inline IBLinkWidth char2width( co
   if (!strcmp(w,"8x")) return IB_LINK_WIDTH_8X;
   if (!strcmp(w,"12x")) return IB_LINK_WIDTH_12X;
   return IB_UNKNOWN_LINK_WIDTH;
-};
+}
 
 static inline char * width2char( const IBLinkWidth w) 
 {
@@ -155,12 +155,12 @@ static inline char * width2char( const I
   case IB_LINK_WIDTH_12X: return("12x"); break;
   default: return("UNKNOWN");
   }
-};
+}
 
 typedef enum {IB_UNKNOWN_LINK_SPEED = 0, 
               IB_LINK_SPEED_2_5 = 1,
               IB_LINK_SPEED_5   = 2,
-              IB_LINK_SPEED_10  = 4,
+              IB_LINK_SPEED_10  = 4
 } IBLinkSpeed;
 
 static inline char * speed2char( const IBLinkSpeed s) 
@@ -172,7 +172,7 @@ static inline char * speed2char( const I
   case IB_LINK_SPEED_10:  return("10"); break;
   default: return("UNKNOWN");
   }
-};
+}
 
 static inline IBLinkSpeed char2speed( const char *s) 
 {
@@ -181,14 +181,14 @@ static inline IBLinkSpeed char2speed( co
   if (!strcmp(s,"5"))   return IB_LINK_SPEED_5;
   if (!strcmp(s,"10"))  return IB_LINK_SPEED_10;
   return IB_UNKNOWN_LINK_SPEED;
-};
+}
 
 static inline string guid2str(uint64_t guid) {
   char buff[18];
   sprintf(buff, "0x%016" PRIx64 , guid);
   //  string res(buff);
   return buff;
-};
+}
 
 // 
 // IB Port class. 
@@ -196,6 +196,7 @@ static inline string guid2str(uint64_t g
 // a node.
 // 
 class IBPort {
+private:
   uint64_t        guid;           // The port GUID (on SW only on Port0)
  public:
   class IBPort    * p_remotePort; // Port connected on the other side of link
@@ -237,6 +238,7 @@ typedef union _PrivateAppData {
 // IB Node class
 // 
 class IBNode {
+private:
   uint64_t guid;
  public:
   string			   name;         // Name of the node (instance name of the chip)
@@ -339,6 +341,7 @@ class IBSysPort {
 // This is normally derived into a system specific class
 //
 class IBSystem {
+private:
   uint64_t        guid;
  public:
   string			   name;       // the "host" name of the system
@@ -474,7 +477,7 @@ class IBFabric {
   int parseFdbFile(string fn);
 
   // Parse an OpenSM MCFDBs file and set the MFT table accordingly
-  int IBFabric::parseMCFdbFile(string fn);
+  int parseMCFdbFile(string fn);
 
   // set a lid port
   inline void setLidPort (unsigned int lid, IBPort *p_port) {
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/ibdmsh_wrap.cpp gen2.2810/utils/src/linux-user/ibdm/datamodel/ibdmsh_wrap.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/ibdmsh_wrap.cpp	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/ibdmsh_wrap.cpp	2005-07-06 15:33:29.782708152 +0200
@@ -459,7 +459,7 @@ SWIGEXPORT(int,Ibdm_Init)(Tcl_Interp *);
   int ibdmGetObjPtrByTclName(Tcl_Obj *objPtr, void **ptr) {
 	 /* we need to parse the name and get the type etc. */
 	 char buf[256];
-	 char *type, *name, *fabIdxStr;
+	 char *type, *name = NULL, *fabIdxStr;
 	 char *colonIdx, *slashIdx;
 	 int fabricIdx;
 	 *ptr = NULL;
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/ibdm_wrap.cpp gen2.2810/utils/src/linux-user/ibdm/datamodel/ibdm_wrap.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/ibdm_wrap.cpp	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/ibdm_wrap.cpp	2005-07-06 15:19:48.172208363 +0200
@@ -459,7 +459,7 @@ SWIGEXPORT(int,Ibdm_Init)(Tcl_Interp *);
   int ibdmGetObjPtrByTclName(Tcl_Obj *objPtr, void **ptr) {
 	 /* we need to parse the name and get the type etc. */
 	 char buf[256];
-	 char *type, *name, *fabIdxStr;
+	 char *type, *name = NULL, *fabIdxStr;
 	 char *colonIdx, *slashIdx;
 	 int fabricIdx;
 	 *ptr = NULL;
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/SubnMgt.cpp gen2.2810/utils/src/linux-user/ibdm/datamodel/SubnMgt.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/SubnMgt.cpp	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/SubnMgt.cpp	2005-07-06 14:56:48.000000000 +0200
@@ -323,7 +323,7 @@ SubnMgtCalcMinHopTables (IBFabric *p_fab
   int maxHops = 0;
   IBNode *p_worstHopNode  = NULL;
   IBPort *p_worstHopPort  = NULL;
-  unsigned int worstHopLid;
+  unsigned int worstHopLid = 0;
   IBNode *p_caNode;
   IBPort *p_caPort;
   vec_int maxHopsHist(50,0);
@@ -1389,7 +1389,7 @@ SubnReportNonUpDownCa2CaPaths(
         int numChanges = 0;
         int prevGoingUp = 1, goingUp; 
         int prevRank = 99, rank;
-        IBNode *p_prevNode;
+        IBNode *p_prevNode = NULL;
 
         // Go through the path and check for up down transitions.
         for (list_pnode::iterator lI = path.begin(); lI!= path.end(); lI++) {
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/TopoMatch.cpp gen2.2810/utils/src/linux-user/ibdm/datamodel/TopoMatch.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/TopoMatch.cpp	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/TopoMatch.cpp	2005-07-06 15:14:34.998761395 +0200
@@ -534,7 +534,7 @@ TopoMatchNodeByAdjacentMatched(
   IBNode *p_sNode 
   )
 {
-  int succeedMatch;
+  int succeedMatch = 0;
 
   // This map will hold the number of matching ports for each candidate 
   // discovered node
@@ -648,6 +648,7 @@ TopoMatchSpecNodesByAdjacentNode(
   {
     numMatched += TopoMatchNodeByAdjacentMatched(*nI);
   }
+  return numMatched;
 }
 
 //////////////////////////////////////////////////////////////////////////////
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/TraceRoute.cpp gen2.2810/utils/src/linux-user/ibdm/datamodel/TraceRoute.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/datamodel/TraceRoute.cpp	2005-07-06 11:52:49.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/datamodel/TraceRoute.cpp	2005-07-06 15:01:31.000000000 +0200
@@ -111,7 +111,8 @@ int TraceDRPathRoute (IBPort *p_smNodePo
 			 // calc the next port
 			 outPortNum = *pI - 1;
 			 // we could get stuck on bad port num
-			 if (outPortNum > p_remNode->numPorts || outPortNum < 0) {
+			 // as outPortNum is unsigned it's always >= 0
+			 if (outPortNum > p_remNode->numPorts) {
 				cout << "-E- Bad port number:" << outPortNum << " hop:" << hop << endl;
 				return 1;
 			 }
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/ibdm/src/osm_check.cpp gen2.2810/utils/src/linux-user/ibdm/src/osm_check.cpp
--- gen2.2810.oorig/utils/src/linux-user/ibdm/src/osm_check.cpp	2005-07-06 11:52:44.000000000 +0200
+++ gen2.2810/utils/src/linux-user/ibdm/src/osm_check.cpp	2005-07-06 15:22:07.999098448 +0200
@@ -327,7 +327,6 @@ int main (int argc, char **argv) {
   }
   while(next_option != -1);
 
-  int status = 0;
   IBFabric fabric;	
     
   /* based on the topology file we decide what mode we are: */

From viswak at yahoo.com  Wed Jul  6 10:28:22 2005
From: viswak at yahoo.com (viswanath krishnamurthy)
Date: Wed, 6 Jul 2005 10:28:22 -0700 (PDT)
Subject: [openib-general] Re: useraccess_cm sample client/server (gen1)
Message-ID: <20050706172822.22677.qmail@web33212.mail.mud.yahoo.com>

I looked further into the whole gen1 source tree.
There is no consumer of this useraccess_cm API
(ioctl). Are there any consumers of this API's. Is it
supported ?

Thanks,
Vish

--- viswanath krishnamurthy <viswak at yahoo.com> wrote:

> Is there a sample code (examples) to use the gen1
> stack user level CM API (ioctls) ? Any pointers is
> appreciated.
> 
> Thanks,
> Vish
> 
> 
> 		
> ____________________________________________________
> 
> Yahoo! Sports 
> Rekindle the Rivalries. Sign up for Fantasy Football
> 
> http://football.fantasysports.yahoo.com
> 



		
____________________________________________________
Sell on Yahoo! Auctions – no fees. Bid on great items.  
http://auctions.yahoo.com/


From krause at cup.hp.com  Wed Jul  6 10:27:17 2005
From: krause at cup.hp.com (Michael Krause)
Date: Wed, 06 Jul 2005 10:27:17 -0700
Subject: [openib-general] [iser]about the target
In-Reply-To: <5D78D28F88822E4D8702BB9EEF1A43670A072B@mercury.infiniconsy s.com>
References: <5D78D28F88822E4D8702BB9EEF1A43670A072B@mercury.infiniconsys.com>
Message-ID: <6.2.0.14.2.20050706102430.02a10e60@esmail.cup.hp.com>

At 06:14 AM 7/6/2005, Rimmer, Todd wrote:
>I would like to clarify the comment on SRP.  There are companies presently 
>shipping and demonstrating SRP native IB storage.  For example:
>     Engenio (formerly LSI)
>     Raytheon
>     Data Direct
>     Mellanox
>
>SRP was designed for highly optimized storage access across an RDMA 
>capable transport, and hence is capable of very high performance.
>
>Longer term the storage vendors anticipate that iSCSI will be the focus 
>for long haul (remote backup, etc) type solutions, while IB Native Storage 
>and FC will be the focus for data center high performance storage solutions.

iSCSI within the data center is quite real for many applications especially 
blades where there is already an Ethernet interface.  The reason for iSER 
was to take advantage of the iSCSI ecosystem while providing a RDMA focused 
data mover.  SRP is primarily a data mover and does not define the rest of 
the management, etc. interfaces.  When used to move data to a FC, the FC 
infrastructure is leveraged but SRP by itself does nothing really in this 
regard.  No one stated that SRP could not deliver performance only that the 
rest of the infrastructure is not defined and must rely upon other 
standards / plug-ins / etc.  I do not want to get into a vision / marketing 
debate - was just explaining why we created iSER instead of just enhancing SRP.

Mike

>
>Todd R.
>-----Original Message-----
>From: Michael Krause [mailto:krause at cup.hp.com]
>Sent: Tuesday, July 05, 2005 12:28 PM
>To: Ian Jiang; openib-general at openib.org
>Subject: Re: [openib-general] [iser]about the target
>
>At 06:07 PM 7/4/2005, Ian Jiang wrote:
>>Hi!
>>I am new to the iSER.
>>On " https://openib.org/tiki/tiki-index.php?page=iSER", it is said that 
>>iSER currently contains initiator only (no target). Will the target come 
>>out later? How did they test the iSER initiator without a iSER target?
>>Could you give some explaination?
>
> From a practical perspective, there are very few iSCSI targets shipping 
> today.  Most people had envisioned iSER over IB to a gateway Ethernet 
> device since native IB storage is also quite rare in terms of real 
> product.  For many of us, our push for iSER over IB was to replace SRP 
> which has a deficient ecosystem thus not really used beyond some basic 
> Fibre Channel gateway cards.
>
>Mike
>
>>Thanks!
>>
>>
>>
>>Ian Jiang
>>ianjiang91 at hotmail.com
>>----
>>Computer Architecture Laboratory
>>Institute of Computing Technology
>>Chinese Academy of Sciences
>>Beijing,P.R.China
>>Zip code: 100080
>>Tel: +86-10-62564394(office)
>>
>>_________________________________________________________________
>>ÓëÁª»úµÄÅóÓѽøÐн»Á÷£¬ÇëʹÓà MSN Messenger:  http://messenger.msn.com/cn
>>
>>_______________________________________________
>>openib-general mailing list
>>openib-general at openib.org
>>http://openib.org/mailman/listinfo/openib-general
>>
>>To unsubscribe, please visit 
>>http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050706/fd159e35/attachment.html>

From caitlinb at siliquent.com  Wed Jul  6 10:38:28 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 6 Jul 2005 10:38:28 -0700
Subject: [openib-general] OpenIB DAT_Return merging and kdat.h
Message-ID: <8508251A6FC08A489844A94261D3693A057D0D@fiona.siliquent.com>

Eliminating the DAT_RETURN detail information is a strange
choice, but certainly within the realm of options available
to any implementation. No DAPL consumer should rely on the
more detailed information. It is strange because it may
encourage people to code from uDAPL in order to get more
detailed diagnostic information while generally you would
expect a kernel client to have access to more detailed info.

However the proposal to merge some of the major error codes
would make it impossible for a wrapper include file (such as
kdat.h) to restore the full API for use by insmodded clients.

Given both of those factors, I believe it makes more sense
to preserve the full DAT_RETURN value (including detail error
data) as an out parameter. kdat.h could then simply turn it 
into the functional return for compatability clients and there
would be no loss of information. Either that or we could work
out on a case by case basis how kdat.h would be able to restore
the original error value. This may be possible with data in
the struct, since multiple outstanding errors on a single 
object would not seem to be likely (or perhaps more appropriately
worthy of providing detailed error diagnostics for, since it
probably represents a totally hosed application).



From rolandd at cisco.com  Wed Jul  6 10:38:46 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 06 Jul 2005 10:38:46 -0700
Subject: [openib-general] Re: [PATCH] signal driven io support for ib_uverbs
In-Reply-To: <20050706121711.GG12680@minantech.com> (Gleb Natapov's message
	of "Wed, 6 Jul 2005 15:17:11 +0300")
References: <20050706121711.GG12680@minantech.com>
Message-ID: <52wto36di1.fsf@topspin.com>

Thanks for adding this.  A few questions:

    > +static int ib_uverbs_event_async (int fd, struct file *filp, int on)
    > +{
    > +	int ret;
    > +	struct ib_uverbs_event_file *file = filp->private_data;
    > +
    > +	if (!file)
    > +		return -EBADFD;

Can this function ever be called on a file with private_data not set?
Why do we need this check?

    > +
    > +	if ((ret = fasync_helper(fd, filp, on, &file->fasync)) < 0)
    > +		return ret;
    > +
    > +	if (on)
    > +		ret = f_setown(filp, current->pid, 0);

Why do we need the f_setown?  Doesn't the core kernel already handle
this for us?

    > +	kill_fasync (&file->async_file.fasync, SIGIO, POLL_IN);
 
We want this in both ib_uverbs_comp_handler() and also
ib_uverbs_async_handler() too, right?

 - R.


From rolandd at cisco.com  Wed Jul  6 10:46:45 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 06 Jul 2005 10:46:45 -0700
Subject: [openib-general] [PATCH v3] user_mad: Add receive side RMPP
	support
In-Reply-To: <1120242686.4371.3804.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "01 Jul 2005 14:31:26 -0400")
References: <1120242686.4371.3804.camel@hal.voltaire.com>
Message-ID: <52oe9f6d4q.fsf@topspin.com>

This looks good. Go ahead and check it in when the userspace side is
ready.

 - R.


From rolandd at cisco.com  Wed Jul  6 10:47:28 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 06 Jul 2005 10:47:28 -0700
Subject: [openib-general] Re: [PATCH v4] sa_query: Add service record support
In-Reply-To: <1120598014.4388.10396.camel@hal.voltaire.com> (Hal
	Rosenstock's message of "05 Jul 2005 17:13:35 -0400")
References: <1120598014.4388.10396.camel@hal.voltaire.com>
Message-ID: <52k6k36d3j.fsf@topspin.com>

This looks good as well.  Go ahead and check it in.

 - R.


From mst at mellanox.co.il  Wed Jul  6 10:59:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 20:59:49 +0300
Subject: [openib-general] [PATCH] sdp_conn_put/sdp_conn_hold race
Message-ID: <20050706175949.GA8783@mellanox.co.il>

The current sdp_conn_put/sdp_conn_hold implementation
seems to be subject to the following race condition:

- thread A calls sdp_conn_put, atomic dec and test returns 0
- thread B looks up the connection and calls sdp_conn_get,
  incrementing the reference count back to 1
- thread A now goes on to call sdp_conn_destroy, which will
  destroy the connection
- thread B is left with an invalid connection pointer 

As a solution
- sdp_conn_put is moved out of line. checking connection
  reference count is done under the connection table spinlock
  atomically, removing the connection from lookup table
  if the reference drops to 0.

- a new call sdp_conn_put_light is provided for when we
  know this isn't the last reference to the connection.
  This is useful not only for performance reasons, but 
  also for documentation/code clarity purposes: when
  sdp_conn_put_light is used, this isn't the last reference,
  when sdp_conn_put is used, this may be the last reference.
  
Patches follow.
Please comment. 

-- 
MST


From mst at mellanox.co.il  Wed Jul  6 11:02:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 21:02:33 +0300
Subject: [openib-general] [PATCH 1/2] sdp_conn_put/sdp_conn_hold race
	condition
Message-ID: <20050706180233.GA8950@mellanox.co.il>

Protect sdp_conn_put by dev_root_s.sock_lock.
Add sdp_conn_put_light for when we know its not the last reference.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/sdp/sdp_conn.c
===================================================================
--- ulp/sdp/sdp_conn.c	(revision 2784)
+++ ulp/sdp/sdp_conn.c	(working copy)
@@ -595,12 +595,6 @@ static int sdp_conn_table_insert(struct 
 int sdp_conn_table_remove(struct sdp_sock *conn)
 {
 	int result = 0;
-	unsigned long flags;
-
-	/*
-	 * lock table
-	 */
-	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * validate entry
 	 */
@@ -619,10 +613,6 @@ int sdp_conn_table_remove(struct sdp_soc
 	conn->hashent = SDP_DEV_SK_INVALID;
 
 done:
-	/*
-	 * unlock table
-	 */
-	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return result;
 }
 
@@ -706,13 +696,30 @@ void sdp_iocb_q_cancel_all(struct sdp_so
  */
 
 /*
- * sdp_conn_destruct - final destructor for connection.
+ * sdp_conn_put - reference counting and final destructor for connection.
  */
-void sdp_conn_destruct(struct sdp_sock *conn)
+void sdp_conn_put(struct sdp_sock *conn)
 {
+	unsigned long flags;
 	int dump = 0;
 	int result;
 
+
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
+
+	if (!atomic_dec_and_test(&conn->refcnt)) {
+		spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
+		return;
+	}
+
+	result = sdp_conn_table_remove(conn);
+	if (result < 0)
+		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
+			     result, dev_root_s.sk_entry,
+			     dev_root_s.sk_size);
+
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
+
 	sdp_dbg_ctrl(conn, "DESTRUCT. <%08x:%04x> <%08x:%04x> <%u:%u>",
 		     conn->src_addr, conn->src_port,
 		     conn->dst_addr, conn->dst_port,
@@ -721,14 +728,6 @@ void sdp_conn_destruct(struct sdp_sock *
 	 * If the socket is bound, return the port
 	 */
 	(void)sdp_inet_port_put(conn);
-	/*
-	 * remove connection from table
-	 */
-	result = sdp_conn_table_remove(conn);
-	if (result < 0)
-		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
-			     result, dev_root_s.sk_entry,
-			     dev_root_s.sk_size);
 
 	sdp_conn_stat_dump(conn);
 	/*
Index: ulp/sdp/sdp_conn.h
===================================================================
--- ulp/sdp/sdp_conn.h	(revision 2784)
+++ ulp/sdp/sdp_conn.h	(working copy)
@@ -431,7 +431,6 @@ struct sdp_sock {
 extern void sdp_conn_internal_lock(struct sdp_sock *conn, unsigned long *flags);
 extern void sdp_conn_internal_unlock(struct sdp_sock *conn);
 extern void sdp_conn_relock(struct sdp_sock *conn);
-extern void sdp_conn_destruct(struct sdp_sock *conn);
 extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_sock *conn);
 
 #define SDP_CONN_LOCK_IRQ(conn, flags) \
@@ -479,12 +478,17 @@ static inline void sdp_conn_hold(struct 
 	atomic_inc(&conn->refcnt);
 }
 
-static inline void sdp_conn_put(struct sdp_sock *conn)
+/*
+ * safe to call if there's another reference on connection.
+ */
+static inline void sdp_conn_put_light(struct sdp_sock *conn)
 {
-	if (atomic_dec_and_test(&conn->refcnt))
-		sdp_conn_destruct(conn);
+	int noref = atomic_dec_and_test(&conn->refcnt);
+	BUG_ON(noref);
 }
 
+void sdp_conn_put(struct sdp_sock *conn);
+
 static inline void *hashent_arg(s32 hashent) 
 {
 	return (void *)(unsigned long)hashent;

-- 
MST


From mst at mellanox.co.il  Wed Jul  6 11:03:32 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 21:03:32 +0300
Subject: [openib-general] [PATCH 2/2] sdp_conn_put/sdp_conn_hold race
	condition
Message-ID: <20050706180332.GA8965@mellanox.co.il>

Use sdp_conn_put_light in cases when we know there is another
reference on the connection.
Rule: sdp_conn_put should never be called on a locked connection:
since its safe to sdp_conn_unlock, this is not the last reference.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/sdp/sdp_actv.c
===================================================================
--- ulp/sdp/sdp_actv.c	(revision 2784)
+++ ulp/sdp/sdp_actv.c	(working copy)
@@ -325,7 +325,7 @@ error:
 
 	if (conn->state == SDP_CONN_ST_CLOSED) {
 		conn->cm_id = NULL;
-		sdp_conn_put(conn); /* CM reference */
+		sdp_conn_put_light(conn); /* CM reference */
 
 		return -EPROTO;
 	}
@@ -547,7 +547,7 @@ int sdp_cm_connect(struct sdp_sock *conn
 		/*
 		 * callback dosn't have this socket.
 		 */
-		sdp_conn_put(conn); /* address resolution reference */
+		sdp_conn_put_light(conn); /* address resolution reference */
 
 		return -EDESTADDRREQ;
 	}
Index: ulp/sdp/sdp_event.c
===================================================================
--- ulp/sdp/sdp_event.c	(revision 2784)
+++ ulp/sdp/sdp_event.c	(working copy)
@@ -187,7 +187,7 @@ static void sdp_cm_to_error(struct sdp_s
 {
 	sdp_conn_inet_error(conn, -ECONNRESET);
 	conn->cm_id = NULL;
-	sdp_conn_put(conn);	/* CM reference */
+	sdp_conn_put_light(conn);	/* CM reference */
 }
 /*
  * Connection establishment IB/CM callback functions
@@ -228,7 +228,7 @@ static int sdp_cm_idle(struct ib_cm_id *
 	}
 
 	conn->cm_id = NULL;
-	sdp_conn_put(conn); /* CM reference */
+	sdp_conn_put_light(conn); /* CM reference */
 	return -ENOENT;     /* ensure CM cleans-up identifier. */
 }
 

-- 
MST


From halr at voltaire.com  Wed Jul  6 11:00:42 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 06 Jul 2005 14:00:42 -0400
Subject: [openib-general] Re: [PATCH v4] sa_query: Add service record support
In-Reply-To: <52k6k36d3j.fsf@topspin.com>
References: <1120598014.4388.10396.camel@hal.voltaire.com>
	<52k6k36d3j.fsf@topspin.com>
Message-ID: <1120672648.16533.1458.camel@hal.voltaire.com>

On Wed, 2005-07-06 at 13:47, Roland Dreier wrote:
> This looks good as well.  Go ahead and check it in.

Thanks. Applied.

-- Hal



From halr at voltaire.com  Wed Jul  6 12:03:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 06 Jul 2005 15:03:23 -0400
Subject: [openib-general] [PATCH v3] user_mad: Add receive side RMPP
	support
In-Reply-To: <52oe9f6d4q.fsf@topspin.com>
References: <1120242686.4371.3804.camel@hal.voltaire.com>
	<52oe9f6d4q.fsf@topspin.com>
Message-ID: <1120676602.16533.1501.camel@hal.voltaire.com>

On Wed, 2005-07-06 at 13:46, Roland Dreier wrote:
> This looks good. Go ahead and check it in when the userspace side is
> ready.

Thanks. Applied.

WARNING:
Note that this is another flag day (ABI_VERSION change for libibumad so
OpenSM and diagnostics need updating and rebulding.

r2814 is the kernel change for this.
r2815 is the user space change for this.

-- Hal



From halr at voltaire.com  Wed Jul  6 12:56:34 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 06 Jul 2005 15:56:34 -0400
Subject: [openib-general] Re: documentation under src dir
In-Reply-To: <20050705072433.GI14130@mellanox.co.il>
References: <20050705072433.GI14130@mellanox.co.il>
Message-ID: <1120679793.16533.1576.camel@hal.voltaire.com>

On Tue, 2005-07-05 at 03:24, Michael S. Tsirkin wrote:
> Hal, Shahar, could you please move the following files
> outside the src directory?
> 
> ./trunk/src/userspace/management/osm/doc/OpenSM_RN.pdf
> ./trunk/src/userspace/management/osm/doc/OpenSM_UM.pdf
> 
> These are not sources, so they dont belong in src.
> I suggest removing OpenSM_RN.pdf and moving OpenSM_UM.pdf to the docs
> section on the web site.

Other userspace tarballs include the documentation under a doc
subdirectory. Guess the inconsistency here is that it is under 
src/userspace/(management/osm). I'm not sure what the best way to handle
this is. One option is as you suggest: move these elsewhere; a second to
leave this alone; and a third to restructure the naming of the tree so
this is not a problem. If they are to be moved elsewhere, would we
create a doc/userspace/... ?

-- Hal



From halr at voltaire.com  Wed Jul  6 13:56:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 06 Jul 2005 16:56:25 -0400
Subject: [openib-general] [PATCH] MAD RMPP: Minor build changes
Message-ID: <1120683383.16533.1631.camel@hal.voltaire.com>

MAD RMPP: Minor build changes

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: mad_rmpp.h
===================================================================
--- mad_rmpp.h	(revision 2815)
+++ mad_rmpp.h	(working copy)
@@ -33,10 +33,8 @@
  */
 
 #ifndef __MAD_RMPP_H__
-#define __MAD_PRIV_H__
+#define __MAD_RMPP_H__
 
-#include "mad_priv.h"
-
 enum {
 	IB_RMPP_RESULT_PROCESSED,
 	IB_RMPP_RESULT_CONSUMED,
Index: mad_rmpp.c
===================================================================
--- mad_rmpp.c	(revision 2815)
+++ mad_rmpp.c	(working copy)
@@ -35,8 +35,8 @@
 
 #include <linux/dma-mapping.h>
 
+#include "mad_priv.h"
 #include "mad_rmpp.h"
-#include "mad_priv.h"
 
 enum rmpp_state {
 	RMPP_STATE_ACTIVE,





From rolandd at cisco.com  Wed Jul  6 14:35:03 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 06 Jul 2005 14:35:03 -0700
Subject: [openib-general] Re: [PATCH] signal driven io support for ib_uverbs
In-Reply-To: <20050706121711.GG12680@minantech.com> (Gleb Natapov's message
	of "Wed, 6 Jul 2005 15:17:11 +0300")
References: <20050706121711.GG12680@minantech.com>
Message-ID: <52wto34nzs.fsf@topspin.com>

How does this patch look?

Do you have a test program you're using to check SIGIO-based event
handling?  If so can you post it?

Thanks,
  Roland

--- infiniband/core/uverbs.h	(revision 2815)
+++ infiniband/core/uverbs.h	(working copy)
@@ -62,6 +62,7 @@ struct ib_uverbs_event_file {
 	int					fd;
 	int					is_async;
 	wait_queue_head_t			poll_wait;
+	struct fasync_struct		       *async_queue;
 	struct list_head			event_list;
 };
 
--- infiniband/core/uverbs_main.c	(revision 2815)
+++ infiniband/core/uverbs_main.c	(working copy)
@@ -258,11 +258,19 @@ static void ib_uverbs_event_release(stru
 	spin_unlock_irq(&file->lock);
 }
 
+static int ib_uverbs_event_fasync(int fd, struct file *filp, int on)
+{
+	struct ib_uverbs_event_file *file = filp->private_data;
+
+	return fasync_helper(fd, filp, on, &file->async_queue);
+}
+
 static int ib_uverbs_event_close(struct inode *inode, struct file *filp)
 {
 	struct ib_uverbs_event_file *file = filp->private_data;
 
 	ib_uverbs_event_release(file);
+	ib_uverbs_event_fasync(-1, filp, 0);
 	kref_put(&file->uverbs_file->ref, ib_uverbs_release_file);
 
 	return 0;
@@ -277,7 +285,8 @@ static struct file_operations uverbs_eve
 	 */
 	.read 	 = ib_uverbs_event_read,
 	.poll    = ib_uverbs_event_poll,
-	.release = ib_uverbs_event_close
+	.release = ib_uverbs_event_close,
+	.fasync  = ib_uverbs_event_fasync
 };
 
 void ib_uverbs_comp_handler(struct ib_cq *cq, void *cq_context)
@@ -297,6 +306,7 @@ void ib_uverbs_comp_handler(struct ib_cq
 	spin_unlock_irqrestore(&file->comp_file[0].lock, flags);
 
 	wake_up_interruptible(&file->comp_file[0].poll_wait);
+	kill_fasync(&file->comp_file[0].async_queue, SIGIO, POLL_IN);
 }
 
 static void ib_uverbs_async_handler(struct ib_uverbs_file *file,
@@ -317,6 +327,7 @@ static void ib_uverbs_async_handler(stru
 	spin_unlock_irqrestore(&file->async_file.lock, flags);
 
 	wake_up_interruptible(&file->async_file.poll_wait);
+	kill_fasync(&file->async_file.async_queue, SIGIO, POLL_IN);
 }
 
 void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr)
@@ -351,6 +362,7 @@ static int ib_uverbs_event_init(struct i
 	INIT_LIST_HEAD(&file->event_list);
 	init_waitqueue_head(&file->poll_wait);
 	file->uverbs_file = uverbs_file;
+	file->async_queue = NULL;
 
 	file->fd = get_unused_fd();
 	if (file->fd < 0)


From mst at mellanox.co.il  Wed Jul  6 15:08:02 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 01:08:02 +0300
Subject: [openib-general] Re: documentation under src dir
In-Reply-To: <1120679793.16533.1576.camel@hal.voltaire.com>
References: <20050705072433.GI14130@mellanox.co.il>
	<1120679793.16533.1576.camel@hal.voltaire.com>
Message-ID: <20050706220802.GA9524@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: Re: documentation under src dir
> 
> On Tue, 2005-07-05 at 03:24, Michael S. Tsirkin wrote:
> > Hal, Shahar, could you please move the following files
> > outside the src directory?
> > 
> > ./trunk/src/userspace/management/osm/doc/OpenSM_RN.pdf
> > ./trunk/src/userspace/management/osm/doc/OpenSM_UM.pdf
> > 
> > These are not sources, so they dont belong in src.
> > I suggest removing OpenSM_RN.pdf and moving OpenSM_UM.pdf to the docs
> > section on the web site.
> 
> Other userspace tarballs include the documentation under a doc
> subdirectory.

pdf? I think its uncommon. where?

It seems to me that source tarballs usually dont include docs in noneditable
format. In my opinion, documentation that isnt in plain text, sgml, or
something like that doesnt belong in src (pdfs can be compiled on demand).

> Guess the inconsistency here is that it is under 
> src/userspace/(management/osm).
> I'm not sure what the best way to handle
> this is.

I think OpenSM_RN.pdf should go since its just release notes
for 1.6.1 opensm from gold cd.
OpenSM_UM.pdf should ideally converted to text or sgml or something
like this so we can generate standard man/info pages.
Short term, I'd advocate moving it to the web.

> One option is as you suggest: move these elsewhere; a second to
> leave this alone; and a third to restructure the naming of the tree so
> this is not a problem.
> If they are to be moved elsewhere, would we
> create a doc/userspace/... ?

Any way that doesnt force me to pull binaries on each
checkout works for me.

> -- Hal
> 


-- 
MST


From surs at cse.ohio-state.edu  Wed Jul  6 16:46:47 2005
From: surs at cse.ohio-state.edu (Sayantan Sur)
Date: Wed, 6 Jul 2005 19:46:47 -0400
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <52pstyvw99.fsf@topspin.com>
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
Message-ID: <20050706234645.GA32470@cse.ohio-state.edu>

Roland and Michael,

* On Jul,4 Roland Dreier<rolandd at cisco.com> wrote :
>     Sayantan> I am facing a compile problem with the following error:
> 
>     Sayantan> drivers/infiniband/core/uverbs_main.c: In function
>     Sayantan> `ib_uverbs_add_one':
>     Sayantan> drivers/infiniband/core/uverbs_main.c:578: error:
>     Sayantan> structure has no member named `devt'
> 
> Sorry, I checked in a change that requires a 2.6.13-rc1 kernel.  I've
> reverted the change, so the tree should compile against 2.6.12 again.

Thanks for the information. I was able to compile the kernel components
against 2.6.12. On installing the entire OpenIB stack, I am getting an
error like this:

[surs at k20-as4:~] ibstat
CA 'mthca0'
        CA type: MT23108
        Number of ports: 2
        Firmware version: 3.3.2
        Hardware version: a1
        Node GUID: 0x0005ad0000013974
        System image GUID: 0x0005ad000100d050
        Port 1:
                State: Down
                Physical state: Disabled <--- What does this mean?
                Rate: 2
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0005ad0000013975
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 2
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0005ad0000013976

The system is based on RH AS4. The machine is a Xeon. If you need any
more information, please let me know.

Thanks,
Sayantan.


> 
>  - R.

-- 
http://www.cse.ohio-state.edu/~surs


From halr at voltaire.com  Wed Jul  6 17:34:29 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 06 Jul 2005 20:34:29 -0400
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <20050706234645.GA32470@cse.ohio-state.edu>
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
Message-ID: <1120696182.16533.1711.camel@hal.voltaire.com>

On Wed, 2005-07-06 at 19:46, Sayantan Sur wrote:
> Thanks for the information. I was able to compile the kernel components
> against 2.6.12. On installing the entire OpenIB stack, I am getting an
> error like this:
> 
> [surs at k20-as4:~] ibstat
> CA 'mthca0'
>         CA type: MT23108
>         Number of ports: 2
>         Firmware version: 3.3.2
>         Hardware version: a1
>         Node GUID: 0x0005ad0000013974
>         System image GUID: 0x0005ad000100d050
>         Port 1:
>                 State: Down
>                 Physical state: Disabled <--- What does this mean?
>                 Rate: 2
>                 Base lid: 0
>                 LMC: 0
>                 SM lid: 0
>                 Capability mask: 0x00500a68
>                 Port GUID: 0x0005ad0000013975
>         Port 2:
>                 State: Down
>                 Physical state: Disabled
>                 Rate: 2
>                 Base lid: 0
>                 LMC: 0
>                 SM lid: 0
>                 Capability mask: 0x00500a68
>                 Port GUID: 0x0005ad0000013976
> 
> The system is based on RH AS4. The machine is a Xeon. If you need any
> more information, please let me know.

Physical state disabled means a command to disable the physical link
state machine was somehow given. That means the physical link won't come
up and hence the logical link won't either. Not sure what would cause
this.

-- Hal





From athabascan at yahoo.com  Wed Jul  6 22:47:53 2005
From: athabascan at yahoo.com (Debbie Hopper)
Date: Thu, 7 Jul 2005 02:47:53 -0300
Subject: [openib-general] FW: FW: your private invitation Je-34293TH.
Message-ID: <9330351508.3386687@decry.handwritten.uri.edu>

Hello,

You have been chosen to participate in an invitation only limited time event! 
Are you currently paying over 3% for your mortgage? STOP! We can help you lower that today!
Answer only a few questions and we can give you an approval in under 30 seconds � it�s that simple! 

http://FYd.pra1s3.net/p2.asp

And stop fighting for lenders � let them fight for you! Make them work for your business by giving you the lowest rates around!
$230,000 loans are available for only $340/month! WE�RE PRACTICALLY GIVING AWAY MONEY!

Think your credit is too bad to get a deal like this? THINK AGAIN! We will have you saving your money in no time!
Are you ready to save your money? http://d4Mo.pra1s3.net/p2.asp

To update our records go pra1s3.com/deletion.asp, database updates can take up to 72 hours.

Regards,
Debbie Hopper



From glebn at voltaire.com  Wed Jul  6 23:29:25 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 09:29:25 +0300
Subject: [openib-general] Re: [PATCH] signal driven io support for ib_uverbs
In-Reply-To: <52wto36di1.fsf@topspin.com>
References: <20050706121711.GG12680@minantech.com> <52wto36di1.fsf@topspin.com>
Message-ID: <20050707062925.GI12680@minantech.com>

On Wed, Jul 06, 2005 at 10:38:46AM -0700, Roland Dreier wrote:
> Thanks for adding this.  A few questions:
> 
>     > +static int ib_uverbs_event_async (int fd, struct file *filp, int on)
>     > +{
>     > +	int ret;
>     > +	struct ib_uverbs_event_file *file = filp->private_data;
>     > +
>     > +	if (!file)
>     > +		return -EBADFD;
> 
> Can this function ever be called on a file with private_data not set?
> Why do we need this check?
I don't think it can, but it is slow path an check doesn't hurt.
> 
>     > +
>     > +	if ((ret = fasync_helper(fd, filp, on, &file->fasync)) < 0)
>     > +		return ret;
>     > +
>     > +	if (on)
>     > +		ret = f_setown(filp, current->pid, 0);
> 
> Why do we need the f_setown?  Doesn't the core kernel already handle
> this for us?
Same here.

> 
>     > +	kill_fasync (&file->async_file.fasync, SIGIO, POLL_IN);
>  
> We want this in both ib_uverbs_comp_handler() and also
> ib_uverbs_async_handler() too, right?
Yes, of cause. I am using signals for async events handling so I missed this
one.

--
			Gleb.


From eitan at mellanox.co.il  Thu Jul  7 00:15:15 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 7 Jul 2005 10:15:15 +0300 
Subject: [openib-general] RE: [PATCH] ibdm nits and a few questions
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30446@mtlex01.yok.mtl.com>

Hi Ber,

Thanks for the patch and the word list. I will apply these right away.

> 
> What about removing all the generated scripts (configure et al) and
> auto-tools helpers (aclocal.m4 etc and most of config/*) from the
> repository?
[EZ] I thought I have done that. I will double check.

> Note that the other userspace IB libraries do not have these in the
> repository but each do provide a toplevel ./autogen.sh -- as opposed
> to ibdm's ./config/bootstrap.
[EZ] We can rename the bootstrap autogen.sh but from what I learned about
auto tools many use ./config/bootstrap ...
Is it really annoying?
> 
> 
> There are a number of other unused variables.
[EZ] We should fix all of those! 
> 
> Is ibdm supposed to be compilable with pure ISO c++ compilers, btw?
> There is at least one place which uses a variable sized array which AFAIK
> is not valid ISO c++.
[EZ] I'm not sure what the answer to that is.
> 
> IBMgtSim doesn't compile for me as iba/ib_types.h isn't installed, so
[EZ] You are missing OpenSM installation on the machine. It is required ... 
You can use --with-osm to point to where it is installed ...

> the above is only compile-tested.
[EZ] I will then take the patch manually into the code... (one at a time)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050707/5af2facd/attachment.html>

From mst at mellanox.co.il  Thu Jul  7 01:31:48 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 11:31:48 +0300
Subject: [openib-general] Re: [PATCH] ibdm nits and a few questions
In-Reply-To: <20050706160222.GA13817@aon.at>
References: <506C3D7B14CDD411A52C00025558DED607C3041C@mtlex01.yok.mtl.com>
	<20050706160222.GA13817@aon.at>
Message-ID: <20050707083148.GQ14130@mellanox.co.il>

Quoting r. Bernhard Fischer <rep.nop at aon.at>:
> Is ibdm supposed to be compilable with pure ISO c++ compilers, btw?
If you care about this, compile with -pedantic.

> There is at least one place which uses a variable sized array which AFAIK
> is not valid ISO c++.

I think Bernhard is right, it isnt, VLA is a C99 feature but not a C++ feature.
http://www.open-std.org/jtc1/sc22/open/n2356/decl.html

For other incompatibilities between ISO C and ISO C++, see
http://david.tribble.com/text/cdiffs.htm#C99-vla

~>cat > xxx.c
void foo(int argc)
{
        int foo[argc];
}

~>g++ -c -pedantic xxx.c
xxx.c: In function `void foo(int)':
xxx.c:3: warning: ISO C++ forbids variable-size array `foo'

-- 
MST


From glebn at voltaire.com  Thu Jul  7 02:07:47 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 12:07:47 +0300
Subject: [openib-general] Re: [PATCH] signal driven io support for ib_uverbs
In-Reply-To: <52wto34nzs.fsf@topspin.com>
References: <20050706121711.GG12680@minantech.com> <52wto34nzs.fsf@topspin.com>
Message-ID: <20050707090747.GA12222@minantech.com>

On Wed, Jul 06, 2005 at 02:35:03PM -0700, Roland Dreier wrote:
> How does this patch look?
Actually I take may words about f_setown() back. Kernel dose not
set it by default. So unless you set it in the kernel or from
userspace using fcntl(F_SETOWN) you don't receive SIGIO signal.

The question is what is the right way? Do it from the kernel or
left it to userspace? Parts of the kernel do f_setown() from the
kernel others don't.

Other then that the patch is working OK.

> 
> Do you have a test program you're using to check SIGIO-based event
> handling?  If so can you post it?
Actually I am using it as part of MPI implementation, but I've crafted
patch for asyncwatch.c to use signals. Try to comment F_SETOWN bit,
you wont get events.


Index: libibverbs/examples/asyncwatch.c
===================================================================
--- libibverbs/examples/asyncwatch.c	(revision 2810)
+++ libibverbs/examples/asyncwatch.c	(working copy)
@@ -39,6 +39,14 @@
 #include <stdio.h>
 #include <endian.h>
 #include <byteswap.h>
+#include <signal.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <stdlib.h>
+#include <getopt.h>
+#include <sys/types.h>
+#include <unistd.h>
+
 
 #include <infiniband/verbs.h>
 
@@ -48,12 +56,120 @@ static inline uint64_t be64_to_cpu(uint6
 static inline uint64_t be64_to_cpu(uint64_t x) { return x; }
 #endif
 
+struct ibv_context *context;
+
+void async_sigaction (int sig, siginfo_t *siginfo, void *ign)
+{
+        int ret;
+        struct ibv_async_event event;
+
+        if (siginfo->si_code != POLL_IN)
+        {
+                fprintf (stderr, "siginfo->si_code == %d\n", siginfo->si_code);
+                return;
+        }
+
+        if (siginfo->si_fd != context->async_fd)
+        {
+                fprintf (stderr, "siginfo->si_fd != context->async_fd\n");
+                return;
+        }
+
+        while(!(ret = ibv_get_async_event (context, &event)))
+        {
+		printf("  event_type %d, port %d\n", event.event_type,
+		       event.element.port_num);
+        }
+
+        if (errno == EAGAIN) /* no more events to process */
+                return;
+
+        /* other error */
+	fprintf (stderr, "Error in ibv_get_async_event\n");
+	exit (1);
+        return;
+}
+
+void async_event_setup (void)
+{
+	struct sigaction act;
+	int fdflags;
+
+	act.sa_sigaction = async_sigaction;
+	sigemptyset (&act.sa_mask);
+	act.sa_flags = SA_RESTART | SA_SIGINFO;
+	act.sa_restorer = NULL;
+
+	siginterrupt (SIGIO, 0); /* do not interrupt systemcalls */
+	if (sigaction (SIGIO, &act, NULL) != 0) {
+		perror ("sigaction");
+		exit (1);
+	}
+
+	fdflags = fcntl (context->async_fd, F_GETFL);
+
+	if (fdflags == -1) {
+		perror ("fcntl (F_GETFL)");
+		exit (1);
+	}
+
+	if (fcntl (context->async_fd, F_SETFL,
+				fdflags | O_NONBLOCK | O_ASYNC) == -1) {
+		perror ("fcntl (F_ETFL)");
+		exit (1);
+	}
+
+	if (fcntl (context->async_fd, F_SETSIG, SIGIO) == -1) {
+		perror ("fcntl (F_SETSIG)");
+		exit (1);
+	}
+
+	if (fcntl (context->async_fd, F_SETOWN, getpid()) == -1) {
+		perror ("fcntl (F_SETOWN)");
+		exit (1);
+	}
+}
+
+static void usage(const char *argv0)
+{
+	printf("Usage:\n");
+	printf("  %s            wait for asynchronous events\n", argv0);
+        printf("\n");
+        printf("Options:\n");
+        printf("  -s, --sig      wait for events using signals\n");
+}
+
 int main(int argc, char *argv[])
 {
 	struct dlist *dev_list;
 	struct ibv_device *ib_dev;
-	struct ibv_context *context;
 	struct ibv_async_event event;
+	int sig = 0;
+
+	while (1)
+	{
+		int c;
+
+		static struct option long_options[] = {
+			{ .name = "sig",    .has_arg = 0, .val = 's' },
+			{ 0 }
+		};
+
+		c = getopt_long(argc, argv, "s", long_options, NULL);
+
+		if (c == -1)
+			break;
+
+		switch (c)
+		{
+		case 's':
+			sig = 1;
+			break;
+		default:
+			usage (argv[0]);
+			return 1;
+		}
+	}
 
 	dev_list = ibv_get_devices();
 
@@ -75,12 +191,20 @@ int main(int argc, char *argv[])
 	printf("%s: async event FD %d\n",
 	       ibv_get_device_name(ib_dev), context->async_fd);
 
+	if (sig)
+		async_event_setup();
+
 	while (1) {
-		if (ibv_get_async_event(context, &event))
-			return 1;
 
-		printf("  event_type %d, port %d\n", event.event_type,
-		       event.element.port_num);
+		if (sig) {
+			pause ();
+		} else {
+			if (ibv_get_async_event(context, &event))
+				return 1;
+
+			printf("  event_type %d, port %d\n", event.event_type,
+		       	event.element.port_num);
+		}
 	}
 
 	return 0;
 
--
			Gleb.


From halr at voltaire.com  Thu Jul  7 03:42:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 06:42:39 -0400
Subject: [openib-general] Re: documentation under src dir
In-Reply-To: <20050706220802.GA9524@mellanox.co.il>
References: <20050705072433.GI14130@mellanox.co.il>
	<1120679793.16533.1576.camel@hal.voltaire.com>
	<20050706220802.GA9524@mellanox.co.il>
Message-ID: <1120732958.16533.1759.camel@hal.voltaire.com>

On Wed, 2005-07-06 at 18:08, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > Subject: Re: documentation under src dir
> > 
> > On Tue, 2005-07-05 at 03:24, Michael S. Tsirkin wrote:
> > > Hal, Shahar, could you please move the following files
> > > outside the src directory?
> > > 
> > > ./trunk/src/userspace/management/osm/doc/OpenSM_RN.pdf
> > > ./trunk/src/userspace/management/osm/doc/OpenSM_UM.pdf
> > > 
> > > These are not sources, so they dont belong in src.
> > > I suggest removing OpenSM_RN.pdf and moving OpenSM_UM.pdf to the docs
> > > section on the web site.
> > 
> > Other userspace tarballs include the documentation under a doc
> > subdirectory.
> 
> pdf? I think its uncommon. where?

It may be uncommon but it is done.
One example is subversion:
subversion-1.1.0/doc/book/book/svn-book.pdf

> It seems to me that source tarballs usually dont include docs in noneditable
> format. In my opinion, documentation that isnt in plain text, sgml, or
> something like that doesnt belong in src (pdfs can be compiled on demand).
> 
> > Guess the inconsistency here is that it is under 
> > src/userspace/(management/osm).
> > I'm not sure what the best way to handle
> > this is.
> 
> I think OpenSM_RN.pdf should go since its just release notes
> for 1.6.1 opensm from gold cd.

It (1.6.1) is also the basis for the current version of the OpenIB
OpenSM and so is pertinent here. There have been some bug fixes and
incorporation of functions beyond this (but I'm not sure they affect the
list in the release notes).

> OpenSM_UM.pdf should ideally converted to text or sgml or something
> like this so we can generate standard man/info pages.
> Short term, I'd advocate moving it to the web.

Ideally both of these would be converted and made into OpenIB documents.
I don't have the sources nor have I had the time to chase this down.

> > One option is as you suggest: move these elsewhere; a second to
> > leave this alone; and a third to restructure the naming of the tree so
> > this is not a problem.
> > If they are to be moved elsewhere, would we
> > create a doc/userspace/... ?
> 
> Any way that doesnt force me to pull binaries on each
> checkout works for me.

-- Hal



From rep.nop at aon.at  Thu Jul  7 04:11:02 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Thu, 7 Jul 2005 13:11:02 +0200
Subject: [openib-general] [PATCH][osm] install iba include
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30446@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30446@mtlex01.yok.mtl.com>
Message-ID: <20050707111102.GA17593@aon.at>

On Thu, Jul 07, 2005 at 10:15:15AM +0300, Eitan Zahavi wrote:

>> IBMgtSim doesn't compile for me as iba/ib_types.h isn't installed, so
>[EZ] You are missing OpenSM installation on the machine. It is required ... 

I did have opensm installed.

>You can use --with-osm to point to where it is installed ...


- install iba/ib_types.h which is needed for IBMgtSim.

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
-------------- next part --------------
Index: trunk/src/userspace/management/Makefile
===================================================================
--- trunk/src/userspace/management/Makefile	(revision 2810)
+++ trunk/src/userspace/management/Makefile	(working copy)
@@ -1,7 +1,7 @@
 #LIBS:=$(wildcard lib*)
 LIBS:=libibcommon libibumad libibmad
 OSM:=osm
-OSMLIBS:=complib libvendor
+OSMLIBS:= include complib libvendor
 UTIL:=$(wildcard util/*)
 DIAG:=$(wildcard diags/*)
 

From rep.nop at aon.at  Thu Jul  7 06:59:52 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Thu, 7 Jul 2005 15:59:52 +0200
Subject: [openib-general] Re: [PATCH] ibdm nits and a few questions
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30446@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30446@mtlex01.yok.mtl.com>
Message-ID: <20050707135952.GB17593@aon.at>

On Thu, Jul 07, 2005 at 10:15:15AM +0300, Eitan Zahavi wrote:
>Hi Ber,
>
>Thanks for the patch and the word list. I will apply these right away.

massage -> message
randesvous -> rendezvous
menager -> manager
contant -> content
accoumulate -> accumulate
somthing -> something
Constractor -> Constructor
naem -> name

>
>> 
>> What about removing all the generated scripts (configure et al) and
>> auto-tools helpers (aclocal.m4 etc and most of config/*) from the
>> repository?
>[EZ] I thought I have done that. I will double check.

In both ibdm,IBMgtSim there is a configure config/share
config/libtoolize.
ibdm/config/depcomp libtool.m4 install-sh are (or should be) all
generated.
>
>> Note that the other userspace IB libraries do not have these in the
>> repository but each do provide a toplevel ./autogen.sh -- as opposed
>> to ibdm's ./config/bootstrap.
>[EZ] We can rename the bootstrap autogen.sh but from what I learned about
>auto tools many use ./config/bootstrap ...
>Is it really annoying?

It's not consistent, but i don't really care either way.

>> There are a number of other unused variables.
>[EZ] We should fix all of those! 

IBMgtSim/src/sim.cpp does not compile for me. I tried gcc-3.x as well as
4.x to no avail. Ideas?
sim.cpp:201: error: expected `}' at end of input

>> Is ibdm supposed to be compilable with pure ISO c++ compilers, btw?
>> There is at least one place which uses a variable sized array which AFAIK
>> is not valid ISO c++.
>[EZ] I'm not sure what the answer to that is.

I guess it's a question of policy. If gcc can deal with it, it's fine
with me, fwiw.


o IBMSClientInMsgs::proccessClientMsg() strip clientSock which from a
  short glance seems to be unused.

Attached patch does:
in ibmgtsim
- turn ibms_disconnect() and ibms_disconnect()) into a void functions
- add static in few places
- opensm off gen2 doesn't seem to have osm_build_id.h so check for
  osm_base.h instead
- correct offset in configure.in's --enable-debug help text

Signed-off-by: Bernhard Fischer <rep.nop at aon.at>

Please review the patch and advise on the sim.cpp compile failure i get
with pristine sources.
TIA,
Bernhard

-------------- next part --------------
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/config/osm.m4 gen2.2810/utils/src/linux-user/IBMgtSim/config/osm.m4
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/config/osm.m4	2005-03-24 17:56:41.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/config/osm.m4	2005-07-06 17:40:30.000000000 +0200
@@ -35,14 +35,14 @@ if test "x$with_osm" = xnone; then 
 fi
 
 dnl validate the defined path
-AC_CHECK_FILE($with_osm/include/opensm/osm_build_id.h,,
-   AC_MSG_ERROR([ could not find $with_osm/include/opensm/osm_build_id.h]))
+AC_CHECK_FILE($with_osm/include/infiniband/opensm/osm_base.h,,
+   AC_MSG_ERROR([ could not find $with_osm/include/infiniband/opensm/osm_base.h]))
 
 AC_SUBST(with_osm)
 
 dnl now figure out somehow if the build was for debug or not
 dnl the header file should have the string debug if OpenSM is debug mode:
-if test `grep debug $with_osm/include/opensm/osm_build_id.h | wc -l` = 1; then
+if test "$debug" != "false"; then
    dnl why did they need so many ???
    OSM_DEBUG_FLAGS='-DDEBUG -D_DEBUG -D_DEBUG_ -DDBG'
    AC_MSG_NOTICE(OSM compiled in DEBUG mode)
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/configure.in gen2.2810/utils/src/linux-user/IBMgtSim/configure.in
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/configure.in	2005-03-24 17:56:44.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/configure.in	2005-07-06 17:39:30.000000000 +0200
@@ -68,7 +68,7 @@ AC_CHECK_FUNCS([strchr strstr strtol str
 
 dnl Define an input config option to control debug compile
 AC_ARG_ENABLE(debug,
-[  --enable-debug    Turn on debugging],
+[  --enable-debug          Turn on debugging],
 [case "${enableval}" in
   yes) debug=true ;;
   no)  debug=false ;;
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/client.cpp gen2.2810/utils/src/linux-user/IBMgtSim/src/client.cpp
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/client.cpp	2005-03-24 17:55:58.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/client.cpp	2005-07-07 14:18:38.000000000 +0200
@@ -302,7 +302,7 @@ ibms_send(
 }
 
 /* disconnect from the simulator */
-int
+void
 ibms_disconnect(
   ibms_conn_handle_t conHdl)
 {
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h gen2.2810/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h	2005-03-24 17:55:59.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h	2005-07-07 14:18:46.000000000 +0200
@@ -114,7 +114,7 @@ int 
 ibms_send(ibms_conn_handle_t conHdl, ibms_mad_msg_t *pMadMsg);
 
 /* disconnect from the simulator */
-int
+void
 ibms_disconnect(ibms_conn_handle_t conHdl);
 
 #ifdef __cplusplus
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/Makefile.am gen2.2810/utils/src/linux-user/IBMgtSim/src/Makefile.am
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/Makefile.am	2005-03-24 17:55:59.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/Makefile.am	2005-07-07 15:35:34.734297717 +0200
@@ -43,7 +43,7 @@ endif
 
 AM_CXXFLAGS = $(DBG)
 
-INCLUDES		= -I$(with_ibdm)/include -I$(with_osm)/include $(TCL_CPPFLAGS)
+INCLUDES= -I$(with_ibdm)/include -I$(with_osm)/include/infiniband -I$(with_osm) $(TCL_CPPFLAGS)
 
 AM_CPPFLAGS = -I$(top_srcdir)/datamodel $(OSM_DEBUG_FLAGS)
 
@@ -63,7 +63,7 @@ ibmssh_SOURCES = sim.cpp server.cpp node
 ibmssh_LDFLAGS = -static -Wl,-rpath -Wl,$(TCL_PREFIX)/lib \
 	-Wl,-rpath -Wl,$(with_osm)/lib \
 	-Wl,-rpath -Wl,$(with_ibdm)/lib 
-
+ibmssh_CXXCPPFLAGS = #-DBUILD_STANDALONE_SIM
 ibmssh_LDADD = -libmscli -L$(with_ibdm)/lib -libdmcom -L$(with_osm)/lib -lcomplib $(TCL_LIBS) 
 
 # SWIG FILES:
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp gen2.2810/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp	2005-03-24 17:55:59.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp	2005-07-07 15:13:26.723783641 +0200
@@ -309,7 +309,7 @@ msgManager &msgMgr(int vl, std::ostream 
   }
   
   return (*pMgr);
-};
+}
 
 // we provide two global definitions for entering and leaving functions:
 int msgMgrEnterFunc = msgMgr().reg('R',"$ [", "top", "msg");
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/node.h gen2.2810/utils/src/linux-user/IBMgtSim/src/node.h
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/node.h	2005-07-06 11:52:53.000000000 +0200
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/node.h	2005-07-07 13:31:16.000000000 +0200
@@ -64,11 +64,11 @@ typedef std::list< uint16_t > list_uint1
 
 inline class IBMSNode * ibmsGetIBNodeSimNode(IBNode *pIBNode) {
   return ((IBMSNode *)pIBNode->appData1.ptr);
-};
+}
 
 inline void ibmsSetIBNodeSimNode(IBNode *pIBNode, class IBMSNode *pSimNode) {
   pIBNode->appData1.ptr = pSimNode;
-};
+}
 
 /* This class holds the variables defining statistical behaviour of the
    port counters */
@@ -227,7 +227,7 @@ class IBMSNode {
     remNode - to the Sim Node of the other side 
     remIBPort - to the remote side IB Fabric Port object on the other side.
   */
-  int IBMSNode::getRemoteNodeByOutPort(
+  int getRemoteNodeByOutPort(
     uint8_t outPortNum,
     IBMSNode **ppRemNode,
     IBPort **ppRemIBPort, int isVl15 = 0);
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/sim.cpp gen2.2810/utils/src/linux-user/IBMgtSim/src/sim.cpp
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/sim.cpp	2005-03-24 17:55:59.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/sim.cpp	2005-07-07 15:36:19.307584427 +0200
@@ -35,9 +35,10 @@
 #include "sim.h"
 #include "msgmgr.h"
 #include "sma.h"
+extern "C" {
 #include <getopt.h>
 #include <inttypes.h>
-
+}
 //////////////////////////////////////////////////////////////
 //
 // CLASS  IBMgtSim
@@ -142,7 +143,7 @@ IBMgtSim::populateFabricNodes()
     ibmsSetIBNodeSimNode(pNode, pSimNode);
 
     // TODO use a list of mgtclass instead
-    IBMSSma *pSma = new IBMSSma(pSimNode, 0x81);
+    IBMSSma *pSma = new IBMSSma(pSimNode, 0x81); //unused?
   }
   
   return 0;
@@ -185,19 +186,21 @@ int IBMgtSim::init(string topoFileName, 
   return 0;
 }
 
+#ifdef BUILD_STANDALONE_SIM
 //////////////////////////////////////////////////////////////
 //
-static char IBMgtSimUsage[] = 
-"Usage: ibmgtsim [-vh] -t <topology file> [-p <server port>][-w <num workers>]";
 
-void
-show_usage() {
+static void
+show_usage(void) {
+ string IBMgtSimUsage = 
+"Usage: ibmgtsim [-vh] -t <topology file> [-p <server port>][-w <num workers>]";
   cout << IBMgtSimUsage << endl;
 }
 
-void 
-show_help() {
-  cout << "HELP" << endl;
+static void
+show_help(void) {
+  string IBMgtSimHelp = "Descriptive HELP text";
+  cout << IBMgtSimHelp << endl;
 }
 
 //////////////////////////////////////////////////////////////
@@ -205,7 +208,6 @@ show_help() {
 // MAIN
 //
 
-#ifdef BUILD_STANDALONE_SIM
 int main(int argc, char **argv) 
 {
   /*
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/simmsg.cpp gen2.2810/utils/src/linux-user/IBMgtSim/src/simmsg.cpp
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/simmsg.cpp	2005-03-24 17:55:58.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/simmsg.cpp	2005-07-07 15:09:04.683521475 +0200
@@ -41,9 +41,9 @@
 #include <stdio.h>
 #include <unistd.h>
 #include <string>
-#include "simmsg.h"
+#include "helper.h"
 
-std::string
+static std::string
 __ibms_dump_conn_msg(
   IN const ibms_client_msg_t *p_msg) 
 {
@@ -62,7 +62,7 @@ __ibms_dump_conn_msg(
   return msg;
 }
 
-std::string
+static std::string
 __ibms_dump_disconn_msg(
   IN const ibms_client_msg_t *p_msg) 
 {
@@ -79,16 +79,18 @@ __ibms_dump_disconn_msg(
   return msg;
 }
 
-std::string
+static std::string
 __ibms_dump_cap_msg(
   IN const ibms_client_msg_t *p_msg)
 {
   char msg[512];
-  sprintf(msg, "MSG: CAP");
+  sprintf(msg, "MSG: CAP %08x MASK %08x",
+	  p_msg->msg.cap.capabilities,
+	  p_msg->msg.cap.mask);
   return msg;
 }
 
-std::string
+static std::string
 __ibms_dump_bind_msg(
   IN const ibms_client_msg_t *p_msg)
 {
@@ -112,7 +114,7 @@ __ibms_dump_bind_msg(
   return msg;
 }
 
-std::string
+static std::string
 __ibms_dump_mad_msg(
   IN const ibms_client_msg_t *p_msg)
 {
@@ -198,7 +200,7 @@ ibms_get_mad_header_str(ib_mad_t madHead
             CL_NTOH16(madHeader.attr_id),
             CL_NTOH32(madHeader.attr_mod));
     return (std::string)msg;
-};
+}
 
 std::string 
 ibms_get_node_info_str(ib_node_info_t*     pNodeInfo)
@@ -236,7 +238,7 @@ ibms_get_port_info_str(ib_port_info_t*  
             ib_port_info_get_neighbor_mtu(pPortInfo),
             ib_port_info_get_vl_cap(pPortInfo));
     return (std::string)msg;
-};
+}
 
 void 
 ibms_dump_msg(
diff -X excl -rduNp gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/tcpcomm.h gen2.2810/utils/src/linux-user/IBMgtSim/src/tcpcomm.h
--- gen2.2810.oorig/utils/src/linux-user/IBMgtSim/src/tcpcomm.h	2005-03-24 17:55:58.000000000 +0100
+++ gen2.2810/utils/src/linux-user/IBMgtSim/src/tcpcomm.h	2005-07-07 13:44:14.000000000 +0200
@@ -119,7 +119,7 @@ class GenServer {
     int &resLen, char *(pResponse[]) );
 
   /* virtual function called when a client is closed - under a lock */
-  virtual int closingClient(int clientSock) {};
+  virtual int closingClient(int clientSock);
 };
 
 /* 

From eitan at mellanox.co.il  Thu Jul  7 07:05:01 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 7 Jul 2005 17:05:01 +0300 
Subject: [openib-general] RE: [PATCH] ibdm nits and a few questions
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30458@mtlex01.yok.mtl.com>

Hi Ber,

I plan a huge change in the simulator core. So I do not want to waste your
time in cleaning up this code. I have some branch that I use now for
debugging.

I appreciate your check and contribution but as the code I'm about to commit
has many diffs I think it make sense to leave the IBMgtSim for 1 week or so.

I also will ispell the code before committing.

EZ  

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Bernhard Fischer [mailto:rep.nop at aon.at]
> Sent: Thursday, July 07, 2005 5:00 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: Re: [PATCH] ibdm nits and a few questions
> 
> On Thu, Jul 07, 2005 at 10:15:15AM +0300, Eitan Zahavi wrote:
> >Hi Ber,
> >
> >Thanks for the patch and the word list. I will apply these right away.
> 
> massage -> message
> randesvous -> rendezvous
> menager -> manager
> contant -> content
> accoumulate -> accumulate
> somthing -> something
> Constractor -> Constructor
> naem -> name
> 
> >
> >>
> >> What about removing all the generated scripts (configure et al) and
> >> auto-tools helpers (aclocal.m4 etc and most of config/*) from the
> >> repository?
> >[EZ] I thought I have done that. I will double check.
> 
> In both ibdm,IBMgtSim there is a configure config/share
> config/libtoolize.
> ibdm/config/depcomp libtool.m4 install-sh are (or should be) all
> generated.
> >
> >> Note that the other userspace IB libraries do not have these in the
> >> repository but each do provide a toplevel ./autogen.sh -- as opposed
> >> to ibdm's ./config/bootstrap.
> >[EZ] We can rename the bootstrap autogen.sh but from what I learned about
> >auto tools many use ./config/bootstrap ...
> >Is it really annoying?
> 
> It's not consistent, but i don't really care either way.
> 
> >> There are a number of other unused variables.
> >[EZ] We should fix all of those!
> 
> IBMgtSim/src/sim.cpp does not compile for me. I tried gcc-3.x as well as
> 4.x to no avail. Ideas?
> sim.cpp:201: error: expected `}' at end of input
> 
> >> Is ibdm supposed to be compilable with pure ISO c++ compilers, btw?
> >> There is at least one place which uses a variable sized array which
AFAIK
> >> is not valid ISO c++.
> >[EZ] I'm not sure what the answer to that is.
> 
> I guess it's a question of policy. If gcc can deal with it, it's fine
> with me, fwiw.
> 
> 
> o IBMSClientInMsgs::proccessClientMsg() strip clientSock which from a
>   short glance seems to be unused.
> 
> Attached patch does:
> in ibmgtsim
> - turn ibms_disconnect() and ibms_disconnect()) into a void functions
> - add static in few places
> - opensm off gen2 doesn't seem to have osm_build_id.h so check for
>   osm_base.h instead
> - correct offset in configure.in's --enable-debug help text
> 
> Signed-off-by: Bernhard Fischer <rep.nop at aon.at>
> 
> Please review the patch and advise on the sim.cpp compile failure i get
> with pristine sources.
> TIA,
> Bernhard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050707/cbb6d961/attachment.html>

From rep.nop at aon.at  Thu Jul  7 07:21:50 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Thu, 7 Jul 2005 16:21:50 +0200
Subject: [openib-general] Re: [PATCH] ibdm nits and a few questions
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30458@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30458@mtlex01.yok.mtl.com>
Message-ID: <20050707142150.GC17593@aon.at>

On Thu, Jul 07, 2005 at 05:05:01PM +0300, Eitan Zahavi wrote:
>Hi Ber,
>
>I plan a huge change in the simulator core. So I do not want to waste your
>time in cleaning up this code. I have some branch that I use now for
>debugging.
>
>I appreciate your check and contribution but as the code I'm about to commit
>has many diffs I think it make sense to leave the IBMgtSim for 1 week or so.

Ok, thanks.


From philippe.bernadat at hp.com  Thu Jul  7 07:46:04 2005
From: philippe.bernadat at hp.com (Philippe Bernadat)
Date: Thu, 07 Jul 2005 16:46:04 +0200
Subject: [openib-general] Re: how to choose appropriate ULPs for application
Message-ID: <42CD402C.3050605@hp.com>

Lustre does support infiniband. 
This is implemented as a specific NAL kernel module.

HP has just announced a new release of SFS that supports 
the Voltaire stack: 
http://www.hp.com/hpinfo/newsroom/press/2005/050621a.html
HP SFS is based on Lustre Technology:
http://www.hp.com/techservers/products/sfs.html

As explained in the Lustre FAQ, this is not an OpenIB NAL:



    When and how will you support InfiniBand?

The reality of InfiniBand today is, unfortunately, a world of three IB 
driver stacks, which effectively means individual support for three 
separate interconnects: the Mellanox/TopSpin/OpenIB-1 stack, the 
InfiniCon stack, and the Voltaire stack.

We have existing network abstraction layers (NALs) for all three, with 
varying degress of performance and stability. Customers are now 
beginning to appear for all three, and we will begin to introduce stable 
and performant versions of these NALs into the Lustre 1.4.x series.

It is anticipated that all three will be ready for production 
deployments on clusters of many hundreds of nodes in the first half of 2005.

When the promised vision of a single, usable, vendor-supported OpenIB 
Generation 2 driver stack becomes available, we will transition to a 
single OpenIB NAL.

Wang Xigui wrote:/

> Thank you very much!
> I'll read the materials that you supply.
>
>     I have not described my plan clearly. I don't konw how to run lustre
> on IB. Neither do I konw the lustre running in kernel space or user
> space. Now the lustre NAL does not supports IB vapi. If I use IPoIB,
> it works. But it doen not take take advantage of RDMA. Supporse
> lustre NAL invoke SDP or uDAP, RDMA is enable.
>
>    Invocation of Verb API to realize lustre NAL is my target. The SDP or
> uDAP is first step.
/

>/ Hello
/>/
/>/     We plan to run lustre on InfiniBand. But now lustre does not support
/>/ IB.
/>/ I know noting about the IB stack. So I think the IB Upper Layer
/>/ Protocol(SDP, DAP) could be used to transport data for lustre.
/>/ 	But there are too many ULPs over IB. I know little about them. As far as
/>/ I konw, iSER/SRP is for SCSI command transportation. So, it seems both
/>/ SDP and uDAP are OK for my plan. I cant tell the difference between them.
/>/ Which one is better, SDP or DAP? I don't konw how to choose?
/>/
/>/ 	Any suggestion is appreciated!
/>




From glebn at voltaire.com  Thu Jul  7 08:20:27 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 18:20:27 +0300
Subject: [openib-general] calculate inline data size
Message-ID: <20050707152026.GB12222@minantech.com>

Hello,

 Can I trust value returned by ibv_create_qp() in cap.max_inline_data
as been maximum message that I can send inline?

I create qp and pass zero as max_inline_data, I use value returned
in max_inline_data to determine whether to send data inline. It appears
that mthca_tavor_post_send() calculates inline size different from 
mthca_return_cap() if rdma is used, thus sometimes ibv_post_send() fails.

Is this should work at all?

--
			Gleb.


From mst at mellanox.co.il  Thu Jul  7 08:38:39 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 18:38:39 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707152026.GB12222@minantech.com>
References: <20050707152026.GB12222@minantech.com>
Message-ID: <20050707153839.GA17994@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> Subject: calculate inline data size
> 
> Hello,
> 
>  Can I trust value returned by ibv_create_qp() in cap.max_inline_data
> as been maximum message that I can send inline?
> 
> I create qp and pass zero as max_inline_data, I use value returned
> in max_inline_data to determine whether to send data inline.

why dont you decide what size you want and use that?


From rolandd at cisco.com  Thu Jul  7 09:04:12 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 09:04:12 -0700
Subject: [openib-general] Re: [PATCH] signal driven io support for ib_uverbs
In-Reply-To: <20050707090747.GA12222@minantech.com> (Gleb Natapov's message
	of "Thu, 7 Jul 2005 12:07:47 +0300")
References: <20050706121711.GG12680@minantech.com>
	<52wto34nzs.fsf@topspin.com> <20050707090747.GA12222@minantech.com>
Message-ID: <52k6k238n7.fsf@topspin.com>

    Gleb> The question is what is the right way? Do it from the kernel
    Gleb> or left it to userspace? Parts of the kernel do f_setown()
    Gleb> from the kernel others don't.

I think that userspace is supposed to do the fcntl.  For example, the
code at <http://kegel.com/c10k.html> looks like:

	/* For each file descriptor, invoke F_SETOWN, F_SETSIG, and set O_ASYNC. */
	fcntl(fd, F_SETOWN, (int) getpid());
	fcntl(fd, F_SETSIG, signum);
	flags = fcntl(fd, F_GETFL);
	flags |= O_NONBLOCK|O_ASYNC;
	fcntl(fd, F_SETFL, flags);

 - R.


From rolandd at cisco.com  Thu Jul  7 09:05:23 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 09:05:23 -0700
Subject: [openib-general] calculate inline data size
In-Reply-To: <20050707152026.GB12222@minantech.com> (Gleb Natapov's message
	of "Thu, 7 Jul 2005 18:20:27 +0300")
References: <20050707152026.GB12222@minantech.com>
Message-ID: <52fyuq38l8.fsf@topspin.com>

    Gleb> Can I trust value returned by ibv_create_qp() in
    Gleb> cap.max_inline_data as been maximum message that I can send
    Gleb> inline?

You should be able to trust it but it sounds like you've hit a bug.

    Gleb> I create qp and pass zero as max_inline_data, I use value
    Gleb> returned in max_inline_data to determine whether to send
    Gleb> data inline. It appears that mthca_tavor_post_send()
    Gleb> calculates inline size different from mthca_return_cap() if
    Gleb> rdma is used, thus sometimes ibv_post_send() fails.

Do you mean that there is less space available for RDMA writes?  In
other words, if you pass inline data of size max_inline_data for an
RDMA write, it fails?

 - R.


From rolandd at cisco.com  Thu Jul  7 09:09:18 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 09:09:18 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707153839.GA17994@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 7 Jul 2005 18:38:39 +0300")
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
Message-ID: <52br5e38ep.fsf@topspin.com>

    Michael> why dont you decide what size you want and use that?

I think we have a bug -- when we calculate the max_inline_data size,
we don't take into account the fact that an RDMA write WQE will have a
remote address segment.

I'm not sure what the best solution is here.  We can report
max_inline_data as 16 bytes smaller to handle the RDMA case -- but
this means someone doing sends only could actually post more inline
data.  Or we could split the cap field into max_inline_send_data and
max_inline_rdma_write_data and report the two values separately.  But
that may be too complicated to be worth it.

 - R.


From mst at mellanox.co.il  Thu Jul  7 09:44:55 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 19:44:55 +0300
Subject: [openib-general] Re: Re: calculate inline data size
In-Reply-To: <52br5e38ep.fsf@topspin.com>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<52br5e38ep.fsf@topspin.com>
Message-ID: <20050707164455.GB17994@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: Re: calculate inline data size
> 
>     Michael> why dont you decide what size you want and use that?
> 
> I think we have a bug -- when we calculate the max_inline_data size,
> we don't take into account the fact that an RDMA write WQE will have a
> remote address segment.

Seems like it.

> I'm not sure what the best solution is here.  We can report
> max_inline_data as 16 bytes smaller to handle the RDMA case -- but
> this means someone doing sends only could actually post more inline
> data.

I expect generally size may be different for work requests
with and without immediate, so we'd need an array for each
possible opcode.

> Or we could split the cap field into max_inline_send_data and
> max_inline_rdma_write_data and report the two values separately.  But
> that may be too complicated to be worth it.
> 
>  - R.

I think its too complicated.

-- 
MST


From mst at mellanox.co.il  Thu Jul  7 09:53:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 19:53:33 +0300
Subject: [openib-general] [PATCH] ipoib: updates to copyright
Message-ID: <20050707165333.GA18363@mellanox.co.il>

Add Mellanox copyright to files modified by Michael S. Tsirkin.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/ipoib/ipoib_verbs.c
===================================================================
--- ulp/ipoib/ipoib_verbs.c	(revision 2817)
+++ ulp/ipoib/ipoib_verbs.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: ulp/ipoib/ipoib_ib.c
===================================================================
--- ulp/ipoib/ipoib_ib.c	(revision 2817)
+++ ulp/ipoib/ipoib_ib.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU

-- 
MST


From halr at voltaire.com  Thu Jul  7 10:55:30 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 13:55:30 -0400
Subject: [openib-general] IBDM and IBMgtSim Proposal Comments
Message-ID: <1120758776.4386.31.camel@hal.voltaire.com>

Hi Eitan,

It appears that all the proposed Mellanox diagnostics work on top of the
"OSM" vendor layer. ["OSM" is in quotes as this layer may have started
out as the OSM vendor specific layer but is used by more than just the
OSM (diagnostics and an implementation of a user space SA client).] 

Where does that leave the OpenIB diagnostics and other applications
which do not use this layer ? Does the simulator support these
applications ?

In the OpenIB architecture, umad is the lowest layer library and the
diagnostics are built on that. The OSM vendor layer is built on top of
this. So when the umad and mad libraries are ported to Windows,
everything on top of this will work. This includes all diagnostics
(OpenIB ones as well as the additional tools you are proposing to add).

Also, it appears that the Mellanox simulator is builtin to each
simulated process and so each instance is separate which limits its
usability.

In the Voltaire architecture proposed to be moved forward to OpenIB, the
simulator is standalone and hence is useable by all applications in the
network which can be on different nodes (TCP connections): OpenSM (as
many as you want in the subnet), diagnostics, SA clients, etc. Also, the
Voltaire simulator architecture can use an input file recorded from a
real IB fabric where it appears that this would be manual in the
Mellanox scheme.

The file formats (netlist and topology) are for further study. There are
some good ideas in these file formats which would be incorporated into
the ibnetdiscover format.

-- Hal



From eitan at mellanox.co.il  Thu Jul  7 11:58:43 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 7 Jul 2005 21:58:43 +0300 
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3045D@mtlex01.yok.mtl.com>

Hi Hal,

> Where does that leave the OpenIB diagnostics and other applications
> which do not use this layer ? Does the simulator support these
> applications ?
[EZ] The simulator is a separate process that is a TCP/IP server.
Any application that will send and receive simulator messages (as defined by
simmsg.h) can connect to it. 
The osmv_sim (osm vendor layer for simulator) is just an example for such
client.
But one can even write perl or TCL applications that will connect. 
Actually the simulator provides a library named ibmscli that provides all
the required TCP/IP (client/server) logic and exposes a simple API for such
client.
Please see ibms_client_api.h for this API definition.  
> 
> In the OpenIB architecture, umad is the lowest layer library and the
> diagnostics are built on that. The OSM vendor layer is built on top of
> this. So when the umad and mad libraries are ported to Windows,
> everything on top of this will work. This includes all diagnostics
> (OpenIB ones as well as the additional tools you are proposing to add).
[EZ] There is no need to port UMAD to windows!!! We already have OSM Vendor
ported to it. It works on top of the existing IBAL API (actually this is the
first OSM Vendor that was ever built).
> 
> Also, it appears that the Mellanox simulator is builtin to each
> simulated process and so each instance is separate which limits its
> usability.
[EZ] This is totally incorrect - as explained before. We were able to
simulate OpenSM, osmtest and IBIS (IB Inband Services - which is a simple
API for sending MADs) all connected and communicating over the simulator. We
were also able to simulate multiple SM's handoff processes. Where did you
get this idea from?
> 
> In the Voltaire architecture proposed to be moved forward to OpenIB, the
> simulator is standalone and hence is useable by all applications in the
> network which can be on different nodes (TCP connections): OpenSM (as
> many as you want in the subnet), diagnostics, SA clients, etc. Also, the
> Voltaire simulator architecture can use an input file recorded from a
> real IB fabric where it appears that this would be manual in the
> Mellanox scheme.
[EZ] The Voltaire simulator was never made public and actually has no
benefit over what is provided by the IBMgtSim. 
> 
> The file formats (netlist and topology) are for further study. There are
> some good ideas in these file formats which would be incorporated into
> the ibnetdiscover format.
[EZ] Thanks.
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050707/353224f8/attachment.html>

From halr at voltaire.com  Thu Jul  7 11:57:12 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 14:57:12 -0400
Subject: [openib-general] A Couple of IPoIB related issues
Message-ID: <1120762632.4386.44.camel@hal.voltaire.com>

Hi,

Two issues have been observed when running IPoIB:

1. NETDEV WATCHDOG: ib0: transmit timed out
ib0: transmit timeout: latency 360052

This occurs once a minute on heavy pings.

2. __alloc_pages() warnings

here is a partial log quote:
---

The following is only an harmless informational message.
Unless you get a _continuous_flood_ of these messages it means
everything is working fine. Allocations from irqs cannot be
perfectly reliable and the kernel is designed to handle that.
ping: page allocation failure. order:0, mode:0x20
 [__alloc_pages+980/1168] __alloc_pages+0x3d4/0x490
 [<c01470d4>] __alloc_pages+0x3d4/0x490
 [kmem_getpages+50/160] kmem_getpages+0x32/0xa0
 [<c0149cf2>] kmem_getpages+0x32/0xa0
 [cache_grow+179/368] cache_grow+0xb3/0x170
 [<c014aa63>] cache_grow+0xb3/0x170
 [cache_alloc_refill+365/576] cache_alloc_refill+0x16d/0x240
 ...

Any idea on what could be causing these or how to go about isolating
them ? Thanks.

-- Hal



From rolandd at cisco.com  Thu Jul  7 12:27:11 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 12:27:11 -0700
Subject: [openib-general] Re: A Couple of IPoIB related issues
In-Reply-To: <1120762632.4386.44.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "07 Jul 2005 14:57:12 -0400")
References: <1120762632.4386.44.camel@hal.voltaire.com>
Message-ID: <52pstu1kog.fsf@topspin.com>

    Hal> 1. NETDEV WATCHDOG: ib0: transmit timed out ib0: transmit
    Hal> timeout: latency 360052

    Hal> This occurs once a minute on heavy pings.

Exactly once a minute, or is this just a rough frequency?  What is a
heavy ping?  Does the IPoIB interface still work when you see this?

    Hal> 2. __alloc_pages() warnings

Is there more traceback that shows who is doing the allocation that
fails?  In any case it looks like you are just running low on memory.

    Hal> Any idea on what could be causing these or how to go about
    Hal> isolating them ?

The __alloc_pages() warnings look fairly benign and can probably be
fixed by tuning /proc/sys/vm/min_free_kbytes appropriately.

The TX timeout is somewhat odd.  I guess we need to figure out why the
netdevice queue is stopped for a long time.

 - R.


From rolandd at cisco.com  Thu Jul  7 12:29:46 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 12:29:46 -0700
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <20050706234645.GA32470@cse.ohio-state.edu> (Sayantan Sur's
	message of "Wed, 6 Jul 2005 19:46:47 -0400")
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
Message-ID: <52d5pu1kk5.fsf@topspin.com>

    Sayantan> Thanks for the information. I was able to compile the
    Sayantan> kernel components against 2.6.12. On installing the
    Sayantan> entire OpenIB stack, I am getting an error like this:

Has this machine worked with other versions of OpenIB software?  Are
the ports connected to anything?

 - R.


From halr at voltaire.com  Thu Jul  7 12:32:38 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 15:32:38 -0400
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <52d5pu1kk5.fsf@topspin.com>
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
	<52d5pu1kk5.fsf@topspin.com>
Message-ID: <1120764553.4386.69.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 15:29, Roland Dreier wrote:
> Are the ports connected to anything?

They should show down rather than disabled (if not connected), right ?

-- Hal



From rolandd at cisco.com  Thu Jul  7 12:41:34 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 12:41:34 -0700
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <1120764553.4386.69.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "07 Jul 2005 15:32:38 -0400")
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
	<52d5pu1kk5.fsf@topspin.com>
	<1120764553.4386.69.camel@hal.voltaire.com>
Message-ID: <524qb61k0h.fsf@topspin.com>

    Roland> Are the ports connected to anything?

    Hal> They should show down rather than disabled (if not
    Hal> connected), right ?

Yes, but it's still worth knowing what the situation is.

 - R.


From halr at voltaire.com  Thu Jul  7 12:45:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 15:45:14 -0400
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3045D@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C3045D@mtlex01.yok.mtl.com>
Message-ID: <1120765125.4386.85.camel@hal.voltaire.com>

Hi Eitan,

On Thu, 2005-07-07 at 14:58, Eitan Zahavi wrote:
[snip]

> > In the OpenIB architecture, umad is the lowest layer library and the
> > diagnostics are built on that. The OSM vendor layer is built on top
> of
> > this. So when the umad and mad libraries are ported to Windows,
> > everything on top of this will work. This includes all diagnostics
> > (OpenIB ones as well as the additional tools you are proposing to
> add).
> [EZ] There is no need to port UMAD to windows!!! We already have OSM
> Vendor ported to it. It works on top of the existing IBAL API
> (actually this is the first OSM Vendor that was ever built).

There is if the OpenIB diagnostics and other applications in the Linux
environment which are not on top of the "OSM" vendor layer are to work
in the Windows environment. That was what started this whole thread.
 
[snip]

I'll comment on the rest separately.

-- Hal



From eitan at mellanox.co.il  Thu Jul  7 13:14:28 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 7 Jul 2005 23:14:28 +0300 
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30460@mtlex01.yok.mtl.com>

The point is that it is better to write user level code that is not
"performance critical" in a way that is not tailored to the actual
driver/Access layer. I think OpenSM vendor layer was proven to be adequate
for this purpose. 

It should be much simpler to port the existing tools on top of the
osm_vendor_api.h rather then porting them to IBAL api. 

EZ

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Thursday, July 07, 2005 10:45 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org; openib-windows at openib.org
> Subject: RE: IBDM and IBMgtSim Proposal Comments
> 
> Hi Eitan,
> 
> On Thu, 2005-07-07 at 14:58, Eitan Zahavi wrote:
> [snip]
> 
> > > In the OpenIB architecture, umad is the lowest layer library and the
> > > diagnostics are built on that. The OSM vendor layer is built on top
> > of
> > > this. So when the umad and mad libraries are ported to Windows,
> > > everything on top of this will work. This includes all diagnostics
> > > (OpenIB ones as well as the additional tools you are proposing to
> > add).
> > [EZ] There is no need to port UMAD to windows!!! We already have OSM
> > Vendor ported to it. It works on top of the existing IBAL API
> > (actually this is the first OSM Vendor that was ever built).
> 
> There is if the OpenIB diagnostics and other applications in the Linux
> environment which are not on top of the "OSM" vendor layer are to work
> in the Windows environment. That was what started this whole thread.
> 
> [snip]
> 
> I'll comment on the rest separately.
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050707/3105323e/attachment.html>

From glebn at voltaire.com  Thu Jul  7 13:20:58 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 23:20:58 +0300
Subject: [openib-general] Re: [PATCH] signal driven io support for ib_uverbs
In-Reply-To: <52k6k238n7.fsf@topspin.com>
References: <20050706121711.GG12680@minantech.com> <52wto34nzs.fsf@topspin.com>
	<20050707090747.GA12222@minantech.com> <52k6k238n7.fsf@topspin.com>
Message-ID: <20050707202058.GA22605@minantech.com>

On Thu, Jul 07, 2005 at 09:04:12AM -0700, Roland Dreier wrote:
>     Gleb> The question is what is the right way? Do it from the kernel
>     Gleb> or left it to userspace? Parts of the kernel do f_setown()
>     Gleb> from the kernel others don't.
> 
> I think that userspace is supposed to do the fcntl.  For example, the
> code at <http://kegel.com/c10k.html> looks like:
> 
> 	/* For each file descriptor, invoke F_SETOWN, F_SETSIG, and set O_ASYNC. */
> 	fcntl(fd, F_SETOWN, (int) getpid());
> 	fcntl(fd, F_SETSIG, signum);
> 	flags = fcntl(fd, F_GETFL);
> 	flags |= O_NONBLOCK|O_ASYNC;
> 	fcntl(fd, F_SETFL, flags);
> 

The kernel is inconsistent in this respect so correct program should set
owner anyway. Lets use your patch.

--
			Gleb.


From glebn at voltaire.com  Thu Jul  7 13:24:04 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 23:24:04 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707153839.GA17994@mellanox.co.il>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
Message-ID: <20050707202404.GB22605@minantech.com>

On Thu, Jul 07, 2005 at 06:38:39PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > Subject: calculate inline data size
> > 
> > Hello,
> > 
> >  Can I trust value returned by ibv_create_qp() in cap.max_inline_data
> > as been maximum message that I can send inline?
> > 
> > I create qp and pass zero as max_inline_data, I use value returned
> > in max_inline_data to determine whether to send data inline.
> 
> why dont you decide what size you want and use that?
Because I want it to be as big as possible.

--
			Gleb.


From mst at mellanox.co.il  Thu Jul  7 13:27:41 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 23:27:41 +0300
Subject: [openib-general] Re: A Couple of IPoIB related issues
In-Reply-To: <1120762632.4386.44.camel@hal.voltaire.com>
References: <1120762632.4386.44.camel@hal.voltaire.com>
Message-ID: <20050707202741.GA22604@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: A Couple of IPoIB related issues
> 
> Hi,
> 
> Two issues have been observed when running IPoIB:
> 
> 1. NETDEV WATCHDOG: ib0: transmit timed out
> ib0: transmit timeout: latency 360052
> 
> This occurs once a minute on heavy pings.

could you be running some kind of daemon with UDP?

> 2. __alloc_pages() warnings
> 
> here is a partial log quote:
> ---
> 
> The following is only an harmless informational message.
> Unless you get a _continuous_flood_ of these messages it means
> everything is working fine. Allocations from irqs cannot be
> perfectly reliable and the kernel is designed to handle that.
> ping: page allocation failure. order:0, mode:0x20

mode 20 is GFP_ATOMIC, so there's no reason to worry.
...

> Any idea on what could be causing these or how to go about isolating
> them ? Thanks.
> 
> -- Hal

-- 
MST


From halr at voltaire.com  Thu Jul  7 13:20:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 16:20:31 -0400
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30460@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30460@mtlex01.yok.mtl.com>
Message-ID: <1120767613.4386.105.camel@hal.voltaire.com>

Hi Eitan,

On Thu, 2005-07-07 at 16:14, Eitan Zahavi wrote:
> The point is that it is better to write user level code that is not
> "performance critical" in a way that is not tailored to the actual
> driver/Access layer.

Have you looked at the umad and mad libraries ? These are not IBAL.

>  I think OpenSM vendor layer was proven to be adequate for this
> purpose. 

and so have the umad and mad libraries used in the OpenIB Linux stack.

> It should be much simpler to port the existing tools on top of the
> osm_vendor_api.h rather then porting them to IBAL api.

Are you proposing to port any applications needed from OpenIB Linux to
the OSM vendor layer ?

-- Hal



From glebn at voltaire.com  Thu Jul  7 13:30:36 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 23:30:36 +0300
Subject: [openib-general] Re: Re: calculate inline data size
In-Reply-To: <20050707164455.GB17994@mellanox.co.il>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<52br5e38ep.fsf@topspin.com>
	<20050707164455.GB17994@mellanox.co.il>
Message-ID: <20050707203036.GC22605@minantech.com>

On Thu, Jul 07, 2005 at 07:44:55PM +0300, Michael S. Tsirkin wrote:
> > I'm not sure what the best solution is here.  We can report
> > max_inline_data as 16 bytes smaller to handle the RDMA case -- but
> > this means someone doing sends only could actually post more inline
> > data.
> 
> I expect generally size may be different for work requests
> with and without immediate, so we'd need an array for each
> possible opcode.
> 
This could work, but what will be the meaning of user requested 
inline size? The size that should work for any request type?

--
			Gleb.


From mst at mellanox.co.il  Thu Jul  7 13:32:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 23:32:54 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707202404.GB22605@minantech.com>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<20050707202404.GB22605@minantech.com>
Message-ID: <20050707203253.GB28506@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > I create qp and pass zero as max_inline_data, I use value returned
> > > in max_inline_data to determine whether to send data inline.
> > 
> > why dont you decide what size you want and use that?
> >
> Because I want it to be as big as possible.

Then you should specify a big max_inline_data.
If you give 0 you get a fairly low number, basically depending
on the max s/g list size that you give.

-- 
MST


From mst at mellanox.co.il  Thu Jul  7 13:34:00 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 7 Jul 2005 23:34:00 +0300
Subject: [openib-general] Re: Re: calculate inline data size
In-Reply-To: <20050707203036.GC22605@minantech.com>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<52br5e38ep.fsf@topspin.com>
	<20050707164455.GB17994@mellanox.co.il>
	<20050707203036.GC22605@minantech.com>
Message-ID: <20050707203400.GC28506@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> Subject: Re: Re: calculate inline data size
> 
> On Thu, Jul 07, 2005 at 07:44:55PM +0300, Michael S. Tsirkin wrote:
> > > I'm not sure what the best solution is here.  We can report
> > > max_inline_data as 16 bytes smaller to handle the RDMA case -- but
> > > this means someone doing sends only could actually post more inline
> > > data.
> > 
> > I expect generally size may be different for work requests
> > with and without immediate, so we'd need an array for each
> > possible opcode.
> > 
> This could work, but what will be the meaning of user requested 
> inline size? The size that should work for any request type?

Hrm. Right. And array there too?
Better go with the simpler approach I guess.

-- 
MST


From eitan at mellanox.co.il  Thu Jul  7 13:33:54 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 7 Jul 2005 23:33:54 +0300 
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30461@mtlex01.yok.mtl.com>

> 
> Have you looked at the umad and mad libraries ? These are not IBAL.
[EZ] Yes I know. This is why they do not work on Windows IBAL.
> 
> >  I think OpenSM vendor layer was proven to be adequate for this
> > purpose.
> 
> and so have the umad and mad libraries used in the OpenIB Linux stack.
[EZ] No they did not - the API they provide was never ported to any other AL
particularly not Windows.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050707/6c69f480/attachment.html>

From ftillier at silverstorm.com  Thu Jul  7 13:37:09 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 13:37:09 -0700
Subject: [openib-general] IBDM and IBMgtSim Proposal Comments
In-Reply-To: <1120758776.4386.31.camel@hal.voltaire.com>
Message-ID: <000a01c58333$a66fe950$6501a8c0@infiniconsys.com>

> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Thursday, July 07, 2005 10:56 AM
> 
> In the OpenIB architecture, umad is the lowest layer library and the
> diagnostics are built on that.

That's only true in the *Linux* OpenIB Architecture.  Windows is different - the
access layer already provides support for user-level MAD clients, and the API is
very close (if not identical) to the IBAL interface OpenSM was originally
written to.

> The OSM vendor layer is built on top of
> this. So when the umad and mad libraries are ported to Windows,
> everything on top of this will work. This includes all diagnostics
> (OpenIB ones as well as the additional tools you are proposing to add).

Is there a plan to migrate the Linux umad and MAD libraries and replace the
user-mode MAD support already in the Windows stack?  If so, does it make sense
or are umad and the MAD libraries too closely tied to Linux?

I haven't had a chance to look carefully at the umad interface - does anyone
know how it compares to the IBAL MAD service interface?

- Fab



From ftillier at silverstorm.com  Thu Jul  7 13:43:10 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 13:43:10 -0700
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <1120765125.4386.85.camel@hal.voltaire.com>
Message-ID: <000b01c58334$7d620100$6501a8c0@infiniconsys.com>

> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Thursday, July 07, 2005 12:45 PM
> 
> On Thu, 2005-07-07 at 14:58, Eitan Zahavi wrote:
> > [EZ] There is no need to port UMAD to windows!!! We already have OSM
> > Vendor ported to it. It works on top of the existing IBAL API
> > (actually this is the first OSM Vendor that was ever built).
> 
> There is if the OpenIB diagnostics and other applications in the Linux
> environment which are not on top of the "OSM" vendor layer are to work
> in the Windows environment. That was what started this whole thread.

The discussion was about porting the diagnostics to the Windows environment.
Whether that's done by porting umad and the MAD libraries to run on top of the
IBAL MAD services APIs or by porting the diagnostics to interface directly to
IBAL hasn't been settled on.  I suppose it will be up to whoever ports the
diagnostics.  If I was porting them, I'd make them interface to the lowest level
available (just like they do in Linux).  If Eitan was porting them, he'd have
them run over the OSM vendor layer.  If you were porting them, you'd port umad.

How is porting umad any different than using the OSM vendor layer?  umad is the
lowest level API in Linux, but not in Windows.  So either the diagnostics
interface to the lowest level layer (umad for Linux, IBAL for Windows), or the
diagnostics interface to some higher abstraction layer.  If a higher abstraction
layer, why not use the existing OSM vendor layer and skip porting umad to
Windows all together?

- Fab



From halr at voltaire.com  Thu Jul  7 13:38:03 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 16:38:03 -0400
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30461@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30461@mtlex01.yok.mtl.com>
Message-ID: <1120768682.4386.118.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 16:33, Eitan Zahavi wrote:
> > 
> > Have you looked at the umad and mad libraries ? These are not IBAL.
> [EZ] Yes I know. This is why they do not work on Windows IBAL.

Is it a requirement for them to work on Windows IBAL ? 

> > >  I think OpenSM vendor layer was proven to be adequate for this
> > > purpose.
> > 
> > and so have the umad and mad libraries used in the OpenIB Linux
> stack.
> [EZ] No they did not - the API they provide was never ported to any
> other AL particularly not Windows.

Are you saying there is something in those APIs where they couldn't be
ported to Windows ?

There is no reason the MAD and UMAD libraries couldn't be ported to
Windows.

-- Hal



From gleb at minantech.com  Thu Jul  7 13:46:30 2005
From: gleb at minantech.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 23:46:30 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707203253.GB28506@mellanox.co.il>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<20050707202404.GB22605@minantech.com>
	<20050707203253.GB28506@mellanox.co.il>
Message-ID: <20050707204630.GD22605@minantech.com>

On Thu, Jul 07, 2005 at 11:32:54PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > I create qp and pass zero as max_inline_data, I use value returned
> > > > in max_inline_data to determine whether to send data inline.
> > > 
> > > why dont you decide what size you want and use that?
> > >
> > Because I want it to be as big as possible.
> 
> Then you should specify a big max_inline_data.
> If you give 0 you get a fairly low number, basically depending
> on the max s/g list size that you give.
> 
Tried that. But with big values create_qp fails. So my next question is
how can I determine the biggest inline data I can send. Binary search?

--
			Gleb.


From glebn at voltaire.com  Thu Jul  7 13:47:56 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 7 Jul 2005 23:47:56 +0300
Subject: [openib-general] Re: Re: calculate inline data size
In-Reply-To: <20050707203400.GC28506@mellanox.co.il>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<52br5e38ep.fsf@topspin.com>
	<20050707164455.GB17994@mellanox.co.il>
	<20050707203036.GC22605@minantech.com>
	<20050707203400.GC28506@mellanox.co.il>
Message-ID: <20050707204756.GE22605@minantech.com>

On Thu, Jul 07, 2005 at 11:34:00PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > Subject: Re: Re: calculate inline data size
> > 
> > On Thu, Jul 07, 2005 at 07:44:55PM +0300, Michael S. Tsirkin wrote:
> > > > I'm not sure what the best solution is here.  We can report
> > > > max_inline_data as 16 bytes smaller to handle the RDMA case -- but
> > > > this means someone doing sends only could actually post more inline
> > > > data.
> > > 
> > > I expect generally size may be different for work requests
> > > with and without immediate, so we'd need an array for each
> > > possible opcode.
> > > 
> > This could work, but what will be the meaning of user requested 
> > inline size? The size that should work for any request type?
> 
> Hrm. Right. And array there too?
I hope not!

> Better go with the simpler approach I guess.
> 
> -- 
> MST

--
			Gleb.


From ftillier at silverstorm.com  Thu Jul  7 13:50:34 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 13:50:34 -0700
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <1120768682.4386.118.camel@hal.voltaire.com>
Message-ID: <000c01c58335$86393a90$6501a8c0@infiniconsys.com>

> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Thursday, July 07, 2005 1:38 PM
> 
> On Thu, 2005-07-07 at 16:33, Eitan Zahavi wrote:
> > >
> > > Have you looked at the umad and mad libraries ? These are not IBAL.
> > [EZ] Yes I know. This is why they do not work on Windows IBAL.
> 
> Is it a requirement for them to work on Windows IBAL ?

The OpenIB Windows stack is IBAL.

> There is no reason the MAD and UMAD libraries couldn't be ported to
> Windows.

Is there any reason to port these when the Windows OpenIB stack already has
fully implemented and functioning user-mode MAD client support?  What's the
value in porting the API?  Is it portable with no changes so applications using
the umad interface don't have to change between Linux and Windows?

I'm guessing the diagnostics will have to change regardless of what MAD API they
interface to unless they interface to an abstraction layer that hides OS
differences.  If the diagnostics are going to change, why not just change them
to interface directly to the native MAD API for the target platform?

- Fab



From eitan at mellanox.co.il  Thu Jul  7 14:02:04 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Fri, 8 Jul 2005 00:02:04 +0300 
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30463@mtlex01.yok.mtl.com>

> On Thu, 2005-07-07 at 16:33, Eitan Zahavi wrote:
> > >
> > > Have you looked at the umad and mad libraries ? These are not IBAL.
> > [EZ] Yes I know. This is why they do not work on Windows IBAL.
> 
> Is it a requirement for them to work on Windows IBAL ?
[EZ] Yes - this is the access layer for win-ib ...
> 
> > > >  I think OpenSM vendor layer was proven to be adequate for this
> > > > purpose.
> > >
> > > and so have the umad and mad libraries used in the OpenIB Linux
> > stack.
> > [EZ] No they did not - the API they provide was never ported to any
> > other AL particularly not Windows.
> 
> Are you saying there is something in those APIs where they couldn't be
> ported to Windows ?
> 
> There is no reason the MAD and UMAD libraries couldn't be ported to
> Windows.
[EZ] It is not impossible - just does not make sense to me. If you port the
applications to OSMV you get the porting to Gen1/OpenIB/WinIB/IBMgtSim for
free.
> 
> -- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050708/3cbe766e/attachment.html>

From mlleinin at hpcn.ca.sandia.gov  Thu Jul  7 14:05:05 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Thu, 07 Jul 2005 14:05:05 -0700
Subject: [openib-general] RE: IBDM and IBMgtSim Proposal Comments
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30463@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30463@mtlex01.yok.mtl.com>
Message-ID: <1120770305.23483.133.camel@localhost>

On Fri, 2005-07-08 at 00:02 +0300, Eitan Zahavi wrote:
> > On Thu, 2005-07-07 at 16:33, Eitan Zahavi wrote: 
> > There is no reason the MAD and UMAD libraries couldn't be ported to 
> > Windows. 
> [EZ] It is not impossible - just does not make sense to me. If you
> port the applications to OSMV you get the porting to
> Gen1/OpenIB/WinIB/IBMgtSim for free.

  OpenIB is not supporting Gen1.  If Mellanox wants to support Gen1
that's your problem.

  - Matt





From viswak at yahoo.com  Thu Jul  7 14:09:14 2005
From: viswak at yahoo.com (viswanath krishnamurthy)
Date: Thu, 7 Jul 2005 14:09:14 -0700 (PDT)
Subject: [openib-general] Re: useraccess_cm sample client/server (gen1 )
In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35DCA@mtvex01.mtv.mtl.com>
Message-ID: <20050707210914.73234.qmail@web33215.mail.mud.yahoo.com>

Itamar,

Thanks.. I was able to use it.

One more question.. Once the connection is
established,
which API's needs to be used from the userland to send
and receive data.. Any sample code/pointers  is
appreciated.

Thanks,
-Vish
--- Itamar Rabenstein <itamar at mellanox.co.il> wrote:

> openib is working now on gen2.
> but if you want you can look at mellanox IBGD 1.7.0
> from  
> www.mellnaox.com follow the link "Download IB GOLD -
> 1.7.0"
> look for udapl code .
> The code is useing the user_cm IF
> 
> Itamar
> 
> > -----Original Message-----
> > From: viswanath krishnamurthy
> [mailto:viswak at yahoo.com]
> > Sent: Wednesday, July 06, 2005 8:28 PM
> > To: viswanath krishnamurthy;
> openib-general at openib.org
> > Subject: [openib-general] Re: useraccess_cm sample
> 
> > client/server (gen1)
> > 
> > 
> > I looked further into the whole gen1 source tree.
> > There is no consumer of this useraccess_cm API
> > (ioctl). Are there any consumers of this API's. Is
> it
> > supported ?
> > 
> > Thanks,
> > Vish
> > 
> > --- viswanath krishnamurthy <viswak at yahoo.com>
> wrote:
> > 
> > > Is there a sample code (examples) to use the
> gen1
> > > stack user level CM API (ioctls) ? Any pointers
> is
> > > appreciated.
> > > 
> > > Thanks,
> > > Vish
> > > 
> > > 
> > > 		
> > >
> ____________________________________________________
> > > 
> > > Yahoo! Sports 
> > > Rekindle the Rivalries. Sign up for Fantasy
> Football
> > > 
> > > http://football.fantasysports.yahoo.com
> > > 
> > 
> > 
> > 
> > 		
> >
> ____________________________________________________
> > Sell on Yahoo! Auctions - no fees. Bid on great
> items.  
> > http://auctions.yahoo.com/
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> > 
> > To unsubscribe, please visit 
> > http://openib.org/mailman/listinfo/openib-general
> > 
> 



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From yaronh at voltaire.com  Thu Jul  7 14:09:11 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Fri, 8 Jul 2005 00:09:11 +0300
Subject: [openib-general] IBDM and IBMgtSim Proposal Comments
Message-ID: <35EA21F54A45CB47B879F21A91F4862F64F524@taurus.voltaire.com>

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Fab Tillier
> Sent: Thursday, July 07, 2005 11:37 PM
> To: Hal Rosenstock; 'Eitan Zahavi'
> Cc: openib-windows at openib.org; openib-general at openib.org
> Subject: RE: [openib-general] IBDM and IBMgtSim Proposal Comments
> 
> > From: Hal Rosenstock [mailto:halr at voltaire.com]
> > Sent: Thursday, July 07, 2005 10:56 AM
> >
> > In the OpenIB architecture, umad is the lowest layer library and the
> > diagnostics are built on that.
> 
> That's only true in the *Linux* OpenIB Architecture.  Windows is
different
> - the
> access layer already provides support for user-level MAD clients, and
the
> API is
> very close (if not identical) to the IBAL interface OpenSM was
originally
> written to.
> 

>From my understanding the main advantage for using the OSM Vendor
specific layer is that it is also present in Windows ? 
or does it have some other advantage over the umad layer (from Hal's
response seems like umad has better layering/functionality) ?

If that is the case than you can also suggest to replace the OpenIB
verbs layer or CM, etc' with the IBAL one because its present in Windows


I believe if we want to do a major change in the management
infrastructure that is live and kicking (can probably improve like
always) 
We need a much better reason than "its done this way in Windows"

Yaron




From eitan at mellanox.co.il  Thu Jul  7 14:09:35 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Fri, 8 Jul 2005 00:09:35 +0300 
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30465@mtlex01.yok.mtl.com>

If Eitan was porting them, he'd have them run over the OSM vendor layer.  If
you were porting them, you'd port umad.
[EZ] This is true. As I posted long ago - we plan to post an extended
version of similat tools that runs on top of OSMV. The tools are under work
but the underling layer of MADs API will be posted next week or so.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050708/bfe28c1e/attachment.html>

From halr at voltaire.com  Thu Jul  7 14:04:45 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 17:04:45 -0400
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <000b01c58334$7d620100$6501a8c0@infiniconsys.com>
References: <000b01c58334$7d620100$6501a8c0@infiniconsys.com>
Message-ID: <1120770090.4386.144.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 16:43, Fab Tillier wrote:
> > From: Hal Rosenstock [mailto:halr at voltaire.com]

> How is porting umad any different than using the OSM vendor layer?

See below.

> umad is the lowest level API in Linux, but not in Windows.  So either the diagnostics
> interface to the lowest level layer (umad for Linux, IBAL for Windows), or the
> diagnostics interface to some higher abstraction layer.  If a higher abstraction
> layer, why not use the existing OSM vendor layer and skip porting umad to
> Windows all together?

In Linux, OSM vendor layer is implemented on top of umad. Whether it is
a higher abstraction layer is another matter. It is an abstraction with
different semantics and may be higher as I am not sure whether the umad
and mad libraries could be put on top of the OSM vendor layer but the
other way 'round works.

I looked at IBAL some time ago but can't comment now on how it compares.

So it appears there are 3 choices:
1. Port OpenIB Linux libraries to Windows and OpenIB Linux "diags" port
as well (second part is less work than next alternative)
2. Port OpenIB Linux diagnostics to OSM vendor layer
3. No OpenIB "Linux" diagnostics in the windows environment

-- Hal



From ftillier at silverstorm.com  Thu Jul  7 14:26:22 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 14:26:22 -0700
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <1120770090.4386.144.camel@hal.voltaire.com>
Message-ID: <000d01c5833a$865e8390$6501a8c0@infiniconsys.com>

> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Thursday, July 07, 2005 2:05 PM
> 
> On Thu, 2005-07-07 at 16:43, Fab Tillier wrote:
> > umad is the lowest level API in Linux, but not in Windows.  So either the
> > diagnostics interface to the lowest level layer (umad for Linux, IBAL for
> > Windows), or the diagnostics interface to some higher abstraction layer.
> > If a higher abstraction layer, why not use the existing OSM vendor layer
> > and skip porting umad to Windows all together?
> 
> In Linux, OSM vendor layer is implemented on top of umad. Whether it is
> a higher abstraction layer is another matter. It is an abstraction with
> different semantics and may be higher as I am not sure whether the umad
> and mad libraries could be put on top of the OSM vendor layer but the
> other way 'round works.

Since OSMV works over umad, IBAL, Gen1 and so forth, to me that makes it a
higher level abstraction.  I'm not here to debate which is higher - I don't
care.  I was just making the point that if the argument for interfacing to umad
in Linux is that it is the lowest level interface possible, that same argument
should be made for Windows.  Consistency in logic what I'm asking for here.

> 
> I looked at IBAL some time ago but can't comment now on how it compares.
> 
> So it appears there are 3 choices:
> 1. Port OpenIB Linux libraries to Windows and OpenIB Linux "diags" port
> as well (second part is less work than next alternative)
> 2. Port OpenIB Linux diagnostics to OSM vendor layer
> 3. No OpenIB "Linux" diagnostics in the windows environment

What about:
4. Port OpenIB Linux "diags" to Windows/IBAL.

Seems like a pretty obvious choice, and what I've been talking about in my
previous emails.

I don't care about OSMV - I care about the access layer in Windows.  The
functionality for user-level MAD clients is already there - why not use it?  If
there are problems with the exiting MAD interface, let's address them.  If that
ends up making the interface more similar to the Linux umad interface, so be it.

However, I'm not in favor of just porting umad to Windows because someone
randomly decreed that it is *the* user-mode MAD interface for the industry.  It
isn't.  umad in Linux is the functional equivalent of IBAL in Windows, period.

- Fab



From mst at mellanox.co.il  Thu Jul  7 14:27:10 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 8 Jul 2005 00:27:10 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707204630.GD22605@minantech.com>
References: <20050707152026.GB12222@minantech.com>
	<20050707153839.GA17994@mellanox.co.il>
	<20050707202404.GB22605@minantech.com>
	<20050707203253.GB28506@mellanox.co.il>
	<20050707204630.GD22605@minantech.com>
Message-ID: <20050707212710.GE28506@mellanox.co.il>

Quoting r. Gleb Natapov <gleb at minantech.com>:
> Tried that. But with big values create_qp fails.

Yes, we need to fix that.

-- 
MST


From mst at mellanox.co.il  Thu Jul  7 14:31:17 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 8 Jul 2005 00:31:17 +0300
Subject: [openib-general] Re: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <000d01c5833a$865e8390$6501a8c0@infiniconsys.com>
References: <1120770090.4386.144.camel@hal.voltaire.com>
	<000d01c5833a$865e8390$6501a8c0@infiniconsys.com>
Message-ID: <20050707213117.GF28506@mellanox.co.il>

Quoting r. Fab Tillier <ftillier at silverstorm.com>:
> umad in Linux is the functional equivalent of IBAL in Windows, period.
 
Doesnt IBAL also include uverbs, ucm and whatnot?

-- 
MST


From sean.hefty at intel.com  Thu Jul  7 14:35:36 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Thu, 7 Jul 2005 14:35:36 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050707212710.GE28506@mellanox.co.il>
Message-ID: <ORSMSX401TRqf69aZbL00000014@orsmsx401.amr.corp.intel.com>

>Quoting r. Gleb Natapov <gleb at minantech.com>:
>> Tried that. But with big values create_qp fails.
>
>Yes, we need to fix that.

To me, the current behavior sounds exactly like what you want.  The maximum or
some sort of "guaranteed value" should be available through the device
properties.  Needing to determine it only by creating a QP doesn't sound like
the right approach.

BTW, in tests that I've seen previously, using the largest possible inline data
size didn't yield the best system performance, but that could have simply been
based on the tests that we being run.

- Sean



From halr at voltaire.com  Thu Jul  7 14:35:32 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 17:35:32 -0400
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <000d01c5833a$865e8390$6501a8c0@infiniconsys.com>
References: <000d01c5833a$865e8390$6501a8c0@infiniconsys.com>
Message-ID: <1120772064.4386.169.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 17:26, Fab Tillier wrote:
> > So it appears there are 3 choices:
> > 1. Port OpenIB Linux libraries to Windows and OpenIB Linux "diags" port
> > as well (second part is less work than next alternative)
> > 2. Port OpenIB Linux diagnostics to OSM vendor layer
> > 3. No OpenIB "Linux" diagnostics in the windows environment
> 
> What about:
> 4. Port OpenIB Linux "diags" to Windows/IBAL.
> 
> Seems like a pretty obvious choice, and what I've been talking about in my
> previous emails.

Yes, that is another alternative but is similar to choice 2 in terms of
affecting all the applications.

-- Hal



From ftillier at silverstorm.com  Thu Jul  7 14:43:43 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 14:43:43 -0700
Subject: [openib-general] IBDM and IBMgtSim Proposal Comments
In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F64F524@taurus.voltaire.com>
Message-ID: <000e01c5833c$f34b1e30$6501a8c0@infiniconsys.com>

> From: Yaron Haviv [mailto:yaronh at voltaire.com]
> Sent: Thursday, July 07, 2005 2:09 PM
> 
> > -----Original Message-----
> > From: openib-general-bounces at openib.org [mailto:openib-general-
> > bounces at openib.org] On Behalf Of Fab Tillier
> > Sent: Thursday, July 07, 2005 11:37 PM
> >
> > > From: Hal Rosenstock [mailto:halr at voltaire.com]
> > > Sent: Thursday, July 07, 2005 10:56 AM
> > >
> > > In the OpenIB architecture, umad is the lowest layer library and the
> > > diagnostics are built on that.
> >
> > That's only true in the *Linux* OpenIB Architecture.  Windows is
> > different - the access layer already provides support for user-level
> > MAD clients, and the API is very close (if not identical) to the
> > IBAL interface OpenSM was originally written to.
> 
> From my understanding the main advantage for using the OSM Vendor
> specific layer is that it is also present in Windows ?

OSMV already exists over a variety of transports, as Eitan mentioned.  It is one
option for developing portable MAD-based tools and diagnostics.

> or does it have some other advantage over the umad layer (from Hal's
> response seems like umad has better layering/functionality) ?

I'm not familiar with it, so I can't answer what functional advantage it has.  I
would expect that as an abstraction layer, it will hide some of the
functionality in umad, just like it likely hides functionality in IBAL.  That's
the price of using a higher level abstraction.  Hal seems to be making the
argument that the lowest layer is the best to use for Linux, but somehow not for
Windows.  I'm just questioning the inconsistency, independent of OSMV.

> If that is the case than you can also suggest to replace the OpenIB
> verbs layer or CM, etc' with the IBAL one because its present in Windows

No, that wasn't the point.  The point was that we have CM, verbs, MAD and so
forth support in Linux and Windows, independent of OSMV, and more importantly
independent of one another.  Just like I'm not suggesting moving the Windows
code to Linux, I'm pushing back against gratuitously moving Linux code to
Windows.  If there's a reason to do it, great.  If it's just because the Linux
code works, I'll be one of the first to point out that the Windows code works
too.

> I believe if we want to do a major change in the management
> infrastructure that is live and kicking (can probably improve like
> always)
> We need a much better reason than "its done this way in Windows"

Turn your above statement around: "I believe if we want to do a major change in
the management infrastructure that is live and kicking (can probably improve
like always) we need a much better reason than "its done this way in Linux"

We're not talking about changing the umad interface in Linux.  Eitan is
proposing having all diagnostics interface to OSVM to facilitate portability.
Hal is proposing porting umad to Windows.  I'm saying just use IBAL - it's
already there and works.

- Fab




From ftillier at silverstorm.com  Thu Jul  7 14:44:58 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 14:44:58 -0700
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <20050707213117.GF28506@mellanox.co.il>
Message-ID: <000f01c5833d$1f6c1d70$6501a8c0@infiniconsys.com>

> From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
> Sent: Thursday, July 07, 2005 2:31 PM
> 
> Quoting r. Fab Tillier <ftillier at silverstorm.com>:
> > umad in Linux is the functional equivalent of IBAL in Windows, period.
> 
> Doesnt IBAL also include uverbs, ucm and whatnot?

Yes, in Windows all the components are integrated into a single access layer and
work in harmony.  So let me rephrase:  umad in Linux is the functional
equivalent of the MAD services of IBAL in Windows, period.

- Fab




From mst at mellanox.co.il  Thu Jul  7 14:46:36 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 8 Jul 2005 00:46:36 +0300
Subject: [openib-general] Re: Re: calculate inline data size
In-Reply-To: <ORSMSX401TRqf69aZbL00000014@orsmsx401.amr.corp.intel.com>
References: <20050707212710.GE28506@mellanox.co.il>
	<ORSMSX401TRqf69aZbL00000014@orsmsx401.amr.corp.intel.com>
Message-ID: <20050707214636.GH28506@mellanox.co.il>

Quoting r. Sean Hefty <sean.hefty at intel.com>:
> BTW, in tests that I've seen previously, using the largest possible inline data
> size didn't yield the best system performance, but that could have simply been
> based on the tests that we being run.

True. inline needs a data copy into the queue.
So for large buffers even though you save a s/g read, you lose 
latency and bandwidth by using inline unless you have a very fast
CPU.
The actual size depends on the CPU/memory.

-- 
MST


From ftillier at silverstorm.com  Thu Jul  7 14:49:30 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 14:49:30 -0700
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <1120772064.4386.169.camel@hal.voltaire.com>
Message-ID: <001001c5833d$c1cdf070$6501a8c0@infiniconsys.com>

> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Thursday, July 07, 2005 2:36 PM
> 
> On Thu, 2005-07-07 at 17:26, Fab Tillier wrote:
> > > So it appears there are 3 choices:
> > > 1. Port OpenIB Linux libraries to Windows and OpenIB Linux "diags" port
> > > as well (second part is less work than next alternative)
> > > 2. Port OpenIB Linux diagnostics to OSM vendor layer
> > > 3. No OpenIB "Linux" diagnostics in the windows environment
> >
> > What about:
> > 4. Port OpenIB Linux "diags" to Windows/IBAL.
> >
> > Seems like a pretty obvious choice, and what I've been talking about in my
> > previous emails.
> 
> Yes, that is another alternative but is similar to choice 2 in terms of
> affecting all the applications.

All the applications will be affected by being ported from Linux to Windows, not
to mention changes in the umad interface necessary to make it work on Windows.
It has the benefits that the tools interface to the lowest API available to
them.  If interfacing at the lowest layer isn't important, the choice is between
porting the diags to OSMV or porting umad to Windows.  I don't know which is
easier, and it probably doesn't matter.  What matters is which gets done first.

However, it doesn't sound that an effort from Eitan to port the diagnostics to
OSMV would be accepted into the Linux SVN repository simply on the basis that it
isn't umad.  What gives?

- Fab



From sean.hefty at intel.com  Thu Jul  7 14:52:47 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Thu, 7 Jul 2005 14:52:47 -0700
Subject: [Openib-windows] RE: [openib-general] IBDM and IBMgtSim
	ProposalComments
In-Reply-To: <000e01c5833c$f34b1e30$6501a8c0@infiniconsys.com>
Message-ID: <ORSMSX401Rqf69aZbLA00000015@orsmsx401.amr.corp.intel.com>

>I'm not familiar with it, so I can't answer what functional advantage it has.
>I
>would expect that as an abstraction layer, it will hide some of the
>functionality in umad, just like it likely hides functionality in IBAL.  That's
>the price of using a higher level abstraction.  Hal seems to be making the
>argument that the lowest layer is the best to use for Linux, but somehow not
>for
>Windows.  I'm just questioning the inconsistency, independent of OSMV.

Unless I'm mistaken, there are more than a dozen diag tools running on Linux
that are coded to umad.  I think that Hal sees porting umad as the easiest way
to convert those utilities to Windows.

>We're not talking about changing the umad interface in Linux.  Eitan is
>proposing having all diagnostics interface to OSVM to facilitate portability.
>Hal is proposing porting umad to Windows.  I'm saying just use IBAL - it's
>already there and works.

How many programs are currently coded to OSMV?

It seems to me that the person writing the code will decide whether they should
interface at the lowest layer or write to a higher level abstraction for
portability.  I don't see that either way is wrong.  And if the higher level
abstraction is the native interface on one platform, that sounds like a benefit.

- Sean



From ftillier at silverstorm.com  Thu Jul  7 15:07:53 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Thu, 7 Jul 2005 15:07:53 -0700
Subject: [Openib-windows] RE: [openib-general] IBDM and IBMgtSim
	ProposalComments
In-Reply-To: <ORSMSX401Rqf69aZbLA00000015@orsmsx401.amr.corp.intel.com>
Message-ID: <001101c58340$53487410$6501a8c0@infiniconsys.com>

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> Sent: Thursday, July 07, 2005 2:53 PM
> 
> It seems to me that the person writing the code will decide whether they
> should
> interface at the lowest layer or write to a higher level abstraction for
> portability.  I don't see that either way is wrong.  And if the higher level
> abstraction is the native interface on one platform, that sounds like a
> benefit.

It seems unclear to me whether changes made to the existing IB diags to make
them run over OSMV would be accepted, even if Eitan made the changes.  That's my
impression given the dialogue in this thread - there's an attitude of umad or
bust.

Aside from that, I agree - whoever gets it done first gets it done their way.

- Fab




From arlin.r.davis at intel.com  Thu Jul  7 15:53:11 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Thu, 7 Jul 2005 15:53:11 -0700
Subject: [openib-general] [PATCH] uDAPL with IB_CM support
Message-ID: <ORSMSX401ryWtIIZS2T00000016@orsmsx401.amr.corp.intel.com>

James,

Patch for OpenIB uDAPL provider with IB_CM support instead of SOCKET_CM. It was a little too messy
to make conditional and keep SOCKET_CM around so this version only supports IB_CM. Path records are
hand rolled until IBAT support is added in the next 1-2 weeks. See README for details on setup and
limitations. 

Signed-off-by: Arlin Davis <ardavis at ichips.intel.com>

Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile	(revision 2811)
+++ dapl/udapl/Makefile	(working copy)
@@ -122,7 +122,7 @@
 #
 ifeq ($(VERBS),openib)
 PROVIDER = $(TOPDIR)/../openib
-CFLAGS   += -DSOCKET_CM -DOPENIB -DCQ_WAIT_OBJECT
+CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
 CFLAGS   += -I/usr/local/include/infiniband
 endif
 
Index: dapl/openib/TODO
===================================================================
--- dapl/openib/TODO	(revision 2818)
+++ dapl/openib/TODO	(working copy)
@@ -4,16 +4,12 @@
 - query call to get current qp state 
 - ibv_get_cq_event() needs timed event call and wakeup
 - query call to get device attributes
-- current implementation only supports one event per device
 - memory window support
 
 DAPL:
-- Build udapl issues with mthca having reverse dependencies to ibverbs
-- When real CM arrives: change modify_qp_state RTS RTR calls
 - reinit EP needs a QP timewait completion notification
-- code disconnect clean 
 - add cq_object wakeup, time based cq_object wait when verbs support arrives
-- update uDAPL code with real CM and ATS support
+- update uDAPL code with real ATS support
 - etc, etc.
 
 Other:
Index: dapl/openib/dapl_ib_dto.h
===================================================================
--- dapl/openib/dapl_ib_dto.h	(revision 2818)
+++ dapl/openib/dapl_ib_dto.h	(working copy)
@@ -143,6 +143,7 @@
 	ib_data_segment_t	*ds_array_p;
 	struct ibv_send_wr	wr;
 	struct ibv_send_wr	*bad_wr;
+	ib_hca_tranport_t	*ibt_ptr = &ep_ptr->header.owner_ia->hca_ptr->ib_trans;
 	DAT_COUNT		i, total_len;
 	
 	dapl_dbg_log (DAPL_DBG_TYPE_EP,
@@ -195,6 +196,11 @@
 				wr.wr.rdma.rkey, wr.wr.rdma.remote_addr );
 	}
 
+	/* inline data for send or write ops */
+	if ((total_len <= ibt_ptr->max_inline_send ) && 
+	   ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE))) 
+		wr.send_flags |= IBV_SEND_INLINE;
+	
 	/* set completion flags in work request */
 	wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG & 
 				completion_flags) ? 0 : IBV_SEND_SIGNALED;
Index: dapl/openib/dapl_ib_util.c
===================================================================
--- dapl/openib/dapl_ib_util.c	(revision 2818)
+++ dapl/openib/dapl_ib_util.c	(working copy)
@@ -58,34 +58,82 @@
 #include <sys/utsname.h>
 #include <unistd.h>	
 #include <fcntl.h>
+#include <strings.h>
 
 int g_dapl_loopback_connection = 0;
 
-#ifdef SOCKET_CM
+/* get lid */
+int dapli_get_lid(struct dapl_hca *hca_ptr, int port, uint16_t *lid )
+{
+	struct ibv_port_attr attr;
+
+	if (ibv_query_port(hca_ptr->ib_hca_handle, port, &attr))
+		return 1;
+
+	*lid = attr.lid;
+
+	return 0;
+}
+
+/* get gid */
+int dapli_get_gid(struct dapl_hca *hca_ptr, int port,
+                         int index, union ibv_gid *gid )
+{
+        /* ibv_query_gid() coming soon, until then HACK */
+        char path[128];
+        char val[40];
+        char name[256];
+        char *token;
+        uint16_t *p_gid;
+
+        if (sysfs_get_mnt_path(path, sizeof path)) {
+                fprintf(stderr, "Couldn't find sysfs mount.\n");
+                return 1;
+        }
+        sprintf(name, "%s/class/infiniband/%s/ports/%d/gids/%d", path,
+                 ibv_get_device_name(hca_ptr->ib_trans.ib_dev), port, index);
+
+        if (sysfs_read_attribute_value(name, val, sizeof val)) {
+                fprintf(stderr, "Couldn't read GID at %s\n", name);
+                return 1;
+        }
+
+        /* get token strings with delimiter */
+        token = strtok(val,":");
+        p_gid = (uint16_t*)gid->raw;
+        while (token) {
+                *p_gid = strtoul(token,NULL,16);
+		*p_gid = htons(*p_gid); /* convert each token to network order */
+                token = strtok(NULL,":");
+                p_gid++;
+        }
+        return 0;
+}
+
+
 /* just get IP address for hostname */
-DAT_RETURN getipaddr( char *addr, int addr_len)
+int dapli_get_addr( char *addr, int addr_len)
 {
 	struct sockaddr_in	*ipv4_addr = (struct sockaddr_in*)addr;
 	struct hostent		*h_ptr;
 	struct utsname		ourname;
 
 	if ( uname( &ourname ) < 0 ) 
-		return DAT_INTERNAL_ERROR;
+		return 1;
 
 	h_ptr = gethostbyname( ourname.nodename );
 	if ( h_ptr == NULL ) 
-		return DAT_INTERNAL_ERROR;
+		return 1;
 
 	if ( h_ptr->h_addrtype == AF_INET ) {
 		ipv4_addr = (struct sockaddr_in*) addr;
 		ipv4_addr->sin_family = AF_INET;
 		dapl_os_memcpy( &ipv4_addr->sin_addr, h_ptr->h_addr_list[0], 4 );
 	} else 
-		return DAT_INVALID_ADDRESS;
+		return 1;
 
-	return DAT_SUCCESS;
+	return 0;
 }
-#endif
 
 /*
  * dapls_ib_init, dapls_ib_release
@@ -104,11 +152,15 @@
  */
 int32_t dapls_ib_init (void)
 {	
-	return 0;
+	if (dapli_cm_thread_init())
+		return -1;
+	else
+		return 0;
 }
 
 int32_t dapls_ib_release (void)
 {
+	dapli_cm_thread_destroy();
 	return 0;
 }
 
@@ -160,48 +212,50 @@
 
 	hca_ptr->ib_hca_handle = ibv_open_device(hca_ptr->ib_trans.ib_dev);
 	if (!hca_ptr->ib_hca_handle) {
-		 dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
+		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
 				" open_hca: IB dev open failed for %s\n", 
 				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
 		return DAT_INTERNAL_ERROR;
 	}
   
-#ifdef SOCKET_CM
-	/* initialize cr_list lock */
-	dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.lock);
-	if (dat_status != DAT_SUCCESS)
-	{
+	/* set inline max with enviromment or default, get local lid and gid 0 */
+	hca_ptr->ib_trans.max_inline_send = 
+		dapl_os_get_env_val ( "DAPL_MAX_INLINE", INLINE_SEND_DEFAULT );
+
+	if ( dapli_get_lid(hca_ptr, hca_ptr->port_num,
+			   &hca_ptr->ib_trans.lid )) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
-			" open_hca: failed to init lock\n");
-		return dat_status;
+				" open_hca: IB get LID failed for %s\n", 
+				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		return DAT_INTERNAL_ERROR;
 	}
 
-	/* initialize CM list for listens on this HCA */
-	dapl_llist_init_head(&hca_ptr->ib_trans.list);
-
-	/* create thread to process inbound connect request */
-	dat_status = dapl_os_thread_create(cr_thread, 
-					   (void*)hca_ptr, 
-					   &hca_ptr->ib_trans.thread );
-	if (dat_status != DAT_SUCCESS)
-	{
+	if ( dapli_get_gid(hca_ptr, hca_ptr->port_num, 0,  
+			   &hca_ptr->ib_trans.gid )) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
-		" open_hca: failed to create thread\n");
-		return dat_status;
+				" open_hca: IB get GID failed for %s\n", 
+				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		return DAT_INTERNAL_ERROR;
 	}
-
+	
 	/* get the IP address of the device */
-	dat_status = getipaddr((char*)&hca_ptr->hca_address, 
-				sizeof(DAT_SOCK_ADDR6) );
+	if ( dapli_get_addr((char*)&hca_ptr->hca_address, 
+			    sizeof(DAT_SOCK_ADDR6) )) {
+		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
+				" open_hca: IB get ADDR failed for %s\n", 
+				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		return DAT_INTERNAL_ERROR;
+	}
+
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
-		" open_hca: %s, port %d, %s  %d.%d.%d.%d\n", 
+		" open_hca: %s, port %d, %s  %d.%d.%d.%d INLINE_MAX=%d\n", 
 		ibv_get_device_name(hca_ptr->ib_trans.ib_dev), hca_ptr->port_num,
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_family == AF_INET ?
"AF_INET":"AF_INET6",
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 0 & 0xff,
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 8 & 0xff,
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 16 & 0xff,
-		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff );
-#endif
+		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff,
+		hca_ptr->ib_trans.max_inline_send );
 
 	return dat_status;
 }
@@ -232,20 +286,6 @@
 			return(dapl_convert_errno(errno,"ib_close_device"));
 		hca_ptr->ib_hca_handle = IB_INVALID_HANDLE;
 	}
-
-#if SOCKET_CM
-	/* destroy cr_thread and lock */
-	hca_ptr->ib_trans.destroy = 1;
-	while (hca_ptr->ib_trans.destroy) {
-		struct timespec	sleep, remain;
-		sleep.tv_sec = 0;
-		sleep.tv_nsec = 10000000; /* 10 ms */
-		dapl_dbg_log(DAPL_DBG_TYPE_UTIL, 
-			     " close_hca: waiting for cr_thread\n");
-		nanosleep (&sleep, &remain);
-	}
-	dapl_os_lock_destroy(&hca_ptr->ib_trans.lock);
-#endif
 	return (DAT_SUCCESS);
 }
   
@@ -407,3 +447,36 @@
     }
     return DAT_SUCCESS;
 }
+
+#ifdef PROVIDER_SPECIFIC_ATTR
+
+/*
+ * dapls_set_provider_specific_attr
+ *
+ * Input:
+ *	attr_ptr		Pointer provider attributes
+ *
+ * Output:
+ * 	none
+ *
+ * Returns:
+ * 	void
+ */
+DAT_NAMED_ATTR	ib_attrs[] = {
+    {
+    	"I_DAT_SEND_INLINE_THRESHOLD",
+    	"128"
+    },
+};
+
+#define SPEC_ATTR_SIZE( x )	(sizeof( x ) / sizeof( DAT_NAMED_ATTR))
+
+void dapls_set_provider_specific_attr(
+	IN DAT_PROVIDER_ATTR	*attr_ptr )
+{
+	attr_ptr->num_provider_specific_attr = SPEC_ATTR_SIZE( ib_attrs );
+	attr_ptr->provider_specific_attr     = ib_attrs;
+}
+
+#endif
+
Index: dapl/openib/dapl_ib_cm.c
===================================================================
--- dapl/openib/dapl_ib_cm.c	(revision 2818)
+++ dapl/openib/dapl_ib_cm.c	(working copy)
@@ -52,451 +52,687 @@
 #include "dapl_cr_util.h"
 #include "dapl_name_service.h"
 #include "dapl_ib_util.h"
+#include <sys/poll.h>
+#include <signal.h>
 
-#ifdef SOCKET_CM
+/* prototypes */
+static void dapli_path_comp_handler(uint64_t req_id, void *context, int rec_num);
+static void dapli_rt_comp_handler(uint64_t req_id, void *context, int rec_num);
+static void dapli_rep_recv(struct dapl_cm_id *conn, struct ib_cm_event *rep_recv_param);
+static struct dapl_cm_id * dapli_req_recv(struct dapl_cm_id *conn, struct ib_cm_event *event);
+static int dapli_cm_active_cb(struct dapl_cm_id *conn, struct ib_cm_event *event);
+static int dapli_cm_passive_cb(struct dapl_cm_id *conn, struct ib_cm_event *event);
+
+/* prototypes */
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+static inline uint64_t cpu_to_be64(uint64_t x) { return bswap_64(x); }
+#elif __BYTE_ORDER == __BIG_ENDIAN
+static inline uint64_t cpu_to_be64(uint64_t x) { return x; }
+#endif
+
+#ifndef IB_AT
 
 #include <stdio.h>
 #include <unistd.h>
 #include <fcntl.h>
 #include <netinet/tcp.h>
 #include <sysfs/libsysfs.h>
+#include <signal.h>
 
-/* prototypes */
-static uint16_t dapli_get_lid( struct ibv_device *dev, int port );
+/* iclust-20 hard coded values, network order */
+#define REMOTE_GID	"fe80:0000:0000:0000:0002:c902:0000:4071"
+#define REMOTE_LID	"0002"
+
+static int			g_cm_destroy;
+static DAPL_OS_THREAD		g_cm_thread;
+static DAPL_OS_LOCK		g_cm_lock;
+static struct dapl_llist_entry	*g_cm_list;	
+
+int dapli_cm_thread_init(void)
+{
+	DAT_RETURN dat_status;
 
-static DAT_RETURN dapli_socket_connect ( DAPL_EP		*ep_ptr,
-					 DAT_IA_ADDRESS_PTR	r_addr,
-					 DAT_CONN_QUAL		r_qual,
-					 DAT_COUNT		p_size,
-					 DAT_PVOID		p_data );
-
-static DAT_RETURN dapli_socket_listen ( DAPL_IA			*ia_ptr,
-					DAT_CONN_QUAL		serviceID,
-					DAPL_SP			*sp_ptr );
-
-static DAT_RETURN dapli_socket_accept(	ib_cm_srvc_handle_t cm_ptr );
-
-static DAT_RETURN dapli_socket_accept_final(	DAPL_EP		*ep_ptr,
-						DAPL_CR		*cr_ptr,
-						DAT_COUNT	p_size,
-						DAT_PVOID	p_data );
-
-/* XXX temporary hack to get lid */
-static uint16_t dapli_get_lid(IN struct ibv_device *dev, IN int port)
-{
-	char path[128];
-	char val[16];
-	char name[256];
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_init(%d)\n", getpid());
 
-	if (sysfs_get_mnt_path(path, sizeof path)) {
-		fprintf(stderr, "Couldn't find sysfs mount.\n");
-		return 0;
+        /* initialize cr_list lock */
+	dapl_os_lock_init(&g_cm_lock);
+	
+	/* initialize CM list for listens on this HCA */
+	dapl_llist_init_head(&g_cm_list);
+
+	/* create thread to process inbound connect request */
+	dat_status = dapl_os_thread_create(cm_thread, NULL, &g_cm_thread);
+	if (dat_status != DAT_SUCCESS)
+	{
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
+			     " cm_thread_init: failed to create thread\n");
+		return 1;
 	}
-	sprintf(name, "%s/class/infiniband/%s/ports/%d/lid", path,
-		 ibv_get_device_name(dev), port);
+	return 0;
+}
 
-	if (sysfs_read_attribute_value(name, val, sizeof val)) {
-		fprintf(stderr, "Couldn't read LID at %s\n", name);
-		return 0;
+void dapli_cm_thread_destroy(void)
+{
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d)\n", getpid());
+
+	/* destroy cr_thread and lock */
+	g_cm_destroy = 1;
+	pthread_kill( g_cm_thread, SIGUSR1 );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d) SIGUSR1 sent\n",getpid());
+	while (g_cm_destroy) {
+		struct timespec	sleep, remain;
+		sleep.tv_sec = 0;
+		sleep.tv_nsec = 200000000; /* 200 ms */
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " cm_thread_destroy: waiting for cm_thread\n");
+		nanosleep (&sleep, &remain);
 	}
-	return strtol(val, NULL, 0);
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d) exit\n",getpid());
 }
 
-/*
- * ACTIVE: Create socket, connect, and exchange QP information 
- */
-static DAT_RETURN 
-dapli_socket_connect (	DAPL_EP			*ep_ptr,
-			DAT_IA_ADDRESS_PTR	r_addr,
-			DAT_CONN_QUAL		r_qual,
-			DAT_COUNT		p_size,
-			DAT_PVOID		p_data )
-{
-	ib_cm_handle_t	cm_ptr;
-	DAPL_IA		*ia_ptr = ep_ptr->header.owner_ia;
-	int		len, opt = 1;
-	struct iovec    iovec[2];
-	short		rtu_data = htons(0x0E0F);
+static int ib_at_route_by_ip(uint32_t dst_ip, uint32_t src_ip, int tos, uint16_t flags,
+			     struct ib_at_ib_route *ib_route,
+			     struct ib_at_completion *async_comp)
+{
+	struct dapl_cm_id *conn = (struct dapl_cm_id *)async_comp->context;
 	
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d\n", r_qual);
-			
-	/*
-	 *  Allocate CM and initialize
-	 */
-	if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL ) {
-		return DAT_INSUFFICIENT_RESOURCES;
+	dapl_dbg_log (
+		DAPL_DBG_TYPE_CM, 
+		" CM at_route_by_ip: conn %p cm_id %d src %d.%d.%d.%d -> dst %d.%d.%d.%d (%d)\n", 
+		conn,conn->cm_id,
+		src_ip >> 0 & 0xff, src_ip >> 8 & 0xff,
+		src_ip >> 16 & 0xff,src_ip >> 24 & 0xff,
+		dst_ip >> 0 & 0xff, dst_ip >> 8 & 0xff,
+		dst_ip >> 16 & 0xff,dst_ip >> 24 & 0xff, conn->service_id);
+
+	/* use req_id for loopback indication */
+	if (( src_ip == dst_ip ) || ( dst_ip == 0x0100007f ))
+		async_comp->req_id = 1;
+	else
+		async_comp->req_id = 0;
+		
+	return 1;			     
+}
+
+static int ib_at_paths_by_route(struct ib_at_ib_route *ib_route, uint32_t mpath_type,
+				struct ib_sa_path_rec *pr, int npath,
+				struct ib_at_completion *async_comp)
+{
+	struct dapl_cm_id *conn = (struct dapl_cm_id *)async_comp->context;
+	char *env, *token;
+	char  dgid[40];
+	uint16_t *p_gid = (uint16_t*)&ib_route->gid;
+
+	/* set local path record values and send to remote */
+	(void)dapl_os_memzero(pr, sizeof(*pr));
+
+	pr->slid = htons(conn->hca->ib_trans.lid);
+	pr->sgid.global.subnet_prefix = conn->hca->ib_trans.gid.global.subnet_prefix;
+	pr->sgid.global.interface_id  = conn->hca->ib_trans.gid.global.interface_id;
+
+	env = getenv("DAPL_REMOTE_LID");
+	if ( env == NULL )
+		env = REMOTE_LID;
+	ib_route->lid = strtol(env,NULL,0);
+
+	env = getenv("DAPL_REMOTE_GID");
+	if ( env == NULL )
+		env = REMOTE_GID;
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " ib_at_paths_by_route: remote LID %x GID %s\n",
+		     ib_route->lid,env);
+
+	dapl_os_memcpy( dgid, env, 40 ); 
+
+	/* get GID with token strings and delimiter */
+	token = strtok(dgid,":");
+	while (token) {
+		*p_gid = strtoul(token,NULL,16);
+		*p_gid = htons(*p_gid); /* convert each token to network order */
+		token = strtok(NULL,":");
+		p_gid++;
+	}
+       
+	/* set remote lid and gid, req_id is indication of loopback */
+	if ( !async_comp->req_id ) {
+		pr->dlid = htons(ib_route->lid);
+		pr->dgid.global.subnet_prefix = ib_route->gid.global.subnet_prefix;
+		pr->dgid.global.interface_id  = ib_route->gid.global.interface_id;
+	} else {
+		pr->dlid = pr->slid;
+		pr->dgid.global.subnet_prefix = pr->sgid.global.subnet_prefix;
+		pr->dgid.global.interface_id  = pr->sgid.global.interface_id;
 	}
 
-	(void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) );
-	cm_ptr->socket = -1;
+	pr->reversible = 0x1000000;
+	pr->pkey = 0xffff;
+	pr->mtu  = IBV_MTU_1024;
+	pr->mtu_selector  = 2;
+	pr->rate_selector = 2;
+	pr->rate          = 3;
+	pr->packet_life_time_selector = 2;
+	pr->packet_life_time          = 2;
+	
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " ib_at_paths_by_route: SRC LID 0x%x GID subnet %016llx id %016llx\n",
+		     pr->slid,(unsigned long long)(pr->sgid.global.subnet_prefix),
+		     (unsigned long long)(pr->sgid.global.interface_id) );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " ib_at_paths_by_route: DST LID 0x%x GID subnet %016llx id %016llx\n",
+		     pr->dlid,(unsigned long long)(pr->dgid.global.subnet_prefix),
+		     (unsigned long long)(pr->dgid.global.interface_id) );
 
-	/* create, connect, sockopt, and exchange QP information */
-	if ((cm_ptr->socket = socket(AF_INET,SOCK_STREAM,0)) < 0 ) {
-		dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-		return DAT_INSUFFICIENT_RESOURCES;
-	}
+	dapli_path_comp_handler( async_comp->req_id, (void*)conn, 1);
 
-	((struct sockaddr_in*)r_addr)->sin_port = htons(r_qual);
+	return 0;
+}
 
-	if ( connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0 ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
-			     " connect: %s on r_qual %d\n",
-			     strerror(errno), (unsigned int)r_qual);
-		dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-		return DAT_INVALID_ADDRESS;
-	}
-	setsockopt(cm_ptr->socket,IPPROTO_TCP,TCP_NODELAY,&opt,sizeof(opt));
-	
-	/* Send QP info, IA address, and private data */
-	cm_ptr->dst.qpn = ep_ptr->qp_handle->qp_num;
-	cm_ptr->dst.port = ia_ptr->hca_ptr->port_num;
-	cm_ptr->dst.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, 
-					 ia_ptr->hca_ptr->port_num );
-	cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address;
-	cm_ptr->dst.p_size = p_size;
-	iovec[0].iov_base = &cm_ptr->dst;
-	iovec[0].iov_len  = sizeof(ib_qp_cm_t);
-	if ( p_size ) {
-		iovec[1].iov_base = p_data;
-		iovec[1].iov_len  = p_size;
-	}
-	len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) );
-    	if ( len != (p_size + sizeof(ib_qp_cm_t)) ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " connect write: ERR %s, wcnt=%d\n",
-			     strerror(errno), len); 
-		goto bail;
-	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		     " connect: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		     cm_ptr->dst.port, cm_ptr->dst.lid, 
-		     cm_ptr->dst.qpn, cm_ptr->dst.p_size ); 
-
-	/* read DST information into cm_ptr, overwrite SRC info */
-	len = readv( cm_ptr->socket, iovec, 1 );
-	if ( len != sizeof(ib_qp_cm_t) ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " connect read: ERR %s, rcnt=%d\n",
-			     strerror(errno), len); 
-		goto bail;
-	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		     " connect: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		     cm_ptr->dst.port, cm_ptr->dst.lid, 
-		     cm_ptr->dst.qpn, cm_ptr->dst.p_size ); 
+#endif /* ifndef IB_AT */
 
-	/* validate private data size before reading */
-	if ( cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " connect read: psize (%d) wrong\n",
-			     cm_ptr->dst.p_size ); 
-		goto bail;
-	}
+static void dapli_path_comp_handler(uint64_t req_id, void *context, int rec_num)
+{
+	struct dapl_cm_id *conn = context;
+	int status;
+	ib_cm_events_t event;
+
+	if (rec_num <= 0) {
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " path_comp_handler: resolution err %d retry %d\n",
+			     rec_num, conn->retries + 1);
+		if (++conn->retries > IB_MAX_AT_RETRY) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				     " path_comp_handler: ep_ptr 0x%p\n",conn->ep);
+			event = IB_CME_DESTINATION_UNREACHABLE;
+			goto bail;
+		}
 
-	/* read private data into cm_handle if any present */
-	if ( cm_ptr->dst.p_size ) {
-		iovec[0].iov_base = cm_ptr->p_data;
-		iovec[0].iov_len  = cm_ptr->dst.p_size;
-		len = readv( cm_ptr->socket, iovec, 1 );
-		if ( len != cm_ptr->dst.p_size ) {
-			dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-				" connect read pdata: ERR %s, rcnt=%d\n",
-				strerror(errno), len); 
+		status = ib_at_paths_by_route(&conn->dapl_rt, 0,
+					      &conn->dapl_path, 1,
+					      &conn->dapl_comp);
+		if (status) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				     " path_by_route: err %d id %lld\n",
+				     status, conn->dapl_comp.req_id);
+			event = IB_CME_LOCAL_FAILURE;
 			goto bail;
 		}
+		return;
 	}
 
-	/* modify QP to RTR and then to RTS with remote info */
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS )
-		goto bail;
-
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS )
+	/* Mellanox performance workaround - best performance is MTU of 1024 */
+	if (conn->dapl_path.mtu > IBV_MTU_1024)    
+		conn->dapl_path.mtu = IBV_MTU_1024;
+
+	conn->req.service_id = cpu_to_be64(conn->service_id);
+	conn->req.primary_path = &conn->dapl_path;
+	conn->req.alternate_path = NULL;
+
+	status = ib_cm_send_req(conn->cm_id, &conn->req);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_send_cm_req failed: %s\n",
+			     strerror(errno));
+		event = IB_CME_LOCAL_FAILURE;
 		goto bail;
-		 
-	ep_ptr->qp_state = IB_QP_STATE_RTS;
-
-	/* complete handshake after final QP state change */
-	write(cm_ptr->socket, &rtu_data, sizeof(rtu_data) );
-
-	/* init cm_handle and post the event with private data */
-	ep_ptr->cm_handle = cm_ptr;
-	dapl_dbg_log( DAPL_DBG_TYPE_EP," ACTIVE: connected!\n" ); 
-	dapl_evd_connection_callback(   ep_ptr->cm_handle, 
-					IB_CME_CONNECTED, 
-					cm_ptr->p_data, 
-					ep_ptr );	
-	return DAT_SUCCESS;
+	}
+	return;
 
 bail:
-	/* close socket, free cm structure and post error event */
-	if ( cm_ptr->socket >= 0 ) 
-		close(cm_ptr->socket);
-	dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */
-
-	dapl_evd_connection_callback(	ep_ptr->cm_handle, 
-					IB_CME_LOCAL_FAILURE, 
-					NULL, 
-					ep_ptr );
-	return DAT_INTERNAL_ERROR;
+	dapl_evd_connection_callback(conn, event, NULL, conn->ep);
 }
 
+static void dapli_rt_comp_handler(uint64_t req_id, void *context, int rec_num)
+{
+	struct dapl_cm_id *conn = context;
+	int status;
+	ib_cm_events_t event;
 
-/*
- * PASSIVE: Create socket, listen, accept, exchange QP information 
- */
-static DAT_RETURN 
-dapli_socket_listen (	DAPL_IA		*ia_ptr,
-			DAT_CONN_QUAL	serviceID,
-			DAPL_SP		*sp_ptr )
-{
-	struct sockaddr_in	addr;
-	ib_cm_srvc_handle_t	cm_ptr = NULL;
-	int			opt = 1;
-	DAT_RETURN		dat_status = DAT_SUCCESS;
+	if (rec_num <= 0) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " dapl_rt_comp_handler: rec %d retry %d\n",
+			     rec_num, conn->retries+1 ); 
 
-	dapl_dbg_log (	DAPL_DBG_TYPE_EP,
-			" listen(ia_ptr %p ServiceID %d sp_ptr %p)\n",
-			ia_ptr, serviceID, sp_ptr);
+		if (++conn->retries > IB_MAX_AT_RETRY) {
+			event = IB_CME_DESTINATION_UNREACHABLE;
+			goto bail;
+		}
 
-	/* Allocate CM and initialize */
-	if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) 
-		return DAT_INSUFFICIENT_RESOURCES;
+		status = ib_at_route_by_ip(((struct sockaddr_in *)&conn->r_addr)->sin_addr.s_addr,
+					   0, 0, 0, &conn->dapl_rt, &conn->dapl_comp);
+		if (status < 0) {
+			dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_rt_comp_handler: "
+				    "ib_at_route_by_ip failed with status %d\n",
+				    status);
+			event = IB_CME_DESTINATION_UNREACHABLE;
+			goto bail;
+		}
 
-	(void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) );
-	
-	cm_ptr->socket = cm_ptr->l_socket = -1;
-	cm_ptr->sp = sp_ptr;
-	cm_ptr->hca_ptr = ia_ptr->hca_ptr;
-	
-	/* bind, listen, set sockopt, accept, exchange data */
-	if ((cm_ptr->l_socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
-		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
-				"socket for listen returned %d\n", errno);
-		dat_status = DAT_INSUFFICIENT_RESOURCES;
-		goto bail;
+		if (status == 1)
+			dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, 1);
+		return;
 	}
 
-	setsockopt(cm_ptr->l_socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt));
-	addr.sin_port        = htons(serviceID);
-	addr.sin_family      = AF_INET;
-	addr.sin_addr.s_addr = INADDR_ANY;
-
-	if (( bind( cm_ptr->l_socket,(struct sockaddr*)&addr, sizeof(addr) ) < 0) ||
-		   (listen( cm_ptr->l_socket, 128 ) < 0) ) {
-	
-		dapl_dbg_log( DAPL_DBG_TYPE_ERR,
-				" listen: ERROR %s on conn_qual 0x%x\n",
-				strerror(errno),serviceID); 
-
-		if ( errno == EADDRINUSE )
-			dat_status = DAT_CONN_QUAL_IN_USE;
-		else
-			dat_status = DAT_CONN_QUAL_UNAVAILABLE;
-
+	conn->dapl_comp.fn = &dapli_path_comp_handler;
+	conn->dapl_comp.context = conn;
+	conn->retries = 0;
+	status = ib_at_paths_by_route(&conn->dapl_rt, 0, &conn->dapl_path, 1,
+				      &conn->dapl_comp);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     "dapl_rt_comp_handler: ib_at_paths_by_route "
+			     "returned %d id %lld\n", status, conn->dapl_comp.req_id);
+		event = IB_CME_LOCAL_FAILURE;
 		goto bail;
 	}
-	
-	/* set cm_handle for this service point, save listen socket */
-	sp_ptr->cm_srvc_handle = cm_ptr;
+	return;
 
-	/* add to SP->CR thread list */
-	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry);
-	dapl_os_lock( &cm_ptr->hca_ptr->ib_trans.lock );
-	dapl_llist_add_tail(&cm_ptr->hca_ptr->ib_trans.list, 
-			    (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr);
-	dapl_os_unlock(&cm_ptr->hca_ptr->ib_trans.lock);
-
-	dapl_dbg_log( DAPL_DBG_TYPE_CM,
-			" listen: qual 0x%x cr %p s_fd %d\n",
-			ntohs(serviceID), cm_ptr, cm_ptr->l_socket ); 
-	
-	return dat_status;
 bail:
-	dapl_dbg_log( DAPL_DBG_TYPE_ERR,
-			" listen: ERROR on conn_qual 0x%x\n",serviceID); 
-	if ( cm_ptr->l_socket >= 0 )
-		close( cm_ptr->l_socket );
-	dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	return dat_status;
+	dapl_evd_connection_callback(conn, event, NULL, conn->ep);
 }
 
-
-/*
- * PASSIVE: send local QP information, private data, and wait for 
- *	    active side to respond with QP RTS/RTR status 
- */
-static DAT_RETURN 
-dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr)
+static void dapli_destroy_cm_id(struct dapl_cm_id *conn)
 {
-	ib_cm_handle_t	acm_ptr;
-	void		*p_data = NULL;
-	int		len;
-	DAT_RETURN	dat_status = DAT_SUCCESS;
-		
-	/* Allocate accept CM and initialize */
-	if ((acm_ptr = dapl_os_alloc(sizeof(*acm_ptr))) == NULL) 
-		return DAT_INSUFFICIENT_RESOURCES;
+	int in_callback;
 
-	(void) dapl_os_memzero( acm_ptr, sizeof( *acm_ptr ) );
-	
-	acm_ptr->socket = -1;
-	acm_ptr->sp = cm_ptr->sp;
-	acm_ptr->hca_ptr = cm_ptr->hca_ptr;
-
-	len = sizeof(acm_ptr->dst.ia_address);
-	acm_ptr->socket = accept(cm_ptr->l_socket, 
-				(struct sockaddr*)&acm_ptr->dst.ia_address, 
-				&len );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " destroy_cm_id: conn %p id %d\n",conn,conn->cm_id);
+
+	dapl_os_lock(&conn->lock);
+	conn->destroy = 1;
+	in_callback = conn->in_callback;
+	dapl_os_unlock(&conn->lock);
+
+	if (!in_callback) {
+		ib_cm_destroy_id(conn->cm_id);
+		if (conn->ep)
+			conn->ep->cm_handle = IB_INVALID_HANDLE;
+		if (conn->sp)
+			conn->sp->cm_srvc_handle = IB_INVALID_HANDLE;
+
+		/* take off the CM thread work queue and free */
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_remove_entry(&g_cm_list, 
+					(DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_unlock(&g_cm_lock);
+
+		dapl_os_free(conn, sizeof(*conn));
+	}
+}
 
-	if ( acm_ptr->socket < 0 ) {
+static void dapli_rep_recv(struct dapl_cm_id  *conn,
+			   struct ib_cm_event *event)
+{
+	int status;
+
+	if (conn->ep->qp_handle == IB_INVALID_HANDLE) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			" accept: ERR %s on FD %d l_cr %p\n",
-			strerror(errno),cm_ptr->l_socket,cm_ptr); 
-		dat_status = DAT_INTERNAL_ERROR;
-		goto bail;
-   	}
+			     " dapli_rep_recv: invalid qp "
+			     "handle\n");
+		goto disc;
+	}
+
+	/* move QP state to RTR and RTS */
+	/* TODO: could use a ib_cm_init_qp_attr() call here */
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " rep_recv: RTR_RTS: cm_id %d r_qp 0x%x r_lid 0x%x r_SID %d\n",
+		     conn->cm_id,event->param.rep_rcvd.remote_qpn,
+		     ntohs(conn->req.primary_path->dlid),conn->service_id );
+
+	if ( dapls_modify_qp_state( conn->ep->qp_handle, 
+				    IBV_QPS_RTR, 
+				    event->param.rep_rcvd.remote_qpn,
+				    ntohs(conn->req.primary_path->dlid),
+				    1 ) != DAT_SUCCESS )
+		goto disc;
+
+	if ( dapls_modify_qp_state( conn->ep->qp_handle, 
+				    IBV_QPS_RTS,0,0,0 ) != DAT_SUCCESS)
+		goto disc; 
+
 
-	/* read in DST QP info, IA address. check for private data */
-	len = read( acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t) );
-	if ( len != sizeof(ib_qp_cm_t) ) {
+	status = ib_cm_send_rtu(conn->cm_id, NULL, 0);
+	if (status) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			" accept read: ERR %s, rcnt=%d\n",
-			strerror(errno), len); 
-		dat_status = DAT_INTERNAL_ERROR;
-		goto bail;
+			     " dapli_rep_recv: ib_send_cm_rtu "
+			     "failed: %d\n", status);
+		goto disc;
+	}
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+     		     " rep_recv: conn %p id %d CONNECTED!\n",
+     		     conn, conn->cm_id );
+
+	dapl_evd_connection_callback(conn, IB_CME_CONNECTED,
+				     event->private_data, conn->ep);
+	return;
+
+disc:
+	dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL,
+				     conn->ep);
+}
 
-	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		" accept: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		acm_ptr->dst.port, acm_ptr->dst.lid, 
-		acm_ptr->dst.qpn, acm_ptr->dst.p_size ); 
+static struct dapl_cm_id * dapli_req_recv(struct dapl_cm_id  *conn,
+					  struct ib_cm_event *event)
+{
+	struct dapl_cm_id 	*new_conn;
 
-	/* validate private data size before reading */
-	if ( acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE ) {
+	if (conn->sp == NULL) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			" accept read: psize (%d) wrong\n",
-			acm_ptr->dst.p_size ); 
-		dat_status = DAT_INTERNAL_ERROR;
-		goto bail;
+			     " dapli_rep_recv: on invalid listen "
+			     "handle\n");
+		return NULL;
+	}
+
+	/* allocate new cm_id and merge listen parameters */
+	new_conn = dapl_os_alloc(sizeof(*new_conn)); 
+	if (new_conn) {
+
+		(void)dapl_os_memzero(new_conn, sizeof(*new_conn));
+		dapl_os_lock_init(&new_conn->lock);
+		new_conn->cm_id = event->cm_id; /* provided by uCM */
+		new_conn->sp = conn->sp;
+		new_conn->hca = conn->hca;
+		new_conn->service_id = conn->service_id;
+
+		/* save request information in new conn */
+		dapl_os_memcpy(&new_conn->req_rcvd, 
+				&event->param.req_rcvd,
+				sizeof(struct ib_cm_req_event_param));
+
+		new_conn->req_rcvd.primary_path = &new_conn->dapl_path;
+		new_conn->req_rcvd.alternate_path = NULL;
+
+		dapl_os_memcpy(new_conn->req_rcvd.primary_path,
+				event->param.req_rcvd.primary_path,
+				sizeof(struct ib_sa_path_rec));
+			 	
+		/* put new CR on CM thread event work queue */
+		dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&new_conn->entry);
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_add_tail(&g_cm_list, 
+				(DAPL_LLIST_ENTRY*)&new_conn->entry, new_conn);
+		dapl_os_unlock(&g_cm_lock);
+
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, " passive_cb: "
+				"REQ on HCA %p SP %p SID %d L_ID %d new_id %d p_data %p\n",
+				new_conn->hca, new_conn->sp, 
+				conn->service_id, conn->cm_id, new_conn->cm_id, 
+				event->private_data );
+
 	}
+	return new_conn;
+}
 
-	/* read private data into cm_handle if any present */
-	if ( acm_ptr->dst.p_size ) {
-		len = read( acm_ptr->socket, 
-			    acm_ptr->p_data, acm_ptr->dst.p_size );
-		if ( len != acm_ptr->dst.p_size ) {
-			dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-				" accept read pdata: ERR %s, rcnt=%d\n",
-				strerror(errno), len ); 
-			dat_status = DAT_INTERNAL_ERROR;
-			goto bail;
-		}
-		dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-				" accept: psize=%d read\n",
-				acm_ptr->dst.p_size); 
-		p_data = acm_ptr->p_data;
+
+static int dapli_cm_active_cb(struct dapl_cm_id *conn,
+			       struct ib_cm_event *event)
+{
+	int destroy;
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " active_cb: conn %p id %d event %d\n",
+		     conn, conn->cm_id, event->event );
+
+	dapl_os_lock(&conn->lock);
+	if (conn->destroy) {
+		dapl_os_unlock(&conn->lock);
+		return 0;
 	}
-	
-	/* trigger CR event and return SUCCESS */
-	dapls_cr_callback(  acm_ptr,
-			    IB_CME_CONNECTION_REQUEST_PENDING,
-		            p_data,
-			    acm_ptr->sp );
+	conn->in_callback = 1;
+	dapl_os_unlock(&conn->lock);
 
-	return DAT_SUCCESS;
+	switch (event->event) {
+	case IB_CM_REQ_ERROR:
+		dapl_evd_connection_callback(conn,
+					     IB_CME_DESTINATION_UNREACHABLE,
+					     NULL, conn->ep);
+		break;
+	case IB_CM_REJ_RECEIVED:
+		dapl_evd_connection_callback(conn, IB_CME_DESTINATION_REJECT,
+					     NULL, conn->ep);
+		break;
+	case IB_CM_REP_RECEIVED:
+		dapli_rep_recv(conn, event);
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		ib_cm_send_drep(conn->cm_id, NULL, 0);
+		break;
+	case IB_CM_DREQ_ERROR:
+	case IB_CM_DREP_RECEIVED:
+		/* Wait to exit timewait. */
+		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		dapl_evd_connection_callback(conn, IB_CME_DISCONNECTED,
+					     NULL, conn->ep);
+		break;
+	default:
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " dapli_cm_active_cb_handler: Unexpected CM "
+			     "event %d on ID 0x%p\n", event->event, conn->cm_id);
+		break;
+	}
 
-bail:
-	if ( acm_ptr->socket >=0 )
-		close( acm_ptr->socket );
-	dapl_os_free( acm_ptr, sizeof( *acm_ptr ) );
-	return DAT_INTERNAL_ERROR;
+	dapl_os_lock(&conn->lock);
+	destroy = conn->destroy;
+	conn->in_callback = conn->destroy;
+	dapl_os_unlock(&conn->lock);
+	if (destroy) {
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " active_cb: DESTROY conn %p id %d \n",
+			     conn, conn->cm_id );
+		if (conn->ep)
+			conn->ep->cm_handle = IB_INVALID_HANDLE;
+		
+		/* take off the CM thread work queue and free */
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_remove_entry(&g_cm_list, 
+					(DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_unlock(&g_cm_lock);
+		dapl_os_free(conn, sizeof(*conn));
+	}
+	return(destroy);
 }
 
+static int dapli_cm_passive_cb(struct dapl_cm_id *conn,
+			       	struct ib_cm_event *event)
+{
+	int destroy;
+	struct dapl_cm_id *new_conn;
 
-static DAT_RETURN 
-dapli_socket_accept_final( DAPL_EP		*ep_ptr,
-			   DAPL_CR		*cr_ptr,
-			   DAT_COUNT		p_size,
-		           DAT_PVOID		p_data )
-{
-	DAPL_IA		*ia_ptr = ep_ptr->header.owner_ia;
-	ib_cm_handle_t	cm_ptr = cr_ptr->ib_cm_handle;
-	ib_qp_cm_t	qp_cm;
-	struct iovec    iovec[2];
-	int		len;
-	short		rtu_data = 0;
-
-	if (p_size >  IB_MAX_REP_PDATA_SIZE) 
-		return DAT_LENGTH_ERROR;
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " passive_cb: conn %p id %d event %d\n",
+		     conn, conn->cm_id, event->event );
 
-	/* must have a accepted socket */
-	if ( cm_ptr->socket < 0 )
-		return DAT_INTERNAL_ERROR;
-	
-	/* modify QP to RTR and then to RTS with remote info already read */
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS )
-		goto bail;
+	if (conn->cm_id == 0)
+		return 0;
 
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS )
-		goto bail;
+	dapl_os_lock(&conn->lock);
+	if (conn->destroy) {
+		dapl_os_unlock(&conn->lock);
+		return 0;
+	}
+	conn->in_callback = 1;
+	dapl_os_unlock(&conn->lock);
 
-	ep_ptr->qp_state = IB_QP_STATE_RTS;
-	
-	/* Send QP info, IA address, and private data */
-	qp_cm.qpn = ep_ptr->qp_handle->qp_num;
-	qp_cm.port = ia_ptr->hca_ptr->port_num;
-	qp_cm.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, 
-				   ia_ptr->hca_ptr->port_num );
-	qp_cm.ia_address = ia_ptr->hca_ptr->hca_address;
-	qp_cm.p_size = p_size;
-	iovec[0].iov_base = &qp_cm;
-	iovec[0].iov_len  = sizeof(ib_qp_cm_t);
-	if (p_size) {
-		iovec[1].iov_base = p_data;
-		iovec[1].iov_len  = p_size;
+	switch (event->event) {
+	case IB_CM_REQ_RECEIVED:
+		/* create new conn object with new conn_id from event */
+		new_conn = dapli_req_recv(conn,event);
+
+		if (new_conn)	
+			dapls_cr_callback(new_conn, IB_CME_CONNECTION_REQUEST_PENDING, 
+				 	  event->private_data, new_conn->sp);
+		break;
+	case IB_CM_REP_ERROR:
+		dapls_cr_callback(conn, IB_CME_DESTINATION_UNREACHABLE,
+				 NULL, conn->sp);
+		break;
+	case IB_CM_REJ_RECEIVED:
+		dapls_cr_callback(conn, IB_CME_DESTINATION_REJECT, NULL,
+				 conn->sp);
+		break;
+	case IB_CM_RTU_RECEIVED:
+		/* move QP to RTS state */
+		if ( dapls_modify_qp_state(conn->ep->qp_handle, 
+					   IBV_QPS_RTS,0,0,0 ) != DAT_SUCCESS) {
+			dapls_cr_callback(conn, IB_CME_LOCAL_FAILURE, 
+					  NULL, conn->sp);
+		} else {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     		     " passive_cb: conn %p id %d CONNECTED!\n",
+		     		     conn, conn->cm_id );
+			dapls_cr_callback(conn, IB_CME_CONNECTED, 
+					  NULL, conn->sp);
+		}
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		ib_cm_send_drep(conn->cm_id, NULL, 0);
+		break;
+	case IB_CM_DREQ_ERROR:
+	case IB_CM_DREP_RECEIVED:
+		/* Wait to exit timewait. */
+		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		dapls_cr_callback(conn, IB_CME_DISCONNECTED, NULL, conn->sp);
+		break;
+	default:
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_cm_passive_cb_handler: "
+			     "Unexpected CM event %d on ID 0x%p\n",
+			     event->event, conn->cm_id);
+		break;
 	}
-	len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) );
-    	if (len != (p_size + sizeof(ib_qp_cm_t))) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " accept_final: ERR %s, wcnt=%d\n",
-			     strerror(errno), len); 
-		goto bail;
+
+	dapl_os_lock(&conn->lock);
+	destroy = conn->destroy;
+	conn->in_callback = conn->destroy;
+	dapl_os_unlock(&conn->lock);
+	if (destroy) {
+		if (conn->ep)
+			conn->ep->cm_handle = IB_INVALID_HANDLE;
+
+		/* take off the CM thread work queue and free */
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_remove_entry(&g_cm_list, 
+					(DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_unlock(&g_cm_lock);
+
+		dapl_os_free(conn, sizeof(*conn));
 	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		     " accept_final: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		     qp_cm.port, qp_cm.lid, qp_cm.qpn, qp_cm.p_size ); 
+	return(destroy);
+}
+
+/* something to catch the signal */
+static void cm_handler(int signum)
+{
+	dapl_dbg_log (DAPL_DBG_TYPE_CM," cm_thread(%d,0x%x): ENTER cm_handler %d\n",
+			getpid(),g_cm_thread,signum);
+	return;
+}
+
+/* async CM processing thread */
+void cm_thread(void *arg) 
+{
+	struct dapl_cm_id	*conn, *next_conn;
+	struct ib_cm_event	*event;
+	struct pollfd           ufds;
+	sigset_t		sigset;
+
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " cm_thread(%d,0x%x): ENTER: cm_fd %d\n",
+		      getpid(), g_cm_thread, ib_cm_get_fd());
+
+    	sigemptyset(&sigset);
+	sigaddset(&sigset, SIGUSR1);
+    	pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
+	signal( SIGUSR1, cm_handler); 
 	
-	/* complete handshake after final QP state change */
-	len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data) );
-	if ( len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " accept_final: ERR %s, rcnt=%d rdata=%x\n",
-			     strerror(errno), len, ntohs(rtu_data) ); 
-		goto bail;
-	}
+	dapl_os_lock( &g_cm_lock );
+	while (!g_cm_destroy) {
+		int cm_id,ret;
+
+		/* select for CM event, all events process via cm_fd */
+                ufds.fd      = ib_cm_get_fd();
+                ufds.events  = POLLIN;
+                ufds.revents = 0;
+
+		dapl_os_unlock(&g_cm_lock);
+                ret = poll(&ufds, 1, -1); 
+		if ((ret <= 0) || (g_cm_destroy)) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				     " cm_thread(%d): ERR %s poll\n",
+				     getpid(),strerror(errno));
+                	dapl_os_lock(&g_cm_lock);
+			break;
+		}
 
-	/* final data exchange if remote QP state is good to go */
-	dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: connected!\n" ); 
-	dapls_cr_callback ( cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp );
-	return DAT_SUCCESS;
+		dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			" cm_thread: GET EVENT fd=%d n=%d\n",
+			ib_cm_get_fd(),ret);
+		if (ib_cm_event_get(&event)) { 
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				" cm_thread: ERR %s eventi_get on %d\n", 
+				strerror(errno), ib_cm_get_fd() );
+			dapl_os_lock(&g_cm_lock);
+			continue;
+		}
+		dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			" cm_thread: GET EVENT fd=%d woke\n",ib_cm_get_fd());
+		dapl_os_lock(&g_cm_lock);
+
+		/* set proper cm_id */
+		if (event->event == IB_CM_REQ_RECEIVED ||
+			event->event == IB_CM_SIDR_REQ_RECEIVED)
+			cm_id = event->param.req_rcvd.listen_id;
+		else
+			cm_id = event->cm_id;
 
-bail:
-	dapl_dbg_log( DAPL_DBG_TYPE_ERR," accept_final: ERR !QP_RTR_RTS \n"); 
-	if ( cm_ptr >= 0 )
-		close( cm_ptr->socket );
-	dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */
+		dapl_dbg_log (DAPL_DBG_TYPE_CM,
+			" cm_thread: EVENT event(%d) cm_id=%d (%d)\n",
+			event->event, event->cm_id, cm_id );
 
-	return DAT_INTERNAL_ERROR;
-}
+		/* 
+		 * Walk cm_list looking for connection id in event
+		 * no need to walk if uCM would provide context with event
+		 */
+		if (!dapl_llist_is_empty(&g_cm_list))
+			next_conn = dapl_llist_peek_head(&g_cm_list);
+		else
+			next_conn = NULL;
 
-#endif
+		ret = 0;
+		while (next_conn) {
+			conn = next_conn;
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			     " cm_thread: LIST cm %p c_id %d e_id %d)\n", 
+			     conn, conn->cm_id, cm_id );
+				
+			next_conn = dapl_llist_next_entry(
+					&g_cm_list,
+					(DAPL_LLIST_ENTRY*)&conn->entry );
+
+			if (cm_id == conn->cm_id) {
+				dapl_os_unlock(&g_cm_lock);
+				if (conn->sp)
+					ret = dapli_cm_passive_cb(conn,event);
+				else
+					ret = dapli_cm_active_cb(conn,event);
+				dapl_os_lock(&g_cm_lock);
+				break;
+			}
+		} 
+		ib_cm_event_put(event);
+		if (ret) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			     " cm_thread: destroy cm_id %d\n",cm_id);
+			ib_cm_destroy_id(cm_id);
+		}
+	}
+	dapl_os_unlock(&g_cm_lock);	
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread(%d) EXIT, cm_list=%s\n",
+		     getpid(),dapl_llist_is_empty(&g_cm_list) ? "EMPTY":"NOT EMPTY");
+	g_cm_destroy = 0;
+}
 
+/************************ DAPL provider entry points **********************/
 
 /*
  * dapls_ib_connect
@@ -522,16 +758,17 @@
 DAT_RETURN
 dapls_ib_connect (
 	IN  DAT_EP_HANDLE		ep_handle,
-	IN  DAT_IA_ADDRESS_PTR		remote_ia_address,
-	IN  DAT_CONN_QUAL		remote_conn_qual,
-	IN  DAT_COUNT			private_data_size,
-	IN  void			*private_data )
+	IN  DAT_IA_ADDRESS_PTR		r_addr,
+	IN  DAT_CONN_QUAL		r_qual,
+	IN  DAT_COUNT			p_size,
+	IN  void			*p_data )
 {
 	DAPL_EP		*ep_ptr;
 	ib_qp_handle_t	qp_ptr;
+	int		status;
+	DAT_RETURN	dat_status = DAT_INTERNAL_ERROR;
+	ib_cm_handle_t	conn;
 	
-	dapl_dbg_log ( DAPL_DBG_TYPE_EP,
-			" connect(ep_handle %p ....)\n", ep_handle);
 	/*
 	 *  Sanity check
 	 */
@@ -541,13 +778,80 @@
 	ep_ptr = (DAPL_EP*)ep_handle;
 	qp_ptr = ep_ptr->qp_handle;
 
-#ifdef SOCKET_CM
-	return (dapli_socket_connect(	ep_ptr, remote_ia_address, 
-					remote_conn_qual,
-					private_data_size, private_data ));
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, " connect: r_SID %d, pdata %p, plen %d\n", 
+		     r_qual,p_data,p_size);
+			
+	/* Allocate CM and initialize lock */
+	if ((conn = dapl_os_alloc(sizeof(*conn))) == NULL) 
+		return DAT_INSUFFICIENT_RESOURCES;
+	
+	(void)dapl_os_memzero(conn, sizeof(*conn));
+
+	dat_status = dapl_os_lock_init(&conn->lock);
+	if (dat_status != DAT_SUCCESS) { 
+		dapl_os_free(conn, sizeof(*conn));
+		return DAT_INTERNAL_ERROR;
+	}
+
+	conn->ep = ep_ptr;
+	conn->hca = ep_ptr->header.owner_ia->hca_ptr;
+	status = ib_cm_create_id(&conn->cm_id);
+	if (status < 0)  {
+		dat_status = dapl_convert_errno(errno,"create_cm_id");
+		dapl_os_free(conn, sizeof(*conn));
+		return dat_status;
+	}
+	conn->ep->cm_handle = conn;
+
+	/* Setup QP/CM parameters */
+	(void)dapl_os_memzero(&conn->req,sizeof(conn->req));
+	conn->service_id = r_qual;
+	conn->req.qp_num = ep_ptr->qp_handle->qp_num;
+	conn->req.qp_type = IBV_QPT_RC;
+	conn->req.starting_psn = 1;
+	conn->req.private_data = p_data;
+	conn->req.private_data_len = p_size;
+	conn->req.peer_to_peer = 0;
+	conn->req.responder_resources = IB_TARGET_MAX;
+	conn->req.initiator_depth = IB_INITIATOR_DEPTH;
+	conn->req.remote_cm_response_timeout = IB_CM_RESPONSE_TIMEOUT;
+	conn->req.flow_control = 1;
+	conn->req.local_cm_response_timeout = IB_CM_RESPONSE_TIMEOUT;
+	conn->req.retry_count = IB_RC_RETRY_COUNT;
+	conn->req.rnr_retry_count = IB_RNR_RETRY_COUNT;
+	conn->req.max_cm_retries = IB_MAX_CM_RETRIES;
+	conn->req.srq = 0;
+	
+	conn->dapl_comp.fn = &dapli_rt_comp_handler;
+	conn->dapl_comp.context = conn;
+	conn->retries = 0;
+	dapl_os_memcpy(&conn->r_addr, r_addr, sizeof(DAT_SOCK_ADDR6));
+
+	status = ib_at_route_by_ip(
+		((struct sockaddr_in *)&conn->r_addr)->sin_addr.s_addr, 
+		((struct sockaddr_in *)&conn->hca->hca_address)->sin_addr.s_addr, 
+		0, 0, &conn->dapl_rt, &conn->dapl_comp);
+
+	if (status < 0) {
+		dat_status = dapl_convert_errno(errno,"ib_at_route_by_ip");
+		goto destroy;
+	}
+	if (status == 1)
+		dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, 1);
+
+	
+	/* put on CM thread work queue */
+	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
+	dapl_os_lock( &g_cm_lock );
+	dapl_llist_add_tail(&g_cm_list, 
+			    (DAPL_LLIST_ENTRY*)&conn->entry, conn);
+	dapl_os_unlock(&g_cm_lock);
+
+	return DAT_SUCCESS;
+
+destroy:
+	dapli_destroy_cm_id(conn); 
+	return dat_status;
 
 }
 
@@ -572,35 +876,26 @@
 	IN	DAPL_EP			*ep_ptr,
 	IN	DAT_CLOSE_FLAGS		close_flags )
 {
-	ib_cm_handle_t	cm_ptr = ep_ptr->cm_handle;
+	ib_cm_handle_t	conn = ep_ptr->cm_handle;
+	int status;
 
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			"dapls_ib_disconnect(ep_handle %p ....)\n",
-			ep_ptr);
-#ifdef SOCKET_CM
-
-	if ( cm_ptr->socket >= 0 ) {
-		close( cm_ptr->socket );
-		cm_ptr->socket = -1;
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+			" disconnect(ep_handle %p, conn %p, cm_id %d flags %x)\n",
+			ep_ptr,conn, (conn?conn->cm_id:0),close_flags);
+
+	if (conn == IB_INVALID_HANDLE)
+		return DAT_SUCCESS;
+
+	if (close_flags == DAT_CLOSE_ABRUPT_FLAG)
+		dapli_destroy_cm_id(conn);
+	else {
+		status = ib_cm_send_dreq(conn->cm_id, NULL, 0);
+		if (status)
+			dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+				     "dapl_ib_disconnect: ID %p status %d\n", 
+				     ep_ptr->cm_handle, status);
 	}
-	
-	/* reinit to modify QP state */
-	dapls_ib_reinit_ep(ep_ptr);
 
-#endif
-	if ( ep_ptr->cr_ptr ) {
-		dapls_cr_callback ( ep_ptr->cm_handle,
-				    IB_CME_DISCONNECTED,
-				    NULL,
-				    ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr );
-	} else {
-		dapl_evd_connection_callback ( ep_ptr->cm_handle,
-						IB_CME_DISCONNECTED,
-						NULL,
-						ep_ptr );
-		ep_ptr->cm_handle = NULL;
-		dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	}	
 	return DAT_SUCCESS;
 }
 
@@ -628,7 +923,14 @@
 	IN  DAT_BOOLEAN			active,
 	IN  const ib_cm_events_t	ib_cm_event )
 {
-    return;
+
+	/*
+	 * Clean up outstanding connection state
+	 */
+	dapls_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG);
+         
+	if (ep_ptr->qp_handle != IB_INVALID_HANDLE) 
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_ERR, 0,0,0);
 }
 
 /*
@@ -657,11 +959,63 @@
 	IN  DAT_UINT64		ServiceID,
 	IN  DAPL_SP		*sp_ptr )
 {
-#ifdef SOCKET_CM
-	return (dapli_socket_listen( ia_ptr, ServiceID, sp_ptr ));
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
+	DAT_RETURN		dat_status = DAT_SUCCESS;
+	int			status;
+	ib_cm_srvc_handle_t     conn;
+
+
+	/* Allocate CM and initialize lock */
+	if ((conn = dapl_os_alloc(sizeof(*conn))) == NULL) 
+		return DAT_INSUFFICIENT_RESOURCES;
+	
+	(void)dapl_os_memzero(conn, sizeof(*conn));
+
+	dat_status = dapl_os_lock_init(&conn->lock);
+	if (dat_status != DAT_SUCCESS)  {
+		dapl_os_free(conn, sizeof(*conn));
+		return DAT_INTERNAL_ERROR;
+	}
+	 
+	status = ib_cm_create_id(&conn->cm_id);
+	if (status < 0)  {
+		dat_status = dapl_convert_errno(errno,"create_cm_id");
+		dapl_os_free(conn, sizeof(*conn));
+		return dat_status;
+	}
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM,
+		     " setup_listener(ia_ptr %p SID %d sp_ptr %p conn %p cm_id %d)\n",
+		     ia_ptr, ServiceID, sp_ptr, conn, conn->cm_id );
+
+	sp_ptr->cm_srvc_handle = conn;
+	conn->sp = sp_ptr;
+	conn->hca = ia_ptr->hca_ptr;
+	conn->service_id = ServiceID;
+
+	dapl_dbg_log(DAPL_DBG_TYPE_EP,
+		     " setup_listener(conn=%p cm_id=%d)\n",
+		     sp_ptr->cm_srvc_handle,conn->cm_id);
+
+	status = ib_cm_listen(conn->cm_id,
+			      cpu_to_be64(ServiceID), 0);
+	if (status) {
+		if (status == -EBUSY)
+			dat_status = DAT_CONN_QUAL_IN_USE;
+		else
+			dat_status = DAT_INSUFFICIENT_RESOURCES;
+	/* success */ 
+	} else  { 
+		/* put on CM thread work queue */
+		dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_add_tail(&g_cm_list, 
+				(DAPL_LLIST_ENTRY*)&conn->entry, conn);
+		dapl_os_unlock(&g_cm_lock);
+		return dat_status;
+	}
+
+        dapli_destroy_cm_id(conn);
+	return dat_status;
 }
 
 
@@ -687,27 +1041,16 @@
 	IN  DAPL_IA		*ia_ptr,
 	IN  DAPL_SP		*sp_ptr )
 {
-	ib_cm_srvc_handle_t	cm_ptr = sp_ptr->cm_srvc_handle;
+	ib_cm_srvc_handle_t	conn = sp_ptr->cm_srvc_handle;
 
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			"dapls_ib_remove_conn_listener(ia_ptr %p sp_ptr %p cm_ptr %p)\n",
-			ia_ptr, sp_ptr, cm_ptr );
-#ifdef SOCKET_CM
-	/* close accepted socket, free cm_srvc_handle and return */
-	if ( cm_ptr != NULL ) {
-		if ( cm_ptr->l_socket >= 0 ) {
-			close( cm_ptr->l_socket );
-			cm_ptr->socket = -1;
-		}
-	    	/* cr_thread will free */
-		sp_ptr->cm_srvc_handle = NULL;
-	}
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+			" remove_listener(ia_ptr %p sp_ptr %p cm_ptr %p)\n",
+			ia_ptr, sp_ptr, conn );
+	
+	if (sp_ptr->cm_srvc_handle != IB_INVALID_HANDLE) 
+        	dapli_destroy_cm_id(conn);
+	
 	return DAT_SUCCESS;
-#else
-	return DAT_NOT_IMPLEMENTED;   
-
-#endif
-
 }
 
 /*
@@ -737,30 +1080,84 @@
 	IN  DAT_COUNT		p_size,
 	IN  const DAT_PVOID	p_data )
 {
-	DAPL_CR			*cr_ptr;
-	DAPL_EP			*ep_ptr;
-	
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-		      "dapls_ib_accept_connection(cr %p ep %p prd %p,%d)\n",
-		      cr_handle, ep_handle, p_data, p_size  );
+	DAPL_CR *cr_ptr;
+	DAPL_EP *ep_ptr;
+	DAPL_IA *ia_ptr;
+	DAT_RETURN dat_status;
+	int status;
+	struct ib_cm_rep_param passive_params;
+	struct dapl_cm_id *conn;
+
+	cr_ptr = (DAPL_CR *) cr_handle;
+	ep_ptr = (DAPL_EP *) ep_handle;
+	ia_ptr = ep_ptr->header.owner_ia;
+	conn = cr_ptr->ib_cm_handle;
+
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " accept_connection(cr %p conn %p, cm_id %d, p_data %p, p_sz=%d)\n",
+		      cr_ptr, conn, conn->cm_id, p_data, p_size );
+
+	/* Obtain size of private data structure & contents */
+	if (p_size > IB_MAX_REP_PDATA_SIZE) {
+		dat_status = DAT_ERROR(DAT_LENGTH_ERROR, DAT_NO_SUBTYPE);
+		goto reject;
+	}
+
+	if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) {
+		/* 
+		 * If we are lazy attaching the QP then we may need to
+		 * hook it up here. Typically, we run this code only for
+		 * DAT_PSP_PROVIDER_FLAG
+		 */
+		dat_status = dapls_ib_qp_alloc(ia_ptr, ep_ptr, NULL);
+		if (dat_status != DAT_SUCCESS) {
+			dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+				     " accept: ib_qp_alloc failed: %d\n",
+				     dat_status);
+			goto reject;
+		}
+	}
 
-	cr_ptr  = (DAPL_CR *) cr_handle;
-	ep_ptr  = (DAPL_EP *) ep_handle;
-	
-	/* allocate and attach a QP if necessary */
-	if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) {
-		DAT_RETURN status;
-		status = dapls_ib_qp_alloc( ep_ptr->header.owner_ia, 
-					    ep_ptr, ep_ptr );
-		if ( status != DAT_SUCCESS )
-    			return status;
-	}
-    
-#ifdef SOCKET_CM
-	return ( dapli_socket_accept_final(ep_ptr, cr_ptr, p_size, p_data) );
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
+	/* move QP to RTR state, TODO fix port setting */
+	/* TODO: could use a ib_cm_init_qp_attr() call here */
+	dat_status = dapls_modify_qp_state(ep_ptr->qp_handle, 
+					   IBV_QPS_RTR, 
+					   conn->req_rcvd.remote_qpn, 
+					   ntohs(conn->req_rcvd.primary_path->dlid),
+					   1 ); 
+	if (dat_status != DAT_SUCCESS ) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+				     " accept: modify_qp_state failed: %d\n",
+				     dat_status);
+		goto reject;
+	}
+
+	cr_ptr->param.local_ep_handle = ep_handle;
+	ep_ptr->cm_handle = conn;
+	conn->ep = ep_ptr;
+
+	memset(&passive_params, 0, sizeof(passive_params));
+	passive_params.private_data = p_data;
+	passive_params.private_data_len = p_size;
+	passive_params.qp_num = ep_ptr->qp_handle->qp_num;
+	passive_params.responder_resources = IB_TARGET_MAX;
+	passive_params.initiator_depth = IB_INITIATOR_DEPTH;
+	passive_params.rnr_retry_count = IB_RNR_RETRY_COUNT;
+
+	status = ib_cm_send_rep(conn->cm_id, &passive_params);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " accept: "
+			     "ib_send_cm_rep failed: %d\n", status);
+		dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0);
+		goto reject;
+	}
+	return DAT_SUCCESS;
+
+reject:
+	ib_cm_send_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0,
+		       NULL, 0);
+	dapli_destroy_cm_id(conn);
+	return dat_status;
    
 }
 
@@ -783,27 +1180,31 @@
  */
 DAT_RETURN
 dapls_ib_reject_connection (
-	IN  ib_cm_handle_t	ib_cm_handle,
+	IN  ib_cm_handle_t	cm_handle,
 	IN  int			reject_reason )
 {
-    	ib_cm_srvc_handle_t	cm_ptr = ib_cm_handle;
+    	int status;
 
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-		      "dapls_ib_reject_connection(cm_handle %p reason %x)\n",
-		      ib_cm_handle, reject_reason );
-#ifdef SOCKET_CM
-
-	/* just close the socket and return */
-	if ( cm_ptr->socket > 0 ) {
-		close( cm_ptr->socket );
-		cm_ptr->socket = -1;
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " reject_connection(cm_handle %p reason %x)\n",
+		      cm_handle, reject_reason );
+
+	if (cm_handle == IB_INVALID_HANDLE) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " reject_conn: invalid handle: reason %d\n",
+			     reject_reason);
+		return DAT_SUCCESS;
 	}
-	return DAT_SUCCESS;
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
-	
 
+	status = ib_cm_send_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED,
+				NULL, 0, NULL, 0);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
+			     " reject_conn: cm_rej failed: %d\n", status);
+		return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0);
+	}
+	dapli_destroy_cm_id(cm_handle);
+	return DAT_SUCCESS;
 }
 
 /*
@@ -844,7 +1245,7 @@
 		return DAT_INVALID_HANDLE;
 
 	dapl_os_memcpy(	remote_ia_address, 
-			&ib_cm_handle->dst.ia_address, 
+			&ib_cm_handle->r_addr, 
 			sizeof(DAT_SOCK_ADDR6) );
 
 	return DAT_SUCCESS;
@@ -914,12 +1315,12 @@
 	return size;
 }
 
-#ifdef SOCKET_CM
+#ifndef SOCKET_CM
 
 /*
  * Map all socket CM event codes to the DAT equivelent.
  */
-#define DAPL_IB_EVENT_CNT	11
+#define DAPL_IB_EVENT_CNT	12
 
 static struct ib_cm_event_map
 {
@@ -936,17 +1337,19 @@
 					DAT_CONNECTION_REQUEST_EVENT},
 	/* 04 */  { IB_CME_CONNECTION_REQUEST_PENDING_PRIVATE_DATA,
 					DAT_CONNECTION_REQUEST_EVENT},
-	/* 05 */  { IB_CME_DESTINATION_REJECT,
+	/* 05 */  { IB_CME_CONNECTION_REQUEST_ACKED,
+					DAT_CONNECTION_REQUEST_EVENT},
+	/* 06 */  { IB_CME_DESTINATION_REJECT,
 					DAT_CONNECTION_EVENT_NON_PEER_REJECTED},
-	/* 06 */  { IB_CME_DESTINATION_REJECT_PRIVATE_DATA,		
+	/* 07 */  { IB_CME_DESTINATION_REJECT_PRIVATE_DATA,		
 					DAT_CONNECTION_EVENT_PEER_REJECTED},
-	/* 07 */  { IB_CME_DESTINATION_UNREACHABLE,	
+	/* 08 */  { IB_CME_DESTINATION_UNREACHABLE,	
 					DAT_CONNECTION_EVENT_UNREACHABLE},
-	/* 08 */  { IB_CME_TOO_MANY_CONNECTION_REQUESTS,
+	/* 09 */  { IB_CME_TOO_MANY_CONNECTION_REQUESTS,
 					DAT_CONNECTION_EVENT_NON_PEER_REJECTED},
-	/* 09 */  { IB_CME_LOCAL_FAILURE,
+	/* 10 */  { IB_CME_LOCAL_FAILURE,
 					DAT_CONNECTION_EVENT_BROKEN},
-	/* 10 */  { IB_CM_LOCAL_FAILURE,
+	/* 11 */  { IB_CME_BROKEN,
 					DAT_CONNECTION_EVENT_BROKEN}
 };
  
@@ -974,7 +1377,7 @@
 	
 	active = active;
 
-	if (ib_cm_event > IB_CM_LOCAL_FAILURE)
+	if (ib_cm_event > IB_CME_BROKEN)
 		return (DAT_EVENT_NUMBER) 0;
 
 	dat_event_num = 0;
@@ -1024,96 +1427,6 @@
     return ib_cm_event;
 }
 
-/* async CR processing thread to avoid blocking applications */
-void cr_thread(void *arg) 
-{
-    struct dapl_hca	*hca_ptr = arg;
-    ib_cm_srvc_handle_t	cr, next_cr;
-    int			max_fd;
-    fd_set		rfd,rfds;
-    struct timeval	to;
-     
-    dapl_os_lock( &hca_ptr->ib_trans.lock );
-    while ( !hca_ptr->ib_trans.destroy ) {
-	
-	FD_ZERO( &rfds ); 
-	max_fd = -1;
-	
-	if (!dapl_llist_is_empty(&hca_ptr->ib_trans.list))
-            next_cr = dapl_llist_peek_head (&hca_ptr->ib_trans.list);
-	else
-	    next_cr = NULL;
-
-	while (next_cr) {
-	    cr = next_cr;
-	    dapl_dbg_log (DAPL_DBG_TYPE_CM," thread: cm_ptr %p\n", cr );
-	    if (cr->l_socket == -1 || hca_ptr->ib_trans.destroy) {
-
-		dapl_dbg_log(DAPL_DBG_TYPE_CM," thread: Freeing %p\n", cr);
-		next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list,
-						(DAPL_LLIST_ENTRY*)&cr->entry );
-		dapl_llist_remove_entry(&hca_ptr->ib_trans.list, 
-					(DAPL_LLIST_ENTRY*)&cr->entry);
-		dapl_os_free( cr, sizeof(*cr) );
-		continue;
-	    }
-	          
-	    FD_SET( cr->l_socket, &rfds ); /* add to select set */
-	    if ( cr->l_socket > max_fd )
-		max_fd = cr->l_socket;
-
-	    /* individual select poll to check for work */
-	    FD_ZERO(&rfd);
-	    FD_SET(cr->l_socket, &rfd);
-	    dapl_os_unlock(&hca_ptr->ib_trans.lock);	
-	    to.tv_sec  = 0;
-	    to.tv_usec = 0;
-	    if ( select(cr->l_socket + 1,&rfd, NULL, NULL, &to) < 0) {
-		dapl_dbg_log (DAPL_DBG_TYPE_CM,
-			  " thread: ERR %s on cr %p sk %d\n", 
-			  strerror(errno), cr, cr->l_socket);
-		close(cr->l_socket);
-		cr->l_socket = -1;
-	    } else if ( FD_ISSET(cr->l_socket, &rfd) && 
-			dapli_socket_accept(cr)) {
-		close(cr->l_socket);
-		cr->l_socket = -1;
-	    }
-	    dapl_os_lock( &hca_ptr->ib_trans.lock );
-	    next_cr =  dapl_llist_next_entry(&hca_ptr->ib_trans.list,
-					     (DAPL_LLIST_ENTRY*)&cr->entry );
-	} 
-	dapl_os_unlock( &hca_ptr->ib_trans.lock );
-	to.tv_sec  = 0;
-	to.tv_usec = 500000; /* wakeup and check destroy */
-	select(max_fd + 1, &rfds, NULL, NULL, &to);
-	dapl_os_lock( &hca_ptr->ib_trans.lock );
-    } 
-    dapl_os_unlock( &hca_ptr->ib_trans.lock );	
-    hca_ptr->ib_trans.destroy = 0;
-    dapl_dbg_log(DAPL_DBG_TYPE_CM," thread(hca %p) exit\n",hca_ptr);
-}
-
-/* Real IBv CM */
-#else 
-
-DAT_EVENT_NUMBER
-dapls_ib_get_dat_event (
-	IN    const ib_cm_events_t	ib_cm_event,
-	IN    DAT_BOOLEAN		active)
-{
-	return ib_cm_event;
-}
-
-ib_cm_events_t
-dapls_ib_get_cm_event (
-	IN    DAT_EVENT_NUMBER		dat_event_num )
-{
-	return dat_event_num;
-}
-
-
-
 #endif
 
 /*
Index: dapl/openib/dapl_ib_qp.c
===================================================================
--- dapl/openib/dapl_ib_qp.c	(revision 2818)
+++ dapl/openib/dapl_ib_qp.c	(working copy)
@@ -107,6 +107,8 @@
 	qp_create.cap.max_recv_wr = attr->max_recv_dtos;
 	qp_create.cap.max_send_sge = attr->max_request_iov;
 	qp_create.cap.max_recv_sge = attr->max_recv_iov;
+	/* TODO: extend ep_attr so user can set */
+	qp_create.cap.max_inline_data = ia_ptr->hca_ptr->ib_trans.max_inline_send; 
 	qp_create.qp_type = IBV_QPT_RC;
 	qp_create.qp_context = (void*)ep_ptr;
 
@@ -122,8 +124,7 @@
 
 	/* Setup QP attributes for INIT state on the way out */ 
 	if (dapls_modify_qp_state(ep_ptr->qp_handle,
-				  IBV_QPS_INIT,
-				  NULL )  != DAT_SUCCESS ) {
+				  IBV_QPS_INIT,0,0,0 ) != DAT_SUCCESS ) {
 		ibv_destroy_qp(ep_ptr->qp_handle);		
 		ep_ptr->qp_handle = IB_INVALID_HANDLE;
 		return DAT_INTERNAL_ERROR;
@@ -160,7 +161,7 @@
 
 	if (ep_ptr->qp_handle != IB_INVALID_HANDLE) {
 		/* force error state to flush queue, then destroy */
-		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_ERR, NULL);
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_ERR, 0,0,0);
 		
 		if (ibv_destroy_qp(ep_ptr->qp_handle)) 
 			return(dapl_convert_errno(errno,"destroy_qp"));
@@ -216,7 +217,7 @@
 	    (ep_ptr->qp_handle->state != IBV_QPS_ERR)) {
 		ep_ptr->qp_state = IB_QP_STATE_ERROR;
 		return (dapls_modify_qp_state(ep_ptr->qp_handle, 
-					      IBV_QPS_ERR, NULL));
+					      IBV_QPS_ERR,0,0,0));
 	}
 
 	/*
@@ -271,8 +272,8 @@
 	
 	if ( ep_ptr->qp_handle != IB_INVALID_HANDLE ) {
 		/* move to RESET state and then to INIT */
-		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_RESET, NULL);
-		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_INIT, NULL );
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_RESET, 0,0,0);
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_INIT, 0,0,0);
 		ep_ptr->qp_state = IB_QP_STATE_INIT;
 	}
 
@@ -287,7 +288,9 @@
 DAT_RETURN
 dapls_modify_qp_state ( IN ib_qp_handle_t	qp_handle,
 			IN ib_qp_state_t	qp_state,
-			IN ib_qp_cm_t		*qp_cm )
+			IN uint32_t		qpn,
+			IN uint16_t		lid,
+			IN uint8_t		port )
 {
 	struct ibv_qp_attr	qp_attr;
 	enum ibv_qp_attr_mask	mask = IBV_QP_STATE;
@@ -299,7 +302,6 @@
 		/* additional attributes with RTR and RTS */
 		case IBV_QPS_RTR:
 		{
-#ifdef SOCKET_CM
 			mask |= IBV_QP_AV                 |
 				IBV_QP_PATH_MTU           |
 				IBV_QP_DEST_QPN           |
@@ -308,25 +310,24 @@
 				IBV_QP_MIN_RNR_TIMER;
 			qp_attr.qp_state 		= IBV_QPS_RTR;
 			qp_attr.path_mtu 		= IBV_MTU_1024;
-			qp_attr.dest_qp_num 		= qp_cm->qpn;
+			qp_attr.dest_qp_num 		= qpn;
 			qp_attr.rq_psn 			= 1;
-			qp_attr.max_dest_rd_atomic	= 8;
+			qp_attr.max_dest_rd_atomic	= IB_TARGET_MAX;
 			qp_attr.min_rnr_timer		= 12;
 			qp_attr.ah_attr.is_global	= 0;
-			qp_attr.ah_attr.dlid		= qp_cm->lid;
+			qp_attr.ah_attr.dlid		= lid;
 			qp_attr.ah_attr.sl		= 0;
 			qp_attr.ah_attr.src_path_bits	= 0;
-			qp_attr.ah_attr.port_num	= qp_cm->port;
+			qp_attr.ah_attr.port_num	= port;
 
 			dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			      " modify_qp_rtr: qpn %x lid %x port %x\n",
-			      qp_cm->qpn,qp_cm->lid,qp_cm->port );
+			      " modify_qp_RTR: qpn %x lid %x port %x, rq_psn %x\n",
+			      qpn,lid,port,ntohl(qp_attr.rq_psn) );
 			break;
-#endif
+
 		}		
 		case IBV_QPS_RTS: 
 		{
-#ifdef SOCKET_CM
 			mask |= IBV_QP_TIMEOUT            |
 				IBV_QP_RETRY_CNT          |
 				IBV_QP_RNR_RETRY          |
@@ -337,12 +338,11 @@
 			qp_attr.retry_cnt	= 7;
 			qp_attr.rnr_retry	= 7;
 			qp_attr.sq_psn		= 1;
-			qp_attr.max_rd_atomic	= 8;
+			qp_attr.max_rd_atomic	= IB_TARGET_MAX;
 			dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			      " modify_qp_rts: psn %x or %x\n",
-			      qp_attr.sq_psn, qp_attr.max_rd_atomic );
+			      " modify_qp_RTS: psn %x or %x\n",
+			      ntohl(qp_attr.sq_psn), qp_attr.max_rd_atomic );
 			break;
-#endif
 		}
 		case IBV_QPS_INIT: 
 		{
@@ -368,7 +368,7 @@
 					IBV_ACCESS_REMOTE_ATOMIC;
 			
 			dapl_dbg_log (DAPL_DBG_TYPE_EP,
-				" modify_qp_init: pi %x port %x acc %x\n",
+				" modify_qp_INIT: pi %x port %x acc %x\n",
 				qp_attr.pkey_index, qp_attr.port_num,
 				qp_attr.qp_access_flags );
 			break;
Index: dapl/openib/README
===================================================================
--- dapl/openib/README	(revision 2818)
+++ dapl/openib/README	(working copy)
@@ -6,10 +6,12 @@
 - CQ_WAIT_OBJECT support added
 - added dapl/openib directory 
 - modify doc/dat.conf to add a example openib configuration 
+- fixes to wait and resize
 
 	dapl/common/dapl_adapter_util.h 
 	dapl/common/dapl_evd_dto_callb.c
 	dapl/common/dapl_evd_util.c 
+	dapl/common/dapl_evd_resize.c
 	dapl/include/dapl.h 
 	dapl/udapl/dapl_evd_set_unwaitable.c
 	dapl/udapl/dapl_evd_wait.c
@@ -29,8 +31,7 @@
 	dapl/openib/dapl_ib_qp.c
 	dapl/openib/dapl_ib_util.c
 	dapl/openib/dapl_ib_util.h
-	dapl/openib/dapl_ib_cm.c
-
+	
 A simple dapl test just for initial openIB testing...
 
 	test/dtest/dtest.c
@@ -38,10 +39,17 @@
 
 	server:	dtest -s 
 	client:	dtest -h hostname
+	
+setup/known issues:
+	
+	First drop with uCM (without IBAT), tested with simple dtest across 2 nodes. 
+	hand rolled path records require remote LID and GID set via enviroment:
+	
+	export DAPL_REMOTE_GID	"fe80:0000:0000:0000:0002:c902:0000:4071"
+	export DAPL_REMOTE_LID	"0002"
 
-known issues:
-
-	early drop, only tested with simple dtest and dapltest SR.
+    Also, hard coded (RTR) for use with port 1 only.
+	   
 	no memory windows support in ibverbs, dat_create_rmr fails.
 	
 
Index: dapl/openib/dapl_ib_util.h
===================================================================
--- dapl/openib/dapl_ib_util.h	(revision 2818)
+++ dapl/openib/dapl_ib_util.h	(working copy)
@@ -51,6 +51,8 @@
 
 #include "verbs.h"
 #include <byteswap.h>
+#include <infiniband/sa.h>
+#include <infiniband/cm.h>
 
 /* Typedefs to map common DAPL provider types to IB verbs */
 typedef	struct ibv_qp		*ib_qp_handle_t;
@@ -64,20 +66,46 @@
 typedef	struct ibv_context	*ib_hca_handle_t;
 typedef ib_hca_handle_t		dapl_ibal_ca_t;
 
-/* CM mappings, user CM not complete use SOCKETS */
+#define IB_RC_RETRY_COUNT      7
+#define IB_RNR_RETRY_COUNT     7
+#define IB_CM_RESPONSE_TIMEOUT 20	/* 4 sec */
+#define IB_MAX_CM_RETRIES      4
+
+#define IB_REQ_MRA_TIMEOUT	27	/* a little over 9 minutes */
+#define IB_MAX_AT_RETRY		3
+
+#define IB_TARGET_MAX		4	/* max_qp_ous_rd_atom */
+#define IB_INITIATOR_DEPTH	4	/* max_qp_init_rd_atom */
+
+typedef enum {
+	IB_CME_CONNECTED,
+	IB_CME_DISCONNECTED,
+	IB_CME_DISCONNECTED_ON_LINK_DOWN,
+	IB_CME_CONNECTION_REQUEST_PENDING,
+	IB_CME_CONNECTION_REQUEST_PENDING_PRIVATE_DATA,
+	IB_CME_CONNECTION_REQUEST_ACKED,
+	IB_CME_DESTINATION_REJECT,
+	IB_CME_DESTINATION_REJECT_PRIVATE_DATA,
+	IB_CME_DESTINATION_UNREACHABLE,
+	IB_CME_TOO_MANY_CONNECTION_REQUESTS,
+	IB_CME_LOCAL_FAILURE,
+	IB_CME_BROKEN
+} ib_cm_events_t;
 
-#ifdef SOCKET_CM
+#ifndef IB_AT
+/* implement a quick hack to exchange GID/LID's until user IB_AT arrives */
+struct ib_at_ib_route {
+        union ibv_gid   gid;
+        uint16_t        lid;
+};
 
-/* destination info to exchange until real IB CM shows up */
-typedef struct _ib_qp_cm
-{ 
-	uint32_t		qpn;
-	uint16_t		lid;
-	uint16_t		port;
-	int			p_size;
-	DAT_SOCK_ADDR6		ia_address;
+struct ib_at_completion {
+        void (*fn)(uint64_t req_id, void *context, int rec_num);
+        void *context;
+        uint64_t req_id;
+};
 
-} ib_qp_cm_t;
+#endif
 
 /* 
  * dapl_llist_entry in dapl.h but dapl.h depends on provider 
@@ -91,52 +119,27 @@
     struct dapl_llist_entry	*list_head;
 };
 
-struct ib_cm_handle
-{ 
-	struct ib_llist_entry	entry;
-	int			socket;
-	int			l_socket; 
-	struct dapl_hca		*hca_ptr;
-	DAT_HANDLE		cr;
-	DAT_HANDLE		sp;	
-	ib_qp_cm_t		dst;
-	unsigned char		p_data[256];
+struct dapl_cm_id {
+	struct ib_llist_entry		entry;
+	DAPL_OS_LOCK			lock;
+	int				retries;
+	int				destroy;
+	int				in_callback;
+	uint32_t			cm_id;
+	DAT_SOCK_ADDR6			r_addr; 
+	DAT_CONN_QUAL			service_id;
+	struct dapl_hca			*hca;
+	struct dapl_sp			*sp;
+	struct dapl_ep			*ep;
+	struct ib_at_ib_route		dapl_rt;
+	struct ib_sa_path_rec		dapl_path;
+	struct ib_cm_req_param		req;
+	struct ib_cm_req_event_param	req_rcvd;
+	struct ib_at_completion		dapl_comp;
 };
 
-typedef struct ib_cm_handle	*ib_cm_handle_t;
-typedef ib_cm_handle_t		ib_cm_srvc_handle_t;
-
-DAT_RETURN getipaddr(char *addr, int addr_len);
-
-/* CM events */
-typedef enum 
-{
-    IB_CME_CONNECTED,
-    IB_CME_DISCONNECTED,
-    IB_CME_DISCONNECTED_ON_LINK_DOWN,
-    IB_CME_CONNECTION_REQUEST_PENDING,
-    IB_CME_CONNECTION_REQUEST_PENDING_PRIVATE_DATA,
-    IB_CME_DESTINATION_REJECT,
-    IB_CME_DESTINATION_REJECT_PRIVATE_DATA,
-    IB_CME_DESTINATION_UNREACHABLE,
-    IB_CME_TOO_MANY_CONNECTION_REQUESTS,
-    IB_CME_LOCAL_FAILURE,
-    IB_CM_LOCAL_FAILURE
-
-} ib_cm_events_t;
-
-/* prototype for cm thread */
-void cr_thread (void *arg);
-
-#else
-
-/* TODO: Waiting for IB CM to define */
-typedef uint32_t		*ib_cm_handle_t;
-typedef uint32_t		*ib_cm_srvc_handle_t;
-typedef uint32_t		ib_qp_cm_t;
-typedef uint32_t		ib_cm_events_t;
-
-#endif
+typedef struct dapl_cm_id	*ib_cm_handle_t;
+typedef struct dapl_cm_id	*ib_cm_srvc_handle_t;
 
 /* Operation and state mappings */
 typedef enum	ibv_send_flags	ib_send_op_type_t;
@@ -167,6 +170,9 @@
 /* Definitions */
 #define IB_INVALID_HANDLE	NULL
 
+/* inline send rdma threshold */
+#define	INLINE_SEND_DEFAULT	128
+
 /* CM private data areas */
 #define	IB_MAX_REQ_PDATA_SIZE	92
 #define	IB_MAX_REP_PDATA_SIZE	196
@@ -174,7 +180,7 @@
 #define	IB_MAX_DREQ_PDATA_SIZE	220
 #define	IB_MAX_DREP_PDATA_SIZE	224
 
-/* DTO OPs, ordered for DAPL ENUM definitions ???*/
+/* DTO OPs, ordered for DAPL ENUM definitions */
 #define OP_RDMA_WRITE           IBV_WR_RDMA_WRITE
 #define OP_RDMA_WRITE_IMM       IBV_WR_RDMA_WRITE_WITH_IMM
 #define OP_SEND                 IBV_WR_SEND
@@ -232,13 +238,9 @@
 { 
 	struct	ibv_device	*ib_dev;
 	ib_cq_handle_t		ib_cq_empty;
-
-#if SOCKET_CM
-	int			destroy;
-	DAPL_OS_THREAD		thread;
-	DAPL_OS_LOCK		lock;	
-	struct dapl_llist_entry	*list;	
-#endif
+	int			max_inline_send;
+	uint16_t		lid;
+	union ibv_gid		gid;
 	ib_async_handler_t	async_unafiliated;
 	ib_async_handler_t	async_cq_error;
 	ib_async_handler_t	async_cq;
@@ -252,11 +254,21 @@
 /* prototypes */
 int32_t	dapls_ib_init (void);
 int32_t	dapls_ib_release (void);
+void cm_thread (void *arg);
+int dapli_cm_thread_init(void);
+void dapli_cm_thread_destroy(void);
+
+int dapli_get_lid(struct dapl_hca *hca_ptr, int port, uint16_t *lid );
+int dapli_get_gid(struct dapl_hca *hca_ptr, int port, int index, 
+		  union ibv_gid *gid );
+int dapli_get_addr(char *addr, int addr_len);
 
 DAT_RETURN
 dapls_modify_qp_state ( IN ib_qp_handle_t	qp_handle,
 			IN ib_qp_state_t	qp_state,
-			IN ib_qp_cm_t		*qp_cm );
+			IN uint32_t		qpn,
+			IN uint16_t		lid,
+			IN uint8_t		port );
 
 /* inline functions */
 STATIC _INLINE_ IB_HCA_NAME dapl_ib_convert_name (IN char *name)
Index: dapl/openib/dapl_ib_cq.c
===================================================================
--- dapl/openib/dapl_ib_cq.c	(revision 2818)
+++ dapl/openib/dapl_ib_cq.c	(working copy)
@@ -50,6 +50,7 @@
 #include "dapl_adapter_util.h"
 #include "dapl_lmr_util.h"
 #include "dapl_evd_util.h"
+#include <sys/poll.h>
 
 
 /*
@@ -448,36 +449,28 @@
 	ib_cq_handle_t		cq = evd_ptr->ib_cq_handle;
 	struct ibv_cq		*ibv_cq;
 	void			*ibv_ctx;
-	int			status = EINVAL; /* invalid handle */
+	int			status = -ETIMEDOUT; 
 
 	dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, 
 			" cq_object_wait: dev %p evd %p cq %p, time %d\n", 
 			cq->context, evd_ptr, cq, timeout );
 
-	/* TODO: add timeout, map each CQ created?? */
 	/* Multiple EVD's sharing one event handle for now */
-	while (evd_ptr->ib_cq_handle) {
+	if (cq) {
+		struct pollfd cq_poll = { 
+        		.fd      = cq->context->cq_fd[0],
+			.events  = POLLIN
+		};
+		int	timeout_ms = -1;
+
+		if (timeout != DAT_TIMEOUT_INFINITE)
+			timeout_ms = timeout/1000;
 		
-		status = ibv_get_cq_event(cq->context, 
-					  0, &ibv_cq, &ibv_ctx);
-		if (status) 
-			break;
-
-		/* EVD mismatch, process DTO callback for this EVD */
-		if (ibv_cq != cq) {
-			ib_hca_tranport_t *hca_ptr = 
-				&evd_ptr->header.owner_ia->hca_ptr->ib_trans;
-							
-			if ( hca_ptr->async_cq ) 
-				hca_ptr->async_cq(cq->context,
-						  (ib_error_record_t*)ibv_cq,
-						  ibv_ctx);
-			
-			continue;
-		} 
-		break;
+		status = poll(&cq_poll, 1, timeout_ms);
+		if (status == 1)
+			status = ibv_get_cq_event(cq->context, 
+						  0, &ibv_cq, &ibv_ctx);
 	}
-	
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
 		      " cq_object_wait: RET cq %p ibv_cq %p ibv_ctx %p %x\n",
 		      cq,ibv_cq,ibv_ctx,status);
Index: dtest/dtest.c
===================================================================
--- dtest/dtest.c	(revision 0)
+++ dtest/dtest.c	(revision 0)
@@ -0,0 +1,1700 @@
+/*
+ * Copyright (c) 2005 Intel Corporation.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: $
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <netdb.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <arpa/inet.h>
+#include <sys/mman.h>
+#include <getopt.h>
+
+#ifndef DAPL_PROVIDER
+#define DAPL_PROVIDER "OpenIB1_2"
+#endif
+
+#define MAX_POLLING_CNT 50000
+
+/* Header files needed for DAT/uDAPL */
+#include    "dat/udat.h"
+
+/* definitions */
+#define SERVER_CONN_QUAL    71123
+#define DTO_TIMEOUT	    (1000*1000*5)
+#define DTO_FLUSH_TIMEOUT   (1000*1000*2)
+#define CONN_TIMEOUT        (1000*1000*10)
+#define SERVER_TIMEOUT      (1000*1000*20)
+#define RDMA_BUFFER_SIZE    (64)
+
+/* Global DAT vars */
+static DAT_IA_HANDLE      h_ia = DAT_HANDLE_NULL;
+static DAT_PZ_HANDLE      h_pz = DAT_HANDLE_NULL;
+static DAT_EP_HANDLE      h_ep = DAT_HANDLE_NULL;
+static DAT_PSP_HANDLE     h_psp = DAT_HANDLE_NULL;
+static DAT_CR_HANDLE      h_cr = DAT_HANDLE_NULL;
+
+static DAT_EVD_HANDLE     h_async_evd = DAT_HANDLE_NULL;
+static DAT_EVD_HANDLE     h_dto_evd = DAT_HANDLE_NULL;
+static DAT_EVD_HANDLE     h_cr_evd = DAT_HANDLE_NULL;
+static DAT_EVD_HANDLE     h_conn_evd = DAT_HANDLE_NULL;
+static DAT_CNO_HANDLE     h_dto_cno = DAT_HANDLE_NULL;
+
+/* RDMA buffers */
+static DAT_LMR_HANDLE     h_lmr_send = DAT_HANDLE_NULL;
+static DAT_LMR_HANDLE     h_lmr_recv = DAT_HANDLE_NULL;
+static DAT_LMR_CONTEXT    lmr_context_send;
+static DAT_LMR_CONTEXT    lmr_context_recv;
+static DAT_RMR_CONTEXT    rmr_context_send;
+static DAT_RMR_CONTEXT    rmr_context_recv;
+static DAT_VLEN		  registered_size_send;
+static DAT_VLEN		  registered_size_recv;
+static DAT_VADDR	  registered_addr_send;
+static DAT_VADDR	  registered_addr_recv;
+
+/* Initial msg receive buf, RMR exchange, and Rdma-write notification */
+#define MSG_BUF_COUNT	  3
+static DAT_RMR_TRIPLET	  rmr_recv_msg[ MSG_BUF_COUNT ];
+static DAT_LMR_HANDLE     h_lmr_recv_msg = DAT_HANDLE_NULL;
+static DAT_LMR_CONTEXT    lmr_context_recv_msg;
+static DAT_RMR_CONTEXT    rmr_context_recv_msg;
+static DAT_VLEN		  registered_size_recv_msg;
+static DAT_VADDR	  registered_addr_recv_msg;
+
+/* message send buffer */
+static DAT_RMR_TRIPLET	    rmr_send_msg;
+static DAT_LMR_HANDLE	    h_lmr_send_msg = DAT_HANDLE_NULL;
+static DAT_LMR_CONTEXT	    lmr_context_send_msg;
+static DAT_RMR_CONTEXT	    rmr_context_send_msg;
+static DAT_VLEN		    registered_size_send_msg;
+static DAT_VADDR	    registered_addr_send_msg;
+static DAT_EP_ATTR	    ep_attr;
+char			    hostname[256] = {0};
+
+/* rdma pointers */
+char 	*rbuf = NULL;
+char	*sbuf = NULL;
+int     status;
+
+/* timers */
+double start,stop,total_us,total_sec;
+struct {
+    double  total;
+    double  open;
+    double  reg;
+    double  unreg;
+    double  pzc;
+    double  pzf;
+    double  evdc;
+    double  evdf;
+    double  cnoc;
+    double  cnof;
+    double  epc;
+    double  epf;
+    double  rdma_wr;
+    double  rdma_rd;
+    double  rtt;
+    double  close;
+} time;
+
+/* defaults */
+static int  parent=1;
+static int  connected=0;
+static int  burst=10;
+static int  server=1;
+static int  verbose=0;
+static int  polling=1;
+static int  poll_count=0;
+static int  rdma_wr_poll_count=0;
+static int  rdma_rd_poll_count=0;
+static int  pin_memory=0;
+static int  delay=0;
+static int  buf_len=RDMA_BUFFER_SIZE;
+static int  use_cno=0;
+static int  post_recv_count=MSG_BUF_COUNT;
+static int  recv_msg_index=0;
+static int  burst_msg_posted=0;
+static int  burst_msg_index=0;
+
+#define MAX_RDMA_RD	4
+#define MAX_PROCS	1000
+
+static pid_t	child[MAX_PROCS+1];
+
+/* forward prototypes */
+const char * DT_RetToString (DAT_RETURN ret_value);
+const char * DT_EventToSTr (DAT_EVENT_NUMBER event_code);
+void	    print_usage();
+double	    get_time();
+void	    init_data();
+
+DAT_RETURN	send_msg(   void                    *data,
+			    DAT_COUNT               size,
+			    DAT_LMR_CONTEXT         context,
+			    DAT_DTO_COOKIE          cookie,
+			    DAT_COMPLETION_FLAGS    flags );
+
+DAT_RETURN	connect_ep( char *hostname, int conn_id );
+void		disconnect_ep( void );
+DAT_RETURN	register_rdma_memory( void );
+DAT_RETURN	unregister_rdma_memory( void );
+DAT_RETURN	create_events( void );
+DAT_RETURN	destroy_events(void);
+DAT_RETURN	do_rdma_write_with_msg( void );
+DAT_RETURN	do_rdma_read_with_msg( void );
+DAT_RETURN	do_ping_pong_msg( void );
+
+#define LOGPRINTF(_format, _aa...) \
+	if (verbose)		   \
+	    printf(_format, ##_aa)
+
+main(int argc, char **argv)
+{
+	int c;
+	DAT_RETURN  ret;
+	        
+	/* parse arguments */
+	while ((c = getopt(argc, argv, "scvpb:d:B:h:")) != -1)
+	{
+		switch(c)
+		{
+			case 's': 
+				server = 1;
+				printf("%d Running as server\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'c': 
+				use_cno = 1;
+				printf("%d Creating CNO for DTO EVD's\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'v': 
+				verbose = 1;
+				printf("%d Verbose\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'p': 
+				polling = 1;
+				printf("%d Polling\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'B': 
+				burst = atoi(optarg);
+				break;
+			case 'd': 
+				delay = atoi(optarg);
+				break;
+			case 'b': 
+				buf_len = atoi(optarg);
+				break;
+			case 'h':
+				server = 0;
+				strcpy (hostname, optarg);
+    				break;
+			default: 
+				print_usage(); 
+				exit(-12);
+		}
+	}
+
+	if (!server) {
+		printf("%d Running as client\n",getpid()); fflush(stdout);
+	} else {
+		printf("%d Running as server\n",getpid()); fflush(stdout);
+	}
+
+	/* allocate send and receive buffers */
+	if (((rbuf = malloc(buf_len*burst)) == NULL) ||
+	    ((sbuf = malloc(buf_len*burst)) == NULL)) {  
+		perror("malloc");
+		exit(1);
+	}
+	memset( &time, sizeof(time), 0);
+	LOGPRINTF("%d Allocated RDMA buffers (r:%p,s:%p) len %d \n",
+			getpid(), rbuf, sbuf, buf_len);
+
+	/* dat_ia_open, dat_pz_create */
+	h_async_evd = DAT_HANDLE_NULL;
+	start = get_time();
+	ret = dat_ia_open( DAPL_PROVIDER, 8, &h_async_evd, &h_ia );
+	stop = get_time();
+	time.open += ((stop - start)*1.0e6);
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: Error Adaptor open: %s\n", 
+			getpid(),DT_RetToString(ret));
+		exit(1);
+	} else 
+		LOGPRINTF("%d Opened Interface Adaptor\n",getpid());
+	
+	/* Create Protection Zone */
+	start = get_time();
+	LOGPRINTF("%d Create Protection Zone\n",getpid());
+	ret = dat_pz_create(h_ia, &h_pz);
+	stop = get_time();
+	time.pzc += ((stop - start)*1.0e6);
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, 
+			"%d Error creating Protection Zone: %s\n", 
+			getpid(),DT_RetToString(ret));
+		exit(1);
+	} else 
+		LOGPRINTF("%d Created Protection Zone\n",getpid());
+	         
+	/* Register memory */
+	LOGPRINTF("%d Register RDMA memory\n", getpid());
+	ret = register_rdma_memory();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error creating events: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d Register RDMA memory done\n", getpid());
+	
+	LOGPRINTF("%d Create events\n", getpid());
+	ret = create_events();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error creating events: %s\n", 
+			getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else {
+		LOGPRINTF("%d Create events done\n", getpid());
+	}
+
+	/* create EP */
+	memset( &ep_attr, 0, sizeof(ep_attr) );
+	ep_attr.service_type                = DAT_SERVICE_TYPE_RC;
+	ep_attr.max_rdma_size               = 0x10000;
+	ep_attr.qos                         = 0;
+	ep_attr.recv_completion_flags       = 0;
+	ep_attr.max_recv_dtos               = MSG_BUF_COUNT + (burst*3);
+	ep_attr.max_request_dtos            = MSG_BUF_COUNT + (burst*3) + MAX_RDMA_RD;
+	ep_attr.max_recv_iov                = 1;
+	ep_attr.max_request_iov             = 1;
+	ep_attr.max_rdma_read_in            = MAX_RDMA_RD;
+	ep_attr.max_rdma_read_out           = MAX_RDMA_RD;
+	ep_attr.request_completion_flags    = DAT_COMPLETION_DEFAULT_FLAG;
+	ep_attr.ep_transport_specific_count = 0;
+	ep_attr.ep_transport_specific       = NULL;
+	ep_attr.ep_provider_specific_count  = 0;
+	ep_attr.ep_provider_specific        = NULL; 
+
+	start = get_time();
+	ret = dat_ep_create( h_ia, h_pz, h_dto_evd, h_dto_evd, h_conn_evd, &ep_attr, &h_ep );
+	stop = get_time();
+	time.epc += ((stop - start)*1.0e6);
+	time.total += time.epc;
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error dat_ep_create: %s\n", 
+			getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d EP created %p \n", getpid(), h_ep);
+	
+	/* 
+	 * register message buffers, establish connection, and
+	 * exchange DMA RMR information info via messages
+	 */
+	ret = connect_ep( hostname, SERVER_CONN_QUAL );
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error connect_ep: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d connect_ep complete\n", getpid());
+	
+	/*********** RDMA write data *************/
+	ret = do_rdma_write_with_msg();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error do_rdma_write_with_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d do_rdma_write_with_msg complete\n", getpid());
+	
+	/*********** RDMA read data *************/
+	ret = do_rdma_read_with_msg();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error do_rdma_read_with_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d do_rdma_read_with_msg complete\n", getpid());
+
+	/*********** PING PING messages ************/
+	ret = do_ping_pong_msg();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error do_ping_pong_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else
+		LOGPRINTF("%d do_ping_pong_msg complete\n", getpid());
+
+cleanup:
+	/* disconnect and free EP resources */
+	if ( h_ep != DAT_HANDLE_NULL ) {
+		/* unregister message buffers and tear down connection */
+		LOGPRINTF("%d Disconnect and Free EP %p \n",getpid(),h_ep);
+		disconnect_ep();
+	}
+
+	/* free EP */
+	LOGPRINTF("%d Free EP %p \n",getpid(),h_ep);
+	start = get_time();
+	ret = dat_ep_free( h_ep );
+	stop = get_time();
+	time.epf += ((stop - start)*1.0e6);
+	time.total += time.epf;
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error freeing EP: %s\n", 
+			getpid(), DT_RetToString(ret));
+	} else {
+		LOGPRINTF("%d Freed EP\n",getpid());
+		h_ep = DAT_HANDLE_NULL;
+	}
+
+	/* free EVDs */
+	LOGPRINTF("%d destroy events\n", getpid());
+	ret = destroy_events();
+	if(ret != DAT_SUCCESS) 
+		fprintf(stderr, "%d Error destroy_events: %s\n", 
+			getpid(),DT_RetToString(ret));
+	else 
+		LOGPRINTF("%d destroy events done\n", getpid());
+	
+
+	ret = unregister_rdma_memory();
+	LOGPRINTF("%d unregister_rdma_memory \n", getpid());
+	if(ret != DAT_SUCCESS) 
+		fprintf(stderr, "%d Error unregister_rdma_memory: %s\n", 
+			getpid(),DT_RetToString(ret));
+	else 
+		LOGPRINTF("%d unregister_rdma_memory done\n", getpid());
+	
+
+	if (delay) sleep(delay);
+    	       
+	/* Free protection domain */
+	LOGPRINTF("%d Freeing pz\n",getpid());
+	start = get_time();
+	ret = dat_pz_free( h_pz );
+	stop = get_time();
+	time.pzf += ((stop - start)*1.0e6);
+	if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error freeing PZ: %s\n", 
+		getpid(), DT_RetToString(ret));
+	} else {
+		LOGPRINTF("%d Freed pz\n",getpid());
+		h_pz = NULL;
+	}
+
+	/* close the device */
+	LOGPRINTF("%d Closing Interface Adaptor\n",getpid());
+	start = get_time();
+	ret = dat_ia_close( h_ia, DAT_CLOSE_ABRUPT_FLAG );
+	stop = get_time();
+	time.close += ((stop - start)*1.0e6);
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: Error Adaptor close: %s\n", 
+			getpid(),DT_RetToString(ret));
+		exit(1);
+	} else 
+		LOGPRINTF("%d Closed Interface Adaptor\n",getpid());
+
+        printf( "\n%d: DAPL Test Complete.\n\n",getpid());
+	printf( "%d: RDMA write:  Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n", 
+		getpid(), time.rdma_wr, burst, time.rdma_wr/burst, rdma_wr_poll_count );
+	printf( "%d: RDMA read:   Total=%10.2lf usec,   %d bursts, itime=%10.2lf usec, pc=%d\n", 
+		getpid(), time.rdma_rd, MAX_RDMA_RD, time.rdma_rd/MAX_RDMA_RD, rdma_rd_poll_count );
+	printf( "%d: Message RTT: Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n\n", 
+		getpid(), time.rtt, burst, time.rtt/burst, poll_count );
+
+        printf( "%d: open:      %10.2lf usec\n", getpid(), time.open  );
+        printf( "%d: close:     %10.2lf usec\n", getpid(), time.close );
+        printf( "%d: PZ create: %10.2lf usec\n", getpid(), time.pzc );
+        printf( "%d: PZ free:   %10.2lf usec\n", getpid(), time.pzf );
+        printf( "%d: LMR create:%10.2lf usec\n", getpid(), time.reg );
+        printf( "%d: LMR free:  %10.2lf usec\n", getpid(), time.unreg );
+        printf( "%d: EVD create:%10.2lf usec\n", getpid(), time.evdc );
+        printf( "%d: EVD free:  %10.2lf usec\n", getpid(), time.evdf );
+        if (use_cno) {
+	    printf( "%d: CNO create:  %10.2lf usec\n", getpid(), time.cnoc );
+	    printf( "%d: CNO free:    %10.2lf usec\n", getpid(), time.cnof );
+        }
+        printf( "%d: EP create: %10.2lf usec\n",getpid(), time.epc );
+        printf( "%d: EP free:   %10.2lf usec\n",getpid(), time.epf );
+        printf( "%d: TOTAL:     %10.2lf usec\n",getpid(), time.total );
+    
+	/* free rdma buffers */  
+	free(rbuf);
+	free(sbuf);
+}
+
+
+double get_time()
+{
+	struct timeval tp;
+
+	gettimeofday(&tp, NULL);
+	return ((double) tp.tv_sec + (double) tp.tv_usec * 1e-6);
+}
+ 
+void init_data()
+{
+	memset(rbuf, 'a', buf_len);
+	memset(sbuf, 'b', buf_len);
+}
+
+
+DAT_RETURN
+send_msg(   void                    *data,
+	    DAT_COUNT               size,
+	    DAT_LMR_CONTEXT         context,
+	    DAT_DTO_COOKIE          cookie,
+	    DAT_COMPLETION_FLAGS    flags )
+{
+    DAT_LMR_TRIPLET	iov;
+    DAT_EVENT		event;
+    DAT_COUNT		nmore;
+    DAT_RETURN		ret;
+
+    iov.lmr_context     = context;
+    iov.pad             = 0;
+    iov.virtual_address = (DAT_VADDR)(unsigned long)data;
+    iov.segment_length  = size;
+    LOGPRINTF("%d calling post_send\n", getpid());
+    cookie.as_64 = 0xaaaa;
+    ret = dat_ep_post_send( h_ep,
+			    1,
+			    &iov,
+			    cookie,
+			    flags );
+
+    if (ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d: ERROR: dat_ep_post_send() %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return ret;
+    }
+
+    if (!(flags & DAT_COMPLETION_SUPPRESS_FLAG)) {
+	if ( polling ) {
+	    printf("%d Polling post send completion...\n",getpid());
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY );
+	} 
+	else {
+	    LOGPRINTF("%d waiting for post_send completion event\n", getpid());
+	    if (use_cno) {
+		DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		if ( evd != h_dto_evd ) {
+	    	    fprintf(stderr, 
+			"%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+			getpid());
+		    return( DAT_ABORT );
+		}
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+	        fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+			getpid(),DT_RetToString(ret));
+	        return ret;
+	    }
+	}
+
+	/* validate event number, len, cookie, and status */
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d: ERROR: DTO event number %s\n", 
+		    getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+	
+	if ((event.event_data.dto_completion_event_data.transfered_length != size ) ||
+	    (event.event_data.dto_completion_event_data.user_cookie.as_64 != 0xaaaa )) {
+	    fprintf(stderr, "%d: ERROR: DTO len %d or cookie %x\n", 
+		getpid(),
+		event.event_data.dto_completion_event_data.transfered_length,
+		event.event_data.dto_completion_event_data.user_cookie.as_64 );
+	    return( DAT_ABORT );
+		
+	}
+	if (event.event_data.dto_completion_event_data.status != DAT_SUCCESS) {
+	    fprintf(stderr, "%d: ERROR: DTO event status %s\n", 
+		    getpid(),DT_RetToString(ret));
+	    return( DAT_ABORT );
+	}
+    }
+
+    return DAT_SUCCESS;
+}
+
+
+DAT_RETURN
+connect_ep( char *hostname, int conn_id )
+{
+	DAT_SOCK_ADDR		remote_addr;
+	DAT_EP_ATTR		ep_attr;
+	DAT_RETURN		ret;
+	DAT_REGION_DESCRIPTION	region;
+	DAT_EVENT		event;
+	DAT_COUNT		nmore;
+	DAT_LMR_TRIPLET		l_iov;
+	DAT_RMR_TRIPLET		r_iov;
+	DAT_DTO_COOKIE		cookie;
+	int			i;
+    
+     /* Register send message buffer */
+    LOGPRINTF("%d Registering send Message Buffer %p, len %d\n",
+		getpid(), &rmr_send_msg, sizeof(DAT_RMR_TRIPLET) );
+    region.for_va = &rmr_send_msg;
+    ret = dat_lmr_create(   h_ia, 
+		            DAT_MEM_TYPE_VIRTUAL, 
+		            region, 
+		            sizeof(DAT_RMR_TRIPLET), 
+		            h_pz, 
+		            DAT_MEM_PRIV_LOCAL_WRITE_FLAG, 
+		            &h_lmr_send_msg, 
+		            &lmr_context_send_msg, 
+		            &rmr_context_send_msg,
+		            &registered_size_send_msg, 
+		            &registered_addr_send_msg );
+
+    if (ret != DAT_SUCCESS) {
+	fprintf(stderr, "%d Error registering send msg buffer: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return(ret);
+    } 
+    else 
+    	LOGPRINTF("%d Registered send Message Buffer %p \n",
+	        getpid(),region.for_va );
+    
+    /* Register Receive buffers */
+    LOGPRINTF("%d Registering Receive Message Buffer %p\n",
+		getpid(), rmr_recv_msg );
+    region.for_va = rmr_recv_msg;
+    ret = dat_lmr_create(   h_ia, 
+		            DAT_MEM_TYPE_VIRTUAL, 
+			    region, 
+			    sizeof(DAT_RMR_TRIPLET)*MSG_BUF_COUNT, 
+			    h_pz, 
+			    DAT_MEM_PRIV_LOCAL_WRITE_FLAG, 
+			    &h_lmr_recv_msg, 
+			    &lmr_context_recv_msg, 
+			    &rmr_context_recv_msg,
+			    &registered_size_recv_msg, 
+			    &registered_addr_recv_msg );
+    if(ret != DAT_SUCCESS) {
+	fprintf(stderr, "%d Error registering recv msg buffer: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return(ret);
+    } 
+    else 
+    	LOGPRINTF("%d Registered Receive Message Buffer %p\n",
+	        getpid(),region.for_va);
+    
+    for ( i = 0; i < MSG_BUF_COUNT; i++ ) {
+	cookie.as_64          = i;
+	l_iov.lmr_context     = lmr_context_recv_msg;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)&rmr_recv_msg[ i ];
+	l_iov.segment_length  = sizeof(DAT_RMR_TRIPLET);
+        
+	LOGPRINTF("%d Posting Receive Message Buffer %p\n", 
+		    getpid(), &rmr_recv_msg[ i ]);
+	ret = dat_ep_post_recv( h_ep,
+				1,
+				&l_iov,
+				cookie,
+				DAT_COMPLETION_DEFAULT_FLAG );
+        
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error registering recv msg buffer: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+        } 
+	else 
+	    LOGPRINTF("%d Registered Receive Message Buffer %p\n",
+			            getpid(),region.for_va);
+		
+    }
+
+    /* setup receive rdma buffer to initial string to be overwritten */
+    strcpy( (char*)rbuf, "blah, blah, blah\n" );
+
+    if ( server ) {  /* SERVER */
+    
+        /* create the service point for server listen */
+        LOGPRINTF("%d Creating service point for listen\n",getpid());
+	ret = dat_psp_create(   h_ia,
+				conn_id,
+				h_cr_evd,
+				DAT_PSP_CONSUMER_FLAG,
+				&h_psp );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_psp_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_psp_created for server listen\n", getpid());
+	
+        printf("%d Server waiting for connect request..\n", getpid());
+	ret = dat_evd_wait( h_cr_evd, SERVER_TIMEOUT, 1, &event, &nmore );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_evd_wait: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_evd_wait for cr_evd completed\n", getpid());
+	
+	if ( event.event_number != DAT_CONNECTION_REQUEST_EVENT ) {
+            fprintf(stderr, "%d Error unexpected cr event : %s\n", 
+				    getpid(),DT_EventToSTr(event.event_number));
+			return( DAT_ABORT );
+	}
+	if ( (event.event_data.cr_arrival_event_data.conn_qual != SERVER_CONN_QUAL) ||
+	     (event.event_data.cr_arrival_event_data.sp_handle.psp_handle != h_psp) ) {
+            fprintf(stderr, "%d Error wrong cr event data : %s\n", 
+		    getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+
+        /* accept connect request from client */
+	h_cr = event.event_data.cr_arrival_event_data.cr_handle;
+        LOGPRINTF("%d Accepting connect request from client\n",getpid());
+	ret = dat_cr_accept( h_cr, h_ep, 0, (DAT_PVOID)0 );
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_cr_accept: %s\n", 
+		    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_cr_accept completed\n", getpid());
+    } 
+    else {  /* CLIENT */ 
+	struct addrinfo	*target;
+	int		rval;
+
+	if (getaddrinfo (hostname, NULL, NULL, &target) != 0) {
+	    printf("\n remote name resolution failed!\n");
+	    exit ( 1 );
+	}
+
+	rval = ((struct sockaddr_in *)target->ai_addr)->sin_addr.s_addr;
+	printf ("%d Server Name: %s \n", getpid(), hostname);
+	printf ("%d Server Net Address: %d.%d.%d.%d\n", getpid(),
+	    (rval >>  0) & 0xff,
+	    (rval >>  8) & 0xff,
+	    (rval >> 16) & 0xff,
+	    (rval >> 24) & 0xff);
+
+	remote_addr = *((DAT_IA_ADDRESS_PTR)target->ai_addr);
+
+	LOGPRINTF("%d Connecting to server\n",getpid());
+	ret = dat_ep_connect(   h_ep,
+			        &remote_addr,
+			        conn_id,
+			        CONN_TIMEOUT,
+			        0,
+			        (DAT_PVOID)0,
+			        0,
+			        DAT_CONNECT_DEFAULT_FLAG  );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_ep_connect: %s\n", 
+				getpid(), DT_RetToString(ret));
+ 	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_ep_connect completed\n", getpid());
+    }
+
+    printf("%d Waiting for connect response\n",getpid());
+    
+    ret = dat_evd_wait( h_conn_evd, DAT_TIMEOUT_INFINITE, 1, &event, &nmore );
+    if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_evd_wait: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+    } 
+    else 
+    	    LOGPRINTF("%d dat_evd_wait for h_conn_evd completed\n", getpid());
+    
+    if ( event.event_number != DAT_CONNECTION_EVENT_ESTABLISHED ) {
+	    fprintf(stderr, "%d Error unexpected conn event : %s\n", 
+				getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+    }
+    printf("\n%d CONNECTED!\n\n",getpid());
+    connected = 1;
+	
+    /*
+     *  Setup our remote memory and tell the other side about it
+     */
+    rmr_send_msg.rmr_context    = rmr_context_recv;
+    rmr_send_msg.pad	         = 0;
+    rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf;
+    rmr_send_msg.segment_length = RDMA_BUFFER_SIZE;
+
+    printf("%d Send RMR to remote: snd_msg: r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n",
+	    getpid(), rmr_send_msg.rmr_context, rmr_send_msg.pad,
+	    rmr_send_msg.target_address, rmr_send_msg.segment_length );
+
+    ret = send_msg( &rmr_send_msg,
+		    sizeof( DAT_RMR_TRIPLET ),
+		    lmr_context_send_msg,
+		    cookie,
+		    DAT_COMPLETION_SUPPRESS_FLAG );
+
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error send_msg: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return(ret);
+    } 
+    else 
+	LOGPRINTF("%d send_msg completed\n", getpid());
+    
+    /*
+     *  Wait for remote RMR information for RDMA
+     */
+    if ( polling ) {
+	printf("%d Polling for remote to send RMR data\n",getpid());
+	while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY );		
+    } 
+    else  {
+	printf("%d Waiting for remote to send RMR data\n",getpid());
+	if (use_cno) 
+	{
+	    DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+	    ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+	    LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+	    if ( evd != h_dto_evd ) {
+	        fprintf(stderr, 
+			"%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+			getpid());
+	        return( DAT_ABORT );
+	    }
+	}
+	/* use wait to dequeue */
+	ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error waiting on h_dto_evd: %s\n", 
+		    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else {
+	    LOGPRINTF("%d dat_evd_wait h_dto_evd completed\n", getpid());
+	}
+    }
+
+    printf("%d remote RMR data arrived!\n",getpid());
+    
+    if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+        fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+		getpid(),DT_EventToSTr(event.event_number));
+        return( DAT_ABORT );
+    }
+    if ((event.event_data.dto_completion_event_data.transfered_length != 
+		sizeof( DAT_RMR_TRIPLET )) ||
+	(event.event_data.dto_completion_event_data.user_cookie.as_64 != 
+		recv_msg_index) ) {
+	fprintf(stderr,"ERR recv event: len=%d cookie=%d expected %d/%d\n",
+	    (int)event.event_data.dto_completion_event_data.transfered_length,
+	    (int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+	    sizeof(DAT_RMR_TRIPLET), recv_msg_index );
+	return( DAT_ABORT );
+    }
+
+    r_iov = rmr_recv_msg[ recv_msg_index ];
+    
+    printf("%d Received RMR from remote: r_iov: r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n",
+	    getpid(), r_iov.rmr_context, r_iov.pad,
+	    r_iov.target_address, r_iov.segment_length );
+
+    recv_msg_index++;
+
+    return ( DAT_SUCCESS );
+}
+
+
+void
+disconnect_ep()
+{
+    DAT_RETURN	ret;
+    DAT_EVENT	event;
+    DAT_COUNT	nmore;
+    int		i,flush_cnt;
+	
+    if (connected) {
+	
+	LOGPRINTF("%d dat_ep_disconnect\n", getpid());
+	ret = dat_ep_disconnect( h_ep, DAT_CLOSE_DEFAULT );
+	if(ret != DAT_SUCCESS) 	{
+		fprintf(stderr, "%d Error dat_ep_disconnect: %s\n", 
+				getpid(),DT_RetToString(ret));
+	} 
+	else {
+		LOGPRINTF("%d dat_ep_disconnect completed\n", getpid());
+	}
+    }
+    
+    /* destroy service point */
+    if (( server ) && ( h_psp != DAT_HANDLE_NULL )) {
+	ret = dat_psp_free( h_psp );
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_psp_free: %s\n", 
+		    getpid(),DT_RetToString(ret));
+	} 
+	else {
+	    LOGPRINTF("%d dat_psp_free completed\n", getpid());
+	}
+    }
+   
+    /* Unregister Send message Buffer */
+    if ( h_lmr_send_msg != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister send message h_lmr %p \n",getpid(),h_lmr_send_msg);
+	ret = dat_lmr_free(h_lmr_send_msg);
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering send msg mr: %s\n", 
+	    getpid(), DT_RetToString(ret));
+	} else {
+	    LOGPRINTF("%d Unregistered send message Buffer\n",getpid());
+	    h_lmr_send_msg = NULL;
+	}
+    }
+
+    /* Unregister recv message Buffer */
+    if ( h_lmr_recv_msg != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister recv message h_lmr %p \n",getpid(),h_lmr_recv_msg);
+	ret = dat_lmr_free(h_lmr_recv_msg);
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering recv msg mr: %s\n", 
+			    getpid(), DT_RetToString(ret));
+	} else {
+	    LOGPRINTF("%d Unregistered recv message Buffer\n",getpid());
+	    h_lmr_recv_msg = NULL;
+	}
+    }
+    return;
+}
+
+
+DAT_RETURN
+do_rdma_write_with_msg( )
+{
+	DAT_REGION_DESCRIPTION	region;
+	DAT_EVENT		event;
+	DAT_COUNT		nmore;
+	DAT_LMR_TRIPLET		l_iov;
+	DAT_RMR_TRIPLET		r_iov;
+	DAT_DTO_COOKIE		cookie;
+	DAT_RMR_CONTEXT		their_context;
+	DAT_RETURN		ret;
+	int			i;
+
+	printf("\n %d RDMA WRITE DATA with SEND MSG\n\n",getpid());
+
+	cookie.as_64 = 0x5555;
+	
+	if ( recv_msg_index >= MSG_BUF_COUNT )
+		return( DAT_ABORT );
+
+	/* get RMR information from previously received message */
+	r_iov = rmr_recv_msg[ recv_msg_index-1 ];
+	
+	if ( server )
+	    strcpy( (char*)sbuf, "server written data..." );
+	else
+	    strcpy( (char*)sbuf, "client written data..." );
+
+	l_iov.lmr_context     = lmr_context_send;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf;
+	l_iov.segment_length  = buf_len;
+	
+	start = get_time();
+	for (i=0;i<burst;i++) {
+	    cookie.as_64 = 0x9999;
+	    ret = dat_ep_post_rdma_write(   h_ep,		// ep_handle
+					    1,			// num_segments
+					    &l_iov,		// LMR
+					    cookie,		// user_cookie
+					    &r_iov,		// RMR
+					    DAT_COMPLETION_SUPPRESS_FLAG );
+	    if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: ERROR: dat_ep_post_rdma_write() %s\n", 
+					getpid(),DT_RetToString(ret));
+		return( DAT_ABORT );
+	    }
+	    LOGPRINTF("%d rdma_write # %d completed\n", getpid(),i+1);
+	}
+	
+	/*
+	 *  Send RMR information a 2nd time to indicate completion
+	 */
+	rmr_send_msg.rmr_context    = rmr_context_recv;
+	rmr_send_msg.pad	    = 0;
+	rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf;
+	rmr_send_msg.segment_length = RDMA_BUFFER_SIZE;
+
+	printf("%d Sending completion message\n",getpid());
+    
+	ret = send_msg(	&rmr_send_msg,
+		        sizeof( DAT_RMR_TRIPLET ),
+		        lmr_context_send_msg,
+		        cookie,
+		        DAT_COMPLETION_SUPPRESS_FLAG );
+
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		return(ret);
+	} else {
+		LOGPRINTF("%d send_msg completed\n", getpid());
+	}
+
+	/*
+	 *  Collect first event, write completion or the inbound recv with immed
+	 */
+	if ( polling ) {
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY )
+		rdma_wr_poll_count++;
+	} 
+	else {
+	    LOGPRINTF("%d waiting for message receive event\n", getpid());
+	    if (use_cno)  {
+		    DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		    ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		    LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		    if ( evd != h_dto_evd ) {
+			    fprintf(stderr, "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+					    getpid());
+			    return( ret );
+		    }
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+		    fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+					    getpid(),DT_RetToString(ret));
+		    return( ret );
+	    }
+	}
+	stop = get_time();
+	time.rdma_wr = ((stop - start)*1.0e6);
+
+	/* validate event number and status */
+	printf("%d inbound rdma_write; send message arrived!\n",getpid());
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+				getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+
+	if ( (event.event_data.dto_completion_event_data.transfered_length != sizeof(
DAT_RMR_TRIPLET )) ||
+	     (event.event_data.dto_completion_event_data.user_cookie.as_64 != recv_msg_index) ) {
+		
+	    fprintf(stderr,"unexpected event data for receive: len=%d cookie=%d exp %d/%d\n",
+		(int)event.event_data.dto_completion_event_data.transfered_length,
+		(int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+		sizeof(DAT_RMR_TRIPLET), recv_msg_index );
+
+	    return( DAT_ABORT );
+	}
+
+	r_iov = rmr_recv_msg[ recv_msg_index ];
+
+	printf("%d Received RMR from remote: r_iov: ctx=%x,pad=%x,va=%p,len=0x%x\n",
+		    getpid(), r_iov.rmr_context, 
+            r_iov.pad,
+            (void*)(unsigned long)r_iov.target_address,
+		    r_iov.segment_length );
+
+	LOGPRINTF("%d inbound rdma_write; send msg event SUCCESS!!!\n", getpid());
+    
+	printf("%d %s RDMA write buffer contains: %s\n", 
+			getpid(),
+			server ? "SERVER:" : "CLIENT:", 
+			rbuf );
+
+	recv_msg_index++;
+
+	return ( DAT_SUCCESS );
+}
+
+DAT_RETURN
+do_rdma_read_with_msg( )
+{
+	DAT_REGION_DESCRIPTION	region;
+	DAT_EVENT		event;
+	DAT_COUNT		nmore;
+	DAT_LMR_TRIPLET		l_iov;
+	DAT_RMR_TRIPLET		r_iov;
+	DAT_DTO_COOKIE		cookie;
+	DAT_RMR_CONTEXT		their_context;
+	DAT_RETURN		ret;
+	int			i;
+
+	printf("\n %d RDMA READ DATA with SEND MSG\n\n",getpid());
+
+	cookie.as_64 = 0x5555;
+	
+	if ( recv_msg_index >= MSG_BUF_COUNT )
+		return( DAT_ABORT );
+
+	/* get RMR information from previously received message */
+	r_iov = rmr_recv_msg[ recv_msg_index-1 ];
+	
+	/* setup rdma read buffer to initial string to be overwritten */
+	strcpy( (char*)sbuf, "blah, blah, blah\n" );
+
+	if ( server )
+	    strcpy( (char*)rbuf, "server read data..." );
+	else
+	    strcpy( (char*)rbuf, "client read data..." );
+
+	l_iov.lmr_context     = lmr_context_send;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf;
+	l_iov.segment_length  = buf_len;
+	
+	start = get_time();
+	for (i=0;i<MAX_RDMA_RD;i++) {
+	    cookie.as_64 = 0x9999;
+	    ret = dat_ep_post_rdma_read(    h_ep,		// ep_handle
+					    1,			// num_segments
+					    &l_iov,		// LMR
+					    cookie,		// user_cookie
+					    &r_iov,		// RMR
+					    DAT_COMPLETION_SUPPRESS_FLAG );
+	    if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: ERROR: dat_ep_post_rdma_read() %s\n", 
+					getpid(),DT_RetToString(ret));
+		return( DAT_ABORT );
+	    }
+	    LOGPRINTF("%d rdma_read # %d completed\n", getpid(),i+1);
+	}
+	
+	/*
+	 *  Send RMR information a 2nd time to indicate completion
+	 */
+	rmr_send_msg.rmr_context    = rmr_context_recv;
+	rmr_send_msg.pad	    = 0;
+	rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf;
+	rmr_send_msg.segment_length = RDMA_BUFFER_SIZE;
+
+	printf("%d Sending completion message\n",getpid());
+    
+	ret = send_msg(	&rmr_send_msg,
+		        sizeof( DAT_RMR_TRIPLET ),
+		        lmr_context_send_msg,
+		        cookie,
+		        DAT_COMPLETION_SUPPRESS_FLAG );
+
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		return(ret);
+	} else {
+		LOGPRINTF("%d send_msg completed\n", getpid());
+	}
+
+	/*
+	 *  Collect first event, write completion or the inbound recv with immed
+	 */
+	printf("%d Waiting for inbound message....\n",getpid());
+	if ( polling ) {
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY )
+		rdma_rd_poll_count++;
+	} 
+	else {
+	    LOGPRINTF("%d waiting for message receive event\n", getpid());
+	    if (use_cno) {
+		    DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		    ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		    LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		    if ( evd != h_dto_evd ) {
+			    fprintf(stderr, "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+					    getpid());
+			    return( ret );
+		    }
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+		    fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+					    getpid(),DT_RetToString(ret));
+		    return( ret );
+	    }
+	}
+	stop = get_time();
+	time.rdma_rd = ((stop - start)*1.0e6);
+
+	/* validate event number and status */
+	printf("%d inbound rdma_read; send message arrived!\n",getpid());
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+				getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+
+	if ( (event.event_data.dto_completion_event_data.transfered_length != sizeof(
DAT_RMR_TRIPLET )) ||
+	     (event.event_data.dto_completion_event_data.user_cookie.as_64 != recv_msg_index) ) {
+		
+	    fprintf(stderr,"unexpected event data for receive: len=%d cookie=%d exp %d/%d\n",
+		(int)event.event_data.dto_completion_event_data.transfered_length,
+		(int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+		sizeof(DAT_RMR_TRIPLET), recv_msg_index );
+
+	    return( DAT_ABORT );
+	}
+
+	r_iov = rmr_recv_msg[ recv_msg_index ];
+
+	printf("%d Received RMR from remote: r_iov: ctx=%x,pad=%x,va=%p,len=0x%x\n",
+		getpid(), r_iov.rmr_context, r_iov.pad,
+		(void*)(unsigned long)r_iov.target_address, r_iov.segment_length );
+
+	LOGPRINTF("%d inbound rdma_write; send msg event SUCCESS!!!\n", getpid());
+    
+	printf("%d %s RCV RDMA read buffer contains: %s\n", 
+			getpid(),
+			server ? "SERVER:" : "CLIENT:", 
+			sbuf );
+
+	recv_msg_index++;
+
+	return ( DAT_SUCCESS );
+}
+
+
+DAT_RETURN
+do_ping_pong_msg( )
+{
+    DAT_EVENT		event;
+    DAT_COUNT		nmore;
+    DAT_DTO_COOKIE	cookie;
+    DAT_LMR_TRIPLET	l_iov;
+    DAT_RETURN		ret;
+    int			i;
+    unsigned char	*snd_buf;
+    unsigned char	*rcv_buf;
+
+    printf("\n %d PING DATA with SEND MSG\n\n",getpid());
+    
+    snd_buf = sbuf;
+    rcv_buf = rbuf;
+
+    /* pre-post all buffers */
+    for ( i=0; i < burst; i++ ) {	
+	burst_msg_posted++;
+	cookie.as_64	      = i;
+	l_iov.lmr_context     = lmr_context_recv;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)rcv_buf;
+	l_iov.segment_length  = buf_len;
+        
+	LOGPRINTF("%d Pre-posting Receive Message Buffers %p\n", 
+		    getpid(), rcv_buf );
+	
+	ret = dat_ep_post_recv( h_ep,
+				1,
+				&l_iov,
+				cookie,
+				DAT_COMPLETION_DEFAULT_FLAG );
+        
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error posting recv msg buffer: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+        } 
+	else {
+	    LOGPRINTF("%d Posted Receive Message Buffer %p\n",
+			            getpid(),rcv_buf);
+        }
+
+	/* next buffer */
+	rcv_buf += buf_len;
+    }
+    sleep(1);
+    
+    /* Initialize recv_buf and index to beginning */
+    rcv_buf = rbuf;
+    burst_msg_index=0;
+
+    /* client ping 0x55, server pong 0xAA in first byte */
+    start = get_time();
+    for ( i=0;i<burst;i++ ) {
+	/* walk the send and recv buffers */
+	if ( !server ) {
+	    *snd_buf = 0x55;
+
+	    LOGPRINTF("%d %s SND buffer %p contains: 0x%x len=%d\n", 
+		    getpid(), server ? "SERVER:" : "CLIENT:", 
+		    snd_buf, *snd_buf, buf_len );
+
+	    ret = send_msg( snd_buf,
+			    buf_len,
+			    lmr_context_send,
+			    cookie,
+			    DAT_COMPLETION_SUPPRESS_FLAG );
+
+	    if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				    getpid(),DT_RetToString(ret));
+	        return(ret);
+	    } 
+	    else {
+	        LOGPRINTF("%d send_msg completed\n", getpid());
+	    }
+	}
+	   
+	/* Wait for recv message */
+	if ( polling ) {
+	    poll_count=0;
+	    LOGPRINTF("%d Polling for message receive event\n", getpid());
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY )
+		poll_count++;
+	} 
+	else {
+	    LOGPRINTF("%d waiting for message receive event\n", getpid());
+	    if (use_cno) {
+		DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		if ( evd != h_dto_evd ) 
+		{
+		    fprintf(stderr, "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+					getpid());
+		    return( ret );
+		}
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+					getpid(),DT_RetToString(ret));
+		return( ret );
+	    }
+	}
+	/* start timer after first message arrives on server */
+	if ( i == 0) {
+	    start = get_time();
+	}
+	/* validate event number and status */
+	LOGPRINTF("%d inbound message; message arrived!\n",getpid());
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+		    getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+	if ((event.event_data.dto_completion_event_data.transfered_length 
+	    != buf_len) ||
+	    (event.event_data.dto_completion_event_data.user_cookie.as_64 
+	    != burst_msg_index) )  {
+    	    fprintf(stderr,"ERR: recv event: len=%d cookie=%d exp %d/%d\n",
+		(int)event.event_data.dto_completion_event_data.transfered_length,
+		(int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+		buf_len, burst_msg_index );
+
+	    return( DAT_ABORT );
+	}
+        
+	LOGPRINTF("%d %s RCV buffer %p contains: 0x%x len=%d\n", 
+		    getpid(), server ? "SERVER:" : "CLIENT:", 
+		    rcv_buf, *rcv_buf, buf_len );
+
+	burst_msg_index++;
+
+	/* If server, change data and send it back to client */
+	if ( server ) {
+	    *snd_buf = 0xaa;
+
+	    LOGPRINTF("%d %s SND buffer %p contains: 0x%x len=%d\n", 
+		    getpid(), server ? "SERVER:" : "CLIENT:", 
+		    snd_buf, *snd_buf, buf_len );
+	    
+	    ret = send_msg( snd_buf,
+			    buf_len,
+			    lmr_context_send,
+			    cookie,
+			    DAT_COMPLETION_SUPPRESS_FLAG );
+
+	    if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				    getpid(),DT_RetToString(ret));
+		return(ret);
+	    } 
+	    else {
+		LOGPRINTF("%d send_msg completed\n", getpid());
+	    }
+	}
+
+	/* next buffers */
+	rcv_buf += buf_len;
+	snd_buf += buf_len;
+    }
+    stop = get_time();
+    time.rtt = ((stop - start)*1.0e6);
+    
+    return ( DAT_SUCCESS );
+}
+
+/* Register RDMA Receive buffer */
+DAT_RETURN
+register_rdma_memory(void)
+{
+    DAT_RETURN				ret;
+    DAT_REGION_DESCRIPTION region;
+
+    region.for_va = rbuf;
+    start = get_time();
+    ret = dat_lmr_create(	h_ia, 
+			    DAT_MEM_TYPE_VIRTUAL, 
+			    region, 
+			    buf_len*burst, 
+			    h_pz, 
+			    DAT_MEM_PRIV_ALL_FLAG, 
+			    &h_lmr_recv, 
+			    &lmr_context_recv, 
+			    &rmr_context_recv,
+			    &registered_size_recv, 
+			    &registered_addr_recv );
+    stop = get_time();
+    time.reg += ((stop - start)*1.0e6);
+    time.total += time.reg;
+
+    if(ret != DAT_SUCCESS) {
+	fprintf(stderr, "%d Error registering recv buffer: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return (ret);
+    } else {
+	    LOGPRINTF("%d Registered Receive RDMA Buffer %p\n",
+			getpid(),region.for_va);
+    }
+
+    /* Register RDMA Send buffer */
+    region.for_va = sbuf;
+    ret = dat_lmr_create(   h_ia, 
+			    DAT_MEM_TYPE_VIRTUAL, 
+			    region, 
+			    buf_len*burst, 
+			    h_pz, 
+			    DAT_MEM_PRIV_ALL_FLAG, 
+			    &h_lmr_send, 
+			    &lmr_context_send, 
+			    &rmr_context_send,
+			    &registered_size_send, 
+			    &registered_addr_send );
+    if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error registering send RDMA buffer: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return (ret);
+    } else {
+	    LOGPRINTF("%d Registered Send RDMA Buffer %p\n",
+		    getpid(),region.for_va);
+    }
+
+    return DAT_SUCCESS;
+}
+
+/* 
+ * Unregister RDMA memory 
+ */
+DAT_RETURN
+unregister_rdma_memory(void)
+{
+    DAT_RETURN	ret;
+
+    /* Unregister Recv Buffer */
+    if ( h_lmr_recv != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister h_lmr %p \n",getpid(),h_lmr_recv);
+        start = get_time();
+	ret = dat_lmr_free(h_lmr_recv);
+        stop = get_time();
+	time.unreg += ((stop - start)*1.0e6);
+	time.total += time.unreg;
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering recv mr: %s\n", 
+			getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Unregistered Recv Buffer\n",getpid());
+	    h_lmr_recv = NULL; 
+	}
+    }
+
+    /* Unregister Send Buffer */
+    if ( h_lmr_send != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister h_lmr %p \n",getpid(),h_lmr_send);
+	ret = dat_lmr_free(h_lmr_send);
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering send mr: %s\n", 
+			    getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Unregistered send Buffer\n",getpid());
+	    h_lmr_send = NULL;
+	}
+    }
+    return DAT_SUCCESS;
+}
+
+ /* 
+  * Create CNO, CR, CONN, and DTO events 
+  */
+DAT_RETURN
+create_events(void)
+{
+    DAT_RETURN	ret;
+        
+    /* create CNO */
+    if (use_cno) {
+        start = get_time();
+        ret = dat_cno_create( h_ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &h_dto_cno  );
+        stop = get_time();
+	time.cnoc += ((stop - start)*1.0e6);
+	time.total += time.cnoc;
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_cno_create: %s\n", 
+				getpid(),DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d cr_evd created, %p\n", getpid(), h_dto_cno);
+	}
+    }
+
+    /* create cr EVD */
+    start = get_time();
+    ret = dat_evd_create( h_ia, 10, DAT_HANDLE_NULL, DAT_EVD_CR_FLAG, &h_cr_evd  );
+    stop = get_time();
+    time.evdc += ((stop - start)*1.0e6);
+    time.total += time.evdc;
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error dat_evd_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return (ret);
+    } 
+    else {
+        LOGPRINTF("%d cr_evd created %p\n", getpid(),h_cr_evd);
+    }
+    /* create conn EVD */
+    ret = dat_evd_create( h_ia, 10, DAT_HANDLE_NULL, DAT_EVD_CONNECTION_FLAG, &h_conn_evd  );
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error dat_evd_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return (ret);
+    } 
+    else {
+        LOGPRINTF("%d con_evd created %p\n", getpid(),h_conn_evd);
+    }     
+	
+    /* create dto EVD, with CNO if use_cno was set */
+    ret = dat_evd_create( h_ia,
+			  (MSG_BUF_COUNT*2)+burst*2, 
+			  h_dto_cno, 
+			  DAT_EVD_DTO_FLAG, 
+			  &h_dto_evd  );
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error dat_evd_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return (ret);
+    } 
+    else {
+        LOGPRINTF("%d dto_evd created %p\n", getpid(), h_dto_evd );
+    }
+
+    return DAT_SUCCESS;
+}
+
+/*
+ * Destroy CR, CONN, CNO, and DTO events 
+ */ 
+DAT_RETURN
+destroy_events(void)
+{
+	DAT_RETURN	ret;
+	
+	/* free cr EVD */
+    if ( h_cr_evd != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free cr EVD %p \n",getpid(),h_cr_evd);
+        ret = dat_evd_free( h_cr_evd );
+        if(ret != DAT_SUCCESS) {
+		    fprintf(stderr, "%d Error freeing cr EVD: %s\n", 
+			    getpid(), DT_RetToString(ret));
+			return (ret);
+	    } else {
+		    LOGPRINTF("%d Freed cr EVD\n",getpid());
+            h_cr_evd = DAT_HANDLE_NULL;
+	    }
+    }
+
+    /* free conn EVD */
+    if ( h_conn_evd != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free conn EVD %p \n",getpid(),h_conn_evd);
+        ret = dat_evd_free( h_conn_evd );
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error freeing conn EVD: %s\n", 
+			getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Freed conn EVD\n",getpid());
+	    h_conn_evd = DAT_HANDLE_NULL;
+	}
+    }
+
+    /* free dto EVD */
+    if ( h_dto_evd != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free dto EVD %p \n",getpid(),h_dto_evd);
+        start = get_time();
+	ret = dat_evd_free( h_dto_evd );
+        stop = get_time();
+	time.evdf += ((stop - start)*1.0e6);
+	time.total += time.evdf;
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error freeing dto EVD: %s\n", 
+			    getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Freed dto EVD\n",getpid());
+            h_dto_evd = DAT_HANDLE_NULL;
+	}
+    }
+
+    /* free CNO */
+    if ( h_dto_cno != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free dto CNO %p \n",getpid(),h_dto_cno);
+        start = get_time();
+        ret = dat_cno_free( h_dto_cno );
+        stop = get_time();
+        time.cnof += ((stop - start)*1.0e6);
+        time.total += time.cnof;
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error freeing dto CNO: %s\n", 
+	    	    getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Freed dto CNO\n",getpid());
+            h_dto_cno = DAT_HANDLE_NULL;
+	}
+    }
+    return DAT_SUCCESS;
+}
+
+/*
+ * Map DAT_RETURN values to readable strings,
+ * but don't assume the values are zero-based or contiguous.
+ */
+char    errmsg[512] = {0};
+const char *
+DT_RetToString (DAT_RETURN ret_value)
+{
+    const char *major_msg, *minor_msg;
+    int sz;
+        
+    dat_strerror (ret_value, &major_msg, &minor_msg);
+  
+    strcpy(errmsg, major_msg);
+    strcat(errmsg, " ");
+    strcat(errmsg, minor_msg);
+
+    return errmsg;
+}
+
+/*
+ * Map DAT_EVENT_CODE values to readable strings
+ */
+const char *
+DT_EventToSTr (DAT_EVENT_NUMBER event_code)
+{
+    unsigned int i;
+    static struct {
+	    const char  *name;
+	    DAT_RETURN  value;
+    }
+    dat_events[] =
+    {
+	    #   define DATxx(x) { # x, x }
+	    DATxx (DAT_DTO_COMPLETION_EVENT),
+	    DATxx (DAT_RMR_BIND_COMPLETION_EVENT),
+	    DATxx (DAT_CONNECTION_REQUEST_EVENT),
+	    DATxx (DAT_CONNECTION_EVENT_ESTABLISHED),
+	    DATxx (DAT_CONNECTION_EVENT_PEER_REJECTED),
+	    DATxx (DAT_CONNECTION_EVENT_NON_PEER_REJECTED),
+	    DATxx (DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR),
+	    DATxx (DAT_CONNECTION_EVENT_DISCONNECTED),
+	    DATxx (DAT_CONNECTION_EVENT_BROKEN),
+	    DATxx (DAT_CONNECTION_EVENT_TIMED_OUT),
+	    DATxx (DAT_CONNECTION_EVENT_UNREACHABLE),
+	    DATxx (DAT_ASYNC_ERROR_EVD_OVERFLOW),
+	    DATxx (DAT_ASYNC_ERROR_IA_CATASTROPHIC),
+	    DATxx (DAT_ASYNC_ERROR_EP_BROKEN),
+	    DATxx (DAT_ASYNC_ERROR_TIMED_OUT),
+	    DATxx (DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR),
+	    DATxx (DAT_SOFTWARE_EVENT)
+	    #   undef DATxx
+    };
+    #   define NUM_EVENTS (sizeof(dat_events)/sizeof(dat_events[0]))
+
+    for (i = 0;  i < NUM_EVENTS;  i++) {
+	    if (dat_events[i].value == event_code)
+	    {
+	        return ( dat_events[i].name );
+	    }
+    }
+
+    return ( "Invalid_DAT_EVENT_NUMBER" );
+}
+
+
+void print_usage()
+{
+    printf("\n DAPL USAGE \n\n");
+    printf("s: server\n");
+    printf("c: use cno\n");
+    printf("v: verbose\n");
+    printf("p: polling\n");
+    printf("d: delay before close\n");
+    printf("b: buf length to allocate\n");
+    printf("B: burst count, rdma and msgs \n");
+    printf("h: hostname\n");
+    printf("\n");
+}
+
Index: dtest/dat.conf
===================================================================
--- dtest/dat.conf	(revision 0)
+++ dtest/dat.conf	(revision 0)
@@ -0,0 +1,11 @@
+#
+# DAT 1.1 and 1.2 configuration file
+#
+# Each entry should have the following fields:
+#
+# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
+#           <provider_version> <ia_params> <platform_params>
+#
+# Example for openib using the first Mellanox adapter,  port 1 and port 2
+
+IB1 u1.2 nonthreadsafe default
/home/ardavis/Proj/openib/gen2/users/jlentini/userspace/dapl/udapl/Target/libdapl.so r.1.2 "mthca0
1" ""
Index: dtest/README
===================================================================
--- dtest/README	(revision 0)
+++ dtest/README	(revision 0)
@@ -0,0 +1,19 @@
+simple dapl test just for initial openIB uDAPL testing...
+
+	dtest/dtest.c
+	dtest/makefile
+	dtest/dat.conf
+
+to build (default uDAPL name == IB1, ib device  ==  mthca0, port == 1)
+       edit makefile and change path (DAT_LIB) to appropriate libdat.so
+       edit dat.conf and change path to appropriate libdapl.so 
+       cp dat.conf to /etc/dat.conf 
+
+to run:
+	server:	dtest  
+	client:	dtest -h hostname
+
+for verbose uDAPL and uDAT debug:
+
+	export DAPL_DBG_TYPE=0xffff
+	export DAT_DBG_TYPE=0xffff
Index: dtest/makefile
===================================================================
--- dtest/makefile	(revision 0)
+++ dtest/makefile	(revision 0)
@@ -0,0 +1,16 @@
+CC         = gcc
+CFLAGS     = -O2 -g 
+
+DAT_INC = ../dat/include 
+DAT_LIB = /usr/lib64 
+
+all: dtest 
+
+clean:
+	rm -f *.o;touch *.c;rm -f dtest
+
+dtest: ./dtest.c 
+	$(CC) $(CFLAGS) ./dtest.c -o dtest \
+        -DDAPL_PROVIDER='"IB1"' \
+        -I $(DAT_INC) -L $(DAT_LIB) -ldat 
+	  







From halr at voltaire.com  Thu Jul  7 17:03:12 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 20:03:12 -0400
Subject: [Openib-windows] RE: [openib-general] IBDM and IBMgtSim
	ProposalComments
In-Reply-To: <ORSMSX401Rqf69aZbLA00000015@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401Rqf69aZbLA00000015@orsmsx401.amr.corp.intel.com>
Message-ID: <1120780747.4386.185.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 17:52, Sean Hefty wrote:
> Unless I'm mistaken, there are more than a dozen diag tools running on Linux
> that are coded to umad.  I think that Hal sees porting umad as the easiest way
> to convert those utilities to Windows.

Exactly and I don't think the application porting should be a big deal.
I think it is more interfacing to what is below it.

> It seems to me that the person writing the code will decide whether they should
> interface at the lowest layer or write to a higher level abstraction for
> portability.  I don't see that either way is wrong.  And if the higher level
> abstraction is the native interface on one platform, that sounds like a benefit.

I agree. Whoever does this work will explain and decide their course on
this.

-- Hal



From halr at voltaire.com  Thu Jul  7 17:06:26 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 07 Jul 2005 20:06:26 -0400
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <001001c5833d$c1cdf070$6501a8c0@infiniconsys.com>
References: <001001c5833d$c1cdf070$6501a8c0@infiniconsys.com>
Message-ID: <1120780793.4386.188.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 17:49, Fab Tillier wrote:
> However, it doesn't sound that an effort from Eitan to port the diagnostics to
> OSMV would be accepted into the Linux SVN repository simply on the basis that it
> isn't umad.  What gives?

I think it is more than that. There are also some data model/file format
issues to be reconciled.

I wonder how many mixed Windows/Linux environments there will be but the
more things are common between the two the better IMO.

-- Hal



From rolandd at cisco.com  Thu Jul  7 19:35:48 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 19:35:48 -0700
Subject: [openib-general] [PATCH] ipoib: updates to copyright
In-Reply-To: <20050707165333.GA18363@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 7 Jul 2005 19:53:33 +0300")
References: <20050707165333.GA18363@mellanox.co.il>
Message-ID: <5264vmyqgr.fsf@topspin.com>

Thanks, applied.

 - R.


From rolandd at cisco.com  Thu Jul  7 20:46:25 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 07 Jul 2005 20:46:25 -0700
Subject: [openib-general] [PATCH] mthca: report board id in sysfs
In-Reply-To: <20050627150308.GD25304@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 27 Jun 2005 18:03:08 +0300")
References: <20050627150308.GD25304@mellanox.co.il>
Message-ID: <521x6ayn72.fsf@topspin.com>

It seems your original patch only reported the board ID for PCI-X
HCAs, because of the code duplication between mthca_init_tavor and
mthca_init_arbel for the call to QUERY_ADAPTER.  This is the patch I
cam up with, which consolidates the code so we only call QUERY_ADAPTER
from one place.

I think this is correct but none of my PCI-X HCAs seem to have a board
ID -- it works fine on my PCIe HCAs but I get all 00000000s on the
PCI-X HCAs that I tried.

Are my HCAs just missing VPD in their flash or is the patch messed up?

 - R.


--- infiniband/hw/mthca/mthca_dev.h	(revision 2815)
+++ infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -68,6 +68,10 @@ enum {
 };
 
 enum {
+	MTHCA_BOARD_ID_LEN = 16
+};
+
+enum {
 	MTHCA_EQ_CONTEXT_SIZE =  0x40,
 	MTHCA_CQ_CONTEXT_SIZE =  0x40,
 	MTHCA_QP_CONTEXT_SIZE = 0x200,
@@ -247,6 +251,7 @@ struct mthca_dev {
 	unsigned long    device_cap_flags;
 
 	u32              rev_id;
+	char             board_id[MTHCA_BOARD_ID_LEN];
 
 	/* firmware info */
 	u64              fw_ver;
--- infiniband/hw/mthca/mthca_main.c	(revision 2815)
+++ infiniband/hw/mthca/mthca_main.c	(working copy)
@@ -213,7 +213,6 @@ static int __devinit mthca_init_tavor(st
 	struct mthca_dev_lim        dev_lim;
 	struct mthca_profile        profile;
 	struct mthca_init_hca_param init_hca;
-	struct mthca_adapter        adapter;
 
 	err = mthca_SYS_EN(mdev, &status);
 	if (err) {
@@ -271,26 +270,8 @@ static int __devinit mthca_init_tavor(st
 		goto err_disable;
 	}
 
-	err = mthca_QUERY_ADAPTER(mdev, &adapter, &status);
-	if (err) {
-		mthca_err(mdev, "QUERY_ADAPTER command failed, aborting.\n");
-		goto err_close;
-	}
-	if (status) {
-		mthca_err(mdev, "QUERY_ADAPTER returned status 0x%02x, "
-			  "aborting.\n", status);
-		err = -EINVAL;
-		goto err_close;
-	}
-
-	mdev->eq_table.inta_pin = adapter.inta_pin;
-	mdev->rev_id            = adapter.revision_id;
-
 	return 0;
 
-err_close:
-	mthca_CLOSE_HCA(mdev, 0, &status);
-
 err_disable:
 	mthca_SYS_DIS(mdev, &status);
 
@@ -507,7 +488,6 @@ static int __devinit mthca_init_arbel(st
 	struct mthca_dev_lim        dev_lim;
 	struct mthca_profile        profile;
 	struct mthca_init_hca_param init_hca;
-	struct mthca_adapter        adapter;
 	u64 icm_size;
 	u8 status;
 	int err;
@@ -575,21 +555,6 @@ static int __devinit mthca_init_arbel(st
 		goto err_free_icm;
 	}
 
-	err = mthca_QUERY_ADAPTER(mdev, &adapter, &status);
-	if (err) {
-		mthca_err(mdev, "QUERY_ADAPTER command failed, aborting.\n");
-		goto err_free_icm;
-	}
-	if (status) {
-		mthca_err(mdev, "QUERY_ADAPTER returned status 0x%02x, "
-			  "aborting.\n", status);
-		err = -EINVAL;
-		goto err_free_icm;
-	}
-
-	mdev->eq_table.inta_pin = adapter.inta_pin;
-	mdev->rev_id            = adapter.revision_id;
-
 	return 0;
 
 err_free_icm:
@@ -615,12 +580,68 @@ err_disable:
 	return err;
 }
 
+static void mthca_close_hca(struct mthca_dev *mdev)
+{
+	u8 status;
+
+	mthca_CLOSE_HCA(mdev, 0, &status);
+
+	if (mthca_is_memfree(mdev)) {
+		mthca_free_icm_table(mdev, mdev->cq_table.table);
+		mthca_free_icm_table(mdev, mdev->qp_table.rdb_table);
+		mthca_free_icm_table(mdev, mdev->qp_table.eqp_table);
+		mthca_free_icm_table(mdev, mdev->qp_table.qp_table);
+		mthca_free_icm_table(mdev, mdev->mr_table.mpt_table);
+		mthca_free_icm_table(mdev, mdev->mr_table.mtt_table);
+		mthca_unmap_eq_icm(mdev);
+
+		mthca_UNMAP_ICM_AUX(mdev, &status);
+		mthca_free_icm(mdev, mdev->fw.arbel.aux_icm);
+
+		mthca_UNMAP_FA(mdev, &status);
+		mthca_free_icm(mdev, mdev->fw.arbel.fw_icm);
+
+		if (!(mdev->mthca_flags & MTHCA_FLAG_NO_LAM))
+			mthca_DISABLE_LAM(mdev, &status);
+	} else
+		mthca_SYS_DIS(mdev, &status);
+}
+
 static int __devinit mthca_init_hca(struct mthca_dev *mdev)
 {
+	u8 status;
+	int err;
+	struct mthca_adapter adapter;
+
 	if (mthca_is_memfree(mdev))
-		return mthca_init_arbel(mdev);
+		err = mthca_init_arbel(mdev);
 	else
-		return mthca_init_tavor(mdev);
+		err = mthca_init_tavor(mdev);
+
+	if (err)
+		return err;
+
+	err = mthca_QUERY_ADAPTER(mdev, &adapter, &status);
+	if (err) {
+		mthca_err(mdev, "QUERY_ADAPTER command failed, aborting.\n");
+		goto err_close;
+	}
+	if (status) {
+		mthca_err(mdev, "QUERY_ADAPTER returned status 0x%02x, "
+			  "aborting.\n", status);
+		err = -EINVAL;
+		goto err_close;
+	}
+
+	mdev->eq_table.inta_pin = adapter.inta_pin;
+	mdev->rev_id            = adapter.revision_id;
+	memcpy(mdev->board_id, adapter.board_id, sizeof mdev->board_id);
+
+	return 0;
+
+err_close:
+	mthca_close_hca(mdev);
+	return err;
 }
 
 static int __devinit mthca_setup_hca(struct mthca_dev *dev)
@@ -845,33 +866,6 @@ static int __devinit mthca_enable_msi_x(
 	return 0;
 }
 
-static void mthca_close_hca(struct mthca_dev *mdev)
-{
-	u8 status;
-
-	mthca_CLOSE_HCA(mdev, 0, &status);
-
-	if (mthca_is_memfree(mdev)) {
-		mthca_free_icm_table(mdev, mdev->cq_table.table);
-		mthca_free_icm_table(mdev, mdev->qp_table.rdb_table);
-		mthca_free_icm_table(mdev, mdev->qp_table.eqp_table);
-		mthca_free_icm_table(mdev, mdev->qp_table.qp_table);
-		mthca_free_icm_table(mdev, mdev->mr_table.mpt_table);
-		mthca_free_icm_table(mdev, mdev->mr_table.mtt_table);
-		mthca_unmap_eq_icm(mdev);
-
-		mthca_UNMAP_ICM_AUX(mdev, &status);
-		mthca_free_icm(mdev, mdev->fw.arbel.aux_icm);
-
-		mthca_UNMAP_FA(mdev, &status);
-		mthca_free_icm(mdev, mdev->fw.arbel.fw_icm);
-
-		if (!(mdev->mthca_flags & MTHCA_FLAG_NO_LAM))
-			mthca_DISABLE_LAM(mdev, &status);
-	} else
-		mthca_SYS_DIS(mdev, &status);
-}
-
 /* Types of supported HCA */
 enum {
 	TAVOR,			/* MT23108                        */
--- infiniband/hw/mthca/mthca_provider.c	(revision 2815)
+++ infiniband/hw/mthca/mthca_provider.c	(working copy)
@@ -956,14 +956,22 @@ static ssize_t show_hca(struct class_dev
 	}
 }
 
+static ssize_t show_board(struct class_device *cdev, char *buf)
+{
+	struct mthca_dev *dev = container_of(cdev, struct mthca_dev, ib_dev.class_dev);
+	return sprintf(buf, "%.*s\n", MTHCA_BOARD_ID_LEN, dev->board_id);
+}
+
 static CLASS_DEVICE_ATTR(hw_rev,   S_IRUGO, show_rev,    NULL);
 static CLASS_DEVICE_ATTR(fw_ver,   S_IRUGO, show_fw_ver, NULL);
 static CLASS_DEVICE_ATTR(hca_type, S_IRUGO, show_hca,    NULL);
+static CLASS_DEVICE_ATTR(board_id, S_IRUGO, show_board,  NULL);
 
 static struct class_device_attribute *mthca_class_attributes[] = {
 	&class_device_attr_hw_rev,
 	&class_device_attr_fw_ver,
-	&class_device_attr_hca_type
+	&class_device_attr_hca_type,
+	&class_device_attr_board_id
 };
 
 int mthca_register_device(struct mthca_dev *dev)
--- infiniband/hw/mthca/mthca_cmd.c	(revision 2815)
+++ infiniband/hw/mthca/mthca_cmd.c	(working copy)
@@ -1088,6 +1088,7 @@ int mthca_QUERY_ADAPTER(struct mthca_dev
 {
 	struct mthca_mailbox *mailbox;
 	u32 *outbox;
+	int i;
 	int err;
 
 #define QUERY_ADAPTER_OUT_SIZE             0x100
@@ -1095,6 +1096,7 @@ int mthca_QUERY_ADAPTER(struct mthca_dev
 #define QUERY_ADAPTER_DEVICE_ID_OFFSET     0x04
 #define QUERY_ADAPTER_REVISION_ID_OFFSET   0x08
 #define QUERY_ADAPTER_INTA_PIN_OFFSET      0x10
+#define QUERY_ADAPTER_BOARD_ID_OFFSET      0xf0
 
 	mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL);
 	if (IS_ERR(mailbox))
@@ -1112,6 +1114,15 @@ int mthca_QUERY_ADAPTER(struct mthca_dev
 	MTHCA_GET(adapter->revision_id, outbox, QUERY_ADAPTER_REVISION_ID_OFFSET);
 	MTHCA_GET(adapter->inta_pin, outbox,    QUERY_ADAPTER_INTA_PIN_OFFSET);
 
+	/*
+	 * The board ID is a string but the firmware byte swaps each
+	 * 4-byte word before passing it back to us.  Therefore we
+	 * need to swab it before printing it.
+	 */
+	for (i = 0; i < sizeof adapter->board_id / 4; ++i)
+		adapter->board_id[i] =
+			swab32(outbox[QUERY_ADAPTER_BOARD_ID_OFFSET / 4 + i]);
+
 out:
 	mthca_free_mailbox(dev, mailbox);
 	return err;
--- infiniband/hw/mthca/mthca_cmd.h	(revision 2815)
+++ infiniband/hw/mthca/mthca_cmd.h	(working copy)
@@ -187,6 +187,7 @@ struct mthca_adapter {
 	u32 vendor_id;
 	u32 device_id;
 	u32 revision_id;
+	u32 board_id[MTHCA_BOARD_ID_LEN / 4];
 	u8  inta_pin;
 };
 


From mst at mellanox.co.il  Thu Jul  7 22:55:06 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 8 Jul 2005 08:55:06 +0300
Subject: [openib-general] [PATCH] mthca: report board id in sysfs
In-Reply-To: <521x6ayn72.fsf@topspin.com>
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com>
Message-ID: <20050708055506.GA4989@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [openib-general] [PATCH] mthca: report board id in sysfs
> 
> It seems your original patch only reported the board ID for PCI-X
> HCAs, because of the code duplication between mthca_init_tavor and
> mthca_init_arbel for the call to QUERY_ADAPTER.  This is the patch I
> cam up with, which consolidates the code so we only call QUERY_ADAPTER
> from one place.
> 
> I think this is correct but none of my PCI-X HCAs seem to have a board
> ID -- it works fine on my PCIe HCAs but I get all 00000000s on the
> PCI-X HCAs that I tried.

Looks ok ... need testing.

> Are my HCAs just missing VPD in their flash or is the patch messed up?
> 
>  - R.

To check, you can type

mstflint -d <bus:dev.fn> q

-- 
MST


From gleb at minantech.com  Thu Jul  7 23:25:57 2005
From: gleb at minantech.com (Gleb Natapov)
Date: Fri, 8 Jul 2005 09:25:57 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <ORSMSX401TRqf69aZbL00000014@orsmsx401.amr.corp.intel.com>
References: <20050707212710.GE28506@mellanox.co.il>
	<ORSMSX401TRqf69aZbL00000014@orsmsx401.amr.corp.intel.com>
Message-ID: <20050708062557.GA30898@minantech.com>

On Thu, Jul 07, 2005 at 02:35:36PM -0700, Sean Hefty wrote:
> >Quoting r. Gleb Natapov <gleb at minantech.com>:
> >> Tried that. But with big values create_qp fails.
> >
> >Yes, we need to fix that.
> 
> To me, the current behavior sounds exactly like what you want.  The maximum or
> some sort of "guaranteed value" should be available through the device
> properties.  Needing to determine it only by creating a QP doesn't sound like
> the right approach.
I am not sure what is the "current behavior" you are referring to.
Currently the only way I know to determine inline data size is create
QP is there other way?

> 
> BTW, in tests that I've seen previously, using the largest possible inline data
> size didn't yield the best system performance, but that could have simply been
> based on the tests that we being run.
I want the clear way to determine inline size then I'll optimise for
performance.

--
			Gleb.


From mst at mellanox.co.il  Fri Jul  8 00:46:39 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 8 Jul 2005 10:46:39 +0300
Subject: [openib-general] Re: IBDM and IBMgtSim Proposal Comments
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30460@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30460@mtlex01.yok.mtl.com>
Message-ID: <20050708074639.GA5204@mellanox.co.il>

Quoting r. Eitan Zahavi <eitan at mellanox.co.il>:
> Subject: RE: IBDM and IBMgtSim Proposal Comments
> 
> The point is that it is better to write user level code that is not
> "performance critical" in a way that is not tailored to the actual driver/
> Access layer.

Performance is not the only reason to have a clean architecture.
Why cant the windows driver match the openib API?

-- 
MST


From clifton at axelero.hu  Thu Jul  7 23:30:37 2005
From: clifton at axelero.hu (Thelma Bermudez)
Date: Fri, 8 Jul 2005 09:30:37 +0300
Subject: [openib-general] Two hundred and thirty thousand dollars for only
	$320/month
Message-ID: <MAILSENDERNG7XsNQpt9384ad52@hotmail.com>

Hello,

You have been chosen to participate in an invitation only limited time event! 
Are you currently paying over 3% for your mortgage? STOP! We can help you lower that today!
Answer only a few questions and we can give you an approval in under 30 seconds – it’s that simple! 

http://gcS.m3ssages.com/p1.asp

And stop fighting for lenders – let them fight for you! Make them work for your business by giving you the lowest rates around!
$230,000 loans are available for only $340/month! WE’RE PRACTICALLY GIVING AWAY MONEY!

Think your credit is too bad to get a deal like this? THINK AGAIN! We will have you saving your money in no time!
Are you ready to save your money? http://d9Xl.m3ssages.com/p1.asp

To update our records go m3ssages.com/deletion.asp, database updates can take up to 72 hours.

Regards,
Thelma Bermudez



From halr at voltaire.com  Fri Jul  8 03:52:38 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 08 Jul 2005 06:52:38 -0400
Subject: [openib-general] RE: [Openib-windows] RE: IBDM and IBMgtSim
	Proposal Comments
In-Reply-To: <1120772064.4386.169.camel@hal.voltaire.com>
References: <000d01c5833a$865e8390$6501a8c0@infiniconsys.com>
	<1120772064.4386.169.camel@hal.voltaire.com>
Message-ID: <1120819958.4386.194.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 17:35, Hal Rosenstock wrote:
> On Thu, 2005-07-07 at 17:26, Fab Tillier wrote:
> > > So it appears there are 3 choices:
> > > 1. Port OpenIB Linux libraries to Windows and OpenIB Linux "diags" port
> > > as well (second part is less work than next alternative)
> > > 2. Port OpenIB Linux diagnostics to OSM vendor layer
> > > 3. No OpenIB "Linux" diagnostics in the windows environment
> > 
> > What about:
> > 4. Port OpenIB Linux "diags" to Windows/IBAL.
> > 
> > Seems like a pretty obvious choice, and what I've been talking about in my
> > previous emails.
> 
> Yes, that is another alternative but is similar to choice 2 in terms of
> affecting all the applications.

One more thing about approach 4:

One downside of this approach is that these tools would not work with
the Mellanox simulator as that simulator relies on the OSM vendor layer.

This is not to advocate approach 2 over approach 4 but just trying to
articulate what the end result would be.

-- Hal



From halr at voltaire.com  Fri Jul  8 05:39:49 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 08 Jul 2005 08:39:49 -0400
Subject: [openib-general] [PATCH] mthca: Add copyright
Message-ID: <1120826389.4386.210.camel@hal.voltaire.com>

mthca: Add copyright to files touched by Voltaire

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: mthca_dev.h
===================================================================
--- mthca_dev.h	(revision 2820)
+++ mthca_dev.h	(working copy)
@@ -3,6 +3,7 @@
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: mthca_provider.c
===================================================================
--- mthca_provider.c	(revision 2820)
+++ mthca_provider.c	(working copy)
@@ -3,6 +3,7 @@
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Cisco Systems. All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: mthca_mad.c
===================================================================
--- mthca_mad.c	(revision 2820)
+++ mthca_mad.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: mthca_cq.c
===================================================================
--- mthca_cq.c	(revision 2820)
+++ mthca_cq.c	(working copy)
@@ -3,6 +3,7 @@
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Cisco Systems, Inc. All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: mthca_qp.c
===================================================================
--- mthca_qp.c	(revision 2820)
+++ mthca_qp.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems. All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved. 
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU





From halr at voltaire.com  Fri Jul  8 05:44:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 08 Jul 2005 08:44:39 -0400
Subject: [openib-general] [PATCH] IPoIB: Add copyright
Message-ID: <1120826390.4386.212.camel@hal.voltaire.com>

IPoIB: Add copyright to files touched by Voltaire

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: ipoib_main.c
===================================================================
--- ipoib_main.c	(revision 2820)
+++ ipoib_main.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: ipoib.h
===================================================================
--- ipoib.h	(revision 2820)
+++ ipoib.h	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: ipoib_multicast.c
===================================================================
--- ipoib_multicast.c	(revision 2820)
+++ ipoib_multicast.c	(working copy)
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: ipoib_ib.c
===================================================================
--- ipoib_ib.c	(revision 2820)
+++ ipoib_ib.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU





From halr at voltaire.com  Fri Jul  8 05:53:36 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 08 Jul 2005 08:53:36 -0400
Subject: [openib-general] [PATCH] core: Add copyright
Message-ID: <1120827215.4386.225.camel@hal.voltaire.com>

core: Add copyright to files touched by Voltaire

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: uverbs.h
===================================================================
--- uverbs.h	(revision 2820)
+++ uverbs.h	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: uverbs_main.c
===================================================================
--- uverbs_main.c	(revision 2820)
+++ uverbs_main.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Cisco Systems.  All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
Index: cache.c
===================================================================
--- cache.c	(revision 2820)
+++ cache.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Intel Corporation. All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU





From halr at voltaire.com  Fri Jul  8 05:59:06 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 08 Jul 2005 08:59:06 -0400
Subject: [openib-general] [PATCH] CM: Add copyright
Message-ID: <1120827545.4386.233.camel@hal.voltaire.com>

CM: Add copyright to file touched by Voltaire

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: cm.c
===================================================================
--- cm.c        (revision 2820)
+++ cm.c        (working copy)
@@ -3,6 +3,7 @@
  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU




From rolandd at cisco.com  Fri Jul  8 09:32:16 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 09:32:16 -0700
Subject: [openib-general] [PATCH] mthca: report board id in sysfs
In-Reply-To: <20050708055506.GA4989@mellanox.co.il> (Michael S. Tsirkin's
	message of "Fri, 8 Jul 2005 08:55:06 +0300")
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
Message-ID: <52pstttg1b.fsf@topspin.com>

    Michael> To check, you can type

    Michael> mstflint -d <bus:dev.fn> q

Looks like the board ID is not there.

I think I understand what's happening now: on Topspin HCAs, the
vendor-specific area is formatted differently, and the board ID string
is not in the last 16 bytes of the VSD.

What's the best way to make this work for Topspin HCAs?

 - R.


From sean.hefty at intel.com  Fri Jul  8 09:32:28 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 09:32:28 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050708062557.GA30898@minantech.com>
Message-ID: <ORSMSX401FRaqbC8wSA00000017@orsmsx401.amr.corp.intel.com>

>> >> Tried that. But with big values create_qp fails.
>> >
>> >Yes, we need to fix that.
>>
>> To me, the current behavior sounds exactly like what you want.  The maximum
>or
>> some sort of "guaranteed value" should be available through the device
>> properties.  Needing to determine it only by creating a QP doesn't sound like
>> the right approach.
>I am not sure what is the "current behavior" you are referring to.
>Currently the only way I know to determine inline data size is create
>QP is there other way?

By "current behavior" I meant that create_qp should fail if an inline size is
specified that is too large.

- Sean



From sean.hefty at intel.com  Fri Jul  8 09:43:28 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 09:43:28 -0700
Subject: [openib-general] RE: [PATCH] CM: Add copyright
In-Reply-To: <1120827545.4386.233.camel@hal.voltaire.com>
Message-ID: <ORSMSX401FRaqbC8wSA00000018@orsmsx401.amr.corp.intel.com>

>CM: Add copyright to file touched by Voltaire
>
>Signed-off-by: Hal Rosenstock <halr at voltaire.com>
>
>Index: cm.c
>===================================================================
>--- cm.c        (revision 2820)
>+++ cm.c        (working copy)
>@@ -3,6 +3,7 @@
>  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
>  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
>  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
>+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
>  *
>  * This software is available to you under a choice of one of two
>  * licenses.  You may choose to be licensed under the terms of the GNU

Do you want to add a new line or just modify the existing one?

- Sean



From arlin.r.davis at intel.com  Fri Jul  8 09:47:58 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Fri, 8 Jul 2005 09:47:58 -0700
Subject: [openib-general] [PATCH] uDAPL with IB_CM support
Message-ID: <ORSMSX401GryWtIIZS200000019@orsmsx401.amr.corp.intel.com>


James,

Patch for OpenIB uDAPL provider with IB_CM support instead of SOCKET_CM. It was a little too messy
to make conditional and keep SOCKET_CM around so this version only supports IB_CM. Path records are
hand rolled until IBAT support is added in the next 1-2 weeks. See README for details on setup and
limitations. 

Signed-off-by: Arlin Davis <ardavis at ichips.intel.com>

Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile	(revision 2811)
+++ dapl/udapl/Makefile	(working copy)
@@ -122,7 +122,7 @@
 #
 ifeq ($(VERBS),openib)
 PROVIDER = $(TOPDIR)/../openib
-CFLAGS   += -DSOCKET_CM -DOPENIB -DCQ_WAIT_OBJECT
+CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
 CFLAGS   += -I/usr/local/include/infiniband
 endif
 
Index: dapl/openib/TODO
===================================================================
--- dapl/openib/TODO	(revision 2818)
+++ dapl/openib/TODO	(working copy)
@@ -4,16 +4,12 @@
 - query call to get current qp state 
 - ibv_get_cq_event() needs timed event call and wakeup
 - query call to get device attributes
-- current implementation only supports one event per device
 - memory window support
 
 DAPL:
-- Build udapl issues with mthca having reverse dependencies to ibverbs
-- When real CM arrives: change modify_qp_state RTS RTR calls
 - reinit EP needs a QP timewait completion notification
-- code disconnect clean 
 - add cq_object wakeup, time based cq_object wait when verbs support arrives
-- update uDAPL code with real CM and ATS support
+- update uDAPL code with real ATS support
 - etc, etc.
 
 Other:
Index: dapl/openib/dapl_ib_dto.h
===================================================================
--- dapl/openib/dapl_ib_dto.h	(revision 2818)
+++ dapl/openib/dapl_ib_dto.h	(working copy)
@@ -143,6 +143,7 @@
 	ib_data_segment_t	*ds_array_p;
 	struct ibv_send_wr	wr;
 	struct ibv_send_wr	*bad_wr;
+	ib_hca_tranport_t	*ibt_ptr = &ep_ptr->header.owner_ia->hca_ptr->ib_trans;
 	DAT_COUNT		i, total_len;
 	
 	dapl_dbg_log (DAPL_DBG_TYPE_EP,
@@ -195,6 +196,11 @@
 				wr.wr.rdma.rkey, wr.wr.rdma.remote_addr );
 	}
 
+	/* inline data for send or write ops */
+	if ((total_len <= ibt_ptr->max_inline_send ) && 
+	   ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE))) 
+		wr.send_flags |= IBV_SEND_INLINE;
+	
 	/* set completion flags in work request */
 	wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG & 
 				completion_flags) ? 0 : IBV_SEND_SIGNALED;
Index: dapl/openib/dapl_ib_util.c
===================================================================
--- dapl/openib/dapl_ib_util.c	(revision 2818)
+++ dapl/openib/dapl_ib_util.c	(working copy)
@@ -58,34 +58,82 @@
 #include <sys/utsname.h>
 #include <unistd.h>	
 #include <fcntl.h>
+#include <strings.h>
 
 int g_dapl_loopback_connection = 0;
 
-#ifdef SOCKET_CM
+/* get lid */
+int dapli_get_lid(struct dapl_hca *hca_ptr, int port, uint16_t *lid )
+{
+	struct ibv_port_attr attr;
+
+	if (ibv_query_port(hca_ptr->ib_hca_handle, port, &attr))
+		return 1;
+
+	*lid = attr.lid;
+
+	return 0;
+}
+
+/* get gid */
+int dapli_get_gid(struct dapl_hca *hca_ptr, int port,
+                         int index, union ibv_gid *gid )
+{
+        /* ibv_query_gid() coming soon, until then HACK */
+        char path[128];
+        char val[40];
+        char name[256];
+        char *token;
+        uint16_t *p_gid;
+
+        if (sysfs_get_mnt_path(path, sizeof path)) {
+                fprintf(stderr, "Couldn't find sysfs mount.\n");
+                return 1;
+        }
+        sprintf(name, "%s/class/infiniband/%s/ports/%d/gids/%d", path,
+                 ibv_get_device_name(hca_ptr->ib_trans.ib_dev), port, index);
+
+        if (sysfs_read_attribute_value(name, val, sizeof val)) {
+                fprintf(stderr, "Couldn't read GID at %s\n", name);
+                return 1;
+        }
+
+        /* get token strings with delimiter */
+        token = strtok(val,":");
+        p_gid = (uint16_t*)gid->raw;
+        while (token) {
+                *p_gid = strtoul(token,NULL,16);
+		*p_gid = htons(*p_gid); /* convert each token to network order */
+                token = strtok(NULL,":");
+                p_gid++;
+        }
+        return 0;
+}
+
+
 /* just get IP address for hostname */
-DAT_RETURN getipaddr( char *addr, int addr_len)
+int dapli_get_addr( char *addr, int addr_len)
 {
 	struct sockaddr_in	*ipv4_addr = (struct sockaddr_in*)addr;
 	struct hostent		*h_ptr;
 	struct utsname		ourname;
 
 	if ( uname( &ourname ) < 0 ) 
-		return DAT_INTERNAL_ERROR;
+		return 1;
 
 	h_ptr = gethostbyname( ourname.nodename );
 	if ( h_ptr == NULL ) 
-		return DAT_INTERNAL_ERROR;
+		return 1;
 
 	if ( h_ptr->h_addrtype == AF_INET ) {
 		ipv4_addr = (struct sockaddr_in*) addr;
 		ipv4_addr->sin_family = AF_INET;
 		dapl_os_memcpy( &ipv4_addr->sin_addr, h_ptr->h_addr_list[0], 4 );
 	} else 
-		return DAT_INVALID_ADDRESS;
+		return 1;
 
-	return DAT_SUCCESS;
+	return 0;
 }
-#endif
 
 /*
  * dapls_ib_init, dapls_ib_release
@@ -104,11 +152,15 @@
  */
 int32_t dapls_ib_init (void)
 {	
-	return 0;
+	if (dapli_cm_thread_init())
+		return -1;
+	else
+		return 0;
 }
 
 int32_t dapls_ib_release (void)
 {
+	dapli_cm_thread_destroy();
 	return 0;
 }
 
@@ -160,48 +212,50 @@
 
 	hca_ptr->ib_hca_handle = ibv_open_device(hca_ptr->ib_trans.ib_dev);
 	if (!hca_ptr->ib_hca_handle) {
-		 dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
+		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
 				" open_hca: IB dev open failed for %s\n", 
 				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
 		return DAT_INTERNAL_ERROR;
 	}
   
-#ifdef SOCKET_CM
-	/* initialize cr_list lock */
-	dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.lock);
-	if (dat_status != DAT_SUCCESS)
-	{
+	/* set inline max with enviromment or default, get local lid and gid 0 */
+	hca_ptr->ib_trans.max_inline_send = 
+		dapl_os_get_env_val ( "DAPL_MAX_INLINE", INLINE_SEND_DEFAULT );
+
+	if ( dapli_get_lid(hca_ptr, hca_ptr->port_num,
+			   &hca_ptr->ib_trans.lid )) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
-			" open_hca: failed to init lock\n");
-		return dat_status;
+				" open_hca: IB get LID failed for %s\n", 
+				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		return DAT_INTERNAL_ERROR;
 	}
 
-	/* initialize CM list for listens on this HCA */
-	dapl_llist_init_head(&hca_ptr->ib_trans.list);
-
-	/* create thread to process inbound connect request */
-	dat_status = dapl_os_thread_create(cr_thread, 
-					   (void*)hca_ptr, 
-					   &hca_ptr->ib_trans.thread );
-	if (dat_status != DAT_SUCCESS)
-	{
+	if ( dapli_get_gid(hca_ptr, hca_ptr->port_num, 0,  
+			   &hca_ptr->ib_trans.gid )) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
-		" open_hca: failed to create thread\n");
-		return dat_status;
+				" open_hca: IB get GID failed for %s\n", 
+				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		return DAT_INTERNAL_ERROR;
 	}
-
+	
 	/* get the IP address of the device */
-	dat_status = getipaddr((char*)&hca_ptr->hca_address, 
-				sizeof(DAT_SOCK_ADDR6) );
+	if ( dapli_get_addr((char*)&hca_ptr->hca_address, 
+			    sizeof(DAT_SOCK_ADDR6) )) {
+		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
+				" open_hca: IB get ADDR failed for %s\n", 
+				ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		return DAT_INTERNAL_ERROR;
+	}
+
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
-		" open_hca: %s, port %d, %s  %d.%d.%d.%d\n", 
+		" open_hca: %s, port %d, %s  %d.%d.%d.%d INLINE_MAX=%d\n", 
 		ibv_get_device_name(hca_ptr->ib_trans.ib_dev), hca_ptr->port_num,
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_family == AF_INET ?
"AF_INET":"AF_INET6",
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 0 & 0xff,
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 8 & 0xff,
 		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 16 & 0xff,
-		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff );
-#endif
+		((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff,
+		hca_ptr->ib_trans.max_inline_send );
 
 	return dat_status;
 }
@@ -232,20 +286,6 @@
 			return(dapl_convert_errno(errno,"ib_close_device"));
 		hca_ptr->ib_hca_handle = IB_INVALID_HANDLE;
 	}
-
-#if SOCKET_CM
-	/* destroy cr_thread and lock */
-	hca_ptr->ib_trans.destroy = 1;
-	while (hca_ptr->ib_trans.destroy) {
-		struct timespec	sleep, remain;
-		sleep.tv_sec = 0;
-		sleep.tv_nsec = 10000000; /* 10 ms */
-		dapl_dbg_log(DAPL_DBG_TYPE_UTIL, 
-			     " close_hca: waiting for cr_thread\n");
-		nanosleep (&sleep, &remain);
-	}
-	dapl_os_lock_destroy(&hca_ptr->ib_trans.lock);
-#endif
 	return (DAT_SUCCESS);
 }
   
@@ -407,3 +447,36 @@
     }
     return DAT_SUCCESS;
 }
+
+#ifdef PROVIDER_SPECIFIC_ATTR
+
+/*
+ * dapls_set_provider_specific_attr
+ *
+ * Input:
+ *	attr_ptr		Pointer provider attributes
+ *
+ * Output:
+ * 	none
+ *
+ * Returns:
+ * 	void
+ */
+DAT_NAMED_ATTR	ib_attrs[] = {
+    {
+    	"I_DAT_SEND_INLINE_THRESHOLD",
+    	"128"
+    },
+};
+
+#define SPEC_ATTR_SIZE( x )	(sizeof( x ) / sizeof( DAT_NAMED_ATTR))
+
+void dapls_set_provider_specific_attr(
+	IN DAT_PROVIDER_ATTR	*attr_ptr )
+{
+	attr_ptr->num_provider_specific_attr = SPEC_ATTR_SIZE( ib_attrs );
+	attr_ptr->provider_specific_attr     = ib_attrs;
+}
+
+#endif
+
Index: dapl/openib/dapl_ib_cm.c
===================================================================
--- dapl/openib/dapl_ib_cm.c	(revision 2818)
+++ dapl/openib/dapl_ib_cm.c	(working copy)
@@ -52,451 +52,687 @@
 #include "dapl_cr_util.h"
 #include "dapl_name_service.h"
 #include "dapl_ib_util.h"
+#include <sys/poll.h>
+#include <signal.h>
 
-#ifdef SOCKET_CM
+/* prototypes */
+static void dapli_path_comp_handler(uint64_t req_id, void *context, int rec_num);
+static void dapli_rt_comp_handler(uint64_t req_id, void *context, int rec_num);
+static void dapli_rep_recv(struct dapl_cm_id *conn, struct ib_cm_event *rep_recv_param);
+static struct dapl_cm_id * dapli_req_recv(struct dapl_cm_id *conn, struct ib_cm_event *event);
+static int dapli_cm_active_cb(struct dapl_cm_id *conn, struct ib_cm_event *event);
+static int dapli_cm_passive_cb(struct dapl_cm_id *conn, struct ib_cm_event *event);
+
+/* prototypes */
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+static inline uint64_t cpu_to_be64(uint64_t x) { return bswap_64(x); }
+#elif __BYTE_ORDER == __BIG_ENDIAN
+static inline uint64_t cpu_to_be64(uint64_t x) { return x; }
+#endif
+
+#ifndef IB_AT
 
 #include <stdio.h>
 #include <unistd.h>
 #include <fcntl.h>
 #include <netinet/tcp.h>
 #include <sysfs/libsysfs.h>
+#include <signal.h>
 
-/* prototypes */
-static uint16_t dapli_get_lid( struct ibv_device *dev, int port );
+/* iclust-20 hard coded values, network order */
+#define REMOTE_GID	"fe80:0000:0000:0000:0002:c902:0000:4071"
+#define REMOTE_LID	"0002"
+
+static int			g_cm_destroy;
+static DAPL_OS_THREAD		g_cm_thread;
+static DAPL_OS_LOCK		g_cm_lock;
+static struct dapl_llist_entry	*g_cm_list;	
+
+int dapli_cm_thread_init(void)
+{
+	DAT_RETURN dat_status;
 
-static DAT_RETURN dapli_socket_connect ( DAPL_EP		*ep_ptr,
-					 DAT_IA_ADDRESS_PTR	r_addr,
-					 DAT_CONN_QUAL		r_qual,
-					 DAT_COUNT		p_size,
-					 DAT_PVOID		p_data );
-
-static DAT_RETURN dapli_socket_listen ( DAPL_IA			*ia_ptr,
-					DAT_CONN_QUAL		serviceID,
-					DAPL_SP			*sp_ptr );
-
-static DAT_RETURN dapli_socket_accept(	ib_cm_srvc_handle_t cm_ptr );
-
-static DAT_RETURN dapli_socket_accept_final(	DAPL_EP		*ep_ptr,
-						DAPL_CR		*cr_ptr,
-						DAT_COUNT	p_size,
-						DAT_PVOID	p_data );
-
-/* XXX temporary hack to get lid */
-static uint16_t dapli_get_lid(IN struct ibv_device *dev, IN int port)
-{
-	char path[128];
-	char val[16];
-	char name[256];
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_init(%d)\n", getpid());
 
-	if (sysfs_get_mnt_path(path, sizeof path)) {
-		fprintf(stderr, "Couldn't find sysfs mount.\n");
-		return 0;
+        /* initialize cr_list lock */
+	dapl_os_lock_init(&g_cm_lock);
+	
+	/* initialize CM list for listens on this HCA */
+	dapl_llist_init_head(&g_cm_list);
+
+	/* create thread to process inbound connect request */
+	dat_status = dapl_os_thread_create(cm_thread, NULL, &g_cm_thread);
+	if (dat_status != DAT_SUCCESS)
+	{
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
+			     " cm_thread_init: failed to create thread\n");
+		return 1;
 	}
-	sprintf(name, "%s/class/infiniband/%s/ports/%d/lid", path,
-		 ibv_get_device_name(dev), port);
+	return 0;
+}
 
-	if (sysfs_read_attribute_value(name, val, sizeof val)) {
-		fprintf(stderr, "Couldn't read LID at %s\n", name);
-		return 0;
+void dapli_cm_thread_destroy(void)
+{
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d)\n", getpid());
+
+	/* destroy cr_thread and lock */
+	g_cm_destroy = 1;
+	pthread_kill( g_cm_thread, SIGUSR1 );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d) SIGUSR1 sent\n",getpid());
+	while (g_cm_destroy) {
+		struct timespec	sleep, remain;
+		sleep.tv_sec = 0;
+		sleep.tv_nsec = 200000000; /* 200 ms */
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " cm_thread_destroy: waiting for cm_thread\n");
+		nanosleep (&sleep, &remain);
 	}
-	return strtol(val, NULL, 0);
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d) exit\n",getpid());
 }
 
-/*
- * ACTIVE: Create socket, connect, and exchange QP information 
- */
-static DAT_RETURN 
-dapli_socket_connect (	DAPL_EP			*ep_ptr,
-			DAT_IA_ADDRESS_PTR	r_addr,
-			DAT_CONN_QUAL		r_qual,
-			DAT_COUNT		p_size,
-			DAT_PVOID		p_data )
-{
-	ib_cm_handle_t	cm_ptr;
-	DAPL_IA		*ia_ptr = ep_ptr->header.owner_ia;
-	int		len, opt = 1;
-	struct iovec    iovec[2];
-	short		rtu_data = htons(0x0E0F);
+static int ib_at_route_by_ip(uint32_t dst_ip, uint32_t src_ip, int tos, uint16_t flags,
+			     struct ib_at_ib_route *ib_route,
+			     struct ib_at_completion *async_comp)
+{
+	struct dapl_cm_id *conn = (struct dapl_cm_id *)async_comp->context;
 	
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d\n", r_qual);
-			
-	/*
-	 *  Allocate CM and initialize
-	 */
-	if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL ) {
-		return DAT_INSUFFICIENT_RESOURCES;
+	dapl_dbg_log (
+		DAPL_DBG_TYPE_CM, 
+		" CM at_route_by_ip: conn %p cm_id %d src %d.%d.%d.%d -> dst %d.%d.%d.%d (%d)\n", 
+		conn,conn->cm_id,
+		src_ip >> 0 & 0xff, src_ip >> 8 & 0xff,
+		src_ip >> 16 & 0xff,src_ip >> 24 & 0xff,
+		dst_ip >> 0 & 0xff, dst_ip >> 8 & 0xff,
+		dst_ip >> 16 & 0xff,dst_ip >> 24 & 0xff, conn->service_id);
+
+	/* use req_id for loopback indication */
+	if (( src_ip == dst_ip ) || ( dst_ip == 0x0100007f ))
+		async_comp->req_id = 1;
+	else
+		async_comp->req_id = 0;
+		
+	return 1;			     
+}
+
+static int ib_at_paths_by_route(struct ib_at_ib_route *ib_route, uint32_t mpath_type,
+				struct ib_sa_path_rec *pr, int npath,
+				struct ib_at_completion *async_comp)
+{
+	struct dapl_cm_id *conn = (struct dapl_cm_id *)async_comp->context;
+	char *env, *token;
+	char  dgid[40];
+	uint16_t *p_gid = (uint16_t*)&ib_route->gid;
+
+	/* set local path record values and send to remote */
+	(void)dapl_os_memzero(pr, sizeof(*pr));
+
+	pr->slid = htons(conn->hca->ib_trans.lid);
+	pr->sgid.global.subnet_prefix = conn->hca->ib_trans.gid.global.subnet_prefix;
+	pr->sgid.global.interface_id  = conn->hca->ib_trans.gid.global.interface_id;
+
+	env = getenv("DAPL_REMOTE_LID");
+	if ( env == NULL )
+		env = REMOTE_LID;
+	ib_route->lid = strtol(env,NULL,0);
+
+	env = getenv("DAPL_REMOTE_GID");
+	if ( env == NULL )
+		env = REMOTE_GID;
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " ib_at_paths_by_route: remote LID %x GID %s\n",
+		     ib_route->lid,env);
+
+	dapl_os_memcpy( dgid, env, 40 ); 
+
+	/* get GID with token strings and delimiter */
+	token = strtok(dgid,":");
+	while (token) {
+		*p_gid = strtoul(token,NULL,16);
+		*p_gid = htons(*p_gid); /* convert each token to network order */
+		token = strtok(NULL,":");
+		p_gid++;
+	}
+       
+	/* set remote lid and gid, req_id is indication of loopback */
+	if ( !async_comp->req_id ) {
+		pr->dlid = htons(ib_route->lid);
+		pr->dgid.global.subnet_prefix = ib_route->gid.global.subnet_prefix;
+		pr->dgid.global.interface_id  = ib_route->gid.global.interface_id;
+	} else {
+		pr->dlid = pr->slid;
+		pr->dgid.global.subnet_prefix = pr->sgid.global.subnet_prefix;
+		pr->dgid.global.interface_id  = pr->sgid.global.interface_id;
 	}
 
-	(void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) );
-	cm_ptr->socket = -1;
+	pr->reversible = 0x1000000;
+	pr->pkey = 0xffff;
+	pr->mtu  = IBV_MTU_1024;
+	pr->mtu_selector  = 2;
+	pr->rate_selector = 2;
+	pr->rate          = 3;
+	pr->packet_life_time_selector = 2;
+	pr->packet_life_time          = 2;
+	
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " ib_at_paths_by_route: SRC LID 0x%x GID subnet %016llx id %016llx\n",
+		     pr->slid,(unsigned long long)(pr->sgid.global.subnet_prefix),
+		     (unsigned long long)(pr->sgid.global.interface_id) );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " ib_at_paths_by_route: DST LID 0x%x GID subnet %016llx id %016llx\n",
+		     pr->dlid,(unsigned long long)(pr->dgid.global.subnet_prefix),
+		     (unsigned long long)(pr->dgid.global.interface_id) );
 
-	/* create, connect, sockopt, and exchange QP information */
-	if ((cm_ptr->socket = socket(AF_INET,SOCK_STREAM,0)) < 0 ) {
-		dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-		return DAT_INSUFFICIENT_RESOURCES;
-	}
+	dapli_path_comp_handler( async_comp->req_id, (void*)conn, 1);
 
-	((struct sockaddr_in*)r_addr)->sin_port = htons(r_qual);
+	return 0;
+}
 
-	if ( connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0 ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
-			     " connect: %s on r_qual %d\n",
-			     strerror(errno), (unsigned int)r_qual);
-		dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-		return DAT_INVALID_ADDRESS;
-	}
-	setsockopt(cm_ptr->socket,IPPROTO_TCP,TCP_NODELAY,&opt,sizeof(opt));
-	
-	/* Send QP info, IA address, and private data */
-	cm_ptr->dst.qpn = ep_ptr->qp_handle->qp_num;
-	cm_ptr->dst.port = ia_ptr->hca_ptr->port_num;
-	cm_ptr->dst.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, 
-					 ia_ptr->hca_ptr->port_num );
-	cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address;
-	cm_ptr->dst.p_size = p_size;
-	iovec[0].iov_base = &cm_ptr->dst;
-	iovec[0].iov_len  = sizeof(ib_qp_cm_t);
-	if ( p_size ) {
-		iovec[1].iov_base = p_data;
-		iovec[1].iov_len  = p_size;
-	}
-	len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) );
-    	if ( len != (p_size + sizeof(ib_qp_cm_t)) ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " connect write: ERR %s, wcnt=%d\n",
-			     strerror(errno), len); 
-		goto bail;
-	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		     " connect: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		     cm_ptr->dst.port, cm_ptr->dst.lid, 
-		     cm_ptr->dst.qpn, cm_ptr->dst.p_size ); 
-
-	/* read DST information into cm_ptr, overwrite SRC info */
-	len = readv( cm_ptr->socket, iovec, 1 );
-	if ( len != sizeof(ib_qp_cm_t) ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " connect read: ERR %s, rcnt=%d\n",
-			     strerror(errno), len); 
-		goto bail;
-	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		     " connect: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		     cm_ptr->dst.port, cm_ptr->dst.lid, 
-		     cm_ptr->dst.qpn, cm_ptr->dst.p_size ); 
+#endif /* ifndef IB_AT */
 
-	/* validate private data size before reading */
-	if ( cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " connect read: psize (%d) wrong\n",
-			     cm_ptr->dst.p_size ); 
-		goto bail;
-	}
+static void dapli_path_comp_handler(uint64_t req_id, void *context, int rec_num)
+{
+	struct dapl_cm_id *conn = context;
+	int status;
+	ib_cm_events_t event;
+
+	if (rec_num <= 0) {
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " path_comp_handler: resolution err %d retry %d\n",
+			     rec_num, conn->retries + 1);
+		if (++conn->retries > IB_MAX_AT_RETRY) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				     " path_comp_handler: ep_ptr 0x%p\n",conn->ep);
+			event = IB_CME_DESTINATION_UNREACHABLE;
+			goto bail;
+		}
 
-	/* read private data into cm_handle if any present */
-	if ( cm_ptr->dst.p_size ) {
-		iovec[0].iov_base = cm_ptr->p_data;
-		iovec[0].iov_len  = cm_ptr->dst.p_size;
-		len = readv( cm_ptr->socket, iovec, 1 );
-		if ( len != cm_ptr->dst.p_size ) {
-			dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-				" connect read pdata: ERR %s, rcnt=%d\n",
-				strerror(errno), len); 
+		status = ib_at_paths_by_route(&conn->dapl_rt, 0,
+					      &conn->dapl_path, 1,
+					      &conn->dapl_comp);
+		if (status) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				     " path_by_route: err %d id %lld\n",
+				     status, conn->dapl_comp.req_id);
+			event = IB_CME_LOCAL_FAILURE;
 			goto bail;
 		}
+		return;
 	}
 
-	/* modify QP to RTR and then to RTS with remote info */
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS )
-		goto bail;
-
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS )
+	/* Mellanox performance workaround - best performance is MTU of 1024 */
+	if (conn->dapl_path.mtu > IBV_MTU_1024)    
+		conn->dapl_path.mtu = IBV_MTU_1024;
+
+	conn->req.service_id = cpu_to_be64(conn->service_id);
+	conn->req.primary_path = &conn->dapl_path;
+	conn->req.alternate_path = NULL;
+
+	status = ib_cm_send_req(conn->cm_id, &conn->req);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_send_cm_req failed: %s\n",
+			     strerror(errno));
+		event = IB_CME_LOCAL_FAILURE;
 		goto bail;
-		 
-	ep_ptr->qp_state = IB_QP_STATE_RTS;
-
-	/* complete handshake after final QP state change */
-	write(cm_ptr->socket, &rtu_data, sizeof(rtu_data) );
-
-	/* init cm_handle and post the event with private data */
-	ep_ptr->cm_handle = cm_ptr;
-	dapl_dbg_log( DAPL_DBG_TYPE_EP," ACTIVE: connected!\n" ); 
-	dapl_evd_connection_callback(   ep_ptr->cm_handle, 
-					IB_CME_CONNECTED, 
-					cm_ptr->p_data, 
-					ep_ptr );	
-	return DAT_SUCCESS;
+	}
+	return;
 
 bail:
-	/* close socket, free cm structure and post error event */
-	if ( cm_ptr->socket >= 0 ) 
-		close(cm_ptr->socket);
-	dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */
-
-	dapl_evd_connection_callback(	ep_ptr->cm_handle, 
-					IB_CME_LOCAL_FAILURE, 
-					NULL, 
-					ep_ptr );
-	return DAT_INTERNAL_ERROR;
+	dapl_evd_connection_callback(conn, event, NULL, conn->ep);
 }
 
+static void dapli_rt_comp_handler(uint64_t req_id, void *context, int rec_num)
+{
+	struct dapl_cm_id *conn = context;
+	int status;
+	ib_cm_events_t event;
 
-/*
- * PASSIVE: Create socket, listen, accept, exchange QP information 
- */
-static DAT_RETURN 
-dapli_socket_listen (	DAPL_IA		*ia_ptr,
-			DAT_CONN_QUAL	serviceID,
-			DAPL_SP		*sp_ptr )
-{
-	struct sockaddr_in	addr;
-	ib_cm_srvc_handle_t	cm_ptr = NULL;
-	int			opt = 1;
-	DAT_RETURN		dat_status = DAT_SUCCESS;
+	if (rec_num <= 0) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " dapl_rt_comp_handler: rec %d retry %d\n",
+			     rec_num, conn->retries+1 ); 
 
-	dapl_dbg_log (	DAPL_DBG_TYPE_EP,
-			" listen(ia_ptr %p ServiceID %d sp_ptr %p)\n",
-			ia_ptr, serviceID, sp_ptr);
+		if (++conn->retries > IB_MAX_AT_RETRY) {
+			event = IB_CME_DESTINATION_UNREACHABLE;
+			goto bail;
+		}
 
-	/* Allocate CM and initialize */
-	if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) 
-		return DAT_INSUFFICIENT_RESOURCES;
+		status = ib_at_route_by_ip(((struct sockaddr_in *)&conn->r_addr)->sin_addr.s_addr,
+					   0, 0, 0, &conn->dapl_rt, &conn->dapl_comp);
+		if (status < 0) {
+			dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_rt_comp_handler: "
+				    "ib_at_route_by_ip failed with status %d\n",
+				    status);
+			event = IB_CME_DESTINATION_UNREACHABLE;
+			goto bail;
+		}
 
-	(void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) );
-	
-	cm_ptr->socket = cm_ptr->l_socket = -1;
-	cm_ptr->sp = sp_ptr;
-	cm_ptr->hca_ptr = ia_ptr->hca_ptr;
-	
-	/* bind, listen, set sockopt, accept, exchange data */
-	if ((cm_ptr->l_socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
-		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
-				"socket for listen returned %d\n", errno);
-		dat_status = DAT_INSUFFICIENT_RESOURCES;
-		goto bail;
+		if (status == 1)
+			dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, 1);
+		return;
 	}
 
-	setsockopt(cm_ptr->l_socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt));
-	addr.sin_port        = htons(serviceID);
-	addr.sin_family      = AF_INET;
-	addr.sin_addr.s_addr = INADDR_ANY;
-
-	if (( bind( cm_ptr->l_socket,(struct sockaddr*)&addr, sizeof(addr) ) < 0) ||
-		   (listen( cm_ptr->l_socket, 128 ) < 0) ) {
-	
-		dapl_dbg_log( DAPL_DBG_TYPE_ERR,
-				" listen: ERROR %s on conn_qual 0x%x\n",
-				strerror(errno),serviceID); 
-
-		if ( errno == EADDRINUSE )
-			dat_status = DAT_CONN_QUAL_IN_USE;
-		else
-			dat_status = DAT_CONN_QUAL_UNAVAILABLE;
-
+	conn->dapl_comp.fn = &dapli_path_comp_handler;
+	conn->dapl_comp.context = conn;
+	conn->retries = 0;
+	status = ib_at_paths_by_route(&conn->dapl_rt, 0, &conn->dapl_path, 1,
+				      &conn->dapl_comp);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     "dapl_rt_comp_handler: ib_at_paths_by_route "
+			     "returned %d id %lld\n", status, conn->dapl_comp.req_id);
+		event = IB_CME_LOCAL_FAILURE;
 		goto bail;
 	}
-	
-	/* set cm_handle for this service point, save listen socket */
-	sp_ptr->cm_srvc_handle = cm_ptr;
+	return;
 
-	/* add to SP->CR thread list */
-	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry);
-	dapl_os_lock( &cm_ptr->hca_ptr->ib_trans.lock );
-	dapl_llist_add_tail(&cm_ptr->hca_ptr->ib_trans.list, 
-			    (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr);
-	dapl_os_unlock(&cm_ptr->hca_ptr->ib_trans.lock);
-
-	dapl_dbg_log( DAPL_DBG_TYPE_CM,
-			" listen: qual 0x%x cr %p s_fd %d\n",
-			ntohs(serviceID), cm_ptr, cm_ptr->l_socket ); 
-	
-	return dat_status;
 bail:
-	dapl_dbg_log( DAPL_DBG_TYPE_ERR,
-			" listen: ERROR on conn_qual 0x%x\n",serviceID); 
-	if ( cm_ptr->l_socket >= 0 )
-		close( cm_ptr->l_socket );
-	dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	return dat_status;
+	dapl_evd_connection_callback(conn, event, NULL, conn->ep);
 }
 
-
-/*
- * PASSIVE: send local QP information, private data, and wait for 
- *	    active side to respond with QP RTS/RTR status 
- */
-static DAT_RETURN 
-dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr)
+static void dapli_destroy_cm_id(struct dapl_cm_id *conn)
 {
-	ib_cm_handle_t	acm_ptr;
-	void		*p_data = NULL;
-	int		len;
-	DAT_RETURN	dat_status = DAT_SUCCESS;
-		
-	/* Allocate accept CM and initialize */
-	if ((acm_ptr = dapl_os_alloc(sizeof(*acm_ptr))) == NULL) 
-		return DAT_INSUFFICIENT_RESOURCES;
+	int in_callback;
 
-	(void) dapl_os_memzero( acm_ptr, sizeof( *acm_ptr ) );
-	
-	acm_ptr->socket = -1;
-	acm_ptr->sp = cm_ptr->sp;
-	acm_ptr->hca_ptr = cm_ptr->hca_ptr;
-
-	len = sizeof(acm_ptr->dst.ia_address);
-	acm_ptr->socket = accept(cm_ptr->l_socket, 
-				(struct sockaddr*)&acm_ptr->dst.ia_address, 
-				&len );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " destroy_cm_id: conn %p id %d\n",conn,conn->cm_id);
+
+	dapl_os_lock(&conn->lock);
+	conn->destroy = 1;
+	in_callback = conn->in_callback;
+	dapl_os_unlock(&conn->lock);
+
+	if (!in_callback) {
+		ib_cm_destroy_id(conn->cm_id);
+		if (conn->ep)
+			conn->ep->cm_handle = IB_INVALID_HANDLE;
+		if (conn->sp)
+			conn->sp->cm_srvc_handle = IB_INVALID_HANDLE;
+
+		/* take off the CM thread work queue and free */
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_remove_entry(&g_cm_list, 
+					(DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_unlock(&g_cm_lock);
+
+		dapl_os_free(conn, sizeof(*conn));
+	}
+}
 
-	if ( acm_ptr->socket < 0 ) {
+static void dapli_rep_recv(struct dapl_cm_id  *conn,
+			   struct ib_cm_event *event)
+{
+	int status;
+
+	if (conn->ep->qp_handle == IB_INVALID_HANDLE) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			" accept: ERR %s on FD %d l_cr %p\n",
-			strerror(errno),cm_ptr->l_socket,cm_ptr); 
-		dat_status = DAT_INTERNAL_ERROR;
-		goto bail;
-   	}
+			     " dapli_rep_recv: invalid qp "
+			     "handle\n");
+		goto disc;
+	}
+
+	/* move QP state to RTR and RTS */
+	/* TODO: could use a ib_cm_init_qp_attr() call here */
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " rep_recv: RTR_RTS: cm_id %d r_qp 0x%x r_lid 0x%x r_SID %d\n",
+		     conn->cm_id,event->param.rep_rcvd.remote_qpn,
+		     ntohs(conn->req.primary_path->dlid),conn->service_id );
+
+	if ( dapls_modify_qp_state( conn->ep->qp_handle, 
+				    IBV_QPS_RTR, 
+				    event->param.rep_rcvd.remote_qpn,
+				    ntohs(conn->req.primary_path->dlid),
+				    1 ) != DAT_SUCCESS )
+		goto disc;
+
+	if ( dapls_modify_qp_state( conn->ep->qp_handle, 
+				    IBV_QPS_RTS,0,0,0 ) != DAT_SUCCESS)
+		goto disc; 
+
 
-	/* read in DST QP info, IA address. check for private data */
-	len = read( acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t) );
-	if ( len != sizeof(ib_qp_cm_t) ) {
+	status = ib_cm_send_rtu(conn->cm_id, NULL, 0);
+	if (status) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			" accept read: ERR %s, rcnt=%d\n",
-			strerror(errno), len); 
-		dat_status = DAT_INTERNAL_ERROR;
-		goto bail;
+			     " dapli_rep_recv: ib_send_cm_rtu "
+			     "failed: %d\n", status);
+		goto disc;
+	}
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+     		     " rep_recv: conn %p id %d CONNECTED!\n",
+     		     conn, conn->cm_id );
+
+	dapl_evd_connection_callback(conn, IB_CME_CONNECTED,
+				     event->private_data, conn->ep);
+	return;
+
+disc:
+	dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL,
+				     conn->ep);
+}
 
-	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		" accept: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		acm_ptr->dst.port, acm_ptr->dst.lid, 
-		acm_ptr->dst.qpn, acm_ptr->dst.p_size ); 
+static struct dapl_cm_id * dapli_req_recv(struct dapl_cm_id  *conn,
+					  struct ib_cm_event *event)
+{
+	struct dapl_cm_id 	*new_conn;
 
-	/* validate private data size before reading */
-	if ( acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE ) {
+	if (conn->sp == NULL) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			" accept read: psize (%d) wrong\n",
-			acm_ptr->dst.p_size ); 
-		dat_status = DAT_INTERNAL_ERROR;
-		goto bail;
+			     " dapli_rep_recv: on invalid listen "
+			     "handle\n");
+		return NULL;
+	}
+
+	/* allocate new cm_id and merge listen parameters */
+	new_conn = dapl_os_alloc(sizeof(*new_conn)); 
+	if (new_conn) {
+
+		(void)dapl_os_memzero(new_conn, sizeof(*new_conn));
+		dapl_os_lock_init(&new_conn->lock);
+		new_conn->cm_id = event->cm_id; /* provided by uCM */
+		new_conn->sp = conn->sp;
+		new_conn->hca = conn->hca;
+		new_conn->service_id = conn->service_id;
+
+		/* save request information in new conn */
+		dapl_os_memcpy(&new_conn->req_rcvd, 
+				&event->param.req_rcvd,
+				sizeof(struct ib_cm_req_event_param));
+
+		new_conn->req_rcvd.primary_path = &new_conn->dapl_path;
+		new_conn->req_rcvd.alternate_path = NULL;
+
+		dapl_os_memcpy(new_conn->req_rcvd.primary_path,
+				event->param.req_rcvd.primary_path,
+				sizeof(struct ib_sa_path_rec));
+			 	
+		/* put new CR on CM thread event work queue */
+		dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&new_conn->entry);
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_add_tail(&g_cm_list, 
+				(DAPL_LLIST_ENTRY*)&new_conn->entry, new_conn);
+		dapl_os_unlock(&g_cm_lock);
+
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, " passive_cb: "
+				"REQ on HCA %p SP %p SID %d L_ID %d new_id %d p_data %p\n",
+				new_conn->hca, new_conn->sp, 
+				conn->service_id, conn->cm_id, new_conn->cm_id, 
+				event->private_data );
+
 	}
+	return new_conn;
+}
 
-	/* read private data into cm_handle if any present */
-	if ( acm_ptr->dst.p_size ) {
-		len = read( acm_ptr->socket, 
-			    acm_ptr->p_data, acm_ptr->dst.p_size );
-		if ( len != acm_ptr->dst.p_size ) {
-			dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-				" accept read pdata: ERR %s, rcnt=%d\n",
-				strerror(errno), len ); 
-			dat_status = DAT_INTERNAL_ERROR;
-			goto bail;
-		}
-		dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-				" accept: psize=%d read\n",
-				acm_ptr->dst.p_size); 
-		p_data = acm_ptr->p_data;
+
+static int dapli_cm_active_cb(struct dapl_cm_id *conn,
+			       struct ib_cm_event *event)
+{
+	int destroy;
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " active_cb: conn %p id %d event %d\n",
+		     conn, conn->cm_id, event->event );
+
+	dapl_os_lock(&conn->lock);
+	if (conn->destroy) {
+		dapl_os_unlock(&conn->lock);
+		return 0;
 	}
-	
-	/* trigger CR event and return SUCCESS */
-	dapls_cr_callback(  acm_ptr,
-			    IB_CME_CONNECTION_REQUEST_PENDING,
-		            p_data,
-			    acm_ptr->sp );
+	conn->in_callback = 1;
+	dapl_os_unlock(&conn->lock);
 
-	return DAT_SUCCESS;
+	switch (event->event) {
+	case IB_CM_REQ_ERROR:
+		dapl_evd_connection_callback(conn,
+					     IB_CME_DESTINATION_UNREACHABLE,
+					     NULL, conn->ep);
+		break;
+	case IB_CM_REJ_RECEIVED:
+		dapl_evd_connection_callback(conn, IB_CME_DESTINATION_REJECT,
+					     NULL, conn->ep);
+		break;
+	case IB_CM_REP_RECEIVED:
+		dapli_rep_recv(conn, event);
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		ib_cm_send_drep(conn->cm_id, NULL, 0);
+		break;
+	case IB_CM_DREQ_ERROR:
+	case IB_CM_DREP_RECEIVED:
+		/* Wait to exit timewait. */
+		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		dapl_evd_connection_callback(conn, IB_CME_DISCONNECTED,
+					     NULL, conn->ep);
+		break;
+	default:
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " dapli_cm_active_cb_handler: Unexpected CM "
+			     "event %d on ID 0x%p\n", event->event, conn->cm_id);
+		break;
+	}
 
-bail:
-	if ( acm_ptr->socket >=0 )
-		close( acm_ptr->socket );
-	dapl_os_free( acm_ptr, sizeof( *acm_ptr ) );
-	return DAT_INTERNAL_ERROR;
+	dapl_os_lock(&conn->lock);
+	destroy = conn->destroy;
+	conn->in_callback = conn->destroy;
+	dapl_os_unlock(&conn->lock);
+	if (destroy) {
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " active_cb: DESTROY conn %p id %d \n",
+			     conn, conn->cm_id );
+		if (conn->ep)
+			conn->ep->cm_handle = IB_INVALID_HANDLE;
+		
+		/* take off the CM thread work queue and free */
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_remove_entry(&g_cm_list, 
+					(DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_unlock(&g_cm_lock);
+		dapl_os_free(conn, sizeof(*conn));
+	}
+	return(destroy);
 }
 
+static int dapli_cm_passive_cb(struct dapl_cm_id *conn,
+			       	struct ib_cm_event *event)
+{
+	int destroy;
+	struct dapl_cm_id *new_conn;
 
-static DAT_RETURN 
-dapli_socket_accept_final( DAPL_EP		*ep_ptr,
-			   DAPL_CR		*cr_ptr,
-			   DAT_COUNT		p_size,
-		           DAT_PVOID		p_data )
-{
-	DAPL_IA		*ia_ptr = ep_ptr->header.owner_ia;
-	ib_cm_handle_t	cm_ptr = cr_ptr->ib_cm_handle;
-	ib_qp_cm_t	qp_cm;
-	struct iovec    iovec[2];
-	int		len;
-	short		rtu_data = 0;
-
-	if (p_size >  IB_MAX_REP_PDATA_SIZE) 
-		return DAT_LENGTH_ERROR;
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " passive_cb: conn %p id %d event %d\n",
+		     conn, conn->cm_id, event->event );
 
-	/* must have a accepted socket */
-	if ( cm_ptr->socket < 0 )
-		return DAT_INTERNAL_ERROR;
-	
-	/* modify QP to RTR and then to RTS with remote info already read */
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS )
-		goto bail;
+	if (conn->cm_id == 0)
+		return 0;
 
-	if ( dapls_modify_qp_state( ep_ptr->qp_handle, 
-				    IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS )
-		goto bail;
+	dapl_os_lock(&conn->lock);
+	if (conn->destroy) {
+		dapl_os_unlock(&conn->lock);
+		return 0;
+	}
+	conn->in_callback = 1;
+	dapl_os_unlock(&conn->lock);
 
-	ep_ptr->qp_state = IB_QP_STATE_RTS;
-	
-	/* Send QP info, IA address, and private data */
-	qp_cm.qpn = ep_ptr->qp_handle->qp_num;
-	qp_cm.port = ia_ptr->hca_ptr->port_num;
-	qp_cm.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, 
-				   ia_ptr->hca_ptr->port_num );
-	qp_cm.ia_address = ia_ptr->hca_ptr->hca_address;
-	qp_cm.p_size = p_size;
-	iovec[0].iov_base = &qp_cm;
-	iovec[0].iov_len  = sizeof(ib_qp_cm_t);
-	if (p_size) {
-		iovec[1].iov_base = p_data;
-		iovec[1].iov_len  = p_size;
+	switch (event->event) {
+	case IB_CM_REQ_RECEIVED:
+		/* create new conn object with new conn_id from event */
+		new_conn = dapli_req_recv(conn,event);
+
+		if (new_conn)	
+			dapls_cr_callback(new_conn, IB_CME_CONNECTION_REQUEST_PENDING, 
+				 	  event->private_data, new_conn->sp);
+		break;
+	case IB_CM_REP_ERROR:
+		dapls_cr_callback(conn, IB_CME_DESTINATION_UNREACHABLE,
+				 NULL, conn->sp);
+		break;
+	case IB_CM_REJ_RECEIVED:
+		dapls_cr_callback(conn, IB_CME_DESTINATION_REJECT, NULL,
+				 conn->sp);
+		break;
+	case IB_CM_RTU_RECEIVED:
+		/* move QP to RTS state */
+		if ( dapls_modify_qp_state(conn->ep->qp_handle, 
+					   IBV_QPS_RTS,0,0,0 ) != DAT_SUCCESS) {
+			dapls_cr_callback(conn, IB_CME_LOCAL_FAILURE, 
+					  NULL, conn->sp);
+		} else {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     		     " passive_cb: conn %p id %d CONNECTED!\n",
+		     		     conn, conn->cm_id );
+			dapls_cr_callback(conn, IB_CME_CONNECTED, 
+					  NULL, conn->sp);
+		}
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		ib_cm_send_drep(conn->cm_id, NULL, 0);
+		break;
+	case IB_CM_DREQ_ERROR:
+	case IB_CM_DREP_RECEIVED:
+		/* Wait to exit timewait. */
+		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		dapls_cr_callback(conn, IB_CME_DISCONNECTED, NULL, conn->sp);
+		break;
+	default:
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_cm_passive_cb_handler: "
+			     "Unexpected CM event %d on ID 0x%p\n",
+			     event->event, conn->cm_id);
+		break;
 	}
-	len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) );
-    	if (len != (p_size + sizeof(ib_qp_cm_t))) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " accept_final: ERR %s, wcnt=%d\n",
-			     strerror(errno), len); 
-		goto bail;
+
+	dapl_os_lock(&conn->lock);
+	destroy = conn->destroy;
+	conn->in_callback = conn->destroy;
+	dapl_os_unlock(&conn->lock);
+	if (destroy) {
+		if (conn->ep)
+			conn->ep->cm_handle = IB_INVALID_HANDLE;
+
+		/* take off the CM thread work queue and free */
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_remove_entry(&g_cm_list, 
+					(DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_unlock(&g_cm_lock);
+
+		dapl_os_free(conn, sizeof(*conn));
 	}
-	dapl_dbg_log(DAPL_DBG_TYPE_EP, 
-		     " accept_final: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n",
-		     qp_cm.port, qp_cm.lid, qp_cm.qpn, qp_cm.p_size ); 
+	return(destroy);
+}
+
+/* something to catch the signal */
+static void cm_handler(int signum)
+{
+	dapl_dbg_log (DAPL_DBG_TYPE_CM," cm_thread(%d,0x%x): ENTER cm_handler %d\n",
+			getpid(),g_cm_thread,signum);
+	return;
+}
+
+/* async CM processing thread */
+void cm_thread(void *arg) 
+{
+	struct dapl_cm_id	*conn, *next_conn;
+	struct ib_cm_event	*event;
+	struct pollfd           ufds;
+	sigset_t		sigset;
+
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " cm_thread(%d,0x%x): ENTER: cm_fd %d\n",
+		      getpid(), g_cm_thread, ib_cm_get_fd());
+
+    	sigemptyset(&sigset);
+	sigaddset(&sigset, SIGUSR1);
+    	pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
+	signal( SIGUSR1, cm_handler); 
 	
-	/* complete handshake after final QP state change */
-	len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data) );
-	if ( len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f ) {
-		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
-			     " accept_final: ERR %s, rcnt=%d rdata=%x\n",
-			     strerror(errno), len, ntohs(rtu_data) ); 
-		goto bail;
-	}
+	dapl_os_lock( &g_cm_lock );
+	while (!g_cm_destroy) {
+		int cm_id,ret;
+
+		/* select for CM event, all events process via cm_fd */
+                ufds.fd      = ib_cm_get_fd();
+                ufds.events  = POLLIN;
+                ufds.revents = 0;
+
+		dapl_os_unlock(&g_cm_lock);
+                ret = poll(&ufds, 1, -1); 
+		if ((ret <= 0) || (g_cm_destroy)) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				     " cm_thread(%d): ERR %s poll\n",
+				     getpid(),strerror(errno));
+                	dapl_os_lock(&g_cm_lock);
+			break;
+		}
 
-	/* final data exchange if remote QP state is good to go */
-	dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: connected!\n" ); 
-	dapls_cr_callback ( cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp );
-	return DAT_SUCCESS;
+		dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			" cm_thread: GET EVENT fd=%d n=%d\n",
+			ib_cm_get_fd(),ret);
+		if (ib_cm_event_get(&event)) { 
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				" cm_thread: ERR %s eventi_get on %d\n", 
+				strerror(errno), ib_cm_get_fd() );
+			dapl_os_lock(&g_cm_lock);
+			continue;
+		}
+		dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			" cm_thread: GET EVENT fd=%d woke\n",ib_cm_get_fd());
+		dapl_os_lock(&g_cm_lock);
+
+		/* set proper cm_id */
+		if (event->event == IB_CM_REQ_RECEIVED ||
+			event->event == IB_CM_SIDR_REQ_RECEIVED)
+			cm_id = event->param.req_rcvd.listen_id;
+		else
+			cm_id = event->cm_id;
 
-bail:
-	dapl_dbg_log( DAPL_DBG_TYPE_ERR," accept_final: ERR !QP_RTR_RTS \n"); 
-	if ( cm_ptr >= 0 )
-		close( cm_ptr->socket );
-	dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */
+		dapl_dbg_log (DAPL_DBG_TYPE_CM,
+			" cm_thread: EVENT event(%d) cm_id=%d (%d)\n",
+			event->event, event->cm_id, cm_id );
 
-	return DAT_INTERNAL_ERROR;
-}
+		/* 
+		 * Walk cm_list looking for connection id in event
+		 * no need to walk if uCM would provide context with event
+		 */
+		if (!dapl_llist_is_empty(&g_cm_list))
+			next_conn = dapl_llist_peek_head(&g_cm_list);
+		else
+			next_conn = NULL;
 
-#endif
+		ret = 0;
+		while (next_conn) {
+			conn = next_conn;
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			     " cm_thread: LIST cm %p c_id %d e_id %d)\n", 
+			     conn, conn->cm_id, cm_id );
+				
+			next_conn = dapl_llist_next_entry(
+					&g_cm_list,
+					(DAPL_LLIST_ENTRY*)&conn->entry );
+
+			if (cm_id == conn->cm_id) {
+				dapl_os_unlock(&g_cm_lock);
+				if (conn->sp)
+					ret = dapli_cm_passive_cb(conn,event);
+				else
+					ret = dapli_cm_active_cb(conn,event);
+				dapl_os_lock(&g_cm_lock);
+				break;
+			}
+		} 
+		ib_cm_event_put(event);
+		if (ret) {
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+			     " cm_thread: destroy cm_id %d\n",cm_id);
+			ib_cm_destroy_id(cm_id);
+		}
+	}
+	dapl_os_unlock(&g_cm_lock);	
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread(%d) EXIT, cm_list=%s\n",
+		     getpid(),dapl_llist_is_empty(&g_cm_list) ? "EMPTY":"NOT EMPTY");
+	g_cm_destroy = 0;
+}
 
+/************************ DAPL provider entry points **********************/
 
 /*
  * dapls_ib_connect
@@ -522,16 +758,17 @@
 DAT_RETURN
 dapls_ib_connect (
 	IN  DAT_EP_HANDLE		ep_handle,
-	IN  DAT_IA_ADDRESS_PTR		remote_ia_address,
-	IN  DAT_CONN_QUAL		remote_conn_qual,
-	IN  DAT_COUNT			private_data_size,
-	IN  void			*private_data )
+	IN  DAT_IA_ADDRESS_PTR		r_addr,
+	IN  DAT_CONN_QUAL		r_qual,
+	IN  DAT_COUNT			p_size,
+	IN  void			*p_data )
 {
 	DAPL_EP		*ep_ptr;
 	ib_qp_handle_t	qp_ptr;
+	int		status;
+	DAT_RETURN	dat_status = DAT_INTERNAL_ERROR;
+	ib_cm_handle_t	conn;
 	
-	dapl_dbg_log ( DAPL_DBG_TYPE_EP,
-			" connect(ep_handle %p ....)\n", ep_handle);
 	/*
 	 *  Sanity check
 	 */
@@ -541,13 +778,80 @@
 	ep_ptr = (DAPL_EP*)ep_handle;
 	qp_ptr = ep_ptr->qp_handle;
 
-#ifdef SOCKET_CM
-	return (dapli_socket_connect(	ep_ptr, remote_ia_address, 
-					remote_conn_qual,
-					private_data_size, private_data ));
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, " connect: r_SID %d, pdata %p, plen %d\n", 
+		     r_qual,p_data,p_size);
+			
+	/* Allocate CM and initialize lock */
+	if ((conn = dapl_os_alloc(sizeof(*conn))) == NULL) 
+		return DAT_INSUFFICIENT_RESOURCES;
+	
+	(void)dapl_os_memzero(conn, sizeof(*conn));
+
+	dat_status = dapl_os_lock_init(&conn->lock);
+	if (dat_status != DAT_SUCCESS) { 
+		dapl_os_free(conn, sizeof(*conn));
+		return DAT_INTERNAL_ERROR;
+	}
+
+	conn->ep = ep_ptr;
+	conn->hca = ep_ptr->header.owner_ia->hca_ptr;
+	status = ib_cm_create_id(&conn->cm_id);
+	if (status < 0)  {
+		dat_status = dapl_convert_errno(errno,"create_cm_id");
+		dapl_os_free(conn, sizeof(*conn));
+		return dat_status;
+	}
+	conn->ep->cm_handle = conn;
+
+	/* Setup QP/CM parameters */
+	(void)dapl_os_memzero(&conn->req,sizeof(conn->req));
+	conn->service_id = r_qual;
+	conn->req.qp_num = ep_ptr->qp_handle->qp_num;
+	conn->req.qp_type = IBV_QPT_RC;
+	conn->req.starting_psn = 1;
+	conn->req.private_data = p_data;
+	conn->req.private_data_len = p_size;
+	conn->req.peer_to_peer = 0;
+	conn->req.responder_resources = IB_TARGET_MAX;
+	conn->req.initiator_depth = IB_INITIATOR_DEPTH;
+	conn->req.remote_cm_response_timeout = IB_CM_RESPONSE_TIMEOUT;
+	conn->req.flow_control = 1;
+	conn->req.local_cm_response_timeout = IB_CM_RESPONSE_TIMEOUT;
+	conn->req.retry_count = IB_RC_RETRY_COUNT;
+	conn->req.rnr_retry_count = IB_RNR_RETRY_COUNT;
+	conn->req.max_cm_retries = IB_MAX_CM_RETRIES;
+	conn->req.srq = 0;
+	
+	conn->dapl_comp.fn = &dapli_rt_comp_handler;
+	conn->dapl_comp.context = conn;
+	conn->retries = 0;
+	dapl_os_memcpy(&conn->r_addr, r_addr, sizeof(DAT_SOCK_ADDR6));
+
+	status = ib_at_route_by_ip(
+		((struct sockaddr_in *)&conn->r_addr)->sin_addr.s_addr, 
+		((struct sockaddr_in *)&conn->hca->hca_address)->sin_addr.s_addr, 
+		0, 0, &conn->dapl_rt, &conn->dapl_comp);
+
+	if (status < 0) {
+		dat_status = dapl_convert_errno(errno,"ib_at_route_by_ip");
+		goto destroy;
+	}
+	if (status == 1)
+		dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, 1);
+
+	
+	/* put on CM thread work queue */
+	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
+	dapl_os_lock( &g_cm_lock );
+	dapl_llist_add_tail(&g_cm_list, 
+			    (DAPL_LLIST_ENTRY*)&conn->entry, conn);
+	dapl_os_unlock(&g_cm_lock);
+
+	return DAT_SUCCESS;
+
+destroy:
+	dapli_destroy_cm_id(conn); 
+	return dat_status;
 
 }
 
@@ -572,35 +876,26 @@
 	IN	DAPL_EP			*ep_ptr,
 	IN	DAT_CLOSE_FLAGS		close_flags )
 {
-	ib_cm_handle_t	cm_ptr = ep_ptr->cm_handle;
+	ib_cm_handle_t	conn = ep_ptr->cm_handle;
+	int status;
 
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			"dapls_ib_disconnect(ep_handle %p ....)\n",
-			ep_ptr);
-#ifdef SOCKET_CM
-
-	if ( cm_ptr->socket >= 0 ) {
-		close( cm_ptr->socket );
-		cm_ptr->socket = -1;
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+			" disconnect(ep_handle %p, conn %p, cm_id %d flags %x)\n",
+			ep_ptr,conn, (conn?conn->cm_id:0),close_flags);
+
+	if (conn == IB_INVALID_HANDLE)
+		return DAT_SUCCESS;
+
+	if (close_flags == DAT_CLOSE_ABRUPT_FLAG)
+		dapli_destroy_cm_id(conn);
+	else {
+		status = ib_cm_send_dreq(conn->cm_id, NULL, 0);
+		if (status)
+			dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+				     "dapl_ib_disconnect: ID %p status %d\n", 
+				     ep_ptr->cm_handle, status);
 	}
-	
-	/* reinit to modify QP state */
-	dapls_ib_reinit_ep(ep_ptr);
 
-#endif
-	if ( ep_ptr->cr_ptr ) {
-		dapls_cr_callback ( ep_ptr->cm_handle,
-				    IB_CME_DISCONNECTED,
-				    NULL,
-				    ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr );
-	} else {
-		dapl_evd_connection_callback ( ep_ptr->cm_handle,
-						IB_CME_DISCONNECTED,
-						NULL,
-						ep_ptr );
-		ep_ptr->cm_handle = NULL;
-		dapl_os_free( cm_ptr, sizeof( *cm_ptr ) );
-	}	
 	return DAT_SUCCESS;
 }
 
@@ -628,7 +923,14 @@
 	IN  DAT_BOOLEAN			active,
 	IN  const ib_cm_events_t	ib_cm_event )
 {
-    return;
+
+	/*
+	 * Clean up outstanding connection state
+	 */
+	dapls_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG);
+         
+	if (ep_ptr->qp_handle != IB_INVALID_HANDLE) 
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_ERR, 0,0,0);
 }
 
 /*
@@ -657,11 +959,63 @@
 	IN  DAT_UINT64		ServiceID,
 	IN  DAPL_SP		*sp_ptr )
 {
-#ifdef SOCKET_CM
-	return (dapli_socket_listen( ia_ptr, ServiceID, sp_ptr ));
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
+	DAT_RETURN		dat_status = DAT_SUCCESS;
+	int			status;
+	ib_cm_srvc_handle_t     conn;
+
+
+	/* Allocate CM and initialize lock */
+	if ((conn = dapl_os_alloc(sizeof(*conn))) == NULL) 
+		return DAT_INSUFFICIENT_RESOURCES;
+	
+	(void)dapl_os_memzero(conn, sizeof(*conn));
+
+	dat_status = dapl_os_lock_init(&conn->lock);
+	if (dat_status != DAT_SUCCESS)  {
+		dapl_os_free(conn, sizeof(*conn));
+		return DAT_INTERNAL_ERROR;
+	}
+	 
+	status = ib_cm_create_id(&conn->cm_id);
+	if (status < 0)  {
+		dat_status = dapl_convert_errno(errno,"create_cm_id");
+		dapl_os_free(conn, sizeof(*conn));
+		return dat_status;
+	}
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM,
+		     " setup_listener(ia_ptr %p SID %d sp_ptr %p conn %p cm_id %d)\n",
+		     ia_ptr, ServiceID, sp_ptr, conn, conn->cm_id );
+
+	sp_ptr->cm_srvc_handle = conn;
+	conn->sp = sp_ptr;
+	conn->hca = ia_ptr->hca_ptr;
+	conn->service_id = ServiceID;
+
+	dapl_dbg_log(DAPL_DBG_TYPE_EP,
+		     " setup_listener(conn=%p cm_id=%d)\n",
+		     sp_ptr->cm_srvc_handle,conn->cm_id);
+
+	status = ib_cm_listen(conn->cm_id,
+			      cpu_to_be64(ServiceID), 0);
+	if (status) {
+		if (status == -EBUSY)
+			dat_status = DAT_CONN_QUAL_IN_USE;
+		else
+			dat_status = DAT_INSUFFICIENT_RESOURCES;
+	/* success */ 
+	} else  { 
+		/* put on CM thread work queue */
+		dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
+		dapl_os_lock( &g_cm_lock );
+		dapl_llist_add_tail(&g_cm_list, 
+				(DAPL_LLIST_ENTRY*)&conn->entry, conn);
+		dapl_os_unlock(&g_cm_lock);
+		return dat_status;
+	}
+
+        dapli_destroy_cm_id(conn);
+	return dat_status;
 }
 
 
@@ -687,27 +1041,16 @@
 	IN  DAPL_IA		*ia_ptr,
 	IN  DAPL_SP		*sp_ptr )
 {
-	ib_cm_srvc_handle_t	cm_ptr = sp_ptr->cm_srvc_handle;
+	ib_cm_srvc_handle_t	conn = sp_ptr->cm_srvc_handle;
 
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			"dapls_ib_remove_conn_listener(ia_ptr %p sp_ptr %p cm_ptr %p)\n",
-			ia_ptr, sp_ptr, cm_ptr );
-#ifdef SOCKET_CM
-	/* close accepted socket, free cm_srvc_handle and return */
-	if ( cm_ptr != NULL ) {
-		if ( cm_ptr->l_socket >= 0 ) {
-			close( cm_ptr->l_socket );
-			cm_ptr->socket = -1;
-		}
-	    	/* cr_thread will free */
-		sp_ptr->cm_srvc_handle = NULL;
-	}
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+			" remove_listener(ia_ptr %p sp_ptr %p cm_ptr %p)\n",
+			ia_ptr, sp_ptr, conn );
+	
+	if (sp_ptr->cm_srvc_handle != IB_INVALID_HANDLE) 
+        	dapli_destroy_cm_id(conn);
+	
 	return DAT_SUCCESS;
-#else
-	return DAT_NOT_IMPLEMENTED;   
-
-#endif
-
 }
 
 /*
@@ -737,30 +1080,84 @@
 	IN  DAT_COUNT		p_size,
 	IN  const DAT_PVOID	p_data )
 {
-	DAPL_CR			*cr_ptr;
-	DAPL_EP			*ep_ptr;
-	
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-		      "dapls_ib_accept_connection(cr %p ep %p prd %p,%d)\n",
-		      cr_handle, ep_handle, p_data, p_size  );
+	DAPL_CR *cr_ptr;
+	DAPL_EP *ep_ptr;
+	DAPL_IA *ia_ptr;
+	DAT_RETURN dat_status;
+	int status;
+	struct ib_cm_rep_param passive_params;
+	struct dapl_cm_id *conn;
+
+	cr_ptr = (DAPL_CR *) cr_handle;
+	ep_ptr = (DAPL_EP *) ep_handle;
+	ia_ptr = ep_ptr->header.owner_ia;
+	conn = cr_ptr->ib_cm_handle;
+
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " accept_connection(cr %p conn %p, cm_id %d, p_data %p, p_sz=%d)\n",
+		      cr_ptr, conn, conn->cm_id, p_data, p_size );
+
+	/* Obtain size of private data structure & contents */
+	if (p_size > IB_MAX_REP_PDATA_SIZE) {
+		dat_status = DAT_ERROR(DAT_LENGTH_ERROR, DAT_NO_SUBTYPE);
+		goto reject;
+	}
+
+	if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) {
+		/* 
+		 * If we are lazy attaching the QP then we may need to
+		 * hook it up here. Typically, we run this code only for
+		 * DAT_PSP_PROVIDER_FLAG
+		 */
+		dat_status = dapls_ib_qp_alloc(ia_ptr, ep_ptr, NULL);
+		if (dat_status != DAT_SUCCESS) {
+			dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+				     " accept: ib_qp_alloc failed: %d\n",
+				     dat_status);
+			goto reject;
+		}
+	}
 
-	cr_ptr  = (DAPL_CR *) cr_handle;
-	ep_ptr  = (DAPL_EP *) ep_handle;
-	
-	/* allocate and attach a QP if necessary */
-	if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) {
-		DAT_RETURN status;
-		status = dapls_ib_qp_alloc( ep_ptr->header.owner_ia, 
-					    ep_ptr, ep_ptr );
-		if ( status != DAT_SUCCESS )
-    			return status;
-	}
-    
-#ifdef SOCKET_CM
-	return ( dapli_socket_accept_final(ep_ptr, cr_ptr, p_size, p_data) );
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
+	/* move QP to RTR state, TODO fix port setting */
+	/* TODO: could use a ib_cm_init_qp_attr() call here */
+	dat_status = dapls_modify_qp_state(ep_ptr->qp_handle, 
+					   IBV_QPS_RTR, 
+					   conn->req_rcvd.remote_qpn, 
+					   ntohs(conn->req_rcvd.primary_path->dlid),
+					   1 ); 
+	if (dat_status != DAT_SUCCESS ) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+				     " accept: modify_qp_state failed: %d\n",
+				     dat_status);
+		goto reject;
+	}
+
+	cr_ptr->param.local_ep_handle = ep_handle;
+	ep_ptr->cm_handle = conn;
+	conn->ep = ep_ptr;
+
+	memset(&passive_params, 0, sizeof(passive_params));
+	passive_params.private_data = p_data;
+	passive_params.private_data_len = p_size;
+	passive_params.qp_num = ep_ptr->qp_handle->qp_num;
+	passive_params.responder_resources = IB_TARGET_MAX;
+	passive_params.initiator_depth = IB_INITIATOR_DEPTH;
+	passive_params.rnr_retry_count = IB_RNR_RETRY_COUNT;
+
+	status = ib_cm_send_rep(conn->cm_id, &passive_params);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " accept: "
+			     "ib_send_cm_rep failed: %d\n", status);
+		dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0);
+		goto reject;
+	}
+	return DAT_SUCCESS;
+
+reject:
+	ib_cm_send_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0,
+		       NULL, 0);
+	dapli_destroy_cm_id(conn);
+	return dat_status;
    
 }
 
@@ -783,27 +1180,31 @@
  */
 DAT_RETURN
 dapls_ib_reject_connection (
-	IN  ib_cm_handle_t	ib_cm_handle,
+	IN  ib_cm_handle_t	cm_handle,
 	IN  int			reject_reason )
 {
-    	ib_cm_srvc_handle_t	cm_ptr = ib_cm_handle;
+    	int status;
 
-	dapl_dbg_log (DAPL_DBG_TYPE_EP,
-		      "dapls_ib_reject_connection(cm_handle %p reason %x)\n",
-		      ib_cm_handle, reject_reason );
-#ifdef SOCKET_CM
-
-	/* just close the socket and return */
-	if ( cm_ptr->socket > 0 ) {
-		close( cm_ptr->socket );
-		cm_ptr->socket = -1;
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " reject_connection(cm_handle %p reason %x)\n",
+		      cm_handle, reject_reason );
+
+	if (cm_handle == IB_INVALID_HANDLE) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " reject_conn: invalid handle: reason %d\n",
+			     reject_reason);
+		return DAT_SUCCESS;
 	}
-	return DAT_SUCCESS;
-#else
-	return DAT_NOT_IMPLEMENTED;   
-#endif
-	
 
+	status = ib_cm_send_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED,
+				NULL, 0, NULL, 0);
+	if (status) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
+			     " reject_conn: cm_rej failed: %d\n", status);
+		return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0);
+	}
+	dapli_destroy_cm_id(cm_handle);
+	return DAT_SUCCESS;
 }
 
 /*
@@ -844,7 +1245,7 @@
 		return DAT_INVALID_HANDLE;
 
 	dapl_os_memcpy(	remote_ia_address, 
-			&ib_cm_handle->dst.ia_address, 
+			&ib_cm_handle->r_addr, 
 			sizeof(DAT_SOCK_ADDR6) );
 
 	return DAT_SUCCESS;
@@ -914,12 +1315,12 @@
 	return size;
 }
 
-#ifdef SOCKET_CM
+#ifndef SOCKET_CM
 
 /*
  * Map all socket CM event codes to the DAT equivelent.
  */
-#define DAPL_IB_EVENT_CNT	11
+#define DAPL_IB_EVENT_CNT	12
 
 static struct ib_cm_event_map
 {
@@ -936,17 +1337,19 @@
 					DAT_CONNECTION_REQUEST_EVENT},
 	/* 04 */  { IB_CME_CONNECTION_REQUEST_PENDING_PRIVATE_DATA,
 					DAT_CONNECTION_REQUEST_EVENT},
-	/* 05 */  { IB_CME_DESTINATION_REJECT,
+	/* 05 */  { IB_CME_CONNECTION_REQUEST_ACKED,
+					DAT_CONNECTION_REQUEST_EVENT},
+	/* 06 */  { IB_CME_DESTINATION_REJECT,
 					DAT_CONNECTION_EVENT_NON_PEER_REJECTED},
-	/* 06 */  { IB_CME_DESTINATION_REJECT_PRIVATE_DATA,		
+	/* 07 */  { IB_CME_DESTINATION_REJECT_PRIVATE_DATA,		
 					DAT_CONNECTION_EVENT_PEER_REJECTED},
-	/* 07 */  { IB_CME_DESTINATION_UNREACHABLE,	
+	/* 08 */  { IB_CME_DESTINATION_UNREACHABLE,	
 					DAT_CONNECTION_EVENT_UNREACHABLE},
-	/* 08 */  { IB_CME_TOO_MANY_CONNECTION_REQUESTS,
+	/* 09 */  { IB_CME_TOO_MANY_CONNECTION_REQUESTS,
 					DAT_CONNECTION_EVENT_NON_PEER_REJECTED},
-	/* 09 */  { IB_CME_LOCAL_FAILURE,
+	/* 10 */  { IB_CME_LOCAL_FAILURE,
 					DAT_CONNECTION_EVENT_BROKEN},
-	/* 10 */  { IB_CM_LOCAL_FAILURE,
+	/* 11 */  { IB_CME_BROKEN,
 					DAT_CONNECTION_EVENT_BROKEN}
 };
  
@@ -974,7 +1377,7 @@
 	
 	active = active;
 
-	if (ib_cm_event > IB_CM_LOCAL_FAILURE)
+	if (ib_cm_event > IB_CME_BROKEN)
 		return (DAT_EVENT_NUMBER) 0;
 
 	dat_event_num = 0;
@@ -1024,96 +1427,6 @@
     return ib_cm_event;
 }
 
-/* async CR processing thread to avoid blocking applications */
-void cr_thread(void *arg) 
-{
-    struct dapl_hca	*hca_ptr = arg;
-    ib_cm_srvc_handle_t	cr, next_cr;
-    int			max_fd;
-    fd_set		rfd,rfds;
-    struct timeval	to;
-     
-    dapl_os_lock( &hca_ptr->ib_trans.lock );
-    while ( !hca_ptr->ib_trans.destroy ) {
-	
-	FD_ZERO( &rfds ); 
-	max_fd = -1;
-	
-	if (!dapl_llist_is_empty(&hca_ptr->ib_trans.list))
-            next_cr = dapl_llist_peek_head (&hca_ptr->ib_trans.list);
-	else
-	    next_cr = NULL;
-
-	while (next_cr) {
-	    cr = next_cr;
-	    dapl_dbg_log (DAPL_DBG_TYPE_CM," thread: cm_ptr %p\n", cr );
-	    if (cr->l_socket == -1 || hca_ptr->ib_trans.destroy) {
-
-		dapl_dbg_log(DAPL_DBG_TYPE_CM," thread: Freeing %p\n", cr);
-		next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list,
-						(DAPL_LLIST_ENTRY*)&cr->entry );
-		dapl_llist_remove_entry(&hca_ptr->ib_trans.list, 
-					(DAPL_LLIST_ENTRY*)&cr->entry);
-		dapl_os_free( cr, sizeof(*cr) );
-		continue;
-	    }
-	          
-	    FD_SET( cr->l_socket, &rfds ); /* add to select set */
-	    if ( cr->l_socket > max_fd )
-		max_fd = cr->l_socket;
-
-	    /* individual select poll to check for work */
-	    FD_ZERO(&rfd);
-	    FD_SET(cr->l_socket, &rfd);
-	    dapl_os_unlock(&hca_ptr->ib_trans.lock);	
-	    to.tv_sec  = 0;
-	    to.tv_usec = 0;
-	    if ( select(cr->l_socket + 1,&rfd, NULL, NULL, &to) < 0) {
-		dapl_dbg_log (DAPL_DBG_TYPE_CM,
-			  " thread: ERR %s on cr %p sk %d\n", 
-			  strerror(errno), cr, cr->l_socket);
-		close(cr->l_socket);
-		cr->l_socket = -1;
-	    } else if ( FD_ISSET(cr->l_socket, &rfd) && 
-			dapli_socket_accept(cr)) {
-		close(cr->l_socket);
-		cr->l_socket = -1;
-	    }
-	    dapl_os_lock( &hca_ptr->ib_trans.lock );
-	    next_cr =  dapl_llist_next_entry(&hca_ptr->ib_trans.list,
-					     (DAPL_LLIST_ENTRY*)&cr->entry );
-	} 
-	dapl_os_unlock( &hca_ptr->ib_trans.lock );
-	to.tv_sec  = 0;
-	to.tv_usec = 500000; /* wakeup and check destroy */
-	select(max_fd + 1, &rfds, NULL, NULL, &to);
-	dapl_os_lock( &hca_ptr->ib_trans.lock );
-    } 
-    dapl_os_unlock( &hca_ptr->ib_trans.lock );	
-    hca_ptr->ib_trans.destroy = 0;
-    dapl_dbg_log(DAPL_DBG_TYPE_CM," thread(hca %p) exit\n",hca_ptr);
-}
-
-/* Real IBv CM */
-#else 
-
-DAT_EVENT_NUMBER
-dapls_ib_get_dat_event (
-	IN    const ib_cm_events_t	ib_cm_event,
-	IN    DAT_BOOLEAN		active)
-{
-	return ib_cm_event;
-}
-
-ib_cm_events_t
-dapls_ib_get_cm_event (
-	IN    DAT_EVENT_NUMBER		dat_event_num )
-{
-	return dat_event_num;
-}
-
-
-
 #endif
 
 /*
Index: dapl/openib/dapl_ib_qp.c
===================================================================
--- dapl/openib/dapl_ib_qp.c	(revision 2818)
+++ dapl/openib/dapl_ib_qp.c	(working copy)
@@ -107,6 +107,8 @@
 	qp_create.cap.max_recv_wr = attr->max_recv_dtos;
 	qp_create.cap.max_send_sge = attr->max_request_iov;
 	qp_create.cap.max_recv_sge = attr->max_recv_iov;
+	/* TODO: extend ep_attr so user can set */
+	qp_create.cap.max_inline_data = ia_ptr->hca_ptr->ib_trans.max_inline_send; 
 	qp_create.qp_type = IBV_QPT_RC;
 	qp_create.qp_context = (void*)ep_ptr;
 
@@ -122,8 +124,7 @@
 
 	/* Setup QP attributes for INIT state on the way out */ 
 	if (dapls_modify_qp_state(ep_ptr->qp_handle,
-				  IBV_QPS_INIT,
-				  NULL )  != DAT_SUCCESS ) {
+				  IBV_QPS_INIT,0,0,0 ) != DAT_SUCCESS ) {
 		ibv_destroy_qp(ep_ptr->qp_handle);		
 		ep_ptr->qp_handle = IB_INVALID_HANDLE;
 		return DAT_INTERNAL_ERROR;
@@ -160,7 +161,7 @@
 
 	if (ep_ptr->qp_handle != IB_INVALID_HANDLE) {
 		/* force error state to flush queue, then destroy */
-		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_ERR, NULL);
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_ERR, 0,0,0);
 		
 		if (ibv_destroy_qp(ep_ptr->qp_handle)) 
 			return(dapl_convert_errno(errno,"destroy_qp"));
@@ -216,7 +217,7 @@
 	    (ep_ptr->qp_handle->state != IBV_QPS_ERR)) {
 		ep_ptr->qp_state = IB_QP_STATE_ERROR;
 		return (dapls_modify_qp_state(ep_ptr->qp_handle, 
-					      IBV_QPS_ERR, NULL));
+					      IBV_QPS_ERR,0,0,0));
 	}
 
 	/*
@@ -271,8 +272,8 @@
 	
 	if ( ep_ptr->qp_handle != IB_INVALID_HANDLE ) {
 		/* move to RESET state and then to INIT */
-		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_RESET, NULL);
-		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_INIT, NULL );
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_RESET, 0,0,0);
+		dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_INIT, 0,0,0);
 		ep_ptr->qp_state = IB_QP_STATE_INIT;
 	}
 
@@ -287,7 +288,9 @@
 DAT_RETURN
 dapls_modify_qp_state ( IN ib_qp_handle_t	qp_handle,
 			IN ib_qp_state_t	qp_state,
-			IN ib_qp_cm_t		*qp_cm )
+			IN uint32_t		qpn,
+			IN uint16_t		lid,
+			IN uint8_t		port )
 {
 	struct ibv_qp_attr	qp_attr;
 	enum ibv_qp_attr_mask	mask = IBV_QP_STATE;
@@ -299,7 +302,6 @@
 		/* additional attributes with RTR and RTS */
 		case IBV_QPS_RTR:
 		{
-#ifdef SOCKET_CM
 			mask |= IBV_QP_AV                 |
 				IBV_QP_PATH_MTU           |
 				IBV_QP_DEST_QPN           |
@@ -308,25 +310,24 @@
 				IBV_QP_MIN_RNR_TIMER;
 			qp_attr.qp_state 		= IBV_QPS_RTR;
 			qp_attr.path_mtu 		= IBV_MTU_1024;
-			qp_attr.dest_qp_num 		= qp_cm->qpn;
+			qp_attr.dest_qp_num 		= qpn;
 			qp_attr.rq_psn 			= 1;
-			qp_attr.max_dest_rd_atomic	= 8;
+			qp_attr.max_dest_rd_atomic	= IB_TARGET_MAX;
 			qp_attr.min_rnr_timer		= 12;
 			qp_attr.ah_attr.is_global	= 0;
-			qp_attr.ah_attr.dlid		= qp_cm->lid;
+			qp_attr.ah_attr.dlid		= lid;
 			qp_attr.ah_attr.sl		= 0;
 			qp_attr.ah_attr.src_path_bits	= 0;
-			qp_attr.ah_attr.port_num	= qp_cm->port;
+			qp_attr.ah_attr.port_num	= port;
 
 			dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			      " modify_qp_rtr: qpn %x lid %x port %x\n",
-			      qp_cm->qpn,qp_cm->lid,qp_cm->port );
+			      " modify_qp_RTR: qpn %x lid %x port %x, rq_psn %x\n",
+			      qpn,lid,port,ntohl(qp_attr.rq_psn) );
 			break;
-#endif
+
 		}		
 		case IBV_QPS_RTS: 
 		{
-#ifdef SOCKET_CM
 			mask |= IBV_QP_TIMEOUT            |
 				IBV_QP_RETRY_CNT          |
 				IBV_QP_RNR_RETRY          |
@@ -337,12 +338,11 @@
 			qp_attr.retry_cnt	= 7;
 			qp_attr.rnr_retry	= 7;
 			qp_attr.sq_psn		= 1;
-			qp_attr.max_rd_atomic	= 8;
+			qp_attr.max_rd_atomic	= IB_TARGET_MAX;
 			dapl_dbg_log (DAPL_DBG_TYPE_EP,
-			      " modify_qp_rts: psn %x or %x\n",
-			      qp_attr.sq_psn, qp_attr.max_rd_atomic );
+			      " modify_qp_RTS: psn %x or %x\n",
+			      ntohl(qp_attr.sq_psn), qp_attr.max_rd_atomic );
 			break;
-#endif
 		}
 		case IBV_QPS_INIT: 
 		{
@@ -368,7 +368,7 @@
 					IBV_ACCESS_REMOTE_ATOMIC;
 			
 			dapl_dbg_log (DAPL_DBG_TYPE_EP,
-				" modify_qp_init: pi %x port %x acc %x\n",
+				" modify_qp_INIT: pi %x port %x acc %x\n",
 				qp_attr.pkey_index, qp_attr.port_num,
 				qp_attr.qp_access_flags );
 			break;
Index: dapl/openib/README
===================================================================
--- dapl/openib/README	(revision 2818)
+++ dapl/openib/README	(working copy)
@@ -6,10 +6,12 @@
 - CQ_WAIT_OBJECT support added
 - added dapl/openib directory 
 - modify doc/dat.conf to add a example openib configuration 
+- fixes to wait and resize
 
 	dapl/common/dapl_adapter_util.h 
 	dapl/common/dapl_evd_dto_callb.c
 	dapl/common/dapl_evd_util.c 
+	dapl/common/dapl_evd_resize.c
 	dapl/include/dapl.h 
 	dapl/udapl/dapl_evd_set_unwaitable.c
 	dapl/udapl/dapl_evd_wait.c
@@ -29,8 +31,7 @@
 	dapl/openib/dapl_ib_qp.c
 	dapl/openib/dapl_ib_util.c
 	dapl/openib/dapl_ib_util.h
-	dapl/openib/dapl_ib_cm.c
-
+	
 A simple dapl test just for initial openIB testing...
 
 	test/dtest/dtest.c
@@ -38,10 +39,17 @@
 
 	server:	dtest -s 
 	client:	dtest -h hostname
+	
+setup/known issues:
+	
+	First drop with uCM (without IBAT), tested with simple dtest across 2 nodes. 
+	hand rolled path records require remote LID and GID set via enviroment:
+	
+	export DAPL_REMOTE_GID	"fe80:0000:0000:0000:0002:c902:0000:4071"
+	export DAPL_REMOTE_LID	"0002"
 
-known issues:
-
-	early drop, only tested with simple dtest and dapltest SR.
+    Also, hard coded (RTR) for use with port 1 only.
+	   
 	no memory windows support in ibverbs, dat_create_rmr fails.
 	
 
Index: dapl/openib/dapl_ib_util.h
===================================================================
--- dapl/openib/dapl_ib_util.h	(revision 2818)
+++ dapl/openib/dapl_ib_util.h	(working copy)
@@ -51,6 +51,8 @@
 
 #include "verbs.h"
 #include <byteswap.h>
+#include <infiniband/sa.h>
+#include <infiniband/cm.h>
 
 /* Typedefs to map common DAPL provider types to IB verbs */
 typedef	struct ibv_qp		*ib_qp_handle_t;
@@ -64,20 +66,46 @@
 typedef	struct ibv_context	*ib_hca_handle_t;
 typedef ib_hca_handle_t		dapl_ibal_ca_t;
 
-/* CM mappings, user CM not complete use SOCKETS */
+#define IB_RC_RETRY_COUNT      7
+#define IB_RNR_RETRY_COUNT     7
+#define IB_CM_RESPONSE_TIMEOUT 20	/* 4 sec */
+#define IB_MAX_CM_RETRIES      4
+
+#define IB_REQ_MRA_TIMEOUT	27	/* a little over 9 minutes */
+#define IB_MAX_AT_RETRY		3
+
+#define IB_TARGET_MAX		4	/* max_qp_ous_rd_atom */
+#define IB_INITIATOR_DEPTH	4	/* max_qp_init_rd_atom */
+
+typedef enum {
+	IB_CME_CONNECTED,
+	IB_CME_DISCONNECTED,
+	IB_CME_DISCONNECTED_ON_LINK_DOWN,
+	IB_CME_CONNECTION_REQUEST_PENDING,
+	IB_CME_CONNECTION_REQUEST_PENDING_PRIVATE_DATA,
+	IB_CME_CONNECTION_REQUEST_ACKED,
+	IB_CME_DESTINATION_REJECT,
+	IB_CME_DESTINATION_REJECT_PRIVATE_DATA,
+	IB_CME_DESTINATION_UNREACHABLE,
+	IB_CME_TOO_MANY_CONNECTION_REQUESTS,
+	IB_CME_LOCAL_FAILURE,
+	IB_CME_BROKEN
+} ib_cm_events_t;
 
-#ifdef SOCKET_CM
+#ifndef IB_AT
+/* implement a quick hack to exchange GID/LID's until user IB_AT arrives */
+struct ib_at_ib_route {
+        union ibv_gid   gid;
+        uint16_t        lid;
+};
 
-/* destination info to exchange until real IB CM shows up */
-typedef struct _ib_qp_cm
-{ 
-	uint32_t		qpn;
-	uint16_t		lid;
-	uint16_t		port;
-	int			p_size;
-	DAT_SOCK_ADDR6		ia_address;
+struct ib_at_completion {
+        void (*fn)(uint64_t req_id, void *context, int rec_num);
+        void *context;
+        uint64_t req_id;
+};
 
-} ib_qp_cm_t;
+#endif
 
 /* 
  * dapl_llist_entry in dapl.h but dapl.h depends on provider 
@@ -91,52 +119,27 @@
     struct dapl_llist_entry	*list_head;
 };
 
-struct ib_cm_handle
-{ 
-	struct ib_llist_entry	entry;
-	int			socket;
-	int			l_socket; 
-	struct dapl_hca		*hca_ptr;
-	DAT_HANDLE		cr;
-	DAT_HANDLE		sp;	
-	ib_qp_cm_t		dst;
-	unsigned char		p_data[256];
+struct dapl_cm_id {
+	struct ib_llist_entry		entry;
+	DAPL_OS_LOCK			lock;
+	int				retries;
+	int				destroy;
+	int				in_callback;
+	uint32_t			cm_id;
+	DAT_SOCK_ADDR6			r_addr; 
+	DAT_CONN_QUAL			service_id;
+	struct dapl_hca			*hca;
+	struct dapl_sp			*sp;
+	struct dapl_ep			*ep;
+	struct ib_at_ib_route		dapl_rt;
+	struct ib_sa_path_rec		dapl_path;
+	struct ib_cm_req_param		req;
+	struct ib_cm_req_event_param	req_rcvd;
+	struct ib_at_completion		dapl_comp;
 };
 
-typedef struct ib_cm_handle	*ib_cm_handle_t;
-typedef ib_cm_handle_t		ib_cm_srvc_handle_t;
-
-DAT_RETURN getipaddr(char *addr, int addr_len);
-
-/* CM events */
-typedef enum 
-{
-    IB_CME_CONNECTED,
-    IB_CME_DISCONNECTED,
-    IB_CME_DISCONNECTED_ON_LINK_DOWN,
-    IB_CME_CONNECTION_REQUEST_PENDING,
-    IB_CME_CONNECTION_REQUEST_PENDING_PRIVATE_DATA,
-    IB_CME_DESTINATION_REJECT,
-    IB_CME_DESTINATION_REJECT_PRIVATE_DATA,
-    IB_CME_DESTINATION_UNREACHABLE,
-    IB_CME_TOO_MANY_CONNECTION_REQUESTS,
-    IB_CME_LOCAL_FAILURE,
-    IB_CM_LOCAL_FAILURE
-
-} ib_cm_events_t;
-
-/* prototype for cm thread */
-void cr_thread (void *arg);
-
-#else
-
-/* TODO: Waiting for IB CM to define */
-typedef uint32_t		*ib_cm_handle_t;
-typedef uint32_t		*ib_cm_srvc_handle_t;
-typedef uint32_t		ib_qp_cm_t;
-typedef uint32_t		ib_cm_events_t;
-
-#endif
+typedef struct dapl_cm_id	*ib_cm_handle_t;
+typedef struct dapl_cm_id	*ib_cm_srvc_handle_t;
 
 /* Operation and state mappings */
 typedef enum	ibv_send_flags	ib_send_op_type_t;
@@ -167,6 +170,9 @@
 /* Definitions */
 #define IB_INVALID_HANDLE	NULL
 
+/* inline send rdma threshold */
+#define	INLINE_SEND_DEFAULT	128
+
 /* CM private data areas */
 #define	IB_MAX_REQ_PDATA_SIZE	92
 #define	IB_MAX_REP_PDATA_SIZE	196
@@ -174,7 +180,7 @@
 #define	IB_MAX_DREQ_PDATA_SIZE	220
 #define	IB_MAX_DREP_PDATA_SIZE	224
 
-/* DTO OPs, ordered for DAPL ENUM definitions ???*/
+/* DTO OPs, ordered for DAPL ENUM definitions */
 #define OP_RDMA_WRITE           IBV_WR_RDMA_WRITE
 #define OP_RDMA_WRITE_IMM       IBV_WR_RDMA_WRITE_WITH_IMM
 #define OP_SEND                 IBV_WR_SEND
@@ -232,13 +238,9 @@
 { 
 	struct	ibv_device	*ib_dev;
 	ib_cq_handle_t		ib_cq_empty;
-
-#if SOCKET_CM
-	int			destroy;
-	DAPL_OS_THREAD		thread;
-	DAPL_OS_LOCK		lock;	
-	struct dapl_llist_entry	*list;	
-#endif
+	int			max_inline_send;
+	uint16_t		lid;
+	union ibv_gid		gid;
 	ib_async_handler_t	async_unafiliated;
 	ib_async_handler_t	async_cq_error;
 	ib_async_handler_t	async_cq;
@@ -252,11 +254,21 @@
 /* prototypes */
 int32_t	dapls_ib_init (void);
 int32_t	dapls_ib_release (void);
+void cm_thread (void *arg);
+int dapli_cm_thread_init(void);
+void dapli_cm_thread_destroy(void);
+
+int dapli_get_lid(struct dapl_hca *hca_ptr, int port, uint16_t *lid );
+int dapli_get_gid(struct dapl_hca *hca_ptr, int port, int index, 
+		  union ibv_gid *gid );
+int dapli_get_addr(char *addr, int addr_len);
 
 DAT_RETURN
 dapls_modify_qp_state ( IN ib_qp_handle_t	qp_handle,
 			IN ib_qp_state_t	qp_state,
-			IN ib_qp_cm_t		*qp_cm );
+			IN uint32_t		qpn,
+			IN uint16_t		lid,
+			IN uint8_t		port );
 
 /* inline functions */
 STATIC _INLINE_ IB_HCA_NAME dapl_ib_convert_name (IN char *name)
Index: dapl/openib/dapl_ib_cq.c
===================================================================
--- dapl/openib/dapl_ib_cq.c	(revision 2818)
+++ dapl/openib/dapl_ib_cq.c	(working copy)
@@ -50,6 +50,7 @@
 #include "dapl_adapter_util.h"
 #include "dapl_lmr_util.h"
 #include "dapl_evd_util.h"
+#include <sys/poll.h>
 
 
 /*
@@ -448,36 +449,28 @@
 	ib_cq_handle_t		cq = evd_ptr->ib_cq_handle;
 	struct ibv_cq		*ibv_cq;
 	void			*ibv_ctx;
-	int			status = EINVAL; /* invalid handle */
+	int			status = -ETIMEDOUT; 
 
 	dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, 
 			" cq_object_wait: dev %p evd %p cq %p, time %d\n", 
 			cq->context, evd_ptr, cq, timeout );
 
-	/* TODO: add timeout, map each CQ created?? */
 	/* Multiple EVD's sharing one event handle for now */
-	while (evd_ptr->ib_cq_handle) {
+	if (cq) {
+		struct pollfd cq_poll = { 
+        		.fd      = cq->context->cq_fd[0],
+			.events  = POLLIN
+		};
+		int	timeout_ms = -1;
+
+		if (timeout != DAT_TIMEOUT_INFINITE)
+			timeout_ms = timeout/1000;
 		
-		status = ibv_get_cq_event(cq->context, 
-					  0, &ibv_cq, &ibv_ctx);
-		if (status) 
-			break;
-
-		/* EVD mismatch, process DTO callback for this EVD */
-		if (ibv_cq != cq) {
-			ib_hca_tranport_t *hca_ptr = 
-				&evd_ptr->header.owner_ia->hca_ptr->ib_trans;
-							
-			if ( hca_ptr->async_cq ) 
-				hca_ptr->async_cq(cq->context,
-						  (ib_error_record_t*)ibv_cq,
-						  ibv_ctx);
-			
-			continue;
-		} 
-		break;
+		status = poll(&cq_poll, 1, timeout_ms);
+		if (status == 1)
+			status = ibv_get_cq_event(cq->context, 
+						  0, &ibv_cq, &ibv_ctx);
 	}
-	
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
 		      " cq_object_wait: RET cq %p ibv_cq %p ibv_ctx %p %x\n",
 		      cq,ibv_cq,ibv_ctx,status);
Index: dtest/dtest.c
===================================================================
--- dtest/dtest.c	(revision 0)
+++ dtest/dtest.c	(revision 0)
@@ -0,0 +1,1700 @@
+/*
+ * Copyright (c) 2005 Intel Corporation.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: $
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <netdb.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <arpa/inet.h>
+#include <sys/mman.h>
+#include <getopt.h>
+
+#ifndef DAPL_PROVIDER
+#define DAPL_PROVIDER "OpenIB1_2"
+#endif
+
+#define MAX_POLLING_CNT 50000
+
+/* Header files needed for DAT/uDAPL */
+#include    "dat/udat.h"
+
+/* definitions */
+#define SERVER_CONN_QUAL    71123
+#define DTO_TIMEOUT	    (1000*1000*5)
+#define DTO_FLUSH_TIMEOUT   (1000*1000*2)
+#define CONN_TIMEOUT        (1000*1000*10)
+#define SERVER_TIMEOUT      (1000*1000*20)
+#define RDMA_BUFFER_SIZE    (64)
+
+/* Global DAT vars */
+static DAT_IA_HANDLE      h_ia = DAT_HANDLE_NULL;
+static DAT_PZ_HANDLE      h_pz = DAT_HANDLE_NULL;
+static DAT_EP_HANDLE      h_ep = DAT_HANDLE_NULL;
+static DAT_PSP_HANDLE     h_psp = DAT_HANDLE_NULL;
+static DAT_CR_HANDLE      h_cr = DAT_HANDLE_NULL;
+
+static DAT_EVD_HANDLE     h_async_evd = DAT_HANDLE_NULL;
+static DAT_EVD_HANDLE     h_dto_evd = DAT_HANDLE_NULL;
+static DAT_EVD_HANDLE     h_cr_evd = DAT_HANDLE_NULL;
+static DAT_EVD_HANDLE     h_conn_evd = DAT_HANDLE_NULL;
+static DAT_CNO_HANDLE     h_dto_cno = DAT_HANDLE_NULL;
+
+/* RDMA buffers */
+static DAT_LMR_HANDLE     h_lmr_send = DAT_HANDLE_NULL;
+static DAT_LMR_HANDLE     h_lmr_recv = DAT_HANDLE_NULL;
+static DAT_LMR_CONTEXT    lmr_context_send;
+static DAT_LMR_CONTEXT    lmr_context_recv;
+static DAT_RMR_CONTEXT    rmr_context_send;
+static DAT_RMR_CONTEXT    rmr_context_recv;
+static DAT_VLEN		  registered_size_send;
+static DAT_VLEN		  registered_size_recv;
+static DAT_VADDR	  registered_addr_send;
+static DAT_VADDR	  registered_addr_recv;
+
+/* Initial msg receive buf, RMR exchange, and Rdma-write notification */
+#define MSG_BUF_COUNT	  3
+static DAT_RMR_TRIPLET	  rmr_recv_msg[ MSG_BUF_COUNT ];
+static DAT_LMR_HANDLE     h_lmr_recv_msg = DAT_HANDLE_NULL;
+static DAT_LMR_CONTEXT    lmr_context_recv_msg;
+static DAT_RMR_CONTEXT    rmr_context_recv_msg;
+static DAT_VLEN		  registered_size_recv_msg;
+static DAT_VADDR	  registered_addr_recv_msg;
+
+/* message send buffer */
+static DAT_RMR_TRIPLET	    rmr_send_msg;
+static DAT_LMR_HANDLE	    h_lmr_send_msg = DAT_HANDLE_NULL;
+static DAT_LMR_CONTEXT	    lmr_context_send_msg;
+static DAT_RMR_CONTEXT	    rmr_context_send_msg;
+static DAT_VLEN		    registered_size_send_msg;
+static DAT_VADDR	    registered_addr_send_msg;
+static DAT_EP_ATTR	    ep_attr;
+char			    hostname[256] = {0};
+
+/* rdma pointers */
+char 	*rbuf = NULL;
+char	*sbuf = NULL;
+int     status;
+
+/* timers */
+double start,stop,total_us,total_sec;
+struct {
+    double  total;
+    double  open;
+    double  reg;
+    double  unreg;
+    double  pzc;
+    double  pzf;
+    double  evdc;
+    double  evdf;
+    double  cnoc;
+    double  cnof;
+    double  epc;
+    double  epf;
+    double  rdma_wr;
+    double  rdma_rd;
+    double  rtt;
+    double  close;
+} time;
+
+/* defaults */
+static int  parent=1;
+static int  connected=0;
+static int  burst=10;
+static int  server=1;
+static int  verbose=0;
+static int  polling=1;
+static int  poll_count=0;
+static int  rdma_wr_poll_count=0;
+static int  rdma_rd_poll_count=0;
+static int  pin_memory=0;
+static int  delay=0;
+static int  buf_len=RDMA_BUFFER_SIZE;
+static int  use_cno=0;
+static int  post_recv_count=MSG_BUF_COUNT;
+static int  recv_msg_index=0;
+static int  burst_msg_posted=0;
+static int  burst_msg_index=0;
+
+#define MAX_RDMA_RD	4
+#define MAX_PROCS	1000
+
+static pid_t	child[MAX_PROCS+1];
+
+/* forward prototypes */
+const char * DT_RetToString (DAT_RETURN ret_value);
+const char * DT_EventToSTr (DAT_EVENT_NUMBER event_code);
+void	    print_usage();
+double	    get_time();
+void	    init_data();
+
+DAT_RETURN	send_msg(   void                    *data,
+			    DAT_COUNT               size,
+			    DAT_LMR_CONTEXT         context,
+			    DAT_DTO_COOKIE          cookie,
+			    DAT_COMPLETION_FLAGS    flags );
+
+DAT_RETURN	connect_ep( char *hostname, int conn_id );
+void		disconnect_ep( void );
+DAT_RETURN	register_rdma_memory( void );
+DAT_RETURN	unregister_rdma_memory( void );
+DAT_RETURN	create_events( void );
+DAT_RETURN	destroy_events(void);
+DAT_RETURN	do_rdma_write_with_msg( void );
+DAT_RETURN	do_rdma_read_with_msg( void );
+DAT_RETURN	do_ping_pong_msg( void );
+
+#define LOGPRINTF(_format, _aa...) \
+	if (verbose)		   \
+	    printf(_format, ##_aa)
+
+main(int argc, char **argv)
+{
+	int c;
+	DAT_RETURN  ret;
+	        
+	/* parse arguments */
+	while ((c = getopt(argc, argv, "scvpb:d:B:h:")) != -1)
+	{
+		switch(c)
+		{
+			case 's': 
+				server = 1;
+				printf("%d Running as server\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'c': 
+				use_cno = 1;
+				printf("%d Creating CNO for DTO EVD's\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'v': 
+				verbose = 1;
+				printf("%d Verbose\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'p': 
+				polling = 1;
+				printf("%d Polling\n",getpid()); 
+				fflush(stdout);
+				break;
+			case 'B': 
+				burst = atoi(optarg);
+				break;
+			case 'd': 
+				delay = atoi(optarg);
+				break;
+			case 'b': 
+				buf_len = atoi(optarg);
+				break;
+			case 'h':
+				server = 0;
+				strcpy (hostname, optarg);
+    				break;
+			default: 
+				print_usage(); 
+				exit(-12);
+		}
+	}
+
+	if (!server) {
+		printf("%d Running as client\n",getpid()); fflush(stdout);
+	} else {
+		printf("%d Running as server\n",getpid()); fflush(stdout);
+	}
+
+	/* allocate send and receive buffers */
+	if (((rbuf = malloc(buf_len*burst)) == NULL) ||
+	    ((sbuf = malloc(buf_len*burst)) == NULL)) {  
+		perror("malloc");
+		exit(1);
+	}
+	memset( &time, sizeof(time), 0);
+	LOGPRINTF("%d Allocated RDMA buffers (r:%p,s:%p) len %d \n",
+			getpid(), rbuf, sbuf, buf_len);
+
+	/* dat_ia_open, dat_pz_create */
+	h_async_evd = DAT_HANDLE_NULL;
+	start = get_time();
+	ret = dat_ia_open( DAPL_PROVIDER, 8, &h_async_evd, &h_ia );
+	stop = get_time();
+	time.open += ((stop - start)*1.0e6);
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: Error Adaptor open: %s\n", 
+			getpid(),DT_RetToString(ret));
+		exit(1);
+	} else 
+		LOGPRINTF("%d Opened Interface Adaptor\n",getpid());
+	
+	/* Create Protection Zone */
+	start = get_time();
+	LOGPRINTF("%d Create Protection Zone\n",getpid());
+	ret = dat_pz_create(h_ia, &h_pz);
+	stop = get_time();
+	time.pzc += ((stop - start)*1.0e6);
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, 
+			"%d Error creating Protection Zone: %s\n", 
+			getpid(),DT_RetToString(ret));
+		exit(1);
+	} else 
+		LOGPRINTF("%d Created Protection Zone\n",getpid());
+	         
+	/* Register memory */
+	LOGPRINTF("%d Register RDMA memory\n", getpid());
+	ret = register_rdma_memory();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error creating events: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d Register RDMA memory done\n", getpid());
+	
+	LOGPRINTF("%d Create events\n", getpid());
+	ret = create_events();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error creating events: %s\n", 
+			getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else {
+		LOGPRINTF("%d Create events done\n", getpid());
+	}
+
+	/* create EP */
+	memset( &ep_attr, 0, sizeof(ep_attr) );
+	ep_attr.service_type                = DAT_SERVICE_TYPE_RC;
+	ep_attr.max_rdma_size               = 0x10000;
+	ep_attr.qos                         = 0;
+	ep_attr.recv_completion_flags       = 0;
+	ep_attr.max_recv_dtos               = MSG_BUF_COUNT + (burst*3);
+	ep_attr.max_request_dtos            = MSG_BUF_COUNT + (burst*3) + MAX_RDMA_RD;
+	ep_attr.max_recv_iov                = 1;
+	ep_attr.max_request_iov             = 1;
+	ep_attr.max_rdma_read_in            = MAX_RDMA_RD;
+	ep_attr.max_rdma_read_out           = MAX_RDMA_RD;
+	ep_attr.request_completion_flags    = DAT_COMPLETION_DEFAULT_FLAG;
+	ep_attr.ep_transport_specific_count = 0;
+	ep_attr.ep_transport_specific       = NULL;
+	ep_attr.ep_provider_specific_count  = 0;
+	ep_attr.ep_provider_specific        = NULL; 
+
+	start = get_time();
+	ret = dat_ep_create( h_ia, h_pz, h_dto_evd, h_dto_evd, h_conn_evd, &ep_attr, &h_ep );
+	stop = get_time();
+	time.epc += ((stop - start)*1.0e6);
+	time.total += time.epc;
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error dat_ep_create: %s\n", 
+			getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d EP created %p \n", getpid(), h_ep);
+	
+	/* 
+	 * register message buffers, establish connection, and
+	 * exchange DMA RMR information info via messages
+	 */
+	ret = connect_ep( hostname, SERVER_CONN_QUAL );
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error connect_ep: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d connect_ep complete\n", getpid());
+	
+	/*********** RDMA write data *************/
+	ret = do_rdma_write_with_msg();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error do_rdma_write_with_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d do_rdma_write_with_msg complete\n", getpid());
+	
+	/*********** RDMA read data *************/
+	ret = do_rdma_read_with_msg();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error do_rdma_read_with_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else 
+		LOGPRINTF("%d do_rdma_read_with_msg complete\n", getpid());
+
+	/*********** PING PING messages ************/
+	ret = do_ping_pong_msg();
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error do_ping_pong_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		goto cleanup;
+	} else
+		LOGPRINTF("%d do_ping_pong_msg complete\n", getpid());
+
+cleanup:
+	/* disconnect and free EP resources */
+	if ( h_ep != DAT_HANDLE_NULL ) {
+		/* unregister message buffers and tear down connection */
+		LOGPRINTF("%d Disconnect and Free EP %p \n",getpid(),h_ep);
+		disconnect_ep();
+	}
+
+	/* free EP */
+	LOGPRINTF("%d Free EP %p \n",getpid(),h_ep);
+	start = get_time();
+	ret = dat_ep_free( h_ep );
+	stop = get_time();
+	time.epf += ((stop - start)*1.0e6);
+	time.total += time.epf;
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error freeing EP: %s\n", 
+			getpid(), DT_RetToString(ret));
+	} else {
+		LOGPRINTF("%d Freed EP\n",getpid());
+		h_ep = DAT_HANDLE_NULL;
+	}
+
+	/* free EVDs */
+	LOGPRINTF("%d destroy events\n", getpid());
+	ret = destroy_events();
+	if(ret != DAT_SUCCESS) 
+		fprintf(stderr, "%d Error destroy_events: %s\n", 
+			getpid(),DT_RetToString(ret));
+	else 
+		LOGPRINTF("%d destroy events done\n", getpid());
+	
+
+	ret = unregister_rdma_memory();
+	LOGPRINTF("%d unregister_rdma_memory \n", getpid());
+	if(ret != DAT_SUCCESS) 
+		fprintf(stderr, "%d Error unregister_rdma_memory: %s\n", 
+			getpid(),DT_RetToString(ret));
+	else 
+		LOGPRINTF("%d unregister_rdma_memory done\n", getpid());
+	
+
+	if (delay) sleep(delay);
+    	       
+	/* Free protection domain */
+	LOGPRINTF("%d Freeing pz\n",getpid());
+	start = get_time();
+	ret = dat_pz_free( h_pz );
+	stop = get_time();
+	time.pzf += ((stop - start)*1.0e6);
+	if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error freeing PZ: %s\n", 
+		getpid(), DT_RetToString(ret));
+	} else {
+		LOGPRINTF("%d Freed pz\n",getpid());
+		h_pz = NULL;
+	}
+
+	/* close the device */
+	LOGPRINTF("%d Closing Interface Adaptor\n",getpid());
+	start = get_time();
+	ret = dat_ia_close( h_ia, DAT_CLOSE_ABRUPT_FLAG );
+	stop = get_time();
+	time.close += ((stop - start)*1.0e6);
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: Error Adaptor close: %s\n", 
+			getpid(),DT_RetToString(ret));
+		exit(1);
+	} else 
+		LOGPRINTF("%d Closed Interface Adaptor\n",getpid());
+
+        printf( "\n%d: DAPL Test Complete.\n\n",getpid());
+	printf( "%d: RDMA write:  Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n", 
+		getpid(), time.rdma_wr, burst, time.rdma_wr/burst, rdma_wr_poll_count );
+	printf( "%d: RDMA read:   Total=%10.2lf usec,   %d bursts, itime=%10.2lf usec, pc=%d\n", 
+		getpid(), time.rdma_rd, MAX_RDMA_RD, time.rdma_rd/MAX_RDMA_RD, rdma_rd_poll_count );
+	printf( "%d: Message RTT: Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n\n", 
+		getpid(), time.rtt, burst, time.rtt/burst, poll_count );
+
+        printf( "%d: open:      %10.2lf usec\n", getpid(), time.open  );
+        printf( "%d: close:     %10.2lf usec\n", getpid(), time.close );
+        printf( "%d: PZ create: %10.2lf usec\n", getpid(), time.pzc );
+        printf( "%d: PZ free:   %10.2lf usec\n", getpid(), time.pzf );
+        printf( "%d: LMR create:%10.2lf usec\n", getpid(), time.reg );
+        printf( "%d: LMR free:  %10.2lf usec\n", getpid(), time.unreg );
+        printf( "%d: EVD create:%10.2lf usec\n", getpid(), time.evdc );
+        printf( "%d: EVD free:  %10.2lf usec\n", getpid(), time.evdf );
+        if (use_cno) {
+	    printf( "%d: CNO create:  %10.2lf usec\n", getpid(), time.cnoc );
+	    printf( "%d: CNO free:    %10.2lf usec\n", getpid(), time.cnof );
+        }
+        printf( "%d: EP create: %10.2lf usec\n",getpid(), time.epc );
+        printf( "%d: EP free:   %10.2lf usec\n",getpid(), time.epf );
+        printf( "%d: TOTAL:     %10.2lf usec\n",getpid(), time.total );
+    
+	/* free rdma buffers */  
+	free(rbuf);
+	free(sbuf);
+}
+
+
+double get_time()
+{
+	struct timeval tp;
+
+	gettimeofday(&tp, NULL);
+	return ((double) tp.tv_sec + (double) tp.tv_usec * 1e-6);
+}
+ 
+void init_data()
+{
+	memset(rbuf, 'a', buf_len);
+	memset(sbuf, 'b', buf_len);
+}
+
+
+DAT_RETURN
+send_msg(   void                    *data,
+	    DAT_COUNT               size,
+	    DAT_LMR_CONTEXT         context,
+	    DAT_DTO_COOKIE          cookie,
+	    DAT_COMPLETION_FLAGS    flags )
+{
+    DAT_LMR_TRIPLET	iov;
+    DAT_EVENT		event;
+    DAT_COUNT		nmore;
+    DAT_RETURN		ret;
+
+    iov.lmr_context     = context;
+    iov.pad             = 0;
+    iov.virtual_address = (DAT_VADDR)(unsigned long)data;
+    iov.segment_length  = size;
+    LOGPRINTF("%d calling post_send\n", getpid());
+    cookie.as_64 = 0xaaaa;
+    ret = dat_ep_post_send( h_ep,
+			    1,
+			    &iov,
+			    cookie,
+			    flags );
+
+    if (ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d: ERROR: dat_ep_post_send() %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return ret;
+    }
+
+    if (!(flags & DAT_COMPLETION_SUPPRESS_FLAG)) {
+	if ( polling ) {
+	    printf("%d Polling post send completion...\n",getpid());
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY );
+	} 
+	else {
+	    LOGPRINTF("%d waiting for post_send completion event\n", getpid());
+	    if (use_cno) {
+		DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		if ( evd != h_dto_evd ) {
+	    	    fprintf(stderr, 
+			"%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+			getpid());
+		    return( DAT_ABORT );
+		}
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+	        fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+			getpid(),DT_RetToString(ret));
+	        return ret;
+	    }
+	}
+
+	/* validate event number, len, cookie, and status */
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d: ERROR: DTO event number %s\n", 
+		    getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+	
+	if ((event.event_data.dto_completion_event_data.transfered_length != size ) ||
+	    (event.event_data.dto_completion_event_data.user_cookie.as_64 != 0xaaaa )) {
+	    fprintf(stderr, "%d: ERROR: DTO len %d or cookie %x\n", 
+		getpid(),
+		event.event_data.dto_completion_event_data.transfered_length,
+		event.event_data.dto_completion_event_data.user_cookie.as_64 );
+	    return( DAT_ABORT );
+		
+	}
+	if (event.event_data.dto_completion_event_data.status != DAT_SUCCESS) {
+	    fprintf(stderr, "%d: ERROR: DTO event status %s\n", 
+		    getpid(),DT_RetToString(ret));
+	    return( DAT_ABORT );
+	}
+    }
+
+    return DAT_SUCCESS;
+}
+
+
+DAT_RETURN
+connect_ep( char *hostname, int conn_id )
+{
+	DAT_SOCK_ADDR		remote_addr;
+	DAT_EP_ATTR		ep_attr;
+	DAT_RETURN		ret;
+	DAT_REGION_DESCRIPTION	region;
+	DAT_EVENT		event;
+	DAT_COUNT		nmore;
+	DAT_LMR_TRIPLET		l_iov;
+	DAT_RMR_TRIPLET		r_iov;
+	DAT_DTO_COOKIE		cookie;
+	int			i;
+    
+     /* Register send message buffer */
+    LOGPRINTF("%d Registering send Message Buffer %p, len %d\n",
+		getpid(), &rmr_send_msg, sizeof(DAT_RMR_TRIPLET) );
+    region.for_va = &rmr_send_msg;
+    ret = dat_lmr_create(   h_ia, 
+		            DAT_MEM_TYPE_VIRTUAL, 
+		            region, 
+		            sizeof(DAT_RMR_TRIPLET), 
+		            h_pz, 
+		            DAT_MEM_PRIV_LOCAL_WRITE_FLAG, 
+		            &h_lmr_send_msg, 
+		            &lmr_context_send_msg, 
+		            &rmr_context_send_msg,
+		            &registered_size_send_msg, 
+		            &registered_addr_send_msg );
+
+    if (ret != DAT_SUCCESS) {
+	fprintf(stderr, "%d Error registering send msg buffer: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return(ret);
+    } 
+    else 
+    	LOGPRINTF("%d Registered send Message Buffer %p \n",
+	        getpid(),region.for_va );
+    
+    /* Register Receive buffers */
+    LOGPRINTF("%d Registering Receive Message Buffer %p\n",
+		getpid(), rmr_recv_msg );
+    region.for_va = rmr_recv_msg;
+    ret = dat_lmr_create(   h_ia, 
+		            DAT_MEM_TYPE_VIRTUAL, 
+			    region, 
+			    sizeof(DAT_RMR_TRIPLET)*MSG_BUF_COUNT, 
+			    h_pz, 
+			    DAT_MEM_PRIV_LOCAL_WRITE_FLAG, 
+			    &h_lmr_recv_msg, 
+			    &lmr_context_recv_msg, 
+			    &rmr_context_recv_msg,
+			    &registered_size_recv_msg, 
+			    &registered_addr_recv_msg );
+    if(ret != DAT_SUCCESS) {
+	fprintf(stderr, "%d Error registering recv msg buffer: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return(ret);
+    } 
+    else 
+    	LOGPRINTF("%d Registered Receive Message Buffer %p\n",
+	        getpid(),region.for_va);
+    
+    for ( i = 0; i < MSG_BUF_COUNT; i++ ) {
+	cookie.as_64          = i;
+	l_iov.lmr_context     = lmr_context_recv_msg;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)&rmr_recv_msg[ i ];
+	l_iov.segment_length  = sizeof(DAT_RMR_TRIPLET);
+        
+	LOGPRINTF("%d Posting Receive Message Buffer %p\n", 
+		    getpid(), &rmr_recv_msg[ i ]);
+	ret = dat_ep_post_recv( h_ep,
+				1,
+				&l_iov,
+				cookie,
+				DAT_COMPLETION_DEFAULT_FLAG );
+        
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error registering recv msg buffer: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+        } 
+	else 
+	    LOGPRINTF("%d Registered Receive Message Buffer %p\n",
+			            getpid(),region.for_va);
+		
+    }
+
+    /* setup receive rdma buffer to initial string to be overwritten */
+    strcpy( (char*)rbuf, "blah, blah, blah\n" );
+
+    if ( server ) {  /* SERVER */
+    
+        /* create the service point for server listen */
+        LOGPRINTF("%d Creating service point for listen\n",getpid());
+	ret = dat_psp_create(   h_ia,
+				conn_id,
+				h_cr_evd,
+				DAT_PSP_CONSUMER_FLAG,
+				&h_psp );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_psp_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_psp_created for server listen\n", getpid());
+	
+        printf("%d Server waiting for connect request..\n", getpid());
+	ret = dat_evd_wait( h_cr_evd, SERVER_TIMEOUT, 1, &event, &nmore );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_evd_wait: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_evd_wait for cr_evd completed\n", getpid());
+	
+	if ( event.event_number != DAT_CONNECTION_REQUEST_EVENT ) {
+            fprintf(stderr, "%d Error unexpected cr event : %s\n", 
+				    getpid(),DT_EventToSTr(event.event_number));
+			return( DAT_ABORT );
+	}
+	if ( (event.event_data.cr_arrival_event_data.conn_qual != SERVER_CONN_QUAL) ||
+	     (event.event_data.cr_arrival_event_data.sp_handle.psp_handle != h_psp) ) {
+            fprintf(stderr, "%d Error wrong cr event data : %s\n", 
+		    getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+
+        /* accept connect request from client */
+	h_cr = event.event_data.cr_arrival_event_data.cr_handle;
+        LOGPRINTF("%d Accepting connect request from client\n",getpid());
+	ret = dat_cr_accept( h_cr, h_ep, 0, (DAT_PVOID)0 );
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_cr_accept: %s\n", 
+		    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_cr_accept completed\n", getpid());
+    } 
+    else {  /* CLIENT */ 
+	struct addrinfo	*target;
+	int		rval;
+
+	if (getaddrinfo (hostname, NULL, NULL, &target) != 0) {
+	    printf("\n remote name resolution failed!\n");
+	    exit ( 1 );
+	}
+
+	rval = ((struct sockaddr_in *)target->ai_addr)->sin_addr.s_addr;
+	printf ("%d Server Name: %s \n", getpid(), hostname);
+	printf ("%d Server Net Address: %d.%d.%d.%d\n", getpid(),
+	    (rval >>  0) & 0xff,
+	    (rval >>  8) & 0xff,
+	    (rval >> 16) & 0xff,
+	    (rval >> 24) & 0xff);
+
+	remote_addr = *((DAT_IA_ADDRESS_PTR)target->ai_addr);
+
+	LOGPRINTF("%d Connecting to server\n",getpid());
+	ret = dat_ep_connect(   h_ep,
+			        &remote_addr,
+			        conn_id,
+			        CONN_TIMEOUT,
+			        0,
+			        (DAT_PVOID)0,
+			        0,
+			        DAT_CONNECT_DEFAULT_FLAG  );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_ep_connect: %s\n", 
+				getpid(), DT_RetToString(ret));
+ 	    return(ret);
+	} 
+	else 
+	    LOGPRINTF("%d dat_ep_connect completed\n", getpid());
+    }
+
+    printf("%d Waiting for connect response\n",getpid());
+    
+    ret = dat_evd_wait( h_conn_evd, DAT_TIMEOUT_INFINITE, 1, &event, &nmore );
+    if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_evd_wait: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+    } 
+    else 
+    	    LOGPRINTF("%d dat_evd_wait for h_conn_evd completed\n", getpid());
+    
+    if ( event.event_number != DAT_CONNECTION_EVENT_ESTABLISHED ) {
+	    fprintf(stderr, "%d Error unexpected conn event : %s\n", 
+				getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+    }
+    printf("\n%d CONNECTED!\n\n",getpid());
+    connected = 1;
+	
+    /*
+     *  Setup our remote memory and tell the other side about it
+     */
+    rmr_send_msg.rmr_context    = rmr_context_recv;
+    rmr_send_msg.pad	         = 0;
+    rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf;
+    rmr_send_msg.segment_length = RDMA_BUFFER_SIZE;
+
+    printf("%d Send RMR to remote: snd_msg: r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n",
+	    getpid(), rmr_send_msg.rmr_context, rmr_send_msg.pad,
+	    rmr_send_msg.target_address, rmr_send_msg.segment_length );
+
+    ret = send_msg( &rmr_send_msg,
+		    sizeof( DAT_RMR_TRIPLET ),
+		    lmr_context_send_msg,
+		    cookie,
+		    DAT_COMPLETION_SUPPRESS_FLAG );
+
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error send_msg: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return(ret);
+    } 
+    else 
+	LOGPRINTF("%d send_msg completed\n", getpid());
+    
+    /*
+     *  Wait for remote RMR information for RDMA
+     */
+    if ( polling ) {
+	printf("%d Polling for remote to send RMR data\n",getpid());
+	while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY );		
+    } 
+    else  {
+	printf("%d Waiting for remote to send RMR data\n",getpid());
+	if (use_cno) 
+	{
+	    DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+	    ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+	    LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+	    if ( evd != h_dto_evd ) {
+	        fprintf(stderr, 
+			"%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+			getpid());
+	        return( DAT_ABORT );
+	    }
+	}
+	/* use wait to dequeue */
+	ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error waiting on h_dto_evd: %s\n", 
+		    getpid(),DT_RetToString(ret));
+	    return(ret);
+	} 
+	else {
+	    LOGPRINTF("%d dat_evd_wait h_dto_evd completed\n", getpid());
+	}
+    }
+
+    printf("%d remote RMR data arrived!\n",getpid());
+    
+    if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+        fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+		getpid(),DT_EventToSTr(event.event_number));
+        return( DAT_ABORT );
+    }
+    if ((event.event_data.dto_completion_event_data.transfered_length != 
+		sizeof( DAT_RMR_TRIPLET )) ||
+	(event.event_data.dto_completion_event_data.user_cookie.as_64 != 
+		recv_msg_index) ) {
+	fprintf(stderr,"ERR recv event: len=%d cookie=%d expected %d/%d\n",
+	    (int)event.event_data.dto_completion_event_data.transfered_length,
+	    (int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+	    sizeof(DAT_RMR_TRIPLET), recv_msg_index );
+	return( DAT_ABORT );
+    }
+
+    r_iov = rmr_recv_msg[ recv_msg_index ];
+    
+    printf("%d Received RMR from remote: r_iov: r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n",
+	    getpid(), r_iov.rmr_context, r_iov.pad,
+	    r_iov.target_address, r_iov.segment_length );
+
+    recv_msg_index++;
+
+    return ( DAT_SUCCESS );
+}
+
+
+void
+disconnect_ep()
+{
+    DAT_RETURN	ret;
+    DAT_EVENT	event;
+    DAT_COUNT	nmore;
+    int		i,flush_cnt;
+	
+    if (connected) {
+	
+	LOGPRINTF("%d dat_ep_disconnect\n", getpid());
+	ret = dat_ep_disconnect( h_ep, DAT_CLOSE_DEFAULT );
+	if(ret != DAT_SUCCESS) 	{
+		fprintf(stderr, "%d Error dat_ep_disconnect: %s\n", 
+				getpid(),DT_RetToString(ret));
+	} 
+	else {
+		LOGPRINTF("%d dat_ep_disconnect completed\n", getpid());
+	}
+    }
+    
+    /* destroy service point */
+    if (( server ) && ( h_psp != DAT_HANDLE_NULL )) {
+	ret = dat_psp_free( h_psp );
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_psp_free: %s\n", 
+		    getpid(),DT_RetToString(ret));
+	} 
+	else {
+	    LOGPRINTF("%d dat_psp_free completed\n", getpid());
+	}
+    }
+   
+    /* Unregister Send message Buffer */
+    if ( h_lmr_send_msg != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister send message h_lmr %p \n",getpid(),h_lmr_send_msg);
+	ret = dat_lmr_free(h_lmr_send_msg);
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering send msg mr: %s\n", 
+	    getpid(), DT_RetToString(ret));
+	} else {
+	    LOGPRINTF("%d Unregistered send message Buffer\n",getpid());
+	    h_lmr_send_msg = NULL;
+	}
+    }
+
+    /* Unregister recv message Buffer */
+    if ( h_lmr_recv_msg != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister recv message h_lmr %p \n",getpid(),h_lmr_recv_msg);
+	ret = dat_lmr_free(h_lmr_recv_msg);
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering recv msg mr: %s\n", 
+			    getpid(), DT_RetToString(ret));
+	} else {
+	    LOGPRINTF("%d Unregistered recv message Buffer\n",getpid());
+	    h_lmr_recv_msg = NULL;
+	}
+    }
+    return;
+}
+
+
+DAT_RETURN
+do_rdma_write_with_msg( )
+{
+	DAT_REGION_DESCRIPTION	region;
+	DAT_EVENT		event;
+	DAT_COUNT		nmore;
+	DAT_LMR_TRIPLET		l_iov;
+	DAT_RMR_TRIPLET		r_iov;
+	DAT_DTO_COOKIE		cookie;
+	DAT_RMR_CONTEXT		their_context;
+	DAT_RETURN		ret;
+	int			i;
+
+	printf("\n %d RDMA WRITE DATA with SEND MSG\n\n",getpid());
+
+	cookie.as_64 = 0x5555;
+	
+	if ( recv_msg_index >= MSG_BUF_COUNT )
+		return( DAT_ABORT );
+
+	/* get RMR information from previously received message */
+	r_iov = rmr_recv_msg[ recv_msg_index-1 ];
+	
+	if ( server )
+	    strcpy( (char*)sbuf, "server written data..." );
+	else
+	    strcpy( (char*)sbuf, "client written data..." );
+
+	l_iov.lmr_context     = lmr_context_send;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf;
+	l_iov.segment_length  = buf_len;
+	
+	start = get_time();
+	for (i=0;i<burst;i++) {
+	    cookie.as_64 = 0x9999;
+	    ret = dat_ep_post_rdma_write(   h_ep,		// ep_handle
+					    1,			// num_segments
+					    &l_iov,		// LMR
+					    cookie,		// user_cookie
+					    &r_iov,		// RMR
+					    DAT_COMPLETION_SUPPRESS_FLAG );
+	    if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: ERROR: dat_ep_post_rdma_write() %s\n", 
+					getpid(),DT_RetToString(ret));
+		return( DAT_ABORT );
+	    }
+	    LOGPRINTF("%d rdma_write # %d completed\n", getpid(),i+1);
+	}
+	
+	/*
+	 *  Send RMR information a 2nd time to indicate completion
+	 */
+	rmr_send_msg.rmr_context    = rmr_context_recv;
+	rmr_send_msg.pad	    = 0;
+	rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf;
+	rmr_send_msg.segment_length = RDMA_BUFFER_SIZE;
+
+	printf("%d Sending completion message\n",getpid());
+    
+	ret = send_msg(	&rmr_send_msg,
+		        sizeof( DAT_RMR_TRIPLET ),
+		        lmr_context_send_msg,
+		        cookie,
+		        DAT_COMPLETION_SUPPRESS_FLAG );
+
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		return(ret);
+	} else {
+		LOGPRINTF("%d send_msg completed\n", getpid());
+	}
+
+	/*
+	 *  Collect first event, write completion or the inbound recv with immed
+	 */
+	if ( polling ) {
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY )
+		rdma_wr_poll_count++;
+	} 
+	else {
+	    LOGPRINTF("%d waiting for message receive event\n", getpid());
+	    if (use_cno)  {
+		    DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		    ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		    LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		    if ( evd != h_dto_evd ) {
+			    fprintf(stderr, "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+					    getpid());
+			    return( ret );
+		    }
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+		    fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+					    getpid(),DT_RetToString(ret));
+		    return( ret );
+	    }
+	}
+	stop = get_time();
+	time.rdma_wr = ((stop - start)*1.0e6);
+
+	/* validate event number and status */
+	printf("%d inbound rdma_write; send message arrived!\n",getpid());
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+				getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+
+	if ( (event.event_data.dto_completion_event_data.transfered_length != sizeof(
DAT_RMR_TRIPLET )) ||
+	     (event.event_data.dto_completion_event_data.user_cookie.as_64 != recv_msg_index) ) {
+		
+	    fprintf(stderr,"unexpected event data for receive: len=%d cookie=%d exp %d/%d\n",
+		(int)event.event_data.dto_completion_event_data.transfered_length,
+		(int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+		sizeof(DAT_RMR_TRIPLET), recv_msg_index );
+
+	    return( DAT_ABORT );
+	}
+
+	r_iov = rmr_recv_msg[ recv_msg_index ];
+
+	printf("%d Received RMR from remote: r_iov: ctx=%x,pad=%x,va=%p,len=0x%x\n",
+		    getpid(), r_iov.rmr_context, 
+            r_iov.pad,
+            (void*)(unsigned long)r_iov.target_address,
+		    r_iov.segment_length );
+
+	LOGPRINTF("%d inbound rdma_write; send msg event SUCCESS!!!\n", getpid());
+    
+	printf("%d %s RDMA write buffer contains: %s\n", 
+			getpid(),
+			server ? "SERVER:" : "CLIENT:", 
+			rbuf );
+
+	recv_msg_index++;
+
+	return ( DAT_SUCCESS );
+}
+
+DAT_RETURN
+do_rdma_read_with_msg( )
+{
+	DAT_REGION_DESCRIPTION	region;
+	DAT_EVENT		event;
+	DAT_COUNT		nmore;
+	DAT_LMR_TRIPLET		l_iov;
+	DAT_RMR_TRIPLET		r_iov;
+	DAT_DTO_COOKIE		cookie;
+	DAT_RMR_CONTEXT		their_context;
+	DAT_RETURN		ret;
+	int			i;
+
+	printf("\n %d RDMA READ DATA with SEND MSG\n\n",getpid());
+
+	cookie.as_64 = 0x5555;
+	
+	if ( recv_msg_index >= MSG_BUF_COUNT )
+		return( DAT_ABORT );
+
+	/* get RMR information from previously received message */
+	r_iov = rmr_recv_msg[ recv_msg_index-1 ];
+	
+	/* setup rdma read buffer to initial string to be overwritten */
+	strcpy( (char*)sbuf, "blah, blah, blah\n" );
+
+	if ( server )
+	    strcpy( (char*)rbuf, "server read data..." );
+	else
+	    strcpy( (char*)rbuf, "client read data..." );
+
+	l_iov.lmr_context     = lmr_context_send;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf;
+	l_iov.segment_length  = buf_len;
+	
+	start = get_time();
+	for (i=0;i<MAX_RDMA_RD;i++) {
+	    cookie.as_64 = 0x9999;
+	    ret = dat_ep_post_rdma_read(    h_ep,		// ep_handle
+					    1,			// num_segments
+					    &l_iov,		// LMR
+					    cookie,		// user_cookie
+					    &r_iov,		// RMR
+					    DAT_COMPLETION_SUPPRESS_FLAG );
+	    if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: ERROR: dat_ep_post_rdma_read() %s\n", 
+					getpid(),DT_RetToString(ret));
+		return( DAT_ABORT );
+	    }
+	    LOGPRINTF("%d rdma_read # %d completed\n", getpid(),i+1);
+	}
+	
+	/*
+	 *  Send RMR information a 2nd time to indicate completion
+	 */
+	rmr_send_msg.rmr_context    = rmr_context_recv;
+	rmr_send_msg.pad	    = 0;
+	rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf;
+	rmr_send_msg.segment_length = RDMA_BUFFER_SIZE;
+
+	printf("%d Sending completion message\n",getpid());
+    
+	ret = send_msg(	&rmr_send_msg,
+		        sizeof( DAT_RMR_TRIPLET ),
+		        lmr_context_send_msg,
+		        cookie,
+		        DAT_COMPLETION_SUPPRESS_FLAG );
+
+	if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				getpid(),DT_RetToString(ret));
+		return(ret);
+	} else {
+		LOGPRINTF("%d send_msg completed\n", getpid());
+	}
+
+	/*
+	 *  Collect first event, write completion or the inbound recv with immed
+	 */
+	printf("%d Waiting for inbound message....\n",getpid());
+	if ( polling ) {
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY )
+		rdma_rd_poll_count++;
+	} 
+	else {
+	    LOGPRINTF("%d waiting for message receive event\n", getpid());
+	    if (use_cno) {
+		    DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		    ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		    LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		    if ( evd != h_dto_evd ) {
+			    fprintf(stderr, "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+					    getpid());
+			    return( ret );
+		    }
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+		    fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+					    getpid(),DT_RetToString(ret));
+		    return( ret );
+	    }
+	}
+	stop = get_time();
+	time.rdma_rd = ((stop - start)*1.0e6);
+
+	/* validate event number and status */
+	printf("%d inbound rdma_read; send message arrived!\n",getpid());
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+				getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+
+	if ( (event.event_data.dto_completion_event_data.transfered_length != sizeof(
DAT_RMR_TRIPLET )) ||
+	     (event.event_data.dto_completion_event_data.user_cookie.as_64 != recv_msg_index) ) {
+		
+	    fprintf(stderr,"unexpected event data for receive: len=%d cookie=%d exp %d/%d\n",
+		(int)event.event_data.dto_completion_event_data.transfered_length,
+		(int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+		sizeof(DAT_RMR_TRIPLET), recv_msg_index );
+
+	    return( DAT_ABORT );
+	}
+
+	r_iov = rmr_recv_msg[ recv_msg_index ];
+
+	printf("%d Received RMR from remote: r_iov: ctx=%x,pad=%x,va=%p,len=0x%x\n",
+		getpid(), r_iov.rmr_context, r_iov.pad,
+		(void*)(unsigned long)r_iov.target_address, r_iov.segment_length );
+
+	LOGPRINTF("%d inbound rdma_write; send msg event SUCCESS!!!\n", getpid());
+    
+	printf("%d %s RCV RDMA read buffer contains: %s\n", 
+			getpid(),
+			server ? "SERVER:" : "CLIENT:", 
+			sbuf );
+
+	recv_msg_index++;
+
+	return ( DAT_SUCCESS );
+}
+
+
+DAT_RETURN
+do_ping_pong_msg( )
+{
+    DAT_EVENT		event;
+    DAT_COUNT		nmore;
+    DAT_DTO_COOKIE	cookie;
+    DAT_LMR_TRIPLET	l_iov;
+    DAT_RETURN		ret;
+    int			i;
+    unsigned char	*snd_buf;
+    unsigned char	*rcv_buf;
+
+    printf("\n %d PING DATA with SEND MSG\n\n",getpid());
+    
+    snd_buf = sbuf;
+    rcv_buf = rbuf;
+
+    /* pre-post all buffers */
+    for ( i=0; i < burst; i++ ) {	
+	burst_msg_posted++;
+	cookie.as_64	      = i;
+	l_iov.lmr_context     = lmr_context_recv;
+	l_iov.pad             = 0;
+	l_iov.virtual_address = (DAT_VADDR)(unsigned long)rcv_buf;
+	l_iov.segment_length  = buf_len;
+        
+	LOGPRINTF("%d Pre-posting Receive Message Buffers %p\n", 
+		    getpid(), rcv_buf );
+	
+	ret = dat_ep_post_recv( h_ep,
+				1,
+				&l_iov,
+				cookie,
+				DAT_COMPLETION_DEFAULT_FLAG );
+        
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error posting recv msg buffer: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return(ret);
+        } 
+	else {
+	    LOGPRINTF("%d Posted Receive Message Buffer %p\n",
+			            getpid(),rcv_buf);
+        }
+
+	/* next buffer */
+	rcv_buf += buf_len;
+    }
+    sleep(1);
+    
+    /* Initialize recv_buf and index to beginning */
+    rcv_buf = rbuf;
+    burst_msg_index=0;
+
+    /* client ping 0x55, server pong 0xAA in first byte */
+    start = get_time();
+    for ( i=0;i<burst;i++ ) {
+	/* walk the send and recv buffers */
+	if ( !server ) {
+	    *snd_buf = 0x55;
+
+	    LOGPRINTF("%d %s SND buffer %p contains: 0x%x len=%d\n", 
+		    getpid(), server ? "SERVER:" : "CLIENT:", 
+		    snd_buf, *snd_buf, buf_len );
+
+	    ret = send_msg( snd_buf,
+			    buf_len,
+			    lmr_context_send,
+			    cookie,
+			    DAT_COMPLETION_SUPPRESS_FLAG );
+
+	    if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				    getpid(),DT_RetToString(ret));
+	        return(ret);
+	    } 
+	    else {
+	        LOGPRINTF("%d send_msg completed\n", getpid());
+	    }
+	}
+	   
+	/* Wait for recv message */
+	if ( polling ) {
+	    poll_count=0;
+	    LOGPRINTF("%d Polling for message receive event\n", getpid());
+	    while (  dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY )
+		poll_count++;
+	} 
+	else {
+	    LOGPRINTF("%d waiting for message receive event\n", getpid());
+	    if (use_cno) {
+		DAT_EVD_HANDLE evd = DAT_HANDLE_NULL;
+		ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd );
+		LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd);
+		if ( evd != h_dto_evd ) 
+		{
+		    fprintf(stderr, "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", 
+					getpid());
+		    return( ret );
+		}
+	    }
+	    /* use wait to dequeue */
+	    ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore );
+	    if (ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", 
+					getpid(),DT_RetToString(ret));
+		return( ret );
+	    }
+	}
+	/* start timer after first message arrives on server */
+	if ( i == 0) {
+	    start = get_time();
+	}
+	/* validate event number and status */
+	LOGPRINTF("%d inbound message; message arrived!\n",getpid());
+	if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) {
+	    fprintf(stderr, "%d Error unexpected DTO event : %s\n", 
+		    getpid(),DT_EventToSTr(event.event_number));
+	    return( DAT_ABORT );
+	}
+	if ((event.event_data.dto_completion_event_data.transfered_length 
+	    != buf_len) ||
+	    (event.event_data.dto_completion_event_data.user_cookie.as_64 
+	    != burst_msg_index) )  {
+    	    fprintf(stderr,"ERR: recv event: len=%d cookie=%d exp %d/%d\n",
+		(int)event.event_data.dto_completion_event_data.transfered_length,
+		(int)event.event_data.dto_completion_event_data.user_cookie.as_64,
+		buf_len, burst_msg_index );
+
+	    return( DAT_ABORT );
+	}
+        
+	LOGPRINTF("%d %s RCV buffer %p contains: 0x%x len=%d\n", 
+		    getpid(), server ? "SERVER:" : "CLIENT:", 
+		    rcv_buf, *rcv_buf, buf_len );
+
+	burst_msg_index++;
+
+	/* If server, change data and send it back to client */
+	if ( server ) {
+	    *snd_buf = 0xaa;
+
+	    LOGPRINTF("%d %s SND buffer %p contains: 0x%x len=%d\n", 
+		    getpid(), server ? "SERVER:" : "CLIENT:", 
+		    snd_buf, *snd_buf, buf_len );
+	    
+	    ret = send_msg( snd_buf,
+			    buf_len,
+			    lmr_context_send,
+			    cookie,
+			    DAT_COMPLETION_SUPPRESS_FLAG );
+
+	    if(ret != DAT_SUCCESS) {
+		fprintf(stderr, "%d Error send_msg: %s\n", 
+				    getpid(),DT_RetToString(ret));
+		return(ret);
+	    } 
+	    else {
+		LOGPRINTF("%d send_msg completed\n", getpid());
+	    }
+	}
+
+	/* next buffers */
+	rcv_buf += buf_len;
+	snd_buf += buf_len;
+    }
+    stop = get_time();
+    time.rtt = ((stop - start)*1.0e6);
+    
+    return ( DAT_SUCCESS );
+}
+
+/* Register RDMA Receive buffer */
+DAT_RETURN
+register_rdma_memory(void)
+{
+    DAT_RETURN				ret;
+    DAT_REGION_DESCRIPTION region;
+
+    region.for_va = rbuf;
+    start = get_time();
+    ret = dat_lmr_create(	h_ia, 
+			    DAT_MEM_TYPE_VIRTUAL, 
+			    region, 
+			    buf_len*burst, 
+			    h_pz, 
+			    DAT_MEM_PRIV_ALL_FLAG, 
+			    &h_lmr_recv, 
+			    &lmr_context_recv, 
+			    &rmr_context_recv,
+			    &registered_size_recv, 
+			    &registered_addr_recv );
+    stop = get_time();
+    time.reg += ((stop - start)*1.0e6);
+    time.total += time.reg;
+
+    if(ret != DAT_SUCCESS) {
+	fprintf(stderr, "%d Error registering recv buffer: %s\n", 
+		getpid(),DT_RetToString(ret));
+	return (ret);
+    } else {
+	    LOGPRINTF("%d Registered Receive RDMA Buffer %p\n",
+			getpid(),region.for_va);
+    }
+
+    /* Register RDMA Send buffer */
+    region.for_va = sbuf;
+    ret = dat_lmr_create(   h_ia, 
+			    DAT_MEM_TYPE_VIRTUAL, 
+			    region, 
+			    buf_len*burst, 
+			    h_pz, 
+			    DAT_MEM_PRIV_ALL_FLAG, 
+			    &h_lmr_send, 
+			    &lmr_context_send, 
+			    &rmr_context_send,
+			    &registered_size_send, 
+			    &registered_addr_send );
+    if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error registering send RDMA buffer: %s\n", 
+			    getpid(),DT_RetToString(ret));
+	    return (ret);
+    } else {
+	    LOGPRINTF("%d Registered Send RDMA Buffer %p\n",
+		    getpid(),region.for_va);
+    }
+
+    return DAT_SUCCESS;
+}
+
+/* 
+ * Unregister RDMA memory 
+ */
+DAT_RETURN
+unregister_rdma_memory(void)
+{
+    DAT_RETURN	ret;
+
+    /* Unregister Recv Buffer */
+    if ( h_lmr_recv != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister h_lmr %p \n",getpid(),h_lmr_recv);
+        start = get_time();
+	ret = dat_lmr_free(h_lmr_recv);
+        stop = get_time();
+	time.unreg += ((stop - start)*1.0e6);
+	time.total += time.unreg;
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering recv mr: %s\n", 
+			getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Unregistered Recv Buffer\n",getpid());
+	    h_lmr_recv = NULL; 
+	}
+    }
+
+    /* Unregister Send Buffer */
+    if ( h_lmr_send != DAT_HANDLE_NULL ) {
+	LOGPRINTF("%d Unregister h_lmr %p \n",getpid(),h_lmr_send);
+	ret = dat_lmr_free(h_lmr_send);
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error deregistering send mr: %s\n", 
+			    getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Unregistered send Buffer\n",getpid());
+	    h_lmr_send = NULL;
+	}
+    }
+    return DAT_SUCCESS;
+}
+
+ /* 
+  * Create CNO, CR, CONN, and DTO events 
+  */
+DAT_RETURN
+create_events(void)
+{
+    DAT_RETURN	ret;
+        
+    /* create CNO */
+    if (use_cno) {
+        start = get_time();
+        ret = dat_cno_create( h_ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &h_dto_cno  );
+        stop = get_time();
+	time.cnoc += ((stop - start)*1.0e6);
+	time.total += time.cnoc;
+	if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error dat_cno_create: %s\n", 
+				getpid(),DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d cr_evd created, %p\n", getpid(), h_dto_cno);
+	}
+    }
+
+    /* create cr EVD */
+    start = get_time();
+    ret = dat_evd_create( h_ia, 10, DAT_HANDLE_NULL, DAT_EVD_CR_FLAG, &h_cr_evd  );
+    stop = get_time();
+    time.evdc += ((stop - start)*1.0e6);
+    time.total += time.evdc;
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error dat_evd_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return (ret);
+    } 
+    else {
+        LOGPRINTF("%d cr_evd created %p\n", getpid(),h_cr_evd);
+    }
+    /* create conn EVD */
+    ret = dat_evd_create( h_ia, 10, DAT_HANDLE_NULL, DAT_EVD_CONNECTION_FLAG, &h_conn_evd  );
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error dat_evd_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return (ret);
+    } 
+    else {
+        LOGPRINTF("%d con_evd created %p\n", getpid(),h_conn_evd);
+    }     
+	
+    /* create dto EVD, with CNO if use_cno was set */
+    ret = dat_evd_create( h_ia,
+			  (MSG_BUF_COUNT*2)+burst*2, 
+			  h_dto_cno, 
+			  DAT_EVD_DTO_FLAG, 
+			  &h_dto_evd  );
+    if(ret != DAT_SUCCESS) {
+        fprintf(stderr, "%d Error dat_evd_create: %s\n", 
+			    getpid(),DT_RetToString(ret));
+        return (ret);
+    } 
+    else {
+        LOGPRINTF("%d dto_evd created %p\n", getpid(), h_dto_evd );
+    }
+
+    return DAT_SUCCESS;
+}
+
+/*
+ * Destroy CR, CONN, CNO, and DTO events 
+ */ 
+DAT_RETURN
+destroy_events(void)
+{
+	DAT_RETURN	ret;
+	
+	/* free cr EVD */
+    if ( h_cr_evd != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free cr EVD %p \n",getpid(),h_cr_evd);
+        ret = dat_evd_free( h_cr_evd );
+        if(ret != DAT_SUCCESS) {
+		    fprintf(stderr, "%d Error freeing cr EVD: %s\n", 
+			    getpid(), DT_RetToString(ret));
+			return (ret);
+	    } else {
+		    LOGPRINTF("%d Freed cr EVD\n",getpid());
+            h_cr_evd = DAT_HANDLE_NULL;
+	    }
+    }
+
+    /* free conn EVD */
+    if ( h_conn_evd != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free conn EVD %p \n",getpid(),h_conn_evd);
+        ret = dat_evd_free( h_conn_evd );
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error freeing conn EVD: %s\n", 
+			getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Freed conn EVD\n",getpid());
+	    h_conn_evd = DAT_HANDLE_NULL;
+	}
+    }
+
+    /* free dto EVD */
+    if ( h_dto_evd != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free dto EVD %p \n",getpid(),h_dto_evd);
+        start = get_time();
+	ret = dat_evd_free( h_dto_evd );
+        stop = get_time();
+	time.evdf += ((stop - start)*1.0e6);
+	time.total += time.evdf;
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error freeing dto EVD: %s\n", 
+			    getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Freed dto EVD\n",getpid());
+            h_dto_evd = DAT_HANDLE_NULL;
+	}
+    }
+
+    /* free CNO */
+    if ( h_dto_cno != DAT_HANDLE_NULL ) {
+        LOGPRINTF("%d Free dto CNO %p \n",getpid(),h_dto_cno);
+        start = get_time();
+        ret = dat_cno_free( h_dto_cno );
+        stop = get_time();
+        time.cnof += ((stop - start)*1.0e6);
+        time.total += time.cnof;
+        if(ret != DAT_SUCCESS) {
+	    fprintf(stderr, "%d Error freeing dto CNO: %s\n", 
+	    	    getpid(), DT_RetToString(ret));
+	    return (ret);
+	} 
+	else {
+	    LOGPRINTF("%d Freed dto CNO\n",getpid());
+            h_dto_cno = DAT_HANDLE_NULL;
+	}
+    }
+    return DAT_SUCCESS;
+}
+
+/*
+ * Map DAT_RETURN values to readable strings,
+ * but don't assume the values are zero-based or contiguous.
+ */
+char    errmsg[512] = {0};
+const char *
+DT_RetToString (DAT_RETURN ret_value)
+{
+    const char *major_msg, *minor_msg;
+    int sz;
+        
+    dat_strerror (ret_value, &major_msg, &minor_msg);
+  
+    strcpy(errmsg, major_msg);
+    strcat(errmsg, " ");
+    strcat(errmsg, minor_msg);
+
+    return errmsg;
+}
+
+/*
+ * Map DAT_EVENT_CODE values to readable strings
+ */
+const char *
+DT_EventToSTr (DAT_EVENT_NUMBER event_code)
+{
+    unsigned int i;
+    static struct {
+	    const char  *name;
+	    DAT_RETURN  value;
+    }
+    dat_events[] =
+    {
+	    #   define DATxx(x) { # x, x }
+	    DATxx (DAT_DTO_COMPLETION_EVENT),
+	    DATxx (DAT_RMR_BIND_COMPLETION_EVENT),
+	    DATxx (DAT_CONNECTION_REQUEST_EVENT),
+	    DATxx (DAT_CONNECTION_EVENT_ESTABLISHED),
+	    DATxx (DAT_CONNECTION_EVENT_PEER_REJECTED),
+	    DATxx (DAT_CONNECTION_EVENT_NON_PEER_REJECTED),
+	    DATxx (DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR),
+	    DATxx (DAT_CONNECTION_EVENT_DISCONNECTED),
+	    DATxx (DAT_CONNECTION_EVENT_BROKEN),
+	    DATxx (DAT_CONNECTION_EVENT_TIMED_OUT),
+	    DATxx (DAT_CONNECTION_EVENT_UNREACHABLE),
+	    DATxx (DAT_ASYNC_ERROR_EVD_OVERFLOW),
+	    DATxx (DAT_ASYNC_ERROR_IA_CATASTROPHIC),
+	    DATxx (DAT_ASYNC_ERROR_EP_BROKEN),
+	    DATxx (DAT_ASYNC_ERROR_TIMED_OUT),
+	    DATxx (DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR),
+	    DATxx (DAT_SOFTWARE_EVENT)
+	    #   undef DATxx
+    };
+    #   define NUM_EVENTS (sizeof(dat_events)/sizeof(dat_events[0]))
+
+    for (i = 0;  i < NUM_EVENTS;  i++) {
+	    if (dat_events[i].value == event_code)
+	    {
+	        return ( dat_events[i].name );
+	    }
+    }
+
+    return ( "Invalid_DAT_EVENT_NUMBER" );
+}
+
+
+void print_usage()
+{
+    printf("\n DAPL USAGE \n\n");
+    printf("s: server\n");
+    printf("c: use cno\n");
+    printf("v: verbose\n");
+    printf("p: polling\n");
+    printf("d: delay before close\n");
+    printf("b: buf length to allocate\n");
+    printf("B: burst count, rdma and msgs \n");
+    printf("h: hostname\n");
+    printf("\n");
+}
+
Index: dtest/dat.conf
===================================================================
--- dtest/dat.conf	(revision 0)
+++ dtest/dat.conf	(revision 0)
@@ -0,0 +1,11 @@
+#
+# DAT 1.1 and 1.2 configuration file
+#
+# Each entry should have the following fields:
+#
+# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
+#           <provider_version> <ia_params> <platform_params>
+#
+# Example for openib using the first Mellanox adapter,  port 1 and port 2
+
+IB1 u1.2 nonthreadsafe default
/home/ardavis/Proj/openib/gen2/users/jlentini/userspace/dapl/udapl/Target/libdapl.so r.1.2 "mthca0
1" ""
Index: dtest/README
===================================================================
--- dtest/README	(revision 0)
+++ dtest/README	(revision 0)
@@ -0,0 +1,19 @@
+simple dapl test just for initial openIB uDAPL testing...
+
+	dtest/dtest.c
+	dtest/makefile
+	dtest/dat.conf
+
+to build (default uDAPL name == IB1, ib device  ==  mthca0, port == 1)
+       edit makefile and change path (DAT_LIB) to appropriate libdat.so
+       edit dat.conf and change path to appropriate libdapl.so 
+       cp dat.conf to /etc/dat.conf 
+
+to run:
+	server:	dtest  
+	client:	dtest -h hostname
+
+for verbose uDAPL and uDAT debug:
+
+	export DAPL_DBG_TYPE=0xffff
+	export DAT_DBG_TYPE=0xffff
Index: dtest/makefile
===================================================================
--- dtest/makefile	(revision 0)
+++ dtest/makefile	(revision 0)
@@ -0,0 +1,16 @@
+CC         = gcc
+CFLAGS     = -O2 -g 
+
+DAT_INC = ../dat/include 
+DAT_LIB = /usr/lib64 
+
+all: dtest 
+
+clean:
+	rm -f *.o;touch *.c;rm -f dtest
+
+dtest: ./dtest.c 
+	$(CC) $(CFLAGS) ./dtest.c -o dtest \
+        -DDAPL_PROVIDER='"IB1"' \
+        -I $(DAT_INC) -L $(DAT_LIB) -ldat 
+	  







From ftillier at silverstorm.com  Fri Jul  8 09:51:24 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Fri, 8 Jul 2005 09:51:24 -0700
Subject: [openib-general] Re: IBDM and IBMgtSim Proposal Comments
In-Reply-To: <20050708074639.GA5204@mellanox.co.il>
Message-ID: <001201c583dd$4a4edaa0$6501a8c0@infiniconsys.com>

> From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
> Sent: Friday, July 08, 2005 12:47 AM
> 
> Performance is not the only reason to have a clean architecture.
> Why cant the windows driver match the openib API?

OSMV is not the Windows API.  The Windows stack has a clean architecture, is
already implemented, and works.  Why rewrite?

Also, the Linux API is not *the* openib API.  It's specific to Linux.

If there are issues with the architecture of the Windows drivers, let's hear
them and fix them.  Otherwise, just changing the API to match doesn't make
sense, especially given how different the driver models are between the two
operating systems.

Remember that code sharing is not a goal.  One reason for this is that the Linux
kernel code will never incorporate features that make sense in Windows but not
Linux for the sake of similarity in APIs.  The Linux stack is designed to fit
into Linux properly and not to be a piece of portable code.  The Windows stack
will do the same, but for Windows.

There's nothing that prevents the Windows or Linux APIs from evolving to more
closely match one another.  In fact, I plan on changing the CQ poll semantics to
match the Linux model more closely.

- Fab



From halr at voltaire.com  Fri Jul  8 09:48:26 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 08 Jul 2005 12:48:26 -0400
Subject: [openib-general] RE: [PATCH] CM: Add copyright
In-Reply-To: <ORSMSX401FRaqbC8wSA00000018@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401FRaqbC8wSA00000018@orsmsx401.amr.corp.intel.com>
Message-ID: <1120840937.4389.140.camel@hal.voltaire.com>

On Fri, 2005-07-08 at 12:43, Sean Hefty wrote:
> >CM: Add copyright to file touched by Voltaire
> >
> >Signed-off-by: Hal Rosenstock <halr at voltaire.com>
> >
> >Index: cm.c
> >===================================================================
> >--- cm.c        (revision 2820)
> >+++ cm.c        (working copy)
> >@@ -3,6 +3,7 @@
> >  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
> >  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
> >  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
> >+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
> >  *
> >  * This software is available to you under a choice of one of two
> >  * licenses.  You may choose to be licensed under the terms of the GNU
> 
> Do you want to add a new line or just modify the existing one?

Just modify. It should be 2004, 2005. I missed that this AM.

Thanks.

-- Hal



From sean.hefty at intel.com  Fri Jul  8 09:58:13 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 09:58:13 -0700
Subject: [openib-general] RE: [PATCH] CM: Add copyright
In-Reply-To: <1120840937.4389.140.camel@hal.voltaire.com>
Message-ID: <ORSMSX401FRaqbC8wSA0000001b@orsmsx401.amr.corp.intel.com>

>> >CM: Add copyright to file touched by Voltaire
>> >
>> >Signed-off-by: Hal Rosenstock <halr at voltaire.com>
>> >
>> >Index: cm.c
>> >===================================================================
>> >--- cm.c        (revision 2820)
>> >+++ cm.c        (working copy)
>> >@@ -3,6 +3,7 @@
>> >  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
>> >  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
>> >  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
>> >+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
>> >  *
>> >  * This software is available to you under a choice of one of two
>> >  * licenses.  You may choose to be licensed under the terms of the GNU
>>
>> Do you want to add a new line or just modify the existing one?
>
>Just modify. It should be 2004, 2005. I missed that this AM.

I've modified the copyright line to include 2005 and committed the change.

- Sean



From glebn at voltaire.com  Fri Jul  8 11:22:55 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Fri, 8 Jul 2005 21:22:55 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <ORSMSX401FRaqbC8wSA00000017@orsmsx401.amr.corp.intel.com>
References: <20050708062557.GA30898@minantech.com>
	<ORSMSX401FRaqbC8wSA00000017@orsmsx401.amr.corp.intel.com>
Message-ID: <20050708182254.GA8335@minantech.com>

On Fri, Jul 08, 2005 at 09:32:28AM -0700, Sean Hefty wrote:
> >> >> Tried that. But with big values create_qp fails.
> >> >
> >> >Yes, we need to fix that.
> >>
> >> To me, the current behavior sounds exactly like what you want.  The maximum
> >or
> >> some sort of "guaranteed value" should be available through the device
> >> properties.  Needing to determine it only by creating a QP doesn't sound like
> >> the right approach.
> >I am not sure what is the "current behavior" you are referring to.
> >Currently the only way I know to determine inline data size is create
> >QP is there other way?
> 
> By "current behavior" I meant that create_qp should fail if an inline size is
> specified that is too large.
>
But how should I know what maximum size that I can use? By trial and
error?

--
			Gleb.


From rolandd at cisco.com  Fri Jul  8 11:32:45 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 11:32:45 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050708182254.GA8335@minantech.com> (Gleb Natapov's message
	of "Fri, 8 Jul 2005 21:22:55 +0300")
References: <20050708062557.GA30898@minantech.com>
	<ORSMSX401FRaqbC8wSA00000017@orsmsx401.amr.corp.intel.com>
	<20050708182254.GA8335@minantech.com>
Message-ID: <524qb5tagi.fsf@topspin.com>

    Gleb> But how should I know what maximum size that I can use? By
    Gleb> trial and error?

This is a good question that I don't have an answer to right now.

However I did check in a change to libmthca that changes the value of
max_inline_data reported back from create_qp so that it will work for
all work requests.  This means it is a little lower than the true
limit for send requests, but it will work for RDMA write as well.

 - R.


From ftillier at silverstorm.com  Fri Jul  8 11:38:52 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Fri, 8 Jul 2005 11:38:52 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <524qb5tagi.fsf@topspin.com>
Message-ID: <001301c583ec$4b7f6d90$6501a8c0@infiniconsys.com>

> From: Roland Dreier [mailto:rolandd at cisco.com]
> Sent: Friday, July 08, 2005 11:33 AM
> 
>     Gleb> But how should I know what maximum size that I can use? By
>     Gleb> trial and error?
> 
> This is a good question that I don't have an answer to right now.

Why not add the max inline size to the CA attributes?  HCAs that don't support
inline data would set it to zero.

It does require the user to query the CA attributes to get the limit, but that's
probably OK.

- Fab




From sean.hefty at intel.com  Fri Jul  8 11:38:48 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 11:38:48 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <20050708182254.GA8335@minantech.com>
Message-ID: <ORSMSX401FRaqbC8wSA0000001f@orsmsx401.amr.corp.intel.com>

>> By "current behavior" I meant that create_qp should fail if an inline size is
>> specified that is too large.
>>
>But how should I know what maximum size that I can use? By trial and
>error?

I don't have a good answer to that yet.  I think that it should come from either
the device properties or some sort of query routine.  But I'm unclear on the
details.

One thought that I had was to rename "inline" sends to "non-registered" sends
and support any arbitrary size.  The device driver could then optimize transfers
based on any method that it chose, whether performing an inline transfer or
copying into pre-allocated buffers.  This seems to push too much policy into the
device driver though.  (I didn't say it was a good thought...)

I'm not sure about tying the inline size together with the number of sge's,
since that implies a specific implementation.  But then I can't think of a
reasonable alternative implementation...

Maybe the best solution is to keep what's there, but provide documentation on
how to calculate the actual amount of inline data that can be sent with any
given work request.  Or, provide a query (possibly an inline function or macro)
that takes a formatted work request as input and returns the maximum inline size
for that request.

- Sean



From sean.hefty at intel.com  Fri Jul  8 11:42:58 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 11:42:58 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <001301c583ec$4b7f6d90$6501a8c0@infiniconsys.com>
Message-ID: <ORSMSX4011XvpFVjCRG00000020@orsmsx401.amr.corp.intel.com>

>>     Gleb> But how should I know what maximum size that I can use? By
>>     Gleb> trial and error?
>>
>> This is a good question that I don't have an answer to right now.
>
>Why not add the max inline size to the CA attributes?  HCAs that don't support
>inline data would set it to zero.
>
>It does require the user to query the CA attributes to get the limit, but
>that's
>probably OK.

In general I agree that the ib_device_attr is a good place to store this.  The
problem is that the maximum changes depending on the work request type (send or
RDMA write), if immediate data is used, and number of sge's.

- Sean




From glebn at voltaire.com  Fri Jul  8 11:50:29 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Fri, 8 Jul 2005 21:50:29 +0300
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <524qb5tagi.fsf@topspin.com>
References: <20050708062557.GA30898@minantech.com>
	<ORSMSX401FRaqbC8wSA00000017@orsmsx401.amr.corp.intel.com>
	<20050708182254.GA8335@minantech.com> <524qb5tagi.fsf@topspin.com>
Message-ID: <20050708185029.GC8335@minantech.com>

On Fri, Jul 08, 2005 at 11:32:45AM -0700, Roland Dreier wrote:
>     Gleb> But how should I know what maximum size that I can use? By
>     Gleb> trial and error?
> 
> This is a good question that I don't have an answer to right now.
> 
> However I did check in a change to libmthca that changes the value of
> max_inline_data reported back from create_qp so that it will work for
> all work requests.  This means it is a little lower than the true
> limit for send requests, but it will work for RDMA write as well.
> 
This is acceptable solution for now I think. Thank you.

--
			Gleb.


From ftillier at silverstorm.com  Fri Jul  8 11:55:35 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Fri, 8 Jul 2005 11:55:35 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <ORSMSX4011XvpFVjCRG00000020@orsmsx401.amr.corp.intel.com>
Message-ID: <001401c583ee$a0b356d0$6501a8c0@infiniconsys.com>

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> Sent: Friday, July 08, 2005 11:43 AM
> 
> >>     Gleb> But how should I know what maximum size that I can use? By
> >>     Gleb> trial and error?
> >>
> >> This is a good question that I don't have an answer to right now.
> >
> >Why not add the max inline size to the CA attributes?  HCAs that don't
> >support inline data would set it to zero.
> >
> >It does require the user to query the CA attributes to get the limit, but
> >that's probably OK.
> 
> In general I agree that the ib_device_attr is a good place to store this.  The
> problem is that the maximum changes depending on the work request type (send
> or RDMA write), if immediate data is used, and number of sge's.

As far as I know, the maximum size supported by the Mellanox HCAs is larger than
optimum - that is, the performance goes down due to the data copy into the WQE
as compared to formatting the data segment and letting the HCA DMA the data.  

If we expect all implementation that support inline data to do so beyond the
optimal point, I don't see an issue with the maximum being the smallest maximum
of all the different types of work requests.

So my vote is for adding the max inline to the device attributes, and making it
the maximum for the most constrained work request (probably RDMA Write with
immediate?).

- Fab




From ogerlitz at voltaire.com  Fri Jul  8 11:54:59 2005
From: ogerlitz at voltaire.com (Or Gerlitz)
Date: Fri, 8 Jul 2005 21:54:59 +0300
Subject: [openib-general] Re: calculate inline data size
Message-ID: <D4F8F0B3820E754C887699BEF26A89404A9EC4@taurus.voltaire.com>

How about this (trivial) approach: given a requestred SG size for the send queue of a created QP, the user gets the max inline
data she can do sends/rdma-writes with this QP. If the user want to have the max inline size supported, she creates the QP
with the max SG size supported, where the latter is an attribute retrieved by the query hca verb. 
 
Typically, inline sends/rdma-writes are cool for latency benchmarks/apps as eliminating the pci transaction saves ~1us 
on PCI-X and ~0.5us on PCI-EXP. Now, latency benchmarks typically test with zero payload size, and generally, latency 
is an important measure for "short" messeges, "short" should be something between 0-256 bytes. With the VAPI gen1 driver
I recall that SG size of 16 gives max inline size of 400 something. If with mthca its more or less the same, i think we are done here.
 
Or.
________________________________

From: openib-general-bounces at openib.org on behalf of Sean Hefty
Sent: Fri 7/8/2005 9:38 PM
To: Gleb Natapov
Cc: openib-general at openib.org
Subject: RE: [openib-general] Re: calculate inline data size



>> By "current behavior" I meant that create_qp should fail if an inline size is
>> specified that is too large.
>>
>But how should I know what maximum size that I can use? By trial and
>error?

I don't have a good answer to that yet.  I think that it should come from either
the device properties or some sort of query routine.  But I'm unclear on the
details.

One thought that I had was to rename "inline" sends to "non-registered" sends
and support any arbitrary size.  The device driver could then optimize transfers
based on any method that it chose, whether performing an inline transfer or
copying into pre-allocated buffers.  This seems to push too much policy into the
device driver though.  (I didn't say it was a good thought...)

I'm not sure about tying the inline size together with the number of sge's,
since that implies a specific implementation.  But then I can't think of a
reasonable alternative implementation...

Maybe the best solution is to keep what's there, but provide documentation on
how to calculate the actual amount of inline data that can be sent with any
given work request.  Or, provide a query (possibly an inline function or macro)
that takes a formatted work request as input and returns the maximum inline size
for that request.

- Sean

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general




From sean.hefty at intel.com  Fri Jul  8 12:01:33 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 12:01:33 -0700
Subject: [openib-general] Re: calculate inline data size
In-Reply-To: <001401c583ee$a0b356d0$6501a8c0@infiniconsys.com>
Message-ID: <ORSMSX401Rqf69aZbLA00000022@orsmsx401.amr.corp.intel.com>

>As far as I know, the maximum size supported by the Mellanox HCAs is larger
>than
>optimum - that is, the performance goes down due to the data copy into the WQE
>as compared to formatting the data segment and letting the HCA DMA the data.

I think that this is only true if you ignore the cost of registering the memory.
If you can avoid registering the memory, then using inline is probably
beneficial beyond the current maximum.

>If we expect all implementation that support inline data to do so beyond the
>optimal point, I don't see an issue with the maximum being the smallest maximum
>of all the different types of work requests.
>
>So my vote is for adding the max inline to the device attributes, and making it
>the maximum for the most constrained work request (probably RDMA Write with
>immediate?).

I think that this makes the most sense until we have a better solution.

- Sean



From rolandd at cisco.com  Fri Jul  8 13:27:08 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 13:27:08 -0700
Subject: [openib-general] Kernel locking rules documentation
Message-ID: <52fyuprqlf.fsf@topspin.com>

I've tried to write up the locking rules used by our kernel IB layer,
so that they're made explicit for people implementing new low-level
drivers and new protocols.  I'd appreciate corrections and all other
comments.  Does this seem worth sending upstream for inclusion under
Documentation/infiniband?

 - R.

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: core_locking.txt
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050708/9bde2fcb/attachment.txt>

From surs at cse.ohio-state.edu  Fri Jul  8 13:39:54 2005
From: surs at cse.ohio-state.edu (Sayantan Sur)
Date: Fri, 8 Jul 2005 16:39:54 -0400
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <524qb61k0h.fsf@topspin.com>
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
	<52d5pu1kk5.fsf@topspin.com>
	<1120764553.4386.69.camel@hal.voltaire.com>
	<524qb61k0h.fsf@topspin.com>
Message-ID: <20050708203954.GB2682@cse.ohio-state.edu>

* On Jul,13 Roland Dreier<rolandd at cisco.com> wrote :
>     Roland> Are the ports connected to anything?
> 
>     Hal> They should show down rather than disabled (if not
>     Hal> connected), right ?
> 
> Yes, but it's still worth knowing what the situation is.

Sorry for the late reply. Actually the machine in question got shipped
out :-(( But yes, it was connected to a switch. We even tried back to
back connections with two of these machines -- with the same result.

One of the partitions on this node had IBGD 1.7.0 installed. But the
OpenIB problem was reported with a RH AS4 partition with a clean OpenIB
install.

We have some machines which work just fine with Gen2, so I was wondering
what the problem here was.

Thanks,
Sayantan.

> 
>  - R.

-- 
http://www.cse.ohio-state.edu/~surs


From rolandd at cisco.com  Fri Jul  8 13:42:41 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 13:42:41 -0700
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <20050708203954.GB2682@cse.ohio-state.edu> (Sayantan Sur's
	message of "Fri, 8 Jul 2005 16:39:54 -0400")
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
	<52d5pu1kk5.fsf@topspin.com>
	<1120764553.4386.69.camel@hal.voltaire.com>
	<524qb61k0h.fsf@topspin.com>
	<20050708203954.GB2682@cse.ohio-state.edu>
Message-ID: <52br5drpvi.fsf@topspin.com>

    Sayantan> One of the partitions on this node had IBGD 1.7.0
    Sayantan> installed. But the OpenIB problem was reported with a RH
    Sayantan> AS4 partition with a clean OpenIB install.

And IBGD 1.7.0 worked OK?

 - R.


From kjreilly at us.ibm.com  Fri Jul  8 13:45:26 2005
From: kjreilly at us.ibm.com (Kevin Reilly)
Date: Fri, 8 Jul 2005 16:45:26 -0400
Subject: [openib-general] CQ peek
Message-ID: <OF759F69EC.AA597D1D-ON85257038.00717046-85257038.0072054D@us.ibm.com>





Does gen2 user access lib have a way to peek at the CQ to see if it has
entries on it like VAPI did? All i see is ways to pop an entry
off the CQ.

Kevin J. Reilly
STSM, HPC Architecture
-Federation/HPS  Chief Engineer
-HPC interconnect architect
(office) 845-433-7976  (tieline) 8-293-7976



From rolandd at cisco.com  Fri Jul  8 13:48:29 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 13:48:29 -0700
Subject: [openib-general] CQ peek
In-Reply-To: <OF759F69EC.AA597D1D-ON85257038.00717046-85257038.0072054D@us.ibm.com>
	(Kevin Reilly's message of "Fri, 8 Jul 2005 16:45:26 -0400")
References: <OF759F69EC.AA597D1D-ON85257038.00717046-85257038.0072054D@us.ibm.com>
Message-ID: <527jg1rplu.fsf@topspin.com>

    Kevin> Does gen2 user access lib have a way to peek at the CQ to
    Kevin> see if it has entries on it like VAPI did? All i see is
    Kevin> ways to pop an entry off the CQ.

No, I never implemented that API.  What is it useful for?

 - R.


From kjreilly at us.ibm.com  Fri Jul  8 13:57:03 2005
From: kjreilly at us.ibm.com (Kevin Reilly)
Date: Fri, 8 Jul 2005 16:57:03 -0400
Subject: [openib-general] vendor_id, vendor_part_id, hw_ver
Message-ID: <OF30DC11F3.AA03CA0A-ON85257038.00720B9B-85257038.00731558@us.ibm.com>





struct ibv_device_attr {
                         ......
                              uint32_t                       vendor_id;
                              uint32_t
vendor_part_id;
                              uint32_t                       hw_ver;
                                 ........
                                                                  }

Is openIB going to do anything to enumerate the vendor_id,vendor_part_id
and hw_ver in a common header fille or is
it the responsiblity of ULP running ontop of the lib to understand these
values?


Kevin J. Reilly
STSM, HPC Architecture
-Federation/HPS  Chief Engineer
-HPC interconnect architect
(office) 845-433-7976  (tieline) 8-293-7976



From sean.hefty at intel.com  Fri Jul  8 13:57:26 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 13:57:26 -0700
Subject: [openib-general] Kernel locking rules documentation
In-Reply-To: <52fyuprqlf.fsf@topspin.com>
Message-ID: <ORSMSX401AX3cf3ednH00000029@orsmsx401.amr.corp.intel.com>

>I've tried to write up the locking rules used by our kernel IB layer,
>so that they're made explicit for people implementing new low-level
>drivers and new protocols.  I'd appreciate corrections and all other
>comments.  Does this seem worth sending upstream for inclusion under
>Documentation/infiniband?

Looks good to me.  I think that it makes sense to include this in the
Documentation area.

The only problem that stood out to me was a minor typo where "all all" appears
in the paragraph under the Hot Plug section.

- Sean



From rolandd at cisco.com  Fri Jul  8 14:00:20 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 14:00:20 -0700
Subject: [openib-general] vendor_id, vendor_part_id, hw_ver
In-Reply-To: <OF30DC11F3.AA03CA0A-ON85257038.00720B9B-85257038.00731558@us.ibm.com>
	(Kevin Reilly's message of "Fri, 8 Jul 2005 16:57:03 -0400")
References: <OF30DC11F3.AA03CA0A-ON85257038.00720B9B-85257038.00731558@us.ibm.com>
Message-ID: <523bqprp23.fsf@topspin.com>

    Kevin> Is openIB going to do anything to enumerate the
    Kevin> vendor_id,vendor_part_id and hw_ver in a common header
    Kevin> fille or is it the responsiblity of ULP running ontop of
    Kevin> the lib to understand these values?

I don't have anything planned to enumerate those values.  I would
guess that only a very few ULPs should even look at them.

 - R.


From surs at cse.ohio-state.edu  Fri Jul  8 14:02:30 2005
From: surs at cse.ohio-state.edu (Sayantan Sur)
Date: Fri, 8 Jul 2005 17:02:30 -0400
Subject: [openib-general] compilation problem (no member devt)
In-Reply-To: <52br5drpvi.fsf@topspin.com>
References: <20050704221903.GA16913@cse.ohio-state.edu>
	<52pstyvw99.fsf@topspin.com>
	<20050706234645.GA32470@cse.ohio-state.edu>
	<52d5pu1kk5.fsf@topspin.com>
	<1120764553.4386.69.camel@hal.voltaire.com>
	<524qb61k0h.fsf@topspin.com>
	<20050708203954.GB2682@cse.ohio-state.edu>
	<52br5drpvi.fsf@topspin.com>
Message-ID: <20050708210230.GB2760@cse.ohio-state.edu>

* On Jul,16 Roland Dreier<rolandd at cisco.com> wrote :
>     Sayantan> One of the partitions on this node had IBGD 1.7.0
>     Sayantan> installed. But the OpenIB problem was reported with a RH
>     Sayantan> AS4 partition with a clean OpenIB install.
> 
> And IBGD 1.7.0 worked OK?

Yep. It was working fine. I am puzzled by what "Disabled" state meant.

Thanks,
Sayantan.

> 
>  - R.

-- 
http://www.cse.ohio-state.edu/~surs


From simei-senyou at ad.netlaputa.com  Wed Jul  6 04:45:49 2005
From: simei-senyou at ad.netlaputa.com (=?ISO-2022-JP?B?GyRCMkRGYBsoQiA=?=)
Date: Wed, 06 Jul 2005 20:45:49 +0900
Subject: [openib-general] =?iso-2022-jp?b?UmU6GyRCOiMkKyRpJUAlYSEpGyhC?=
Message-ID: <20050706.1145490385@simei-senyou-ad.netlaputa.com>

あなた様への援助額：３０万円です。
岡本仁美さんより貴方様を指名するメールが来ましたのでご連絡致しました。
『迷惑じゃなければ電話で話したりして、話聞いて欲しいんです。
　今日これからでも会えますか？』
簡単なプロフィールを載せます。
34歳、年収1000万円以上。T154・B84、CカップW60、H86
貴方に紹介する事を岡本仁美さんに通知してあります。
http://chao.hosting-geomax.jp/index.php/tak006/
貴方の登録(無料)確認後は岡本仁美さんに
貴方が登録をした事を通知するので、登録(無料)後は
岡本仁美さんから直接返事が来ます。
その為サイトを通す必要はありません。
無料登録をしてお待ちください。
http://chao.hosting-geomax.jp/index.php/tak006/
※yahooなどの一般アドレスからでも登録可能です。

拒否・問い合わせ
bunbunbun at ad.netlaputa.com


From krause at cup.hp.com  Fri Jul  8 14:38:28 2005
From: krause at cup.hp.com (Michael Krause)
Date: Fri, 08 Jul 2005 14:38:28 -0700
Subject: [openib-general] vendor_id, vendor_part_id, hw_ver
In-Reply-To: <523bqprp23.fsf@topspin.com>
References: <OF30DC11F3.AA03CA0A-ON85257038.00720B9B-85257038.00731558@us.ibm.com>
	<523bqprp23.fsf@topspin.com>
Message-ID: <6.2.0.14.2.20050708143717.0209e738@esmail.cup.hp.com>

At 02:00 PM 7/8/2005, Roland Dreier wrote:
>     Kevin> Is openIB going to do anything to enumerate the
>     Kevin> vendor_id,vendor_part_id and hw_ver in a common header
>     Kevin> fille or is it the responsiblity of ULP running ontop of
>     Kevin> the lib to understand these values?
>
>I don't have anything planned to enumerate those values.  I would
>guess that only a very few ULPs should even look at them.

Why would this ever have to be examined by a ULP?  This seems like a 
low-level driver issue.  Given the verbs semantics abstract the hardware, 
aside from the HCA-specific driver, the rest of the stack should be unaware 
of any thing below.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050708/6f8ae35b/attachment.html>

From kjreilly at us.ibm.com  Fri Jul  8 15:08:41 2005
From: kjreilly at us.ibm.com (Kevin Reilly)
Date: Fri, 8 Jul 2005 18:08:41 -0400
Subject: [openib-general] CQ peek
In-Reply-To: <527jg1rplu.fsf@topspin.com>
Message-ID: <OF92075E78.F188E71C-ON85257038.007655FB-85257038.0079A491@us.ibm.com>





Peeks are often used to determine what processing to perform next in code
that doesn't want to do the processing itself.
A simple example would be if your in a send thread was sending lots on
packets every so often it may peek at a CQ used for receive completions and
if their are receives ready  then either signal the receive thread or just
put a marker on the wall and exit to allow receive
processing to happen. Usually ULP that support polling modes give priority
to receives over sends so peek is an easy way for the send
thread to cool it and switch gears to received.

Peek may also be useful to batch together completion processing into a
bigger therefore potentially more efficient unit. For example,
you have a send fifo where the completions are steered into a completion
queue.  You may only want to process the send completions
after some number of send fifo slots have completed therefore batching up
the processing into big units.

My last and least convincing reason for peek is if we want the gen2 user
space libraries to have a broad set of ULP layer ontop of
it.  Some of those ULPs may have implemented peek in their interface and
not having it in the lower layer makes it difficult.

Kevin J. Reilly
STSM, HPC Architecture
-Federation/HPS  Chief Engineer
-HPC interconnect architect
(office) 845-433-7976  (tieline) 8-293-7976



                                                                           
             Roland Dreier                                                 
             <rolandd at cisco.co                                             
             m>                                                         To 
                                       Kevin Reilly/Poughkeepsie/IBM at IBMUS 
             07/08/2005 04:48                                           cc 
             PM                        "openib-general at openib.org"         
                                       <openib-general at openib.org>         
                                                                   Subject 
                                       Re: [openib-general] CQ peek        
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




    Kevin> Does gen2 user access lib have a way to peek at the CQ to
    Kevin> see if it has entries on it like VAPI did? All i see is
    Kevin> ways to pop an entry off the CQ.

No, I never implemented that API.  What is it useful for?

 - R.




From rolandd at cisco.com  Fri Jul  8 15:34:15 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 15:34:15 -0700
Subject: [openib-general] CQ peek
In-Reply-To: <OF92075E78.F188E71C-ON85257038.007655FB-85257038.0079A491@us.ibm.com>
	(Kevin Reilly's message of "Fri, 8 Jul 2005 18:08:41 -0400")
References: <OF92075E78.F188E71C-ON85257038.007655FB-85257038.0079A491@us.ibm.com>
Message-ID: <52hdf5q654.fsf@topspin.com>

First, a general point about peek CQ.  For all RDMA device
architectures that I know of, doing "peek CQ" is just as expensive as
doing "poll CQ," since the driver needs to do the same locking and
incur the same cache misses either way.  And then the app will always
have to do "poll CQ" later anyway.

    Kevin> Peeks are often used to determine what processing to
    Kevin> perform next in code that doesn't want to do the processing
    Kevin> itself.  A simple example would be if your in a send thread
    Kevin> was sending lots on packets every so often it may peek at a
    Kevin> CQ used for receive completions and if their are receives
    Kevin> ready then either signal the receive thread or just put a
    Kevin> marker on the wall and exit to allow receive processing to
    Kevin> happen. Usually ULP that support polling modes give
    Kevin> priority to receives over sends so peek is an easy way for
    Kevin> the send thread to cool it and switch gears to received.

It seems better to use a completion event on the receive CQ to wake
up the receive thread, or just poll the receive CQ and pass any
completions you find to the receive queue.  Doing a peek in one thread
and a poll in another is almost guaranteed to be terrible for performance.

    Kevin> Peek may also be useful to batch together completion
    Kevin> processing into a bigger therefore potentially more
    Kevin> efficient unit. For example, you have a send fifo where the
    Kevin> completions are steered into a completion queue.  You may
    Kevin> only want to process the send completions after some number
    Kevin> of send fifo slots have completed therefore batching up the
    Kevin> processing into big units.

Again, I think peek is the wrong interface here.  Either continue
polling and accumulating completions until you get the number of
completions you want to batch, or have only every Nth send generate a
completion at all.

    Kevin> My last and least convincing reason for peek is if we want
    Kevin> the gen2 user space libraries to have a broad set of ULP
    Kevin> layer ontop of it.  Some of those ULPs may have implemented
    Kevin> peek in their interface and not having it in the lower
    Kevin> layer makes it difficult.

This is moderately compelling, and if someone has a concrete use for
poll CQ I'll probably end up implementing it someday (unless someone
beats me to it).  But I'd be somewhat surprised if a ULP uses a
primitive that is not defined in either the IB or iWARP verbs.

 - R.


From sean.hefty at intel.com  Fri Jul  8 15:50:20 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 15:50:20 -0700
Subject: [openib-general] OpenIB maintainers and iWarp
Message-ID: <ORSMSX401M3SOlbpH710000002a@orsmsx401.amr.corp.intel.com>

I was asked by one of the OpenIB promoters to send an e-mail directed towards
the iWarp vendors with respect to maintaining the software stack.  I believe
that the current OpenIB development process is sufficient for supporting iWarp.

The OpenIB stack has several maintainers focused on different components.
Typically whoever submits new code becomes the maintainer, but write access to
the main repository is restricted to specific individuals.  I believe that Matt
Leininger controls the access.

According to the initial agreements made by the OpenIB members, I was chosen as
a vendor neutral maintainer of the kernel verbs API, and Roland was selected as
the lead liaison between OpenIB and LKML.  Since then I believe that OpenIB has
evolved to a true open source development project where the maintainers simply
do what's right based on their technical knowledge and community feedback.

If there are any questions, please post them to the list.

Thanks,
- Sean



From sean.hefty at intel.com  Fri Jul  8 15:57:46 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 8 Jul 2005 15:57:46 -0700
Subject: [openib-general] CQ peek
In-Reply-To: <52hdf5q654.fsf@topspin.com>
Message-ID: <ORSMSX401X3cf3ednHM0000002b@orsmsx401.amr.corp.intel.com>

>First, a general point about peek CQ.  For all RDMA device
>architectures that I know of, doing "peek CQ" is just as expensive as
>doing "poll CQ," since the driver needs to do the same locking and
>incur the same cache misses either way.  And then the app will always
>have to do "poll CQ" later anyway.
{snip}
>This is moderately compelling, and if someone has a concrete use for
>poll CQ I'll probably end up implementing it someday (unless someone
>beats me to it).  But I'd be somewhat surprised if a ULP uses a
>primitive that is not defined in either the IB or iWARP verbs.

I vaguely remember discussing peek CQ before, but I don't remember what the
outcome was.  I tend to agree with Roland though.  There are likely mechanisms
to perform the same tasks that can provide better performance.  I would lean
more towards removing peek CQ from the API than implementing it, especially if
it leads an application to be less efficient.

- Sean



From kjreilly at us.ibm.com  Fri Jul  8 16:20:52 2005
From: kjreilly at us.ibm.com (Kevin Reilly)
Date: Fri, 8 Jul 2005 19:20:52 -0400
Subject: [openib-general] vendor_id, vendor_part_id, hw_ver
In-Reply-To: <6.2.0.14.2.20050708143717.0209e738@esmail.cup.hp.com>
Message-ID: <OFD5438117.E723E5B8-ON85257038.007F6D0A-85257038.0080400E@us.ibm.com>





Mike,
   Ideally your right the ULP shouldn't care what HCA it's running on.
There are some practical reasons why an ULP might
want to know the vendor and part number it was using like for debug or
taking advantage a perform nuance of a particular
HCA.

Kevin J. Reilly
STSM, HPC Architecture
-Federation/HPS  Chief Engineer
-HPC interconnect architect
(office) 845-433-7976  (tieline) 8-293-7976



                                                                           
             Michael Krause                                                
             <krause at cup.hp.co                                             
             m>                                                         To 
                                       Kevin Reilly/Poughkeepsie/IBM at IBMUS 
             07/08/2005 05:38                                           cc 
             PM                        "openib-general at openib.org"         
                                       <openib-general at openib.org>         
                                                                   Subject 
                                       Re: [openib-general] vendor_id,     
                                       vendor_part_id, hw_ver              
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




At 02:00 PM 7/8/2005, Roland Dreier wrote:
          Kevin> Is openIB going to do anything to enumerate the
          Kevin> vendor_id,vendor_part_id and hw_ver in a common header
          Kevin> fille or is it the responsiblity of ULP running ontop of
          Kevin> the lib to understand these values?

      I don't have anything planned to enumerate those values.  I would
      guess that only a very few ULPs should even look at them.

Why would this ever have to be examined by a ULP?  This seems like a
low-level driver issue.  Given the verbs semantics abstract the hardware,
aside from the HCA-specific driver, the rest of the stack should be unaware
of any thing below.

Mike



From kjreilly at us.ibm.com  Fri Jul  8 17:06:50 2005
From: kjreilly at us.ibm.com (Kevin Reilly)
Date: Fri, 8 Jul 2005 20:06:50 -0400
Subject: [openib-general] CQ peek
In-Reply-To: <ORSMSX401X3cf3ednHM0000002b@orsmsx401.amr.corp.intel.com>
Message-ID: <OF5A3DA88F.0DD12DF5-ON85257038.0081E228-85257039.00009F84@us.ibm.com>





Thanks Sean and Roland for your response. You right it's not a primitive in
the IB verbs or iWARP unless you think the verbs define a
non-consuming poll(). Which is a stretch for sure.  We fell into the trap
of structuring with it because it was in VAPI and maybe other
that coded over VAPI fell into this deadend too.

I guess you can make arguments that using a peek is sub optimal performance
vise but since there is a peek in sockets and there is a
peek function in VAPI some ULPs may be structures to use it.

Kevin J. Reilly
STSM, HPC Architecture
-Federation/HPS  Chief Engineer
-HPC interconnect architect
(office) 845-433-7976  (tieline) 8-293-7976



                                                                           
             "Sean Hefty"                                                  
             <sean.hefty at intel                                             
             .com>                                                      To 
                                       "'Roland Dreier'"                   
             07/08/2005 06:57          <rolandd at cisco.com>, Kevin          
             PM                        Reilly/Poughkeepsie/IBM at IBMUS       
                                                                        cc 
                                       <openib-general at openib.org>         
                                                                   Subject 
                                       RE: [openib-general] CQ peek        
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




>First, a general point about peek CQ.  For all RDMA device
>architectures that I know of, doing "peek CQ" is just as expensive as
>doing "poll CQ," since the driver needs to do the same locking and
>incur the same cache misses either way.  And then the app will always
>have to do "poll CQ" later anyway.
{snip}
>This is moderately compelling, and if someone has a concrete use for
>poll CQ I'll probably end up implementing it someday (unless someone
>beats me to it).  But I'd be somewhat surprised if a ULP uses a
>primitive that is not defined in either the IB or iWARP verbs.

I vaguely remember discussing peek CQ before, but I don't remember what the
outcome was.  I tend to agree with Roland though.  There are likely
mechanisms
to perform the same tasks that can provide better performance.  I would
lean
more towards removing peek CQ from the API than implementing it, especially
if
it leads an application to be less efficient.

- Sean





From rolandd at cisco.com  Fri Jul  8 22:13:22 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 08 Jul 2005 22:13:22 -0700
Subject: [openib-general] [ANNOUNCE] Initial checkin of SRP initiator
Message-ID: <52r7e8muj1.fsf@topspin.com>

I just checked in a first version of a SCSI RDMA Protocol (SRP)
initiator under infiniband/ulp/srp.  Error handling is generously
described as minimal, but normal operation is reasonably solid.  The
initiator has been tested against the Topspin/Cisco FC/SRP gateway as
well as another vendor's native IB storage.

The initiator does not do any device management queries to discover
targets in the kernel.  Instead, userspace needs to pass in target
information through an add_target file under /sys/class/infiniband_srp/.
See below for an example.

I'll be on vacation Monday through Wednesday of next week, and in
Ottawa for the Kernel Summit and Ottawa Linux Symposium the whole week
after that, but more documentation and better code will appear once
I'm back.  In the meantime, please fill my mailbox with feedback.

I'm attaching some (very hacky) python scripts to this email that can
be used as a device management client to discover storage.  Just untar
the package and follow the README to use it.

Finally, here's an example SRP in action: I discover a target and then
connect to it and do some I/O.  Don't take the performance numbers
seriously, since this is a single old 36 GB drive in a JBOD.  I see
_much_ higher numbers talking to a fancy native IB RAID box.

    # dmcli -t /sys/class/infiniband/mthca1/ports/1 -d /dev/infiniband/umad1 2
    IO Unit Info:
        max controllers:  1
    
        controller[  1]
        GUID:      0005ad00000015dd
        vendor ID: 0005ad
        device ID: 0005ad
        ID:         Topspin SRP/FC TCA
        service entries:  11
            service[  0]: 0000000000000066 / SRP.T10:21000004CF758882
            service[  1]: 0000000000000066 / SRP.T10:21000004CF92BDDA
            service[  2]: 0000000000000066 / SRP.T10:21000004CF92BE07
            service[  3]: 0000000000000066 / SRP.T10:21000004CF92BE12
            service[  4]: 0000000000000066 / SRP.T10:21000004CF92BE15
            service[  5]: 0000000000000066 / SRP.T10:21000004CF92BE24
            service[  6]: 0000000000000066 / SRP.T10:21000004CF92BE33
            service[  7]: 0000000000000066 / SRP.T10:21000004CF92C110
            service[  8]: 0000000000000066 / SRP.T10:21000004CF92C354
            service[  9]: 0000000000000066 / SRP.T10:21000004CF92C360
            service[ 10]: 0000000000000066 / SRP.T10:21000004CFE7A949
    
    # modprobe ib_srp
    # echo id_ext=21000004CFE7A949,ioc_gguid=0005ad00000015dd,dgid=fe800000000000000005ad00000015dd,pkey=ffff,service_id=0000000000000066 > /sys/class/infiniband_srp/srp-mthca1-1/add_target
    # dmesg
    [45839.392438] ib_srp: new target: id_ext 21000004cfe7a949 ioc_guid 0005ad00000015dd pkey ffff service_id 0000000000000066 dgid fe80:0000:0000:0000:0005:ad00:0000:15dd
    [45839.393309] ib_srp: Topspin/Cisco initiator port ID workaround activated for target GUID 0005ad00000015dd
    [45839.393814] ib_srp: REJ received
    [45839.393821] ib_srp: Topspin/Cisco redirect to target port GID fe800000000000000005ad0000012371
    [45839.394428] ib_srp: Topspin/Cisco initiator port ID workaround activated for target GUID 0005ad00000015dd
    [45839.394675] ib_srp: MRA received
    [45839.405305] ib_srp: REP received
    [45839.405309]   req_lim_delta 32
    [45839.405311]   max_it_iu_len 384
    [45839.405312]   max_ti_iu_len 128
    [45839.405483] scsi2 : SRP.T10:21000004CFE7A949
    [45839.420978]   Vendor: SEAGATE   Model: ST336607FC        Rev: 0004
    [45839.442096]   Type:   Direct-Access                      ANSI SCSI revision: 03
    [45839.468006] SCSI device sdb: 71687372 512-byte hdwr sectors (36704 MB)
    [45839.490459] SCSI device sdb: drive cache: write through
    [45839.508436] SCSI device sdb: 71687372 512-byte hdwr sectors (36704 MB)
    [45839.530869] SCSI device sdb: drive cache: write through
    [45839.548097]  sdb: sdb1 sdb2 sdb3 sdb4
    [45839.571821] Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0
    # hdparm -tT /dev/sdb
    
    /dev/sdb:
     Timing cached reads:   2312 MB in  2.00 seconds = 1156.18 MB/sec
     Timing buffered disk reads:  194 MB in  3.02 seconds =  64.16 MB/sec

    # mount /dev/sdb1 /mnt
    # cd /mnt
    # iozone -s 512m -r64 -i0 -i1         Iozone: Performance Test of File I/O
                    Version $Revision: 3.221 $                 Compiled for 32 bit mode.
                    Build: linux
    
            Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                         Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                         Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                         Randy Dunlap, Mark Montague, Dan Million,
                         Jean-Marc Zucconi, Jeff Blomberg,
                         Erik Habbinga, Kris Strecker.
    
            Run began: Fri Jul  8 22:05:24 2005
    
            File size set to 524288 KB
            Record Size 64 KB
            Command line used: iozone -s 512m -r64 -i0 -i1
            Output is in Kbytes/sec
            Time Resolution = 0.000001 seconds.
            Processor cache size set to 1024 Kbytes.
            Processor cache line size set to 32 bytes.
            File stride size set to 17 * record size.
                                                                random  random    bkwd  record  stride 
                  KB  reclen   write rewrite    read    reread    read   write    read rewrite    read  fwrite frewrite   fread  freread
              524288      64   46894   52908    27904    28061

 - R.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PyOpenIB-1.0.tar.gz
Type: application/x-compressed-tar
Size: 4925 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050708/3e37162d/attachment.bin>

From amelia_boucher at jerome.com  Sat Jul  9 18:59:37 2005
From: amelia_boucher at jerome.com (Delta ODX Service Env.)
Date: Sat, 09 Jul 2005 19:59:37 -0600
Subject: [openib-general] Delta ODX Service Env.
Message-ID: <067735187.92499252988013@jerome.com>

Libreville above has been dependably, his were fixate interlacing. They howled up she outsiders lasses did derby yours. 
Perseus they has been industrialists, mine regale. He onset they community's mars has fixture's your. I at hereinabove are Redstone him. 
   
Gasped disposable, it sections rubs did Eskimo them. 
Barricades is orbiting his brat's September. Bottleneck archival I may until Curtis back him evolutionary. 
Reveals it must grape's, me domesticating. Troutman lamps I being pecuniary. Mediums can calculated them illusive pastry. 
Apprentices has been shrewish near mine epidemiology Waters. 
Drenching aquarium you may Wier. 
Brent altruist, we coincidences from flyer's had been competes mine. Orbiters elbow it had been formalizations mine.  
Petticoats laughingstock we have sidings. Ballpark's you is assertions, theirs Rumania. 
It cultural could detectable. 
Elmer I has been headed, him portrayed. Robbery's curtsy's she is incapable its paucity. 
Jive are Lippincott until his quantitative. Gruesome down is glandular, yours would circular Malton. Ballast is displeasing, yours has feathered chiefs. 
Adheres creeper, she codeword potato could pullback your. It up philosophically must idyllic them. 
 Shearing we has been Paoli, theirs when feathered. 
Japanese be barnacle mine recreational. Puppeteer you would disfiguring, him to plodding. 
We parch up you clocks admix has been Holbrook his. Impetuous might persecution, your were armed rearrangement's. Axioms did bothersome, them could entrench Anderson. 
Billboard's Thuban we has barrow. 
He Saracen who he ascribed seaquake have cyclic them. I as primate are obliterates him. Phantom would cantankerously, mine may Wichita down plebiscite. 
Sharpness are relying yours periods. Backed she being debilitates, hers from anchoring. Incinerate anywhere you be Gilbertson. 
Bachelor's you are situationally, mine monarchs. 
Bolster has been minuscule down her scamper. 
It deposits are alma. Ruptures Ogden, you Cretan who loathing could habitual me. 
Dionysian bonny, I does awesome him.  
You towards cures could radiotelephone me. We down coefficient does board them. Evelyn are Pollard, theirs be gall whose impracticable. 
Crochet had been Valhalla into its choreography balustrade's. We dumped are Adelia. 
Consanguine has display his credulousness clone. She nips until it harmonies outer have been racketeering her. 
Monitored they have prostitution, her into nap's. 
 Humiliate would mothers, his are jig's that deportee. 
Rides redirecting she can grudges. 
We over metropolitan can Laplacian your.  McKenna would auscultations his fights. 
 
Inhospitable he had been pin's, theirs regents. 
Veneto Norwich I is for carpenter's away them antipode's. 
Viking have forsaking while hers hemisphere. 
Hardscrabble Maureen, it could scaled her. Sanctification that be paean, hers would minerals Coddington. It draftee I slant Pillsbury being desertion its. 
We from shiningly be Olivia me. 
She aimer he commonality collegiate would don't hers. Roaring iniquitous, they Calvary phrases have been regularities her. Ruin any he can near armpit's its. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050709/cce13897/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: coves.gif
Type: image/gif
Size: 5895 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050709/cce13897/attachment.gif>

From mst at mellanox.co.il  Sun Jul 10 00:05:13 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 10 Jul 2005 10:05:13 +0300
Subject: [openib-general] [PATCH] mthca: report board id in sysfs
In-Reply-To: <52pstttg1b.fsf@topspin.com>
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
	<52pstttg1b.fsf@topspin.com>
Message-ID: <20050710070512.GW14130@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [openib-general] [PATCH] mthca: report board id in sysfs
> 
>     Michael> To check, you can type
> 
>     Michael> mstflint -d <bus:dev.fn> q
> 
> Looks like the board ID is not there.
> 
> I think I understand what's happening now: on Topspin HCAs, the
> vendor-specific area is formatted differently, and the board ID string
> is not in the last 16 bytes of the VSD.
> 
> What's the best way to make this work for Topspin HCAs?
> 
>  - R.
> 

Since topspin cards have a distinct vendor id, its possible they dont
need the board id at all.

What'd you like to do?
- Report empty board id (thats what current code does). User can follow
  the device link and use vendor id to figure out its a topspin card.
- Check topspin vendor id and report some hard-coded string in board_id.
- Check topspin vendor id and report some other field from the vpd.


-- 
MST


From olh at suse.de  Sun Jul 10 12:35:26 2005
From: olh at suse.de (Olaf Hering)
Date: Sun, 10 Jul 2005 19:35:26 +0000
Subject: [openib-general] [PATCH 18/82] remove linux/version.h from
	drivers/infiniband/
In-Reply-To: <20050710193508.0.PmFpst2252.2247.olh@nectarine.suse.de>  
Message-ID: <20050710193526.18.UYMiOz2756.2247.olh@nectarine.suse.de> 


changing CONFIG_LOCALVERSION rebuilds too much, for no appearent reason.

Signed-off-by: Olaf Hering <olh at suse.de>

drivers/infiniband/core/cache.c           |    1 -
drivers/infiniband/hw/mthca/mthca_main.c  |    1 -
drivers/infiniband/ulp/ipoib/ipoib_main.c |    1 -
drivers/infiniband/ulp/ipoib/ipoib_vlan.c |    1 -
4 files changed, 4 deletions(-)

Index: linux-2.6.13-rc2-mm1/drivers/infiniband/core/cache.c
===================================================================
--- linux-2.6.13-rc2-mm1.orig/drivers/infiniband/core/cache.c
+++ linux-2.6.13-rc2-mm1/drivers/infiniband/core/cache.c
@@ -32,7 +32,6 @@
* $Id: cache.c 1349 2004-12-16 21:09:43Z roland $
*/

-#include <linux/version.h>
#include <linux/module.h>
#include <linux/errno.h>
#include <linux/slab.h>
Index: linux-2.6.13-rc2-mm1/drivers/infiniband/hw/mthca/mthca_main.c
===================================================================
--- linux-2.6.13-rc2-mm1.orig/drivers/infiniband/hw/mthca/mthca_main.c
+++ linux-2.6.13-rc2-mm1/drivers/infiniband/hw/mthca/mthca_main.c
@@ -34,7 +34,6 @@
*/

#include <linux/config.h>
-#include <linux/version.h>
#include <linux/module.h>
#include <linux/init.h>
#include <linux/errno.h>
Index: linux-2.6.13-rc2-mm1/drivers/infiniband/ulp/ipoib/ipoib_main.c
===================================================================
--- linux-2.6.13-rc2-mm1.orig/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ linux-2.6.13-rc2-mm1/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -34,7 +34,6 @@

#include "ipoib.h"

-#include <linux/version.h>
#include <linux/module.h>

#include <linux/init.h>
Index: linux-2.6.13-rc2-mm1/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
===================================================================
--- linux-2.6.13-rc2-mm1.orig/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
+++ linux-2.6.13-rc2-mm1/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
@@ -32,7 +32,6 @@
* $Id: ipoib_vlan.c 1349 2004-12-16 21:09:43Z roland $
*/

-#include <linux/version.h>
#include <linux/module.h>

#include <linux/init.h>


From talal.jaafar at gmail.com  Sun Jul 10 17:23:58 2005
From: talal.jaafar at gmail.com (Talal jaafar)
Date: Sun, 10 Jul 2005 20:23:58 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
Message-ID: <cd9967e2050710172345f01051@mail.gmail.com>

hi all,

   I am fairly new to OpenIB and I have downloaded and built the
2.6.12 kernel that has OpenIB support. My question now is how to
compile the OpenIB modules in that kernel; There is no documentation,
and all what I need to do is to test the IPoIB, so anyhelp would be
appreciated.

Thanks in advance,

 Talal


From rjwalsh at pathscale.com  Sun Jul 10 18:40:54 2005
From: rjwalsh at pathscale.com (Robert Walsh)
Date: Sun, 10 Jul 2005 18:40:54 -0700
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <cd9967e2050710172345f01051@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
Message-ID: <1121046055.14196.5.camel@phosphene.durables.org>

>    I am fairly new to OpenIB and I have downloaded and built the
> 2.6.12 kernel that has OpenIB support. My question now is how to
> compile the OpenIB modules in that kernel; There is no documentation,
> and all what I need to do is to test the IPoIB, so anyhelp would be
> appreciated.

The way I've been doing it (and I don't know if this is the "official"
way or not) is I blow away the drivers/infiniband directory from the
2.6.12 directory and remake it as a symlink to the
gen2/trunk/src/linux-kernel/infiniband directory in the svn tree.  Then
just configure and build the kernel as normal.

Regards,
 Robert.

-- 
Robert Walsh                                 Email: rjwalsh at pathscale.com
PathScale, Inc.                              Phone: +1 650 934 8117
2071 Stierlin Court, Suite 200                 Fax: +1 650 428 1969
Mountain View, CA 94043.




From jlentini at netapp.com  Mon Jul 11 06:24:58 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 11 Jul 2005 09:24:58 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] [udapl dtest]: Fix broken statement
In-Reply-To: <1120261202.4389.32.camel@hal.voltaire.com>
References: <1120261202.4389.32.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507110924370.25603@jlentini-linux.nane.netapp.com>


Committed in revision 2830.


From jlentini at netapp.com  Mon Jul 11 06:37:30 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 11 Jul 2005 09:37:30 -0400 (EDT)
Subject: [openib-general] Re: IP addressing on InfiniBand networks (Caitlin
	Bestler)
In-Reply-To: <6.2.0.14.2.20050705093014.0291a9d0@esmail.cup.hp.com>
References: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com>
	<52r7ej68ga.fsf@topspin.com>
	<6.2.0.14.2.20050705093014.0291a9d0@esmail.cup.hp.com>
Message-ID: <Pine.LNX.4.61.0507110933030.25603@jlentini-linux.nane.netapp.com>



On Tue, 5 Jul 2005, Michael Krause wrote:

> The intention was to allow one to manage the fabric by having 
> mapping functions from traditional IP management applications to IB 
> GID to minimize the amount of work to enable IB within a solution.

I was unaware of this. What happened to the mapping functions? What 
did the API look like and how was it going to be implemented?


From halr at voltaire.com  Mon Jul 11 06:37:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:37:33 -0400
Subject: [openib-general] [PATCH 0/27] InfiniBand core update
Message-ID: <1121089053.4389.4505.camel@hal.voltaire.com>

This patch series gets the Infiniband core up to date. Aside from bug
fixes, the following new functionality is also introduced:
	MAD (Management Datagram) support for RMPP
			(Reliable MultiPacket Protocol)
	CM (Communications Manager) support
	User CM support
	User MAD changes for RMPP
	Service Record support in SA query

Thanks.

-- Hal



From halr at voltaire.com  Mon Jul 11 06:41:54 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:41:54 -0400
Subject: [openib-general] [PATCH 1/27] Update FMR functions
Message-ID: <1121089070.4389.4507.camel@hal.voltaire.com>

Change some functions to return void rather than an int since they are
always returning 0, thus making checking return values rather pointless.

Signed-off-by: Tom Duffy <tduffy at sun.com>
Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

-- 
 core/fmr_pool.c       |    7 +++--
 include/ib_fmr_pool.h |    5 +++--
 2 files changed, 6 insertions(+), 6 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband/core/fmr_pool.c linux-2.6.13-rc2-mm1/drivers/infiniband1/core/fmr_pool.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband/core/fmr_pool.c	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband1/core/fmr_pool.c	2005-06-28 13:07:35.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: fmr_pool.c 1349 2004-12-16 21:09:43Z roland $
+ * $Id: fmr_pool.c 2730 2005-06-28 16:43:03Z sean.hefty $
  */
 
 #include <linux/errno.h>
@@ -329,7 +330,7 @@ EXPORT_SYMBOL(ib_create_fmr_pool);
  *
  * Destroy an FMR pool and free all associated resources.
  */
-int ib_destroy_fmr_pool(struct ib_fmr_pool *pool)
+void ib_destroy_fmr_pool(struct ib_fmr_pool *pool)
 {
 	struct ib_pool_fmr *fmr;
 	struct ib_pool_fmr *tmp;
@@ -352,8 +353,6 @@ int ib_destroy_fmr_pool(struct ib_fmr_po
 
 	kfree(pool->cache_bucket);
 	kfree(pool);
-
-	return 0;
 }
 EXPORT_SYMBOL(ib_destroy_fmr_pool);
 
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband/include/ib_fmr_pool.h linux-2.6.13-rc2-mm1/drivers/infiniband1/include/ib_fmr_pool.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband/include/ib_fmr_pool.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband1/include/ib_fmr_pool.h	2005-06-28 13:07:34.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_fmr_pool.h 1349 2004-12-16 21:09:43Z roland $
+ * $Id: ib_fmr_pool.h 2730 2005-06-28 16:43:03Z sean.hefty $
  */
 
 #if !defined(IB_FMR_POOL_H)
@@ -78,7 +79,7 @@ struct ib_pool_fmr {
 struct ib_fmr_pool *ib_create_fmr_pool(struct ib_pd             *pd,
 				       struct ib_fmr_pool_param *params);
 
-int ib_destroy_fmr_pool(struct ib_fmr_pool *pool);
+void ib_destroy_fmr_pool(struct ib_fmr_pool *pool);
 
 int ib_flush_fmr_pool(struct ib_fmr_pool *pool);
 




From halr at voltaire.com  Mon Jul 11 06:45:08 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:45:08 -0400
Subject: [openib-general] [PATCH 2/27] Update MAD client API
Message-ID: <1121089074.4389.4509.camel@hal.voltaire.com>

Automatically allocate a MR when registering a MAD agent. 
MAD clients are modified to use this updated API.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

--- 
 core/agent.c      |   16 +--
 core/agent_priv.h |    3 +--
 core/mad.c        |   31 +++++++++++++++++++--
 core/sa_query.c   |   15 ++--
 include/ib_mad.h  |    1 +
 5 files changed, 24 insertions(+), 42 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband1/core/agent.c linux-2.6.13-rc2-mm1/drivers/infiniband2/core/agent.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband1/core/agent.c	2005-07-08 14:12:17.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband2/core/agent.c	2005-07-09 13:22:55.000000000 -0400
@@ -129,13 +129,12 @@ static int agent_mad_send(struct ib_mad_
 		goto out;
 	agent_send_wr->mad = mad_priv;
 
-	/* PCI mapping */
 	gather_list.addr = dma_map_single(mad_agent->device->dma_device,
 					  &mad_priv->mad,
 					  sizeof(mad_priv->mad),
 					  DMA_TO_DEVICE);
 	gather_list.length = sizeof(mad_priv->mad);
-	gather_list.lkey = (*port_priv->mr).lkey;
+	gather_list.lkey = mad_agent->mr->lkey;
 
 	send_wr.next = NULL;
 	send_wr.opcode = IB_WR_SEND;
@@ -261,7 +260,6 @@ static void agent_send_handler(struct ib
 	list_del(&agent_send_wr->send_list);
 	spin_unlock_irqrestore(&port_priv->send_list_lock, flags);
 
-	/* Unmap PCI */
 	dma_unmap_single(mad_agent->device->dma_device,
 			 pci_unmap_addr(agent_send_wr, mapping),
 			 sizeof(agent_send_wr->mad->mad),
@@ -324,22 +322,12 @@ int ib_agent_port_open(struct ib_device 
 		goto error3;
 	}
 
-	port_priv->mr = ib_get_dma_mr(port_priv->smp_agent->qp->pd,
-				      IB_ACCESS_LOCAL_WRITE);
-	if (IS_ERR(port_priv->mr)) {
-		printk(KERN_ERR SPFX "Couldn't get DMA MR\n");
-		ret = PTR_ERR(port_priv->mr);
-		goto error4;
-	}
-
 	spin_lock_irqsave(&ib_agent_port_list_lock, flags);
 	list_add_tail(&port_priv->port_list, &ib_agent_port_list);
 	spin_unlock_irqrestore(&ib_agent_port_list_lock, flags);
 
 	return 0;
 
-error4:
-	ib_unregister_mad_agent(port_priv->perf_mgmt_agent);
 error3:
 	ib_unregister_mad_agent(port_priv->smp_agent);
 error2:
@@ -363,8 +351,6 @@ int ib_agent_port_close(struct ib_device
 	list_del(&port_priv->port_list);
 	spin_unlock_irqrestore(&ib_agent_port_list_lock, flags);
 
-	ib_dereg_mr(port_priv->mr);
-
 	ib_unregister_mad_agent(port_priv->perf_mgmt_agent);
 	ib_unregister_mad_agent(port_priv->smp_agent);
 	kfree(port_priv);
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband1/core/agent_priv.h linux-2.6.13-rc2-mm1/drivers/infiniband2/core/agent_priv.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband1/core/agent_priv.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband2/core/agent_priv.h	2005-07-09 12:14:49.000000000 -0400
@@ -33,7 +33,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: agent_priv.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: agent_priv.h 1640 2005-01-24 22:39:02Z halr $
  */
 
 #ifndef __IB_AGENT_PRIV_H__
@@ -57,7 +57,6 @@ struct ib_agent_port_private {
 	int port_num;
 	struct ib_mad_agent *smp_agent;	      /* SM class */
 	struct ib_mad_agent *perf_mgmt_agent; /* PerfMgmt class */
-	struct ib_mr *mr;
 };
 
 #endif	/* __IB_AGENT_PRIV_H__ */
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband1/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband1/core/mad.c	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c	2005-07-09 12:58:23.000000000 -0400
@@ -261,19 +261,26 @@ struct ib_mad_agent *ib_register_mad_age
 		ret = ERR_PTR(-ENOMEM);
 		goto error1;
 	}
+	memset(mad_agent_priv, 0, sizeof *mad_agent_priv);
+
+	mad_agent_priv->agent.mr = ib_get_dma_mr(port_priv->qp_info[qpn].qp->pd,
+						 IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(mad_agent_priv->agent.mr)) {
+		ret = ERR_PTR(-ENOMEM);
+		goto error2;
+	}
 
 	if (mad_reg_req) {
 		reg_req = kmalloc(sizeof *reg_req, GFP_KERNEL);
 		if (!reg_req) {
 			ret = ERR_PTR(-ENOMEM);
-			goto error2;
+			goto error3;
 		}
 		/* Make a copy of the MAD registration request */
 		memcpy(reg_req, mad_reg_req, sizeof *reg_req);
 	}
 
 	/* Now, fill in the various structures */
-	memset(mad_agent_priv, 0, sizeof *mad_agent_priv);
 	mad_agent_priv->qp_info = &port_priv->qp_info[qpn];
 	mad_agent_priv->reg_req = reg_req;
 	mad_agent_priv->rmpp_version = rmpp_version;
@@ -301,7 +308,7 @@ struct ib_mad_agent *ib_register_mad_age
 				if (method) {
 					if (method_in_use(&method,
 							   mad_reg_req))
-						goto error3;
+						goto error4;
 				}
 			}
 			ret2 = add_nonoui_reg_req(mad_reg_req, mad_agent_priv,
@@ -317,14 +324,14 @@ struct ib_mad_agent *ib_register_mad_age
 					if (is_vendor_method_in_use(
 							vendor_class,
 							mad_reg_req))
-						goto error3;
+						goto error4;
 				}
 			}
 			ret2 = add_oui_reg_req(mad_reg_req, mad_agent_priv);
 		}
 		if (ret2) {
 			ret = ERR_PTR(ret2);
-			goto error3;
+			goto error4;
 		}
 	}
 
@@ -346,11 +353,13 @@ struct ib_mad_agent *ib_register_mad_age
 
 	return &mad_agent_priv->agent;
 
-error3:
+error4:
 	spin_unlock_irqrestore(&port_priv->reg_lock, flags);
 	kfree(reg_req);
-error2:
+error3:
 	kfree(mad_agent_priv);
+error2:
+	ib_dereg_mr(mad_agent_priv->agent.mr);
 error1:
 	return ret;
 }
@@ -487,18 +496,15 @@ static void unregister_mad_agent(struct 
 	 * MADs, preventing us from queuing additional work
 	 */
 	cancel_mads(mad_agent_priv);
-
 	port_priv = mad_agent_priv->qp_info->port_priv;
-
 	cancel_delayed_work(&mad_agent_priv->timed_work);
-	flush_workqueue(port_priv->wq);
 
 	spin_lock_irqsave(&port_priv->reg_lock, flags);
 	remove_mad_reg_req(mad_agent_priv);
 	list_del(&mad_agent_priv->agent_list);
 	spin_unlock_irqrestore(&port_priv->reg_lock, flags);
 
-	/* XXX: Cleanup pending RMPP receives for this agent */
+	flush_workqueue(port_priv->wq);
 
 	atomic_dec(&mad_agent_priv->refcount);
 	wait_event(mad_agent_priv->wait,
@@ -506,6 +512,7 @@ static void unregister_mad_agent(struct 
 
 	if (mad_agent_priv->reg_req)
 		kfree(mad_agent_priv->reg_req);
+	ib_dereg_mr(mad_agent_priv->agent.mr);
 	kfree(mad_agent_priv);
 }
 
@@ -750,7 +757,7 @@ static int handle_outgoing_dr_smp(struct
 	list_add_tail(&local->completion_list, &mad_agent_priv->local_list);
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 	queue_work(mad_agent_priv->qp_info->port_priv->wq,
-		  &mad_agent_priv->local_work);
+		   &mad_agent_priv->local_work);
 	ret = 1;
 out:
 	return ret;
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband1/core/sa_query.c linux-2.6.13-rc2-mm1/drivers/infiniband2/core/sa_query.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband1/core/sa_query.c	2005-07-10 16:21:40.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband2/core/sa_query.c	2005-07-10 16:21:55.000000000 -0400
@@ -77,7 +77,6 @@ struct ib_sa_sm_ah {
 
 struct ib_sa_port {
 	struct ib_mad_agent *agent;
-	struct ib_mr        *mr;
 	struct ib_sa_sm_ah  *sm_ah;
 	struct work_struct   update_task;
 	spinlock_t           ah_lock;
@@ -492,7 +491,7 @@ retry:
 					    sizeof (struct ib_sa_mad),
 					    DMA_TO_DEVICE);
 	gather_list.length = sizeof (struct ib_sa_mad);
-	gather_list.lkey   = port->mr->lkey;
+	gather_list.lkey   = port->agent->mr->lkey;
 	pci_unmap_addr_set(query, mapping, gather_list.addr);
 
 	ret = ib_post_send_mad(port->agent, &wr, &bad_wr);
@@ -780,7 +779,6 @@ static void ib_sa_add_one(struct ib_devi
 	sa_dev->end_port   = e;
 
 	for (i = 0; i <= e - s; ++i) {
-		sa_dev->port[i].mr       = NULL;
 		sa_dev->port[i].sm_ah    = NULL;
 		sa_dev->port[i].port_num = i + s;
 		spin_lock_init(&sa_dev->port[i].ah_lock);
@@ -792,13 +790,6 @@ static void ib_sa_add_one(struct ib_devi
 		if (IS_ERR(sa_dev->port[i].agent))
 			goto err;
 
-		sa_dev->port[i].mr = ib_get_dma_mr(sa_dev->port[i].agent->qp->pd,
-						   IB_ACCESS_LOCAL_WRITE);
-		if (IS_ERR(sa_dev->port[i].mr)) {
-			ib_unregister_mad_agent(sa_dev->port[i].agent);
-			goto err;
-		}
-
 		INIT_WORK(&sa_dev->port[i].update_task,
 			  update_sm_ah, &sa_dev->port[i]);
 	}
@@ -822,10 +813,8 @@ static void ib_sa_add_one(struct ib_devi
 	return;
 
 err:
-	while (--i >= 0) {
-		ib_dereg_mr(sa_dev->port[i].mr);
+	while (--i >= 0)
 		ib_unregister_mad_agent(sa_dev->port[i].agent);
-	}
 
 	kfree(sa_dev);
 
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband1/include/ib_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband2/include/ib_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband1/include/ib_mad.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband2/include/ib_mad.h	2005-07-09 12:06:49.000000000 -0400
@@ -180,6 +180,7 @@ typedef void (*ib_mad_recv_handler)(stru
 struct ib_mad_agent {
 	struct ib_device	*device;
 	struct ib_qp		*qp;
+	struct ib_mr		*mr;
 	ib_mad_recv_handler	recv_handler;
 	ib_mad_send_handler	send_handler;
 	ib_mad_snoop_handler	snoop_handler;




From halr at voltaire.com  Mon Jul 11 06:48:22 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:48:22 -0400
Subject: [openib-general] [PATCH 3/27] Add MAD helper functions
Message-ID: <1121089079.4389.4511.camel@hal.voltaire.com>

Add new helper routines for allocating MADs for sending and formatting
a send WR.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 2/27.

-- 
 core/mad.c       |   76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/ib_mad.h |   60 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 136 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c	2005-07-09 12:58:23.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c	2005-07-09 14:31:49.000000000 -0400
@@ -763,6 +763,82 @@ out:
 	return ret;
 }
 
+static int get_buf_length(int hdr_len, int data_len)
+{
+	int seg_size, pad;
+
+	seg_size = sizeof(struct ib_mad) - hdr_len;
+	if (data_len && seg_size) {
+		pad = seg_size - data_len % seg_size;
+		if (pad == seg_size)
+			pad = 0;
+	} else
+		pad = seg_size;
+	return hdr_len + data_len + pad;
+}
+
+struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
+					    u32 remote_qpn, u16 pkey_index,
+					    struct ib_ah *ah,
+					    int hdr_len, int data_len,
+					    int gfp_mask)
+{
+	struct ib_mad_agent_private *mad_agent_priv;
+	struct ib_mad_send_buf *send_buf;
+	int buf_size;
+	void *buf;
+
+	mad_agent_priv = container_of(mad_agent,
+				      struct ib_mad_agent_private, agent);
+	buf_size = get_buf_length(hdr_len, data_len);
+
+	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);
+	if (!buf)
+		return ERR_PTR(-ENOMEM);
+
+	send_buf = buf + buf_size;
+	memset(send_buf, 0, sizeof *send_buf);
+	send_buf->mad = buf;
+
+	send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device,
+					    buf, buf_size, DMA_TO_DEVICE);
+	pci_unmap_addr_set(send_buf, mapping, send_buf->sge.addr);
+	send_buf->sge.length = buf_size;
+	send_buf->sge.lkey = mad_agent->mr->lkey;
+
+	send_buf->send_wr.wr_id = (unsigned long) send_buf;
+	send_buf->send_wr.sg_list = &send_buf->sge;
+	send_buf->send_wr.num_sge = 1;
+	send_buf->send_wr.opcode = IB_WR_SEND;
+	send_buf->send_wr.send_flags = IB_SEND_SIGNALED;
+	send_buf->send_wr.wr.ud.ah = ah;
+	send_buf->send_wr.wr.ud.mad_hdr = &send_buf->mad->mad_hdr;
+	send_buf->send_wr.wr.ud.remote_qpn = remote_qpn;
+	send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY;
+	send_buf->send_wr.wr.ud.pkey_index = pkey_index;
+	send_buf->mad_agent = mad_agent;
+	atomic_inc(&mad_agent_priv->refcount);
+	return send_buf;
+}
+EXPORT_SYMBOL(ib_create_send_mad);
+
+void ib_free_send_mad(struct ib_mad_send_buf *send_buf)
+{
+	struct ib_mad_agent_private *mad_agent_priv;
+
+	mad_agent_priv = container_of(send_buf->mad_agent,
+				      struct ib_mad_agent_private, agent);
+
+	dma_unmap_single(send_buf->mad_agent->device->dma_device,
+			 pci_unmap_addr(send_buf, mapping),
+			 send_buf->sge.length, DMA_TO_DEVICE);
+	kfree(send_buf->mad);
+
+	if (atomic_dec_and_test(&mad_agent_priv->refcount))
+		wake_up(&mad_agent_priv->wait);
+}
+EXPORT_SYMBOL(ib_free_send_mad);
+
 static int ib_send_mad(struct ib_mad_agent_private *mad_agent_priv,
 		       struct ib_mad_send_wr_private *mad_send_wr)
 {
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband2/include/ib_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband3/include/ib_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband2/include/ib_mad.h	2005-07-09 12:06:49.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband3/include/ib_mad.h	2005-07-09 14:26:49.000000000 -0400
@@ -39,6 +39,8 @@
 #if !defined( IB_MAD_H )
 #define IB_MAD_H
 
+#include <linux/pci.h>
+
 #include <ib_verbs.h>
 
 /* Management base version */
@@ -73,6 +75,7 @@
 #define IB_QP0		0
 #define IB_QP1		__constant_htonl(1)
 #define IB_QP1_QKEY	0x80010000
+#define IB_QP_SET_QKEY	0x80000000
 
 struct ib_grh {
 	u32		version_tclass_flow;
@@ -124,6 +127,30 @@ struct ib_vendor_mad {
 	u8			data[216];
 } __attribute__ ((packed));
 
+/**
+ * ib_mad_send_buf - MAD data buffer and work request for sends.
+ * @mad: References an allocated MAD data buffer.  The size of the data
+ *   buffer is specified in the @send_wr.length field.
+ * @mapping: DMA mapping information.
+ * @mad_agent: MAD agent that allocated the buffer.
+ * @context: User-controlled context fields.
+ * @send_wr: An initialized work request structure used when sending the MAD.
+ *   The wr_id field of the work request is initialized to reference this
+ *   data structure.
+ * @sge: A scatter-gather list referenced by the work request.
+ *
+ * Users are responsible for initializing the MAD buffer itself, with the
+ * exception of specifying the payload length field in any RMPP MAD.
+ */
+struct ib_mad_send_buf {
+	struct ib_mad		*mad;
+	DECLARE_PCI_UNMAP_ADDR(mapping)
+	struct ib_mad_agent	*mad_agent;
+	void			*context[2];
+	struct ib_send_wr	send_wr;
+	struct ib_sge		sge;
+};
+
 struct ib_mad_agent;
 struct ib_mad_send_wc;
 struct ib_mad_recv_wc;
@@ -402,4 +429,37 @@ struct ib_mad_agent *ib_redirect_mad_qp(
 int ib_process_mad_wc(struct ib_mad_agent *mad_agent,
 		      struct ib_wc *wc);
 
+/**
+ * ib_create_send_mad - Allocate and initialize a data buffer and work request
+ *   for sending a MAD.
+ * @mad_agent: Specifies the registered MAD service to associate with the MAD.
+ * @remote_qpn: Specifies the QPN of the receiving node.
+ * @pkey_index: Specifies which PKey the MAD will be sent using.  This field
+ *   is valid only if the remote_qpn is QP 1.
+ * @ah: References the address handle used to transfer to the remote node.
+ * @hdr_len: Indicates the size of the data header of the MAD.  This length
+ *   should include the common MAD header, RMPP header, plus any class
+ *   specific header.
+ * @data_len: Indicates the size of any user-transfered data.  The call will
+ *   automatically adjust the allocated buffer size to account for any
+ *   additional padding that may be necessary.
+ * @gfp_mask: GFP mask used for the memory allocation.
+ *
+ * This is a helper routine that may be used to allocate a MAD.  Users are
+ * not required to allocate outbound MADs using this call.  The returned
+ * MAD send buffer will reference a data buffer usable for sending a MAD, along
+ * with an intialized work request structure.
+ */
+struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
+					    u32 remote_qpn, u16 pkey_index,
+					    struct ib_ah *ah,
+					    int hdr_len, int data_len,
+					    int gfp_mask);
+
+/**
+ * ib_free_send_mad - Returns data buffers used to send a MAD.
+ * @send_buf: Previously allocated send data buffer.
+ */
+void ib_free_send_mad(struct ib_mad_send_buf *send_buf);
+
 #endif /* IB_MAD_H */




From info at specialsuperservice.com  Mon Jul 11 05:27:06 2005
From: info at specialsuperservice.com (info at specialsuperservice.com)
Date: 11 Jul 2005 21:27:06 +0900
Subject: [openib-general] $B!yD69k2Z%W%l%<%s%H<B;\Cf!*!y(B
Message-ID: <20050711122706.11115.qmail@mail.specialsuperservice.com>

$BEv%5%$%H$O#2<~G/$r7^$(!"8=:_6H3&$K$*$$$FBgJQ0B?4$7$FMxMQ$G$-$k%5%$%H$H$7$F$N(B
$B8|$$?.MQ$r!"$*0~MM$G3MF@$9$k$3$H$,$G$-$^$7$?!#K\Ev$KM-Fq$&$4$6$$$^$9!#(B
$B$3$NEY!"%5%$%H$r%j%K%e!<%"%kCW$7$^$7$?$N$G@'Hs0lEY$*1[$72<$5$$$^$;!#(B
$B!~(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,!~(B
$B(-(B    $B!y!#!&!,!z(B   $B%j%K%e!<%"%k4k2hBh0lCF!!!y!#!&!,!z(B   $B!!(B  $B!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!(-(B
$B!~(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(B http://awg.webchu.com/?kensyo  $B!~(B

$B"'8=:_!V?75,$4EPO?!W$N3'MM$NCf$+$iCjA*$G(B100$BL>$NJ}$K9k2Z%W%l%<%s%H$rB#DhCf!*(B

1. Louis Vuitton $B%H%j%W%k%3%$%k(B
2. Louis Vuitton $B%@%_%((B $B%H%j%"%J(B
3. ROREX $B%7!<%I%%%(%i!<(B
[ W$B%A%c%s%9"v(B]
SONY NW-507L $B%"!<%:%k%V%k!<(B $B%a%b%j!<%*!<%G%#%*(B 1GB

$B!}$9$01~Jg"v"*(B http://awg.webchu.com/?kensyo

$B!,!}!#!y!,!#!~!!!y(B $B40A4L5NA%(%s%H%j!<8e!"!V(B10,000$B1_!WJ,L5NA%]%$%s%H!JMW(BTel$BG'>Z!KB#DhCf!*(B
$B!@%W%l%<%s%H!?(B $B(.(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(/(B
$B!!!@!@!!!?!?!!!V9k2Z%W%l%<%s%H!W$H!V??2F$N%(%C%AAj<j!W(BGET$B"v$O$3$3$G7h$^$j$G$9$M"v(B
$B!!(#(!(((!($!!!!!!!!(B               http://awg.webchu.com/?kensyo$B!!(B
$B!!('(B  $B=K(B   |$B!!(B $B(1(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(0(B
$B!!(&(!(*(!(%!!!!!y(B $B;n$7$FD:$$$F at dBPB;$O$"$j$^$;$s!*<+?.$r;}$C$F$*>)$a$7$^$9!*@'Hs$*;n$72<$5$$$^$;!#(B


$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"($=$NB>HqMQ$O0l at Z3]$+$i$J$$$3$H$rL at 8@$5$;$FD:$-$^$9$N$G0B?4$7$F$*;n$72<$5$$!#(B
$B"(L5NA%]%$%s%H>C2=8e!"5$$KF~$C$FD:$1$^$7$?$i!"<c43M-NA$H$J$j$^$9$,%]%$%s%H$r$49XF~2<$5$$$^$;!#(B
$B"(%5%$%H$H$7$F!V2q$($k$^$GJ]>Z!W$b<B;\$5$;$FD:$$$F$*$j$^$9!#AGE($J=P2q$$$r$"$J$?$X!#(B

------------------------------------------------------------------------------
$B""9XFI2r=|J}K!(B
$B9XFI2r=|$r$44uK>$NJ}$O!"$*<j?t$G$9$,2<5-(BURL$B08$K$4O"Mm$/$@$5$$!#(B
free_awg at mail.ru
==============================================================================
$B"((B18$B6X(B


From halr at voltaire.com  Mon Jul 11 06:51:35 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:51:35 -0400
Subject: [openib-general] [PATCH 4/27] Combine some MAD routines
Message-ID: <1121089083.4389.4513.camel@hal.voltaire.com>

Combine response_mad() and solicited_mad() routines into a single
function and simplify/encapsulate its usage.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 3/27.

-- 
 mad.c |  105 ++++++++++++++++--
 1 files changed, 27 insertions(+), 78 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband4/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c	2005-07-09 14:31:49.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband4/core/mad.c	2005-07-09 15:06:29.000000000 -0400
@@ -58,7 +58,7 @@ static int method_in_use(struct ib_mad_m
 static void remove_mad_reg_req(struct ib_mad_agent_private *priv);
 static struct ib_mad_agent_private *find_mad_agent(
 					struct ib_mad_port_private *port_priv,
-					struct ib_mad *mad, int solicited);
+					struct ib_mad *mad);
 static int ib_mad_post_receive_mads(struct ib_mad_qp_info *qp_info,
 				    struct ib_mad_private *mad);
 static void cancel_mads(struct ib_mad_agent_private *mad_agent_priv);
@@ -67,7 +67,6 @@ static void ib_mad_complete_send_wr(stru
 static void timeout_sends(void *data);
 static void cancel_sends(void *data);
 static void local_completions(void *data);
-static int solicited_mad(struct ib_mad *mad);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
 			      struct ib_mad_agent_private *agent_priv,
 			      u8 mgmt_class);
@@ -558,6 +557,13 @@ int ib_unregister_mad_agent(struct ib_ma
 }
 EXPORT_SYMBOL(ib_unregister_mad_agent);
 
+static inline int response_mad(struct ib_mad *mad)
+{
+	/* Trap represses are responses although response bit is reset */
+	return ((mad->mad_hdr.method == IB_MGMT_METHOD_TRAP_REPRESS) ||
+		(mad->mad_hdr.method & IB_MGMT_METHOD_RESP));
+}
+
 static void dequeue_mad(struct ib_mad_list_head *mad_list)
 {
 	struct ib_mad_queue *mad_queue;
@@ -650,7 +656,7 @@ static int handle_outgoing_dr_smp(struct
 				  struct ib_smp *smp,
 				  struct ib_send_wr *send_wr)
 {
-	int ret, solicited;
+	int ret;
 	unsigned long flags;
 	struct ib_mad_local_private *local;
 	struct ib_mad_private *mad_priv;
@@ -696,11 +702,7 @@ static int handle_outgoing_dr_smp(struct
 	switch (ret)
 	{
 	case IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY:
-		/*
-		 * See if response is solicited and
-		 * there is a recv handler
-		 */
-		if (solicited_mad(&mad_priv->mad.mad) &&
+		if (response_mad(&mad_priv->mad.mad) &&
 		    mad_agent_priv->agent.recv_handler) {
 			local->mad_priv = mad_priv;
 			local->recv_mad_agent = mad_agent_priv;
@@ -717,15 +719,13 @@ static int handle_outgoing_dr_smp(struct
 		break;
 	case IB_MAD_RESULT_SUCCESS:
 		/* Treat like an incoming receive MAD */
-		solicited = solicited_mad(&mad_priv->mad.mad);
 		port_priv = ib_get_mad_port(mad_agent_priv->agent.device,
 					    mad_agent_priv->agent.port_num);
 		if (port_priv) {
 			mad_priv->mad.mad.mad_hdr.tid =
 				((struct ib_mad *)smp)->mad_hdr.tid;
 			recv_mad_agent = find_mad_agent(port_priv,
-						       &mad_priv->mad.mad,
-							solicited);
+						        &mad_priv->mad.mad);
 		}
 		if (!port_priv || !recv_mad_agent) {
 			kmem_cache_free(ib_mad_cache, mad_priv);
@@ -1421,42 +1421,15 @@ out:
 	return;
 }
 
-static int response_mad(struct ib_mad *mad)
-{
-	/* Trap represses are responses although response bit is reset */
-	return ((mad->mad_hdr.method == IB_MGMT_METHOD_TRAP_REPRESS) ||
-		(mad->mad_hdr.method & IB_MGMT_METHOD_RESP));
-}
-
-static int solicited_mad(struct ib_mad *mad)
-{
-	/* CM MADs are never solicited */
-	if (mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_CM) {
-		return 0;
-	}
-
-	/* XXX: Determine whether MAD is using RMPP */
-
-	/* Not using RMPP */
-	/* Is this MAD a response to a previous MAD ? */
-	return response_mad(mad);
-}
-
 static struct ib_mad_agent_private *
 find_mad_agent(struct ib_mad_port_private *port_priv,
-	       struct ib_mad *mad,
-	       int solicited)
+	       struct ib_mad *mad)
 {
 	struct ib_mad_agent_private *mad_agent = NULL;
 	unsigned long flags;
 
 	spin_lock_irqsave(&port_priv->reg_lock, flags);
-
-	/*
-	 * Whether MAD was solicited determines type of routing to
-	 * MAD client.
-	 */
-	if (solicited) {
+	if (response_mad(mad)) {
 		u32 hi_tid;
 		struct ib_mad_agent_private *entry;
 
@@ -1560,18 +1533,6 @@ out:
 	return valid;
 }
 
-/*
- * Return start of fully reassembled MAD, or NULL, if MAD isn't assembled yet
- */
-static struct ib_mad_private *
-reassemble_recv(struct ib_mad_agent_private *mad_agent_priv,
-		struct ib_mad_private *recv)
-{
-	/* Until we have RMPP, all receives are reassembled!... */
-	INIT_LIST_HEAD(&recv->header.recv_wc.recv_buf.list);
-	return recv;
-}
-
 static struct ib_mad_send_wr_private*
 find_send_req(struct ib_mad_agent_private *mad_agent_priv,
 	      u64 tid)
@@ -1600,29 +1561,22 @@ find_send_req(struct ib_mad_agent_privat
 }
 
 static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
-				 struct ib_mad_private *recv,
-				 int solicited)
+				 struct ib_mad_recv_wc *mad_recv_wc)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 	struct ib_mad_send_wc mad_send_wc;
 	unsigned long flags;
+	u64 tid;
 
-	/* Fully reassemble receive before processing */
-	recv = reassemble_recv(mad_agent_priv, recv);
-	if (!recv) {
-		if (atomic_dec_and_test(&mad_agent_priv->refcount))
-			wake_up(&mad_agent_priv->wait);
-		return;
-	}
-
+	INIT_LIST_HEAD(&mad_recv_wc->recv_buf.list);
 	/* Complete corresponding request */
-	if (solicited) {
+	if (response_mad(mad_recv_wc->recv_buf.mad)) {
+		tid = mad_recv_wc->recv_buf.mad->mad_hdr.tid;
 		spin_lock_irqsave(&mad_agent_priv->lock, flags);
-		mad_send_wr = find_send_req(mad_agent_priv,
-					    recv->mad.mad.mad_hdr.tid);
+		mad_send_wr = find_send_req(mad_agent_priv, tid);
 		if (!mad_send_wr) {
 			spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-			ib_free_recv_mad(&recv->header.recv_wc);
+			ib_free_recv_mad(mad_recv_wc);
 			if (atomic_dec_and_test(&mad_agent_priv->refcount))
 				wake_up(&mad_agent_priv->wait);
 			return;
@@ -1632,10 +1586,9 @@ static void ib_mad_complete_recv(struct 
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		/* Defined behavior is to complete response before request */
-		recv->header.recv_wc.wc->wr_id = mad_send_wr->wr_id;
-		mad_agent_priv->agent.recv_handler(
-						&mad_agent_priv->agent,
-						&recv->header.recv_wc);
+		mad_recv_wc->wc->wr_id = mad_send_wr->wr_id;
+		mad_agent_priv->agent.recv_handler(&mad_agent_priv->agent,
+						   mad_recv_wc);
 		atomic_dec(&mad_agent_priv->refcount);
 
 		mad_send_wc.status = IB_WC_SUCCESS;
@@ -1643,9 +1596,8 @@ static void ib_mad_complete_recv(struct 
 		mad_send_wc.wr_id = mad_send_wr->wr_id;
 		ib_mad_complete_send_wr(mad_send_wr, &mad_send_wc);
 	} else {
-		mad_agent_priv->agent.recv_handler(
-						&mad_agent_priv->agent,
-						&recv->header.recv_wc);
+		mad_agent_priv->agent.recv_handler(&mad_agent_priv->agent,
+						   mad_recv_wc);
 		if (atomic_dec_and_test(&mad_agent_priv->refcount))
 			wake_up(&mad_agent_priv->wait);
 	}
@@ -1659,7 +1611,6 @@ static void ib_mad_recv_done_handler(str
 	struct ib_mad_private *recv, *response;
 	struct ib_mad_list_head *mad_list;
 	struct ib_mad_agent_private *mad_agent;
-	int solicited;
 
 	response = kmem_cache_alloc(ib_mad_cache, GFP_KERNEL);
 	if (!response)
@@ -1745,11 +1696,9 @@ local:
 		}
 	}
 
-	/* Determine corresponding MAD agent for incoming receive MAD */
-	solicited = solicited_mad(&recv->mad.mad);
-	mad_agent = find_mad_agent(port_priv, &recv->mad.mad, solicited);
+	mad_agent = find_mad_agent(port_priv, &recv->mad.mad);
 	if (mad_agent) {
-		ib_mad_complete_recv(mad_agent, recv, solicited);
+		ib_mad_complete_recv(mad_agent, &recv->header.recv_wc);
 		/*
 		 * recv is freed up in error cases in ib_mad_complete_recv
 		 * or via recv_handler in ib_mad_complete_recv()




From halr at voltaire.com  Mon Jul 11 06:54:49 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:54:49 -0400
Subject: [openib-general] [PATCH 5/27] Change saving of user's send wr_id in
	MAD
Message-ID: <1121089087.4389.4515.camel@hal.voltaire.com>

Move saving of user's send wr_id to better match layering of received
response handling.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 4/27.

-- 
 mad.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband4/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband5/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband4/core/mad.c	2005-07-09 15:06:29.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband5/core/mad.c	2005-07-09 15:08:31.000000000 -0400
@@ -847,9 +847,8 @@ static int ib_send_mad(struct ib_mad_age
 	unsigned long flags;
 	int ret;
 
-	/* Replace user's WR ID with our own to find WR upon completion */
+	/* Set WR ID to find mad_send_wr upon completion */
 	qp_info = mad_agent_priv->qp_info;
-	mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
 	mad_send_wr->send_wr.wr_id = (unsigned long)&mad_send_wr->mad_list;
 	mad_send_wr->mad_list.mad_queue = &qp_info->send_queue;
 
@@ -948,6 +947,7 @@ int ib_post_send_mad(struct ib_mad_agent
 		mad_send_wr->send_wr.sg_list = mad_send_wr->sg_list;
 		memcpy(mad_send_wr->sg_list, send_wr->sg_list,
 		       sizeof *send_wr->sg_list * send_wr->num_sge);
+		mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
 		mad_send_wr->send_wr.next = NULL;
 		mad_send_wr->tid = send_wr->wr.ud.mad_hdr->tid;
 		mad_send_wr->agent = mad_agent;




From halr at voltaire.com  Mon Jul 11 06:58:03 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 09:58:03 -0400
Subject: [openib-general] PATCH [6/27] Change ib_mad_send_wr_private struct
Message-ID: <1121089092.4389.4517.camel@hal.voltaire.com>

Have ib_mad_send_wr_private reference the private agent structure
directly, rather than the exposed agent definition.  Remove unneeded
parameters to functions and simplify code were possible from this
change.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 5/27.

-- 
 mad.c      |   22 ++++++++++--
 mad_priv.h |    4 ++--
 2 files changed, 12 insertions(+), 14 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband5/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband5/core/mad.c	2005-07-09 15:08:31.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad.c	2005-07-09 15:12:32.000000000 -0400
@@ -839,8 +839,7 @@ void ib_free_send_mad(struct ib_mad_send
 }
 EXPORT_SYMBOL(ib_free_send_mad);
 
-static int ib_send_mad(struct ib_mad_agent_private *mad_agent_priv,
-		       struct ib_mad_send_wr_private *mad_send_wr)
+static int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr)
 {
 	struct ib_mad_qp_info *qp_info;
 	struct ib_send_wr *bad_send_wr;
@@ -848,7 +847,7 @@ static int ib_send_mad(struct ib_mad_age
 	int ret;
 
 	/* Set WR ID to find mad_send_wr upon completion */
-	qp_info = mad_agent_priv->qp_info;
+	qp_info = mad_send_wr->mad_agent_priv->qp_info;
 	mad_send_wr->send_wr.wr_id = (unsigned long)&mad_send_wr->mad_list;
 	mad_send_wr->mad_list.mad_queue = &qp_info->send_queue;
 
@@ -857,7 +856,7 @@ static int ib_send_mad(struct ib_mad_age
 		list_add_tail(&mad_send_wr->mad_list.list,
 			      &qp_info->send_queue.list);
 		spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
-		ret = ib_post_send(mad_agent_priv->agent.qp,
+		ret = ib_post_send(mad_send_wr->mad_agent_priv->agent.qp,
 				   &mad_send_wr->send_wr, &bad_send_wr);
 		if (ret) {
 			printk(KERN_ERR PFX "ib_post_send failed: %d\n", ret);
@@ -950,7 +949,7 @@ int ib_post_send_mad(struct ib_mad_agent
 		mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
 		mad_send_wr->send_wr.next = NULL;
 		mad_send_wr->tid = send_wr->wr.ud.mad_hdr->tid;
-		mad_send_wr->agent = mad_agent;
+		mad_send_wr->mad_agent_priv = mad_agent_priv;
 		/* Timeout will be updated after send completes */
 		mad_send_wr->timeout = msecs_to_jiffies(send_wr->wr.
 							ud.timeout_ms);
@@ -966,7 +965,7 @@ int ib_post_send_mad(struct ib_mad_agent
 			      &mad_agent_priv->send_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
-		ret = ib_send_mad(mad_agent_priv, mad_send_wr);
+		ret = ib_send_mad(mad_send_wr);
 		if (ret) {
 			/* Fail send request */
 			spin_lock_irqsave(&mad_agent_priv->lock, flags);
@@ -1742,13 +1741,14 @@ static void adjust_timeout(struct ib_mad
 	}
 }
 
-static void wait_for_response(struct ib_mad_agent_private *mad_agent_priv,
-			      struct ib_mad_send_wr_private *mad_send_wr )
+static void wait_for_response(struct ib_mad_send_wr_private *mad_send_wr)
 {
+	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *temp_mad_send_wr;
 	struct list_head *list_item;
 	unsigned long delay;
 
+	mad_agent_priv = mad_send_wr->mad_agent_priv;
 	list_del(&mad_send_wr->agent_list);
 
 	delay = mad_send_wr->timeout;
@@ -1781,9 +1781,7 @@ static void ib_mad_complete_send_wr(stru
 	struct ib_mad_agent_private	*mad_agent_priv;
 	unsigned long			flags;
 
-	mad_agent_priv = container_of(mad_send_wr->agent,
-				      struct ib_mad_agent_private, agent);
-
+	mad_agent_priv = mad_send_wr->mad_agent_priv;
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
 	if (mad_send_wc->status != IB_WC_SUCCESS &&
 	    mad_send_wr->status == IB_WC_SUCCESS) {
@@ -1794,7 +1792,7 @@ static void ib_mad_complete_send_wr(stru
 	if (--mad_send_wr->refcount > 0) {
 		if (mad_send_wr->refcount == 1 && mad_send_wr->timeout &&
 		    mad_send_wr->status == IB_WC_SUCCESS) {
-			wait_for_response(mad_agent_priv, mad_send_wr);
+			wait_for_response(mad_send_wr);
 		}
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 		return;
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband5/core/mad_priv.h linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad_priv.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband5/core/mad_priv.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad_priv.h	2005-07-09 15:09:53.000000000 -0400
@@ -29,7 +29,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: mad_priv.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: mad_priv.h 1980 2005-03-11 22:33:53Z sean.hefty $
  */
 
 #ifndef __IB_MAD_PRIV_H__
@@ -116,7 +116,7 @@ struct ib_mad_snoop_private {
 struct ib_mad_send_wr_private {
 	struct ib_mad_list_head mad_list;
 	struct list_head agent_list;
-	struct ib_mad_agent *agent;
+	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_send_wr send_wr;
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */




From halr at voltaire.com  Mon Jul 11 07:01:17 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:01:17 -0400
Subject: [openib-general] PATCH [7/27] Fix timeout/cancelled MAD handling
Message-ID: <1121089096.4389.4519.camel@hal.voltaire.com>

Fixes an issue processing a sent MAD after it has timed out or 
been canceled.  The race occurs when a response MAD matches with
the send request.  The request could time out or be canceled after
the response MAD matches with the request, but before the request
completion can be processed.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 6/27.

-- 
 mad.c      |   14 ++++++++++++--
 mad_priv.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband7/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad.c	2005-07-09 15:12:32.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband7/core/mad.c	2005-07-09 16:48:45.000000000 -0400
@@ -341,6 +341,7 @@ struct ib_mad_agent *ib_register_mad_age
 	spin_lock_init(&mad_agent_priv->lock);
 	INIT_LIST_HEAD(&mad_agent_priv->send_list);
 	INIT_LIST_HEAD(&mad_agent_priv->wait_list);
+	INIT_LIST_HEAD(&mad_agent_priv->done_list);
 	INIT_WORK(&mad_agent_priv->timed_work, timeout_sends, mad_agent_priv);
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
@@ -1559,6 +1560,16 @@ find_send_req(struct ib_mad_agent_privat
 	return NULL;
 }
 
+static void ib_mark_req_done(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	mad_send_wr->timeout = 0;
+	if (mad_send_wr->refcount == 1) {
+		list_del(&mad_send_wr->agent_list);
+		list_add_tail(&mad_send_wr->agent_list,
+			      &mad_send_wr->mad_agent_priv->done_list);
+	}
+}
+
 static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
 				 struct ib_mad_recv_wc *mad_recv_wc)
 {
@@ -1580,8 +1591,7 @@ static void ib_mad_complete_recv(struct 
 				wake_up(&mad_agent_priv->wait);
 			return;
 		}
-		/* Timeout = 0 means that we won't wait for a response */
-		mad_send_wr->timeout = 0;
+		ib_mark_req_done(mad_send_wr);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		/* Defined behavior is to complete response before request */
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad_priv.h linux-2.6.13-rc2-mm1/drivers/infiniband7/core/mad_priv.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband6/core/mad_priv.h	2005-07-09 15:09:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband7/core/mad_priv.h	2005-07-09 16:48:05.000000000 -0400
@@ -92,6 +92,7 @@ struct ib_mad_agent_private {
 	spinlock_t lock;
 	struct list_head send_list;
 	struct list_head wait_list;
+	struct list_head done_list;
 	struct work_struct timed_work;
 	unsigned long timeout;
 	struct list_head local_list;




From halr at voltaire.com  Mon Jul 11 07:04:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:04:31 -0400
Subject: [openib-general] [PATCH 8/27] Minor cleanup during MAD startup and
	shutdown
Message-ID: <1121089102.4389.4521.camel@hal.voltaire.com>

Minor cleanup during startup and shutdown

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 7/27.

-- 
 mad.c |   44 +++++++++--
 1 files changed, 9 insertions(+), 35 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband7/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband8/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband7/core/mad.c	2005-07-09 16:48:45.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband8/core/mad.c	2005-07-09 16:51:21.000000000 -0400
@@ -2487,14 +2487,6 @@ static int ib_mad_port_open(struct ib_de
 	unsigned long flags;
 	char name[sizeof "ib_mad123"];
 
-	/* First, check if port already open at MAD layer */
-	port_priv = ib_get_mad_port(device, port_num);
-	if (port_priv) {
-		printk(KERN_DEBUG PFX "%s port %d already open\n",
-		       device->name, port_num);
-		return 0;
-	}
-
 	/* Create new device info */
 	port_priv = kmalloc(sizeof *port_priv, GFP_KERNEL);
 	if (!port_priv) {
@@ -2619,7 +2611,7 @@ static int ib_mad_port_close(struct ib_d
 
 static void ib_mad_init_device(struct ib_device *device)
 {
-	int ret, num_ports, cur_port, i, ret2;
+	int num_ports, cur_port, i;
 
 	if (device->node_type == IB_NODE_SWITCH) {
 		num_ports = 1;
@@ -2629,47 +2621,37 @@ static void ib_mad_init_device(struct ib
 		cur_port = 1;
 	}
 	for (i = 0; i < num_ports; i++, cur_port++) {
-		ret = ib_mad_port_open(device, cur_port);
-		if (ret) {
+		if (ib_mad_port_open(device, cur_port)) {
 			printk(KERN_ERR PFX "Couldn't open %s port %d\n",
 			       device->name, cur_port);
 			goto error_device_open;
 		}
-		ret = ib_agent_port_open(device, cur_port);
-		if (ret) {
+		if (ib_agent_port_open(device, cur_port)) {
 			printk(KERN_ERR PFX "Couldn't open %s port %d "
 			       "for agents\n",
 			       device->name, cur_port);
 			goto error_device_open;
 		}
 	}
-
-	goto error_device_query;
+	return;
 
 error_device_open:
 	while (i > 0) {
 		cur_port--;
-		ret2 = ib_agent_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_agent_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d "
 			       "for agents\n",
 			       device->name, cur_port);
-		}
-		ret2 = ib_mad_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_mad_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d\n",
 			       device->name, cur_port);
-		}
 		i--;
 	}
-
-error_device_query:
-	return;
 }
 
 static void ib_mad_remove_device(struct ib_device *device)
 {
-	int ret = 0, i, num_ports, cur_port, ret2;
+	int i, num_ports, cur_port;
 
 	if (device->node_type == IB_NODE_SWITCH) {
 		num_ports = 1;
@@ -2679,21 +2661,13 @@ static void ib_mad_remove_device(struct 
 		cur_port = 1;
 	}
 	for (i = 0; i < num_ports; i++, cur_port++) {
-		ret2 = ib_agent_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_agent_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d "
 			       "for agents\n",
 			       device->name, cur_port);
-			if (!ret)
-				ret = ret2;
-		}
-		ret2 = ib_mad_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_mad_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d\n",
 			       device->name, cur_port);
-			if (!ret)
-				ret = ret2;
-		}
 	}
 }
 




From halr at voltaire.com  Mon Jul 11 07:07:45 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:07:45 -0400
Subject: [openib-general] [PATCH 9/27] Add ib_coalesce_recv_mad to MAD
Message-ID: <1121089106.4389.4523.camel@hal.voltaire.com>

Add implementation for ib_coalesce_recv_mad. 
Also, clear allocated MAD data buffer in ib_create_send_mad.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 8/27.

-- 
 core/mad.c       |    9 +--
 include/ib_mad.h |    3 +--
 2 files changed, 2 insertions(+), 10 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband8/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband9/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband8/core/mad.c	2005-07-09 16:51:21.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband9/core/mad.c	2005-07-09 17:25:36.000000000 -0400
@@ -796,9 +796,9 @@ struct ib_mad_send_buf * ib_create_send_
 	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);
 	if (!buf)
 		return ERR_PTR(-ENOMEM);
+	memset(buf, 0, sizeof *send_buf + buf_size);
 
 	send_buf = buf + buf_size;
-	memset(send_buf, 0, sizeof *send_buf);
 	send_buf->mad = buf;
 
 	send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device,
@@ -1021,13 +1021,6 @@ void ib_free_recv_mad(struct ib_mad_recv
 }
 EXPORT_SYMBOL(ib_free_recv_mad);
 
-void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc,
-			  void *buf)
-{
-	printk(KERN_ERR PFX "ib_coalesce_recv_mad() not implemented yet\n");
-}
-EXPORT_SYMBOL(ib_coalesce_recv_mad);
-
 struct ib_mad_agent *ib_redirect_mad_qp(struct ib_qp *qp,
 					u8 rmpp_version,
 					ib_mad_send_handler send_handler,
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband8/include/ib_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband9/include/ib_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband8/include/ib_mad.h	2005-07-09 14:26:49.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband9/include/ib_mad.h	2005-07-09 16:59:22.000000000 -0400
@@ -365,8 +365,7 @@ int ib_post_send_mad(struct ib_mad_agent
  * This call copies a chain of received RMPP MADs into a single data buffer,
  * removing duplicated headers.
  */
-void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc,
-			  void *buf);
+void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf);
 
 /**
  * ib_free_recv_mad - Returns data buffers used to receive a MAD to the




From info at syutaro-mendo.com  Mon Jul 11 05:05:16 2005
From: info at syutaro-mendo.com (info at syutaro-mendo.com)
Date: 11 Jul 2005 21:05:16 +0900
Subject: [openib-general] $B!y!z0lK|1_J,%W%l%<%s%H!z!y(B
Message-ID: <20050711120516.21168.qmail@mail.syutaro-mendo.com>

http://awg.webchu.com/nanpara/?present1

$B=i$a$^$7$F!*!*FMA3$N%a!<%k$4LBOG$G$7$?$i$*5v$72<$5$$!#(B
$B?75,%5%$%HN)$A>e$2$KH<$$!"0lK|1_(B(1000$B%]%$%s%H(B)$BJ,%W%l%<%s%H$N$*CN$i$;$G$9!#(B

http://awg.webchu.com/nanpara/?present1

$B!X$^$?=P2q$$7O$+$!!Y$H;W$o$J$$$G2<$5$$!*!*!#(B
$B;dC#$b3'$5$s$HF1$8=P2q$$7O$K<:K>$7$F$$$?%f!<%6!<$G$7$?!#(B
$B!X$*6b$, at K$7$$$s$8$c$J$$!"0)$($J$$;v$,2y$7$$$s$@!*!Y(B
$B$=$3$G<+J,C#$G!XNI$$%5%$%H$r:n$m$&$8$c$J$$$+!*!Y$H;W$C$?Lu$G$9!#(B
$B0lK|1_(B(1000$B%]%$%s%H(B)$BJ,%W%l%<%s%H$O3'$5$s$X$N5$;}$A$G$9!#(B

http://awg.webchu.com/nanpara/?present1

$B;dC#$HNI$+$C$?:"$N=P2q$$7O$r<h$jLa$9$Y$-$G$9!*!#(B
$B at 5D>!"NI$$%9%]%s%5!<$,8+$D$+$i$:B?>/M-NA2=$H$J$C$?$N$,;DG0$G$9!#(B
$B%9%?%C%U0lF1!"3'$5$s$HD9$$$*IU$-9g$$$,$G$-$k;v$r4uK>$7$^$9!#(B

http://awg.webchu.com/nanpara/?present1


From halr at voltaire.com  Mon Jul 11 07:10:58 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:10:58 -0400
Subject: [openib-general] [PATCH 10/27] Add automatic retries to MAD layer
Message-ID: <1121089111.4389.4525.camel@hal.voltaire.com>

Add automatic retries to MAD layer. 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 9/27.

-- 
 core/mad.c         |   26 +++++++++++++++++++++++++-
 core/mad_priv.h    |    2 ++
 core/sa_query.c    |    3 ++-
 core/user_mad.c    |    1 +
 include/ib_verbs.h |    1 +
 5 files changed, 31 insertions(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband9/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband10/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband9/core/mad.c	2005-07-09 17:25:36.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband10/core/mad.c	2005-07-09 17:25:28.000000000 -0400
@@ -954,7 +954,7 @@ int ib_post_send_mad(struct ib_mad_agent
 		/* Timeout will be updated after send completes */
 		mad_send_wr->timeout = msecs_to_jiffies(send_wr->wr.
 							ud.timeout_ms);
-		mad_send_wr->retry = 0;
+		mad_send_wr->retries = mad_send_wr->send_wr.wr.ud.retries;
 		/* One reference for each work request to QP + response */
 		mad_send_wr->refcount = 1 + (mad_send_wr->timeout > 0);
 		mad_send_wr->status = IB_WC_SUCCESS;
@@ -2174,6 +2174,27 @@ local_send_completion:
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 }
 
+static int retry_send(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	int ret;
+
+	if (!mad_send_wr->retries--)
+		return -ETIMEDOUT;
+
+	mad_send_wr->timeout = msecs_to_jiffies(mad_send_wr->send_wr.
+						wr.ud.timeout_ms);
+
+	ret = ib_send_mad(mad_send_wr);
+
+	if (!ret) {
+		mad_send_wr->refcount++;
+		list_del(&mad_send_wr->agent_list);
+		list_add_tail(&mad_send_wr->agent_list,
+			      &mad_send_wr->mad_agent_priv->send_list);
+	}
+	return ret;
+}
+
 static void timeout_sends(void *data)
 {
 	struct ib_mad_agent_private *mad_agent_priv;
@@ -2202,6 +2223,9 @@ static void timeout_sends(void *data)
 			break;
 		}
 
+		if (!retry_send(mad_send_wr))
+			continue;
+
 		list_del(&mad_send_wr->agent_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband9/core/mad_priv.h linux-2.6.13-rc2-mm1/drivers/infiniband10/core/mad_priv.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband9/core/mad_priv.h	2005-07-09 16:48:05.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband10/core/mad_priv.h	2005-07-09 17:15:40.000000000 -0400
@@ -123,6 +123,7 @@ struct ib_mad_send_wr_private {
 	u64 wr_id;			/* client WR ID */
 	u64 tid;
 	unsigned long timeout;
+	int retries;
 	int retry;
 	int refcount;
 	enum ib_wc_status status;
@@ -136,6 +137,7 @@ struct ib_mad_local_private {
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */
 	u64 tid;
+	int retries;
 };
 
 struct ib_mad_mgmt_method_table {
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband9/core/sa_query.c linux-2.6.13-rc2-mm1/drivers/infiniband10/core/sa_query.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband9/core/sa_query.c	2005-07-10 16:21:55.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband10/core/sa_query.c	2005-07-10 16:22:13.000000000 -0400
@@ -462,7 +462,8 @@ static int send_mad(struct ib_sa_query *
 				 .mad_hdr     = &query->mad->mad_hdr,
 				 .remote_qpn  = 1,
 				 .remote_qkey = IB_QP1_QKEY,
-				 .timeout_ms  = timeout_ms
+				 .timeout_ms  = timeout_ms,
+				 .retries     = 0 
 			 }
 		 }
 	};
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband9/core/user_mad.c linux-2.6.13-rc2-mm1/drivers/infiniband10/core/user_mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband9/core/user_mad.c	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband10/core/user_mad.c	2005-07-09 17:14:46.000000000 -0400
@@ -322,6 +322,7 @@ static ssize_t ib_umad_write(struct file
 	wr.wr.ud.remote_qpn  = be32_to_cpu(packet->mad.qpn);
 	wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey);
 	wr.wr.ud.timeout_ms  = packet->mad.timeout_ms;
+	wr.wr.ud.retries     = 0;
 
 	wr.wr_id            = (unsigned long) packet;
 
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband9/include/ib_verbs.h linux-2.6.13-rc2-mm1/drivers/infiniband10/include/ib_verbs.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband9/include/ib_verbs.h	2005-07-10 11:11:43.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband10/include/ib_verbs.h	2005-07-10 10:55:59.000000000 -0400
@@ -566,6 +566,7 @@ struct ib_send_wr {
 			u32	remote_qpn;
 			u32	remote_qkey;
 			int	timeout_ms; /* valid for MADs only */
+			int	retries;    /* valid for MADs only */
 			u16	pkey_index; /* valid for GSI only */
 			u8	port_num;   /* valid for DR SMPs on switch only */
 		} ud;




From halr at voltaire.com  Mon Jul 11 07:14:11 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:14:11 -0400
Subject: [openib-general] [PATCH 11/27] Simplify calling of list_del in MAD
Message-ID: <1121089115.4389.4527.camel@hal.voltaire.com>

Simplify calling of list_del. 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 10/27.

-- 
 mad.c |    3 +--
 1 files changed, 1 insertion(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband10/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband11/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband10/core/mad.c	2005-07-09 17:25:28.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband11/core/mad.c	2005-07-09 17:41:24.000000000 -0400
@@ -2188,7 +2188,6 @@ static int retry_send(struct ib_mad_send
 
 	if (!ret) {
 		mad_send_wr->refcount++;
-		list_del(&mad_send_wr->agent_list);
 		list_add_tail(&mad_send_wr->agent_list,
 			      &mad_send_wr->mad_agent_priv->send_list);
 	}
@@ -2223,10 +2222,10 @@ static void timeout_sends(void *data)
 			break;
 		}
 
+		list_del(&mad_send_wr->agent_list);
 		if (!retry_send(mad_send_wr))
 			continue;
 
-		list_del(&mad_send_wr->agent_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		mad_send_wc.wr_id = mad_send_wr->wr_id;




From halr at voltaire.com  Mon Jul 11 07:17:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:17:25 -0400
Subject: [openib-general] [PATCH 12/27] Eliminate MAD cache leak associated
	with local completions
Message-ID: <1121089118.4389.4529.camel@hal.voltaire.com>

Eliminate MAD cache leak associated with local completions. 
Also, when canceling MAD, empty local completion list as well.

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 11/27.

-- 
 mad.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletion(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband11/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband12/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband11/core/mad.c	2005-07-09 17:41:24.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband12/core/mad.c	2005-07-09 17:46:10.000000000 -0400
@@ -1994,6 +1994,8 @@ static void cancel_mads(struct ib_mad_ag
 
 	/* Empty wait list to prevent receives from finding a request */
 	list_splice_init(&mad_agent_priv->wait_list, &cancel_list);
+	/* Empty local completion list as well */
+	list_splice_init(&mad_agent_priv->local_list, &cancel_list);
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 	/* Report all cancelled requests */
@@ -2108,6 +2110,7 @@ static void local_completions(void *data
 	struct ib_mad_local_private *local;
 	struct ib_mad_agent_private *recv_mad_agent;
 	unsigned long flags;
+	int recv = 0;
 	struct ib_wc wc;
 	struct ib_mad_send_wc mad_send_wc;
 
@@ -2123,10 +2126,10 @@ static void local_completions(void *data
 			recv_mad_agent = local->recv_mad_agent;
 			if (!recv_mad_agent) {
 				printk(KERN_ERR PFX "No receive MAD agent for local completion\n");
-				kmem_cache_free(ib_mad_cache, local->mad_priv);
 				goto local_send_completion;
 			}
 
+			recv = 1;
 			/*
 			 * Defined behavior is to complete response
 			 * before request
@@ -2169,6 +2172,8 @@ local_send_completion:
 		spin_lock_irqsave(&mad_agent_priv->lock, flags);
 		list_del(&local->completion_list);
 		atomic_dec(&mad_agent_priv->refcount);
+		if (!recv)
+			kmem_cache_free(ib_mad_cache, local->mad_priv);
 		kfree(local);
 	}
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);




From halr at voltaire.com  Mon Jul 11 07:20:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:20:39 -0400
Subject: [openib-general] [PATCH 13/27] Add ib_modify_mad API to MAD
Message-ID: <1121089121.4389.4531.camel@hal.voltaire.com>

Add new MAD layer call to modify (ib_modify_mad) the timeout of a sent 
MAD, and simplify cancel code.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 12/27.

-- 
 core/mad.c       |   83 ++++++++++++++++++--
 core/mad_priv.h  |    2 -
 include/ib_mad.h |   14 +++++++--
 3 files changed, 40 insertions(+), 59 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband12/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband13/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband12/core/mad.c	2005-07-09 17:46:10.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband13/core/mad.c	2005-07-09 18:16:36.000000000 -0400
@@ -65,7 +65,6 @@ static void cancel_mads(struct ib_mad_ag
 static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
 				    struct ib_mad_send_wc *mad_send_wc);
 static void timeout_sends(void *data);
-static void cancel_sends(void *data);
 static void local_completions(void *data);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
 			      struct ib_mad_agent_private *agent_priv,
@@ -346,8 +345,6 @@ struct ib_mad_agent *ib_register_mad_age
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
 		   mad_agent_priv);
-	INIT_LIST_HEAD(&mad_agent_priv->canceled_list);
-	INIT_WORK(&mad_agent_priv->canceled_work, cancel_sends, mad_agent_priv);
 	atomic_set(&mad_agent_priv->refcount, 1);
 	init_waitqueue_head(&mad_agent_priv->wait);
 
@@ -1775,6 +1772,13 @@ static void wait_for_response(struct ib_
 	}
 }
 
+void ib_reset_mad_timeout(struct ib_mad_send_wr_private *mad_send_wr,
+			  int timeout_ms)
+{
+	mad_send_wr->timeout = msecs_to_jiffies(timeout_ms);
+	wait_for_response(mad_send_wr);
+}
+
 /*
  * Process a send work completion
  */
@@ -2034,41 +2038,7 @@ find_send_by_wr_id(struct ib_mad_agent_p
 	return NULL;
 }
 
-void cancel_sends(void *data)
-{
-	struct ib_mad_agent_private *mad_agent_priv;
-	struct ib_mad_send_wr_private *mad_send_wr;
-	struct ib_mad_send_wc mad_send_wc;
-	unsigned long flags;
-
-	mad_agent_priv = data;
-
-	mad_send_wc.status = IB_WC_WR_FLUSH_ERR;
-	mad_send_wc.vendor_err = 0;
-
-	spin_lock_irqsave(&mad_agent_priv->lock, flags);
-	while (!list_empty(&mad_agent_priv->canceled_list)) {
-		mad_send_wr = list_entry(mad_agent_priv->canceled_list.next,
-					 struct ib_mad_send_wr_private,
-					 agent_list);
-
-		list_del(&mad_send_wr->agent_list);
-		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-
-		mad_send_wc.wr_id = mad_send_wr->wr_id;
-		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
-						   &mad_send_wc);
-
-		kfree(mad_send_wr);
-		if (atomic_dec_and_test(&mad_agent_priv->refcount))
-			wake_up(&mad_agent_priv->wait);
-		spin_lock_irqsave(&mad_agent_priv->lock, flags);
-	}
-	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-}
-
-void ib_cancel_mad(struct ib_mad_agent *mad_agent,
-		  u64 wr_id)
+int ib_modify_mad(struct ib_mad_agent *mad_agent, u64 wr_id, u32 timeout_ms)
 {
 	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *mad_send_wr;
@@ -2078,29 +2048,30 @@ void ib_cancel_mad(struct ib_mad_agent *
 				      agent);
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
 	mad_send_wr = find_send_by_wr_id(mad_agent_priv, wr_id);
-	if (!mad_send_wr) {
+	if (!mad_send_wr || mad_send_wr->status != IB_WC_SUCCESS) {
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-		goto out;
+		return -EINVAL;
 	}
 
-	if (mad_send_wr->status == IB_WC_SUCCESS)
-		mad_send_wr->refcount -= (mad_send_wr->timeout > 0);
-
-	if (mad_send_wr->refcount != 0) {
+	if (!timeout_ms) {
 		mad_send_wr->status = IB_WC_WR_FLUSH_ERR;
-		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-		goto out;
+		mad_send_wr->refcount -= (mad_send_wr->timeout > 0);
 	}
 
-	list_del(&mad_send_wr->agent_list);
-	list_add_tail(&mad_send_wr->agent_list, &mad_agent_priv->canceled_list);
-	adjust_timeout(mad_agent_priv);
+	mad_send_wr->send_wr.wr.ud.timeout_ms = timeout_ms;
+	if (!mad_send_wr->timeout || mad_send_wr->refcount > 1)
+		mad_send_wr->timeout = msecs_to_jiffies(timeout_ms);
+	else
+		ib_reset_mad_timeout(mad_send_wr, timeout_ms);
+
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
+	return 0;
+}
+EXPORT_SYMBOL(ib_modify_mad);
 
-	queue_work(mad_agent_priv->qp_info->port_priv->wq,
-		   &mad_agent_priv->canceled_work);
-out:
-	return;
+void ib_cancel_mad(struct ib_mad_agent *mad_agent, u64 wr_id)
+{
+	ib_modify_mad(mad_agent, wr_id, 0);
 }
 EXPORT_SYMBOL(ib_cancel_mad);
 
@@ -2207,8 +2178,6 @@ static void timeout_sends(void *data)
 	unsigned long flags, delay;
 
 	mad_agent_priv = (struct ib_mad_agent_private *)data;
-
-	mad_send_wc.status = IB_WC_RESP_TIMEOUT_ERR;
 	mad_send_wc.vendor_err = 0;
 
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
@@ -2233,6 +2202,10 @@ static void timeout_sends(void *data)
 
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
+		if (mad_send_wr->status == IB_WC_SUCCESS)
+			mad_send_wc.status = IB_WC_RESP_TIMEOUT_ERR;
+		else
+			mad_send_wc.status = mad_send_wr->status;
 		mad_send_wc.wr_id = mad_send_wr->wr_id;
 		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
 						   &mad_send_wc);
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband12/core/mad_priv.h linux-2.6.13-rc2-mm1/drivers/infiniband13/core/mad_priv.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband12/core/mad_priv.h	2005-07-09 17:15:40.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband13/core/mad_priv.h	2005-07-09 17:57:55.000000000 -0400
@@ -97,8 +97,6 @@ struct ib_mad_agent_private {
 	unsigned long timeout;
 	struct list_head local_list;
 	struct work_struct local_work;
-	struct list_head canceled_list;
-	struct work_struct canceled_work;
 
 	atomic_t refcount;
 	wait_queue_head_t wait;
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband12/include/ib_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband13/include/ib_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband12/include/ib_mad.h	2005-07-09 16:59:22.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband13/include/ib_mad.h	2005-07-09 17:57:11.000000000 -0400
@@ -385,8 +385,18 @@ void ib_free_recv_mad(struct ib_mad_recv
  * MADs will be returned to the user through the corresponding
  * ib_mad_send_handler.
  */
-void ib_cancel_mad(struct ib_mad_agent *mad_agent,
-		   u64 wr_id);
+void ib_cancel_mad(struct ib_mad_agent *mad_agent, u64 wr_id);
+
+/**
+ * ib_modify_mad - Modifies an outstanding send MAD operation.
+ * @mad_agent: Specifies the registration associated with sent MAD.
+ * @wr_id: Indicates the work request identifier of the MAD to modify.
+ * @timeout_ms: New timeout value for sent MAD.
+ *
+ * This call will reset the timeout value for a sent MAD to the specified
+ * value.
+ */
+int ib_modify_mad(struct ib_mad_agent *mad_agent, u64 wr_id, u32 timeout_ms);
 
 /**
  * ib_redirect_mad_qp - Registers a QP for MAD services.




From halr at voltaire.com  Mon Jul 11 07:23:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:23:53 -0400
Subject: [openib-general] [PATCH 14/27] Optimize canceling a MAD
Message-ID: <1121089126.4389.4533.camel@hal.voltaire.com>

Optimize canceling a MAD. 
- Eliminate searching timeout list in cancel case.
- Remove duplicate calls to queue work item.
- Eliminate resending a MAD before MAD is completed.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 13/27.

-- 
 mad.c |   21 +++++++++++++--
 1 files changed, 13 insertions(+), 8 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband13/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband14/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband13/core/mad.c	2005-07-09 18:16:36.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband14/core/mad.c	2005-07-09 18:19:51.000000000 -0400
@@ -1754,14 +1754,18 @@ static void wait_for_response(struct ib_
 	delay = mad_send_wr->timeout;
 	mad_send_wr->timeout += jiffies;
 
-	list_for_each_prev(list_item, &mad_agent_priv->wait_list) {
-		temp_mad_send_wr = list_entry(list_item,
-					      struct ib_mad_send_wr_private,
-					      agent_list);
-		if (time_after(mad_send_wr->timeout,
-			       temp_mad_send_wr->timeout))
-			break;
+	if (delay) {
+		list_for_each_prev(list_item, &mad_agent_priv->wait_list) {
+			temp_mad_send_wr = list_entry(list_item,
+						struct ib_mad_send_wr_private,
+						agent_list);
+			if (time_after(mad_send_wr->timeout,
+				       temp_mad_send_wr->timeout))
+				break;
+		}
 	}
+	else
+		list_item = &mad_agent_priv->wait_list;
 	list_add(&mad_send_wr->agent_list, list_item);
 
 	/* Reschedule a work item if we have a shorter timeout */
@@ -2197,7 +2201,8 @@ static void timeout_sends(void *data)
 		}
 
 		list_del(&mad_send_wr->agent_list);
-		if (!retry_send(mad_send_wr))
+		if (mad_send_wr->status == IB_WC_SUCCESS &&
+		    !retry_send(mad_send_wr))
 			continue;
 
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);




From adobriyan at gmail.com  Mon Jul 11 07:39:41 2005
From: adobriyan at gmail.com (Alexey Dobriyan)
Date: Mon, 11 Jul 2005 18:39:41 +0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <1121089079.4389.4511.camel@hal.voltaire.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
Message-ID: <200507111839.41807.adobriyan@gmail.com>

On Monday 11 July 2005 17:48, Hal Rosenstock wrote:
> Add new helper routines for allocating MADs for sending and formatting
> a send WR.

> -- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
> +++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
				   ^^^^^^^^^^^
Ick. You'd better have linux-2.6.13-rc2-mm1-[0123...].

> +struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
> +					    u32 remote_qpn, u16 pkey_index,
> +					    struct ib_ah *ah,
> +					    int hdr_len, int data_len,
> +					    int gfp_mask)

unsigned int __nocast gfp_mask, please. 430 or so infiniband sparse warnings
is not a reason to add more.

> +{

> +	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);


From halr at voltaire.com  Mon Jul 11 07:27:06 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:27:06 -0400
Subject: [openib-general] [PATCH 15/27] Fix a couple of MAD code paths
Message-ID: <1121089129.4389.4535.camel@hal.voltaire.com>

Fixed locking to handle error posting MAD send work requests. 
Fixed handling canceling a MAD with an active work request.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 14/27.

-- 
 mad.c |   28 ++++++++++++++--
 1 files changed, 14 insertions(+), 14 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband14/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband15/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband14/core/mad.c	2005-07-09 18:19:51.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband15/core/mad.c	2005-07-09 18:24:43.000000000 -0400
@@ -841,6 +841,7 @@ static int ib_send_mad(struct ib_mad_sen
 {
 	struct ib_mad_qp_info *qp_info;
 	struct ib_send_wr *bad_send_wr;
+	struct list_head *list;
 	unsigned long flags;
 	int ret;
 
@@ -850,22 +851,20 @@ static int ib_send_mad(struct ib_mad_sen
 	mad_send_wr->mad_list.mad_queue = &qp_info->send_queue;
 
 	spin_lock_irqsave(&qp_info->send_queue.lock, flags);
-	if (qp_info->send_queue.count++ < qp_info->send_queue.max_active) {
-		list_add_tail(&mad_send_wr->mad_list.list,
-			      &qp_info->send_queue.list);
-		spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
+	if (qp_info->send_queue.count < qp_info->send_queue.max_active) {
 		ret = ib_post_send(mad_send_wr->mad_agent_priv->agent.qp,
 				   &mad_send_wr->send_wr, &bad_send_wr);
-		if (ret) {
-			printk(KERN_ERR PFX "ib_post_send failed: %d\n", ret);
-			dequeue_mad(&mad_send_wr->mad_list);
-		}
+		list = &qp_info->send_queue.list;
 	} else {
-		list_add_tail(&mad_send_wr->mad_list.list,
-			      &qp_info->overflow_list);
-		spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
 		ret = 0;
+		list = &qp_info->overflow_list;
 	}
+
+	if (!ret) {
+		qp_info->send_queue.count++;
+		list_add_tail(&mad_send_wr->mad_list.list, list);
+	}
+	spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
 	return ret;
 }
 
@@ -2023,8 +2022,7 @@ static void cancel_mads(struct ib_mad_ag
 }
 
 static struct ib_mad_send_wr_private*
-find_send_by_wr_id(struct ib_mad_agent_private *mad_agent_priv,
-		   u64 wr_id)
+find_send_by_wr_id(struct ib_mad_agent_private *mad_agent_priv, u64 wr_id)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 
@@ -2047,6 +2045,7 @@ int ib_modify_mad(struct ib_mad_agent *m
 	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *mad_send_wr;
 	unsigned long flags;
+	int active;
 
 	mad_agent_priv = container_of(mad_agent, struct ib_mad_agent_private,
 				      agent);
@@ -2057,13 +2056,14 @@ int ib_modify_mad(struct ib_mad_agent *m
 		return -EINVAL;
 	}
 
+	active = (!mad_send_wr->timeout || mad_send_wr->refcount > 1);
 	if (!timeout_ms) {
 		mad_send_wr->status = IB_WC_WR_FLUSH_ERR;
 		mad_send_wr->refcount -= (mad_send_wr->timeout > 0);
 	}
 
 	mad_send_wr->send_wr.wr.ud.timeout_ms = timeout_ms;
-	if (!mad_send_wr->timeout || mad_send_wr->refcount > 1)
+	if (active)
 		mad_send_wr->timeout = msecs_to_jiffies(timeout_ms);
 	else
 		ib_reset_mad_timeout(mad_send_wr, timeout_ms);




From halr at voltaire.com  Mon Jul 11 07:30:19 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:30:19 -0400
Subject: [openib-general] [PATCH 16/27] Add ib_create_ah_from_wc to IB verbs
Message-ID: <1121089135.4389.4537.camel@hal.voltaire.com>

Added new call: ib_create_ah_from_wc. Call will allocate an 
address handle given work completion information, including any
received GRH.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 15/27.

-- 
 core/verbs.c       |   35 +++++++++++++++++++++++++++++++++++
 include/ib_mad.h   |    9 --
 include/ib_verbs.h |   24 ++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 9 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband15/core/verbs.c linux-2.6.13-rc2-mm1/drivers/infiniband16/core/verbs.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband15/core/verbs.c	2005-07-10 11:09:37.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband16/core/verbs.c	2005-07-10 11:43:44.000000000 -0400
@@ -41,6 +41,7 @@
 #include <linux/err.h>
 
 #include <ib_verbs.h>
+#include <ib_cache.h>
 
 /* Protection domains */
 
@@ -88,6 +89,40 @@ struct ib_ah *ib_create_ah(struct ib_pd 
 }
 EXPORT_SYMBOL(ib_create_ah);
 
+struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, struct ib_wc *wc,
+				   struct ib_grh *grh, u8 port_num)
+{
+	struct ib_ah_attr ah_attr;
+	u32 flow_class;
+	u16 gid_index;
+	int ret;
+
+	memset(&ah_attr, 0, sizeof ah_attr);
+	ah_attr.dlid = wc->slid;
+	ah_attr.sl = wc->sl;
+	ah_attr.src_path_bits = wc->dlid_path_bits;
+	ah_attr.port_num = port_num;
+	
+	if (wc->wc_flags & IB_WC_GRH) {
+		ah_attr.ah_flags = IB_AH_GRH;
+		ah_attr.grh.dgid = grh->dgid;
+
+		ret = ib_find_cached_gid(pd->device, &grh->sgid, &port_num,
+					 &gid_index);
+		if (ret)
+			return ERR_PTR(ret);
+
+		ah_attr.grh.sgid_index = (u8) gid_index;
+		flow_class = be32_to_cpu(&grh->version_tclass_flow);
+		ah_attr.grh.flow_label = flow_class & 0xFFFFF;
+		ah_attr.grh.traffic_class = (flow_class >> 20) & 0xFF;
+		ah_attr.grh.hop_limit = grh->hop_limit;
+	}
+
+	return ib_create_ah(pd, &ah_attr);
+}
+EXPORT_SYMBOL(ib_create_ah_from_wc);
+
 int ib_modify_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr)
 {
 	return ah->device->modify_ah ?
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband15/include/ib_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband16/include/ib_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband15/include/ib_mad.h	2005-07-09 17:57:11.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband16/include/ib_mad.h	2005-07-10 11:43:45.000000000 -0400
@@ -77,15 +77,6 @@
 #define IB_QP1_QKEY	0x80010000
 #define IB_QP_SET_QKEY	0x80000000
 
-struct ib_grh {
-	u32		version_tclass_flow;
-	u16		paylen;
-	u8		next_hdr;
-	u8		hop_limit;
-	union ib_gid	sgid;
-	union ib_gid	dgid;
-} __attribute__ ((packed));
-
 struct ib_mad_hdr {
 	u8	base_version;
 	u8	mgmt_class;
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband15/include/ib_verbs.h linux-2.6.13-rc2-mm1/drivers/infiniband16/include/ib_verbs.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband15/include/ib_verbs.h	2005-07-10 10:55:59.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband16/include/ib_verbs.h	2005-07-10 11:43:45.000000000 -0400
@@ -289,6 +289,15 @@ struct ib_global_route {
 	u8		traffic_class;
 };
 
+struct ib_grh {
+	u32		version_tclass_flow;
+	u16		paylen;
+	u8		next_hdr;
+	u8		hop_limit;
+	union ib_gid	sgid;
+	union ib_gid	dgid;
+};
+
 enum {
 	IB_MULTICAST_QPN = 0xffffff
 };
@@ -991,6 +1000,21 @@ int ib_dealloc_pd(struct ib_pd *pd);
 struct ib_ah *ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr);
 
 /**
+ * ib_create_ah_from_wc - Creates an address handle associated with the
+ *   sender of the specified work completion.
+ * @pd: The protection domain associated with the address handle.
+ * @wc: Work completion information associated with a received message.
+ * @grh: References the received global route header.  This parameter is
+ *   ignored unless the work completion indicates that the GRH is valid.
+ * @port_num: The outbound port number to associate with the address.
+ *
+ * The address handle is used to reference a local or global destination
+ * in all UD QP post sends.
+ */
+struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, struct ib_wc *wc,
+				   struct ib_grh *grh, u8 port_num);
+
+/**
  * ib_modify_ah - Modifies the address vector associated with an address
  *   handle.
  * @ah: The address handle to modify.




From jlentini at netapp.com  Mon Jul 11 07:38:32 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 11 Jul 2005 10:38:32 -0400 (EDT)
Subject: [openib-general] how to choose appropriate ULPs for application
In-Reply-To: <2322.159.226.40.170.1120658965.squirrel@webmail.ict.ac.cn>
References: <8E9D028761D8264D910612167E8457E8742D7B@mail2.ammasso.com>
	<2322.159.226.40.170.1120658965.squirrel@webmail.ict.ac.cn>
Message-ID: <Pine.LNX.4.61.0507111035120.25603@jlentini-linux.nane.netapp.com>


FYI, you can find a kDAPL implementation for OpenIB at

svn/gen2/users/jlentini/linux-kernel

and a uDAPL implementation for OpenIB at

svn/gen2/users/jlentini/userspace

I can answer any questions you might have about these ULPs.

I'm not familiar with NAL, so I can't comment on which ULP is right 
for you.

On Wed, 6 Jul 2005, wangxigui wrote:

> Thank you very much!
> I'll read the materials that you supply.
>
>     I have not described my plan clearly. I don't konw how to run lustre
> on IB. Neither do I konw the lustre running in kernel space or user
> space. Now the lustre NAL does not supports IB vapi. If I use IPoIB,
> it works. But it doen not take take advantage of RDMA. Supporse
> lustre NAL invoke SDP or uDAP, RDMA is enable.
>
>    Invocation of Verb API to realize lustre NAL is my target. The SDP or
> uDAP is first step.
>
> Clem Cole said:
>> The quick answer here is none of SDP, iSER/SRP or uDAPL for different
>> reasons on each case
>>
>> kDAPL is the kernel API that is designed to handle a kernel application
>> such as NFS, Lustre or the like.  See:
>> 	http://www.datcollaborative.org/kdapl.html
>>
>> There is are already implementations of NFS/RDMA available:
>> 	http://www.sourceforge.net/projects/nfs-rdma
>>
>> This should give you some hints on how to a distrubuted file system
>> uses the kDAPL interface.
>>
>>
>>
>> -----Original Message-----
>> From: openib-general-bounces at openib.org
>> [mailto:openib-general-bounces at openib.org]On Behalf Of wxg at ict.ac.cn
>> Sent: Wednesday, July 06, 2005 9:26 AM
>> To: openib-general
>> Subject: [openib-general] how to choose appropriate ULPs for application
>>
>>
>> Hello！
>>
>>     We plan to run lustre on InfiniBand. But now lustre does not support
>> IB.
>> I know noting about the IB stack. So I think the IB Upper Layer
>> Protocol(SDP, DAP) could be used to transport data for lustre.
>> 	But there are too many ULPs over IB. I know little about them. As far as
>> I konw, iSER/SRP is for SCSI command transportation. So, it seems both
>> SDP and uDAP are OK for my plan. I cant tell the difference between them.
>> Which one is better, SDP or DAP? I don't konw how to choose?
>>
>> 	Any suggestion is appreciated!
>>
>> 　　
>>
>> 　　　　　　　　致
>> 礼！
>>
>> Sincerely yours
>> Wang Xigui 王锡贵
>> 2005-07-06
>> --------------------
>> Computer Architecture Laboratory
>> Institute of Computing Technology
>> Chinese Academy of Sciences
>> Beijing,P.R.China
>> Zip code: 100080
>> Tel: +86-10-62564394(office)
>> Email: wxg at ict.ac.cn
>> 　　　　　　　　　　　　　　
>> 　　　　　　　　　　　　　　
>> 　　　　　　　　　　　　　　　　　
>>
>
>
> -- 
> Sincerely yours
> Wang Xigui 王锡贵
> 2005-07-06
> --------------------
> Computer Architecture Laboratory
> Institute of Computing Technology
> Chinese Academy of Sciences
> Beijing,P.R.China
> Zip code: 100080
> Tel: +86-10-62565533-9314(office)
> Email: wxg at ict.ac.cn
>
> -- 
> Sincerely yours
> Wang Xigui 王锡贵
> 2005-07-06
> --------------------
> Computer Architecture Laboratory
> Institute of Computing Technology
> Chinese Academy of Sciences
> Beijing,P.R.China
> Zip code: 100080
> Tel: +86-10-62565533-9314(office)
> Email: wxg at ict.ac.cn
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>

From halr at voltaire.com  Mon Jul 11 07:33:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:33:33 -0400
Subject: [openib-general] [PATCH 17/27] A couple of IB core bug fixes
Message-ID: <1121089138.4389.4539.camel@hal.voltaire.com>

Replace be32_to_cpup with be32_to_cpu and fix bug referencing pointer 
rather than value in ib_create_ah_from_wc().

Signed-off-by: Tom Duffy <tduffy at sun.com>
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 16/27.

-- 
 agent.c |    8 ++++--
 verbs.c |    2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband16/core/agent.c linux-2.6.13-rc2-mm1/drivers/infiniband17/core/agent.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband16/core/agent.c	2005-07-09 13:22:55.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband17/core/agent.c	2005-07-10 11:50:26.000000000 -0400
@@ -156,10 +156,10 @@ static int agent_mad_send(struct ib_mad_
 			/* Should sgid be looked up ? */
 			ah_attr.grh.sgid_index = 0;
 			ah_attr.grh.hop_limit = grh->hop_limit;
-			ah_attr.grh.flow_label = be32_to_cpup(
-				&grh->version_tclass_flow)  & 0xfffff;
-			ah_attr.grh.traffic_class = (be32_to_cpup(
-				&grh->version_tclass_flow) >> 20) & 0xff;
+			ah_attr.grh.flow_label = be32_to_cpu(
+				grh->version_tclass_flow)  & 0xfffff;
+			ah_attr.grh.traffic_class = (be32_to_cpu(
+				grh->version_tclass_flow) >> 20) & 0xff;
 			memcpy(ah_attr.grh.dgid.raw,
 			       grh->sgid.raw,
 			       sizeof(ah_attr.grh.dgid));
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband16/core/verbs.c linux-2.6.13-rc2-mm1/drivers/infiniband17/core/verbs.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband16/core/verbs.c	2005-07-10 11:43:44.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband17/core/verbs.c	2005-07-10 11:50:26.000000000 -0400
@@ -113,7 +113,7 @@ struct ib_ah *ib_create_ah_from_wc(struc
 			return ERR_PTR(ret);
 
 		ah_attr.grh.sgid_index = (u8) gid_index;
-		flow_class = be32_to_cpu(&grh->version_tclass_flow);
+		flow_class = be32_to_cpu(grh->version_tclass_flow);
 		ah_attr.grh.flow_label = flow_class & 0xFFFFF;
 		ah_attr.grh.traffic_class = (flow_class >> 20) & 0xFF;
 		ah_attr.grh.hop_limit = grh->hop_limit;




From halr at voltaire.com  Mon Jul 11 07:36:46 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:36:46 -0400
Subject: [openib-general] [PATCH 18/27] Introduce RMPP APIs
Message-ID: <1121089154.4389.4541.camel@hal.voltaire.com>

Introduce RMPP APIs 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 17/27.

-- 
 core/mad.c       |    4 +
 core/sa_query.c  |   20 --
 include/ib_mad.h |  132 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 include/ib_sa.h  |    4 -
 4 files changed, 125 insertions(+), 35 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband17/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband17/core/mad.c	2005-07-09 18:24:43.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad.c	2005-07-10 11:57:01.000000000 -0400
@@ -777,7 +777,7 @@ static int get_buf_length(int hdr_len, i
 
 struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
 					    u32 remote_qpn, u16 pkey_index,
-					    struct ib_ah *ah,
+					    struct ib_ah *ah, int rmpp_active,
 					    int hdr_len, int data_len,
 					    int gfp_mask)
 {
@@ -786,6 +786,8 @@ struct ib_mad_send_buf * ib_create_send_
 	int buf_size;
 	void *buf;
 
+	if (rmpp_active)
+		return ERR_PTR(-EINVAL);	/* until RMPP implemented */
 	mad_agent_priv = container_of(mad_agent,
 				      struct ib_mad_agent_private, agent);
 	buf_size = get_buf_length(hdr_len, data_len);
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband17/core/sa_query.c linux-2.6.13-rc2-mm1/drivers/infiniband18/core/sa_query.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband17/core/sa_query.c	2005-07-10 16:22:13.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband18/core/sa_query.c	2005-07-10 16:22:18.000000000 -0400
@@ -50,26 +50,6 @@ MODULE_AUTHOR("Roland Dreier");
 MODULE_DESCRIPTION("InfiniBand subnet administration query support");
 MODULE_LICENSE("Dual BSD/GPL");
 
-/*
- * These two structures must be packed because they have 64-bit fields
- * that are only 32-bit aligned.  64-bit architectures will lay them
- * out wrong otherwise.  (And unfortunately they are sent on the wire
- * so we can't change the layout)
- */
-struct ib_sa_hdr {
-	u64			sm_key;
-	u16			attr_offset;
-	u16			reserved;
-	ib_sa_comp_mask		comp_mask;
-} __attribute__ ((packed));
-
-struct ib_sa_mad {
-	struct ib_mad_hdr	mad_hdr;
-	struct ib_rmpp_hdr	rmpp_hdr;
-	struct ib_sa_hdr	sa_hdr;
-	u8			data[200];
-} __attribute__ ((packed));
-
 struct ib_sa_sm_ah {
 	struct ib_ah        *ah;
 	struct kref          ref;
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband17/include/ib_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband18/include/ib_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband17/include/ib_mad.h	2005-07-10 11:43:45.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband18/include/ib_mad.h	2005-07-10 12:04:23.000000000 -0400
@@ -33,7 +33,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_mad.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: ib_mad.h 2775 2005-07-02 13:42:12Z halr $
  */
 
 #if !defined( IB_MAD_H )
@@ -58,6 +58,8 @@
 #define IB_MGMT_CLASS_VENDOR_RANGE2_START	0x30
 #define IB_MGMT_CLASS_VENDOR_RANGE2_END		0x4F
 
+#define	IB_OPENIB_OUI				(0x001405)
+
 /* Management methods */
 #define IB_MGMT_METHOD_GET			0x01
 #define IB_MGMT_METHOD_SET			0x02
@@ -72,6 +74,33 @@
 
 #define IB_MGMT_MAX_METHODS			128
 
+/* RMPP information */
+#define IB_MGMT_RMPP_VERSION			1
+
+#define IB_MGMT_RMPP_TYPE_DATA			1
+#define IB_MGMT_RMPP_TYPE_ACK			2
+#define IB_MGMT_RMPP_TYPE_STOP			3
+#define IB_MGMT_RMPP_TYPE_ABORT			4
+
+#define IB_MGMT_RMPP_FLAG_ACTIVE		1
+#define IB_MGMT_RMPP_FLAG_FIRST			(1<<1)
+#define IB_MGMT_RMPP_FLAG_LAST			(1<<2)
+
+#define IB_MGMT_RMPP_NO_RESPTIME		0x1F
+
+#define	IB_MGMT_RMPP_STATUS_SUCCESS		0
+#define	IB_MGMT_RMPP_STATUS_RESX		1
+#define	IB_MGMT_RMPP_STATUS_T2L			118
+#define	IB_MGMT_RMPP_STATUS_BAD_LEN		119
+#define	IB_MGMT_RMPP_STATUS_BAD_SEG		120
+#define	IB_MGMT_RMPP_STATUS_BADT		121
+#define	IB_MGMT_RMPP_STATUS_W2S			122
+#define	IB_MGMT_RMPP_STATUS_S2B			123
+#define	IB_MGMT_RMPP_STATUS_BAD_STATUS		124
+#define	IB_MGMT_RMPP_STATUS_UNV			125
+#define	IB_MGMT_RMPP_STATUS_TMR			126
+#define	IB_MGMT_RMPP_STATUS_UNSPEC		127
+
 #define IB_QP0		0
 #define IB_QP1		__constant_htonl(1)
 #define IB_QP1_QKEY	0x80010000
@@ -88,7 +117,7 @@ struct ib_mad_hdr {
 	u16	attr_id;
 	u16	resv;
 	u32	attr_mod;
-} __attribute__ ((packed));
+};
 
 struct ib_rmpp_hdr {
 	u8	rmpp_version;
@@ -97,17 +126,41 @@ struct ib_rmpp_hdr {
 	u8	rmpp_status;
 	u32	seg_num;
 	u32	paylen_newwin;
+};
+
+typedef u64 __bitwise ib_sa_comp_mask;
+
+#define IB_SA_COMP_MASK(n) ((__force ib_sa_comp_mask) cpu_to_be64(1ull << n))
+
+/*
+ * ib_sa_hdr and ib_sa_mad structures must be packed because they have 
+ * 64-bit fields that are only 32-bit aligned. 64-bit architectures will
+ * lay them out wrong otherwise.  (And unfortunately they are sent on 
+ * the wire so we can't change the layout)
+ */
+struct ib_sa_hdr {
+	u64			sm_key;
+	u16			attr_offset;
+	u16			reserved;
+	ib_sa_comp_mask		comp_mask;
 } __attribute__ ((packed));
 
 struct ib_mad {
 	struct ib_mad_hdr	mad_hdr;
 	u8			data[232];
-} __attribute__ ((packed));
+};
 
 struct ib_rmpp_mad {
 	struct ib_mad_hdr	mad_hdr;
 	struct ib_rmpp_hdr	rmpp_hdr;
 	u8			data[220];
+};
+
+struct ib_sa_mad {
+	struct ib_mad_hdr	mad_hdr;
+	struct ib_rmpp_hdr	rmpp_hdr;
+	struct ib_sa_hdr	sa_hdr;
+	u8			data[200];
 } __attribute__ ((packed));
 
 struct ib_vendor_mad {
@@ -116,7 +169,7 @@ struct ib_vendor_mad {
 	u8			reserved;
 	u8			oui[3];
 	u8			data[216];
-} __attribute__ ((packed));
+};
 
 /**
  * ib_mad_send_buf - MAD data buffer and work request for sends.
@@ -142,6 +195,45 @@ struct ib_mad_send_buf {
 	struct ib_sge		sge;
 };
 
+/**
+ * ib_get_rmpp_resptime - Returns the RMPP response time.
+ * @rmpp_hdr: An RMPP header.
+ */
+static inline u8 ib_get_rmpp_resptime(struct ib_rmpp_hdr *rmpp_hdr)
+{
+	return rmpp_hdr->rmpp_rtime_flags >> 3;
+}
+
+/**
+ * ib_get_rmpp_flags - Returns the RMPP flags.
+ * @rmpp_hdr: An RMPP header.
+ */
+static inline u8 ib_get_rmpp_flags(struct ib_rmpp_hdr *rmpp_hdr)
+{
+	return rmpp_hdr->rmpp_rtime_flags & 0x7;
+}
+
+/**
+ * ib_set_rmpp_resptime - Sets the response time in an RMPP header.
+ * @rmpp_hdr: An RMPP header.
+ * @rtime: The response time to set.
+ */
+static inline void ib_set_rmpp_resptime(struct ib_rmpp_hdr *rmpp_hdr, u8 rtime)
+{
+	rmpp_hdr->rmpp_rtime_flags = ib_get_rmpp_flags(rmpp_hdr) | (rtime << 3);
+}
+
+/**
+ * ib_set_rmpp_flags - Sets the flags in an RMPP header.
+ * @rmpp_hdr: An RMPP header.
+ * @flags: The flags to set.
+ */
+static inline void ib_set_rmpp_flags(struct ib_rmpp_hdr *rmpp_hdr, u8 flags)
+{
+	rmpp_hdr->rmpp_rtime_flags = (rmpp_hdr->rmpp_rtime_flags & 0xF1) |
+				     (flags & 0x7);
+}
+
 struct ib_mad_agent;
 struct ib_mad_send_wc;
 struct ib_mad_recv_wc;
@@ -186,6 +278,7 @@ typedef void (*ib_mad_recv_handler)(stru
  * ib_mad_agent - Used to track MAD registration with the access layer.
  * @device: Reference to device registration is on.
  * @qp: Reference to QP used for sending and receiving MADs.
+ * @mr: Memory region for system memory usable for DMA.
  * @recv_handler: Callback handler for a received MAD.
  * @send_handler: Callback handler for a sent MAD.
  * @snoop_handler: Callback handler for snooped sent MADs.
@@ -194,6 +287,7 @@ typedef void (*ib_mad_recv_handler)(stru
  *   Unsolicited MADs sent by this client will have the upper 32-bits
  *   of their TID set to this value.
  * @port_num: Port number on which QP is registered
+ * @rmpp_version: If set, indicates the RMPP version used by this agent.
  */
 struct ib_mad_agent {
 	struct ib_device	*device;
@@ -205,6 +299,7 @@ struct ib_mad_agent {
 	void			*context;
 	u32			hi_tid;
 	u8			port_num;
+	u8			rmpp_version;
 };
 
 /**
@@ -238,6 +333,7 @@ struct ib_mad_recv_buf {
  * ib_mad_recv_wc - received MAD information.
  * @wc: Completion information for the received data.
  * @recv_buf: Specifies the location of the received data buffer(s).
+ * @rmpp_list: Specifies a list of RMPP reassembled received MAD buffers.
  * @mad_len: The length of the received MAD, without duplicated headers.
  *
  * For received response, the wr_id field of the wc is set to the wr_id
@@ -246,6 +342,7 @@ struct ib_mad_recv_buf {
 struct ib_mad_recv_wc {
 	struct ib_wc		*wc;
 	struct ib_mad_recv_buf	recv_buf;
+	struct list_head	rmpp_list;
 	int			mad_len;
 };
 
@@ -341,6 +438,16 @@ int ib_unregister_mad_agent(struct ib_ma
  * @bad_send_wr: Specifies the MAD on which an error was encountered.
  *
  * Sent MADs are not guaranteed to complete in the order that they were posted.
+ *
+ * If the MAD requires RMPP, the data buffer should contain a single copy
+ * of the common MAD, RMPP, and class specific headers, followed by the class
+ * defined data.  If the class defined data would not divide evenly into
+ * RMPP segments, then space must be allocated at the end of the referenced
+ * buffer for any required padding.  To indicate the amount of class defined
+ * data being transferred, the paylen_newwin field in the RMPP header should
+ * be set to the size of the class specific header plus the amount of class
+ * defined data being transferred.  The paylen_newwin field should be
+ * specified in network-byte order.
  */
 int ib_post_send_mad(struct ib_mad_agent *mad_agent,
 		     struct ib_send_wr *send_wr,
@@ -353,14 +460,13 @@ int ib_post_send_mad(struct ib_mad_agent
  *   referenced buffer should be at least the size of the mad_len specified
  *   by @mad_recv_wc.
  *
- * This call copies a chain of received RMPP MADs into a single data buffer,
+ * This call copies a chain of received MAD segments into a single data buffer,
  * removing duplicated headers.
  */
 void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf);
 
 /**
- * ib_free_recv_mad - Returns data buffers used to receive a MAD to the
- *   access layer.
+ * ib_free_recv_mad - Returns data buffers used to receive a MAD.
  * @mad_recv_wc: Work completion information for a received MAD.
  *
  * Clients receiving MADs through their ib_mad_recv_handler must call this
@@ -437,10 +543,11 @@ int ib_process_mad_wc(struct ib_mad_agen
  * @pkey_index: Specifies which PKey the MAD will be sent using.  This field
  *   is valid only if the remote_qpn is QP 1.
  * @ah: References the address handle used to transfer to the remote node.
+ * @rmpp_active: Indicates if the send will enable RMPP.
  * @hdr_len: Indicates the size of the data header of the MAD.  This length
  *   should include the common MAD header, RMPP header, plus any class
  *   specific header.
- * @data_len: Indicates the size of any user-transfered data.  The call will
+ * @data_len: Indicates the size of any user-transferred data.  The call will
  *   automatically adjust the allocated buffer size to account for any
  *   additional padding that may be necessary.
  * @gfp_mask: GFP mask used for the memory allocation.
@@ -448,11 +555,16 @@ int ib_process_mad_wc(struct ib_mad_agen
  * This is a helper routine that may be used to allocate a MAD.  Users are
  * not required to allocate outbound MADs using this call.  The returned
  * MAD send buffer will reference a data buffer usable for sending a MAD, along
- * with an intialized work request structure.
+ * with an initialized work request structure.  Users may modify the returned
+ * MAD data buffer or work request before posting the send.
+ *
+ * The returned data buffer will be cleared.  Users are responsible for
+ * initializing the common MAD and any class specific headers.  If @rmpp_active
+ * is set, the RMPP header will be initialized for sending.
  */
 struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
 					    u32 remote_qpn, u16 pkey_index,
-					    struct ib_ah *ah,
+					    struct ib_ah *ah, int rmpp_active,
 					    int hdr_len, int data_len,
 					    int gfp_mask);
 
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband17/include/ib_sa.h linux-2.6.13-rc2-mm1/drivers/infiniband18/include/ib_sa.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband17/include/ib_sa.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband18/include/ib_sa.h	2005-07-10 12:07:41.000000000 -0400
@@ -87,10 +87,6 @@ static inline int ib_sa_rate_enum_to_int
 	}
 }
 
-typedef u64 __bitwise ib_sa_comp_mask;
-
-#define IB_SA_COMP_MASK(n)	((__force ib_sa_comp_mask) cpu_to_be64(1ull << n))
-
 /*
  * Structures for SA records are named "struct ib_sa_xxx_rec."  No
  * attempt is made to pack structures to match the physical layout of



From halr at voltaire.com  Mon Jul 11 07:40:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:40:00 -0400
Subject: [openib-general] [PATCH 19/27] Add RMPP implementation
Message-ID: <1121089164.4389.4543.camel@hal.voltaire.com>

Add RMPP implementation 


Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 18/27.

-- 
 Makefile   |    2 
 mad.c      |  167 +++++++++--
 mad_priv.h |   28 +-
 mad_rmpp.c |  765 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 mad_rmpp.h |   58 ++++
 5 files changed, 968 insertions(+), 52 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad.c linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad.c	2005-07-10 11:57:01.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad.c	2005-07-10 12:12:10.000000000 -0400
@@ -1,5 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved.
+ * Copyright (c) 2005 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies Ltd.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,12 +31,12 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: mad.c 1389 2004-12-27 22:56:47Z roland $
+ * $Id: mad.c 2817 2005-07-07 11:29:26Z halr $
  */
-
 #include <linux/dma-mapping.h>
 
 #include "mad_priv.h"
+#include "mad_rmpp.h"
 #include "smi.h"
 #include "agent.h"
 
@@ -45,6 +47,7 @@ MODULE_AUTHOR("Sean Hefty");
 

 kmem_cache_t *ib_mad_cache;
+
 static struct list_head ib_mad_port_list;
 static u32 ib_mad_client_id = 0;
 
@@ -62,8 +65,6 @@ static struct ib_mad_agent_private *find
 static int ib_mad_post_receive_mads(struct ib_mad_qp_info *qp_info,
 				    struct ib_mad_private *mad);
 static void cancel_mads(struct ib_mad_agent_private *mad_agent_priv);
-static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
-				    struct ib_mad_send_wc *mad_send_wc);
 static void timeout_sends(void *data);
 static void local_completions(void *data);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
@@ -195,8 +196,8 @@ struct ib_mad_agent *ib_register_mad_age
 	if (qpn == -1)
 		goto error1;
 
-	if (rmpp_version)
-		goto error1;	/* XXX: until RMPP implemented */
+	if (rmpp_version && rmpp_version != IB_MGMT_RMPP_VERSION)
+		goto error1;
 
 	/* Validate MAD registration request if supplied */
 	if (mad_reg_req) {
@@ -281,7 +282,7 @@ struct ib_mad_agent *ib_register_mad_age
 	/* Now, fill in the various structures */
 	mad_agent_priv->qp_info = &port_priv->qp_info[qpn];
 	mad_agent_priv->reg_req = reg_req;
-	mad_agent_priv->rmpp_version = rmpp_version;
+	mad_agent_priv->agent.rmpp_version = rmpp_version;
 	mad_agent_priv->agent.device = device;
 	mad_agent_priv->agent.recv_handler = recv_handler;
 	mad_agent_priv->agent.send_handler = send_handler;
@@ -341,6 +342,7 @@ struct ib_mad_agent *ib_register_mad_age
 	INIT_LIST_HEAD(&mad_agent_priv->send_list);
 	INIT_LIST_HEAD(&mad_agent_priv->wait_list);
 	INIT_LIST_HEAD(&mad_agent_priv->done_list);
+	INIT_LIST_HEAD(&mad_agent_priv->rmpp_list);
 	INIT_WORK(&mad_agent_priv->timed_work, timeout_sends, mad_agent_priv);
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
@@ -502,6 +504,7 @@ static void unregister_mad_agent(struct 
 	spin_unlock_irqrestore(&port_priv->reg_lock, flags);
 
 	flush_workqueue(port_priv->wq);
+	ib_cancel_rmpp_recvs(mad_agent_priv);
 
 	atomic_dec(&mad_agent_priv->refcount);
 	wait_event(mad_agent_priv->wait,
@@ -689,7 +692,7 @@ static int handle_outgoing_dr_smp(struct
 		kfree(local);
 		goto out;
 	}
-
+	
 	build_smp_wc(send_wr->wr_id, smp->dr_slid, send_wr->wr.ud.pkey_index,
 		     send_wr->wr.ud.port_num, &mad_wc);
 
@@ -705,7 +708,7 @@ static int handle_outgoing_dr_smp(struct
 			local->mad_priv = mad_priv;
 			local->recv_mad_agent = mad_agent_priv;
 			/*
-			 * Reference MAD agent until receive
+			 * Reference MAD agent until receive 
 			 * side of local completion handled
 			 */
 			atomic_inc(&mad_agent_priv->refcount);
@@ -786,12 +789,15 @@ struct ib_mad_send_buf * ib_create_send_
 	int buf_size;
 	void *buf;
 
-	if (rmpp_active)
-		return ERR_PTR(-EINVAL);	/* until RMPP implemented */
 	mad_agent_priv = container_of(mad_agent,
 				      struct ib_mad_agent_private, agent);
 	buf_size = get_buf_length(hdr_len, data_len);
 
+	if ((!mad_agent->rmpp_version &&
+	     (rmpp_active || buf_size > sizeof(struct ib_mad))) ||
+	    (!rmpp_active && buf_size > sizeof(struct ib_mad)))
+		return ERR_PTR(-EINVAL);
+
 	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);
 	if (!buf)
 		return ERR_PTR(-ENOMEM);
@@ -816,6 +822,18 @@ struct ib_mad_send_buf * ib_create_send_
 	send_buf->send_wr.wr.ud.remote_qpn = remote_qpn;
 	send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY;
 	send_buf->send_wr.wr.ud.pkey_index = pkey_index;
+
+	if (rmpp_active) {
+		struct ib_rmpp_mad *rmpp_mad;
+		rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad;
+		rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len -
+			offsetof(struct ib_rmpp_mad, data) + data_len);
+		rmpp_mad->rmpp_hdr.rmpp_version = mad_agent->rmpp_version;
+		rmpp_mad->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_DATA;
+		ib_set_rmpp_flags(&rmpp_mad->rmpp_hdr,
+				  IB_MGMT_RMPP_FLAG_ACTIVE);
+	}
+
 	send_buf->mad_agent = mad_agent;
 	atomic_inc(&mad_agent_priv->refcount);
 	return send_buf;
@@ -839,7 +857,7 @@ void ib_free_send_mad(struct ib_mad_send
 }
 EXPORT_SYMBOL(ib_free_send_mad);
 
-static int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr)
+int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr)
 {
 	struct ib_mad_qp_info *qp_info;
 	struct ib_send_wr *bad_send_wr;
@@ -940,13 +958,13 @@ int ib_post_send_mad(struct ib_mad_agent
 			ret = -ENOMEM;
 			goto error2;
 		}
+		memset(mad_send_wr, 0, sizeof *mad_send_wr);
 
 		mad_send_wr->send_wr = *send_wr;
 		mad_send_wr->send_wr.sg_list = mad_send_wr->sg_list;
 		memcpy(mad_send_wr->sg_list, send_wr->sg_list,
 		       sizeof *send_wr->sg_list * send_wr->num_sge);
-		mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
-		mad_send_wr->send_wr.next = NULL;
+		mad_send_wr->wr_id = send_wr->wr_id;
 		mad_send_wr->tid = send_wr->wr.ud.mad_hdr->tid;
 		mad_send_wr->mad_agent_priv = mad_agent_priv;
 		/* Timeout will be updated after send completes */
@@ -964,8 +982,13 @@ int ib_post_send_mad(struct ib_mad_agent
 			      &mad_agent_priv->send_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
-		ret = ib_send_mad(mad_send_wr);
-		if (ret) {
+		if (mad_agent_priv->agent.rmpp_version) {
+			ret = ib_send_rmpp_mad(mad_send_wr);
+			if (ret >= 0 && ret != IB_RMPP_RESULT_CONSUMED)
+				ret = ib_send_mad(mad_send_wr);
+		} else
+			ret = ib_send_mad(mad_send_wr);
+		if (ret < 0) {
 			/* Fail send request */
 			spin_lock_irqsave(&mad_agent_priv->lock, flags);
 			list_del(&mad_send_wr->agent_list);
@@ -991,31 +1014,25 @@ EXPORT_SYMBOL(ib_post_send_mad);
  */
 void ib_free_recv_mad(struct ib_mad_recv_wc *mad_recv_wc)
 {
-	struct ib_mad_recv_buf *entry;
+	struct ib_mad_recv_buf *mad_recv_buf, *temp_recv_buf;
 	struct ib_mad_private_header *mad_priv_hdr;
 	struct ib_mad_private *priv;
+	struct list_head free_list;
 
-	mad_priv_hdr = container_of(mad_recv_wc,
-				    struct ib_mad_private_header,
-				    recv_wc);
-	priv = container_of(mad_priv_hdr, struct ib_mad_private, header);
+	INIT_LIST_HEAD(&free_list);
+	list_splice_init(&mad_recv_wc->rmpp_list, &free_list);
 
-	/*
-	 * Walk receive buffer list associated with this WC
-	 * No need to remove them from list of receive buffers
-	 */
-	list_for_each_entry(entry, &mad_recv_wc->recv_buf.list, list) {
-		/* Free previous receive buffer */
-		kmem_cache_free(ib_mad_cache, priv);
+	list_for_each_entry_safe(mad_recv_buf, temp_recv_buf,
+					&free_list, list) {
+		mad_recv_wc = container_of(mad_recv_buf, struct ib_mad_recv_wc,
+					   recv_buf);
 		mad_priv_hdr = container_of(mad_recv_wc,
 					    struct ib_mad_private_header,
 					    recv_wc);
 		priv = container_of(mad_priv_hdr, struct ib_mad_private,
 				    header);
+		kmem_cache_free(ib_mad_cache, priv);
 	}
-
-	/* Free last buffer */
-	kmem_cache_free(ib_mad_cache, priv);
 }
 EXPORT_SYMBOL(ib_free_recv_mad);
 
@@ -1524,9 +1541,20 @@ out:
 	return valid;
 }
 
-static struct ib_mad_send_wr_private*
-find_send_req(struct ib_mad_agent_private *mad_agent_priv,
-	      u64 tid)
+static int is_data_mad(struct ib_mad_agent_private *mad_agent_priv,
+		       struct ib_mad_hdr *mad_hdr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_hdr;
+	return !mad_agent_priv->agent.rmpp_version ||
+		!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) &
+				    IB_MGMT_RMPP_FLAG_ACTIVE) ||
+		(rmpp_mad->rmpp_hdr.rmpp_type == IB_MGMT_RMPP_TYPE_DATA);
+}
+
+struct ib_mad_send_wr_private*
+ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, u64 tid)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 
@@ -1542,7 +1570,9 @@ find_send_req(struct ib_mad_agent_privat
 	 */
 	list_for_each_entry(mad_send_wr, &mad_agent_priv->send_list,
 			    agent_list) {
-		if (mad_send_wr->tid == tid && mad_send_wr->timeout) {
+		if (is_data_mad(mad_agent_priv,
+				mad_send_wr->send_wr.wr.ud.mad_hdr) &&
+		    mad_send_wr->tid == tid && mad_send_wr->timeout) {
 			/* Verify request has not been canceled */
 			return (mad_send_wr->status == IB_WC_SUCCESS) ?
 				mad_send_wr : NULL;
@@ -1551,7 +1581,7 @@ find_send_req(struct ib_mad_agent_privat
 	return NULL;
 }
 
-static void ib_mark_req_done(struct ib_mad_send_wr_private *mad_send_wr)
+void ib_mark_mad_done(struct ib_mad_send_wr_private *mad_send_wr)
 {
 	mad_send_wr->timeout = 0;
 	if (mad_send_wr->refcount == 1) {
@@ -1569,12 +1599,23 @@ static void ib_mad_complete_recv(struct 
 	unsigned long flags;
 	u64 tid;
 
-	INIT_LIST_HEAD(&mad_recv_wc->recv_buf.list);
+	INIT_LIST_HEAD(&mad_recv_wc->rmpp_list);
+	list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list);
+	if (mad_agent_priv->agent.rmpp_version) {
+		mad_recv_wc = ib_process_rmpp_recv_wc(mad_agent_priv,
+						      mad_recv_wc);
+		if (!mad_recv_wc) {
+			if (atomic_dec_and_test(&mad_agent_priv->refcount))
+				wake_up(&mad_agent_priv->wait);
+			return;
+		}
+	}
+
 	/* Complete corresponding request */
 	if (response_mad(mad_recv_wc->recv_buf.mad)) {
 		tid = mad_recv_wc->recv_buf.mad->mad_hdr.tid;
 		spin_lock_irqsave(&mad_agent_priv->lock, flags);
-		mad_send_wr = find_send_req(mad_agent_priv, tid);
+		mad_send_wr = ib_find_send_mad(mad_agent_priv, tid);
 		if (!mad_send_wr) {
 			spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 			ib_free_recv_mad(mad_recv_wc);
@@ -1582,7 +1623,7 @@ static void ib_mad_complete_recv(struct 
 				wake_up(&mad_agent_priv->wait);
 			return;
 		}
-		ib_mark_req_done(mad_send_wr);
+		ib_mark_mad_done(mad_send_wr);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		/* Defined behavior is to complete response before request */
@@ -1787,14 +1828,22 @@ void ib_reset_mad_timeout(struct ib_mad_
 /*
  * Process a send work completion
  */
-static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
-				    struct ib_mad_send_wc *mad_send_wc)
+void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
+			     struct ib_mad_send_wc *mad_send_wc)
 {
 	struct ib_mad_agent_private	*mad_agent_priv;
 	unsigned long			flags;
+	int				ret;
 
 	mad_agent_priv = mad_send_wr->mad_agent_priv;
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
+	if (mad_agent_priv->agent.rmpp_version) {
+		ret = ib_process_rmpp_send_wc(mad_send_wr, mad_send_wc);
+		if (ret == IB_RMPP_RESULT_CONSUMED)
+			goto done;
+	} else
+		ret = IB_RMPP_RESULT_UNHANDLED;
+
 	if (mad_send_wc->status != IB_WC_SUCCESS &&
 	    mad_send_wr->status == IB_WC_SUCCESS) {
 		mad_send_wr->status = mad_send_wc->status;
@@ -1806,8 +1855,7 @@ static void ib_mad_complete_send_wr(stru
 		    mad_send_wr->status == IB_WC_SUCCESS) {
 			wait_for_response(mad_send_wr);
 		}
-		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-		return;
+		goto done;
 	}
 
 	/* Remove send from MAD agent and notify client of completion */
@@ -1817,14 +1865,18 @@ static void ib_mad_complete_send_wr(stru
 
 	if (mad_send_wr->status != IB_WC_SUCCESS )
 		mad_send_wc->status = mad_send_wr->status;
-	mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
-					    mad_send_wc);
+	if (ret != IB_RMPP_RESULT_INTERNAL)
+		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
+						   mad_send_wc);
 
 	/* Release reference on agent taken when sending */
 	if (atomic_dec_and_test(&mad_agent_priv->refcount))
 		wake_up(&mad_agent_priv->wait);
 
 	kfree(mad_send_wr);
+	return;
+done:
+	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 }
 
 static void ib_mad_send_done_handler(struct ib_mad_port_private *port_priv,
@@ -2036,7 +2088,9 @@ find_send_by_wr_id(struct ib_mad_agent_p
 
 	list_for_each_entry(mad_send_wr, &mad_agent_priv->send_list,
 			    agent_list) {
-		if (mad_send_wr->wr_id == wr_id)
+		if (is_data_mad(mad_agent_priv,
+				mad_send_wr->send_wr.wr.ud.mad_hdr) &&
+		    mad_send_wr->wr_id == wr_id)
 			return mad_send_wr;
 	}
 	return NULL;
@@ -2118,7 +2172,9 @@ static void local_completions(void *data
 			local->mad_priv->header.recv_wc.wc = &wc;
 			local->mad_priv->header.recv_wc.mad_len =
 						sizeof(struct ib_mad);
-			INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.recv_buf.list);
+			INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.rmpp_list);
+			list_add(&local->mad_priv->header.recv_wc.recv_buf.list,
+				 &local->mad_priv->header.recv_wc.rmpp_list);
 			local->mad_priv->header.recv_wc.recv_buf.grh = NULL;
 			local->mad_priv->header.recv_wc.recv_buf.mad =
 						&local->mad_priv->mad.mad;
@@ -2166,7 +2222,21 @@ static int retry_send(struct ib_mad_send
 	mad_send_wr->timeout = msecs_to_jiffies(mad_send_wr->send_wr.
 						wr.ud.timeout_ms);
 
-	ret = ib_send_mad(mad_send_wr);
+	if (mad_send_wr->mad_agent_priv->agent.rmpp_version) {
+		ret = ib_retry_rmpp(mad_send_wr);
+		switch (ret) {
+		case IB_RMPP_RESULT_UNHANDLED:
+			ret = ib_send_mad(mad_send_wr);
+			break;
+		case IB_RMPP_RESULT_CONSUMED:
+			ret = 0;
+			break;
+		default:
+			ret = -ECOMM;
+			break;
+		}
+	} else
+		ret = ib_send_mad(mad_send_wr);
 
 	if (!ret) {
 		mad_send_wr->refcount++;
@@ -2724,3 +2794,4 @@ static void __exit ib_mad_cleanup_module
 
 module_init(ib_mad_init_module);
 module_exit(ib_mad_cleanup_module);
+
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad_priv.h linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad_priv.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad_priv.h	2005-07-09 17:57:55.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad_priv.h	2005-07-10 12:12:10.000000000 -0400
@@ -1,5 +1,7 @@
 /*
  * Copyright (c) 2004, 2005, Voltaire, Inc. All rights reserved.
+ * Copyright (c) 2005 Intel Corporation. All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +31,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: mad_priv.h 1980 2005-03-11 22:33:53Z sean.hefty $
+ * $Id: mad_priv.h 2730 2005-06-28 16:43:03Z sean.hefty $
  */
 
 #ifndef __IB_MAD_PRIV_H__
@@ -97,10 +99,10 @@ struct ib_mad_agent_private {
 	unsigned long timeout;
 	struct list_head local_list;
 	struct work_struct local_work;
+	struct list_head rmpp_list;
 
 	atomic_t refcount;
 	wait_queue_head_t wait;
-	u8 rmpp_version;
 };
 
 struct ib_mad_snoop_private {
@@ -125,6 +127,14 @@ struct ib_mad_send_wr_private {
 	int retry;
 	int refcount;
 	enum ib_wc_status status;
+
+	/* RMPP control */
+	int last_ack;
+	int seg_num;
+	int newwin;
+	int total_seg;
+	int data_offset;
+	int pad;
 };
 
 struct ib_mad_local_private {
@@ -135,7 +145,6 @@ struct ib_mad_local_private {
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */
 	u64 tid;
-	int retries;
 };
 
 struct ib_mad_mgmt_method_table {
@@ -198,4 +207,17 @@ struct ib_mad_port_private {
 
 extern kmem_cache_t *ib_mad_cache;
 
+int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr);
+
+struct ib_mad_send_wr_private *
+ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, u64 tid);
+
+void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
+			     struct ib_mad_send_wc *mad_send_wc);
+
+void ib_mark_mad_done(struct ib_mad_send_wr_private *mad_send_wr);
+
+void ib_reset_mad_timeout(struct ib_mad_send_wr_private *mad_send_wr,
+			  int timeout_ms);
+
 #endif	/* __IB_MAD_PRIV_H__ */
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad_rmpp.c linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad_rmpp.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad_rmpp.c	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad_rmpp.c	2005-07-06 16:52:21.000000000 -0400
@@ -0,0 +1,765 @@
+/*
+ * Copyright (c) 2005 Intel Inc. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: mad_rmpp.c 1921 2005-03-02 22:58:44Z sean.hefty $
+ */
+
+#include <linux/dma-mapping.h>
+
+#include "mad_priv.h"
+#include "mad_rmpp.h"
+
+enum rmpp_state {
+	RMPP_STATE_ACTIVE,
+	RMPP_STATE_TIMEOUT,
+	RMPP_STATE_COMPLETE
+};
+
+struct mad_rmpp_recv {
+	struct ib_mad_agent_private *agent;
+	struct list_head list;
+	struct work_struct timeout_work;
+	struct work_struct cleanup_work;
+	wait_queue_head_t wait;
+	enum rmpp_state state;
+	spinlock_t lock;
+	atomic_t refcount;
+
+	struct ib_ah *ah;
+	struct ib_mad_recv_wc *rmpp_wc;
+	struct ib_mad_recv_buf *cur_seg_buf;
+	int last_ack;
+	int seg_num;
+	int newwin;
+
+	u64 tid;
+	u32 src_qp;
+	u16 slid;
+	u8 mgmt_class;
+	u8 class_version;
+	u8 method;
+};
+
+static void destroy_rmpp_recv(struct mad_rmpp_recv *rmpp_recv)
+{
+	atomic_dec(&rmpp_recv->refcount);
+	wait_event(rmpp_recv->wait, !atomic_read(&rmpp_recv->refcount));
+	ib_destroy_ah(rmpp_recv->ah);
+	kfree(rmpp_recv);
+}
+
+void ib_cancel_rmpp_recvs(struct ib_mad_agent_private *agent)
+{
+	struct mad_rmpp_recv *rmpp_recv, *temp_rmpp_recv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&agent->lock, flags);
+	list_for_each_entry(rmpp_recv, &agent->rmpp_list, list) {
+		cancel_delayed_work(&rmpp_recv->timeout_work);
+		cancel_delayed_work(&rmpp_recv->cleanup_work);
+	}
+	spin_unlock_irqrestore(&agent->lock, flags);
+
+	flush_workqueue(agent->qp_info->port_priv->wq);
+
+	list_for_each_entry_safe(rmpp_recv, temp_rmpp_recv,
+				 &agent->rmpp_list, list) {
+		list_del(&rmpp_recv->list);
+		if (rmpp_recv->state != RMPP_STATE_COMPLETE)
+			ib_free_recv_mad(rmpp_recv->rmpp_wc);
+		destroy_rmpp_recv(rmpp_recv);
+	}
+}
+
+static void recv_timeout_handler(void *data)
+{
+	struct mad_rmpp_recv *rmpp_recv = data;
+	struct ib_mad_recv_wc *rmpp_wc;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rmpp_recv->agent->lock, flags);
+	if (rmpp_recv->state != RMPP_STATE_ACTIVE) {
+		spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
+		return;
+	}
+	rmpp_recv->state = RMPP_STATE_TIMEOUT;
+	list_del(&rmpp_recv->list);
+	spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
+
+	/* TODO: send abort. */
+	rmpp_wc = rmpp_recv->rmpp_wc;
+	destroy_rmpp_recv(rmpp_recv);
+	ib_free_recv_mad(rmpp_wc);
+}
+
+static void recv_cleanup_handler(void *data)
+{
+	struct mad_rmpp_recv *rmpp_recv = data;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rmpp_recv->agent->lock, flags);
+	list_del(&rmpp_recv->list);
+	spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
+	destroy_rmpp_recv(rmpp_recv);
+}
+
+static struct mad_rmpp_recv *
+create_rmpp_recv(struct ib_mad_agent_private *agent,
+		 struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	struct ib_mad_hdr *mad_hdr;
+
+	rmpp_recv = kmalloc(sizeof *rmpp_recv, GFP_KERNEL);
+	if (!rmpp_recv)
+		return NULL;
+
+	rmpp_recv->ah = ib_create_ah_from_wc(agent->agent.qp->pd,
+					     mad_recv_wc->wc,
+					     mad_recv_wc->recv_buf.grh,
+					     agent->agent.port_num);
+	if (IS_ERR(rmpp_recv->ah))
+		goto error;
+
+	rmpp_recv->agent = agent;
+	init_waitqueue_head(&rmpp_recv->wait);
+	INIT_WORK(&rmpp_recv->timeout_work, recv_timeout_handler, rmpp_recv);
+	INIT_WORK(&rmpp_recv->cleanup_work, recv_cleanup_handler, rmpp_recv);
+	spin_lock_init(&rmpp_recv->lock);
+	rmpp_recv->state = RMPP_STATE_ACTIVE;
+	atomic_set(&rmpp_recv->refcount, 1);
+
+	rmpp_recv->rmpp_wc = mad_recv_wc;
+	rmpp_recv->cur_seg_buf = &mad_recv_wc->recv_buf;
+	rmpp_recv->newwin = 1;
+	rmpp_recv->seg_num = 1;
+	rmpp_recv->last_ack = 0;
+
+	mad_hdr = &mad_recv_wc->recv_buf.mad->mad_hdr;
+	rmpp_recv->tid = mad_hdr->tid;
+	rmpp_recv->src_qp = mad_recv_wc->wc->src_qp;
+	rmpp_recv->slid = mad_recv_wc->wc->slid;
+	rmpp_recv->mgmt_class = mad_hdr->mgmt_class;
+	rmpp_recv->class_version = mad_hdr->class_version;
+	rmpp_recv->method  = mad_hdr->method;
+	return rmpp_recv;
+
+error:	kfree(rmpp_recv);
+	return NULL;
+}
+
+static inline void deref_rmpp_recv(struct mad_rmpp_recv *rmpp_recv)
+{
+	if (atomic_dec_and_test(&rmpp_recv->refcount))
+		wake_up(&rmpp_recv->wait);
+}
+
+static struct mad_rmpp_recv *
+find_rmpp_recv(struct ib_mad_agent_private *agent,
+	       struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	struct ib_mad_hdr *mad_hdr = &mad_recv_wc->recv_buf.mad->mad_hdr;
+
+	list_for_each_entry(rmpp_recv, &agent->rmpp_list, list) {
+		if (rmpp_recv->tid == mad_hdr->tid &&
+		    rmpp_recv->src_qp == mad_recv_wc->wc->src_qp &&
+		    rmpp_recv->slid == mad_recv_wc->wc->slid &&
+		    rmpp_recv->mgmt_class == mad_hdr->mgmt_class &&
+		    rmpp_recv->class_version == mad_hdr->class_version &&
+		    rmpp_recv->method == mad_hdr->method)
+			return rmpp_recv;
+	}
+	return NULL;
+}
+
+static struct mad_rmpp_recv *
+acquire_rmpp_recv(struct ib_mad_agent_private *agent,
+		  struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&agent->lock, flags);
+	rmpp_recv = find_rmpp_recv(agent, mad_recv_wc);
+	if (rmpp_recv)
+		atomic_inc(&rmpp_recv->refcount);
+	spin_unlock_irqrestore(&agent->lock, flags);
+	return rmpp_recv;
+}
+
+static struct mad_rmpp_recv *
+insert_rmpp_recv(struct ib_mad_agent_private *agent,
+		 struct mad_rmpp_recv *rmpp_recv)
+{
+	struct mad_rmpp_recv *cur_rmpp_recv;
+
+	cur_rmpp_recv = find_rmpp_recv(agent, rmpp_recv->rmpp_wc);
+	if (!cur_rmpp_recv)
+		list_add_tail(&rmpp_recv->list, &agent->rmpp_list);
+
+	return cur_rmpp_recv;
+}
+
+static int data_offset(u8 mgmt_class)
+{
+	if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM)
+		return offsetof(struct ib_sa_mad, data);
+	else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) &&
+		 (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END))
+		return offsetof(struct ib_vendor_mad, data);
+	else
+		return offsetof(struct ib_rmpp_mad, data);
+}
+
+static void format_ack(struct ib_rmpp_mad *ack,
+		       struct ib_rmpp_mad *data,
+		       struct mad_rmpp_recv *rmpp_recv)
+{
+	unsigned long flags;
+
+	memcpy(&ack->mad_hdr, &data->mad_hdr,
+	       data_offset(data->mad_hdr.mgmt_class));
+
+	ack->mad_hdr.method ^= IB_MGMT_METHOD_RESP;
+	ack->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_ACK;
+	ib_set_rmpp_flags(&ack->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
+
+	spin_lock_irqsave(&rmpp_recv->lock, flags);
+	rmpp_recv->last_ack = rmpp_recv->seg_num;
+	ack->rmpp_hdr.seg_num = cpu_to_be32(rmpp_recv->seg_num);
+	ack->rmpp_hdr.paylen_newwin = cpu_to_be32(rmpp_recv->newwin);
+	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+}
+
+static void ack_recv(struct mad_rmpp_recv *rmpp_recv,
+		     struct ib_mad_recv_wc *recv_wc)
+{
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	int hdr_len, ret;
+
+	hdr_len = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr);
+	msg = ib_create_send_mad(&rmpp_recv->agent->agent, recv_wc->wc->src_qp,
+				 recv_wc->wc->pkey_index, rmpp_recv->ah, 1,
+				 hdr_len, sizeof(struct ib_rmpp_mad) - hdr_len,
+				 GFP_KERNEL);
+	if (!msg)
+		return;
+
+	format_ack((struct ib_rmpp_mad *) msg->mad,
+		   (struct ib_rmpp_mad *) recv_wc->recv_buf.mad, rmpp_recv);
+	ret = ib_post_send_mad(&rmpp_recv->agent->agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		ib_free_send_mad(msg);
+}
+
+static inline int get_last_flag(struct ib_mad_recv_buf *seg)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *) seg->mad;
+	return ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_LAST;
+}
+
+static inline int get_seg_num(struct ib_mad_recv_buf *seg)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *) seg->mad;
+	return be32_to_cpu(rmpp_mad->rmpp_hdr.seg_num);
+}
+
+static inline struct ib_mad_recv_buf * get_next_seg(struct list_head *rmpp_list,
+						    struct ib_mad_recv_buf *seg)
+{
+	if (seg->list.next == rmpp_list)
+		return NULL;
+
+	return container_of(seg->list.next, struct ib_mad_recv_buf, list);
+}
+
+static inline int window_size(struct ib_mad_agent_private *agent)
+{
+	return max(agent->qp_info->recv_queue.max_active >> 3, 1);
+}
+
+static struct ib_mad_recv_buf * find_seg_location(struct list_head *rmpp_list,
+						  int seg_num)
+{
+        struct ib_mad_recv_buf *seg_buf;
+	int cur_seg_num;
+
+	list_for_each_entry_reverse(seg_buf, rmpp_list, list) {
+		cur_seg_num = get_seg_num(seg_buf);
+		if (seg_num > cur_seg_num)
+			return seg_buf;
+		if (seg_num == cur_seg_num)
+			break;
+	}
+	return NULL;
+}
+
+static void update_seg_num(struct mad_rmpp_recv *rmpp_recv,
+			   struct ib_mad_recv_buf *new_buf)
+{
+	struct list_head *rmpp_list = &rmpp_recv->rmpp_wc->rmpp_list;
+	
+	while (new_buf && (get_seg_num(new_buf) == rmpp_recv->seg_num + 1)) {
+		rmpp_recv->cur_seg_buf = new_buf;
+		rmpp_recv->seg_num++;
+		new_buf = get_next_seg(rmpp_list, new_buf);
+	}
+}
+
+static inline int get_mad_len(struct mad_rmpp_recv *rmpp_recv)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int hdr_size, data_size, pad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)rmpp_recv->cur_seg_buf->mad;
+
+	hdr_size = data_offset(rmpp_mad->mad_hdr.mgmt_class);
+	data_size = sizeof(struct ib_rmpp_mad) - hdr_size;
+	pad = data_size - be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+	if (pad > data_size || pad < 0)
+		pad = 0;
+
+	return hdr_size + rmpp_recv->seg_num * data_size - pad;
+}
+
+static struct ib_mad_recv_wc * complete_rmpp(struct mad_rmpp_recv *rmpp_recv)
+{
+	struct ib_mad_recv_wc *rmpp_wc;
+
+	ack_recv(rmpp_recv, rmpp_recv->rmpp_wc);
+	if (rmpp_recv->seg_num > 1)
+		cancel_delayed_work(&rmpp_recv->timeout_work);
+
+	rmpp_wc = rmpp_recv->rmpp_wc;
+	rmpp_wc->mad_len = get_mad_len(rmpp_recv);
+	/* 10 seconds until we can find the packet lifetime */
+	queue_delayed_work(rmpp_recv->agent->qp_info->port_priv->wq,
+			   &rmpp_recv->cleanup_work, msecs_to_jiffies(10000));
+	return rmpp_wc;
+}
+
+void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf)
+{
+	struct ib_mad_recv_buf *seg_buf;
+	struct ib_rmpp_mad *rmpp_mad;
+	void *data;
+	int size, len, offset;
+	u8 flags;
+
+	len = mad_recv_wc->mad_len;
+	if (len <= sizeof(struct ib_mad)) {
+		memcpy(buf, mad_recv_wc->recv_buf.mad, len);
+		return;
+	}
+
+	offset = data_offset(mad_recv_wc->recv_buf.mad->mad_hdr.mgmt_class);
+
+	list_for_each_entry(seg_buf, &mad_recv_wc->rmpp_list, list) {
+		rmpp_mad = (struct ib_rmpp_mad *)seg_buf->mad;
+		flags = ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr);
+
+		if (flags & IB_MGMT_RMPP_FLAG_FIRST) {
+			data = rmpp_mad;
+			size = sizeof(*rmpp_mad);
+		} else {
+			data = (void *) rmpp_mad + offset;
+			if (flags & IB_MGMT_RMPP_FLAG_LAST) 
+				size = len;
+			else
+				size = sizeof(*rmpp_mad) - offset;
+		}
+
+		memcpy(buf, data, size);
+		len -= size;
+		buf += size;
+	}
+}
+EXPORT_SYMBOL(ib_coalesce_recv_mad);
+
+static struct ib_mad_recv_wc *
+continue_rmpp(struct ib_mad_agent_private *agent,
+	      struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	struct ib_mad_recv_buf *prev_buf;
+	struct ib_mad_recv_wc *done_wc;
+	int seg_num;
+	unsigned long flags;
+
+	rmpp_recv = acquire_rmpp_recv(agent, mad_recv_wc);
+	if (!rmpp_recv)
+		goto drop1;
+
+	seg_num = get_seg_num(&mad_recv_wc->recv_buf);
+
+	spin_lock_irqsave(&rmpp_recv->lock, flags);
+	if ((rmpp_recv->state == RMPP_STATE_TIMEOUT) ||
+	    (seg_num > rmpp_recv->newwin))
+		goto drop3;
+
+	if ((seg_num <= rmpp_recv->last_ack) ||
+	    (rmpp_recv->state == RMPP_STATE_COMPLETE)) {
+		spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+		ack_recv(rmpp_recv, mad_recv_wc);
+		goto drop2;
+	}
+
+	prev_buf = find_seg_location(&rmpp_recv->rmpp_wc->rmpp_list, seg_num);
+	if (!prev_buf)
+		goto drop3;
+
+	done_wc = NULL;
+	list_add(&mad_recv_wc->recv_buf.list, &prev_buf->list);
+	if (rmpp_recv->cur_seg_buf == prev_buf) {
+		update_seg_num(rmpp_recv, &mad_recv_wc->recv_buf);
+		if (get_last_flag(rmpp_recv->cur_seg_buf)) {
+			rmpp_recv->state = RMPP_STATE_COMPLETE;
+			spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+			done_wc = complete_rmpp(rmpp_recv);
+			goto out;
+		} else if (rmpp_recv->seg_num == rmpp_recv->newwin) {
+			rmpp_recv->newwin += window_size(agent);
+			spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+			ack_recv(rmpp_recv, mad_recv_wc);
+			goto out;
+		}
+	}
+	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+out:
+	deref_rmpp_recv(rmpp_recv);
+	return done_wc;
+
+drop3:	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+drop2:	deref_rmpp_recv(rmpp_recv);
+drop1:	ib_free_recv_mad(mad_recv_wc);
+	return NULL;
+}
+
+static struct ib_mad_recv_wc *
+start_rmpp(struct ib_mad_agent_private *agent,
+	   struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	unsigned long flags;
+
+	rmpp_recv = create_rmpp_recv(agent, mad_recv_wc);
+	if (!rmpp_recv) {
+		ib_free_recv_mad(mad_recv_wc);
+		return NULL;
+	}
+
+	spin_lock_irqsave(&agent->lock, flags);
+	if (insert_rmpp_recv(agent, rmpp_recv)) {
+		spin_unlock_irqrestore(&agent->lock, flags);
+		/* duplicate first MAD */
+		destroy_rmpp_recv(rmpp_recv);
+		return continue_rmpp(agent, mad_recv_wc);
+	}
+	atomic_inc(&rmpp_recv->refcount);
+
+	if (get_last_flag(&mad_recv_wc->recv_buf)) {
+		rmpp_recv->state = RMPP_STATE_COMPLETE;
+		spin_unlock_irqrestore(&agent->lock, flags);
+		complete_rmpp(rmpp_recv);
+	} else {
+		spin_unlock_irqrestore(&agent->lock, flags);
+		/* 40 seconds until we can find the packet lifetimes */
+		queue_delayed_work(agent->qp_info->port_priv->wq,
+				   &rmpp_recv->timeout_work,
+				   msecs_to_jiffies(40000));
+		rmpp_recv->newwin += window_size(agent);
+		ack_recv(rmpp_recv, mad_recv_wc);
+		mad_recv_wc = NULL;
+	}
+	deref_rmpp_recv(rmpp_recv);
+	return mad_recv_wc;
+}
+
+static inline u64 get_seg_addr(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	return mad_send_wr->sg_list[0].addr + mad_send_wr->data_offset +
+	       (sizeof(struct ib_rmpp_mad) - mad_send_wr->data_offset) *
+	       (mad_send_wr->seg_num - 1);
+}
+
+static int send_next_seg(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int timeout;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	ib_set_rmpp_flags(&rmpp_mad->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
+	rmpp_mad->rmpp_hdr.seg_num = cpu_to_be32(mad_send_wr->seg_num);
+
+	if (mad_send_wr->seg_num == 1) {
+		rmpp_mad->rmpp_hdr.rmpp_rtime_flags |= IB_MGMT_RMPP_FLAG_FIRST;
+		rmpp_mad->rmpp_hdr.paylen_newwin =
+			cpu_to_be32(mad_send_wr->total_seg *
+				    (sizeof(struct ib_rmpp_mad) -
+				       offsetof(struct ib_rmpp_mad, data)));
+		mad_send_wr->sg_list[0].length = sizeof(struct ib_rmpp_mad);
+	} else {
+		mad_send_wr->send_wr.num_sge = 2;
+		mad_send_wr->sg_list[0].length = mad_send_wr->data_offset;
+		mad_send_wr->sg_list[1].addr = get_seg_addr(mad_send_wr);
+		mad_send_wr->sg_list[1].length = sizeof(struct ib_rmpp_mad) -
+						 mad_send_wr->data_offset;
+		mad_send_wr->sg_list[1].lkey = mad_send_wr->sg_list[0].lkey;
+	}
+
+	if (mad_send_wr->seg_num == mad_send_wr->total_seg) {
+		rmpp_mad->rmpp_hdr.rmpp_rtime_flags |= IB_MGMT_RMPP_FLAG_LAST;
+		rmpp_mad->rmpp_hdr.paylen_newwin =
+			cpu_to_be32(sizeof(struct ib_rmpp_mad) -
+				    offsetof(struct ib_rmpp_mad, data) -
+				    mad_send_wr->pad);
+	}
+
+	/* 2 seconds for an ACK until we can find the packet lifetime */
+	timeout = mad_send_wr->send_wr.wr.ud.timeout_ms;
+	if (!timeout || timeout > 2000)
+		mad_send_wr->timeout = msecs_to_jiffies(2000);
+	mad_send_wr->seg_num++;
+	return ib_send_mad(mad_send_wr);
+}
+
+static void process_rmpp_ack(struct ib_mad_agent_private *agent,
+			     struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_mad_send_wr_private *mad_send_wr;
+	struct ib_rmpp_mad *rmpp_mad;
+	unsigned long flags;
+	int seg_num, newwin, ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+	if (rmpp_mad->rmpp_hdr.rmpp_status)
+		return;
+
+	seg_num = be32_to_cpu(rmpp_mad->rmpp_hdr.seg_num);
+	newwin = be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+
+	spin_lock_irqsave(&agent->lock, flags);
+	mad_send_wr = ib_find_send_mad(agent, rmpp_mad->mad_hdr.tid);
+	if (!mad_send_wr)
+		goto out;	/* Unmatched ACK */
+
+	if ((mad_send_wr->last_ack == mad_send_wr->total_seg) ||
+	    (!mad_send_wr->timeout) || (mad_send_wr->status != IB_WC_SUCCESS))
+		goto out;	/* Send is already done */
+
+	if (seg_num > mad_send_wr->total_seg)
+		goto out;	/* Bad ACK */
+
+	if (newwin < mad_send_wr->newwin || seg_num < mad_send_wr->last_ack)
+		goto out;	/* Old ACK */
+
+	if (seg_num > mad_send_wr->last_ack) {
+		mad_send_wr->last_ack = seg_num;
+		mad_send_wr->retries = mad_send_wr->send_wr.wr.ud.retries;
+	}
+	mad_send_wr->newwin = newwin;
+	if (mad_send_wr->last_ack == mad_send_wr->total_seg) {
+		/* If no response is expected, the ACK completes the send */
+		if (!mad_send_wr->send_wr.wr.ud.timeout_ms) {
+			struct ib_mad_send_wc wc;
+
+			ib_mark_mad_done(mad_send_wr);
+			spin_unlock_irqrestore(&agent->lock, flags);
+
+			wc.status = IB_WC_SUCCESS;
+			wc.vendor_err = 0;
+			wc.wr_id = mad_send_wr->wr_id;
+			ib_mad_complete_send_wr(mad_send_wr, &wc);
+			return;
+		}
+		if (mad_send_wr->refcount == 1)
+			ib_reset_mad_timeout(mad_send_wr, mad_send_wr->
+					     send_wr.wr.ud.timeout_ms);
+	} else if (mad_send_wr->refcount == 1 &&
+		   mad_send_wr->seg_num < mad_send_wr->newwin &&
+		   mad_send_wr->seg_num <= mad_send_wr->total_seg) {
+		/* Send failure will just result in a timeout/retry */
+		ret = send_next_seg(mad_send_wr);
+		if (ret)
+			goto out;
+
+		mad_send_wr->refcount++;
+		list_del(&mad_send_wr->agent_list);
+		list_add_tail(&mad_send_wr->agent_list,
+			      &mad_send_wr->mad_agent_priv->send_list);
+	}
+out:
+	spin_unlock_irqrestore(&agent->lock, flags);
+}
+
+struct ib_mad_recv_wc *
+ib_process_rmpp_recv_wc(struct ib_mad_agent_private *agent,
+			struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+	if (!(rmpp_mad->rmpp_hdr.rmpp_rtime_flags & IB_MGMT_RMPP_FLAG_ACTIVE))
+		return mad_recv_wc;
+
+	if (rmpp_mad->rmpp_hdr.rmpp_version != IB_MGMT_RMPP_VERSION)
+		goto out;
+
+	switch (rmpp_mad->rmpp_hdr.rmpp_type) {
+	case IB_MGMT_RMPP_TYPE_DATA:
+		if (rmpp_mad->rmpp_hdr.seg_num == __constant_htonl(1))
+			return start_rmpp(agent, mad_recv_wc);
+		else
+			return continue_rmpp(agent, mad_recv_wc);
+	case IB_MGMT_RMPP_TYPE_ACK:
+		process_rmpp_ack(agent, mad_recv_wc);
+		break;
+	case IB_MGMT_RMPP_TYPE_STOP:
+	case IB_MGMT_RMPP_TYPE_ABORT:
+		/* TODO: process_rmpp_nack(agent, mad_recv_wc); */
+		break;
+	default:
+		break;
+	}
+out:
+	ib_free_recv_mad(mad_recv_wc);
+	return NULL;
+}
+
+int ib_send_rmpp_mad(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int i, total_len, ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	if (!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) &
+	      IB_MGMT_RMPP_FLAG_ACTIVE))
+		return IB_RMPP_RESULT_UNHANDLED;
+
+	if (rmpp_mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA)
+		return IB_RMPP_RESULT_INTERNAL;
+
+	if (mad_send_wr->send_wr.num_sge > 1)
+		return -EINVAL;		/* TODO: support num_sge > 1 */
+
+	mad_send_wr->seg_num = 1;
+	mad_send_wr->newwin = 1;
+	mad_send_wr->data_offset = data_offset(rmpp_mad->mad_hdr.mgmt_class);
+
+	total_len = 0;
+	for (i = 0; i < mad_send_wr->send_wr.num_sge; i++)
+		total_len += mad_send_wr->send_wr.sg_list[i].length;
+
+        mad_send_wr->total_seg = (total_len - mad_send_wr->data_offset) /
+			(sizeof(struct ib_rmpp_mad) - mad_send_wr->data_offset);
+	mad_send_wr->pad = total_len - offsetof(struct ib_rmpp_mad, data) -
+			   be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+
+	/* We need to wait for the final ACK even if there isn't a response */
+	mad_send_wr->refcount += (mad_send_wr->timeout == 0);
+	ret = send_next_seg(mad_send_wr);
+	if (!ret)
+		return IB_RMPP_RESULT_CONSUMED;
+	return ret;
+}
+
+int ib_process_rmpp_send_wc(struct ib_mad_send_wr_private *mad_send_wr,
+			    struct ib_mad_send_wc *mad_send_wc)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	struct ib_mad_send_buf *msg;
+	int ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	if (!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & 
+	      IB_MGMT_RMPP_FLAG_ACTIVE))
+		return IB_RMPP_RESULT_UNHANDLED; /* RMPP not active */
+
+	if (rmpp_mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA) {
+		msg = (struct ib_mad_send_buf *) (unsigned long)
+		      mad_send_wc->wr_id;
+		ib_free_send_mad(msg);
+		return IB_RMPP_RESULT_INTERNAL;	 /* ACK, STOP, or ABORT */
+	}
+
+	if (mad_send_wc->status != IB_WC_SUCCESS ||
+	    mad_send_wr->status != IB_WC_SUCCESS)
+		return IB_RMPP_RESULT_PROCESSED; /* Canceled or send error */
+
+	if (!mad_send_wr->timeout)
+		return IB_RMPP_RESULT_PROCESSED; /* Response received */
+
+	if (mad_send_wr->last_ack == mad_send_wr->total_seg) {
+		mad_send_wr->timeout =
+			msecs_to_jiffies(mad_send_wr->send_wr.wr.ud.timeout_ms);
+		return IB_RMPP_RESULT_PROCESSED; /* Send done */
+	}
+
+	if (mad_send_wr->seg_num > mad_send_wr->newwin ||
+	    mad_send_wr->seg_num > mad_send_wr->total_seg)
+		return IB_RMPP_RESULT_PROCESSED; /* Wait for ACK */
+
+	ret = send_next_seg(mad_send_wr);
+	if (ret) {
+		mad_send_wc->status = IB_WC_GENERAL_ERR;
+		return IB_RMPP_RESULT_PROCESSED;
+	}
+	return IB_RMPP_RESULT_CONSUMED;
+}
+
+int ib_retry_rmpp(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	if (!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & 
+	      IB_MGMT_RMPP_FLAG_ACTIVE))
+		return IB_RMPP_RESULT_UNHANDLED; /* RMPP not active */
+
+	if (mad_send_wr->last_ack == mad_send_wr->total_seg)
+		return IB_RMPP_RESULT_PROCESSED;
+
+	mad_send_wr->seg_num = mad_send_wr->last_ack + 1;
+	ret = send_next_seg(mad_send_wr);
+	if (ret)
+		return IB_RMPP_RESULT_PROCESSED;
+
+	return IB_RMPP_RESULT_CONSUMED;
+}
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad_rmpp.h linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad_rmpp.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband18/core/mad_rmpp.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband19/core/mad_rmpp.h	2005-07-06 16:55:32.000000000 -0400
@@ -0,0 +1,58 @@
+/*
+ * Copyright (c) 2005 Intel Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: mad_rmpp.h 1921 2005-02-25 22:58:44Z sean.hefty $
+ */
+
+#ifndef __MAD_RMPP_H__
+#define __MAD_RMPP_H__
+
+enum {
+	IB_RMPP_RESULT_PROCESSED,
+	IB_RMPP_RESULT_CONSUMED,
+	IB_RMPP_RESULT_INTERNAL,
+	IB_RMPP_RESULT_UNHANDLED
+};
+
+int ib_send_rmpp_mad(struct ib_mad_send_wr_private *mad_send_wr);
+
+struct ib_mad_recv_wc *
+ib_process_rmpp_recv_wc(struct ib_mad_agent_private *agent,
+			struct ib_mad_recv_wc *mad_recv_wc);
+
+int ib_process_rmpp_send_wc(struct ib_mad_send_wr_private *mad_send_wr,
+			    struct ib_mad_send_wc *mad_send_wc);
+
+void ib_cancel_rmpp_recvs(struct ib_mad_agent_private *agent);
+
+int ib_retry_rmpp(struct ib_mad_send_wr_private *mad_send_wr);
+
+#endif	/* __MAD_RMPP_H__ */
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband18/core/Makefile linux-2.6.13-rc2-mm1/drivers/infiniband19/core/Makefile
-- linux-2.6.13-rc2-mm1/drivers/infiniband18/core/Makefile	2005-07-10 16:30:24.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband19/core/Makefile	2005-07-10 16:32:15.000000000 -0400
@@ -6,7 +6,7 @@ obj-$(CONFIG_INFINIBAND_USER_VERBS) +=	i
 ib_core-y :=			packer.o ud_header.o verbs.o sysfs.o \
 				device.o fmr_pool.o cache.o
 
-ib_mad-y :=			mad.o smi.o agent.o
+ib_mad-y :=			mad.o smi.o agent.o mad_rmpp.o
 
 ib_sa-y :=			sa_query.o
 



From halr at voltaire.com  Mon Jul 11 07:43:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:43:14 -0400
Subject: [openib-general] [PATCH 20/27] Add Service Record support to SA
	client
Message-ID: <1121089173.4389.4545.camel@hal.voltaire.com>

Add Service Record support to SA client

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 19/27.

-- 
 core/sa_query.c |  166 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 include/ib_sa.h |   75 ++++++++++++++++++++++++-
 2 files changed, 236 insertions(+), 5 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband19/core/sa_query.c linux-2.6.13-rc2-mm1/drivers/infiniband20/core/sa_query.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband19/core/sa_query.c	2005-07-10 16:22:18.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband20/core/sa_query.c	2005-07-10 16:41:32.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: sa_query.c 1389 2004-12-27 22:56:47Z roland $
+ * $Id: sa_query.c 2811 2005-07-06 18:11:43Z halr $
  */
 
 #include <linux/module.h>
@@ -79,6 +80,12 @@ struct ib_sa_query {
 	int                 id;
 };
 
+struct ib_sa_service_query {
+	void (*callback)(int, struct ib_sa_service_rec *, void *);
+	void *context;
+	struct ib_sa_query sa_query;
+};
+
 struct ib_sa_path_query {
 	void (*callback)(int, struct ib_sa_path_rec *, void *);
 	void *context;
@@ -320,6 +327,54 @@ static const struct ib_field mcmember_re
 	  .size_bits    = 23 },
 };
 
+#define SERVICE_REC_FIELD(field) \
+	.struct_offset_bytes = offsetof(struct ib_sa_service_rec, field),	\
+	.struct_size_bytes   = sizeof ((struct ib_sa_service_rec *) 0)->field,	\
+	.field_name          = "sa_service_rec:" #field
+
+static const struct ib_field service_rec_table[] = {
+	{ SERVICE_REC_FIELD(id),
+	  .offset_words = 0,
+	  .offset_bits  = 0,
+	  .size_bits    = 64 },
+	{ SERVICE_REC_FIELD(gid),
+	  .offset_words = 2,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(pkey),
+	  .offset_words = 6,
+	  .offset_bits  = 0,
+	  .size_bits    = 16 },
+	{ SERVICE_REC_FIELD(lease),
+	  .offset_words = 7,
+	  .offset_bits  = 0,
+	  .size_bits    = 32 },
+	{ SERVICE_REC_FIELD(key),
+	  .offset_words = 8,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(name),
+	  .offset_words = 12,
+	  .offset_bits  = 0,
+	  .size_bits    = 64*8 },
+	{ SERVICE_REC_FIELD(data8),
+	  .offset_words = 28,
+	  .offset_bits  = 0,
+	  .size_bits    = 16*8 },
+	{ SERVICE_REC_FIELD(data16),
+	  .offset_words = 32,
+	  .offset_bits  = 0,
+	  .size_bits    = 8*16 },
+	{ SERVICE_REC_FIELD(data32),
+	  .offset_words = 36,
+	  .offset_bits  = 0,
+	  .size_bits    = 4*32 },
+	{ SERVICE_REC_FIELD(data64),
+	  .offset_words = 40,
+	  .offset_bits  = 0,
+	  .size_bits    = 2*64 },
+};
+
 static void free_sm_ah(struct kref *kref)
 {
 	struct ib_sa_sm_ah *sm_ah = container_of(kref, struct ib_sa_sm_ah, ref);
@@ -443,7 +498,6 @@ static int send_mad(struct ib_sa_query *
 				 .remote_qpn  = 1,
 				 .remote_qkey = IB_QP1_QKEY,
 				 .timeout_ms  = timeout_ms,
-				 .retries     = 0 
 			 }
 		 }
 	};
@@ -596,6 +650,114 @@ int ib_sa_path_rec_get(struct ib_device 
 }
 EXPORT_SYMBOL(ib_sa_path_rec_get);
 
+static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query,
+				    int status,
+				    struct ib_sa_mad *mad)
+{
+	struct ib_sa_service_query *query =
+		container_of(sa_query, struct ib_sa_service_query, sa_query);
+
+	if (mad) {
+		struct ib_sa_service_rec rec;
+
+		ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table),
+			  mad->data, &rec);
+		query->callback(status, &rec, query->context);
+	} else
+		query->callback(status, NULL, query->context);
+}
+
+static void ib_sa_service_rec_release(struct ib_sa_query *sa_query)
+{
+	kfree(sa_query->mad);
+	kfree(container_of(sa_query, struct ib_sa_service_query, sa_query));
+}
+
+/**
+ * ib_sa_service_rec_query - Start Service Record operation
+ * @device:device to send request on
+ * @port_num: port number to send request on
+ * @method:SA method - should be get, set, or delete
+ * @rec:Service Record to send in request
+ * @comp_mask:component mask to send in request
+ * @timeout_ms:time to wait for response
+ * @gfp_mask:GFP mask to use for internal allocations
+ * @callback:function called when request completes, times out or is
+ * canceled
+ * @context:opaque user context passed to callback
+ * @sa_query:request context, used to cancel request
+ *
+ * Send a Service Record set/get/delete to the SA to register,
+ * unregister or query a service record.
+ * The callback function will be called when the request completes (or
+ * fails); status is 0 for a successful response, -EINTR if the query
+ * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error
+ * occurred sending the query.  The resp parameter of the callback is
+ * only valid if status is 0.
+ *
+ * If the return value of ib_sa_service_rec_query() is negative, it is an
+ * error code.  Otherwise it is a request ID that can be used to cancel
+ * the query.
+ */
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
+			    struct ib_sa_service_rec *rec,
+			    ib_sa_comp_mask comp_mask,
+			    int timeout_ms, int gfp_mask,
+			    void (*callback)(int status,
+					     struct ib_sa_service_rec *resp,
+					     void *context),
+			    void *context,
+			    struct ib_sa_query **sa_query)
+{
+	struct ib_sa_service_query *query;
+	struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client);
+	struct ib_sa_port   *port   = &sa_dev->port[port_num - sa_dev->start_port];
+	struct ib_mad_agent *agent  = port->agent;
+	int ret;
+
+	if (method != IB_MGMT_METHOD_GET &&
+	    method != IB_MGMT_METHOD_SET &&
+	    method != IB_SA_METHOD_DELETE)
+		return -EINVAL;
+
+	query = kmalloc(sizeof *query, gfp_mask);
+	if (!query)
+		return -ENOMEM;
+	query->sa_query.mad = kmalloc(sizeof *query->sa_query.mad, gfp_mask);
+	if (!query->sa_query.mad) {
+		kfree(query);
+		return -ENOMEM;
+	}
+
+	query->callback = callback;
+	query->context  = context;
+
+	init_mad(query->sa_query.mad, agent);
+
+	query->sa_query.callback              = callback ? ib_sa_service_rec_callback : NULL;
+	query->sa_query.release               = ib_sa_service_rec_release;
+	query->sa_query.port                  = port;
+	query->sa_query.mad->mad_hdr.method   = method;
+	query->sa_query.mad->mad_hdr.attr_id  =
+				cpu_to_be16(IB_SA_ATTR_SERVICE_REC);
+	query->sa_query.mad->sa_hdr.comp_mask = comp_mask;
+
+	ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table),
+		rec, query->sa_query.mad->data);
+
+	*sa_query = &query->sa_query;
+
+	ret = send_mad(&query->sa_query, timeout_ms);
+	if (ret < 0) {
+		*sa_query = NULL;
+		kfree(query->sa_query.mad);
+		kfree(query);
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(ib_sa_service_rec_query);
+
 static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query,
 					int status,
 					struct ib_sa_mad *mad)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband19/include/ib_sa.h linux-2.6.13-rc2-mm1/drivers/infiniband20/include/ib_sa.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband19/include/ib_sa.h	2005-07-10 12:07:41.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband20/include/ib_sa.h	2005-07-10 16:40:43.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_sa.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: ib_sa.h 2811 2005-07-06 18:11:43Z halr $
  */
 
 #ifndef IB_SA_H
@@ -41,9 +42,11 @@
 #include <ib_mad.h>
 
 enum {
-	IB_SA_CLASS_VERSION	= 2,	/* IB spec version 1.1/1.2 */
+	IB_SA_CLASS_VERSION		= 2,	/* IB spec version 1.1/1.2 */
 
-	IB_SA_METHOD_DELETE	= 0x15
+	IB_SA_METHOD_GET_TABLE		= 0x12,
+	IB_SA_METHOD_GET_TABLE_RESP	= 0x92,
+	IB_SA_METHOD_DELETE		= 0x15
 };
 
 enum ib_sa_selector {
@@ -191,6 +194,61 @@ struct ib_sa_mcmember_rec {
 	int          proxy_join;
 };
 
+/* Service Record Component Mask Sec 15.2.5.14 Ver 1.1	*/
+#define IB_SA_SERVICE_REC_SERVICE_ID			IB_SA_COMP_MASK( 0)
+#define IB_SA_SERVICE_REC_SERVICE_GID			IB_SA_COMP_MASK( 1)
+#define IB_SA_SERVICE_REC_SERVICE_PKEY			IB_SA_COMP_MASK( 2)
+/* reserved:								 3 */
+#define IB_SA_SERVICE_REC_SERVICE_LEASE			IB_SA_COMP_MASK( 4)
+#define IB_SA_SERVICE_REC_SERVICE_KEY			IB_SA_COMP_MASK( 5)
+#define IB_SA_SERVICE_REC_SERVICE_NAME			IB_SA_COMP_MASK( 6)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_0		IB_SA_COMP_MASK( 7)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_1		IB_SA_COMP_MASK( 8)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_2		IB_SA_COMP_MASK( 9)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_3		IB_SA_COMP_MASK(10)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_4		IB_SA_COMP_MASK(11)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_5		IB_SA_COMP_MASK(12)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_6		IB_SA_COMP_MASK(13)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_7		IB_SA_COMP_MASK(14)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_8		IB_SA_COMP_MASK(15)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_9		IB_SA_COMP_MASK(16)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_10		IB_SA_COMP_MASK(17)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_11		IB_SA_COMP_MASK(18)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_12		IB_SA_COMP_MASK(19)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_13		IB_SA_COMP_MASK(20)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_14		IB_SA_COMP_MASK(21)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_15		IB_SA_COMP_MASK(22)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_0		IB_SA_COMP_MASK(23)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_1		IB_SA_COMP_MASK(24)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_2		IB_SA_COMP_MASK(25)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_3		IB_SA_COMP_MASK(26)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_4		IB_SA_COMP_MASK(27)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_5		IB_SA_COMP_MASK(28)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_6		IB_SA_COMP_MASK(29)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_7		IB_SA_COMP_MASK(30)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_0		IB_SA_COMP_MASK(31)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_1		IB_SA_COMP_MASK(32)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_2		IB_SA_COMP_MASK(33)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_3		IB_SA_COMP_MASK(34)
+#define IB_SA_SERVICE_REC_SERVICE_DATA64_0		IB_SA_COMP_MASK(35)
+#define IB_SA_SERVICE_REC_SERVICE_DATA64_1		IB_SA_COMP_MASK(36)
+
+#define IB_DEFAULT_SERVICE_LEASE 	0xFFFFFFFF
+
+struct ib_sa_service_rec {
+	u64		id;
+	union ib_gid	gid;		
+	u16 		pkey;
+	/* reserved */
+	u32		lease;
+	u8		key[16];
+	u8		name[64];
+	u8		data8[16];
+	u16		data16[8];
+	u32		data32[4];
+	u64		data64[2];
+};
+
 struct ib_sa_query;
 
 void ib_sa_cancel_query(int id, struct ib_sa_query *query);
@@ -216,6 +274,17 @@ int ib_sa_mcmember_rec_query(struct ib_d
 			     void *context,
 			     struct ib_sa_query **query);
 
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num,
+			 u8 method,
+			 struct ib_sa_service_rec *rec,
+			 ib_sa_comp_mask comp_mask,
+			 int timeout_ms, int gfp_mask,
+			 void (*callback)(int status,
+					  struct ib_sa_service_rec *resp,
+					  void *context),
+			 void *context,
+			 struct ib_sa_query **sa_query);
+
 /**
  * ib_sa_mcmember_rec_set - Start an MCMember set query
  * @device:device to send query on




From halr at voltaire.com  Mon Jul 11 07:46:28 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:46:28 -0400
Subject: [openib-general] [PATCH 21/27] Add the header file for kernel CM
	(Communications Manager)
Message-ID: <1121089182.4389.4547.camel@hal.voltaire.com>

Add the header file for kernel CM (Communications Manager)

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

-- 
 ib_cm.h |  568 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 568 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband20/include/ib_cm.h linux-2.6.13-rc2-mm1/drivers/infiniband21/include/ib_cm.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband20/include/ib_cm.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband21/include/ib_cm.h	2005-06-28 13:07:34.000000000 -0400
@@ -0,0 +1,568 @@
+/*
+ * Copyright (c) 2004 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ib_cm.h 2730 2005-06-28 16:43:03Z sean.hefty $
+ */
+#if !defined(IB_CM_H)
+#define IB_CM_H
+
+#include <ib_mad.h>
+#include <ib_sa.h>
+
+enum ib_cm_state {
+	IB_CM_IDLE,
+	IB_CM_LISTEN,
+	IB_CM_REQ_SENT,
+	IB_CM_REQ_RCVD,
+	IB_CM_MRA_REQ_SENT,
+	IB_CM_MRA_REQ_RCVD,
+	IB_CM_REP_SENT,
+	IB_CM_REP_RCVD,
+	IB_CM_MRA_REP_SENT,
+	IB_CM_MRA_REP_RCVD,
+	IB_CM_ESTABLISHED,
+	IB_CM_DREQ_SENT,
+	IB_CM_DREQ_RCVD,
+	IB_CM_TIMEWAIT,
+	IB_CM_SIDR_REQ_SENT,
+	IB_CM_SIDR_REQ_RCVD
+};
+
+enum ib_cm_lap_state {
+	IB_CM_LAP_IDLE,
+	IB_CM_LAP_SENT,
+	IB_CM_LAP_RCVD,
+	IB_CM_MRA_LAP_SENT,
+	IB_CM_MRA_LAP_RCVD,
+};
+
+enum ib_cm_event_type {
+	IB_CM_REQ_ERROR,
+	IB_CM_REQ_RECEIVED,
+	IB_CM_REP_ERROR,
+	IB_CM_REP_RECEIVED,
+	IB_CM_RTU_RECEIVED,
+	IB_CM_USER_ESTABLISHED,
+	IB_CM_DREQ_ERROR,
+	IB_CM_DREQ_RECEIVED,
+	IB_CM_DREP_RECEIVED,
+	IB_CM_TIMEWAIT_EXIT,
+	IB_CM_MRA_RECEIVED,
+	IB_CM_REJ_RECEIVED,
+	IB_CM_LAP_ERROR,
+	IB_CM_LAP_RECEIVED,
+	IB_CM_APR_RECEIVED,
+	IB_CM_SIDR_REQ_ERROR,
+	IB_CM_SIDR_REQ_RECEIVED,
+	IB_CM_SIDR_REP_RECEIVED
+};
+
+enum ib_cm_data_size {
+	IB_CM_REQ_PRIVATE_DATA_SIZE	 = 92,
+	IB_CM_MRA_PRIVATE_DATA_SIZE	 = 222,
+	IB_CM_REJ_PRIVATE_DATA_SIZE	 = 148,
+	IB_CM_REP_PRIVATE_DATA_SIZE	 = 196,
+	IB_CM_RTU_PRIVATE_DATA_SIZE	 = 224,
+	IB_CM_DREQ_PRIVATE_DATA_SIZE	 = 220,
+	IB_CM_DREP_PRIVATE_DATA_SIZE	 = 224,
+	IB_CM_REJ_ARI_LENGTH		 = 72,
+	IB_CM_LAP_PRIVATE_DATA_SIZE	 = 168,
+	IB_CM_APR_PRIVATE_DATA_SIZE	 = 148,
+	IB_CM_APR_INFO_LENGTH		 = 72,
+	IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE = 216,
+	IB_CM_SIDR_REP_PRIVATE_DATA_SIZE = 136,
+	IB_CM_SIDR_REP_INFO_LENGTH	 = 72
+};
+
+struct ib_cm_id;
+
+struct ib_cm_req_event_param {
+	struct ib_cm_id		*listen_id;
+	struct ib_device	*device;
+	u8			port;
+
+	struct ib_sa_path_rec	*primary_path;
+	struct ib_sa_path_rec	*alternate_path;
+
+	u64			remote_ca_guid;
+	u32			remote_qkey;
+	u32			remote_qpn;
+	enum ib_qp_type		qp_type;
+
+	u32			starting_psn;
+	u8			responder_resources;
+	u8			initiator_depth;
+	unsigned int		local_cm_response_timeout:5;
+	unsigned int		flow_control:1;
+	unsigned int		remote_cm_response_timeout:5;
+	unsigned int		retry_count:3;
+	unsigned int		rnr_retry_count:3;
+	unsigned int		srq:1;
+};
+
+struct ib_cm_rep_event_param {
+	u64			remote_ca_guid;
+	u32			remote_qkey;
+	u32			remote_qpn;
+	u32			starting_psn;
+	u8			responder_resources;
+	u8			initiator_depth;
+	unsigned int		target_ack_delay:5;
+	unsigned int		failover_accepted:2;
+	unsigned int		flow_control:1;
+	unsigned int		rnr_retry_count:3;
+	unsigned int		srq:1;
+};
+
+enum ib_cm_rej_reason {
+	IB_CM_REJ_NO_QP				= __constant_htons(1),
+	IB_CM_REJ_NO_EEC			= __constant_htons(2),
+	IB_CM_REJ_NO_RESOURCES			= __constant_htons(3),
+	IB_CM_REJ_TIMEOUT			= __constant_htons(4),
+	IB_CM_REJ_UNSUPPORTED			= __constant_htons(5),
+	IB_CM_REJ_INVALID_COMM_ID		= __constant_htons(6),
+	IB_CM_REJ_INVALID_COMM_INSTANCE		= __constant_htons(7),
+	IB_CM_REJ_INVALID_SERVICE_ID		= __constant_htons(8),
+	IB_CM_REJ_INVALID_TRANSPORT_TYPE	= __constant_htons(9),
+	IB_CM_REJ_STALE_CONN			= __constant_htons(10),
+	IB_CM_REJ_RDC_NOT_EXIST			= __constant_htons(11),
+	IB_CM_REJ_INVALID_GID			= __constant_htons(12),
+	IB_CM_REJ_INVALID_LID			= __constant_htons(13),
+	IB_CM_REJ_INVALID_SL			= __constant_htons(14),
+	IB_CM_REJ_INVALID_TRAFFIC_CLASS		= __constant_htons(15),
+	IB_CM_REJ_INVALID_HOP_LIMIT		= __constant_htons(16),
+	IB_CM_REJ_INVALID_PACKET_RATE		= __constant_htons(17),
+	IB_CM_REJ_INVALID_ALT_GID		= __constant_htons(18),
+	IB_CM_REJ_INVALID_ALT_LID		= __constant_htons(19),
+	IB_CM_REJ_INVALID_ALT_SL		= __constant_htons(20),
+	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= __constant_htons(21),
+	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= __constant_htons(22),
+	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= __constant_htons(23),
+	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(24),
+	IB_CM_REJ_INVALID_MTU			= __constant_htons(26),
+	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= __constant_htons(27),
+	IB_CM_REJ_CONSUMER_DEFINED		= __constant_htons(28),
+	IB_CM_REJ_INVALID_RNR_RETRY		= __constant_htons(29),
+	IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID	= __constant_htons(30),
+	IB_CM_REJ_INVALID_CLASS_VERSION		= __constant_htons(31),
+	IB_CM_REJ_INVALID_FLOW_LABEL		= __constant_htons(32),
+	IB_CM_REJ_INVALID_ALT_FLOW_LABEL	= __constant_htons(33)
+};
+
+struct ib_cm_rej_event_param {
+	enum ib_cm_rej_reason	reason;
+	void			*ari;
+	u8			ari_length;
+};
+
+struct ib_cm_mra_event_param {
+	u8	service_timeout;
+};
+
+struct ib_cm_lap_event_param {
+	struct ib_sa_path_rec	*alternate_path;
+};
+
+enum ib_cm_apr_status {
+	IB_CM_APR_SUCCESS,
+	IB_CM_APR_INVALID_COMM_ID,
+	IB_CM_APR_UNSUPPORTED,
+	IB_CM_APR_REJECT,
+	IB_CM_APR_REDIRECT,
+	IB_CM_APR_IS_CURRENT,
+	IB_CM_APR_INVALID_QPN_EECN,
+	IB_CM_APR_INVALID_LID,
+	IB_CM_APR_INVALID_GID,
+	IB_CM_APR_INVALID_FLOW_LABEL,
+	IB_CM_APR_INVALID_TCLASS,
+	IB_CM_APR_INVALID_HOP_LIMIT,
+	IB_CM_APR_INVALID_PACKET_RATE,
+	IB_CM_APR_INVALID_SL
+};
+
+struct ib_cm_apr_event_param {
+	enum ib_cm_apr_status	ap_status;
+	void			*apr_info;
+	u8			info_len;
+};
+
+struct ib_cm_sidr_req_event_param {
+	struct ib_cm_id		*listen_id;
+	struct ib_device	*device;
+	u8			port;
+
+	u16	pkey;
+};
+
+enum ib_cm_sidr_status {
+	IB_SIDR_SUCCESS,
+	IB_SIDR_UNSUPPORTED,
+	IB_SIDR_REJECT,
+	IB_SIDR_NO_QP,
+	IB_SIDR_REDIRECT,
+	IB_SIDR_UNSUPPORTED_VERSION
+};
+
+struct ib_cm_sidr_rep_event_param {
+	enum ib_cm_sidr_status	status;
+	u32			qkey;
+	u32			qpn;
+	void			*info;
+	u8			info_len;
+
+};
+
+struct ib_cm_event {
+	enum ib_cm_event_type	event;
+	union {
+		struct ib_cm_req_event_param	req_rcvd;
+		struct ib_cm_rep_event_param	rep_rcvd;
+		/* No data for RTU received events. */
+		struct ib_cm_rej_event_param	rej_rcvd;
+		struct ib_cm_mra_event_param	mra_rcvd;
+		struct ib_cm_lap_event_param	lap_rcvd;
+		struct ib_cm_apr_event_param	apr_rcvd;
+		/* No data for DREQ/DREP received events. */
+		struct ib_cm_sidr_req_event_param sidr_req_rcvd;
+		struct ib_cm_sidr_rep_event_param sidr_rep_rcvd;
+		enum ib_wc_status		send_status;
+	} param;
+
+	void			*private_data;
+};
+
+/**
+ * ib_cm_handler - User-defined callback to process communication events.
+ * @cm_id: Communication identifier associated with the reported event.
+ * @event: Information about the communication event.
+ *
+ * IB_CM_REQ_RECEIVED and IB_CM_SIDR_REQ_RECEIVED communication events 
+ * generated as a result of listen requests result in the allocation of a 
+ * new @cm_id.  The new @cm_id is returned to the user through this callback.
+ * Clients are responsible for destroying the new @cm_id.  For peer-to-peer
+ * IB_CM_REQ_RECEIVED and all other events, the returned @cm_id corresponds
+ * to a user's existing communication identifier.
+ *
+ * Users may not call ib_destroy_cm_id while in the context of this callback;
+ * however, returning a non-zero value instructs the communication manager to
+ * destroy the @cm_id after the callback completes. 
+ */
+typedef int (*ib_cm_handler)(struct ib_cm_id *cm_id,
+			     struct ib_cm_event *event);
+
+struct ib_cm_id {
+	ib_cm_handler		cm_handler;
+	void			*context;
+	u64			service_id;
+	u64			service_mask;
+	enum ib_cm_state	state;		/* internal CM/debug use */
+	enum ib_cm_lap_state	lap_state;	/* internal CM/debug use */
+	u32			local_id;
+	u32			remote_id;
+};
+
+/**
+ * ib_create_cm_id - Allocate a communication identifier.
+ * @cm_handler: Callback invoked to notify the user of CM events.
+ * @context: User specified context associated with the communication
+ *   identifier.
+ *
+ * Communication identifiers are used to track connection states, service
+ * ID resolution requests, and listen requests.
+ */
+struct ib_cm_id *ib_create_cm_id(ib_cm_handler cm_handler,
+				 void *context);
+
+/**
+ * ib_destroy_cm_id - Destroy a connection identifier.
+ * @cm_id: Connection identifier to destroy.
+ *
+ * This call blocks until the connection identifier is destroyed.
+ */
+void ib_destroy_cm_id(struct ib_cm_id *cm_id);
+
+#define IB_SERVICE_ID_AGN_MASK	__constant_cpu_to_be64(0xFF00000000000000ULL)
+#define IB_CM_ASSIGN_SERVICE_ID __constant_cpu_to_be64(0x0200000000000000ULL)
+
+/**
+ * ib_cm_listen - Initiates listening on the specified service ID for
+ *   connection and service ID resolution requests.
+ * @cm_id: Connection identifier associated with the listen request.
+ * @service_id: Service identifier matched against incoming connection
+ *   and service ID resolution requests.  The service ID should be specified
+ *   network-byte order.  If set to IB_CM_ASSIGN_SERVICE_ID, the CM will
+ *   assign a service ID to the caller.
+ * @service_mask: Mask applied to service ID used to listen across a
+ *   range of service IDs.  If set to 0, the service ID is matched
+ *   exactly.  This parameter is ignored if %service_id is set to
+ *   IB_CM_ASSIGN_SERVICE_ID.
+ */
+int ib_cm_listen(struct ib_cm_id *cm_id,
+		 u64 service_id,
+		 u64 service_mask);
+
+struct ib_cm_req_param {
+	struct ib_sa_path_rec	*primary_path;
+	struct ib_sa_path_rec	*alternate_path;
+	u64			service_id;
+	u32			qp_num;
+	enum ib_qp_type		qp_type;
+	u32			starting_psn;
+	const void		*private_data;
+	u8			private_data_len;
+	u8			peer_to_peer;
+	u8			responder_resources;
+	u8			initiator_depth;
+	u8			remote_cm_response_timeout;
+	u8			flow_control;
+	u8			local_cm_response_timeout;
+	u8			retry_count;
+	u8			rnr_retry_count;
+	u8			max_cm_retries;
+	u8			srq;
+};
+
+/**
+ * ib_send_cm_req - Sends a connection request to the remote node.
+ * @cm_id: Connection identifier that will be associated with the
+ *   connection request.
+ * @param: Connection request information needed to establish the
+ *   connection.
+ */
+int ib_send_cm_req(struct ib_cm_id *cm_id,
+		   struct ib_cm_req_param *param);
+
+struct ib_cm_rep_param {
+	u32		qp_num;
+	u32		starting_psn;
+	const void	*private_data;
+	u8		private_data_len;
+	u8		responder_resources;
+	u8		initiator_depth;
+	u8		target_ack_delay;
+	u8		failover_accepted;
+	u8		flow_control;
+	u8		rnr_retry_count;
+	u8		srq;
+};
+
+/**
+ * ib_send_cm_rep - Sends a connection reply in response to a connection
+ *   request.
+ * @cm_id: Connection identifier that will be associated with the
+ *   connection request.
+ * @param: Connection reply information needed to establish the
+ *   connection.
+ */
+int ib_send_cm_rep(struct ib_cm_id *cm_id,
+		   struct ib_cm_rep_param *param);
+
+/**
+ * ib_send_cm_rtu - Sends a connection ready to use message in response
+ *   to a connection reply message.
+ * @cm_id: Connection identifier associated with the connection request.
+ * @private_data: Optional user-defined private data sent with the
+ *   ready to use message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_rtu(struct ib_cm_id *cm_id,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_send_cm_dreq - Sends a disconnection request for an existing
+ *   connection.
+ * @cm_id: Connection identifier associated with the connection being
+ *   released.
+ * @private_data: Optional user-defined private data sent with the
+ *   disconnection request message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_dreq(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len);
+
+/**
+ * ib_send_cm_drep - Sends a disconnection reply to a disconnection request.
+ * @cm_id: Connection identifier associated with the connection being
+ *   released.
+ * @private_data: Optional user-defined private data sent with the
+ *   disconnection reply message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ *
+ * If the cm_id is in the correct state, the CM will transition the connection
+ * to the timewait state, even if an error occurs sending the DREP message.
+ */
+int ib_send_cm_drep(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len);
+
+/**
+ * ib_cm_establish - Forces a connection state to established.
+ * @cm_id: Connection identifier to transition to established.
+ *
+ * This routine should be invoked by users who receive messages on a
+ * connected QP before an RTU has been received.
+ */
+int ib_cm_establish(struct ib_cm_id *cm_id);
+
+/**
+ * ib_send_cm_rej - Sends a connection rejection message to the
+ *   remote node.
+ * @cm_id: Connection identifier associated with the connection being
+ *   rejected.
+ * @reason: Reason for the connection request rejection.
+ * @ari: Optional additional rejection information.
+ * @ari_length: Size of the additional rejection information, in bytes.
+ * @private_data: Optional user-defined private data sent with the
+ *   rejection message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_rej(struct ib_cm_id *cm_id,
+		   enum ib_cm_rej_reason reason,
+		   void *ari,
+		   u8 ari_length,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_send_cm_mra - Sends a message receipt acknowledgement to a connection
+ *   message.
+ * @cm_id: Connection identifier associated with the connection message.
+ * @service_timeout: The maximum time required for the sender to reply to
+ *   to the connection message.
+ * @private_data: Optional user-defined private data sent with the
+ *   message receipt acknowledgement.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_mra(struct ib_cm_id *cm_id,
+		   u8 service_timeout,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_send_cm_lap - Sends a load alternate path request.
+ * @cm_id: Connection identifier associated with the load alternate path
+ *   message.
+ * @alternate_path: A path record that identifies the alternate path to
+ *   load.
+ * @private_data: Optional user-defined private data sent with the
+ *   load alternate path message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_lap(struct ib_cm_id *cm_id,
+		   struct ib_sa_path_rec *alternate_path,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_cm_init_qp_attr - Initializes the QP attributes for use in transitioning
+ *   to a specified QP state.
+ * @cm_id: Communication identifier associated with the QP attributes to
+ *   initialize.
+ * @qp_attr: On input, specifies the desired QP state.  On output, the
+ *   mandatory and desired optional attributes will be set in order to
+ *   modify the QP to the specified state.
+ * @qp_attr_mask: The QP attribute mask that may be used to transition the
+ *   QP to the specified state.
+ *
+ * Users must set the @qp_attr->qp_state to the desired QP state.  This call
+ * will set all required attributes for the given transition, along with
+ * known optional attributes.  Users may override the attributes returned from
+ * this call before calling ib_modify_qp.
+ */
+int ib_cm_init_qp_attr(struct ib_cm_id *cm_id,
+		       struct ib_qp_attr *qp_attr,
+		       int *qp_attr_mask);
+
+/**
+ * ib_send_cm_apr - Sends an alternate path response message in response to
+ *   a load alternate path request.
+ * @cm_id: Connection identifier associated with the alternate path response.
+ * @status: Reply status sent with the alternate path response.
+ * @info: Optional additional information sent with the alternate path
+ *   response.
+ * @info_length: Size of the additional information, in bytes.
+ * @private_data: Optional user-defined private data sent with the
+ *   alternate path response message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_apr(struct ib_cm_id *cm_id,
+		   enum ib_cm_apr_status status,
+		   void *info,
+		   u8 info_length,
+		   const void *private_data,
+		   u8 private_data_len);
+
+struct ib_cm_sidr_req_param {
+	struct ib_sa_path_rec	*path;
+	u64			service_id;
+	int			timeout_ms;
+	const void		*private_data;
+	u8			private_data_len;
+	u8			max_cm_retries;
+	u16			pkey;
+};
+
+/**
+ * ib_send_cm_sidr_req - Sends a service ID resolution request to the
+ *   remote node.
+ * @cm_id: Communication identifier that will be associated with the
+ *   service ID resolution request.
+ * @param: Service ID resolution request information.
+ */
+int ib_send_cm_sidr_req(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_req_param *param);
+
+struct ib_cm_sidr_rep_param {
+	u32			qp_num;
+	u32			qkey;
+	enum ib_cm_sidr_status	status;
+	const void		*info;
+	u8			info_length;
+	const void		*private_data;
+	u8			private_data_len;
+};
+
+/**
+ * ib_send_cm_sidr_rep - Sends a service ID resolution request to the
+ *   remote node.
+ * @cm_id: Communication identifier associated with the received service ID
+ *   resolution request.
+ * @param: Service ID resolution reply information.
+ */
+int ib_send_cm_sidr_rep(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_rep_param *param);
+
+#endif /* IB_CM_H */




From halr at voltaire.com  Mon Jul 11 07:49:42 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:49:42 -0400
Subject: [openib-general] [PATCH 22/27] Add the CM implementation
Message-ID: <1121089235.4389.4549.camel@hal.voltaire.com>

Add the CM implementation 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 21/27.

-- 
 Makefile  |    5 
 cm.c      | 3324 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 cm_msgs.h |  819 +++++++++++++++
 3 files changed, 4147 insertions(+), 1 deletion(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband21/core/cm.c linux-2.6.13-rc2-mm1/drivers/infiniband22/core/cm.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband21/core/cm.c	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband22/core/cm.c	2005-07-10 16:14:28.000000000 -0400
@@ -0,0 +1,3324 @@
+/*
+ * Copyright (c) 2004, 2005 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004, 2005 Voltaire Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: cm.c 2821 2005-07-08 17:07:28Z sean.hefty $
+ */
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/idr.h>
+#include <linux/interrupt.h>
+#include <linux/pci.h>
+#include <linux/rbtree.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#include <ib_cache.h>
+#include <ib_cm.h>
+#include "cm_msgs.h"
+
+MODULE_AUTHOR("Sean Hefty");
+MODULE_DESCRIPTION("InfiniBand CM");
+MODULE_LICENSE("Dual BSD/GPL");
+
+static void cm_add_one(struct ib_device *device);
+static void cm_remove_one(struct ib_device *device);
+
+static struct ib_client cm_client = {
+	.name   = "cm",
+	.add    = cm_add_one,
+	.remove = cm_remove_one
+};
+
+static struct ib_cm {
+	spinlock_t lock;
+	struct list_head device_list;
+	rwlock_t device_lock;
+	struct rb_root listen_service_table;
+	u64 listen_service_id;
+	/* struct rb_root peer_service_table; todo: fix peer to peer */
+	struct rb_root remote_qp_table;
+	struct rb_root remote_id_table;
+	struct rb_root remote_sidr_table;
+	struct idr local_id_table;
+	struct workqueue_struct *wq;
+} cm;
+
+struct cm_port {
+	struct cm_device *cm_dev;
+	struct ib_mad_agent *mad_agent;
+	u8 port_num;
+};
+
+struct cm_device {
+	struct list_head list;
+	struct ib_device *device;
+	u64 ca_guid;
+	struct cm_port port[0];
+};
+
+struct cm_av {
+	struct cm_port *port;
+	union ib_gid dgid;
+	struct ib_ah_attr ah_attr;
+	u16 pkey_index;
+	u8 packet_life_time;
+};
+
+struct cm_work {
+	struct work_struct work;
+	struct list_head list;
+	struct cm_port *port;
+	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
+	u32 local_id;				/* Established / timewait */
+	u32 remote_id;
+	struct ib_cm_event cm_event;
+	struct ib_sa_path_rec path[0];
+};
+
+struct cm_timewait_info {
+	struct cm_work work;			/* Must be first. */
+	struct rb_node remote_qp_node;
+	struct rb_node remote_id_node;
+	u64 remote_ca_guid;
+	u32 remote_qpn;
+	u8 inserted_remote_qp;
+	u8 inserted_remote_id;
+};
+
+struct cm_id_private {
+	struct ib_cm_id	id;
+
+	struct rb_node service_node;
+	struct rb_node sidr_id_node;
+	spinlock_t lock;
+	wait_queue_head_t wait;
+	atomic_t refcount;
+
+	struct ib_mad_send_buf *msg;
+	struct cm_timewait_info *timewait_info;
+	/* todo: use alternate port on send failure */
+	struct cm_av av;
+	struct cm_av alt_av;
+
+	void *private_data;
+	u64 tid;
+	u32 local_qpn;
+	u32 remote_qpn;
+	u32 sq_psn;
+	u32 rq_psn;
+	int timeout_ms;
+	enum ib_mtu path_mtu;
+	u8 private_data_len;
+	u8 max_cm_retries;
+	u8 peer_to_peer;
+	u8 responder_resources;
+	u8 initiator_depth;
+	u8 local_ack_timeout;
+	u8 retry_count;
+	u8 rnr_retry_count;
+	u8 service_timeout;
+
+	struct list_head work_list;
+	atomic_t work_count;
+};
+
+static void cm_work_handler(void *data);
+
+static inline void cm_deref_id(struct cm_id_private *cm_id_priv)
+{
+	if (atomic_dec_and_test(&cm_id_priv->refcount))
+		wake_up(&cm_id_priv->wait);
+}
+
+static int cm_alloc_msg(struct cm_id_private *cm_id_priv,
+			struct ib_mad_send_buf **msg)
+{
+	struct ib_mad_agent *mad_agent;
+	struct ib_mad_send_buf *m;
+	struct ib_ah *ah;
+
+	mad_agent = cm_id_priv->av.port->mad_agent;
+	ah = ib_create_ah(mad_agent->qp->pd, &cm_id_priv->av.ah_attr);
+	if (IS_ERR(ah))
+		return PTR_ERR(ah);
+
+	m = ib_create_send_mad(mad_agent, 1, cm_id_priv->av.pkey_index,
+			       ah, 0, sizeof(struct ib_mad_hdr),
+			       sizeof(struct ib_mad)-sizeof(struct ib_mad_hdr),
+			       GFP_ATOMIC);
+	if (IS_ERR(m)) {
+		ib_destroy_ah(ah);
+		return PTR_ERR(m);
+	}
+
+	/* Timeout set by caller if response is expected. */
+	m->send_wr.wr.ud.retries = cm_id_priv->max_cm_retries;
+
+	atomic_inc(&cm_id_priv->refcount);
+	m->context[0] = cm_id_priv;
+	*msg = m;
+	return 0;
+}
+
+static int cm_alloc_response_msg(struct cm_port *port,
+				 struct ib_mad_recv_wc *mad_recv_wc,
+				 struct ib_mad_send_buf **msg)
+{
+	struct ib_mad_send_buf *m;
+	struct ib_ah *ah;
+
+	ah = ib_create_ah_from_wc(port->mad_agent->qp->pd, mad_recv_wc->wc,
+				  mad_recv_wc->recv_buf.grh, port->port_num);
+	if (IS_ERR(ah))
+		return PTR_ERR(ah);
+
+	m = ib_create_send_mad(port->mad_agent, 1, mad_recv_wc->wc->pkey_index,
+			       ah, 0, sizeof(struct ib_mad_hdr),
+			       sizeof(struct ib_mad)-sizeof(struct ib_mad_hdr),
+			       GFP_ATOMIC);
+	if (IS_ERR(m)) {
+		ib_destroy_ah(ah);
+		return PTR_ERR(m);
+	}
+	*msg = m;
+	return 0;
+}
+
+static void cm_free_msg(struct ib_mad_send_buf *msg)
+{
+	ib_destroy_ah(msg->send_wr.wr.ud.ah);
+	if (msg->context[0])
+		cm_deref_id(msg->context[0]);
+	ib_free_send_mad(msg);
+}
+
+static void * cm_copy_private_data(const void *private_data,
+				   u8 private_data_len)
+{
+	void *data;
+
+	if (!private_data || !private_data_len)
+		return NULL;
+
+	data = kmalloc(private_data_len, GFP_KERNEL);
+	if (!data)
+		return ERR_PTR(-ENOMEM);
+
+	memcpy(data, private_data, private_data_len);
+	return data;
+}
+
+static void cm_set_private_data(struct cm_id_private *cm_id_priv,
+				 void *private_data, u8 private_data_len)
+{
+	if (cm_id_priv->private_data && cm_id_priv->private_data_len)
+		kfree(cm_id_priv->private_data);
+
+	cm_id_priv->private_data = private_data;
+	cm_id_priv->private_data_len = private_data_len;
+}
+
+static void cm_set_ah_attr(struct ib_ah_attr *ah_attr, u8 port_num,
+			   u16 dlid, u8 sl, u16 src_path_bits)
+{
+	memset(ah_attr, 0, sizeof ah_attr);
+	ah_attr->dlid = be16_to_cpu(dlid);
+	ah_attr->sl = sl;
+	ah_attr->src_path_bits = src_path_bits;
+	ah_attr->port_num = port_num;
+}
+
+static void cm_init_av_for_response(struct cm_port *port,
+				    struct ib_wc *wc, struct cm_av *av)
+{
+	av->port = port;
+	av->pkey_index = wc->pkey_index;
+	cm_set_ah_attr(&av->ah_attr, port->port_num, cpu_to_be16(wc->slid),
+		       wc->sl, wc->dlid_path_bits);
+}
+
+static int cm_init_av_by_path(struct ib_sa_path_rec *path, struct cm_av *av)
+{
+	struct cm_device *cm_dev;
+	struct cm_port *port = NULL;
+	unsigned long flags;
+	int ret;
+	u8 p;
+
+	read_lock_irqsave(&cm.device_lock, flags);
+	list_for_each_entry(cm_dev, &cm.device_list, list) {
+		if (!ib_find_cached_gid(cm_dev->device, &path->sgid,
+					&p, NULL)) {
+			port = &cm_dev->port[p-1];
+			break;
+		}
+	}
+	read_unlock_irqrestore(&cm.device_lock, flags);
+
+	if (!port)
+		return -EINVAL;
+
+	ret = ib_find_cached_pkey(cm_dev->device, port->port_num,
+				  be16_to_cpu(path->pkey), &av->pkey_index);
+	if (ret)
+		return ret;
+
+	av->port = port;
+	cm_set_ah_attr(&av->ah_attr, av->port->port_num, path->dlid,
+		       path->sl, path->slid & 0x7F);
+	av->packet_life_time = path->packet_life_time;
+	return 0;
+}
+
+static int cm_alloc_id(struct cm_id_private *cm_id_priv)
+{
+	unsigned long flags;
+	int ret;
+
+	do {
+		spin_lock_irqsave(&cm.lock, flags);
+		ret = idr_get_new_above(&cm.local_id_table, cm_id_priv, 1,
+					(int *) &cm_id_priv->id.local_id);
+		spin_unlock_irqrestore(&cm.lock, flags);
+	} while( (ret == -EAGAIN) && idr_pre_get(&cm.local_id_table, GFP_KERNEL) );
+	return ret;
+}
+
+static void cm_free_id(u32 local_id)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	idr_remove(&cm.local_id_table, (int) local_id);
+	spin_unlock_irqrestore(&cm.lock, flags);
+}
+
+static struct cm_id_private * cm_get_id(u32 local_id, u32 remote_id)
+{
+	struct cm_id_private *cm_id_priv;
+
+	cm_id_priv = idr_find(&cm.local_id_table, (int) local_id);
+	if (cm_id_priv) {
+		if (cm_id_priv->id.remote_id == remote_id)
+			atomic_inc(&cm_id_priv->refcount);
+		else
+			cm_id_priv = NULL;
+	}
+
+	return cm_id_priv;
+}
+
+static struct cm_id_private * cm_acquire_id(u32 local_id, u32 remote_id)
+{
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	cm_id_priv = cm_get_id(local_id, remote_id);
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	return cm_id_priv;
+}
+
+static struct cm_id_private * cm_insert_listen(struct cm_id_private *cm_id_priv)
+{
+	struct rb_node **link = &cm.listen_service_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_id_private *cur_cm_id_priv;
+	u64 service_id = cm_id_priv->id.service_id;
+	u64 service_mask = cm_id_priv->id.service_mask;
+
+	while (*link) {
+		parent = *link;
+		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
+					  service_node);
+		if ((cur_cm_id_priv->id.service_mask & service_id) ==
+		    (service_mask & cur_cm_id_priv->id.service_id))
+			return cm_id_priv;
+		if (service_id < cur_cm_id_priv->id.service_id)
+			link = &(*link)->rb_left;
+		else
+			link = &(*link)->rb_right;
+	}
+	rb_link_node(&cm_id_priv->service_node, parent, link);
+	rb_insert_color(&cm_id_priv->service_node, &cm.listen_service_table);
+	return NULL;
+}
+
+static struct cm_id_private * cm_find_listen(u64 service_id)
+{
+	struct rb_node *node = cm.listen_service_table.rb_node;
+	struct cm_id_private *cm_id_priv;
+
+	while (node) {
+		cm_id_priv = rb_entry(node, struct cm_id_private, service_node);
+		if ((cm_id_priv->id.service_mask & service_id) ==
+		    (cm_id_priv->id.service_mask & cm_id_priv->id.service_id))
+			return cm_id_priv;
+		if (service_id < cm_id_priv->id.service_id)
+			node = node->rb_left;
+		else
+			node = node->rb_right;
+	}
+	return NULL;
+}
+
+static struct cm_timewait_info * cm_insert_remote_id(struct cm_timewait_info
+						     *timewait_info)
+{
+	struct rb_node **link = &cm.remote_id_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_timewait_info *cur_timewait_info;
+	u64 remote_ca_guid = timewait_info->remote_ca_guid;
+	u32 remote_id = timewait_info->work.remote_id;
+
+	while (*link) {
+		parent = *link;
+		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
+					     remote_id_node);
+		if (remote_id < cur_timewait_info->work.remote_id)
+			link = &(*link)->rb_left;
+		else if (remote_id > cur_timewait_info->work.remote_id)
+			link = &(*link)->rb_right;
+		else if (remote_ca_guid < cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_left;
+		else if (remote_ca_guid > cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_right;
+		else
+			return cur_timewait_info;
+	}
+	timewait_info->inserted_remote_id = 1;
+	rb_link_node(&timewait_info->remote_id_node, parent, link);
+	rb_insert_color(&timewait_info->remote_id_node, &cm.remote_id_table);
+	return NULL;
+}
+
+static struct cm_timewait_info * cm_find_remote_id(u64 remote_ca_guid,
+						   u32 remote_id)
+{
+	struct rb_node *node = cm.remote_id_table.rb_node;
+	struct cm_timewait_info *timewait_info;
+
+	while (node) {
+		timewait_info = rb_entry(node, struct cm_timewait_info,
+					 remote_id_node);
+		if (remote_id < timewait_info->work.remote_id)
+			node = node->rb_left;
+		else if (remote_id > timewait_info->work.remote_id)
+			node = node->rb_right;
+		else if (remote_ca_guid < timewait_info->remote_ca_guid)
+			node = node->rb_left;
+		else if (remote_ca_guid > timewait_info->remote_ca_guid)
+			node = node->rb_right;
+		else
+			return timewait_info;
+	}
+	return NULL;
+}
+
+static struct cm_timewait_info * cm_insert_remote_qpn(struct cm_timewait_info
+						      *timewait_info)
+{
+	struct rb_node **link = &cm.remote_qp_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_timewait_info *cur_timewait_info;
+	u64 remote_ca_guid = timewait_info->remote_ca_guid;
+	u32 remote_qpn = timewait_info->remote_qpn;
+
+	while (*link) {
+		parent = *link;
+		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
+					     remote_qp_node);
+		if (remote_qpn < cur_timewait_info->remote_qpn)
+			link = &(*link)->rb_left;
+		else if (remote_qpn > cur_timewait_info->remote_qpn)
+			link = &(*link)->rb_right;
+		else if (remote_ca_guid < cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_left;
+		else if (remote_ca_guid > cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_right;
+		else
+			return cur_timewait_info;
+	}
+	timewait_info->inserted_remote_qp = 1;
+	rb_link_node(&timewait_info->remote_qp_node, parent, link);
+	rb_insert_color(&timewait_info->remote_qp_node, &cm.remote_qp_table);
+	return NULL;
+}
+
+static struct cm_id_private * cm_insert_remote_sidr(struct cm_id_private
+						    *cm_id_priv)
+{
+	struct rb_node **link = &cm.remote_sidr_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_id_private *cur_cm_id_priv;
+	union ib_gid *port_gid = &cm_id_priv->av.dgid;
+	u32 remote_id = cm_id_priv->id.remote_id;
+
+	while (*link) {
+		parent = *link;
+		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
+					  sidr_id_node);
+		if (remote_id < cur_cm_id_priv->id.remote_id)
+			link = &(*link)->rb_left;
+		else if (remote_id > cur_cm_id_priv->id.remote_id)
+			link = &(*link)->rb_right;
+		else {
+			int cmp;
+			cmp = memcmp(port_gid, &cur_cm_id_priv->av.dgid,
+				     sizeof *port_gid);
+			if (cmp < 0)
+				link = &(*link)->rb_left;
+			else if (cmp > 0)
+				link = &(*link)->rb_right;
+			else
+				return cur_cm_id_priv;
+		}
+	}
+	rb_link_node(&cm_id_priv->sidr_id_node, parent, link);
+	rb_insert_color(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
+	return NULL;
+}
+
+static void cm_reject_sidr_req(struct cm_id_private *cm_id_priv,
+			       enum ib_cm_sidr_status status)
+{
+	struct ib_cm_sidr_rep_param param;
+
+	memset(&param, 0, sizeof param);
+	param.status = status;
+	ib_send_cm_sidr_rep(&cm_id_priv->id, &param);
+}
+
+struct ib_cm_id *ib_create_cm_id(ib_cm_handler cm_handler,
+				 void *context)
+{
+	struct cm_id_private *cm_id_priv;
+	int ret;
+
+	cm_id_priv = kmalloc(sizeof *cm_id_priv, GFP_KERNEL);
+	if (!cm_id_priv)
+		return ERR_PTR(-ENOMEM);
+
+	memset(cm_id_priv, 0, sizeof *cm_id_priv);
+	cm_id_priv->id.state = IB_CM_IDLE;
+	cm_id_priv->id.cm_handler = cm_handler;
+	cm_id_priv->id.context = context;
+	ret = cm_alloc_id(cm_id_priv);
+	if (ret)
+		goto error;
+
+	spin_lock_init(&cm_id_priv->lock);
+	init_waitqueue_head(&cm_id_priv->wait);
+	INIT_LIST_HEAD(&cm_id_priv->work_list);
+	atomic_set(&cm_id_priv->work_count, -1);
+	atomic_set(&cm_id_priv->refcount, 1);
+	return &cm_id_priv->id;
+
+error:
+	kfree(cm_id_priv);
+	return ERR_PTR(-ENOMEM);
+}
+EXPORT_SYMBOL(ib_create_cm_id);
+
+static struct cm_work * cm_dequeue_work(struct cm_id_private *cm_id_priv)
+{
+	struct cm_work *work;
+
+	if (list_empty(&cm_id_priv->work_list))
+		return NULL;
+
+	work = list_entry(cm_id_priv->work_list.next, struct cm_work, list);
+	list_del(&work->list);
+	return work;
+}
+
+static void cm_free_work(struct cm_work *work)
+{
+	if (work->mad_recv_wc)
+		ib_free_recv_mad(work->mad_recv_wc);
+	kfree(work);
+}
+
+static inline int cm_convert_to_ms(int iba_time)
+{
+	/* approximate conversion to ms from 4.096us x 2^iba_time */
+	return 1 << max(iba_time - 8, 0);
+}
+
+static void cm_cleanup_timewait(struct cm_timewait_info *timewait_info)
+{
+	unsigned long flags;
+
+	if (!timewait_info->inserted_remote_id &&
+	    !timewait_info->inserted_remote_qp)
+	    return;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	if (timewait_info->inserted_remote_id) {
+		rb_erase(&timewait_info->remote_id_node, &cm.remote_id_table);
+		timewait_info->inserted_remote_id = 0;
+	}
+
+	if (timewait_info->inserted_remote_qp) {
+		rb_erase(&timewait_info->remote_qp_node, &cm.remote_qp_table);
+		timewait_info->inserted_remote_qp = 0;
+	}
+	spin_unlock_irqrestore(&cm.lock, flags);
+}
+
+static struct cm_timewait_info * cm_create_timewait_info(u32 local_id)
+{
+	struct cm_timewait_info *timewait_info;
+
+	timewait_info = kmalloc(sizeof *timewait_info, GFP_KERNEL);
+	if (!timewait_info)
+		return ERR_PTR(-ENOMEM);
+	memset(timewait_info, 0, sizeof *timewait_info);
+
+	timewait_info->work.local_id = local_id;
+	INIT_WORK(&timewait_info->work.work, cm_work_handler,
+		  &timewait_info->work);
+	timewait_info->work.cm_event.event = IB_CM_TIMEWAIT_EXIT;
+	return timewait_info;
+}
+
+static void cm_enter_timewait(struct cm_id_private *cm_id_priv)
+{
+	int wait_time;
+
+	/*
+	 * The cm_id could be destroyed by the user before we exit timewait.
+	 * To protect against this, we search for the cm_id after exiting
+	 * timewait before notifying the user that we've exited timewait.
+	 */
+	cm_id_priv->id.state = IB_CM_TIMEWAIT;
+	wait_time = cm_convert_to_ms(cm_id_priv->local_ack_timeout);
+	queue_delayed_work(cm.wq, &cm_id_priv->timewait_info->work.work,
+			   msecs_to_jiffies(wait_time));
+	cm_id_priv->timewait_info = NULL;
+}
+
+static void cm_reset_to_idle(struct cm_id_private *cm_id_priv)
+{
+	cm_id_priv->id.state = IB_CM_IDLE;
+	if (cm_id_priv->timewait_info) {
+		cm_cleanup_timewait(cm_id_priv->timewait_info);
+		kfree(cm_id_priv->timewait_info);
+		cm_id_priv->timewait_info = NULL;
+	}
+}
+
+void ib_destroy_cm_id(struct ib_cm_id *cm_id)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_work *work;
+	unsigned long flags;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+retest:
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state) {
+	case IB_CM_LISTEN:
+		cm_id->state = IB_CM_IDLE;
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		spin_lock_irqsave(&cm.lock, flags);
+		rb_erase(&cm_id_priv->service_node, &cm.listen_service_table);
+		spin_unlock_irqrestore(&cm.lock, flags);
+		break;
+	case IB_CM_SIDR_REQ_SENT:
+		cm_id->state = IB_CM_IDLE;
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		break;
+	case IB_CM_SIDR_REQ_RCVD:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_reject_sidr_req(cm_id_priv, IB_SIDR_REJECT);
+		break;
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		/* Fall through */
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ib_send_cm_rej(cm_id, IB_CM_REJ_TIMEOUT, 
+			       &cm_id_priv->av.port->cm_dev->ca_guid,
+			       sizeof cm_id_priv->av.port->cm_dev->ca_guid,
+			       NULL, 0);
+		break;
+	case IB_CM_ESTABLISHED:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ib_send_cm_dreq(cm_id, NULL, 0);
+		goto retest;
+	case IB_CM_DREQ_SENT:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		cm_enter_timewait(cm_id_priv);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		break;
+	case IB_CM_DREQ_RCVD:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ib_send_cm_drep(cm_id, NULL, 0);
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		break;
+	}
+
+	cm_free_id(cm_id->local_id);
+	atomic_dec(&cm_id_priv->refcount);
+	wait_event(cm_id_priv->wait, !atomic_read(&cm_id_priv->refcount));
+	while ((work = cm_dequeue_work(cm_id_priv)) != NULL)
+		cm_free_work(work);
+	if (cm_id_priv->private_data && cm_id_priv->private_data_len)
+		kfree(cm_id_priv->private_data);
+	kfree(cm_id_priv);
+}
+EXPORT_SYMBOL(ib_destroy_cm_id);
+
+int ib_cm_listen(struct ib_cm_id *cm_id,
+		 u64 service_id,
+		 u64 service_mask)
+{
+	struct cm_id_private *cm_id_priv, *cur_cm_id_priv;
+	unsigned long flags;
+	int ret = 0;
+
+	service_mask = service_mask ? service_mask : ~0ULL;
+	service_id &= service_mask;
+	if ((service_id & IB_SERVICE_ID_AGN_MASK) == IB_CM_ASSIGN_SERVICE_ID &&
+	    (service_id != IB_CM_ASSIGN_SERVICE_ID))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	BUG_ON(cm_id->state != IB_CM_IDLE);
+
+	cm_id->state = IB_CM_LISTEN;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	if (service_id == IB_CM_ASSIGN_SERVICE_ID) {
+		cm_id->service_id = __cpu_to_be64(cm.listen_service_id++);
+		cm_id->service_mask = ~0ULL;
+	} else {
+		cm_id->service_id = service_id;
+		cm_id->service_mask = service_mask;
+	}
+	cur_cm_id_priv = cm_insert_listen(cm_id_priv);
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	if (cur_cm_id_priv) {
+		cm_id->state = IB_CM_IDLE;
+		ret = -EBUSY;
+	}
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_listen);
+
+static u64 cm_form_tid(struct cm_id_private *cm_id_priv,
+		       enum cm_msg_sequence msg_seq)
+{
+	u64 hi_tid, low_tid;
+
+	hi_tid   = ((u64) cm_id_priv->av.port->mad_agent->hi_tid) << 32;
+	low_tid  = (u64) (cm_id_priv->id.local_id | (msg_seq << 30));
+	return cpu_to_be64(hi_tid | low_tid);
+}
+
+static void cm_format_mad_hdr(struct ib_mad_hdr *hdr,
+			      enum cm_msg_attr_id attr_id, u64 tid)
+{
+	hdr->base_version  = IB_MGMT_BASE_VERSION;
+	hdr->mgmt_class	   = IB_MGMT_CLASS_CM;
+	hdr->class_version = IB_CM_CLASS_VERSION;
+	hdr->method	   = IB_MGMT_METHOD_SEND;
+	hdr->attr_id	   = attr_id;
+	hdr->tid	   = tid;
+}
+
+static void cm_format_req(struct cm_req_msg *req_msg,
+			  struct cm_id_private *cm_id_priv,
+			  struct ib_cm_req_param *param)
+{
+	cm_format_mad_hdr(&req_msg->hdr, CM_REQ_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ));
+
+	req_msg->local_comm_id = cm_id_priv->id.local_id;
+	req_msg->service_id = param->service_id;
+	req_msg->local_ca_guid = cm_id_priv->av.port->cm_dev->ca_guid;
+	cm_req_set_local_qpn(req_msg, cpu_to_be32(param->qp_num));
+	cm_req_set_resp_res(req_msg, param->responder_resources);
+	cm_req_set_init_depth(req_msg, param->initiator_depth);
+	cm_req_set_remote_resp_timeout(req_msg,
+				       param->remote_cm_response_timeout);
+	cm_req_set_qp_type(req_msg, param->qp_type);
+	cm_req_set_flow_ctrl(req_msg, param->flow_control);
+	cm_req_set_starting_psn(req_msg, cpu_to_be32(param->starting_psn));
+	cm_req_set_local_resp_timeout(req_msg,
+				      param->local_cm_response_timeout);
+	cm_req_set_retry_count(req_msg, param->retry_count);
+	req_msg->pkey = param->primary_path->pkey;
+	cm_req_set_path_mtu(req_msg, param->primary_path->mtu);
+	cm_req_set_rnr_retry_count(req_msg, param->rnr_retry_count);
+	cm_req_set_max_cm_retries(req_msg, param->max_cm_retries);
+	cm_req_set_srq(req_msg, param->srq);
+
+	req_msg->primary_local_lid = param->primary_path->slid;
+	req_msg->primary_remote_lid = param->primary_path->dlid;
+	req_msg->primary_local_gid = param->primary_path->sgid;
+	req_msg->primary_remote_gid = param->primary_path->dgid;
+	cm_req_set_primary_flow_label(req_msg, param->primary_path->flow_label);
+	cm_req_set_primary_packet_rate(req_msg, param->primary_path->rate);
+	req_msg->primary_traffic_class = param->primary_path->traffic_class;
+	req_msg->primary_hop_limit = param->primary_path->hop_limit;
+	cm_req_set_primary_sl(req_msg, param->primary_path->sl);
+	cm_req_set_primary_subnet_local(req_msg, 1); /* local only... */
+	cm_req_set_primary_local_ack_timeout(req_msg,
+		min(31, param->primary_path->packet_life_time + 1));
+
+	if (param->alternate_path) {
+		req_msg->alt_local_lid = param->alternate_path->slid;
+		req_msg->alt_remote_lid = param->alternate_path->dlid;
+		req_msg->alt_local_gid = param->alternate_path->sgid;
+		req_msg->alt_remote_gid = param->alternate_path->dgid;
+		cm_req_set_alt_flow_label(req_msg,
+					  param->alternate_path->flow_label);
+		cm_req_set_alt_packet_rate(req_msg, param->alternate_path->rate);
+		req_msg->alt_traffic_class = param->alternate_path->traffic_class;
+		req_msg->alt_hop_limit = param->alternate_path->hop_limit;
+		cm_req_set_alt_sl(req_msg, param->alternate_path->sl);
+		cm_req_set_alt_subnet_local(req_msg, 1); /* local only... */
+		cm_req_set_alt_local_ack_timeout(req_msg,
+			min(31, param->alternate_path->packet_life_time + 1));
+	}
+
+	if (param->private_data && param->private_data_len)
+		memcpy(req_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+static inline int cm_validate_req_param(struct ib_cm_req_param *param)
+{
+	/* peer-to-peer not supported */
+	if (param->peer_to_peer)
+		return -EINVAL;
+
+	if (!param->primary_path)
+		return -EINVAL;
+
+	if (param->qp_type != IB_QPT_RC && param->qp_type != IB_QPT_UC)
+		return -EINVAL;
+
+	if (param->private_data &&
+	    param->private_data_len > IB_CM_REQ_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	if (param->alternate_path &&
+	    (param->alternate_path->pkey != param->primary_path->pkey ||
+	     param->alternate_path->mtu != param->primary_path->mtu))
+		return -EINVAL;
+
+	return 0;
+}
+
+int ib_send_cm_req(struct ib_cm_id *cm_id,
+		   struct ib_cm_req_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_send_wr *bad_send_wr;
+	struct cm_req_msg *req_msg;
+	unsigned long flags;
+	int ret;
+
+	ret = cm_validate_req_param(param);
+	if (ret)
+		return ret;
+
+	/* Verify that we're not in timewait. */
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_IDLE) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto out;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv->
+							    id.local_id);
+	if (IS_ERR(cm_id_priv->timewait_info))
+		goto out;
+
+	ret = cm_init_av_by_path(param->primary_path, &cm_id_priv->av);
+	if (ret)
+		goto error1;
+	if (param->alternate_path) {
+		ret = cm_init_av_by_path(param->alternate_path,
+					 &cm_id_priv->alt_av);
+		if (ret)
+			goto error1;
+	}
+	cm_id->service_id = param->service_id;
+	cm_id->service_mask = ~0ULL;
+	cm_id_priv->timeout_ms = cm_convert_to_ms(
+				    param->primary_path->packet_life_time) * 2 +
+				 cm_convert_to_ms(
+				    param->remote_cm_response_timeout);
+	cm_id_priv->max_cm_retries = param->max_cm_retries;
+	cm_id_priv->initiator_depth = param->initiator_depth;
+	cm_id_priv->responder_resources = param->responder_resources;
+	cm_id_priv->retry_count = param->retry_count;
+	cm_id_priv->path_mtu = param->primary_path->mtu;
+
+	ret = cm_alloc_msg(cm_id_priv, &cm_id_priv->msg);
+	if (ret)
+		goto error1;
+
+	req_msg = (struct cm_req_msg *) cm_id_priv->msg->mad;
+	cm_format_req(req_msg, cm_id_priv, param);
+	cm_id_priv->tid = req_msg->hdr.tid;
+	cm_id_priv->msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	cm_id_priv->msg->context[1] = (void *) (unsigned long) IB_CM_REQ_SENT;
+
+	cm_id_priv->local_qpn = cm_req_get_local_qpn(req_msg);
+	cm_id_priv->rq_psn = cm_req_get_starting_psn(req_msg);
+	cm_id_priv->local_ack_timeout =
+				cm_req_get_primary_local_ack_timeout(req_msg);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				&cm_id_priv->msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto error2;
+	}
+	BUG_ON(cm_id->state != IB_CM_IDLE);
+	cm_id->state = IB_CM_REQ_SENT;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return 0;
+
+error2:	cm_free_msg(cm_id_priv->msg);
+error1:	kfree(cm_id_priv->timewait_info);
+out:	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_req);
+
+static int cm_issue_rej(struct cm_port *port,
+			struct ib_mad_recv_wc *mad_recv_wc,
+			enum ib_cm_rej_reason reason,
+			enum cm_msg_response msg_rejected,
+			void *ari, u8 ari_length)
+{
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	struct cm_rej_msg *rej_msg, *rcv_msg;
+	int ret;
+
+	ret = cm_alloc_response_msg(port, mad_recv_wc, &msg);
+	if (ret)
+		return ret;
+	
+	/* We just need common CM header information.  Cast to any message. */
+	rcv_msg = (struct cm_rej_msg *) mad_recv_wc->recv_buf.mad;
+	rej_msg = (struct cm_rej_msg *) msg->mad;
+
+	cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, rcv_msg->hdr.tid);
+	rej_msg->remote_comm_id = rcv_msg->local_comm_id;
+	rej_msg->local_comm_id = rcv_msg->remote_comm_id;
+	cm_rej_set_msg_rejected(rej_msg, msg_rejected);
+	rej_msg->reason = reason;
+
+	if (ari && ari_length) {
+		cm_rej_set_reject_info_len(rej_msg, ari_length);
+		memcpy(rej_msg->ari, ari, ari_length);
+	}
+
+	ret = ib_post_send_mad(port->mad_agent, &msg->send_wr, &bad_send_wr);
+	if (ret)
+		cm_free_msg(msg);
+
+	return ret;
+}
+
+static inline int cm_is_active_peer(u64 local_ca_guid, u64 remote_ca_guid,
+				    u32 local_qpn, u32 remote_qpn)
+{
+	return (be64_to_cpu(local_ca_guid) > be64_to_cpu(remote_ca_guid) ||
+		((local_ca_guid == remote_ca_guid) &&
+		 (be32_to_cpu(local_qpn) > be32_to_cpu(remote_qpn))));
+}
+
+static inline void cm_format_paths_from_req(struct cm_req_msg *req_msg,
+					    struct ib_sa_path_rec *primary_path,
+					    struct ib_sa_path_rec *alt_path)
+{
+	memset(primary_path, 0, sizeof *primary_path);
+	primary_path->dgid = req_msg->primary_local_gid;
+	primary_path->sgid = req_msg->primary_remote_gid;
+	primary_path->dlid = req_msg->primary_local_lid;
+	primary_path->slid = req_msg->primary_remote_lid;
+	primary_path->flow_label = cm_req_get_primary_flow_label(req_msg);
+	primary_path->hop_limit = req_msg->primary_hop_limit;
+	primary_path->traffic_class = req_msg->primary_traffic_class;
+	primary_path->reversible = 1;
+	primary_path->pkey = req_msg->pkey;
+	primary_path->sl = cm_req_get_primary_sl(req_msg);
+	primary_path->mtu_selector = IB_SA_EQ;
+	primary_path->mtu = cm_req_get_path_mtu(req_msg);
+	primary_path->rate_selector = IB_SA_EQ;
+	primary_path->rate = cm_req_get_primary_packet_rate(req_msg);
+	primary_path->packet_life_time_selector = IB_SA_EQ;
+	primary_path->packet_life_time =
+		cm_req_get_primary_local_ack_timeout(req_msg);
+	primary_path->packet_life_time -= (primary_path->packet_life_time > 0);
+
+	if (req_msg->alt_local_lid) {
+		memset(alt_path, 0, sizeof *alt_path);
+		alt_path->dgid = req_msg->alt_local_gid;
+		alt_path->sgid = req_msg->alt_remote_gid;
+		alt_path->dlid = req_msg->alt_local_lid;
+		alt_path->slid = req_msg->alt_remote_lid;
+		alt_path->flow_label = cm_req_get_alt_flow_label(req_msg);
+		alt_path->hop_limit = req_msg->alt_hop_limit;
+		alt_path->traffic_class = req_msg->alt_traffic_class;
+		alt_path->reversible = 1;
+		alt_path->pkey = req_msg->pkey;
+		alt_path->sl = cm_req_get_alt_sl(req_msg);
+		alt_path->mtu_selector = IB_SA_EQ;
+		alt_path->mtu = cm_req_get_path_mtu(req_msg);
+		alt_path->rate_selector = IB_SA_EQ;
+		alt_path->rate = cm_req_get_alt_packet_rate(req_msg);
+		alt_path->packet_life_time_selector = IB_SA_EQ;
+		alt_path->packet_life_time =
+			cm_req_get_alt_local_ack_timeout(req_msg);
+		alt_path->packet_life_time -= (alt_path->packet_life_time > 0);
+	}
+}
+
+static void cm_format_req_event(struct cm_work *work,
+				struct cm_id_private *cm_id_priv,
+				struct ib_cm_id *listen_id)
+{
+	struct cm_req_msg *req_msg;
+	struct ib_cm_req_event_param *param;
+
+	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.req_rcvd;
+	param->listen_id = listen_id;
+	param->device = cm_id_priv->av.port->mad_agent->device;
+	param->port = cm_id_priv->av.port->port_num;
+	param->primary_path = &work->path[0];
+	if (req_msg->alt_local_lid)
+		param->alternate_path = &work->path[1];
+	else
+		param->alternate_path = NULL;
+	param->remote_ca_guid = req_msg->local_ca_guid;
+	param->remote_qkey = be32_to_cpu(req_msg->local_qkey);
+	param->remote_qpn = be32_to_cpu(cm_req_get_local_qpn(req_msg));
+	param->qp_type = cm_req_get_qp_type(req_msg);
+	param->starting_psn = be32_to_cpu(cm_req_get_starting_psn(req_msg));
+	param->responder_resources = cm_req_get_init_depth(req_msg);
+	param->initiator_depth = cm_req_get_resp_res(req_msg);
+	param->local_cm_response_timeout = 
+					cm_req_get_remote_resp_timeout(req_msg);
+	param->flow_control = cm_req_get_flow_ctrl(req_msg);
+	param->remote_cm_response_timeout = 
+					cm_req_get_local_resp_timeout(req_msg);
+	param->retry_count = cm_req_get_retry_count(req_msg);
+	param->rnr_retry_count = cm_req_get_rnr_retry_count(req_msg);
+	param->srq = cm_req_get_srq(req_msg);
+	work->cm_event.private_data = &req_msg->private_data;
+}
+
+static void cm_process_work(struct cm_id_private *cm_id_priv,
+			    struct cm_work *work)
+{
+	unsigned long flags;
+	int ret;
+
+	/* We will typically only have the current event to report. */
+	ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, &work->cm_event);
+	cm_free_work(work);
+
+	while (!ret && !atomic_add_negative(-1, &cm_id_priv->work_count)) {
+		spin_lock_irqsave(&cm_id_priv->lock, flags);
+		work = cm_dequeue_work(cm_id_priv);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		BUG_ON(!work);
+		ret = cm_id_priv->id.cm_handler(&cm_id_priv->id,
+						&work->cm_event);
+		cm_free_work(work);
+	}
+	cm_deref_id(cm_id_priv);
+	if (ret)
+		ib_destroy_cm_id(&cm_id_priv->id);	
+}
+
+static void cm_format_mra(struct cm_mra_msg *mra_msg,
+			  struct cm_id_private *cm_id_priv,
+			  enum cm_msg_response msg_mraed, u8 service_timeout,
+			  const void *private_data, u8 private_data_len)
+{
+	cm_format_mad_hdr(&mra_msg->hdr, CM_MRA_ATTR_ID, cm_id_priv->tid);
+	cm_mra_set_msg_mraed(mra_msg, msg_mraed);
+	mra_msg->local_comm_id = cm_id_priv->id.local_id;
+	mra_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_mra_set_service_timeout(mra_msg, service_timeout);
+
+	if (private_data && private_data_len)
+		memcpy(mra_msg->private_data, private_data, private_data_len);
+}
+
+static void cm_format_rej(struct cm_rej_msg *rej_msg,
+			  struct cm_id_private *cm_id_priv,
+			  enum ib_cm_rej_reason reason,
+			  void *ari,
+			  u8 ari_length,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, cm_id_priv->tid);
+	rej_msg->remote_comm_id = cm_id_priv->id.remote_id;
+
+	switch(cm_id_priv->id.state) {
+	case IB_CM_REQ_RCVD:
+		rej_msg->local_comm_id = 0;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ);
+		break;
+	case IB_CM_MRA_REQ_SENT:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ);
+		break;
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REP);
+		break;
+	default:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_OTHER);
+		break;
+	}
+
+	rej_msg->reason = reason;
+	if (ari && ari_length) {
+		cm_rej_set_reject_info_len(rej_msg, ari_length);
+		memcpy(rej_msg->ari, ari, ari_length);
+	}
+
+	if (private_data && private_data_len)
+		memcpy(rej_msg->private_data, private_data, private_data_len);
+}
+
+static void cm_dup_req_handler(struct cm_work *work,
+			       struct cm_id_private *cm_id_priv)
+{
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	/* Quick state check to discard duplicate REQs. */
+	if (cm_id_priv->id.state == IB_CM_REQ_RCVD)
+		return;
+
+	ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg);
+	if (ret)
+		return;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_MRA_REQ_SENT:
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REQ, cm_id_priv->service_timeout,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+		break;
+	case IB_CM_TIMEWAIT:
+		cm_format_rej((struct cm_rej_msg *) msg->mad, cm_id_priv,
+			      IB_CM_REJ_STALE_CONN, NULL, 0, NULL, 0);
+		break;
+	default:
+		goto unlock;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		goto free;
+	return;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+free:	cm_free_msg(msg);
+}
+
+static struct cm_id_private * cm_match_req(struct cm_work *work,
+					   struct cm_id_private *cm_id_priv)
+{
+	struct cm_id_private *listen_cm_id_priv, *cur_cm_id_priv;
+	struct cm_timewait_info *timewait_info;
+	struct cm_req_msg *req_msg;
+	unsigned long flags;
+
+	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
+
+	/* Check for duplicate REQ and stale connections. */
+	spin_lock_irqsave(&cm.lock, flags);
+	timewait_info = cm_insert_remote_id(cm_id_priv->timewait_info);
+	if (!timewait_info)
+		timewait_info = cm_insert_remote_qpn(cm_id_priv->timewait_info);
+
+	if (timewait_info) {
+		cur_cm_id_priv = cm_get_id(timewait_info->work.local_id,
+					   timewait_info->work.remote_id);
+		spin_unlock_irqrestore(&cm.lock, flags);
+		if (cur_cm_id_priv) {
+			cm_dup_req_handler(work, cur_cm_id_priv);
+			cm_deref_id(cur_cm_id_priv);
+		} else
+			cm_issue_rej(work->port, work->mad_recv_wc,
+				     IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REQ,
+				     NULL, 0);
+		goto error;
+	}
+
+	/* Find matching listen request. */
+	listen_cm_id_priv = cm_find_listen(req_msg->service_id);
+	if (!listen_cm_id_priv) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		cm_issue_rej(work->port, work->mad_recv_wc,
+			     IB_CM_REJ_INVALID_SERVICE_ID, CM_MSG_RESPONSE_REQ,
+			     NULL, 0);
+		goto error;
+	}
+	atomic_inc(&listen_cm_id_priv->refcount);
+	atomic_inc(&cm_id_priv->refcount);
+	cm_id_priv->id.state = IB_CM_REQ_RCVD;
+	atomic_inc(&cm_id_priv->work_count);
+	spin_unlock_irqrestore(&cm.lock, flags);
+	return listen_cm_id_priv;
+
+error:	cm_cleanup_timewait(cm_id_priv->timewait_info);
+	return NULL;
+}
+
+static int cm_req_handler(struct cm_work *work)
+{
+	struct ib_cm_id *cm_id;
+	struct cm_id_private *cm_id_priv, *listen_cm_id_priv;
+	struct cm_req_msg *req_msg;
+	int ret;
+
+	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
+
+	cm_id = ib_create_cm_id(NULL, NULL);
+	if (IS_ERR(cm_id))
+		return PTR_ERR(cm_id);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	cm_id_priv->id.remote_id = req_msg->local_comm_id;
+	cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
+				&cm_id_priv->av);
+	cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv->
+							    id.local_id);
+	if (IS_ERR(cm_id_priv->timewait_info)) {
+		ret = PTR_ERR(cm_id_priv->timewait_info);
+		goto error1;
+	}
+	cm_id_priv->timewait_info->work.remote_id = req_msg->local_comm_id;
+	cm_id_priv->timewait_info->remote_ca_guid = req_msg->local_ca_guid;
+	cm_id_priv->timewait_info->remote_qpn = cm_req_get_local_qpn(req_msg);
+
+	listen_cm_id_priv = cm_match_req(work, cm_id_priv);
+	if (!listen_cm_id_priv) {
+		ret = -EINVAL;
+		goto error2;
+	}
+
+	cm_id_priv->id.cm_handler = listen_cm_id_priv->id.cm_handler;
+	cm_id_priv->id.context = listen_cm_id_priv->id.context;
+	cm_id_priv->id.service_id = req_msg->service_id;
+	cm_id_priv->id.service_mask = ~0ULL;
+
+	cm_format_paths_from_req(req_msg, &work->path[0], &work->path[1]);
+	ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av);
+	if (ret)
+		goto error3;
+	if (req_msg->alt_local_lid) {
+		ret = cm_init_av_by_path(&work->path[1], &cm_id_priv->alt_av);
+		if (ret)
+			goto error3;
+	}
+	cm_id_priv->tid = req_msg->hdr.tid;
+	cm_id_priv->timeout_ms = cm_convert_to_ms(
+					cm_req_get_local_resp_timeout(req_msg));
+	cm_id_priv->max_cm_retries = cm_req_get_max_cm_retries(req_msg);
+	cm_id_priv->remote_qpn = cm_req_get_local_qpn(req_msg);
+	cm_id_priv->initiator_depth = cm_req_get_resp_res(req_msg);
+	cm_id_priv->responder_resources = cm_req_get_init_depth(req_msg);
+	cm_id_priv->path_mtu = cm_req_get_path_mtu(req_msg);
+	cm_id_priv->sq_psn = cm_req_get_starting_psn(req_msg);
+	cm_id_priv->local_ack_timeout =
+				cm_req_get_primary_local_ack_timeout(req_msg);
+	cm_id_priv->retry_count = cm_req_get_retry_count(req_msg);
+	cm_id_priv->rnr_retry_count = cm_req_get_rnr_retry_count(req_msg);
+
+	cm_format_req_event(work, cm_id_priv, &listen_cm_id_priv->id);
+	cm_process_work(cm_id_priv, work);
+	cm_deref_id(listen_cm_id_priv);
+	return 0;
+
+error3:	atomic_dec(&cm_id_priv->refcount);
+	cm_deref_id(listen_cm_id_priv);
+	cm_cleanup_timewait(cm_id_priv->timewait_info);
+error2:	kfree(cm_id_priv->timewait_info);
+error1:	ib_destroy_cm_id(&cm_id_priv->id);
+	return ret;
+}
+
+static void cm_format_rep(struct cm_rep_msg *rep_msg,
+			  struct cm_id_private *cm_id_priv,
+			  struct ib_cm_rep_param *param)
+{
+	cm_format_mad_hdr(&rep_msg->hdr, CM_REP_ATTR_ID, cm_id_priv->tid);
+	rep_msg->local_comm_id = cm_id_priv->id.local_id;
+	rep_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_rep_set_local_qpn(rep_msg, cpu_to_be32(param->qp_num));
+	cm_rep_set_starting_psn(rep_msg, cpu_to_be32(param->starting_psn));
+	rep_msg->resp_resources = param->responder_resources;
+	rep_msg->initiator_depth = param->initiator_depth;
+	cm_rep_set_target_ack_delay(rep_msg, param->target_ack_delay);
+	cm_rep_set_failover(rep_msg, param->failover_accepted);
+	cm_rep_set_flow_ctrl(rep_msg, param->flow_control);
+	cm_rep_set_rnr_retry_count(rep_msg, param->rnr_retry_count);
+	cm_rep_set_srq(rep_msg, param->srq);
+	rep_msg->local_ca_guid = cm_id_priv->av.port->cm_dev->ca_guid;
+
+	if (param->private_data && param->private_data_len)
+		memcpy(rep_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+int ib_send_cm_rep(struct ib_cm_id *cm_id,
+		   struct ib_cm_rep_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct cm_rep_msg *rep_msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (param->private_data &&
+	    param->private_data_len > IB_CM_REP_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_REQ_RCVD &&
+	    cm_id->state != IB_CM_MRA_REQ_SENT) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	rep_msg = (struct cm_rep_msg *) msg->mad;
+	cm_format_rep(rep_msg, cm_id_priv, param);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_REP_SENT;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->state = IB_CM_REP_SENT;
+	cm_id_priv->msg = msg;
+	cm_id_priv->initiator_depth = param->initiator_depth;
+	cm_id_priv->responder_resources = param->responder_resources;
+	cm_id_priv->rq_psn = cm_rep_get_starting_psn(rep_msg);
+	cm_id_priv->local_qpn = cm_rep_get_local_qpn(rep_msg);
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_rep);
+
+static void cm_format_rtu(struct cm_rtu_msg *rtu_msg,
+			  struct cm_id_private *cm_id_priv,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&rtu_msg->hdr, CM_RTU_ATTR_ID, cm_id_priv->tid);
+	rtu_msg->local_comm_id = cm_id_priv->id.local_id;
+	rtu_msg->remote_comm_id = cm_id_priv->id.remote_id;
+
+	if (private_data && private_data_len)
+		memcpy(rtu_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_rtu(struct ib_cm_id *cm_id,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	void *data;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_RTU_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	data = cm_copy_private_data(private_data, private_data_len);
+	if (IS_ERR(data))
+		return PTR_ERR(data);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_REP_RCVD &&
+	    cm_id->state != IB_CM_MRA_REP_SENT) {
+		ret = -EINVAL;
+		goto error;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto error;
+
+	cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv,
+		      private_data, private_data_len);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		kfree(data);
+		return ret;
+	}
+
+	cm_id->state = IB_CM_ESTABLISHED;
+	cm_set_private_data(cm_id_priv, data, private_data_len);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return 0;
+
+error:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	kfree(data);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_rtu);
+
+static void cm_format_rep_event(struct cm_work *work)
+{
+	struct cm_rep_msg *rep_msg;
+	struct ib_cm_rep_event_param *param;
+
+	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.rep_rcvd;
+	param->remote_ca_guid = rep_msg->local_ca_guid;
+	param->remote_qkey = be32_to_cpu(rep_msg->local_qkey);
+	param->remote_qpn = be32_to_cpu(cm_rep_get_local_qpn(rep_msg));
+	param->starting_psn = be32_to_cpu(cm_rep_get_starting_psn(rep_msg));
+	param->responder_resources = rep_msg->initiator_depth;
+	param->initiator_depth = rep_msg->resp_resources;
+	param->target_ack_delay = cm_rep_get_target_ack_delay(rep_msg);
+	param->failover_accepted = cm_rep_get_failover(rep_msg);
+	param->flow_control = cm_rep_get_flow_ctrl(rep_msg);
+	param->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg);
+	param->srq = cm_rep_get_srq(rep_msg);
+	work->cm_event.private_data = &rep_msg->private_data;
+}
+
+static void cm_dup_rep_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rep_msg *rep_msg;
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	rep_msg = (struct cm_rep_msg *) work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id,
+				   rep_msg->local_comm_id);
+	if (!cm_id_priv)
+		return;
+
+	ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg);
+	if (ret)
+		goto deref;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state == IB_CM_ESTABLISHED)
+		cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+	else if (cm_id_priv->id.state == IB_CM_MRA_REP_SENT)
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REP, cm_id_priv->service_timeout,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+	else
+		goto unlock;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		goto free;
+	goto deref;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+free:	cm_free_msg(msg);
+deref:	cm_deref_id(cm_id_priv);
+}
+
+static int cm_rep_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rep_msg *rep_msg;
+	unsigned long flags;
+	int ret;
+
+	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id, 0);
+	if (!cm_id_priv) {
+		cm_dup_rep_handler(work);
+		return -EINVAL;
+	}
+
+	cm_id_priv->timewait_info->work.remote_id = rep_msg->local_comm_id;
+	cm_id_priv->timewait_info->remote_ca_guid = rep_msg->local_ca_guid;
+	cm_id_priv->timewait_info->remote_qpn = cm_rep_get_local_qpn(rep_msg);
+
+	spin_lock_irqsave(&cm.lock, flags);
+	/* Check for duplicate REP. */
+	if (cm_insert_remote_id(cm_id_priv->timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		ret = -EINVAL;
+		goto error;
+	}
+	/* Check for a stale connection. */
+	if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		cm_issue_rej(work->port, work->mad_recv_wc,
+			     IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REP,
+			     NULL, 0);
+		ret = -EINVAL;
+		goto error;
+	}
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	cm_format_rep_event(work);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto error;
+	}
+	cm_id_priv->id.state = IB_CM_REP_RCVD;
+	cm_id_priv->id.remote_id = rep_msg->local_comm_id;
+	cm_id_priv->remote_qpn = cm_rep_get_local_qpn(rep_msg);
+	cm_id_priv->initiator_depth = rep_msg->resp_resources;
+	cm_id_priv->responder_resources = rep_msg->initiator_depth;
+	cm_id_priv->sq_psn = cm_rep_get_starting_psn(rep_msg);
+	cm_id_priv->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg);
+
+	/* todo: handle peer_to_peer */
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+
+error:	cm_cleanup_timewait(cm_id_priv->timewait_info);
+	cm_deref_id(cm_id_priv);
+	return ret;
+}
+
+static int cm_establish_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	int ret;
+
+	/* See comment in ib_cm_establish about lookup. */
+	cm_id_priv = cm_acquire_id(work->local_id, work->remote_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static int cm_rtu_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rtu_msg *rtu_msg;
+	unsigned long flags;
+	int ret;
+
+	rtu_msg = (struct cm_rtu_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(rtu_msg->remote_comm_id,
+				   rtu_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &rtu_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_REP_SENT &&
+	    cm_id_priv->id.state != IB_CM_MRA_REP_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_ESTABLISHED;
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_dreq(struct cm_dreq_msg *dreq_msg,
+			  struct cm_id_private *cm_id_priv,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&dreq_msg->hdr, CM_DREQ_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_DREQ));
+	dreq_msg->local_comm_id = cm_id_priv->id.local_id;
+	dreq_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_dreq_set_remote_qpn(dreq_msg, cm_id_priv->remote_qpn);
+
+	if (private_data && private_data_len)
+		memcpy(dreq_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_dreq(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_DREQ_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_ESTABLISHED) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret) {
+		cm_enter_timewait(cm_id_priv);
+		goto out;
+	}
+
+	cm_format_dreq((struct cm_dreq_msg *) msg->mad, cm_id_priv,
+		       private_data, private_data_len);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_DREQ_SENT;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		cm_enter_timewait(cm_id_priv);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->state = IB_CM_DREQ_SENT;
+	cm_id_priv->msg = msg;
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_dreq);
+
+static void cm_format_drep(struct cm_drep_msg *drep_msg,
+			  struct cm_id_private *cm_id_priv,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&drep_msg->hdr, CM_DREP_ATTR_ID, cm_id_priv->tid);
+	drep_msg->local_comm_id = cm_id_priv->id.local_id;
+	drep_msg->remote_comm_id = cm_id_priv->id.remote_id;
+
+	if (private_data && private_data_len)
+		memcpy(drep_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_drep(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	void *data;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_DREP_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	data = cm_copy_private_data(private_data, private_data_len);
+	if (IS_ERR(data))
+		return PTR_ERR(data);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_DREQ_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		kfree(data);
+		return -EINVAL;
+	}
+
+	cm_set_private_data(cm_id_priv, data, private_data_len);
+	cm_enter_timewait(cm_id_priv);
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv,
+		       private_data, private_data_len);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_drep);
+
+static int cm_dreq_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_dreq_msg *dreq_msg;
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	dreq_msg = (struct cm_dreq_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(dreq_msg->remote_comm_id,
+				   dreq_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &dreq_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->local_qpn != cm_dreq_get_remote_qpn(dreq_msg))
+		goto unlock;
+
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REP_SENT:
+	case IB_CM_DREQ_SENT:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		break;
+	case IB_CM_ESTABLISHED:
+	case IB_CM_MRA_REP_RCVD:
+		break;
+	case IB_CM_TIMEWAIT:
+		if (cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg))
+			goto unlock;
+
+		cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv,
+			       cm_id_priv->private_data,
+			       cm_id_priv->private_data_len);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+		if (ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				     &msg->send_wr, &bad_send_wr))
+			cm_free_msg(msg);
+		goto deref;
+	default:
+		goto unlock;
+	}
+	cm_id_priv->id.state = IB_CM_DREQ_RCVD;
+	cm_id_priv->tid = dreq_msg->hdr.tid;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+deref:	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static int cm_drep_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_drep_msg *drep_msg;
+	unsigned long flags;
+	int ret;
+
+	drep_msg = (struct cm_drep_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(drep_msg->remote_comm_id,
+				   drep_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &drep_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_DREQ_SENT &&
+	    cm_id_priv->id.state != IB_CM_DREQ_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_enter_timewait(cm_id_priv);
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+int ib_send_cm_rej(struct ib_cm_id *cm_id,
+		   enum ib_cm_rej_reason reason,
+		   void *ari,
+		   u8 ari_length,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if ((private_data && private_data_len > IB_CM_REJ_PRIVATE_DATA_SIZE) ||
+	    (ari && ari_length > IB_CM_REJ_ARI_LENGTH))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (!ret)
+			cm_format_rej((struct cm_rej_msg *) msg->mad,
+				      cm_id_priv, reason, ari, ari_length,
+				      private_data, private_data_len);
+
+		cm_reset_to_idle(cm_id_priv);
+		break;
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (!ret)
+			cm_format_rej((struct cm_rej_msg *) msg->mad,
+				      cm_id_priv, reason, ari, ari_length,
+				      private_data, private_data_len);
+
+		cm_enter_timewait(cm_id_priv);
+		break;
+	default:
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (ret)
+		goto out;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret)
+		cm_free_msg(msg);
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_rej);
+
+static void cm_format_rej_event(struct cm_work *work)
+{
+	struct cm_rej_msg *rej_msg;
+	struct ib_cm_rej_event_param *param;
+
+	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.rej_rcvd;
+	param->ari = rej_msg->ari;
+	param->ari_length = cm_rej_get_reject_info_len(rej_msg);
+	param->reason = rej_msg->reason;
+	work->cm_event.private_data = &rej_msg->private_data;
+}
+
+static struct cm_id_private * cm_acquire_rejected_id(struct cm_rej_msg *rej_msg)
+{
+	struct cm_timewait_info *timewait_info;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	u32 remote_id;
+
+	remote_id = rej_msg->local_comm_id;
+
+	if (rej_msg->reason == IB_CM_REJ_TIMEOUT) {
+		spin_lock_irqsave(&cm.lock, flags);
+		timewait_info = cm_find_remote_id( *((u64 *) rej_msg->ari),
+						  remote_id);
+		if (!timewait_info) {
+			spin_unlock_irqrestore(&cm.lock, flags);
+			return NULL;
+		}
+		cm_id_priv = idr_find(&cm.local_id_table,
+				      (int) timewait_info->work.local_id);
+		if (cm_id_priv) {
+			if (cm_id_priv->id.remote_id == remote_id)
+				atomic_inc(&cm_id_priv->refcount);
+			else
+				cm_id_priv = NULL;
+		}
+		spin_unlock_irqrestore(&cm.lock, flags);
+	} else if (cm_rej_get_msg_rejected(rej_msg) == CM_MSG_RESPONSE_REQ)
+		cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, 0);
+	else
+		cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, remote_id);
+
+	return cm_id_priv;
+}
+
+static int cm_rej_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rej_msg *rej_msg;
+	unsigned long flags;
+	int ret;
+
+	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_rejected_id(rej_msg);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	cm_format_rej_event(work);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		/* fall through */
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+		if (rej_msg->reason == IB_CM_REJ_STALE_CONN)
+			cm_enter_timewait(cm_id_priv);
+		else
+			cm_reset_to_idle(cm_id_priv);
+		break;
+	case IB_CM_DREQ_SENT:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		/* fall through */
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_ESTABLISHED:
+		cm_enter_timewait(cm_id_priv);
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+int ib_send_cm_mra(struct ib_cm_id *cm_id,
+		   u8 service_timeout,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	void *data;
+	unsigned long flags;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_MRA_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	data = cm_copy_private_data(private_data, private_data_len);
+	if (IS_ERR(data))
+		return PTR_ERR(data);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch(cm_id_priv->id.state) {
+	case IB_CM_REQ_RCVD:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (ret)
+			goto error1;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REQ, service_timeout,
+			      private_data, private_data_len);
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+		if (ret)
+			goto error2;
+		cm_id->state = IB_CM_MRA_REQ_SENT;
+		break;
+	case IB_CM_REP_RCVD:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (ret)
+			goto error1;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REP, service_timeout,
+			      private_data, private_data_len);
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+		if (ret)
+			goto error2;
+		cm_id->state = IB_CM_MRA_REP_SENT;
+		break;
+	case IB_CM_ESTABLISHED:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (ret)
+			goto error1;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_OTHER, service_timeout,
+			      private_data, private_data_len);
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+		if (ret)
+			goto error2;
+		cm_id->lap_state = IB_CM_MRA_LAP_SENT;
+		break;
+	default:
+		ret = -EINVAL;
+		goto error1;
+	}
+	cm_id_priv->service_timeout = service_timeout;
+	cm_set_private_data(cm_id_priv, data, private_data_len);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return 0;
+
+error1:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	kfree(data);
+	return ret;
+
+error2:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	kfree(data);
+	cm_free_msg(msg);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_mra);
+
+static struct cm_id_private * cm_acquire_mraed_id(struct cm_mra_msg *mra_msg)
+{
+	switch (cm_mra_get_msg_mraed(mra_msg)) {
+	case CM_MSG_RESPONSE_REQ:
+		return cm_acquire_id(mra_msg->remote_comm_id, 0);
+	case CM_MSG_RESPONSE_REP:
+	case CM_MSG_RESPONSE_OTHER:
+		return cm_acquire_id(mra_msg->remote_comm_id,
+				     mra_msg->local_comm_id);
+	default:
+		return NULL;
+	}
+}
+
+static int cm_mra_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_mra_msg *mra_msg;
+	unsigned long flags;
+	int timeout, ret;
+
+	mra_msg = (struct cm_mra_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_mraed_id(mra_msg);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &mra_msg->private_data;
+	work->cm_event.param.mra_rcvd.service_timeout =
+					cm_mra_get_service_timeout(mra_msg);
+	timeout = cm_convert_to_ms(cm_mra_get_service_timeout(mra_msg)) +
+		  cm_convert_to_ms(cm_id_priv->av.packet_life_time);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+		if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_REQ ||
+		    ib_modify_mad(cm_id_priv->av.port->mad_agent,
+				  (unsigned long) cm_id_priv->msg, timeout))
+			goto out;
+		cm_id_priv->id.state = IB_CM_MRA_REQ_RCVD;
+		break;
+	case IB_CM_REP_SENT:
+		if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_REP ||
+		    ib_modify_mad(cm_id_priv->av.port->mad_agent,
+				  (unsigned long) cm_id_priv->msg, timeout))
+			goto out;
+		cm_id_priv->id.state = IB_CM_MRA_REP_RCVD;
+		break;
+	case IB_CM_ESTABLISHED:
+		if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_OTHER ||
+		    cm_id_priv->id.lap_state != IB_CM_LAP_SENT ||
+		    ib_modify_mad(cm_id_priv->av.port->mad_agent,
+				  (unsigned long) cm_id_priv->msg, timeout))
+			goto out;
+		cm_id_priv->id.lap_state = IB_CM_MRA_LAP_RCVD;
+		break;
+	default:
+		goto out;
+	}
+
+	cm_id_priv->msg->context[1] = (void *) (unsigned long)
+				      cm_id_priv->id.state;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_lap(struct cm_lap_msg *lap_msg,
+			  struct cm_id_private *cm_id_priv,
+			  struct ib_sa_path_rec *alternate_path,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&lap_msg->hdr, CM_LAP_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_LAP));
+	lap_msg->local_comm_id = cm_id_priv->id.local_id;
+	lap_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_lap_set_remote_qpn(lap_msg, cm_id_priv->remote_qpn);
+	/* todo: need remote CM response timeout */
+	cm_lap_set_remote_resp_timeout(lap_msg, 0x1F);
+	lap_msg->alt_local_lid = alternate_path->slid;
+	lap_msg->alt_remote_lid = alternate_path->dlid;
+	lap_msg->alt_local_gid = alternate_path->sgid;
+	lap_msg->alt_remote_gid = alternate_path->dgid;
+	cm_lap_set_flow_label(lap_msg, alternate_path->flow_label);
+	cm_lap_set_traffic_class(lap_msg, alternate_path->traffic_class);
+	lap_msg->alt_hop_limit = alternate_path->hop_limit;
+	cm_lap_set_packet_rate(lap_msg, alternate_path->rate);
+	cm_lap_set_sl(lap_msg, alternate_path->sl);
+	cm_lap_set_subnet_local(lap_msg, 1); /* local only... */
+	cm_lap_set_local_ack_timeout(lap_msg,
+		min(31, alternate_path->packet_life_time + 1));
+
+	if (private_data && private_data_len)
+		memcpy(lap_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_lap(struct ib_cm_id *cm_id,
+		   struct ib_sa_path_rec *alternate_path,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_LAP_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_ESTABLISHED ||
+	    cm_id->lap_state != IB_CM_LAP_IDLE) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_lap((struct cm_lap_msg *) msg->mad, cm_id_priv,
+		      alternate_path, private_data, private_data_len);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_ESTABLISHED;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->lap_state = IB_CM_LAP_SENT;
+	cm_id_priv->msg = msg;
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_lap);
+
+static void cm_format_path_from_lap(struct ib_sa_path_rec *path,
+				    struct cm_lap_msg *lap_msg)
+{
+	memset(path, 0, sizeof *path);
+	path->dgid = lap_msg->alt_local_gid;
+	path->sgid = lap_msg->alt_remote_gid;
+	path->dlid = lap_msg->alt_local_lid;
+	path->slid = lap_msg->alt_remote_lid;
+	path->flow_label = cm_lap_get_flow_label(lap_msg);
+	path->hop_limit = lap_msg->alt_hop_limit;
+	path->traffic_class = cm_lap_get_traffic_class(lap_msg);
+	path->reversible = 1;
+	/* pkey is same as in REQ */
+	path->sl = cm_lap_get_sl(lap_msg);
+	path->mtu_selector = IB_SA_EQ;
+	/* mtu is same as in REQ */
+	path->rate_selector = IB_SA_EQ;
+	path->rate = cm_lap_get_packet_rate(lap_msg);
+	path->packet_life_time_selector = IB_SA_EQ;
+	path->packet_life_time = cm_lap_get_local_ack_timeout(lap_msg);
+	path->packet_life_time -= (path->packet_life_time > 0);
+}
+
+static int cm_lap_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_lap_msg *lap_msg;
+	struct ib_cm_lap_event_param *param;
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	/* todo: verify LAP request and send reject APR if invalid. */
+	lap_msg = (struct cm_lap_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(lap_msg->remote_comm_id,
+				   lap_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+	
+	param = &work->cm_event.param.lap_rcvd;
+	param->alternate_path = &work->path[0];
+	cm_format_path_from_lap(param->alternate_path, lap_msg);
+	work->cm_event.private_data = &lap_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED)
+		goto unlock;
+	
+	switch (cm_id_priv->id.lap_state) {
+	case IB_CM_LAP_IDLE:
+		break;
+	case IB_CM_MRA_LAP_SENT:
+		if (cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg))
+			goto unlock;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_OTHER,
+			      cm_id_priv->service_timeout,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+		if (ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				     &msg->send_wr, &bad_send_wr))
+			cm_free_msg(msg);
+		goto deref;
+	default:
+		goto unlock;
+	}
+
+	cm_id_priv->id.lap_state = IB_CM_LAP_RCVD;
+	cm_id_priv->tid = lap_msg->hdr.tid;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+deref:	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_apr(struct cm_apr_msg *apr_msg,
+			  struct cm_id_private *cm_id_priv,
+			  enum ib_cm_apr_status status,
+			  void *info,
+			  u8 info_length,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&apr_msg->hdr, CM_APR_ATTR_ID, cm_id_priv->tid);
+	apr_msg->local_comm_id = cm_id_priv->id.local_id;
+	apr_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	apr_msg->ap_status = (u8) status;
+
+	if (info && info_length) {
+		apr_msg->info_length = info_length;
+		memcpy(apr_msg->info, info, info_length);
+	}
+
+	if (private_data && private_data_len)
+		memcpy(apr_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_apr(struct ib_cm_id *cm_id,
+		   enum ib_cm_apr_status status,
+		   void *info,
+		   u8 info_length,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if ((private_data && private_data_len > IB_CM_APR_PRIVATE_DATA_SIZE) ||
+	    (info && info_length > IB_CM_APR_INFO_LENGTH))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_ESTABLISHED ||
+	    (cm_id->lap_state != IB_CM_LAP_RCVD &&
+	     cm_id->lap_state != IB_CM_MRA_LAP_SENT)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_apr((struct cm_apr_msg *) msg->mad, cm_id_priv, status,
+		      info, info_length, private_data, private_data_len);
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->lap_state = IB_CM_LAP_IDLE;
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_apr);
+
+static int cm_apr_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_apr_msg *apr_msg;
+	unsigned long flags;
+	int ret;
+
+	apr_msg = (struct cm_apr_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(apr_msg->remote_comm_id,
+				   apr_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL; /* Unmatched reply. */
+
+	work->cm_event.param.apr_rcvd.ap_status = apr_msg->ap_status;
+	work->cm_event.param.apr_rcvd.apr_info = &apr_msg->info;
+	work->cm_event.param.apr_rcvd.info_len = apr_msg->info_length;
+	work->cm_event.private_data = &apr_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED || 
+	    (cm_id_priv->id.lap_state != IB_CM_LAP_SENT &&
+	     cm_id_priv->id.lap_state != IB_CM_MRA_LAP_RCVD)) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.lap_state = IB_CM_LAP_IDLE;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	cm_id_priv->msg = NULL;
+
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static int cm_timewait_handler(struct cm_work *work)
+{
+	struct cm_timewait_info *timewait_info;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	int ret;
+
+	timewait_info = (struct cm_timewait_info *)work;
+	cm_cleanup_timewait(timewait_info);
+
+	cm_id_priv = cm_acquire_id(timewait_info->work.local_id,
+				   timewait_info->work.remote_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_TIMEWAIT ||
+	    cm_id_priv->remote_qpn != timewait_info->remote_qpn) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_IDLE;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_sidr_req(struct cm_sidr_req_msg *sidr_req_msg,
+			       struct cm_id_private *cm_id_priv,
+			       struct ib_cm_sidr_req_param *param)
+{
+	cm_format_mad_hdr(&sidr_req_msg->hdr, CM_SIDR_REQ_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_SIDR));
+	sidr_req_msg->request_id = cm_id_priv->id.local_id;
+	sidr_req_msg->pkey = param->pkey;
+	sidr_req_msg->service_id = param->service_id;
+
+	if (param->private_data && param->private_data_len)
+		memcpy(sidr_req_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+int ib_send_cm_sidr_req(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_req_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (!param->path || (param->private_data && 
+	     param->private_data_len > IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	ret = cm_init_av_by_path(param->path, &cm_id_priv->av);
+	if (ret)
+		goto out;
+
+	cm_id->service_id = param->service_id;
+	cm_id->service_mask = ~0ULL;
+	cm_id_priv->timeout_ms = param->timeout_ms;
+	cm_id_priv->max_cm_retries = param->max_cm_retries;
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_sidr_req((struct cm_sidr_req_msg *) msg->mad, cm_id_priv,
+			   param);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_SIDR_REQ_SENT;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state == IB_CM_IDLE)
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+	else
+		ret = -EINVAL;
+
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		goto out;
+	}
+	cm_id->state = IB_CM_SIDR_REQ_SENT;
+	cm_id_priv->msg = msg;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+out:
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_sidr_req);
+
+static void cm_format_sidr_req_event(struct cm_work *work,
+				     struct ib_cm_id *listen_id)
+{
+	struct cm_sidr_req_msg *sidr_req_msg;
+	struct ib_cm_sidr_req_event_param *param;
+
+	sidr_req_msg = (struct cm_sidr_req_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.sidr_req_rcvd;
+	param->pkey = sidr_req_msg->pkey;
+	param->listen_id = listen_id;
+	param->device = work->port->mad_agent->device;
+	param->port = work->port->port_num;
+	work->cm_event.private_data = &sidr_req_msg->private_data;
+}
+
+static int cm_sidr_req_handler(struct cm_work *work)
+{
+	struct ib_cm_id *cm_id;
+	struct cm_id_private *cm_id_priv, *cur_cm_id_priv;
+	struct cm_sidr_req_msg *sidr_req_msg;
+	struct ib_wc *wc;
+	unsigned long flags;
+
+	cm_id = ib_create_cm_id(NULL, NULL);
+	if (IS_ERR(cm_id))
+		return PTR_ERR(cm_id);
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+
+	/* Record SGID/SLID and request ID for lookup. */
+	sidr_req_msg = (struct cm_sidr_req_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	wc = work->mad_recv_wc->wc;
+	cm_id_priv->av.dgid.global.subnet_prefix = wc->slid;
+	cm_id_priv->av.dgid.global.interface_id = 0;
+	cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
+				&cm_id_priv->av);
+	cm_id_priv->id.remote_id = sidr_req_msg->request_id;
+	cm_id_priv->id.state = IB_CM_SIDR_REQ_RCVD;
+	cm_id_priv->tid = sidr_req_msg->hdr.tid;
+	atomic_inc(&cm_id_priv->work_count);
+
+	spin_lock_irqsave(&cm.lock, flags);
+	cur_cm_id_priv = cm_insert_remote_sidr(cm_id_priv);
+	if (cur_cm_id_priv) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		goto out; /* Duplicate message. */
+	}
+	cur_cm_id_priv = cm_find_listen(sidr_req_msg->service_id);
+	if (!cur_cm_id_priv) {
+		rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
+		spin_unlock_irqrestore(&cm.lock, flags);
+		/* todo: reply with no match */
+		goto out; /* No match. */
+	}
+	atomic_inc(&cur_cm_id_priv->refcount);
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	cm_id_priv->id.cm_handler = cur_cm_id_priv->id.cm_handler;
+	cm_id_priv->id.context = cur_cm_id_priv->id.context;
+	cm_id_priv->id.service_id = sidr_req_msg->service_id;
+	cm_id_priv->id.service_mask = ~0ULL;
+
+	cm_format_sidr_req_event(work, &cur_cm_id_priv->id);
+	cm_process_work(cm_id_priv, work);
+	cm_deref_id(cur_cm_id_priv);
+	return 0;
+out:
+	ib_destroy_cm_id(&cm_id_priv->id);
+	return -EINVAL;
+}
+
+static void cm_format_sidr_rep(struct cm_sidr_rep_msg *sidr_rep_msg,
+			       struct cm_id_private *cm_id_priv,
+			       struct ib_cm_sidr_rep_param *param)
+{
+	cm_format_mad_hdr(&sidr_rep_msg->hdr, CM_SIDR_REP_ATTR_ID,
+			  cm_id_priv->tid);
+	sidr_rep_msg->request_id = cm_id_priv->id.remote_id;
+	sidr_rep_msg->status = param->status;
+	cm_sidr_rep_set_qpn(sidr_rep_msg, cpu_to_be32(param->qp_num));
+	sidr_rep_msg->service_id = cm_id_priv->id.service_id;
+	sidr_rep_msg->qkey = cpu_to_be32(param->qkey);
+
+	if (param->info && param->info_length)
+		memcpy(sidr_rep_msg->info, param->info, param->info_length);
+
+	if (param->private_data && param->private_data_len)
+		memcpy(sidr_rep_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+int ib_send_cm_sidr_rep(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_rep_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if ((param->info && param->info_length > IB_CM_SIDR_REP_INFO_LENGTH) ||
+	    (param->private_data &&
+	     param->private_data_len > IB_CM_SIDR_REP_PRIVATE_DATA_SIZE))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_SIDR_REQ_RCVD) {
+		ret = -EINVAL;
+		goto error;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto error;
+
+	cm_format_sidr_rep((struct cm_sidr_rep_msg *) msg->mad, cm_id_priv,
+			   param);
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+	cm_id->state = IB_CM_IDLE;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	spin_lock_irqsave(&cm.lock, flags);
+	rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
+	spin_unlock_irqrestore(&cm.lock, flags);
+	return 0;
+
+error:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_sidr_rep);
+
+static void cm_format_sidr_rep_event(struct cm_work *work)
+{
+	struct cm_sidr_rep_msg *sidr_rep_msg;
+	struct ib_cm_sidr_rep_event_param *param;
+
+	sidr_rep_msg = (struct cm_sidr_rep_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.sidr_rep_rcvd;
+	param->status = sidr_rep_msg->status;
+	param->qkey = be32_to_cpu(sidr_rep_msg->qkey);
+	param->qpn = be32_to_cpu(cm_sidr_rep_get_qpn(sidr_rep_msg));
+	param->info = &sidr_rep_msg->info;
+	param->info_len = sidr_rep_msg->info_length;
+	work->cm_event.private_data = &sidr_rep_msg->private_data;
+}
+
+static int cm_sidr_rep_handler(struct cm_work *work)
+{
+	struct cm_sidr_rep_msg *sidr_rep_msg;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+
+	sidr_rep_msg = (struct cm_sidr_rep_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(sidr_rep_msg->request_id, 0);
+	if (!cm_id_priv)
+		return -EINVAL; /* Unmatched reply. */
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_SIDR_REQ_SENT) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_IDLE;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	cm_format_sidr_rep_event(work);
+	cm_process_work(cm_id_priv, work);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_process_send_error(struct ib_mad_send_buf *msg,
+				  enum ib_wc_status wc_status)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_cm_event cm_event;
+	enum ib_cm_state state;
+	unsigned long flags;
+	int ret;
+
+	memset(&cm_event, 0, sizeof cm_event);
+	cm_id_priv = msg->context[0];
+
+	/* Discard old sends or ones without a response. */
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	state = (enum ib_cm_state) (unsigned long) msg->context[1];
+	if (msg != cm_id_priv->msg || state != cm_id_priv->id.state)
+		goto discard;
+
+	switch (state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+		cm_reset_to_idle(cm_id_priv);
+		cm_event.event = IB_CM_REQ_ERROR;
+		break;
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		cm_reset_to_idle(cm_id_priv);
+		cm_event.event = IB_CM_REP_ERROR;
+		break;
+	case IB_CM_DREQ_SENT:
+		cm_enter_timewait(cm_id_priv);
+		cm_event.event = IB_CM_DREQ_ERROR;
+		break;
+	case IB_CM_SIDR_REQ_SENT:
+		cm_id_priv->id.state = IB_CM_IDLE;
+		cm_event.event = IB_CM_SIDR_REQ_ERROR;
+		break;
+	default:
+		goto discard;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	cm_event.param.send_status = wc_status;
+
+	/* No other events can occur on the cm_id at this point. */
+	ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, &cm_event);
+	cm_free_msg(msg);
+	if (ret)
+		ib_destroy_cm_id(&cm_id_priv->id);
+	return;
+discard:
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	cm_free_msg(msg);
+}
+
+static void cm_send_handler(struct ib_mad_agent *mad_agent,
+			    struct ib_mad_send_wc *mad_send_wc)
+{
+	struct ib_mad_send_buf *msg;
+
+	msg = (struct ib_mad_send_buf *)(unsigned long)mad_send_wc->wr_id;
+
+	switch (mad_send_wc->status) {
+	case IB_WC_SUCCESS:
+	case IB_WC_WR_FLUSH_ERR:
+		cm_free_msg(msg);
+		break;
+	default:
+		if (msg->context[0] && msg->context[1])
+			cm_process_send_error(msg, mad_send_wc->status);
+		else
+			cm_free_msg(msg);
+		break;
+	}
+}
+
+static void cm_work_handler(void *data)
+{
+	struct cm_work *work = data;
+	int ret;
+
+	switch (work->cm_event.event) {
+	case IB_CM_REQ_RECEIVED:
+		ret = cm_req_handler(work);
+		break;
+	case IB_CM_MRA_RECEIVED:
+		ret = cm_mra_handler(work);
+		break;
+	case IB_CM_REJ_RECEIVED:
+		ret = cm_rej_handler(work);
+		break;
+	case IB_CM_REP_RECEIVED:
+		ret = cm_rep_handler(work);
+		break;
+	case IB_CM_RTU_RECEIVED:
+		ret = cm_rtu_handler(work);
+		break;
+	case IB_CM_USER_ESTABLISHED:
+		ret = cm_establish_handler(work);
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		ret = cm_dreq_handler(work);
+		break;
+	case IB_CM_DREP_RECEIVED:
+		ret = cm_drep_handler(work);
+		break;
+	case IB_CM_SIDR_REQ_RECEIVED:
+		ret = cm_sidr_req_handler(work);
+		break;
+	case IB_CM_SIDR_REP_RECEIVED:
+		ret = cm_sidr_rep_handler(work);
+		break;
+	case IB_CM_LAP_RECEIVED:
+		ret = cm_lap_handler(work);
+		break;
+	case IB_CM_APR_RECEIVED:
+		ret = cm_apr_handler(work);
+		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		ret = cm_timewait_handler(work);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	if (ret)
+		cm_free_work(work);
+}
+
+int ib_cm_establish(struct ib_cm_id *cm_id)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_work *work;
+	unsigned long flags;
+	int ret = 0;
+
+	work = kmalloc(sizeof *work, GFP_ATOMIC);
+	if (!work)
+		return -ENOMEM;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state)
+	{
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		cm_id->state = IB_CM_ESTABLISHED;
+		break;
+	case IB_CM_ESTABLISHED:
+		ret = -EISCONN;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret) {
+		kfree(work);
+		goto out;
+	}
+
+	/*
+	 * The CM worker thread may try to destroy the cm_id before it
+	 * can execute this work item.  To prevent potential deadlock,
+	 * we need to find the cm_id once we're in the context of the 
+	 * worker thread, rather than holding a reference on it.
+	 */
+	INIT_WORK(&work->work, cm_work_handler, work);
+	work->local_id = cm_id->local_id;
+	work->remote_id = cm_id->remote_id;
+	work->mad_recv_wc = NULL;
+	work->cm_event.event = IB_CM_USER_ESTABLISHED;
+	queue_work(cm.wq, &work->work);
+out:
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_establish);
+
+static void cm_recv_handler(struct ib_mad_agent *mad_agent,
+			    struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct cm_work *work;
+	enum ib_cm_event_type event;
+	int paths = 0;
+
+	switch (mad_recv_wc->recv_buf.mad->mad_hdr.attr_id) {
+	case CM_REQ_ATTR_ID:
+		paths = 1 + (((struct cm_req_msg *) mad_recv_wc->recv_buf.mad)->
+						    alt_local_lid != 0);
+		event = IB_CM_REQ_RECEIVED;
+		break;
+	case CM_MRA_ATTR_ID:
+		event = IB_CM_MRA_RECEIVED;
+		break;
+	case CM_REJ_ATTR_ID:
+		event = IB_CM_REJ_RECEIVED;
+		break;
+	case CM_REP_ATTR_ID:
+		event = IB_CM_REP_RECEIVED;
+		break;
+	case CM_RTU_ATTR_ID:
+		event = IB_CM_RTU_RECEIVED;
+		break;
+	case CM_DREQ_ATTR_ID:
+		event = IB_CM_DREQ_RECEIVED;
+		break;
+	case CM_DREP_ATTR_ID:
+		event = IB_CM_DREP_RECEIVED;
+		break;
+	case CM_SIDR_REQ_ATTR_ID:
+		event = IB_CM_SIDR_REQ_RECEIVED;
+		break;
+	case CM_SIDR_REP_ATTR_ID:
+		event = IB_CM_SIDR_REP_RECEIVED;
+		break;
+	case CM_LAP_ATTR_ID:
+		paths = 1;
+		event = IB_CM_LAP_RECEIVED;
+		break;
+	case CM_APR_ATTR_ID:
+		event = IB_CM_APR_RECEIVED;
+		break;
+	default:
+		ib_free_recv_mad(mad_recv_wc);
+		return;
+	}
+
+	work = kmalloc(sizeof *work + sizeof(struct ib_sa_path_rec) * paths,
+		       GFP_KERNEL);
+	if (!work) {
+		ib_free_recv_mad(mad_recv_wc);
+		return;
+	}
+
+	INIT_WORK(&work->work, cm_work_handler, work);
+	work->cm_event.event = event;
+	work->mad_recv_wc = mad_recv_wc;
+	work->port = (struct cm_port *)mad_agent->context;
+	queue_work(cm.wq, &work->work);
+}
+
+static int cm_init_qp_init_attr(struct cm_id_private *cm_id_priv,
+				struct ib_qp_attr *qp_attr,
+				int *qp_attr_mask)
+{
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+	case IB_CM_ESTABLISHED:
+		*qp_attr_mask = IB_QP_STATE | IB_QP_ACCESS_FLAGS |
+				IB_QP_PKEY_INDEX | IB_QP_PORT;
+		qp_attr->qp_access_flags = IB_ACCESS_LOCAL_WRITE;
+		if (cm_id_priv->responder_resources)
+			qp_attr->qp_access_flags |= IB_ACCESS_REMOTE_WRITE |
+						    IB_ACCESS_REMOTE_READ;
+		qp_attr->pkey_index = cm_id_priv->av.pkey_index;
+		qp_attr->port_num = cm_id_priv->av.port->port_num;
+		ret = 0;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+
+static int cm_init_qp_rtr_attr(struct cm_id_private *cm_id_priv,
+			       struct ib_qp_attr *qp_attr,
+			       int *qp_attr_mask)
+{
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+	case IB_CM_ESTABLISHED:
+		*qp_attr_mask = IB_QP_STATE | IB_QP_AV | IB_QP_PATH_MTU |
+				IB_QP_DEST_QPN | IB_QP_RQ_PSN |
+				IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER;
+		qp_attr->ah_attr = cm_id_priv->av.ah_attr;
+		qp_attr->path_mtu = cm_id_priv->path_mtu;
+		qp_attr->dest_qp_num = be32_to_cpu(cm_id_priv->remote_qpn);
+		qp_attr->rq_psn = be32_to_cpu(cm_id_priv->rq_psn);
+		qp_attr->max_dest_rd_atomic = cm_id_priv->responder_resources;
+		qp_attr->min_rnr_timer = 0;
+		if (cm_id_priv->alt_av.ah_attr.dlid) {
+			*qp_attr_mask |= IB_QP_ALT_PATH;
+			qp_attr->alt_ah_attr = cm_id_priv->alt_av.ah_attr;
+		}
+		ret = 0;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+
+static int cm_init_qp_rts_attr(struct cm_id_private *cm_id_priv,
+			       struct ib_qp_attr *qp_attr,
+			       int *qp_attr_mask)
+{
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+	case IB_CM_ESTABLISHED:
+		*qp_attr_mask = IB_QP_STATE | IB_QP_TIMEOUT | IB_QP_RETRY_CNT |
+				IB_QP_RNR_RETRY | IB_QP_SQ_PSN |
+				IB_QP_MAX_QP_RD_ATOMIC;
+		qp_attr->timeout = cm_id_priv->local_ack_timeout;
+		qp_attr->retry_cnt = cm_id_priv->retry_count;
+		qp_attr->rnr_retry = cm_id_priv->rnr_retry_count;
+		qp_attr->sq_psn = be32_to_cpu(cm_id_priv->sq_psn);
+		qp_attr->max_rd_atomic = cm_id_priv->initiator_depth;
+		if (cm_id_priv->alt_av.ah_attr.dlid) {
+			*qp_attr_mask |= IB_QP_PATH_MIG_STATE;
+			qp_attr->path_mig_state = IB_MIG_REARM;
+		}
+		ret = 0;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+
+int ib_cm_init_qp_attr(struct ib_cm_id *cm_id,
+		       struct ib_qp_attr *qp_attr,
+		       int *qp_attr_mask)
+{
+	struct cm_id_private *cm_id_priv;
+	int ret;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	switch (qp_attr->qp_state) {
+	case IB_QPS_INIT:
+		ret = cm_init_qp_init_attr(cm_id_priv, qp_attr, qp_attr_mask);
+		break;
+	case IB_QPS_RTR:
+		ret = cm_init_qp_rtr_attr(cm_id_priv, qp_attr, qp_attr_mask);
+		break;
+	case IB_QPS_RTS:
+		ret = cm_init_qp_rts_attr(cm_id_priv, qp_attr, qp_attr_mask);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_init_qp_attr);
+
+static u64 cm_get_ca_guid(struct ib_device *device)
+{
+	struct ib_device_attr *device_attr;
+	u64 guid;
+	int ret;
+
+	device_attr = kmalloc(sizeof *device_attr, GFP_KERNEL);
+	if (!device_attr)
+		return 0;
+
+	ret = ib_query_device(device, device_attr);
+	guid = ret ? 0 : device_attr->node_guid;
+	kfree(device_attr);
+	return guid;
+}
+
+static void cm_add_one(struct ib_device *device)
+{
+	struct cm_device *cm_dev;
+	struct cm_port *port;
+	struct ib_mad_reg_req reg_req = {
+		.mgmt_class = IB_MGMT_CLASS_CM,
+		.mgmt_class_version = IB_CM_CLASS_VERSION
+	};
+	struct ib_port_modify port_modify = {
+		.set_port_cap_mask = IB_PORT_CM_SUP
+	};
+	unsigned long flags;
+	int ret;
+	u8 i;
+
+	cm_dev = kmalloc(sizeof(*cm_dev) + sizeof(*port) *
+			 device->phys_port_cnt, GFP_KERNEL);
+	if (!cm_dev)
+		return;
+
+	cm_dev->device = device;
+	cm_dev->ca_guid = cm_get_ca_guid(device);
+	if (!cm_dev->ca_guid)
+		goto error1;
+
+	set_bit(IB_MGMT_METHOD_SEND, reg_req.method_mask);
+	for (i = 1; i <= device->phys_port_cnt; i++) {
+		port = &cm_dev->port[i-1];
+		port->cm_dev = cm_dev;
+		port->port_num = i;
+		port->mad_agent = ib_register_mad_agent(device, i,
+							IB_QPT_GSI,
+							&reg_req,
+							0,
+							cm_send_handler,
+							cm_recv_handler,
+							port);
+		if (IS_ERR(port->mad_agent))
+			goto error2;
+
+		ret = ib_modify_port(device, i, 0, &port_modify);
+		if (ret)
+			goto error3;
+	}
+	ib_set_client_data(device, &cm_client, cm_dev);
+	
+	write_lock_irqsave(&cm.device_lock, flags);
+	list_add_tail(&cm_dev->list, &cm.device_list);
+	write_unlock_irqrestore(&cm.device_lock, flags);
+	return;
+
+error3:
+	ib_unregister_mad_agent(port->mad_agent);
+error2:
+	port_modify.set_port_cap_mask = 0;
+	port_modify.clr_port_cap_mask = IB_PORT_CM_SUP;
+	while (--i) {
+		port = &cm_dev->port[i-1];
+		ib_modify_port(device, port->port_num, 0, &port_modify);
+		ib_unregister_mad_agent(port->mad_agent);
+	}
+error1:
+	kfree(cm_dev);
+}
+
+static void cm_remove_one(struct ib_device *device)
+{
+	struct cm_device *cm_dev;
+	struct cm_port *port;
+	struct ib_port_modify port_modify = {
+		.clr_port_cap_mask = IB_PORT_CM_SUP
+	};
+	unsigned long flags;
+	int i;
+
+	cm_dev = ib_get_client_data(device, &cm_client);
+	if (!cm_dev)
+		return;
+
+	write_lock_irqsave(&cm.device_lock, flags);
+	list_del(&cm_dev->list);
+	write_unlock_irqrestore(&cm.device_lock, flags);
+
+	for (i = 1; i <= device->phys_port_cnt; i++) {
+		port = &cm_dev->port[i-1];
+		ib_modify_port(device, port->port_num, 0, &port_modify);
+		ib_unregister_mad_agent(port->mad_agent);
+	}
+	kfree(cm_dev);
+}
+
+static int __init ib_cm_init(void)
+{
+	int ret;
+
+	memset(&cm, 0, sizeof cm);
+	INIT_LIST_HEAD(&cm.device_list);
+	rwlock_init(&cm.device_lock);
+	spin_lock_init(&cm.lock);
+	cm.listen_service_table = RB_ROOT;
+	cm.listen_service_id = __constant_be64_to_cpu(IB_CM_ASSIGN_SERVICE_ID);
+	cm.remote_id_table = RB_ROOT;
+	cm.remote_qp_table = RB_ROOT;
+	cm.remote_sidr_table = RB_ROOT;
+	idr_init(&cm.local_id_table);
+	idr_pre_get(&cm.local_id_table, GFP_KERNEL);
+
+	cm.wq = create_workqueue("ib_cm");
+	if (!cm.wq)
+		return -ENOMEM;
+
+	ret = ib_register_client(&cm_client);
+	if (ret)
+		goto error;
+
+	return 0;
+error:
+	destroy_workqueue(cm.wq);
+	return ret;
+}
+
+static void __exit ib_cm_cleanup(void)
+{
+	flush_workqueue(cm.wq);
+	destroy_workqueue(cm.wq);
+	ib_unregister_client(&cm_client);
+}
+
+module_init(ib_cm_init);
+module_exit(ib_cm_cleanup);
+
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband21/core/cm_msgs.h linux-2.6.13-rc2-mm1/drivers/infiniband22/core/cm_msgs.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband21/core/cm_msgs.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband22/core/cm_msgs.h	2005-02-03 14:05:21.000000000 -0500
@@ -0,0 +1,819 @@
+/*
+ * Copyright (c) 2004 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING the madirectory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use source and binary forms, with or
+ *     withmodification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retathe above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHWARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS THE
+ * SOFTWARE.
+ */
+#if !defined(CM_MSGS_H)
+#define CM_MSGS_H
+
+#include <ib_mad.h>
+
+/*
+ * Parameters to routines below should be in network-byte order, and values
+ * are returned in network-byte order.
+ */
+
+#define IB_CM_CLASS_VERSION	2 /* IB specification 1.2 */
+
+enum cm_msg_attr_id {
+	CM_REQ_ATTR_ID	    = __constant_htons(0x0010),
+	CM_MRA_ATTR_ID	    = __constant_htons(0x0011),
+	CM_REJ_ATTR_ID	    = __constant_htons(0x0012),
+	CM_REP_ATTR_ID	    = __constant_htons(0x0013),
+	CM_RTU_ATTR_ID	    = __constant_htons(0x0014),
+	CM_DREQ_ATTR_ID	    = __constant_htons(0x0015),
+	CM_DREP_ATTR_ID	    = __constant_htons(0x0016),
+	CM_SIDR_REQ_ATTR_ID = __constant_htons(0x0017),
+	CM_SIDR_REP_ATTR_ID = __constant_htons(0x0018),
+	CM_LAP_ATTR_ID      = __constant_htons(0x0019),
+	CM_APR_ATTR_ID      = __constant_htons(0x001A)
+};
+
+enum cm_msg_sequence {
+	CM_MSG_SEQUENCE_REQ,
+	CM_MSG_SEQUENCE_LAP,
+	CM_MSG_SEQUENCE_DREQ,
+	CM_MSG_SEQUENCE_SIDR
+};
+
+struct cm_req_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 rsvd4;
+	u64 service_id;
+	u64 local_ca_guid;
+	u32 rsvd24;
+	u32 local_qkey;
+	/* local QPN:24, responder resources:8 */
+	u32 offset32;
+	/* local EECN:24, initiator depth:8 */
+	u32 offset36;
+	/*
+	 * remote EECN:24, remote CM response timeout:5,
+	 * transport service type:2, end-to-end flow control:1
+	 */
+	u32 offset40;
+	/* starting PSN:24, local CM response timeout:5, retry count:3 */
+	u32 offset44;
+	u16 pkey;
+	/* path MTU:4, RDC exists:1, RNR retry count:3. */
+	u8 offset50;
+	/* max CM Retries:4, SRQ:1, rsvd:3 */
+	u8 offset51;
+
+	u16 primary_local_lid;
+	u16 primary_remote_lid;
+	union ib_gid primary_local_gid;
+	union ib_gid primary_remote_gid;
+	/* flow label:20, rsvd:6, packet rate:6 */
+	u32 primary_offset88;
+	u8 primary_traffic_class;
+	u8 primary_hop_limit;
+	/* SL:4, subnet local:1, rsvd:3 */
+	u8 primary_offset94;
+	/* local ACK timeout:5, rsvd:3 */
+	u8 primary_offset95;
+
+	u16 alt_local_lid;
+	u16 alt_remote_lid;
+	union ib_gid alt_local_gid;
+	union ib_gid alt_remote_gid;
+	/* flow label:20, rsvd:6, packet rate:6 */
+	u32 alt_offset132;
+	u8 alt_traffic_class;
+	u8 alt_hop_limit;
+	/* SL:4, subnet local:1, rsvd:3 */
+	u8 alt_offset138;
+	/* local ACK timeout:5, rsvd:3 */
+	u8 alt_offset139;
+
+	u8 private_data[IB_CM_REQ_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u32 cm_req_get_local_qpn(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32(be32_to_cpu(req_msg->offset32) >> 8);
+}
+
+static inline void cm_req_set_local_qpn(struct cm_req_msg *req_msg, u32 qpn)
+{
+	req_msg->offset32 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+					 (be32_to_cpu(req_msg->offset32) &
+					  0x000000FF));
+}
+
+static inline u8 cm_req_get_resp_res(struct cm_req_msg *req_msg)
+{
+	return (u8) be32_to_cpu(req_msg->offset32);
+}
+
+static inline void cm_req_set_resp_res(struct cm_req_msg *req_msg, u8 resp_res)
+{
+	req_msg->offset32 = cpu_to_be32(resp_res |
+					(be32_to_cpu(req_msg->offset32) &
+					 0xFFFFFF00));
+}
+
+static inline u8 cm_req_get_init_depth(struct cm_req_msg *req_msg)
+{
+	return (u8) be32_to_cpu(req_msg->offset36);
+}
+
+static inline void cm_req_set_init_depth(struct cm_req_msg *req_msg,
+					 u8 init_depth)
+{
+	req_msg->offset36 = cpu_to_be32(init_depth |
+					(be32_to_cpu(req_msg->offset36) &
+					 0xFFFFFF00));
+}
+
+static inline u8 cm_req_get_remote_resp_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) ((be32_to_cpu(req_msg->offset40) & 0xF8) >> 3);
+}
+
+static inline void cm_req_set_remote_resp_timeout(struct cm_req_msg *req_msg,
+						  u8 resp_timeout)
+{
+	req_msg->offset40 = cpu_to_be32((resp_timeout << 3) |
+					 (be32_to_cpu(req_msg->offset40) &
+					  0xFFFFFF07));
+}
+
+static inline enum ib_qp_type cm_req_get_qp_type(struct cm_req_msg *req_msg)
+{
+	u8 transport_type = (u8) (be32_to_cpu(req_msg->offset40) & 0x06) >> 1;
+	switch(transport_type) {
+	case 0: return IB_QPT_RC;
+	case 1: return IB_QPT_UC;
+	default: return 0;
+	}
+}
+
+static inline void cm_req_set_qp_type(struct cm_req_msg *req_msg,
+				      enum ib_qp_type qp_type)
+{
+	switch(qp_type) {
+	case IB_QPT_UC:
+		req_msg->offset40 = cpu_to_be32((be32_to_cpu(
+						  req_msg->offset40) &
+						   0xFFFFFFF9) | 0x2);
+	default: 
+		req_msg->offset40 = cpu_to_be32(be32_to_cpu(
+						 req_msg->offset40) &
+						  0xFFFFFFF9);
+	}
+}
+
+static inline u8 cm_req_get_flow_ctrl(struct cm_req_msg *req_msg)
+{
+	return be32_to_cpu(req_msg->offset40) & 0x1;
+}
+
+static inline void cm_req_set_flow_ctrl(struct cm_req_msg *req_msg,
+					u8 flow_ctrl)
+{
+	req_msg->offset40 = cpu_to_be32((flow_ctrl & 0x1) |
+					 (be32_to_cpu(req_msg->offset40) &
+					  0xFFFFFFFE));
+}
+
+static inline u32 cm_req_get_starting_psn(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32(be32_to_cpu(req_msg->offset44) >> 8);
+}
+
+static inline void cm_req_set_starting_psn(struct cm_req_msg *req_msg,
+					   u32 starting_psn)
+{
+	req_msg->offset44 = cpu_to_be32((be32_to_cpu(starting_psn) << 8) |
+			    (be32_to_cpu(req_msg->offset44) & 0x000000FF));
+}
+
+static inline u8 cm_req_get_local_resp_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) ((be32_to_cpu(req_msg->offset44) & 0xF8) >> 3);
+}
+
+static inline void cm_req_set_local_resp_timeout(struct cm_req_msg *req_msg,
+						 u8 resp_timeout)
+{
+	req_msg->offset44 = cpu_to_be32((resp_timeout << 3) |
+			    (be32_to_cpu(req_msg->offset44) & 0xFFFFFF07));
+}
+
+static inline u8 cm_req_get_retry_count(struct cm_req_msg *req_msg)
+{
+	return (u8) (be32_to_cpu(req_msg->offset44) & 0x7);
+}
+
+static inline void cm_req_set_retry_count(struct cm_req_msg *req_msg,
+					  u8 retry_count)
+{
+	req_msg->offset44 = cpu_to_be32((retry_count & 0x7) |
+			    (be32_to_cpu(req_msg->offset44) & 0xFFFFFFF8));
+}
+
+static inline u8 cm_req_get_path_mtu(struct cm_req_msg *req_msg)
+{
+	return req_msg->offset50 >> 4;
+}
+
+static inline void cm_req_set_path_mtu(struct cm_req_msg *req_msg, u8 path_mtu)
+{
+	req_msg->offset50 = (u8) ((req_msg->offset50 & 0xF) | (path_mtu << 4));
+}
+
+static inline u8 cm_req_get_rnr_retry_count(struct cm_req_msg *req_msg)
+{
+	return req_msg->offset50 & 0x7;
+}
+
+static inline void cm_req_set_rnr_retry_count(struct cm_req_msg *req_msg,
+					      u8 rnr_retry_count)
+{
+	req_msg->offset50 = (u8) ((req_msg->offset50 & 0xF8) |
+				  (rnr_retry_count & 0x7));
+}
+
+static inline u8 cm_req_get_max_cm_retries(struct cm_req_msg *req_msg)
+{
+	return req_msg->offset51 >> 4;
+}
+
+static inline void cm_req_set_max_cm_retries(struct cm_req_msg *req_msg,
+					     u8 retries)
+{
+	req_msg->offset51 = (u8) ((req_msg->offset51 & 0xF) | (retries << 4));
+}
+
+static inline u8 cm_req_get_srq(struct cm_req_msg *req_msg)
+{
+	return (req_msg->offset51 & 0x8) >> 3;
+}
+
+static inline void cm_req_set_srq(struct cm_req_msg *req_msg, u8 srq)
+{
+	req_msg->offset51 = (u8) ((req_msg->offset51 & 0xF7) |
+				  ((srq & 0x1) << 3));
+}
+
+static inline u32 cm_req_get_primary_flow_label(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32((be32_to_cpu(req_msg->primary_offset88) >> 12));
+}
+
+static inline void cm_req_set_primary_flow_label(struct cm_req_msg *req_msg,
+						 u32 flow_label)
+{
+	req_msg->primary_offset88 = cpu_to_be32(
+				    (be32_to_cpu(req_msg->primary_offset88) &
+				     0x00000FFF) |
+				     (be32_to_cpu(flow_label) << 12));
+}
+
+static inline u8 cm_req_get_primary_packet_rate(struct cm_req_msg *req_msg)
+{
+	return (u8) (be32_to_cpu(req_msg->primary_offset88) & 0x3F);
+}
+
+static inline void cm_req_set_primary_packet_rate(struct cm_req_msg *req_msg,
+						  u8 rate)
+{
+	req_msg->primary_offset88 = cpu_to_be32(
+				    (be32_to_cpu(req_msg->primary_offset88) &
+				     0xFFFFFFC0) | (rate & 0x3F));
+}
+
+static inline u8 cm_req_get_primary_sl(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->primary_offset94 >> 4);
+}
+
+static inline void cm_req_set_primary_sl(struct cm_req_msg *req_msg, u8 sl)
+{
+	req_msg->primary_offset94 = (u8) ((req_msg->primary_offset94 & 0x0F) |
+					  (sl << 4));
+}
+
+static inline u8 cm_req_get_primary_subnet_local(struct cm_req_msg *req_msg)
+{
+	return (u8) ((req_msg->primary_offset94 & 0x08) >> 3);
+}
+
+static inline void cm_req_set_primary_subnet_local(struct cm_req_msg *req_msg,
+						   u8 subnet_local)
+{
+	req_msg->primary_offset94 = (u8) ((req_msg->primary_offset94 & 0xF7) |
+					  ((subnet_local & 0x1) << 3));
+}
+
+static inline u8 cm_req_get_primary_local_ack_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->primary_offset95 >> 3);
+}
+
+static inline void cm_req_set_primary_local_ack_timeout(struct cm_req_msg *req_msg,
+							u8 local_ack_timeout)
+{
+	req_msg->primary_offset95 = (u8) ((req_msg->primary_offset95 & 0x07) |
+					  (local_ack_timeout << 3));
+}
+
+static inline u32 cm_req_get_alt_flow_label(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32((be32_to_cpu(req_msg->alt_offset132) >> 12));
+}
+
+static inline void cm_req_set_alt_flow_label(struct cm_req_msg *req_msg,
+					     u32 flow_label)
+{
+	req_msg->alt_offset132 = cpu_to_be32(
+				 (be32_to_cpu(req_msg->alt_offset132) &
+				  0x00000FFF) |
+				  (be32_to_cpu(flow_label) << 12));
+}
+
+static inline u8 cm_req_get_alt_packet_rate(struct cm_req_msg *req_msg)
+{
+	return (u8) (be32_to_cpu(req_msg->alt_offset132) & 0x3F);
+}
+
+static inline void cm_req_set_alt_packet_rate(struct cm_req_msg *req_msg,
+					      u8 rate)
+{
+	req_msg->alt_offset132 = cpu_to_be32(
+				 (be32_to_cpu(req_msg->alt_offset132) &
+				  0xFFFFFFC0) | (rate & 0x3F));
+}
+
+static inline u8 cm_req_get_alt_sl(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->alt_offset138 >> 4);
+}
+
+static inline void cm_req_set_alt_sl(struct cm_req_msg *req_msg, u8 sl)
+{
+	req_msg->alt_offset138 = (u8) ((req_msg->alt_offset138 & 0x0F) |
+				       (sl << 4));
+}
+
+static inline u8 cm_req_get_alt_subnet_local(struct cm_req_msg *req_msg)
+{
+	return (u8) ((req_msg->alt_offset138 & 0x08) >> 3);
+}
+
+static inline void cm_req_set_alt_subnet_local(struct cm_req_msg *req_msg,
+					       u8 subnet_local)
+{
+	req_msg->alt_offset138 = (u8) ((req_msg->alt_offset138 & 0xF7) |
+				       ((subnet_local & 0x1) << 3));
+}
+
+static inline u8 cm_req_get_alt_local_ack_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->alt_offset139 >> 3);
+}
+
+static inline void cm_req_set_alt_local_ack_timeout(struct cm_req_msg *req_msg,
+						    u8 local_ack_timeout)
+{
+	req_msg->alt_offset139 = (u8) ((req_msg->alt_offset139 & 0x07) |
+				       (local_ack_timeout << 3));
+}
+
+/* Message REJected or MRAed */
+enum cm_msg_response {
+	CM_MSG_RESPONSE_REQ = 0x0,
+	CM_MSG_RESPONSE_REP = 0x1,
+	CM_MSG_RESPONSE_OTHER = 0x2
+};
+
+ struct cm_mra_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	/* message MRAed:2, rsvd:6 */
+	u8 offset8;
+	/* service timeout:5, rsvd:3 */
+	u8 offset9;
+
+	u8 private_data[IB_CM_MRA_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u8 cm_mra_get_msg_mraed(struct cm_mra_msg *mra_msg)
+{
+	return (u8) (mra_msg->offset8 >> 6);
+}
+
+static inline void cm_mra_set_msg_mraed(struct cm_mra_msg *mra_msg, u8 msg)
+{
+	mra_msg->offset8 = (u8) ((mra_msg->offset8 & 0x3F) | (msg << 6));
+}
+
+static inline u8 cm_mra_get_service_timeout(struct cm_mra_msg *mra_msg)
+{
+	return (u8) (mra_msg->offset9 >> 3);
+}
+
+static inline void cm_mra_set_service_timeout(struct cm_mra_msg *mra_msg,
+					      u8 service_timeout)
+{
+	mra_msg->offset9 = (u8) ((mra_msg->offset9 & 0x07) |
+				 (service_timeout << 3));
+}
+
+struct cm_rej_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	/* message REJected:2, rsvd:6 */
+	u8 offset8;
+	/* reject info length:7, rsvd:1. */
+	u8 offset9;
+	u16 reason;
+	u8 ari[IB_CM_REJ_ARI_LENGTH];
+
+	u8 private_data[IB_CM_REJ_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u8 cm_rej_get_msg_rejected(struct cm_rej_msg *rej_msg)
+{
+	return (u8) (rej_msg->offset8 >> 6);
+}
+
+static inline void cm_rej_set_msg_rejected(struct cm_rej_msg *rej_msg, u8 msg)
+{
+	rej_msg->offset8 = (u8) ((rej_msg->offset8 & 0x3F) | (msg << 6));
+}
+
+static inline u8 cm_rej_get_reject_info_len(struct cm_rej_msg *rej_msg)
+{
+	return (u8) (rej_msg->offset9 >> 1);
+}
+
+static inline void cm_rej_set_reject_info_len(struct cm_rej_msg *rej_msg,
+					      u8 len)
+{
+	rej_msg->offset9 = (u8) ((rej_msg->offset9 & 0x1) | (len << 1));
+}
+
+struct cm_rep_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	u32 local_qkey;
+	/* local QPN:24, rsvd:8 */
+	u32 offset12;
+	/* local EECN:24, rsvd:8 */
+	u32 offset16;
+	/* starting PSN:24 rsvd:8 */
+	u32 offset20;
+	u8 resp_resources;
+	u8 initiator_depth;
+	/* target ACK delay:5, failover accepted:2, end-to-end flow control:1 */
+	u8 offset26;
+	/* RNR retry count:3, SRQ:1, rsvd:5 */
+	u8 offset27;
+	u64 local_ca_guid;
+
+	u8 private_data[IB_CM_REP_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u32 cm_rep_get_local_qpn(struct cm_rep_msg *rep_msg)
+{
+	return cpu_to_be32(be32_to_cpu(rep_msg->offset12) >> 8);
+}
+
+static inline void cm_rep_set_local_qpn(struct cm_rep_msg *rep_msg, u32 qpn)
+{
+	rep_msg->offset12 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+			    (be32_to_cpu(rep_msg->offset12) & 0x000000FF));
+}
+
+static inline u32 cm_rep_get_starting_psn(struct cm_rep_msg *rep_msg)
+{
+	return cpu_to_be32(be32_to_cpu(rep_msg->offset20) >> 8);
+}
+
+static inline void cm_rep_set_starting_psn(struct cm_rep_msg *rep_msg,
+					   u32 starting_psn)
+{
+	rep_msg->offset20 = cpu_to_be32((be32_to_cpu(starting_psn) << 8) |
+			    (be32_to_cpu(rep_msg->offset20) & 0x000000FF));
+}
+
+static inline u8 cm_rep_get_target_ack_delay(struct cm_rep_msg *rep_msg)
+{
+	return (u8) (rep_msg->offset26 >> 3);
+}
+
+static inline void cm_rep_set_target_ack_delay(struct cm_rep_msg *rep_msg,
+					       u8 target_ack_delay)
+{
+	rep_msg->offset26 = (u8) ((rep_msg->offset26 & 0x07) |
+				  (target_ack_delay << 3));
+}
+
+static inline u8 cm_rep_get_failover(struct cm_rep_msg *rep_msg)
+{
+	return (u8) ((rep_msg->offset26 & 0x06) >> 1);
+}
+
+static inline void cm_rep_set_failover(struct cm_rep_msg *rep_msg, u8 failover)
+{
+	rep_msg->offset26 = (u8) ((rep_msg->offset26 & 0xF9) |
+				  ((failover & 0x3) << 1));
+}
+
+static inline u8 cm_rep_get_flow_ctrl(struct cm_rep_msg *rep_msg)
+{
+	return (u8) (rep_msg->offset26 & 0x01);
+}
+
+static inline void cm_rep_set_flow_ctrl(struct cm_rep_msg *rep_msg,
+					    u8 flow_ctrl)
+{
+	rep_msg->offset26 = (u8) ((rep_msg->offset26 & 0xFE) |
+				  (flow_ctrl & 0x1));
+}
+
+static inline u8 cm_rep_get_rnr_retry_count(struct cm_rep_msg *rep_msg)
+{
+	return (u8) (rep_msg->offset27 >> 5);
+}
+
+static inline void cm_rep_set_rnr_retry_count(struct cm_rep_msg *rep_msg,
+					      u8 rnr_retry_count)
+{
+	rep_msg->offset27 = (u8) ((rep_msg->offset27 & 0x1F) |
+				  (rnr_retry_count << 5));
+}
+
+static inline u8 cm_rep_get_srq(struct cm_rep_msg *rep_msg)
+{
+	return (u8) ((rep_msg->offset27 >> 4) & 0x1);
+}
+
+static inline void cm_rep_set_srq(struct cm_rep_msg *rep_msg, u8 srq)
+{
+	rep_msg->offset27 = (u8) ((rep_msg->offset27 & 0xEF) |
+				  ((srq & 0x1) << 4));
+}
+
+struct cm_rtu_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u8 private_data[IB_CM_RTU_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+struct cm_dreq_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	/* remote QPN/EECN:24, rsvd:8 */
+	u32 offset8;
+
+	u8 private_data[IB_CM_DREQ_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u32 cm_dreq_get_remote_qpn(struct cm_dreq_msg *dreq_msg)
+{
+	return cpu_to_be32(be32_to_cpu(dreq_msg->offset8) >> 8);
+}
+
+static inline void cm_dreq_set_remote_qpn(struct cm_dreq_msg *dreq_msg, u32 qpn)
+{
+	dreq_msg->offset8 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+			    (be32_to_cpu(dreq_msg->offset8) & 0x000000FF));
+}
+
+struct cm_drep_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u8 private_data[IB_CM_DREP_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+struct cm_lap_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u32 rsvd8;
+	/* remote QPN/EECN:24, remote CM response timeout:5, rsvd:3 */
+	u32 offset12;
+	u32 rsvd16;
+
+	u16 alt_local_lid;
+	u16 alt_remote_lid;
+	union ib_gid alt_local_gid;
+	union ib_gid alt_remote_gid;
+	/* flow label:20, rsvd:4, traffic class:8 */
+	u32 offset56;
+	u8 alt_hop_limit;
+	/* rsvd:2, packet rate:6 */
+	uint8_t offset61;
+	/* SL:4, subnet local:1, rsvd:3 */
+	uint8_t offset62;
+	/* local ACK timeout:5, rsvd:3 */
+	uint8_t offset63;
+
+	u8 private_data[IB_CM_LAP_PRIVATE_DATA_SIZE];
+} __attribute__  ((packed));
+
+static inline u32 cm_lap_get_remote_qpn(struct cm_lap_msg *lap_msg)
+{
+	return cpu_to_be32(be32_to_cpu(lap_msg->offset12) >> 8);
+}
+
+static inline void cm_lap_set_remote_qpn(struct cm_lap_msg *lap_msg, u32 qpn)
+{
+	lap_msg->offset12 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+					 (be32_to_cpu(lap_msg->offset12) &
+					  0x000000FF));
+}
+
+static inline u8 cm_lap_get_remote_resp_timeout(struct cm_lap_msg *lap_msg)
+{
+	return (u8) ((be32_to_cpu(lap_msg->offset12) & 0xF8) >> 3);
+}
+
+static inline void cm_lap_set_remote_resp_timeout(struct cm_lap_msg *lap_msg,
+						  u8 resp_timeout)
+{
+	lap_msg->offset12 = cpu_to_be32((resp_timeout << 3) |
+					 (be32_to_cpu(lap_msg->offset12) &
+					  0xFFFFFF07));
+}
+
+static inline u32 cm_lap_get_flow_label(struct cm_lap_msg *lap_msg)
+{
+	return be32_to_cpu(lap_msg->offset56) >> 12;
+}
+
+static inline void cm_lap_set_flow_label(struct cm_lap_msg *lap_msg,
+					 u32 flow_label)
+{
+	lap_msg->offset56 = cpu_to_be32((flow_label << 12) |
+					 (be32_to_cpu(lap_msg->offset56) &
+					  0x00000FFF));
+}
+
+static inline u8 cm_lap_get_traffic_class(struct cm_lap_msg *lap_msg)
+{
+	return (u8) be32_to_cpu(lap_msg->offset56);
+}
+
+static inline void cm_lap_set_traffic_class(struct cm_lap_msg *lap_msg,
+					    u8 traffic_class)
+{
+	lap_msg->offset56 = cpu_to_be32(traffic_class |
+					 (be32_to_cpu(lap_msg->offset56) &
+					  0xFFFFFF00));
+}
+
+static inline u8 cm_lap_get_packet_rate(struct cm_lap_msg *lap_msg)
+{
+	return lap_msg->offset61 & 0x3F;
+}
+
+static inline void cm_lap_set_packet_rate(struct cm_lap_msg *lap_msg,
+					  u8 packet_rate)
+{
+	lap_msg->offset61 = (packet_rate & 0x3F) | (lap_msg->offset61 & 0xC0);
+}
+
+static inline u8 cm_lap_get_sl(struct cm_lap_msg *lap_msg)
+{
+	return lap_msg->offset62 >> 4;
+}
+
+static inline void cm_lap_set_sl(struct cm_lap_msg *lap_msg, u8 sl)
+{
+	lap_msg->offset62 = (sl << 4) | (lap_msg->offset62 & 0x0F);
+}
+
+static inline u8 cm_lap_get_subnet_local(struct cm_lap_msg *lap_msg)
+{
+	return (lap_msg->offset62 >> 3) & 0x1;
+}
+
+static inline void cm_lap_set_subnet_local(struct cm_lap_msg *lap_msg,
+					   u8 subnet_local)
+{
+	lap_msg->offset62 = ((subnet_local & 0x1) << 3) |
+			     (lap_msg->offset61 & 0xF7);
+}
+static inline u8 cm_lap_get_local_ack_timeout(struct cm_lap_msg *lap_msg)
+{
+	return lap_msg->offset63 >> 3;
+}
+
+static inline void cm_lap_set_local_ack_timeout(struct cm_lap_msg *lap_msg,
+						u8 local_ack_timeout)
+{
+	lap_msg->offset63 = (local_ack_timeout << 3) |
+			    (lap_msg->offset63 & 0x07);
+}
+
+struct cm_apr_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u8 info_length;
+	u8 ap_status;
+	u8 info[IB_CM_APR_INFO_LENGTH];
+
+	u8 private_data[IB_CM_APR_PRIVATE_DATA_SIZE];
+} __attribute__ ((packed));
+
+struct cm_sidr_req_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 request_id;
+	u16 pkey;
+	u16 rsvd;
+	u64 service_id;
+
+	u8 private_data[IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE];
+} __attribute__ ((packed));
+
+struct cm_sidr_rep_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 request_id;
+	u8 status;
+	u8 info_length;
+	u16 rsvd;
+	/* QPN:24, rsvd:8 */
+	u32 offset8;
+	u64 service_id;
+	u32 qkey;
+	u8 info[IB_CM_SIDR_REP_INFO_LENGTH];
+
+	u8 private_data[IB_CM_SIDR_REP_PRIVATE_DATA_SIZE];
+} __attribute__ ((packed));
+
+static inline u32 cm_sidr_rep_get_qpn(struct cm_sidr_rep_msg *sidr_rep_msg)
+{
+	return cpu_to_be32(be32_to_cpu(sidr_rep_msg->offset8) >> 8);
+}
+
+static inline void cm_sidr_rep_set_qpn(struct cm_sidr_rep_msg *sidr_rep_msg,
+				       u32 qpn)
+{
+	sidr_rep_msg->offset8 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+					(be32_to_cpu(sidr_rep_msg->offset8) &
+					 0x000000FF));
+}
+
+#endif /* CM_MSGS_H */
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband21/core/Makefile linux-2.6.13-rc2-mm1/drivers/infiniband22/core/Makefile
-- linux-2.6.13-rc2-mm1/drivers/infiniband21/core/Makefile	2005-07-10 16:32:15.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband22/core/Makefile	2005-07-10 16:50:04.000000000 -0400
@@ -1,6 +1,7 @@
 EXTRA_CFLAGS += -Idrivers/infiniband/include
 
-obj-$(CONFIG_INFINIBAND) +=		ib_core.o ib_mad.o ib_sa.o ib_umad.o
+obj-$(CONFIG_INFINIBAND) +=		ib_core.o ib_mad.o ib_sa.o \
+					ib_cm.o ib_umad.o
 obj-$(CONFIG_INFINIBAND_USER_VERBS) +=	ib_uverbs.o
 
 ib_core-y :=			packer.o ud_header.o verbs.o sysfs.o \
@@ -10,6 +11,8 @@ ib_mad-y :=			mad.o smi.o agent.o mad_rm
 
 ib_sa-y :=			sa_query.o
 
+ib_cm-y :=			cm.o
+
 ib_umad-y :=			user_mad.o
 
 ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_mem.o




From halr at voltaire.com  Mon Jul 11 07:52:58 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 10:52:58 -0400
Subject: [openib-general] [PATCH 23/27] User MAD ABI changes to support RMPP
Message-ID: <1121089249.4389.4551.camel@hal.voltaire.com>

User MAD ABI changes to support RMPP

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 22/27.

-- 
 ib_user_mad.h |   28 ++++++++++++++++++++++--
 1 files changed, 22 insertions(+), 6 deletions(-)
 user_mad.txt |   51 +++++++++++++++++++++++++++++++++++++++--
 1 files changed, 39 insertions(+), 12 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband22/include/ib_user_mad.h linux-2.6.13-rc2-mm1/drivers/infiniband23/include/ib_user_mad.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband22/include/ib_user_mad.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband23/include/ib_user_mad.h	2005-07-10 16:55:43.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_user_mad.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: ib_user_mad.h 2814 2005-07-06 19:14:09Z halr $
  */
 
 #ifndef IB_USER_MAD_H
@@ -42,7 +43,7 @@
  * Increment this value if any changes that break userspace ABI
  * compatibility are made.
  */
-#define IB_USER_MAD_ABI_VERSION	2
+#define IB_USER_MAD_ABI_VERSION	5
 
 /*
  * Make sure that all structs defined in this file remain laid out so
@@ -51,13 +52,13 @@
  */
 
 /**
- * ib_user_mad - MAD packet
- * @data - Contents of MAD
+ * ib_user_mad_hdr - MAD packet header
  * @id - ID of agent MAD received with/to be sent with
  * @status - 0 on successful receive, ETIMEDOUT if no response
  *   received (transaction ID in data[] will be set to TID of original
  *   request) (ignored on send)
  * @timeout_ms - Milliseconds to wait for response (unset on receive)
+ * @retries - Number of automatic retries to attempt
  * @qpn - Remote QP number received from/to be sent to
  * @qkey - Remote Q_Key to be sent with (unset on receive)
  * @lid - Remote lid received from/to be sent to
@@ -72,11 +73,12 @@
  *
  * All multi-byte quantities are stored in network (big endian) byte order.
  */
-struct ib_user_mad {
-	__u8	data[256];
+struct ib_user_mad_hdr {
 	__u32	id;
 	__u32	status;
 	__u32	timeout_ms;
+	__u32	retries;
+	__u32	length;
 	__u32	qpn;
 	__u32   qkey;
 	__u16	lid;
@@ -91,6 +93,17 @@ struct ib_user_mad {
 };
 
 /**
+ * ib_user_mad - MAD packet
+ * @hdr - MAD packet header
+ * @data - Contents of MAD
+ *
+ */
+struct ib_user_mad {
+	struct ib_user_mad_hdr hdr;
+	__u8	data[0];
+};
+
+/**
  * ib_user_mad_reg_req - MAD registration request
  * @id - Set by the kernel; used to identify agent in future requests.
  * @qpn - Queue pair number; must be 0 or 1.
@@ -103,6 +116,8 @@ struct ib_user_mad {
  *   management class to receive.
  * @oui: Indicates IEEE OUI when mgmt_class is a vendor class
  *   in the range from 0x30 to 0x4f. Otherwise not used.
+ * @rmpp_version: If set, indicates the RMPP version used.
+ * 
  */
 struct ib_user_mad_reg_req {
 	__u32	id;
@@ -111,6 +126,7 @@ struct ib_user_mad_reg_req {
 	__u8	mgmt_class;
 	__u8	mgmt_class_version;
 	__u8    oui[3];
+	__u8	rmpp_version;
 };
 
 #define IB_IOCTL_MAGIC		0x1b
diff -uprN linux-2.6.13-rc2-mm1/Documentation/infiniband/user_mad.txt linux-2.6.13-rc2-mm1/Documentation23/infiniband/user_mad.txt
-- linux-2.6.13-rc2-mm1/Documentation/infiniband/user_mad.txt	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1/Documentation23/infiniband/user_mad.txt	2005-07-01 14:27:50.000000000 -0400
@@ -28,13 +28,37 @@ Creating MAD agents
 
 Receiving MADs
 
-  MADs are received using read().  The buffer passed to read() must be
-  large enough to hold at least one struct ib_user_mad.  For example:
+  MADs are received using read().  The receive side now supports
+  RMPP. The buffer passed to read() must be at least one 
+  struct ib_user_mad + 256 bytes. For example:
+
+  If the buffer passed is not large enough to hold the received
+  MAD (RMPP), the errno is set to ENOSPC and the length of the
+  buffer needed is set in mad.length.
+
+  Example for normal MAD (non RMPP) reads:
+	struct ib_user_mad *mad;
+	mad = malloc(sizeof *mad + 256);
+	ret = read(fd, mad, sizeof *mad + 256);
+	if (ret != sizeof mad + 256) {
+		perror("read");
+		free(mad);
+	}
 
-	struct ib_user_mad mad;
-	ret = read(fd, &mad, sizeof mad);
-	if (ret != sizeof mad)
+  Example for RMPP reads:
+	struct ib_user_mad *mad;
+	mad = malloc(sizeof *mad + 256);
+	ret = read(fd, mad, sizeof *mad + 256);
+	if (ret == -ENOSPC)) {
+		length = mad.length;
+		free(mad);
+		mad = malloc(sizeof *mad + length);
+		ret = read(fd, mad, sizeof *mad + length);
+	}
+	if (ret < 0) {
 		perror("read");
+		free(mad);
+	}
 
   In addition to the actual MAD contents, the other struct ib_user_mad
   fields will be filled in with information on the received MAD.  For
@@ -50,18 +74,21 @@ Sending MADs
 
   MADs are sent using write().  The agent ID for sending should be
   filled into the id field of the MAD, the destination LID should be
-  filled into the lid field, and so on.  For example:
+  filled into the lid field, and so on.  The send side does support
+  RMPP so arbitrary length MAD can be sent. For example:
+
+	struct ib_user_mad *mad;
 
-	struct ib_user_mad mad;
+	mad = malloc(sizeof *mad + mad_length);
 
-	/* fill in mad.data */
+	/* fill in mad->data */
 
-	mad.id  = my_agent;	/* req.id from agent registration */
-	mad.lid = my_dest;	/* in network byte order... */
+	mad->hdr.id  = my_agent;	/* req.id from agent registration */
+	mad->hdr.lid = my_dest;		/* in network byte order... */
 	/* etc. */
 
-	ret = write(fd, &mad, sizeof mad);
-	if (ret != sizeof mad)
+	ret = write(fd, &mad, sizeof *mad + mad_length);
+	if (ret != sizeof *mad + mad_length)
 		perror("write");
 
 Setting IsSM Capability Bit




From halr at voltaire.com  Mon Jul 11 08:03:06 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:03:06 -0400
Subject: [openib-general] [PATCH 24/27] Implementation for RMPP support in
	user MAD
Message-ID: <1121089254.4389.4553.camel@hal.voltaire.com>

Implementation for RMPP support in user MAD

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 23/27.

-- 
 user_mad.c |  300 +++++++++++++++++++++++++++++++++++++++--
 1 files changed, 194 insertions(+), 106 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband23/core/user_mad.c linux-2.6.13-rc2-mm1/drivers/infiniband24/core/user_mad.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband23/core/user_mad.c	2005-07-09 17:14:46.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband24/core/user_mad.c	2005-07-10 16:56:39.000000000 -0400
@@ -1,5 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved. 
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +31,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: user_mad.c 1389 2004-12-27 22:56:47Z roland $
+ * $Id: user_mad.c 2814 2005-07-06 19:14:09Z halr $
  */
 
 #include <linux/module.h>
@@ -94,10 +96,12 @@ struct ib_umad_file {
 };
 
 struct ib_umad_packet {
-	struct ib_user_mad mad;
 	struct ib_ah      *ah;
+	struct ib_mad_send_buf *msg;
 	struct list_head   list;
+	int		   length;
 	DECLARE_PCI_UNMAP_ADDR(mapping)
+	struct ib_user_mad mad;
 };
 
 static const dev_t base_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE);
@@ -114,10 +118,10 @@ static int queue_packet(struct ib_umad_f
 	int ret = 1;
 
 	down_read(&file->agent_mutex);
-	for (packet->mad.id = 0;
-	     packet->mad.id < IB_UMAD_MAX_AGENTS;
-	     packet->mad.id++)
-		if (agent == file->agent[packet->mad.id]) {
+	for (packet->mad.hdr.id = 0;
+	     packet->mad.hdr.id < IB_UMAD_MAX_AGENTS;
+	     packet->mad.hdr.id++)
+		if (agent == file->agent[packet->mad.hdr.id]) {
 			spin_lock_irq(&file->recv_lock);
 			list_add_tail(&packet->list, &file->recv_list);
 			spin_unlock_irq(&file->recv_lock);
@@ -135,22 +139,30 @@ static void send_handler(struct ib_mad_a
 			 struct ib_mad_send_wc *send_wc)
 {
 	struct ib_umad_file *file = agent->context;
-	struct ib_umad_packet *packet =
+	struct ib_umad_packet *timeout, *packet =
 		(void *) (unsigned long) send_wc->wr_id;
 
-	dma_unmap_single(agent->device->dma_device,
-			 pci_unmap_addr(packet, mapping),
-			 sizeof packet->mad.data,
-			 DMA_TO_DEVICE);
-	ib_destroy_ah(packet->ah);
+	ib_destroy_ah(packet->msg->send_wr.wr.ud.ah);
+	ib_free_send_mad(packet->msg);
 
 	if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) {
-		packet->mad.status = ETIMEDOUT;
+		timeout = kmalloc(sizeof *timeout + sizeof (struct ib_mad_hdr),
+				  GFP_KERNEL);
+		if (!timeout)
+			goto out;
+
+		memset(timeout, 0, sizeof *timeout + sizeof (struct ib_mad_hdr));
+
+		timeout->length = sizeof (struct ib_mad_hdr);
+		timeout->mad.hdr.id = packet->mad.hdr.id;
+		timeout->mad.hdr.status = ETIMEDOUT;
+		memcpy(timeout->mad.data, packet->mad.data,
+		       sizeof (struct ib_mad_hdr));
 
-		if (!queue_packet(file, agent, packet))
-			return;
+		if (!queue_packet(file, agent, timeout))
+				return;
 	}
-
+out:
 	kfree(packet);
 }
 
@@ -159,30 +171,35 @@ static void recv_handler(struct ib_mad_a
 {
 	struct ib_umad_file *file = agent->context;
 	struct ib_umad_packet *packet;
+	int length;
 
 	if (mad_recv_wc->wc->status != IB_WC_SUCCESS)
 		goto out;
 
-	packet = kmalloc(sizeof *packet, GFP_KERNEL);
+	length = mad_recv_wc->mad_len; 
+	packet = kmalloc(sizeof *packet + length, GFP_KERNEL);
 	if (!packet)
 		goto out;
 
-	memset(packet, 0, sizeof *packet);
+	memset(packet, 0, sizeof *packet + length);
+	packet->length = length;
+
+	ib_coalesce_recv_mad(mad_recv_wc, packet->mad.data);
 
-	memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, sizeof packet->mad.data);
-	packet->mad.status        = 0;
-	packet->mad.qpn 	  = cpu_to_be32(mad_recv_wc->wc->src_qp);
-	packet->mad.lid 	  = cpu_to_be16(mad_recv_wc->wc->slid);
-	packet->mad.sl  	  = mad_recv_wc->wc->sl;
-	packet->mad.path_bits 	  = mad_recv_wc->wc->dlid_path_bits;
-	packet->mad.grh_present   = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH);
-	if (packet->mad.grh_present) {
+	packet->mad.hdr.status    = 0;
+	packet->mad.hdr.length    = length + sizeof (struct ib_user_mad);
+	packet->mad.hdr.qpn 	  = cpu_to_be32(mad_recv_wc->wc->src_qp);
+	packet->mad.hdr.lid 	  = cpu_to_be16(mad_recv_wc->wc->slid);
+	packet->mad.hdr.sl  	  = mad_recv_wc->wc->sl;
+	packet->mad.hdr.path_bits = mad_recv_wc->wc->dlid_path_bits;
+	packet->mad.hdr.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH);
+	if (packet->mad.hdr.grh_present) {
 		/* XXX parse GRH */
-		packet->mad.gid_index 	  = 0;
-		packet->mad.hop_limit 	  = 0;
-		packet->mad.traffic_class = 0;
-		memset(packet->mad.gid, 0, 16);
-		packet->mad.flow_label 	  = 0;
+		packet->mad.hdr.gid_index 	= 0;
+		packet->mad.hdr.hop_limit 	= 0;
+		packet->mad.hdr.traffic_class	= 0;
+		memset(packet->mad.hdr.gid, 0, 16);
+		packet->mad.hdr.flow_label	= 0;
 	}
 
 	if (queue_packet(file, agent, packet))
@@ -199,7 +216,7 @@ static ssize_t ib_umad_read(struct file 
 	struct ib_umad_packet *packet;
 	ssize_t ret;
 
-	if (count < sizeof (struct ib_user_mad))
+	if (count < sizeof (struct ib_user_mad) + sizeof (struct ib_mad))
 		return -EINVAL;
 
 	spin_lock_irq(&file->recv_lock);
@@ -222,12 +239,25 @@ static ssize_t ib_umad_read(struct file 
 
 	spin_unlock_irq(&file->recv_lock);
 
-	if (copy_to_user(buf, &packet->mad, sizeof packet->mad))
+	if (count < packet->length + sizeof (struct ib_user_mad)) {
+		/* Return length needed (and first RMPP segment) if too small */
+		if (copy_to_user(buf, &packet->mad,
+				 sizeof (struct ib_user_mad) + sizeof (struct ib_mad)))
+			ret = -EFAULT;
+		else
+			ret = -ENOSPC;
+	} else if (copy_to_user(buf, &packet->mad,
+			      packet->length + sizeof (struct ib_user_mad)))
 		ret = -EFAULT;
 	else
-		ret = sizeof packet->mad;
-
-	kfree(packet);
+		ret = packet->length + sizeof (struct ib_user_mad);
+	if (ret < 0) {
+		/* Requeue packet */
+		spin_lock_irq(&file->recv_lock);
+		list_add(&packet->list, &file->recv_list);
+		spin_unlock_irq(&file->recv_lock);
+	} else
+		kfree(packet);
 	return ret;
 }
 
@@ -238,106 +268,163 @@ static ssize_t ib_umad_write(struct file
 	struct ib_umad_packet *packet;
 	struct ib_mad_agent *agent;
 	struct ib_ah_attr ah_attr;
-	struct ib_sge      gather_list;
-	struct ib_send_wr *bad_wr, wr = {
-		.opcode      = IB_WR_SEND,
-		.sg_list     = &gather_list,
-		.num_sge     = 1,
-		.send_flags  = IB_SEND_SIGNALED,
-	};
+	struct ib_send_wr *bad_wr;
+	struct ib_rmpp_mad *rmpp_mad;
 	u8 method;
 	u64 *tid;
-	int ret;
+	int ret, length, hdr_len, data_len, rmpp_hdr_size;
+	int rmpp_active = 0;
 
 	if (count < sizeof (struct ib_user_mad))
 		return -EINVAL;
 
-	packet = kmalloc(sizeof *packet, GFP_KERNEL);
+	length = count - sizeof (struct ib_user_mad);
+	packet = kmalloc(sizeof *packet + sizeof(struct ib_mad_hdr) +
+			 sizeof(struct ib_rmpp_hdr), GFP_KERNEL);
 	if (!packet)
 		return -ENOMEM;
 
-	if (copy_from_user(&packet->mad, buf, sizeof packet->mad)) {
-		kfree(packet);
-		return -EFAULT;
+	if (copy_from_user(&packet->mad, buf,
+			    sizeof (struct ib_user_mad) +
+			    sizeof(struct ib_mad_hdr) +
+			    sizeof(struct ib_rmpp_hdr))) {
+		ret = -EFAULT;
+		goto err;
 	}
 
-	if (packet->mad.id < 0 || packet->mad.id >= IB_UMAD_MAX_AGENTS) {
+	if (packet->mad.hdr.id < 0 ||
+	    packet->mad.hdr.id >= IB_UMAD_MAX_AGENTS) {
 		ret = -EINVAL;
 		goto err;
 	}
 
+	packet->length = length;
+
 	down_read(&file->agent_mutex);
 
-	agent = file->agent[packet->mad.id];
+	agent = file->agent[packet->mad.hdr.id];
 	if (!agent) {
 		ret = -EINVAL;
 		goto err_up;
 	}
 
+	memset(&ah_attr, 0, sizeof ah_attr);
+	ah_attr.dlid          = be16_to_cpu(packet->mad.hdr.lid);
+	ah_attr.sl            = packet->mad.hdr.sl;
+	ah_attr.src_path_bits = packet->mad.hdr.path_bits;
+	ah_attr.port_num      = file->port->port_num;
+	if (packet->mad.hdr.grh_present) {
+		ah_attr.ah_flags = IB_AH_GRH;
+		memcpy(ah_attr.grh.dgid.raw, packet->mad.hdr.gid, 16);
+		ah_attr.grh.flow_label 	   = packet->mad.hdr.flow_label;
+		ah_attr.grh.hop_limit  	   = packet->mad.hdr.hop_limit;
+		ah_attr.grh.traffic_class  = packet->mad.hdr.traffic_class;
+	}
+
+	packet->ah = ib_create_ah(agent->qp->pd, &ah_attr);
+	if (IS_ERR(packet->ah)) {
+		ret = PTR_ERR(packet->ah);
+		goto err_up;
+	}
+
+	rmpp_mad = (struct ib_rmpp_mad *) packet->mad.data;
+	if (ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_ACTIVE) {
+		/* RMPP active */
+		if (!agent->rmpp_version) {
+			ret = -EINVAL;
+			goto err_ah;
+		}
+		/* Validate that management class can support RMPP */
+		if (rmpp_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_ADM) {
+			hdr_len = offsetof(struct ib_sa_mad, data);
+			data_len = length;
+		} else if ((rmpp_mad->mad_hdr.mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) &&
+			    (rmpp_mad->mad_hdr.mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) {
+				hdr_len = offsetof(struct ib_vendor_mad, data);
+				data_len = length - hdr_len;
+		} else {
+			ret = -EINVAL;
+			goto err_ah;
+		}
+		rmpp_active = 1;
+	} else {
+		if (length > sizeof(struct ib_mad)) {
+			ret = -EINVAL;
+			goto err_ah;
+		}
+		hdr_len = offsetof(struct ib_mad, data);
+		data_len = length - hdr_len;
+	}
+	
+	packet->msg = ib_create_send_mad(agent,
+					 be32_to_cpu(packet->mad.hdr.qpn),
+					 0, packet->ah, rmpp_active,
+					 hdr_len, data_len,
+					 GFP_KERNEL);
+	if (IS_ERR(packet->msg)) {
+		ret = PTR_ERR(packet->msg);
+		goto err_ah;
+	}
+
+	packet->msg->send_wr.wr.ud.timeout_ms  = packet->mad.hdr.timeout_ms;
+	packet->msg->send_wr.wr.ud.retries = packet->mad.hdr.retries;
+
+	/* Override send WR WRID initialized in ib_create_send_mad */
+	packet->msg->send_wr.wr_id = (unsigned long) packet;
+
+	if (!rmpp_active) {
+		/* Copy message from user into send buffer */
+		if (copy_from_user(packet->msg->mad,
+				   buf + sizeof(struct ib_user_mad), length)) {
+			ret = -EFAULT;
+			goto err_msg;
+		}
+	} else {
+		rmpp_hdr_size = sizeof(struct ib_mad_hdr) +
+				sizeof(struct ib_rmpp_hdr);
+
+		/* Only copy MAD headers (RMPP header in place) */
+		memcpy(packet->msg->mad, packet->mad.data,
+		       sizeof(struct ib_mad_hdr));
+
+		/* Now, copy rest of message from user into send buffer */
+		if (copy_from_user(((struct ib_rmpp_mad *) packet->msg->mad)->data,
+				   buf + sizeof (struct ib_user_mad) + rmpp_hdr_size,
+				   length - rmpp_hdr_size)) {
+			ret = -EFAULT;
+			goto err_msg;
+		}
+	}
+
 	/*
 	 * If userspace is generating a request that will generate a
 	 * response, we need to make sure the high-order part of the
 	 * transaction ID matches the agent being used to send the
 	 * MAD.
 	 */
-	method = ((struct ib_mad_hdr *) packet->mad.data)->method;
+	method = packet->msg->mad->mad_hdr.method;
 
 	if (!(method & IB_MGMT_METHOD_RESP)       &&
 	    method != IB_MGMT_METHOD_TRAP_REPRESS &&
 	    method != IB_MGMT_METHOD_SEND) {
-		tid = &((struct ib_mad_hdr *) packet->mad.data)->tid;
+		tid = &packet->msg->mad->mad_hdr.tid;
 		*tid = cpu_to_be64(((u64) agent->hi_tid) << 32 |
 				   (be64_to_cpup(tid) & 0xffffffff));
 	}
 
-	memset(&ah_attr, 0, sizeof ah_attr);
-	ah_attr.dlid          = be16_to_cpu(packet->mad.lid);
-	ah_attr.sl            = packet->mad.sl;
-	ah_attr.src_path_bits = packet->mad.path_bits;
-	ah_attr.port_num      = file->port->port_num;
-	if (packet->mad.grh_present) {
-		ah_attr.ah_flags = IB_AH_GRH;
-		memcpy(ah_attr.grh.dgid.raw, packet->mad.gid, 16);
-		ah_attr.grh.flow_label 	   = packet->mad.flow_label;
-		ah_attr.grh.hop_limit  	   = packet->mad.hop_limit;
-		ah_attr.grh.traffic_class  = packet->mad.traffic_class;
-	}
-
-	packet->ah = ib_create_ah(agent->qp->pd, &ah_attr);
-	if (IS_ERR(packet->ah)) {
-		ret = PTR_ERR(packet->ah);
-		goto err_up;
-	}
-
-	gather_list.addr = dma_map_single(agent->device->dma_device,
-					  packet->mad.data,
-					  sizeof packet->mad.data,
-					  DMA_TO_DEVICE);
-	gather_list.length = sizeof packet->mad.data;
-	gather_list.lkey   = file->mr[packet->mad.id]->lkey;
-	pci_unmap_addr_set(packet, mapping, gather_list.addr);
-
-	wr.wr.ud.mad_hdr     = (struct ib_mad_hdr *) packet->mad.data;
-	wr.wr.ud.ah          = packet->ah;
-	wr.wr.ud.remote_qpn  = be32_to_cpu(packet->mad.qpn);
-	wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey);
-	wr.wr.ud.timeout_ms  = packet->mad.timeout_ms;
-	wr.wr.ud.retries     = 0;
+	ret = ib_post_send_mad(agent, &packet->msg->send_wr, &bad_wr);
+	if (ret)
+		goto err_msg;
 
-	wr.wr_id            = (unsigned long) packet;
+	up_read(&file->agent_mutex);
 
-	ret = ib_post_send_mad(agent, &wr, &bad_wr);
-	if (ret) {
-		dma_unmap_single(agent->device->dma_device,
-				 pci_unmap_addr(packet, mapping),
-				 sizeof packet->mad.data,
-				 DMA_TO_DEVICE);
-		goto err_up;
-	}
+	return sizeof (struct ib_user_mad_hdr) + packet->length;
 
-	up_read(&file->agent_mutex);
+err_msg:
+	ib_free_send_mad(packet->msg);
 
-	return sizeof packet->mad;
+err_ah:
+	ib_destroy_ah(packet->ah);
 
 err_up:
 	up_read(&file->agent_mutex);
@@ -400,7 +487,8 @@ found:
 	agent = ib_register_mad_agent(file->port->ib_dev, file->port->port_num,
 				      ureq.qpn ? IB_QPT_GSI : IB_QPT_SMI,
 				      ureq.mgmt_class ? &req : NULL,
-				      0, send_handler, recv_handler, file);
+				      ureq.rmpp_version,
+				      send_handler, recv_handler, file);
 	if (IS_ERR(agent)) {
 		ret = PTR_ERR(agent);
 		goto out;
@@ -461,8 +549,8 @@ out:
 	return ret;
 }
 
-static long ib_umad_ioctl(struct file *filp,
-			 unsigned int cmd, unsigned long arg)
+static long ib_umad_ioctl(struct file *filp, unsigned int cmd,
+			  unsigned long arg)
 {
 	switch (cmd) {
 	case IB_USER_MAD_REGISTER_AGENT:
@@ -518,14 +606,14 @@ static int ib_umad_close(struct inode *i
 }
 
 static struct file_operations umad_fops = {
-	.owner 	        = THIS_MODULE,
-	.read 	        = ib_umad_read,
-	.write 	        = ib_umad_write,
-	.poll 	        = ib_umad_poll,
+	.owner 	 	= THIS_MODULE,
+	.read 	 	= ib_umad_read,
+	.write 	 	= ib_umad_write,
+	.poll 	 	= ib_umad_poll,
 	.unlocked_ioctl = ib_umad_ioctl,
-	.compat_ioctl   = ib_umad_ioctl,
-	.open 	        = ib_umad_open,
-	.release        = ib_umad_close
+	.compat_ioctl 	= ib_umad_ioctl,
+	.open 	 	= ib_umad_open,
+	.release 	= ib_umad_close
 };
 
 static int ib_umad_sm_open(struct inode *inode, struct file *filp)




From halr at voltaire.com  Mon Jul 11 08:06:20 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:06:20 -0400
Subject: [openib-general] [PATCH 25/27] Add the header file for user space CM
Message-ID: <1121089260.4389.4555.camel@hal.voltaire.com>

Add the header file for user space CM. This file defines 
the ABI used by the CM for kernel/user communication.

Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

-- 
 ib_user_cm.h |  328 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 328 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband24/include/ib_user_cm.h linux-2.6.13-rc2-mm1/drivers/infiniband25/include/ib_user_cm.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband24/include/ib_user_cm.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband25/include/ib_user_cm.h	2005-06-10 08:16:25.000000000 -0400
@@ -0,0 +1,328 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ib_user_cm.h 2576 2005-06-09 17:00:30Z libor $
+ */
+
+#ifndef IB_USER_CM_H
+#define IB_USER_CM_H
+
+#include <linux/types.h>
+
+#define IB_USER_CM_ABI_VERSION 1
+
+enum {
+	IB_USER_CM_CMD_CREATE_ID,
+	IB_USER_CM_CMD_DESTROY_ID,
+	IB_USER_CM_CMD_ATTR_ID,
+
+	IB_USER_CM_CMD_LISTEN,
+	IB_USER_CM_CMD_ESTABLISH,
+	
+	IB_USER_CM_CMD_SEND_REQ,
+	IB_USER_CM_CMD_SEND_REP,
+	IB_USER_CM_CMD_SEND_RTU,
+	IB_USER_CM_CMD_SEND_DREQ,
+	IB_USER_CM_CMD_SEND_DREP,
+	IB_USER_CM_CMD_SEND_REJ,
+	IB_USER_CM_CMD_SEND_MRA,
+	IB_USER_CM_CMD_SEND_LAP,
+	IB_USER_CM_CMD_SEND_APR,
+	IB_USER_CM_CMD_SEND_SIDR_REQ,
+	IB_USER_CM_CMD_SEND_SIDR_REP,
+
+	IB_USER_CM_CMD_EVENT,
+};
+/*
+ * command ABI structures.
+ */
+struct ib_ucm_cmd_hdr {
+	__u32 cmd;
+	__u16 in;
+	__u16 out;
+};
+
+struct ib_ucm_create_id {
+	__u64 response;
+};
+
+struct ib_ucm_create_id_resp {
+	__u32 id;
+};
+
+struct ib_ucm_destroy_id {
+	__u32 id;
+};
+
+struct ib_ucm_attr_id {
+	__u64 response;
+	__u32 id;
+};
+
+struct ib_ucm_attr_id_resp {
+	__u64 service_id;
+	__u64 service_mask;
+	__u32 local_id;
+	__u32 remote_id;
+};
+
+struct ib_ucm_listen {
+	__u64 service_id;
+	__u64 service_mask;
+	__u32 id;
+};
+
+struct ib_ucm_establish {
+	__u32 id;
+};
+
+struct ib_ucm_private_data {
+	__u64 data;
+	__u32 id;
+	__u8  len;
+	__u8  reserved[3];
+};
+
+struct ib_ucm_path_rec {
+	__u8  dgid[16];
+	__u8  sgid[16];
+	__u16 dlid;
+	__u16 slid;
+	__u32 raw_traffic;
+	__u32 flow_label;
+	__u32 reversible;
+	__u32 mtu;
+	__u16 pkey;
+	__u8  hop_limit;
+	__u8  traffic_class;
+	__u8  numb_path;
+	__u8  sl;
+	__u8  mtu_selector;
+	__u8  rate_selector;
+	__u8  rate;
+	__u8  packet_life_time_selector;
+	__u8  packet_life_time;
+	__u8  preference;
+};
+
+struct ib_ucm_req {
+	__u32 id;
+	__u32 qpn;
+	__u32 qp_type;
+	__u32 psn;
+	__u64 sid;
+	__u64 data;
+	__u64 primary_path;
+	__u64 alternate_path;
+	__u8  len;
+	__u8  peer_to_peer;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  remote_cm_response_timeout;
+	__u8  flow_control;
+	__u8  local_cm_response_timeout;
+	__u8  retry_count;
+	__u8  rnr_retry_count;
+	__u8  max_cm_retries;
+	__u8  srq;
+	__u8  reserved[1];
+};
+
+struct ib_ucm_rep {
+	__u64 data;
+	__u32 id;
+	__u32 qpn;
+	__u32 psn;
+	__u8  len;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  target_ack_delay;
+	__u8  failover_accepted;
+	__u8  flow_control;
+	__u8  rnr_retry_count;
+	__u8  srq;
+};
+
+struct ib_ucm_info {
+	__u32 id;
+	__u32 status;
+	__u64 info;
+	__u64 data;
+	__u8  info_len;
+	__u8  data_len;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_mra {
+	__u64 data;
+	__u32 id;
+	__u8  len;
+	__u8  timeout;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_lap {
+	__u64 path;
+	__u64 data;
+	__u32 id;
+	__u8  len;
+	__u8  reserved[3];
+};
+
+struct ib_ucm_sidr_req {
+	__u32 id;
+	__u32 timeout;
+	__u64 sid;
+	__u64 data;
+	__u64 path;
+	__u16 pkey;
+	__u8  len;
+	__u8  max_cm_retries;
+};
+
+struct ib_ucm_sidr_rep {
+	__u32 id;
+	__u32 qpn;
+	__u32 qkey;
+	__u32 status;
+	__u64 info;
+	__u64 data;
+	__u8  info_len;
+	__u8  data_len;
+	__u8  reserved[2];
+};
+/*
+ * event notification ABI structures.
+ */
+struct ib_ucm_event_get {
+	__u64 response;
+	__u64 data;
+	__u64 info;
+	__u8  data_len;
+	__u8  info_len;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_req_event_resp {
+	__u32                  listen_id;
+	/* device */
+	/* port */
+	struct ib_ucm_path_rec primary_path;
+	struct ib_ucm_path_rec alternate_path;
+	__u64                  remote_ca_guid;
+	__u32                  remote_qkey;
+	__u32                  remote_qpn;
+	__u32                  qp_type;
+	__u32                  starting_psn;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  local_cm_response_timeout;
+	__u8  flow_control;
+	__u8  remote_cm_response_timeout;
+	__u8  retry_count;
+	__u8  rnr_retry_count;
+	__u8  srq;
+};
+
+struct ib_ucm_rep_event_resp {
+	__u64 remote_ca_guid;
+	__u32 remote_qkey;
+	__u32 remote_qpn;
+	__u32 starting_psn;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  target_ack_delay;
+	__u8  failover_accepted;
+	__u8  flow_control;
+	__u8  rnr_retry_count;
+	__u8  srq;
+	__u8  reserved[1];
+};
+
+struct ib_ucm_rej_event_resp {
+	__u32 reason;
+	/* ari in ib_ucm_event_get info field. */
+};
+
+struct ib_ucm_mra_event_resp {
+	__u8  timeout;
+	__u8  reserved[3];
+};
+
+struct ib_ucm_lap_event_resp {
+	struct ib_ucm_path_rec path;
+};
+
+struct ib_ucm_apr_event_resp {
+	__u32 status;
+	/* apr info in ib_ucm_event_get info field. */
+};
+
+struct ib_ucm_sidr_req_event_resp {
+	__u32 listen_id;
+	/* device */
+	/* port */
+	__u16 pkey;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_sidr_rep_event_resp {
+	__u32 status;
+	__u32 qkey;
+	__u32 qpn;
+	/* info in ib_ucm_event_get info field. */
+};
+
+#define IB_UCM_PRES_DATA      0x01
+#define IB_UCM_PRES_INFO      0x02
+#define IB_UCM_PRES_PRIMARY   0x04
+#define IB_UCM_PRES_ALTERNATE 0x08
+
+struct ib_ucm_event_resp {
+	__u32 id;
+	__u32 event;
+	__u32 present;
+	union {
+		struct ib_ucm_req_event_resp req_resp;
+		struct ib_ucm_rep_event_resp rep_resp;
+		struct ib_ucm_rej_event_resp rej_resp;
+		struct ib_ucm_mra_event_resp mra_resp;
+		struct ib_ucm_lap_event_resp lap_resp;
+		struct ib_ucm_apr_event_resp apr_resp;
+
+		struct ib_ucm_sidr_req_event_resp sidr_req_resp;
+		struct ib_ucm_sidr_rep_event_resp sidr_rep_resp;
+
+		__u32                             send_status;
+	} u;
+};
+
+#endif /* IB_USER_CM_H */




From halr at voltaire.com  Mon Jul 11 08:09:33 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:09:33 -0400
Subject: [openib-general] [PATCH 26/27] Add kernel portion of user CM
	implementation
Message-ID: <1121089270.4389.4557.camel@hal.voltaire.com>

Add kernel portion of user CM implementation 
Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 25/27.

-- 
 ucm.c | 1396 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 ucm.h |   89 ++++
 2 files changed, 1485 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband25/core/ucm.c linux-2.6.13-rc2-mm1/drivers/infiniband26/core/ucm.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband25/core/ucm.c	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband26/core/ucm.c	2005-07-10 16:58:39.000000000 -0400
@@ -0,0 +1,1396 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *	copyright notice, this list of conditions and the following
+ *	disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *	copyright notice, this list of conditions and the following
+ *	disclaimer in the documentation and/or other materials
+ *	provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ucm.c 2594 2005-06-13 19:46:02Z libor $
+ */
+#include <linux/init.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/poll.h>
+#include <linux/file.h>
+#include <linux/mount.h>
+#include <linux/cdev.h>
+
+#include <asm/uaccess.h>
+
+#include "ucm.h"
+	
+MODULE_AUTHOR("Libor Michalek");
+MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
+MODULE_LICENSE("Dual BSD/GPL");
+
+enum {
+	IB_UCM_MAJOR = 231,
+	IB_UCM_MINOR = 255
+};
+
+#define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
+
+static struct semaphore ctx_id_mutex;
+static struct idr       ctx_id_table;
+static int              ctx_id_rover = 0;
+
+static struct ib_ucm_context *ib_ucm_ctx_get(int id)
+{
+	struct ib_ucm_context *ctx;
+
+	down(&ctx_id_mutex);
+	ctx = idr_find(&ctx_id_table, id);
+	if (ctx)
+		ctx->ref++;
+	up(&ctx_id_mutex);
+
+	return ctx;
+}
+
+static void ib_ucm_ctx_put(struct ib_ucm_context *ctx)
+{
+	struct ib_ucm_event *uevent;
+
+	down(&ctx_id_mutex);
+
+	ctx->ref--;
+	if (!ctx->ref)
+		idr_remove(&ctx_id_table, ctx->id);
+
+	up(&ctx_id_mutex);
+
+	if (ctx->ref)
+		return;
+
+	down(&ctx->file->mutex);
+		
+	list_del(&ctx->file_list);
+	while (!list_empty(&ctx->events)) {
+
+		uevent = list_entry(ctx->events.next,
+				    struct ib_ucm_event, ctx_list);
+		list_del(&uevent->file_list);
+		list_del(&uevent->ctx_list);
+
+		/* clear incoming connections. */
+		if (uevent->cm_id)
+			ib_destroy_cm_id(uevent->cm_id);
+
+		kfree(uevent);
+	}
+
+	up(&ctx->file->mutex);
+			
+	printk(KERN_ERR "UCM: Destroyed CM ID <%d>\n", ctx->id);
+
+	ib_destroy_cm_id(ctx->cm_id);
+	kfree(ctx);
+}
+
+static struct ib_ucm_context *ib_ucm_ctx_alloc(struct ib_ucm_file *file)
+{
+	struct ib_ucm_context *ctx;
+	int result;
+
+	ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)
+		return NULL;
+
+	ctx->ref  = 1; /* user reference */
+	ctx->file = file;
+
+	INIT_LIST_HEAD(&ctx->events);
+	init_MUTEX(&ctx->mutex);
+
+	list_add_tail(&ctx->file_list, &file->ctxs);
+
+	ctx_id_rover = (ctx_id_rover + 1) & INT_MAX;
+retry:
+	result = idr_pre_get(&ctx_id_table, GFP_KERNEL);
+	if (!result)
+		goto error;
+	
+	down(&ctx_id_mutex);
+	result = idr_get_new_above(&ctx_id_table, ctx, ctx_id_rover, &ctx->id);
+	up(&ctx_id_mutex);
+
+	if (result == -EAGAIN)
+		goto retry;
+	if (result)
+		goto error;
+
+	printk(KERN_ERR "UCM: Allocated CM ID <%d>\n", ctx->id);
+
+	return ctx;
+error:
+	list_del(&ctx->file_list);
+	kfree(ctx);
+
+	return NULL;
+}
+/*
+ * Event portion of the API, handle CM events
+ * and allow event polling.
+ */
+static void ib_ucm_event_path_get(struct ib_ucm_path_rec *upath,
+				  struct ib_sa_path_rec	 *kpath)
+{
+	if (!kpath || !upath)
+		return;
+
+	memcpy(upath->dgid, kpath->dgid.raw, sizeof(union ib_gid));
+	memcpy(upath->sgid, kpath->sgid.raw, sizeof(union ib_gid));
+	
+	upath->dlid             = kpath->dlid;
+	upath->slid             = kpath->slid;
+	upath->raw_traffic      = kpath->raw_traffic;
+	upath->flow_label       = kpath->flow_label;
+	upath->hop_limit        = kpath->hop_limit;
+	upath->traffic_class    = kpath->traffic_class;
+	upath->reversible       = kpath->reversible;
+	upath->numb_path        = kpath->numb_path;
+	upath->pkey             = kpath->pkey;
+	upath->sl	        = kpath->sl;
+	upath->mtu_selector     = kpath->mtu_selector;
+	upath->mtu              = kpath->mtu;
+	upath->rate_selector    = kpath->rate_selector;
+	upath->rate             = kpath->rate;
+	upath->packet_life_time = kpath->packet_life_time;
+	upath->preference       = kpath->preference;
+
+	upath->packet_life_time_selector = 
+		kpath->packet_life_time_selector;
+}
+
+static void ib_ucm_event_req_get(struct ib_ucm_req_event_resp *ureq,
+				 struct ib_cm_req_event_param *kreq)
+{
+	ureq->listen_id = (long)kreq->listen_id->context;
+
+	ureq->remote_ca_guid             = kreq->remote_ca_guid;
+	ureq->remote_qkey                = kreq->remote_qkey;
+	ureq->remote_qpn                 = kreq->remote_qpn;
+	ureq->qp_type                    = kreq->qp_type;
+	ureq->starting_psn               = kreq->starting_psn;
+	ureq->responder_resources        = kreq->responder_resources;
+	ureq->initiator_depth            = kreq->initiator_depth;
+	ureq->local_cm_response_timeout  = kreq->local_cm_response_timeout;
+	ureq->flow_control               = kreq->flow_control;
+	ureq->remote_cm_response_timeout = kreq->remote_cm_response_timeout;
+	ureq->retry_count                = kreq->retry_count;
+	ureq->rnr_retry_count            = kreq->rnr_retry_count;
+	ureq->srq                        = kreq->srq;
+
+	ib_ucm_event_path_get(&ureq->primary_path, kreq->primary_path);
+	ib_ucm_event_path_get(&ureq->alternate_path, kreq->alternate_path);
+}
+
+static void ib_ucm_event_rep_get(struct ib_ucm_rep_event_resp *urep,
+				 struct ib_cm_rep_event_param *krep)
+{
+	urep->remote_ca_guid      = krep->remote_ca_guid;
+	urep->remote_qkey         = krep->remote_qkey;
+	urep->remote_qpn          = krep->remote_qpn;
+	urep->starting_psn        = krep->starting_psn;
+	urep->responder_resources = krep->responder_resources;
+	urep->initiator_depth     = krep->initiator_depth;
+	urep->target_ack_delay    = krep->target_ack_delay;
+	urep->failover_accepted   = krep->failover_accepted;
+	urep->flow_control        = krep->flow_control;
+	urep->rnr_retry_count     = krep->rnr_retry_count;
+	urep->srq                 = krep->srq;
+}
+
+static void ib_ucm_event_rej_get(struct ib_ucm_rej_event_resp *urej,
+				 struct ib_cm_rej_event_param *krej)
+{
+	urej->reason = krej->reason;
+}
+
+static void ib_ucm_event_mra_get(struct ib_ucm_mra_event_resp *umra,
+				 struct ib_cm_mra_event_param *kmra)
+{
+	umra->timeout = kmra->service_timeout;
+}
+
+static void ib_ucm_event_lap_get(struct ib_ucm_lap_event_resp *ulap,
+				 struct ib_cm_lap_event_param *klap)
+{
+	ib_ucm_event_path_get(&ulap->path, klap->alternate_path);
+}
+
+static void ib_ucm_event_apr_get(struct ib_ucm_apr_event_resp *uapr,
+				 struct ib_cm_apr_event_param *kapr)
+{
+	uapr->status = kapr->ap_status;
+}
+
+static void ib_ucm_event_sidr_req_get(struct ib_ucm_sidr_req_event_resp *ureq,
+				      struct ib_cm_sidr_req_event_param *kreq)
+{
+	ureq->listen_id = (long)kreq->listen_id->context;
+	ureq->pkey      = kreq->pkey;
+}
+
+static void ib_ucm_event_sidr_rep_get(struct ib_ucm_sidr_rep_event_resp *urep,
+				      struct ib_cm_sidr_rep_event_param *krep)
+{
+	urep->status = krep->status;
+	urep->qkey   = krep->qkey;
+	urep->qpn    = krep->qpn;
+};
+
+static int ib_ucm_event_process(struct ib_cm_event *evt,
+				struct ib_ucm_event *uvt)
+{
+	void *info = NULL;
+	int result;
+
+	switch (evt->event) {
+	case IB_CM_REQ_RECEIVED:
+		ib_ucm_event_req_get(&uvt->resp.u.req_resp, 
+				     &evt->param.req_rcvd);
+		uvt->data_len      = IB_CM_REQ_PRIVATE_DATA_SIZE;
+		uvt->resp.present |= (evt->param.req_rcvd.primary_path ? 
+				      IB_UCM_PRES_PRIMARY : 0);
+		uvt->resp.present |= (evt->param.req_rcvd.alternate_path ? 
+				      IB_UCM_PRES_ALTERNATE : 0);
+		break;
+	case IB_CM_REP_RECEIVED:
+		ib_ucm_event_rep_get(&uvt->resp.u.rep_resp, 
+				     &evt->param.rep_rcvd);
+		uvt->data_len = IB_CM_REP_PRIVATE_DATA_SIZE;
+
+		break;
+	case IB_CM_RTU_RECEIVED:
+		uvt->data_len = IB_CM_RTU_PRIVATE_DATA_SIZE;
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		uvt->data_len = IB_CM_DREQ_PRIVATE_DATA_SIZE;
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	case IB_CM_DREP_RECEIVED:
+		uvt->data_len = IB_CM_DREP_PRIVATE_DATA_SIZE;
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	case IB_CM_MRA_RECEIVED:
+		ib_ucm_event_mra_get(&uvt->resp.u.mra_resp,
+				     &evt->param.mra_rcvd);
+		uvt->data_len = IB_CM_MRA_PRIVATE_DATA_SIZE;
+
+		break;
+	case IB_CM_REJ_RECEIVED:
+		ib_ucm_event_rej_get(&uvt->resp.u.rej_resp,
+				     &evt->param.rej_rcvd);
+		uvt->data_len = IB_CM_REJ_PRIVATE_DATA_SIZE;
+		uvt->info_len = evt->param.rej_rcvd.ari_length;
+		info	      = evt->param.rej_rcvd.ari;
+
+		break;
+	case IB_CM_LAP_RECEIVED:
+		ib_ucm_event_lap_get(&uvt->resp.u.lap_resp,
+				     &evt->param.lap_rcvd);
+		uvt->data_len = IB_CM_LAP_PRIVATE_DATA_SIZE;
+		uvt->resp.present |= (evt->param.lap_rcvd.alternate_path ? 
+				      IB_UCM_PRES_ALTERNATE : 0);
+		break;
+	case IB_CM_APR_RECEIVED:
+		ib_ucm_event_apr_get(&uvt->resp.u.apr_resp,
+				     &evt->param.apr_rcvd);
+		uvt->data_len = IB_CM_APR_PRIVATE_DATA_SIZE;
+		uvt->info_len = evt->param.apr_rcvd.info_len;
+		info	      = evt->param.apr_rcvd.apr_info;
+
+		break;
+	case IB_CM_SIDR_REQ_RECEIVED:
+		ib_ucm_event_sidr_req_get(&uvt->resp.u.sidr_req_resp, 
+					  &evt->param.sidr_req_rcvd);
+		uvt->data_len = IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE;
+
+		break;
+	case IB_CM_SIDR_REP_RECEIVED:
+		ib_ucm_event_sidr_rep_get(&uvt->resp.u.sidr_rep_resp, 
+					  &evt->param.sidr_rep_rcvd);
+		uvt->data_len = IB_CM_SIDR_REP_PRIVATE_DATA_SIZE;
+		uvt->info_len = evt->param.sidr_rep_rcvd.info_len;
+		info	      = evt->param.sidr_rep_rcvd.info;
+
+		break;
+	default:
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	}
+
+	if (uvt->data_len && evt->private_data) {
+
+		uvt->data = kmalloc(uvt->data_len, GFP_KERNEL);
+		if (!uvt->data) {
+			result = -ENOMEM;
+			goto error;
+		}
+
+		memcpy(uvt->data, evt->private_data, uvt->data_len);
+		uvt->resp.present |= IB_UCM_PRES_DATA;
+	}
+
+	if (uvt->info_len && info) {
+
+		uvt->info = kmalloc(uvt->info_len, GFP_KERNEL);
+		if (!uvt->info) {
+			result = -ENOMEM;
+			goto error;
+		}
+
+		memcpy(uvt->info, info, uvt->info_len);
+		uvt->resp.present |= IB_UCM_PRES_INFO;
+	}
+
+	return 0;
+error:
+	if (uvt->info)
+		kfree(uvt->info);
+	if (uvt->data)
+		kfree(uvt->data);
+	return result;
+}
+	
+static int ib_ucm_event_handler(struct ib_cm_id *cm_id,
+				struct ib_cm_event *event)
+{
+	struct ib_ucm_event *uevent;
+	struct ib_ucm_context *ctx;
+	int result = 0;
+	int id;
+	/*
+	 * lookup correct context based on event type.
+	 */
+	switch (event->event) {
+	case IB_CM_REQ_RECEIVED:
+		id = (long)event->param.req_rcvd.listen_id->context;
+		break;
+	case IB_CM_SIDR_REQ_RECEIVED:
+		id = (long)event->param.sidr_req_rcvd.listen_id->context;
+		break;
+	default:
+		id = (long)cm_id->context;
+		break;
+	}
+
+	printk(KERN_ERR "UCM: Event. CM ID <%d> event <%d>\n",
+	       id, event->event);
+
+	ctx = ib_ucm_ctx_get(id);
+	if (!ctx)
+		return -ENOENT;
+
+	if (event->event == IB_CM_REQ_RECEIVED ||
+	    event->event == IB_CM_SIDR_REQ_RECEIVED)
+		id = IB_UCM_CM_ID_INVALID;
+
+	uevent = kmalloc(sizeof(*uevent), GFP_KERNEL);
+	if (!uevent) {
+		result = -ENOMEM;
+		goto done;
+	}
+
+	memset(uevent, 0, sizeof(*uevent));
+
+	uevent->resp.id    = id;
+	uevent->resp.event = event->event;
+
+	result = ib_ucm_event_process(event, uevent);
+	if (result)
+		goto done;
+
+	uevent->ctx   = ctx;
+	uevent->cm_id = ((event->event == IB_CM_REQ_RECEIVED ||
+			  event->event == IB_CM_SIDR_REQ_RECEIVED ) ?
+			 cm_id : NULL);
+
+	down(&ctx->file->mutex);
+
+	list_add_tail(&uevent->file_list, &ctx->file->events);
+	list_add_tail(&uevent->ctx_list, &ctx->events);
+
+	wake_up_interruptible(&ctx->file->poll_wait);
+
+	up(&ctx->file->mutex);
+done:
+	ctx->error = result;
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static ssize_t ib_ucm_event(struct ib_ucm_file *file,
+			    const char __user *inbuf, 
+			    int in_len, int out_len)
+{
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_event_get cmd;
+	struct ib_ucm_event *uevent = NULL;
+	int result = 0;
+	DEFINE_WAIT(wait);
+	
+	if (out_len < sizeof(struct ib_ucm_event_resp))
+		return -ENOSPC;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+	/*
+	 * wait
+	 */
+	down(&file->mutex);
+
+	while (list_empty(&file->events)) {
+
+		if (file->filp->f_flags & O_NONBLOCK) {
+			result = -EAGAIN;
+			break;
+		}
+
+		if (signal_pending(current)) {
+			result = -ERESTARTSYS;
+			break;
+		}
+
+		prepare_to_wait(&file->poll_wait, &wait, TASK_INTERRUPTIBLE);
+
+		up(&file->mutex);
+		schedule();
+		down(&file->mutex);
+
+		finish_wait(&file->poll_wait, &wait);
+	}
+
+	if (result)
+		goto done;
+	
+	uevent = list_entry(file->events.next, struct ib_ucm_event, file_list);
+
+	if (!uevent->cm_id)
+		goto user;
+
+	ctx = ib_ucm_ctx_alloc(file);
+	if (!ctx) {
+		result = -ENOMEM;
+		goto done;
+	}
+
+	ctx->cm_id             = uevent->cm_id;
+	ctx->cm_id->cm_handler = ib_ucm_event_handler;
+	ctx->cm_id->context    = (void *)(unsigned long)ctx->id;
+
+	uevent->resp.id = ctx->id;
+
+user:
+	if (copy_to_user((void __user *)(unsigned long)cmd.response,
+			 &uevent->resp, sizeof(uevent->resp))) {
+		result = -EFAULT;
+		goto done;
+	}
+
+	if (uevent->data) {
+
+		if (cmd.data_len < uevent->data_len) {
+			result = -ENOMEM;
+			goto done;
+		}
+
+		if (copy_to_user((void __user *)(unsigned long)cmd.data,
+				 uevent->data, uevent->data_len)) {
+			result = -EFAULT;
+			goto done;
+		}
+	}
+
+	if (uevent->info) {
+
+		if (cmd.info_len < uevent->info_len) {
+			result = -ENOMEM;
+			goto done;
+		}
+
+		if (copy_to_user((void __user *)(unsigned long)cmd.info,
+				 uevent->info, uevent->info_len)) {
+			result = -EFAULT;
+			goto done;
+		}
+	}
+
+	list_del(&uevent->file_list);
+	list_del(&uevent->ctx_list);
+
+	if (uevent->data)
+		kfree(uevent->data);
+	if (uevent->info)
+		kfree(uevent->info);
+	kfree(uevent);
+done:
+	up(&file->mutex);
+	return result;
+}
+
+
+static ssize_t ib_ucm_create_id(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	struct ib_ucm_create_id cmd;
+	struct ib_ucm_create_id_resp resp;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	if (out_len < sizeof(resp))
+		return -ENOSPC;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	ctx = ib_ucm_ctx_alloc(file);
+	if (!ctx)
+		return -ENOMEM;
+
+	ctx->cm_id = ib_create_cm_id(ib_ucm_event_handler, 
+				     (void *)(unsigned long)ctx->id);
+	if (!ctx->cm_id) {
+		result = -ENOMEM;
+		goto err_cm;
+	}
+
+	resp.id = ctx->id;
+	if (copy_to_user((void __user *)(unsigned long)cmd.response,
+			 &resp, sizeof(resp))) {
+		result = -EFAULT;
+		goto err_ret;
+	}
+
+	return 0;
+err_ret:
+	ib_destroy_cm_id(ctx->cm_id);
+err_cm:
+	ib_ucm_ctx_put(ctx); /* user reference */
+
+	return result;
+}
+
+static ssize_t ib_ucm_destroy_id(struct ib_ucm_file *file,
+				 const char __user *inbuf, 
+				 int in_len, int out_len)
+{
+	struct ib_ucm_destroy_id cmd;
+	struct ib_ucm_context *ctx;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+	ib_ucm_ctx_put(ctx); /* user reference */
+	ib_ucm_ctx_put(ctx); /* func reference */
+
+	return 0;
+}
+
+static ssize_t ib_ucm_attr_id(struct ib_ucm_file *file,
+			      const char __user *inbuf, 
+			      int in_len, int out_len)
+{
+	struct ib_ucm_attr_id_resp resp;
+	struct ib_ucm_attr_id cmd;
+	struct ib_ucm_context *ctx;
+	int result = 0;
+
+	if (out_len < sizeof(resp))
+		return -ENOSPC;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file) {
+		result = -EINVAL;
+		goto done;
+	}
+
+	resp.service_id   = ctx->cm_id->service_id;
+	resp.service_mask = ctx->cm_id->service_mask;
+	resp.local_id     = ctx->cm_id->local_id;
+	resp.remote_id    = ctx->cm_id->remote_id;
+
+	if (copy_to_user((void __user *)(unsigned long)cmd.response,
+			 &resp, sizeof(resp)))
+		result = -EFAULT;
+
+done:
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static ssize_t ib_ucm_listen(struct ib_ucm_file *file,
+			     const char __user *inbuf, 
+			     int in_len, int out_len)
+{
+	struct ib_ucm_listen cmd;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+	
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_cm_listen(ctx->cm_id, cmd.service_id,
+				      cmd.service_mask);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static ssize_t ib_ucm_establish(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	struct ib_ucm_establish cmd;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+	
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_cm_establish(ctx->cm_id);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static int ib_ucm_alloc_data(const void **dest, u64 src, u32 len)
+{
+	void *data;
+
+	*dest = NULL;
+
+	if (!len)
+		return 0;
+
+	data = kmalloc(len, GFP_KERNEL);
+	if (!data)
+		return -ENOMEM;
+	
+	if (copy_from_user(data, (void __user *)(unsigned long)src, len)) {
+		kfree(data);
+		return -EFAULT;
+	}
+
+	*dest = data;
+	return 0;
+}
+
+static int ib_ucm_path_get(struct ib_sa_path_rec **path, u64 src)
+{
+	struct ib_ucm_path_rec ucm_path;
+	struct ib_sa_path_rec  *sa_path;
+
+	*path = NULL;
+
+	if (!src)
+		return 0;
+
+	sa_path = kmalloc(sizeof(*sa_path), GFP_KERNEL);
+	if (!sa_path)
+		return -ENOMEM;
+
+	if (copy_from_user(&ucm_path, (void __user *)(unsigned long)src,
+			   sizeof(ucm_path))) {
+
+		kfree(sa_path);
+		return -EFAULT;
+	}
+
+	memcpy(sa_path->dgid.raw, ucm_path.dgid, sizeof(union ib_gid));
+	memcpy(sa_path->sgid.raw, ucm_path.sgid, sizeof(union ib_gid));
+
+	sa_path->dlid	          = ucm_path.dlid;
+	sa_path->slid	          = ucm_path.slid;
+	sa_path->raw_traffic      = ucm_path.raw_traffic;
+	sa_path->flow_label       = ucm_path.flow_label;
+	sa_path->hop_limit        = ucm_path.hop_limit;
+	sa_path->traffic_class    = ucm_path.traffic_class;
+	sa_path->reversible       = ucm_path.reversible;
+	sa_path->numb_path        = ucm_path.numb_path;
+	sa_path->pkey             = ucm_path.pkey;
+	sa_path->sl               = ucm_path.sl;
+	sa_path->mtu_selector     = ucm_path.mtu_selector;
+	sa_path->mtu              = ucm_path.mtu;
+	sa_path->rate_selector    = ucm_path.rate_selector;
+	sa_path->rate             = ucm_path.rate;
+	sa_path->packet_life_time = ucm_path.packet_life_time;
+	sa_path->preference       = ucm_path.preference;
+
+	sa_path->packet_life_time_selector = 
+		ucm_path.packet_life_time_selector;
+
+	*path = sa_path;
+	return 0;
+}
+
+static ssize_t ib_ucm_send_req(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_cm_req_param param;
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_req cmd;
+	int result;
+
+	param.private_data   = NULL;
+	param.primary_path   = NULL;
+	param.alternate_path = NULL;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data, cmd.data, cmd.len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&param.primary_path, cmd.primary_path);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&param.alternate_path, cmd.alternate_path);
+	if (result)
+		goto done;
+
+	param.private_data_len           = cmd.len;
+	param.service_id                 = cmd.sid;
+	param.qp_num                     = cmd.qpn;
+	param.qp_type                    = cmd.qp_type;
+	param.starting_psn               = cmd.psn;
+	param.peer_to_peer               = cmd.peer_to_peer;
+	param.responder_resources        = cmd.responder_resources;
+	param.initiator_depth            = cmd.initiator_depth;
+	param.remote_cm_response_timeout = cmd.remote_cm_response_timeout;
+	param.flow_control               = cmd.flow_control;
+	param.local_cm_response_timeout  = cmd.local_cm_response_timeout;
+	param.retry_count                = cmd.retry_count;
+	param.rnr_retry_count            = cmd.rnr_retry_count;
+	param.max_cm_retries             = cmd.max_cm_retries;
+	param.srq                        = cmd.srq;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_req(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+	if (param.primary_path)
+		kfree(param.primary_path);
+	if (param.alternate_path)
+		kfree(param.alternate_path);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_rep(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_cm_rep_param param;
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_rep cmd;
+	int result;
+
+	param.private_data = NULL;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data, cmd.data, cmd.len);
+	if (result)
+		return result;
+
+	param.qp_num              = cmd.qpn;
+	param.starting_psn        = cmd.psn;
+	param.private_data_len    = cmd.len;
+	param.responder_resources = cmd.responder_resources;
+	param.initiator_depth     = cmd.initiator_depth;
+	param.target_ack_delay    = cmd.target_ack_delay;
+	param.failover_accepted   = cmd.failover_accepted;
+	param.flow_control        = cmd.flow_control;
+	param.rnr_retry_count     = cmd.rnr_retry_count;
+	param.srq                 = cmd.srq;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_rep(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_private_data(struct ib_ucm_file *file,
+					const char __user *inbuf, int in_len,
+					int (*func)(struct ib_cm_id *cm_id,
+						    const void *private_data,
+						    u8 private_data_len))
+{
+	struct ib_ucm_private_data cmd;
+	struct ib_ucm_context *ctx;
+	const void *private_data = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&private_data, cmd.data, cmd.len);
+	if (result)
+		return result;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = func(ctx->cm_id, private_data, cmd.len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (private_data)
+		kfree(private_data);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_rtu(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	return ib_ucm_send_private_data(file, inbuf, in_len, ib_send_cm_rtu);
+}
+
+static ssize_t ib_ucm_send_dreq(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	return ib_ucm_send_private_data(file, inbuf, in_len, ib_send_cm_dreq);
+}
+
+static ssize_t ib_ucm_send_drep(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	return ib_ucm_send_private_data(file, inbuf, in_len, ib_send_cm_drep);
+}
+
+static ssize_t ib_ucm_send_info(struct ib_ucm_file *file,
+				const char __user *inbuf, int in_len,
+				int (*func)(struct ib_cm_id *cm_id,
+					    int status,
+					    const void *info,
+					    u8 info_len,
+					    const void *data,
+					    u8 data_len))
+{
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_info cmd;
+	const void *data = NULL;
+	const void *info = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&data, cmd.data, cmd.data_len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_alloc_data(&info, cmd.info, cmd.info_len);
+	if (result)
+		goto done;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = func(ctx->cm_id, cmd.status,
+			      info, cmd.info_len,
+			      data, cmd.data_len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (data)
+		kfree(data);
+	if (info)
+		kfree(info);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_rej(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	return ib_ucm_send_info(file, inbuf, in_len, (void *)ib_send_cm_rej);
+}
+
+static ssize_t ib_ucm_send_apr(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	return ib_ucm_send_info(file, inbuf, in_len, (void *)ib_send_cm_apr);
+}
+
+static ssize_t ib_ucm_send_mra(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_mra cmd;
+	const void *data = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&data, cmd.data, cmd.len);
+	if (result)
+		return result;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_mra(ctx->cm_id, cmd.timeout, 
+					data, cmd.len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (data)
+		kfree(data);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_lap(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_ucm_context *ctx;
+	struct ib_sa_path_rec *path = NULL;
+	struct ib_ucm_lap cmd;
+	const void *data = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&data, cmd.data, cmd.len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&path, cmd.path);
+	if (result)
+		goto done;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_lap(ctx->cm_id, path, data, cmd.len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (data)
+		kfree(data);
+	if (path)
+		kfree(path);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_sidr_req(struct ib_ucm_file *file,
+				    const char __user *inbuf, 
+				    int in_len, int out_len)
+{
+	struct ib_cm_sidr_req_param param;
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_sidr_req cmd;
+	int result;
+
+	param.private_data = NULL;
+	param.path = NULL;
+	
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data, cmd.data, cmd.len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&param.path, cmd.path);
+	if (result)
+		goto done;
+
+	param.private_data_len = cmd.len;
+	param.service_id       = cmd.sid;
+	param.timeout_ms       = cmd.timeout;
+	param.max_cm_retries   = cmd.max_cm_retries;
+	param.pkey             = cmd.pkey;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_sidr_req(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+	if (param.path)
+		kfree(param.path);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_sidr_rep(struct ib_ucm_file *file,
+				    const char __user *inbuf, 
+				    int in_len, int out_len)
+{
+	struct ib_cm_sidr_rep_param param;
+	struct ib_ucm_sidr_rep cmd;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	param.info = NULL;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data,
+				   cmd.data, cmd.data_len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_alloc_data(&param.info, cmd.info, cmd.info_len);
+	if (result)
+		goto done;
+
+	param.qp_num	   = cmd.qpn;
+	param.qkey	     = cmd.qkey;
+	param.status	   = cmd.status;
+	param.info_length      = cmd.info_len;
+	param.private_data_len = cmd.data_len;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_sidr_rep(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+	if (param.info)
+		kfree(param.info);
+
+	return result;
+}
+
+static ssize_t (*ucm_cmd_table[])(struct ib_ucm_file *file,
+				  const char __user *inbuf,
+				  int in_len, int out_len) = {
+	[IB_USER_CM_CMD_CREATE_ID]     = ib_ucm_create_id,
+	[IB_USER_CM_CMD_DESTROY_ID]    = ib_ucm_destroy_id,
+	[IB_USER_CM_CMD_ATTR_ID]       = ib_ucm_attr_id,
+	[IB_USER_CM_CMD_LISTEN]        = ib_ucm_listen,
+	[IB_USER_CM_CMD_ESTABLISH]     = ib_ucm_establish,
+	[IB_USER_CM_CMD_SEND_REQ]      = ib_ucm_send_req,
+	[IB_USER_CM_CMD_SEND_REP]      = ib_ucm_send_rep,
+	[IB_USER_CM_CMD_SEND_RTU]      = ib_ucm_send_rtu,
+	[IB_USER_CM_CMD_SEND_DREQ]     = ib_ucm_send_dreq,
+	[IB_USER_CM_CMD_SEND_DREP]     = ib_ucm_send_drep,
+	[IB_USER_CM_CMD_SEND_REJ]      = ib_ucm_send_rej,
+	[IB_USER_CM_CMD_SEND_MRA]      = ib_ucm_send_mra,
+	[IB_USER_CM_CMD_SEND_LAP]      = ib_ucm_send_lap,
+	[IB_USER_CM_CMD_SEND_APR]      = ib_ucm_send_apr,
+	[IB_USER_CM_CMD_SEND_SIDR_REQ] = ib_ucm_send_sidr_req,
+	[IB_USER_CM_CMD_SEND_SIDR_REP] = ib_ucm_send_sidr_rep,
+	[IB_USER_CM_CMD_EVENT]	       = ib_ucm_event,
+};
+
+static ssize_t ib_ucm_write(struct file *filp, const char __user *buf,
+			    size_t len, loff_t *pos)
+{
+	struct ib_ucm_file *file = filp->private_data;
+	struct ib_ucm_cmd_hdr hdr;
+	ssize_t result;
+
+	if (len < sizeof(hdr))
+		return -EINVAL;
+
+	if (copy_from_user(&hdr, buf, sizeof(hdr)))
+		return -EFAULT;
+
+	printk(KERN_ERR "UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
+	       hdr.cmd, hdr.in, hdr.out, len);
+
+	if (hdr.cmd < 0 || hdr.cmd >= ARRAY_SIZE(ucm_cmd_table))
+		return -EINVAL;
+
+	if (hdr.in + sizeof(hdr) > len)
+		return -EINVAL;
+
+	result = ucm_cmd_table[hdr.cmd](file, buf + sizeof(hdr),
+					hdr.in, hdr.out);
+	if (!result)
+		result = len;
+
+	return result;
+}
+
+static unsigned int ib_ucm_poll(struct file *filp,
+				struct poll_table_struct *wait)
+{
+	struct ib_ucm_file *file = filp->private_data;
+	unsigned int mask = 0;
+
+	poll_wait(filp, &file->poll_wait, wait);
+
+	if (!list_empty(&file->events))
+		mask = POLLIN | POLLRDNORM;
+
+	return mask;
+}
+
+static int ib_ucm_open(struct inode *inode, struct file *filp)
+{
+	struct ib_ucm_file *file;
+
+	file = kmalloc(sizeof(*file), GFP_KERNEL);
+	if (!file)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&file->events);
+	INIT_LIST_HEAD(&file->ctxs);
+	init_waitqueue_head(&file->poll_wait);
+
+	init_MUTEX(&file->mutex);
+
+	filp->private_data = file;
+	file->filp = filp;
+
+	printk(KERN_ERR "UCM: Created struct\n");
+
+	return 0;
+}
+
+static int ib_ucm_close(struct inode *inode, struct file *filp)
+{
+	struct ib_ucm_file *file = filp->private_data;
+	struct ib_ucm_context *ctx;
+
+	down(&file->mutex);
+
+	while (!list_empty(&file->ctxs)) {
+
+		ctx = list_entry(file->ctxs.next,
+				 struct ib_ucm_context, file_list);
+		
+		up(&ctx->file->mutex);
+		ib_ucm_ctx_put(ctx); /* user reference */
+		down(&file->mutex);
+	}
+
+	up(&file->mutex);
+
+	kfree(file);
+
+	printk(KERN_ERR "UCM: Deleted struct\n");
+	return 0;
+}
+
+static struct file_operations ib_ucm_fops = {
+	.owner 	 = THIS_MODULE,
+	.open 	 = ib_ucm_open,
+	.release = ib_ucm_close,
+	.write 	 = ib_ucm_write,
+	.poll    = ib_ucm_poll,
+};
+
+
+static struct class_simple *ib_ucm_class;
+static struct cdev	  ib_ucm_cdev;
+
+static int __init ib_ucm_init(void)
+{
+	int result;
+
+	result = register_chrdev_region(IB_UCM_DEV, 1, "infiniband_cm");
+	if (result) {
+		printk(KERN_ERR "UCM: Error <%d> registering dev\n", result);
+		goto err_chr;
+	}
+
+	cdev_init(&ib_ucm_cdev, &ib_ucm_fops);
+
+	result = cdev_add(&ib_ucm_cdev, IB_UCM_DEV, 1);
+	if (result) {
+		printk(KERN_ERR "UCM: Error <%d> adding cdev\n", result);
+		goto err_cdev;
+	}
+
+	ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm");
+	if (IS_ERR(ib_ucm_class)) {
+		result = PTR_ERR(ib_ucm_class);
+		printk(KERN_ERR "UCM: Error <%d> creating class\n", result);
+		goto err_class;
+	}
+
+	class_simple_device_add(ib_ucm_class,
+				IB_UCM_DEV,
+				NULL,
+				"ucm");
+	
+	idr_init(&ctx_id_table);
+	init_MUTEX(&ctx_id_mutex);
+
+	return 0;
+err_class:
+	cdev_del(&ib_ucm_cdev);
+err_cdev:
+	unregister_chrdev_region(IB_UCM_DEV, 1);
+err_chr:
+	return result;
+}
+
+static void __exit ib_ucm_cleanup(void)
+{
+	class_simple_device_remove(IB_UCM_DEV);
+	class_simple_destroy(ib_ucm_class);
+	cdev_del(&ib_ucm_cdev);
+	unregister_chrdev_region(IB_UCM_DEV, 1);
+}
+
+module_init(ib_ucm_init);
+module_exit(ib_ucm_cleanup);
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband25/core/ucm.h linux-2.6.13-rc2-mm1/drivers/infiniband26/core/ucm.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband25/core/ucm.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1/drivers/infiniband26/core/ucm.h	2005-04-22 23:23:15.000000000 -0400
@@ -0,0 +1,89 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ucm.h 2208 2005-04-22 23:24:31Z libor $
+ */
+
+#ifndef UCM_H
+#define UCM_H
+
+#include <linux/fs.h>
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/idr.h>
+
+#include <ib_cm.h>
+#include <ib_user_cm.h>
+
+#define IB_UCM_CM_ID_INVALID 0xffffffff
+
+struct ib_ucm_file {
+	struct semaphore mutex;
+	struct file *filp;
+	/*
+	 * list of pending events
+	 */
+	struct list_head  ctxs;   /* list of active connections */
+	struct list_head  events; /* list of pending events */
+	wait_queue_head_t poll_wait;
+};
+
+struct ib_ucm_context {
+	int                 id;
+	int                 ref;
+	int                 error;
+
+	struct ib_ucm_file *file;
+	struct ib_cm_id    *cm_id;
+	struct semaphore    mutex;
+
+	struct list_head    events;    /* list of pending events. */
+	struct list_head    file_list; /* member in file ctx list */
+};
+
+struct ib_ucm_event {
+	struct ib_ucm_context *ctx;
+	struct list_head file_list; /* member in file event list */
+	struct list_head ctx_list;  /* member in ctx event list */
+
+	struct ib_ucm_event_resp resp;
+	void *data;
+	void *info;
+	int data_len;
+	int info_len;
+	/*
+	 * new connection identifiers needs to be saved until
+	 * userspace can get a handle on them.
+	 */
+	struct ib_cm_id *cm_id;
+};
+
+#endif /* UCM_H */



From halr at voltaire.com  Mon Jul 11 08:12:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:12:53 -0400
Subject: [openib-general] [PATCH 27/27] Hook up userspace CM to the make
	system
Message-ID: <1121089282.4389.4559.camel@hal.voltaire.com>

Hook up userspace CM to the make system 

Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 26/27.

-- 
 Makefile |    4 +++-
 1 files changed, 3 insertions(+), 1 deletion(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband26/core/Makefile linux-2.6.13-rc2-mm1/drivers/infiniband27/core/Makefile
-- linux-2.6.13-rc2-mm1/drivers/infiniband26/core/Makefile	2005-07-10 16:50:04.000000000 -0400
+++ linux-2.6.13-rc2-mm1/drivers/infiniband27/core/Makefile	2005-07-10 16:59:52.000000000 -0400
@@ -1,7 +1,7 @@
 EXTRA_CFLAGS += -Idrivers/infiniband/include
 
 obj-$(CONFIG_INFINIBAND) +=		ib_core.o ib_mad.o ib_sa.o \
-					ib_cm.o ib_umad.o
+					ib_cm.o ib_umad.o ib_ucm.o
 obj-$(CONFIG_INFINIBAND_USER_VERBS) +=	ib_uverbs.o
 
 ib_core-y :=			packer.o ud_header.o verbs.o sysfs.o \
@@ -15,4 +15,6 @@ ib_cm-y :=			cm.o
 
 ib_umad-y :=			user_mad.o
 
+ib_ucm-y :=			ucm.o
+
 ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_mem.o




From halr at voltaire.com  Mon Jul 11 08:19:24 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:19:24 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <cd9967e2050710172345f01051@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
Message-ID: <1121090768.4389.4566.camel@hal.voltaire.com>

On Sun, 2005-07-10 at 20:23, Talal jaafar wrote:
> hi all,
> 
>    I am fairly new to OpenIB and I have downloaded and built the
> 2.6.12 kernel that has OpenIB support. My question now is how to
> compile the OpenIB modules in that kernel; There is no documentation,
> and all what I need to do is to test the IPoIB, so anyhelp would be
> appreciated.

There is no "official" way. Either you can use the support already
included in the kernel or use the latest and greatest (from the OpenIB
svn tree). IPoIB is pretty stable. Some of the other ULPs are undergoing
more change so that might influence your choice.

In terms of building OpenIB modules, you will need to go into the kernel
config, under device drivers, and configure the InfiniBand things you
want for m (module). Mellanox HCA and IPoIB sound like they will be
sufficient for what you are doing.

-- Hal



From halr at voltaire.com  Mon Jul 11 08:25:50 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:25:50 -0400
Subject: [openib-general] [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/IBMgtSim: . config src tests utils]
Message-ID: <1121091238.4389.4581.camel@hal.voltaire.com>

Hi Eitan,

A few nits with the latest:

1. Still get:
g++ -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include -I/home/hal/openib/trunk/src/userspace/management/osm/include -I/usr/include -I../datamodel   -O6 -Wall -Werror  -c -o sim.o `test -f 'sim.cpp' || echo './'`sim.cpp
sim.cpp:342: parse error at end of input

2. Still uneeded executable permissions on a number of simulator and
ibdm files.

Some questions:

1. What is BUILD_STANDALONE_SIM for ?

2. Will you now be doing your development in OpenIB for ibdm and the
simulator or will this be done elsewhere and dropped in here ?

Thanks.

-- Hal

--Forwarded Message--

From: eitan at openib.org
To: openib-commits at openib.org
Subject: [openib-commits] r2826 - in gen2/utils/src/linux-user/IBMgtSim: . config src tests utils
Date: 10 Jul 2005 02:00:57 -0700

Author: eitan
Date: 2005-07-10 02:00:54 -0700 (Sun, 10 Jul 2005)
New Revision: 2826

Added:
   gen2/utils/src/linux-user/IBMgtSim/src/inttypes.i
   gen2/utils/src/linux-user/IBMgtSim/src/mads.i
   gen2/utils/src/linux-user/IBMgtSim/src/pma.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/pma.h
   gen2/utils/src/linux-user/IBMgtSim/src/randmgr.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/randmgr.h
   gen2/utils/src/linux-user/IBMgtSim/src/vsa.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/vsa.h
   gen2/utils/src/linux-user/IBMgtSim/tests/
   gen2/utils/src/linux-user/IBMgtSim/tests/Gnu16NodeOsmTest.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/tests/OsmTest.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/OsmTest.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/README
   gen2/utils/src/linux-user/IBMgtSim/tests/ReindeerBased128Nodes.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBased10K.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBased512Nodes.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBasedCreditLoop.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoReindeer1100.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/discover.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/discover.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/duplicateGuid.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/init.test.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmLidAssignment.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmLidAssignment.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticast.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticastRoutingTest.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticastRoutingTest.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmStability.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmStability.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osm_no_quit.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osm_no_quit.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/pkey.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/pkey.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/utils/LICENSE.txt
   gen2/utils/src/linux-user/IBMgtSim/utils/fprogbar.sh
   gen2/utils/src/linux-user/IBMgtSim/utils/install.sh
   gen2/utils/src/linux-user/IBMgtSim/utils/uninstall.sh
Modified:
   gen2/utils/src/linux-user/IBMgtSim/INSTALL
   gen2/utils/src/linux-user/IBMgtSim/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/README
   gen2/utils/src/linux-user/IBMgtSim/config/bootstrap
   gen2/utils/src/linux-user/IBMgtSim/config/osm.m4
   gen2/utils/src/linux-user/IBMgtSim/configure.in
   gen2/utils/src/linux-user/IBMgtSim/src/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/src/client.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/dispatcher.h
   gen2/utils/src/linux-user/IBMgtSim/src/helper.h
   gen2/utils/src/linux-user/IBMgtSim/src/ib_types.i
   gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i
   gen2/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h
   gen2/utils/src/linux-user/IBMgtSim/src/ibmssh_wrap.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/msgmgr.h
   gen2/utils/src/linux-user/IBMgtSim/src/node.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/node.h
   gen2/utils/src/linux-user/IBMgtSim/src/server.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/server.h
   gen2/utils/src/linux-user/IBMgtSim/src/sim.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/sim.h
   gen2/utils/src/linux-user/IBMgtSim/src/sim.i
   gen2/utils/src/linux-user/IBMgtSim/src/simmsg.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/simmsg.h
   gen2/utils/src/linux-user/IBMgtSim/src/sma.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/sma.h
   gen2/utils/src/linux-user/IBMgtSim/src/swig_alternate_mangling.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/tcpcomm.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/tcpcomm.h
   gen2/utils/src/linux-user/IBMgtSim/utils/IBMgtSim
   gen2/utils/src/linux-user/IBMgtSim/utils/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest
Log:
Major update:
* Introduce RunSimTest and tests directory with a set of example tests 
  and a simple way to run them
* PMA integrated into the simulator
* VSA integrated into the simulator
* All randomization in test flows should be done through a thread safe
  randomization angine 

Many bug fixes.





From halr at voltaire.com  Mon Jul 11 08:30:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 11:30:23 -0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <200507111839.41807.adobriyan@gmail.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com>
Message-ID: <1121094791.4389.4591.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 10:39, Alexey Dobriyan wrote:
> On Monday 11 July 2005 17:48, Hal Rosenstock wrote:
> > Add new helper routines for allocating MADs for sending and formatting
> > a send WR.
> 
> > -- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
> > +++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
> 				   ^^^^^^^^^^^
> Ick. You'd better have linux-2.6.13-rc2-mm1-[0123...].

Shall I resubmit with linux-2.6.13-rc2-mm1-[0123...] ?

> > +struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
> > +					    u32 remote_qpn, u16 pkey_index,
> > +					    struct ib_ah *ah,
> > +					    int hdr_len, int data_len,
> > +					    int gfp_mask)
> 
> unsigned int __nocast gfp_mask, please. 430 or so infiniband sparse warnings
> is not a reason to add more.

I'll fix this in a subsequent patch. Is that OK ?

Thanks.

-- Hal



From bjordan.ics at gmail.com  Mon Jul 11 08:38:31 2005
From: bjordan.ics at gmail.com (William Jordan)
Date: Mon, 11 Jul 2005 11:38:31 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <cd9967e2050710172345f01051@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
Message-ID: <78d18e2050711083826cffc63@mail.gmail.com>

On 7/10/05, Talal jaafar <talal.jaafar at gmail.com> wrote:
> hi all,
> 
>   I am fairly new to OpenIB and I have downloaded and built the
> 2.6.12 kernel that has OpenIB support. My question now is how to
> compile the OpenIB modules in that kernel; There is no documentation,
> and all what I need to do is to test the IPoIB, so anyhelp would be
> appreciated.
> 
> Thanks in advance,
> 
>  Talal

It sounds like you just want to use the OpenIB support that is part of
the downloaded kernel. Correct? If so, just rebuild the kernel. In the
config phase, specify "InfiniBand support" under "Device Drivers". You
will need "Mellanox HCA support", and "IP-over-InfiniBand support".

If you alias ib0 and ib1 to ib_ipoib in the /etc/modprobe.conf, and
configure the ib0 and ib1 interfaces, the drivers should load at
boottime (assuming you have a Mellanox HCA).

If you want the latest OpenIB support, then you want to do what Robert
Walsh suggested, and replace the drivers/infiniband tree in the 2.6.12
source with the latest gen2/trunk/src/linux-kernel/infiniband
downloaded from svn.

-- 
Bill Jordan
SilverStorm Technologies


From nacc at us.ibm.com  Mon Jul 11 09:05:05 2005
From: nacc at us.ibm.com (Nishanth Aravamudan)
Date: Mon, 11 Jul 2005 09:05:05 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <1121094791.4389.4591.camel@hal.voltaire.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com>
	<1121094791.4389.4591.camel@hal.voltaire.com>
Message-ID: <20050711160505.GA15937@us.ibm.com>

On 11.07.2005 [11:30:23 -0400], Hal Rosenstock wrote:
> On Mon, 2005-07-11 at 10:39, Alexey Dobriyan wrote:
> > On Monday 11 July 2005 17:48, Hal Rosenstock wrote:
> > > Add new helper routines for allocating MADs for sending and formatting
> > > a send WR.
> > 
> > > -- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
> > > +++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
> > 				   ^^^^^^^^^^^
> > Ick. You'd better have linux-2.6.13-rc2-mm1-[0123...].
> 
> Shall I resubmit with linux-2.6.13-rc2-mm1-[0123...] ?

Do these patches even apply with -p1 with these odd directories? I think
it will try to find the addition context in a file which doesn't exist
(because the directory, e.g. infiniband3, does not). I notice that every
patch increments these values. I think you only want to differentiate
between the kernels, not between particular directories in the kernel.

Thanks,
Nish


From robert.j.woodruff at intel.com  Mon Jul 11 09:10:17 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Mon, 11 Jul 2005 09:10:17 -0700
Subject: [openib-general] how to choose appropriate ULPs for application
Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004E39E21@orsmsx408>

There exists a NAL that supports the old gen1 IBAL stack and
I thought that there was one for the gen1 topspin stack, but
there is not one yet for gen2. I suspect that someone will write
one pretty soon, but do not know if anyone has started this effort
yet or not.

 

-----Original Message-----
From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of wangxigui
Sent: Wednesday, July 06, 2005 7:09 AM
To: openib-general at openib.org
Subject: RE: [openib-general] how to choose appropriate ULPs for application

Thank you very much!
I'll read the materials that you supply.

     I have not described my plan clearly. I don't konw how to run lustre
on IB. Neither do I konw the lustre running in kernel space or user
space. Now the lustre NAL does not supports IB vapi. If I use IPoIB,
it works. But it doen not take take advantage of RDMA. Supporse
lustre NAL invoke SDP or uDAP, RDMA is enable.

    Invocation of Verb API to realize lustre NAL is my target. The SDP or
uDAP is first step.

Clem Cole said:
> The quick answer here is none of SDP, iSER/SRP or uDAPL for different
> reasons on each case
>
> kDAPL is the kernel API that is designed to handle a kernel application
> such as NFS, Lustre or the like.  See:
> 	http://www.datcollaborative.org/kdapl.html
>
> There is are already implementations of NFS/RDMA available:
> 	http://www.sourceforge.net/projects/nfs-rdma
>
> This should give you some hints on how to a distrubuted file system
> uses the kDAPL interface.
>
>
>
> -----Original Message-----
> From: openib-general-bounces at openib.org
> [mailto:openib-general-bounces at openib.org]On Behalf Of wxg at ict.ac.cn
> Sent: Wednesday, July 06, 2005 9:26 AM
> To: openib-general
> Subject: [openib-general] how to choose appropriate ULPs for application
>
>
> Hello！
>
>     We plan to run lustre on InfiniBand. But now lustre does not support
> IB.
> I know noting about the IB stack. So I think the IB Upper Layer
> Protocol(SDP, DAP) could be used to transport data for lustre.
> 	But there are too many ULPs over IB. I know little about them. As far as
> I konw, iSER/SRP is for SCSI command transportation. So, it seems both
> SDP and uDAP are OK for my plan. I cant tell the difference between them.
> Which one is better, SDP or DAP? I don't konw how to choose?
>
> 	Any suggestion is appreciated!
>
> 　　
>
> 　　　　　　　　致
> 礼！
>
> Sincerely yours
> Wang Xigui 王锡贵
> 2005-07-06
> --------------------
> Computer Architecture Laboratory
> Institute of Computing Technology
> Chinese Academy of Sciences
> Beijing,P.R.China
> Zip code: 100080
> Tel: +86-10-62564394(office)
> Email: wxg at ict.ac.cn
> 　　　　　　　　　　　　　　
> 　　　　　　　　　　　　　　
> 　　　　　　　　　　　　　　　　　
>


-- 
Sincerely yours
Wang Xigui 王锡贵
2005-07-06
--------------------
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62565533-9314(office)
Email: wxg at ict.ac.cn

-- 
Sincerely yours
Wang Xigui 王锡贵
2005-07-06
--------------------
Computer Architecture Laboratory
Institute of Computing Technology
Chinese Academy of Sciences
Beijing,P.R.China
Zip code: 100080
Tel: +86-10-62565533-9314(office)
Email: wxg at ict.ac.cn

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

From halr at voltaire.com  Mon Jul 11 09:14:42 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 12:14:42 -0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <20050711160505.GA15937@us.ibm.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com>
	<1121094791.4389.4591.camel@hal.voltaire.com>
	<20050711160505.GA15937@us.ibm.com>
Message-ID: <1121098399.4389.4602.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 12:05, Nishanth Aravamudan wrote:
> On 11.07.2005 [11:30:23 -0400], Hal Rosenstock wrote:
> > On Mon, 2005-07-11 at 10:39, Alexey Dobriyan wrote:
> > > On Monday 11 July 2005 17:48, Hal Rosenstock wrote:
> > > > -- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
> > > > +++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
> > > 				   ^^^^^^^^^^^
> > > Ick. You'd better have linux-2.6.13-rc2-mm1-[0123...].
> > 
> > Shall I resubmit with linux-2.6.13-rc2-mm1-[0123...] ?
> 
> Do these patches even apply with -p1 with these odd directories? I think
> it will try to find the addition context in a file which doesn't exist
> (because the directory, e.g. infiniband3, does not). I notice that every
> patch increments these values. I think you only want to differentiate
> between the kernels, not between particular directories in the kernel.

I will resubmit. Sorry.

-- Hal



From adobriyan at gmail.com  Mon Jul 11 09:29:26 2005
From: adobriyan at gmail.com (Alexey Dobriyan)
Date: Mon, 11 Jul 2005 20:29:26 +0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <1121094791.4389.4591.camel@hal.voltaire.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com>
	<1121094791.4389.4591.camel@hal.voltaire.com>
Message-ID: <200507112029.26890.adobriyan@gmail.com>

On Monday 11 July 2005 19:30, Hal Rosenstock wrote:
> On Mon, 2005-07-11 at 10:39, Alexey Dobriyan wrote:
> > On Monday 11 July 2005 17:48, Hal Rosenstock wrote:
> > > Add new helper routines for allocating MADs for sending and formatting
> > > a send WR.
> > 
> > > -- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
> > > +++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
> > 				   ^^^^^^^^^^^
> > Ick. You'd better have linux-2.6.13-rc2-mm1-[0123...].
> 
> Shall I resubmit with linux-2.6.13-rc2-mm1-[0123...] ?

I'd wait for comments and then resubmit clean series which can be applied
with patch -p1.

> > > +struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
> > > +					    u32 remote_qpn, u16 pkey_index,
> > > +					    struct ib_ah *ah,
> > > +					    int hdr_len, int data_len,
> > > +					    int gfp_mask)
> > 
> > unsigned int __nocast gfp_mask, please. 430 or so infiniband sparse warnings
> > is not a reason to add more.
> 
> I'll fix this in a subsequent patch. Is that OK ?

If Andrew will fix patches locally, OK.


From info at senpu-ki.net  Mon Jul 11 09:25:59 2005
From: info at senpu-ki.net (info at senpu-ki.net)
Date: 12 Jul 2005 01:25:59 +0900
Subject: [openib-general] $B40A4L5NA$NAG?M=PD%%[%9%H$N%"%k%P%$%H(B
Message-ID: <20050711162559.27711.qmail@mail.senpu-ki.net>

$B(.(,(/(.(,(/(.(,(/(.(,(/(B
$B(-AG(-(-?M(-(-=P(-(-D%(-(B
$B(1(,(0(1(,(0(1(,(0(1(,(0(B
http://awg.webchu.com/?freehost
$B"(%[%9%HBgJg=8!J$*;n$740A4L5NAEPO?#O#K!*!K(B
$B"($9$Y$F%(%C%AL\E*$@$+$iNI$$;W$$$r$7$F>.8/$$2T$.!*(B
$B"($*Aj<j$O(B10$BBe!&#O#L!&Bg3X at 8!&@lLg3X9;@8!&<c:J!&=O=w!&IwB/>n!&#S>n!&#M>n!&(B
$B%J!<%9!&%9%C%A!<$J$I(B
$B(.(,(/(.(,(/(.(,(/(.(,(/(B
$B(-40(-(-A4(-(-L5(-(-NA(-(B
$B(1(,(0(1(,(0(1(,(0(1(,(0(B
http://awg.webchu.com/?freehost
$B(.(,(/(.(,(/(B
$B(-$*;n$7#O#K(-(B
$B(1(,(0(1(,(0(B
$B"(;E;v$N9g4V!"5YF|$rMxMQ$7$F<}F~%"%C%W$r$7$^$;$s$+!*D64JC1$JI{6H$G$9!#(B
$B!y5.J}$b%j%C%A$K$J$j$^$;$s$+!)(B
http://awg.webchu.com/?freehost

$B!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a(B
$B:#8e!"<u?.$r5qH]$9$k>l9g$OJV?.2<$5$$!#(B
a245_goawg at poppymail.com
$B!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a!&!a(B

18$B:PL$K~$OMxMQ6X;_$G$9!#(B


From jlentini at netapp.com  Mon Jul 11 09:35:46 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 11 Jul 2005 12:35:46 -0400 (EDT)
Subject: [openib-general] Kernel locking rules documentation
In-Reply-To: <52fyuprqlf.fsf@topspin.com>
References: <52fyuprqlf.fsf@topspin.com>
Message-ID: <Pine.LNX.4.61.0507111218410.25603@jlentini-linux.nane.netapp.com>


Thanks for writing this up Roland. I would recommend sending it 
upstream.

Will this document be expanded to include the functions 
declared in ib_cm.h?

On Fri, 8 Jul 2005, Roland Dreier wrote:

> I've tried to write up the locking rules used by our kernel IB layer,
> so that they're made explicit for people implementing new low-level
> drivers and new protocols.  I'd appreciate corrections and all other
> comments.  Does this seem worth sending upstream for inclusion under
> Documentation/infiniband?
>
> - R.
>
>


From mshefty at ichips.intel.com  Mon Jul 11 09:45:37 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Mon, 11 Jul 2005 09:45:37 -0700
Subject: [openib-general] Kernel locking rules documentation
In-Reply-To: <Pine.LNX.4.61.0507111218410.25603@jlentini-linux.nane.netapp.com>
References: <52fyuprqlf.fsf@topspin.com>
	<Pine.LNX.4.61.0507111218410.25603@jlentini-linux.nane.netapp.com>
Message-ID: <42D2A231.1090808@ichips.intel.com>

James Lentini wrote:
> 
> Thanks for writing this up Roland. I would recommend sending it upstream.
> 
> Will this document be expanded to include the functions declared in 
> ib_cm.h?

I can either expand the documentation in ib_cm.h directly or add to this 
file.  The CM seems separate from what's contained here, since it's not 
expected that anyone will write another CM.

- Sean


From caitlinb at siliquent.com  Mon Jul 11 09:47:04 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Mon, 11 Jul 2005 09:47:04 -0700
Subject: [openib-general] makiing ibverb.h  transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>

The attached .h files follow the alternate approach where each
transport can define its own specifics in a supplemental .h file
and avoid the massive union approach.

This is mostly cosmetic, except for the work request. Because
some work request opcodes are transport specific the work request
itself has to follow the same approach of deferred sub-typing
rather than unioning.

The impact of this on the existing code would be quite wide,
but quite shallow. Many types would be renamed form 'ib_xyz'
to 'rdma_xyz', if being used in transport neutral code. Or
when staying infiniband specific fields that now qualified
as transport neutral would require "common." to be inserted
before the field name. Some constants are shifted from IB_XXX
to RDMA_XXX.

The goal of these changes is to allow users of the core
services to code *most* of their logic without having to
reference transport specific fields.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: rdma_verbs.h
Type: application/octet-stream
Size: 33780 bytes
Desc: rdma_verbs.h
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/442c161f/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ibonly_verbs.h
Type: application/octet-stream
Size: 15717 bytes
Desc: ibonly_verbs.h
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/442c161f/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iwarp_verbs.h
Type: application/octet-stream
Size: 4661 bytes
Desc: iwarp_verbs.h
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/442c161f/attachment-0002.obj>

From halr at voltaire.com  Mon Jul 11 09:47:52 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 12:47:52 -0400
Subject: [openib-general] Kernel locking rules documentation
In-Reply-To: <Pine.LNX.4.61.0507111218410.25603@jlentini-linux.nane.netapp.com>
References: <52fyuprqlf.fsf@topspin.com>
	<Pine.LNX.4.61.0507111218410.25603@jlentini-linux.nane.netapp.com>
Message-ID: <1121100472.4389.4607.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 12:35, James Lentini wrote:
> Thanks for writing this up Roland. I would recommend sending it 
> upstream.

Since it seems like I will be resubmitting the core patches again, I
could include this. Any objections ?

-- Hal



From halr at voltaire.com  Mon Jul 11 09:58:43 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 12:58:43 -0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <200507112029.26890.adobriyan@gmail.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com>
	<1121094791.4389.4591.camel@hal.voltaire.com>
	<200507112029.26890.adobriyan@gmail.com>
Message-ID: <1121100534.4389.4609.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 12:29, Alexey Dobriyan wrote:
> On Monday 11 July 2005 19:30, Hal Rosenstock wrote:
> > On Mon, 2005-07-11 at 10:39, Alexey Dobriyan wrote:
> > > On Monday 11 July 2005 17:48, Hal Rosenstock wrote:
> > > > Add new helper routines for allocating MADs for sending and formatting
> > > > a send WR.
> > > 
> > > > -- linux-2.6.13-rc2-mm1/drivers/infiniband2/core/mad.c
> > > > +++ linux-2.6.13-rc2-mm1/drivers/infiniband3/core/mad.c
> > > 				   ^^^^^^^^^^^
> > > Ick. You'd better have linux-2.6.13-rc2-mm1-[0123...].
> > 
> > Shall I resubmit with linux-2.6.13-rc2-mm1-[0123...] ?
> 
> I'd wait for comments and then resubmit clean series which can be applied
> with patch -p1.
> 
> > > > +struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
> > > > +					    u32 remote_qpn, u16 pkey_index,
> > > > +					    struct ib_ah *ah,
> > > > +					    int hdr_len, int data_len,
> > > > +					    int gfp_mask)
> > > 
> > > unsigned int __nocast gfp_mask, please. 430 or so infiniband sparse warnings
> > > is not a reason to add more.
> > 
> > I'll fix this in a subsequent patch. Is that OK ?
> 
> If Andrew will fix patches locally, OK.

I will work this into the resubmitted patches.

-- Hal



From jlentini at netapp.com  Mon Jul 11 10:21:26 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 11 Jul 2005 13:21:26 -0400 (EDT)
Subject: [openib-general] Kernel locking rules documentation
In-Reply-To: <42D2A231.1090808@ichips.intel.com>
References: <52fyuprqlf.fsf@topspin.com>
	<Pine.LNX.4.61.0507111218410.25603@jlentini-linux.nane.netapp.com>
	<42D2A231.1090808@ichips.intel.com>
Message-ID: <Pine.LNX.4.61.0507111303370.25603@jlentini-linux.nane.netapp.com>



On Mon, 11 Jul 2005, Sean Hefty wrote:

> James Lentini wrote:
>> 
>> Thanks for writing this up Roland. I would recommend sending it upstream.
>> 
>> Will this document be expanded to include the functions declared in 
>> ib_cm.h?
>
> I can either expand the documentation in ib_cm.h directly or add to this 
> file.  The CM seems separate from what's contained here, since it's not 
> expected that anyone will write another CM.

My interpretation of the document is that all IB verbs implementations 
will adhere to these rules, so the multiple verbs vs. single cm 
distinction wasn't as great in my mind.

If you combine the descriptions, ULP writers only need to read one 
piece of documentation to understand these synchronization issues.

I can understand why you would want to keep all of the CM 
documentation separate. That makes a lot of sense.


From krause at cup.hp.com  Mon Jul 11 09:16:53 2005
From: krause at cup.hp.com (Michael Krause)
Date: Mon, 11 Jul 2005 09:16:53 -0700
Subject: [openib-general] Re: IP addressing on InfiniBand networks
	(Caitlin Bestler)
In-Reply-To: <Pine.LNX.4.61.0507110933030.25603@jlentini-linux.nane.neta
	pp.com>
References: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com>
	<52r7ej68ga.fsf@topspin.com>
	<6.2.0.14.2.20050705093014.0291a9d0@esmail.cup.hp.com>
	<Pine.LNX.4.61.0507110933030.25603@jlentini-linux.nane.netapp.com>
Message-ID: <6.2.0.14.2.20050711090646.026aa9e8@esmail.cup.hp.com>

At 06:37 AM 7/11/2005, James Lentini wrote:


>On Tue, 5 Jul 2005, Michael Krause wrote:
>
>>The intention was to allow one to manage the fabric by having mapping 
>>functions from traditional IP management applications to IB GID to 
>>minimize the amount of work to enable IB within a solution.
>
>I was unaware of this. What happened to the mapping functions? What did 
>the API look like and how was it going to be implemented?

The IBTA specs are not API specifications.  They define semantics and wire 
protocols.  As for what was envisioned which guided the spec, most data 
center management applications understand a variety of management objects 
that represent various IP based attributes.  A GID is close enough to an 
IPv6 address that many of these objects could be easily modified via a 
plug-in such that the IB would have been easily slid into the associated 
management applications.  Investigations occurred, e.g. in to providing the 
necessary plug ins for OpenView, Tivoli, etc.  This is one area where IB is 
insufficient in terms of a viable ecosystem since there really isn't any 
way to manage IB in the enterprise without requiring significant amounts of 
training and a new tool chain to be deployed.

Mike

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/d08fccef/attachment.html>

From krause at cup.hp.com  Mon Jul 11 09:21:21 2005
From: krause at cup.hp.com (Michael Krause)
Date: Mon, 11 Jul 2005 09:21:21 -0700
Subject: [openib-general] vendor_id, vendor_part_id, hw_ver
In-Reply-To: <OFD5438117.E723E5B8-ON85257038.007F6D0A-85257038.0080400E@
	us.ibm.com>
References: <6.2.0.14.2.20050708143717.0209e738@esmail.cup.hp.com>
	<OFD5438117.E723E5B8-ON85257038.007F6D0A-85257038.0080400E@us.ibm.com>
Message-ID: <6.2.0.14.2.20050711091915.0263a5a8@esmail.cup.hp.com>

At 04:20 PM 7/8/2005, Kevin Reilly wrote:




>Mike,
>    Ideally your right the ULP shouldn't care what HCA it's running on. 
> There are some practical reasons why an ULP might want to know the vendor 
> and part number it was using like for debug or taking advantage a perform 
> nuance of a particular HCA.

Create a private interface since it is unique per HCA.  This type of 
information was rejected during the IB and iWARP verbs creation as 
something best handled out of band / out of spec.  Please do not push for 
this to be in a ULP itself or any standard RDMA API since it isn't required 
for all - most likely is very rare given it was rejected by all companies 
involved in creating these technologies to date.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/7e55d24a/attachment.html>

From Tom.Duffy at Sun.COM  Mon Jul 11 10:52:49 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 11 Jul 2005 10:52:49 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <200507111839.41807.adobriyan@gmail.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com>
Message-ID: <42D2B1F1.7000408@sun.com>

Alexey Dobriyan wrote:

>unsigned int __nocast gfp_mask, please. 430 or so infiniband sparse warnings
>is not a reason to add more.
>  
>
Can you please elaborate on the sparse warnings that you are seeing 
throughout the rest of infiniband?

Thanks,

-tduffy


From adobriyan at gmail.com  Mon Jul 11 11:38:27 2005
From: adobriyan at gmail.com (Alexey Dobriyan)
Date: Mon, 11 Jul 2005 22:38:27 +0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <42D2B1F1.7000408@sun.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com> <42D2B1F1.7000408@sun.com>
Message-ID: <200507112238.27856.adobriyan@gmail.com>

On Monday 11 July 2005 21:52, Tom Duffy wrote:
> Alexey Dobriyan wrote:
> 
> >unsigned int __nocast gfp_mask, please. 430 or so infiniband sparse warnings
> >is not a reason to add more.
> >  
> >
> Can you please elaborate on the sparse warnings that you are seeing 
> throughout the rest of infiniband?

$ make allmodconfig >/dev/null
$ make C=2 CHECK="sparse -Wbitwise" drivers/infiniband/ 2>&1 | tee ../W_infiniband
	[snip]
$ grep -c "warning: " ../W_infiniband
430


From surs at cse.ohio-state.edu  Mon Jul 11 12:03:10 2005
From: surs at cse.ohio-state.edu (Sayantan Sur)
Date: Mon, 11 Jul 2005 15:03:10 -0400
Subject: [openib-general] Gen2 build/update process
Message-ID: <20050711190308.GA7758@cse.ohio-state.edu>

Hi,

Is there any plan to have any formal install procedure instead of the
current process of going into subdirectories and building?

If someone has build scripts to share, that'll be real nice.

TIA,
Sayantan.


-- 
http://www.cse.ohio-state.edu/~surs


From hermannv at supermicro.com  Mon Jul 11 12:07:01 2005
From: hermannv at supermicro.com (Hermann von Drateln)
Date: Mon, 11 Jul 2005 12:07:01 -0700
Subject: [openib-general] RE: Infiniband embedded dual Xeon 800 MHz MBD
Message-ID: <!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAE4LjNMw16EaNXEizozGxbcKAAAAQAAAA9g3WQ/f3hkCyA1px0KEs5wEAAAAA@supermicro.com>

Attention Open IB member:

 

            For the members that received the technical information for this
12*10 inch MBD that supports dual socket for Xeon with embedded Infiniband.

 

            This MBD will be release in two models!

 

            Version for Irwindale

 

            And Version for Paxville.

 

            Please send me feedback on your preference.

 

            Also take in consideration that the second Paxville support MBD
would also work with Nocona, and Irwindale!

 

            It is also critical to know you volume requirement of each MBD.

 

            For Paxville only the CSE- 811T-420 Special edition for embedded
Infiniband will support this MBD

 

            http://www.supermicro.com/products/chassis/1U/811/SC811T-420.cfm

 

 

            Thank you in advance for your feedback.

 

Hermann von Drateln

Director Business Development

  

  USA TEL  1 408  503 8110

  USA CEL  1 408 306 8110

 

   Eu Tel  + 49 173 286 6883

   Eu Fax + 49 69 255 77303

 



  _____  

From: Hermann von Drateln [mailto:hermannv at supermicro.com] 
Sent: Monday, April 18, 2005 6:38 PM
To: 'openib-general at openib.org'
Subject: Infiniband embedded dual Xeon 800 MHz MBD
Importance: High

 

To all Open IB Members!

 

This week we are commencing the verification and validation of our Dual Xeon
800 MHz FSB MBD.

 

The board is very similar to our 

 

http://www.supermicro.com/products/motherboard/Xeon800/E7320/X6DVA-EG.cfm

 

And it made to be fitted on our  CSE-513

 

http://www.supermicro.com/products/chassis/1U/?chs=513

 

For any one that would like to obtain more detail information on this new
board and our new product road map send me an email to provide you with
Power point presentation if information.

 

 

Best regards,

 

 

Hermann von Drateln

Director Business Development

  

  USA TEL  1 408  503 8110

  USA CEL  1 408 306 8110

 

   Eu Tel  + 49 173 286 6883

   Eu Fax + 49 69 255 77303

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/41af9e88/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.gif
Type: image/gif
Size: 52199 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/41af9e88/attachment.gif>

From Tom.Duffy at Sun.COM  Mon Jul 11 12:08:21 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 11 Jul 2005 12:08:21 -0700
Subject: [openib-general] OpenIB DAT_Return merging and kdat.h
In-Reply-To: <8508251A6FC08A489844A94261D3693A057D0D@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057D0D@fiona.siliquent.com>
Message-ID: <42D2C3A5.9010104@sun.com>

Caitlin Bestler wrote:

>Given both of those factors, I believe it makes more sense
>to preserve the full DAT_RETURN value (including detail error
>data) as an out parameter. kdat.h could then simply turn it 
>into the functional return for compatability clients and there
>would be no loss of information. Either that or we could work
>out on a case by case basis how kdat.h would be able to restore
>the original error value. This may be possible with data in
>the struct, since multiple outstanding errors on a single 
>object would not seem to be likely (or perhaps more appropriately
>worthy of providing detailed error diagnostics for, since it
>probably represents a totally hosed application).
>  
>

If we decide to add in a compatibility layer with a dat_return 
*parameter, or some other mechanism, this should be a compile time option.

-tduffy


From Tom.Duffy at Sun.COM  Mon Jul 11 12:10:48 2005
From: Tom.Duffy at Sun.COM (Tom Duffy)
Date: Mon, 11 Jul 2005 12:10:48 -0700
Subject: [openib-general] Gen2 build/update process
In-Reply-To: <20050711190308.GA7758@cse.ohio-state.edu>
References: <20050711190308.GA7758@cse.ohio-state.edu>
Message-ID: <42D2C438.3020707@sun.com>

Sayantan Sur wrote:

>Hi,
>
>Is there any plan to have any formal install procedure instead of the
>current process of going into subdirectories and building?
>
>If someone has build scripts to share, that'll be real nice.
>  
>

What are you trying to build?  Kernel build should not need to do this.

-tduffy


From halr at voltaire.com  Mon Jul 11 12:30:50 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:30:50 -0400
Subject: [openib-general] [PATCH 0/29v2] InfiniBand core update
Message-ID: <1121110249.4389.4984.camel@hal.voltaire.com>

This is version 2 of a patch series to get the Infiniband core up to
date. Aside from bug fixes, the following new functionality is also
introduced: 

	MAD (Management Datagram) support for RMPP
			(Reliable MultiPacket Protocol)
	CM (Communications Manager) support
	User CM support
	User MAD changes for RMPP
	Service Record support in SA query

Thanks.

-- Hal



From halr at voltaire.com  Mon Jul 11 12:34:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:34:10 -0400
Subject: [openib-general] [PATCH 1/29v2] Update FMR functions
Message-ID: <1121110253.4389.4986.camel@hal.voltaire.com>

Change some functions to return void rather than an int since they are
always returning 0, thus making checking return values rather pointless.

Signed-off-by: Tom Duffy <tduffy at sun.com>
Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

--- 
 core/fmr_pool.c       |    7 +++--
 include/ib_fmr_pool.h |    5 +++--
 2 files changed, 6 insertions(+), 6 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband/core/fmr_pool.c linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/fmr_pool.c
-- linux-2.6.13-rc2-mm1/drivers/infiniband/core/fmr_pool.c	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/fmr_pool.c	2005-06-28 13:07:35.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: fmr_pool.c 1349 2004-12-16 21:09:43Z roland $
+ * $Id: fmr_pool.c 2730 2005-06-28 16:43:03Z sean.hefty $
  */
 
 #include <linux/errno.h>
@@ -329,7 +330,7 @@ EXPORT_SYMBOL(ib_create_fmr_pool);
  *
  * Destroy an FMR pool and free all associated resources.
  */
-int ib_destroy_fmr_pool(struct ib_fmr_pool *pool)
+void ib_destroy_fmr_pool(struct ib_fmr_pool *pool)
 {
 	struct ib_pool_fmr *fmr;
 	struct ib_pool_fmr *tmp;
@@ -352,8 +353,6 @@ int ib_destroy_fmr_pool(struct ib_fmr_po
 
 	kfree(pool->cache_bucket);
 	kfree(pool);
-
-	return 0;
 }
 EXPORT_SYMBOL(ib_destroy_fmr_pool);
 
diff -uprN linux-2.6.13-rc2-mm1/drivers/infiniband/include/ib_fmr_pool.h linux-2.6.13-rc2-mm1-1/drivers/infiniband/include/ib_fmr_pool.h
-- linux-2.6.13-rc2-mm1/drivers/infiniband/include/ib_fmr_pool.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-1/drivers/infiniband/include/ib_fmr_pool.h	2005-06-28 13:07:34.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_fmr_pool.h 1349 2004-12-16 21:09:43Z roland $
+ * $Id: ib_fmr_pool.h 2730 2005-06-28 16:43:03Z sean.hefty $
  */
 
 #if !defined(IB_FMR_POOL_H)
@@ -78,7 +79,7 @@ struct ib_pool_fmr {
 struct ib_fmr_pool *ib_create_fmr_pool(struct ib_pd             *pd,
 				       struct ib_fmr_pool_param *params);
 
-int ib_destroy_fmr_pool(struct ib_fmr_pool *pool);
+void ib_destroy_fmr_pool(struct ib_fmr_pool *pool);
 
 int ib_flush_fmr_pool(struct ib_fmr_pool *pool);
 




From halr at voltaire.com  Mon Jul 11 12:37:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:37:25 -0400
Subject: [openib-general] [PATCH 2/29v2] Update MAD client API
Message-ID: <1121110257.4389.4988.camel@hal.voltaire.com>

Automatically allocate a MR when registering a MAD agent. 
MAD clients are modified to use this updated API.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

--
 core/agent.c      |   16 +--
 core/agent_priv.h |    3 +--
 core/mad.c        |   31 +++++++++++++++++++--
 core/sa_query.c   |   15 ++--
 include/ib_mad.h  |    1 +
 5 files changed, 24 insertions(+), 42 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/agent.c linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/agent.c
-- linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/agent.c	2005-07-08 14:12:17.000000000 -0400
+++ linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/agent.c	2005-07-09 13:22:55.000000000 -0400
@@ -129,13 +129,12 @@ static int agent_mad_send(struct ib_mad_
 		goto out;
 	agent_send_wr->mad = mad_priv;
 
-	/* PCI mapping */
 	gather_list.addr = dma_map_single(mad_agent->device->dma_device,
 					  &mad_priv->mad,
 					  sizeof(mad_priv->mad),
 					  DMA_TO_DEVICE);
 	gather_list.length = sizeof(mad_priv->mad);
-	gather_list.lkey = (*port_priv->mr).lkey;
+	gather_list.lkey = mad_agent->mr->lkey;
 
 	send_wr.next = NULL;
 	send_wr.opcode = IB_WR_SEND;
@@ -261,7 +260,6 @@ static void agent_send_handler(struct ib
 	list_del(&agent_send_wr->send_list);
 	spin_unlock_irqrestore(&port_priv->send_list_lock, flags);
 
-	/* Unmap PCI */
 	dma_unmap_single(mad_agent->device->dma_device,
 			 pci_unmap_addr(agent_send_wr, mapping),
 			 sizeof(agent_send_wr->mad->mad),
@@ -324,22 +322,12 @@ int ib_agent_port_open(struct ib_device 
 		goto error3;
 	}
 
-	port_priv->mr = ib_get_dma_mr(port_priv->smp_agent->qp->pd,
-				      IB_ACCESS_LOCAL_WRITE);
-	if (IS_ERR(port_priv->mr)) {
-		printk(KERN_ERR SPFX "Couldn't get DMA MR\n");
-		ret = PTR_ERR(port_priv->mr);
-		goto error4;
-	}
-
 	spin_lock_irqsave(&ib_agent_port_list_lock, flags);
 	list_add_tail(&port_priv->port_list, &ib_agent_port_list);
 	spin_unlock_irqrestore(&ib_agent_port_list_lock, flags);
 
 	return 0;
 
-error4:
-	ib_unregister_mad_agent(port_priv->perf_mgmt_agent);
 error3:
 	ib_unregister_mad_agent(port_priv->smp_agent);
 error2:
@@ -363,8 +351,6 @@ int ib_agent_port_close(struct ib_device
 	list_del(&port_priv->port_list);
 	spin_unlock_irqrestore(&ib_agent_port_list_lock, flags);
 
-	ib_dereg_mr(port_priv->mr);
-
 	ib_unregister_mad_agent(port_priv->perf_mgmt_agent);
 	ib_unregister_mad_agent(port_priv->smp_agent);
 	kfree(port_priv);
diff -uprN linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/agent_priv.h linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/agent_priv.h
-- linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/agent_priv.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/agent_priv.h	2005-07-09 12:14:49.000000000 -0400
@@ -33,7 +33,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: agent_priv.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: agent_priv.h 1640 2005-01-24 22:39:02Z halr $
  */
 
 #ifndef __IB_AGENT_PRIV_H__
@@ -57,7 +57,6 @@ struct ib_agent_port_private {
 	int port_num;
 	struct ib_mad_agent *smp_agent;	      /* SM class */
 	struct ib_mad_agent *perf_mgmt_agent; /* PerfMgmt class */
-	struct ib_mr *mr;
 };
 
 #endif	/* __IB_AGENT_PRIV_H__ */
diff -uprN linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/mad.c	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/mad.c	2005-07-09 12:58:23.000000000 -0400
@@ -261,19 +261,26 @@ struct ib_mad_agent *ib_register_mad_age
 		ret = ERR_PTR(-ENOMEM);
 		goto error1;
 	}
+	memset(mad_agent_priv, 0, sizeof *mad_agent_priv);
+
+	mad_agent_priv->agent.mr = ib_get_dma_mr(port_priv->qp_info[qpn].qp->pd,
+						 IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(mad_agent_priv->agent.mr)) {
+		ret = ERR_PTR(-ENOMEM);
+		goto error2;
+	}
 
 	if (mad_reg_req) {
 		reg_req = kmalloc(sizeof *reg_req, GFP_KERNEL);
 		if (!reg_req) {
 			ret = ERR_PTR(-ENOMEM);
-			goto error2;
+			goto error3;
 		}
 		/* Make a copy of the MAD registration request */
 		memcpy(reg_req, mad_reg_req, sizeof *reg_req);
 	}
 
 	/* Now, fill in the various structures */
-	memset(mad_agent_priv, 0, sizeof *mad_agent_priv);
 	mad_agent_priv->qp_info = &port_priv->qp_info[qpn];
 	mad_agent_priv->reg_req = reg_req;
 	mad_agent_priv->rmpp_version = rmpp_version;
@@ -301,7 +308,7 @@ struct ib_mad_agent *ib_register_mad_age
 				if (method) {
 					if (method_in_use(&method,
 							   mad_reg_req))
-						goto error3;
+						goto error4;
 				}
 			}
 			ret2 = add_nonoui_reg_req(mad_reg_req, mad_agent_priv,
@@ -317,14 +324,14 @@ struct ib_mad_agent *ib_register_mad_age
 					if (is_vendor_method_in_use(
 							vendor_class,
 							mad_reg_req))
-						goto error3;
+						goto error4;
 				}
 			}
 			ret2 = add_oui_reg_req(mad_reg_req, mad_agent_priv);
 		}
 		if (ret2) {
 			ret = ERR_PTR(ret2);
-			goto error3;
+			goto error4;
 		}
 	}
 
@@ -346,11 +353,13 @@ struct ib_mad_agent *ib_register_mad_age
 
 	return &mad_agent_priv->agent;
 
-error3:
+error4:
 	spin_unlock_irqrestore(&port_priv->reg_lock, flags);
 	kfree(reg_req);
-error2:
+error3:
 	kfree(mad_agent_priv);
+error2:
+	ib_dereg_mr(mad_agent_priv->agent.mr);
 error1:
 	return ret;
 }
@@ -487,18 +496,15 @@ static void unregister_mad_agent(struct 
 	 * MADs, preventing us from queuing additional work
 	 */
 	cancel_mads(mad_agent_priv);
-
 	port_priv = mad_agent_priv->qp_info->port_priv;
-
 	cancel_delayed_work(&mad_agent_priv->timed_work);
-	flush_workqueue(port_priv->wq);
 
 	spin_lock_irqsave(&port_priv->reg_lock, flags);
 	remove_mad_reg_req(mad_agent_priv);
 	list_del(&mad_agent_priv->agent_list);
 	spin_unlock_irqrestore(&port_priv->reg_lock, flags);
 
-	/* XXX: Cleanup pending RMPP receives for this agent */
+	flush_workqueue(port_priv->wq);
 
 	atomic_dec(&mad_agent_priv->refcount);
 	wait_event(mad_agent_priv->wait,
@@ -506,6 +512,7 @@ static void unregister_mad_agent(struct 
 
 	if (mad_agent_priv->reg_req)
 		kfree(mad_agent_priv->reg_req);
+	ib_dereg_mr(mad_agent_priv->agent.mr);
 	kfree(mad_agent_priv);
 }
 
@@ -750,7 +757,7 @@ static int handle_outgoing_dr_smp(struct
 	list_add_tail(&local->completion_list, &mad_agent_priv->local_list);
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 	queue_work(mad_agent_priv->qp_info->port_priv->wq,
-		  &mad_agent_priv->local_work);
+		   &mad_agent_priv->local_work);
 	ret = 1;
 out:
 	return ret;
diff -uprN linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/sa_query.c linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/sa_query.c
-- linux-2.6.13-rc2-mm1-1/drivers/infiniband/core/sa_query.c	2005-07-10 16:21:40.000000000 -0400
+++ linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/sa_query.c	2005-07-10 16:21:55.000000000 -0400
@@ -77,7 +77,6 @@ struct ib_sa_sm_ah {
 
 struct ib_sa_port {
 	struct ib_mad_agent *agent;
-	struct ib_mr        *mr;
 	struct ib_sa_sm_ah  *sm_ah;
 	struct work_struct   update_task;
 	spinlock_t           ah_lock;
@@ -492,7 +491,7 @@ retry:
 					    sizeof (struct ib_sa_mad),
 					    DMA_TO_DEVICE);
 	gather_list.length = sizeof (struct ib_sa_mad);
-	gather_list.lkey   = port->mr->lkey;
+	gather_list.lkey   = port->agent->mr->lkey;
 	pci_unmap_addr_set(query, mapping, gather_list.addr);
 
 	ret = ib_post_send_mad(port->agent, &wr, &bad_wr);
@@ -780,7 +779,6 @@ static void ib_sa_add_one(struct ib_devi
 	sa_dev->end_port   = e;
 
 	for (i = 0; i <= e - s; ++i) {
-		sa_dev->port[i].mr       = NULL;
 		sa_dev->port[i].sm_ah    = NULL;
 		sa_dev->port[i].port_num = i + s;
 		spin_lock_init(&sa_dev->port[i].ah_lock);
@@ -792,13 +790,6 @@ static void ib_sa_add_one(struct ib_devi
 		if (IS_ERR(sa_dev->port[i].agent))
 			goto err;
 
-		sa_dev->port[i].mr = ib_get_dma_mr(sa_dev->port[i].agent->qp->pd,
-						   IB_ACCESS_LOCAL_WRITE);
-		if (IS_ERR(sa_dev->port[i].mr)) {
-			ib_unregister_mad_agent(sa_dev->port[i].agent);
-			goto err;
-		}
-
 		INIT_WORK(&sa_dev->port[i].update_task,
 			  update_sm_ah, &sa_dev->port[i]);
 	}
@@ -822,10 +813,8 @@ static void ib_sa_add_one(struct ib_devi
 	return;
 
 err:
-	while (--i >= 0) {
-		ib_dereg_mr(sa_dev->port[i].mr);
+	while (--i >= 0)
 		ib_unregister_mad_agent(sa_dev->port[i].agent);
-	}
 
 	kfree(sa_dev);
 
diff -uprN linux-2.6.13-rc2-mm1-1/drivers/infiniband/include/ib_mad.h linux-2.6.13-rc2-mm1-2/drivers/infiniband/include/ib_mad.h
-- linux-2.6.13-rc2-mm1-1/drivers/infiniband/include/ib_mad.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-2/drivers/infiniband/include/ib_mad.h	2005-07-09 12:06:49.000000000 -0400
@@ -180,6 +180,7 @@ typedef void (*ib_mad_recv_handler)(stru
 struct ib_mad_agent {
 	struct ib_device	*device;
 	struct ib_qp		*qp;
+	struct ib_mr		*mr;
 	ib_mad_recv_handler	recv_handler;
 	ib_mad_send_handler	send_handler;
 	ib_mad_snoop_handler	snoop_handler;




From surs at cse.ohio-state.edu  Mon Jul 11 12:46:59 2005
From: surs at cse.ohio-state.edu (Sayantan Sur)
Date: Mon, 11 Jul 2005 15:46:59 -0400
Subject: [openib-general] Gen2 build/update process
In-Reply-To: <42D2C438.3020707@sun.com>
References: <20050711190308.GA7758@cse.ohio-state.edu>
	<42D2C438.3020707@sun.com>
Message-ID: <20050711194659.GA7892@cse.ohio-state.edu>

* On Jul,2 Tom Duffy<Tom.Duffy at Sun.COM> wrote :
> Sayantan Sur wrote:
> 
> >Hi,
> >
> >Is there any plan to have any formal install procedure instead of the
> >current process of going into subdirectories and building?
> >
> >If someone has build scripts to share, that'll be real nice.
> > 
> >
> 
> What are you trying to build?  Kernel build should not need to do this.

I am building the user level support. Currently, that requires manual
autogen, config, make, make install in lots of subdirectories. I was
wondering if someone has a script to automate the build process. This
will also ensure that new adopters don't get any dependency wrong.


Thanks,
Sayantan.


> 
> -tduffy

-- 
http://www.cse.ohio-state.edu/~surs


From halr at voltaire.com  Mon Jul 11 12:40:38 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:40:38 -0400
Subject: [openib-general] [PATCH 3/29v2] Add MAD helper functions
Message-ID: <1121110260.4389.4990.camel@hal.voltaire.com>

Add new helper routines for allocating MADs for sending and formatting 
a send WR.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 2/29.

--
 core/mad.c       |   76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/ib_mad.h |   60 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 136 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-3/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-2/drivers/infiniband/core/mad.c	2005-07-09 12:58:23.000000000 -0400
+++ linux-2.6.13-rc2-mm1-3/drivers/infiniband/core/mad.c	2005-07-11 13:34:57.000000000 -0400
@@ -763,6 +763,82 @@ out:
 	return ret;
 }
 
+static int get_buf_length(int hdr_len, int data_len)
+{
+	int seg_size, pad;
+
+	seg_size = sizeof(struct ib_mad) - hdr_len;
+	if (data_len && seg_size) {
+		pad = seg_size - data_len % seg_size;
+		if (pad == seg_size)
+			pad = 0;
+	} else
+		pad = seg_size;
+	return hdr_len + data_len + pad;
+}
+
+struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
+					    u32 remote_qpn, u16 pkey_index,
+					    struct ib_ah *ah,
+					    int hdr_len, int data_len,
+					    unsigned int __nocast gfp_mask)
+{
+	struct ib_mad_agent_private *mad_agent_priv;
+	struct ib_mad_send_buf *send_buf;
+	int buf_size;
+	void *buf;
+
+	mad_agent_priv = container_of(mad_agent,
+				      struct ib_mad_agent_private, agent);
+	buf_size = get_buf_length(hdr_len, data_len);
+
+	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);
+	if (!buf)
+		return ERR_PTR(-ENOMEM);
+
+	send_buf = buf + buf_size;
+	memset(send_buf, 0, sizeof *send_buf);
+	send_buf->mad = buf;
+
+	send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device,
+					    buf, buf_size, DMA_TO_DEVICE);
+	pci_unmap_addr_set(send_buf, mapping, send_buf->sge.addr);
+	send_buf->sge.length = buf_size;
+	send_buf->sge.lkey = mad_agent->mr->lkey;
+
+	send_buf->send_wr.wr_id = (unsigned long) send_buf;
+	send_buf->send_wr.sg_list = &send_buf->sge;
+	send_buf->send_wr.num_sge = 1;
+	send_buf->send_wr.opcode = IB_WR_SEND;
+	send_buf->send_wr.send_flags = IB_SEND_SIGNALED;
+	send_buf->send_wr.wr.ud.ah = ah;
+	send_buf->send_wr.wr.ud.mad_hdr = &send_buf->mad->mad_hdr;
+	send_buf->send_wr.wr.ud.remote_qpn = remote_qpn;
+	send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY;
+	send_buf->send_wr.wr.ud.pkey_index = pkey_index;
+	send_buf->mad_agent = mad_agent;
+	atomic_inc(&mad_agent_priv->refcount);
+	return send_buf;
+}
+EXPORT_SYMBOL(ib_create_send_mad);
+
+void ib_free_send_mad(struct ib_mad_send_buf *send_buf)
+{
+	struct ib_mad_agent_private *mad_agent_priv;
+
+	mad_agent_priv = container_of(send_buf->mad_agent,
+				      struct ib_mad_agent_private, agent);
+
+	dma_unmap_single(send_buf->mad_agent->device->dma_device,
+			 pci_unmap_addr(send_buf, mapping),
+			 send_buf->sge.length, DMA_TO_DEVICE);
+	kfree(send_buf->mad);
+
+	if (atomic_dec_and_test(&mad_agent_priv->refcount))
+		wake_up(&mad_agent_priv->wait);
+}
+EXPORT_SYMBOL(ib_free_send_mad);
+
 static int ib_send_mad(struct ib_mad_agent_private *mad_agent_priv,
 		       struct ib_mad_send_wr_private *mad_send_wr)
 {
diff -uprN linux-2.6.13-rc2-mm1-2/drivers/infiniband/include/ib_mad.h linux-2.6.13-rc2-mm1-3/drivers/infiniband/include/ib_mad.h
-- linux-2.6.13-rc2-mm1-2/drivers/infiniband/include/ib_mad.h	2005-07-09 12:06:49.000000000 -0400
+++ linux-2.6.13-rc2-mm1-3/drivers/infiniband/include/ib_mad.h	2005-07-11 13:34:54.000000000 -0400
@@ -39,6 +39,8 @@
 #if !defined( IB_MAD_H )
 #define IB_MAD_H
 
+#include <linux/pci.h>
+
 #include <ib_verbs.h>
 
 /* Management base version */
@@ -73,6 +75,7 @@
 #define IB_QP0		0
 #define IB_QP1		__constant_htonl(1)
 #define IB_QP1_QKEY	0x80010000
+#define IB_QP_SET_QKEY	0x80000000
 
 struct ib_grh {
 	u32		version_tclass_flow;
@@ -124,6 +127,30 @@ struct ib_vendor_mad {
 	u8			data[216];
 } __attribute__ ((packed));
 
+/**
+ * ib_mad_send_buf - MAD data buffer and work request for sends.
+ * @mad: References an allocated MAD data buffer.  The size of the data
+ *   buffer is specified in the @send_wr.length field.
+ * @mapping: DMA mapping information.
+ * @mad_agent: MAD agent that allocated the buffer.
+ * @context: User-controlled context fields.
+ * @send_wr: An initialized work request structure used when sending the MAD.
+ *   The wr_id field of the work request is initialized to reference this
+ *   data structure.
+ * @sge: A scatter-gather list referenced by the work request.
+ *
+ * Users are responsible for initializing the MAD buffer itself, with the
+ * exception of specifying the payload length field in any RMPP MAD.
+ */
+struct ib_mad_send_buf {
+	struct ib_mad		*mad;
+	DECLARE_PCI_UNMAP_ADDR(mapping)
+	struct ib_mad_agent	*mad_agent;
+	void			*context[2];
+	struct ib_send_wr	send_wr;
+	struct ib_sge		sge;
+};
+
 struct ib_mad_agent;
 struct ib_mad_send_wc;
 struct ib_mad_recv_wc;
@@ -402,4 +429,37 @@ struct ib_mad_agent *ib_redirect_mad_qp(
 int ib_process_mad_wc(struct ib_mad_agent *mad_agent,
 		      struct ib_wc *wc);
 
+/**
+ * ib_create_send_mad - Allocate and initialize a data buffer and work request
+ *   for sending a MAD.
+ * @mad_agent: Specifies the registered MAD service to associate with the MAD.
+ * @remote_qpn: Specifies the QPN of the receiving node.
+ * @pkey_index: Specifies which PKey the MAD will be sent using.  This field
+ *   is valid only if the remote_qpn is QP 1.
+ * @ah: References the address handle used to transfer to the remote node.
+ * @hdr_len: Indicates the size of the data header of the MAD.  This length
+ *   should include the common MAD header, RMPP header, plus any class
+ *   specific header.
+ * @data_len: Indicates the size of any user-transfered data.  The call will
+ *   automatically adjust the allocated buffer size to account for any
+ *   additional padding that may be necessary.
+ * @gfp_mask: GFP mask used for the memory allocation.
+ *
+ * This is a helper routine that may be used to allocate a MAD.  Users are
+ * not required to allocate outbound MADs using this call.  The returned
+ * MAD send buffer will reference a data buffer usable for sending a MAD, along
+ * with an intialized work request structure.
+ */
+struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
+					    u32 remote_qpn, u16 pkey_index,
+					    struct ib_ah *ah,
+					    int hdr_len, int data_len,
+					    unsigned int __nocast gfp_mask);
+
+/**
+ * ib_free_send_mad - Returns data buffers used to send a MAD.
+ * @send_buf: Previously allocated send data buffer.
+ */
+void ib_free_send_mad(struct ib_mad_send_buf *send_buf);
+
 #endif /* IB_MAD_H */




From halr at voltaire.com  Mon Jul 11 12:43:52 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:43:52 -0400
Subject: [openib-general] [PATCH 4/29v2] Combine some MAD routines
Message-ID: <1121110267.4389.4992.camel@hal.voltaire.com>

Combine response_mad() and solicited_mad() routines into a single 
function and simplify/encapsulate its usage.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 3/29.

-- 
 mad.c |  105 ++++++++++++++++--
 1 files changed, 27 insertions(+), 78 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-3/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-4/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-3/drivers/infiniband/core/mad.c	2005-07-11 13:34:57.000000000 -0400
+++ linux-2.6.13-rc2-mm1-4/drivers/infiniband/core/mad.c	2005-07-11 13:35:26.000000000 -0400
@@ -58,7 +58,7 @@ static int method_in_use(struct ib_mad_m
 static void remove_mad_reg_req(struct ib_mad_agent_private *priv);
 static struct ib_mad_agent_private *find_mad_agent(
 					struct ib_mad_port_private *port_priv,
-					struct ib_mad *mad, int solicited);
+					struct ib_mad *mad);
 static int ib_mad_post_receive_mads(struct ib_mad_qp_info *qp_info,
 				    struct ib_mad_private *mad);
 static void cancel_mads(struct ib_mad_agent_private *mad_agent_priv);
@@ -67,7 +67,6 @@ static void ib_mad_complete_send_wr(stru
 static void timeout_sends(void *data);
 static void cancel_sends(void *data);
 static void local_completions(void *data);
-static int solicited_mad(struct ib_mad *mad);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
 			      struct ib_mad_agent_private *agent_priv,
 			      u8 mgmt_class);
@@ -558,6 +557,13 @@ int ib_unregister_mad_agent(struct ib_ma
 }
 EXPORT_SYMBOL(ib_unregister_mad_agent);
 
+static inline int response_mad(struct ib_mad *mad)
+{
+	/* Trap represses are responses although response bit is reset */
+	return ((mad->mad_hdr.method == IB_MGMT_METHOD_TRAP_REPRESS) ||
+		(mad->mad_hdr.method & IB_MGMT_METHOD_RESP));
+}
+
 static void dequeue_mad(struct ib_mad_list_head *mad_list)
 {
 	struct ib_mad_queue *mad_queue;
@@ -650,7 +656,7 @@ static int handle_outgoing_dr_smp(struct
 				  struct ib_smp *smp,
 				  struct ib_send_wr *send_wr)
 {
-	int ret, solicited;
+	int ret;
 	unsigned long flags;
 	struct ib_mad_local_private *local;
 	struct ib_mad_private *mad_priv;
@@ -696,11 +702,7 @@ static int handle_outgoing_dr_smp(struct
 	switch (ret)
 	{
 	case IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY:
-		/*
-		 * See if response is solicited and
-		 * there is a recv handler
-		 */
-		if (solicited_mad(&mad_priv->mad.mad) &&
+		if (response_mad(&mad_priv->mad.mad) &&
 		    mad_agent_priv->agent.recv_handler) {
 			local->mad_priv = mad_priv;
 			local->recv_mad_agent = mad_agent_priv;
@@ -717,15 +719,13 @@ static int handle_outgoing_dr_smp(struct
 		break;
 	case IB_MAD_RESULT_SUCCESS:
 		/* Treat like an incoming receive MAD */
-		solicited = solicited_mad(&mad_priv->mad.mad);
 		port_priv = ib_get_mad_port(mad_agent_priv->agent.device,
 					    mad_agent_priv->agent.port_num);
 		if (port_priv) {
 			mad_priv->mad.mad.mad_hdr.tid =
 				((struct ib_mad *)smp)->mad_hdr.tid;
 			recv_mad_agent = find_mad_agent(port_priv,
-						       &mad_priv->mad.mad,
-							solicited);
+						        &mad_priv->mad.mad);
 		}
 		if (!port_priv || !recv_mad_agent) {
 			kmem_cache_free(ib_mad_cache, mad_priv);
@@ -1421,42 +1421,15 @@ out:
 	return;
 }
 
-static int response_mad(struct ib_mad *mad)
-{
-	/* Trap represses are responses although response bit is reset */
-	return ((mad->mad_hdr.method == IB_MGMT_METHOD_TRAP_REPRESS) ||
-		(mad->mad_hdr.method & IB_MGMT_METHOD_RESP));
-}
-
-static int solicited_mad(struct ib_mad *mad)
-{
-	/* CM MADs are never solicited */
-	if (mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_CM) {
-		return 0;
-	}
-
-	/* XXX: Determine whether MAD is using RMPP */
-
-	/* Not using RMPP */
-	/* Is this MAD a response to a previous MAD ? */
-	return response_mad(mad);
-}
-
 static struct ib_mad_agent_private *
 find_mad_agent(struct ib_mad_port_private *port_priv,
-	       struct ib_mad *mad,
-	       int solicited)
+	       struct ib_mad *mad)
 {
 	struct ib_mad_agent_private *mad_agent = NULL;
 	unsigned long flags;
 
 	spin_lock_irqsave(&port_priv->reg_lock, flags);
-
-	/*
-	 * Whether MAD was solicited determines type of routing to
-	 * MAD client.
-	 */
-	if (solicited) {
+	if (response_mad(mad)) {
 		u32 hi_tid;
 		struct ib_mad_agent_private *entry;
 
@@ -1560,18 +1533,6 @@ out:
 	return valid;
 }
 
-/*
- * Return start of fully reassembled MAD, or NULL, if MAD isn't assembled yet
- */
-static struct ib_mad_private *
-reassemble_recv(struct ib_mad_agent_private *mad_agent_priv,
-		struct ib_mad_private *recv)
-{
-	/* Until we have RMPP, all receives are reassembled!... */
-	INIT_LIST_HEAD(&recv->header.recv_wc.recv_buf.list);
-	return recv;
-}
-
 static struct ib_mad_send_wr_private*
 find_send_req(struct ib_mad_agent_private *mad_agent_priv,
 	      u64 tid)
@@ -1600,29 +1561,22 @@ find_send_req(struct ib_mad_agent_privat
 }
 
 static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
-				 struct ib_mad_private *recv,
-				 int solicited)
+				 struct ib_mad_recv_wc *mad_recv_wc)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 	struct ib_mad_send_wc mad_send_wc;
 	unsigned long flags;
+	u64 tid;
 
-	/* Fully reassemble receive before processing */
-	recv = reassemble_recv(mad_agent_priv, recv);
-	if (!recv) {
-		if (atomic_dec_and_test(&mad_agent_priv->refcount))
-			wake_up(&mad_agent_priv->wait);
-		return;
-	}
-
+	INIT_LIST_HEAD(&mad_recv_wc->recv_buf.list);
 	/* Complete corresponding request */
-	if (solicited) {
+	if (response_mad(mad_recv_wc->recv_buf.mad)) {
+		tid = mad_recv_wc->recv_buf.mad->mad_hdr.tid;
 		spin_lock_irqsave(&mad_agent_priv->lock, flags);
-		mad_send_wr = find_send_req(mad_agent_priv,
-					    recv->mad.mad.mad_hdr.tid);
+		mad_send_wr = find_send_req(mad_agent_priv, tid);
 		if (!mad_send_wr) {
 			spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-			ib_free_recv_mad(&recv->header.recv_wc);
+			ib_free_recv_mad(mad_recv_wc);
 			if (atomic_dec_and_test(&mad_agent_priv->refcount))
 				wake_up(&mad_agent_priv->wait);
 			return;
@@ -1632,10 +1586,9 @@ static void ib_mad_complete_recv(struct 
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		/* Defined behavior is to complete response before request */
-		recv->header.recv_wc.wc->wr_id = mad_send_wr->wr_id;
-		mad_agent_priv->agent.recv_handler(
-						&mad_agent_priv->agent,
-						&recv->header.recv_wc);
+		mad_recv_wc->wc->wr_id = mad_send_wr->wr_id;
+		mad_agent_priv->agent.recv_handler(&mad_agent_priv->agent,
+						   mad_recv_wc);
 		atomic_dec(&mad_agent_priv->refcount);
 
 		mad_send_wc.status = IB_WC_SUCCESS;
@@ -1643,9 +1596,8 @@ static void ib_mad_complete_recv(struct 
 		mad_send_wc.wr_id = mad_send_wr->wr_id;
 		ib_mad_complete_send_wr(mad_send_wr, &mad_send_wc);
 	} else {
-		mad_agent_priv->agent.recv_handler(
-						&mad_agent_priv->agent,
-						&recv->header.recv_wc);
+		mad_agent_priv->agent.recv_handler(&mad_agent_priv->agent,
+						   mad_recv_wc);
 		if (atomic_dec_and_test(&mad_agent_priv->refcount))
 			wake_up(&mad_agent_priv->wait);
 	}
@@ -1659,7 +1611,6 @@ static void ib_mad_recv_done_handler(str
 	struct ib_mad_private *recv, *response;
 	struct ib_mad_list_head *mad_list;
 	struct ib_mad_agent_private *mad_agent;
-	int solicited;
 
 	response = kmem_cache_alloc(ib_mad_cache, GFP_KERNEL);
 	if (!response)
@@ -1745,11 +1696,9 @@ local:
 		}
 	}
 
-	/* Determine corresponding MAD agent for incoming receive MAD */
-	solicited = solicited_mad(&recv->mad.mad);
-	mad_agent = find_mad_agent(port_priv, &recv->mad.mad, solicited);
+	mad_agent = find_mad_agent(port_priv, &recv->mad.mad);
 	if (mad_agent) {
-		ib_mad_complete_recv(mad_agent, recv, solicited);
+		ib_mad_complete_recv(mad_agent, &recv->header.recv_wc);
 		/*
 		 * recv is freed up in error cases in ib_mad_complete_recv
 		 * or via recv_handler in ib_mad_complete_recv()




From tduffy at sun.com  Mon Jul 11 12:52:06 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 11 Jul 2005 12:52:06 -0700
Subject: [openib-general] Gen2 build/update process
In-Reply-To: <20050711194659.GA7892@cse.ohio-state.edu>
References: <20050711190308.GA7758@cse.ohio-state.edu>
	<42D2C438.3020707@sun.com> <20050711194659.GA7892@cse.ohio-state.edu>
Message-ID: <1121111526.27796.1.camel@duffman>

On Mon, 2005-07-11 at 15:46 -0400, Sayantan Sur wrote:
> I am building the user level support. Currently, that requires manual
> autogen, config, make, make install in lots of subdirectories. I was
> wondering if someone has a script to automate the build process. This
> will also ensure that new adopters don't get any dependency wrong.

ideally, these will be packaged as rpms, deps, or ebuilds and included
with your favorite distribution.

btw, if you do learn a recipe that works for you, please update the wiki
to share your experiences.

Thanks,

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/1cbe0e48/attachment.sig>

From davem at davemloft.net  Mon Jul 11 12:53:05 2005
From: davem at davemloft.net (David S. Miller)
Date: Mon, 11 Jul 2005 12:53:05 -0700 (PDT)
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <1121110249.4389.4984.camel@hal.voltaire.com>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
Message-ID: <20050711.125305.08322243.davem@davemloft.net>


NO!!!! No more huge patch bombs to linux-kernel at vger.kernel.org
people!  Condense your changes into more manageable pieces
or only submit smaller groups of changes at a time.

You people doing this are absolutely killing vger.kernel.org.

Every patch you post has to go to 5000+ subscribers, so please
keep this in mind when posting multiple patches.  If you feel
the need to send, say, more than 15 patches at once, reconsider.


From halr at voltaire.com  Mon Jul 11 12:47:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:47:05 -0400
Subject: [openib-general] [PATCH 5/29v2] Change saving of user's send wr_id
	in MAD
Message-ID: <1121110274.4389.4994.camel@hal.voltaire.com>

Move saving of user's send wr_id to better match layering of received 
response handling.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 4/29.

--
 mad.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-4/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-5/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-4/drivers/infiniband/core/mad.c	2005-07-11 13:35:26.000000000 -0400
+++ linux-2.6.13-rc2-mm1-5/drivers/infiniband/core/mad.c	2005-07-11 13:35:56.000000000 -0400
@@ -847,9 +847,8 @@ static int ib_send_mad(struct ib_mad_age
 	unsigned long flags;
 	int ret;
 
-	/* Replace user's WR ID with our own to find WR upon completion */
+	/* Set WR ID to find mad_send_wr upon completion */
 	qp_info = mad_agent_priv->qp_info;
-	mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
 	mad_send_wr->send_wr.wr_id = (unsigned long)&mad_send_wr->mad_list;
 	mad_send_wr->mad_list.mad_queue = &qp_info->send_queue;
 
@@ -948,6 +947,7 @@ int ib_post_send_mad(struct ib_mad_agent
 		mad_send_wr->send_wr.sg_list = mad_send_wr->sg_list;
 		memcpy(mad_send_wr->sg_list, send_wr->sg_list,
 		       sizeof *send_wr->sg_list * send_wr->num_sge);
+		mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
 		mad_send_wr->send_wr.next = NULL;
 		mad_send_wr->tid = send_wr->wr.ud.mad_hdr->tid;
 		mad_send_wr->agent = mad_agent;




From mst at mellanox.co.il  Mon Jul 11 12:57:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 11 Jul 2005 22:57:49 +0300
Subject: [openib-general] Re: Gen2 build/update process
In-Reply-To: <20050711190308.GA7758@cse.ohio-state.edu>
References: <20050711190308.GA7758@cse.ohio-state.edu>
Message-ID: <20050711195749.GA25791@mellanox.co.il>

Quoting r. Sayantan Sur <surs at cse.ohio-state.edu>:
> Subject: Gen2 build/update process
> 
> Hi,
> 
> Is there any plan to have any formal install procedure instead of the
> current process of going into subdirectories and building?
> 
> If someone has build scripts to share, that'll be real nice.
> 
> TIA,
> Sayantan.

Sure, I have such scripts.
Where's the good place to put them?

-- 
MST


From halr at voltaire.com  Mon Jul 11 12:50:18 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:50:18 -0400
Subject: [openib-general] [PATCH 6/29v2] Change ib_mad_send_wr_private struct
Message-ID: <1121110277.4389.4996.camel@hal.voltaire.com>

Have ib_mad_send_wr_private reference the private agent structure 
directly, rather than the exposed agent definition.  Remove unneeded
parameters to functions and simplify code were possible from this
change.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 5/29.

--
 mad.c      |   22 ++++++++++--
 mad_priv.h |    4 ++--
 2 files changed, 12 insertions(+), 14 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-5/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-5/drivers/infiniband/core/mad.c	2005-07-11 13:35:56.000000000 -0400
+++ linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad.c	2005-07-11 13:36:16.000000000 -0400
@@ -839,8 +839,7 @@ void ib_free_send_mad(struct ib_mad_send
 }
 EXPORT_SYMBOL(ib_free_send_mad);
 
-static int ib_send_mad(struct ib_mad_agent_private *mad_agent_priv,
-		       struct ib_mad_send_wr_private *mad_send_wr)
+static int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr)
 {
 	struct ib_mad_qp_info *qp_info;
 	struct ib_send_wr *bad_send_wr;
@@ -848,7 +847,7 @@ static int ib_send_mad(struct ib_mad_age
 	int ret;
 
 	/* Set WR ID to find mad_send_wr upon completion */
-	qp_info = mad_agent_priv->qp_info;
+	qp_info = mad_send_wr->mad_agent_priv->qp_info;
 	mad_send_wr->send_wr.wr_id = (unsigned long)&mad_send_wr->mad_list;
 	mad_send_wr->mad_list.mad_queue = &qp_info->send_queue;
 
@@ -857,7 +856,7 @@ static int ib_send_mad(struct ib_mad_age
 		list_add_tail(&mad_send_wr->mad_list.list,
 			      &qp_info->send_queue.list);
 		spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
-		ret = ib_post_send(mad_agent_priv->agent.qp,
+		ret = ib_post_send(mad_send_wr->mad_agent_priv->agent.qp,
 				   &mad_send_wr->send_wr, &bad_send_wr);
 		if (ret) {
 			printk(KERN_ERR PFX "ib_post_send failed: %d\n", ret);
@@ -950,7 +949,7 @@ int ib_post_send_mad(struct ib_mad_agent
 		mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
 		mad_send_wr->send_wr.next = NULL;
 		mad_send_wr->tid = send_wr->wr.ud.mad_hdr->tid;
-		mad_send_wr->agent = mad_agent;
+		mad_send_wr->mad_agent_priv = mad_agent_priv;
 		/* Timeout will be updated after send completes */
 		mad_send_wr->timeout = msecs_to_jiffies(send_wr->wr.
 							ud.timeout_ms);
@@ -966,7 +965,7 @@ int ib_post_send_mad(struct ib_mad_agent
 			      &mad_agent_priv->send_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
-		ret = ib_send_mad(mad_agent_priv, mad_send_wr);
+		ret = ib_send_mad(mad_send_wr);
 		if (ret) {
 			/* Fail send request */
 			spin_lock_irqsave(&mad_agent_priv->lock, flags);
@@ -1742,13 +1741,14 @@ static void adjust_timeout(struct ib_mad
 	}
 }
 
-static void wait_for_response(struct ib_mad_agent_private *mad_agent_priv,
-			      struct ib_mad_send_wr_private *mad_send_wr )
+static void wait_for_response(struct ib_mad_send_wr_private *mad_send_wr)
 {
+	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *temp_mad_send_wr;
 	struct list_head *list_item;
 	unsigned long delay;
 
+	mad_agent_priv = mad_send_wr->mad_agent_priv;
 	list_del(&mad_send_wr->agent_list);
 
 	delay = mad_send_wr->timeout;
@@ -1781,9 +1781,7 @@ static void ib_mad_complete_send_wr(stru
 	struct ib_mad_agent_private	*mad_agent_priv;
 	unsigned long			flags;
 
-	mad_agent_priv = container_of(mad_send_wr->agent,
-				      struct ib_mad_agent_private, agent);
-
+	mad_agent_priv = mad_send_wr->mad_agent_priv;
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
 	if (mad_send_wc->status != IB_WC_SUCCESS &&
 	    mad_send_wr->status == IB_WC_SUCCESS) {
@@ -1794,7 +1792,7 @@ static void ib_mad_complete_send_wr(stru
 	if (--mad_send_wr->refcount > 0) {
 		if (mad_send_wr->refcount == 1 && mad_send_wr->timeout &&
 		    mad_send_wr->status == IB_WC_SUCCESS) {
-			wait_for_response(mad_agent_priv, mad_send_wr);
+			wait_for_response(mad_send_wr);
 		}
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 		return;
diff -uprN linux-2.6.13-rc2-mm1-5/drivers/infiniband/core/mad_priv.h linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad_priv.h
-- linux-2.6.13-rc2-mm1-5/drivers/infiniband/core/mad_priv.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad_priv.h	2005-07-09 15:09:53.000000000 -0400
@@ -29,7 +29,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: mad_priv.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: mad_priv.h 1980 2005-03-11 22:33:53Z sean.hefty $
  */
 
 #ifndef __IB_MAD_PRIV_H__
@@ -116,7 +116,7 @@ struct ib_mad_snoop_private {
 struct ib_mad_send_wr_private {
 	struct ib_mad_list_head mad_list;
 	struct list_head agent_list;
-	struct ib_mad_agent *agent;
+	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_send_wr send_wr;
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */




From halr at voltaire.com  Mon Jul 11 12:53:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:53:31 -0400
Subject: [openib-general] [PATCH 7/29v2] Fix timeout/cancelled MAD handling
Message-ID: <1121110279.4389.4998.camel@hal.voltaire.com>

Fixes an issue processing a sent MAD after it has timed out or 
been canceled.  The race occurs when a response MAD matches with
the send request.  The request could time out or be canceled after
the response MAD matches with the request, but before the request
completion can be processed.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 6/29.

--
 mad.c      |   14 ++++++++++++--
 mad_priv.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-7/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad.c	2005-07-11 13:36:16.000000000 -0400
+++ linux-2.6.13-rc2-mm1-7/drivers/infiniband/core/mad.c	2005-07-11 13:36:29.000000000 -0400
@@ -341,6 +341,7 @@ struct ib_mad_agent *ib_register_mad_age
 	spin_lock_init(&mad_agent_priv->lock);
 	INIT_LIST_HEAD(&mad_agent_priv->send_list);
 	INIT_LIST_HEAD(&mad_agent_priv->wait_list);
+	INIT_LIST_HEAD(&mad_agent_priv->done_list);
 	INIT_WORK(&mad_agent_priv->timed_work, timeout_sends, mad_agent_priv);
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
@@ -1559,6 +1560,16 @@ find_send_req(struct ib_mad_agent_privat
 	return NULL;
 }
 
+static void ib_mark_req_done(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	mad_send_wr->timeout = 0;
+	if (mad_send_wr->refcount == 1) {
+		list_del(&mad_send_wr->agent_list);
+		list_add_tail(&mad_send_wr->agent_list,
+			      &mad_send_wr->mad_agent_priv->done_list);
+	}
+}
+
 static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
 				 struct ib_mad_recv_wc *mad_recv_wc)
 {
@@ -1580,8 +1591,7 @@ static void ib_mad_complete_recv(struct 
 				wake_up(&mad_agent_priv->wait);
 			return;
 		}
-		/* Timeout = 0 means that we won't wait for a response */
-		mad_send_wr->timeout = 0;
+		ib_mark_req_done(mad_send_wr);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		/* Defined behavior is to complete response before request */
diff -uprN linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad_priv.h linux-2.6.13-rc2-mm1-7/drivers/infiniband/core/mad_priv.h
-- linux-2.6.13-rc2-mm1-6/drivers/infiniband/core/mad_priv.h	2005-07-09 15:09:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-7/drivers/infiniband/core/mad_priv.h	2005-07-09 16:48:05.000000000 -0400
@@ -92,6 +92,7 @@ struct ib_mad_agent_private {
 	spinlock_t lock;
 	struct list_head send_list;
 	struct list_head wait_list;
+	struct list_head done_list;
 	struct work_struct timed_work;
 	unsigned long timeout;
 	struct list_head local_list;




From halr at voltaire.com  Mon Jul 11 12:57:59 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 15:57:59 -0400
Subject: [openib-general] [PATCH 8/29v2] Minor cleanup during MAD startup
	and shutdown
Message-ID: <1121110285.4389.5000.camel@hal.voltaire.com>

Minor cleanup during startup and shutdown

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 7/29.

--
 mad.c |   44 +++++++++--
 1 files changed, 9 insertions(+), 35 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-7/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-8/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-7/drivers/infiniband/core/mad.c	2005-07-11 13:36:29.000000000 -0400
+++ linux-2.6.13-rc2-mm1-8/drivers/infiniband/core/mad.c	2005-07-11 13:36:47.000000000 -0400
@@ -2487,14 +2487,6 @@ static int ib_mad_port_open(struct ib_de
 	unsigned long flags;
 	char name[sizeof "ib_mad123"];
 
-	/* First, check if port already open at MAD layer */
-	port_priv = ib_get_mad_port(device, port_num);
-	if (port_priv) {
-		printk(KERN_DEBUG PFX "%s port %d already open\n",
-		       device->name, port_num);
-		return 0;
-	}
-
 	/* Create new device info */
 	port_priv = kmalloc(sizeof *port_priv, GFP_KERNEL);
 	if (!port_priv) {
@@ -2619,7 +2611,7 @@ static int ib_mad_port_close(struct ib_d
 
 static void ib_mad_init_device(struct ib_device *device)
 {
-	int ret, num_ports, cur_port, i, ret2;
+	int num_ports, cur_port, i;
 
 	if (device->node_type == IB_NODE_SWITCH) {
 		num_ports = 1;
@@ -2629,47 +2621,37 @@ static void ib_mad_init_device(struct ib
 		cur_port = 1;
 	}
 	for (i = 0; i < num_ports; i++, cur_port++) {
-		ret = ib_mad_port_open(device, cur_port);
-		if (ret) {
+		if (ib_mad_port_open(device, cur_port)) {
 			printk(KERN_ERR PFX "Couldn't open %s port %d\n",
 			       device->name, cur_port);
 			goto error_device_open;
 		}
-		ret = ib_agent_port_open(device, cur_port);
-		if (ret) {
+		if (ib_agent_port_open(device, cur_port)) {
 			printk(KERN_ERR PFX "Couldn't open %s port %d "
 			       "for agents\n",
 			       device->name, cur_port);
 			goto error_device_open;
 		}
 	}
-
-	goto error_device_query;
+	return;
 
 error_device_open:
 	while (i > 0) {
 		cur_port--;
-		ret2 = ib_agent_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_agent_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d "
 			       "for agents\n",
 			       device->name, cur_port);
-		}
-		ret2 = ib_mad_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_mad_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d\n",
 			       device->name, cur_port);
-		}
 		i--;
 	}
-
-error_device_query:
-	return;
 }
 
 static void ib_mad_remove_device(struct ib_device *device)
 {
-	int ret = 0, i, num_ports, cur_port, ret2;
+	int i, num_ports, cur_port;
 
 	if (device->node_type == IB_NODE_SWITCH) {
 		num_ports = 1;
@@ -2679,21 +2661,13 @@ static void ib_mad_remove_device(struct 
 		cur_port = 1;
 	}
 	for (i = 0; i < num_ports; i++, cur_port++) {
-		ret2 = ib_agent_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_agent_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d "
 			       "for agents\n",
 			       device->name, cur_port);
-			if (!ret)
-				ret = ret2;
-		}
-		ret2 = ib_mad_port_close(device, cur_port);
-		if (ret2) {
+		if (ib_mad_port_close(device, cur_port))
 			printk(KERN_ERR PFX "Couldn't close %s port %d\n",
 			       device->name, cur_port);
-			if (!ret)
-				ret = ret2;
-		}
 	}
 }
 




From rep.nop at aon.at  Mon Jul 11 13:06:42 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Mon, 11 Jul 2005 22:06:42 +0200
Subject: [openib-general] Gen2 build/update process
In-Reply-To: <42D2C438.3020707@sun.com>
References: <20050711190308.GA7758@cse.ohio-state.edu>
	<42D2C438.3020707@sun.com>
Message-ID: <20050711200642.GD15261@aon.at>

On Mon, Jul 11, 2005 at 12:10:48PM -0700, Tom Duffy wrote:
>Sayantan Sur wrote:
>
>>Hi,
>>
>>Is there any plan to have any formal install procedure instead of the
>>current process of going into subdirectories and building?
>>
>>If someone has build scripts to share, that'll be real nice.
>> 
>>
>
>What are you trying to build?  Kernel build should not need to do this.

.. and src/userspace/management has a 'automake' target.

In case it's helpful for you, i'm attaching a simple script which can
be easily extended to cover the rest. I usually call it like this:
nohup ./mk.openib2.usr.sh 01a
It's not pretty but you get the idea.

hth,
Bernhard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mk.openib2.usr.sh
Type: application/x-sh
Size: 1225 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/24fff9eb/attachment.sh>

From halr at voltaire.com  Mon Jul 11 13:02:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:02:25 -0400
Subject: [openib-general] PATCH 9/29v2] Add ib_coalesce_recv_mad to MAD
Message-ID: <1121110291.4389.5002.camel@hal.voltaire.com>

Add implementation for ib_coalesce_recv_mad. 
Also, clear allocated MAD data buffer in ib_create_send_mad.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 8/29.

--
 core/mad.c       |    9 +--
 include/ib_mad.h |    3 +--
 2 files changed, 2 insertions(+), 10 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-8/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-8/drivers/infiniband/core/mad.c	2005-07-11 13:36:47.000000000 -0400
+++ linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/mad.c	2005-07-11 13:37:10.000000000 -0400
@@ -796,9 +796,9 @@ struct ib_mad_send_buf * ib_create_send_
 	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);
 	if (!buf)
 		return ERR_PTR(-ENOMEM);
+	memset(buf, 0, sizeof *send_buf + buf_size);
 
 	send_buf = buf + buf_size;
-	memset(send_buf, 0, sizeof *send_buf);
 	send_buf->mad = buf;
 
 	send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device,
@@ -1021,13 +1021,6 @@ void ib_free_recv_mad(struct ib_mad_recv
 }
 EXPORT_SYMBOL(ib_free_recv_mad);
 
-void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc,
-			  void *buf)
-{
-	printk(KERN_ERR PFX "ib_coalesce_recv_mad() not implemented yet\n");
-}
-EXPORT_SYMBOL(ib_coalesce_recv_mad);
-
 struct ib_mad_agent *ib_redirect_mad_qp(struct ib_qp *qp,
 					u8 rmpp_version,
 					ib_mad_send_handler send_handler,
diff -uprN linux-2.6.13-rc2-mm1-8/drivers/infiniband/include/ib_mad.h linux-2.6.13-rc2-mm1-9/drivers/infiniband/include/ib_mad.h
-- linux-2.6.13-rc2-mm1-8/drivers/infiniband/include/ib_mad.h	2005-07-11 13:36:45.000000000 -0400
+++ linux-2.6.13-rc2-mm1-9/drivers/infiniband/include/ib_mad.h	2005-07-11 13:36:58.000000000 -0400
@@ -365,8 +365,7 @@ int ib_post_send_mad(struct ib_mad_agent
  * This call copies a chain of received RMPP MADs into a single data buffer,
  * removing duplicated headers.
  */
-void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc,
-			  void *buf);
+void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf);
 
 /**
  * ib_free_recv_mad - Returns data buffers used to receive a MAD to the




From mst at mellanox.co.il  Mon Jul 11 13:11:05 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 11 Jul 2005 23:11:05 +0300
Subject: [openib-general] Re: Gen2 build/update process
In-Reply-To: <20050711200642.GD15261@aon.at>
References: <20050711190308.GA7758@cse.ohio-state.edu>
	<42D2C438.3020707@sun.com> <20050711200642.GD15261@aon.at>
Message-ID: <20050711201105.GF25791@mellanox.co.il>

Quoting r. Bernhard Fischer <rep.nop at aon.at>:
> Subject: Re: Gen2 build/update process
> 
> On Mon, Jul 11, 2005 at 12:10:48PM -0700, Tom Duffy wrote:
> >Sayantan Sur wrote:
> >
> >>Hi,
> >>
> >>Is there any plan to have any formal install procedure instead of the
> >>current process of going into subdirectories and building?
> >>
> >>If someone has build scripts to share, that'll be real nice.
> >> 
> >>
> >
> >What are you trying to build?  Kernel build should not need to do this.
> 
> .. and src/userspace/management has a 'automake' target.
> 
> In case it's helpful for you, i'm attaching a simple script which can
> be easily extended to cover the rest. I usually call it like this:
> nohup ./mk.openib2.usr.sh 01a
> It's not pretty but you get the idea.
> 
> hth,
> Bernhard

My scripts are simpler: e.g. for management (I have one for libibverbs too):

#!/bin/sh -x
#1. In libibcommon, libibumad, and libibmad (in that order), run:
( cd libibcommon && ./autogen.sh && ./configure && make && make install )
( cd libibumad && ./autogen.sh && ./configure && make && make install )
( cd libibmad && ./autogen.sh && ./configure && make && make install )

#2. In osm/complib and osm/libvendor, run:
( cd osm/complib && ./autogen.sh && ./configure && make && make install )
( cd osm/libvendor && ./autogen.sh && ./configure && make && make install )
#3. In all util/mad_test, diags, and osm/opensm subdirectories, run:
( cd util/mad_test && ./autogen.sh && ./configure )
( cd diags && ./autogen.sh && ./configure )
( cd osm/opensm && ./autogen.sh && ./configure )
#4. At top level of management, run:
( make && make install )

-- 
MST


From halr at voltaire.com  Mon Jul 11 13:07:12 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:07:12 -0400
Subject: [openib-general] [PATCH 10/29v2] Add automatic retries to MAD layer
Message-ID: <1121110298.4389.5004.camel@hal.voltaire.com>

Add automatic retries to MAD layer. 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 9/29.

--
 core/mad.c         |   26 +++++++++++++++++++++++++-
 core/mad_priv.h    |    2 ++
 core/sa_query.c    |    3 ++-
 core/user_mad.c    |    1 +
 include/ib_verbs.h |    1 +
 5 files changed, 31 insertions(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/mad.c	2005-07-11 13:37:10.000000000 -0400
+++ linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/mad.c	2005-07-11 13:37:54.000000000 -0400
@@ -954,7 +954,7 @@ int ib_post_send_mad(struct ib_mad_agent
 		/* Timeout will be updated after send completes */
 		mad_send_wr->timeout = msecs_to_jiffies(send_wr->wr.
 							ud.timeout_ms);
-		mad_send_wr->retry = 0;
+		mad_send_wr->retries = mad_send_wr->send_wr.wr.ud.retries;
 		/* One reference for each work request to QP + response */
 		mad_send_wr->refcount = 1 + (mad_send_wr->timeout > 0);
 		mad_send_wr->status = IB_WC_SUCCESS;
@@ -2174,6 +2174,27 @@ local_send_completion:
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 }
 
+static int retry_send(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	int ret;
+
+	if (!mad_send_wr->retries--)
+		return -ETIMEDOUT;
+
+	mad_send_wr->timeout = msecs_to_jiffies(mad_send_wr->send_wr.
+						wr.ud.timeout_ms);
+
+	ret = ib_send_mad(mad_send_wr);
+
+	if (!ret) {
+		mad_send_wr->refcount++;
+		list_del(&mad_send_wr->agent_list);
+		list_add_tail(&mad_send_wr->agent_list,
+			      &mad_send_wr->mad_agent_priv->send_list);
+	}
+	return ret;
+}
+
 static void timeout_sends(void *data)
 {
 	struct ib_mad_agent_private *mad_agent_priv;
@@ -2202,6 +2223,9 @@ static void timeout_sends(void *data)
 			break;
 		}
 
+		if (!retry_send(mad_send_wr))
+			continue;
+
 		list_del(&mad_send_wr->agent_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
diff -uprN linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/mad_priv.h linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/mad_priv.h
-- linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/mad_priv.h	2005-07-09 16:48:05.000000000 -0400
+++ linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/mad_priv.h	2005-07-09 17:15:40.000000000 -0400
@@ -123,6 +123,7 @@ struct ib_mad_send_wr_private {
 	u64 wr_id;			/* client WR ID */
 	u64 tid;
 	unsigned long timeout;
+	int retries;
 	int retry;
 	int refcount;
 	enum ib_wc_status status;
@@ -136,6 +137,7 @@ struct ib_mad_local_private {
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */
 	u64 tid;
+	int retries;
 };
 
 struct ib_mad_mgmt_method_table {
diff -uprN linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/sa_query.c linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/sa_query.c
-- linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/sa_query.c	2005-07-10 16:21:55.000000000 -0400
+++ linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/sa_query.c	2005-07-10 16:22:13.000000000 -0400
@@ -462,7 +462,8 @@ static int send_mad(struct ib_sa_query *
 				 .mad_hdr     = &query->mad->mad_hdr,
 				 .remote_qpn  = 1,
 				 .remote_qkey = IB_QP1_QKEY,
-				 .timeout_ms  = timeout_ms
+				 .timeout_ms  = timeout_ms,
+				 .retries     = 0 
 			 }
 		 }
 	};
diff -uprN linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/user_mad.c linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/user_mad.c
-- linux-2.6.13-rc2-mm1-9/drivers/infiniband/core/user_mad.c	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/user_mad.c	2005-07-09 17:14:46.000000000 -0400
@@ -322,6 +322,7 @@ static ssize_t ib_umad_write(struct file
 	wr.wr.ud.remote_qpn  = be32_to_cpu(packet->mad.qpn);
 	wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey);
 	wr.wr.ud.timeout_ms  = packet->mad.timeout_ms;
+	wr.wr.ud.retries     = 0;
 
 	wr.wr_id            = (unsigned long) packet;
 
diff -uprN linux-2.6.13-rc2-mm1-9/drivers/infiniband/include/ib_verbs.h linux-2.6.13-rc2-mm1-10/drivers/infiniband/include/ib_verbs.h
-- linux-2.6.13-rc2-mm1-9/drivers/infiniband/include/ib_verbs.h	2005-07-10 11:11:43.000000000 -0400
+++ linux-2.6.13-rc2-mm1-10/drivers/infiniband/include/ib_verbs.h	2005-07-10 10:55:59.000000000 -0400
@@ -566,6 +566,7 @@ struct ib_send_wr {
 			u32	remote_qpn;
 			u32	remote_qkey;
 			int	timeout_ms; /* valid for MADs only */
+			int	retries;    /* valid for MADs only */
 			u16	pkey_index; /* valid for GSI only */
 			u8	port_num;   /* valid for DR SMPs on switch only */
 		} ud;




From halr at voltaire.com  Mon Jul 11 13:10:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:10:25 -0400
Subject: [openib-general] [PATCH 11/29v2] Simplify calling of list_del in MAD
Message-ID: <1121110302.4389.5006.camel@hal.voltaire.com>

Simplify calling of list_del.


Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 10/29.

--
 mad.c |    3 +--
 1 files changed, 1 insertion(+), 2 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-11/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-10/drivers/infiniband/core/mad.c	2005-07-11 13:37:54.000000000 -0400
+++ linux-2.6.13-rc2-mm1-11/drivers/infiniband/core/mad.c	2005-07-11 13:38:06.000000000 -0400
@@ -2188,7 +2188,6 @@ static int retry_send(struct ib_mad_send
 
 	if (!ret) {
 		mad_send_wr->refcount++;
-		list_del(&mad_send_wr->agent_list);
 		list_add_tail(&mad_send_wr->agent_list,
 			      &mad_send_wr->mad_agent_priv->send_list);
 	}
@@ -2223,10 +2222,10 @@ static void timeout_sends(void *data)
 			break;
 		}
 
+		list_del(&mad_send_wr->agent_list);
 		if (!retry_send(mad_send_wr))
 			continue;
 
-		list_del(&mad_send_wr->agent_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		mad_send_wc.wr_id = mad_send_wr->wr_id;




From halr at voltaire.com  Mon Jul 11 13:13:38 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:13:38 -0400
Subject: [openib-general] [PATCH 12/29v2] Eliminate MAD cache leak
	associated with local completions
Message-ID: <1121110308.4389.5008.camel@hal.voltaire.com>

Eliminate MAD cache leak associated with local completions. 
Also, when canceling MAD, empty local completion list as well.

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 11/29.

-- 
 mad.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletion(-)
diff -uprN linux-2.6.13-rc2-mm1-11/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-12/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-11/drivers/infiniband/core/mad.c	2005-07-11 13:38:06.000000000 -0400
+++ linux-2.6.13-rc2-mm1-12/drivers/infiniband/core/mad.c	2005-07-11 13:38:18.000000000 -0400
@@ -1994,6 +1994,8 @@ static void cancel_mads(struct ib_mad_ag
 
 	/* Empty wait list to prevent receives from finding a request */
 	list_splice_init(&mad_agent_priv->wait_list, &cancel_list);
+	/* Empty local completion list as well */
+	list_splice_init(&mad_agent_priv->local_list, &cancel_list);
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 	/* Report all cancelled requests */
@@ -2108,6 +2110,7 @@ static void local_completions(void *data
 	struct ib_mad_local_private *local;
 	struct ib_mad_agent_private *recv_mad_agent;
 	unsigned long flags;
+	int recv = 0;
 	struct ib_wc wc;
 	struct ib_mad_send_wc mad_send_wc;
 
@@ -2123,10 +2126,10 @@ static void local_completions(void *data
 			recv_mad_agent = local->recv_mad_agent;
 			if (!recv_mad_agent) {
 				printk(KERN_ERR PFX "No receive MAD agent for local completion\n");
-				kmem_cache_free(ib_mad_cache, local->mad_priv);
 				goto local_send_completion;
 			}
 
+			recv = 1;
 			/*
 			 * Defined behavior is to complete response
 			 * before request
@@ -2169,6 +2172,8 @@ local_send_completion:
 		spin_lock_irqsave(&mad_agent_priv->lock, flags);
 		list_del(&local->completion_list);
 		atomic_dec(&mad_agent_priv->refcount);
+		if (!recv)
+			kmem_cache_free(ib_mad_cache, local->mad_priv);
 		kfree(local);
 	}
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);




From halr at voltaire.com  Mon Jul 11 13:16:51 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:16:51 -0400
Subject: [openib-general] [PATCH 13/29v2] Add ib_modify_mad API to MAD
Message-ID: <1121110313.4389.5010.camel@hal.voltaire.com>

Add new MAD layer call to modify (ib_modify_mad) the timeout of a sent 
MAD, and simplify cancel code.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 12/29.

-- 
 core/mad.c       |   83 ++++++++++++++++++--
 core/mad_priv.h  |    2 -
 include/ib_mad.h |   14 +++++++--
 3 files changed, 40 insertions(+), 59 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-12/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-13/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-12/drivers/infiniband/core/mad.c	2005-07-11 13:38:18.000000000 -0400
+++ linux-2.6.13-rc2-mm1-13/drivers/infiniband/core/mad.c	2005-07-11 13:38:30.000000000 -0400
@@ -65,7 +65,6 @@ static void cancel_mads(struct ib_mad_ag
 static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
 				    struct ib_mad_send_wc *mad_send_wc);
 static void timeout_sends(void *data);
-static void cancel_sends(void *data);
 static void local_completions(void *data);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
 			      struct ib_mad_agent_private *agent_priv,
@@ -346,8 +345,6 @@ struct ib_mad_agent *ib_register_mad_age
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
 		   mad_agent_priv);
-	INIT_LIST_HEAD(&mad_agent_priv->canceled_list);
-	INIT_WORK(&mad_agent_priv->canceled_work, cancel_sends, mad_agent_priv);
 	atomic_set(&mad_agent_priv->refcount, 1);
 	init_waitqueue_head(&mad_agent_priv->wait);
 
@@ -1775,6 +1772,13 @@ static void wait_for_response(struct ib_
 	}
 }
 
+void ib_reset_mad_timeout(struct ib_mad_send_wr_private *mad_send_wr,
+			  int timeout_ms)
+{
+	mad_send_wr->timeout = msecs_to_jiffies(timeout_ms);
+	wait_for_response(mad_send_wr);
+}
+
 /*
  * Process a send work completion
  */
@@ -2034,41 +2038,7 @@ find_send_by_wr_id(struct ib_mad_agent_p
 	return NULL;
 }
 
-void cancel_sends(void *data)
-{
-	struct ib_mad_agent_private *mad_agent_priv;
-	struct ib_mad_send_wr_private *mad_send_wr;
-	struct ib_mad_send_wc mad_send_wc;
-	unsigned long flags;
-
-	mad_agent_priv = data;
-
-	mad_send_wc.status = IB_WC_WR_FLUSH_ERR;
-	mad_send_wc.vendor_err = 0;
-
-	spin_lock_irqsave(&mad_agent_priv->lock, flags);
-	while (!list_empty(&mad_agent_priv->canceled_list)) {
-		mad_send_wr = list_entry(mad_agent_priv->canceled_list.next,
-					 struct ib_mad_send_wr_private,
-					 agent_list);
-
-		list_del(&mad_send_wr->agent_list);
-		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-
-		mad_send_wc.wr_id = mad_send_wr->wr_id;
-		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
-						   &mad_send_wc);
-
-		kfree(mad_send_wr);
-		if (atomic_dec_and_test(&mad_agent_priv->refcount))
-			wake_up(&mad_agent_priv->wait);
-		spin_lock_irqsave(&mad_agent_priv->lock, flags);
-	}
-	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-}
-
-void ib_cancel_mad(struct ib_mad_agent *mad_agent,
-		  u64 wr_id)
+int ib_modify_mad(struct ib_mad_agent *mad_agent, u64 wr_id, u32 timeout_ms)
 {
 	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *mad_send_wr;
@@ -2078,29 +2048,30 @@ void ib_cancel_mad(struct ib_mad_agent *
 				      agent);
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
 	mad_send_wr = find_send_by_wr_id(mad_agent_priv, wr_id);
-	if (!mad_send_wr) {
+	if (!mad_send_wr || mad_send_wr->status != IB_WC_SUCCESS) {
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-		goto out;
+		return -EINVAL;
 	}
 
-	if (mad_send_wr->status == IB_WC_SUCCESS)
-		mad_send_wr->refcount -= (mad_send_wr->timeout > 0);
-
-	if (mad_send_wr->refcount != 0) {
+	if (!timeout_ms) {
 		mad_send_wr->status = IB_WC_WR_FLUSH_ERR;
-		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-		goto out;
+		mad_send_wr->refcount -= (mad_send_wr->timeout > 0);
 	}
 
-	list_del(&mad_send_wr->agent_list);
-	list_add_tail(&mad_send_wr->agent_list, &mad_agent_priv->canceled_list);
-	adjust_timeout(mad_agent_priv);
+	mad_send_wr->send_wr.wr.ud.timeout_ms = timeout_ms;
+	if (!mad_send_wr->timeout || mad_send_wr->refcount > 1)
+		mad_send_wr->timeout = msecs_to_jiffies(timeout_ms);
+	else
+		ib_reset_mad_timeout(mad_send_wr, timeout_ms);
+
 	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
+	return 0;
+}
+EXPORT_SYMBOL(ib_modify_mad);
 
-	queue_work(mad_agent_priv->qp_info->port_priv->wq,
-		   &mad_agent_priv->canceled_work);
-out:
-	return;
+void ib_cancel_mad(struct ib_mad_agent *mad_agent, u64 wr_id)
+{
+	ib_modify_mad(mad_agent, wr_id, 0);
 }
 EXPORT_SYMBOL(ib_cancel_mad);
 
@@ -2207,8 +2178,6 @@ static void timeout_sends(void *data)
 	unsigned long flags, delay;
 
 	mad_agent_priv = (struct ib_mad_agent_private *)data;
-
-	mad_send_wc.status = IB_WC_RESP_TIMEOUT_ERR;
 	mad_send_wc.vendor_err = 0;
 
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
@@ -2233,6 +2202,10 @@ static void timeout_sends(void *data)
 
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
+		if (mad_send_wr->status == IB_WC_SUCCESS)
+			mad_send_wc.status = IB_WC_RESP_TIMEOUT_ERR;
+		else
+			mad_send_wc.status = mad_send_wr->status;
 		mad_send_wc.wr_id = mad_send_wr->wr_id;
 		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
 						   &mad_send_wc);
diff -uprN linux-2.6.13-rc2-mm1-12/drivers/infiniband/core/mad_priv.h linux-2.6.13-rc2-mm1-13/drivers/infiniband/core/mad_priv.h
-- linux-2.6.13-rc2-mm1-12/drivers/infiniband/core/mad_priv.h	2005-07-09 17:15:40.000000000 -0400
+++ linux-2.6.13-rc2-mm1-13/drivers/infiniband/core/mad_priv.h	2005-07-09 17:57:55.000000000 -0400
@@ -97,8 +97,6 @@ struct ib_mad_agent_private {
 	unsigned long timeout;
 	struct list_head local_list;
 	struct work_struct local_work;
-	struct list_head canceled_list;
-	struct work_struct canceled_work;
 
 	atomic_t refcount;
 	wait_queue_head_t wait;
diff -uprN linux-2.6.13-rc2-mm1-12/drivers/infiniband/include/ib_mad.h linux-2.6.13-rc2-mm1-13/drivers/infiniband/include/ib_mad.h
-- linux-2.6.13-rc2-mm1-12/drivers/infiniband/include/ib_mad.h	2005-07-11 13:38:16.000000000 -0400
+++ linux-2.6.13-rc2-mm1-13/drivers/infiniband/include/ib_mad.h	2005-07-11 13:38:28.000000000 -0400
@@ -385,8 +385,18 @@ void ib_free_recv_mad(struct ib_mad_recv
  * MADs will be returned to the user through the corresponding
  * ib_mad_send_handler.
  */
-void ib_cancel_mad(struct ib_mad_agent *mad_agent,
-		   u64 wr_id);
+void ib_cancel_mad(struct ib_mad_agent *mad_agent, u64 wr_id);
+
+/**
+ * ib_modify_mad - Modifies an outstanding send MAD operation.
+ * @mad_agent: Specifies the registration associated with sent MAD.
+ * @wr_id: Indicates the work request identifier of the MAD to modify.
+ * @timeout_ms: New timeout value for sent MAD.
+ *
+ * This call will reset the timeout value for a sent MAD to the specified
+ * value.
+ */
+int ib_modify_mad(struct ib_mad_agent *mad_agent, u64 wr_id, u32 timeout_ms);
 
 /**
  * ib_redirect_mad_qp - Registers a QP for MAD services.




From halr at voltaire.com  Mon Jul 11 13:20:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:20:05 -0400
Subject: [openib-general] [PATCH 14/29v2] Optimize canceling a MAD
Message-ID: <1121110318.4389.5012.camel@hal.voltaire.com>

Optimize canceling a MAD. 
- Eliminate searching timeout list in cancel case.
- Remove duplicate calls to queue work item.
- Eliminate resending a MAD before MAD is completed.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 13/29.

--
 mad.c |   21 +++++++++++++--
 1 files changed, 13 insertions(+), 8 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-13/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-14/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-13/drivers/infiniband/core/mad.c	2005-07-11 13:38:30.000000000 -0400
+++ linux-2.6.13-rc2-mm1-14/drivers/infiniband/core/mad.c	2005-07-11 13:38:42.000000000 -0400
@@ -1754,14 +1754,18 @@ static void wait_for_response(struct ib_
 	delay = mad_send_wr->timeout;
 	mad_send_wr->timeout += jiffies;
 
-	list_for_each_prev(list_item, &mad_agent_priv->wait_list) {
-		temp_mad_send_wr = list_entry(list_item,
-					      struct ib_mad_send_wr_private,
-					      agent_list);
-		if (time_after(mad_send_wr->timeout,
-			       temp_mad_send_wr->timeout))
-			break;
+	if (delay) {
+		list_for_each_prev(list_item, &mad_agent_priv->wait_list) {
+			temp_mad_send_wr = list_entry(list_item,
+						struct ib_mad_send_wr_private,
+						agent_list);
+			if (time_after(mad_send_wr->timeout,
+				       temp_mad_send_wr->timeout))
+				break;
+		}
 	}
+	else
+		list_item = &mad_agent_priv->wait_list;
 	list_add(&mad_send_wr->agent_list, list_item);
 
 	/* Reschedule a work item if we have a shorter timeout */
@@ -2197,7 +2201,8 @@ static void timeout_sends(void *data)
 		}
 
 		list_del(&mad_send_wr->agent_list);
-		if (!retry_send(mad_send_wr))
+		if (mad_send_wr->status == IB_WC_SUCCESS &&
+		    !retry_send(mad_send_wr))
 			continue;
 
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);




From tduffy at sun.com  Mon Jul 11 13:29:45 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 11 Jul 2005 13:29:45 -0700
Subject: [openib-general] Re: Gen2 build/update process
In-Reply-To: <20050711195749.GA25791@mellanox.co.il>
References: <20050711190308.GA7758@cse.ohio-state.edu>
	<20050711195749.GA25791@mellanox.co.il>
Message-ID: <1121113785.27796.3.camel@duffman>

On Mon, 2005-07-11 at 22:57 +0300, Michael S. Tsirkin wrote:
> Sure, I have such scripts.
> Where's the good place to put them?

wiki

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/e30e889d/attachment.sig>

From halr at voltaire.com  Mon Jul 11 13:23:18 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:23:18 -0400
Subject: [openib-general] [PATCH 15/29v2] Fix a couple of MAD code paths
Message-ID: <1121110325.4389.5014.camel@hal.voltaire.com>

Fixed locking to handle error posting MAD send work requests. 
Fixed handling canceling a MAD with an active work request.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 14/29.

-- 
 mad.c |   28 ++++++++++++++--
 1 files changed, 14 insertions(+), 14 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-14/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-15/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-14/drivers/infiniband/core/mad.c	2005-07-11 13:38:42.000000000 -0400
+++ linux-2.6.13-rc2-mm1-15/drivers/infiniband/core/mad.c	2005-07-11 13:38:54.000000000 -0400
@@ -841,6 +841,7 @@ static int ib_send_mad(struct ib_mad_sen
 {
 	struct ib_mad_qp_info *qp_info;
 	struct ib_send_wr *bad_send_wr;
+	struct list_head *list;
 	unsigned long flags;
 	int ret;
 
@@ -850,22 +851,20 @@ static int ib_send_mad(struct ib_mad_sen
 	mad_send_wr->mad_list.mad_queue = &qp_info->send_queue;
 
 	spin_lock_irqsave(&qp_info->send_queue.lock, flags);
-	if (qp_info->send_queue.count++ < qp_info->send_queue.max_active) {
-		list_add_tail(&mad_send_wr->mad_list.list,
-			      &qp_info->send_queue.list);
-		spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
+	if (qp_info->send_queue.count < qp_info->send_queue.max_active) {
 		ret = ib_post_send(mad_send_wr->mad_agent_priv->agent.qp,
 				   &mad_send_wr->send_wr, &bad_send_wr);
-		if (ret) {
-			printk(KERN_ERR PFX "ib_post_send failed: %d\n", ret);
-			dequeue_mad(&mad_send_wr->mad_list);
-		}
+		list = &qp_info->send_queue.list;
 	} else {
-		list_add_tail(&mad_send_wr->mad_list.list,
-			      &qp_info->overflow_list);
-		spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
 		ret = 0;
+		list = &qp_info->overflow_list;
 	}
+
+	if (!ret) {
+		qp_info->send_queue.count++;
+		list_add_tail(&mad_send_wr->mad_list.list, list);
+	}
+	spin_unlock_irqrestore(&qp_info->send_queue.lock, flags);
 	return ret;
 }
 
@@ -2023,8 +2022,7 @@ static void cancel_mads(struct ib_mad_ag
 }
 
 static struct ib_mad_send_wr_private*
-find_send_by_wr_id(struct ib_mad_agent_private *mad_agent_priv,
-		   u64 wr_id)
+find_send_by_wr_id(struct ib_mad_agent_private *mad_agent_priv, u64 wr_id)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 
@@ -2047,6 +2045,7 @@ int ib_modify_mad(struct ib_mad_agent *m
 	struct ib_mad_agent_private *mad_agent_priv;
 	struct ib_mad_send_wr_private *mad_send_wr;
 	unsigned long flags;
+	int active;
 
 	mad_agent_priv = container_of(mad_agent, struct ib_mad_agent_private,
 				      agent);
@@ -2057,13 +2056,14 @@ int ib_modify_mad(struct ib_mad_agent *m
 		return -EINVAL;
 	}
 
+	active = (!mad_send_wr->timeout || mad_send_wr->refcount > 1);
 	if (!timeout_ms) {
 		mad_send_wr->status = IB_WC_WR_FLUSH_ERR;
 		mad_send_wr->refcount -= (mad_send_wr->timeout > 0);
 	}
 
 	mad_send_wr->send_wr.wr.ud.timeout_ms = timeout_ms;
-	if (!mad_send_wr->timeout || mad_send_wr->refcount > 1)
+	if (active)
 		mad_send_wr->timeout = msecs_to_jiffies(timeout_ms);
 	else
 		ib_reset_mad_timeout(mad_send_wr, timeout_ms);




From halr at voltaire.com  Mon Jul 11 13:26:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:26:31 -0400
Subject: [openib-general] [PATCH 16/29v2] Add ib_create_ah_from_wc to IB
	verbs
Message-ID: <1121110331.4389.5016.camel@hal.voltaire.com>

Added new call: ib_create_ah_from_wc. Call will allocate an 
address handle given work completion information, including any
received GRH.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 15/29.

--
 core/verbs.c       |   35 +++++++++++++++++++++++++++++++++++
 include/ib_mad.h   |    9 --
 include/ib_verbs.h |   24 ++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 9 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-15/drivers/infiniband/core/verbs.c linux-2.6.13-rc2-mm1-16/drivers/infiniband/core/verbs.c
-- linux-2.6.13-rc2-mm1-15/drivers/infiniband/core/verbs.c	2005-07-10 11:09:37.000000000 -0400
+++ linux-2.6.13-rc2-mm1-16/drivers/infiniband/core/verbs.c	2005-07-10 11:43:44.000000000 -0400
@@ -41,6 +41,7 @@
 #include <linux/err.h>
 
 #include <ib_verbs.h>
+#include <ib_cache.h>
 
 /* Protection domains */
 
@@ -88,6 +89,40 @@ struct ib_ah *ib_create_ah(struct ib_pd 
 }
 EXPORT_SYMBOL(ib_create_ah);
 
+struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, struct ib_wc *wc,
+				   struct ib_grh *grh, u8 port_num)
+{
+	struct ib_ah_attr ah_attr;
+	u32 flow_class;
+	u16 gid_index;
+	int ret;
+
+	memset(&ah_attr, 0, sizeof ah_attr);
+	ah_attr.dlid = wc->slid;
+	ah_attr.sl = wc->sl;
+	ah_attr.src_path_bits = wc->dlid_path_bits;
+	ah_attr.port_num = port_num;
+	
+	if (wc->wc_flags & IB_WC_GRH) {
+		ah_attr.ah_flags = IB_AH_GRH;
+		ah_attr.grh.dgid = grh->dgid;
+
+		ret = ib_find_cached_gid(pd->device, &grh->sgid, &port_num,
+					 &gid_index);
+		if (ret)
+			return ERR_PTR(ret);
+
+		ah_attr.grh.sgid_index = (u8) gid_index;
+		flow_class = be32_to_cpu(&grh->version_tclass_flow);
+		ah_attr.grh.flow_label = flow_class & 0xFFFFF;
+		ah_attr.grh.traffic_class = (flow_class >> 20) & 0xFF;
+		ah_attr.grh.hop_limit = grh->hop_limit;
+	}
+
+	return ib_create_ah(pd, &ah_attr);
+}
+EXPORT_SYMBOL(ib_create_ah_from_wc);
+
 int ib_modify_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr)
 {
 	return ah->device->modify_ah ?
diff -uprN linux-2.6.13-rc2-mm1-15/drivers/infiniband/include/ib_mad.h linux-2.6.13-rc2-mm1-16/drivers/infiniband/include/ib_mad.h
-- linux-2.6.13-rc2-mm1-15/drivers/infiniband/include/ib_mad.h	2005-07-11 13:38:52.000000000 -0400
+++ linux-2.6.13-rc2-mm1-16/drivers/infiniband/include/ib_mad.h	2005-07-11 13:39:04.000000000 -0400
@@ -77,15 +77,6 @@
 #define IB_QP1_QKEY	0x80010000
 #define IB_QP_SET_QKEY	0x80000000
 
-struct ib_grh {
-	u32		version_tclass_flow;
-	u16		paylen;
-	u8		next_hdr;
-	u8		hop_limit;
-	union ib_gid	sgid;
-	union ib_gid	dgid;
-} __attribute__ ((packed));
-
 struct ib_mad_hdr {
 	u8	base_version;
 	u8	mgmt_class;
diff -uprN linux-2.6.13-rc2-mm1-15/drivers/infiniband/include/ib_verbs.h linux-2.6.13-rc2-mm1-16/drivers/infiniband/include/ib_verbs.h
-- linux-2.6.13-rc2-mm1-15/drivers/infiniband/include/ib_verbs.h	2005-07-10 10:55:59.000000000 -0400
+++ linux-2.6.13-rc2-mm1-16/drivers/infiniband/include/ib_verbs.h	2005-07-10 11:43:45.000000000 -0400
@@ -289,6 +289,15 @@ struct ib_global_route {
 	u8		traffic_class;
 };
 
+struct ib_grh {
+	u32		version_tclass_flow;
+	u16		paylen;
+	u8		next_hdr;
+	u8		hop_limit;
+	union ib_gid	sgid;
+	union ib_gid	dgid;
+};
+
 enum {
 	IB_MULTICAST_QPN = 0xffffff
 };
@@ -991,6 +1000,21 @@ int ib_dealloc_pd(struct ib_pd *pd);
 struct ib_ah *ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr);
 
 /**
+ * ib_create_ah_from_wc - Creates an address handle associated with the
+ *   sender of the specified work completion.
+ * @pd: The protection domain associated with the address handle.
+ * @wc: Work completion information associated with a received message.
+ * @grh: References the received global route header.  This parameter is
+ *   ignored unless the work completion indicates that the GRH is valid.
+ * @port_num: The outbound port number to associate with the address.
+ *
+ * The address handle is used to reference a local or global destination
+ * in all UD QP post sends.
+ */
+struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, struct ib_wc *wc,
+				   struct ib_grh *grh, u8 port_num);
+
+/**
  * ib_modify_ah - Modifies the address vector associated with an address
  *   handle.
  * @ah: The address handle to modify.




From halr at voltaire.com  Mon Jul 11 13:29:45 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:29:45 -0400
Subject: [openib-general] [PATCH 17/29v2] A couple of IB core bug fixes
Message-ID: <1121110334.4389.5018.camel@hal.voltaire.com>

Replace be32_to_cpup with be32_to_cpu and fix bug referencing pointer 
rather than value in ib_create_ah_from_wc().

Signed-off-by: Tom Duffy <tduffy at sun.com>
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 16/29.

-- 
 agent.c |    8 ++++--
 verbs.c |    2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-16/drivers/infiniband/core/agent.c linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/agent.c
-- linux-2.6.13-rc2-mm1-16/drivers/infiniband/core/agent.c	2005-07-09 13:22:55.000000000 -0400
+++ linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/agent.c	2005-07-10 11:50:26.000000000 -0400
@@ -156,10 +156,10 @@ static int agent_mad_send(struct ib_mad_
 			/* Should sgid be looked up ? */
 			ah_attr.grh.sgid_index = 0;
 			ah_attr.grh.hop_limit = grh->hop_limit;
-			ah_attr.grh.flow_label = be32_to_cpup(
-				&grh->version_tclass_flow)  & 0xfffff;
-			ah_attr.grh.traffic_class = (be32_to_cpup(
-				&grh->version_tclass_flow) >> 20) & 0xff;
+			ah_attr.grh.flow_label = be32_to_cpu(
+				grh->version_tclass_flow)  & 0xfffff;
+			ah_attr.grh.traffic_class = (be32_to_cpu(
+				grh->version_tclass_flow) >> 20) & 0xff;
 			memcpy(ah_attr.grh.dgid.raw,
 			       grh->sgid.raw,
 			       sizeof(ah_attr.grh.dgid));
diff -uprN linux-2.6.13-rc2-mm1-16/drivers/infiniband/core/verbs.c linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/verbs.c
-- linux-2.6.13-rc2-mm1-16/drivers/infiniband/core/verbs.c	2005-07-10 11:43:44.000000000 -0400
+++ linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/verbs.c	2005-07-10 11:50:26.000000000 -0400
@@ -113,7 +113,7 @@ struct ib_ah *ib_create_ah_from_wc(struc
 			return ERR_PTR(ret);
 
 		ah_attr.grh.sgid_index = (u8) gid_index;
-		flow_class = be32_to_cpu(&grh->version_tclass_flow);
+		flow_class = be32_to_cpu(grh->version_tclass_flow);
 		ah_attr.grh.flow_label = flow_class & 0xFFFFF;
 		ah_attr.grh.traffic_class = (flow_class >> 20) & 0xFF;
 		ah_attr.grh.hop_limit = grh->hop_limit;




From halr at voltaire.com  Mon Jul 11 13:32:58 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:32:58 -0400
Subject: [openib-general] [PATCH 18/29v2] Introduce RMPP APIs
Message-ID: <1121110345.4389.5020.camel@hal.voltaire.com>

Introduce RMPP APIs 

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 17/29.

--
 core/mad.c       |    4 +
 core/sa_query.c  |   20 --
 include/ib_mad.h |  132 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 include/ib_sa.h  |    4 -
 4 files changed, 125 insertions(+), 35 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/mad.c	2005-07-11 13:39:42.000000000 -0400
+++ linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad.c	2005-07-11 13:39:53.000000000 -0400
@@ -777,7 +777,7 @@ static int get_buf_length(int hdr_len, i
 
 struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
 					    u32 remote_qpn, u16 pkey_index,
-					    struct ib_ah *ah,
+					    struct ib_ah *ah, int rmpp_active,
 					    int hdr_len, int data_len,
 					    unsigned int __nocast gfp_mask)
 {
@@ -786,6 +786,8 @@ struct ib_mad_send_buf * ib_create_send_
 	int buf_size;
 	void *buf;
 
+	if (rmpp_active)
+		return ERR_PTR(-EINVAL);	/* until RMPP implemented */
 	mad_agent_priv = container_of(mad_agent,
 				      struct ib_mad_agent_private, agent);
 	buf_size = get_buf_length(hdr_len, data_len);
diff -uprN linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/sa_query.c linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/sa_query.c
-- linux-2.6.13-rc2-mm1-17/drivers/infiniband/core/sa_query.c	2005-07-10 16:22:13.000000000 -0400
+++ linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/sa_query.c	2005-07-10 16:22:18.000000000 -0400
@@ -50,26 +50,6 @@ MODULE_AUTHOR("Roland Dreier");
 MODULE_DESCRIPTION("InfiniBand subnet administration query support");
 MODULE_LICENSE("Dual BSD/GPL");
 
-/*
- * These two structures must be packed because they have 64-bit fields
- * that are only 32-bit aligned.  64-bit architectures will lay them
- * out wrong otherwise.  (And unfortunately they are sent on the wire
- * so we can't change the layout)
- */
-struct ib_sa_hdr {
-	u64			sm_key;
-	u16			attr_offset;
-	u16			reserved;
-	ib_sa_comp_mask		comp_mask;
-} __attribute__ ((packed));
-
-struct ib_sa_mad {
-	struct ib_mad_hdr	mad_hdr;
-	struct ib_rmpp_hdr	rmpp_hdr;
-	struct ib_sa_hdr	sa_hdr;
-	u8			data[200];
-} __attribute__ ((packed));
-
 struct ib_sa_sm_ah {
 	struct ib_ah        *ah;
 	struct kref          ref;
diff -uprN linux-2.6.13-rc2-mm1-17/drivers/infiniband/include/ib_mad.h linux-2.6.13-rc2-mm1-18/drivers/infiniband/include/ib_mad.h
-- linux-2.6.13-rc2-mm1-17/drivers/infiniband/include/ib_mad.h	2005-07-11 13:39:40.000000000 -0400
+++ linux-2.6.13-rc2-mm1-18/drivers/infiniband/include/ib_mad.h	2005-07-11 13:39:52.000000000 -0400
@@ -33,7 +33,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_mad.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: ib_mad.h 2775 2005-07-02 13:42:12Z halr $
  */
 
 #if !defined( IB_MAD_H )
@@ -58,6 +58,8 @@
 #define IB_MGMT_CLASS_VENDOR_RANGE2_START	0x30
 #define IB_MGMT_CLASS_VENDOR_RANGE2_END		0x4F
 
+#define	IB_OPENIB_OUI				(0x001405)
+
 /* Management methods */
 #define IB_MGMT_METHOD_GET			0x01
 #define IB_MGMT_METHOD_SET			0x02
@@ -72,6 +74,33 @@
 
 #define IB_MGMT_MAX_METHODS			128
 
+/* RMPP information */
+#define IB_MGMT_RMPP_VERSION			1
+
+#define IB_MGMT_RMPP_TYPE_DATA			1
+#define IB_MGMT_RMPP_TYPE_ACK			2
+#define IB_MGMT_RMPP_TYPE_STOP			3
+#define IB_MGMT_RMPP_TYPE_ABORT			4
+
+#define IB_MGMT_RMPP_FLAG_ACTIVE		1
+#define IB_MGMT_RMPP_FLAG_FIRST			(1<<1)
+#define IB_MGMT_RMPP_FLAG_LAST			(1<<2)
+
+#define IB_MGMT_RMPP_NO_RESPTIME		0x1F
+
+#define	IB_MGMT_RMPP_STATUS_SUCCESS		0
+#define	IB_MGMT_RMPP_STATUS_RESX		1
+#define	IB_MGMT_RMPP_STATUS_T2L			118
+#define	IB_MGMT_RMPP_STATUS_BAD_LEN		119
+#define	IB_MGMT_RMPP_STATUS_BAD_SEG		120
+#define	IB_MGMT_RMPP_STATUS_BADT		121
+#define	IB_MGMT_RMPP_STATUS_W2S			122
+#define	IB_MGMT_RMPP_STATUS_S2B			123
+#define	IB_MGMT_RMPP_STATUS_BAD_STATUS		124
+#define	IB_MGMT_RMPP_STATUS_UNV			125
+#define	IB_MGMT_RMPP_STATUS_TMR			126
+#define	IB_MGMT_RMPP_STATUS_UNSPEC		127
+
 #define IB_QP0		0
 #define IB_QP1		__constant_htonl(1)
 #define IB_QP1_QKEY	0x80010000
@@ -88,7 +117,7 @@ struct ib_mad_hdr {
 	u16	attr_id;
 	u16	resv;
 	u32	attr_mod;
-} __attribute__ ((packed));
+};
 
 struct ib_rmpp_hdr {
 	u8	rmpp_version;
@@ -97,17 +126,41 @@ struct ib_rmpp_hdr {
 	u8	rmpp_status;
 	u32	seg_num;
 	u32	paylen_newwin;
+};
+
+typedef u64 __bitwise ib_sa_comp_mask;
+
+#define IB_SA_COMP_MASK(n) ((__force ib_sa_comp_mask) cpu_to_be64(1ull << n))
+
+/*
+ * ib_sa_hdr and ib_sa_mad structures must be packed because they have 
+ * 64-bit fields that are only 32-bit aligned. 64-bit architectures will
+ * lay them out wrong otherwise.  (And unfortunately they are sent on 
+ * the wire so we can't change the layout)
+ */
+struct ib_sa_hdr {
+	u64			sm_key;
+	u16			attr_offset;
+	u16			reserved;
+	ib_sa_comp_mask		comp_mask;
 } __attribute__ ((packed));
 
 struct ib_mad {
 	struct ib_mad_hdr	mad_hdr;
 	u8			data[232];
-} __attribute__ ((packed));
+};
 
 struct ib_rmpp_mad {
 	struct ib_mad_hdr	mad_hdr;
 	struct ib_rmpp_hdr	rmpp_hdr;
 	u8			data[220];
+};
+
+struct ib_sa_mad {
+	struct ib_mad_hdr	mad_hdr;
+	struct ib_rmpp_hdr	rmpp_hdr;
+	struct ib_sa_hdr	sa_hdr;
+	u8			data[200];
 } __attribute__ ((packed));
 
 struct ib_vendor_mad {
@@ -116,7 +169,7 @@ struct ib_vendor_mad {
 	u8			reserved;
 	u8			oui[3];
 	u8			data[216];
-} __attribute__ ((packed));
+};
 
 /**
  * ib_mad_send_buf - MAD data buffer and work request for sends.
@@ -142,6 +195,45 @@ struct ib_mad_send_buf {
 	struct ib_sge		sge;
 };
 
+/**
+ * ib_get_rmpp_resptime - Returns the RMPP response time.
+ * @rmpp_hdr: An RMPP header.
+ */
+static inline u8 ib_get_rmpp_resptime(struct ib_rmpp_hdr *rmpp_hdr)
+{
+	return rmpp_hdr->rmpp_rtime_flags >> 3;
+}
+
+/**
+ * ib_get_rmpp_flags - Returns the RMPP flags.
+ * @rmpp_hdr: An RMPP header.
+ */
+static inline u8 ib_get_rmpp_flags(struct ib_rmpp_hdr *rmpp_hdr)
+{
+	return rmpp_hdr->rmpp_rtime_flags & 0x7;
+}
+
+/**
+ * ib_set_rmpp_resptime - Sets the response time in an RMPP header.
+ * @rmpp_hdr: An RMPP header.
+ * @rtime: The response time to set.
+ */
+static inline void ib_set_rmpp_resptime(struct ib_rmpp_hdr *rmpp_hdr, u8 rtime)
+{
+	rmpp_hdr->rmpp_rtime_flags = ib_get_rmpp_flags(rmpp_hdr) | (rtime << 3);
+}
+
+/**
+ * ib_set_rmpp_flags - Sets the flags in an RMPP header.
+ * @rmpp_hdr: An RMPP header.
+ * @flags: The flags to set.
+ */
+static inline void ib_set_rmpp_flags(struct ib_rmpp_hdr *rmpp_hdr, u8 flags)
+{
+	rmpp_hdr->rmpp_rtime_flags = (rmpp_hdr->rmpp_rtime_flags & 0xF1) |
+				     (flags & 0x7);
+}
+
 struct ib_mad_agent;
 struct ib_mad_send_wc;
 struct ib_mad_recv_wc;
@@ -186,6 +278,7 @@ typedef void (*ib_mad_recv_handler)(stru
  * ib_mad_agent - Used to track MAD registration with the access layer.
  * @device: Reference to device registration is on.
  * @qp: Reference to QP used for sending and receiving MADs.
+ * @mr: Memory region for system memory usable for DMA.
  * @recv_handler: Callback handler for a received MAD.
  * @send_handler: Callback handler for a sent MAD.
  * @snoop_handler: Callback handler for snooped sent MADs.
@@ -194,6 +287,7 @@ typedef void (*ib_mad_recv_handler)(stru
  *   Unsolicited MADs sent by this client will have the upper 32-bits
  *   of their TID set to this value.
  * @port_num: Port number on which QP is registered
+ * @rmpp_version: If set, indicates the RMPP version used by this agent.
  */
 struct ib_mad_agent {
 	struct ib_device	*device;
@@ -205,6 +299,7 @@ struct ib_mad_agent {
 	void			*context;
 	u32			hi_tid;
 	u8			port_num;
+	u8			rmpp_version;
 };
 
 /**
@@ -238,6 +333,7 @@ struct ib_mad_recv_buf {
  * ib_mad_recv_wc - received MAD information.
  * @wc: Completion information for the received data.
  * @recv_buf: Specifies the location of the received data buffer(s).
+ * @rmpp_list: Specifies a list of RMPP reassembled received MAD buffers.
  * @mad_len: The length of the received MAD, without duplicated headers.
  *
  * For received response, the wr_id field of the wc is set to the wr_id
@@ -246,6 +342,7 @@ struct ib_mad_recv_buf {
 struct ib_mad_recv_wc {
 	struct ib_wc		*wc;
 	struct ib_mad_recv_buf	recv_buf;
+	struct list_head	rmpp_list;
 	int			mad_len;
 };
 
@@ -341,6 +438,16 @@ int ib_unregister_mad_agent(struct ib_ma
  * @bad_send_wr: Specifies the MAD on which an error was encountered.
  *
  * Sent MADs are not guaranteed to complete in the order that they were posted.
+ *
+ * If the MAD requires RMPP, the data buffer should contain a single copy
+ * of the common MAD, RMPP, and class specific headers, followed by the class
+ * defined data.  If the class defined data would not divide evenly into
+ * RMPP segments, then space must be allocated at the end of the referenced
+ * buffer for any required padding.  To indicate the amount of class defined
+ * data being transferred, the paylen_newwin field in the RMPP header should
+ * be set to the size of the class specific header plus the amount of class
+ * defined data being transferred.  The paylen_newwin field should be
+ * specified in network-byte order.
  */
 int ib_post_send_mad(struct ib_mad_agent *mad_agent,
 		     struct ib_send_wr *send_wr,
@@ -353,14 +460,13 @@ int ib_post_send_mad(struct ib_mad_agent
  *   referenced buffer should be at least the size of the mad_len specified
  *   by @mad_recv_wc.
  *
- * This call copies a chain of received RMPP MADs into a single data buffer,
+ * This call copies a chain of received MAD segments into a single data buffer,
  * removing duplicated headers.
  */
 void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf);
 
 /**
- * ib_free_recv_mad - Returns data buffers used to receive a MAD to the
- *   access layer.
+ * ib_free_recv_mad - Returns data buffers used to receive a MAD.
  * @mad_recv_wc: Work completion information for a received MAD.
  *
  * Clients receiving MADs through their ib_mad_recv_handler must call this
@@ -437,10 +543,11 @@ int ib_process_mad_wc(struct ib_mad_agen
  * @pkey_index: Specifies which PKey the MAD will be sent using.  This field
  *   is valid only if the remote_qpn is QP 1.
  * @ah: References the address handle used to transfer to the remote node.
+ * @rmpp_active: Indicates if the send will enable RMPP.
  * @hdr_len: Indicates the size of the data header of the MAD.  This length
  *   should include the common MAD header, RMPP header, plus any class
  *   specific header.
- * @data_len: Indicates the size of any user-transfered data.  The call will
+ * @data_len: Indicates the size of any user-transferred data.  The call will
  *   automatically adjust the allocated buffer size to account for any
  *   additional padding that may be necessary.
  * @gfp_mask: GFP mask used for the memory allocation.
@@ -448,11 +555,16 @@ int ib_process_mad_wc(struct ib_mad_agen
  * This is a helper routine that may be used to allocate a MAD.  Users are
  * not required to allocate outbound MADs using this call.  The returned
  * MAD send buffer will reference a data buffer usable for sending a MAD, along
- * with an intialized work request structure.
+ * with an initialized work request structure.  Users may modify the returned
+ * MAD data buffer or work request before posting the send.
+ *
+ * The returned data buffer will be cleared.  Users are responsible for
+ * initializing the common MAD and any class specific headers.  If @rmpp_active
+ * is set, the RMPP header will be initialized for sending.
  */
 struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent,
 					    u32 remote_qpn, u16 pkey_index,
-					    struct ib_ah *ah,
+					    struct ib_ah *ah, int rmpp_active,
 					    int hdr_len, int data_len,
 					    unsigned int __nocast gfp_mask);
 
diff -uprN linux-2.6.13-rc2-mm1-17/drivers/infiniband/include/ib_sa.h linux-2.6.13-rc2-mm1-18/drivers/infiniband/include/ib_sa.h
-- linux-2.6.13-rc2-mm1-17/drivers/infiniband/include/ib_sa.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-18/drivers/infiniband/include/ib_sa.h	2005-07-10 12:07:41.000000000 -0400
@@ -87,10 +87,6 @@ static inline int ib_sa_rate_enum_to_int
 	}
 }
 
-typedef u64 __bitwise ib_sa_comp_mask;
-
-#define IB_SA_COMP_MASK(n)	((__force ib_sa_comp_mask) cpu_to_be64(1ull << n))
-
 /*
  * Structures for SA records are named "struct ib_sa_xxx_rec."  No
  * attempt is made to pack structures to match the physical layout of




From halr at voltaire.com  Mon Jul 11 13:36:11 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:36:11 -0400
Subject: [openib-general] [PATCH 19/29v2] Add RMPP implementation
Message-ID: <1121110354.4389.5022.camel@hal.voltaire.com>

Add RMPP implementation 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 18/29.

-- 
 Makefile   |    2 
 mad.c      |  167 +++++++++--
 mad_priv.h |   28 +-
 mad_rmpp.c |  765 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 mad_rmpp.h |   58 ++++
 5 files changed, 968 insertions(+), 52 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad.c linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad.c
-- linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad.c	2005-07-11 13:39:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad.c	2005-07-11 13:40:04.000000000 -0400
@@ -1,5 +1,7 @@
 /*
  * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved.
+ * Copyright (c) 2005 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2005 Mellanox Technologies Ltd.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,12 +31,12 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: mad.c 1389 2004-12-27 22:56:47Z roland $
+ * $Id: mad.c 2817 2005-07-07 11:29:26Z halr $
  */
-
 #include <linux/dma-mapping.h>
 
 #include "mad_priv.h"
+#include "mad_rmpp.h"
 #include "smi.h"
 #include "agent.h"
 
@@ -45,6 +47,7 @@ MODULE_AUTHOR("Sean Hefty");
 

 kmem_cache_t *ib_mad_cache;
+
 static struct list_head ib_mad_port_list;
 static u32 ib_mad_client_id = 0;
 
@@ -62,8 +65,6 @@ static struct ib_mad_agent_private *find
 static int ib_mad_post_receive_mads(struct ib_mad_qp_info *qp_info,
 				    struct ib_mad_private *mad);
 static void cancel_mads(struct ib_mad_agent_private *mad_agent_priv);
-static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
-				    struct ib_mad_send_wc *mad_send_wc);
 static void timeout_sends(void *data);
 static void local_completions(void *data);
 static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req,
@@ -195,8 +196,8 @@ struct ib_mad_agent *ib_register_mad_age
 	if (qpn == -1)
 		goto error1;
 
-	if (rmpp_version)
-		goto error1;	/* XXX: until RMPP implemented */
+	if (rmpp_version && rmpp_version != IB_MGMT_RMPP_VERSION)
+		goto error1;
 
 	/* Validate MAD registration request if supplied */
 	if (mad_reg_req) {
@@ -281,7 +282,7 @@ struct ib_mad_agent *ib_register_mad_age
 	/* Now, fill in the various structures */
 	mad_agent_priv->qp_info = &port_priv->qp_info[qpn];
 	mad_agent_priv->reg_req = reg_req;
-	mad_agent_priv->rmpp_version = rmpp_version;
+	mad_agent_priv->agent.rmpp_version = rmpp_version;
 	mad_agent_priv->agent.device = device;
 	mad_agent_priv->agent.recv_handler = recv_handler;
 	mad_agent_priv->agent.send_handler = send_handler;
@@ -341,6 +342,7 @@ struct ib_mad_agent *ib_register_mad_age
 	INIT_LIST_HEAD(&mad_agent_priv->send_list);
 	INIT_LIST_HEAD(&mad_agent_priv->wait_list);
 	INIT_LIST_HEAD(&mad_agent_priv->done_list);
+	INIT_LIST_HEAD(&mad_agent_priv->rmpp_list);
 	INIT_WORK(&mad_agent_priv->timed_work, timeout_sends, mad_agent_priv);
 	INIT_LIST_HEAD(&mad_agent_priv->local_list);
 	INIT_WORK(&mad_agent_priv->local_work, local_completions,
@@ -502,6 +504,7 @@ static void unregister_mad_agent(struct 
 	spin_unlock_irqrestore(&port_priv->reg_lock, flags);
 
 	flush_workqueue(port_priv->wq);
+	ib_cancel_rmpp_recvs(mad_agent_priv);
 
 	atomic_dec(&mad_agent_priv->refcount);
 	wait_event(mad_agent_priv->wait,
@@ -689,7 +692,7 @@ static int handle_outgoing_dr_smp(struct
 		kfree(local);
 		goto out;
 	}
-
+	
 	build_smp_wc(send_wr->wr_id, smp->dr_slid, send_wr->wr.ud.pkey_index,
 		     send_wr->wr.ud.port_num, &mad_wc);
 
@@ -705,7 +708,7 @@ static int handle_outgoing_dr_smp(struct
 			local->mad_priv = mad_priv;
 			local->recv_mad_agent = mad_agent_priv;
 			/*
-			 * Reference MAD agent until receive
+			 * Reference MAD agent until receive 
 			 * side of local completion handled
 			 */
 			atomic_inc(&mad_agent_priv->refcount);
@@ -786,12 +789,15 @@ struct ib_mad_send_buf * ib_create_send_
 	int buf_size;
 	void *buf;
 
-	if (rmpp_active)
-		return ERR_PTR(-EINVAL);	/* until RMPP implemented */
 	mad_agent_priv = container_of(mad_agent,
 				      struct ib_mad_agent_private, agent);
 	buf_size = get_buf_length(hdr_len, data_len);
 
+	if ((!mad_agent->rmpp_version &&
+	     (rmpp_active || buf_size > sizeof(struct ib_mad))) ||
+	    (!rmpp_active && buf_size > sizeof(struct ib_mad)))
+		return ERR_PTR(-EINVAL);
+
 	buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask);
 	if (!buf)
 		return ERR_PTR(-ENOMEM);
@@ -816,6 +822,18 @@ struct ib_mad_send_buf * ib_create_send_
 	send_buf->send_wr.wr.ud.remote_qpn = remote_qpn;
 	send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY;
 	send_buf->send_wr.wr.ud.pkey_index = pkey_index;
+
+	if (rmpp_active) {
+		struct ib_rmpp_mad *rmpp_mad;
+		rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad;
+		rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len -
+			offsetof(struct ib_rmpp_mad, data) + data_len);
+		rmpp_mad->rmpp_hdr.rmpp_version = mad_agent->rmpp_version;
+		rmpp_mad->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_DATA;
+		ib_set_rmpp_flags(&rmpp_mad->rmpp_hdr,
+				  IB_MGMT_RMPP_FLAG_ACTIVE);
+	}
+
 	send_buf->mad_agent = mad_agent;
 	atomic_inc(&mad_agent_priv->refcount);
 	return send_buf;
@@ -839,7 +857,7 @@ void ib_free_send_mad(struct ib_mad_send
 }
 EXPORT_SYMBOL(ib_free_send_mad);
 
-static int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr)
+int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr)
 {
 	struct ib_mad_qp_info *qp_info;
 	struct ib_send_wr *bad_send_wr;
@@ -940,13 +958,13 @@ int ib_post_send_mad(struct ib_mad_agent
 			ret = -ENOMEM;
 			goto error2;
 		}
+		memset(mad_send_wr, 0, sizeof *mad_send_wr);
 
 		mad_send_wr->send_wr = *send_wr;
 		mad_send_wr->send_wr.sg_list = mad_send_wr->sg_list;
 		memcpy(mad_send_wr->sg_list, send_wr->sg_list,
 		       sizeof *send_wr->sg_list * send_wr->num_sge);
-		mad_send_wr->wr_id = mad_send_wr->send_wr.wr_id;
-		mad_send_wr->send_wr.next = NULL;
+		mad_send_wr->wr_id = send_wr->wr_id;
 		mad_send_wr->tid = send_wr->wr.ud.mad_hdr->tid;
 		mad_send_wr->mad_agent_priv = mad_agent_priv;
 		/* Timeout will be updated after send completes */
@@ -964,8 +982,13 @@ int ib_post_send_mad(struct ib_mad_agent
 			      &mad_agent_priv->send_list);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
-		ret = ib_send_mad(mad_send_wr);
-		if (ret) {
+		if (mad_agent_priv->agent.rmpp_version) {
+			ret = ib_send_rmpp_mad(mad_send_wr);
+			if (ret >= 0 && ret != IB_RMPP_RESULT_CONSUMED)
+				ret = ib_send_mad(mad_send_wr);
+		} else
+			ret = ib_send_mad(mad_send_wr);
+		if (ret < 0) {
 			/* Fail send request */
 			spin_lock_irqsave(&mad_agent_priv->lock, flags);
 			list_del(&mad_send_wr->agent_list);
@@ -991,31 +1014,25 @@ EXPORT_SYMBOL(ib_post_send_mad);
  */
 void ib_free_recv_mad(struct ib_mad_recv_wc *mad_recv_wc)
 {
-	struct ib_mad_recv_buf *entry;
+	struct ib_mad_recv_buf *mad_recv_buf, *temp_recv_buf;
 	struct ib_mad_private_header *mad_priv_hdr;
 	struct ib_mad_private *priv;
+	struct list_head free_list;
 
-	mad_priv_hdr = container_of(mad_recv_wc,
-				    struct ib_mad_private_header,
-				    recv_wc);
-	priv = container_of(mad_priv_hdr, struct ib_mad_private, header);
+	INIT_LIST_HEAD(&free_list);
+	list_splice_init(&mad_recv_wc->rmpp_list, &free_list);
 
-	/*
-	 * Walk receive buffer list associated with this WC
-	 * No need to remove them from list of receive buffers
-	 */
-	list_for_each_entry(entry, &mad_recv_wc->recv_buf.list, list) {
-		/* Free previous receive buffer */
-		kmem_cache_free(ib_mad_cache, priv);
+	list_for_each_entry_safe(mad_recv_buf, temp_recv_buf,
+					&free_list, list) {
+		mad_recv_wc = container_of(mad_recv_buf, struct ib_mad_recv_wc,
+					   recv_buf);
 		mad_priv_hdr = container_of(mad_recv_wc,
 					    struct ib_mad_private_header,
 					    recv_wc);
 		priv = container_of(mad_priv_hdr, struct ib_mad_private,
 				    header);
+		kmem_cache_free(ib_mad_cache, priv);
 	}
-
-	/* Free last buffer */
-	kmem_cache_free(ib_mad_cache, priv);
 }
 EXPORT_SYMBOL(ib_free_recv_mad);
 
@@ -1524,9 +1541,20 @@ out:
 	return valid;
 }
 
-static struct ib_mad_send_wr_private*
-find_send_req(struct ib_mad_agent_private *mad_agent_priv,
-	      u64 tid)
+static int is_data_mad(struct ib_mad_agent_private *mad_agent_priv,
+		       struct ib_mad_hdr *mad_hdr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_hdr;
+	return !mad_agent_priv->agent.rmpp_version ||
+		!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) &
+				    IB_MGMT_RMPP_FLAG_ACTIVE) ||
+		(rmpp_mad->rmpp_hdr.rmpp_type == IB_MGMT_RMPP_TYPE_DATA);
+}
+
+struct ib_mad_send_wr_private*
+ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, u64 tid)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 
@@ -1542,7 +1570,9 @@ find_send_req(struct ib_mad_agent_privat
 	 */
 	list_for_each_entry(mad_send_wr, &mad_agent_priv->send_list,
 			    agent_list) {
-		if (mad_send_wr->tid == tid && mad_send_wr->timeout) {
+		if (is_data_mad(mad_agent_priv,
+				mad_send_wr->send_wr.wr.ud.mad_hdr) &&
+		    mad_send_wr->tid == tid && mad_send_wr->timeout) {
 			/* Verify request has not been canceled */
 			return (mad_send_wr->status == IB_WC_SUCCESS) ?
 				mad_send_wr : NULL;
@@ -1551,7 +1581,7 @@ find_send_req(struct ib_mad_agent_privat
 	return NULL;
 }
 
-static void ib_mark_req_done(struct ib_mad_send_wr_private *mad_send_wr)
+void ib_mark_mad_done(struct ib_mad_send_wr_private *mad_send_wr)
 {
 	mad_send_wr->timeout = 0;
 	if (mad_send_wr->refcount == 1) {
@@ -1569,12 +1599,23 @@ static void ib_mad_complete_recv(struct 
 	unsigned long flags;
 	u64 tid;
 
-	INIT_LIST_HEAD(&mad_recv_wc->recv_buf.list);
+	INIT_LIST_HEAD(&mad_recv_wc->rmpp_list);
+	list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list);
+	if (mad_agent_priv->agent.rmpp_version) {
+		mad_recv_wc = ib_process_rmpp_recv_wc(mad_agent_priv,
+						      mad_recv_wc);
+		if (!mad_recv_wc) {
+			if (atomic_dec_and_test(&mad_agent_priv->refcount))
+				wake_up(&mad_agent_priv->wait);
+			return;
+		}
+	}
+
 	/* Complete corresponding request */
 	if (response_mad(mad_recv_wc->recv_buf.mad)) {
 		tid = mad_recv_wc->recv_buf.mad->mad_hdr.tid;
 		spin_lock_irqsave(&mad_agent_priv->lock, flags);
-		mad_send_wr = find_send_req(mad_agent_priv, tid);
+		mad_send_wr = ib_find_send_mad(mad_agent_priv, tid);
 		if (!mad_send_wr) {
 			spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 			ib_free_recv_mad(mad_recv_wc);
@@ -1582,7 +1623,7 @@ static void ib_mad_complete_recv(struct 
 				wake_up(&mad_agent_priv->wait);
 			return;
 		}
-		ib_mark_req_done(mad_send_wr);
+		ib_mark_mad_done(mad_send_wr);
 		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 
 		/* Defined behavior is to complete response before request */
@@ -1787,14 +1828,22 @@ void ib_reset_mad_timeout(struct ib_mad_
 /*
  * Process a send work completion
  */
-static void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
-				    struct ib_mad_send_wc *mad_send_wc)
+void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
+			     struct ib_mad_send_wc *mad_send_wc)
 {
 	struct ib_mad_agent_private	*mad_agent_priv;
 	unsigned long			flags;
+	int				ret;
 
 	mad_agent_priv = mad_send_wr->mad_agent_priv;
 	spin_lock_irqsave(&mad_agent_priv->lock, flags);
+	if (mad_agent_priv->agent.rmpp_version) {
+		ret = ib_process_rmpp_send_wc(mad_send_wr, mad_send_wc);
+		if (ret == IB_RMPP_RESULT_CONSUMED)
+			goto done;
+	} else
+		ret = IB_RMPP_RESULT_UNHANDLED;
+
 	if (mad_send_wc->status != IB_WC_SUCCESS &&
 	    mad_send_wr->status == IB_WC_SUCCESS) {
 		mad_send_wr->status = mad_send_wc->status;
@@ -1806,8 +1855,7 @@ static void ib_mad_complete_send_wr(stru
 		    mad_send_wr->status == IB_WC_SUCCESS) {
 			wait_for_response(mad_send_wr);
 		}
-		spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
-		return;
+		goto done;
 	}
 
 	/* Remove send from MAD agent and notify client of completion */
@@ -1817,14 +1865,18 @@ static void ib_mad_complete_send_wr(stru
 
 	if (mad_send_wr->status != IB_WC_SUCCESS )
 		mad_send_wc->status = mad_send_wr->status;
-	mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
-					    mad_send_wc);
+	if (ret != IB_RMPP_RESULT_INTERNAL)
+		mad_agent_priv->agent.send_handler(&mad_agent_priv->agent,
+						   mad_send_wc);
 
 	/* Release reference on agent taken when sending */
 	if (atomic_dec_and_test(&mad_agent_priv->refcount))
 		wake_up(&mad_agent_priv->wait);
 
 	kfree(mad_send_wr);
+	return;
+done:
+	spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
 }
 
 static void ib_mad_send_done_handler(struct ib_mad_port_private *port_priv,
@@ -2036,7 +2088,9 @@ find_send_by_wr_id(struct ib_mad_agent_p
 
 	list_for_each_entry(mad_send_wr, &mad_agent_priv->send_list,
 			    agent_list) {
-		if (mad_send_wr->wr_id == wr_id)
+		if (is_data_mad(mad_agent_priv,
+				mad_send_wr->send_wr.wr.ud.mad_hdr) &&
+		    mad_send_wr->wr_id == wr_id)
 			return mad_send_wr;
 	}
 	return NULL;
@@ -2118,7 +2172,9 @@ static void local_completions(void *data
 			local->mad_priv->header.recv_wc.wc = &wc;
 			local->mad_priv->header.recv_wc.mad_len =
 						sizeof(struct ib_mad);
-			INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.recv_buf.list);
+			INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.rmpp_list);
+			list_add(&local->mad_priv->header.recv_wc.recv_buf.list,
+				 &local->mad_priv->header.recv_wc.rmpp_list);
 			local->mad_priv->header.recv_wc.recv_buf.grh = NULL;
 			local->mad_priv->header.recv_wc.recv_buf.mad =
 						&local->mad_priv->mad.mad;
@@ -2166,7 +2222,21 @@ static int retry_send(struct ib_mad_send
 	mad_send_wr->timeout = msecs_to_jiffies(mad_send_wr->send_wr.
 						wr.ud.timeout_ms);
 
-	ret = ib_send_mad(mad_send_wr);
+	if (mad_send_wr->mad_agent_priv->agent.rmpp_version) {
+		ret = ib_retry_rmpp(mad_send_wr);
+		switch (ret) {
+		case IB_RMPP_RESULT_UNHANDLED:
+			ret = ib_send_mad(mad_send_wr);
+			break;
+		case IB_RMPP_RESULT_CONSUMED:
+			ret = 0;
+			break;
+		default:
+			ret = -ECOMM;
+			break;
+		}
+	} else
+		ret = ib_send_mad(mad_send_wr);
 
 	if (!ret) {
 		mad_send_wr->refcount++;
@@ -2724,3 +2794,4 @@ static void __exit ib_mad_cleanup_module
 
 module_init(ib_mad_init_module);
 module_exit(ib_mad_cleanup_module);
+
diff -uprN linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad_priv.h linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad_priv.h
-- linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad_priv.h	2005-07-09 17:57:55.000000000 -0400
+++ linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad_priv.h	2005-07-10 12:12:10.000000000 -0400
@@ -1,5 +1,7 @@
 /*
  * Copyright (c) 2004, 2005, Voltaire, Inc. All rights reserved.
+ * Copyright (c) 2005 Intel Corporation. All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +31,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: mad_priv.h 1980 2005-03-11 22:33:53Z sean.hefty $
+ * $Id: mad_priv.h 2730 2005-06-28 16:43:03Z sean.hefty $
  */
 
 #ifndef __IB_MAD_PRIV_H__
@@ -97,10 +99,10 @@ struct ib_mad_agent_private {
 	unsigned long timeout;
 	struct list_head local_list;
 	struct work_struct local_work;
+	struct list_head rmpp_list;
 
 	atomic_t refcount;
 	wait_queue_head_t wait;
-	u8 rmpp_version;
 };
 
 struct ib_mad_snoop_private {
@@ -125,6 +127,14 @@ struct ib_mad_send_wr_private {
 	int retry;
 	int refcount;
 	enum ib_wc_status status;
+
+	/* RMPP control */
+	int last_ack;
+	int seg_num;
+	int newwin;
+	int total_seg;
+	int data_offset;
+	int pad;
 };
 
 struct ib_mad_local_private {
@@ -135,7 +145,6 @@ struct ib_mad_local_private {
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */
 	u64 tid;
-	int retries;
 };
 
 struct ib_mad_mgmt_method_table {
@@ -198,4 +207,17 @@ struct ib_mad_port_private {
 
 extern kmem_cache_t *ib_mad_cache;
 
+int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr);
+
+struct ib_mad_send_wr_private *
+ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, u64 tid);
+
+void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
+			     struct ib_mad_send_wc *mad_send_wc);
+
+void ib_mark_mad_done(struct ib_mad_send_wr_private *mad_send_wr);
+
+void ib_reset_mad_timeout(struct ib_mad_send_wr_private *mad_send_wr,
+			  int timeout_ms);
+
 #endif	/* __IB_MAD_PRIV_H__ */
diff -uprN linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad_rmpp.c linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad_rmpp.c
-- linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad_rmpp.c	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad_rmpp.c	2005-07-06 16:52:21.000000000 -0400
@@ -0,0 +1,765 @@
+/*
+ * Copyright (c) 2005 Intel Inc. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: mad_rmpp.c 1921 2005-03-02 22:58:44Z sean.hefty $
+ */
+
+#include <linux/dma-mapping.h>
+
+#include "mad_priv.h"
+#include "mad_rmpp.h"
+
+enum rmpp_state {
+	RMPP_STATE_ACTIVE,
+	RMPP_STATE_TIMEOUT,
+	RMPP_STATE_COMPLETE
+};
+
+struct mad_rmpp_recv {
+	struct ib_mad_agent_private *agent;
+	struct list_head list;
+	struct work_struct timeout_work;
+	struct work_struct cleanup_work;
+	wait_queue_head_t wait;
+	enum rmpp_state state;
+	spinlock_t lock;
+	atomic_t refcount;
+
+	struct ib_ah *ah;
+	struct ib_mad_recv_wc *rmpp_wc;
+	struct ib_mad_recv_buf *cur_seg_buf;
+	int last_ack;
+	int seg_num;
+	int newwin;
+
+	u64 tid;
+	u32 src_qp;
+	u16 slid;
+	u8 mgmt_class;
+	u8 class_version;
+	u8 method;
+};
+
+static void destroy_rmpp_recv(struct mad_rmpp_recv *rmpp_recv)
+{
+	atomic_dec(&rmpp_recv->refcount);
+	wait_event(rmpp_recv->wait, !atomic_read(&rmpp_recv->refcount));
+	ib_destroy_ah(rmpp_recv->ah);
+	kfree(rmpp_recv);
+}
+
+void ib_cancel_rmpp_recvs(struct ib_mad_agent_private *agent)
+{
+	struct mad_rmpp_recv *rmpp_recv, *temp_rmpp_recv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&agent->lock, flags);
+	list_for_each_entry(rmpp_recv, &agent->rmpp_list, list) {
+		cancel_delayed_work(&rmpp_recv->timeout_work);
+		cancel_delayed_work(&rmpp_recv->cleanup_work);
+	}
+	spin_unlock_irqrestore(&agent->lock, flags);
+
+	flush_workqueue(agent->qp_info->port_priv->wq);
+
+	list_for_each_entry_safe(rmpp_recv, temp_rmpp_recv,
+				 &agent->rmpp_list, list) {
+		list_del(&rmpp_recv->list);
+		if (rmpp_recv->state != RMPP_STATE_COMPLETE)
+			ib_free_recv_mad(rmpp_recv->rmpp_wc);
+		destroy_rmpp_recv(rmpp_recv);
+	}
+}
+
+static void recv_timeout_handler(void *data)
+{
+	struct mad_rmpp_recv *rmpp_recv = data;
+	struct ib_mad_recv_wc *rmpp_wc;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rmpp_recv->agent->lock, flags);
+	if (rmpp_recv->state != RMPP_STATE_ACTIVE) {
+		spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
+		return;
+	}
+	rmpp_recv->state = RMPP_STATE_TIMEOUT;
+	list_del(&rmpp_recv->list);
+	spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
+
+	/* TODO: send abort. */
+	rmpp_wc = rmpp_recv->rmpp_wc;
+	destroy_rmpp_recv(rmpp_recv);
+	ib_free_recv_mad(rmpp_wc);
+}
+
+static void recv_cleanup_handler(void *data)
+{
+	struct mad_rmpp_recv *rmpp_recv = data;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rmpp_recv->agent->lock, flags);
+	list_del(&rmpp_recv->list);
+	spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
+	destroy_rmpp_recv(rmpp_recv);
+}
+
+static struct mad_rmpp_recv *
+create_rmpp_recv(struct ib_mad_agent_private *agent,
+		 struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	struct ib_mad_hdr *mad_hdr;
+
+	rmpp_recv = kmalloc(sizeof *rmpp_recv, GFP_KERNEL);
+	if (!rmpp_recv)
+		return NULL;
+
+	rmpp_recv->ah = ib_create_ah_from_wc(agent->agent.qp->pd,
+					     mad_recv_wc->wc,
+					     mad_recv_wc->recv_buf.grh,
+					     agent->agent.port_num);
+	if (IS_ERR(rmpp_recv->ah))
+		goto error;
+
+	rmpp_recv->agent = agent;
+	init_waitqueue_head(&rmpp_recv->wait);
+	INIT_WORK(&rmpp_recv->timeout_work, recv_timeout_handler, rmpp_recv);
+	INIT_WORK(&rmpp_recv->cleanup_work, recv_cleanup_handler, rmpp_recv);
+	spin_lock_init(&rmpp_recv->lock);
+	rmpp_recv->state = RMPP_STATE_ACTIVE;
+	atomic_set(&rmpp_recv->refcount, 1);
+
+	rmpp_recv->rmpp_wc = mad_recv_wc;
+	rmpp_recv->cur_seg_buf = &mad_recv_wc->recv_buf;
+	rmpp_recv->newwin = 1;
+	rmpp_recv->seg_num = 1;
+	rmpp_recv->last_ack = 0;
+
+	mad_hdr = &mad_recv_wc->recv_buf.mad->mad_hdr;
+	rmpp_recv->tid = mad_hdr->tid;
+	rmpp_recv->src_qp = mad_recv_wc->wc->src_qp;
+	rmpp_recv->slid = mad_recv_wc->wc->slid;
+	rmpp_recv->mgmt_class = mad_hdr->mgmt_class;
+	rmpp_recv->class_version = mad_hdr->class_version;
+	rmpp_recv->method  = mad_hdr->method;
+	return rmpp_recv;
+
+error:	kfree(rmpp_recv);
+	return NULL;
+}
+
+static inline void deref_rmpp_recv(struct mad_rmpp_recv *rmpp_recv)
+{
+	if (atomic_dec_and_test(&rmpp_recv->refcount))
+		wake_up(&rmpp_recv->wait);
+}
+
+static struct mad_rmpp_recv *
+find_rmpp_recv(struct ib_mad_agent_private *agent,
+	       struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	struct ib_mad_hdr *mad_hdr = &mad_recv_wc->recv_buf.mad->mad_hdr;
+
+	list_for_each_entry(rmpp_recv, &agent->rmpp_list, list) {
+		if (rmpp_recv->tid == mad_hdr->tid &&
+		    rmpp_recv->src_qp == mad_recv_wc->wc->src_qp &&
+		    rmpp_recv->slid == mad_recv_wc->wc->slid &&
+		    rmpp_recv->mgmt_class == mad_hdr->mgmt_class &&
+		    rmpp_recv->class_version == mad_hdr->class_version &&
+		    rmpp_recv->method == mad_hdr->method)
+			return rmpp_recv;
+	}
+	return NULL;
+}
+
+static struct mad_rmpp_recv *
+acquire_rmpp_recv(struct ib_mad_agent_private *agent,
+		  struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&agent->lock, flags);
+	rmpp_recv = find_rmpp_recv(agent, mad_recv_wc);
+	if (rmpp_recv)
+		atomic_inc(&rmpp_recv->refcount);
+	spin_unlock_irqrestore(&agent->lock, flags);
+	return rmpp_recv;
+}
+
+static struct mad_rmpp_recv *
+insert_rmpp_recv(struct ib_mad_agent_private *agent,
+		 struct mad_rmpp_recv *rmpp_recv)
+{
+	struct mad_rmpp_recv *cur_rmpp_recv;
+
+	cur_rmpp_recv = find_rmpp_recv(agent, rmpp_recv->rmpp_wc);
+	if (!cur_rmpp_recv)
+		list_add_tail(&rmpp_recv->list, &agent->rmpp_list);
+
+	return cur_rmpp_recv;
+}
+
+static int data_offset(u8 mgmt_class)
+{
+	if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM)
+		return offsetof(struct ib_sa_mad, data);
+	else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) &&
+		 (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END))
+		return offsetof(struct ib_vendor_mad, data);
+	else
+		return offsetof(struct ib_rmpp_mad, data);
+}
+
+static void format_ack(struct ib_rmpp_mad *ack,
+		       struct ib_rmpp_mad *data,
+		       struct mad_rmpp_recv *rmpp_recv)
+{
+	unsigned long flags;
+
+	memcpy(&ack->mad_hdr, &data->mad_hdr,
+	       data_offset(data->mad_hdr.mgmt_class));
+
+	ack->mad_hdr.method ^= IB_MGMT_METHOD_RESP;
+	ack->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_ACK;
+	ib_set_rmpp_flags(&ack->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
+
+	spin_lock_irqsave(&rmpp_recv->lock, flags);
+	rmpp_recv->last_ack = rmpp_recv->seg_num;
+	ack->rmpp_hdr.seg_num = cpu_to_be32(rmpp_recv->seg_num);
+	ack->rmpp_hdr.paylen_newwin = cpu_to_be32(rmpp_recv->newwin);
+	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+}
+
+static void ack_recv(struct mad_rmpp_recv *rmpp_recv,
+		     struct ib_mad_recv_wc *recv_wc)
+{
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	int hdr_len, ret;
+
+	hdr_len = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr);
+	msg = ib_create_send_mad(&rmpp_recv->agent->agent, recv_wc->wc->src_qp,
+				 recv_wc->wc->pkey_index, rmpp_recv->ah, 1,
+				 hdr_len, sizeof(struct ib_rmpp_mad) - hdr_len,
+				 GFP_KERNEL);
+	if (!msg)
+		return;
+
+	format_ack((struct ib_rmpp_mad *) msg->mad,
+		   (struct ib_rmpp_mad *) recv_wc->recv_buf.mad, rmpp_recv);
+	ret = ib_post_send_mad(&rmpp_recv->agent->agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		ib_free_send_mad(msg);
+}
+
+static inline int get_last_flag(struct ib_mad_recv_buf *seg)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *) seg->mad;
+	return ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_LAST;
+}
+
+static inline int get_seg_num(struct ib_mad_recv_buf *seg)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *) seg->mad;
+	return be32_to_cpu(rmpp_mad->rmpp_hdr.seg_num);
+}
+
+static inline struct ib_mad_recv_buf * get_next_seg(struct list_head *rmpp_list,
+						    struct ib_mad_recv_buf *seg)
+{
+	if (seg->list.next == rmpp_list)
+		return NULL;
+
+	return container_of(seg->list.next, struct ib_mad_recv_buf, list);
+}
+
+static inline int window_size(struct ib_mad_agent_private *agent)
+{
+	return max(agent->qp_info->recv_queue.max_active >> 3, 1);
+}
+
+static struct ib_mad_recv_buf * find_seg_location(struct list_head *rmpp_list,
+						  int seg_num)
+{
+        struct ib_mad_recv_buf *seg_buf;
+	int cur_seg_num;
+
+	list_for_each_entry_reverse(seg_buf, rmpp_list, list) {
+		cur_seg_num = get_seg_num(seg_buf);
+		if (seg_num > cur_seg_num)
+			return seg_buf;
+		if (seg_num == cur_seg_num)
+			break;
+	}
+	return NULL;
+}
+
+static void update_seg_num(struct mad_rmpp_recv *rmpp_recv,
+			   struct ib_mad_recv_buf *new_buf)
+{
+	struct list_head *rmpp_list = &rmpp_recv->rmpp_wc->rmpp_list;
+	
+	while (new_buf && (get_seg_num(new_buf) == rmpp_recv->seg_num + 1)) {
+		rmpp_recv->cur_seg_buf = new_buf;
+		rmpp_recv->seg_num++;
+		new_buf = get_next_seg(rmpp_list, new_buf);
+	}
+}
+
+static inline int get_mad_len(struct mad_rmpp_recv *rmpp_recv)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int hdr_size, data_size, pad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)rmpp_recv->cur_seg_buf->mad;
+
+	hdr_size = data_offset(rmpp_mad->mad_hdr.mgmt_class);
+	data_size = sizeof(struct ib_rmpp_mad) - hdr_size;
+	pad = data_size - be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+	if (pad > data_size || pad < 0)
+		pad = 0;
+
+	return hdr_size + rmpp_recv->seg_num * data_size - pad;
+}
+
+static struct ib_mad_recv_wc * complete_rmpp(struct mad_rmpp_recv *rmpp_recv)
+{
+	struct ib_mad_recv_wc *rmpp_wc;
+
+	ack_recv(rmpp_recv, rmpp_recv->rmpp_wc);
+	if (rmpp_recv->seg_num > 1)
+		cancel_delayed_work(&rmpp_recv->timeout_work);
+
+	rmpp_wc = rmpp_recv->rmpp_wc;
+	rmpp_wc->mad_len = get_mad_len(rmpp_recv);
+	/* 10 seconds until we can find the packet lifetime */
+	queue_delayed_work(rmpp_recv->agent->qp_info->port_priv->wq,
+			   &rmpp_recv->cleanup_work, msecs_to_jiffies(10000));
+	return rmpp_wc;
+}
+
+void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf)
+{
+	struct ib_mad_recv_buf *seg_buf;
+	struct ib_rmpp_mad *rmpp_mad;
+	void *data;
+	int size, len, offset;
+	u8 flags;
+
+	len = mad_recv_wc->mad_len;
+	if (len <= sizeof(struct ib_mad)) {
+		memcpy(buf, mad_recv_wc->recv_buf.mad, len);
+		return;
+	}
+
+	offset = data_offset(mad_recv_wc->recv_buf.mad->mad_hdr.mgmt_class);
+
+	list_for_each_entry(seg_buf, &mad_recv_wc->rmpp_list, list) {
+		rmpp_mad = (struct ib_rmpp_mad *)seg_buf->mad;
+		flags = ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr);
+
+		if (flags & IB_MGMT_RMPP_FLAG_FIRST) {
+			data = rmpp_mad;
+			size = sizeof(*rmpp_mad);
+		} else {
+			data = (void *) rmpp_mad + offset;
+			if (flags & IB_MGMT_RMPP_FLAG_LAST) 
+				size = len;
+			else
+				size = sizeof(*rmpp_mad) - offset;
+		}
+
+		memcpy(buf, data, size);
+		len -= size;
+		buf += size;
+	}
+}
+EXPORT_SYMBOL(ib_coalesce_recv_mad);
+
+static struct ib_mad_recv_wc *
+continue_rmpp(struct ib_mad_agent_private *agent,
+	      struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	struct ib_mad_recv_buf *prev_buf;
+	struct ib_mad_recv_wc *done_wc;
+	int seg_num;
+	unsigned long flags;
+
+	rmpp_recv = acquire_rmpp_recv(agent, mad_recv_wc);
+	if (!rmpp_recv)
+		goto drop1;
+
+	seg_num = get_seg_num(&mad_recv_wc->recv_buf);
+
+	spin_lock_irqsave(&rmpp_recv->lock, flags);
+	if ((rmpp_recv->state == RMPP_STATE_TIMEOUT) ||
+	    (seg_num > rmpp_recv->newwin))
+		goto drop3;
+
+	if ((seg_num <= rmpp_recv->last_ack) ||
+	    (rmpp_recv->state == RMPP_STATE_COMPLETE)) {
+		spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+		ack_recv(rmpp_recv, mad_recv_wc);
+		goto drop2;
+	}
+
+	prev_buf = find_seg_location(&rmpp_recv->rmpp_wc->rmpp_list, seg_num);
+	if (!prev_buf)
+		goto drop3;
+
+	done_wc = NULL;
+	list_add(&mad_recv_wc->recv_buf.list, &prev_buf->list);
+	if (rmpp_recv->cur_seg_buf == prev_buf) {
+		update_seg_num(rmpp_recv, &mad_recv_wc->recv_buf);
+		if (get_last_flag(rmpp_recv->cur_seg_buf)) {
+			rmpp_recv->state = RMPP_STATE_COMPLETE;
+			spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+			done_wc = complete_rmpp(rmpp_recv);
+			goto out;
+		} else if (rmpp_recv->seg_num == rmpp_recv->newwin) {
+			rmpp_recv->newwin += window_size(agent);
+			spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+			ack_recv(rmpp_recv, mad_recv_wc);
+			goto out;
+		}
+	}
+	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+out:
+	deref_rmpp_recv(rmpp_recv);
+	return done_wc;
+
+drop3:	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+drop2:	deref_rmpp_recv(rmpp_recv);
+drop1:	ib_free_recv_mad(mad_recv_wc);
+	return NULL;
+}
+
+static struct ib_mad_recv_wc *
+start_rmpp(struct ib_mad_agent_private *agent,
+	   struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct mad_rmpp_recv *rmpp_recv;
+	unsigned long flags;
+
+	rmpp_recv = create_rmpp_recv(agent, mad_recv_wc);
+	if (!rmpp_recv) {
+		ib_free_recv_mad(mad_recv_wc);
+		return NULL;
+	}
+
+	spin_lock_irqsave(&agent->lock, flags);
+	if (insert_rmpp_recv(agent, rmpp_recv)) {
+		spin_unlock_irqrestore(&agent->lock, flags);
+		/* duplicate first MAD */
+		destroy_rmpp_recv(rmpp_recv);
+		return continue_rmpp(agent, mad_recv_wc);
+	}
+	atomic_inc(&rmpp_recv->refcount);
+
+	if (get_last_flag(&mad_recv_wc->recv_buf)) {
+		rmpp_recv->state = RMPP_STATE_COMPLETE;
+		spin_unlock_irqrestore(&agent->lock, flags);
+		complete_rmpp(rmpp_recv);
+	} else {
+		spin_unlock_irqrestore(&agent->lock, flags);
+		/* 40 seconds until we can find the packet lifetimes */
+		queue_delayed_work(agent->qp_info->port_priv->wq,
+				   &rmpp_recv->timeout_work,
+				   msecs_to_jiffies(40000));
+		rmpp_recv->newwin += window_size(agent);
+		ack_recv(rmpp_recv, mad_recv_wc);
+		mad_recv_wc = NULL;
+	}
+	deref_rmpp_recv(rmpp_recv);
+	return mad_recv_wc;
+}
+
+static inline u64 get_seg_addr(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	return mad_send_wr->sg_list[0].addr + mad_send_wr->data_offset +
+	       (sizeof(struct ib_rmpp_mad) - mad_send_wr->data_offset) *
+	       (mad_send_wr->seg_num - 1);
+}
+
+static int send_next_seg(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int timeout;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	ib_set_rmpp_flags(&rmpp_mad->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
+	rmpp_mad->rmpp_hdr.seg_num = cpu_to_be32(mad_send_wr->seg_num);
+
+	if (mad_send_wr->seg_num == 1) {
+		rmpp_mad->rmpp_hdr.rmpp_rtime_flags |= IB_MGMT_RMPP_FLAG_FIRST;
+		rmpp_mad->rmpp_hdr.paylen_newwin =
+			cpu_to_be32(mad_send_wr->total_seg *
+				    (sizeof(struct ib_rmpp_mad) -
+				       offsetof(struct ib_rmpp_mad, data)));
+		mad_send_wr->sg_list[0].length = sizeof(struct ib_rmpp_mad);
+	} else {
+		mad_send_wr->send_wr.num_sge = 2;
+		mad_send_wr->sg_list[0].length = mad_send_wr->data_offset;
+		mad_send_wr->sg_list[1].addr = get_seg_addr(mad_send_wr);
+		mad_send_wr->sg_list[1].length = sizeof(struct ib_rmpp_mad) -
+						 mad_send_wr->data_offset;
+		mad_send_wr->sg_list[1].lkey = mad_send_wr->sg_list[0].lkey;
+	}
+
+	if (mad_send_wr->seg_num == mad_send_wr->total_seg) {
+		rmpp_mad->rmpp_hdr.rmpp_rtime_flags |= IB_MGMT_RMPP_FLAG_LAST;
+		rmpp_mad->rmpp_hdr.paylen_newwin =
+			cpu_to_be32(sizeof(struct ib_rmpp_mad) -
+				    offsetof(struct ib_rmpp_mad, data) -
+				    mad_send_wr->pad);
+	}
+
+	/* 2 seconds for an ACK until we can find the packet lifetime */
+	timeout = mad_send_wr->send_wr.wr.ud.timeout_ms;
+	if (!timeout || timeout > 2000)
+		mad_send_wr->timeout = msecs_to_jiffies(2000);
+	mad_send_wr->seg_num++;
+	return ib_send_mad(mad_send_wr);
+}
+
+static void process_rmpp_ack(struct ib_mad_agent_private *agent,
+			     struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_mad_send_wr_private *mad_send_wr;
+	struct ib_rmpp_mad *rmpp_mad;
+	unsigned long flags;
+	int seg_num, newwin, ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+	if (rmpp_mad->rmpp_hdr.rmpp_status)
+		return;
+
+	seg_num = be32_to_cpu(rmpp_mad->rmpp_hdr.seg_num);
+	newwin = be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+
+	spin_lock_irqsave(&agent->lock, flags);
+	mad_send_wr = ib_find_send_mad(agent, rmpp_mad->mad_hdr.tid);
+	if (!mad_send_wr)
+		goto out;	/* Unmatched ACK */
+
+	if ((mad_send_wr->last_ack == mad_send_wr->total_seg) ||
+	    (!mad_send_wr->timeout) || (mad_send_wr->status != IB_WC_SUCCESS))
+		goto out;	/* Send is already done */
+
+	if (seg_num > mad_send_wr->total_seg)
+		goto out;	/* Bad ACK */
+
+	if (newwin < mad_send_wr->newwin || seg_num < mad_send_wr->last_ack)
+		goto out;	/* Old ACK */
+
+	if (seg_num > mad_send_wr->last_ack) {
+		mad_send_wr->last_ack = seg_num;
+		mad_send_wr->retries = mad_send_wr->send_wr.wr.ud.retries;
+	}
+	mad_send_wr->newwin = newwin;
+	if (mad_send_wr->last_ack == mad_send_wr->total_seg) {
+		/* If no response is expected, the ACK completes the send */
+		if (!mad_send_wr->send_wr.wr.ud.timeout_ms) {
+			struct ib_mad_send_wc wc;
+
+			ib_mark_mad_done(mad_send_wr);
+			spin_unlock_irqrestore(&agent->lock, flags);
+
+			wc.status = IB_WC_SUCCESS;
+			wc.vendor_err = 0;
+			wc.wr_id = mad_send_wr->wr_id;
+			ib_mad_complete_send_wr(mad_send_wr, &wc);
+			return;
+		}
+		if (mad_send_wr->refcount == 1)
+			ib_reset_mad_timeout(mad_send_wr, mad_send_wr->
+					     send_wr.wr.ud.timeout_ms);
+	} else if (mad_send_wr->refcount == 1 &&
+		   mad_send_wr->seg_num < mad_send_wr->newwin &&
+		   mad_send_wr->seg_num <= mad_send_wr->total_seg) {
+		/* Send failure will just result in a timeout/retry */
+		ret = send_next_seg(mad_send_wr);
+		if (ret)
+			goto out;
+
+		mad_send_wr->refcount++;
+		list_del(&mad_send_wr->agent_list);
+		list_add_tail(&mad_send_wr->agent_list,
+			      &mad_send_wr->mad_agent_priv->send_list);
+	}
+out:
+	spin_unlock_irqrestore(&agent->lock, flags);
+}
+
+struct ib_mad_recv_wc *
+ib_process_rmpp_recv_wc(struct ib_mad_agent_private *agent,
+			struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+	if (!(rmpp_mad->rmpp_hdr.rmpp_rtime_flags & IB_MGMT_RMPP_FLAG_ACTIVE))
+		return mad_recv_wc;
+
+	if (rmpp_mad->rmpp_hdr.rmpp_version != IB_MGMT_RMPP_VERSION)
+		goto out;
+
+	switch (rmpp_mad->rmpp_hdr.rmpp_type) {
+	case IB_MGMT_RMPP_TYPE_DATA:
+		if (rmpp_mad->rmpp_hdr.seg_num == __constant_htonl(1))
+			return start_rmpp(agent, mad_recv_wc);
+		else
+			return continue_rmpp(agent, mad_recv_wc);
+	case IB_MGMT_RMPP_TYPE_ACK:
+		process_rmpp_ack(agent, mad_recv_wc);
+		break;
+	case IB_MGMT_RMPP_TYPE_STOP:
+	case IB_MGMT_RMPP_TYPE_ABORT:
+		/* TODO: process_rmpp_nack(agent, mad_recv_wc); */
+		break;
+	default:
+		break;
+	}
+out:
+	ib_free_recv_mad(mad_recv_wc);
+	return NULL;
+}
+
+int ib_send_rmpp_mad(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int i, total_len, ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	if (!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) &
+	      IB_MGMT_RMPP_FLAG_ACTIVE))
+		return IB_RMPP_RESULT_UNHANDLED;
+
+	if (rmpp_mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA)
+		return IB_RMPP_RESULT_INTERNAL;
+
+	if (mad_send_wr->send_wr.num_sge > 1)
+		return -EINVAL;		/* TODO: support num_sge > 1 */
+
+	mad_send_wr->seg_num = 1;
+	mad_send_wr->newwin = 1;
+	mad_send_wr->data_offset = data_offset(rmpp_mad->mad_hdr.mgmt_class);
+
+	total_len = 0;
+	for (i = 0; i < mad_send_wr->send_wr.num_sge; i++)
+		total_len += mad_send_wr->send_wr.sg_list[i].length;
+
+        mad_send_wr->total_seg = (total_len - mad_send_wr->data_offset) /
+			(sizeof(struct ib_rmpp_mad) - mad_send_wr->data_offset);
+	mad_send_wr->pad = total_len - offsetof(struct ib_rmpp_mad, data) -
+			   be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+
+	/* We need to wait for the final ACK even if there isn't a response */
+	mad_send_wr->refcount += (mad_send_wr->timeout == 0);
+	ret = send_next_seg(mad_send_wr);
+	if (!ret)
+		return IB_RMPP_RESULT_CONSUMED;
+	return ret;
+}
+
+int ib_process_rmpp_send_wc(struct ib_mad_send_wr_private *mad_send_wr,
+			    struct ib_mad_send_wc *mad_send_wc)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	struct ib_mad_send_buf *msg;
+	int ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	if (!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & 
+	      IB_MGMT_RMPP_FLAG_ACTIVE))
+		return IB_RMPP_RESULT_UNHANDLED; /* RMPP not active */
+
+	if (rmpp_mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA) {
+		msg = (struct ib_mad_send_buf *) (unsigned long)
+		      mad_send_wc->wr_id;
+		ib_free_send_mad(msg);
+		return IB_RMPP_RESULT_INTERNAL;	 /* ACK, STOP, or ABORT */
+	}
+
+	if (mad_send_wc->status != IB_WC_SUCCESS ||
+	    mad_send_wr->status != IB_WC_SUCCESS)
+		return IB_RMPP_RESULT_PROCESSED; /* Canceled or send error */
+
+	if (!mad_send_wr->timeout)
+		return IB_RMPP_RESULT_PROCESSED; /* Response received */
+
+	if (mad_send_wr->last_ack == mad_send_wr->total_seg) {
+		mad_send_wr->timeout =
+			msecs_to_jiffies(mad_send_wr->send_wr.wr.ud.timeout_ms);
+		return IB_RMPP_RESULT_PROCESSED; /* Send done */
+	}
+
+	if (mad_send_wr->seg_num > mad_send_wr->newwin ||
+	    mad_send_wr->seg_num > mad_send_wr->total_seg)
+		return IB_RMPP_RESULT_PROCESSED; /* Wait for ACK */
+
+	ret = send_next_seg(mad_send_wr);
+	if (ret) {
+		mad_send_wc->status = IB_WC_GENERAL_ERR;
+		return IB_RMPP_RESULT_PROCESSED;
+	}
+	return IB_RMPP_RESULT_CONSUMED;
+}
+
+int ib_retry_rmpp(struct ib_mad_send_wr_private *mad_send_wr)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+	int ret;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr;
+	if (!(ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & 
+	      IB_MGMT_RMPP_FLAG_ACTIVE))
+		return IB_RMPP_RESULT_UNHANDLED; /* RMPP not active */
+
+	if (mad_send_wr->last_ack == mad_send_wr->total_seg)
+		return IB_RMPP_RESULT_PROCESSED;
+
+	mad_send_wr->seg_num = mad_send_wr->last_ack + 1;
+	ret = send_next_seg(mad_send_wr);
+	if (ret)
+		return IB_RMPP_RESULT_PROCESSED;
+
+	return IB_RMPP_RESULT_CONSUMED;
+}
diff -uprN linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad_rmpp.h linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad_rmpp.h
-- linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/mad_rmpp.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/mad_rmpp.h	2005-07-06 16:55:32.000000000 -0400
@@ -0,0 +1,58 @@
+/*
+ * Copyright (c) 2005 Intel Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: mad_rmpp.h 1921 2005-02-25 22:58:44Z sean.hefty $
+ */
+
+#ifndef __MAD_RMPP_H__
+#define __MAD_RMPP_H__
+
+enum {
+	IB_RMPP_RESULT_PROCESSED,
+	IB_RMPP_RESULT_CONSUMED,
+	IB_RMPP_RESULT_INTERNAL,
+	IB_RMPP_RESULT_UNHANDLED
+};
+
+int ib_send_rmpp_mad(struct ib_mad_send_wr_private *mad_send_wr);
+
+struct ib_mad_recv_wc *
+ib_process_rmpp_recv_wc(struct ib_mad_agent_private *agent,
+			struct ib_mad_recv_wc *mad_recv_wc);
+
+int ib_process_rmpp_send_wc(struct ib_mad_send_wr_private *mad_send_wr,
+			    struct ib_mad_send_wc *mad_send_wc);
+
+void ib_cancel_rmpp_recvs(struct ib_mad_agent_private *agent);
+
+int ib_retry_rmpp(struct ib_mad_send_wr_private *mad_send_wr);
+
+#endif	/* __MAD_RMPP_H__ */
diff -uprN linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/Makefile linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/Makefile
-- linux-2.6.13-rc2-mm1-18/drivers/infiniband/core/Makefile	2005-07-10 16:30:24.000000000 -0400
+++ linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/Makefile	2005-07-10 16:32:15.000000000 -0400
@@ -6,7 +6,7 @@ obj-$(CONFIG_INFINIBAND_USER_VERBS) +=	i
 ib_core-y :=			packer.o ud_header.o verbs.o sysfs.o \
 				device.o fmr_pool.o cache.o
 
-ib_mad-y :=			mad.o smi.o agent.o
+ib_mad-y :=			mad.o smi.o agent.o mad_rmpp.o
 
 ib_sa-y :=			sa_query.o
 




From halr at voltaire.com  Mon Jul 11 13:40:17 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:40:17 -0400
Subject: [openib-general] [PATCH 20/29v2] Add Service Record support to SA
	client
Message-ID: <1121110368.4389.5024.camel@hal.voltaire.com>

Add Service Record support to SA client

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 19/29.

--
 core/sa_query.c |  166 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 include/ib_sa.h |   75 ++++++++++++++++++++++++-
 2 files changed, 236 insertions(+), 5 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/sa_query.c linux-2.6.13-rc2-mm1-20/drivers/infiniband/core/sa_query.c
-- linux-2.6.13-rc2-mm1-19/drivers/infiniband/core/sa_query.c	2005-07-10 16:22:18.000000000 -0400
+++ linux-2.6.13-rc2-mm1-20/drivers/infiniband/core/sa_query.c	2005-07-10 16:41:32.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: sa_query.c 1389 2004-12-27 22:56:47Z roland $
+ * $Id: sa_query.c 2811 2005-07-06 18:11:43Z halr $
  */
 
 #include <linux/module.h>
@@ -79,6 +80,12 @@ struct ib_sa_query {
 	int                 id;
 };
 
+struct ib_sa_service_query {
+	void (*callback)(int, struct ib_sa_service_rec *, void *);
+	void *context;
+	struct ib_sa_query sa_query;
+};
+
 struct ib_sa_path_query {
 	void (*callback)(int, struct ib_sa_path_rec *, void *);
 	void *context;
@@ -320,6 +327,54 @@ static const struct ib_field mcmember_re
 	  .size_bits    = 23 },
 };
 
+#define SERVICE_REC_FIELD(field) \
+	.struct_offset_bytes = offsetof(struct ib_sa_service_rec, field),	\
+	.struct_size_bytes   = sizeof ((struct ib_sa_service_rec *) 0)->field,	\
+	.field_name          = "sa_service_rec:" #field
+
+static const struct ib_field service_rec_table[] = {
+	{ SERVICE_REC_FIELD(id),
+	  .offset_words = 0,
+	  .offset_bits  = 0,
+	  .size_bits    = 64 },
+	{ SERVICE_REC_FIELD(gid),
+	  .offset_words = 2,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(pkey),
+	  .offset_words = 6,
+	  .offset_bits  = 0,
+	  .size_bits    = 16 },
+	{ SERVICE_REC_FIELD(lease),
+	  .offset_words = 7,
+	  .offset_bits  = 0,
+	  .size_bits    = 32 },
+	{ SERVICE_REC_FIELD(key),
+	  .offset_words = 8,
+	  .offset_bits  = 0,
+	  .size_bits    = 128 },
+	{ SERVICE_REC_FIELD(name),
+	  .offset_words = 12,
+	  .offset_bits  = 0,
+	  .size_bits    = 64*8 },
+	{ SERVICE_REC_FIELD(data8),
+	  .offset_words = 28,
+	  .offset_bits  = 0,
+	  .size_bits    = 16*8 },
+	{ SERVICE_REC_FIELD(data16),
+	  .offset_words = 32,
+	  .offset_bits  = 0,
+	  .size_bits    = 8*16 },
+	{ SERVICE_REC_FIELD(data32),
+	  .offset_words = 36,
+	  .offset_bits  = 0,
+	  .size_bits    = 4*32 },
+	{ SERVICE_REC_FIELD(data64),
+	  .offset_words = 40,
+	  .offset_bits  = 0,
+	  .size_bits    = 2*64 },
+};
+
 static void free_sm_ah(struct kref *kref)
 {
 	struct ib_sa_sm_ah *sm_ah = container_of(kref, struct ib_sa_sm_ah, ref);
@@ -443,7 +498,6 @@ static int send_mad(struct ib_sa_query *
 				 .remote_qpn  = 1,
 				 .remote_qkey = IB_QP1_QKEY,
 				 .timeout_ms  = timeout_ms,
-				 .retries     = 0 
 			 }
 		 }
 	};
@@ -596,6 +650,114 @@ int ib_sa_path_rec_get(struct ib_device 
 }
 EXPORT_SYMBOL(ib_sa_path_rec_get);
 
+static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query,
+				    int status,
+				    struct ib_sa_mad *mad)
+{
+	struct ib_sa_service_query *query =
+		container_of(sa_query, struct ib_sa_service_query, sa_query);
+
+	if (mad) {
+		struct ib_sa_service_rec rec;
+
+		ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table),
+			  mad->data, &rec);
+		query->callback(status, &rec, query->context);
+	} else
+		query->callback(status, NULL, query->context);
+}
+
+static void ib_sa_service_rec_release(struct ib_sa_query *sa_query)
+{
+	kfree(sa_query->mad);
+	kfree(container_of(sa_query, struct ib_sa_service_query, sa_query));
+}
+
+/**
+ * ib_sa_service_rec_query - Start Service Record operation
+ * @device:device to send request on
+ * @port_num: port number to send request on
+ * @method:SA method - should be get, set, or delete
+ * @rec:Service Record to send in request
+ * @comp_mask:component mask to send in request
+ * @timeout_ms:time to wait for response
+ * @gfp_mask:GFP mask to use for internal allocations
+ * @callback:function called when request completes, times out or is
+ * canceled
+ * @context:opaque user context passed to callback
+ * @sa_query:request context, used to cancel request
+ *
+ * Send a Service Record set/get/delete to the SA to register,
+ * unregister or query a service record.
+ * The callback function will be called when the request completes (or
+ * fails); status is 0 for a successful response, -EINTR if the query
+ * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error
+ * occurred sending the query.  The resp parameter of the callback is
+ * only valid if status is 0.
+ *
+ * If the return value of ib_sa_service_rec_query() is negative, it is an
+ * error code.  Otherwise it is a request ID that can be used to cancel
+ * the query.
+ */
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
+			    struct ib_sa_service_rec *rec,
+			    ib_sa_comp_mask comp_mask,
+			    int timeout_ms, int gfp_mask,
+			    void (*callback)(int status,
+					     struct ib_sa_service_rec *resp,
+					     void *context),
+			    void *context,
+			    struct ib_sa_query **sa_query)
+{
+	struct ib_sa_service_query *query;
+	struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client);
+	struct ib_sa_port   *port   = &sa_dev->port[port_num - sa_dev->start_port];
+	struct ib_mad_agent *agent  = port->agent;
+	int ret;
+
+	if (method != IB_MGMT_METHOD_GET &&
+	    method != IB_MGMT_METHOD_SET &&
+	    method != IB_SA_METHOD_DELETE)
+		return -EINVAL;
+
+	query = kmalloc(sizeof *query, gfp_mask);
+	if (!query)
+		return -ENOMEM;
+	query->sa_query.mad = kmalloc(sizeof *query->sa_query.mad, gfp_mask);
+	if (!query->sa_query.mad) {
+		kfree(query);
+		return -ENOMEM;
+	}
+
+	query->callback = callback;
+	query->context  = context;
+
+	init_mad(query->sa_query.mad, agent);
+
+	query->sa_query.callback              = callback ? ib_sa_service_rec_callback : NULL;
+	query->sa_query.release               = ib_sa_service_rec_release;
+	query->sa_query.port                  = port;
+	query->sa_query.mad->mad_hdr.method   = method;
+	query->sa_query.mad->mad_hdr.attr_id  =
+				cpu_to_be16(IB_SA_ATTR_SERVICE_REC);
+	query->sa_query.mad->sa_hdr.comp_mask = comp_mask;
+
+	ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table),
+		rec, query->sa_query.mad->data);
+
+	*sa_query = &query->sa_query;
+
+	ret = send_mad(&query->sa_query, timeout_ms);
+	if (ret < 0) {
+		*sa_query = NULL;
+		kfree(query->sa_query.mad);
+		kfree(query);
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(ib_sa_service_rec_query);
+
 static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query,
 					int status,
 					struct ib_sa_mad *mad)
diff -uprN linux-2.6.13-rc2-mm1-19/drivers/infiniband/include/ib_sa.h linux-2.6.13-rc2-mm1-20/drivers/infiniband/include/ib_sa.h
-- linux-2.6.13-rc2-mm1-19/drivers/infiniband/include/ib_sa.h	2005-07-10 12:07:41.000000000 -0400
+++ linux-2.6.13-rc2-mm1-20/drivers/infiniband/include/ib_sa.h	2005-07-10 16:40:43.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_sa.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: ib_sa.h 2811 2005-07-06 18:11:43Z halr $
  */
 
 #ifndef IB_SA_H
@@ -41,9 +42,11 @@
 #include <ib_mad.h>
 
 enum {
-	IB_SA_CLASS_VERSION	= 2,	/* IB spec version 1.1/1.2 */
+	IB_SA_CLASS_VERSION		= 2,	/* IB spec version 1.1/1.2 */
 
-	IB_SA_METHOD_DELETE	= 0x15
+	IB_SA_METHOD_GET_TABLE		= 0x12,
+	IB_SA_METHOD_GET_TABLE_RESP	= 0x92,
+	IB_SA_METHOD_DELETE		= 0x15
 };
 
 enum ib_sa_selector {
@@ -191,6 +194,61 @@ struct ib_sa_mcmember_rec {
 	int          proxy_join;
 };
 
+/* Service Record Component Mask Sec 15.2.5.14 Ver 1.1	*/
+#define IB_SA_SERVICE_REC_SERVICE_ID			IB_SA_COMP_MASK( 0)
+#define IB_SA_SERVICE_REC_SERVICE_GID			IB_SA_COMP_MASK( 1)
+#define IB_SA_SERVICE_REC_SERVICE_PKEY			IB_SA_COMP_MASK( 2)
+/* reserved:								 3 */
+#define IB_SA_SERVICE_REC_SERVICE_LEASE			IB_SA_COMP_MASK( 4)
+#define IB_SA_SERVICE_REC_SERVICE_KEY			IB_SA_COMP_MASK( 5)
+#define IB_SA_SERVICE_REC_SERVICE_NAME			IB_SA_COMP_MASK( 6)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_0		IB_SA_COMP_MASK( 7)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_1		IB_SA_COMP_MASK( 8)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_2		IB_SA_COMP_MASK( 9)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_3		IB_SA_COMP_MASK(10)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_4		IB_SA_COMP_MASK(11)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_5		IB_SA_COMP_MASK(12)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_6		IB_SA_COMP_MASK(13)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_7		IB_SA_COMP_MASK(14)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_8		IB_SA_COMP_MASK(15)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_9		IB_SA_COMP_MASK(16)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_10		IB_SA_COMP_MASK(17)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_11		IB_SA_COMP_MASK(18)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_12		IB_SA_COMP_MASK(19)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_13		IB_SA_COMP_MASK(20)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_14		IB_SA_COMP_MASK(21)
+#define IB_SA_SERVICE_REC_SERVICE_DATA8_15		IB_SA_COMP_MASK(22)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_0		IB_SA_COMP_MASK(23)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_1		IB_SA_COMP_MASK(24)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_2		IB_SA_COMP_MASK(25)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_3		IB_SA_COMP_MASK(26)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_4		IB_SA_COMP_MASK(27)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_5		IB_SA_COMP_MASK(28)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_6		IB_SA_COMP_MASK(29)
+#define IB_SA_SERVICE_REC_SERVICE_DATA16_7		IB_SA_COMP_MASK(30)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_0		IB_SA_COMP_MASK(31)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_1		IB_SA_COMP_MASK(32)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_2		IB_SA_COMP_MASK(33)
+#define IB_SA_SERVICE_REC_SERVICE_DATA32_3		IB_SA_COMP_MASK(34)
+#define IB_SA_SERVICE_REC_SERVICE_DATA64_0		IB_SA_COMP_MASK(35)
+#define IB_SA_SERVICE_REC_SERVICE_DATA64_1		IB_SA_COMP_MASK(36)
+
+#define IB_DEFAULT_SERVICE_LEASE 	0xFFFFFFFF
+
+struct ib_sa_service_rec {
+	u64		id;
+	union ib_gid	gid;		
+	u16 		pkey;
+	/* reserved */
+	u32		lease;
+	u8		key[16];
+	u8		name[64];
+	u8		data8[16];
+	u16		data16[8];
+	u32		data32[4];
+	u64		data64[2];
+};
+
 struct ib_sa_query;
 
 void ib_sa_cancel_query(int id, struct ib_sa_query *query);
@@ -216,6 +274,17 @@ int ib_sa_mcmember_rec_query(struct ib_d
 			     void *context,
 			     struct ib_sa_query **query);
 
+int ib_sa_service_rec_query(struct ib_device *device, u8 port_num,
+			 u8 method,
+			 struct ib_sa_service_rec *rec,
+			 ib_sa_comp_mask comp_mask,
+			 int timeout_ms, int gfp_mask,
+			 void (*callback)(int status,
+					  struct ib_sa_service_rec *resp,
+					  void *context),
+			 void *context,
+			 struct ib_sa_query **sa_query);
+
 /**
  * ib_sa_mcmember_rec_set - Start an MCMember set query
  * @device:device to send query on




From halr at voltaire.com  Mon Jul 11 13:43:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:43:31 -0400
Subject: [openib-general] [PATCH 21/29v2] Add the header file for kernel CM
	(Communications Manager)
Message-ID: <1121110375.4389.5026.camel@hal.voltaire.com>

Add the header file for kernel CM (Communications Manager) 
Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 20/29.

--
 ib_cm.h |  568 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 568 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1-20/drivers/infiniband/include/ib_cm.h linux-2.6.13-rc2-mm1-21/drivers/infiniband/include/ib_cm.h
-- linux-2.6.13-rc2-mm1-20/drivers/infiniband/include/ib_cm.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-21/drivers/infiniband/include/ib_cm.h	2005-06-28 13:07:34.000000000 -0400
@@ -0,0 +1,568 @@
+/*
+ * Copyright (c) 2004 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ib_cm.h 2730 2005-06-28 16:43:03Z sean.hefty $
+ */
+#if !defined(IB_CM_H)
+#define IB_CM_H
+
+#include <ib_mad.h>
+#include <ib_sa.h>
+
+enum ib_cm_state {
+	IB_CM_IDLE,
+	IB_CM_LISTEN,
+	IB_CM_REQ_SENT,
+	IB_CM_REQ_RCVD,
+	IB_CM_MRA_REQ_SENT,
+	IB_CM_MRA_REQ_RCVD,
+	IB_CM_REP_SENT,
+	IB_CM_REP_RCVD,
+	IB_CM_MRA_REP_SENT,
+	IB_CM_MRA_REP_RCVD,
+	IB_CM_ESTABLISHED,
+	IB_CM_DREQ_SENT,
+	IB_CM_DREQ_RCVD,
+	IB_CM_TIMEWAIT,
+	IB_CM_SIDR_REQ_SENT,
+	IB_CM_SIDR_REQ_RCVD
+};
+
+enum ib_cm_lap_state {
+	IB_CM_LAP_IDLE,
+	IB_CM_LAP_SENT,
+	IB_CM_LAP_RCVD,
+	IB_CM_MRA_LAP_SENT,
+	IB_CM_MRA_LAP_RCVD,
+};
+
+enum ib_cm_event_type {
+	IB_CM_REQ_ERROR,
+	IB_CM_REQ_RECEIVED,
+	IB_CM_REP_ERROR,
+	IB_CM_REP_RECEIVED,
+	IB_CM_RTU_RECEIVED,
+	IB_CM_USER_ESTABLISHED,
+	IB_CM_DREQ_ERROR,
+	IB_CM_DREQ_RECEIVED,
+	IB_CM_DREP_RECEIVED,
+	IB_CM_TIMEWAIT_EXIT,
+	IB_CM_MRA_RECEIVED,
+	IB_CM_REJ_RECEIVED,
+	IB_CM_LAP_ERROR,
+	IB_CM_LAP_RECEIVED,
+	IB_CM_APR_RECEIVED,
+	IB_CM_SIDR_REQ_ERROR,
+	IB_CM_SIDR_REQ_RECEIVED,
+	IB_CM_SIDR_REP_RECEIVED
+};
+
+enum ib_cm_data_size {
+	IB_CM_REQ_PRIVATE_DATA_SIZE	 = 92,
+	IB_CM_MRA_PRIVATE_DATA_SIZE	 = 222,
+	IB_CM_REJ_PRIVATE_DATA_SIZE	 = 148,
+	IB_CM_REP_PRIVATE_DATA_SIZE	 = 196,
+	IB_CM_RTU_PRIVATE_DATA_SIZE	 = 224,
+	IB_CM_DREQ_PRIVATE_DATA_SIZE	 = 220,
+	IB_CM_DREP_PRIVATE_DATA_SIZE	 = 224,
+	IB_CM_REJ_ARI_LENGTH		 = 72,
+	IB_CM_LAP_PRIVATE_DATA_SIZE	 = 168,
+	IB_CM_APR_PRIVATE_DATA_SIZE	 = 148,
+	IB_CM_APR_INFO_LENGTH		 = 72,
+	IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE = 216,
+	IB_CM_SIDR_REP_PRIVATE_DATA_SIZE = 136,
+	IB_CM_SIDR_REP_INFO_LENGTH	 = 72
+};
+
+struct ib_cm_id;
+
+struct ib_cm_req_event_param {
+	struct ib_cm_id		*listen_id;
+	struct ib_device	*device;
+	u8			port;
+
+	struct ib_sa_path_rec	*primary_path;
+	struct ib_sa_path_rec	*alternate_path;
+
+	u64			remote_ca_guid;
+	u32			remote_qkey;
+	u32			remote_qpn;
+	enum ib_qp_type		qp_type;
+
+	u32			starting_psn;
+	u8			responder_resources;
+	u8			initiator_depth;
+	unsigned int		local_cm_response_timeout:5;
+	unsigned int		flow_control:1;
+	unsigned int		remote_cm_response_timeout:5;
+	unsigned int		retry_count:3;
+	unsigned int		rnr_retry_count:3;
+	unsigned int		srq:1;
+};
+
+struct ib_cm_rep_event_param {
+	u64			remote_ca_guid;
+	u32			remote_qkey;
+	u32			remote_qpn;
+	u32			starting_psn;
+	u8			responder_resources;
+	u8			initiator_depth;
+	unsigned int		target_ack_delay:5;
+	unsigned int		failover_accepted:2;
+	unsigned int		flow_control:1;
+	unsigned int		rnr_retry_count:3;
+	unsigned int		srq:1;
+};
+
+enum ib_cm_rej_reason {
+	IB_CM_REJ_NO_QP				= __constant_htons(1),
+	IB_CM_REJ_NO_EEC			= __constant_htons(2),
+	IB_CM_REJ_NO_RESOURCES			= __constant_htons(3),
+	IB_CM_REJ_TIMEOUT			= __constant_htons(4),
+	IB_CM_REJ_UNSUPPORTED			= __constant_htons(5),
+	IB_CM_REJ_INVALID_COMM_ID		= __constant_htons(6),
+	IB_CM_REJ_INVALID_COMM_INSTANCE		= __constant_htons(7),
+	IB_CM_REJ_INVALID_SERVICE_ID		= __constant_htons(8),
+	IB_CM_REJ_INVALID_TRANSPORT_TYPE	= __constant_htons(9),
+	IB_CM_REJ_STALE_CONN			= __constant_htons(10),
+	IB_CM_REJ_RDC_NOT_EXIST			= __constant_htons(11),
+	IB_CM_REJ_INVALID_GID			= __constant_htons(12),
+	IB_CM_REJ_INVALID_LID			= __constant_htons(13),
+	IB_CM_REJ_INVALID_SL			= __constant_htons(14),
+	IB_CM_REJ_INVALID_TRAFFIC_CLASS		= __constant_htons(15),
+	IB_CM_REJ_INVALID_HOP_LIMIT		= __constant_htons(16),
+	IB_CM_REJ_INVALID_PACKET_RATE		= __constant_htons(17),
+	IB_CM_REJ_INVALID_ALT_GID		= __constant_htons(18),
+	IB_CM_REJ_INVALID_ALT_LID		= __constant_htons(19),
+	IB_CM_REJ_INVALID_ALT_SL		= __constant_htons(20),
+	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= __constant_htons(21),
+	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= __constant_htons(22),
+	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= __constant_htons(23),
+	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(24),
+	IB_CM_REJ_INVALID_MTU			= __constant_htons(26),
+	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= __constant_htons(27),
+	IB_CM_REJ_CONSUMER_DEFINED		= __constant_htons(28),
+	IB_CM_REJ_INVALID_RNR_RETRY		= __constant_htons(29),
+	IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID	= __constant_htons(30),
+	IB_CM_REJ_INVALID_CLASS_VERSION		= __constant_htons(31),
+	IB_CM_REJ_INVALID_FLOW_LABEL		= __constant_htons(32),
+	IB_CM_REJ_INVALID_ALT_FLOW_LABEL	= __constant_htons(33)
+};
+
+struct ib_cm_rej_event_param {
+	enum ib_cm_rej_reason	reason;
+	void			*ari;
+	u8			ari_length;
+};
+
+struct ib_cm_mra_event_param {
+	u8	service_timeout;
+};
+
+struct ib_cm_lap_event_param {
+	struct ib_sa_path_rec	*alternate_path;
+};
+
+enum ib_cm_apr_status {
+	IB_CM_APR_SUCCESS,
+	IB_CM_APR_INVALID_COMM_ID,
+	IB_CM_APR_UNSUPPORTED,
+	IB_CM_APR_REJECT,
+	IB_CM_APR_REDIRECT,
+	IB_CM_APR_IS_CURRENT,
+	IB_CM_APR_INVALID_QPN_EECN,
+	IB_CM_APR_INVALID_LID,
+	IB_CM_APR_INVALID_GID,
+	IB_CM_APR_INVALID_FLOW_LABEL,
+	IB_CM_APR_INVALID_TCLASS,
+	IB_CM_APR_INVALID_HOP_LIMIT,
+	IB_CM_APR_INVALID_PACKET_RATE,
+	IB_CM_APR_INVALID_SL
+};
+
+struct ib_cm_apr_event_param {
+	enum ib_cm_apr_status	ap_status;
+	void			*apr_info;
+	u8			info_len;
+};
+
+struct ib_cm_sidr_req_event_param {
+	struct ib_cm_id		*listen_id;
+	struct ib_device	*device;
+	u8			port;
+
+	u16	pkey;
+};
+
+enum ib_cm_sidr_status {
+	IB_SIDR_SUCCESS,
+	IB_SIDR_UNSUPPORTED,
+	IB_SIDR_REJECT,
+	IB_SIDR_NO_QP,
+	IB_SIDR_REDIRECT,
+	IB_SIDR_UNSUPPORTED_VERSION
+};
+
+struct ib_cm_sidr_rep_event_param {
+	enum ib_cm_sidr_status	status;
+	u32			qkey;
+	u32			qpn;
+	void			*info;
+	u8			info_len;
+
+};
+
+struct ib_cm_event {
+	enum ib_cm_event_type	event;
+	union {
+		struct ib_cm_req_event_param	req_rcvd;
+		struct ib_cm_rep_event_param	rep_rcvd;
+		/* No data for RTU received events. */
+		struct ib_cm_rej_event_param	rej_rcvd;
+		struct ib_cm_mra_event_param	mra_rcvd;
+		struct ib_cm_lap_event_param	lap_rcvd;
+		struct ib_cm_apr_event_param	apr_rcvd;
+		/* No data for DREQ/DREP received events. */
+		struct ib_cm_sidr_req_event_param sidr_req_rcvd;
+		struct ib_cm_sidr_rep_event_param sidr_rep_rcvd;
+		enum ib_wc_status		send_status;
+	} param;
+
+	void			*private_data;
+};
+
+/**
+ * ib_cm_handler - User-defined callback to process communication events.
+ * @cm_id: Communication identifier associated with the reported event.
+ * @event: Information about the communication event.
+ *
+ * IB_CM_REQ_RECEIVED and IB_CM_SIDR_REQ_RECEIVED communication events 
+ * generated as a result of listen requests result in the allocation of a 
+ * new @cm_id.  The new @cm_id is returned to the user through this callback.
+ * Clients are responsible for destroying the new @cm_id.  For peer-to-peer
+ * IB_CM_REQ_RECEIVED and all other events, the returned @cm_id corresponds
+ * to a user's existing communication identifier.
+ *
+ * Users may not call ib_destroy_cm_id while in the context of this callback;
+ * however, returning a non-zero value instructs the communication manager to
+ * destroy the @cm_id after the callback completes. 
+ */
+typedef int (*ib_cm_handler)(struct ib_cm_id *cm_id,
+			     struct ib_cm_event *event);
+
+struct ib_cm_id {
+	ib_cm_handler		cm_handler;
+	void			*context;
+	u64			service_id;
+	u64			service_mask;
+	enum ib_cm_state	state;		/* internal CM/debug use */
+	enum ib_cm_lap_state	lap_state;	/* internal CM/debug use */
+	u32			local_id;
+	u32			remote_id;
+};
+
+/**
+ * ib_create_cm_id - Allocate a communication identifier.
+ * @cm_handler: Callback invoked to notify the user of CM events.
+ * @context: User specified context associated with the communication
+ *   identifier.
+ *
+ * Communication identifiers are used to track connection states, service
+ * ID resolution requests, and listen requests.
+ */
+struct ib_cm_id *ib_create_cm_id(ib_cm_handler cm_handler,
+				 void *context);
+
+/**
+ * ib_destroy_cm_id - Destroy a connection identifier.
+ * @cm_id: Connection identifier to destroy.
+ *
+ * This call blocks until the connection identifier is destroyed.
+ */
+void ib_destroy_cm_id(struct ib_cm_id *cm_id);
+
+#define IB_SERVICE_ID_AGN_MASK	__constant_cpu_to_be64(0xFF00000000000000ULL)
+#define IB_CM_ASSIGN_SERVICE_ID __constant_cpu_to_be64(0x0200000000000000ULL)
+
+/**
+ * ib_cm_listen - Initiates listening on the specified service ID for
+ *   connection and service ID resolution requests.
+ * @cm_id: Connection identifier associated with the listen request.
+ * @service_id: Service identifier matched against incoming connection
+ *   and service ID resolution requests.  The service ID should be specified
+ *   network-byte order.  If set to IB_CM_ASSIGN_SERVICE_ID, the CM will
+ *   assign a service ID to the caller.
+ * @service_mask: Mask applied to service ID used to listen across a
+ *   range of service IDs.  If set to 0, the service ID is matched
+ *   exactly.  This parameter is ignored if %service_id is set to
+ *   IB_CM_ASSIGN_SERVICE_ID.
+ */
+int ib_cm_listen(struct ib_cm_id *cm_id,
+		 u64 service_id,
+		 u64 service_mask);
+
+struct ib_cm_req_param {
+	struct ib_sa_path_rec	*primary_path;
+	struct ib_sa_path_rec	*alternate_path;
+	u64			service_id;
+	u32			qp_num;
+	enum ib_qp_type		qp_type;
+	u32			starting_psn;
+	const void		*private_data;
+	u8			private_data_len;
+	u8			peer_to_peer;
+	u8			responder_resources;
+	u8			initiator_depth;
+	u8			remote_cm_response_timeout;
+	u8			flow_control;
+	u8			local_cm_response_timeout;
+	u8			retry_count;
+	u8			rnr_retry_count;
+	u8			max_cm_retries;
+	u8			srq;
+};
+
+/**
+ * ib_send_cm_req - Sends a connection request to the remote node.
+ * @cm_id: Connection identifier that will be associated with the
+ *   connection request.
+ * @param: Connection request information needed to establish the
+ *   connection.
+ */
+int ib_send_cm_req(struct ib_cm_id *cm_id,
+		   struct ib_cm_req_param *param);
+
+struct ib_cm_rep_param {
+	u32		qp_num;
+	u32		starting_psn;
+	const void	*private_data;
+	u8		private_data_len;
+	u8		responder_resources;
+	u8		initiator_depth;
+	u8		target_ack_delay;
+	u8		failover_accepted;
+	u8		flow_control;
+	u8		rnr_retry_count;
+	u8		srq;
+};
+
+/**
+ * ib_send_cm_rep - Sends a connection reply in response to a connection
+ *   request.
+ * @cm_id: Connection identifier that will be associated with the
+ *   connection request.
+ * @param: Connection reply information needed to establish the
+ *   connection.
+ */
+int ib_send_cm_rep(struct ib_cm_id *cm_id,
+		   struct ib_cm_rep_param *param);
+
+/**
+ * ib_send_cm_rtu - Sends a connection ready to use message in response
+ *   to a connection reply message.
+ * @cm_id: Connection identifier associated with the connection request.
+ * @private_data: Optional user-defined private data sent with the
+ *   ready to use message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_rtu(struct ib_cm_id *cm_id,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_send_cm_dreq - Sends a disconnection request for an existing
+ *   connection.
+ * @cm_id: Connection identifier associated with the connection being
+ *   released.
+ * @private_data: Optional user-defined private data sent with the
+ *   disconnection request message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_dreq(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len);
+
+/**
+ * ib_send_cm_drep - Sends a disconnection reply to a disconnection request.
+ * @cm_id: Connection identifier associated with the connection being
+ *   released.
+ * @private_data: Optional user-defined private data sent with the
+ *   disconnection reply message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ *
+ * If the cm_id is in the correct state, the CM will transition the connection
+ * to the timewait state, even if an error occurs sending the DREP message.
+ */
+int ib_send_cm_drep(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len);
+
+/**
+ * ib_cm_establish - Forces a connection state to established.
+ * @cm_id: Connection identifier to transition to established.
+ *
+ * This routine should be invoked by users who receive messages on a
+ * connected QP before an RTU has been received.
+ */
+int ib_cm_establish(struct ib_cm_id *cm_id);
+
+/**
+ * ib_send_cm_rej - Sends a connection rejection message to the
+ *   remote node.
+ * @cm_id: Connection identifier associated with the connection being
+ *   rejected.
+ * @reason: Reason for the connection request rejection.
+ * @ari: Optional additional rejection information.
+ * @ari_length: Size of the additional rejection information, in bytes.
+ * @private_data: Optional user-defined private data sent with the
+ *   rejection message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_rej(struct ib_cm_id *cm_id,
+		   enum ib_cm_rej_reason reason,
+		   void *ari,
+		   u8 ari_length,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_send_cm_mra - Sends a message receipt acknowledgement to a connection
+ *   message.
+ * @cm_id: Connection identifier associated with the connection message.
+ * @service_timeout: The maximum time required for the sender to reply to
+ *   to the connection message.
+ * @private_data: Optional user-defined private data sent with the
+ *   message receipt acknowledgement.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_mra(struct ib_cm_id *cm_id,
+		   u8 service_timeout,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_send_cm_lap - Sends a load alternate path request.
+ * @cm_id: Connection identifier associated with the load alternate path
+ *   message.
+ * @alternate_path: A path record that identifies the alternate path to
+ *   load.
+ * @private_data: Optional user-defined private data sent with the
+ *   load alternate path message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_lap(struct ib_cm_id *cm_id,
+		   struct ib_sa_path_rec *alternate_path,
+		   const void *private_data,
+		   u8 private_data_len);
+
+/**
+ * ib_cm_init_qp_attr - Initializes the QP attributes for use in transitioning
+ *   to a specified QP state.
+ * @cm_id: Communication identifier associated with the QP attributes to
+ *   initialize.
+ * @qp_attr: On input, specifies the desired QP state.  On output, the
+ *   mandatory and desired optional attributes will be set in order to
+ *   modify the QP to the specified state.
+ * @qp_attr_mask: The QP attribute mask that may be used to transition the
+ *   QP to the specified state.
+ *
+ * Users must set the @qp_attr->qp_state to the desired QP state.  This call
+ * will set all required attributes for the given transition, along with
+ * known optional attributes.  Users may override the attributes returned from
+ * this call before calling ib_modify_qp.
+ */
+int ib_cm_init_qp_attr(struct ib_cm_id *cm_id,
+		       struct ib_qp_attr *qp_attr,
+		       int *qp_attr_mask);
+
+/**
+ * ib_send_cm_apr - Sends an alternate path response message in response to
+ *   a load alternate path request.
+ * @cm_id: Connection identifier associated with the alternate path response.
+ * @status: Reply status sent with the alternate path response.
+ * @info: Optional additional information sent with the alternate path
+ *   response.
+ * @info_length: Size of the additional information, in bytes.
+ * @private_data: Optional user-defined private data sent with the
+ *   alternate path response message.
+ * @private_data_len: Size of the private data buffer, in bytes.
+ */
+int ib_send_cm_apr(struct ib_cm_id *cm_id,
+		   enum ib_cm_apr_status status,
+		   void *info,
+		   u8 info_length,
+		   const void *private_data,
+		   u8 private_data_len);
+
+struct ib_cm_sidr_req_param {
+	struct ib_sa_path_rec	*path;
+	u64			service_id;
+	int			timeout_ms;
+	const void		*private_data;
+	u8			private_data_len;
+	u8			max_cm_retries;
+	u16			pkey;
+};
+
+/**
+ * ib_send_cm_sidr_req - Sends a service ID resolution request to the
+ *   remote node.
+ * @cm_id: Communication identifier that will be associated with the
+ *   service ID resolution request.
+ * @param: Service ID resolution request information.
+ */
+int ib_send_cm_sidr_req(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_req_param *param);
+
+struct ib_cm_sidr_rep_param {
+	u32			qp_num;
+	u32			qkey;
+	enum ib_cm_sidr_status	status;
+	const void		*info;
+	u8			info_length;
+	const void		*private_data;
+	u8			private_data_len;
+};
+
+/**
+ * ib_send_cm_sidr_rep - Sends a service ID resolution request to the
+ *   remote node.
+ * @cm_id: Communication identifier associated with the received service ID
+ *   resolution request.
+ * @param: Service ID resolution reply information.
+ */
+int ib_send_cm_sidr_rep(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_rep_param *param);
+
+#endif /* IB_CM_H */




From halr at voltaire.com  Mon Jul 11 13:46:44 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:46:44 -0400
Subject: [openib-general] [PATCH 22/29v2] Add the kernel CM implementation
Message-ID: <1121110392.4389.5028.camel@hal.voltaire.com>

Add the kernel CM implementation

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 21/29.

--
 Makefile  |    5 
 cm.c      | 3324 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 cm_msgs.h |  819 +++++++++++++++
 3 files changed, 4147 insertions(+), 1 deletion(-)
diff -uprN linux-2.6.13-rc2-mm1-21/drivers/infiniband/core/cm.c linux-2.6.13-rc2-mm1-22/drivers/infiniband/core/cm.c
-- linux-2.6.13-rc2-mm1-21/drivers/infiniband/core/cm.c	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-22/drivers/infiniband/core/cm.c	2005-07-10 16:14:28.000000000 -0400
@@ -0,0 +1,3324 @@
+/*
+ * Copyright (c) 2004, 2005 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004, 2005 Voltaire Corporation.  All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: cm.c 2821 2005-07-08 17:07:28Z sean.hefty $
+ */
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/idr.h>
+#include <linux/interrupt.h>
+#include <linux/pci.h>
+#include <linux/rbtree.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#include <ib_cache.h>
+#include <ib_cm.h>
+#include "cm_msgs.h"
+
+MODULE_AUTHOR("Sean Hefty");
+MODULE_DESCRIPTION("InfiniBand CM");
+MODULE_LICENSE("Dual BSD/GPL");
+
+static void cm_add_one(struct ib_device *device);
+static void cm_remove_one(struct ib_device *device);
+
+static struct ib_client cm_client = {
+	.name   = "cm",
+	.add    = cm_add_one,
+	.remove = cm_remove_one
+};
+
+static struct ib_cm {
+	spinlock_t lock;
+	struct list_head device_list;
+	rwlock_t device_lock;
+	struct rb_root listen_service_table;
+	u64 listen_service_id;
+	/* struct rb_root peer_service_table; todo: fix peer to peer */
+	struct rb_root remote_qp_table;
+	struct rb_root remote_id_table;
+	struct rb_root remote_sidr_table;
+	struct idr local_id_table;
+	struct workqueue_struct *wq;
+} cm;
+
+struct cm_port {
+	struct cm_device *cm_dev;
+	struct ib_mad_agent *mad_agent;
+	u8 port_num;
+};
+
+struct cm_device {
+	struct list_head list;
+	struct ib_device *device;
+	u64 ca_guid;
+	struct cm_port port[0];
+};
+
+struct cm_av {
+	struct cm_port *port;
+	union ib_gid dgid;
+	struct ib_ah_attr ah_attr;
+	u16 pkey_index;
+	u8 packet_life_time;
+};
+
+struct cm_work {
+	struct work_struct work;
+	struct list_head list;
+	struct cm_port *port;
+	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
+	u32 local_id;				/* Established / timewait */
+	u32 remote_id;
+	struct ib_cm_event cm_event;
+	struct ib_sa_path_rec path[0];
+};
+
+struct cm_timewait_info {
+	struct cm_work work;			/* Must be first. */
+	struct rb_node remote_qp_node;
+	struct rb_node remote_id_node;
+	u64 remote_ca_guid;
+	u32 remote_qpn;
+	u8 inserted_remote_qp;
+	u8 inserted_remote_id;
+};
+
+struct cm_id_private {
+	struct ib_cm_id	id;
+
+	struct rb_node service_node;
+	struct rb_node sidr_id_node;
+	spinlock_t lock;
+	wait_queue_head_t wait;
+	atomic_t refcount;
+
+	struct ib_mad_send_buf *msg;
+	struct cm_timewait_info *timewait_info;
+	/* todo: use alternate port on send failure */
+	struct cm_av av;
+	struct cm_av alt_av;
+
+	void *private_data;
+	u64 tid;
+	u32 local_qpn;
+	u32 remote_qpn;
+	u32 sq_psn;
+	u32 rq_psn;
+	int timeout_ms;
+	enum ib_mtu path_mtu;
+	u8 private_data_len;
+	u8 max_cm_retries;
+	u8 peer_to_peer;
+	u8 responder_resources;
+	u8 initiator_depth;
+	u8 local_ack_timeout;
+	u8 retry_count;
+	u8 rnr_retry_count;
+	u8 service_timeout;
+
+	struct list_head work_list;
+	atomic_t work_count;
+};
+
+static void cm_work_handler(void *data);
+
+static inline void cm_deref_id(struct cm_id_private *cm_id_priv)
+{
+	if (atomic_dec_and_test(&cm_id_priv->refcount))
+		wake_up(&cm_id_priv->wait);
+}
+
+static int cm_alloc_msg(struct cm_id_private *cm_id_priv,
+			struct ib_mad_send_buf **msg)
+{
+	struct ib_mad_agent *mad_agent;
+	struct ib_mad_send_buf *m;
+	struct ib_ah *ah;
+
+	mad_agent = cm_id_priv->av.port->mad_agent;
+	ah = ib_create_ah(mad_agent->qp->pd, &cm_id_priv->av.ah_attr);
+	if (IS_ERR(ah))
+		return PTR_ERR(ah);
+
+	m = ib_create_send_mad(mad_agent, 1, cm_id_priv->av.pkey_index,
+			       ah, 0, sizeof(struct ib_mad_hdr),
+			       sizeof(struct ib_mad)-sizeof(struct ib_mad_hdr),
+			       GFP_ATOMIC);
+	if (IS_ERR(m)) {
+		ib_destroy_ah(ah);
+		return PTR_ERR(m);
+	}
+
+	/* Timeout set by caller if response is expected. */
+	m->send_wr.wr.ud.retries = cm_id_priv->max_cm_retries;
+
+	atomic_inc(&cm_id_priv->refcount);
+	m->context[0] = cm_id_priv;
+	*msg = m;
+	return 0;
+}
+
+static int cm_alloc_response_msg(struct cm_port *port,
+				 struct ib_mad_recv_wc *mad_recv_wc,
+				 struct ib_mad_send_buf **msg)
+{
+	struct ib_mad_send_buf *m;
+	struct ib_ah *ah;
+
+	ah = ib_create_ah_from_wc(port->mad_agent->qp->pd, mad_recv_wc->wc,
+				  mad_recv_wc->recv_buf.grh, port->port_num);
+	if (IS_ERR(ah))
+		return PTR_ERR(ah);
+
+	m = ib_create_send_mad(port->mad_agent, 1, mad_recv_wc->wc->pkey_index,
+			       ah, 0, sizeof(struct ib_mad_hdr),
+			       sizeof(struct ib_mad)-sizeof(struct ib_mad_hdr),
+			       GFP_ATOMIC);
+	if (IS_ERR(m)) {
+		ib_destroy_ah(ah);
+		return PTR_ERR(m);
+	}
+	*msg = m;
+	return 0;
+}
+
+static void cm_free_msg(struct ib_mad_send_buf *msg)
+{
+	ib_destroy_ah(msg->send_wr.wr.ud.ah);
+	if (msg->context[0])
+		cm_deref_id(msg->context[0]);
+	ib_free_send_mad(msg);
+}
+
+static void * cm_copy_private_data(const void *private_data,
+				   u8 private_data_len)
+{
+	void *data;
+
+	if (!private_data || !private_data_len)
+		return NULL;
+
+	data = kmalloc(private_data_len, GFP_KERNEL);
+	if (!data)
+		return ERR_PTR(-ENOMEM);
+
+	memcpy(data, private_data, private_data_len);
+	return data;
+}
+
+static void cm_set_private_data(struct cm_id_private *cm_id_priv,
+				 void *private_data, u8 private_data_len)
+{
+	if (cm_id_priv->private_data && cm_id_priv->private_data_len)
+		kfree(cm_id_priv->private_data);
+
+	cm_id_priv->private_data = private_data;
+	cm_id_priv->private_data_len = private_data_len;
+}
+
+static void cm_set_ah_attr(struct ib_ah_attr *ah_attr, u8 port_num,
+			   u16 dlid, u8 sl, u16 src_path_bits)
+{
+	memset(ah_attr, 0, sizeof ah_attr);
+	ah_attr->dlid = be16_to_cpu(dlid);
+	ah_attr->sl = sl;
+	ah_attr->src_path_bits = src_path_bits;
+	ah_attr->port_num = port_num;
+}
+
+static void cm_init_av_for_response(struct cm_port *port,
+				    struct ib_wc *wc, struct cm_av *av)
+{
+	av->port = port;
+	av->pkey_index = wc->pkey_index;
+	cm_set_ah_attr(&av->ah_attr, port->port_num, cpu_to_be16(wc->slid),
+		       wc->sl, wc->dlid_path_bits);
+}
+
+static int cm_init_av_by_path(struct ib_sa_path_rec *path, struct cm_av *av)
+{
+	struct cm_device *cm_dev;
+	struct cm_port *port = NULL;
+	unsigned long flags;
+	int ret;
+	u8 p;
+
+	read_lock_irqsave(&cm.device_lock, flags);
+	list_for_each_entry(cm_dev, &cm.device_list, list) {
+		if (!ib_find_cached_gid(cm_dev->device, &path->sgid,
+					&p, NULL)) {
+			port = &cm_dev->port[p-1];
+			break;
+		}
+	}
+	read_unlock_irqrestore(&cm.device_lock, flags);
+
+	if (!port)
+		return -EINVAL;
+
+	ret = ib_find_cached_pkey(cm_dev->device, port->port_num,
+				  be16_to_cpu(path->pkey), &av->pkey_index);
+	if (ret)
+		return ret;
+
+	av->port = port;
+	cm_set_ah_attr(&av->ah_attr, av->port->port_num, path->dlid,
+		       path->sl, path->slid & 0x7F);
+	av->packet_life_time = path->packet_life_time;
+	return 0;
+}
+
+static int cm_alloc_id(struct cm_id_private *cm_id_priv)
+{
+	unsigned long flags;
+	int ret;
+
+	do {
+		spin_lock_irqsave(&cm.lock, flags);
+		ret = idr_get_new_above(&cm.local_id_table, cm_id_priv, 1,
+					(int *) &cm_id_priv->id.local_id);
+		spin_unlock_irqrestore(&cm.lock, flags);
+	} while( (ret == -EAGAIN) && idr_pre_get(&cm.local_id_table, GFP_KERNEL) );
+	return ret;
+}
+
+static void cm_free_id(u32 local_id)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	idr_remove(&cm.local_id_table, (int) local_id);
+	spin_unlock_irqrestore(&cm.lock, flags);
+}
+
+static struct cm_id_private * cm_get_id(u32 local_id, u32 remote_id)
+{
+	struct cm_id_private *cm_id_priv;
+
+	cm_id_priv = idr_find(&cm.local_id_table, (int) local_id);
+	if (cm_id_priv) {
+		if (cm_id_priv->id.remote_id == remote_id)
+			atomic_inc(&cm_id_priv->refcount);
+		else
+			cm_id_priv = NULL;
+	}
+
+	return cm_id_priv;
+}
+
+static struct cm_id_private * cm_acquire_id(u32 local_id, u32 remote_id)
+{
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	cm_id_priv = cm_get_id(local_id, remote_id);
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	return cm_id_priv;
+}
+
+static struct cm_id_private * cm_insert_listen(struct cm_id_private *cm_id_priv)
+{
+	struct rb_node **link = &cm.listen_service_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_id_private *cur_cm_id_priv;
+	u64 service_id = cm_id_priv->id.service_id;
+	u64 service_mask = cm_id_priv->id.service_mask;
+
+	while (*link) {
+		parent = *link;
+		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
+					  service_node);
+		if ((cur_cm_id_priv->id.service_mask & service_id) ==
+		    (service_mask & cur_cm_id_priv->id.service_id))
+			return cm_id_priv;
+		if (service_id < cur_cm_id_priv->id.service_id)
+			link = &(*link)->rb_left;
+		else
+			link = &(*link)->rb_right;
+	}
+	rb_link_node(&cm_id_priv->service_node, parent, link);
+	rb_insert_color(&cm_id_priv->service_node, &cm.listen_service_table);
+	return NULL;
+}
+
+static struct cm_id_private * cm_find_listen(u64 service_id)
+{
+	struct rb_node *node = cm.listen_service_table.rb_node;
+	struct cm_id_private *cm_id_priv;
+
+	while (node) {
+		cm_id_priv = rb_entry(node, struct cm_id_private, service_node);
+		if ((cm_id_priv->id.service_mask & service_id) ==
+		    (cm_id_priv->id.service_mask & cm_id_priv->id.service_id))
+			return cm_id_priv;
+		if (service_id < cm_id_priv->id.service_id)
+			node = node->rb_left;
+		else
+			node = node->rb_right;
+	}
+	return NULL;
+}
+
+static struct cm_timewait_info * cm_insert_remote_id(struct cm_timewait_info
+						     *timewait_info)
+{
+	struct rb_node **link = &cm.remote_id_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_timewait_info *cur_timewait_info;
+	u64 remote_ca_guid = timewait_info->remote_ca_guid;
+	u32 remote_id = timewait_info->work.remote_id;
+
+	while (*link) {
+		parent = *link;
+		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
+					     remote_id_node);
+		if (remote_id < cur_timewait_info->work.remote_id)
+			link = &(*link)->rb_left;
+		else if (remote_id > cur_timewait_info->work.remote_id)
+			link = &(*link)->rb_right;
+		else if (remote_ca_guid < cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_left;
+		else if (remote_ca_guid > cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_right;
+		else
+			return cur_timewait_info;
+	}
+	timewait_info->inserted_remote_id = 1;
+	rb_link_node(&timewait_info->remote_id_node, parent, link);
+	rb_insert_color(&timewait_info->remote_id_node, &cm.remote_id_table);
+	return NULL;
+}
+
+static struct cm_timewait_info * cm_find_remote_id(u64 remote_ca_guid,
+						   u32 remote_id)
+{
+	struct rb_node *node = cm.remote_id_table.rb_node;
+	struct cm_timewait_info *timewait_info;
+
+	while (node) {
+		timewait_info = rb_entry(node, struct cm_timewait_info,
+					 remote_id_node);
+		if (remote_id < timewait_info->work.remote_id)
+			node = node->rb_left;
+		else if (remote_id > timewait_info->work.remote_id)
+			node = node->rb_right;
+		else if (remote_ca_guid < timewait_info->remote_ca_guid)
+			node = node->rb_left;
+		else if (remote_ca_guid > timewait_info->remote_ca_guid)
+			node = node->rb_right;
+		else
+			return timewait_info;
+	}
+	return NULL;
+}
+
+static struct cm_timewait_info * cm_insert_remote_qpn(struct cm_timewait_info
+						      *timewait_info)
+{
+	struct rb_node **link = &cm.remote_qp_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_timewait_info *cur_timewait_info;
+	u64 remote_ca_guid = timewait_info->remote_ca_guid;
+	u32 remote_qpn = timewait_info->remote_qpn;
+
+	while (*link) {
+		parent = *link;
+		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
+					     remote_qp_node);
+		if (remote_qpn < cur_timewait_info->remote_qpn)
+			link = &(*link)->rb_left;
+		else if (remote_qpn > cur_timewait_info->remote_qpn)
+			link = &(*link)->rb_right;
+		else if (remote_ca_guid < cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_left;
+		else if (remote_ca_guid > cur_timewait_info->remote_ca_guid)
+			link = &(*link)->rb_right;
+		else
+			return cur_timewait_info;
+	}
+	timewait_info->inserted_remote_qp = 1;
+	rb_link_node(&timewait_info->remote_qp_node, parent, link);
+	rb_insert_color(&timewait_info->remote_qp_node, &cm.remote_qp_table);
+	return NULL;
+}
+
+static struct cm_id_private * cm_insert_remote_sidr(struct cm_id_private
+						    *cm_id_priv)
+{
+	struct rb_node **link = &cm.remote_sidr_table.rb_node;
+	struct rb_node *parent = NULL;
+	struct cm_id_private *cur_cm_id_priv;
+	union ib_gid *port_gid = &cm_id_priv->av.dgid;
+	u32 remote_id = cm_id_priv->id.remote_id;
+
+	while (*link) {
+		parent = *link;
+		cur_cm_id_priv = rb_entry(parent, struct cm_id_private,
+					  sidr_id_node);
+		if (remote_id < cur_cm_id_priv->id.remote_id)
+			link = &(*link)->rb_left;
+		else if (remote_id > cur_cm_id_priv->id.remote_id)
+			link = &(*link)->rb_right;
+		else {
+			int cmp;
+			cmp = memcmp(port_gid, &cur_cm_id_priv->av.dgid,
+				     sizeof *port_gid);
+			if (cmp < 0)
+				link = &(*link)->rb_left;
+			else if (cmp > 0)
+				link = &(*link)->rb_right;
+			else
+				return cur_cm_id_priv;
+		}
+	}
+	rb_link_node(&cm_id_priv->sidr_id_node, parent, link);
+	rb_insert_color(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
+	return NULL;
+}
+
+static void cm_reject_sidr_req(struct cm_id_private *cm_id_priv,
+			       enum ib_cm_sidr_status status)
+{
+	struct ib_cm_sidr_rep_param param;
+
+	memset(&param, 0, sizeof param);
+	param.status = status;
+	ib_send_cm_sidr_rep(&cm_id_priv->id, &param);
+}
+
+struct ib_cm_id *ib_create_cm_id(ib_cm_handler cm_handler,
+				 void *context)
+{
+	struct cm_id_private *cm_id_priv;
+	int ret;
+
+	cm_id_priv = kmalloc(sizeof *cm_id_priv, GFP_KERNEL);
+	if (!cm_id_priv)
+		return ERR_PTR(-ENOMEM);
+
+	memset(cm_id_priv, 0, sizeof *cm_id_priv);
+	cm_id_priv->id.state = IB_CM_IDLE;
+	cm_id_priv->id.cm_handler = cm_handler;
+	cm_id_priv->id.context = context;
+	ret = cm_alloc_id(cm_id_priv);
+	if (ret)
+		goto error;
+
+	spin_lock_init(&cm_id_priv->lock);
+	init_waitqueue_head(&cm_id_priv->wait);
+	INIT_LIST_HEAD(&cm_id_priv->work_list);
+	atomic_set(&cm_id_priv->work_count, -1);
+	atomic_set(&cm_id_priv->refcount, 1);
+	return &cm_id_priv->id;
+
+error:
+	kfree(cm_id_priv);
+	return ERR_PTR(-ENOMEM);
+}
+EXPORT_SYMBOL(ib_create_cm_id);
+
+static struct cm_work * cm_dequeue_work(struct cm_id_private *cm_id_priv)
+{
+	struct cm_work *work;
+
+	if (list_empty(&cm_id_priv->work_list))
+		return NULL;
+
+	work = list_entry(cm_id_priv->work_list.next, struct cm_work, list);
+	list_del(&work->list);
+	return work;
+}
+
+static void cm_free_work(struct cm_work *work)
+{
+	if (work->mad_recv_wc)
+		ib_free_recv_mad(work->mad_recv_wc);
+	kfree(work);
+}
+
+static inline int cm_convert_to_ms(int iba_time)
+{
+	/* approximate conversion to ms from 4.096us x 2^iba_time */
+	return 1 << max(iba_time - 8, 0);
+}
+
+static void cm_cleanup_timewait(struct cm_timewait_info *timewait_info)
+{
+	unsigned long flags;
+
+	if (!timewait_info->inserted_remote_id &&
+	    !timewait_info->inserted_remote_qp)
+	    return;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	if (timewait_info->inserted_remote_id) {
+		rb_erase(&timewait_info->remote_id_node, &cm.remote_id_table);
+		timewait_info->inserted_remote_id = 0;
+	}
+
+	if (timewait_info->inserted_remote_qp) {
+		rb_erase(&timewait_info->remote_qp_node, &cm.remote_qp_table);
+		timewait_info->inserted_remote_qp = 0;
+	}
+	spin_unlock_irqrestore(&cm.lock, flags);
+}
+
+static struct cm_timewait_info * cm_create_timewait_info(u32 local_id)
+{
+	struct cm_timewait_info *timewait_info;
+
+	timewait_info = kmalloc(sizeof *timewait_info, GFP_KERNEL);
+	if (!timewait_info)
+		return ERR_PTR(-ENOMEM);
+	memset(timewait_info, 0, sizeof *timewait_info);
+
+	timewait_info->work.local_id = local_id;
+	INIT_WORK(&timewait_info->work.work, cm_work_handler,
+		  &timewait_info->work);
+	timewait_info->work.cm_event.event = IB_CM_TIMEWAIT_EXIT;
+	return timewait_info;
+}
+
+static void cm_enter_timewait(struct cm_id_private *cm_id_priv)
+{
+	int wait_time;
+
+	/*
+	 * The cm_id could be destroyed by the user before we exit timewait.
+	 * To protect against this, we search for the cm_id after exiting
+	 * timewait before notifying the user that we've exited timewait.
+	 */
+	cm_id_priv->id.state = IB_CM_TIMEWAIT;
+	wait_time = cm_convert_to_ms(cm_id_priv->local_ack_timeout);
+	queue_delayed_work(cm.wq, &cm_id_priv->timewait_info->work.work,
+			   msecs_to_jiffies(wait_time));
+	cm_id_priv->timewait_info = NULL;
+}
+
+static void cm_reset_to_idle(struct cm_id_private *cm_id_priv)
+{
+	cm_id_priv->id.state = IB_CM_IDLE;
+	if (cm_id_priv->timewait_info) {
+		cm_cleanup_timewait(cm_id_priv->timewait_info);
+		kfree(cm_id_priv->timewait_info);
+		cm_id_priv->timewait_info = NULL;
+	}
+}
+
+void ib_destroy_cm_id(struct ib_cm_id *cm_id)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_work *work;
+	unsigned long flags;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+retest:
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state) {
+	case IB_CM_LISTEN:
+		cm_id->state = IB_CM_IDLE;
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		spin_lock_irqsave(&cm.lock, flags);
+		rb_erase(&cm_id_priv->service_node, &cm.listen_service_table);
+		spin_unlock_irqrestore(&cm.lock, flags);
+		break;
+	case IB_CM_SIDR_REQ_SENT:
+		cm_id->state = IB_CM_IDLE;
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		break;
+	case IB_CM_SIDR_REQ_RCVD:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_reject_sidr_req(cm_id_priv, IB_SIDR_REJECT);
+		break;
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		/* Fall through */
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ib_send_cm_rej(cm_id, IB_CM_REJ_TIMEOUT, 
+			       &cm_id_priv->av.port->cm_dev->ca_guid,
+			       sizeof cm_id_priv->av.port->cm_dev->ca_guid,
+			       NULL, 0);
+		break;
+	case IB_CM_ESTABLISHED:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ib_send_cm_dreq(cm_id, NULL, 0);
+		goto retest;
+	case IB_CM_DREQ_SENT:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		cm_enter_timewait(cm_id_priv);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		break;
+	case IB_CM_DREQ_RCVD:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ib_send_cm_drep(cm_id, NULL, 0);
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		break;
+	}
+
+	cm_free_id(cm_id->local_id);
+	atomic_dec(&cm_id_priv->refcount);
+	wait_event(cm_id_priv->wait, !atomic_read(&cm_id_priv->refcount));
+	while ((work = cm_dequeue_work(cm_id_priv)) != NULL)
+		cm_free_work(work);
+	if (cm_id_priv->private_data && cm_id_priv->private_data_len)
+		kfree(cm_id_priv->private_data);
+	kfree(cm_id_priv);
+}
+EXPORT_SYMBOL(ib_destroy_cm_id);
+
+int ib_cm_listen(struct ib_cm_id *cm_id,
+		 u64 service_id,
+		 u64 service_mask)
+{
+	struct cm_id_private *cm_id_priv, *cur_cm_id_priv;
+	unsigned long flags;
+	int ret = 0;
+
+	service_mask = service_mask ? service_mask : ~0ULL;
+	service_id &= service_mask;
+	if ((service_id & IB_SERVICE_ID_AGN_MASK) == IB_CM_ASSIGN_SERVICE_ID &&
+	    (service_id != IB_CM_ASSIGN_SERVICE_ID))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	BUG_ON(cm_id->state != IB_CM_IDLE);
+
+	cm_id->state = IB_CM_LISTEN;
+
+	spin_lock_irqsave(&cm.lock, flags);
+	if (service_id == IB_CM_ASSIGN_SERVICE_ID) {
+		cm_id->service_id = __cpu_to_be64(cm.listen_service_id++);
+		cm_id->service_mask = ~0ULL;
+	} else {
+		cm_id->service_id = service_id;
+		cm_id->service_mask = service_mask;
+	}
+	cur_cm_id_priv = cm_insert_listen(cm_id_priv);
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	if (cur_cm_id_priv) {
+		cm_id->state = IB_CM_IDLE;
+		ret = -EBUSY;
+	}
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_listen);
+
+static u64 cm_form_tid(struct cm_id_private *cm_id_priv,
+		       enum cm_msg_sequence msg_seq)
+{
+	u64 hi_tid, low_tid;
+
+	hi_tid   = ((u64) cm_id_priv->av.port->mad_agent->hi_tid) << 32;
+	low_tid  = (u64) (cm_id_priv->id.local_id | (msg_seq << 30));
+	return cpu_to_be64(hi_tid | low_tid);
+}
+
+static void cm_format_mad_hdr(struct ib_mad_hdr *hdr,
+			      enum cm_msg_attr_id attr_id, u64 tid)
+{
+	hdr->base_version  = IB_MGMT_BASE_VERSION;
+	hdr->mgmt_class	   = IB_MGMT_CLASS_CM;
+	hdr->class_version = IB_CM_CLASS_VERSION;
+	hdr->method	   = IB_MGMT_METHOD_SEND;
+	hdr->attr_id	   = attr_id;
+	hdr->tid	   = tid;
+}
+
+static void cm_format_req(struct cm_req_msg *req_msg,
+			  struct cm_id_private *cm_id_priv,
+			  struct ib_cm_req_param *param)
+{
+	cm_format_mad_hdr(&req_msg->hdr, CM_REQ_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ));
+
+	req_msg->local_comm_id = cm_id_priv->id.local_id;
+	req_msg->service_id = param->service_id;
+	req_msg->local_ca_guid = cm_id_priv->av.port->cm_dev->ca_guid;
+	cm_req_set_local_qpn(req_msg, cpu_to_be32(param->qp_num));
+	cm_req_set_resp_res(req_msg, param->responder_resources);
+	cm_req_set_init_depth(req_msg, param->initiator_depth);
+	cm_req_set_remote_resp_timeout(req_msg,
+				       param->remote_cm_response_timeout);
+	cm_req_set_qp_type(req_msg, param->qp_type);
+	cm_req_set_flow_ctrl(req_msg, param->flow_control);
+	cm_req_set_starting_psn(req_msg, cpu_to_be32(param->starting_psn));
+	cm_req_set_local_resp_timeout(req_msg,
+				      param->local_cm_response_timeout);
+	cm_req_set_retry_count(req_msg, param->retry_count);
+	req_msg->pkey = param->primary_path->pkey;
+	cm_req_set_path_mtu(req_msg, param->primary_path->mtu);
+	cm_req_set_rnr_retry_count(req_msg, param->rnr_retry_count);
+	cm_req_set_max_cm_retries(req_msg, param->max_cm_retries);
+	cm_req_set_srq(req_msg, param->srq);
+
+	req_msg->primary_local_lid = param->primary_path->slid;
+	req_msg->primary_remote_lid = param->primary_path->dlid;
+	req_msg->primary_local_gid = param->primary_path->sgid;
+	req_msg->primary_remote_gid = param->primary_path->dgid;
+	cm_req_set_primary_flow_label(req_msg, param->primary_path->flow_label);
+	cm_req_set_primary_packet_rate(req_msg, param->primary_path->rate);
+	req_msg->primary_traffic_class = param->primary_path->traffic_class;
+	req_msg->primary_hop_limit = param->primary_path->hop_limit;
+	cm_req_set_primary_sl(req_msg, param->primary_path->sl);
+	cm_req_set_primary_subnet_local(req_msg, 1); /* local only... */
+	cm_req_set_primary_local_ack_timeout(req_msg,
+		min(31, param->primary_path->packet_life_time + 1));
+
+	if (param->alternate_path) {
+		req_msg->alt_local_lid = param->alternate_path->slid;
+		req_msg->alt_remote_lid = param->alternate_path->dlid;
+		req_msg->alt_local_gid = param->alternate_path->sgid;
+		req_msg->alt_remote_gid = param->alternate_path->dgid;
+		cm_req_set_alt_flow_label(req_msg,
+					  param->alternate_path->flow_label);
+		cm_req_set_alt_packet_rate(req_msg, param->alternate_path->rate);
+		req_msg->alt_traffic_class = param->alternate_path->traffic_class;
+		req_msg->alt_hop_limit = param->alternate_path->hop_limit;
+		cm_req_set_alt_sl(req_msg, param->alternate_path->sl);
+		cm_req_set_alt_subnet_local(req_msg, 1); /* local only... */
+		cm_req_set_alt_local_ack_timeout(req_msg,
+			min(31, param->alternate_path->packet_life_time + 1));
+	}
+
+	if (param->private_data && param->private_data_len)
+		memcpy(req_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+static inline int cm_validate_req_param(struct ib_cm_req_param *param)
+{
+	/* peer-to-peer not supported */
+	if (param->peer_to_peer)
+		return -EINVAL;
+
+	if (!param->primary_path)
+		return -EINVAL;
+
+	if (param->qp_type != IB_QPT_RC && param->qp_type != IB_QPT_UC)
+		return -EINVAL;
+
+	if (param->private_data &&
+	    param->private_data_len > IB_CM_REQ_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	if (param->alternate_path &&
+	    (param->alternate_path->pkey != param->primary_path->pkey ||
+	     param->alternate_path->mtu != param->primary_path->mtu))
+		return -EINVAL;
+
+	return 0;
+}
+
+int ib_send_cm_req(struct ib_cm_id *cm_id,
+		   struct ib_cm_req_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_send_wr *bad_send_wr;
+	struct cm_req_msg *req_msg;
+	unsigned long flags;
+	int ret;
+
+	ret = cm_validate_req_param(param);
+	if (ret)
+		return ret;
+
+	/* Verify that we're not in timewait. */
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_IDLE) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto out;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv->
+							    id.local_id);
+	if (IS_ERR(cm_id_priv->timewait_info))
+		goto out;
+
+	ret = cm_init_av_by_path(param->primary_path, &cm_id_priv->av);
+	if (ret)
+		goto error1;
+	if (param->alternate_path) {
+		ret = cm_init_av_by_path(param->alternate_path,
+					 &cm_id_priv->alt_av);
+		if (ret)
+			goto error1;
+	}
+	cm_id->service_id = param->service_id;
+	cm_id->service_mask = ~0ULL;
+	cm_id_priv->timeout_ms = cm_convert_to_ms(
+				    param->primary_path->packet_life_time) * 2 +
+				 cm_convert_to_ms(
+				    param->remote_cm_response_timeout);
+	cm_id_priv->max_cm_retries = param->max_cm_retries;
+	cm_id_priv->initiator_depth = param->initiator_depth;
+	cm_id_priv->responder_resources = param->responder_resources;
+	cm_id_priv->retry_count = param->retry_count;
+	cm_id_priv->path_mtu = param->primary_path->mtu;
+
+	ret = cm_alloc_msg(cm_id_priv, &cm_id_priv->msg);
+	if (ret)
+		goto error1;
+
+	req_msg = (struct cm_req_msg *) cm_id_priv->msg->mad;
+	cm_format_req(req_msg, cm_id_priv, param);
+	cm_id_priv->tid = req_msg->hdr.tid;
+	cm_id_priv->msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	cm_id_priv->msg->context[1] = (void *) (unsigned long) IB_CM_REQ_SENT;
+
+	cm_id_priv->local_qpn = cm_req_get_local_qpn(req_msg);
+	cm_id_priv->rq_psn = cm_req_get_starting_psn(req_msg);
+	cm_id_priv->local_ack_timeout =
+				cm_req_get_primary_local_ack_timeout(req_msg);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				&cm_id_priv->msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto error2;
+	}
+	BUG_ON(cm_id->state != IB_CM_IDLE);
+	cm_id->state = IB_CM_REQ_SENT;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return 0;
+
+error2:	cm_free_msg(cm_id_priv->msg);
+error1:	kfree(cm_id_priv->timewait_info);
+out:	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_req);
+
+static int cm_issue_rej(struct cm_port *port,
+			struct ib_mad_recv_wc *mad_recv_wc,
+			enum ib_cm_rej_reason reason,
+			enum cm_msg_response msg_rejected,
+			void *ari, u8 ari_length)
+{
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	struct cm_rej_msg *rej_msg, *rcv_msg;
+	int ret;
+
+	ret = cm_alloc_response_msg(port, mad_recv_wc, &msg);
+	if (ret)
+		return ret;
+	
+	/* We just need common CM header information.  Cast to any message. */
+	rcv_msg = (struct cm_rej_msg *) mad_recv_wc->recv_buf.mad;
+	rej_msg = (struct cm_rej_msg *) msg->mad;
+
+	cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, rcv_msg->hdr.tid);
+	rej_msg->remote_comm_id = rcv_msg->local_comm_id;
+	rej_msg->local_comm_id = rcv_msg->remote_comm_id;
+	cm_rej_set_msg_rejected(rej_msg, msg_rejected);
+	rej_msg->reason = reason;
+
+	if (ari && ari_length) {
+		cm_rej_set_reject_info_len(rej_msg, ari_length);
+		memcpy(rej_msg->ari, ari, ari_length);
+	}
+
+	ret = ib_post_send_mad(port->mad_agent, &msg->send_wr, &bad_send_wr);
+	if (ret)
+		cm_free_msg(msg);
+
+	return ret;
+}
+
+static inline int cm_is_active_peer(u64 local_ca_guid, u64 remote_ca_guid,
+				    u32 local_qpn, u32 remote_qpn)
+{
+	return (be64_to_cpu(local_ca_guid) > be64_to_cpu(remote_ca_guid) ||
+		((local_ca_guid == remote_ca_guid) &&
+		 (be32_to_cpu(local_qpn) > be32_to_cpu(remote_qpn))));
+}
+
+static inline void cm_format_paths_from_req(struct cm_req_msg *req_msg,
+					    struct ib_sa_path_rec *primary_path,
+					    struct ib_sa_path_rec *alt_path)
+{
+	memset(primary_path, 0, sizeof *primary_path);
+	primary_path->dgid = req_msg->primary_local_gid;
+	primary_path->sgid = req_msg->primary_remote_gid;
+	primary_path->dlid = req_msg->primary_local_lid;
+	primary_path->slid = req_msg->primary_remote_lid;
+	primary_path->flow_label = cm_req_get_primary_flow_label(req_msg);
+	primary_path->hop_limit = req_msg->primary_hop_limit;
+	primary_path->traffic_class = req_msg->primary_traffic_class;
+	primary_path->reversible = 1;
+	primary_path->pkey = req_msg->pkey;
+	primary_path->sl = cm_req_get_primary_sl(req_msg);
+	primary_path->mtu_selector = IB_SA_EQ;
+	primary_path->mtu = cm_req_get_path_mtu(req_msg);
+	primary_path->rate_selector = IB_SA_EQ;
+	primary_path->rate = cm_req_get_primary_packet_rate(req_msg);
+	primary_path->packet_life_time_selector = IB_SA_EQ;
+	primary_path->packet_life_time =
+		cm_req_get_primary_local_ack_timeout(req_msg);
+	primary_path->packet_life_time -= (primary_path->packet_life_time > 0);
+
+	if (req_msg->alt_local_lid) {
+		memset(alt_path, 0, sizeof *alt_path);
+		alt_path->dgid = req_msg->alt_local_gid;
+		alt_path->sgid = req_msg->alt_remote_gid;
+		alt_path->dlid = req_msg->alt_local_lid;
+		alt_path->slid = req_msg->alt_remote_lid;
+		alt_path->flow_label = cm_req_get_alt_flow_label(req_msg);
+		alt_path->hop_limit = req_msg->alt_hop_limit;
+		alt_path->traffic_class = req_msg->alt_traffic_class;
+		alt_path->reversible = 1;
+		alt_path->pkey = req_msg->pkey;
+		alt_path->sl = cm_req_get_alt_sl(req_msg);
+		alt_path->mtu_selector = IB_SA_EQ;
+		alt_path->mtu = cm_req_get_path_mtu(req_msg);
+		alt_path->rate_selector = IB_SA_EQ;
+		alt_path->rate = cm_req_get_alt_packet_rate(req_msg);
+		alt_path->packet_life_time_selector = IB_SA_EQ;
+		alt_path->packet_life_time =
+			cm_req_get_alt_local_ack_timeout(req_msg);
+		alt_path->packet_life_time -= (alt_path->packet_life_time > 0);
+	}
+}
+
+static void cm_format_req_event(struct cm_work *work,
+				struct cm_id_private *cm_id_priv,
+				struct ib_cm_id *listen_id)
+{
+	struct cm_req_msg *req_msg;
+	struct ib_cm_req_event_param *param;
+
+	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.req_rcvd;
+	param->listen_id = listen_id;
+	param->device = cm_id_priv->av.port->mad_agent->device;
+	param->port = cm_id_priv->av.port->port_num;
+	param->primary_path = &work->path[0];
+	if (req_msg->alt_local_lid)
+		param->alternate_path = &work->path[1];
+	else
+		param->alternate_path = NULL;
+	param->remote_ca_guid = req_msg->local_ca_guid;
+	param->remote_qkey = be32_to_cpu(req_msg->local_qkey);
+	param->remote_qpn = be32_to_cpu(cm_req_get_local_qpn(req_msg));
+	param->qp_type = cm_req_get_qp_type(req_msg);
+	param->starting_psn = be32_to_cpu(cm_req_get_starting_psn(req_msg));
+	param->responder_resources = cm_req_get_init_depth(req_msg);
+	param->initiator_depth = cm_req_get_resp_res(req_msg);
+	param->local_cm_response_timeout = 
+					cm_req_get_remote_resp_timeout(req_msg);
+	param->flow_control = cm_req_get_flow_ctrl(req_msg);
+	param->remote_cm_response_timeout = 
+					cm_req_get_local_resp_timeout(req_msg);
+	param->retry_count = cm_req_get_retry_count(req_msg);
+	param->rnr_retry_count = cm_req_get_rnr_retry_count(req_msg);
+	param->srq = cm_req_get_srq(req_msg);
+	work->cm_event.private_data = &req_msg->private_data;
+}
+
+static void cm_process_work(struct cm_id_private *cm_id_priv,
+			    struct cm_work *work)
+{
+	unsigned long flags;
+	int ret;
+
+	/* We will typically only have the current event to report. */
+	ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, &work->cm_event);
+	cm_free_work(work);
+
+	while (!ret && !atomic_add_negative(-1, &cm_id_priv->work_count)) {
+		spin_lock_irqsave(&cm_id_priv->lock, flags);
+		work = cm_dequeue_work(cm_id_priv);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		BUG_ON(!work);
+		ret = cm_id_priv->id.cm_handler(&cm_id_priv->id,
+						&work->cm_event);
+		cm_free_work(work);
+	}
+	cm_deref_id(cm_id_priv);
+	if (ret)
+		ib_destroy_cm_id(&cm_id_priv->id);	
+}
+
+static void cm_format_mra(struct cm_mra_msg *mra_msg,
+			  struct cm_id_private *cm_id_priv,
+			  enum cm_msg_response msg_mraed, u8 service_timeout,
+			  const void *private_data, u8 private_data_len)
+{
+	cm_format_mad_hdr(&mra_msg->hdr, CM_MRA_ATTR_ID, cm_id_priv->tid);
+	cm_mra_set_msg_mraed(mra_msg, msg_mraed);
+	mra_msg->local_comm_id = cm_id_priv->id.local_id;
+	mra_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_mra_set_service_timeout(mra_msg, service_timeout);
+
+	if (private_data && private_data_len)
+		memcpy(mra_msg->private_data, private_data, private_data_len);
+}
+
+static void cm_format_rej(struct cm_rej_msg *rej_msg,
+			  struct cm_id_private *cm_id_priv,
+			  enum ib_cm_rej_reason reason,
+			  void *ari,
+			  u8 ari_length,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, cm_id_priv->tid);
+	rej_msg->remote_comm_id = cm_id_priv->id.remote_id;
+
+	switch(cm_id_priv->id.state) {
+	case IB_CM_REQ_RCVD:
+		rej_msg->local_comm_id = 0;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ);
+		break;
+	case IB_CM_MRA_REQ_SENT:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ);
+		break;
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REP);
+		break;
+	default:
+		rej_msg->local_comm_id = cm_id_priv->id.local_id;
+		cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_OTHER);
+		break;
+	}
+
+	rej_msg->reason = reason;
+	if (ari && ari_length) {
+		cm_rej_set_reject_info_len(rej_msg, ari_length);
+		memcpy(rej_msg->ari, ari, ari_length);
+	}
+
+	if (private_data && private_data_len)
+		memcpy(rej_msg->private_data, private_data, private_data_len);
+}
+
+static void cm_dup_req_handler(struct cm_work *work,
+			       struct cm_id_private *cm_id_priv)
+{
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	/* Quick state check to discard duplicate REQs. */
+	if (cm_id_priv->id.state == IB_CM_REQ_RCVD)
+		return;
+
+	ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg);
+	if (ret)
+		return;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_MRA_REQ_SENT:
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REQ, cm_id_priv->service_timeout,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+		break;
+	case IB_CM_TIMEWAIT:
+		cm_format_rej((struct cm_rej_msg *) msg->mad, cm_id_priv,
+			      IB_CM_REJ_STALE_CONN, NULL, 0, NULL, 0);
+		break;
+	default:
+		goto unlock;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		goto free;
+	return;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+free:	cm_free_msg(msg);
+}
+
+static struct cm_id_private * cm_match_req(struct cm_work *work,
+					   struct cm_id_private *cm_id_priv)
+{
+	struct cm_id_private *listen_cm_id_priv, *cur_cm_id_priv;
+	struct cm_timewait_info *timewait_info;
+	struct cm_req_msg *req_msg;
+	unsigned long flags;
+
+	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
+
+	/* Check for duplicate REQ and stale connections. */
+	spin_lock_irqsave(&cm.lock, flags);
+	timewait_info = cm_insert_remote_id(cm_id_priv->timewait_info);
+	if (!timewait_info)
+		timewait_info = cm_insert_remote_qpn(cm_id_priv->timewait_info);
+
+	if (timewait_info) {
+		cur_cm_id_priv = cm_get_id(timewait_info->work.local_id,
+					   timewait_info->work.remote_id);
+		spin_unlock_irqrestore(&cm.lock, flags);
+		if (cur_cm_id_priv) {
+			cm_dup_req_handler(work, cur_cm_id_priv);
+			cm_deref_id(cur_cm_id_priv);
+		} else
+			cm_issue_rej(work->port, work->mad_recv_wc,
+				     IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REQ,
+				     NULL, 0);
+		goto error;
+	}
+
+	/* Find matching listen request. */
+	listen_cm_id_priv = cm_find_listen(req_msg->service_id);
+	if (!listen_cm_id_priv) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		cm_issue_rej(work->port, work->mad_recv_wc,
+			     IB_CM_REJ_INVALID_SERVICE_ID, CM_MSG_RESPONSE_REQ,
+			     NULL, 0);
+		goto error;
+	}
+	atomic_inc(&listen_cm_id_priv->refcount);
+	atomic_inc(&cm_id_priv->refcount);
+	cm_id_priv->id.state = IB_CM_REQ_RCVD;
+	atomic_inc(&cm_id_priv->work_count);
+	spin_unlock_irqrestore(&cm.lock, flags);
+	return listen_cm_id_priv;
+
+error:	cm_cleanup_timewait(cm_id_priv->timewait_info);
+	return NULL;
+}
+
+static int cm_req_handler(struct cm_work *work)
+{
+	struct ib_cm_id *cm_id;
+	struct cm_id_private *cm_id_priv, *listen_cm_id_priv;
+	struct cm_req_msg *req_msg;
+	int ret;
+
+	req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad;
+
+	cm_id = ib_create_cm_id(NULL, NULL);
+	if (IS_ERR(cm_id))
+		return PTR_ERR(cm_id);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	cm_id_priv->id.remote_id = req_msg->local_comm_id;
+	cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
+				&cm_id_priv->av);
+	cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv->
+							    id.local_id);
+	if (IS_ERR(cm_id_priv->timewait_info)) {
+		ret = PTR_ERR(cm_id_priv->timewait_info);
+		goto error1;
+	}
+	cm_id_priv->timewait_info->work.remote_id = req_msg->local_comm_id;
+	cm_id_priv->timewait_info->remote_ca_guid = req_msg->local_ca_guid;
+	cm_id_priv->timewait_info->remote_qpn = cm_req_get_local_qpn(req_msg);
+
+	listen_cm_id_priv = cm_match_req(work, cm_id_priv);
+	if (!listen_cm_id_priv) {
+		ret = -EINVAL;
+		goto error2;
+	}
+
+	cm_id_priv->id.cm_handler = listen_cm_id_priv->id.cm_handler;
+	cm_id_priv->id.context = listen_cm_id_priv->id.context;
+	cm_id_priv->id.service_id = req_msg->service_id;
+	cm_id_priv->id.service_mask = ~0ULL;
+
+	cm_format_paths_from_req(req_msg, &work->path[0], &work->path[1]);
+	ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av);
+	if (ret)
+		goto error3;
+	if (req_msg->alt_local_lid) {
+		ret = cm_init_av_by_path(&work->path[1], &cm_id_priv->alt_av);
+		if (ret)
+			goto error3;
+	}
+	cm_id_priv->tid = req_msg->hdr.tid;
+	cm_id_priv->timeout_ms = cm_convert_to_ms(
+					cm_req_get_local_resp_timeout(req_msg));
+	cm_id_priv->max_cm_retries = cm_req_get_max_cm_retries(req_msg);
+	cm_id_priv->remote_qpn = cm_req_get_local_qpn(req_msg);
+	cm_id_priv->initiator_depth = cm_req_get_resp_res(req_msg);
+	cm_id_priv->responder_resources = cm_req_get_init_depth(req_msg);
+	cm_id_priv->path_mtu = cm_req_get_path_mtu(req_msg);
+	cm_id_priv->sq_psn = cm_req_get_starting_psn(req_msg);
+	cm_id_priv->local_ack_timeout =
+				cm_req_get_primary_local_ack_timeout(req_msg);
+	cm_id_priv->retry_count = cm_req_get_retry_count(req_msg);
+	cm_id_priv->rnr_retry_count = cm_req_get_rnr_retry_count(req_msg);
+
+	cm_format_req_event(work, cm_id_priv, &listen_cm_id_priv->id);
+	cm_process_work(cm_id_priv, work);
+	cm_deref_id(listen_cm_id_priv);
+	return 0;
+
+error3:	atomic_dec(&cm_id_priv->refcount);
+	cm_deref_id(listen_cm_id_priv);
+	cm_cleanup_timewait(cm_id_priv->timewait_info);
+error2:	kfree(cm_id_priv->timewait_info);
+error1:	ib_destroy_cm_id(&cm_id_priv->id);
+	return ret;
+}
+
+static void cm_format_rep(struct cm_rep_msg *rep_msg,
+			  struct cm_id_private *cm_id_priv,
+			  struct ib_cm_rep_param *param)
+{
+	cm_format_mad_hdr(&rep_msg->hdr, CM_REP_ATTR_ID, cm_id_priv->tid);
+	rep_msg->local_comm_id = cm_id_priv->id.local_id;
+	rep_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_rep_set_local_qpn(rep_msg, cpu_to_be32(param->qp_num));
+	cm_rep_set_starting_psn(rep_msg, cpu_to_be32(param->starting_psn));
+	rep_msg->resp_resources = param->responder_resources;
+	rep_msg->initiator_depth = param->initiator_depth;
+	cm_rep_set_target_ack_delay(rep_msg, param->target_ack_delay);
+	cm_rep_set_failover(rep_msg, param->failover_accepted);
+	cm_rep_set_flow_ctrl(rep_msg, param->flow_control);
+	cm_rep_set_rnr_retry_count(rep_msg, param->rnr_retry_count);
+	cm_rep_set_srq(rep_msg, param->srq);
+	rep_msg->local_ca_guid = cm_id_priv->av.port->cm_dev->ca_guid;
+
+	if (param->private_data && param->private_data_len)
+		memcpy(rep_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+int ib_send_cm_rep(struct ib_cm_id *cm_id,
+		   struct ib_cm_rep_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct cm_rep_msg *rep_msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (param->private_data &&
+	    param->private_data_len > IB_CM_REP_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_REQ_RCVD &&
+	    cm_id->state != IB_CM_MRA_REQ_SENT) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	rep_msg = (struct cm_rep_msg *) msg->mad;
+	cm_format_rep(rep_msg, cm_id_priv, param);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_REP_SENT;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->state = IB_CM_REP_SENT;
+	cm_id_priv->msg = msg;
+	cm_id_priv->initiator_depth = param->initiator_depth;
+	cm_id_priv->responder_resources = param->responder_resources;
+	cm_id_priv->rq_psn = cm_rep_get_starting_psn(rep_msg);
+	cm_id_priv->local_qpn = cm_rep_get_local_qpn(rep_msg);
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_rep);
+
+static void cm_format_rtu(struct cm_rtu_msg *rtu_msg,
+			  struct cm_id_private *cm_id_priv,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&rtu_msg->hdr, CM_RTU_ATTR_ID, cm_id_priv->tid);
+	rtu_msg->local_comm_id = cm_id_priv->id.local_id;
+	rtu_msg->remote_comm_id = cm_id_priv->id.remote_id;
+
+	if (private_data && private_data_len)
+		memcpy(rtu_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_rtu(struct ib_cm_id *cm_id,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	void *data;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_RTU_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	data = cm_copy_private_data(private_data, private_data_len);
+	if (IS_ERR(data))
+		return PTR_ERR(data);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_REP_RCVD &&
+	    cm_id->state != IB_CM_MRA_REP_SENT) {
+		ret = -EINVAL;
+		goto error;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto error;
+
+	cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv,
+		      private_data, private_data_len);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		kfree(data);
+		return ret;
+	}
+
+	cm_id->state = IB_CM_ESTABLISHED;
+	cm_set_private_data(cm_id_priv, data, private_data_len);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return 0;
+
+error:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	kfree(data);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_rtu);
+
+static void cm_format_rep_event(struct cm_work *work)
+{
+	struct cm_rep_msg *rep_msg;
+	struct ib_cm_rep_event_param *param;
+
+	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.rep_rcvd;
+	param->remote_ca_guid = rep_msg->local_ca_guid;
+	param->remote_qkey = be32_to_cpu(rep_msg->local_qkey);
+	param->remote_qpn = be32_to_cpu(cm_rep_get_local_qpn(rep_msg));
+	param->starting_psn = be32_to_cpu(cm_rep_get_starting_psn(rep_msg));
+	param->responder_resources = rep_msg->initiator_depth;
+	param->initiator_depth = rep_msg->resp_resources;
+	param->target_ack_delay = cm_rep_get_target_ack_delay(rep_msg);
+	param->failover_accepted = cm_rep_get_failover(rep_msg);
+	param->flow_control = cm_rep_get_flow_ctrl(rep_msg);
+	param->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg);
+	param->srq = cm_rep_get_srq(rep_msg);
+	work->cm_event.private_data = &rep_msg->private_data;
+}
+
+static void cm_dup_rep_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rep_msg *rep_msg;
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	rep_msg = (struct cm_rep_msg *) work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id,
+				   rep_msg->local_comm_id);
+	if (!cm_id_priv)
+		return;
+
+	ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg);
+	if (ret)
+		goto deref;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state == IB_CM_ESTABLISHED)
+		cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+	else if (cm_id_priv->id.state == IB_CM_MRA_REP_SENT)
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REP, cm_id_priv->service_timeout,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+	else
+		goto unlock;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		goto free;
+	goto deref;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+free:	cm_free_msg(msg);
+deref:	cm_deref_id(cm_id_priv);
+}
+
+static int cm_rep_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rep_msg *rep_msg;
+	unsigned long flags;
+	int ret;
+
+	rep_msg = (struct cm_rep_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(rep_msg->remote_comm_id, 0);
+	if (!cm_id_priv) {
+		cm_dup_rep_handler(work);
+		return -EINVAL;
+	}
+
+	cm_id_priv->timewait_info->work.remote_id = rep_msg->local_comm_id;
+	cm_id_priv->timewait_info->remote_ca_guid = rep_msg->local_ca_guid;
+	cm_id_priv->timewait_info->remote_qpn = cm_rep_get_local_qpn(rep_msg);
+
+	spin_lock_irqsave(&cm.lock, flags);
+	/* Check for duplicate REP. */
+	if (cm_insert_remote_id(cm_id_priv->timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		ret = -EINVAL;
+		goto error;
+	}
+	/* Check for a stale connection. */
+	if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		cm_issue_rej(work->port, work->mad_recv_wc,
+			     IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REP,
+			     NULL, 0);
+		ret = -EINVAL;
+		goto error;
+	}
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	cm_format_rep_event(work);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto error;
+	}
+	cm_id_priv->id.state = IB_CM_REP_RCVD;
+	cm_id_priv->id.remote_id = rep_msg->local_comm_id;
+	cm_id_priv->remote_qpn = cm_rep_get_local_qpn(rep_msg);
+	cm_id_priv->initiator_depth = rep_msg->resp_resources;
+	cm_id_priv->responder_resources = rep_msg->initiator_depth;
+	cm_id_priv->sq_psn = cm_rep_get_starting_psn(rep_msg);
+	cm_id_priv->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg);
+
+	/* todo: handle peer_to_peer */
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+
+error:	cm_cleanup_timewait(cm_id_priv->timewait_info);
+	cm_deref_id(cm_id_priv);
+	return ret;
+}
+
+static int cm_establish_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	int ret;
+
+	/* See comment in ib_cm_establish about lookup. */
+	cm_id_priv = cm_acquire_id(work->local_id, work->remote_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static int cm_rtu_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rtu_msg *rtu_msg;
+	unsigned long flags;
+	int ret;
+
+	rtu_msg = (struct cm_rtu_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(rtu_msg->remote_comm_id,
+				   rtu_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &rtu_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_REP_SENT &&
+	    cm_id_priv->id.state != IB_CM_MRA_REP_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_ESTABLISHED;
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_dreq(struct cm_dreq_msg *dreq_msg,
+			  struct cm_id_private *cm_id_priv,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&dreq_msg->hdr, CM_DREQ_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_DREQ));
+	dreq_msg->local_comm_id = cm_id_priv->id.local_id;
+	dreq_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_dreq_set_remote_qpn(dreq_msg, cm_id_priv->remote_qpn);
+
+	if (private_data && private_data_len)
+		memcpy(dreq_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_dreq(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_DREQ_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_ESTABLISHED) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret) {
+		cm_enter_timewait(cm_id_priv);
+		goto out;
+	}
+
+	cm_format_dreq((struct cm_dreq_msg *) msg->mad, cm_id_priv,
+		       private_data, private_data_len);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_DREQ_SENT;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		cm_enter_timewait(cm_id_priv);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->state = IB_CM_DREQ_SENT;
+	cm_id_priv->msg = msg;
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_dreq);
+
+static void cm_format_drep(struct cm_drep_msg *drep_msg,
+			  struct cm_id_private *cm_id_priv,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&drep_msg->hdr, CM_DREP_ATTR_ID, cm_id_priv->tid);
+	drep_msg->local_comm_id = cm_id_priv->id.local_id;
+	drep_msg->remote_comm_id = cm_id_priv->id.remote_id;
+
+	if (private_data && private_data_len)
+		memcpy(drep_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_drep(struct ib_cm_id *cm_id,
+		    const void *private_data,
+		    u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	void *data;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_DREP_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	data = cm_copy_private_data(private_data, private_data_len);
+	if (IS_ERR(data))
+		return PTR_ERR(data);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_DREQ_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		kfree(data);
+		return -EINVAL;
+	}
+
+	cm_set_private_data(cm_id_priv, data, private_data_len);
+	cm_enter_timewait(cm_id_priv);
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv,
+		       private_data, private_data_len);
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_drep);
+
+static int cm_dreq_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_dreq_msg *dreq_msg;
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	dreq_msg = (struct cm_dreq_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(dreq_msg->remote_comm_id,
+				   dreq_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &dreq_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->local_qpn != cm_dreq_get_remote_qpn(dreq_msg))
+		goto unlock;
+
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REP_SENT:
+	case IB_CM_DREQ_SENT:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		break;
+	case IB_CM_ESTABLISHED:
+	case IB_CM_MRA_REP_RCVD:
+		break;
+	case IB_CM_TIMEWAIT:
+		if (cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg))
+			goto unlock;
+
+		cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv,
+			       cm_id_priv->private_data,
+			       cm_id_priv->private_data_len);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+		if (ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				     &msg->send_wr, &bad_send_wr))
+			cm_free_msg(msg);
+		goto deref;
+	default:
+		goto unlock;
+	}
+	cm_id_priv->id.state = IB_CM_DREQ_RCVD;
+	cm_id_priv->tid = dreq_msg->hdr.tid;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+deref:	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static int cm_drep_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_drep_msg *drep_msg;
+	unsigned long flags;
+	int ret;
+
+	drep_msg = (struct cm_drep_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(drep_msg->remote_comm_id,
+				   drep_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &drep_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_DREQ_SENT &&
+	    cm_id_priv->id.state != IB_CM_DREQ_RCVD) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_enter_timewait(cm_id_priv);
+
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+int ib_send_cm_rej(struct ib_cm_id *cm_id,
+		   enum ib_cm_rej_reason reason,
+		   void *ari,
+		   u8 ari_length,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if ((private_data && private_data_len > IB_CM_REJ_PRIVATE_DATA_SIZE) ||
+	    (ari && ari_length > IB_CM_REJ_ARI_LENGTH))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (!ret)
+			cm_format_rej((struct cm_rej_msg *) msg->mad,
+				      cm_id_priv, reason, ari, ari_length,
+				      private_data, private_data_len);
+
+		cm_reset_to_idle(cm_id_priv);
+		break;
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (!ret)
+			cm_format_rej((struct cm_rej_msg *) msg->mad,
+				      cm_id_priv, reason, ari, ari_length,
+				      private_data, private_data_len);
+
+		cm_enter_timewait(cm_id_priv);
+		break;
+	default:
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (ret)
+		goto out;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret)
+		cm_free_msg(msg);
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_rej);
+
+static void cm_format_rej_event(struct cm_work *work)
+{
+	struct cm_rej_msg *rej_msg;
+	struct ib_cm_rej_event_param *param;
+
+	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.rej_rcvd;
+	param->ari = rej_msg->ari;
+	param->ari_length = cm_rej_get_reject_info_len(rej_msg);
+	param->reason = rej_msg->reason;
+	work->cm_event.private_data = &rej_msg->private_data;
+}
+
+static struct cm_id_private * cm_acquire_rejected_id(struct cm_rej_msg *rej_msg)
+{
+	struct cm_timewait_info *timewait_info;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	u32 remote_id;
+
+	remote_id = rej_msg->local_comm_id;
+
+	if (rej_msg->reason == IB_CM_REJ_TIMEOUT) {
+		spin_lock_irqsave(&cm.lock, flags);
+		timewait_info = cm_find_remote_id( *((u64 *) rej_msg->ari),
+						  remote_id);
+		if (!timewait_info) {
+			spin_unlock_irqrestore(&cm.lock, flags);
+			return NULL;
+		}
+		cm_id_priv = idr_find(&cm.local_id_table,
+				      (int) timewait_info->work.local_id);
+		if (cm_id_priv) {
+			if (cm_id_priv->id.remote_id == remote_id)
+				atomic_inc(&cm_id_priv->refcount);
+			else
+				cm_id_priv = NULL;
+		}
+		spin_unlock_irqrestore(&cm.lock, flags);
+	} else if (cm_rej_get_msg_rejected(rej_msg) == CM_MSG_RESPONSE_REQ)
+		cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, 0);
+	else
+		cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, remote_id);
+
+	return cm_id_priv;
+}
+
+static int cm_rej_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_rej_msg *rej_msg;
+	unsigned long flags;
+	int ret;
+
+	rej_msg = (struct cm_rej_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_rejected_id(rej_msg);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	cm_format_rej_event(work);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		/* fall through */
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+		if (rej_msg->reason == IB_CM_REJ_STALE_CONN)
+			cm_enter_timewait(cm_id_priv);
+		else
+			cm_reset_to_idle(cm_id_priv);
+		break;
+	case IB_CM_DREQ_SENT:
+		ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+			      (unsigned long) cm_id_priv->msg);
+		/* fall through */
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_ESTABLISHED:
+		cm_enter_timewait(cm_id_priv);
+		break;
+	default:
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+int ib_send_cm_mra(struct ib_cm_id *cm_id,
+		   u8 service_timeout,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	void *data;
+	unsigned long flags;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_MRA_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	data = cm_copy_private_data(private_data, private_data_len);
+	if (IS_ERR(data))
+		return PTR_ERR(data);
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch(cm_id_priv->id.state) {
+	case IB_CM_REQ_RCVD:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (ret)
+			goto error1;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REQ, service_timeout,
+			      private_data, private_data_len);
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+		if (ret)
+			goto error2;
+		cm_id->state = IB_CM_MRA_REQ_SENT;
+		break;
+	case IB_CM_REP_RCVD:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (ret)
+			goto error1;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_REP, service_timeout,
+			      private_data, private_data_len);
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+		if (ret)
+			goto error2;
+		cm_id->state = IB_CM_MRA_REP_SENT;
+		break;
+	case IB_CM_ESTABLISHED:
+		ret = cm_alloc_msg(cm_id_priv, &msg);
+		if (ret)
+			goto error1;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_OTHER, service_timeout,
+			      private_data, private_data_len);
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+		if (ret)
+			goto error2;
+		cm_id->lap_state = IB_CM_MRA_LAP_SENT;
+		break;
+	default:
+		ret = -EINVAL;
+		goto error1;
+	}
+	cm_id_priv->service_timeout = service_timeout;
+	cm_set_private_data(cm_id_priv, data, private_data_len);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return 0;
+
+error1:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	kfree(data);
+	return ret;
+
+error2:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	kfree(data);
+	cm_free_msg(msg);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_mra);
+
+static struct cm_id_private * cm_acquire_mraed_id(struct cm_mra_msg *mra_msg)
+{
+	switch (cm_mra_get_msg_mraed(mra_msg)) {
+	case CM_MSG_RESPONSE_REQ:
+		return cm_acquire_id(mra_msg->remote_comm_id, 0);
+	case CM_MSG_RESPONSE_REP:
+	case CM_MSG_RESPONSE_OTHER:
+		return cm_acquire_id(mra_msg->remote_comm_id,
+				     mra_msg->local_comm_id);
+	default:
+		return NULL;
+	}
+}
+
+static int cm_mra_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_mra_msg *mra_msg;
+	unsigned long flags;
+	int timeout, ret;
+
+	mra_msg = (struct cm_mra_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_mraed_id(mra_msg);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	work->cm_event.private_data = &mra_msg->private_data;
+	work->cm_event.param.mra_rcvd.service_timeout =
+					cm_mra_get_service_timeout(mra_msg);
+	timeout = cm_convert_to_ms(cm_mra_get_service_timeout(mra_msg)) +
+		  cm_convert_to_ms(cm_id_priv->av.packet_life_time);
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+		if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_REQ ||
+		    ib_modify_mad(cm_id_priv->av.port->mad_agent,
+				  (unsigned long) cm_id_priv->msg, timeout))
+			goto out;
+		cm_id_priv->id.state = IB_CM_MRA_REQ_RCVD;
+		break;
+	case IB_CM_REP_SENT:
+		if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_REP ||
+		    ib_modify_mad(cm_id_priv->av.port->mad_agent,
+				  (unsigned long) cm_id_priv->msg, timeout))
+			goto out;
+		cm_id_priv->id.state = IB_CM_MRA_REP_RCVD;
+		break;
+	case IB_CM_ESTABLISHED:
+		if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_OTHER ||
+		    cm_id_priv->id.lap_state != IB_CM_LAP_SENT ||
+		    ib_modify_mad(cm_id_priv->av.port->mad_agent,
+				  (unsigned long) cm_id_priv->msg, timeout))
+			goto out;
+		cm_id_priv->id.lap_state = IB_CM_MRA_LAP_RCVD;
+		break;
+	default:
+		goto out;
+	}
+
+	cm_id_priv->msg->context[1] = (void *) (unsigned long)
+				      cm_id_priv->id.state;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_lap(struct cm_lap_msg *lap_msg,
+			  struct cm_id_private *cm_id_priv,
+			  struct ib_sa_path_rec *alternate_path,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&lap_msg->hdr, CM_LAP_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_LAP));
+	lap_msg->local_comm_id = cm_id_priv->id.local_id;
+	lap_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	cm_lap_set_remote_qpn(lap_msg, cm_id_priv->remote_qpn);
+	/* todo: need remote CM response timeout */
+	cm_lap_set_remote_resp_timeout(lap_msg, 0x1F);
+	lap_msg->alt_local_lid = alternate_path->slid;
+	lap_msg->alt_remote_lid = alternate_path->dlid;
+	lap_msg->alt_local_gid = alternate_path->sgid;
+	lap_msg->alt_remote_gid = alternate_path->dgid;
+	cm_lap_set_flow_label(lap_msg, alternate_path->flow_label);
+	cm_lap_set_traffic_class(lap_msg, alternate_path->traffic_class);
+	lap_msg->alt_hop_limit = alternate_path->hop_limit;
+	cm_lap_set_packet_rate(lap_msg, alternate_path->rate);
+	cm_lap_set_sl(lap_msg, alternate_path->sl);
+	cm_lap_set_subnet_local(lap_msg, 1); /* local only... */
+	cm_lap_set_local_ack_timeout(lap_msg,
+		min(31, alternate_path->packet_life_time + 1));
+
+	if (private_data && private_data_len)
+		memcpy(lap_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_lap(struct ib_cm_id *cm_id,
+		   struct ib_sa_path_rec *alternate_path,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (private_data && private_data_len > IB_CM_LAP_PRIVATE_DATA_SIZE)
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_ESTABLISHED ||
+	    cm_id->lap_state != IB_CM_LAP_IDLE) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_lap((struct cm_lap_msg *) msg->mad, cm_id_priv,
+		      alternate_path, private_data, private_data_len);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_ESTABLISHED;
+
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->lap_state = IB_CM_LAP_SENT;
+	cm_id_priv->msg = msg;
+
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_lap);
+
+static void cm_format_path_from_lap(struct ib_sa_path_rec *path,
+				    struct cm_lap_msg *lap_msg)
+{
+	memset(path, 0, sizeof *path);
+	path->dgid = lap_msg->alt_local_gid;
+	path->sgid = lap_msg->alt_remote_gid;
+	path->dlid = lap_msg->alt_local_lid;
+	path->slid = lap_msg->alt_remote_lid;
+	path->flow_label = cm_lap_get_flow_label(lap_msg);
+	path->hop_limit = lap_msg->alt_hop_limit;
+	path->traffic_class = cm_lap_get_traffic_class(lap_msg);
+	path->reversible = 1;
+	/* pkey is same as in REQ */
+	path->sl = cm_lap_get_sl(lap_msg);
+	path->mtu_selector = IB_SA_EQ;
+	/* mtu is same as in REQ */
+	path->rate_selector = IB_SA_EQ;
+	path->rate = cm_lap_get_packet_rate(lap_msg);
+	path->packet_life_time_selector = IB_SA_EQ;
+	path->packet_life_time = cm_lap_get_local_ack_timeout(lap_msg);
+	path->packet_life_time -= (path->packet_life_time > 0);
+}
+
+static int cm_lap_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_lap_msg *lap_msg;
+	struct ib_cm_lap_event_param *param;
+	struct ib_mad_send_buf *msg = NULL;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	/* todo: verify LAP request and send reject APR if invalid. */
+	lap_msg = (struct cm_lap_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(lap_msg->remote_comm_id,
+				   lap_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+	
+	param = &work->cm_event.param.lap_rcvd;
+	param->alternate_path = &work->path[0];
+	cm_format_path_from_lap(param->alternate_path, lap_msg);
+	work->cm_event.private_data = &lap_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED)
+		goto unlock;
+	
+	switch (cm_id_priv->id.lap_state) {
+	case IB_CM_LAP_IDLE:
+		break;
+	case IB_CM_MRA_LAP_SENT:
+		if (cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg))
+			goto unlock;
+
+		cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv,
+			      CM_MSG_RESPONSE_OTHER,
+			      cm_id_priv->service_timeout,
+			      cm_id_priv->private_data,
+			      cm_id_priv->private_data_len);
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+		if (ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				     &msg->send_wr, &bad_send_wr))
+			cm_free_msg(msg);
+		goto deref;
+	default:
+		goto unlock;
+	}
+
+	cm_id_priv->id.lap_state = IB_CM_LAP_RCVD;
+	cm_id_priv->tid = lap_msg->hdr.tid;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+
+unlock:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+deref:	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_apr(struct cm_apr_msg *apr_msg,
+			  struct cm_id_private *cm_id_priv,
+			  enum ib_cm_apr_status status,
+			  void *info,
+			  u8 info_length,
+			  const void *private_data,
+			  u8 private_data_len)
+{
+	cm_format_mad_hdr(&apr_msg->hdr, CM_APR_ATTR_ID, cm_id_priv->tid);
+	apr_msg->local_comm_id = cm_id_priv->id.local_id;
+	apr_msg->remote_comm_id = cm_id_priv->id.remote_id;
+	apr_msg->ap_status = (u8) status;
+
+	if (info && info_length) {
+		apr_msg->info_length = info_length;
+		memcpy(apr_msg->info, info, info_length);
+	}
+
+	if (private_data && private_data_len)
+		memcpy(apr_msg->private_data, private_data, private_data_len);
+}
+
+int ib_send_cm_apr(struct ib_cm_id *cm_id,
+		   enum ib_cm_apr_status status,
+		   void *info,
+		   u8 info_length,
+		   const void *private_data,
+		   u8 private_data_len)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if ((private_data && private_data_len > IB_CM_APR_PRIVATE_DATA_SIZE) ||
+	    (info && info_length > IB_CM_APR_INFO_LENGTH))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_ESTABLISHED ||
+	    (cm_id->lap_state != IB_CM_LAP_RCVD &&
+	     cm_id->lap_state != IB_CM_MRA_LAP_SENT)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_apr((struct cm_apr_msg *) msg->mad, cm_id_priv, status,
+		      info, info_length, private_data, private_data_len);
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+
+	cm_id->lap_state = IB_CM_LAP_IDLE;
+out:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_apr);
+
+static int cm_apr_handler(struct cm_work *work)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_apr_msg *apr_msg;
+	unsigned long flags;
+	int ret;
+
+	apr_msg = (struct cm_apr_msg *)work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(apr_msg->remote_comm_id,
+				   apr_msg->local_comm_id);
+	if (!cm_id_priv)
+		return -EINVAL; /* Unmatched reply. */
+
+	work->cm_event.param.apr_rcvd.ap_status = apr_msg->ap_status;
+	work->cm_event.param.apr_rcvd.apr_info = &apr_msg->info;
+	work->cm_event.param.apr_rcvd.info_len = apr_msg->info_length;
+	work->cm_event.private_data = &apr_msg->private_data;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_ESTABLISHED || 
+	    (cm_id_priv->id.lap_state != IB_CM_LAP_SENT &&
+	     cm_id_priv->id.lap_state != IB_CM_MRA_LAP_RCVD)) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.lap_state = IB_CM_LAP_IDLE;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	cm_id_priv->msg = NULL;
+
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static int cm_timewait_handler(struct cm_work *work)
+{
+	struct cm_timewait_info *timewait_info;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+	int ret;
+
+	timewait_info = (struct cm_timewait_info *)work;
+	cm_cleanup_timewait(timewait_info);
+
+	cm_id_priv = cm_acquire_id(timewait_info->work.local_id,
+				   timewait_info->work.remote_id);
+	if (!cm_id_priv)
+		return -EINVAL;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_TIMEWAIT ||
+	    cm_id_priv->remote_qpn != timewait_info->remote_qpn) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_IDLE;
+	ret = atomic_inc_and_test(&cm_id_priv->work_count);
+	if (!ret)
+		list_add_tail(&work->list, &cm_id_priv->work_list);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret)
+		cm_process_work(cm_id_priv, work);
+	else
+		cm_deref_id(cm_id_priv);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_format_sidr_req(struct cm_sidr_req_msg *sidr_req_msg,
+			       struct cm_id_private *cm_id_priv,
+			       struct ib_cm_sidr_req_param *param)
+{
+	cm_format_mad_hdr(&sidr_req_msg->hdr, CM_SIDR_REQ_ATTR_ID,
+			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_SIDR));
+	sidr_req_msg->request_id = cm_id_priv->id.local_id;
+	sidr_req_msg->pkey = param->pkey;
+	sidr_req_msg->service_id = param->service_id;
+
+	if (param->private_data && param->private_data_len)
+		memcpy(sidr_req_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+int ib_send_cm_sidr_req(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_req_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if (!param->path || (param->private_data && 
+	     param->private_data_len > IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	ret = cm_init_av_by_path(param->path, &cm_id_priv->av);
+	if (ret)
+		goto out;
+
+	cm_id->service_id = param->service_id;
+	cm_id->service_mask = ~0ULL;
+	cm_id_priv->timeout_ms = param->timeout_ms;
+	cm_id_priv->max_cm_retries = param->max_cm_retries;
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto out;
+
+	cm_format_sidr_req((struct cm_sidr_req_msg *) msg->mad, cm_id_priv,
+			   param);
+	msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms;
+	msg->context[1] = (void *) (unsigned long) IB_CM_SIDR_REQ_SENT;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state == IB_CM_IDLE)
+		ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+				       &msg->send_wr, &bad_send_wr);
+	else
+		ret = -EINVAL;
+
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		goto out;
+	}
+	cm_id->state = IB_CM_SIDR_REQ_SENT;
+	cm_id_priv->msg = msg;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+out:
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_sidr_req);
+
+static void cm_format_sidr_req_event(struct cm_work *work,
+				     struct ib_cm_id *listen_id)
+{
+	struct cm_sidr_req_msg *sidr_req_msg;
+	struct ib_cm_sidr_req_event_param *param;
+
+	sidr_req_msg = (struct cm_sidr_req_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.sidr_req_rcvd;
+	param->pkey = sidr_req_msg->pkey;
+	param->listen_id = listen_id;
+	param->device = work->port->mad_agent->device;
+	param->port = work->port->port_num;
+	work->cm_event.private_data = &sidr_req_msg->private_data;
+}
+
+static int cm_sidr_req_handler(struct cm_work *work)
+{
+	struct ib_cm_id *cm_id;
+	struct cm_id_private *cm_id_priv, *cur_cm_id_priv;
+	struct cm_sidr_req_msg *sidr_req_msg;
+	struct ib_wc *wc;
+	unsigned long flags;
+
+	cm_id = ib_create_cm_id(NULL, NULL);
+	if (IS_ERR(cm_id))
+		return PTR_ERR(cm_id);
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+
+	/* Record SGID/SLID and request ID for lookup. */
+	sidr_req_msg = (struct cm_sidr_req_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	wc = work->mad_recv_wc->wc;
+	cm_id_priv->av.dgid.global.subnet_prefix = wc->slid;
+	cm_id_priv->av.dgid.global.interface_id = 0;
+	cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
+				&cm_id_priv->av);
+	cm_id_priv->id.remote_id = sidr_req_msg->request_id;
+	cm_id_priv->id.state = IB_CM_SIDR_REQ_RCVD;
+	cm_id_priv->tid = sidr_req_msg->hdr.tid;
+	atomic_inc(&cm_id_priv->work_count);
+
+	spin_lock_irqsave(&cm.lock, flags);
+	cur_cm_id_priv = cm_insert_remote_sidr(cm_id_priv);
+	if (cur_cm_id_priv) {
+		spin_unlock_irqrestore(&cm.lock, flags);
+		goto out; /* Duplicate message. */
+	}
+	cur_cm_id_priv = cm_find_listen(sidr_req_msg->service_id);
+	if (!cur_cm_id_priv) {
+		rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
+		spin_unlock_irqrestore(&cm.lock, flags);
+		/* todo: reply with no match */
+		goto out; /* No match. */
+	}
+	atomic_inc(&cur_cm_id_priv->refcount);
+	spin_unlock_irqrestore(&cm.lock, flags);
+
+	cm_id_priv->id.cm_handler = cur_cm_id_priv->id.cm_handler;
+	cm_id_priv->id.context = cur_cm_id_priv->id.context;
+	cm_id_priv->id.service_id = sidr_req_msg->service_id;
+	cm_id_priv->id.service_mask = ~0ULL;
+
+	cm_format_sidr_req_event(work, &cur_cm_id_priv->id);
+	cm_process_work(cm_id_priv, work);
+	cm_deref_id(cur_cm_id_priv);
+	return 0;
+out:
+	ib_destroy_cm_id(&cm_id_priv->id);
+	return -EINVAL;
+}
+
+static void cm_format_sidr_rep(struct cm_sidr_rep_msg *sidr_rep_msg,
+			       struct cm_id_private *cm_id_priv,
+			       struct ib_cm_sidr_rep_param *param)
+{
+	cm_format_mad_hdr(&sidr_rep_msg->hdr, CM_SIDR_REP_ATTR_ID,
+			  cm_id_priv->tid);
+	sidr_rep_msg->request_id = cm_id_priv->id.remote_id;
+	sidr_rep_msg->status = param->status;
+	cm_sidr_rep_set_qpn(sidr_rep_msg, cpu_to_be32(param->qp_num));
+	sidr_rep_msg->service_id = cm_id_priv->id.service_id;
+	sidr_rep_msg->qkey = cpu_to_be32(param->qkey);
+
+	if (param->info && param->info_length)
+		memcpy(sidr_rep_msg->info, param->info, param->info_length);
+
+	if (param->private_data && param->private_data_len)
+		memcpy(sidr_rep_msg->private_data, param->private_data,
+		       param->private_data_len);
+}
+
+int ib_send_cm_sidr_rep(struct ib_cm_id *cm_id,
+			struct ib_cm_sidr_rep_param *param)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	unsigned long flags;
+	int ret;
+
+	if ((param->info && param->info_length > IB_CM_SIDR_REP_INFO_LENGTH) ||
+	    (param->private_data &&
+	     param->private_data_len > IB_CM_SIDR_REP_PRIVATE_DATA_SIZE))
+		return -EINVAL;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id->state != IB_CM_SIDR_REQ_RCVD) {
+		ret = -EINVAL;
+		goto error;
+	}
+
+	ret = cm_alloc_msg(cm_id_priv, &msg);
+	if (ret)
+		goto error;
+
+	cm_format_sidr_rep((struct cm_sidr_rep_msg *) msg->mad, cm_id_priv,
+			   param);
+	ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent,
+			       &msg->send_wr, &bad_send_wr);
+	if (ret) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		cm_free_msg(msg);
+		return ret;
+	}
+	cm_id->state = IB_CM_IDLE;
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	spin_lock_irqsave(&cm.lock, flags);
+	rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table);
+	spin_unlock_irqrestore(&cm.lock, flags);
+	return 0;
+
+error:	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL(ib_send_cm_sidr_rep);
+
+static void cm_format_sidr_rep_event(struct cm_work *work)
+{
+	struct cm_sidr_rep_msg *sidr_rep_msg;
+	struct ib_cm_sidr_rep_event_param *param;
+
+	sidr_rep_msg = (struct cm_sidr_rep_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	param = &work->cm_event.param.sidr_rep_rcvd;
+	param->status = sidr_rep_msg->status;
+	param->qkey = be32_to_cpu(sidr_rep_msg->qkey);
+	param->qpn = be32_to_cpu(cm_sidr_rep_get_qpn(sidr_rep_msg));
+	param->info = &sidr_rep_msg->info;
+	param->info_len = sidr_rep_msg->info_length;
+	work->cm_event.private_data = &sidr_rep_msg->private_data;
+}
+
+static int cm_sidr_rep_handler(struct cm_work *work)
+{
+	struct cm_sidr_rep_msg *sidr_rep_msg;
+	struct cm_id_private *cm_id_priv;
+	unsigned long flags;
+
+	sidr_rep_msg = (struct cm_sidr_rep_msg *)
+				work->mad_recv_wc->recv_buf.mad;
+	cm_id_priv = cm_acquire_id(sidr_rep_msg->request_id, 0);
+	if (!cm_id_priv)
+		return -EINVAL; /* Unmatched reply. */
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	if (cm_id_priv->id.state != IB_CM_SIDR_REQ_SENT) {
+		spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+		goto out;
+	}
+	cm_id_priv->id.state = IB_CM_IDLE;
+	ib_cancel_mad(cm_id_priv->av.port->mad_agent,
+		      (unsigned long) cm_id_priv->msg);
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	cm_format_sidr_rep_event(work);
+	cm_process_work(cm_id_priv, work);
+	return 0;
+out:
+	cm_deref_id(cm_id_priv);
+	return -EINVAL;
+}
+
+static void cm_process_send_error(struct ib_mad_send_buf *msg,
+				  enum ib_wc_status wc_status)
+{
+	struct cm_id_private *cm_id_priv;
+	struct ib_cm_event cm_event;
+	enum ib_cm_state state;
+	unsigned long flags;
+	int ret;
+
+	memset(&cm_event, 0, sizeof cm_event);
+	cm_id_priv = msg->context[0];
+
+	/* Discard old sends or ones without a response. */
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	state = (enum ib_cm_state) (unsigned long) msg->context[1];
+	if (msg != cm_id_priv->msg || state != cm_id_priv->id.state)
+		goto discard;
+
+	switch (state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+		cm_reset_to_idle(cm_id_priv);
+		cm_event.event = IB_CM_REQ_ERROR;
+		break;
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		cm_reset_to_idle(cm_id_priv);
+		cm_event.event = IB_CM_REP_ERROR;
+		break;
+	case IB_CM_DREQ_SENT:
+		cm_enter_timewait(cm_id_priv);
+		cm_event.event = IB_CM_DREQ_ERROR;
+		break;
+	case IB_CM_SIDR_REQ_SENT:
+		cm_id_priv->id.state = IB_CM_IDLE;
+		cm_event.event = IB_CM_SIDR_REQ_ERROR;
+		break;
+	default:
+		goto discard;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	cm_event.param.send_status = wc_status;
+
+	/* No other events can occur on the cm_id at this point. */
+	ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, &cm_event);
+	cm_free_msg(msg);
+	if (ret)
+		ib_destroy_cm_id(&cm_id_priv->id);
+	return;
+discard:
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	cm_free_msg(msg);
+}
+
+static void cm_send_handler(struct ib_mad_agent *mad_agent,
+			    struct ib_mad_send_wc *mad_send_wc)
+{
+	struct ib_mad_send_buf *msg;
+
+	msg = (struct ib_mad_send_buf *)(unsigned long)mad_send_wc->wr_id;
+
+	switch (mad_send_wc->status) {
+	case IB_WC_SUCCESS:
+	case IB_WC_WR_FLUSH_ERR:
+		cm_free_msg(msg);
+		break;
+	default:
+		if (msg->context[0] && msg->context[1])
+			cm_process_send_error(msg, mad_send_wc->status);
+		else
+			cm_free_msg(msg);
+		break;
+	}
+}
+
+static void cm_work_handler(void *data)
+{
+	struct cm_work *work = data;
+	int ret;
+
+	switch (work->cm_event.event) {
+	case IB_CM_REQ_RECEIVED:
+		ret = cm_req_handler(work);
+		break;
+	case IB_CM_MRA_RECEIVED:
+		ret = cm_mra_handler(work);
+		break;
+	case IB_CM_REJ_RECEIVED:
+		ret = cm_rej_handler(work);
+		break;
+	case IB_CM_REP_RECEIVED:
+		ret = cm_rep_handler(work);
+		break;
+	case IB_CM_RTU_RECEIVED:
+		ret = cm_rtu_handler(work);
+		break;
+	case IB_CM_USER_ESTABLISHED:
+		ret = cm_establish_handler(work);
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		ret = cm_dreq_handler(work);
+		break;
+	case IB_CM_DREP_RECEIVED:
+		ret = cm_drep_handler(work);
+		break;
+	case IB_CM_SIDR_REQ_RECEIVED:
+		ret = cm_sidr_req_handler(work);
+		break;
+	case IB_CM_SIDR_REP_RECEIVED:
+		ret = cm_sidr_rep_handler(work);
+		break;
+	case IB_CM_LAP_RECEIVED:
+		ret = cm_lap_handler(work);
+		break;
+	case IB_CM_APR_RECEIVED:
+		ret = cm_apr_handler(work);
+		break;
+	case IB_CM_TIMEWAIT_EXIT:
+		ret = cm_timewait_handler(work);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	if (ret)
+		cm_free_work(work);
+}
+
+int ib_cm_establish(struct ib_cm_id *cm_id)
+{
+	struct cm_id_private *cm_id_priv;
+	struct cm_work *work;
+	unsigned long flags;
+	int ret = 0;
+
+	work = kmalloc(sizeof *work, GFP_ATOMIC);
+	if (!work)
+		return -ENOMEM;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id->state)
+	{
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+		cm_id->state = IB_CM_ESTABLISHED;
+		break;
+	case IB_CM_ESTABLISHED:
+		ret = -EISCONN;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+
+	if (ret) {
+		kfree(work);
+		goto out;
+	}
+
+	/*
+	 * The CM worker thread may try to destroy the cm_id before it
+	 * can execute this work item.  To prevent potential deadlock,
+	 * we need to find the cm_id once we're in the context of the 
+	 * worker thread, rather than holding a reference on it.
+	 */
+	INIT_WORK(&work->work, cm_work_handler, work);
+	work->local_id = cm_id->local_id;
+	work->remote_id = cm_id->remote_id;
+	work->mad_recv_wc = NULL;
+	work->cm_event.event = IB_CM_USER_ESTABLISHED;
+	queue_work(cm.wq, &work->work);
+out:
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_establish);
+
+static void cm_recv_handler(struct ib_mad_agent *mad_agent,
+			    struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct cm_work *work;
+	enum ib_cm_event_type event;
+	int paths = 0;
+
+	switch (mad_recv_wc->recv_buf.mad->mad_hdr.attr_id) {
+	case CM_REQ_ATTR_ID:
+		paths = 1 + (((struct cm_req_msg *) mad_recv_wc->recv_buf.mad)->
+						    alt_local_lid != 0);
+		event = IB_CM_REQ_RECEIVED;
+		break;
+	case CM_MRA_ATTR_ID:
+		event = IB_CM_MRA_RECEIVED;
+		break;
+	case CM_REJ_ATTR_ID:
+		event = IB_CM_REJ_RECEIVED;
+		break;
+	case CM_REP_ATTR_ID:
+		event = IB_CM_REP_RECEIVED;
+		break;
+	case CM_RTU_ATTR_ID:
+		event = IB_CM_RTU_RECEIVED;
+		break;
+	case CM_DREQ_ATTR_ID:
+		event = IB_CM_DREQ_RECEIVED;
+		break;
+	case CM_DREP_ATTR_ID:
+		event = IB_CM_DREP_RECEIVED;
+		break;
+	case CM_SIDR_REQ_ATTR_ID:
+		event = IB_CM_SIDR_REQ_RECEIVED;
+		break;
+	case CM_SIDR_REP_ATTR_ID:
+		event = IB_CM_SIDR_REP_RECEIVED;
+		break;
+	case CM_LAP_ATTR_ID:
+		paths = 1;
+		event = IB_CM_LAP_RECEIVED;
+		break;
+	case CM_APR_ATTR_ID:
+		event = IB_CM_APR_RECEIVED;
+		break;
+	default:
+		ib_free_recv_mad(mad_recv_wc);
+		return;
+	}
+
+	work = kmalloc(sizeof *work + sizeof(struct ib_sa_path_rec) * paths,
+		       GFP_KERNEL);
+	if (!work) {
+		ib_free_recv_mad(mad_recv_wc);
+		return;
+	}
+
+	INIT_WORK(&work->work, cm_work_handler, work);
+	work->cm_event.event = event;
+	work->mad_recv_wc = mad_recv_wc;
+	work->port = (struct cm_port *)mad_agent->context;
+	queue_work(cm.wq, &work->work);
+}
+
+static int cm_init_qp_init_attr(struct cm_id_private *cm_id_priv,
+				struct ib_qp_attr *qp_attr,
+				int *qp_attr_mask)
+{
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_SENT:
+	case IB_CM_MRA_REQ_RCVD:
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+	case IB_CM_ESTABLISHED:
+		*qp_attr_mask = IB_QP_STATE | IB_QP_ACCESS_FLAGS |
+				IB_QP_PKEY_INDEX | IB_QP_PORT;
+		qp_attr->qp_access_flags = IB_ACCESS_LOCAL_WRITE;
+		if (cm_id_priv->responder_resources)
+			qp_attr->qp_access_flags |= IB_ACCESS_REMOTE_WRITE |
+						    IB_ACCESS_REMOTE_READ;
+		qp_attr->pkey_index = cm_id_priv->av.pkey_index;
+		qp_attr->port_num = cm_id_priv->av.port->port_num;
+		ret = 0;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+
+static int cm_init_qp_rtr_attr(struct cm_id_private *cm_id_priv,
+			       struct ib_qp_attr *qp_attr,
+			       int *qp_attr_mask)
+{
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REQ_RCVD:
+	case IB_CM_MRA_REQ_SENT:
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+	case IB_CM_ESTABLISHED:
+		*qp_attr_mask = IB_QP_STATE | IB_QP_AV | IB_QP_PATH_MTU |
+				IB_QP_DEST_QPN | IB_QP_RQ_PSN |
+				IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER;
+		qp_attr->ah_attr = cm_id_priv->av.ah_attr;
+		qp_attr->path_mtu = cm_id_priv->path_mtu;
+		qp_attr->dest_qp_num = be32_to_cpu(cm_id_priv->remote_qpn);
+		qp_attr->rq_psn = be32_to_cpu(cm_id_priv->rq_psn);
+		qp_attr->max_dest_rd_atomic = cm_id_priv->responder_resources;
+		qp_attr->min_rnr_timer = 0;
+		if (cm_id_priv->alt_av.ah_attr.dlid) {
+			*qp_attr_mask |= IB_QP_ALT_PATH;
+			qp_attr->alt_ah_attr = cm_id_priv->alt_av.ah_attr;
+		}
+		ret = 0;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+
+static int cm_init_qp_rts_attr(struct cm_id_private *cm_id_priv,
+			       struct ib_qp_attr *qp_attr,
+			       int *qp_attr_mask)
+{
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&cm_id_priv->lock, flags);
+	switch (cm_id_priv->id.state) {
+	case IB_CM_REP_RCVD:
+	case IB_CM_MRA_REP_SENT:
+	case IB_CM_REP_SENT:
+	case IB_CM_MRA_REP_RCVD:
+	case IB_CM_ESTABLISHED:
+		*qp_attr_mask = IB_QP_STATE | IB_QP_TIMEOUT | IB_QP_RETRY_CNT |
+				IB_QP_RNR_RETRY | IB_QP_SQ_PSN |
+				IB_QP_MAX_QP_RD_ATOMIC;
+		qp_attr->timeout = cm_id_priv->local_ack_timeout;
+		qp_attr->retry_cnt = cm_id_priv->retry_count;
+		qp_attr->rnr_retry = cm_id_priv->rnr_retry_count;
+		qp_attr->sq_psn = be32_to_cpu(cm_id_priv->sq_psn);
+		qp_attr->max_rd_atomic = cm_id_priv->initiator_depth;
+		if (cm_id_priv->alt_av.ah_attr.dlid) {
+			*qp_attr_mask |= IB_QP_PATH_MIG_STATE;
+			qp_attr->path_mig_state = IB_MIG_REARM;
+		}
+		ret = 0;
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	spin_unlock_irqrestore(&cm_id_priv->lock, flags);
+	return ret;
+}
+
+int ib_cm_init_qp_attr(struct ib_cm_id *cm_id,
+		       struct ib_qp_attr *qp_attr,
+		       int *qp_attr_mask)
+{
+	struct cm_id_private *cm_id_priv;
+	int ret;
+
+	cm_id_priv = container_of(cm_id, struct cm_id_private, id);
+	switch (qp_attr->qp_state) {
+	case IB_QPS_INIT:
+		ret = cm_init_qp_init_attr(cm_id_priv, qp_attr, qp_attr_mask);
+		break;
+	case IB_QPS_RTR:
+		ret = cm_init_qp_rtr_attr(cm_id_priv, qp_attr, qp_attr_mask);
+		break;
+	case IB_QPS_RTS:
+		ret = cm_init_qp_rts_attr(cm_id_priv, qp_attr, qp_attr_mask);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	return ret;
+}
+EXPORT_SYMBOL(ib_cm_init_qp_attr);
+
+static u64 cm_get_ca_guid(struct ib_device *device)
+{
+	struct ib_device_attr *device_attr;
+	u64 guid;
+	int ret;
+
+	device_attr = kmalloc(sizeof *device_attr, GFP_KERNEL);
+	if (!device_attr)
+		return 0;
+
+	ret = ib_query_device(device, device_attr);
+	guid = ret ? 0 : device_attr->node_guid;
+	kfree(device_attr);
+	return guid;
+}
+
+static void cm_add_one(struct ib_device *device)
+{
+	struct cm_device *cm_dev;
+	struct cm_port *port;
+	struct ib_mad_reg_req reg_req = {
+		.mgmt_class = IB_MGMT_CLASS_CM,
+		.mgmt_class_version = IB_CM_CLASS_VERSION
+	};
+	struct ib_port_modify port_modify = {
+		.set_port_cap_mask = IB_PORT_CM_SUP
+	};
+	unsigned long flags;
+	int ret;
+	u8 i;
+
+	cm_dev = kmalloc(sizeof(*cm_dev) + sizeof(*port) *
+			 device->phys_port_cnt, GFP_KERNEL);
+	if (!cm_dev)
+		return;
+
+	cm_dev->device = device;
+	cm_dev->ca_guid = cm_get_ca_guid(device);
+	if (!cm_dev->ca_guid)
+		goto error1;
+
+	set_bit(IB_MGMT_METHOD_SEND, reg_req.method_mask);
+	for (i = 1; i <= device->phys_port_cnt; i++) {
+		port = &cm_dev->port[i-1];
+		port->cm_dev = cm_dev;
+		port->port_num = i;
+		port->mad_agent = ib_register_mad_agent(device, i,
+							IB_QPT_GSI,
+							&reg_req,
+							0,
+							cm_send_handler,
+							cm_recv_handler,
+							port);
+		if (IS_ERR(port->mad_agent))
+			goto error2;
+
+		ret = ib_modify_port(device, i, 0, &port_modify);
+		if (ret)
+			goto error3;
+	}
+	ib_set_client_data(device, &cm_client, cm_dev);
+	
+	write_lock_irqsave(&cm.device_lock, flags);
+	list_add_tail(&cm_dev->list, &cm.device_list);
+	write_unlock_irqrestore(&cm.device_lock, flags);
+	return;
+
+error3:
+	ib_unregister_mad_agent(port->mad_agent);
+error2:
+	port_modify.set_port_cap_mask = 0;
+	port_modify.clr_port_cap_mask = IB_PORT_CM_SUP;
+	while (--i) {
+		port = &cm_dev->port[i-1];
+		ib_modify_port(device, port->port_num, 0, &port_modify);
+		ib_unregister_mad_agent(port->mad_agent);
+	}
+error1:
+	kfree(cm_dev);
+}
+
+static void cm_remove_one(struct ib_device *device)
+{
+	struct cm_device *cm_dev;
+	struct cm_port *port;
+	struct ib_port_modify port_modify = {
+		.clr_port_cap_mask = IB_PORT_CM_SUP
+	};
+	unsigned long flags;
+	int i;
+
+	cm_dev = ib_get_client_data(device, &cm_client);
+	if (!cm_dev)
+		return;
+
+	write_lock_irqsave(&cm.device_lock, flags);
+	list_del(&cm_dev->list);
+	write_unlock_irqrestore(&cm.device_lock, flags);
+
+	for (i = 1; i <= device->phys_port_cnt; i++) {
+		port = &cm_dev->port[i-1];
+		ib_modify_port(device, port->port_num, 0, &port_modify);
+		ib_unregister_mad_agent(port->mad_agent);
+	}
+	kfree(cm_dev);
+}
+
+static int __init ib_cm_init(void)
+{
+	int ret;
+
+	memset(&cm, 0, sizeof cm);
+	INIT_LIST_HEAD(&cm.device_list);
+	rwlock_init(&cm.device_lock);
+	spin_lock_init(&cm.lock);
+	cm.listen_service_table = RB_ROOT;
+	cm.listen_service_id = __constant_be64_to_cpu(IB_CM_ASSIGN_SERVICE_ID);
+	cm.remote_id_table = RB_ROOT;
+	cm.remote_qp_table = RB_ROOT;
+	cm.remote_sidr_table = RB_ROOT;
+	idr_init(&cm.local_id_table);
+	idr_pre_get(&cm.local_id_table, GFP_KERNEL);
+
+	cm.wq = create_workqueue("ib_cm");
+	if (!cm.wq)
+		return -ENOMEM;
+
+	ret = ib_register_client(&cm_client);
+	if (ret)
+		goto error;
+
+	return 0;
+error:
+	destroy_workqueue(cm.wq);
+	return ret;
+}
+
+static void __exit ib_cm_cleanup(void)
+{
+	flush_workqueue(cm.wq);
+	destroy_workqueue(cm.wq);
+	ib_unregister_client(&cm_client);
+}
+
+module_init(ib_cm_init);
+module_exit(ib_cm_cleanup);
+
diff -uprN linux-2.6.13-rc2-mm1-21/drivers/infiniband/core/cm_msgs.h linux-2.6.13-rc2-mm1-22/drivers/infiniband/core/cm_msgs.h
-- linux-2.6.13-rc2-mm1-21/drivers/infiniband/core/cm_msgs.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-22/drivers/infiniband/core/cm_msgs.h	2005-02-03 14:05:21.000000000 -0500
@@ -0,0 +1,819 @@
+/*
+ * Copyright (c) 2004 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
+ * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING the madirectory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use source and binary forms, with or
+ *     withmodification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retathe above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHWARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS THE
+ * SOFTWARE.
+ */
+#if !defined(CM_MSGS_H)
+#define CM_MSGS_H
+
+#include <ib_mad.h>
+
+/*
+ * Parameters to routines below should be in network-byte order, and values
+ * are returned in network-byte order.
+ */
+
+#define IB_CM_CLASS_VERSION	2 /* IB specification 1.2 */
+
+enum cm_msg_attr_id {
+	CM_REQ_ATTR_ID	    = __constant_htons(0x0010),
+	CM_MRA_ATTR_ID	    = __constant_htons(0x0011),
+	CM_REJ_ATTR_ID	    = __constant_htons(0x0012),
+	CM_REP_ATTR_ID	    = __constant_htons(0x0013),
+	CM_RTU_ATTR_ID	    = __constant_htons(0x0014),
+	CM_DREQ_ATTR_ID	    = __constant_htons(0x0015),
+	CM_DREP_ATTR_ID	    = __constant_htons(0x0016),
+	CM_SIDR_REQ_ATTR_ID = __constant_htons(0x0017),
+	CM_SIDR_REP_ATTR_ID = __constant_htons(0x0018),
+	CM_LAP_ATTR_ID      = __constant_htons(0x0019),
+	CM_APR_ATTR_ID      = __constant_htons(0x001A)
+};
+
+enum cm_msg_sequence {
+	CM_MSG_SEQUENCE_REQ,
+	CM_MSG_SEQUENCE_LAP,
+	CM_MSG_SEQUENCE_DREQ,
+	CM_MSG_SEQUENCE_SIDR
+};
+
+struct cm_req_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 rsvd4;
+	u64 service_id;
+	u64 local_ca_guid;
+	u32 rsvd24;
+	u32 local_qkey;
+	/* local QPN:24, responder resources:8 */
+	u32 offset32;
+	/* local EECN:24, initiator depth:8 */
+	u32 offset36;
+	/*
+	 * remote EECN:24, remote CM response timeout:5,
+	 * transport service type:2, end-to-end flow control:1
+	 */
+	u32 offset40;
+	/* starting PSN:24, local CM response timeout:5, retry count:3 */
+	u32 offset44;
+	u16 pkey;
+	/* path MTU:4, RDC exists:1, RNR retry count:3. */
+	u8 offset50;
+	/* max CM Retries:4, SRQ:1, rsvd:3 */
+	u8 offset51;
+
+	u16 primary_local_lid;
+	u16 primary_remote_lid;
+	union ib_gid primary_local_gid;
+	union ib_gid primary_remote_gid;
+	/* flow label:20, rsvd:6, packet rate:6 */
+	u32 primary_offset88;
+	u8 primary_traffic_class;
+	u8 primary_hop_limit;
+	/* SL:4, subnet local:1, rsvd:3 */
+	u8 primary_offset94;
+	/* local ACK timeout:5, rsvd:3 */
+	u8 primary_offset95;
+
+	u16 alt_local_lid;
+	u16 alt_remote_lid;
+	union ib_gid alt_local_gid;
+	union ib_gid alt_remote_gid;
+	/* flow label:20, rsvd:6, packet rate:6 */
+	u32 alt_offset132;
+	u8 alt_traffic_class;
+	u8 alt_hop_limit;
+	/* SL:4, subnet local:1, rsvd:3 */
+	u8 alt_offset138;
+	/* local ACK timeout:5, rsvd:3 */
+	u8 alt_offset139;
+
+	u8 private_data[IB_CM_REQ_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u32 cm_req_get_local_qpn(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32(be32_to_cpu(req_msg->offset32) >> 8);
+}
+
+static inline void cm_req_set_local_qpn(struct cm_req_msg *req_msg, u32 qpn)
+{
+	req_msg->offset32 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+					 (be32_to_cpu(req_msg->offset32) &
+					  0x000000FF));
+}
+
+static inline u8 cm_req_get_resp_res(struct cm_req_msg *req_msg)
+{
+	return (u8) be32_to_cpu(req_msg->offset32);
+}
+
+static inline void cm_req_set_resp_res(struct cm_req_msg *req_msg, u8 resp_res)
+{
+	req_msg->offset32 = cpu_to_be32(resp_res |
+					(be32_to_cpu(req_msg->offset32) &
+					 0xFFFFFF00));
+}
+
+static inline u8 cm_req_get_init_depth(struct cm_req_msg *req_msg)
+{
+	return (u8) be32_to_cpu(req_msg->offset36);
+}
+
+static inline void cm_req_set_init_depth(struct cm_req_msg *req_msg,
+					 u8 init_depth)
+{
+	req_msg->offset36 = cpu_to_be32(init_depth |
+					(be32_to_cpu(req_msg->offset36) &
+					 0xFFFFFF00));
+}
+
+static inline u8 cm_req_get_remote_resp_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) ((be32_to_cpu(req_msg->offset40) & 0xF8) >> 3);
+}
+
+static inline void cm_req_set_remote_resp_timeout(struct cm_req_msg *req_msg,
+						  u8 resp_timeout)
+{
+	req_msg->offset40 = cpu_to_be32((resp_timeout << 3) |
+					 (be32_to_cpu(req_msg->offset40) &
+					  0xFFFFFF07));
+}
+
+static inline enum ib_qp_type cm_req_get_qp_type(struct cm_req_msg *req_msg)
+{
+	u8 transport_type = (u8) (be32_to_cpu(req_msg->offset40) & 0x06) >> 1;
+	switch(transport_type) {
+	case 0: return IB_QPT_RC;
+	case 1: return IB_QPT_UC;
+	default: return 0;
+	}
+}
+
+static inline void cm_req_set_qp_type(struct cm_req_msg *req_msg,
+				      enum ib_qp_type qp_type)
+{
+	switch(qp_type) {
+	case IB_QPT_UC:
+		req_msg->offset40 = cpu_to_be32((be32_to_cpu(
+						  req_msg->offset40) &
+						   0xFFFFFFF9) | 0x2);
+	default: 
+		req_msg->offset40 = cpu_to_be32(be32_to_cpu(
+						 req_msg->offset40) &
+						  0xFFFFFFF9);
+	}
+}
+
+static inline u8 cm_req_get_flow_ctrl(struct cm_req_msg *req_msg)
+{
+	return be32_to_cpu(req_msg->offset40) & 0x1;
+}
+
+static inline void cm_req_set_flow_ctrl(struct cm_req_msg *req_msg,
+					u8 flow_ctrl)
+{
+	req_msg->offset40 = cpu_to_be32((flow_ctrl & 0x1) |
+					 (be32_to_cpu(req_msg->offset40) &
+					  0xFFFFFFFE));
+}
+
+static inline u32 cm_req_get_starting_psn(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32(be32_to_cpu(req_msg->offset44) >> 8);
+}
+
+static inline void cm_req_set_starting_psn(struct cm_req_msg *req_msg,
+					   u32 starting_psn)
+{
+	req_msg->offset44 = cpu_to_be32((be32_to_cpu(starting_psn) << 8) |
+			    (be32_to_cpu(req_msg->offset44) & 0x000000FF));
+}
+
+static inline u8 cm_req_get_local_resp_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) ((be32_to_cpu(req_msg->offset44) & 0xF8) >> 3);
+}
+
+static inline void cm_req_set_local_resp_timeout(struct cm_req_msg *req_msg,
+						 u8 resp_timeout)
+{
+	req_msg->offset44 = cpu_to_be32((resp_timeout << 3) |
+			    (be32_to_cpu(req_msg->offset44) & 0xFFFFFF07));
+}
+
+static inline u8 cm_req_get_retry_count(struct cm_req_msg *req_msg)
+{
+	return (u8) (be32_to_cpu(req_msg->offset44) & 0x7);
+}
+
+static inline void cm_req_set_retry_count(struct cm_req_msg *req_msg,
+					  u8 retry_count)
+{
+	req_msg->offset44 = cpu_to_be32((retry_count & 0x7) |
+			    (be32_to_cpu(req_msg->offset44) & 0xFFFFFFF8));
+}
+
+static inline u8 cm_req_get_path_mtu(struct cm_req_msg *req_msg)
+{
+	return req_msg->offset50 >> 4;
+}
+
+static inline void cm_req_set_path_mtu(struct cm_req_msg *req_msg, u8 path_mtu)
+{
+	req_msg->offset50 = (u8) ((req_msg->offset50 & 0xF) | (path_mtu << 4));
+}
+
+static inline u8 cm_req_get_rnr_retry_count(struct cm_req_msg *req_msg)
+{
+	return req_msg->offset50 & 0x7;
+}
+
+static inline void cm_req_set_rnr_retry_count(struct cm_req_msg *req_msg,
+					      u8 rnr_retry_count)
+{
+	req_msg->offset50 = (u8) ((req_msg->offset50 & 0xF8) |
+				  (rnr_retry_count & 0x7));
+}
+
+static inline u8 cm_req_get_max_cm_retries(struct cm_req_msg *req_msg)
+{
+	return req_msg->offset51 >> 4;
+}
+
+static inline void cm_req_set_max_cm_retries(struct cm_req_msg *req_msg,
+					     u8 retries)
+{
+	req_msg->offset51 = (u8) ((req_msg->offset51 & 0xF) | (retries << 4));
+}
+
+static inline u8 cm_req_get_srq(struct cm_req_msg *req_msg)
+{
+	return (req_msg->offset51 & 0x8) >> 3;
+}
+
+static inline void cm_req_set_srq(struct cm_req_msg *req_msg, u8 srq)
+{
+	req_msg->offset51 = (u8) ((req_msg->offset51 & 0xF7) |
+				  ((srq & 0x1) << 3));
+}
+
+static inline u32 cm_req_get_primary_flow_label(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32((be32_to_cpu(req_msg->primary_offset88) >> 12));
+}
+
+static inline void cm_req_set_primary_flow_label(struct cm_req_msg *req_msg,
+						 u32 flow_label)
+{
+	req_msg->primary_offset88 = cpu_to_be32(
+				    (be32_to_cpu(req_msg->primary_offset88) &
+				     0x00000FFF) |
+				     (be32_to_cpu(flow_label) << 12));
+}
+
+static inline u8 cm_req_get_primary_packet_rate(struct cm_req_msg *req_msg)
+{
+	return (u8) (be32_to_cpu(req_msg->primary_offset88) & 0x3F);
+}
+
+static inline void cm_req_set_primary_packet_rate(struct cm_req_msg *req_msg,
+						  u8 rate)
+{
+	req_msg->primary_offset88 = cpu_to_be32(
+				    (be32_to_cpu(req_msg->primary_offset88) &
+				     0xFFFFFFC0) | (rate & 0x3F));
+}
+
+static inline u8 cm_req_get_primary_sl(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->primary_offset94 >> 4);
+}
+
+static inline void cm_req_set_primary_sl(struct cm_req_msg *req_msg, u8 sl)
+{
+	req_msg->primary_offset94 = (u8) ((req_msg->primary_offset94 & 0x0F) |
+					  (sl << 4));
+}
+
+static inline u8 cm_req_get_primary_subnet_local(struct cm_req_msg *req_msg)
+{
+	return (u8) ((req_msg->primary_offset94 & 0x08) >> 3);
+}
+
+static inline void cm_req_set_primary_subnet_local(struct cm_req_msg *req_msg,
+						   u8 subnet_local)
+{
+	req_msg->primary_offset94 = (u8) ((req_msg->primary_offset94 & 0xF7) |
+					  ((subnet_local & 0x1) << 3));
+}
+
+static inline u8 cm_req_get_primary_local_ack_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->primary_offset95 >> 3);
+}
+
+static inline void cm_req_set_primary_local_ack_timeout(struct cm_req_msg *req_msg,
+							u8 local_ack_timeout)
+{
+	req_msg->primary_offset95 = (u8) ((req_msg->primary_offset95 & 0x07) |
+					  (local_ack_timeout << 3));
+}
+
+static inline u32 cm_req_get_alt_flow_label(struct cm_req_msg *req_msg)
+{
+	return cpu_to_be32((be32_to_cpu(req_msg->alt_offset132) >> 12));
+}
+
+static inline void cm_req_set_alt_flow_label(struct cm_req_msg *req_msg,
+					     u32 flow_label)
+{
+	req_msg->alt_offset132 = cpu_to_be32(
+				 (be32_to_cpu(req_msg->alt_offset132) &
+				  0x00000FFF) |
+				  (be32_to_cpu(flow_label) << 12));
+}
+
+static inline u8 cm_req_get_alt_packet_rate(struct cm_req_msg *req_msg)
+{
+	return (u8) (be32_to_cpu(req_msg->alt_offset132) & 0x3F);
+}
+
+static inline void cm_req_set_alt_packet_rate(struct cm_req_msg *req_msg,
+					      u8 rate)
+{
+	req_msg->alt_offset132 = cpu_to_be32(
+				 (be32_to_cpu(req_msg->alt_offset132) &
+				  0xFFFFFFC0) | (rate & 0x3F));
+}
+
+static inline u8 cm_req_get_alt_sl(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->alt_offset138 >> 4);
+}
+
+static inline void cm_req_set_alt_sl(struct cm_req_msg *req_msg, u8 sl)
+{
+	req_msg->alt_offset138 = (u8) ((req_msg->alt_offset138 & 0x0F) |
+				       (sl << 4));
+}
+
+static inline u8 cm_req_get_alt_subnet_local(struct cm_req_msg *req_msg)
+{
+	return (u8) ((req_msg->alt_offset138 & 0x08) >> 3);
+}
+
+static inline void cm_req_set_alt_subnet_local(struct cm_req_msg *req_msg,
+					       u8 subnet_local)
+{
+	req_msg->alt_offset138 = (u8) ((req_msg->alt_offset138 & 0xF7) |
+				       ((subnet_local & 0x1) << 3));
+}
+
+static inline u8 cm_req_get_alt_local_ack_timeout(struct cm_req_msg *req_msg)
+{
+	return (u8) (req_msg->alt_offset139 >> 3);
+}
+
+static inline void cm_req_set_alt_local_ack_timeout(struct cm_req_msg *req_msg,
+						    u8 local_ack_timeout)
+{
+	req_msg->alt_offset139 = (u8) ((req_msg->alt_offset139 & 0x07) |
+				       (local_ack_timeout << 3));
+}
+
+/* Message REJected or MRAed */
+enum cm_msg_response {
+	CM_MSG_RESPONSE_REQ = 0x0,
+	CM_MSG_RESPONSE_REP = 0x1,
+	CM_MSG_RESPONSE_OTHER = 0x2
+};
+
+ struct cm_mra_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	/* message MRAed:2, rsvd:6 */
+	u8 offset8;
+	/* service timeout:5, rsvd:3 */
+	u8 offset9;
+
+	u8 private_data[IB_CM_MRA_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u8 cm_mra_get_msg_mraed(struct cm_mra_msg *mra_msg)
+{
+	return (u8) (mra_msg->offset8 >> 6);
+}
+
+static inline void cm_mra_set_msg_mraed(struct cm_mra_msg *mra_msg, u8 msg)
+{
+	mra_msg->offset8 = (u8) ((mra_msg->offset8 & 0x3F) | (msg << 6));
+}
+
+static inline u8 cm_mra_get_service_timeout(struct cm_mra_msg *mra_msg)
+{
+	return (u8) (mra_msg->offset9 >> 3);
+}
+
+static inline void cm_mra_set_service_timeout(struct cm_mra_msg *mra_msg,
+					      u8 service_timeout)
+{
+	mra_msg->offset9 = (u8) ((mra_msg->offset9 & 0x07) |
+				 (service_timeout << 3));
+}
+
+struct cm_rej_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	/* message REJected:2, rsvd:6 */
+	u8 offset8;
+	/* reject info length:7, rsvd:1. */
+	u8 offset9;
+	u16 reason;
+	u8 ari[IB_CM_REJ_ARI_LENGTH];
+
+	u8 private_data[IB_CM_REJ_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u8 cm_rej_get_msg_rejected(struct cm_rej_msg *rej_msg)
+{
+	return (u8) (rej_msg->offset8 >> 6);
+}
+
+static inline void cm_rej_set_msg_rejected(struct cm_rej_msg *rej_msg, u8 msg)
+{
+	rej_msg->offset8 = (u8) ((rej_msg->offset8 & 0x3F) | (msg << 6));
+}
+
+static inline u8 cm_rej_get_reject_info_len(struct cm_rej_msg *rej_msg)
+{
+	return (u8) (rej_msg->offset9 >> 1);
+}
+
+static inline void cm_rej_set_reject_info_len(struct cm_rej_msg *rej_msg,
+					      u8 len)
+{
+	rej_msg->offset9 = (u8) ((rej_msg->offset9 & 0x1) | (len << 1));
+}
+
+struct cm_rep_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	u32 local_qkey;
+	/* local QPN:24, rsvd:8 */
+	u32 offset12;
+	/* local EECN:24, rsvd:8 */
+	u32 offset16;
+	/* starting PSN:24 rsvd:8 */
+	u32 offset20;
+	u8 resp_resources;
+	u8 initiator_depth;
+	/* target ACK delay:5, failover accepted:2, end-to-end flow control:1 */
+	u8 offset26;
+	/* RNR retry count:3, SRQ:1, rsvd:5 */
+	u8 offset27;
+	u64 local_ca_guid;
+
+	u8 private_data[IB_CM_REP_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u32 cm_rep_get_local_qpn(struct cm_rep_msg *rep_msg)
+{
+	return cpu_to_be32(be32_to_cpu(rep_msg->offset12) >> 8);
+}
+
+static inline void cm_rep_set_local_qpn(struct cm_rep_msg *rep_msg, u32 qpn)
+{
+	rep_msg->offset12 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+			    (be32_to_cpu(rep_msg->offset12) & 0x000000FF));
+}
+
+static inline u32 cm_rep_get_starting_psn(struct cm_rep_msg *rep_msg)
+{
+	return cpu_to_be32(be32_to_cpu(rep_msg->offset20) >> 8);
+}
+
+static inline void cm_rep_set_starting_psn(struct cm_rep_msg *rep_msg,
+					   u32 starting_psn)
+{
+	rep_msg->offset20 = cpu_to_be32((be32_to_cpu(starting_psn) << 8) |
+			    (be32_to_cpu(rep_msg->offset20) & 0x000000FF));
+}
+
+static inline u8 cm_rep_get_target_ack_delay(struct cm_rep_msg *rep_msg)
+{
+	return (u8) (rep_msg->offset26 >> 3);
+}
+
+static inline void cm_rep_set_target_ack_delay(struct cm_rep_msg *rep_msg,
+					       u8 target_ack_delay)
+{
+	rep_msg->offset26 = (u8) ((rep_msg->offset26 & 0x07) |
+				  (target_ack_delay << 3));
+}
+
+static inline u8 cm_rep_get_failover(struct cm_rep_msg *rep_msg)
+{
+	return (u8) ((rep_msg->offset26 & 0x06) >> 1);
+}
+
+static inline void cm_rep_set_failover(struct cm_rep_msg *rep_msg, u8 failover)
+{
+	rep_msg->offset26 = (u8) ((rep_msg->offset26 & 0xF9) |
+				  ((failover & 0x3) << 1));
+}
+
+static inline u8 cm_rep_get_flow_ctrl(struct cm_rep_msg *rep_msg)
+{
+	return (u8) (rep_msg->offset26 & 0x01);
+}
+
+static inline void cm_rep_set_flow_ctrl(struct cm_rep_msg *rep_msg,
+					    u8 flow_ctrl)
+{
+	rep_msg->offset26 = (u8) ((rep_msg->offset26 & 0xFE) |
+				  (flow_ctrl & 0x1));
+}
+
+static inline u8 cm_rep_get_rnr_retry_count(struct cm_rep_msg *rep_msg)
+{
+	return (u8) (rep_msg->offset27 >> 5);
+}
+
+static inline void cm_rep_set_rnr_retry_count(struct cm_rep_msg *rep_msg,
+					      u8 rnr_retry_count)
+{
+	rep_msg->offset27 = (u8) ((rep_msg->offset27 & 0x1F) |
+				  (rnr_retry_count << 5));
+}
+
+static inline u8 cm_rep_get_srq(struct cm_rep_msg *rep_msg)
+{
+	return (u8) ((rep_msg->offset27 >> 4) & 0x1);
+}
+
+static inline void cm_rep_set_srq(struct cm_rep_msg *rep_msg, u8 srq)
+{
+	rep_msg->offset27 = (u8) ((rep_msg->offset27 & 0xEF) |
+				  ((srq & 0x1) << 4));
+}
+
+struct cm_rtu_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u8 private_data[IB_CM_RTU_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+struct cm_dreq_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+	/* remote QPN/EECN:24, rsvd:8 */
+	u32 offset8;
+
+	u8 private_data[IB_CM_DREQ_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+static inline u32 cm_dreq_get_remote_qpn(struct cm_dreq_msg *dreq_msg)
+{
+	return cpu_to_be32(be32_to_cpu(dreq_msg->offset8) >> 8);
+}
+
+static inline void cm_dreq_set_remote_qpn(struct cm_dreq_msg *dreq_msg, u32 qpn)
+{
+	dreq_msg->offset8 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+			    (be32_to_cpu(dreq_msg->offset8) & 0x000000FF));
+}
+
+struct cm_drep_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u8 private_data[IB_CM_DREP_PRIVATE_DATA_SIZE];
+
+} __attribute__ ((packed));
+
+struct cm_lap_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u32 rsvd8;
+	/* remote QPN/EECN:24, remote CM response timeout:5, rsvd:3 */
+	u32 offset12;
+	u32 rsvd16;
+
+	u16 alt_local_lid;
+	u16 alt_remote_lid;
+	union ib_gid alt_local_gid;
+	union ib_gid alt_remote_gid;
+	/* flow label:20, rsvd:4, traffic class:8 */
+	u32 offset56;
+	u8 alt_hop_limit;
+	/* rsvd:2, packet rate:6 */
+	uint8_t offset61;
+	/* SL:4, subnet local:1, rsvd:3 */
+	uint8_t offset62;
+	/* local ACK timeout:5, rsvd:3 */
+	uint8_t offset63;
+
+	u8 private_data[IB_CM_LAP_PRIVATE_DATA_SIZE];
+} __attribute__  ((packed));
+
+static inline u32 cm_lap_get_remote_qpn(struct cm_lap_msg *lap_msg)
+{
+	return cpu_to_be32(be32_to_cpu(lap_msg->offset12) >> 8);
+}
+
+static inline void cm_lap_set_remote_qpn(struct cm_lap_msg *lap_msg, u32 qpn)
+{
+	lap_msg->offset12 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+					 (be32_to_cpu(lap_msg->offset12) &
+					  0x000000FF));
+}
+
+static inline u8 cm_lap_get_remote_resp_timeout(struct cm_lap_msg *lap_msg)
+{
+	return (u8) ((be32_to_cpu(lap_msg->offset12) & 0xF8) >> 3);
+}
+
+static inline void cm_lap_set_remote_resp_timeout(struct cm_lap_msg *lap_msg,
+						  u8 resp_timeout)
+{
+	lap_msg->offset12 = cpu_to_be32((resp_timeout << 3) |
+					 (be32_to_cpu(lap_msg->offset12) &
+					  0xFFFFFF07));
+}
+
+static inline u32 cm_lap_get_flow_label(struct cm_lap_msg *lap_msg)
+{
+	return be32_to_cpu(lap_msg->offset56) >> 12;
+}
+
+static inline void cm_lap_set_flow_label(struct cm_lap_msg *lap_msg,
+					 u32 flow_label)
+{
+	lap_msg->offset56 = cpu_to_be32((flow_label << 12) |
+					 (be32_to_cpu(lap_msg->offset56) &
+					  0x00000FFF));
+}
+
+static inline u8 cm_lap_get_traffic_class(struct cm_lap_msg *lap_msg)
+{
+	return (u8) be32_to_cpu(lap_msg->offset56);
+}
+
+static inline void cm_lap_set_traffic_class(struct cm_lap_msg *lap_msg,
+					    u8 traffic_class)
+{
+	lap_msg->offset56 = cpu_to_be32(traffic_class |
+					 (be32_to_cpu(lap_msg->offset56) &
+					  0xFFFFFF00));
+}
+
+static inline u8 cm_lap_get_packet_rate(struct cm_lap_msg *lap_msg)
+{
+	return lap_msg->offset61 & 0x3F;
+}
+
+static inline void cm_lap_set_packet_rate(struct cm_lap_msg *lap_msg,
+					  u8 packet_rate)
+{
+	lap_msg->offset61 = (packet_rate & 0x3F) | (lap_msg->offset61 & 0xC0);
+}
+
+static inline u8 cm_lap_get_sl(struct cm_lap_msg *lap_msg)
+{
+	return lap_msg->offset62 >> 4;
+}
+
+static inline void cm_lap_set_sl(struct cm_lap_msg *lap_msg, u8 sl)
+{
+	lap_msg->offset62 = (sl << 4) | (lap_msg->offset62 & 0x0F);
+}
+
+static inline u8 cm_lap_get_subnet_local(struct cm_lap_msg *lap_msg)
+{
+	return (lap_msg->offset62 >> 3) & 0x1;
+}
+
+static inline void cm_lap_set_subnet_local(struct cm_lap_msg *lap_msg,
+					   u8 subnet_local)
+{
+	lap_msg->offset62 = ((subnet_local & 0x1) << 3) |
+			     (lap_msg->offset61 & 0xF7);
+}
+static inline u8 cm_lap_get_local_ack_timeout(struct cm_lap_msg *lap_msg)
+{
+	return lap_msg->offset63 >> 3;
+}
+
+static inline void cm_lap_set_local_ack_timeout(struct cm_lap_msg *lap_msg,
+						u8 local_ack_timeout)
+{
+	lap_msg->offset63 = (local_ack_timeout << 3) |
+			    (lap_msg->offset63 & 0x07);
+}
+
+struct cm_apr_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 local_comm_id;
+	u32 remote_comm_id;
+
+	u8 info_length;
+	u8 ap_status;
+	u8 info[IB_CM_APR_INFO_LENGTH];
+
+	u8 private_data[IB_CM_APR_PRIVATE_DATA_SIZE];
+} __attribute__ ((packed));
+
+struct cm_sidr_req_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 request_id;
+	u16 pkey;
+	u16 rsvd;
+	u64 service_id;
+
+	u8 private_data[IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE];
+} __attribute__ ((packed));
+
+struct cm_sidr_rep_msg {
+	struct ib_mad_hdr hdr;
+
+	u32 request_id;
+	u8 status;
+	u8 info_length;
+	u16 rsvd;
+	/* QPN:24, rsvd:8 */
+	u32 offset8;
+	u64 service_id;
+	u32 qkey;
+	u8 info[IB_CM_SIDR_REP_INFO_LENGTH];
+
+	u8 private_data[IB_CM_SIDR_REP_PRIVATE_DATA_SIZE];
+} __attribute__ ((packed));
+
+static inline u32 cm_sidr_rep_get_qpn(struct cm_sidr_rep_msg *sidr_rep_msg)
+{
+	return cpu_to_be32(be32_to_cpu(sidr_rep_msg->offset8) >> 8);
+}
+
+static inline void cm_sidr_rep_set_qpn(struct cm_sidr_rep_msg *sidr_rep_msg,
+				       u32 qpn)
+{
+	sidr_rep_msg->offset8 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
+					(be32_to_cpu(sidr_rep_msg->offset8) &
+					 0x000000FF));
+}
+
+#endif /* CM_MSGS_H */
diff -uprN linux-2.6.13-rc2-mm1-21/drivers/infiniband/core/Makefile linux-2.6.13-rc2-mm1-22/drivers/infiniband/core/Makefile
-- linux-2.6.13-rc2-mm1-21/drivers/infiniband/core/Makefile	2005-07-10 16:32:15.000000000 -0400
+++ linux-2.6.13-rc2-mm1-22/drivers/infiniband/core/Makefile	2005-07-10 16:50:04.000000000 -0400
@@ -1,6 +1,7 @@
 EXTRA_CFLAGS += -Idrivers/infiniband/include
 
-obj-$(CONFIG_INFINIBAND) +=		ib_core.o ib_mad.o ib_sa.o ib_umad.o
+obj-$(CONFIG_INFINIBAND) +=		ib_core.o ib_mad.o ib_sa.o \
+					ib_cm.o ib_umad.o
 obj-$(CONFIG_INFINIBAND_USER_VERBS) +=	ib_uverbs.o
 
 ib_core-y :=			packer.o ud_header.o verbs.o sysfs.o \
@@ -10,6 +11,8 @@ ib_mad-y :=			mad.o smi.o agent.o mad_rm
 
 ib_sa-y :=			sa_query.o
 
+ib_cm-y :=			cm.o
+
 ib_umad-y :=			user_mad.o
 
 ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_mem.o




From halr at voltaire.com  Mon Jul 11 13:50:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:50:00 -0400
Subject: [openib-general] [PATCH 23/29v2] User MAD ABI changes to support
	RMPP
Message-ID: <1121110406.4389.5030.camel@hal.voltaire.com>

User MAD ABI changes to support RMPP

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 22/29.

--
 Documentation/infiniband/user_mad.txt    |   51 +++++++++++++++++++++++--
 drivers/infiniband/include/ib_user_mad.h |   28 +++++++++++++--
 2 files changed, 61 insertions(+), 18 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-22/Documentation/infiniband/user_mad.txt linux-2.6.13-rc2-mm1-23/Documentation/infiniband/user_mad.txt
-- linux-2.6.13-rc2-mm1-22/Documentation/infiniband/user_mad.txt	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-23/Documentation/infiniband/user_mad.txt	2005-07-01 14:27:50.000000000 -0400
@@ -28,13 +28,37 @@ Creating MAD agents
 
 Receiving MADs
 
-  MADs are received using read().  The buffer passed to read() must be
-  large enough to hold at least one struct ib_user_mad.  For example:
+  MADs are received using read().  The receive side now supports
+  RMPP. The buffer passed to read() must be at least one 
+  struct ib_user_mad + 256 bytes. For example:
+
+  If the buffer passed is not large enough to hold the received
+  MAD (RMPP), the errno is set to ENOSPC and the length of the
+  buffer needed is set in mad.length.
+
+  Example for normal MAD (non RMPP) reads:
+	struct ib_user_mad *mad;
+	mad = malloc(sizeof *mad + 256);
+	ret = read(fd, mad, sizeof *mad + 256);
+	if (ret != sizeof mad + 256) {
+		perror("read");
+		free(mad);
+	}
 
-	struct ib_user_mad mad;
-	ret = read(fd, &mad, sizeof mad);
-	if (ret != sizeof mad)
+  Example for RMPP reads:
+	struct ib_user_mad *mad;
+	mad = malloc(sizeof *mad + 256);
+	ret = read(fd, mad, sizeof *mad + 256);
+	if (ret == -ENOSPC)) {
+		length = mad.length;
+		free(mad);
+		mad = malloc(sizeof *mad + length);
+		ret = read(fd, mad, sizeof *mad + length);
+	}
+	if (ret < 0) {
 		perror("read");
+		free(mad);
+	}
 
   In addition to the actual MAD contents, the other struct ib_user_mad
   fields will be filled in with information on the received MAD.  For
@@ -50,18 +74,21 @@ Sending MADs
 
   MADs are sent using write().  The agent ID for sending should be
   filled into the id field of the MAD, the destination LID should be
-  filled into the lid field, and so on.  For example:
+  filled into the lid field, and so on.  The send side does support
+  RMPP so arbitrary length MAD can be sent. For example:
+
+	struct ib_user_mad *mad;
 
-	struct ib_user_mad mad;
+	mad = malloc(sizeof *mad + mad_length);
 
-	/* fill in mad.data */
+	/* fill in mad->data */
 
-	mad.id  = my_agent;	/* req.id from agent registration */
-	mad.lid = my_dest;	/* in network byte order... */
+	mad->hdr.id  = my_agent;	/* req.id from agent registration */
+	mad->hdr.lid = my_dest;		/* in network byte order... */
 	/* etc. */
 
-	ret = write(fd, &mad, sizeof mad);
-	if (ret != sizeof mad)
+	ret = write(fd, &mad, sizeof *mad + mad_length);
+	if (ret != sizeof *mad + mad_length)
 		perror("write");
 
 Setting IsSM Capability Bit
diff -uprN linux-2.6.13-rc2-mm1-22/drivers/infiniband/include/ib_user_mad.h linux-2.6.13-rc2-mm1-23/drivers/infiniband/include/ib_user_mad.h
-- linux-2.6.13-rc2-mm1-22/drivers/infiniband/include/ib_user_mad.h	2005-06-29 19:00:53.000000000 -0400
+++ linux-2.6.13-rc2-mm1-23/drivers/infiniband/include/ib_user_mad.h	2005-07-10 16:55:43.000000000 -0400
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +30,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: ib_user_mad.h 1389 2004-12-27 22:56:47Z roland $
+ * $Id: ib_user_mad.h 2814 2005-07-06 19:14:09Z halr $
  */
 
 #ifndef IB_USER_MAD_H
@@ -42,7 +43,7 @@
  * Increment this value if any changes that break userspace ABI
  * compatibility are made.
  */
-#define IB_USER_MAD_ABI_VERSION	2
+#define IB_USER_MAD_ABI_VERSION	5
 
 /*
  * Make sure that all structs defined in this file remain laid out so
@@ -51,13 +52,13 @@
  */
 
 /**
- * ib_user_mad - MAD packet
- * @data - Contents of MAD
+ * ib_user_mad_hdr - MAD packet header
  * @id - ID of agent MAD received with/to be sent with
  * @status - 0 on successful receive, ETIMEDOUT if no response
  *   received (transaction ID in data[] will be set to TID of original
  *   request) (ignored on send)
  * @timeout_ms - Milliseconds to wait for response (unset on receive)
+ * @retries - Number of automatic retries to attempt
  * @qpn - Remote QP number received from/to be sent to
  * @qkey - Remote Q_Key to be sent with (unset on receive)
  * @lid - Remote lid received from/to be sent to
@@ -72,11 +73,12 @@
  *
  * All multi-byte quantities are stored in network (big endian) byte order.
  */
-struct ib_user_mad {
-	__u8	data[256];
+struct ib_user_mad_hdr {
 	__u32	id;
 	__u32	status;
 	__u32	timeout_ms;
+	__u32	retries;
+	__u32	length;
 	__u32	qpn;
 	__u32   qkey;
 	__u16	lid;
@@ -91,6 +93,17 @@ struct ib_user_mad {
 };
 
 /**
+ * ib_user_mad - MAD packet
+ * @hdr - MAD packet header
+ * @data - Contents of MAD
+ *
+ */
+struct ib_user_mad {
+	struct ib_user_mad_hdr hdr;
+	__u8	data[0];
+};
+
+/**
  * ib_user_mad_reg_req - MAD registration request
  * @id - Set by the kernel; used to identify agent in future requests.
  * @qpn - Queue pair number; must be 0 or 1.
@@ -103,6 +116,8 @@ struct ib_user_mad {
  *   management class to receive.
  * @oui: Indicates IEEE OUI when mgmt_class is a vendor class
  *   in the range from 0x30 to 0x4f. Otherwise not used.
+ * @rmpp_version: If set, indicates the RMPP version used.
+ * 
  */
 struct ib_user_mad_reg_req {
 	__u32	id;
@@ -111,6 +126,7 @@ struct ib_user_mad_reg_req {
 	__u8	mgmt_class;
 	__u8	mgmt_class_version;
 	__u8    oui[3];
+	__u8	rmpp_version;
 };
 
 #define IB_IOCTL_MAGIC		0x1b




From halr at voltaire.com  Mon Jul 11 13:53:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:53:14 -0400
Subject: [openib-general] [PATCH 24/29v2] Implementation for RMPP support in
	user MAD
Message-ID: <1121110412.4389.5032.camel@hal.voltaire.com>

Implementation for RMPP support in user MAD

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 23/29.

--
 user_mad.c |  300 +++++++++++++++++++++++++++++++++++++++--
 1 files changed, 194 insertions(+), 106 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-23/drivers/infiniband/core/user_mad.c linux-2.6.13-rc2-mm1-24/drivers/infiniband/core/user_mad.c
-- linux-2.6.13-rc2-mm1-23/drivers/infiniband/core/user_mad.c	2005-07-09 17:14:46.000000000 -0400
+++ linux-2.6.13-rc2-mm1-24/drivers/infiniband/core/user_mad.c	2005-07-10 16:56:39.000000000 -0400
@@ -1,5 +1,7 @@
 /*
  * Copyright (c) 2004 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved. 
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -29,7 +31,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id: user_mad.c 1389 2004-12-27 22:56:47Z roland $
+ * $Id: user_mad.c 2814 2005-07-06 19:14:09Z halr $
  */
 
 #include <linux/module.h>
@@ -94,10 +96,12 @@ struct ib_umad_file {
 };
 
 struct ib_umad_packet {
-	struct ib_user_mad mad;
 	struct ib_ah      *ah;
+	struct ib_mad_send_buf *msg;
 	struct list_head   list;
+	int		   length;
 	DECLARE_PCI_UNMAP_ADDR(mapping)
+	struct ib_user_mad mad;
 };
 
 static const dev_t base_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE);
@@ -114,10 +118,10 @@ static int queue_packet(struct ib_umad_f
 	int ret = 1;
 
 	down_read(&file->agent_mutex);
-	for (packet->mad.id = 0;
-	     packet->mad.id < IB_UMAD_MAX_AGENTS;
-	     packet->mad.id++)
-		if (agent == file->agent[packet->mad.id]) {
+	for (packet->mad.hdr.id = 0;
+	     packet->mad.hdr.id < IB_UMAD_MAX_AGENTS;
+	     packet->mad.hdr.id++)
+		if (agent == file->agent[packet->mad.hdr.id]) {
 			spin_lock_irq(&file->recv_lock);
 			list_add_tail(&packet->list, &file->recv_list);
 			spin_unlock_irq(&file->recv_lock);
@@ -135,22 +139,30 @@ static void send_handler(struct ib_mad_a
 			 struct ib_mad_send_wc *send_wc)
 {
 	struct ib_umad_file *file = agent->context;
-	struct ib_umad_packet *packet =
+	struct ib_umad_packet *timeout, *packet =
 		(void *) (unsigned long) send_wc->wr_id;
 
-	dma_unmap_single(agent->device->dma_device,
-			 pci_unmap_addr(packet, mapping),
-			 sizeof packet->mad.data,
-			 DMA_TO_DEVICE);
-	ib_destroy_ah(packet->ah);
+	ib_destroy_ah(packet->msg->send_wr.wr.ud.ah);
+	ib_free_send_mad(packet->msg);
 
 	if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) {
-		packet->mad.status = ETIMEDOUT;
+		timeout = kmalloc(sizeof *timeout + sizeof (struct ib_mad_hdr),
+				  GFP_KERNEL);
+		if (!timeout)
+			goto out;
+
+		memset(timeout, 0, sizeof *timeout + sizeof (struct ib_mad_hdr));
+
+		timeout->length = sizeof (struct ib_mad_hdr);
+		timeout->mad.hdr.id = packet->mad.hdr.id;
+		timeout->mad.hdr.status = ETIMEDOUT;
+		memcpy(timeout->mad.data, packet->mad.data,
+		       sizeof (struct ib_mad_hdr));
 
-		if (!queue_packet(file, agent, packet))
-			return;
+		if (!queue_packet(file, agent, timeout))
+				return;
 	}
-
+out:
 	kfree(packet);
 }
 
@@ -159,30 +171,35 @@ static void recv_handler(struct ib_mad_a
 {
 	struct ib_umad_file *file = agent->context;
 	struct ib_umad_packet *packet;
+	int length;
 
 	if (mad_recv_wc->wc->status != IB_WC_SUCCESS)
 		goto out;
 
-	packet = kmalloc(sizeof *packet, GFP_KERNEL);
+	length = mad_recv_wc->mad_len; 
+	packet = kmalloc(sizeof *packet + length, GFP_KERNEL);
 	if (!packet)
 		goto out;
 
-	memset(packet, 0, sizeof *packet);
+	memset(packet, 0, sizeof *packet + length);
+	packet->length = length;
+
+	ib_coalesce_recv_mad(mad_recv_wc, packet->mad.data);
 
-	memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, sizeof packet->mad.data);
-	packet->mad.status        = 0;
-	packet->mad.qpn 	  = cpu_to_be32(mad_recv_wc->wc->src_qp);
-	packet->mad.lid 	  = cpu_to_be16(mad_recv_wc->wc->slid);
-	packet->mad.sl  	  = mad_recv_wc->wc->sl;
-	packet->mad.path_bits 	  = mad_recv_wc->wc->dlid_path_bits;
-	packet->mad.grh_present   = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH);
-	if (packet->mad.grh_present) {
+	packet->mad.hdr.status    = 0;
+	packet->mad.hdr.length    = length + sizeof (struct ib_user_mad);
+	packet->mad.hdr.qpn 	  = cpu_to_be32(mad_recv_wc->wc->src_qp);
+	packet->mad.hdr.lid 	  = cpu_to_be16(mad_recv_wc->wc->slid);
+	packet->mad.hdr.sl  	  = mad_recv_wc->wc->sl;
+	packet->mad.hdr.path_bits = mad_recv_wc->wc->dlid_path_bits;
+	packet->mad.hdr.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH);
+	if (packet->mad.hdr.grh_present) {
 		/* XXX parse GRH */
-		packet->mad.gid_index 	  = 0;
-		packet->mad.hop_limit 	  = 0;
-		packet->mad.traffic_class = 0;
-		memset(packet->mad.gid, 0, 16);
-		packet->mad.flow_label 	  = 0;
+		packet->mad.hdr.gid_index 	= 0;
+		packet->mad.hdr.hop_limit 	= 0;
+		packet->mad.hdr.traffic_class	= 0;
+		memset(packet->mad.hdr.gid, 0, 16);
+		packet->mad.hdr.flow_label	= 0;
 	}
 
 	if (queue_packet(file, agent, packet))
@@ -199,7 +216,7 @@ static ssize_t ib_umad_read(struct file 
 	struct ib_umad_packet *packet;
 	ssize_t ret;
 
-	if (count < sizeof (struct ib_user_mad))
+	if (count < sizeof (struct ib_user_mad) + sizeof (struct ib_mad))
 		return -EINVAL;
 
 	spin_lock_irq(&file->recv_lock);
@@ -222,12 +239,25 @@ static ssize_t ib_umad_read(struct file 
 
 	spin_unlock_irq(&file->recv_lock);
 
-	if (copy_to_user(buf, &packet->mad, sizeof packet->mad))
+	if (count < packet->length + sizeof (struct ib_user_mad)) {
+		/* Return length needed (and first RMPP segment) if too small */
+		if (copy_to_user(buf, &packet->mad,
+				 sizeof (struct ib_user_mad) + sizeof (struct ib_mad)))
+			ret = -EFAULT;
+		else
+			ret = -ENOSPC;
+	} else if (copy_to_user(buf, &packet->mad,
+			      packet->length + sizeof (struct ib_user_mad)))
 		ret = -EFAULT;
 	else
-		ret = sizeof packet->mad;
-
-	kfree(packet);
+		ret = packet->length + sizeof (struct ib_user_mad);
+	if (ret < 0) {
+		/* Requeue packet */
+		spin_lock_irq(&file->recv_lock);
+		list_add(&packet->list, &file->recv_list);
+		spin_unlock_irq(&file->recv_lock);
+	} else
+		kfree(packet);
 	return ret;
 }
 
@@ -238,106 +268,163 @@ static ssize_t ib_umad_write(struct file
 	struct ib_umad_packet *packet;
 	struct ib_mad_agent *agent;
 	struct ib_ah_attr ah_attr;
-	struct ib_sge      gather_list;
-	struct ib_send_wr *bad_wr, wr = {
-		.opcode      = IB_WR_SEND,
-		.sg_list     = &gather_list,
-		.num_sge     = 1,
-		.send_flags  = IB_SEND_SIGNALED,
-	};
+	struct ib_send_wr *bad_wr;
+	struct ib_rmpp_mad *rmpp_mad;
 	u8 method;
 	u64 *tid;
-	int ret;
+	int ret, length, hdr_len, data_len, rmpp_hdr_size;
+	int rmpp_active = 0;
 
 	if (count < sizeof (struct ib_user_mad))
 		return -EINVAL;
 
-	packet = kmalloc(sizeof *packet, GFP_KERNEL);
+	length = count - sizeof (struct ib_user_mad);
+	packet = kmalloc(sizeof *packet + sizeof(struct ib_mad_hdr) +
+			 sizeof(struct ib_rmpp_hdr), GFP_KERNEL);
 	if (!packet)
 		return -ENOMEM;
 
-	if (copy_from_user(&packet->mad, buf, sizeof packet->mad)) {
-		kfree(packet);
-		return -EFAULT;
+	if (copy_from_user(&packet->mad, buf,
+			    sizeof (struct ib_user_mad) +
+			    sizeof(struct ib_mad_hdr) +
+			    sizeof(struct ib_rmpp_hdr))) {
+		ret = -EFAULT;
+		goto err;
 	}
 
-	if (packet->mad.id < 0 || packet->mad.id >= IB_UMAD_MAX_AGENTS) {
+	if (packet->mad.hdr.id < 0 ||
+	    packet->mad.hdr.id >= IB_UMAD_MAX_AGENTS) {
 		ret = -EINVAL;
 		goto err;
 	}
 
+	packet->length = length;
+
 	down_read(&file->agent_mutex);
 
-	agent = file->agent[packet->mad.id];
+	agent = file->agent[packet->mad.hdr.id];
 	if (!agent) {
 		ret = -EINVAL;
 		goto err_up;
 	}
 
+	memset(&ah_attr, 0, sizeof ah_attr);
+	ah_attr.dlid          = be16_to_cpu(packet->mad.hdr.lid);
+	ah_attr.sl            = packet->mad.hdr.sl;
+	ah_attr.src_path_bits = packet->mad.hdr.path_bits;
+	ah_attr.port_num      = file->port->port_num;
+	if (packet->mad.hdr.grh_present) {
+		ah_attr.ah_flags = IB_AH_GRH;
+		memcpy(ah_attr.grh.dgid.raw, packet->mad.hdr.gid, 16);
+		ah_attr.grh.flow_label 	   = packet->mad.hdr.flow_label;
+		ah_attr.grh.hop_limit  	   = packet->mad.hdr.hop_limit;
+		ah_attr.grh.traffic_class  = packet->mad.hdr.traffic_class;
+	}
+
+	packet->ah = ib_create_ah(agent->qp->pd, &ah_attr);
+	if (IS_ERR(packet->ah)) {
+		ret = PTR_ERR(packet->ah);
+		goto err_up;
+	}
+
+	rmpp_mad = (struct ib_rmpp_mad *) packet->mad.data;
+	if (ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_ACTIVE) {
+		/* RMPP active */
+		if (!agent->rmpp_version) {
+			ret = -EINVAL;
+			goto err_ah;
+		}
+		/* Validate that management class can support RMPP */
+		if (rmpp_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_ADM) {
+			hdr_len = offsetof(struct ib_sa_mad, data);
+			data_len = length;
+		} else if ((rmpp_mad->mad_hdr.mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) &&
+			    (rmpp_mad->mad_hdr.mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) {
+				hdr_len = offsetof(struct ib_vendor_mad, data);
+				data_len = length - hdr_len;
+		} else {
+			ret = -EINVAL;
+			goto err_ah;
+		}
+		rmpp_active = 1;
+	} else {
+		if (length > sizeof(struct ib_mad)) {
+			ret = -EINVAL;
+			goto err_ah;
+		}
+		hdr_len = offsetof(struct ib_mad, data);
+		data_len = length - hdr_len;
+	}
+	
+	packet->msg = ib_create_send_mad(agent,
+					 be32_to_cpu(packet->mad.hdr.qpn),
+					 0, packet->ah, rmpp_active,
+					 hdr_len, data_len,
+					 GFP_KERNEL);
+	if (IS_ERR(packet->msg)) {
+		ret = PTR_ERR(packet->msg);
+		goto err_ah;
+	}
+
+	packet->msg->send_wr.wr.ud.timeout_ms  = packet->mad.hdr.timeout_ms;
+	packet->msg->send_wr.wr.ud.retries = packet->mad.hdr.retries;
+
+	/* Override send WR WRID initialized in ib_create_send_mad */
+	packet->msg->send_wr.wr_id = (unsigned long) packet;
+
+	if (!rmpp_active) {
+		/* Copy message from user into send buffer */
+		if (copy_from_user(packet->msg->mad,
+				   buf + sizeof(struct ib_user_mad), length)) {
+			ret = -EFAULT;
+			goto err_msg;
+		}
+	} else {
+		rmpp_hdr_size = sizeof(struct ib_mad_hdr) +
+				sizeof(struct ib_rmpp_hdr);
+
+		/* Only copy MAD headers (RMPP header in place) */
+		memcpy(packet->msg->mad, packet->mad.data,
+		       sizeof(struct ib_mad_hdr));
+
+		/* Now, copy rest of message from user into send buffer */
+		if (copy_from_user(((struct ib_rmpp_mad *) packet->msg->mad)->data,
+				   buf + sizeof (struct ib_user_mad) + rmpp_hdr_size,
+				   length - rmpp_hdr_size)) {
+			ret = -EFAULT;
+			goto err_msg;
+		}
+	}
+
 	/*
 	 * If userspace is generating a request that will generate a
 	 * response, we need to make sure the high-order part of the
 	 * transaction ID matches the agent being used to send the
 	 * MAD.
 	 */
-	method = ((struct ib_mad_hdr *) packet->mad.data)->method;
+	method = packet->msg->mad->mad_hdr.method;
 
 	if (!(method & IB_MGMT_METHOD_RESP)       &&
 	    method != IB_MGMT_METHOD_TRAP_REPRESS &&
 	    method != IB_MGMT_METHOD_SEND) {
-		tid = &((struct ib_mad_hdr *) packet->mad.data)->tid;
+		tid = &packet->msg->mad->mad_hdr.tid;
 		*tid = cpu_to_be64(((u64) agent->hi_tid) << 32 |
 				   (be64_to_cpup(tid) & 0xffffffff));
 	}
 
-	memset(&ah_attr, 0, sizeof ah_attr);
-	ah_attr.dlid          = be16_to_cpu(packet->mad.lid);
-	ah_attr.sl            = packet->mad.sl;
-	ah_attr.src_path_bits = packet->mad.path_bits;
-	ah_attr.port_num      = file->port->port_num;
-	if (packet->mad.grh_present) {
-		ah_attr.ah_flags = IB_AH_GRH;
-		memcpy(ah_attr.grh.dgid.raw, packet->mad.gid, 16);
-		ah_attr.grh.flow_label 	   = packet->mad.flow_label;
-		ah_attr.grh.hop_limit  	   = packet->mad.hop_limit;
-		ah_attr.grh.traffic_class  = packet->mad.traffic_class;
-	}
-
-	packet->ah = ib_create_ah(agent->qp->pd, &ah_attr);
-	if (IS_ERR(packet->ah)) {
-		ret = PTR_ERR(packet->ah);
-		goto err_up;
-	}
-
-	gather_list.addr = dma_map_single(agent->device->dma_device,
-					  packet->mad.data,
-					  sizeof packet->mad.data,
-					  DMA_TO_DEVICE);
-	gather_list.length = sizeof packet->mad.data;
-	gather_list.lkey   = file->mr[packet->mad.id]->lkey;
-	pci_unmap_addr_set(packet, mapping, gather_list.addr);
-
-	wr.wr.ud.mad_hdr     = (struct ib_mad_hdr *) packet->mad.data;
-	wr.wr.ud.ah          = packet->ah;
-	wr.wr.ud.remote_qpn  = be32_to_cpu(packet->mad.qpn);
-	wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey);
-	wr.wr.ud.timeout_ms  = packet->mad.timeout_ms;
-	wr.wr.ud.retries     = 0;
+	ret = ib_post_send_mad(agent, &packet->msg->send_wr, &bad_wr);
+	if (ret)
+		goto err_msg;
 
-	wr.wr_id            = (unsigned long) packet;
+	up_read(&file->agent_mutex);
 
-	ret = ib_post_send_mad(agent, &wr, &bad_wr);
-	if (ret) {
-		dma_unmap_single(agent->device->dma_device,
-				 pci_unmap_addr(packet, mapping),
-				 sizeof packet->mad.data,
-				 DMA_TO_DEVICE);
-		goto err_up;
-	}
+	return sizeof (struct ib_user_mad_hdr) + packet->length;
 
-	up_read(&file->agent_mutex);
+err_msg:
+	ib_free_send_mad(packet->msg);
 
-	return sizeof packet->mad;
+err_ah:
+	ib_destroy_ah(packet->ah);
 
 err_up:
 	up_read(&file->agent_mutex);
@@ -400,7 +487,8 @@ found:
 	agent = ib_register_mad_agent(file->port->ib_dev, file->port->port_num,
 				      ureq.qpn ? IB_QPT_GSI : IB_QPT_SMI,
 				      ureq.mgmt_class ? &req : NULL,
-				      0, send_handler, recv_handler, file);
+				      ureq.rmpp_version,
+				      send_handler, recv_handler, file);
 	if (IS_ERR(agent)) {
 		ret = PTR_ERR(agent);
 		goto out;
@@ -461,8 +549,8 @@ out:
 	return ret;
 }
 
-static long ib_umad_ioctl(struct file *filp,
-			 unsigned int cmd, unsigned long arg)
+static long ib_umad_ioctl(struct file *filp, unsigned int cmd,
+			  unsigned long arg)
 {
 	switch (cmd) {
 	case IB_USER_MAD_REGISTER_AGENT:
@@ -518,14 +606,14 @@ static int ib_umad_close(struct inode *i
 }
 
 static struct file_operations umad_fops = {
-	.owner 	        = THIS_MODULE,
-	.read 	        = ib_umad_read,
-	.write 	        = ib_umad_write,
-	.poll 	        = ib_umad_poll,
+	.owner 	 	= THIS_MODULE,
+	.read 	 	= ib_umad_read,
+	.write 	 	= ib_umad_write,
+	.poll 	 	= ib_umad_poll,
 	.unlocked_ioctl = ib_umad_ioctl,
-	.compat_ioctl   = ib_umad_ioctl,
-	.open 	        = ib_umad_open,
-	.release        = ib_umad_close
+	.compat_ioctl 	= ib_umad_ioctl,
+	.open 	 	= ib_umad_open,
+	.release 	= ib_umad_close
 };
 
 static int ib_umad_sm_open(struct inode *inode, struct file *filp)




From halr at voltaire.com  Mon Jul 11 13:56:27 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:56:27 -0400
Subject: [openib-general] [PATCH 25/29v2] Add the header file for user space
	CM
Message-ID: <1121110419.4389.5034.camel@hal.voltaire.com>

Add the header file for user space CM. This file defines 
the ABI used by the CM for kernel/user communication.

Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

--
 ib_user_cm.h |  328 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 328 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1-24/drivers/infiniband/include/ib_user_cm.h linux-2.6.13-rc2-mm1-25/drivers/infiniband/include/ib_user_cm.h
-- linux-2.6.13-rc2-mm1-24/drivers/infiniband/include/ib_user_cm.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-25/drivers/infiniband/include/ib_user_cm.h	2005-06-10 08:16:25.000000000 -0400
@@ -0,0 +1,328 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ib_user_cm.h 2576 2005-06-09 17:00:30Z libor $
+ */
+
+#ifndef IB_USER_CM_H
+#define IB_USER_CM_H
+
+#include <linux/types.h>
+
+#define IB_USER_CM_ABI_VERSION 1
+
+enum {
+	IB_USER_CM_CMD_CREATE_ID,
+	IB_USER_CM_CMD_DESTROY_ID,
+	IB_USER_CM_CMD_ATTR_ID,
+
+	IB_USER_CM_CMD_LISTEN,
+	IB_USER_CM_CMD_ESTABLISH,
+	
+	IB_USER_CM_CMD_SEND_REQ,
+	IB_USER_CM_CMD_SEND_REP,
+	IB_USER_CM_CMD_SEND_RTU,
+	IB_USER_CM_CMD_SEND_DREQ,
+	IB_USER_CM_CMD_SEND_DREP,
+	IB_USER_CM_CMD_SEND_REJ,
+	IB_USER_CM_CMD_SEND_MRA,
+	IB_USER_CM_CMD_SEND_LAP,
+	IB_USER_CM_CMD_SEND_APR,
+	IB_USER_CM_CMD_SEND_SIDR_REQ,
+	IB_USER_CM_CMD_SEND_SIDR_REP,
+
+	IB_USER_CM_CMD_EVENT,
+};
+/*
+ * command ABI structures.
+ */
+struct ib_ucm_cmd_hdr {
+	__u32 cmd;
+	__u16 in;
+	__u16 out;
+};
+
+struct ib_ucm_create_id {
+	__u64 response;
+};
+
+struct ib_ucm_create_id_resp {
+	__u32 id;
+};
+
+struct ib_ucm_destroy_id {
+	__u32 id;
+};
+
+struct ib_ucm_attr_id {
+	__u64 response;
+	__u32 id;
+};
+
+struct ib_ucm_attr_id_resp {
+	__u64 service_id;
+	__u64 service_mask;
+	__u32 local_id;
+	__u32 remote_id;
+};
+
+struct ib_ucm_listen {
+	__u64 service_id;
+	__u64 service_mask;
+	__u32 id;
+};
+
+struct ib_ucm_establish {
+	__u32 id;
+};
+
+struct ib_ucm_private_data {
+	__u64 data;
+	__u32 id;
+	__u8  len;
+	__u8  reserved[3];
+};
+
+struct ib_ucm_path_rec {
+	__u8  dgid[16];
+	__u8  sgid[16];
+	__u16 dlid;
+	__u16 slid;
+	__u32 raw_traffic;
+	__u32 flow_label;
+	__u32 reversible;
+	__u32 mtu;
+	__u16 pkey;
+	__u8  hop_limit;
+	__u8  traffic_class;
+	__u8  numb_path;
+	__u8  sl;
+	__u8  mtu_selector;
+	__u8  rate_selector;
+	__u8  rate;
+	__u8  packet_life_time_selector;
+	__u8  packet_life_time;
+	__u8  preference;
+};
+
+struct ib_ucm_req {
+	__u32 id;
+	__u32 qpn;
+	__u32 qp_type;
+	__u32 psn;
+	__u64 sid;
+	__u64 data;
+	__u64 primary_path;
+	__u64 alternate_path;
+	__u8  len;
+	__u8  peer_to_peer;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  remote_cm_response_timeout;
+	__u8  flow_control;
+	__u8  local_cm_response_timeout;
+	__u8  retry_count;
+	__u8  rnr_retry_count;
+	__u8  max_cm_retries;
+	__u8  srq;
+	__u8  reserved[1];
+};
+
+struct ib_ucm_rep {
+	__u64 data;
+	__u32 id;
+	__u32 qpn;
+	__u32 psn;
+	__u8  len;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  target_ack_delay;
+	__u8  failover_accepted;
+	__u8  flow_control;
+	__u8  rnr_retry_count;
+	__u8  srq;
+};
+
+struct ib_ucm_info {
+	__u32 id;
+	__u32 status;
+	__u64 info;
+	__u64 data;
+	__u8  info_len;
+	__u8  data_len;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_mra {
+	__u64 data;
+	__u32 id;
+	__u8  len;
+	__u8  timeout;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_lap {
+	__u64 path;
+	__u64 data;
+	__u32 id;
+	__u8  len;
+	__u8  reserved[3];
+};
+
+struct ib_ucm_sidr_req {
+	__u32 id;
+	__u32 timeout;
+	__u64 sid;
+	__u64 data;
+	__u64 path;
+	__u16 pkey;
+	__u8  len;
+	__u8  max_cm_retries;
+};
+
+struct ib_ucm_sidr_rep {
+	__u32 id;
+	__u32 qpn;
+	__u32 qkey;
+	__u32 status;
+	__u64 info;
+	__u64 data;
+	__u8  info_len;
+	__u8  data_len;
+	__u8  reserved[2];
+};
+/*
+ * event notification ABI structures.
+ */
+struct ib_ucm_event_get {
+	__u64 response;
+	__u64 data;
+	__u64 info;
+	__u8  data_len;
+	__u8  info_len;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_req_event_resp {
+	__u32                  listen_id;
+	/* device */
+	/* port */
+	struct ib_ucm_path_rec primary_path;
+	struct ib_ucm_path_rec alternate_path;
+	__u64                  remote_ca_guid;
+	__u32                  remote_qkey;
+	__u32                  remote_qpn;
+	__u32                  qp_type;
+	__u32                  starting_psn;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  local_cm_response_timeout;
+	__u8  flow_control;
+	__u8  remote_cm_response_timeout;
+	__u8  retry_count;
+	__u8  rnr_retry_count;
+	__u8  srq;
+};
+
+struct ib_ucm_rep_event_resp {
+	__u64 remote_ca_guid;
+	__u32 remote_qkey;
+	__u32 remote_qpn;
+	__u32 starting_psn;
+	__u8  responder_resources;
+	__u8  initiator_depth;
+	__u8  target_ack_delay;
+	__u8  failover_accepted;
+	__u8  flow_control;
+	__u8  rnr_retry_count;
+	__u8  srq;
+	__u8  reserved[1];
+};
+
+struct ib_ucm_rej_event_resp {
+	__u32 reason;
+	/* ari in ib_ucm_event_get info field. */
+};
+
+struct ib_ucm_mra_event_resp {
+	__u8  timeout;
+	__u8  reserved[3];
+};
+
+struct ib_ucm_lap_event_resp {
+	struct ib_ucm_path_rec path;
+};
+
+struct ib_ucm_apr_event_resp {
+	__u32 status;
+	/* apr info in ib_ucm_event_get info field. */
+};
+
+struct ib_ucm_sidr_req_event_resp {
+	__u32 listen_id;
+	/* device */
+	/* port */
+	__u16 pkey;
+	__u8  reserved[2];
+};
+
+struct ib_ucm_sidr_rep_event_resp {
+	__u32 status;
+	__u32 qkey;
+	__u32 qpn;
+	/* info in ib_ucm_event_get info field. */
+};
+
+#define IB_UCM_PRES_DATA      0x01
+#define IB_UCM_PRES_INFO      0x02
+#define IB_UCM_PRES_PRIMARY   0x04
+#define IB_UCM_PRES_ALTERNATE 0x08
+
+struct ib_ucm_event_resp {
+	__u32 id;
+	__u32 event;
+	__u32 present;
+	union {
+		struct ib_ucm_req_event_resp req_resp;
+		struct ib_ucm_rep_event_resp rep_resp;
+		struct ib_ucm_rej_event_resp rej_resp;
+		struct ib_ucm_mra_event_resp mra_resp;
+		struct ib_ucm_lap_event_resp lap_resp;
+		struct ib_ucm_apr_event_resp apr_resp;
+
+		struct ib_ucm_sidr_req_event_resp sidr_req_resp;
+		struct ib_ucm_sidr_rep_event_resp sidr_rep_resp;
+
+		__u32                             send_status;
+	} u;
+};
+
+#endif /* IB_USER_CM_H */




From halr at voltaire.com  Mon Jul 11 13:59:41 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 16:59:41 -0400
Subject: [openib-general] [PATCH 26/29v2] Add kernel portion of user CM
	implementation
Message-ID: <1121110427.4389.5036.camel@hal.voltaire.com>

Add kernel portion of user CM implementation


Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 25/29.

--
 ucm.c | 1396 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 ucm.h |   89 ++++
 2 files changed, 1485 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1-25/drivers/infiniband/core/ucm.c linux-2.6.13-rc2-mm1-26/drivers/infiniband/core/ucm.c
-- linux-2.6.13-rc2-mm1-25/drivers/infiniband/core/ucm.c	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-26/drivers/infiniband/core/ucm.c	2005-07-10 16:58:39.000000000 -0400
@@ -0,0 +1,1396 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *	copyright notice, this list of conditions and the following
+ *	disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *	copyright notice, this list of conditions and the following
+ *	disclaimer in the documentation and/or other materials
+ *	provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ucm.c 2594 2005-06-13 19:46:02Z libor $
+ */
+#include <linux/init.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/poll.h>
+#include <linux/file.h>
+#include <linux/mount.h>
+#include <linux/cdev.h>
+
+#include <asm/uaccess.h>
+
+#include "ucm.h"
+	
+MODULE_AUTHOR("Libor Michalek");
+MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
+MODULE_LICENSE("Dual BSD/GPL");
+
+enum {
+	IB_UCM_MAJOR = 231,
+	IB_UCM_MINOR = 255
+};
+
+#define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
+
+static struct semaphore ctx_id_mutex;
+static struct idr       ctx_id_table;
+static int              ctx_id_rover = 0;
+
+static struct ib_ucm_context *ib_ucm_ctx_get(int id)
+{
+	struct ib_ucm_context *ctx;
+
+	down(&ctx_id_mutex);
+	ctx = idr_find(&ctx_id_table, id);
+	if (ctx)
+		ctx->ref++;
+	up(&ctx_id_mutex);
+
+	return ctx;
+}
+
+static void ib_ucm_ctx_put(struct ib_ucm_context *ctx)
+{
+	struct ib_ucm_event *uevent;
+
+	down(&ctx_id_mutex);
+
+	ctx->ref--;
+	if (!ctx->ref)
+		idr_remove(&ctx_id_table, ctx->id);
+
+	up(&ctx_id_mutex);
+
+	if (ctx->ref)
+		return;
+
+	down(&ctx->file->mutex);
+		
+	list_del(&ctx->file_list);
+	while (!list_empty(&ctx->events)) {
+
+		uevent = list_entry(ctx->events.next,
+				    struct ib_ucm_event, ctx_list);
+		list_del(&uevent->file_list);
+		list_del(&uevent->ctx_list);
+
+		/* clear incoming connections. */
+		if (uevent->cm_id)
+			ib_destroy_cm_id(uevent->cm_id);
+
+		kfree(uevent);
+	}
+
+	up(&ctx->file->mutex);
+			
+	printk(KERN_ERR "UCM: Destroyed CM ID <%d>\n", ctx->id);
+
+	ib_destroy_cm_id(ctx->cm_id);
+	kfree(ctx);
+}
+
+static struct ib_ucm_context *ib_ucm_ctx_alloc(struct ib_ucm_file *file)
+{
+	struct ib_ucm_context *ctx;
+	int result;
+
+	ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)
+		return NULL;
+
+	ctx->ref  = 1; /* user reference */
+	ctx->file = file;
+
+	INIT_LIST_HEAD(&ctx->events);
+	init_MUTEX(&ctx->mutex);
+
+	list_add_tail(&ctx->file_list, &file->ctxs);
+
+	ctx_id_rover = (ctx_id_rover + 1) & INT_MAX;
+retry:
+	result = idr_pre_get(&ctx_id_table, GFP_KERNEL);
+	if (!result)
+		goto error;
+	
+	down(&ctx_id_mutex);
+	result = idr_get_new_above(&ctx_id_table, ctx, ctx_id_rover, &ctx->id);
+	up(&ctx_id_mutex);
+
+	if (result == -EAGAIN)
+		goto retry;
+	if (result)
+		goto error;
+
+	printk(KERN_ERR "UCM: Allocated CM ID <%d>\n", ctx->id);
+
+	return ctx;
+error:
+	list_del(&ctx->file_list);
+	kfree(ctx);
+
+	return NULL;
+}
+/*
+ * Event portion of the API, handle CM events
+ * and allow event polling.
+ */
+static void ib_ucm_event_path_get(struct ib_ucm_path_rec *upath,
+				  struct ib_sa_path_rec	 *kpath)
+{
+	if (!kpath || !upath)
+		return;
+
+	memcpy(upath->dgid, kpath->dgid.raw, sizeof(union ib_gid));
+	memcpy(upath->sgid, kpath->sgid.raw, sizeof(union ib_gid));
+	
+	upath->dlid             = kpath->dlid;
+	upath->slid             = kpath->slid;
+	upath->raw_traffic      = kpath->raw_traffic;
+	upath->flow_label       = kpath->flow_label;
+	upath->hop_limit        = kpath->hop_limit;
+	upath->traffic_class    = kpath->traffic_class;
+	upath->reversible       = kpath->reversible;
+	upath->numb_path        = kpath->numb_path;
+	upath->pkey             = kpath->pkey;
+	upath->sl	        = kpath->sl;
+	upath->mtu_selector     = kpath->mtu_selector;
+	upath->mtu              = kpath->mtu;
+	upath->rate_selector    = kpath->rate_selector;
+	upath->rate             = kpath->rate;
+	upath->packet_life_time = kpath->packet_life_time;
+	upath->preference       = kpath->preference;
+
+	upath->packet_life_time_selector = 
+		kpath->packet_life_time_selector;
+}
+
+static void ib_ucm_event_req_get(struct ib_ucm_req_event_resp *ureq,
+				 struct ib_cm_req_event_param *kreq)
+{
+	ureq->listen_id = (long)kreq->listen_id->context;
+
+	ureq->remote_ca_guid             = kreq->remote_ca_guid;
+	ureq->remote_qkey                = kreq->remote_qkey;
+	ureq->remote_qpn                 = kreq->remote_qpn;
+	ureq->qp_type                    = kreq->qp_type;
+	ureq->starting_psn               = kreq->starting_psn;
+	ureq->responder_resources        = kreq->responder_resources;
+	ureq->initiator_depth            = kreq->initiator_depth;
+	ureq->local_cm_response_timeout  = kreq->local_cm_response_timeout;
+	ureq->flow_control               = kreq->flow_control;
+	ureq->remote_cm_response_timeout = kreq->remote_cm_response_timeout;
+	ureq->retry_count                = kreq->retry_count;
+	ureq->rnr_retry_count            = kreq->rnr_retry_count;
+	ureq->srq                        = kreq->srq;
+
+	ib_ucm_event_path_get(&ureq->primary_path, kreq->primary_path);
+	ib_ucm_event_path_get(&ureq->alternate_path, kreq->alternate_path);
+}
+
+static void ib_ucm_event_rep_get(struct ib_ucm_rep_event_resp *urep,
+				 struct ib_cm_rep_event_param *krep)
+{
+	urep->remote_ca_guid      = krep->remote_ca_guid;
+	urep->remote_qkey         = krep->remote_qkey;
+	urep->remote_qpn          = krep->remote_qpn;
+	urep->starting_psn        = krep->starting_psn;
+	urep->responder_resources = krep->responder_resources;
+	urep->initiator_depth     = krep->initiator_depth;
+	urep->target_ack_delay    = krep->target_ack_delay;
+	urep->failover_accepted   = krep->failover_accepted;
+	urep->flow_control        = krep->flow_control;
+	urep->rnr_retry_count     = krep->rnr_retry_count;
+	urep->srq                 = krep->srq;
+}
+
+static void ib_ucm_event_rej_get(struct ib_ucm_rej_event_resp *urej,
+				 struct ib_cm_rej_event_param *krej)
+{
+	urej->reason = krej->reason;
+}
+
+static void ib_ucm_event_mra_get(struct ib_ucm_mra_event_resp *umra,
+				 struct ib_cm_mra_event_param *kmra)
+{
+	umra->timeout = kmra->service_timeout;
+}
+
+static void ib_ucm_event_lap_get(struct ib_ucm_lap_event_resp *ulap,
+				 struct ib_cm_lap_event_param *klap)
+{
+	ib_ucm_event_path_get(&ulap->path, klap->alternate_path);
+}
+
+static void ib_ucm_event_apr_get(struct ib_ucm_apr_event_resp *uapr,
+				 struct ib_cm_apr_event_param *kapr)
+{
+	uapr->status = kapr->ap_status;
+}
+
+static void ib_ucm_event_sidr_req_get(struct ib_ucm_sidr_req_event_resp *ureq,
+				      struct ib_cm_sidr_req_event_param *kreq)
+{
+	ureq->listen_id = (long)kreq->listen_id->context;
+	ureq->pkey      = kreq->pkey;
+}
+
+static void ib_ucm_event_sidr_rep_get(struct ib_ucm_sidr_rep_event_resp *urep,
+				      struct ib_cm_sidr_rep_event_param *krep)
+{
+	urep->status = krep->status;
+	urep->qkey   = krep->qkey;
+	urep->qpn    = krep->qpn;
+};
+
+static int ib_ucm_event_process(struct ib_cm_event *evt,
+				struct ib_ucm_event *uvt)
+{
+	void *info = NULL;
+	int result;
+
+	switch (evt->event) {
+	case IB_CM_REQ_RECEIVED:
+		ib_ucm_event_req_get(&uvt->resp.u.req_resp, 
+				     &evt->param.req_rcvd);
+		uvt->data_len      = IB_CM_REQ_PRIVATE_DATA_SIZE;
+		uvt->resp.present |= (evt->param.req_rcvd.primary_path ? 
+				      IB_UCM_PRES_PRIMARY : 0);
+		uvt->resp.present |= (evt->param.req_rcvd.alternate_path ? 
+				      IB_UCM_PRES_ALTERNATE : 0);
+		break;
+	case IB_CM_REP_RECEIVED:
+		ib_ucm_event_rep_get(&uvt->resp.u.rep_resp, 
+				     &evt->param.rep_rcvd);
+		uvt->data_len = IB_CM_REP_PRIVATE_DATA_SIZE;
+
+		break;
+	case IB_CM_RTU_RECEIVED:
+		uvt->data_len = IB_CM_RTU_PRIVATE_DATA_SIZE;
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	case IB_CM_DREQ_RECEIVED:
+		uvt->data_len = IB_CM_DREQ_PRIVATE_DATA_SIZE;
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	case IB_CM_DREP_RECEIVED:
+		uvt->data_len = IB_CM_DREP_PRIVATE_DATA_SIZE;
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	case IB_CM_MRA_RECEIVED:
+		ib_ucm_event_mra_get(&uvt->resp.u.mra_resp,
+				     &evt->param.mra_rcvd);
+		uvt->data_len = IB_CM_MRA_PRIVATE_DATA_SIZE;
+
+		break;
+	case IB_CM_REJ_RECEIVED:
+		ib_ucm_event_rej_get(&uvt->resp.u.rej_resp,
+				     &evt->param.rej_rcvd);
+		uvt->data_len = IB_CM_REJ_PRIVATE_DATA_SIZE;
+		uvt->info_len = evt->param.rej_rcvd.ari_length;
+		info	      = evt->param.rej_rcvd.ari;
+
+		break;
+	case IB_CM_LAP_RECEIVED:
+		ib_ucm_event_lap_get(&uvt->resp.u.lap_resp,
+				     &evt->param.lap_rcvd);
+		uvt->data_len = IB_CM_LAP_PRIVATE_DATA_SIZE;
+		uvt->resp.present |= (evt->param.lap_rcvd.alternate_path ? 
+				      IB_UCM_PRES_ALTERNATE : 0);
+		break;
+	case IB_CM_APR_RECEIVED:
+		ib_ucm_event_apr_get(&uvt->resp.u.apr_resp,
+				     &evt->param.apr_rcvd);
+		uvt->data_len = IB_CM_APR_PRIVATE_DATA_SIZE;
+		uvt->info_len = evt->param.apr_rcvd.info_len;
+		info	      = evt->param.apr_rcvd.apr_info;
+
+		break;
+	case IB_CM_SIDR_REQ_RECEIVED:
+		ib_ucm_event_sidr_req_get(&uvt->resp.u.sidr_req_resp, 
+					  &evt->param.sidr_req_rcvd);
+		uvt->data_len = IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE;
+
+		break;
+	case IB_CM_SIDR_REP_RECEIVED:
+		ib_ucm_event_sidr_rep_get(&uvt->resp.u.sidr_rep_resp, 
+					  &evt->param.sidr_rep_rcvd);
+		uvt->data_len = IB_CM_SIDR_REP_PRIVATE_DATA_SIZE;
+		uvt->info_len = evt->param.sidr_rep_rcvd.info_len;
+		info	      = evt->param.sidr_rep_rcvd.info;
+
+		break;
+	default:
+		uvt->resp.u.send_status = evt->param.send_status;
+
+		break;
+	}
+
+	if (uvt->data_len && evt->private_data) {
+
+		uvt->data = kmalloc(uvt->data_len, GFP_KERNEL);
+		if (!uvt->data) {
+			result = -ENOMEM;
+			goto error;
+		}
+
+		memcpy(uvt->data, evt->private_data, uvt->data_len);
+		uvt->resp.present |= IB_UCM_PRES_DATA;
+	}
+
+	if (uvt->info_len && info) {
+
+		uvt->info = kmalloc(uvt->info_len, GFP_KERNEL);
+		if (!uvt->info) {
+			result = -ENOMEM;
+			goto error;
+		}
+
+		memcpy(uvt->info, info, uvt->info_len);
+		uvt->resp.present |= IB_UCM_PRES_INFO;
+	}
+
+	return 0;
+error:
+	if (uvt->info)
+		kfree(uvt->info);
+	if (uvt->data)
+		kfree(uvt->data);
+	return result;
+}
+	
+static int ib_ucm_event_handler(struct ib_cm_id *cm_id,
+				struct ib_cm_event *event)
+{
+	struct ib_ucm_event *uevent;
+	struct ib_ucm_context *ctx;
+	int result = 0;
+	int id;
+	/*
+	 * lookup correct context based on event type.
+	 */
+	switch (event->event) {
+	case IB_CM_REQ_RECEIVED:
+		id = (long)event->param.req_rcvd.listen_id->context;
+		break;
+	case IB_CM_SIDR_REQ_RECEIVED:
+		id = (long)event->param.sidr_req_rcvd.listen_id->context;
+		break;
+	default:
+		id = (long)cm_id->context;
+		break;
+	}
+
+	printk(KERN_ERR "UCM: Event. CM ID <%d> event <%d>\n",
+	       id, event->event);
+
+	ctx = ib_ucm_ctx_get(id);
+	if (!ctx)
+		return -ENOENT;
+
+	if (event->event == IB_CM_REQ_RECEIVED ||
+	    event->event == IB_CM_SIDR_REQ_RECEIVED)
+		id = IB_UCM_CM_ID_INVALID;
+
+	uevent = kmalloc(sizeof(*uevent), GFP_KERNEL);
+	if (!uevent) {
+		result = -ENOMEM;
+		goto done;
+	}
+
+	memset(uevent, 0, sizeof(*uevent));
+
+	uevent->resp.id    = id;
+	uevent->resp.event = event->event;
+
+	result = ib_ucm_event_process(event, uevent);
+	if (result)
+		goto done;
+
+	uevent->ctx   = ctx;
+	uevent->cm_id = ((event->event == IB_CM_REQ_RECEIVED ||
+			  event->event == IB_CM_SIDR_REQ_RECEIVED ) ?
+			 cm_id : NULL);
+
+	down(&ctx->file->mutex);
+
+	list_add_tail(&uevent->file_list, &ctx->file->events);
+	list_add_tail(&uevent->ctx_list, &ctx->events);
+
+	wake_up_interruptible(&ctx->file->poll_wait);
+
+	up(&ctx->file->mutex);
+done:
+	ctx->error = result;
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static ssize_t ib_ucm_event(struct ib_ucm_file *file,
+			    const char __user *inbuf, 
+			    int in_len, int out_len)
+{
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_event_get cmd;
+	struct ib_ucm_event *uevent = NULL;
+	int result = 0;
+	DEFINE_WAIT(wait);
+	
+	if (out_len < sizeof(struct ib_ucm_event_resp))
+		return -ENOSPC;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+	/*
+	 * wait
+	 */
+	down(&file->mutex);
+
+	while (list_empty(&file->events)) {
+
+		if (file->filp->f_flags & O_NONBLOCK) {
+			result = -EAGAIN;
+			break;
+		}
+
+		if (signal_pending(current)) {
+			result = -ERESTARTSYS;
+			break;
+		}
+
+		prepare_to_wait(&file->poll_wait, &wait, TASK_INTERRUPTIBLE);
+
+		up(&file->mutex);
+		schedule();
+		down(&file->mutex);
+
+		finish_wait(&file->poll_wait, &wait);
+	}
+
+	if (result)
+		goto done;
+	
+	uevent = list_entry(file->events.next, struct ib_ucm_event, file_list);
+
+	if (!uevent->cm_id)
+		goto user;
+
+	ctx = ib_ucm_ctx_alloc(file);
+	if (!ctx) {
+		result = -ENOMEM;
+		goto done;
+	}
+
+	ctx->cm_id             = uevent->cm_id;
+	ctx->cm_id->cm_handler = ib_ucm_event_handler;
+	ctx->cm_id->context    = (void *)(unsigned long)ctx->id;
+
+	uevent->resp.id = ctx->id;
+
+user:
+	if (copy_to_user((void __user *)(unsigned long)cmd.response,
+			 &uevent->resp, sizeof(uevent->resp))) {
+		result = -EFAULT;
+		goto done;
+	}
+
+	if (uevent->data) {
+
+		if (cmd.data_len < uevent->data_len) {
+			result = -ENOMEM;
+			goto done;
+		}
+
+		if (copy_to_user((void __user *)(unsigned long)cmd.data,
+				 uevent->data, uevent->data_len)) {
+			result = -EFAULT;
+			goto done;
+		}
+	}
+
+	if (uevent->info) {
+
+		if (cmd.info_len < uevent->info_len) {
+			result = -ENOMEM;
+			goto done;
+		}
+
+		if (copy_to_user((void __user *)(unsigned long)cmd.info,
+				 uevent->info, uevent->info_len)) {
+			result = -EFAULT;
+			goto done;
+		}
+	}
+
+	list_del(&uevent->file_list);
+	list_del(&uevent->ctx_list);
+
+	if (uevent->data)
+		kfree(uevent->data);
+	if (uevent->info)
+		kfree(uevent->info);
+	kfree(uevent);
+done:
+	up(&file->mutex);
+	return result;
+}
+
+
+static ssize_t ib_ucm_create_id(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	struct ib_ucm_create_id cmd;
+	struct ib_ucm_create_id_resp resp;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	if (out_len < sizeof(resp))
+		return -ENOSPC;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	ctx = ib_ucm_ctx_alloc(file);
+	if (!ctx)
+		return -ENOMEM;
+
+	ctx->cm_id = ib_create_cm_id(ib_ucm_event_handler, 
+				     (void *)(unsigned long)ctx->id);
+	if (!ctx->cm_id) {
+		result = -ENOMEM;
+		goto err_cm;
+	}
+
+	resp.id = ctx->id;
+	if (copy_to_user((void __user *)(unsigned long)cmd.response,
+			 &resp, sizeof(resp))) {
+		result = -EFAULT;
+		goto err_ret;
+	}
+
+	return 0;
+err_ret:
+	ib_destroy_cm_id(ctx->cm_id);
+err_cm:
+	ib_ucm_ctx_put(ctx); /* user reference */
+
+	return result;
+}
+
+static ssize_t ib_ucm_destroy_id(struct ib_ucm_file *file,
+				 const char __user *inbuf, 
+				 int in_len, int out_len)
+{
+	struct ib_ucm_destroy_id cmd;
+	struct ib_ucm_context *ctx;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+	ib_ucm_ctx_put(ctx); /* user reference */
+	ib_ucm_ctx_put(ctx); /* func reference */
+
+	return 0;
+}
+
+static ssize_t ib_ucm_attr_id(struct ib_ucm_file *file,
+			      const char __user *inbuf, 
+			      int in_len, int out_len)
+{
+	struct ib_ucm_attr_id_resp resp;
+	struct ib_ucm_attr_id cmd;
+	struct ib_ucm_context *ctx;
+	int result = 0;
+
+	if (out_len < sizeof(resp))
+		return -ENOSPC;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file) {
+		result = -EINVAL;
+		goto done;
+	}
+
+	resp.service_id   = ctx->cm_id->service_id;
+	resp.service_mask = ctx->cm_id->service_mask;
+	resp.local_id     = ctx->cm_id->local_id;
+	resp.remote_id    = ctx->cm_id->remote_id;
+
+	if (copy_to_user((void __user *)(unsigned long)cmd.response,
+			 &resp, sizeof(resp)))
+		result = -EFAULT;
+
+done:
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static ssize_t ib_ucm_listen(struct ib_ucm_file *file,
+			     const char __user *inbuf, 
+			     int in_len, int out_len)
+{
+	struct ib_ucm_listen cmd;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+	
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_cm_listen(ctx->cm_id, cmd.service_id,
+				      cmd.service_mask);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static ssize_t ib_ucm_establish(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	struct ib_ucm_establish cmd;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+	
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx)
+		return -ENOENT;
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_cm_establish(ctx->cm_id);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+	return result;
+}
+
+static int ib_ucm_alloc_data(const void **dest, u64 src, u32 len)
+{
+	void *data;
+
+	*dest = NULL;
+
+	if (!len)
+		return 0;
+
+	data = kmalloc(len, GFP_KERNEL);
+	if (!data)
+		return -ENOMEM;
+	
+	if (copy_from_user(data, (void __user *)(unsigned long)src, len)) {
+		kfree(data);
+		return -EFAULT;
+	}
+
+	*dest = data;
+	return 0;
+}
+
+static int ib_ucm_path_get(struct ib_sa_path_rec **path, u64 src)
+{
+	struct ib_ucm_path_rec ucm_path;
+	struct ib_sa_path_rec  *sa_path;
+
+	*path = NULL;
+
+	if (!src)
+		return 0;
+
+	sa_path = kmalloc(sizeof(*sa_path), GFP_KERNEL);
+	if (!sa_path)
+		return -ENOMEM;
+
+	if (copy_from_user(&ucm_path, (void __user *)(unsigned long)src,
+			   sizeof(ucm_path))) {
+
+		kfree(sa_path);
+		return -EFAULT;
+	}
+
+	memcpy(sa_path->dgid.raw, ucm_path.dgid, sizeof(union ib_gid));
+	memcpy(sa_path->sgid.raw, ucm_path.sgid, sizeof(union ib_gid));
+
+	sa_path->dlid	          = ucm_path.dlid;
+	sa_path->slid	          = ucm_path.slid;
+	sa_path->raw_traffic      = ucm_path.raw_traffic;
+	sa_path->flow_label       = ucm_path.flow_label;
+	sa_path->hop_limit        = ucm_path.hop_limit;
+	sa_path->traffic_class    = ucm_path.traffic_class;
+	sa_path->reversible       = ucm_path.reversible;
+	sa_path->numb_path        = ucm_path.numb_path;
+	sa_path->pkey             = ucm_path.pkey;
+	sa_path->sl               = ucm_path.sl;
+	sa_path->mtu_selector     = ucm_path.mtu_selector;
+	sa_path->mtu              = ucm_path.mtu;
+	sa_path->rate_selector    = ucm_path.rate_selector;
+	sa_path->rate             = ucm_path.rate;
+	sa_path->packet_life_time = ucm_path.packet_life_time;
+	sa_path->preference       = ucm_path.preference;
+
+	sa_path->packet_life_time_selector = 
+		ucm_path.packet_life_time_selector;
+
+	*path = sa_path;
+	return 0;
+}
+
+static ssize_t ib_ucm_send_req(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_cm_req_param param;
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_req cmd;
+	int result;
+
+	param.private_data   = NULL;
+	param.primary_path   = NULL;
+	param.alternate_path = NULL;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data, cmd.data, cmd.len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&param.primary_path, cmd.primary_path);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&param.alternate_path, cmd.alternate_path);
+	if (result)
+		goto done;
+
+	param.private_data_len           = cmd.len;
+	param.service_id                 = cmd.sid;
+	param.qp_num                     = cmd.qpn;
+	param.qp_type                    = cmd.qp_type;
+	param.starting_psn               = cmd.psn;
+	param.peer_to_peer               = cmd.peer_to_peer;
+	param.responder_resources        = cmd.responder_resources;
+	param.initiator_depth            = cmd.initiator_depth;
+	param.remote_cm_response_timeout = cmd.remote_cm_response_timeout;
+	param.flow_control               = cmd.flow_control;
+	param.local_cm_response_timeout  = cmd.local_cm_response_timeout;
+	param.retry_count                = cmd.retry_count;
+	param.rnr_retry_count            = cmd.rnr_retry_count;
+	param.max_cm_retries             = cmd.max_cm_retries;
+	param.srq                        = cmd.srq;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_req(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+	if (param.primary_path)
+		kfree(param.primary_path);
+	if (param.alternate_path)
+		kfree(param.alternate_path);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_rep(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_cm_rep_param param;
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_rep cmd;
+	int result;
+
+	param.private_data = NULL;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data, cmd.data, cmd.len);
+	if (result)
+		return result;
+
+	param.qp_num              = cmd.qpn;
+	param.starting_psn        = cmd.psn;
+	param.private_data_len    = cmd.len;
+	param.responder_resources = cmd.responder_resources;
+	param.initiator_depth     = cmd.initiator_depth;
+	param.target_ack_delay    = cmd.target_ack_delay;
+	param.failover_accepted   = cmd.failover_accepted;
+	param.flow_control        = cmd.flow_control;
+	param.rnr_retry_count     = cmd.rnr_retry_count;
+	param.srq                 = cmd.srq;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_rep(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_private_data(struct ib_ucm_file *file,
+					const char __user *inbuf, int in_len,
+					int (*func)(struct ib_cm_id *cm_id,
+						    const void *private_data,
+						    u8 private_data_len))
+{
+	struct ib_ucm_private_data cmd;
+	struct ib_ucm_context *ctx;
+	const void *private_data = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&private_data, cmd.data, cmd.len);
+	if (result)
+		return result;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = func(ctx->cm_id, private_data, cmd.len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (private_data)
+		kfree(private_data);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_rtu(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	return ib_ucm_send_private_data(file, inbuf, in_len, ib_send_cm_rtu);
+}
+
+static ssize_t ib_ucm_send_dreq(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	return ib_ucm_send_private_data(file, inbuf, in_len, ib_send_cm_dreq);
+}
+
+static ssize_t ib_ucm_send_drep(struct ib_ucm_file *file,
+				const char __user *inbuf, 
+				int in_len, int out_len)
+{
+	return ib_ucm_send_private_data(file, inbuf, in_len, ib_send_cm_drep);
+}
+
+static ssize_t ib_ucm_send_info(struct ib_ucm_file *file,
+				const char __user *inbuf, int in_len,
+				int (*func)(struct ib_cm_id *cm_id,
+					    int status,
+					    const void *info,
+					    u8 info_len,
+					    const void *data,
+					    u8 data_len))
+{
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_info cmd;
+	const void *data = NULL;
+	const void *info = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&data, cmd.data, cmd.data_len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_alloc_data(&info, cmd.info, cmd.info_len);
+	if (result)
+		goto done;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = func(ctx->cm_id, cmd.status,
+			      info, cmd.info_len,
+			      data, cmd.data_len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (data)
+		kfree(data);
+	if (info)
+		kfree(info);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_rej(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	return ib_ucm_send_info(file, inbuf, in_len, (void *)ib_send_cm_rej);
+}
+
+static ssize_t ib_ucm_send_apr(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	return ib_ucm_send_info(file, inbuf, in_len, (void *)ib_send_cm_apr);
+}
+
+static ssize_t ib_ucm_send_mra(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_mra cmd;
+	const void *data = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&data, cmd.data, cmd.len);
+	if (result)
+		return result;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_mra(ctx->cm_id, cmd.timeout, 
+					data, cmd.len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (data)
+		kfree(data);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_lap(struct ib_ucm_file *file,
+			       const char __user *inbuf, 
+			       int in_len, int out_len)
+{
+	struct ib_ucm_context *ctx;
+	struct ib_sa_path_rec *path = NULL;
+	struct ib_ucm_lap cmd;
+	const void *data = NULL;
+	int result;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&data, cmd.data, cmd.len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&path, cmd.path);
+	if (result)
+		goto done;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_lap(ctx->cm_id, path, data, cmd.len);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (data)
+		kfree(data);
+	if (path)
+		kfree(path);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_sidr_req(struct ib_ucm_file *file,
+				    const char __user *inbuf, 
+				    int in_len, int out_len)
+{
+	struct ib_cm_sidr_req_param param;
+	struct ib_ucm_context *ctx;
+	struct ib_ucm_sidr_req cmd;
+	int result;
+
+	param.private_data = NULL;
+	param.path = NULL;
+	
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data, cmd.data, cmd.len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_path_get(&param.path, cmd.path);
+	if (result)
+		goto done;
+
+	param.private_data_len = cmd.len;
+	param.service_id       = cmd.sid;
+	param.timeout_ms       = cmd.timeout;
+	param.max_cm_retries   = cmd.max_cm_retries;
+	param.pkey             = cmd.pkey;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_sidr_req(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+	if (param.path)
+		kfree(param.path);
+
+	return result;
+}
+
+static ssize_t ib_ucm_send_sidr_rep(struct ib_ucm_file *file,
+				    const char __user *inbuf, 
+				    int in_len, int out_len)
+{
+	struct ib_cm_sidr_rep_param param;
+	struct ib_ucm_sidr_rep cmd;
+	struct ib_ucm_context *ctx;
+	int result;
+
+	param.info = NULL;
+
+	if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
+		return -EFAULT;
+
+	result = ib_ucm_alloc_data(&param.private_data,
+				   cmd.data, cmd.data_len);
+	if (result)
+		goto done;
+
+	result = ib_ucm_alloc_data(&param.info, cmd.info, cmd.info_len);
+	if (result)
+		goto done;
+
+	param.qp_num	   = cmd.qpn;
+	param.qkey	     = cmd.qkey;
+	param.status	   = cmd.status;
+	param.info_length      = cmd.info_len;
+	param.private_data_len = cmd.data_len;
+
+	ctx = ib_ucm_ctx_get(cmd.id);
+	if (!ctx) {
+		result = -ENOENT;
+		goto done;
+	}
+
+ 	down(&ctx->file->mutex);
+	if (ctx->file != file)
+		result = -EINVAL;
+	else
+		result = ib_send_cm_sidr_rep(ctx->cm_id, &param);
+
+	up(&ctx->file->mutex);
+	ib_ucm_ctx_put(ctx); /* func reference */
+done:
+	if (param.private_data)
+		kfree(param.private_data);
+	if (param.info)
+		kfree(param.info);
+
+	return result;
+}
+
+static ssize_t (*ucm_cmd_table[])(struct ib_ucm_file *file,
+				  const char __user *inbuf,
+				  int in_len, int out_len) = {
+	[IB_USER_CM_CMD_CREATE_ID]     = ib_ucm_create_id,
+	[IB_USER_CM_CMD_DESTROY_ID]    = ib_ucm_destroy_id,
+	[IB_USER_CM_CMD_ATTR_ID]       = ib_ucm_attr_id,
+	[IB_USER_CM_CMD_LISTEN]        = ib_ucm_listen,
+	[IB_USER_CM_CMD_ESTABLISH]     = ib_ucm_establish,
+	[IB_USER_CM_CMD_SEND_REQ]      = ib_ucm_send_req,
+	[IB_USER_CM_CMD_SEND_REP]      = ib_ucm_send_rep,
+	[IB_USER_CM_CMD_SEND_RTU]      = ib_ucm_send_rtu,
+	[IB_USER_CM_CMD_SEND_DREQ]     = ib_ucm_send_dreq,
+	[IB_USER_CM_CMD_SEND_DREP]     = ib_ucm_send_drep,
+	[IB_USER_CM_CMD_SEND_REJ]      = ib_ucm_send_rej,
+	[IB_USER_CM_CMD_SEND_MRA]      = ib_ucm_send_mra,
+	[IB_USER_CM_CMD_SEND_LAP]      = ib_ucm_send_lap,
+	[IB_USER_CM_CMD_SEND_APR]      = ib_ucm_send_apr,
+	[IB_USER_CM_CMD_SEND_SIDR_REQ] = ib_ucm_send_sidr_req,
+	[IB_USER_CM_CMD_SEND_SIDR_REP] = ib_ucm_send_sidr_rep,
+	[IB_USER_CM_CMD_EVENT]	       = ib_ucm_event,
+};
+
+static ssize_t ib_ucm_write(struct file *filp, const char __user *buf,
+			    size_t len, loff_t *pos)
+{
+	struct ib_ucm_file *file = filp->private_data;
+	struct ib_ucm_cmd_hdr hdr;
+	ssize_t result;
+
+	if (len < sizeof(hdr))
+		return -EINVAL;
+
+	if (copy_from_user(&hdr, buf, sizeof(hdr)))
+		return -EFAULT;
+
+	printk(KERN_ERR "UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
+	       hdr.cmd, hdr.in, hdr.out, len);
+
+	if (hdr.cmd < 0 || hdr.cmd >= ARRAY_SIZE(ucm_cmd_table))
+		return -EINVAL;
+
+	if (hdr.in + sizeof(hdr) > len)
+		return -EINVAL;
+
+	result = ucm_cmd_table[hdr.cmd](file, buf + sizeof(hdr),
+					hdr.in, hdr.out);
+	if (!result)
+		result = len;
+
+	return result;
+}
+
+static unsigned int ib_ucm_poll(struct file *filp,
+				struct poll_table_struct *wait)
+{
+	struct ib_ucm_file *file = filp->private_data;
+	unsigned int mask = 0;
+
+	poll_wait(filp, &file->poll_wait, wait);
+
+	if (!list_empty(&file->events))
+		mask = POLLIN | POLLRDNORM;
+
+	return mask;
+}
+
+static int ib_ucm_open(struct inode *inode, struct file *filp)
+{
+	struct ib_ucm_file *file;
+
+	file = kmalloc(sizeof(*file), GFP_KERNEL);
+	if (!file)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&file->events);
+	INIT_LIST_HEAD(&file->ctxs);
+	init_waitqueue_head(&file->poll_wait);
+
+	init_MUTEX(&file->mutex);
+
+	filp->private_data = file;
+	file->filp = filp;
+
+	printk(KERN_ERR "UCM: Created struct\n");
+
+	return 0;
+}
+
+static int ib_ucm_close(struct inode *inode, struct file *filp)
+{
+	struct ib_ucm_file *file = filp->private_data;
+	struct ib_ucm_context *ctx;
+
+	down(&file->mutex);
+
+	while (!list_empty(&file->ctxs)) {
+
+		ctx = list_entry(file->ctxs.next,
+				 struct ib_ucm_context, file_list);
+		
+		up(&ctx->file->mutex);
+		ib_ucm_ctx_put(ctx); /* user reference */
+		down(&file->mutex);
+	}
+
+	up(&file->mutex);
+
+	kfree(file);
+
+	printk(KERN_ERR "UCM: Deleted struct\n");
+	return 0;
+}
+
+static struct file_operations ib_ucm_fops = {
+	.owner 	 = THIS_MODULE,
+	.open 	 = ib_ucm_open,
+	.release = ib_ucm_close,
+	.write 	 = ib_ucm_write,
+	.poll    = ib_ucm_poll,
+};
+
+
+static struct class_simple *ib_ucm_class;
+static struct cdev	  ib_ucm_cdev;
+
+static int __init ib_ucm_init(void)
+{
+	int result;
+
+	result = register_chrdev_region(IB_UCM_DEV, 1, "infiniband_cm");
+	if (result) {
+		printk(KERN_ERR "UCM: Error <%d> registering dev\n", result);
+		goto err_chr;
+	}
+
+	cdev_init(&ib_ucm_cdev, &ib_ucm_fops);
+
+	result = cdev_add(&ib_ucm_cdev, IB_UCM_DEV, 1);
+	if (result) {
+		printk(KERN_ERR "UCM: Error <%d> adding cdev\n", result);
+		goto err_cdev;
+	}
+
+	ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm");
+	if (IS_ERR(ib_ucm_class)) {
+		result = PTR_ERR(ib_ucm_class);
+		printk(KERN_ERR "UCM: Error <%d> creating class\n", result);
+		goto err_class;
+	}
+
+	class_simple_device_add(ib_ucm_class,
+				IB_UCM_DEV,
+				NULL,
+				"ucm");
+	
+	idr_init(&ctx_id_table);
+	init_MUTEX(&ctx_id_mutex);
+
+	return 0;
+err_class:
+	cdev_del(&ib_ucm_cdev);
+err_cdev:
+	unregister_chrdev_region(IB_UCM_DEV, 1);
+err_chr:
+	return result;
+}
+
+static void __exit ib_ucm_cleanup(void)
+{
+	class_simple_device_remove(IB_UCM_DEV);
+	class_simple_destroy(ib_ucm_class);
+	cdev_del(&ib_ucm_cdev);
+	unregister_chrdev_region(IB_UCM_DEV, 1);
+}
+
+module_init(ib_ucm_init);
+module_exit(ib_ucm_cleanup);
diff -uprN linux-2.6.13-rc2-mm1-25/drivers/infiniband/core/ucm.h linux-2.6.13-rc2-mm1-26/drivers/infiniband/core/ucm.h
-- linux-2.6.13-rc2-mm1-25/drivers/infiniband/core/ucm.h	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-26/drivers/infiniband/core/ucm.h	2005-04-22 23:23:15.000000000 -0400
@@ -0,0 +1,89 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ucm.h 2208 2005-04-22 23:24:31Z libor $
+ */
+
+#ifndef UCM_H
+#define UCM_H
+
+#include <linux/fs.h>
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/idr.h>
+
+#include <ib_cm.h>
+#include <ib_user_cm.h>
+
+#define IB_UCM_CM_ID_INVALID 0xffffffff
+
+struct ib_ucm_file {
+	struct semaphore mutex;
+	struct file *filp;
+	/*
+	 * list of pending events
+	 */
+	struct list_head  ctxs;   /* list of active connections */
+	struct list_head  events; /* list of pending events */
+	wait_queue_head_t poll_wait;
+};
+
+struct ib_ucm_context {
+	int                 id;
+	int                 ref;
+	int                 error;
+
+	struct ib_ucm_file *file;
+	struct ib_cm_id    *cm_id;
+	struct semaphore    mutex;
+
+	struct list_head    events;    /* list of pending events. */
+	struct list_head    file_list; /* member in file ctx list */
+};
+
+struct ib_ucm_event {
+	struct ib_ucm_context *ctx;
+	struct list_head file_list; /* member in file event list */
+	struct list_head ctx_list;  /* member in ctx event list */
+
+	struct ib_ucm_event_resp resp;
+	void *data;
+	void *info;
+	int data_len;
+	int info_len;
+	/*
+	 * new connection identifiers needs to be saved until
+	 * userspace can get a handle on them.
+	 */
+	struct ib_cm_id *cm_id;
+};
+
+#endif /* UCM_H */




From halr at voltaire.com  Mon Jul 11 14:02:57 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 17:02:57 -0400
Subject: [openib-general] [PATCH 27/29v2] Hook up userspace CM to the make
	system
Message-ID: <1121110436.4389.5038.camel@hal.voltaire.com>

Hook up userspace CM to the make system 
Signed-off-by: Libor Michalek <libor at topspin.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 26/29.

--
 Makefile |    4 +++-
 1 files changed, 3 insertions(+), 1 deletion(-)
diff -uprN linux-2.6.13-rc2-mm1-26/drivers/infiniband/core/Makefile linux-2.6.13-rc2-mm1-27/drivers/infiniband/core/Makefile
-- linux-2.6.13-rc2-mm1-26/drivers/infiniband/core/Makefile	2005-07-10 16:50:04.000000000 -0400
+++ linux-2.6.13-rc2-mm1-27/drivers/infiniband/core/Makefile	2005-07-10 16:59:52.000000000 -0400
@@ -1,7 +1,7 @@
 EXTRA_CFLAGS += -Idrivers/infiniband/include
 
 obj-$(CONFIG_INFINIBAND) +=		ib_core.o ib_mad.o ib_sa.o \
-					ib_cm.o ib_umad.o
+					ib_cm.o ib_umad.o ib_ucm.o
 obj-$(CONFIG_INFINIBAND_USER_VERBS) +=	ib_uverbs.o
 
 ib_core-y :=			packer.o ud_header.o verbs.o sysfs.o \
@@ -15,4 +15,6 @@ ib_cm-y :=			cm.o
 
 ib_umad-y :=			user_mad.o
 
+ib_ucm-y :=			ucm.o
+
 ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_mem.o




From halr at voltaire.com  Mon Jul 11 14:06:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 17:06:10 -0400
Subject: [openib-general] [PATCH 28/29v2] Eliminate sparse warnings in SA
	client
Message-ID: <1121110441.4389.5040.camel@hal.voltaire.com>

Eliminate sparse warnings in SA client

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch depends on patch 20/29.

-- 
 core/sa_query.c |    6 +++--
 include/ib_sa.h |   10 +++++--
 2 files changed, 8 insertions(+), 8 deletions(-)
diff -uprN linux-2.6.13-rc2-mm1-27/drivers/infiniband/core/sa_query.c linux-2.6.13-rc2-mm1-28/drivers/infiniband/core/sa_query.c
-- linux-2.6.13-rc2-mm1-27/drivers/infiniband/core/sa_query.c	2005-07-10 16:41:32.000000000 -0400
+++ linux-2.6.13-rc2-mm1-28/drivers/infiniband/core/sa_query.c	2005-07-11 13:55:40.000000000 -0400
@@ -600,7 +600,7 @@ static void ib_sa_path_rec_release(struc
 int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
 		       struct ib_sa_path_rec *rec,
 		       ib_sa_comp_mask comp_mask,
-		       int timeout_ms, int gfp_mask,
+		       int timeout_ms, unsigned int __nocast gfp_mask,
 		       void (*callback)(int status,
 					struct ib_sa_path_rec *resp,
 					void *context),
@@ -702,7 +702,7 @@ static void ib_sa_service_rec_release(st
 int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
 			    struct ib_sa_service_rec *rec,
 			    ib_sa_comp_mask comp_mask,
-			    int timeout_ms, int gfp_mask,
+			    int timeout_ms, unsigned int __nocast gfp_mask,
 			    void (*callback)(int status,
 					     struct ib_sa_service_rec *resp,
 					     void *context),
@@ -785,7 +785,7 @@ int ib_sa_mcmember_rec_query(struct ib_d
 			     u8 method,
 			     struct ib_sa_mcmember_rec *rec,
 			     ib_sa_comp_mask comp_mask,
-			     int timeout_ms, int gfp_mask,
+			     int timeout_ms, unsigned int __nocast gfp_mask,
 			     void (*callback)(int status,
 					      struct ib_sa_mcmember_rec *resp,
 					      void *context),
diff -uprN linux-2.6.13-rc2-mm1-27/drivers/infiniband/include/ib_sa.h linux-2.6.13-rc2-mm1-28/drivers/infiniband/include/ib_sa.h
-- linux-2.6.13-rc2-mm1-27/drivers/infiniband/include/ib_sa.h	2005-07-10 16:40:43.000000000 -0400
+++ linux-2.6.13-rc2-mm1-28/drivers/infiniband/include/ib_sa.h	2005-07-11 13:55:37.000000000 -0400
@@ -256,7 +256,7 @@ void ib_sa_cancel_query(int id, struct i
 int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
 		       struct ib_sa_path_rec *rec,
 		       ib_sa_comp_mask comp_mask,
-		       int timeout_ms, int gfp_mask,
+		       int timeout_ms, unsigned int __nocast gfp_mask,
 		       void (*callback)(int status,
 					struct ib_sa_path_rec *resp,
 					void *context),
@@ -267,7 +267,7 @@ int ib_sa_mcmember_rec_query(struct ib_d
 			     u8 method,
 			     struct ib_sa_mcmember_rec *rec,
 			     ib_sa_comp_mask comp_mask,
-			     int timeout_ms, int gfp_mask,
+			     int timeout_ms, unsigned int __nocast gfp_mask,
 			     void (*callback)(int status,
 					      struct ib_sa_mcmember_rec *resp,
 					      void *context),
@@ -278,7 +278,7 @@ int ib_sa_service_rec_query(struct ib_de
 			 u8 method,
 			 struct ib_sa_service_rec *rec,
 			 ib_sa_comp_mask comp_mask,
-			 int timeout_ms, int gfp_mask,
+			 int timeout_ms, unsigned int __nocast gfp_mask,
 			 void (*callback)(int status,
 					  struct ib_sa_service_rec *resp,
 					  void *context),
@@ -313,7 +313,7 @@ static inline int
 ib_sa_mcmember_rec_set(struct ib_device *device, u8 port_num,
 		       struct ib_sa_mcmember_rec *rec,
 		       ib_sa_comp_mask comp_mask,
-		       int timeout_ms, int gfp_mask,
+		       int timeout_ms, unsigned int __nocast gfp_mask,
 		       void (*callback)(int status,
 					struct ib_sa_mcmember_rec *resp,
 					void *context),
@@ -355,7 +355,7 @@ static inline int
 ib_sa_mcmember_rec_delete(struct ib_device *device, u8 port_num,
 			  struct ib_sa_mcmember_rec *rec,
 			  ib_sa_comp_mask comp_mask,
-			  int timeout_ms, int gfp_mask,
+			  int timeout_ms, unsigned int __nocast gfp_mask,
 			  void (*callback)(int status,
 					   struct ib_sa_mcmember_rec *resp,
 					   void *context),




From halr at voltaire.com  Mon Jul 11 14:09:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 17:09:23 -0400
Subject: [openib-general] [PATCH 29/29v2] Add core locking documentation to
	Infiniband
Message-ID: <1121110449.4389.5042.camel@hal.voltaire.com>

Add core locking documentation to Infiniband

Signed-off-by: Roland Dreier <rolandd at cisco.com>
Signed-off-by: Hal Rosenstock <halr at voltaire.com>

-- 
 core_locking.txt |  114 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 114 insertions(+)
diff -uprN linux-2.6.13-rc2-mm1-28/Documentation/infiniband/core_locking.txt linux-2.6.13-rc2-mm1-29/Documentation/infiniband/core_locking.txt
-- linux-2.6.13-rc2-mm1-28/Documentation/infiniband/core_locking.txt	1969-12-31 19:00:00.000000000 -0500
+++ linux-2.6.13-rc2-mm1-29/Documentation/infiniband/core_locking.txt	2005-07-09 07:33:43.000000000 -0400
@@ -0,0 +1,114 @@
+INFINIBAND MIDLAYER LOCKING
+
+  This guide is an attempt to make explicit the locking assumptions
+  made by the InfiniBand midlayer.  It describes the requirements on
+  both low-level drivers that sit below the midlayer and upper level
+  protocols that use the midlayer.
+
+Sleeping and interrupt context
+
+  With the following exceptions, a low-level driver implementation of
+  all of the methods in struct ib_device may sleep.  The exceptions
+  are any methods from the list:
+
+    create_ah
+    modify_ah
+    query_ah
+    destroy_ah
+    bind_mw
+    post_send
+    post_recv
+    poll_cq
+    req_notify_cq
+    map_phys_fmr
+
+  which may not sleep and must be callable from any context.
+
+  The corresponding functions exported to upper level protocol
+  consumers:
+
+    ib_create_ah
+    ib_modify_ah
+    ib_query_ah
+    ib_destroy_ah
+    ib_bind_mw
+    ib_post_send
+    ib_post_recv
+    ib_req_notify_cq
+    ib_map_phys_fmr
+
+  are therefore safe to call from any context.
+
+  In addition, the function
+
+    ib_dispatch_event
+
+  used by low-level drivers to dispatch asynchronous events through
+  the midlayer is also safe to call from any context.
+
+Reentrancy
+
+  All of the methods in struct ib_device exported by a low-level
+  driver must be fully reentrant.  The low-level driver is required to
+  perform all synchronization necessary to maintain consistency, even
+  if multiple function calls using the same object are run
+  simultaneously.
+
+  The IB midlayer does not perform any serialization of function calls.
+
+  Because low-level drivers are reentrant, upper level protocol
+  consumers are not required to perform any serialization.  However,
+  some serialization may be required to get sensible results.  For
+  example, a consumer may safely call ib_poll_cq() on multiple CPUs
+  simultaneously.  However, the ordering of the work completion
+  information between different calls of ib_poll_cq() is not defined.
+
+Callbacks
+
+  A low-level driver must not perform a callback directly from the
+  same callchain as an ib_device method call.  For example, it is not
+  allowed for a low-level driver to call a consumer's completion event
+  handler directly from its post_send method.  Instead, the low-level
+  driver should defer this callback by, for example, scheduling a
+  tasklet to perform the callback.
+
+  The low-level driver is responsible for ensuring that multiple
+  completion event handlers for the same CQ are not called
+  simultaneously.  The driver must guarantee that only one CQ event
+  handler for a given CQ is running at a time.  In other words, the
+  following situation is not allowed:
+
+        CPU1                                    CPU2
+
+  low-level driver ->
+    consumer CQ event callback:
+      /* ... */
+      ib_req_notify_cq(cq, ...);
+                                        low-level driver ->
+      /* ... */                           consumer CQ event callback:
+                                            /* ... */
+      return from CQ event handler
+                                        
+  The context in which completion event and asynchronous event
+  callbacks run is not defined.  Depending on the low-level driver, it
+  may be process context, softirq context, or interrupt context.
+  Upper level protocol consumers may not sleep in a callback.
+
+Hot-plug
+
+  A low-level driver announces that a device is ready for use by
+  consumers when it calls ib_register_device(), all initialization
+  must be complete before this call.  The device must remain usable
+  until the driver's call to ib_unregister_device() has returned.
+
+  A low-level driver must call ib_register_device() and
+  ib_unregister_device() from process context.  It must not hold any
+  semaphores that could cause deadlock if a consumer calls back into
+  the driver across these calls.
+
+  An upper level protocol consumer may begin using an IB device as
+  soon as the add method of its struct ib_client is called for that
+  device.  A consumer must finish all cleanup and free all resources
+  relating to a device before returning from the remove method.
+
+  A consumer is permitted to sleep in its add and remove methods.




From tduffy at sun.com  Mon Jul 11 14:19:28 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 11 Jul 2005 14:19:28 -0700
Subject: [openib-general] [PATCH 26/29v2] Add kernel portion of user CM
	implementation
In-Reply-To: <1121110427.4389.5036.camel@hal.voltaire.com>
References: <1121110427.4389.5036.camel@hal.voltaire.com>
Message-ID: <1121116768.3028.9.camel@duffman>

On Mon, 2005-07-11 at 16:59 -0400, Hal Rosenstock wrote:
> Add kernel portion of user CM implementation

Hal, does this compile?  As it doesn't seem to include the patch I sent
to openib-general changing class_simple to class, I don't think it
should work on 2.6.13-rc.

[ also, in future patch bombs, could you please set the references
header so that the message thread properly, thanks ]

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.13-rc2-openib/drivers/infiniband/core/ucm.c
===================================================================
--- linux-2.6.13-rc2-openib/drivers/infiniband/core/ucm.c	(revision 2832)
+++ linux-2.6.13-rc2-openib/drivers/infiniband/core/ucm.c	(working copy)
@@ -1339,7 +1339,7 @@ static struct file_operations ib_ucm_fop
 };
 
 
-static struct class_simple *ib_ucm_class;
+static struct class *ib_ucm_class;
 static struct cdev	  ib_ucm_cdev;
 
 static int __init ib_ucm_init(void)
@@ -1360,17 +1360,14 @@ static int __init ib_ucm_init(void)
 		goto err_cdev;
 	}
 
-	ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm");
+	ib_ucm_class = class_create(THIS_MODULE, "infiniband_cm");
 	if (IS_ERR(ib_ucm_class)) {
 		result = PTR_ERR(ib_ucm_class);
 		printk(KERN_ERR "UCM: Error <%d> creating class\n", result);
 		goto err_class;
 	}
 
-	class_simple_device_add(ib_ucm_class,
-				IB_UCM_DEV,
-				NULL,
-				"ucm");
+	class_device_create(ib_ucm_class, IB_UCM_DEV, NULL, "ucm");
 	
 	idr_init(&ctx_id_table);
 	init_MUTEX(&ctx_id_mutex);
@@ -1386,8 +1383,8 @@ err_chr:
 
 static void __exit ib_ucm_cleanup(void)
 {
-	class_simple_device_remove(IB_UCM_DEV);
-	class_simple_destroy(ib_ucm_class);
+	class_device_destroy(ib_ucm_class, IB_UCM_DEV);
+	class_destroy(ib_ucm_class);
 	cdev_del(&ib_ucm_cdev);
 	unregister_chrdev_region(IB_UCM_DEV, 1);
 }



From arlin.r.davis at intel.com  Mon Jul 11 15:37:55 2005
From: arlin.r.davis at intel.com (Davis, Arlin R)
Date: Mon, 11 Jul 2005 15:37:55 -0700
Subject: [openib-general] ib_uat kernel panic without async callback
Message-ID: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>

Hal,

I was attempting to call ib_at_route_by_ip() in sync mode and get the
following oops:  

You can change the att.c (line 193) example as follow to duplicate:

  r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
                                      att_rt + i, NULL, NULL);

I will go back to async mode for now.

Thanks,

-arlin

Jul 11 15:01:17 iclust-20 kernel: Unable to handle kernel NULL pointer
dereference at 0000000000000000 RIP:
Jul 11 15:01:17 iclust-20 kernel:
<ffffffff880a449c>{:ib_uat:ib_uat_route_by_ip+204}
Jul 11 15:01:17 iclust-20 kernel: PGD 38fa3067 PUD 2e7c7067 PMD 0
Jul 11 15:01:17 iclust-20 kernel: Oops: 0000 [1] SMP
Jul 11 15:01:17 iclust-20 kernel: CPU 0
Jul 11 15:01:17 iclust-20 kernel: Modules linked in: ib_att ib_uat ib_at
ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib ib_sa ib_mthca ib_mad ib_core
dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod
Jul 11 15:01:17 iclust-20 kernel: Pid: 6320, comm: dtestib Tainted: P
2.6.11
Jul 11 15:01:17 iclust-20 kernel: RIP: 0010:[<ffffffff880a449c>]
<ffffffff880a449c>{:ib_uat:ib_uat_route_by_ip+204}
Jul 11 15:01:17 iclust-20 kernel: RSP: 0018:ffff810030c9be78  EFLAGS:
00010282
Jul 11 15:01:17 iclust-20 kernel: RAX: 0000000000000000 RBX:
00000000fffffff4 RCX: 0000000000040000
Jul 11 15:01:17 iclust-20 kernel: RDX: 00000000ffffff01 RSI:
0000000000000292 RDI: ffffffff80453f80
Jul 11 15:01:17 iclust-20 kernel: RBP: ffff81002f342380 R08:
ffff810038d54d80 R09: ffff81003e498c80
Jul 11 15:01:17 iclust-20 kernel: R10: 00000000ffffffff R11:
0000000000000000 R12: ffff81003e498c80
Jul 11 15:01:17 iclust-20 kernel: R13: ffff810038d54d80 R14:
ffff810030c9bf50 R15: 00007ffffffff7c0
Jul 11 15:01:17 iclust-20 kernel: FS:  00002aaaaabc53e0(0000)
GS:ffffffff80580280(0000) knlGS:0000000000000000
Jul 11 15:01:17 iclust-20 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
000000008005003b
Jul 11 15:01:17 iclust-20 kernel: CR2: 0000000000000000 CR3:
0000000033975000 CR4: 00000000000006e0
Jul 11 15:01:17 iclust-20 kernel: Process dtestib (pid: 6320, threadinfo
ffff810030c9a000, task ffff810038a88b00)
Jul 11 15:01:17 iclust-20 kernel: Stack: ffff81003e890940
ffff810030c9bf70 2ca2070a1300a8c0 0000000000000000
Jul 11 15:01:17 iclust-20 kernel:        0000000000518cb8
0000000000000000 00007ffffffff210 ffffffff88099b82
Jul 11 15:01:17 iclust-20 kernel:        ffff810039167820
0000000000000030
Jul 11 15:01:17 iclust-20 kernel: Call
Trace:<ffffffff88099b82>{:ib_ucm:ib_ucm_create_id+130}
<ffffffff880a4eed>{:ib_uat:ib_uat_write+157}
Jul 11 15:01:17 iclust-20 kernel:
<ffffffff8017aca4>{vfs_write+196} <ffffffff8017ae03>{sys_write+83}
Jul 11 15:01:17 iclust-20 kernel:
<ffffffff8010e30a>{system_call+126}
Jul 11 15:01:17 iclust-20 kernel:
Jul 11 15:01:17 iclust-20 kernel: Code: 48 8b 00 48 89 45 78 48 8b 44 24
28 48 8b 40 08 48 89 85 80
Jul 11 15:01:17 iclust-20 kernel: RIP
<ffffffff880a449c>{:ib_uat:ib_uat_route_by_ip+204} RSP
<ffff810030c9be78>
Jul 11 15:01:17 iclust-20 kernel: CR2: 0000000000000000

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/ef297065/attachment.html>

From tduffy at sun.com  Mon Jul 11 15:43:14 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 11 Jul 2005 15:43:14 -0700
Subject: [openib-general] Proposal for mapping DAT_RETURN values to
	ERRNO values
In-Reply-To: <Pine.LNX.4.61.0506301042090.11230@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0506281406010.4300@jlentini-linux.nane.netapp.com>
	<9d3b7de7050629193430c53865@mail.gmail.com>
	<Pine.LNX.4.61.0506301042090.11230@jlentini-linux.nane.netapp.com>
Message-ID: <1121121794.3028.18.camel@duffman>

On Thu, 2005-06-30 at 10:44 -0400, James Lentini wrote:
> I'm going to be on vacation next week. I'll make the conversion when I 
> return on the 11th. If anyone else has comments, please send them by 
> July 10th.

Do you have a patch that implements the errno changes?  Are you going to
add in a new OUT dat_return *int parameter for kdat.h compat?  If so,
please make this a CONFIG_DAT_COMPAT.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050711/c447f634/attachment.sig>

From halr at voltaire.com  Mon Jul 11 16:16:13 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 19:16:13 -0400
Subject: [openib-general] [PATCH 26/29v2] Add kernel portion of user CM
	implementation
In-Reply-To: <1121116768.3028.9.camel@duffman>
References: <1121110427.4389.5036.camel@hal.voltaire.com>
	<1121116768.3028.9.camel@duffman>
Message-ID: <1121123772.4389.5061.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 17:19, Tom Duffy wrote:
> On Mon, 2005-07-11 at 16:59 -0400, Hal Rosenstock wrote:
> > Add kernel portion of user CM implementation
> 
> Hal, does this compile?  As it doesn't seem to include the patch I sent
> to openib-general changing class_simple to class, I don't think it
> should work on 2.6.13-rc.

I did compile and build things at each step along the way. I must have
missed some changes with 2.6.13-rc.

> [ also, in future patch bombs, could you please set the references
> header so that the message thread properly, thanks ]

How do I do this ?

-- Hal



From halr at voltaire.com  Mon Jul 11 16:21:26 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 19:21:26 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
Message-ID: <1121123963.4389.5069.camel@hal.voltaire.com>

Hi Arlin,

On Mon, 2005-07-11 at 18:37, Davis, Arlin R wrote:
> I was attempting to call ib_at_route_by_ip() in sync mode and get the
> following oops:  
> 
> You can change the att.c (line 193) example as follow to duplicate:
> 
>   r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
>                                       att_rt + i, NULL, NULL);
> 
> I will go back to async mode for now.
> 
> Thanks,
> 
> -arlin
> 
> Jul 11 15:01:17 iclust-20 kernel: Unable to handle kernel NULL pointer
> dereference at 0000000000000000 RIP:
> Jul 11 15:01:17 iclust-20 kernel:
> <ffffffff880a449c>{:ib_uat:ib_uat_route_by_ip+204}
> Jul 11 15:01:17 iclust-20 kernel: PGD 38fa3067 PUD 2e7c7067 PMD 0
> Jul 11 15:01:17 iclust-20 kernel: Oops: 0000 [1] SMP
> Jul 11 15:01:17 iclust-20 kernel: CPU 0
> Jul 11 15:01:17 iclust-20 kernel: Modules linked in: ib_att ib_uat
> ib_at ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib ib_sa ib_mthca ib_mad
> ib_core dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod
> Jul 11 15:01:17 iclust-20 kernel: Pid: 6320, comm: dtestib Tainted:
> P      2.6.11
> Jul 11 15:01:17 iclust-20 kernel: RIP: 0010:[<ffffffff880a449c>]
> <ffffffff880a449c>{:ib_uat:ib_uat_route_by_ip+204}
> Jul 11 15:01:17 iclust-20 kernel: RSP: 0018:ffff810030c9be78  EFLAGS:
> 00010282
> Jul 11 15:01:17 iclust-20 kernel: RAX: 0000000000000000 RBX:
> 00000000fffffff4 RCX: 0000000000040000
> Jul 11 15:01:17 iclust-20 kernel: RDX: 00000000ffffff01 RSI:
> 0000000000000292 RDI: ffffffff80453f80
> Jul 11 15:01:17 iclust-20 kernel: RBP: ffff81002f342380 R08:
> ffff810038d54d80 R09: ffff81003e498c80
> Jul 11 15:01:17 iclust-20 kernel: R10: 00000000ffffffff R11:
> 0000000000000000 R12: ffff81003e498c80
> Jul 11 15:01:17 iclust-20 kernel: R13: ffff810038d54d80 R14:
> ffff810030c9bf50 R15: 00007ffffffff7c0
> Jul 11 15:01:17 iclust-20 kernel: FS:  00002aaaaabc53e0(0000)
> GS:ffffffff80580280(0000) knlGS:0000000000000000
> Jul 11 15:01:17 iclust-20 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Jul 11 15:01:17 iclust-20 kernel: CR2: 0000000000000000 CR3:
> 0000000033975000 CR4: 00000000000006e0
> Jul 11 15:01:17 iclust-20 kernel: Process dtestib (pid: 6320,
> threadinfo ffff810030c9a000, task ffff810038a88b00)
> Jul 11 15:01:17 iclust-20 kernel: Stack: ffff81003e890940
> ffff810030c9bf70 2ca2070a1300a8c0 0000000000000000
> Jul 11 15:01:17 iclust-20 kernel:        0000000000518cb8
> 0000000000000000 00007ffffffff210 ffffffff88099b82
> Jul 11 15:01:17 iclust-20 kernel:        ffff810039167820
> 0000000000000030
> Jul 11 15:01:17 iclust-20 kernel: Call
> Trace:<ffffffff88099b82>{:ib_ucm:ib_ucm_create_id+130}
> <ffffffff880a4eed>{:ib_uat:ib_uat_write+157}
> Jul 11 15:01:17 iclust-20 kernel:       
> <ffffffff8017aca4>{vfs_write+196} <ffffffff8017ae03>{sys_write+83}
> Jul 11 15:01:17 iclust-20 kernel:       
> <ffffffff8010e30a>{system_call+126}
> Jul 11 15:01:17 iclust-20 kernel:
> Jul 11 15:01:17 iclust-20 kernel: Code: 48 8b 00 48 89 45 78 48 8b 44
> 24 28 48 8b 40 08 48 89 85 80
> Jul 11 15:01:17 iclust-20 kernel: RIP
> <ffffffff880a449c>{:ib_uat:ib_uat_route_by_ip+204} RSP
> <ffff810030c9be78>
> Jul 11 15:01:17 iclust-20 kernel: CR2: 0000000000000000

How was IPoIB configured ? Was ib0 on the 192.168.0.x subnet ?

What do you mean by "back to async mode" ? Did you use async mode before
?

-- Hal





From tduffy at sun.com  Mon Jul 11 16:30:37 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 11 Jul 2005 16:30:37 -0700
Subject: [openib-general] [PATCH 26/29v2] Add kernel portion of user CM
	implementation
In-Reply-To: <1121123772.4389.5061.camel@hal.voltaire.com>
References: <1121110427.4389.5036.camel@hal.voltaire.com>
	<1121116768.3028.9.camel@duffman>
	<1121123772.4389.5061.camel@hal.voltaire.com>
Message-ID: <1121124637.3028.30.camel@duffman>

[ un-cc lkml ]

On Mon, 2005-07-11 at 19:16 -0400, Hal Rosenstock wrote:
> > [ also, in future patch bombs, could you please set the references
> > header so that the message thread properly, thanks ]
> 
> How do I do this ?

Attached is the script I use to sent patchbombs.  Thanks to Greg Kroah
and Roland Dreier.

-tduffy

#!/usr/bin/perl -w

# horrible hack of a script to send off a large number of email messages,
# one after each other, all chained together.  This is useful for large
# numbers of patches.
#
# Use at your own risk!!!!
#
# greg kroah-hartman Jan 8, 2002
# <greg at kroah.com>
#
# modifications by Roland Dreier <roland at topspin.com> June 30, 2004 
# changes by Tom Duffy <tduffy at sun.com> July 15, 2004
# 	- include references header for better threading
#
# Released under the artistic license.
#

#
# modify these options each time you run the script
#
$to = 'openib-general at openib.org';
#$to = 'tomduffy at gmail.com';
#$to = 'tduffy at sun.com';

# If you want to chain the first post, fill this in
$initial_reply_to = '';

# a list of patches to send out
@files = (
	['0-message','Use Linux standard linked lists'],
	['dat1','convert the cr list to linux native'],
	['dat2','convert the lmr list to linux native'],
	['dat3','convert the rmr list to linux native'],
	['dat4','convert the pz list to linux native'],
	['dat5','convert the evd list to linux native'],
	['dat6','convert the psp and rsp lists to linux native'],
	['dat7','convert the srq list to linux native'],
	['dat8','convert the hcas ia list to linux native'],
	['dat9','remove all vestiges of dapls linked list implementation'],
);

# count to start subject line [PATCH][n/N] part from
$initial_count = 0;

# Put your name and address here
$from = 'Tom Duffy <tduffy at sun.com>';

# Don't need to change anything below here...

use Mail::Sendmail;

# we make a "fake" message id by taking the current number
# of seconds since the beginning of Unix time and tacking on
# a random number to the end, in case we are called quicker than
# 1 second since the last time we were called.
sub make_message_id
{
        my $date = `date "+\%s"`;
        chomp($date);
        my $pseudo_rand = int (rand(4200));
        $message_id = "<$date$pseudo_rand\@sun.com>";
        print "new message id = $message_id\n";
}

sub send_message
{
        %mail = (       To      =>      $to,
                        From    =>      $from,
                        Subject =>      $subject,
                        Message =>      $message,
                        'In-Reply-To'   =>      $reply_to,
			'References'	=>	$references,
                        'Message-ID'    =>      $message_id,
                        'X-Mailer'      =>      "Evolution 2.2.2",
			'Content-Transfer-Encoding' => "8bit",
                );

        $mail{smtp} = 'localhost';

        sendmail(%mail) or die $Mail::Sendmail::error;

        print "OK. Log says:\n", $Mail::Sendmail::log;
        print "\n\n"
}


$reply_to = $initial_reply_to;
$references = $reply_to;
make_message_id();
$count = $initial_count;
$tot = $#files + $initial_count;

foreach $t (@files) {
        $F = $$t[0];
        open F or die "can't open file $$t[0]";
        undef $/;
        $message = <F>; # slurp the whole file in
        close F;
        $/ = "\n";
        $subject = "[PATCH $count/$tot] kDAPL: " . $$t[1];
        send_message();

#	sleep 10;

        # set up for the next message
        $reply_to = $message_id;
	$references = $references . " " . $message_id;
        make_message_id();
        ++$count;
}



From arlin.r.davis at intel.com  Mon Jul 11 17:03:27 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Mon, 11 Jul 2005 17:03:27 -0700
Subject: [openib-general] RE: ib_uat kernel panic without async callback
In-Reply-To: <1121123963.4389.5069.camel@hal.voltaire.com>
Message-ID: <ORSMSX401M3SOlbpH7100000036@orsmsx401.amr.corp.intel.com>

Hal,

>-----Original Message-----
>From: Hal Rosenstock [mailto:halr at voltaire.com]
>Sent: Monday, July 11, 2005 4:21 PM
>To: Davis, Arlin R
>Cc: openib-general at openib.org
>Subject: Re: ib_uat kernel panic without async callback
>
>Hi Arlin,
>
>On Mon, 2005-07-11 at 18:37, Davis, Arlin R wrote:
>> I was attempting to call ib_at_route_by_ip() in sync mode and get the
>> following oops:
>>
>> You can change the att.c (line 193) example as follow to duplicate:
>>
>>   r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
>>                                       att_rt + i, NULL, NULL);
>>
>> I will go back to async mode for now.
>>
>> Thanks,
>>
>> -arlin
>
>How was IPoIB configured ? Was ib0 on the 192.168.0.x subnet ?
>

ib0       Link encap:UNSPEC  HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:192.168.0.19  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c902:0:409d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:128
          RX bytes:1204 (1.1 KiB)  TX bytes:1108 (1.0 KiB)

>What do you mean by "back to async mode" ? Did you use async mode before
>?

Yes, I used async mode similar to the example before trying the call synchronously. If I provide the
async_comp with the ib_at_route_by_ip() followed by ib_at_callback_get() then it seems to work. If
async_comp == NULL then it blows up in the ib_at_route_by_ip call.

-arlin

>
>-- Hal
>
>




From akpm at osdl.org  Mon Jul 11 17:05:48 2005
From: akpm at osdl.org (Andrew Morton)
Date: Mon, 11 Jul 2005 17:05:48 -0700
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <1121110249.4389.4984.camel@hal.voltaire.com>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
Message-ID: <20050711170548.31605e23.akpm@osdl.org>

Hal Rosenstock <halr at voltaire.com> wrote:
>
> This is version 2 of a patch series to get the Infiniband core up to
> date.

Well that was interesting.

- All the patches had mangled headers:

-- linux-2.6.13-rc2-mm1-16/...
+++ linux-2.6.13-rc2-mm1-17/...

  instead of

--- linux-2.6.13-rc2-mm1-16/...
+++ linux-2.6.13-rc2-mm1-17/...

  Which I fixed up.

- The second patch didn't apply.  I fixed that too.

- The patches add lots of trailing whitespace.  I habitually trim that
  off, figuring that any downstream merging hassle which that causes is
  just punishment ;)


Please check that it all landed OK in the next -mm.

I've hung on to Tom Duffy's patch pending a test compilation.

I'll tentatively consider this material to be not-for-2.6.13?



From halr at voltaire.com  Mon Jul 11 19:45:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 11 Jul 2005 22:45:31 -0400
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <20050711170548.31605e23.akpm@osdl.org>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
Message-ID: <1121136330.4389.5093.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 20:05, Andrew Morton wrote:
> Hal Rosenstock <halr at voltaire.com> wrote:
> >
> > This is version 2 of a patch series to get the Infiniband core up to
> > date.
> 
> Well that was interesting.
> 
> - All the patches had mangled headers:
> 
> -- linux-2.6.13-rc2-mm1-16/...
> +++ linux-2.6.13-rc2-mm1-17/...
> 
>   instead of
> 
> --- linux-2.6.13-rc2-mm1-16/...
> +++ linux-2.6.13-rc2-mm1-17/...

Not sure how that happened.

>   Which I fixed up.

Thanks.

> - The second patch didn't apply.  I fixed that too.

Not sure how this was broken. Thanks for fixing this.

> - The patches add lots of trailing whitespace.  I habitually trim that
>   off, figuring that any downstream merging hassle which that causes is
>   just punishment ;)

I'll work on merging this back downstream :-(

> Please check that it all landed OK in the next -mm.

Will do.

> I've hung on to Tom Duffy's patch pending a test compilation.
> 
> I'll tentatively consider this material to be not-for-2.6.13?

Presuming that "this material" refers to the patch to add the kernel CM
implementation, if kernel CM does not make 2.6.13, then user CM should
not either as it is dependent on it.

-- Hal



From hch at lst.de  Mon Jul 11 19:54:34 2005
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 12 Jul 2005 04:54:34 +0200
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
Message-ID: <20050712025434.GA31175@lst.de>

On Mon, Jul 11, 2005 at 09:47:04AM -0700, Caitlin Bestler wrote:
> The attached .h files follow the alternate approach where each
> transport can define its own specifics in a supplemental .h file
> and avoid the massive union approach.
> 
> This is mostly cosmetic, except for the work request. Because
> some work request opcodes are transport specific the work request
> itself has to follow the same approach of deferred sub-typing
> rather than unioning.
> 
> The impact of this on the existing code would be quite wide,
> but quite shallow. Many types would be renamed form 'ib_xyz'
> to 'rdma_xyz', if being used in transport neutral code. Or
> when staying infiniband specific fields that now qualified
> as transport neutral would require "common." to be inserted
> before the field name. Some constants are shifted from IB_XXX
> to RDMA_XXX.
> 
> The goal of these changes is to allow users of the core
> services to code *most* of their logic without having to
> reference transport specific fields.

This looks very nice to me, one nitpick:  I think ibonly_verbs.h should
be called just ib_verbs.h



From akpm at osdl.org  Mon Jul 11 20:11:17 2005
From: akpm at osdl.org (Andrew Morton)
Date: Mon, 11 Jul 2005 20:11:17 -0700
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <1121136330.4389.5093.camel@hal.voltaire.com>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
Message-ID: <20050711201117.72539977.akpm@osdl.org>

Hal Rosenstock <halr at voltaire.com> wrote:
>
> > I'll tentatively consider this material to be not-for-2.6.13?
> 
>  Presuming that "this material" refers to the patch to add the kernel CM
>  implementation, if kernel CM does not make 2.6.13, then user CM should
>  not either as it is dependent on it.

Well I was asking.  Do you guys think that this material is appropriate to
and safe enough for 2.6.13?

What are "user CM" and "kernel CM"?


From davem at davemloft.net  Mon Jul 11 20:12:25 2005
From: davem at davemloft.net (David S. Miller)
Date: Mon, 11 Jul 2005 20:12:25 -0700 (PDT)
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <1121136330.4389.5093.camel@hal.voltaire.com>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
Message-ID: <20050711.201225.34760317.davem@davemloft.net>


Please acknowledge that you understand how inappropriate
and problem causing your huge patch bomb was today to this
mailing list.

It is nearly 8 hours later, and vger.kernel.org is still
trying mightily to spit out all of your patches to the
5000+ people subscribed to linux-kernel.

There is about ~6 hour posting latency as a result of all
of this.


From sato at make-love.cx  Mon Jul 11 21:26:54 2005
From: sato at make-love.cx (make-love  )
Date: Tue, 12 Jul 2005 13:26:54 +0900
Subject: [openib-general] =?iso-2022-jp?b?GyRCOkZFWSQ0QXc/LiQ1JDsbKEI=?=
	=?iso-2022-jp?b?GyRCJEZEOiQtJF4kOSEjGyhC?=
Message-ID: <20050712111634.281.MAIL>

先日は私共の手違いによりメールが届いていない方が多数いらっしゃったそうですので
この場を借りてお詫び申し上げます。

今後はこのようなことがないよう全力で努めさせて頂く所存ですのでこれからも
よろしくお願い致します。

今回の企画は『身元の確かな出会いに縁のない女性会員様と
誠実で秘密厳守を守れる男性会員様』が出会うという素晴らしい企画でございます。

先日会員様を募集したところ、8000名以上の女性会員の方が集いました。
しかし残念なことに男性会員様は700名という形で終りました。
そこで今回は男性会員様を大募集致します。

Make-up Loveでは女性会員様の会費で運営されております。
お会いする場合にはくれぐれも失礼のないようお願い致します。
また男性会員様におきまして登録料・紹介料は一切頂いておりません。

当クラブには様々な女性会員様が在籍しております。
学生、ＯＬ、フリーター、ニート、セレブ、淫乱マダム、ＳＭ、３Ｐ、合コン
などなど夢のようです。


日本全国に女性会員様が多数在籍しておりますので、
まずは貴方様の地域に該当している女性会員様をご紹介させて頂きたいと思います。

簡単なシートですので、まずはお気軽にお書き下さいませ。


Ｑ：ＮＯ,１
　　貴方様のお会いするにあたってのご希望の地域(都道府県等)


Ｑ：ＮＯ,２
 　　当クラブをご利用するにあたってのご利用タイプ
【タイプ1】真面目に一般女性とお付き合い　　（真面目な交際・結婚前提） 
【タイプ2】女性スポンサーとお付き合い　　　（事業資金などサポートして欲しい） 
【タイプ3】割り切った大人のお付き合い　　　（愛人・セフレ・人妻） 
【タイプ4】多趣味にそして過激なお付き合い　（3P、乱交・ＳＭ） 
【タイプ特別】40歳以上同士の新しい交際



以上のご質問にお答え頂きご返信頂ければ、当クラブの各地域担当女性から
貴方様のご希望に沿った女性をご案内させて頂きます。

メール会報に登録した覚えがないという方や同様のメールが届いていると
いう場合もあるかと思いますがご了承下さい。

当クラブ「Make-up Love」の会報メールがご不要な場合にはそのまま破棄してください。
尚、「配信不要」とお送りいただければ会報メールは二度と配信致しません。

ここまで読んで「嘘くさい」「配信不要」と思った方々。
少々お考え下さい。
ここで削除、配信不要とすることは簡単です。
しかし、こんな機会は一生に一度あるかどうかと思われます。

やはりお金の心配をなさっている方もいるのではないでしょうか？
皆様の喜ぶの声を聞かせて頂ければ、それだけで充分でございます。

最後になりますが、「配信不要」と書こうと思った方も
どうか地域、ご利用のタイプをご明記ください。
直ちに女性会員の方々をご紹介させて頂きます。

ご連絡お待ちしております。

 『club　Make-up Love【メイク・ラブ】』代表　藁科美智子




From eitan at mellanox.co.il  Mon Jul 11 22:28:03 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 12 Jul 2005 08:28:03 +0300
Subject: [openib-general] Re: Gen2 build/update process
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3048E@mtlex01.yok.mtl.com>

I propose to build a top level auto-tools project that will call the sub
directories recursively. In the meanwhile it is just a convinience issue

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
> Sent: Monday, July 11, 2005 11:11 PM
> To: Bernhard Fischer
> Cc: Jason 'vanRijn' Kasper; Sayantan Sur; openib-general at openib.org
> Subject: [openib-general] Re: Gen2 build/update process
> 
> Quoting r. Bernhard Fischer <rep.nop at aon.at>:
> > Subject: Re: Gen2 build/update process
> >
> > On Mon, Jul 11, 2005 at 12:10:48PM -0700, Tom Duffy wrote:
> > >Sayantan Sur wrote:
> > >
> > >>Hi,
> > >>
> > >>Is there any plan to have any formal install procedure instead of the
> > >>current process of going into subdirectories and building?
> > >>
> > >>If someone has build scripts to share, that'll be real nice.
> > >>
> > >>
> > >
> > >What are you trying to build?  Kernel build should not need to do this.
> >
> > .. and src/userspace/management has a 'automake' target.
> >
> > In case it's helpful for you, i'm attaching a simple script which can
> > be easily extended to cover the rest. I usually call it like this:
> > nohup ./mk.openib2.usr.sh 01a
> > It's not pretty but you get the idea.
> >
> > hth,
> > Bernhard
> 
> My scripts are simpler: e.g. for management (I have one for libibverbs
too):
> 
> #!/bin/sh -x
> #1. In libibcommon, libibumad, and libibmad (in that order), run:
> ( cd libibcommon && ./autogen.sh && ./configure && make && make install )
> ( cd libibumad && ./autogen.sh && ./configure && make && make install )
> ( cd libibmad && ./autogen.sh && ./configure && make && make install )
> 
> #2. In osm/complib and osm/libvendor, run:
> ( cd osm/complib && ./autogen.sh && ./configure && make && make install )
> ( cd osm/libvendor && ./autogen.sh && ./configure && make && make install
)
> #3. In all util/mad_test, diags, and osm/opensm subdirectories, run:
> ( cd util/mad_test && ./autogen.sh && ./configure )
> ( cd diags && ./autogen.sh && ./configure )
> ( cd osm/opensm && ./autogen.sh && ./configure )
> #4. At top level of management, run:
> ( make && make install )
> 
> --
> MST
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050712/0e0212ee/attachment.html>

From eitan at mellanox.co.il  Mon Jul 11 22:34:52 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Tue, 12 Jul 2005 08:34:52 +0300
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3048F@mtlex01.yok.mtl.com>

Hi Hal

I plan to do the development of IBDM and IBMgtSim in OpenIB.

I also think they should be moved to the main gen2 trunk.

I will deal with the errors today.

Eitan Zahavi


> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com]
> Sent: Monday, July 11, 2005 6:26 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: [Fwd: [openib-commits] r2826 - in
gen2/utils/src/linux-user/IBMgtSim: .
> config src tests utils]
> 
> Hi Eitan,
> 
> A few nits with the latest:
> 
> 1. Still get:
> g++ -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include -
> I/home/hal/openib/trunk/src/userspace/management/osm/include
-I/usr/include -
> I../datamodel   -O6 -Wall -Werror  -c -o sim.o `test -f 'sim.cpp' || echo
'./'`sim.cpp
> sim.cpp:342: parse error at end of input
> 
> 2. Still uneeded executable permissions on a number of simulator and
> ibdm files.
> 
> Some questions:
> 
> 1. What is BUILD_STANDALONE_SIM for ?
> 
> 2. Will you now be doing your development in OpenIB for ibdm and the
> simulator or will this be done elsewhere and dropped in here ?
> 
> Thanks.
> 
> -- Hal
> 
> --Forwarded Message--
> 
> From: eitan at openib.org
> To: openib-commits at openib.org
> Subject: [openib-commits] r2826 - in gen2/utils/src/linux-user/IBMgtSim: .
config src
> tests utils
> Date: 10 Jul 2005 02:00:57 -0700
> 
> Author: eitan
> Date: 2005-07-10 02:00:54 -0700 (Sun, 10 Jul 2005)
> New Revision: 2826
> 
> Added:
>    gen2/utils/src/linux-user/IBMgtSim/src/inttypes.i
>    gen2/utils/src/linux-user/IBMgtSim/src/mads.i
>    gen2/utils/src/linux-user/IBMgtSim/src/pma.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/pma.h
>    gen2/utils/src/linux-user/IBMgtSim/src/randmgr.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/randmgr.h
>    gen2/utils/src/linux-user/IBMgtSim/src/vsa.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/vsa.h
>    gen2/utils/src/linux-user/IBMgtSim/tests/
>    gen2/utils/src/linux-user/IBMgtSim/tests/Gnu16NodeOsmTest.topo
>    gen2/utils/src/linux-user/IBMgtSim/tests/Makefile.am
>    gen2/utils/src/linux-user/IBMgtSim/tests/OsmTest.check.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/OsmTest.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/README
>    gen2/utils/src/linux-user/IBMgtSim/tests/ReindeerBased128Nodes.topo
>    gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBased10K.topo
>    gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBased512Nodes.topo
>    gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBasedCreditLoop.topo
>    gen2/utils/src/linux-user/IBMgtSim/tests/RhinoReindeer1100.topo
>    gen2/utils/src/linux-user/IBMgtSim/tests/discover.check.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/discover.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/duplicateGuid.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/init.test.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osmLidAssignment.check.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osmLidAssignment.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticast.sim.tcl
>
gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticastRoutingTest.check.tcl
>
gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticastRoutingTest.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osmStability.check.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osmStability.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osm_no_quit.check.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/osm_no_quit.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/pkey.check.tcl
>    gen2/utils/src/linux-user/IBMgtSim/tests/pkey.sim.tcl
>    gen2/utils/src/linux-user/IBMgtSim/utils/LICENSE.txt
>    gen2/utils/src/linux-user/IBMgtSim/utils/fprogbar.sh
>    gen2/utils/src/linux-user/IBMgtSim/utils/install.sh
>    gen2/utils/src/linux-user/IBMgtSim/utils/uninstall.sh
> Modified:
>    gen2/utils/src/linux-user/IBMgtSim/INSTALL
>    gen2/utils/src/linux-user/IBMgtSim/Makefile.am
>    gen2/utils/src/linux-user/IBMgtSim/README
>    gen2/utils/src/linux-user/IBMgtSim/config/bootstrap
>    gen2/utils/src/linux-user/IBMgtSim/config/osm.m4
>    gen2/utils/src/linux-user/IBMgtSim/configure.in
>    gen2/utils/src/linux-user/IBMgtSim/src/Makefile.am
>    gen2/utils/src/linux-user/IBMgtSim/src/client.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/dispatcher.h
>    gen2/utils/src/linux-user/IBMgtSim/src/helper.h
>    gen2/utils/src/linux-user/IBMgtSim/src/ib_types.i
>    gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i
>    gen2/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h
>    gen2/utils/src/linux-user/IBMgtSim/src/ibmssh_wrap.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/msgmgr.h
>    gen2/utils/src/linux-user/IBMgtSim/src/node.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/node.h
>    gen2/utils/src/linux-user/IBMgtSim/src/server.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/server.h
>    gen2/utils/src/linux-user/IBMgtSim/src/sim.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/sim.h
>    gen2/utils/src/linux-user/IBMgtSim/src/sim.i
>    gen2/utils/src/linux-user/IBMgtSim/src/simmsg.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/simmsg.h
>    gen2/utils/src/linux-user/IBMgtSim/src/sma.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/sma.h
>    gen2/utils/src/linux-user/IBMgtSim/src/swig_alternate_mangling.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/tcpcomm.cpp
>    gen2/utils/src/linux-user/IBMgtSim/src/tcpcomm.h
>    gen2/utils/src/linux-user/IBMgtSim/utils/IBMgtSim
>    gen2/utils/src/linux-user/IBMgtSim/utils/Makefile.am
>    gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest
> Log:
> Major update:
> * Introduce RunSimTest and tests directory with a set of example tests
>   and a simple way to run them
> * PMA integrated into the simulator
> * VSA integrated into the simulator
> * All randomization in test flows should be done through a thread safe
>   randomization angine
> 
> Many bug fixes.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050712/7525ed16/attachment.html>

From caitlinb at siliquent.com  Mon Jul 11 23:24:47 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Mon, 11 Jul 2005 23:24:47 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057DA4@fiona.siliquent.com>

 
> 
> This looks very nice to me, one nitpick:  I think 
> ibonly_verbs.h should be called just ib_verbs.h
> 
>

Agreed. I should have commented on that. I used
the 'ibonly_verbs.h' as a stand-in so I could 
borrow the existing make when I tested. I didn't
want to pre-empt 'ib_verbs.h' yet.
 


From halr at voltaire.com  Tue Jul 12 02:11:03 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 05:11:03 -0400
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <20050711.201225.34760317.davem@davemloft.net>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
	<20050711.201225.34760317.davem@davemloft.net>
Message-ID: <1121159462.4389.5108.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 23:12, David S. Miller wrote:
> Please acknowledge that you understand how inappropriate
> and problem causing your huge patch bomb was today to this
> mailing list.
> 
> It is nearly 8 hours later, and vger.kernel.org is still
> trying mightily to spit out all of your patches to the
> 5000+ people subscribed to linux-kernel.
> 
> There is about ~6 hour posting latency as a result of all
> of this.

I understand the problem it caused. The MAD layer of InfiniBand was 
quite a bit behind and we wanted to catch up. I tried to make them in
smaller manageable pieces and that was why there were so many patches in
the series. In the future, I will endeavor to make the changes smaller.
Sorry for the problems this caused.

-- Hal



From halr at voltaire.com  Tue Jul 12 02:43:09 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 05:43:09 -0400
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <20050711201117.72539977.akpm@osdl.org>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
	<20050711201117.72539977.akpm@osdl.org>
Message-ID: <1121160475.4389.5154.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 23:11, Andrew Morton wrote:
> Well I was asking.  Do you guys think that this material is appropriate to
> and safe enough for 2.6.13?

We think so.

> What are "user CM" and "kernel CM"?

CM is the InfiniBand Communications Manager. It is (primarily)
responsible for setting up and managing connections between endpoints. 
Connections. IB communication occurs over the queue pairs (QPs) set up
by the CM protocol.

kernel CM is CM functionality in the kernel. user CM is to enable
userspace applications to utilize this functionality.

-- Hal



From halr at voltaire.com  Tue Jul 12 02:49:44 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 05:49:44 -0400
Subject: [openib-general] RE: ib_uat kernel panic without async callback
In-Reply-To: <ORSMSX401M3SOlbpH7100000036@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401M3SOlbpH7100000036@orsmsx401.amr.corp.intel.com>
Message-ID: <1121161137.4389.5167.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 20:03, Arlin Davis wrote:
> Hal,
> 
> >-----Original Message-----
> >From: Hal Rosenstock [mailto:halr at voltaire.com]
> >Sent: Monday, July 11, 2005 4:21 PM
> >To: Davis, Arlin R
> >Cc: openib-general at openib.org
> >Subject: Re: ib_uat kernel panic without async callback
> >
> >Hi Arlin,
> >
> >On Mon, 2005-07-11 at 18:37, Davis, Arlin R wrote:
> >> I was attempting to call ib_at_route_by_ip() in sync mode and get the
> >> following oops:
> >>
> >> You can change the att.c (line 193) example as follow to duplicate:
> >>
> >>   r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
> >>                                       att_rt + i, NULL, NULL);

BTW, did this compile ? I think there is 1 too many NULLs here.

> >> I will go back to async mode for now.
> >>
> >> Thanks,
> >>
> >> -arlin
> >
> >How was IPoIB configured ? Was ib0 on the 192.168.0.x subnet ?
> >
> 
> ib0       Link encap:UNSPEC  HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
>           inet addr:192.168.0.19  Bcast:192.168.0.255  Mask:255.255.255.0
>           inet6 addr: fe80::202:c902:0:409d/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
>           RX packets:19 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:128
>           RX bytes:1204 (1.1 KiB)  TX bytes:1108 (1.0 KiB)
> 
> >What do you mean by "back to async mode" ? Did you use async mode before
> >?
> 
> Yes, I used async mode similar to the example before trying the call synchronously. If I provide the
> async_comp with the ib_at_route_by_ip() followed by ib_at_callback_get() then it seems to work. If
> async_comp == NULL then it blows up in the ib_at_route_by_ip call.

I understand. I was confused by the use of sync and async here. It is
just that the async callback is not supplied. My bad. I'll look into it
and get back on this.

-- Hal



From glebn at voltaire.com  Tue Jul 12 04:48:46 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Tue, 12 Jul 2005 14:48:46 +0300
Subject: [openib-general] [PATCH] crash in ibv_destroy_ah
Message-ID: <20050712114846.GG9125@minantech.com>

Hello,

Included patch fixes one potential crash and one real one in ibv_destroy_ah.


Index: src/userspace/libmthca/src/verbs.c
===================================================================
--- src/userspace/libmthca/src/verbs.c	(revision 2833)
+++ src/userspace/libmthca/src/verbs.c	(working copy)
@@ -440,7 +440,7 @@ int mthca_destroy_ah(struct ibv_ah *ah)
 {
 	mthca_free_av(to_mah(ah));
 
-	free(ah);
+	free(to_mah(ah));
 	return 0;
 }
 
Index: src/userspace/libmthca/src/ah.c
===================================================================
--- src/userspace/libmthca/src/ah.c	(revision 2833)
+++ src/userspace/libmthca/src/ah.c	(working copy)
@@ -126,6 +126,7 @@ int mthca_alloc_av(struct mthca_pd *pd, 
 			}
 
 		ah->key = page->mr->lkey;
+		ah->page = page;
 
 		pthread_mutex_unlock(&pd->ah_mutex);
 	}
--
			Gleb.


From jlentini at netapp.com  Tue Jul 12 07:54:16 2005
From: jlentini at netapp.com (James Lentini)
Date: Tue, 12 Jul 2005 10:54:16 -0400 (EDT)
Subject: [openib-general] Proposal for mapping DAT_RETURN values to ERRNO
	values
In-Reply-To: <1121121794.3028.18.camel@duffman>
References: <Pine.LNX.4.61.0506281406010.4300@jlentini-linux.nane.netapp.com>
	<9d3b7de7050629193430c53865@mail.gmail.com> 
	<Pine.LNX.4.61.0506301042090.11230@jlentini-linux.nane.netapp.com>
	<1121121794.3028.18.camel@duffman>
Message-ID: <Pine.LNX.4.61.0507121038370.25603@jlentini-linux.nane.netapp.com>



On Mon, 11 Jul 2005, Tom Duffy wrote:

> On Thu, 2005-06-30 at 10:44 -0400, James Lentini wrote:
>> I'm going to be on vacation next week. I'll make the conversion when I
>> return on the 11th. If anyone else has comments, please send them by
>> July 10th.
>
> Do you have a patch that implements the errno changes?  Are you going to
> add in a new OUT dat_return *int parameter for kdat.h compat?  If so,
> please make this a CONFIG_DAT_COMPAT.

I've committed the update in revision 2841. Let me know if you have 
any problems.

I'm thinking of ways to unobtrusively preserve the details contained 
in the old error code style. I'll have to give the solution you've 
described some more thought.


From halr at voltaire.com  Tue Jul 12 07:57:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 10:57:23 -0400
Subject: [openib-general] RE: ib_uat kernel panic without async callback
In-Reply-To: <1121161137.4389.5167.camel@hal.voltaire.com>
References: <ORSMSX401M3SOlbpH7100000036@orsmsx401.amr.corp.intel.com>
	<1121161137.4389.5167.camel@hal.voltaire.com>
Message-ID: <1121180243.4369.43.camel@hal.voltaire.com>

On Tue, 2005-07-12 at 05:49, Hal Rosenstock wrote:

> > >> You can change the att.c (line 193) example as follow to duplicate:
> > >>
> > >>   r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
> > >>                                       att_rt + i, NULL, NULL);
> 
> BTW, did this compile ? I think there is 1 too many NULLs here.

I had the wrong att.c (kernel one rather than user example). I see what
you are referring to now (and at least see the first level problem).

-- Hal



From halr at voltaire.com  Tue Jul 12 08:45:18 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 11:45:18 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
Message-ID: <1121183117.4369.97.camel@hal.voltaire.com>

Hi again Arlin,

On Mon, 2005-07-11 at 18:37, Davis, Arlin R wrote:
> I was attempting to call ib_at_route_by_ip() in sync mode and get the
> following oops:  
> 
> You can change the att.c (line 193) example as follow to duplicate:
> 
>   r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
>                                       att_rt + i, NULL, NULL);

I just checked in a new version of uat.c and also the userspace library
(libibat/src/at.c) which should fix this oops.

Note that I did not unit test not supplying async completion routine (as
you noticed) so there may be further issues here :-(

> I will go back to async mode for now.

This is the safer option right now.

-- Hal



From mst at mellanox.co.il  Tue Jul 12 10:55:14 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 12 Jul 2005 20:55:14 +0300
Subject: [openib-general] Re: Gen2 build/update process
In-Reply-To: <20050711190308.GA7758@cse.ohio-state.edu>
References: <20050711190308.GA7758@cse.ohio-state.edu>
Message-ID: <20050712175514.GE5793@mellanox.co.il>

Quoting r. Sayantan Sur <surs at cse.ohio-state.edu>:
> Subject: Gen2 build/update process
> 
> Hi,
> 
> Is there any plan to have any formal install procedure instead of the
> current process of going into subdirectories and building?
> 
> If someone has build scripts to share, that'll be real nice.
> 
> TIA,
> Sayantan.
> 

Check out here
https://openib.org/svn/trunk/contrib/mellanox/scripts

there are also 2 scripts to load/unload all modules.

-- 
MST


From sean.hefty at intel.com  Tue Jul 12 11:53:06 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 11:53:06 -0700
Subject: [openib-general] userspace event reporting
Message-ID: <ORSMSX401IiGXix4A2D0000003c@orsmsx401.amr.corp.intel.com>

This fell out of the uCM connection ID discussion...

There's an issue reporting events to userspace clients for an object that a user
may have destroyed.  The problem exists with user verbs, but is much more likely
to be seen by a userspace CM client.  To avoid reporting events for a destroyed
object, I think that something similar to the following could be used from
userspace:

destroy() should set a state marking destruction and wait for a reference count
to go to 0 before transitioning to the kernel.  The kernel code should destroy
the associated kernel object and then discard any unclaimed events.

get_event() should check the object state and discard the event if the object is
being destroyed.  It should increment a reference count before reporting any
events.

put_event() should decrement the reference count and unblock destroy if the
reference count goes to 0.

I'm at least a week away from trying to work on a patch for this, but I think
that this would work.  (I'm not even sure that we need to set a destroying
state.)  We could then provide user context that could be safely accessed from
an event handler, as long as the user frees the related object after destroy
returns.  Comments?

- Sean



From ftillier at silverstorm.com  Tue Jul 12 12:12:57 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Tue, 12 Jul 2005 12:12:57 -0700
Subject: [openib-general] userspace event reporting
In-Reply-To: <ORSMSX401IiGXix4A2D0000003c@orsmsx401.amr.corp.intel.com>
Message-ID: <001d01c58715$b75681d0$6501a8c0@infiniconsys.com>

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> Sent: Tuesday, July 12, 2005 11:53 AM
> 
> This fell out of the uCM connection ID discussion...
> 
> There's an issue reporting events to userspace clients for an object that a
> user
> may have destroyed.  The problem exists with user verbs, but is much more
> likely
> to be seen by a userspace CM client.  To avoid reporting events for a
> destroyed
> object, I think that something similar to the following could be used from
> userspace:

I had to put something like this in place in the Windows CM, except that I had
to use async destroy semantics to allow preserving the existing CM API.

> destroy() should set a state marking destruction and wait for a reference
> count
> to go to 0 before transitioning to the kernel.  The kernel code should destroy
> the associated kernel object and then discard any unclaimed events.

You don't need a state here - just preset the reference count to 1.  The
sequence should follow something like:

1. Tell kernel to destroy CM ID.
2. Kernel flushes all pending events, completes (with failure) any requests from
a previous call to get_event().
3. deref, and wait for zero.

> get_event() should check the object state and discard the event if the object
> is
> being destroyed.  It should increment a reference count before reporting any
> events.

No need to check for the state.  If the object was destroyed already, the user
has a bug and there's not much you can do about it.  Something like this:
1. inc ref_cnt
2. call kernel to get event.
3. if failure dec ref_cnt.

You need to bump the ref_cnt before making the kernel call to get the event to
prevent a call to destroy() from blowing the CM ID away while an event request
is outstanding (and the status isn't known yet).

> put_event() should decrement the reference count and unblock destroy if the
> reference count goes to 0.

Yep.

- Fab



From sean.hefty at intel.com  Tue Jul 12 12:29:01 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 12:29:01 -0700
Subject: [openib-general] userspace event reporting
In-Reply-To: <001d01c58715$b75681d0$6501a8c0@infiniconsys.com>
Message-ID: <ORSMSX401Rqf69aZbLA0000003e@orsmsx401.amr.corp.intel.com>

>I had to put something like this in place in the Windows CM, except that I had
>to use async destroy semantics to allow preserving the existing CM API.
>
>You don't need a state here - just preset the reference count to 1.  The
>sequence should follow something like:

This is what the kernel does, and I think that it would work for userspace as
well.  I agree that a state probably isn't needed.

Currently uCM doesn't associate a data structure with the cm_id.  It passes an
index directly between the application and the kernel, so adding this would
require abstracting away the cm_id, which is a fair amount of work.  I don't see
any other reasonable alternative however.

- Sean



From mst at mellanox.co.il  Tue Jul 12 12:53:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 12 Jul 2005 22:53:12 +0300
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <ORSMSX401IiGXix4A2D0000003c@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401IiGXix4A2D0000003c@orsmsx401.amr.corp.intel.com>
Message-ID: <20050712195312.GA12896@mellanox.co.il>

Quoting r. Sean Hefty <sean.hefty at intel.com>:
> Subject: userspace event reporting
> 
> This fell out of the uCM connection ID discussion...
> 
> There's an issue reporting events to userspace clients for an object
> that a user may have destroyed.  The problem exists with user verbs,
> but is much more likely to be seen by a userspace CM client.  To avoid
> reporting events for a destroyed object, I think that something
> similar to the following could be used from userspace:

Sean, what do you mean when you say "object"?
For example, cq?

I think the proper solution is to record the event in the object itself.
For example, on cq event set bit "event received".
Then after object goes away events go away without flags, states and
ugliness.

-- 
MST


From sean.hefty at intel.com  Tue Jul 12 12:56:55 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 12:56:55 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050712195312.GA12896@mellanox.co.il>
Message-ID: <ORSMSX401yEXUIEAOeI0000003f@orsmsx401.amr.corp.intel.com>

>> This fell out of the uCM connection ID discussion...
>>
>> There's an issue reporting events to userspace clients for an object
>> that a user may have destroyed.  The problem exists with user verbs,
>> but is much more likely to be seen by a userspace CM client.  To avoid
>> reporting events for a destroyed object, I think that something
>> similar to the following could be used from userspace:
>
>Sean, what do you mean when you say "object"?
>For example, cq?

In this case, I was referring to the cm_id, but it also applies to the CQ.

>I think the proper solution is to record the event in the object itself.
>For example, on cq event set bit "event received".
>Then after object goes away events go away without flags, states and
>ugliness.

What we're suggesting is that the "event set bit" be a reference count.  Some
objects, like a listening cm_id, can have multiple events outstanding at once.

- Sean



From ftillier at silverstorm.com  Tue Jul 12 13:05:05 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Tue, 12 Jul 2005 13:05:05 -0700
Subject: [openib-general] userspace event reporting
In-Reply-To: <ORSMSX401Rqf69aZbLA0000003e@orsmsx401.amr.corp.intel.com>
Message-ID: <001e01c5871c$ff98df90$6501a8c0@infiniconsys.com>

> From: Sean Hefty [mailto:sean.hefty at intel.com]
> Sent: Tuesday, July 12, 2005 12:29 PM
> 
> Currently uCM doesn't associate a data structure with the cm_id.  It passes an
> index directly between the application and the kernel, so adding this would
> require abstracting away the cm_id, which is a fair amount of work.  I don't
> see any other reasonable alternative however.

Actually, it gets a bit messier, since the get_event call would needs an extra
input parameter to identify which CM ID to perform the operation on.  This is a
pretty significant semantic change since an application with many connections
will now need to make many get_event calls, one for each CM ID it wants events
from.  Unless the get_event operation can work asynchronously, this requires a
thread per CM ID, which sucks.

I don't see a clean way to solve this without the above semantic change - the
users can't perform their own reference counting because they don't know which
CM ID will get the next event.

- Fab



From suri at baymicrosystems.com  Tue Jul 12 13:12:32 2005
From: suri at baymicrosystems.com (Suresh Shelvapille)
Date: Tue, 12 Jul 2005 16:12:32 -0400
Subject: [openib-general] Support for Linux 2.6.10
Message-ID: <200507122012.j6CKCWC8017110@ns1.baymicrosystems.com>

Hi:
 
I am very new to Infiniband. I see from the FAQ that the IB stack has been
integrated into 2.6.11 Linux kernel. I also see some patches for 2.6.9.
Unfortunately we are using a 2.6.10 kernel. Is there a patch for 2.6.10? 
 
Thanks a lot,
Suri
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050712/80bc6ecf/attachment.html>

From mst at mellanox.co.il  Tue Jul 12 13:22:40 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 12 Jul 2005 23:22:40 +0300
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <ORSMSX401yEXUIEAOeI0000003f@orsmsx401.amr.corp.intel.com>
References: <20050712195312.GA12896@mellanox.co.il>
	<ORSMSX401yEXUIEAOeI0000003f@orsmsx401.amr.corp.intel.com>
Message-ID: <20050712202240.GA13543@mellanox.co.il>

Quoting r. Sean Hefty <sean.hefty at intel.com>:
> >I think the proper solution is to record the event in the object itself.
> >For example, on cq event set bit "event received".
> >Then after object goes away events go away without flags, states and
> >ugliness.
> 
> What we're suggesting is that the "event set bit" be a reference count.  Some
> objects, like a listening cm_id, can have multiple events outstanding at once.

For cq, there can only be one outstanding event of each kind at a time.
So you can just keep a mask of events that happened, avoiding any mallocs.
Is that not the same for cm?
Anyway, once you can get all events from a given object, you can just go
and remove all events, without state, reference counting or whatever.

Thats what I advocate instead of waiting until all events are delivered.
Makes sense?

-- 
MST


From sean.hefty at intel.com  Tue Jul 12 13:24:13 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 13:24:13 -0700
Subject: [openib-general] userspace event reporting
In-Reply-To: <001e01c5871c$ff98df90$6501a8c0@infiniconsys.com>
Message-ID: <ORSMSX401X3cf3ednHM00000040@orsmsx401.amr.corp.intel.com>

>Actually, it gets a bit messier, since the get_event call would needs an extra
>input parameter to identify which CM ID to perform the operation on.  This is a
>pretty significant semantic change since an application with many connections
>will now need to make many get_event calls, one for each CM ID it wants events
>from.  Unless the get_event operation can work asynchronously, this requires a
>thread per CM ID, which sucks.

I don't think that we'd want to add the cm_id to the get_event routine.
Get_event() could always take the reference on the cm_id before invoking the
callback.  The uCM would then be responsible for verifying that events are for
existing cm_id's, rather than the users doing the check.

Hmm... I think that this will still require some more thought.  Maybe one way to
handle this is to have destruction flush events first, then perform the actual
destroy...

- Sean



From sean.hefty at intel.com  Tue Jul 12 13:31:08 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 13:31:08 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050712202240.GA13543@mellanox.co.il>
Message-ID: <ORSMSX4013SOlbpH71y00000041@orsmsx401.amr.corp.intel.com>

>For cq, there can only be one outstanding event of each kind at a time.
>So you can just keep a mask of events that happened, avoiding any mallocs.
>Is that not the same for cm?

For a listening cm_id, there could be multiple connection requests associated
with the listen in process at the same time.  You also need to report additional
information versus just indicating which event occurred.


>Anyway, once you can get all events from a given object, you can just go
>and remove all events, without state, reference counting or whatever.
>
>Thats what I advocate instead of waiting until all events are delivered.
>Makes sense?

I don't think that it avoids the issue.  Are you referring to setting a bit in a
kernel or userspace structure?  I just don't see how this helps indicate to a
userspace app that an event has occurred, provide them the necessary information
regarding the event, plus give them some context so that they can associate the
event with something.

- Sean



From mst at mellanox.co.il  Tue Jul 12 13:33:46 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 12 Jul 2005 23:33:46 +0300
Subject: [openib-general] Re: Support for Linux 2.6.10
In-Reply-To: <200507122012.j6CKCWC8017110@ns1.baymicrosystems.com>
References: <200507122012.j6CKCWC8017110@ns1.baymicrosystems.com>
Message-ID: <20050712203346.GB12896@mellanox.co.il>

Quoting r. Suresh Shelvapille <suri at baymicrosystems.com>:
> Subject: Support for Linux 2.6.10
> 
> Hi:
> 
>  
> 
> I am very new to Infiniband. I see from the FAQ that the IB stack has been
> integrated into 2.6.11 Linux kernel. I also see some patches for 2.6.9.
> Unfortunately we are using a 2.6.10 kernel. Is there a patch for 2.6.10?
> 
>  
> 
> Thanks a lot,
> 
> Suri

I'd suggest
- check out the svn tree
- apply stuff under branches/backport-to-2.6.11
- try compiling and see what errors come up

Shouldnt be too hard to get IP over IB/SDP to compile, and once it does
its most likely to work more or less out of box.
SRP may be harder.

-- 
MST


From mst at mellanox.co.il  Tue Jul 12 13:36:13 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 12 Jul 2005 23:36:13 +0300
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <ORSMSX401X3cf3ednHM00000040@orsmsx401.amr.corp.intel.com>
References: <001e01c5871c$ff98df90$6501a8c0@infiniconsys.com>
	<ORSMSX401X3cf3ednHM00000040@orsmsx401.amr.corp.intel.com>
Message-ID: <20050712203613.GC12896@mellanox.co.il>

Quoting r. Sean Hefty <sean.hefty at intel.com>:
> one way to handle this is to have destruction flush events first, then
> perform the actual destroy...

Right, what I was trying to say in the other thread was: walk the event
list and kill events for the id that is destroyed.

Does this make more sense?

-- 
MST


From sean.hefty at intel.com  Tue Jul 12 13:45:43 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 13:45:43 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050712203613.GC12896@mellanox.co.il>
Message-ID: <ORSMSX401reBBsQwY6Y00000043@orsmsx401.amr.corp.intel.com>

>> one way to handle this is to have destruction flush events first, then
>> perform the actual destroy...
>
>Right, what I was trying to say in the other thread was: walk the event
>list and kill events for the id that is destroyed.
>
>Does this make more sense?

I think this is needed, just not sure if it's sufficient.  Suppose get_event()
has retrieved an event for a cm_id and is returning the information to
userspace.  The application is unaware that an event is pending and destroys the
cm_id.  The uCM allows the destroy to proceed, since it's not yet aware of the
event.  The get_event() begins executing userspace code.

How does the app or uCM locate the data structure associated with the cm_id?  If
destroy succeeded, the structure could have been destroyed.  A solution is to
block the destroy, but I'm not sure what would block it.

- Sean



From mst at mellanox.co.il  Tue Jul 12 16:04:06 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 02:04:06 +0300
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <ORSMSX401reBBsQwY6Y00000043@orsmsx401.amr.corp.intel.com>
References: <20050712203613.GC12896@mellanox.co.il>
	<ORSMSX401reBBsQwY6Y00000043@orsmsx401.amr.corp.intel.com>
Message-ID: <20050712230406.GA19615@mellanox.co.il>

Quoting r. Sean Hefty <sean.hefty at intel.com>:
> Subject: RE: userspace event reporting
> 
> >> one way to handle this is to have destruction flush events first, then
> >> perform the actual destroy...
> >
> >Right, what I was trying to say in the other thread was: walk the event
> >list and kill events for the id that is destroyed.
> >
> >Does this make more sense?
> 
> I think this is needed, just not sure if it's sufficient.  Suppose get_event()
> has retrieved an event for a cm_id and is returning the information to
> userspace.  The application is unaware that an event is pending and destroys the
> cm_id.  The uCM allows the destroy to proceed, since it's not yet aware of the
> event.  The get_event() begins executing userspace code.
> 
> How does the app or uCM locate the data structure associated with the cm_id?  If
> destroy succeeded, the structure could have been destroyed.  A solution is to
> block the destroy, but I'm not sure what would block it.
> 
> - Sean
> 

Are we talking about multithreaded app? 
So this is purely userspace issue - cant userspace just take some mutex
in get_event and destroy paths?

-- 
MST


From sean.hefty at intel.com  Tue Jul 12 14:08:06 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 14:08:06 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050712230406.GA19615@mellanox.co.il>
Message-ID: <ORSMSX401FRaqbC8wSA00000045@orsmsx401.amr.corp.intel.com>

>Are we talking about multithreaded app?

Correct.  The app is trying to get events while needing to destroy a cm_id (or
other object) at the same time.

>So this is purely userspace issue - cant userspace just take some mutex
>in get_event and destroy paths?

I believe that this is purely a userspace issue.  I can't see why using a mutex
wouldn't work, but I believe that get_event() currently blocks waiting for an
event.

Note that get_event() may be reporting events associated with an object other
than the one being destroyed.

- Sean



From tduffy at sun.com  Tue Jul 12 14:21:02 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 12 Jul 2005 14:21:02 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
Message-ID: <1121203262.14638.16.camel@duffman>

Now that the majority of the cleanup is complete, I think kDAPL is ready
to be moved to gen2/trunk.

This would also be a good time to get NFSoRDMA ported to kDAPL as the
API is unlikely to change too much in the future.

What do others think?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050712/a58814df/attachment.sig>

From tduffy at sun.com  Tue Jul 12 14:32:14 2005
From: tduffy at sun.com (Tom Duffy)
Date: Tue, 12 Jul 2005 14:32:14 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <200507112238.27856.adobriyan@gmail.com>
References: <1121089079.4389.4511.camel@hal.voltaire.com>
	<200507111839.41807.adobriyan@gmail.com> <42D2B1F1.7000408@sun.com>
	<200507112238.27856.adobriyan@gmail.com>
Message-ID: <1121203934.14638.27.camel@duffman>

On Mon, 2005-07-11 at 22:38 +0400, Alexey Dobriyan wrote:
> $ make allmodconfig >/dev/null
> $ make C=2 CHECK="sparse -Wbitwise" drivers/infiniband/ 2>&1 | tee ../W_infiniband
> 	[snip]
> $ grep -c "warning: " ../W_infiniband
> 430

These seem to be mostly coming from cpu_to_be*() and be*_to_cpu().  Is
there a good rule of thumb for fixing these warnings?

Thanks,

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050712/4b4a7104/attachment.sig>

From hch at lst.de  Tue Jul 12 14:52:59 2005
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 12 Jul 2005 23:52:59 +0200
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <1121203262.14638.16.camel@duffman>
References: <1121203262.14638.16.camel@duffman>
Message-ID: <20050712215259.GB19448@lst.de>

On Tue, Jul 12, 2005 at 02:21:02PM -0700, Tom Duffy wrote:
> Now that the majority of the cleanup is complete, I think kDAPL is ready
> to be moved to gen2/trunk.
> 
> This would also be a good time to get NFSoRDMA ported to kDAPL as the
> API is unlikely to change too much in the future.

What's the policy on putting stuff to gen2/trunk?

The code is certainly not going anywhere near the kernel tree, but I
don't know if that's a requirement for the trunk, sdp has been on it
for a long timer either without going to the kernel tree.


From sean.hefty at intel.com  Tue Jul 12 15:00:59 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 15:00:59 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <1121203262.14638.16.camel@duffman>
Message-ID: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>

>This would also be a good time to get NFSoRDMA ported to kDAPL as the
>API is unlikely to change too much in the future.

I'm not as sure about the API.  Is it necessary to abstract away QPs, CQs, etc.
from the user?  (The answer may depend on the intersection of the Infiniband and
RNIC APIs.)  Is it possible to just extract out the kDAPL connection semantics
and layer it directly over verbs?  Thus an app could talk directly to verbs and
the kDAPL connection manager, rather than verbs and the kernel CM and SA client
modules.

- Sean



From sean.hefty at intel.com  Tue Jul 12 15:03:23 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Tue, 12 Jul 2005 15:03:23 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <20050712215259.GB19448@lst.de>
Message-ID: <ORSMSX401Rqf69aZbLA00000047@orsmsx401.amr.corp.intel.com>

>What's the policy on putting stuff to gen2/trunk?

Just my take, but I think that gen2/trunk should be components that are part of
an openib.org release, which could be separate from what gets merged upstream.

- Sean



From mst at mellanox.co.il  Tue Jul 12 15:14:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 01:14:49 +0300
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <ORSMSX401FRaqbC8wSA00000045@orsmsx401.amr.corp.intel.com>
References: <20050712230406.GA19615@mellanox.co.il>
	<ORSMSX401FRaqbC8wSA00000045@orsmsx401.amr.corp.intel.com>
Message-ID: <20050712221449.GA14316@mellanox.co.il>

Quoting r. Sean Hefty <sean.hefty at intel.com>:
> Subject: RE: userspace event reporting
> 
> >Are we talking about multithreaded app?
> 
> Correct.  The app is trying to get events while needing to destroy a
> cm_id (or other object) at the same time.
> 
> >So this is purely userspace issue - cant userspace just take some
> >mutex in get_event and destroy paths?
> 
> I believe that this is purely a userspace issue.  I can't see why
> using a mutex wouldn't work, but I believe that get_event() currently
> blocks waiting for an event.
> 
> Note that get_event() may be reporting events associated with an
> object other than the one being destroyed.
> 
> - Sean
> 

Maybe, create a special kind of event after the object has been destroyed?


-- 
MST


From mst at mellanox.co.il  Tue Jul 12 15:17:25 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 01:17:25 +0300
Subject: [openib-general] Re: Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <1121203934.14638.27.camel@duffman>
References: <1121203934.14638.27.camel@duffman>
Message-ID: <20050712221725.GB14316@mellanox.co.il>

Quoting r. Tom Duffy <tduffy at sun.com>:
> These seem to be mostly coming from cpu_to_be*() and be*_to_cpu().  Is
> there a good rule of thumb for fixing these warnings?

Yes.
Use attributes like __be32 and friends appropriately.

-- 
MST


From halr at voltaire.com  Tue Jul 12 15:15:49 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 18:15:49 -0400
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <20050711201117.72539977.akpm@osdl.org>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
	<20050711201117.72539977.akpm@osdl.org>
Message-ID: <1121206549.4382.10.camel@hal.voltaire.com>

On Mon, 2005-07-11 at 23:11, Andrew Morton wrote:
> Well I was asking.  Do you guys think that this material is appropriate to
> and safe enough for 2.6.13?

I used your versions of the patches (Tom's ucm one is needed and you
added that). I also back ported the trailing whitespace elimination
changes.

I just completed a regression test including uCM, CM, RMPP, OpenSM,
IPoIB and it looks good to me.

I'll check this again when the next -mm comes out.

Thanks.

-- Hal



From adobriyan at gmail.com  Tue Jul 12 15:48:07 2005
From: adobriyan at gmail.com (Alexey Dobriyan)
Date: Wed, 13 Jul 2005 02:48:07 +0400
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <20050712221725.GB14316@mellanox.co.il>
References: <1121203934.14638.27.camel@duffman>
	<20050712221725.GB14316@mellanox.co.il>
Message-ID: <200507130248.08387.adobriyan@gmail.com>

On Wednesday 13 July 2005 02:17, Michael S. Tsirkin wrote:
> Quoting r. Tom Duffy <tduffy at sun.com>:
> > These seem to be mostly coming from cpu_to_be*() and be*_to_cpu().  Is
> > there a good rule of thumb for fixing these warnings?
> 
> Yes.
> Use attributes like __be32 and friends appropriately.

ITYM types. ;-)

Tom, see, for example, drivers/infiniband/core/sysfs.c:
----------------------------------------------------------------------------
   313          in_mad->mad_hdr.attr_id       = cpu_to_be16(0x12); /* PortCounters */
----------------------------------------------------------------------------
drivers/infiniband/core/sysfs.c:313:32: warning: incorrect type in assignment (different base types)
drivers/infiniband/core/sysfs.c:313:32:    expected unsigned short [unsigned] [usertype] attr_id
drivers/infiniband/core/sysfs.c:313:32:    got restricted unsigned short [usertype] [force] <noident>
----------------------------------------------------------------------------
Grepping for attr_id in drivers/infiniband/ shows that:
1) in_mad->attr_id is set to IB_SMP_ATTR_NODE_INFO (network order)
2) mad->mad_hdr.attr_id is compared with IB_SMP_ATTR_PORT_INFO (network order)
3) *->mad_hdr.attr_id is set to big-endian value

All this suggests that struct ib_mad_hdr::attr_id should be __be16 instead of
u16. So, if attr_id is really something big-endian (infiniband people should
know), convert it. If not (unlikely) all those cpu_to_be16() and htons() are
bogus.

HO-WE-VER, be careful and don't blindly shut up sparse. Its warnings exist to
uncover real bugs.


From akpm at osdl.org  Tue Jul 12 15:38:59 2005
From: akpm at osdl.org (Andrew Morton)
Date: Tue, 12 Jul 2005 15:38:59 -0700
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <1121206549.4382.10.camel@hal.voltaire.com>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
	<20050711201117.72539977.akpm@osdl.org>
	<1121206549.4382.10.camel@hal.voltaire.com>
Message-ID: <20050712153859.7b757c4a.akpm@osdl.org>

Hal Rosenstock <halr at voltaire.com> wrote:
>
> On Mon, 2005-07-11 at 23:11, Andrew Morton wrote:
> > Well I was asking.  Do you guys think that this material is appropriate to
> > and safe enough for 2.6.13?
> 
> I used your versions of the patches (Tom's ucm one is needed and you
> added that). I also back ported the trailing whitespace elimination
> changes.
> 
> I just completed a regression test including uCM, CM, RMPP, OpenSM,
> IPoIB and it looks good to me.
> 

OK, well the timing of a merge is mainly up to you guys, especially as the
subsystem is pretty raw and you're the only people who use it ;)

Two things from a quick scan:

a) In many places the patch does

	if (p)
		kfree(p);

   But kfree(0) is permitted.  The cleanup police will be after you at
   some stage - it'd be best to fix those things up immediately.

b) The patch exports a ton of symbols to non-GPL modules:

+EXPORT_SYMBOL(ib_create_ah_from_wc);
+EXPORT_SYMBOL(ib_modify_mad);
+EXPORT_SYMBOL(ib_create_send_mad);
+EXPORT_SYMBOL(ib_free_send_mad);
+EXPORT_SYMBOL(ib_coalesce_recv_mad);
+EXPORT_SYMBOL(ib_sa_service_rec_query);
+EXPORT_SYMBOL(ib_create_cm_id);
+EXPORT_SYMBOL(ib_destroy_cm_id);
+EXPORT_SYMBOL(ib_cm_listen);
+EXPORT_SYMBOL(ib_send_cm_req);
+EXPORT_SYMBOL(ib_send_cm_rep);
+EXPORT_SYMBOL(ib_send_cm_rtu);
+EXPORT_SYMBOL(ib_send_cm_dreq);
+EXPORT_SYMBOL(ib_send_cm_drep);
+EXPORT_SYMBOL(ib_send_cm_rej);
+EXPORT_SYMBOL(ib_send_cm_mra);
+EXPORT_SYMBOL(ib_send_cm_lap);
+EXPORT_SYMBOL(ib_send_cm_apr);
+EXPORT_SYMBOL(ib_send_cm_sidr_req);
+EXPORT_SYMBOL(ib_send_cm_sidr_rep);
+EXPORT_SYMBOL(ib_cm_establish);
+EXPORT_SYMBOL(ib_cm_init_qp_attr);

   Why?


From halr at voltaire.com  Tue Jul 12 15:51:58 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 18:51:58 -0400
Subject: [openib-general] Re: [PATCH 0/29v2] InfiniBand core update
In-Reply-To: <20050712153859.7b757c4a.akpm@osdl.org>
References: <1121110249.4389.4984.camel@hal.voltaire.com>
	<20050711170548.31605e23.akpm@osdl.org>
	<1121136330.4389.5093.camel@hal.voltaire.com>
	<20050711201117.72539977.akpm@osdl.org>
	<1121206549.4382.10.camel@hal.voltaire.com>
	<20050712153859.7b757c4a.akpm@osdl.org>
Message-ID: <1121208717.4382.61.camel@hal.voltaire.com>

On Tue, 2005-07-12 at 18:38, Andrew Morton wrote:
> OK, well the timing of a merge is mainly up to you guys, especially as the
> subsystem is pretty raw and you're the only people who use it ;)
> 
> Two things from a quick scan:
> 
> a) In many places the patch does
> 
> 	if (p)
> 		kfree(p);
> 
>    But kfree(0) is permitted.  The cleanup police will be after you at
>    some stage - it'd be best to fix those things up immediately.

I'll/We'll work on eradicating these and pushing a patch for this
upstream.

> b) The patch exports a ton of symbols to non-GPL modules:
> 
> +EXPORT_SYMBOL(ib_create_ah_from_wc);
> +EXPORT_SYMBOL(ib_modify_mad);
> +EXPORT_SYMBOL(ib_create_send_mad);
> +EXPORT_SYMBOL(ib_free_send_mad);
> +EXPORT_SYMBOL(ib_coalesce_recv_mad);
> +EXPORT_SYMBOL(ib_sa_service_rec_query);
> +EXPORT_SYMBOL(ib_create_cm_id);
> +EXPORT_SYMBOL(ib_destroy_cm_id);
> +EXPORT_SYMBOL(ib_cm_listen);
> +EXPORT_SYMBOL(ib_send_cm_req);
> +EXPORT_SYMBOL(ib_send_cm_rep);
> +EXPORT_SYMBOL(ib_send_cm_rtu);
> +EXPORT_SYMBOL(ib_send_cm_dreq);
> +EXPORT_SYMBOL(ib_send_cm_drep);
> +EXPORT_SYMBOL(ib_send_cm_rej);
> +EXPORT_SYMBOL(ib_send_cm_mra);
> +EXPORT_SYMBOL(ib_send_cm_lap);
> +EXPORT_SYMBOL(ib_send_cm_apr);
> +EXPORT_SYMBOL(ib_send_cm_sidr_req);
> +EXPORT_SYMBOL(ib_send_cm_sidr_rep);
> +EXPORT_SYMBOL(ib_cm_establish);
> +EXPORT_SYMBOL(ib_cm_init_qp_attr);
> 
>    Why?

I think that is because OpenIB has a dual GPL/BSD license.

-- Hal



From ardavis at ichips.intel.com  Tue Jul 12 16:35:30 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Tue, 12 Jul 2005 16:35:30 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121183117.4369.97.camel@hal.voltaire.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
	<1121183117.4369.97.camel@hal.voltaire.com>
Message-ID: <42D453C2.2090402@ichips.intel.com>

Hal,

>I just checked in a new version of uat.c and also the userspace library
>(libibat/src/at.c) which should fix this oops.
>  
>
Thanks, this fixes my oops.

>Note that I did not unit test not supplying async completion routine (as
>you noticed) so there may be further issues here :-(
>  
>
>>I will go back to async mode for now.
>>    
>>
>
>This is the safer option right now.
>  
>
Ok, I now have uDAPL (dtest) up running with IB_CM and AT. :-)

I occassionally see the ib_at_route_by_ip() return a 1 but the ib_route 
data is not written out when returned but my separate at_thread that is 
waiting on a  ib_at_callback_get_timed(-1) seems to pick up the correct 
data.

Here is my output from uDAPL.

dapl_ep_connect (0x516c60, {4294967232.4294967208.0.20}, 115D3, 5000000, 
0, (nil), 0, 0)
 connect: r_SID 71123, pdata (nil), plen 0
 connect: conn 0x51a200, cm_id 4
 connect: at_route ret=1,Success req_id 0
 rt_comp_handler: conn 0x51a200, req_id 0 rec_num 1
 rt_comp_handler: SRC GID subnet 0000000000000000 id 0000000000000000
 rt_comp_handler: DST GID subnet 0000000000000000 id 0000000000000000
 dapl_ep_connect () returns 0x0

3848 Waiting for connect response
dapl_evd_wait (0x5161d0, -1, 1, 0x7ffffffff710, 0x7ffffffff6cc)
dapl_evd_wait: EVD 0x5161d0, CQ (nil), Timeout -1, Threshold 1

Output from at_thread doing the
 rt_comp_handler: conn 0x51a200, req_id 0 rec_num 1
 rt_comp_handler: SRC GID subnet 00000000000080fe id 9d40000002c90200
 rt_comp_handler: DST GID subnet 00000000000080fe id 7140000002c90200
 at_thread: callback woke
 path_comp_handler: ctxt 0x51a200, req_id 4 rec_num 1
...

Thanks for the help!

-arlin

>-- Hal
>
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>  
>



From halr at voltaire.com  Tue Jul 12 16:46:04 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 12 Jul 2005 19:46:04 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42D453C2.2090402@ichips.intel.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
	<1121183117.4369.97.camel@hal.voltaire.com>
	<42D453C2.2090402@ichips.intel.com>
Message-ID: <1121211964.4382.110.camel@hal.voltaire.com>

On Tue, 2005-07-12 at 19:35, Arlin Davis wrote:
> Ok, I now have uDAPL (dtest) up running with IB_CM and AT. :-)

Woohoo :-) Is this with a async callback or without one ?

> I occassionally see the ib_at_route_by_ip() return a 1 but the ib_route 
> data is not written out when returned but my separate at_thread that is 
> waiting on a  ib_at_callback_get_timed(-1) seems to pick up the correct 
> data.
> 
> Here is my output from uDAPL.
> 
> dapl_ep_connect (0x516c60, {4294967232.4294967208.0.20}, 115D3, 5000000, 
> 0, (nil), 0, 0)
>  connect: r_SID 71123, pdata (nil), plen 0
>  connect: conn 0x51a200, cm_id 4
>  connect: at_route ret=1,Success req_id 0
>  rt_comp_handler: conn 0x51a200, req_id 0 rec_num 1
>  rt_comp_handler: SRC GID subnet 0000000000000000 id 0000000000000000
>  rt_comp_handler: DST GID subnet 0000000000000000 id 0000000000000000
>  dapl_ep_connect () returns 0x0
> 
> 3848 Waiting for connect response
> dapl_evd_wait (0x5161d0, -1, 1, 0x7ffffffff710, 0x7ffffffff6cc)
> dapl_evd_wait: EVD 0x5161d0, CQ (nil), Timeout -1, Threshold 1
> 
> Output from at_thread doing the
>  rt_comp_handler: conn 0x51a200, req_id 0 rec_num 1
>  rt_comp_handler: SRC GID subnet 00000000000080fe id 9d40000002c90200
>  rt_comp_handler: DST GID subnet 00000000000080fe id 7140000002c90200
>  at_thread: callback woke
>  path_comp_handler: ctxt 0x51a200, req_id 4 rec_num 1
> ...

I look at this and get back as soon as I have something.

-- Hal



From ardavis at ichips.intel.com  Tue Jul 12 17:11:34 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Tue, 12 Jul 2005 17:11:34 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <ORSMSX401FRaqbC8wSA00000045@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401FRaqbC8wSA00000045@orsmsx401.amr.corp.intel.com>
Message-ID: <42D45C36.5020001@ichips.intel.com>

Sean Hefty wrote:

>>Are we talking about multithreaded app?
>>    
>>
>
>Correct.  The app is trying to get events while needing to destroy a cm_id (or
>other object) at the same time.
>
>  
>
>>So this is purely userspace issue - cant userspace just take some mutex
>>in get_event and destroy paths?
>>    
>>
>
>I believe that this is purely a userspace issue.  I can't see why using a mutex
>wouldn't work, but I believe that get_event() currently blocks waiting for an
>event.
>  
>
there is a new ib_cm_event_get_timed call that I can use that will not 
block.

>Note that get_event() may be reporting events associated with an object other
>than the one being destroyed.
>
>- Sean
>
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>  
>



From ardavis at ichips.intel.com  Tue Jul 12 18:23:39 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Tue, 12 Jul 2005 18:23:39 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121211964.4382.110.camel@hal.voltaire.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>	
	<1121183117.4369.97.camel@hal.voltaire.com>	
	<42D453C2.2090402@ichips.intel.com>
	<1121211964.4382.110.camel@hal.voltaire.com>
Message-ID: <42D46D1B.6040404@ichips.intel.com>

Hal Rosenstock wrote:

>On Tue, 2005-07-12 at 19:35, Arlin Davis wrote:
>  
>
>>Ok, I now have uDAPL (dtest) up running with IB_CM and AT. :-)
>>    
>>
>
>Woohoo :-) Is this with a async callback or without one ?
>
>  
>
with the async callback

>>I occassionally see the ib_at_route_by_ip() return a 1 but the ib_route 
>>data is not written out when returned but my separate at_thread that is 
>>waiting on a  ib_at_callback_get_timed(-1) seems to pick up the correct 
>>data.
>>
>>Here is my output from uDAPL.
>>
>>dapl_ep_connect (0x516c60, {4294967232.4294967208.0.20}, 115D3, 5000000, 
>>0, (nil), 0, 0)
>> connect: r_SID 71123, pdata (nil), plen 0
>> connect: conn 0x51a200, cm_id 4
>> connect: at_route ret=1,Success req_id 0
>> rt_comp_handler: conn 0x51a200, req_id 0 rec_num 1
>> rt_comp_handler: SRC GID subnet 0000000000000000 id 0000000000000000
>> rt_comp_handler: DST GID subnet 0000000000000000 id 0000000000000000
>> dapl_ep_connect () returns 0x0
>>
>>3848 Waiting for connect response
>>dapl_evd_wait (0x5161d0, -1, 1, 0x7ffffffff710, 0x7ffffffff6cc)
>>dapl_evd_wait: EVD 0x5161d0, CQ (nil), Timeout -1, Threshold 1
>>
>>Output from at_thread doing the
>> rt_comp_handler: conn 0x51a200, req_id 0 rec_num 1
>> rt_comp_handler: SRC GID subnet 00000000000080fe id 9d40000002c90200
>> rt_comp_handler: DST GID subnet 00000000000080fe id 7140000002c90200
>> at_thread: callback woke
>> path_comp_handler: ctxt 0x51a200, req_id 4 rec_num 1
>>...
>>    
>>
>
>I look at this and get back as soon as I have something.
>
>-- Hal
>
>  
>



From ftillier at silverstorm.com  Tue Jul 12 22:20:33 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Tue, 12 Jul 2005 22:20:33 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050712221449.GA14316@mellanox.co.il>
Message-ID: <002201c5876a$99b66dc0$6501a8c0@infiniconsys.com>

> From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
> Sent: Tuesday, July 12, 2005 3:15 PM
> 
> Quoting r. Sean Hefty <sean.hefty at intel.com>:
> >
> > I believe that this is purely a userspace issue.  I can't see why
> > using a mutex wouldn't work, but I believe that get_event() currently
> > blocks waiting for an event.
> >
> > Note that get_event() may be reporting events associated with an
> > object other than the one being destroyed.
> 
> Maybe, create a special kind of event after the object has been destroyed?

Adding such an event in effect makes destruction asynchronous - the user must
wait until the destroy event before freeing their context.  This is similar to
what is done in Windows, but still requires tracking each CM ID in user-mode to
provide the proper serialization of events.  All non-destroyed events must have
been put back to the CM before the destroy event can be generated.  Note also
that having the kernel take references to track get_event/put_event pairs can
lead to issues if an app dies before performing the put unless there's a good
way for the kernel to distinguish graceful from abortive CM ID destruction.

So I think anyway I look at it, the uCM ends up needing to shadow the CM IDs in
user-land to maintain references.  If it does that, then Sean's earlier idea of
performing reference counting and blocking destruction should work fine and
maintain the synchronous behavior that exists today.

- Fab



From talal.jaafar at gmail.com  Tue Jul 12 22:55:02 2005
From: talal.jaafar at gmail.com (Talal jaafar)
Date: Wed, 13 Jul 2005 01:55:02 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <78d18e2050711083826cffc63@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
Message-ID: <cd9967e20507122255444ca465@mail.gmail.com>

Thanks for the help, I was able to load the infiniband drivers using
your suggestion. However, I am still stuck at one point. I do not
understand why is it happening but here what I am getting when I try
to run "service network restart"

 ------------------------------------------------------------------
Shutting down interface eth0:      [OK]
Shutting down interface ib0:       [OK]
Shutting down interface ib1:       [OK]
Shutting down loopback interface:  [OK]
Setting network parameters:        [OK]
Bringing up loopback interface:    [OK]
Bringing up interface eth0:        [OK]
Bringing up interface ib0:  Error, some other host already uses
address 20.1.1.10.
Bringing up interface ib1:  Error, some other host already uses
address 20.1.1.11.
-------------------------------------------------------------------


I am 100% sure that no other host is already using the same address,
and here is how I am setting the ib interfaces:

The output of /etc/sysconfig/network-scripts/ifcgf-ib0 :
---------------------------------------------
DEVICE=ib0
BOOTPROTO=static
ONBOOT=yes
IPADDR=20.1.1.10
NETMASK=255.0.0.0
---------------------------------------------

and the output of /etc/sysconfig/network-scripts/ifcgf-ib1 :
---------------------------------------------
DEVICE=ib1
BOOTPROTO=static
ONBOOT=yes
IPADDR=20.1.1.11
NETMASK=255.0.0.0
--------------------------------------------


Had anybody run into this problem??


Regards,

 Talal


On 7/11/05, William Jordan <bjordan.ics at gmail.com> wrote:
> On 7/10/05, Talal jaafar <talal.jaafar at gmail.com> wrote:
> > hi all,
> >
> >   I am fairly new to OpenIB and I have downloaded and built the
> > 2.6.12 kernel that has OpenIB support. My question now is how to
> > compile the OpenIB modules in that kernel; There is no documentation,
> > and all what I need to do is to test the IPoIB, so anyhelp would be
> > appreciated.
> >
> > Thanks in advance,
> >
> >  Talal
> 
> It sounds like you just want to use the OpenIB support that is part of
> the downloaded kernel. Correct? If so, just rebuild the kernel. In the
> config phase, specify "InfiniBand support" under "Device Drivers". You
> will need "Mellanox HCA support", and "IP-over-InfiniBand support".
> 
> If you alias ib0 and ib1 to ib_ipoib in the /etc/modprobe.conf, and
> configure the ib0 and ib1 interfaces, the drivers should load at
> boottime (assuming you have a Mellanox HCA).
> 
> If you want the latest OpenIB support, then you want to do what Robert
> Walsh suggested, and replace the drivers/infiniband tree in the 2.6.12
> source with the latest gen2/trunk/src/linux-kernel/infiniband
> downloaded from svn.
> 
> --
> Bill Jordan
> SilverStorm Technologies
>


From info at tanpakushitsu.com  Tue Jul 12 16:50:48 2005
From: info at tanpakushitsu.com (info at tanpakushitsu.com)
Date: 13 Jul 2005 08:50:48 +0900
Subject: [openib-general] $B!yD69k2Z%W%l%<%s%H<B;\Cf!*!y(B
Message-ID: <20050712235048.7432.qmail@mail.tanpakushitsu.com>

$BEv%5%$%H$O#2<~G/$r7^$(!"8=:_6H3&$K$*$$$FBgJQ0B?4$7$FMxMQ$G$-$k%5%$%H$H$7$F$N(B
$B8|$$?.MQ$r!"$*0~MM$G3MF@$9$k$3$H$,$G$-$^$7$?!#K\Ev$KM-Fq$&$4$6$$$^$9!#(B
$B$3$NEY!"%5%$%H$r%j%K%e!<%"%kCW$7$^$7$?$N$G@'Hs0lEY$*1[$72<$5$$$^$;!#(B
$B!~(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,!~(B
$B(-(B    $B!y!#!&!,!z(B   $B%j%K%e!<%"%k4k2hBh0lCF!!!y!#!&!,!z(B   $B!!(B  $B!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!(-(B
$B!~(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(B http://awg.webchu.com/?kensyo  $B!~(B

$B"'8=:_!V?75,$4EPO?!W$N3'MM$NCf$+$iCjA*$G(B100$BL>$NJ}$K9k2Z%W%l%<%s%H$rB#DhCf!*(B

1. Louis Vuitton $B%H%j%W%k%3%$%k(B
2. Louis Vuitton $B%@%_%((B $B%H%j%"%J(B
3. ROREX $B%7!<%I%%%(%i!<(B
[ W$B%A%c%s%9"v(B]
SONY NW-507L $B%"!<%:%k%V%k!<(B $B%a%b%j!<%*!<%G%#%*(B 1GB

$B!}$9$01~Jg"v"*(B http://awg.webchu.com/?kensyo

$B!,!}!#!y!,!#!~!!!y(B $B40A4L5NA%(%s%H%j!<8e!"!V(B10,000$B1_!WJ,L5NA%]%$%s%H!JMW(BTel$BG'>Z!KB#DhCf!*(B
$B!@%W%l%<%s%H!?(B $B(.(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(/(B
$B!!!@!@!!!?!?!!!V9k2Z%W%l%<%s%H!W$H!V??2F$N%(%C%AAj<j!W(BGET$B"v$O$3$3$G7h$^$j$G$9$M"v(B
$B!!(#(!(((!($!!!!!!!!(B               http://awg.webchu.com/?kensyo$B!!(B
$B!!('(B  $B=K(B   |$B!!(B $B(1(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(0(B
$B!!(&(!(*(!(%!!!!!y(B $B;n$7$FD:$$$F at dBPB;$O$"$j$^$;$s!*<+?.$r;}$C$F$*>)$a$7$^$9!*@'Hs$*;n$72<$5$$$^$;!#(B


$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"($=$NB>HqMQ$O0l at Z3]$+$i$J$$$3$H$rL at 8@$5$;$FD:$-$^$9$N$G0B?4$7$F$*;n$72<$5$$!#(B
$B"(L5NA%]%$%s%H>C2=8e!"5$$KF~$C$FD:$1$^$7$?$i!"<c43M-NA$H$J$j$^$9$,%]%$%s%H$r$49XF~2<$5$$$^$;!#(B
$B"(%5%$%H$H$7$F!V2q$($k$^$GJ]>Z!W$b<B;\$5$;$FD:$$$F$*$j$^$9!#AGE($J=P2q$$$r$"$J$?$X!#(B

------------------------------------------------------------------------------
$B""9XFI2r=|J}K!(B
$B9XFI2r=|$r$44uK>$NJ}$O!"$*<j?t$G$9$,2<5-(BURL$B08$K$4O"Mm$/$@$5$$!#(B
free_awg at mail.ru
==============================================================================
$B"((B18$B6X(B


From mst at mellanox.co.il  Tue Jul 12 23:26:46 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 09:26:46 +0300
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <002201c5876a$99b66dc0$6501a8c0@infiniconsys.com>
References: <20050712221449.GA14316@mellanox.co.il>
	<002201c5876a$99b66dc0$6501a8c0@infiniconsys.com>
Message-ID: <20050713062646.GA22076@mellanox.co.il>

Quoting r. Fab Tillier <ftillier at silverstorm.com>:
> Subject: RE: userspace event reporting
> 
> > From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
> > Sent: Tuesday, July 12, 2005 3:15 PM
> > 
> > Quoting r. Sean Hefty <sean.hefty at intel.com>:
> > >
> > > I believe that this is purely a userspace issue.  I can't see why
> > > using a mutex wouldn't work, but I believe that get_event() currently
> > > blocks waiting for an event.
> > >
> > > Note that get_event() may be reporting events associated with an
> > > object other than the one being destroyed.
> > 
> > Maybe, create a special kind of event after the object has been destroyed?
> 
> Adding such an event in effect makes destruction asynchronous - the user must
> wait until the destroy event before freeing their context.

I think you can make this part of destroy: this event simply
solves a deadlock where get_event is blocked so you cant destroy
an object.

destroy
take mutex
get_event(non blocking)
free mutex

-- 
MST


From halr at voltaire.com  Wed Jul 13 05:07:46 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 08:07:46 -0400
Subject: [openib-general] [PATCH] sysfs: Use enum for PortCounters attribute
Message-ID: <1121256465.4401.31.camel@hal.voltaire.com>

sysfs: Use enum rather than hardcoded constant for PortCounters
attribute

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: sysfs.c
===================================================================
--- sysfs.c	(revision 2824)
+++ sysfs.c	(working copy)
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005 Mellanox Technologies Ltd.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -38,6 +39,13 @@
 
 #include <ib_mad.h>
 
+enum {
+	IB_PERF_MGMT_ATTR_CLASS_PORTINFO	= 0x01,
+	IB_PERF_MGMT_ATTR_PORTSAMPLES_CONTROL	= 0x10,
+	IB_PERF_MGMT_ATTR_PORTSAMPLES_RESULT	= 0x11,
+	IB_PERF_MGMT_ATTR_PORTCOUNTERS		= 0x12,
+};
+
 struct ib_port {
 	struct kobject         kobj;
 	struct ib_device      *ibdev;
@@ -313,7 +321,7 @@
 	in_mad->mad_hdr.mgmt_class    = IB_MGMT_CLASS_PERF_MGMT;
 	in_mad->mad_hdr.class_version = 1;
 	in_mad->mad_hdr.method        = IB_MGMT_METHOD_GET;
-	in_mad->mad_hdr.attr_id       = cpu_to_be16(0x12); /* PortCounters */
+	in_mad->mad_hdr.attr_id       = cpu_to_be16(IB_PERF_MGMT_ATTR_PORTCOUNTERS);
 
 	in_mad->data[41] = p->port_num;	/* PortSelect field */
 





From halr at voltaire.com  Wed Jul 13 05:18:34 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 08:18:34 -0400
Subject: [openib-general] [PATCH] ucm: Eliminate NULL checks prior to
	calling kfree
Message-ID: <1121257113.4401.60.camel@hal.voltaire.com>

ucm: Eliminate NULL checks prior to calling kfree

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: ucm.c
===================================================================
--- ucm.c	(revision 2838)
+++ ucm.c	(working copy)
@@ -378,10 +378,8 @@
 
 	return 0;
 error:
-	if (uvt->info)
-		kfree(uvt->info);
-	if (uvt->data)
-		kfree(uvt->data);
+	kfree(uvt->info);
+	kfree(uvt->data);
 	return result;
 }
 
@@ -551,10 +549,8 @@
 	list_del(&uevent->file_list);
 	list_del(&uevent->ctx_list);
 
-	if (uevent->data)
-		kfree(uevent->data);
-	if (uevent->info)
-		kfree(uevent->info);
+	kfree(uevent->data);
+	kfree(uevent->info);
 	kfree(uevent);
 done:
 	up(&file->mutex);
@@ -846,12 +842,9 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (param.private_data)
-		kfree(param.private_data);
-	if (param.primary_path)
-		kfree(param.primary_path);
-	if (param.alternate_path)
-		kfree(param.alternate_path);
+	kfree(param.private_data);
+	kfree(param.primary_path);
+	kfree(param.alternate_path);
 
 	return result;
 }
@@ -900,8 +893,7 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (param.private_data)
-		kfree(param.private_data);
+	kfree(param.private_data);
 
 	return result;
 }
@@ -939,8 +931,7 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (private_data)
-		kfree(private_data);
+	kfree(private_data);
 
 	return result;
 }
@@ -1009,10 +1000,8 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (data)
-		kfree(data);
-	if (info)
-		kfree(info);
+	kfree(data);
+	kfree(info);
 
 	return result;
 }
@@ -1063,8 +1052,7 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (data)
-		kfree(data);
+	kfree(data);
 
 	return result;
 }
@@ -1105,10 +1093,8 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (data)
-		kfree(data);
-	if (path)
-		kfree(path);
+	kfree(data);
+	kfree(path);
 
 	return result;
 }
@@ -1157,10 +1143,8 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (param.private_data)
-		kfree(param.private_data);
-	if (param.path)
-		kfree(param.path);
+	kfree(param.private_data);
+	kfree(param.path);
 
 	return result;
 }
@@ -1209,10 +1193,8 @@
 	up(&ctx->file->mutex);
 	ib_ucm_ctx_put(ctx); /* func reference */
 done:
-	if (param.private_data)
-		kfree(param.private_data);
-	if (param.info)
-		kfree(param.info);
+	kfree(param.private_data);
+	kfree(param.info);
 
 	return result;
 }





From halr at voltaire.com  Wed Jul 13 05:24:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 08:24:53 -0400
Subject: [openib-general] [PATCH] IPoIB: Eliminate NULL checks prior to
	calling kfree
Message-ID: <1121257492.4401.83.camel@hal.voltaire.com>

IPoIB: Eliminate NULL checks prior to calling kfree

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: ipoib_main.c
===================================================================
--- ipoib_main.c	(revision 2824)
+++ ipoib_main.c	(working copy)
@@ -780,15 +780,10 @@
 
 	ipoib_ib_dev_cleanup(dev);
 
-	if (priv->rx_ring) {
-		kfree(priv->rx_ring);
-		priv->rx_ring = NULL;
-	}
-
-	if (priv->tx_ring) {
-		kfree(priv->tx_ring);
-		priv->tx_ring = NULL;
-	}
+	kfree(priv->rx_ring);
+	priv->rx_ring = NULL;
+	kfree(priv->tx_ring);
+	priv->tx_ring = NULL;
 }
 
 static void ipoib_setup(struct net_device *dev)





From bohra at cs.rutgers.edu  Wed Jul 13 06:25:25 2005
From: bohra at cs.rutgers.edu (Aniruddha Bohra)
Date: Wed, 13 Jul 2005 09:25:25 -0400
Subject: [openib-general] perftest
Message-ID: <42D51645.3000304@cs.rutgers.edu>

Hello all,
  
    I tried running perftest. And it fails with the following error :

[bohra at hora-7 perftest]$ ./rdma_bw
libucm: Error <-1:2> couldn't open IB cm device </dev/infiniband/ucm>
libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
No IB devices found

[bohra at hora-7 perftest]$ ldd rdma_bw
        linux-gate.so.1 =>  (0xffffe000)
        libibverbs.so.1 => /usr/local/lib/libibverbs.so.1 (0xb7ff4000)
        libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
        libsysfs.so.1 => /usr/local/lib/libsysfs.so.1 (0xb7fe9000)
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
        libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
        /lib/ld-linux.so.2 (0x002d9000)

My lsmod looks like :
Module                  Size  Used by
ib_ucm                 19468  0
ib_cm                  36676  1 ib_ucm
ib_mthca              105884  0
ib_umad                18072  0
ib_ipoib               43524  0
ib_sa                  15124  1 ib_ipoib
ib_mad                 40224  4 ib_cm,ib_mthca,ib_umad,ib_sa
ib_core                45312  6 ib_cm,ib_mthca,ib_umad,ib_ipoib,ib_sa,ib_mad

uname -a :
Linux 2.6.11.9bohra #3 SMP Wed Jun 8 15:12:26 EDT 2005 i686 i686 i386 
GNU/Linux

Is there something I am missing here?

Thanks
Aniruddha



From halr at voltaire.com  Wed Jul 13 06:24:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 09:24:00 -0400
Subject: [openib-general] [PATCH] ucm: Only print messages when debug level
	set
Message-ID: <1121260511.4401.211.camel@hal.voltaire.com>

ucm: Only print messages when debug level set

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

This patch is dependent on the previous kfree ucm patch.

--- ucm.c.kfree	2005-07-13 08:10:00.000000000 -0400
+++ ucm.c	2005-07-13 09:12:03.000000000 -0400
@@ -49,6 +49,11 @@ MODULE_AUTHOR("Libor Michalek");
 MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
 MODULE_LICENSE("Dual BSD/GPL");
 
+int ucm_debug_level;
+
+module_param_named(debug_level, ucm_debug_level, int, 0644);
+MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0");
+
 enum {
 	IB_UCM_MAJOR = 231,
 	IB_UCM_MINOR = 255
@@ -56,6 +61,10 @@ enum {
 
 #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
 
+#define ucm_dbg(format, arg...)                 \
+	if (ucm_debug_level > 0)        \
+		printk(KERN_DEBUG format, ## arg)
+
 static struct semaphore ctx_id_mutex;
 static struct idr       ctx_id_table;
 static int              ctx_id_rover = 0;
@@ -107,7 +116,7 @@ static void ib_ucm_ctx_put(struct ib_ucm
 
 	up(&ctx->file->mutex);
 
-	printk(KERN_ERR "UCM: Destroyed CM ID <%d>\n", ctx->id);
+	ucm_dbg("UCM: Destroyed CM ID <%d>\n", ctx->id);
 
 	ib_destroy_cm_id(ctx->cm_id);
 	kfree(ctx);
@@ -145,7 +154,7 @@ retry:
 	if (result)
 		goto error;
 
-	printk(KERN_ERR "UCM: Allocated CM ID <%d>\n", ctx->id);
+	ucm_dbg("UCM: Allocated CM ID <%d>\n", ctx->id);
 
 	return ctx;
 error:
@@ -405,8 +414,8 @@ static int ib_ucm_event_handler(struct i
 		break;
 	}
 
-	printk(KERN_ERR "UCM: Event. CM ID <%d> event <%d>\n",
-	       id, event->event);
+	ucm_dbg("UCM: Event. CM ID <%d> event <%d>\n",
+		id, event->event);
 
 	ctx = ib_ucm_ctx_get(id);
 	if (!ctx)
@@ -1234,8 +1243,8 @@ static ssize_t ib_ucm_write(struct file 
 	if (copy_from_user(&hdr, buf, sizeof(hdr)))
 		return -EFAULT;
 
-	printk(KERN_ERR "UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
-	       hdr.cmd, hdr.in, hdr.out, len);
+	ucm_dbg("UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
+		hdr.cmd, hdr.in, hdr.out, len);
 
 	if (hdr.cmd < 0 || hdr.cmd >= ARRAY_SIZE(ucm_cmd_table))
 		return -EINVAL;
@@ -1282,7 +1291,7 @@ static int ib_ucm_open(struct inode *ino
 	filp->private_data = file;
 	file->filp = filp;
 
-	printk(KERN_ERR "UCM: Created struct\n");
+	ucm_dbg("UCM: Created struct\n");
 
 	return 0;
 }
@@ -1308,7 +1317,7 @@ static int ib_ucm_close(struct inode *in
 
 	kfree(file);
 
-	printk(KERN_ERR "UCM: Deleted struct\n");
+	ucm_dbg("UCM: Deleted struct\n");
 	return 0;
 }
 
@@ -1330,7 +1339,7 @@ static int __init ib_ucm_init(void)
 
 	result = register_chrdev_region(IB_UCM_DEV, 1, "infiniband_cm");
 	if (result) {
-		printk(KERN_ERR "UCM: Error <%d> registering dev\n", result);
+		ucm_dbg("UCM: Error <%d> registering dev\n", result);
 		goto err_chr;
 	}
 
@@ -1338,14 +1347,14 @@ static int __init ib_ucm_init(void)
 
 	result = cdev_add(&ib_ucm_cdev, IB_UCM_DEV, 1);
 	if (result) {
-		printk(KERN_ERR "UCM: Error <%d> adding cdev\n", result);
+		ucm_dbg("UCM: Error <%d> adding cdev\n", result);
 		goto err_cdev;
 	}
 
 	ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm");
 	if (IS_ERR(ib_ucm_class)) {
 		result = PTR_ERR(ib_ucm_class);
-		printk(KERN_ERR "UCM: Error <%d> creating class\n", result);
+		ucm_dbg("UCM: Error <%d> creating class\n", result);
 		goto err_class;
 	}
 





From halr at voltaire.com  Wed Jul 13 06:30:46 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 09:30:46 -0400
Subject: [openib-general] [PATCH] libibcm: Fix some typos
Message-ID: <1121261058.4401.232.camel@hal.voltaire.com>

libibcm: Fix some typos

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: include/infiniband/cm.h
===================================================================
--- include/infiniband/cm.h	(revision 2824)
+++ include/infiniband/cm.h	(working copy)
@@ -274,7 +274,7 @@ int ib_cm_event_put(struct ib_cm_event *
  * The primary use of the file descriptor is to test for CM readiness
  * events. When the CM becomes ready to READ there is a pending event
  * ready, and a subsequent call to ib_cm_event_get will not block.
- * Note: The user should not read or writ directly to the CM file
+ * Note: The user should not read or write directly to the CM file
  *       descriptor, it will likely result in an error or unexpected
  *       results.
  */
@@ -300,6 +300,7 @@ struct ib_cm_attr_param {
 	uint32_t		local_id;
 	uint32_t		remote_id;
 };
+
 /**
  * ib_cm_attr_id - Get connection identifier attributes.
  * @cm_id: Connection identifier to retrieve attributes.
Index: src/cm.c
===================================================================
--- src/cm.c	(revision 2824)
+++ src/cm.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -181,7 +182,6 @@ int ib_cm_attr_id(uint32_t cm_id, struct
 
 	cmd->id = cm_id;
 
-
 	result = write(fd, msg, size);
 	if (result != size)
 		return (result > 0) ? -ENODATA : result;
Index: examples/simple.c
===================================================================
--- examples/simple.c	(revision 2824)
+++ examples/simple.c	(working copy)
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -124,6 +125,7 @@ int main(int argc, char **argv)
 	int param_c = 0;
 	int status = 0;
 	int mode;
+
 	/*
 	 * read command line.
 	 */
@@ -153,7 +155,7 @@ int main(int argc, char **argv)
 	else {
 		result = ib_cm_listen(cm_id, TEST_SID, 0);
 		if (result) {
-			printf("Error <%d:%d> listneing <%d>\n", 
+			printf("Error <%d:%d> listening <%d>\n", 
 			       result, errno, cm_id);
 			goto done;
 		}





From jlentini at netapp.com  Wed Jul 13 06:39:09 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 09:39:09 -0400 (EDT)
Subject: [openib-general] Re:[ULP] how to choose appropriate ULPs for
	application
In-Reply-To: <BAY14-F68ED824B92D65272CBCE0A1DF0@phx.gbl>
References: <BAY14-F68ED824B92D65272CBCE0A1DF0@phx.gbl>
Message-ID: <Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.netapp.com>



On Tue, 12 Jul 2005, xg wang wrote:

>     Frankly speaking, I can not distinguish the function of SDP and DAPL. 
> Since Lustre is a file system, it runs on kernel. So I think maybe kDAPL is 
> better.

SDP stands for the Sockets Direct Protocol. The protocol is 
designed to support the Berkley Sockets API. This allows code already 
using the Sockets API to easily use InfiniBand by simply changing the 
socket type.

kDAPL is the kernel Direct Access Provider Library. It is an API 
that supports RDMA networks (InfiniBand, iWARP, etc.).

>     But for ULP application, what is the advantage and disadvantage of SDP 
> and DAP ?  While you implementation an application, will you use SDP or DAPL, 
> and why?  I just wonder the difference between them from the application 
> view.

First off, SDP is a protocol and kDAPL is an API. Since SDP is a 
protocol, you will only be able to communicate with other nodes that 
implement SDP.

Another thing to consider is the differences in the APIs. SDP 
accessed with traditional Sockets API. This makes porting applications 
to it easy, but doesn't give you much fine grained control over how 
the RDMA network is used. kDAPL was designed specifically for RDMA 
networks with lots of features that allow you to control how the 
network is used. This is good if you are writing new code, but means 
that old code needs substantial porting.


From halr at voltaire.com  Wed Jul 13 06:36:16 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 09:36:16 -0400
Subject: [openib-general] perftest
In-Reply-To: <42D51645.3000304@cs.rutgers.edu>
References: <42D51645.3000304@cs.rutgers.edu>
Message-ID: <1121261556.4401.249.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 09:25, Aniruddha Bohra wrote:
> Hello all,
>   
>     I tried running perftest. And it fails with the following error :
> 
> [bohra at hora-7 perftest]$ ./rdma_bw
> libucm: Error <-1:2> couldn't open IB cm device </dev/infiniband/ucm>
> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found
> 
> [bohra at hora-7 perftest]$ ldd rdma_bw
>         linux-gate.so.1 =>  (0xffffe000)
>         libibverbs.so.1 => /usr/local/lib/libibverbs.so.1 (0xb7ff4000)
>         libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
>         libsysfs.so.1 => /usr/local/lib/libsysfs.so.1 (0xb7fe9000)
>         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
>         libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
>         /lib/ld-linux.so.2 (0x002d9000)
> 
> My lsmod looks like :
> Module                  Size  Used by
> ib_ucm                 19468  0
> ib_cm                  36676  1 ib_ucm
> ib_mthca              105884  0
> ib_umad                18072  0
> ib_ipoib               43524  0
> ib_sa                  15124  1 ib_ipoib
> ib_mad                 40224  4 ib_cm,ib_mthca,ib_umad,ib_sa
> ib_core                45312  6 ib_cm,ib_mthca,ib_umad,ib_ipoib,ib_sa,ib_mad
> 
> uname -a :
> Linux 2.6.11.9bohra #3 SMP Wed Jun 8 15:12:26 EDT 2005 i686 i686 i386 
> GNU/Linux
> 
> Is there something I am missing here?

What's in /dev/infiniband and what are the permissions ?

-- Hal



From glebn at voltaire.com  Wed Jul 13 06:48:45 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 13 Jul 2005 16:48:45 +0300
Subject: [openib-general] perftest
In-Reply-To: <42D51645.3000304@cs.rutgers.edu>
References: <42D51645.3000304@cs.rutgers.edu>
Message-ID: <20050713134844.GN9125@minantech.com>

Try "modprobe ib_uverbs" before running the test.
And have you configured udev?

On Wed, Jul 13, 2005 at 09:25:25AM -0400, Aniruddha Bohra wrote:
> Hello all,
>  
>    I tried running perftest. And it fails with the following error :
> 
> [bohra at hora-7 perftest]$ ./rdma_bw
> libucm: Error <-1:2> couldn't open IB cm device </dev/infiniband/ucm>
> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found
> 
> [bohra at hora-7 perftest]$ ldd rdma_bw
>        linux-gate.so.1 =>  (0xffffe000)
>        libibverbs.so.1 => /usr/local/lib/libibverbs.so.1 (0xb7ff4000)
>        libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
>        libsysfs.so.1 => /usr/local/lib/libsysfs.so.1 (0xb7fe9000)
>        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
>        libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
>        /lib/ld-linux.so.2 (0x002d9000)
> 
> My lsmod looks like :
> Module                  Size  Used by
> ib_ucm                 19468  0
> ib_cm                  36676  1 ib_ucm
> ib_mthca              105884  0
> ib_umad                18072  0
> ib_ipoib               43524  0
> ib_sa                  15124  1 ib_ipoib
> ib_mad                 40224  4 ib_cm,ib_mthca,ib_umad,ib_sa
> ib_core                45312  6 ib_cm,ib_mthca,ib_umad,ib_ipoib,ib_sa,ib_mad
> 
> uname -a :
> Linux 2.6.11.9bohra #3 SMP Wed Jun 8 15:12:26 EDT 2005 i686 i686 i386 
> GNU/Linux
> 
> Is there something I am missing here?
> 
> Thanks
> Aniruddha
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general

--
			Gleb.


From jlentini at netapp.com  Wed Jul 13 07:04:10 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 10:04:10 -0400 (EDT)
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <1121203262.14638.16.camel@duffman>
References: <1121203262.14638.16.camel@duffman>
Message-ID: <Pine.LNX.4.61.0507130956450.25603@jlentini-linux.nane.netapp.com>



On Tue, 12 Jul 2005, Tom Duffy wrote:

> Now that the majority of the cleanup is complete, I think kDAPL is ready
> to be moved to gen2/trunk.

I agree. The components in question are the kDAT registry, kDAT IB 
provider, and the IBAT module. I would not include the kdapltest test 
tool since there are no other test tools in the trunk.

> This would also be a good time to get NFSoRDMA ported to kDAPL as the
> API is unlikely to change too much in the future.

I agree. Tom Talpey is planning on going through your NFSoRDMA patch 
when he returns from vacation.


From jlentini at netapp.com  Wed Jul 13 07:12:17 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 10:12:17 -0400 (EDT)
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <20050712215259.GB19448@lst.de>
References: <1121203262.14638.16.camel@duffman> <20050712215259.GB19448@lst.de>
Message-ID: <Pine.LNX.4.61.0507131004420.25603@jlentini-linux.nane.netapp.com>



On Tue, 12 Jul 2005, Christoph Hellwig wrote:

> On Tue, Jul 12, 2005 at 02:21:02PM -0700, Tom Duffy wrote:
>> Now that the majority of the cleanup is complete, I think kDAPL is ready
>> to be moved to gen2/trunk.
>>
>> This would also be a good time to get NFSoRDMA ported to kDAPL as the
>> API is unlikely to change too much in the future.
>
> What's the policy on putting stuff to gen2/trunk?
>
> The code is certainly not going anywhere near the kernel tree,

What additional changes do you recommend?

Here are the major issues I'm aware of:

  - move the consumer to using a register/callback interface for
    discovering providers (this will better support hot plug).
  - choose a IP <-> GID mapping strategy

> but I don't know if that's a requirement for the trunk, sdp has been 
> on it for a long timer either without going to the kernel tree.


From jlentini at netapp.com  Wed Jul 13 07:18:41 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 10:18:41 -0400 (EDT)
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
Message-ID: <Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>



On Tue, 12 Jul 2005, Sean Hefty wrote:

>> This would also be a good time to get NFSoRDMA ported to kDAPL as the
>> API is unlikely to change too much in the future.
>
> I'm not as sure about the API.  Is it necessary to abstract away 
> QPs, CQs, etc. from the user? (The answer may depend on the 
> intersection of the Infiniband and RNIC APIs.)  Is it possible to 
> just extract out the kDAPL connection semantics and layer it 
> directly over verbs?

In addition to the streamlined connection management interface, kDAPL 
also provides a unified event model. This is another feature that the 
verbs do not provide.

> Thus an app could talk directly to verbs and the kDAPL connection 
> manager, rather than verbs and the kernel CM and SA client modules.
>
> - Sean
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>


From hch at lst.de  Wed Jul 13 07:19:48 2005
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 13 Jul 2005 16:19:48 +0200
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507131004420.25603@jlentini-linux.nane.netapp.com>
References: <1121203262.14638.16.camel@duffman> <20050712215259.GB19448@lst.de>
	<Pine.LNX.4.61.0507131004420.25603@jlentini-linux.nane.netapp.com>
Message-ID: <20050713141948.GA31981@lst.de>

On Wed, Jul 13, 2005 at 10:12:17AM -0400, James Lentini wrote:
> >The code is certainly not going anywhere near the kernel tree,
> 
> What additional changes do you recommend?

After re-evaluting things: none, just don't merged it.

We want an RDMA layer in the kernel, and it looks like Catalin's
extension of the IB layer to cover iWarp aswell is the better choice,
in combination of taking some utility routines from kdapl and adjust
them to work directly ontop of that.



From bjordan.ics at gmail.com  Wed Jul 13 07:21:12 2005
From: bjordan.ics at gmail.com (William Jordan)
Date: Wed, 13 Jul 2005 10:21:12 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <cd9967e20507122255444ca465@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
Message-ID: <78d18e2050713072155462f61@mail.gmail.com>

On 7/13/05, Talal jaafar <talal.jaafar at gmail.com> wrote:
> Thanks for the help, I was able to load the infiniband drivers using
> your suggestion. However, I am still stuck at one point. I do not
> understand why is it happening but here what I am getting when I try
> to run "service network restart"
> 
>  ------------------------------------------------------------------
> Shutting down interface eth0:      [OK]
> Shutting down interface ib0:       [OK]
> Shutting down interface ib1:       [OK]
> Shutting down loopback interface:  [OK]
> Setting network parameters:        [OK]
> Bringing up loopback interface:    [OK]
> Bringing up interface eth0:        [OK]
> Bringing up interface ib0:  Error, some other host already uses
> address 20.1.1.10.
> Bringing up interface ib1:  Error, some other host already uses
> address 20.1.1.11.
> -------------------------------------------------------------------
> 
> 
> I am 100% sure that no other host is already using the same address,
> and here is how I am setting the ib interfaces:
> 
> The output of /etc/sysconfig/network-scripts/ifcgf-ib0 :
> ---------------------------------------------
> DEVICE=ib0
> BOOTPROTO=static
> ONBOOT=yes
> IPADDR=20.1.1.10
> NETMASK=255.0.0.0
> ---------------------------------------------
> 
> and the output of /etc/sysconfig/network-scripts/ifcgf-ib1 :
> ---------------------------------------------
> DEVICE=ib1
> BOOTPROTO=static
> ONBOOT=yes
> IPADDR=20.1.1.11
> NETMASK=255.0.0.0
> --------------------------------------------
> 
> 
> Had anybody run into this problem??
> 
> 
> Regards,
> 
>  Talal

Yes, I see the same problem with ifup. It is an issue with arping
which thinks that it has seen -1 responses to the IP address. I
haven't looked to see if there is a fixed version of  arping which
works with IPoIB (anyone know?).

You can get the the interface up by just specifying "ifconfig ib0 20.1.1.10".

-- 
Bill Jordan
SilverStorm Technologies


From halr at voltaire.com  Wed Jul 13 07:15:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 10:15:31 -0400
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507130956450.25603@jlentini-linux.nane.netapp.com>
References: <1121203262.14638.16.camel@duffman>
	<Pine.LNX.4.61.0507130956450.25603@jlentini-linux.nane.netapp.com>
Message-ID: <1121263744.4401.299.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 10:04, James Lentini wrote:
> On Tue, 12 Jul 2005, Tom Duffy wrote:
> 
> > Now that the majority of the cleanup is complete, I think kDAPL is ready
> > to be moved to gen2/trunk.
> 
> I agree. The components in question are the kDAT registry, kDAT IB 
> provider, and the IBAT module. I would not include the kdapltest test 
> tool since there are no other test tools in the trunk.

Test tools have gone into gen2/utils. There are both kernel and
userspace tools there. I would recommend putting the dapltests there.

[snip...]

-- Hal



From bjordan.ics at gmail.com  Wed Jul 13 07:24:53 2005
From: bjordan.ics at gmail.com (William Jordan)
Date: Wed, 13 Jul 2005 10:24:53 -0400
Subject: [openib-general] [ANNOUNCE] Initial checkin of SRP initiator
In-Reply-To: <52r7e8muj1.fsf@topspin.com>
References: <52r7e8muj1.fsf@topspin.com>
Message-ID: <78d18e2050713072416e026d@mail.gmail.com>

On 7/9/05, Roland Dreier <rolandd at cisco.com> wrote:
> I'm attaching some (very hacky) python scripts to this email that can
> be used as a device management client to discover storage.  Just untar
> the package and follow the README to use it.

The README appears to be missing from the PyOpenIB tar file. It
doesn't look too difficult to figure out if you know python, but
unfortunately I don't. I'm sure I need to build ioctl.c into
something, but I don't know what.

-- 
Bill Jordan
SilverStorm Technologies


From mst at mellanox.co.il  Wed Jul 13 07:34:18 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 17:34:18 +0300
Subject: [openib-general] Re: [PATCH] ucm: Only print messages when debug
	level set
In-Reply-To: <1121260511.4401.211.camel@hal.voltaire.com>
References: <1121260511.4401.211.camel@hal.voltaire.com>
Message-ID: <20050713143418.GA3576@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: [PATCH] ucm: Only print messages when debug level set
> ...
> This patch is dependent on the previous kfree ucm patch.

Is there really a dependency? It doesnt seem to conflict without it.

> +int ucm_debug_level;

shouldnt this be static?


-- 
MST


From halr at voltaire.com  Wed Jul 13 07:34:36 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 10:34:36 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <78d18e2050713072155462f61@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
Message-ID: <1121265275.4401.321.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 10:21, William Jordan wrote:
> On 7/13/05, Talal jaafar <talal.jaafar at gmail.com> wrote:
> > Thanks for the help, I was able to load the infiniband drivers using
> > your suggestion. However, I am still stuck at one point. I do not
> > understand why is it happening but here what I am getting when I try
> > to run "service network restart"
> > 
> >  ------------------------------------------------------------------
> > Shutting down interface eth0:      [OK]
> > Shutting down interface ib0:       [OK]
> > Shutting down interface ib1:       [OK]
> > Shutting down loopback interface:  [OK]
> > Setting network parameters:        [OK]
> > Bringing up loopback interface:    [OK]
> > Bringing up interface eth0:        [OK]
> > Bringing up interface ib0:  Error, some other host already uses
> > address 20.1.1.10.
> > Bringing up interface ib1:  Error, some other host already uses
> > address 20.1.1.11.
> > -------------------------------------------------------------------
> > 
> > 
> > I am 100% sure that no other host is already using the same address,
> > and here is how I am setting the ib interfaces:
> > 
> > The output of /etc/sysconfig/network-scripts/ifcgf-ib0 :
> > ---------------------------------------------
> > DEVICE=ib0
> > BOOTPROTO=static
> > ONBOOT=yes
> > IPADDR=20.1.1.10
> > NETMASK=255.0.0.0
> > ---------------------------------------------
> > 
> > and the output of /etc/sysconfig/network-scripts/ifcgf-ib1 :
> > ---------------------------------------------
> > DEVICE=ib1
> > BOOTPROTO=static
> > ONBOOT=yes
> > IPADDR=20.1.1.11
> > NETMASK=255.0.0.0
> > --------------------------------------------
> > 
> > 
> > Had anybody run into this problem??
> > 
> > 
> > Regards,
> > 
> >  Talal
> 
> Yes, I see the same problem with ifup. It is an issue with arping
> which thinks that it has seen -1 responses to the IP address. I
> haven't looked to see if there is a fixed version of  arping which
> works with IPoIB (anyone know?).
> 
> You can get the the interface up by just specifying "ifconfig ib0 20.1.1.10".

I think arping needs a number of changes. Aside from handling IB
hardware addresses (in addition to MAC addresses), I think it relies on
link level support from userspace (sending a broadcast). There was a
thread on this back in early March entitled "arping" failing over ipoib.

-- Hal




From mst at mellanox.co.il  Wed Jul 13 07:43:21 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 17:43:21 +0300
Subject: [openib-general] Re: [PATCH] libibcm: Fix some typos
In-Reply-To: <1121261058.4401.232.camel@hal.voltaire.com>
References: <1121261058.4401.232.camel@hal.voltaire.com>
Message-ID: <20050713144321.GC3576@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Index: src/cm.c
> ===================================================================
> --- src/cm.c	(revision 2824)
> +++ src/cm.c	(working copy)
> @@ -1,5 +1,6 @@
>  /*
>   * Copyright (c) 2005 Topspin Communications.  All rights reserved.
> + * Copyright (c) 2005 Voltaire, Inc.  All rights reserved.
>   *
>   * This software is available to you under a choice of one of two
>   * licenses.  You may choose to be licensed under the terms of the GNU
> @@ -181,7 +182,6 @@ int ib_cm_attr_id(uint32_t cm_id, struct
>  
>  	cmd->id = cm_id;
>  
> -
>  	result = write(fd, msg, size);
>  	if (result != size)
>  		return (result > 0) ? -ENODATA : result;

I dont think you need to add copyright statements when you trim some whitespace.

-- 
MST


From jlentini at netapp.com  Wed Jul 13 07:48:05 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 10:48:05 -0400 (EDT)
Subject: [openib-general] makiing ibverb.h  transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
Message-ID: <Pine.LNX.4.61.0507131041430.25603@jlentini-linux.nane.netapp.com>


Why do the transport dependent include files, iwarp_verbs.h and 
ibonly_verbs.h, include the generic include file, rdma_verbs.h? This 
will require consumer code to be aware of the different iwarp and ib 
verbs.

Consumers want to use a generic RDMA interface that hides the 
differences between the various transports. Is this proposal intended 
to address that requirement?


From halr at voltaire.com  Wed Jul 13 07:41:13 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 10:41:13 -0400
Subject: [openib-general] Re: [PATCH] ucm: Only print messages when debug
	level set
In-Reply-To: <20050713143418.GA3576@mellanox.co.il>
References: <1121260511.4401.211.camel@hal.voltaire.com>
	<20050713143418.GA3576@mellanox.co.il>
Message-ID: <1121265335.4401.323.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 10:34, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > Subject: [PATCH] ucm: Only print messages when debug level set
> > ...
> > This patch is dependent on the previous kfree ucm patch.
> 
> Is there really a dependency? It doesnt seem to conflict without it.
> 
> > +int ucm_debug_level;
> 
> shouldnt this be static?

Right on both. I'll reissue.

-- Hal



From halr at voltaire.com  Wed Jul 13 07:44:29 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 10:44:29 -0400
Subject: [openib-general] [PATCHv2} ucm: Only print messages when debug
	level set
Message-ID: <1121265670.4401.328.camel@hal.voltaire.com>

ucm: Only print messages when debug level set

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

--- ucm.c.kfree	2005-07-13 08:10:00.000000000 -0400
+++ ucm.c	2005-07-13 10:36:22.000000000 -0400
@@ -49,6 +49,11 @@ MODULE_AUTHOR("Libor Michalek");
 MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
 MODULE_LICENSE("Dual BSD/GPL");
 
+static int ucm_debug_level;
+
+module_param_named(debug_level, ucm_debug_level, int, 0644);
+MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0");
+
 enum {
 	IB_UCM_MAJOR = 231,
 	IB_UCM_MINOR = 255
@@ -56,6 +61,10 @@ enum {
 
 #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
 
+#define ucm_dbg(format, arg...)                 \
+	if (ucm_debug_level > 0)        \
+		printk(KERN_DEBUG format, ## arg)
+
 static struct semaphore ctx_id_mutex;
 static struct idr       ctx_id_table;
 static int              ctx_id_rover = 0;
@@ -107,7 +116,7 @@ static void ib_ucm_ctx_put(struct ib_ucm
 
 	up(&ctx->file->mutex);
 
-	printk(KERN_ERR "UCM: Destroyed CM ID <%d>\n", ctx->id);
+	ucm_dbg("UCM: Destroyed CM ID <%d>\n", ctx->id);
 
 	ib_destroy_cm_id(ctx->cm_id);
 	kfree(ctx);
@@ -145,7 +154,7 @@ retry:
 	if (result)
 		goto error;
 
-	printk(KERN_ERR "UCM: Allocated CM ID <%d>\n", ctx->id);
+	ucm_dbg("UCM: Allocated CM ID <%d>\n", ctx->id);
 
 	return ctx;
 error:
@@ -405,8 +414,8 @@ static int ib_ucm_event_handler(struct i
 		break;
 	}
 
-	printk(KERN_ERR "UCM: Event. CM ID <%d> event <%d>\n",
-	       id, event->event);
+	ucm_dbg("UCM: Event. CM ID <%d> event <%d>\n",
+		id, event->event);
 
 	ctx = ib_ucm_ctx_get(id);
 	if (!ctx)
@@ -1234,8 +1243,8 @@ static ssize_t ib_ucm_write(struct file 
 	if (copy_from_user(&hdr, buf, sizeof(hdr)))
 		return -EFAULT;
 
-	printk(KERN_ERR "UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
-	       hdr.cmd, hdr.in, hdr.out, len);
+	ucm_dbg("UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
+		hdr.cmd, hdr.in, hdr.out, len);
 
 	if (hdr.cmd < 0 || hdr.cmd >= ARRAY_SIZE(ucm_cmd_table))
 		return -EINVAL;
@@ -1282,7 +1291,7 @@ static int ib_ucm_open(struct inode *ino
 	filp->private_data = file;
 	file->filp = filp;
 
-	printk(KERN_ERR "UCM: Created struct\n");
+	ucm_dbg("UCM: Created struct\n");
 
 	return 0;
 }
@@ -1308,7 +1317,7 @@ static int ib_ucm_close(struct inode *in
 
 	kfree(file);
 
-	printk(KERN_ERR "UCM: Deleted struct\n");
+	ucm_dbg("UCM: Deleted struct\n");
 	return 0;
 }
 
@@ -1330,7 +1339,7 @@ static int __init ib_ucm_init(void)
 
 	result = register_chrdev_region(IB_UCM_DEV, 1, "infiniband_cm");
 	if (result) {
-		printk(KERN_ERR "UCM: Error <%d> registering dev\n", result);
+		ucm_dbg("UCM: Error <%d> registering dev\n", result);
 		goto err_chr;
 	}
 
@@ -1338,14 +1347,14 @@ static int __init ib_ucm_init(void)
 
 	result = cdev_add(&ib_ucm_cdev, IB_UCM_DEV, 1);
 	if (result) {
-		printk(KERN_ERR "UCM: Error <%d> adding cdev\n", result);
+		ucm_dbg("UCM: Error <%d> adding cdev\n", result);
 		goto err_cdev;
 	}
 
 	ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm");
 	if (IS_ERR(ib_ucm_class)) {
 		result = PTR_ERR(ib_ucm_class);
-		printk(KERN_ERR "UCM: Error <%d> creating class\n", result);
+		ucm_dbg("UCM: Error <%d> creating class\n", result);
 		goto err_class;
 	}
 





From halr at voltaire.com  Wed Jul 13 07:48:43 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 10:48:43 -0400
Subject: [openib-general] Re: [PATCH] libibcm: Fix some typos
In-Reply-To: <20050713144321.GC3576@mellanox.co.il>
References: <1121261058.4401.232.camel@hal.voltaire.com>
	<20050713144321.GC3576@mellanox.co.il>
Message-ID: <1121265817.4401.330.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 10:43, Michael S. Tsirkin wrote:
> I dont think you need to add copyright statements when you trim some
> whitespace.

What about typos ? Just wondering...

I was told to put a copyright on any files that were touched.

-- Hal



From halr at voltaire.com  Wed Jul 13 07:59:03 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 10:59:03 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42D46D1B.6040404@ichips.intel.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
	<1121183117.4369.97.camel@hal.voltaire.com>
	<42D453C2.2090402@ichips.intel.com>
	<1121211964.4382.110.camel@hal.voltaire.com>
	<42D46D1B.6040404@ichips.intel.com>
Message-ID: <1121266630.4401.343.camel@hal.voltaire.com>

On Tue, 2005-07-12 at 21:23, Arlin Davis wrote:
> with the async callback

One more question: Are you using IB_AT_ROUTE_FORCE_RESOLVE or not in the
flags to the ib_at_route_by_ip call ? (What is the flag setting ?)

-- Hal



From hch at lst.de  Wed Jul 13 08:11:16 2005
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 13 Jul 2005 17:11:16 +0200
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <Pine.LNX.4.61.0507131041430.25603@jlentini-linux.nane.netapp.com>
References: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
	<Pine.LNX.4.61.0507131041430.25603@jlentini-linux.nane.netapp.com>
Message-ID: <20050713151116.GA649@lst.de>

On Wed, Jul 13, 2005 at 10:48:05AM -0400, James Lentini wrote:
> 
> Why do the transport dependent include files, iwarp_verbs.h and 
> ibonly_verbs.h, include the generic include file, rdma_verbs.h?

Because they are using types from rdma_verbs.h, and thus need to include
it to compile without includinging other files first.

> This 
> will require consumer code to be aware of the different iwarp and ib 
> verbs.

If that code uses verbs specific to either iWarp or IB: yes.



From hch at lst.de  Wed Jul 13 08:12:16 2005
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 13 Jul 2005 17:12:16 +0200
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
Message-ID: <20050713151216.GB649@lst.de>

On Wed, Jul 13, 2005 at 10:18:41AM -0400, James Lentini wrote:
> In addition to the streamlined connection management interface, kDAPL 
> also provides a unified event model. This is another feature that the 
> verbs do not provide.

It's probably the biggest mis-feature and would have to go aswell if
we m�ssage what started as kdapl into the kernel rdma layer.



From eitan at mellanox.co.il  Wed Jul 13 08:29:27 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Wed, 13 Jul 2005 18:29:27 +0300
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
Message-ID: <506C3D7B14CDD411A52C00025558DED607C30497@mtlex01.yok.mtl.com>

Hi Hal,

I could not reproduce the error you see in compiling sim.cpp.

Can you elaborate on what kernel/gcc versions you see this?

Thanks

EZ

-----Original Message-----
From: Hal Rosenstock [mailto:halr at voltaire.com]
Sent: Monday, July 11, 2005 6:26 PM
To: Eitan Zahavi
Cc: openib-general at openib.org
Subject: [Fwd: [openib-commits] r2826 - in
gen2/utils/src/linux-user/IBMgtSim: . config src tests utils]


Hi Eitan,

A few nits with the latest:

1. Still get:
g++ -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include
-I/home/hal/openib/trunk/src/userspace/management/osm/include -I/usr/include
-I../datamodel   -O6 -Wall -Werror  -c -o sim.o `test -f 'sim.cpp' || echo
'./'`sim.cpp
sim.cpp:342: parse error at end of input

2. Still uneeded executable permissions on a number of simulator and
ibdm files.

Some questions:

1. What is BUILD_STANDALONE_SIM for ?

2. Will you now be doing your development in OpenIB for ibdm and the
simulator or will this be done elsewhere and dropped in here ?

Thanks.

-- Hal

--Forwarded Message--

From: eitan at openib.org
To: openib-commits at openib.org
Subject: [openib-commits] r2826 - in gen2/utils/src/linux-user/IBMgtSim: .
config src tests utils
Date: 10 Jul 2005 02:00:57 -0700

Author: eitan
Date: 2005-07-10 02:00:54 -0700 (Sun, 10 Jul 2005)
New Revision: 2826

Added:
   gen2/utils/src/linux-user/IBMgtSim/src/inttypes.i
   gen2/utils/src/linux-user/IBMgtSim/src/mads.i
   gen2/utils/src/linux-user/IBMgtSim/src/pma.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/pma.h
   gen2/utils/src/linux-user/IBMgtSim/src/randmgr.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/randmgr.h
   gen2/utils/src/linux-user/IBMgtSim/src/vsa.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/vsa.h
   gen2/utils/src/linux-user/IBMgtSim/tests/
   gen2/utils/src/linux-user/IBMgtSim/tests/Gnu16NodeOsmTest.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/tests/OsmTest.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/OsmTest.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/README
   gen2/utils/src/linux-user/IBMgtSim/tests/ReindeerBased128Nodes.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBased10K.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBased512Nodes.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoBasedCreditLoop.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/RhinoReindeer1100.topo
   gen2/utils/src/linux-user/IBMgtSim/tests/discover.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/discover.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/duplicateGuid.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/init.test.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmLidAssignment.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmLidAssignment.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticast.sim.tcl
 
gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticastRoutingTest.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmMulticastRoutingTest.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmStability.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osmStability.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osm_no_quit.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/osm_no_quit.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/pkey.check.tcl
   gen2/utils/src/linux-user/IBMgtSim/tests/pkey.sim.tcl
   gen2/utils/src/linux-user/IBMgtSim/utils/LICENSE.txt
   gen2/utils/src/linux-user/IBMgtSim/utils/fprogbar.sh
   gen2/utils/src/linux-user/IBMgtSim/utils/install.sh
   gen2/utils/src/linux-user/IBMgtSim/utils/uninstall.sh
Modified:
   gen2/utils/src/linux-user/IBMgtSim/INSTALL
   gen2/utils/src/linux-user/IBMgtSim/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/README
   gen2/utils/src/linux-user/IBMgtSim/config/bootstrap
   gen2/utils/src/linux-user/IBMgtSim/config/osm.m4
   gen2/utils/src/linux-user/IBMgtSim/configure.in
   gen2/utils/src/linux-user/IBMgtSim/src/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/src/client.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/dispatcher.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/dispatcher.h
   gen2/utils/src/linux-user/IBMgtSim/src/helper.h
   gen2/utils/src/linux-user/IBMgtSim/src/ib_types.i
   gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i
   gen2/utils/src/linux-user/IBMgtSim/src/ibms_client_api.h
   gen2/utils/src/linux-user/IBMgtSim/src/ibmssh_wrap.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/msgmgr.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/msgmgr.h
   gen2/utils/src/linux-user/IBMgtSim/src/node.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/node.h
   gen2/utils/src/linux-user/IBMgtSim/src/server.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/server.h
   gen2/utils/src/linux-user/IBMgtSim/src/sim.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/sim.h
   gen2/utils/src/linux-user/IBMgtSim/src/sim.i
   gen2/utils/src/linux-user/IBMgtSim/src/simmsg.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/simmsg.h
   gen2/utils/src/linux-user/IBMgtSim/src/sma.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/sma.h
   gen2/utils/src/linux-user/IBMgtSim/src/swig_alternate_mangling.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/tcpcomm.cpp
   gen2/utils/src/linux-user/IBMgtSim/src/tcpcomm.h
   gen2/utils/src/linux-user/IBMgtSim/utils/IBMgtSim
   gen2/utils/src/linux-user/IBMgtSim/utils/Makefile.am
   gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest
Log:
Major update:
* Introduce RunSimTest and tests directory with a set of example tests 
  and a simple way to run them
* PMA integrated into the simulator
* VSA integrated into the simulator
* All randomization in test flows should be done through a thread safe
  randomization angine 

Many bug fixes.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050713/d748ae33/attachment.html>

From caitlinb at siliquent.com  Wed Jul 13 08:31:45 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 13 Jul 2005 08:31:45 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057DE1@fiona.siliquent.com>

 

> -----Original Message-----
> From: James Lentini [mailto:jlentini at netapp.com] 
> Sent: Wednesday, July 13, 2005 7:48 AM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: Re: [openib-general] makiing ibverb.h transport 
> neutral -- 2nd draft
> 
> 
> Why do the transport dependent include files, iwarp_verbs.h 
> and ibonly_verbs.h, include the generic include file, 
> rdma_verbs.h? This will require consumer code to be aware of 
> the different iwarp and ib verbs.
> 
> Consumers want to use a generic RDMA interface that hides the 
> differences between the various transports. Is this proposal 
> intended to address that requirement?
> 

The transport dependent include file reference rdma_verbs.h
because they are dependent on it. My usual style is to explicitly
include such dependencies and rely on the #ifdefs to filter out
redundant includes.

Most consumer code would only #include rdma_verbs.h. Only those
that needed to perform transport specific conneciton establishment
or error recovery would use the transport specific headers.
Providers, on the other hand, would inlude their transport specific
header file.


From halr at voltaire.com  Wed Jul 13 08:46:49 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 11:46:49 -0400
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30497@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C30497@mtlex01.yok.mtl.com>
Message-ID: <1121269608.4401.362.camel@hal.voltaire.com>

Hi Eitan,

On Wed, 2005-07-13 at 11:29, Eitan Zahavi wrote:
> Hi Hal,
> 
> I could not reproduce the error you see in compiling sim.cpp.
> 
> Can you elaborate on what kernel/gcc versions you see this?

It may be due to an older gcc. I am using:
gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)

Bernhard also saw this.

-- Hal



From jlentini at netapp.com  Wed Jul 13 09:38:21 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 12:38:21 -0400 (EDT)
Subject: [openib-general] [PATCH][dat] commentary typos
In-Reply-To: <20050704213014.GI5997@aon.at>
References: <20050704213014.GI5997@aon.at>
Message-ID: <Pine.LNX.4.61.0507131238160.25603@jlentini-linux.nane.netapp.com>


Committed revision 2850.



From sean.hefty at intel.com  Wed Jul 13 09:41:37 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Wed, 13 Jul 2005 09:41:37 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050713062646.GA22076@mellanox.co.il>
Message-ID: <ORSMSX401FRaqbC8wSA0000004d@orsmsx401.amr.corp.intel.com>

>> > From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
>> > Sent: Tuesday, July 12, 2005 3:15 PM
>> >
>> > Quoting r. Sean Hefty <sean.hefty at intel.com>:
>> > >
>> > > I believe that this is purely a userspace issue.  I can't see why
>> > > using a mutex wouldn't work, but I believe that get_event() currently
>> > > blocks waiting for an event.
>> > >
>> > > Note that get_event() may be reporting events associated with an
>> > > object other than the one being destroyed.
>> >
>> > Maybe, create a special kind of event after the object has been destroyed?
>>
>> Adding such an event in effect makes destruction asynchronous - the user must
>> wait until the destroy event before freeing their context.
>
>I think you can make this part of destroy: this event simply
>solves a deadlock where get_event is blocked so you cant destroy
>an object.
>
>destroy
>take mutex
>get_event(non blocking)
>free mutex

I'm not sure that a destroy event helps here either.  I don't think that
anything prevents a user from calling get_event() from multiple threads.  Also,
currently get_event() is not tied to a specific object.

- Sean



From krause at cup.hp.com  Wed Jul 13 10:17:18 2005
From: krause at cup.hp.com (Michael Krause)
Date: Wed, 13 Jul 2005 10:17:18 -0700
Subject: [openib-general] Re:[ULP] how to choose appropriate ULPs
	for application
In-Reply-To: <Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.neta
	pp.com>
References: <BAY14-F68ED824B92D65272CBCE0A1DF0@phx.gbl>
	<Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.netapp.com>
Message-ID: <6.2.0.14.2.20050713100906.0263f6f8@esmail.cup.hp.com>

At 06:39 AM 7/13/2005, James Lentini wrote:


>On Tue, 12 Jul 2005, xg wang wrote:
>
>>     Frankly speaking, I can not distinguish the function of SDP and 
>> DAPL. Since Lustre is a file system, it runs on kernel. So I think maybe 
>> kDAPL is better.
>
>SDP stands for the Sockets Direct Protocol. The protocol is designed to 
>support the Berkley Sockets API. This allows code already using the 
>Sockets API to easily use InfiniBand by simply changing the socket type.

One clarification: SDP supports both synchronous and asynchronous 
sockets.  The OpenGroup ICSC released the sockets extensions API a number 
of months back that enables the full performance provided by SDP to be 
tapped.  The SDP specifications can be found at the IBTA for IB and at the 
RDMAC for RNIC web sites.


>kDAPL is the kernel Direct Access Provider Library. It is an API that 
>supports RDMA networks (InfiniBand, iWARP, etc.).
>
>>     But for ULP application, what is the advantage and disadvantage of 
>> SDP and DAP ?  While you implementation an application, will you use SDP 
>> or DAPL, and why?  I just wonder the difference between them from the 
>> application view.
>
>First off, SDP is a protocol and kDAPL is an API. Since SDP is a protocol, 
>you will only be able to communicate with other nodes that implement SDP.
>
>Another thing to consider is the differences in the APIs. SDP accessed 
>with traditional Sockets API. This makes porting applications to it easy, 
>but doesn't give you much fine grained control over how the RDMA network 
>is used.

Sockets by definition is interconnect and topology independent.  Network 
controls are managed separately.  The best that an application should do is 
signal its requirements, e.g. using diffserv or similar standard.

>kDAPL was designed specifically for RDMA networks with lots of features 
>that allow you to control how the network is used. This is good if you are 
>writing new code, but means that old code needs substantial porting.

Ideally, applications stay out of such decisions.  Middleware's job is to 
handle application optimization, etc. so that the end consumer stays as 
ignorant as possible thus focused on their application's needs not the 
networks.  The middleware API - whether DAPL, IT API, RNIC PI, whatever - 
can provide the hooks needed to manage the usage from a given endnode's 
perspective.  But even here, the real network management, what routes are 
actually used, the arbitration for QoS, etc. should also be outside of the 
middleware's control.  It simply manages a set of local resources and 
allows the fabric management to do the rest.  There is more to this than 
that but that is how IB was constructed which is no different in many 
respects from how IP works as well.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050713/3059de5f/attachment.html>

From arlin.r.davis at intel.com  Wed Jul 13 09:57:52 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Wed, 13 Jul 2005 09:57:52 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121266630.4401.343.camel@hal.voltaire.com>
Message-ID: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>



>-----Original Message-----
>From: Hal Rosenstock [mailto:halr at voltaire.com]
>Sent: Wednesday, July 13, 2005 7:59 AM
>To: Arlin Davis
>Cc: Davis, Arlin R; openib-general at openib.org
>Subject: Re: [openib-general] Re: ib_uat kernel panic without async callback
>
>On Tue, 2005-07-12 at 21:23, Arlin Davis wrote:
>> with the async callback
>
>One more question: Are you using IB_AT_ROUTE_FORCE_RESOLVE or not in the
>flags to the ib_at_route_by_ip call ? (What is the flag setting ?)

no, tos and flags are both set to 0.

>
>-- Hal




From bohra at cs.rutgers.edu  Wed Jul 13 11:12:47 2005
From: bohra at cs.rutgers.edu (Aniruddha Bohra)
Date: Wed, 13 Jul 2005 14:12:47 -0400
Subject: [openib-general] uDAPL on OpenIB gen2
Message-ID: <42D5599F.8020202@cs.rutgers.edu>

Hello,
    Is there a HOWTO or README to set up uDAPL for the gen2 code?
I have compiled and installed the libraries, however keep get the error :
DT_cs_Server: Could not open  (DAT_PROVIDER_NOT_FOUND 
DAT_NAME_NOT_REGISTERED)

when I try to run dapltest.

Do I need to install the kernel bits before I use uDAPL?

Thanks
Aniruddha




From jlentini at netapp.com  Wed Jul 13 11:18:51 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 14:18:51 -0400 (EDT)
Subject: [openib-general] Re:[ULP] how to choose appropriate ULPs  for
	application
In-Reply-To: <6.2.0.14.2.20050713100906.0263f6f8@esmail.cup.hp.com>
References: <BAY14-F68ED824B92D65272CBCE0A1DF0@phx.gbl>
	<Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.netapp.com>
	<6.2.0.14.2.20050713100906.0263f6f8@esmail.cup.hp.com>
Message-ID: <Pine.LNX.4.61.0507131411520.25603@jlentini-linux.nane.netapp.com>



On Wed, 13 Jul 2005, Michael Krause wrote:

> At 06:39 AM 7/13/2005, James Lentini wrote:
>> 
>> kDAPL was designed specifically for RDMA networks with lots of features 
>> that allow you to control how the network is used. This is good if you are 
>> writing new code, but means that old code needs substantial porting.
>
> Ideally, applications stay out of such decisions.  Middleware's job is to 
> handle application optimization, etc. so that the end consumer stays as 
> ignorant as possible thus focused on their application's needs not the 
> networks.  The middleware API - whether DAPL, IT API, RNIC PI, whatever - can 
> provide the hooks needed to manage the usage from a given endnode's 
> perspective.  But even here, the real network management, what routes are 
> actually used, the arbitration for QoS, etc. should also be outside of the 
> middleware's control.  It simply manages a set of local resources and allows 
> the fabric management to do the rest.  There is more to this than that but 
> that is how IB was constructed which is no different in many respects from 
> how IP works as well.

Let me clarify: kDAPL users can specify exactly how data is transfered 
(SEND, RDMA write, RDMA read), completion events are processed, memory 
is registered, etc. This is the "network control" I was referring to. 
In retrospect, it would be more correct refer to this as "adapter 
control".


From mshefty at ichips.intel.com  Wed Jul 13 11:24:02 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 13 Jul 2005 11:24:02 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057D7F@fiona.siliquent.com>
Message-ID: <42D55C42.2060100@ichips.intel.com>

Caitlin Bestler wrote:
> The impact of this on the existing code would be quite wide,
> but quite shallow. Many types would be renamed form 'ib_xyz'
> to 'rdma_xyz', if being used in transport neutral code. Or
> when staying infiniband specific fields that now qualified
> as transport neutral would require "common." to be inserted
> before the field name. Some constants are shifted from IB_XXX
> to RDMA_XXX.

Embedding a struct rdma_object inside a struct ib_object essentially removes 
type checking from the API.  For example, a user calling rdma_poll_cq() must 
know that they are using IB and reference the correct data structure, such as:

struct ib_wc wc;

rdma_poll_cq(cq, 1, &wc.common);

Similarly for posting RDMA requests through rdma_post_send().

- Sean


From sean.hefty at intel.com  Wed Jul 13 09:26:11 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Wed, 13 Jul 2005 09:26:11 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
Message-ID: <ORSMSX401iGXix4A2Dr0000004b@orsmsx401.amr.corp.intel.com>

>In addition to the streamlined connection management interface, kDAPL
>also provides a unified event model. This is another feature that the
>verbs do not provide.

Can someone educate me on the benefits of a "unified event model" and why a
consumer couldn't easily perform the same functionality?

- Sean



From tduffy at sun.com  Wed Jul 13 11:25:32 2005
From: tduffy at sun.com (Tom Duffy)
Date: Wed, 13 Jul 2005 11:25:32 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <20050713141948.GA31981@lst.de>
References: <1121203262.14638.16.camel@duffman> <20050712215259.GB19448@lst.de>
	<Pine.LNX.4.61.0507131004420.25603@jlentini-linux.nane.netapp.com>
	<20050713141948.GA31981@lst.de>
Message-ID: <1121279132.5366.14.camel@duffman>

On Wed, 2005-07-13 at 16:19 +0200, Christoph Hellwig wrote:
> After re-evaluting things: none, just don't merged it.

into gen2/trunk?

> We want an RDMA layer in the kernel, and it looks like Catalin's
> extension of the IB layer to cover iWarp aswell is the better choice,
> in combination of taking some utility routines from kdapl and adjust
> them to work directly ontop of that.

I agree Caitlin's proposal looks good.  I appreciate concrete examples
of how to move forward.  Let's then create a branch of trunk to start
the merge work.

What routines from kDAPL are you happy with?  What is worth keeping?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050713/d54ee80e/attachment.sig>

From jlentini at netapp.com  Wed Jul 13 11:28:06 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 14:28:06 -0400 (EDT)
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <42D5599F.8020202@cs.rutgers.edu>
References: <42D5599F.8020202@cs.rutgers.edu>
Message-ID: <Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>


On Wed, 13 Jul 2005, Aniruddha Bohra wrote:

> Hello,
>   Is there a HOWTO or README to set up uDAPL for the gen2 code?

There is one for kDAPL but not uDAPL.

> I have compiled and installed the libraries, however keep get the error :
> DT_cs_Server: Could not open  (DAT_PROVIDER_NOT_FOUND 
> DAT_NAME_NOT_REGISTERED)
>
> when I try to run dapltest.
>
> Do I need to install the kernel bits before I use uDAPL?

You don't need the kDAPL code to use uDAPL. You will need the IB 
drivers and userspace libraries though.

Does your system have the IB drivers loaded? To check:

# lsmod | grep ib_


>
> Thanks
> Aniruddha
>
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>


From jlentini at netapp.com  Wed Jul 13 11:48:35 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 14:48:35 -0400 (EDT)
Subject: [openib-general] makiing ibverb.h  transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057DE1@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057DE1@fiona.siliquent.com>
Message-ID: <Pine.LNX.4.61.0507131435130.25603@jlentini-linux.nane.netapp.com>



On Wed, 13 Jul 2005, Caitlin Bestler wrote:

>
>
>> -----Original Message-----
>> From: James Lentini [mailto:jlentini at netapp.com]
>> Sent: Wednesday, July 13, 2005 7:48 AM
>> To: Caitlin Bestler
>> Cc: openib-general; rdma-developers at lists.sourceforge.net
>> Subject: Re: [openib-general] makiing ibverb.h transport
>> neutral -- 2nd draft
>>
>>
>> Why do the transport dependent include files, iwarp_verbs.h
>> and ibonly_verbs.h, include the generic include file,
>> rdma_verbs.h? This will require consumer code to be aware of
>> the different iwarp and ib verbs.
>>
>> Consumers want to use a generic RDMA interface that hides the
>> differences between the various transports. Is this proposal
>> intended to address that requirement?
>>
>
> The transport dependent include file reference rdma_verbs.h
> because they are dependent on it. My usual style is to explicitly
> include such dependencies and rely on the #ifdefs to filter out
> redundant includes.
>
> Most consumer code would only #include rdma_verbs.h. Only those
> that needed to perform transport specific conneciton establishment
> or error recovery would use the transport specific headers.
> Providers, on the other hand, would inlude their transport specific
> header file.

Ok, so each file will be (directly) included by a different type of 
user: rdma_verbs.h by generic RDMA consumers, ibonly_verbs.h (which 
will become ib_verbs.h) by IB consumers, and iwarp_verbs.h by iwarp 
consumers.

In that case, why does the rdma_device structure in rdma_verbs.h 
contain entries for functions with IB specific types (query_gid, 
modify_device, process_mad, etc.)? Shouldn't these be in the ib 
specific header?

Keeping knowledge of the IB types in ib_verbs.h would remove the need 
to define the ib_gid type before including rdma_verbs.h in ib_verbs.h. 
Include files that require certain symbols to be defined before they 
can be included can quickly become difficult to maintain.


From rangam at gmail.com  Wed Jul 13 11:51:39 2005
From: rangam at gmail.com (Srirangam Addepalli)
Date: Wed, 13 Jul 2005 13:51:39 -0500
Subject: [openib-general] VAPI_RETRY_EXC_ERR, vendor code=81
Message-ID: <8c02c345050713115167677fe3@mail.gmail.com>

Hello All,
 I was trying to test bandwidth using the following command
 /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 minigar c0-0 
/usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
 How ever i get the following error messages.
 /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 c0-1 c0-0 
/usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
Got completion with error, code=VAPI_RETRY_EXC_ERR, vendor code=81
mpi_bandwidth: mpid/ch_vapi/viacheck.c:2109: viutil_spinandwaitcq: Assertion 
`sc->status == VAPI_SUCCESS' failed.
bash: line 1: 6813 Aborted /usr/bin/env MPIRUN_HOST=c0-1 MPIRUN_PORT=34474 
MPIRUN_RANK=1 MPIRUN_NPROCS=2 MPIRUN_ID=13416 MPIRUN_NODENAME=c0-0 
/usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
Timeout alarm signaled
Cleaning up all processes ...done.
Killed by signal 15.
 why do i get this error. 
 Rangam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050713/03f66ac5/attachment.html>

From caitlinb at siliquent.com  Wed Jul 13 11:52:44 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 13 Jul 2005 11:52:44 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057DF3@fiona.siliquent.com>

That's pretty much an inherent problem.

The presumption is that most callers of rdma_poll_cq want
to apply transport neutral logic. This is especially easy
when no specific recovery plans are in place based upon
the specific error (the specific error just gets logged).

The weakness of the extensible sub-type approach is ensuring
that the receiving buffer is large enough -- struct sockaddr
being the classic example of this problem. The type 'struct
rdma_wc_storage' is intended to address this problem by
defining a type that can hold any of the results. The
presumption is that the consumer will note the type, and
if they wish to process tht type shift to a routine that
casts the wc to the required matching struct.

The alternative would be to use extensivel sub-types on
input, and unions for work completions. That works, but
could easily be confusing to developers.

On a lesser scale, the naming convention coulde be flipped
so that 'rdma_wc' represented adequate storage for the
worst case work completion, while 'struct rdma_wc_min'
would represent the minimal common header only.
 

> -----Original Message-----
> From: Sean Hefty [mailto:mshefty at ichips.intel.com] 
> Sent: Wednesday, July 13, 2005 11:24 AM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: Re: [openib-general] makiing ibverb.h transport 
> neutral -- 2nd draft
> 
> Caitlin Bestler wrote:
> > The impact of this on the existing code would be quite 
> wide, but quite 
> > shallow. Many types would be renamed form 'ib_xyz'
> > to 'rdma_xyz', if being used in transport neutral code. Or when 
> > staying infiniband specific fields that now qualified as transport 
> > neutral would require "common." to be inserted before the 
> field name. 
> > Some constants are shifted from IB_XXX to RDMA_XXX.
> 
> Embedding a struct rdma_object inside a struct ib_object 
> essentially removes type checking from the API.  For example, 
> a user calling rdma_poll_cq() must know that they are using 
> IB and reference the correct data structure, such as:
> 
> struct ib_wc wc;
> 
> rdma_poll_cq(cq, 1, &wc.common);
> 
> Similarly for posting RDMA requests through rdma_post_send().
> 
> - Sean
> 


From caitlinb at siliquent.com  Wed Jul 13 12:02:28 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 13 Jul 2005 12:02:28 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057DF6@fiona.siliquent.com>

 

> -----Original Message-----
> From: James Lentini [mailto:jlentini at netapp.com] 
> Sent: Wednesday, July 13, 2005 11:49 AM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: RE: [openib-general] makiing ibverb.h transport 
> neutral -- 2nd draft

> 
> Ok, so each file will be (directly) included by a different type of
> user: rdma_verbs.h by generic RDMA consumers, ibonly_verbs.h 
> (which will become ib_verbs.h) by IB consumers, and 
> iwarp_verbs.h by iwarp consumers.
> 
> In that case, why does the rdma_device structure in 
> rdma_verbs.h contain entries for functions with IB specific 
> types (query_gid, modify_device, process_mad, etc.)? 
> Shouldn't these be in the ib specific header?
> 
> Keeping knowledge of the IB types in ib_verbs.h would remove 
> the need to define the ib_gid type before including 
> rdma_verbs.h in ib_verbs.h. 
> Include files that require certain symbols to be defined 
> before they can be included can quickly become difficult to maintain.
> 

The trade-off here is whether there is a single device struct defined,
or whether some existing IB-only methods can remain in their current
home in order to simplify use of the method table.

The current proposal opts for the latter, on the theory that defining
empty struct types and not filling in meaningless method points is not
really a burder for iWARP providers.

Basically, iWARP providers don't have modifiable port attributes, so 
it doesn't really matter what the type of the argument is since there
wouldn't be an implementation.

My assumption here is that all meaningful port attributes for an iWARP
RNIC would already be fully controlled through the IP stack (IP Address,
MTU, etc.) and SHOULD NOT/MUST NOT have conflicting controls being
introduced via the RDMA stack. Or put another way, there are no RDMA
layer port attributes for iWARP. There are at best some transport
layer attributes that might be reported as a convenience to RDMA
developers.

Given that, it seemed silly to rename these iWARP-irrelevant types
to "rdma_xxx" just so that they fit the naming profile a bit better
when all code that actually used them would be IB specific.



From mshefty at ichips.intel.com  Wed Jul 13 12:04:46 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 13 Jul 2005 12:04:46 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057DF3@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057DF3@fiona.siliquent.com>
Message-ID: <42D565CE.5020302@ichips.intel.com>

Caitlin Bestler wrote:
> The presumption is that most callers of rdma_poll_cq want
> to apply transport neutral logic. This is especially easy
> when no specific recovery plans are in place based upon
> the specific error (the specific error just gets logged).

The logic surrounding the call could be transport neutral, but the data 
structure into the call itself cannot be.  I don't think that it makes sense 
to provide a neutral call that requires a transport specific input 
parameter.  To use rdma_poll_cq() in a neutral fashion, the application 
needs to allocate largest possible data structure, and the data structures 
must match in the layout of the common portions.

> defining a type that can hold any of the results. The
> presumption is that the consumer will note the type, and
> if they wish to process tht type shift to a routine that
> casts the wc to the required matching struct.

The provider needs to perform the casts as well, such as when posting work 
requests.

> The alternative would be to use extensivel sub-types on
> input, and unions for work completions. That works, but
> could easily be confusing to developers.

I'm not sure that's any more confusing than requiring the developer to know 
which data type is actually expected.  It does make the data structures 
larger than they may need to be.

- Sean


From suri at baymicrosystems.com  Wed Jul 13 12:19:55 2005
From: suri at baymicrosystems.com (Suresh Shelvapille)
Date: Wed, 13 Jul 2005 15:19:55 -0400
Subject: [openib-general] Generic device driver
In-Reply-To: <1121265275.4401.321.camel@hal.voltaire.com>
Message-ID: <200507131919.j6DJJtC8015460@ns1.baymicrosystems.com>

Another newbee question. 

I only see a Mellanox HCA driver in the gen2/trunk, which is a PCI device.
Where as, we have a proprietary implementation of the HCA, where the PHY is
accessed by the host via an FPGA (Host-bridge). 

1. Is there a Generic Device Driver (I see it being mentioned in many
places).
2. If none, can I replace some files/functions within the Mellanox driver
with the low level addresses etc, to achieve the same. 

Thanks in advance for the advice,
Suri

 



From caitlinb at siliquent.com  Wed Jul 13 12:23:52 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 13 Jul 2005 12:23:52 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057DF8@fiona.siliquent.com>

 

> -----Original Message-----
> From: Sean Hefty [mailto:mshefty at ichips.intel.com] 
> Sent: Wednesday, July 13, 2005 12:05 PM
> To: Caitlin Bestler
> Cc: openib-general; rdma-developers at lists.sourceforge.net
> Subject: Re: [openib-general] makiing ibverb.h transport 
> neutral -- 2nd draft

> 
> > The alternative would be to use extensivel sub-types on input, and 
> > unions for work completions. That works, but could easily 
> be confusing 
> > to developers.
> 
> I'm not sure that's any more confusing than requiring the 
> developer to know which data type is actually expected.  It 
> does make the data structures larger than they may need to be.
> 

The consumer will know which transport. But will it want
to do an "if (ib) ... else (iwarp) ..." wrapper just to
minimize the size of the work completion buf? I doubt it.
The work completions delivered to the user are unlikely
to be kept in arrays. More likely they will be immediately
translated to a middleware event (DAPL, MPI, RPC ...).

And even if the transport is known, the caller will frequently
not know the type (receive, send, bind ...) anyway.

Now it would be real nifty if C defined an "extensible union"
where various different include files could add variants to 
a parent type, and then application code could use the parent
type knowing that it would be large enough for the worst case
and yet only have to know about the variants that the specific
application actually wanted to deal with.

But since ANSI C does not have such a construct we can either
force *all* the variants to be visible in a big overbloated
union (but at least know that they are all there), or we can
allow sub-types to be declared as necessary and hope that all
sub-type declarers double-check that their sizeof() fits within
an allotted maximum.

The current draft follows the philosophy of avoiding overbloated
centralized unions, which seemed to be the consensus from the
first round of discussion.

The overbloated union also works, but it has its own pains.
I have no problem changing it back to the union approach on 
either the forward and/or reverse path, if that is the consensus.
Personally, I think using a different style for forware and
reverse would be more confusing, that using a cast in response
to an opcode is no more inherently suspect than selecting a
union sub-variant in response to an opcode and that many or
most applications will need to support multiple work completion
sub-types anyway.




From bohra at cs.rutgers.edu  Wed Jul 13 12:25:36 2005
From: bohra at cs.rutgers.edu (Aniruddha Bohra)
Date: Wed, 13 Jul 2005 15:25:36 -0400
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
References: <42D5599F.8020202@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
Message-ID: <42D56AB0.6070804@cs.rutgers.edu>

James Lentini wrote:

>
> On Wed, 13 Jul 2005, Aniruddha Bohra wrote:
>
>> Hello,
>>   Is there a HOWTO or README to set up uDAPL for the gen2 code?
>
>
>
> There is one for kDAPL but not uDAPL.
>
>> I have compiled and installed the libraries, however keep get the 
>> error :
>> DT_cs_Server: Could not open  (DAT_PROVIDER_NOT_FOUND 
>> DAT_NAME_NOT_REGISTERED)
>>
>> when I try to run dapltest.
>>
>> Do I need to install the kernel bits before I use uDAPL?
>
>
>
> You don't need the kDAPL code to use uDAPL. You will need the IB 
> drivers and userspace libraries though.
>
> Does your system have the IB drivers loaded? To check:
>
> # lsmod | grep ib_
>
Hello,
   Here is the relevant info :

> [bohra at hora-7 kdapl]$ lsmod | grep ib_
> ib_mthca              105884  0
> ib_uverbs              27152  0
> ib_ucm                 19468  0
> ib_cm                  36676  1 ib_ucm
> ib_umad                18072  0
> ib_ipoib               43524  0
> ib_sa                  15124  1 ib_ipoib
> ib_mad                 40224  4 ib_mthca,ib_cm,ib_umad,ib_sa
> ib_core                45312  7 
> ib_mthca,ib_uverbs,ib_cm,ib_umad,ib_ipoib,ib_sa,ib_mad


[bohra at hora-7 Target]$ pwd
/home/bohra/jlentini/userspace/test/dapltest/udapl/Target
[bohra at hora-7 Target]$ ldd dapltest
       linux-gate.so.1 =>  (0xffffe000)
       libdat.so => /usr/local/lib/libdat.so (0xb7ff6000)
       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
       libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
       libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
       /lib/ld-linux.so.2 (0x002d9000)

[bohra at hora-7 Target]$ ./dapltest -T S -D mthca0a
DT_cs_Server: Could not open mthca0a (DAT_PROVIDER_NOT_FOUND 
DAT_NAME_NOT_REGISTERED)
DT_cs_Server (mthca0a):  Exiting.


Thanks
Aniruddha





From brandons at owenmedia.com  Wed Jul 13 12:28:19 2005
From: brandons at owenmedia.com (Brandon Sanford)
Date: Wed, 13 Jul 2005 12:28:19 -0700
Subject: [openib-general] OIB Member Announcement of Datacenter Fabric
	Workshop 2005
Message-ID: <008901c587e1$067f3b00$1601000a@omi.owenmedia.com>

Dear OpenIB Member, 

 

DATACENTER FABRIC WORKSHOP: Join the InfiniBandSM Trade Association and the OpenIB Alliance at the Datacenter Fabric Workshop, August 22 from 8:00 a.m. to 5:30 p.m. with a reception following.  The workshop will be held in conjunction with the Intel® Developer Forum at the Moscone Center in San Francisco, August 23 - 25.

 

Hosted by the InfiniBand Trade Association and OpenIB, the Datacenter Fabric Workshop will bring together leading fabric developers and users to discuss the future of fabric computing. 

 

Discussions will include RDMA architectures in grid computing, virtualization, storage in RDMA interconnects, interoperability and application interface consistency between stacks and architectures, and the progress of the OpenIB Alliance in delivering Windows and Linux stacks.  Speakers will include representatives from the fabric computing industry including leadership from both organizations.

 

The Datacenter Fabric Workshop registration fee is $250.  Now is the time to register; visit: http://www.acteva.com/booking.cfm?bevaid=91491.

 

INTEL DEVELOPER FORUM DISCOUNT: Also, don't miss out on your OpenIB membership discount when you register for the Intel Developer Forum.  The Intel Developer Forum is the perfect place for you to gain key insights on Intel's plans for multi-core processors and the platforms they enable, as well as broadband wireless technologies.  As an OpenIB member, your Intel Developer Forum registration fee is only $750.  Saving you $945!  

 

Take advantage of this great opportunity and register now.  Please visit http://www28.cplan.com/pls/pg_intel/c82_reg_entry.idfa_sys and type in your OpenIB Priority Code: SGREMPEN.

 

To reserve your hotel room and take advantage of your OpenIB membership rates, please contact: Lilybeth V. FlorCruz, Registration Housing Manager, via phone: 650-226-0668 or email: lilybeth.florcruz at gpjco.com.

 

Hotel list

·         Renaissance Parc55 $164 - 4 Blocks from Moscone West

·         Courtyard By Marriott $159 - 3 Blocks from Moscone West

·         Hotel Nikko $160 - 4 Blocks from Moscone West

·         W Hotel $219 - 0.5 Blocks from Moscone West

·         Grand Hyatt $169 - 5 Blocks from Moscone West

·         The Mosser $ 99 - .5 Block from Moscone West

·         Hotel Milano $119 - 3 Blocks from Moscone West

·         Palace Hotel $199 - 2 Blocks from Moscone West

 

See you there!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050713/e9a8f3f9/attachment.html>

From bjordan.ics at gmail.com  Wed Jul 13 12:32:41 2005
From: bjordan.ics at gmail.com (William Jordan)
Date: Wed, 13 Jul 2005 15:32:41 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <1121265275.4401.321.camel@hal.voltaire.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
Message-ID: <78d18e2050713123235f668bc@mail.gmail.com>

On 13 Jul 2005 10:34:36 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> On Wed, 2005-07-13 at 10:21, William Jordan wrote:
> > On 7/13/05, Talal jaafar <talal.jaafar at gmail.com> wrote:
> > > Thanks for the help, I was able to load the infiniband drivers using
> > > your suggestion. However, I am still stuck at one point. I do not
> > > understand why is it happening but here what I am getting when I try
> > > to run "service network restart"
> > >
> > >  ------------------------------------------------------------------
> > > Shutting down interface eth0:      [OK]
> > > Shutting down interface ib0:       [OK]
> > > Shutting down interface ib1:       [OK]
> > > Shutting down loopback interface:  [OK]
> > > Setting network parameters:        [OK]
> > > Bringing up loopback interface:    [OK]
> > > Bringing up interface eth0:        [OK]
> > > Bringing up interface ib0:  Error, some other host already uses
> > > address 20.1.1.10.
> > > Bringing up interface ib1:  Error, some other host already uses
> > > address 20.1.1.11.
> > > -------------------------------------------------------------------
> > >
> > > Regards,
> > >
> > >  Talal
> >
> > Yes, I see the same problem with ifup. It is an issue with arping
> > which thinks that it has seen -1 responses to the IP address. I
> > haven't looked to see if there is a fixed version of  arping which
> > works with IPoIB (anyone know?).
> 
> I think arping needs a number of changes. Aside from handling IB
> hardware addresses (in addition to MAC addresses), I think it relies on
> link level support from userspace (sending a broadcast). There was a
> thread on this back in early March entitled "arping" failing over ipoib.

I upgraded my iputils (from the Fedora Core 4 source RPM,
iputils-20020927-22). The new version doesn't work. But, instead of
arping returning -1 responses, it reports 0 responses, so at least
ifup works without complaining about the address already being in use.
However, if the address really is in use, it doesn't catch it.

-- 
Bill Jordan
SilverStorm Technologies


From robert.j.woodruff at intel.com  Wed Jul 13 11:24:38 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Wed, 13 Jul 2005 11:24:38 -0700
Subject: [openib-general] uDAPL on OpenIB gen2
Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004EF880A@orsmsx408>

AB wrote,
>DAT_NAME_NOT_REGISTERED)

>when I try to run dapltest.

>Do I need to install the kernel bits before I use uDAPL?


Yes. Install the kernel bits and user-mode libraries 
and then make sure your dat.conf is also set up. 


From halr at voltaire.com  Wed Jul 13 12:37:32 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 15:37:32 -0400
Subject: [openib-general] Generic device driver
In-Reply-To: <200507131919.j6DJJtC8015460@ns1.baymicrosystems.com>
References: <200507131919.j6DJJtC8015460@ns1.baymicrosystems.com>
Message-ID: <1121283452.4423.24.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 15:19, Suresh Shelvapille wrote:
> Another newbee question. 
> 
> I only see a Mellanox HCA driver in the gen2/trunk, which is a PCI device.
> Where as, we have a proprietary implementation of the HCA, where the PHY is
> accessed by the host via an FPGA (Host-bridge). 
> 
> 1. Is there a Generic Device Driver (I see it being mentioned in many
> places).

The generic device driver is a conccept. It is a set of required
(mandatory) APIs which the device driver needs to supply (at
registration time). Look at core/device.c.

> 2. If none, can I replace some files/functions within the Mellanox driver
> with the low level addresses etc, to achieve the same. 

Yes, you can use the Mellanox HCA driver structure (strip out the
implementation). You might want to change the names from mthca_xxxx to
fpga_name_xxx where fpga_name is some name used by your FPGA based
implementation.

Start with mthca_provider.c (and registration/deregistration) and build
up.

-- Hal




From mst at mellanox.co.il  Wed Jul 13 13:13:34 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 23:13:34 +0300
Subject: [openib-general] Re: [PATCH] libibcm: Fix some typos
In-Reply-To: <1121265817.4401.330.camel@hal.voltaire.com>
References: <1121261058.4401.232.camel@hal.voltaire.com>
	<20050713144321.GC3576@mellanox.co.il>
	<1121265817.4401.330.camel@hal.voltaire.com>
Message-ID: <20050713201334.GA26048@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: Re: [PATCH] libibcm: Fix some typos
> 
> On Wed, 2005-07-13 at 10:43, Michael S. Tsirkin wrote:
> > I dont think you need to add copyright statements when you trim some
> > whitespace.
> 
> What about typos ? Just wondering...

I would think not.

> I was told to put a copyright on any files that were touched.

Clearly overboard.
I'm not a lawyer, lets just keep things sensible: copyrights in Linux
files would be insanely long if any trivial fix requires a copyright.

-- 
MST


From jlentini at netapp.com  Wed Jul 13 13:13:05 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 13 Jul 2005 16:13:05 -0400 (EDT)
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <42D56AB0.6070804@cs.rutgers.edu>
References: <42D5599F.8020202@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
	<42D56AB0.6070804@cs.rutgers.edu>
Message-ID: <Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>


Hmm...That looks good to me. Let's start at the bottom of the stack 
and work our way up. Have you checked your userspace IB configuration? 
What is the output of ib_devices and ib_devinfo? Does ibv_ud_pingpong 
work?

On Wed, 13 Jul 2005, Aniruddha Bohra wrote:

bohra> James Lentini wrote:
bohra> 
bohra> > 
bohra> > On Wed, 13 Jul 2005, Aniruddha Bohra wrote:
bohra> > 
bohra> > > Hello,
bohra> > >   Is there a HOWTO or README to set up uDAPL for the gen2 code?
bohra> > 
bohra> > 
bohra> > 
bohra> > There is one for kDAPL but not uDAPL.
bohra> > 
bohra> > > I have compiled and installed the libraries, however keep get the
bohra> > > error :
bohra> > > DT_cs_Server: Could not open  (DAT_PROVIDER_NOT_FOUND
bohra> > > DAT_NAME_NOT_REGISTERED)
bohra> > > 
bohra> > > when I try to run dapltest.
bohra> > > 
bohra> > > Do I need to install the kernel bits before I use uDAPL?
bohra> > 
bohra> > 
bohra> > 
bohra> > You don't need the kDAPL code to use uDAPL. You will need the IB
bohra> > drivers and userspace libraries though.
bohra> > 
bohra> > Does your system have the IB drivers loaded? To check:
bohra> > 
bohra> > # lsmod | grep ib_
bohra> > 
bohra> Hello,
bohra>   Here is the relevant info :
bohra> 
bohra> > [bohra at hora-7 kdapl]$ lsmod | grep ib_
bohra> > ib_mthca              105884  0
bohra> > ib_uverbs              27152  0
bohra> > ib_ucm                 19468  0
bohra> > ib_cm                  36676  1 ib_ucm
bohra> > ib_umad                18072  0
bohra> > ib_ipoib               43524  0
bohra> > ib_sa                  15124  1 ib_ipoib
bohra> > ib_mad                 40224  4 ib_mthca,ib_cm,ib_umad,ib_sa
bohra> > ib_core                45312  7
bohra> > ib_mthca,ib_uverbs,ib_cm,ib_umad,ib_ipoib,ib_sa,ib_mad
bohra> 
bohra> 
bohra> [bohra at hora-7 Target]$ pwd
bohra> /home/bohra/jlentini/userspace/test/dapltest/udapl/Target
bohra> [bohra at hora-7 Target]$ ldd dapltest
bohra>       linux-gate.so.1 =>  (0xffffe000)
bohra>       libdat.so => /usr/local/lib/libdat.so (0xb7ff6000)
bohra>       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
bohra>       libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
bohra>       libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
bohra>       /lib/ld-linux.so.2 (0x002d9000)
bohra> 
bohra> [bohra at hora-7 Target]$ ./dapltest -T S -D mthca0a
bohra> DT_cs_Server: Could not open mthca0a (DAT_PROVIDER_NOT_FOUND
bohra> DAT_NAME_NOT_REGISTERED)
bohra> DT_cs_Server (mthca0a):  Exiting.
bohra> 
bohra> 
bohra> Thanks
bohra> Aniruddha
bohra> 
bohra> 
bohra> 
bohra> _______________________________________________
bohra> openib-general mailing list
bohra> openib-general at openib.org
bohra> http://openib.org/mailman/listinfo/openib-general
bohra> 
bohra> To unsubscribe, please visit
bohra> http://openib.org/mailman/listinfo/openib-general
bohra> 


From mst at mellanox.co.il  Wed Jul 13 13:29:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 13 Jul 2005 23:29:33 +0300
Subject: [openib-general] Re: [PATCHv2} ucm: Only print messages when debug
	level set
In-Reply-To: <1121265670.4401.328.camel@hal.voltaire.com>
References: <1121265670.4401.328.camel@hal.voltaire.com>
Message-ID: <20050713202933.GC26048@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> @@ -56,6 +61,10 @@ enum {
>  
>  #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
>  
> +#define ucm_dbg(format, arg...)                 \
> +	if (ucm_debug_level > 0)        \
> +		printk(KERN_DEBUG format, ## arg)
> +

What happened to whitespace here?
Also, may it be a good idea for ucm_dbg to add the UCM: prefix automatically?

-- 
MST


From mulix at mulix.org  Wed Jul 13 13:43:46 2005
From: mulix at mulix.org (Muli Ben-Yehuda)
Date: Wed, 13 Jul 2005 23:43:46 +0300
Subject: [openib-general] Re: [PATCHv2} ucm: Only print messages when
	debug level set
In-Reply-To: <20050713202933.GC26048@mellanox.co.il>
References: <1121265670.4401.328.camel@hal.voltaire.com>
	<20050713202933.GC26048@mellanox.co.il>
Message-ID: <20050713204346.GC9093@granada.merseine.nu>

On Wed, Jul 13, 2005 at 11:29:33PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > @@ -56,6 +61,10 @@ enum {
> >  
> >  #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
> >  
> > +#define ucm_dbg(format, arg...)                 \
> > +	if (ucm_debug_level > 0)        \
> > +		printk(KERN_DEBUG format, ## arg)
> > +
> 
> What happened to whitespace here?
> Also, may it be a good idea for ucm_dbg to add the UCM: prefix
automatically?

You also want to wrap this thing up in a do { } while (0) loop (see
http://www.kernelnewbies.org/faq/, "Why do a lot of #defines in the
kernel use do { ... } while(0)?")

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/



From eitan at mellanox.co.il  Wed Jul 13 14:04:37 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 00:04:37 +0300
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
Message-ID: <506C3D7B14CDD411A52C00025558DED607C3049C@mtlex01.yok.mtl.com>

Hi Hal,

I will work on it tomorrow
I'm not too familiar with RH release numbers. 
Is there a code name for RH 3.2.2-5 ? Is it a Fedora? AS?

BTW: The directory of IBMgtSim is now called ibmgtsim -
I got flanked for using upper case...

Eitan
-----Original Message-----
From: Hal Rosenstock [mailto:halr at voltaire.com]
Sent: Wednesday, July 13, 2005 6:47 PM
To: Eitan Zahavi
Cc: openib-general at openib.org
Subject: RE: [Fwd: [openib-commits] r2826 - in
gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]


Hi Eitan,

On Wed, 2005-07-13 at 11:29, Eitan Zahavi wrote:
> Hi Hal,
> 
> I could not reproduce the error you see in compiling sim.cpp.
> 
> Can you elaborate on what kernel/gcc versions you see this?

It may be due to an older gcc. I am using:
gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)

Bernhard also saw this.

-- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/98b302ad/attachment.html>

From krause at cup.hp.com  Wed Jul 13 13:45:57 2005
From: krause at cup.hp.com (Michael Krause)
Date: Wed, 13 Jul 2005 13:45:57 -0700
Subject: [openib-general] Re:[ULP] how to choose appropriate ULPs 
	for application
In-Reply-To: <Pine.LNX.4.61.0507131411520.25603@jlentini-linux.nane.neta
	pp.com>
References: <BAY14-F68ED824B92D65272CBCE0A1DF0@phx.gbl>
	<Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.netapp.com>
	<6.2.0.14.2.20050713100906.0263f6f8@esmail.cup.hp.com>
	<Pine.LNX.4.61.0507131411520.25603@jlentini-linux.nane.netapp.com>
Message-ID: <6.2.0.14.2.20050713133832.02a81c80@esmail.cup.hp.com>

At 11:18 AM 7/13/2005, James Lentini wrote:


>On Wed, 13 Jul 2005, Michael Krause wrote:
>
>>At 06:39 AM 7/13/2005, James Lentini wrote:
>>>kDAPL was designed specifically for RDMA networks with lots of features 
>>>that allow you to control how the network is used. This is good if you 
>>>are writing new code, but means that old code needs substantial porting.
>>
>>Ideally, applications stay out of such decisions.  Middleware's job is to 
>>handle application optimization, etc. so that the end consumer stays as 
>>ignorant as possible thus focused on their application's needs not the 
>>networks.  The middleware API - whether DAPL, IT API, RNIC PI, whatever - 
>>can provide the hooks needed to manage the usage from a given endnode's 
>>perspective.  But even here, the real network management, what routes are 
>>actually used, the arbitration for QoS, etc. should also be outside of 
>>the middleware's control.  It simply manages a set of local resources and 
>>allows the fabric management to do the rest.  There is more to this than 
>>that but that is how IB was constructed which is no different in many 
>>respects from how IP works as well.
>
>Let me clarify: kDAPL users can specify exactly how data is transfered 
>(SEND, RDMA write, RDMA read), completion events are processed, memory is 
>registered, etc. This is the "network control" I was referring to. In 
>retrospect, it would be more correct refer to this as "adapter control".

Just to nit-pick this a bit.  The ULP determines what type of operation to 
use - SEND, RDMA Write, RDMA Read, or Atomic (where supported by the 
interconnect).  The middleware API or the verbs API provide an interface to 
abstract the IHV hardware-specifics from the ULP allowing the ULP to be 
implemented across a variety of technologies.  Thus, the API is just an 
abstraction of the underlying semantics and does not make decisions or 
provide much in the way of controls in any regard unless additional 
value-add beyond the underlying hardware semantics are transparently 
implemented within the API implementation itself   For example, SDP defines 
specifically when to use a SEND for control operations, when to use a RDMA 
for a zcopy operation.  SDP itself does not care what the underlying API is 
used to access the associated hardware resources but it does define what 
resources and associated services are used.  SDP can be implemented 
directly on the verbs API (just like MPI) and operate quite nicely without 
the additional middleware API in the execution path.

Apologies of the nit pick but the API does not provide any type of control 
other than to act as an abstraction funnel between the ULP / application 
and the underlying hardware.  There are opportunities to provide 
transparent value-add controls but I don't believe this open source effort 
is focused on these this time.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050713/ac4950d3/attachment.html>

From mshefty at ichips.intel.com  Wed Jul 13 14:23:21 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 13 Jul 2005 14:23:21 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057DF8@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057DF8@fiona.siliquent.com>
Message-ID: <42D58649.7040601@ichips.intel.com>

Caitlin Bestler wrote:
> The overbloated union also works, but it has its own pains.
> I have no problem changing it back to the union approach on 
> either the forward and/or reverse path, if that is the consensus.
> Personally, I think using a different style for forware and
> reverse would be more confusing, that using a cast in response
> to an opcode is no more inherently suspect than selecting a
> union sub-variant in response to an opcode and that many or
> most applications will need to support multiple work completion
> sub-types anyway.

(Note that I'm just using rdma_poll_cq() as an example.  I didn't go into 
the other calls in as much detail.)

I'm not suggesting the use of unions everywhere.  I think that there's a 
problem defining a 256 byte structure that is likely to be declared on the 
stack so an application can call poll_cq in a transport independent fashion.

Reading a field in a structure and then casting the structure to a parent 
structure does not require any specific field layout.  Taking an input 
parameter to a function and casting it to a parent structure does.  For 
example, redefine struct ib_wc and move the common field to the end of the 
structure.

I don't have an issue with how post_send/post_recv would work; just pointing 
out that type checking is lost, which can lead to accessing invalid memory. 
  For example, an application cannot just define struct rdma_wc and then 
call rdma_poll_cq() and have it work.  This would result in stack 
corruption, yet matches the defined API.

- Sean


From ardavis at ichips.intel.com  Wed Jul 13 14:43:05 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Wed, 13 Jul 2005 14:43:05 -0700
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <42D56AB0.6070804@cs.rutgers.edu>
References: <42D5599F.8020202@cs.rutgers.edu>	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
	<42D56AB0.6070804@cs.rutgers.edu>
Message-ID: <42D58AE9.60901@ichips.intel.com>

Aniruddha Bohra wrote:

>
> [bohra at hora-7 Target]$ pwd
> /home/bohra/jlentini/userspace/test/dapltest/udapl/Target
> [bohra at hora-7 Target]$ ldd dapltest
>       linux-gate.so.1 =>  (0xffffe000)
>       libdat.so => /usr/local/lib/libdat.so (0xb7ff6000)
>       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
>       libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
>       libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
>       /lib/ld-linux.so.2 (0x002d9000)
>
> [bohra at hora-7 Target]$ ./dapltest -T S -D mthca0a
> DT_cs_Server: Could not open mthca0a (DAT_PROVIDER_NOT_FOUND 
> DAT_NAME_NOT_REGISTERED)
> DT_cs_Server (mthca0a):  Exiting.
>
What does your /etc/dat.conf  look like?  Does it have an entry for 
mthca0a with a path to your libdapl.so?

If you built DAT and DAPL debug versions you can set 
DAT_DBG_TYPE=0xffff  for verbose debug output.

Here is an example that should work with your mthca0a name and your 
uDAPL build path

#
# DAT 1.2 configuration file
#
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
#           <provider_version> <ia_params> <platform_params>
#
# Example for openib using the first Mellanox adapter,  port 1 and port 2

mthca0a u1.2 nonthreadsafe default  
/home/bohra/jlentini/userspace/dapl/udapl/Target/libdapl.so ri.1.2 
"mthca0 1" "pd_params"


-arlin

>
> Thanks
> Aniruddha
>
>
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>



From caitlinb at siliquent.com  Wed Jul 13 14:51:16 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 13 Jul 2005 14:51:16 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057DFE@fiona.siliquent.com>


> 
> I don't have an issue with how post_send/post_recv would 
> work; just pointing out that type checking is lost, which can 
> lead to accessing invalid memory. 
>   For example, an application cannot just define struct 
> rdma_wc and then call rdma_poll_cq() and have it work.  This 
> would result in stack corruption, yet matches the defined API.
> 
> - Sean
> 

So would you favor changing the signature to match the largest
size, with  comment to the effect that you *could* use a cast
and a smaller object but only if you are sure.

I can see the danger of someone just using 'rdma_wc', so flipping
the naming convention would definitely make sense.

As to the size. Is there a size smaller than 256 that crosses
over into a significantly safer stack allocation but is still
*very* safe in terms of 'sure to be big enough'? Keep in mind
that 'struct sockaddr' has problems precisely because 'biggest
possible address' turned out to be too small.


From bohra at cs.rutgers.edu  Wed Jul 13 15:13:15 2005
From: bohra at cs.rutgers.edu (Aniruddha Bohra)
Date: Wed, 13 Jul 2005 18:13:15 -0400
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>
References: <42D5599F.8020202@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
	<42D56AB0.6070804@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>
Message-ID: <42D591FB.6010106@cs.rutgers.edu>

Hello
    I enabled the DAT_DBG_TYPE=0xffff and DAPL_DBG_TYPE=0xffff.
That gave me the answer.

    Basically libibcm is not linked with dapltest and when the dat_provider
tries to load, and calls ib_cm_send_req, it fails.

    So, just adding -libcm to the dapltest link commands make it work.
It would be great if it could be added to the svn repo.

Thanks for your help.

Regards,
Aniruddha

James Lentini wrote:

>Hmm...That looks good to me. Let's start at the bottom of the stack 
>and work our way up. Have you checked your userspace IB configuration? 
>What is the output of ib_devices and ib_devinfo? Does ibv_ud_pingpong 
>work?
>
>On Wed, 13 Jul 2005, Aniruddha Bohra wrote:
>
>bohra> James Lentini wrote:
>bohra> 
>bohra> > 
>bohra> > On Wed, 13 Jul 2005, Aniruddha Bohra wrote:
>bohra> > 
>bohra> > > Hello,
>bohra> > >   Is there a HOWTO or README to set up uDAPL for the gen2 code?
>bohra> > 
>bohra> > 
>bohra> > 
>bohra> > There is one for kDAPL but not uDAPL.
>bohra> > 
>bohra> > > I have compiled and installed the libraries, however keep get the
>bohra> > > error :
>bohra> > > DT_cs_Server: Could not open  (DAT_PROVIDER_NOT_FOUND
>bohra> > > DAT_NAME_NOT_REGISTERED)
>bohra> > > 
>bohra> > > when I try to run dapltest.
>bohra> > > 
>bohra> > > Do I need to install the kernel bits before I use uDAPL?
>bohra> > 
>bohra> > 
>bohra> > 
>bohra> > You don't need the kDAPL code to use uDAPL. You will need the IB
>bohra> > drivers and userspace libraries though.
>bohra> > 
>bohra> > Does your system have the IB drivers loaded? To check:
>bohra> > 
>bohra> > # lsmod | grep ib_
>bohra> > 
>bohra> Hello,
>bohra>   Here is the relevant info :
>bohra> 
>bohra> > [bohra at hora-7 kdapl]$ lsmod | grep ib_
>bohra> > ib_mthca              105884  0
>bohra> > ib_uverbs              27152  0
>bohra> > ib_ucm                 19468  0
>bohra> > ib_cm                  36676  1 ib_ucm
>bohra> > ib_umad                18072  0
>bohra> > ib_ipoib               43524  0
>bohra> > ib_sa                  15124  1 ib_ipoib
>bohra> > ib_mad                 40224  4 ib_mthca,ib_cm,ib_umad,ib_sa
>bohra> > ib_core                45312  7
>bohra> > ib_mthca,ib_uverbs,ib_cm,ib_umad,ib_ipoib,ib_sa,ib_mad
>bohra> 
>bohra> 
>bohra> [bohra at hora-7 Target]$ pwd
>bohra> /home/bohra/jlentini/userspace/test/dapltest/udapl/Target
>bohra> [bohra at hora-7 Target]$ ldd dapltest
>bohra>       linux-gate.so.1 =>  (0xffffe000)
>bohra>       libdat.so => /usr/local/lib/libdat.so (0xb7ff6000)
>bohra>       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00498000)
>bohra>       libc.so.6 => /lib/tls/libc.so.6 (0x002f2000)
>bohra>       libdl.so.2 => /lib/libdl.so.2 (0x0041b000)
>bohra>       /lib/ld-linux.so.2 (0x002d9000)
>bohra> 
>bohra> [bohra at hora-7 Target]$ ./dapltest -T S -D mthca0a
>bohra> DT_cs_Server: Could not open mthca0a (DAT_PROVIDER_NOT_FOUND
>bohra> DAT_NAME_NOT_REGISTERED)
>bohra> DT_cs_Server (mthca0a):  Exiting.
>bohra> 
>bohra> 
>bohra> Thanks
>bohra> Aniruddha
>bohra> 
>bohra> 
>bohra> 
>bohra> _______________________________________________
>bohra> openib-general mailing list
>bohra> openib-general at openib.org
>bohra> http://openib.org/mailman/listinfo/openib-general
>bohra> 
>bohra> To unsubscribe, please visit
>bohra> http://openib.org/mailman/listinfo/openib-general
>bohra> 
>  
>



From mshefty at ichips.intel.com  Wed Jul 13 15:31:48 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Wed, 13 Jul 2005 15:31:48 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <8508251A6FC08A489844A94261D3693A057DFE@fiona.siliquent.com>
References: <8508251A6FC08A489844A94261D3693A057DFE@fiona.siliquent.com>
Message-ID: <42D59654.7030104@ichips.intel.com>

Caitlin Bestler wrote:
> So would you favor changing the signature to match the largest
> size, with  comment to the effect that you *could* use a cast
> and a smaller object but only if you are sure.

I haven't decided on what might be the best approach for this call yet, I'd 
just like to ensure that the API provides the best performance.  If QPs and 
CQs are both transport specific, I'm wondering now if verbs is the best 
place to abstract RDMA functionality, or if it will result in losing some of 
the details provided by the underlying architecture.

There doesn't seem to be much to iwarp_verbs.h.  Assuming that the API is 
complete, iWarp seems to use a subset of the structures defined by IB.  Why 
couldn't the current data structures just be renamed from ib_blah to 
rdma_blah, with notes that some fields apply only to IB?

Some of the data structures already have fields that only apply for specific 
completion types, work requests, QPs, etc.  So, marking them for IB only 
doesn't seem like that much of a stretch, and transport neutral code would 
need to allocate the space anyway.  This way we would only need to change a 
few enums and move IB specific functions to a new file.

> I can see the danger of someone just using 'rdma_wc', so flipping
> the naming convention would definitely make sense.
> 
> As to the size. Is there a size smaller than 256 that crosses
> over into a significantly safer stack allocation but is still
> *very* safe in terms of 'sure to be big enough'? Keep in mind
> that 'struct sockaddr' has problems precisely because 'biggest
> possible address' turned out to be too small.

Work completions are used in speed path operations, so should be optimized 
for caching, retrieving multiple completions, etc., versus optimizing it for 
transport neutral code.  struct rdma_wc_storage isn't used anywhere in the 
API, so why not make that a union of struct ib_wc and struct iwarp_wc?  (Of 
course this leads us back to unions if swapping the naming convention...)

- Sean



From caitlinb at siliquent.com  Wed Jul 13 16:11:02 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Wed, 13 Jul 2005 16:11:02 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057E06@fiona.siliquent.com>

 

> 
> Work completions are used in speed path operations, so should 
> be optimized for caching, retrieving multiple completions, 
> etc., versus optimizing it for transport neutral code.  
> struct rdma_wc_storage isn't used anywhere in the API, so why 
> not make that a union of struct ib_wc and struct iwarp_wc?  
> (Of course this leads us back to unions if swapping the 
> naming convention...)
> 

Keep in mind that the most completions will be successful
SQ or recv completions, and these are transport neutral.

It is also something that is essential if you are trying
to write transport neutral middleware. Any given CQ will
be attached to a specific transport, but the middleware
would probably prefer not to think about that fact.

Of course they *could* think about it, by filling in 
either method A or method B in their higher level ops
vector based on whether they would call cq poll with
an iwarp_wc or ib_wc.

But I'm not seeing the benefit to the middleware in
terms of performance. My expectation is that they
will poll into a full size wc, then switch based
upon the information in the standard header. One chunk
won't deal with *any* non-successful completions (other
than to log the error), and others will deal with each
of the variants in a case statement (and/or function
that it invokes). After the split referencing the
specific sub-variant, by union or cast, has no
real performance penalty (just a typing nuisance penalty).

Having a union of all _wc types would be great, if there
was a way to get it without having to pick up all of 
the transport specific information. As noted in my
prior post, "open unions" aren't supported (nor are
"open enums" which would be great for accumulating
error codes).





From halr at voltaire.com  Wed Jul 13 16:14:24 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 19:14:24 -0400
Subject: [openib-general] [PATCHv2] libibcm: Fix some typos
Message-ID: <1121296463.4423.33.camel@hal.voltaire.com>

libibcm: Fix some typos

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: include/infiniband/cm.h
===================================================================
--- include/infiniband/cm.h	(revision 2824)
+++ include/infiniband/cm.h	(working copy)
@@ -274,7 +274,7 @@ int ib_cm_event_put(struct ib_cm_event *
  * The primary use of the file descriptor is to test for CM readiness
  * events. When the CM becomes ready to READ there is a pending event
  * ready, and a subsequent call to ib_cm_event_get will not block.
- * Note: The user should not read or writ directly to the CM file
+ * Note: The user should not read or write directly to the CM file
  *       descriptor, it will likely result in an error or unexpected
  *       results.
  */
@@ -300,6 +300,7 @@ struct ib_cm_attr_param {
 	uint32_t		local_id;
 	uint32_t		remote_id;
 };
+
 /**
  * ib_cm_attr_id - Get connection identifier attributes.
  * @cm_id: Connection identifier to retrieve attributes.
Index: src/cm.c
===================================================================
--- src/cm.c	(revision 2824)
+++ src/cm.c	(working copy)
@@ -181,7 +181,6 @@ int ib_cm_attr_id(uint32_t cm_id, struct
 
 	cmd->id = cm_id;
 
-
 	result = write(fd, msg, size);
 	if (result != size)
 		return (result > 0) ? -ENODATA : result;
Index: examples/simple.c
===================================================================
--- examples/simple.c	(revision 2824)
+++ examples/simple.c	(working copy)
@@ -124,6 +124,7 @@ int main(int argc, char **argv)
 	int param_c = 0;
 	int status = 0;
 	int mode;
+
 	/*
 	 * read command line.
 	 */
@@ -153,7 +154,7 @@ int main(int argc, char **argv)
 	else {
 		result = ib_cm_listen(cm_id, TEST_SID, 0);
 		if (result) {
-			printf("Error <%d:%d> listneing <%d>\n", 
+			printf("Error <%d:%d> listening <%d>\n", 
 			       result, errno, cm_id);
 			goto done;
 		}





From halr at voltaire.com  Wed Jul 13 16:24:48 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 19:24:48 -0400
Subject: [openib-general] Re: [PATCHv2} ucm: Only print messages when
	debug level set
In-Reply-To: <20050713204346.GC9093@granada.merseine.nu>
References: <1121265670.4401.328.camel@hal.voltaire.com>
	<20050713202933.GC26048@mellanox.co.il>
	<20050713204346.GC9093@granada.merseine.nu>
Message-ID: <1121297088.4423.35.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 16:43, Muli Ben-Yehuda wrote:
> On Wed, Jul 13, 2005 at 11:29:33PM +0300, Michael S. Tsirkin wrote:
> > Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > @@ -56,6 +61,10 @@ enum {
> > >  
> > >  #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
> > >  
> > > +#define ucm_dbg(format, arg...)                 \
> > > +	if (ucm_debug_level > 0)        \
> > > +		printk(KERN_DEBUG format, ## arg)
> > > +
> > 
> > What happened to whitespace here?
> > Also, may it be a good idea for ucm_dbg to add the UCM: prefix
> automatically?
> 
> You also want to wrap this thing up in a do { } while (0) loop (see
> http://www.kernelnewbies.org/faq/, "Why do a lot of #defines in the
> kernel use do { ... } while(0)?")

Thanks. I'll do that in my next version of the patch.

-- Hal



From halr at voltaire.com  Wed Jul 13 16:31:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 19:31:31 -0400
Subject: [openib-general] Re: [PATCHv2} ucm: Only print messages when debug
	level set
In-Reply-To: <20050713202933.GC26048@mellanox.co.il>
References: <1121265670.4401.328.camel@hal.voltaire.com>
	<20050713202933.GC26048@mellanox.co.il>
Message-ID: <1121297490.4423.45.camel@hal.voltaire.com>

On Wed, 2005-07-13 at 16:29, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > @@ -56,6 +61,10 @@ enum {
> >  
> >  #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
> >  
> > +#define ucm_dbg(format, arg...)                 \
> > +	if (ucm_debug_level > 0)        \
> > +		printk(KERN_DEBUG format, ## arg)
> > +
> 
> What happened to whitespace here?

Not sure. Should be fixed in next version of patch.

> Also, may it be a good idea for ucm_dbg to add the UCM: prefix automatically?

OK.

-- Hal



From halr at voltaire.com  Wed Jul 13 16:40:07 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 13 Jul 2005 19:40:07 -0400
Subject: [openib-general] [PATCHv3] ucm: Only print messages when debug
	level set
Message-ID: <1121298005.4423.56.camel@hal.voltaire.com>

ucm: Only print messages when debug level set

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

--- ucm.c.kfree	2005-07-13 08:10:00.000000000 -0400
+++ ucm.c	2005-07-13 19:39:04.000000000 -0400
@@ -49,6 +49,11 @@ MODULE_AUTHOR("Libor Michalek");
 MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
 MODULE_LICENSE("Dual BSD/GPL");
 
+static int ucm_debug_level;
+
+module_param_named(debug_level, ucm_debug_level, int, 0644);
+MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0");
+
 enum {
 	IB_UCM_MAJOR = 231,
 	IB_UCM_MINOR = 255
@@ -56,6 +61,14 @@ enum {
 
 #define IB_UCM_DEV MKDEV(IB_UCM_MAJOR, IB_UCM_MINOR)
 
+#define PFX "UCM: "
+
+#define ucm_dbg(format, arg...)			\
+	do {					\
+		if (ucm_debug_level > 0)	\
+			printk(KERN_DEBUG PFX format, ## arg); \
+	} while (0)
+
 static struct semaphore ctx_id_mutex;
 static struct idr       ctx_id_table;
 static int              ctx_id_rover = 0;
@@ -107,7 +120,7 @@ static void ib_ucm_ctx_put(struct ib_ucm
 
 	up(&ctx->file->mutex);
 
-	printk(KERN_ERR "UCM: Destroyed CM ID <%d>\n", ctx->id);
+	ucm_dbg("Destroyed CM ID <%d>\n", ctx->id);
 
 	ib_destroy_cm_id(ctx->cm_id);
 	kfree(ctx);
@@ -145,7 +158,7 @@ retry:
 	if (result)
 		goto error;
 
-	printk(KERN_ERR "UCM: Allocated CM ID <%d>\n", ctx->id);
+	ucm_dbg("Allocated CM ID <%d>\n", ctx->id);
 
 	return ctx;
 error:
@@ -405,8 +418,7 @@ static int ib_ucm_event_handler(struct i
 		break;
 	}
 
-	printk(KERN_ERR "UCM: Event. CM ID <%d> event <%d>\n",
-	       id, event->event);
+	ucm_dbg("Event. CM ID <%d> event <%d>\n", id, event->event);
 
 	ctx = ib_ucm_ctx_get(id);
 	if (!ctx)
@@ -1234,8 +1246,8 @@ static ssize_t ib_ucm_write(struct file 
 	if (copy_from_user(&hdr, buf, sizeof(hdr)))
 		return -EFAULT;
 
-	printk(KERN_ERR "UCM: Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
-	       hdr.cmd, hdr.in, hdr.out, len);
+	ucm_dbg("Write. cmd <%d> in <%d> out <%d> len <%Zu>\n",
+		hdr.cmd, hdr.in, hdr.out, len);
 
 	if (hdr.cmd < 0 || hdr.cmd >= ARRAY_SIZE(ucm_cmd_table))
 		return -EINVAL;
@@ -1282,7 +1294,7 @@ static int ib_ucm_open(struct inode *ino
 	filp->private_data = file;
 	file->filp = filp;
 
-	printk(KERN_ERR "UCM: Created struct\n");
+	ucm_dbg("Created struct\n");
 
 	return 0;
 }
@@ -1308,7 +1320,7 @@ static int ib_ucm_close(struct inode *in
 
 	kfree(file);
 
-	printk(KERN_ERR "UCM: Deleted struct\n");
+	ucm_dbg("Deleted struct\n");
 	return 0;
 }
 
@@ -1330,7 +1342,7 @@ static int __init ib_ucm_init(void)
 
 	result = register_chrdev_region(IB_UCM_DEV, 1, "infiniband_cm");
 	if (result) {
-		printk(KERN_ERR "UCM: Error <%d> registering dev\n", result);
+		ucm_dbg("Error <%d> registering dev\n", result);
 		goto err_chr;
 	}
 
@@ -1338,14 +1350,14 @@ static int __init ib_ucm_init(void)
 
 	result = cdev_add(&ib_ucm_cdev, IB_UCM_DEV, 1);
 	if (result) {
-		printk(KERN_ERR "UCM: Error <%d> adding cdev\n", result);
+		ucm_dbg("Error <%d> adding cdev\n", result);
 		goto err_cdev;
 	}
 
 	ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm");
 	if (IS_ERR(ib_ucm_class)) {
 		result = PTR_ERR(ib_ucm_class);
-		printk(KERN_ERR "UCM: Error <%d> creating class\n", result);
+		ucm_dbg("Error <%d> creating class\n", result);
 		goto err_class;
 	}
 





From eitan at mellanox.co.il  Thu Jul 14 00:25:59 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 10:25:59 +0300
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304A2@mtlex01.yok.mtl.com>

Hi Hal,

I was able to figure out the gcc version you use is available under RH9.0.
I compiled the code successfully under that machine .
I wonder if the issue is due to complib (which is part of the OpenSM code).
I will start building OpenIB stack shortly and will let you know.
Meanwhile I attached the log file of build I have done
Please try a fresh checkout too.

EZ

 -----Original Message-----
From: 	Hal Rosenstock [mailto:halr at voltaire.com] 
Sent:	Wednesday, July 13, 2005 6:47 PM
To:	Eitan Zahavi
Cc:	openib-general at openib.org
Subject:	RE: [Fwd: [openib-commits] r2826 - in
gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]

Hi Eitan,

On Wed, 2005-07-13 at 11:29, Eitan Zahavi wrote:
> Hi Hal,
> 
> I could not reproduce the error you see in compiling sim.cpp.
> 
> Can you elaborate on what kernel/gcc versions you see this?

It may be due to an older gcc. I am using:
gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)

Bernhard also saw this.

-- Hal
 <<ibmgtsim.build.log>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/73b65811/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ibmgtsim.build.log
Type: application/octet-stream
Size: 15857 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/73b65811/attachment.obj>

From tziporet at mellanox.co.il  Thu Jul 14 04:14:04 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Thu, 14 Jul 2005 14:14:04 +0300
Subject: [openib-general] VAPI_RETRY_EXC_ERR, vendor code=81
Message-ID: <506C3D7B14CDD411A52C00025558DED6064BF8B1@mtlex01.yok.mtl.com>

Its means that you have some connectivity problems thus got retry exceeded.
Check your cables or see if the switch dropped packets.
 
Tziporet

-----Original Message-----
From: Srirangam Addepalli [mailto:rangam at gmail.com]
Sent: Wednesday, July 13, 2005 9:52 PM
To: openib-general at openib.org
Subject: [openib-general] VAPI_RETRY_EXC_ERR, vendor code=81


Hello All,
            I was trying to test bandwidth using the following command
 
/usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 minigar c0-0
/usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
 
How ever i get the following error messages.
 
/usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 c0-1 c0-0
/usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
Got completion with error, code=VAPI_RETRY_EXC_ERR, vendor code=81
mpi_bandwidth: mpid/ch_vapi/viacheck.c:2109: viutil_spinandwaitcq: Assertion
`sc->status == VAPI_SUCCESS' failed. 
bash: line 1:  6813 Aborted                 /usr/bin/env MPIRUN_HOST=c0-1
MPIRUN_PORT=34474 MPIRUN_RANK=1 MPIRUN_NPROCS=2 MPIRUN_ID=13416
MPIRUN_NODENAME=c0-0 /usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000
262144 
Timeout alarm signaled
Cleaning up all processes ...done.
Killed by signal 15.
 
why do i get this error. 
 
Rangam
 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/89c01e2a/attachment.html>

From halr at voltaire.com  Thu Jul 14 04:12:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 14 Jul 2005 07:12:39 -0400
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C304A2@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C304A2@mtlex01.yok.mtl.com>
Message-ID: <1121339559.4423.930.camel@hal.voltaire.com>

On Thu, 2005-07-14 at 03:25, Eitan Zahavi wrote:
> Hi Hal,
> 
> I was able to figure out the gcc version you use is available under
> RH9.0.
> I compiled the code successfully under that machine .

Are you sure it's the same version ?

> I wonder if the issue is due to complib (which is part of the OpenSM
> code).
> I will start building OpenIB stack shortly and will let you know.
> Meanwhile I attached the log file of build I have done
> Please try a fresh checkout too.

Did something change that would fix this ? I did this again and still
get the same error:

g++ -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include -I/home/hal/openib/trunk/src/userspace/management/osm/include -I/usr/include -I../datamodel   -O6 -Wall -Werror  -c -o sim.o `test -f 'sim.cpp' || echo './'`sim.cpp
sim.cpp:342: parse error at end of input

-- Hal




From eitan at mellanox.co.il  Thu Jul 14 04:19:58 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 14:19:58 +0300
Subject: [openib-general] RE: [Fwd: [openib-commits] r2826 - in
	gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304A9@mtlex01.yok.mtl.com>

Hi Hal,

I did not reproduce the problem ever.
But I did many changes too.

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL



-----Original Message-----
From: Hal Rosenstock [mailto:halr at voltaire.com]
Sent: Thursday, July 14, 2005 2:13 PM
To: Eitan Zahavi
Cc: openib-general at openib.org
Subject: RE: [Fwd: [openib-commits] r2826 - in
gen2/utils/src/linux-user/I BMgtSim: . config src tests utils]


On Thu, 2005-07-14 at 03:25, Eitan Zahavi wrote:
> Hi Hal,
> 
> I was able to figure out the gcc version you use is available under
> RH9.0.
> I compiled the code successfully under that machine .

Are you sure it's the same version ?

> I wonder if the issue is due to complib (which is part of the OpenSM
> code).
> I will start building OpenIB stack shortly and will let you know.
> Meanwhile I attached the log file of build I have done
> Please try a fresh checkout too.

Did something change that would fix this ? I did this again and still
get the same error:

g++ -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include
-I/home/hal/openib/trunk/src/userspace/management/osm/include -I/usr/include
-I../datamodel   -O6 -Wall -Werror  -c -o sim.o `test -f 'sim.cpp' || echo
'./'`sim.cpp
sim.cpp:342: parse error at end of input

-- Hal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/7d07f4e2/attachment.html>

From halr at voltaire.com  Thu Jul 14 04:31:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 14 Jul 2005 07:31:00 -0400
Subject: [openib-general] Re: A Couple of IPoIB related issues
In-Reply-To: <52pstu1kog.fsf@topspin.com>
References: <1120762632.4386.44.camel@hal.voltaire.com>
	<52pstu1kog.fsf@topspin.com>
Message-ID: <1121339564.4423.932.camel@hal.voltaire.com>

On Thu, 2005-07-07 at 15:27, Roland Dreier wrote:
>     Hal> 1. NETDEV WATCHDOG: ib0: transmit timed out ib0: transmit
>     Hal> timeout: latency 360052
> 
>     Hal> This occurs once a minute on heavy pings.
> 
> Exactly once a minute, or is this just a rough frequency?  What is a
> heavy ping?  Does the IPoIB interface still work when you see this?

I should have said they are exactly once per second (rather than
1/minute).

Jun 19 19:34:01 mo1 kernel: NETDEV WATCHDOG: ib0: transmit timed out
Jun 19 19:34:01 mo1 kernel: ib0: transmit timeout: latency 3879
Jun 19 19:34:02 mo1 kernel: NETDEV WATCHDOG: ib0: transmit timed out
Jun 19 19:34:02 mo1 kernel: ib0: transmit timeout: latency 4879
Jun 19 19:34:03 mo1 kernel: NETDEV WATCHDOG: ib0: transmit timed out
Jun 19 19:34:03 mo1 kernel: ib0: transmit timeout: latency 5879
...

A "heavy ping" is twenty concurrent flood pings.
During these warnings the ipoib interface does seem to work.

> Is there more traceback that shows who is doing the allocation that
> fails?  In any case it looks like you are just running low on memory.
> 
>     Hal> Any idea on what could be causing these or how to go about
>     Hal> isolating them ?
> 
> The __alloc_pages() warnings look fairly benign and can probably be
> fixed by tuning /proc/sys/vm/min_free_kbytes appropriately.
> 
> The TX timeout is somewhat odd.  I guess we need to figure out why the
> netdevice queue is stopped for a long time.

How can that be determined ?

Also, is there missing code on handling ipoib timeouts ?

-- Hal



From rolandd at cisco.com  Thu Jul 14 07:27:39 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 07:27:39 -0700
Subject: [openib-general] [ANNOUNCE] Initial checkin of SRP initiator
References: <52r7e8muj1.fsf@topspin.com>
	<78d18e2050713072416e026d@mail.gmail.com>
Message-ID: <52u0ixea3o.fsf@topspin.com>

    William> The README appears to be missing from the PyOpenIB tar
    William> file. It doesn't look too difficult to figure out if you
    William> know python, but unfortunately I don't. I'm sure I need
    William> to build ioctl.c into something, but I don't know what.

Sorry, I left the README out.  Here it is (along with the tarball again).

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: README
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/2102a1e1/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PyOpenIB-1.0.tar.bz2
Type: application/x-bzip-compressed-tar
Size: 5876 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/2102a1e1/attachment.bin>

From eitan at mellanox.co.il  Thu Jul 14 07:28:14 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 17:28:14 +0300
Subject: [openib-general] Introducing IB MAD TCL API
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304AE@mtlex01.yok.mtl.com>

Hi All,

I have uploaded a new sub project for TCL MAD API into the directory
https://openib.org/svn/gen2/utils/src/linux-user/
<https://openib.org/svn/gen2/utils/src/linux-user/> 
This API enables development of IB Management utilities and scripts in TCL.

I have also introduced a "top level" autotools project for the utilities sub
tree including: 
ibdm - IB datamodel (in memory data model for nodes, links, systems, etc and
analysis utilities )
ibmgtsim - IB management simulator
ibis - TCL API for IB MADs

I think that this autotools method for building top level projects can be
utilized for the rest of the OpenIB user level code.

I plan to actively develop the above components in the OpenIB tree. 

Also I call for moving this sub-tree of user-level utilities into the OpenIB
main trunk.

Eitan 

Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/26a06d07/attachment.html>

From halr at voltaire.com  Thu Jul 14 07:53:04 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 14 Jul 2005 10:53:04 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
Message-ID: <1121351537.4423.1162.camel@hal.voltaire.com>

Hi Arlin,

On Wed, 2005-07-13 at 12:57, Arlin Davis wrote:
> >One more question: Are you using IB_AT_ROUTE_FORCE_RESOLVE or not in the
> >flags to the ib_at_route_by_ip call ? (What is the flag setting ?)
> 
> no, tos and flags are both set to 0.

OK. I looked into this and it appears to me that the route is not copied
back to userspace until the callback is made.

-- Hal



From halr at voltaire.com  Thu Jul 14 08:04:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 14 Jul 2005 11:04:53 -0400
Subject: [openib-general] Introducing IB MAD TCL API
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C304AE@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C304AE@mtlex01.yok.mtl.com>
Message-ID: <1121353492.4423.1225.camel@hal.voltaire.com>

On Thu, 2005-07-14 at 10:28, Eitan Zahavi wrote:
> Hi All,
> 
> I have uploaded a new sub project for TCL MAD API into the directory
> https://openib.org/svn/gen2/utils/src/linux-user/
> 
> This API enables development of IB Management utilities and scripts in
> TCL.

Does this build ? 

I get the following when I run autogen:
src/Makefile.am:46: AM_CFLAGS was already defined in condition TRUE,
which implies condition OSM_VENDOR_MTL_TRUE

> I have also introduced a "top level" autotools project for the
> utilities sub tree including: 
> 
> ibdm - IB datamodel (in memory data model for nodes, links, systems,
> etc and analysis utilities )
> 
> ibmgtsim - IB management simulator
> 
> ibis - TCL API for IB MADs

Are these all ported and working in OpenIB ?

Do the files have the correct licenses ?

> I think that this autotools method for building top level projects can
> be utilized for the rest of the OpenIB user level code.
> 
> I plan to actively develop the above components in the OpenIB tree. 

Glad to hear this.

> Also I call for moving this sub-tree of user-level utilities into the
> OpenIB main trunk.

The trunk is for eligible for release. IMO, it seems a little early yet
for this. I would like to get them all built for OpenIB and play wit
them a little. Also, I fwould like to understand these better and their
implications. I would also be interested in hearing other opinions and
how wide the community of interest for these tools is.

-- Hal



From rolandd at cisco.com  Thu Jul 14 08:12:39 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 08:12:39 -0700
Subject: [openib-general] Generic device driver
References: <200507131919.j6DJJtC8015460@ns1.baymicrosystems.com>
Message-ID: <52d5ple80o.fsf@topspin.com>

    Suresh> Another newbee question. I only see a Mellanox HCA driver
    Suresh> in the gen2/trunk, which is a PCI device.  Where as, we
    Suresh> have a proprietary implementation of the HCA, where the
    Suresh> PHY is accessed by the host via an FPGA (Host-bridge).

    Suresh> 1. Is there a Generic Device Driver (I see it being
    Suresh> mentioned in many places).  2. If none, can I replace some
    Suresh> files/functions within the Mellanox driver with the low
    Suresh> level addresses etc, to achieve the same.

You will need to write a driver for your hardware.  One way to think
of it is if you have an ethernet interface, then you need to write a
NIC driver for your specific hardware.

For InfiniBand, your driver needs to discover the hardware.  For each
IB device, the driver must fill in a struct ib_device with function
pointers for the methods it implements, and pass the ib_device to
ib_register_device().  In mthca, this is done in mthca_register_device().

 - R.


From mst at mellanox.co.il  Thu Jul 14 08:21:11 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 14 Jul 2005 18:21:11 +0300
Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs
In-Reply-To: <20050710070512.GW14130@mellanox.co.il>
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
	<52pstttg1b.fsf@topspin.com>
	<20050710070512.GW14130@mellanox.co.il>
Message-ID: <20050714152111.GB2856@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> Subject: Re: [PATCH] mthca: report board id in sysfs
> 
> Quoting r. Roland Dreier <rolandd at cisco.com>:
> > Subject: Re: [openib-general] [PATCH] mthca: report board id in sysfs
> > 
> >     Michael> To check, you can type
> > 
> >     Michael> mstflint -d <bus:dev.fn> q
> > 
> > Looks like the board ID is not there.
> > 
> > I think I understand what's happening now: on Topspin HCAs, the
> > vendor-specific area is formatted differently, and the board ID string
> > is not in the last 16 bytes of the VSD.
> > 
> > What's the best way to make this work for Topspin HCAs?
> > 
> >  - R.
> > 
> 
> Since topspin cards have a distinct vendor id, its possible they dont
> need the board id at all.
> 
> What'd you like to do?
> - Report empty board id (thats what current code does). User can follow
>   the device link and use vendor id to figure out its a topspin card.
> - Check topspin vendor id and report some hard-coded string in board_id.
> - Check topspin vendor id and report some other field from the vpd.

Any ideas yet?
I'd like to add that what this patch does is basically get parity with the
VAPI vstat functionality.

-- 
MST


From rolandd at cisco.com  Thu Jul 14 08:37:01 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 08:37:01 -0700
Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs
In-Reply-To: <20050714152111.GB2856@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 14 Jul 2005 18:21:11 +0300")
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
	<52pstttg1b.fsf@topspin.com> <20050710070512.GW14130@mellanox.co.il>
	<20050714152111.GB2856@mellanox.co.il>
Message-ID: <52pstlcsbm.fsf@topspin.com>

    Michael> Since topspin cards have a distinct vendor id, its
    Michael> possible they dont need the board id at all.
    Michael> 
    Michael> What'd you like to do?  - Report empty board id (thats
    Michael> what current code does). User can follow the device link
    Michael> and use vendor id to figure out its a topspin card.  -
    Michael> Check topspin vendor id and report some hard-coded string
    Michael> in board_id.  - Check topspin vendor id and report some
    Michael> other field from the vpd.

    Michael> Any ideas yet?  I'd like to add that what this patch does
    Michael> is basically get parity with the VAPI vstat
    Michael> functionality.

Sorry, I've been on vacation...

Anyway, unfortunately most Topspin/Cisco cards don't have a distinct
vendor ID -- they ship with the Mellanox vendor ID.

The only difference is our VSD format.  We do have a board ID in the
VSD, but it's at a different offset and so QUERY_ADAPTER doesn't
return it to the driver, as far as I know.

 - R.


From rolandd at cisco.com  Thu Jul 14 08:38:52 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 08:38:52 -0700
Subject: [openib-general] Re: A Couple of IPoIB related issues
In-Reply-To: <1121339564.4423.932.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "14 Jul 2005 07:31:00 -0400")
References: <1120762632.4386.44.camel@hal.voltaire.com>
	<52pstu1kog.fsf@topspin.com>
	<1121339564.4423.932.camel@hal.voltaire.com>
Message-ID: <52ll49cs8j.fsf@topspin.com>

    Hal> A "heavy ping" is twenty concurrent flood pings.  During
    Hal> these warnings the ipoib interface does seem to work.

>From 20 separate sources?  I don't have that big a test network, but I
can try to reproduce it.

    Hal> Also, is there missing code on handling ipoib timeouts ?

We might want to add something to handle the hardware getting wedged,
but I'm not sure what.

This case seems to be that the system is so busy handling receives
that it doesn't get a chance to handle transmits.  Maybe I need to
dust off my NAPI patch after all.

 - R.


From mst at mellanox.co.il  Thu Jul 14 08:45:56 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 14 Jul 2005 18:45:56 +0300
Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs
In-Reply-To: <52pstlcsbm.fsf@topspin.com>
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
	<52pstttg1b.fsf@topspin.com>
	<20050710070512.GW14130@mellanox.co.il>
	<20050714152111.GB2856@mellanox.co.il> <52pstlcsbm.fsf@topspin.com>
Message-ID: <20050714154556.GA3044@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [PATCH] mthca: report board id in sysfs
> 
>     Michael> Since topspin cards have a distinct vendor id, its
>     Michael> possible they dont need the board id at all.
>     Michael> 
>     Michael> What'd you like to do?  - Report empty board id (thats
>     Michael> what current code does). User can follow the device link
>     Michael> and use vendor id to figure out its a topspin card.  -
>     Michael> Check topspin vendor id and report some hard-coded string
>     Michael> in board_id.  - Check topspin vendor id and report some
>     Michael> other field from the vpd.
> 
>     Michael> Any ideas yet?  I'd like to add that what this patch does
>     Michael> is basically get parity with the VAPI vstat
>     Michael> functionality.
> 
> Sorry, I've been on vacation...
> 
> Anyway, unfortunately most Topspin/Cisco cards don't have a distinct
> vendor ID -- they ship with the Mellanox vendor ID.
> 
> The only difference is our VSD format.  We do have a board ID in the
> VSD, but it's at a different offset and so QUERY_ADAPTER doesn't
> return it to the driver, as far as I know.
> 
>  - R.
> 

AFAIK, all of VSD gets reported back to the driver.
What do you say about merging my patch, and then either

- report empty board id as vapi does

- add support for topspin cards along the following lines:
  if board_id is empty, get it at another offset.

  I could code that up myself but I dont have any topspin hardware.

-- 
MST


From rolandd at cisco.com  Thu Jul 14 08:54:24 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 08:54:24 -0700
Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs
In-Reply-To: <20050714154556.GA3044@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 14 Jul 2005 18:45:56 +0300")
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
	<52pstttg1b.fsf@topspin.com> <20050710070512.GW14130@mellanox.co.il>
	<20050714152111.GB2856@mellanox.co.il> <52pstlcsbm.fsf@topspin.com>
	<20050714154556.GA3044@mellanox.co.il>
Message-ID: <52hdexcrin.fsf@topspin.com>

    Michael> AFAIK, all of VSD gets reported back to the driver.  What
    Michael> do you say about merging my patch, and then either

Yes, I just tried it and it seems to be there.  I think I misread the
PRM before.

    Michael> - add support for topspin cards along the following
    Michael> lines: if board_id is empty, get it at another offset.

I'll add code like that.  There's actually a Topspin signature in the
VSD, so we can tell when to use Topspin format.

Can anyone from Voltaire and Silverstorm comment on what VSD format is
used on your HCAs?

 - R.


From shault at voltaire.com  Thu Jul 14 08:57:16 2005
From: shault at voltaire.com (Shaul Tamari)
Date: Thu, 14 Jul 2005 18:57:16 +0300
Subject: [openib-general] Re: A Couple of IPoIB related issues
Message-ID: <D4F8F0B3820E754C887699BEF26A8940905717@taurus.voltaire.com>

[snip]
>     Hal> A "heavy ping" is twenty concurrent flood pings.  During
>     Hal> these warnings the ipoib interface does seem to work.
Roland> >From 20 separate sources?

nope


Roland> I don't have that big a test network, but I can try to reproduce
it.
[snip]

it was two hosts only. one of them flood pinged the other.

example (10 concurrent flood pings):
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &
host1# ping -f host2 > /dev/null &

(the /dev/null redirection is used reduce the ssh overhead)


saul


From jlentini at netapp.com  Thu Jul 14 09:27:19 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 14 Jul 2005 12:27:19 -0400 (EDT)
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <42D591FB.6010106@cs.rutgers.edu>
References: <42D5599F.8020202@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
	<42D56AB0.6070804@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>
	<42D591FB.6010106@cs.rutgers.edu>
Message-ID: <Pine.LNX.4.61.0507141226140.25603@jlentini-linux.nane.netapp.com>


On Wed, 13 Jul 2005, Aniruddha Bohra wrote:

>   Basically libibcm is not linked with dapltest and when the dat_provider
> tries to load, and calls ib_cm_send_req, it fails.
>
>   So, just adding -libcm to the dapltest link commands make it work.
> It would be great if it could be added to the svn repo.

Thanks for pointing this out. I've just updated the makefile (revision 
2859).

james


From rolandd at cisco.com  Thu Jul 14 09:28:06 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 09:28:06 -0700
Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs
In-Reply-To: <20050714152111.GB2856@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 14 Jul 2005 18:21:11 +0300")
References: <20050627150308.GD25304@mellanox.co.il>
	<521x6ayn72.fsf@topspin.com> <20050708055506.GA4989@mellanox.co.il>
	<52pstttg1b.fsf@topspin.com> <20050710070512.GW14130@mellanox.co.il>
	<20050714152111.GB2856@mellanox.co.il>
Message-ID: <52d5plcpyh.fsf@topspin.com>

OK, here's my latest diff.  Seems to work on both Topspin and Mellanox HCAs...

Index: infiniband/hw/mthca/mthca_dev.h
===================================================================
--- infiniband/hw/mthca/mthca_dev.h	(revision 2822)
+++ infiniband/hw/mthca/mthca_dev.h	(working copy)
@@ -68,6 +68,10 @@ enum {
 };
 
 enum {
+	MTHCA_BOARD_ID_LEN = 64
+};
+
+enum {
 	MTHCA_EQ_CONTEXT_SIZE =  0x40,
 	MTHCA_CQ_CONTEXT_SIZE =  0x40,
 	MTHCA_QP_CONTEXT_SIZE = 0x200,
@@ -247,6 +251,7 @@ struct mthca_dev {
 	unsigned long    device_cap_flags;
 
 	u32              rev_id;
+	char             board_id[MTHCA_BOARD_ID_LEN];
 
 	/* firmware info */
 	u64              fw_ver;
Index: infiniband/hw/mthca/mthca_main.c
===================================================================
--- infiniband/hw/mthca/mthca_main.c	(revision 2822)
+++ infiniband/hw/mthca/mthca_main.c	(working copy)
@@ -213,7 +213,6 @@ static int __devinit mthca_init_tavor(st
 	struct mthca_dev_lim        dev_lim;
 	struct mthca_profile        profile;
 	struct mthca_init_hca_param init_hca;
-	struct mthca_adapter        adapter;
 
 	err = mthca_SYS_EN(mdev, &status);
 	if (err) {
@@ -271,26 +270,8 @@ static int __devinit mthca_init_tavor(st
 		goto err_disable;
 	}
 
-	err = mthca_QUERY_ADAPTER(mdev, &adapter, &status);
-	if (err) {
-		mthca_err(mdev, "QUERY_ADAPTER command failed, aborting.\n");
-		goto err_close;
-	}
-	if (status) {
-		mthca_err(mdev, "QUERY_ADAPTER returned status 0x%02x, "
-			  "aborting.\n", status);
-		err = -EINVAL;
-		goto err_close;
-	}
-
-	mdev->eq_table.inta_pin = adapter.inta_pin;
-	mdev->rev_id            = adapter.revision_id;
-
 	return 0;
 
-err_close:
-	mthca_CLOSE_HCA(mdev, 0, &status);
-
 err_disable:
 	mthca_SYS_DIS(mdev, &status);
 
@@ -507,7 +488,6 @@ static int __devinit mthca_init_arbel(st
 	struct mthca_dev_lim        dev_lim;
 	struct mthca_profile        profile;
 	struct mthca_init_hca_param init_hca;
-	struct mthca_adapter        adapter;
 	u64 icm_size;
 	u8 status;
 	int err;
@@ -575,21 +555,6 @@ static int __devinit mthca_init_arbel(st
 		goto err_free_icm;
 	}
 
-	err = mthca_QUERY_ADAPTER(mdev, &adapter, &status);
-	if (err) {
-		mthca_err(mdev, "QUERY_ADAPTER command failed, aborting.\n");
-		goto err_free_icm;
-	}
-	if (status) {
-		mthca_err(mdev, "QUERY_ADAPTER returned status 0x%02x, "
-			  "aborting.\n", status);
-		err = -EINVAL;
-		goto err_free_icm;
-	}
-
-	mdev->eq_table.inta_pin = adapter.inta_pin;
-	mdev->rev_id            = adapter.revision_id;
-
 	return 0;
 
 err_free_icm:
@@ -615,12 +580,68 @@ err_disable:
 	return err;
 }
 
+static void mthca_close_hca(struct mthca_dev *mdev)
+{
+	u8 status;
+
+	mthca_CLOSE_HCA(mdev, 0, &status);
+
+	if (mthca_is_memfree(mdev)) {
+		mthca_free_icm_table(mdev, mdev->cq_table.table);
+		mthca_free_icm_table(mdev, mdev->qp_table.rdb_table);
+		mthca_free_icm_table(mdev, mdev->qp_table.eqp_table);
+		mthca_free_icm_table(mdev, mdev->qp_table.qp_table);
+		mthca_free_icm_table(mdev, mdev->mr_table.mpt_table);
+		mthca_free_icm_table(mdev, mdev->mr_table.mtt_table);
+		mthca_unmap_eq_icm(mdev);
+
+		mthca_UNMAP_ICM_AUX(mdev, &status);
+		mthca_free_icm(mdev, mdev->fw.arbel.aux_icm);
+
+		mthca_UNMAP_FA(mdev, &status);
+		mthca_free_icm(mdev, mdev->fw.arbel.fw_icm);
+
+		if (!(mdev->mthca_flags & MTHCA_FLAG_NO_LAM))
+			mthca_DISABLE_LAM(mdev, &status);
+	} else
+		mthca_SYS_DIS(mdev, &status);
+}
+
 static int __devinit mthca_init_hca(struct mthca_dev *mdev)
 {
+	u8 status;
+	int err;
+	struct mthca_adapter adapter;
+
 	if (mthca_is_memfree(mdev))
-		return mthca_init_arbel(mdev);
+		err = mthca_init_arbel(mdev);
 	else
-		return mthca_init_tavor(mdev);
+		err = mthca_init_tavor(mdev);
+
+	if (err)
+		return err;
+
+	err = mthca_QUERY_ADAPTER(mdev, &adapter, &status);
+	if (err) {
+		mthca_err(mdev, "QUERY_ADAPTER command failed, aborting.\n");
+		goto err_close;
+	}
+	if (status) {
+		mthca_err(mdev, "QUERY_ADAPTER returned status 0x%02x, "
+			  "aborting.\n", status);
+		err = -EINVAL;
+		goto err_close;
+	}
+
+	mdev->eq_table.inta_pin = adapter.inta_pin;
+	mdev->rev_id            = adapter.revision_id;
+	memcpy(mdev->board_id, adapter.board_id, sizeof mdev->board_id);
+
+	return 0;
+
+err_close:
+	mthca_close_hca(mdev);
+	return err;
 }
 
 static int __devinit mthca_setup_hca(struct mthca_dev *dev)
@@ -845,33 +866,6 @@ static int __devinit mthca_enable_msi_x(
 	return 0;
 }
 
-static void mthca_close_hca(struct mthca_dev *mdev)
-{
-	u8 status;
-
-	mthca_CLOSE_HCA(mdev, 0, &status);
-
-	if (mthca_is_memfree(mdev)) {
-		mthca_free_icm_table(mdev, mdev->cq_table.table);
-		mthca_free_icm_table(mdev, mdev->qp_table.rdb_table);
-		mthca_free_icm_table(mdev, mdev->qp_table.eqp_table);
-		mthca_free_icm_table(mdev, mdev->qp_table.qp_table);
-		mthca_free_icm_table(mdev, mdev->mr_table.mpt_table);
-		mthca_free_icm_table(mdev, mdev->mr_table.mtt_table);
-		mthca_unmap_eq_icm(mdev);
-
-		mthca_UNMAP_ICM_AUX(mdev, &status);
-		mthca_free_icm(mdev, mdev->fw.arbel.aux_icm);
-
-		mthca_UNMAP_FA(mdev, &status);
-		mthca_free_icm(mdev, mdev->fw.arbel.fw_icm);
-
-		if (!(mdev->mthca_flags & MTHCA_FLAG_NO_LAM))
-			mthca_DISABLE_LAM(mdev, &status);
-	} else
-		mthca_SYS_DIS(mdev, &status);
-}
-
 /* Types of supported HCA */
 enum {
 	TAVOR,			/* MT23108                        */
Index: infiniband/hw/mthca/mthca_provider.c
===================================================================
--- infiniband/hw/mthca/mthca_provider.c	(revision 2822)
+++ infiniband/hw/mthca/mthca_provider.c	(working copy)
@@ -956,14 +956,22 @@ static ssize_t show_hca(struct class_dev
 	}
 }
 
+static ssize_t show_board(struct class_device *cdev, char *buf)
+{
+	struct mthca_dev *dev = container_of(cdev, struct mthca_dev, ib_dev.class_dev);
+	return sprintf(buf, "%.*s\n", MTHCA_BOARD_ID_LEN, dev->board_id);
+}
+
 static CLASS_DEVICE_ATTR(hw_rev,   S_IRUGO, show_rev,    NULL);
 static CLASS_DEVICE_ATTR(fw_ver,   S_IRUGO, show_fw_ver, NULL);
 static CLASS_DEVICE_ATTR(hca_type, S_IRUGO, show_hca,    NULL);
+static CLASS_DEVICE_ATTR(board_id, S_IRUGO, show_board,  NULL);
 
 static struct class_device_attribute *mthca_class_attributes[] = {
 	&class_device_attr_hw_rev,
 	&class_device_attr_fw_ver,
-	&class_device_attr_hca_type
+	&class_device_attr_hca_type,
+	&class_device_attr_board_id
 };
 
 int mthca_register_device(struct mthca_dev *dev)
Index: infiniband/hw/mthca/mthca_cmd.c
===================================================================
--- infiniband/hw/mthca/mthca_cmd.c	(revision 2822)
+++ infiniband/hw/mthca/mthca_cmd.c	(working copy)
@@ -1083,6 +1083,34 @@ out:
 	return err;
 }
 
+static void get_board_id(void *vsd, char *board_id)
+{
+	int i;
+
+#define VSD_OFFSET_SIG1		0x00
+#define VSD_OFFSET_SIG2		0xde
+#define VSD_OFFSET_MLX_BOARD_ID	0xd0
+#define VSD_OFFSET_TS_BOARD_ID	0x20
+
+#define VSD_SIGNATURE_TOPSPIN	0x5ad
+
+	memset(board_id, 0, MTHCA_BOARD_ID_LEN);
+
+	if (be16_to_cpup(vsd + VSD_OFFSET_SIG1) == VSD_SIGNATURE_TOPSPIN &&
+	    be16_to_cpup(vsd + VSD_OFFSET_SIG2) == VSD_SIGNATURE_TOPSPIN) {
+		strlcpy(board_id, vsd + VSD_OFFSET_TS_BOARD_ID, MTHCA_BOARD_ID_LEN);
+	} else {
+		/*
+		 * The board ID is a string but the firmware byte
+		 * swaps each 4-byte word before passing it back to
+		 * us.  Therefore we need to swab it before printing.
+		 */
+		for (i = 0; i < 4; ++i)
+			((u32 *) board_id)[i] =
+				swab32(*(u32 *) (vsd + VSD_OFFSET_MLX_BOARD_ID + i * 4));
+	}
+}
+
 int mthca_QUERY_ADAPTER(struct mthca_dev *dev,
 			struct mthca_adapter *adapter, u8 *status)
 {
@@ -1095,6 +1123,7 @@ int mthca_QUERY_ADAPTER(struct mthca_dev
 #define QUERY_ADAPTER_DEVICE_ID_OFFSET     0x04
 #define QUERY_ADAPTER_REVISION_ID_OFFSET   0x08
 #define QUERY_ADAPTER_INTA_PIN_OFFSET      0x10
+#define QUERY_ADAPTER_VSD_OFFSET           0x20
 
 	mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL);
 	if (IS_ERR(mailbox))
@@ -1112,6 +1141,9 @@ int mthca_QUERY_ADAPTER(struct mthca_dev
 	MTHCA_GET(adapter->revision_id, outbox, QUERY_ADAPTER_REVISION_ID_OFFSET);
 	MTHCA_GET(adapter->inta_pin, outbox,    QUERY_ADAPTER_INTA_PIN_OFFSET);
 
+	get_board_id(outbox + QUERY_ADAPTER_VSD_OFFSET / 4,
+		     adapter->board_id);
+
 out:
 	mthca_free_mailbox(dev, mailbox);
 	return err;
Index: infiniband/hw/mthca/mthca_cmd.h
===================================================================
--- infiniband/hw/mthca/mthca_cmd.h	(revision 2822)
+++ infiniband/hw/mthca/mthca_cmd.h	(working copy)
@@ -184,10 +184,11 @@ struct mthca_dev_lim {
 };
 
 struct mthca_adapter {
-	u32 vendor_id;
-	u32 device_id;
-	u32 revision_id;
-	u8  inta_pin;
+	u32  vendor_id;
+	u32  device_id;
+	u32  revision_id;
+	char board_id[MTHCA_BOARD_ID_LEN];
+	u8   inta_pin;
 };
 
 struct mthca_init_hca_param {


From jlentini at netapp.com  Thu Jul 14 11:04:52 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 14 Jul 2005 14:04:52 -0400 (EDT)
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <20050713151216.GB649@lst.de>
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
	<20050713151216.GB649@lst.de>
Message-ID: <Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>


When the DAT Collaborative solicited requirements for their API, RDMA 
consumers requested this feature. The idea is that a single 
event source is a cleaner, simpler programming model.

On Wed, 13 Jul 2005, Christoph Hellwig wrote:

> On Wed, Jul 13, 2005 at 10:18:41AM -0400, James Lentini wrote:
>> In addition to the streamlined connection management interface, kDAPL
>> also provides a unified event model. This is another feature that the
>> verbs do not provide.
>
> It's probably the biggest mis-feature and would have to go aswell if
> we m�ssage what started as kdapl into the kernel rdma layer.

From jlentini at netapp.com  Thu Jul 14 11:25:35 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 14 Jul 2005 14:25:35 -0400 (EDT)
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <ORSMSX401iGXix4A2Dr0000004b@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401iGXix4A2Dr0000004b@orsmsx401.amr.corp.intel.com>
Message-ID: <Pine.LNX.4.61.0507141405240.25603@jlentini-linux.nane.netapp.com>


The benefit is that all RDMA related events are produced by the same 
source.

Contrasting this with the OpenIB API may useful. In the OpenIB 
API, events are delivered via both CQ work completions and callbacks.

If you look through the DAPL event subsystem, you'll see that it does 
take a bit of code to implement this feature. Duplicating this 
functionality in each consumer wouldn't be efficient. Consumers would 
be better off handling the work completions and callbacks directly.

james

On Wed, 13 Jul 2005, Sean Hefty wrote:

>> In addition to the streamlined connection management interface, kDAPL
>> also provides a unified event model. This is another feature that the
>> verbs do not provide.
>
> Can someone educate me on the benefits of a "unified event model" and why a
> consumer couldn't easily perform the same functionality?


From tduffy at sun.com  Thu Jul 14 11:25:46 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 14 Jul 2005 11:25:46 -0700
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <78d18e2050713123235f668bc@mail.gmail.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
Message-ID: <1121365546.17302.3.camel@duffman>

On Wed, 2005-07-13 at 15:32 -0400, William Jordan wrote:
> I upgraded my iputils (from the Fedora Core 4 source RPM,
> iputils-20020927-22). The new version doesn't work. But, instead of
> arping returning -1 responses, it reports 0 responses, so at least
> ifup works without complaining about the address already being in use.
> However, if the address really is in use, it doesn't catch it.

Hrm, my arping on Fedora Core 4 now does real bad stuff:

Bringing up interface ib0:  *** buffer overflow detected ***: arping terminated
======= Backtrace: =========
/lib64/libc.so.6(__chk_fail+0x2f)[0x2aaaaacc923f]
arping(main+0x48f)[0x5555555567df]
/lib64/libc.so.6(__libc_start_main+0xdc)[0x2aaaaac084cc]
arping[0x5555555559b9]
======= Memory map: ========
3b77c00000-3b77c1a000 r-xp 00000000 08:02 168058                         /lib64/ld-2.3.5.so
3b77d19000-3b77d1a000 r--p 00019000 08:02 168058                         /lib64/ld-2.3.5.so
3b77d1a000-3b77d1b000 rw-p 0001a000 08:02 168058                         /lib64/ld-2.3.5.so
2aaaaaaab000-2aaaaaaac000 rw-p 2aaaaaaab000 00:00 0
2aaaaaad6000-2aaaaaad7000 rw-p 2aaaaaad6000 00:00 0
2aaaaaad7000-2aaaaaae8000 r-xp 00000000 08:02 684                        /lib64/libresolv-2.3.5.so
2aaaaaae8000-2aaaaabe8000 ---p 00011000 08:02 684                        /lib64/libresolv-2.3.5.so
2aaaaabe8000-2aaaaabe9000 r--p 00011000 08:02 684                        /lib64/libresolv-2.3.5.so
2aaaaabe9000-2aaaaabea000 rw-p 00012000 08:02 684                        /lib64/libresolv-2.3.5.so
2aaaaabea000-2aaaaabec000 rw-p 2aaaaabea000 00:00 0
2aaaaabec000-2aaaaad1a000 r-xp 00000000 08:02 168062                     /lib64/libc-2.3.5.so
2aaaaad1a000-2aaaaae19000 ---p 0012e000 08:02 168062                     /lib64/libc-2.3.5.so
2aaaaae19000-2aaaaae1d000 r--p 0012d000 08:02 168062                     /lib64/libc-2.3.5.so
2aaaaae1d000-2aaaaae1f000 rw-p 00131000 08:02 168062                     /lib64/libc-2.3.5.so
2aaaaae1f000-2aaaaae25000 rw-p 2aaaaae1f000 00:00 0
2aaaaae25000-2aaaaae32000 r-xp 00000000 08:02 168371                     /lib64/libgcc_s-4.0.0-20050520.so.1
2aaaaae32000-2aaaaaf31000 ---p 0000d000 08:02 168371                     /lib64/libgcc_s-4.0.0-20050520.so.1
2aaaaaf31000-2aaaaaf32000 rw-p 0000c000 08:02 168371                     /lib64/libgcc_s-4.0.0-20050520.so.1
555555554000-555555558000 r-xp 00000000 08:02 16803329                   /sbin/arping
555555657000-555555658000 rw-p 00003000 08:02 16803329                   /sbin/arping
555555658000-555555679000 rw-p 555555658000 00:00 0                      [heap]
7fffffbb6000-7fffffbcc000 rw-p 7fffffbb6000 00:00 0                      [stack]ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/047d312d/attachment.sig>

From tduffy at sun.com  Thu Jul 14 11:34:06 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 14 Jul 2005 11:34:06 -0700
Subject: [openib-general] [PATCH] SDP: use linux/list.h for binds
Message-ID: <1121366046.17302.9.camel@duffman>

This patch replaces the null terminated linked list for connections
bound to a port and instead uses the list.h library from core Linux.

I have tested this on a few machines doing several ttcp's at once, but
please inspect to verify correctness.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 2861)
+++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -392,8 +392,8 @@ int sdp_inet_port_get(struct sdp_sock *c
 	 * simple linked list of sockets ordered on local port number.
 	 */
 	if (port > 0) {
-		for (look = dev_root_s.bind_list, port_ok = 1;
-		     look; look = look->bind_next) {
+		port_ok = 1;
+		list_for_each_entry(look, &dev_root_s.bind_list, bind_next) {
 			srch = sk_sdp(look);
 			/*
 			 * 1) same port
@@ -451,16 +451,19 @@ int sdp_inet_port_get(struct sdp_sock *c
 
 		for (counter = (top_port - low_port) + 1; counter > 0;
 		     counter--) {
+			int found = 0;
 			rover++;
 			if (rover < low_port || rover > top_port)
 				rover = low_port;
 
-			for (look = dev_root_s.bind_list;
-			     look && look->src_port != port;
-			     look = look->bind_next)
-				do {} while(0); /* pass */
+			list_for_each_entry(look, &dev_root_s.bind_list,
+					    bind_next)
+				if (look->src_port == port) {
+					found = 1;
+					break;
+				}
 
-			if (!look) {
+			if (!found) {
 				port = rover;
 				break;
 			}
@@ -474,14 +477,9 @@ int sdp_inet_port_get(struct sdp_sock *c
 
 	conn->src_port = port;
 	/*
-	 * insert into listening list.
+	 * insert into bind list.
 	 */
-	conn->bind_next = dev_root_s.bind_list;
-	dev_root_s.bind_list = conn;
-	conn->bind_p_next = &dev_root_s.bind_list;
-
-	if (conn->bind_next)
-		conn->bind_next->bind_p_next = &conn->bind_next;
+	list_add(&conn->bind_next, &dev_root_s.bind_list);
 
 	result = 0;
 done:
@@ -496,25 +494,14 @@ int sdp_inet_port_put(struct sdp_sock *c
 {
 	unsigned long flags;
 
-	if (!conn->bind_p_next)
+	if (list_empty(&dev_root_s.bind_list))
 		return -EADDRNOTAVAIL;
-	/*
-	 * lock table
-	 */
-	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
-	/*
-	 * remove from bind list.
-	 */
-	if (conn->bind_next)
-		conn->bind_next->bind_p_next = conn->bind_p_next;
-
-	*(conn->bind_p_next) = conn->bind_next;
 
-	conn->bind_p_next = NULL;
-	conn->bind_next = NULL;
+	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
+	list_del(&conn->bind_next);
 	conn->src_port = 0;
-
 	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
+
 	return 0;
 }
 
@@ -530,18 +517,11 @@ void sdp_inet_port_inherit(struct sdp_so
 	 */
 	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
 
-	BUG_ON(child->bind_p_next);
 	BUG_ON(child->src_port != parent->src_port);
 	/*
-	 * insert into listening list.
+	 * insert into bind list.
 	 */
-	child->bind_next = parent->bind_next;
-	parent->bind_next = child;
-	child->bind_p_next = &parent->bind_next;
-
-	if (child->bind_next)
-		child->bind_next->bind_p_next = &child->bind_next;
-
+	list_add(&child->bind_next, &dev_root_s.bind_list);
 	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
 }
 
@@ -1917,7 +1897,7 @@ int sdp_conn_table_init(int proto_family
 	 * list
 	 */
 	INIT_LIST_HEAD(&dev_root_s.listen_list);
-	dev_root_s.bind_list = NULL;
+	INIT_LIST_HEAD(&dev_root_s.bind_list);
 
 	spin_lock_init(&dev_root_s.sock_lock);
 	spin_lock_init(&dev_root_s.bind_lock);
Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h
===================================================================
--- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h	(revision 2861)
+++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h	(working copy)
@@ -339,9 +339,8 @@ struct sdp_sock {
 	 * table managment
 	 */
 	struct list_head lstn_next;
+	struct list_head bind_next;
 
-	struct sdp_sock *bind_next;    /* next conn in the chain */
-	struct sdp_sock **bind_p_next; /* previous next conn in the chain */
 	/*
 	 * listen/accept managment
 	 */
Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h
===================================================================
--- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h	(revision 2861)
+++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h	(working copy)
@@ -188,7 +188,7 @@ struct sdev_root {
 	 * connection managment
 	 */
 	struct list_head listen_list;
-	struct sdp_sock *bind_list;	/* connections bound to a port. */
+	struct list_head bind_list;	/* connections bound to a port. */
 	/*
 	 * list locks
 	 */



From jlentini at netapp.com  Thu Jul 14 11:40:26 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 14 Jul 2005 14:40:26 -0400 (EDT)
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <1121279132.5366.14.camel@duffman>
References: <1121203262.14638.16.camel@duffman> <20050712215259.GB19448@lst.de>
	<Pine.LNX.4.61.0507131004420.25603@jlentini-linux.nane.netapp.com> 
	<20050713141948.GA31981@lst.de> <1121279132.5366.14.camel@duffman>
Message-ID: <Pine.LNX.4.61.0507141428550.25603@jlentini-linux.nane.netapp.com>


The ability to transparently support multiple transports is key. With 
the DAT registry, consumers can use a single API to interact with IB, 
iWARP, or some other RDMA provider.

This needs to be retained.

On Wed, 13 Jul 2005, Tom Duffy wrote:

> On Wed, 2005-07-13 at 16:19 +0200, Christoph Hellwig wrote:
>> After re-evaluting things: none, just don't merged it.
>
> into gen2/trunk?
>
>> We want an RDMA layer in the kernel, and it looks like Catalin's
>> extension of the IB layer to cover iWarp aswell is the better choice,
>> in combination of taking some utility routines from kdapl and adjust
>> them to work directly ontop of that.
>
> I agree Caitlin's proposal looks good.  I appreciate concrete examples
> of how to move forward.  Let's then create a branch of trunk to start
> the merge work.
>
> What routines from kDAPL are you happy with?  What is worth keeping?
>
> -tduffy
>


From mshefty at ichips.intel.com  Thu Jul 14 11:40:57 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 14 Jul 2005 11:40:57 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507141405240.25603@jlentini-linux.nane.netapp.com>
References: <ORSMSX401iGXix4A2Dr0000004b@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507141405240.25603@jlentini-linux.nane.netapp.com>
Message-ID: <42D6B1B9.8030503@ichips.intel.com>

James Lentini wrote:
> 
> The benefit is that all RDMA related events are produced by the same 
> source.

I understand what it does.  But why is that a benefit?  It restricts clients 
from setting event handling priorities.

> Consumers would be 
> better off handling the work completions and callbacks directly.

Exactly.  What makes this handling so difficult that it needs to be abstracted?

- Sean


From caitlin.bestler at gmail.com  Thu Jul 14 12:16:35 2005
From: caitlin.bestler at gmail.com (Caitlin Bestler)
Date: Thu, 14 Jul 2005 12:16:35 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <42D6B1B9.8030503@ichips.intel.com>
References: <ORSMSX401iGXix4A2Dr0000004b@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507141405240.25603@jlentini-linux.nane.netapp.com>
	<42D6B1B9.8030503@ichips.intel.com>
Message-ID: <469958e00507141216342204c3@mail.gmail.com>

On 7/14/05, Sean Hefty <mshefty at ichips.intel.com> wrote:
> James Lentini wrote:
> >
> > The benefit is that all RDMA related events are produced by the same
> > source.
> 
> I understand what it does.  But why is that a benefit?  It restricts clients
> from setting event handling priorities.
> 
> > Consumers would be
> > better off handling the work completions and callbacks directly.
> 
> Exactly.  What makes this handling so difficult that it needs to be abstracted?
> 

There are two important features of the EVD, even in KDAPL callback mode,
that were considered crucial by all of the early participants. These were based
on very negative experiences working with VIPL that lacked these features:

1) unifornmity. The application does not have to deal with one rule for the CQ,
    one for Connection Establishment, one for asynch affiliated events, one for
   asynch unaffilliated events, and one for IPC.

2) serialization: The application does not have to worry that an asynch event
    will interrupt the processing of a work completion, or vise versa.
Or whether
   affilliated, unaffilliated and  connection management events come from one,
   two or three sources.

I'd to re-iterate that this was based on feedback of people developing
applications to run in kernel mode based on actual problems they had
experienced with prior APIs such as VIPL -- not abstract theory.


From sean.hefty at intel.com  Thu Jul 14 12:25:54 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Thu, 14 Jul 2005 12:25:54 -0700
Subject: [openib-general] matching an RMPP NACK to send or receive?
Message-ID: <ORSMSX401eBBsQwY6YH00000051@orsmsx401.amr.corp.intel.com>

I'm probably just missing something simple here, but does anyone know of a way
to determine if a received RMPP ABORT or STOP message should match to a MAD
being sent or received?  That is, should an active send be aborted, or should
processing halt on a receive?  It doesn't seem that the TID or response
bit/method is sufficient.

- Sean



From hch at lst.de  Thu Jul 14 12:46:03 2005
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 14 Jul 2005 21:46:03 +0200
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
	<20050713151216.GB649@lst.de>
	<Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>
Message-ID: <20050714194603.GA26814@lst.de>

On Thu, Jul 14, 2005 at 02:04:52PM -0400, James Lentini wrote:
> 
> When the DAT Collaborative solicited requirements for their API, RDMA 
> consumers requested this feature. The idea is that a single 
> event source is a cleaner, simpler programming model.

The DAT Collaborative is responsible for all that mes� so I don't take
them serious at all.  If you feel so strongly about this feature
implement it ontop of the new rdma stack and we can see whether it makes
the clients simpler or more complex.



From jlentini at netapp.com  Thu Jul 14 13:06:27 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 14 Jul 2005 16:06:27 -0400 (EDT)
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <42D59654.7030104@ichips.intel.com>
References: <8508251A6FC08A489844A94261D3693A057DFE@fiona.siliquent.com>
	<42D59654.7030104@ichips.intel.com>
Message-ID: <Pine.LNX.4.61.0507141555570.25603@jlentini-linux.nane.netapp.com>



On Wed, 13 Jul 2005, Sean Hefty wrote:

> Caitlin Bestler wrote:
>> So would you favor changing the signature to match the largest
>> size, with  comment to the effect that you *could* use a cast
>> and a smaller object but only if you are sure.
>
> I haven't decided on what might be the best approach for this call yet, I'd 
> just like to ensure that the API provides the best performance.  If QPs and 
> CQs are both transport specific, I'm wondering now if verbs is the best place 
> to abstract RDMA functionality, or if it will result in losing some of the 
> details provided by the underlying architecture.
>
> There doesn't seem to be much to iwarp_verbs.h.  Assuming that the API is 
> complete, iWarp seems to use a subset of the structures defined by IB.  Why 
> couldn't the current data structures just be renamed from ib_blah to 
> rdma_blah, with notes that some fields apply only to IB?
>
> Some of the data structures already have fields that only apply for specific 
> completion types, work requests, QPs, etc.  So, marking them for IB only 
> doesn't seem like that much of a stretch, and transport neutral code would 
> need to allocate the space anyway.  This way we would only need to change a 
> few enums and move IB specific functions to a new file.

A "generic" consumer would never need to fill in such fields, 
correct?

>> I can see the danger of someone just using 'rdma_wc', so flipping
>> the naming convention would definitely make sense.
>> 
>> As to the size. Is there a size smaller than 256 that crosses
>> over into a significantly safer stack allocation but is still
>> *very* safe in terms of 'sure to be big enough'? Keep in mind
>> that 'struct sockaddr' has problems precisely because 'biggest
>> possible address' turned out to be too small.
>
> Work completions are used in speed path operations, so should be optimized 
> for caching, retrieving multiple completions, etc., versus optimizing it for 
> transport neutral code.  struct rdma_wc_storage isn't used anywhere in the 
> API, so why not make that a union of struct ib_wc and struct iwarp_wc?  (Of 
> course this leads us back to unions if swapping the naming convention...)
>
> - Sean
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>


From mshefty at ichips.intel.com  Thu Jul 14 13:16:09 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 14 Jul 2005 13:16:09 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
In-Reply-To: <Pine.LNX.4.61.0507141555570.25603@jlentini-linux.nane.netapp.com>
References: <8508251A6FC08A489844A94261D3693A057DFE@fiona.siliquent.com>
	<42D59654.7030104@ichips.intel.com>
	<Pine.LNX.4.61.0507141555570.25603@jlentini-linux.nane.netapp.com>
Message-ID: <42D6C809.4070708@ichips.intel.com>

James Lentini wrote:
>> There doesn't seem to be much to iwarp_verbs.h.  Assuming that the API 
>> is complete, iWarp seems to use a subset of the structures defined by 
>> IB.  Why couldn't the current data structures just be renamed from 
>> ib_blah to rdma_blah, with notes that some fields apply only to IB?
>>
>> Some of the data structures already have fields that only apply for 
>> specific completion types, work requests, QPs, etc.  So, marking them 
>> for IB only doesn't seem like that much of a stretch, and transport 
>> neutral code would need to allocate the space anyway.  This way we 
>> would only need to change a few enums and move IB specific functions 
>> to a new file.
> 
> A "generic" consumer would never need to fill in such fields, correct?

That would be correct.  Fields that did not apply would simply be ignored. 
For instance, the current verbs API does this for work requests and 
completions; the structures contain fields that apply only to MADs.

If iwarp_verbs.h is complete, then it only extends 4 enums and encapsulates 
3 data structures without extending them at all.  This makes me suspect that 
iwarp could just plug into the existing verbs API without widespread changes.

- Sean



From eitan at mellanox.co.il  Thu Jul 14 13:28:00 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 23:28:00 +0300
Subject: [openib-general] matching an RMPP NACK to send or receive?
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304AF@mtlex01.yok.mtl.com>

Hi Sean,

I do not understand your question.
If you have an RMPP transaction active - it carries the TID with it. 
So when the ABORT or STOP is received - that transaction has to end.
Further incomming MADs of that transaction should be dropped.

I must miss something in your question.

EZ

-----Original Message-----
From: Sean Hefty [mailto:sean.hefty at intel.com]
Sent: Thursday, July 14, 2005 10:26 PM
To: openib-general at openib.org
Subject: [openib-general] matching an RMPP NACK to send or receive?


I'm probably just missing something simple here, but does anyone know of a
way
to determine if a received RMPP ABORT or STOP message should match to a MAD
being sent or received?  That is, should an active send be aborted, or
should
processing halt on a receive?  It doesn't seem that the TID or response
bit/method is sufficient.

- Sean

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/6e8ade8e/attachment.html>

From caitlinb at siliquent.com  Thu Jul 14 13:30:01 2005
From: caitlinb at siliquent.com (Caitlin Bestler)
Date: Thu, 14 Jul 2005 13:30:01 -0700
Subject: [openib-general] makiing ibverb.h transport neutral -- 2nd draft
Message-ID: <8508251A6FC08A489844A94261D3693A057E1A@fiona.siliquent.com>

 


> 
> If iwarp_verbs.h is complete, then it only extends 4 enums 
> and encapsulates
> 3 data structures without extending them at all.  This makes 
> me suspect that iwarp could just plug into the existing verbs 
> API without widespread changes.
> 
> - Sean
> 
You're looking at the wrong file.

What's important to a developer is not how seldom they need
to deal with iWARP specifics, but how much IB specifics they
can ignore once the data structures cleanly separate the
transport neutral problem-related fields from the IB-specific
solution-oriented fields.

I do not think it is reasonable for developers to have to wade
through long structures and read comments to determine field
by field which fields they can ignore. Rather transport neutral
fields should be isolated either by a union or by a sub-type.

That way, someone who wants to write transport neutral code
is prevented from accidentally referencing transport specific
fields.


From mshefty at ichips.intel.com  Thu Jul 14 13:37:34 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 14 Jul 2005 13:37:34 -0700
Subject: [openib-general] matching an RMPP NACK to send or receive?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C304AF@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C304AF@mtlex01.yok.mtl.com>
Message-ID: <42D6CD0E.1060508@ichips.intel.com>

Eitan Zahavi wrote:
> I do not understand your question.
> If you have an RMPP transaction active - it carries the TID with it. 
> So when the ABORT or STOP is received - that transaction has to end.
> Further incomming MADs of that transaction should be dropped.
> 
> I must miss something in your question.

To clarify:

Suppose two systems initiate RMPP transfers to each other.  Each system 
would be sending and reassembling RMPP messages.  Both could select the same 
TID.

Upon receiving an ACK, a system knows that it matches with the outbound 
send.  A NACK however could match with either the outbound send or the 
inbound receive.  System A needs to know if system B is attempting to stop 
transaction from A to B or B to A.

Does that make more sense?

- Sean


From eitan at mellanox.co.il  Thu Jul 14 13:46:01 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 23:46:01 +0300
Subject: [openib-general] Introducing IB MAD TCL API
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304B1@mtlex01.yok.mtl.com>

Hi Hal,

Hal wrote:
> Does this build ? 

> I get the following when I run autogen:
> src/Makefile.am:46: AM_CFLAGS was already defined in condition TRUE,
> which implies condition OSM_VENDOR_MTL_TRUE

EZ: It built for me. On many types of machines/systems.
I guess the issue is really with the fact it did not build on a clean
environment.
Can you please tar and gzip the source tree you are using such that I can
double check.
Also please cat /etc/redhat-release and gcc --version


> Are these all ported and working in OpenIB ?
EZ: They should - try only rely on osm vendor - and it should not have
changed API. Did it?

> Do the files have the correct licenses ?
EZ: Yes. Did you see a wrong one?

> The trunk is for eligible for release. IMO, it seems a little early yet
> for this. I would like to get them all built for OpenIB and play wit
> them a little. 
EZ: OK, but in order to integrate eventually to the trunk I will need to put
them first on a branch - but probably in the right hierarchy - not a side
branch - on a totally different place in the tree.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/d66358eb/attachment.html>

From caitlin.bestler at gmail.com  Thu Jul 14 13:46:37 2005
From: caitlin.bestler at gmail.com (Caitlin Bestler)
Date: Thu, 14 Jul 2005 13:46:37 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <20050714194603.GA26814@lst.de>
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
	<20050713151216.GB649@lst.de>
	<Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>
	<20050714194603.GA26814@lst.de>
Message-ID: <469958e0050714134618bbb83e@mail.gmail.com>

I believe that the EVD is an extremely valuable feature, but
I do agree that is one that should be optional to use.

That is why I believe there needs to be a transport neutral
API that is lower than kDAPL. However I believe that such
an API should be cleanly separated from kDAPL so that
it does not cause confusion for developers.

There are two basic approaches to this:

a) define a subset of kDAPL that provides only transport
    neutrality and none of the "extra" features such as EVDs.
    Then provide an open source implementation of the missing
    features written on top of the core API.

b) define a verb layer interface that is usable by kDAPL, other
   middleware and advanced applications. Implement kDAPL
   and other middleware using it.


Both are starting from an existing interface. A is working down
from kDAPL. B is working up from gen2. Either works. My 
current estimation is that B is easier. But I don't see any
point in doing both.


On 7/14/05, Christoph Hellwig <hch at lst.de> wrote:
> On Thu, Jul 14, 2005 at 02:04:52PM -0400, James Lentini wrote:
> >
> > When the DAT Collaborative solicited requirements for their API, RDMA
> > consumers requested this feature. The idea is that a single
> > event source is a cleaner, simpler programming model.
> 
> The DAT Collaborative is responsible for all that mesß so I don't take
> them serious at all.  If you feel so strongly about this feature
> implement it ontop of the new rdma stack and we can see whether it makes
> the clients simpler or more complex.
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>


From eitan at mellanox.co.il  Thu Jul 14 13:52:47 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Thu, 14 Jul 2005 23:52:47 +0300
Subject: [openib-general] matching an RMPP NACK to send or receive?
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>

Hi Sean,


> Suppose two systems initiate RMPP transfers to each other.  Each system 
> would be sending and reassembling RMPP messages.  Both could select the
same 
> TID.

EZ: TID should consist of a client selected part and process ID. For kernels
I would select some part of the GUID too.
We need to minimize cases of same TID use. But in such case - abort both of
them.

> Does that make more sense?
EZ: Yes - but once the TID is the same I do not think there is an answer.
Another thought might be that if an active TID exists in the system a new
RMPP transaction with same TID should be denied...

- Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/e4d148e3/attachment.html>

From mshefty at ichips.intel.com  Thu Jul 14 14:01:58 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 14 Jul 2005 14:01:58 -0700
Subject: [openib-general] matching an RMPP NACK to send or receive?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>
Message-ID: <42D6D2C6.8040903@ichips.intel.com>

Eitan Zahavi wrote:
> EZ: TID should consist of a client selected part and process ID. For kernels
> I would select some part of the GUID too.
> We need to minimize cases of same TID use. But in such case - abort both of
> them.

I don't think that the receive handling code should make any assumptions 
about the format of the TID.  Aborting both transactions seems a little 
harsh, but not sure if there's another way.  For now, the code will only 
abort a send, which should be the more common case.  I will need to add 
aborting receives later and assume that actual TID collisions would be 
extremely rare.

The RMPP status does help indicate whether a send or receive is being 
aborted, but not in all cases.

- Sean


From eitan at mellanox.co.il  Thu Jul 14 14:08:52 2005
From: eitan at mellanox.co.il (Eitan Zahavi)
Date: Fri, 15 Jul 2005 00:08:52 +0300
Subject: [openib-general] matching an RMPP NACK to send or receive?
Message-ID: <506C3D7B14CDD411A52C00025558DED607C304B6@mtlex01.yok.mtl.com>

I was referring to the way TID is built only to say that TID made by the
kernel should try and use some way to identify kernel A from kernel B.
If a TID is built out of PID and sequential number then using a TID with PID
prefix of zero for Kernel generated transaction would cause more TID
collisions. 

 -----Original Message-----
From: 	Sean Hefty [mailto:mshefty at ichips.intel.com] 
Sent:	Friday, July 15, 2005 12:02 AM
To:	Eitan Zahavi
Cc:	'Sean Hefty'; openib-general at openib.org
Subject:	Re: [openib-general] matching an RMPP NACK to send or
receive?

Eitan Zahavi wrote:
> EZ: TID should consist of a client selected part and process ID. For
kernels
> I would select some part of the GUID too.
> We need to minimize cases of same TID use. But in such case - abort both
of
> them.

I don't think that the receive handling code should make any assumptions 
about the format of the TID.  Aborting both transactions seems a little 
harsh, but not sure if there's another way.  For now, the code will only 
abort a send, which should be the more common case.  I will need to add 
aborting receives later and assume that actual TID collisions would be 
extremely rare.

The RMPP status does help indicate whether a send or receive is being 
aborted, but not in all cases.

- Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050715/a89039ce/attachment.html>

From tduffy at sun.com  Thu Jul 14 15:30:12 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 14 Jul 2005 15:30:12 -0700
Subject: [openib-general] [PATCH] SDP: use linux/list.h for hca port list
Message-ID: <1121380212.17302.30.camel@duffman>

This patch changes the NULL terminated port list in the sdp hca object
to a Linux list_head.  It also removes the sdev_hca *next member from
the sdev_hca struct as this is never used, AFAICT.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 2861)
+++ drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -994,10 +974,10 @@ int sdp_conn_alloc_ib(struct sdp_sock *c
 {
 	struct ib_qp_init_attr *init_attr;
 	struct ib_qp_attr     *qp_attr;
-	struct sdev_hca_port  *port = NULL;
-	struct sdev_hca       *hca = NULL;
+	struct sdev_hca_port  *port;
+	struct sdev_hca       *hca;
 	int                    attr_mask = 0;
-	int                    result;
+	int                    result = 0;
 
 	/*
 	 * look up correct HCA and port
@@ -1006,11 +986,13 @@ int sdp_conn_alloc_ib(struct sdp_sock *c
 	if (!hca)
 		return -ERANGE;
 
-	for (port = hca->port_list; port; port = port->next)
-		if (hw_port == port->index)
+	list_for_each_entry(port, &hca->port_list, list)
+		if (hw_port == port->index) {
+			result = 1;
 			break;
+		}
 
-	if (!port)
+	if (!result)
 		return -ERANGE;
 	/*
 	 * allocate creation parameters
@@ -1740,14 +1722,14 @@ int sdp_proc_dump_device(char *buffer, i
 static void sdp_device_init_one(struct ib_device *device)
 {
 	struct ib_fmr_pool_param fmr_param_s;
-	struct sdev_hca_port *port;
+	struct sdev_hca_port *port, *tmp;
 	struct sdev_hca *hca;
 	int result;
 	int port_count;
 	/*
 	 * allocate per-HCA structure
 	 */
-	hca = kmalloc(sizeof(struct sdev_hca), GFP_KERNEL);
+	hca = kmalloc(sizeof *hca, GFP_KERNEL);
 	if (!hca) {
 		sdp_warn("Error allocating HCA <%s> memory.", device->name);
 		return;
@@ -1755,12 +1737,10 @@ static void sdp_device_init_one(struct i
 	/*
 	 * init and insert into list.
 	 */
-	memset(hca, 0, sizeof(struct sdev_hca));
+	memset(hca, 0, sizeof *hca);
 
-	hca->fmr_pool = NULL;
-	hca->mem_h    = NULL;
-	hca->pd       = NULL;
-	hca->ca       = device;
+	hca->ca = device;
+	INIT_LIST_HEAD(&hca->port_list);
 	/*
 	 * protection domain
 	 */
@@ -1798,17 +1778,15 @@ static void sdp_device_init_one(struct i
 	 * create SDP memory pool
 	 */
 	hca->fmr_pool = ib_create_fmr_pool(hca->pd, &fmr_param_s);
-	if (IS_ERR(hca->fmr_pool)) {
-		sdp_warn("Warning, could not creating HCA <%s> FMR pool <%ld>",
+	if (IS_ERR(hca->fmr_pool))
+		sdp_warn("Warning, could not create HCA <%s> FMR pool <%ld>",
 			 device->name, PTR_ERR(hca->fmr_pool));
-	}
+
 	/*
 	 * port allocation
 	 */
-	for (port_count = 0; 
-	     port_count < device->phys_port_cnt;
-	     port_count++) {
-		port = kmalloc(sizeof(struct sdev_hca_port), GFP_KERNEL);
+	for (port_count = 0; port_count < device->phys_port_cnt; port_count++) {
+		port = kmalloc(sizeof *port, GFP_KERNEL);
 		if (!port) {
 			sdp_warn("Error allocating HCA <%s> port <%d:%d>",
 				 device->name, port_count,
@@ -1817,11 +1795,10 @@ static void sdp_device_init_one(struct i
 			goto error;
 		}
 
-		memset(port, 0, sizeof(struct sdev_hca_port));
+		memset(port, 0, sizeof *port);
 
-		port->index    = port_count + 1;
-		port->next     = hca->port_list;
-		hca->port_list = port;
+		port->index = port_count + 1;
+		list_add(&port->list, &hca->port_list);
 
 		result = ib_query_gid(hca->ca, 
 				      port->index, 
@@ -1840,11 +1817,8 @@ static void sdp_device_init_one(struct i
 	return;
 
 error:
-	while (hca->port_list) {
-		port = hca->port_list;
-		hca->port_list = port->next;
-		port->next = NULL;
-
+	list_for_each_entry_safe(port, tmp, &hca->port_list, list) {
+		list_del(&port->list);
 		kfree(port);
 	}
 
@@ -1865,7 +1839,7 @@ error:
  */
 static void sdp_device_remove_one(struct ib_device *device)
 {
-	struct sdev_hca_port *port;
+	struct sdev_hca_port *port, *tmp;
 	struct sdev_hca *hca;
 
 	hca = ib_get_client_data(device, &sdp_client);
@@ -1875,11 +1849,8 @@ static void sdp_device_remove_one(struct
 		return;
 	}
 	
-	while (hca->port_list) {
-		port = hca->port_list;
-		hca->port_list = port->next;
-		port->next = NULL;
-
+	list_for_each_entry_safe(port, tmp, &hca->port_list, list) {
+		list_del(&port->list);
 		kfree(port);
 	}
 
Index: drivers/infiniband/ulp/sdp/sdp_dev.h
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_dev.h	(revision 2861)
+++ drivers/infiniband/ulp/sdp/sdp_dev.h	(working copy)
@@ -150,7 +150,7 @@
 struct sdev_hca_port {
 	u8                    index; /* port ID */
 	union ib_gid          gid;   /* port GID */
-	struct sdev_hca_port *next;  /* next port in the list */
+	struct list_head      list;
 };
 
 struct sdev_hca {
@@ -160,8 +160,7 @@ struct sdev_hca {
 	u32                   l_key;     /* local key */
 	u32                   r_key;     /* remote key */
 	struct ib_fmr_pool   *fmr_pool;  /* fast memory for Zcopy */
-	struct sdev_hca_port *port_list; /* ports on this HCA */
-	struct sdev_hca      *next;      /* next HCA in the list */
+	struct list_head      port_list; /* ports on this HCA */
 };
 
 struct sdev_root {



From tduffy at sun.com  Thu Jul 14 15:59:45 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 14 Jul 2005 15:59:45 -0700
Subject: [openib-general] [ANNOUNCE] Initial checkin of SRP initiator
In-Reply-To: <52r7e8muj1.fsf@topspin.com>
References: <52r7e8muj1.fsf@topspin.com>
Message-ID: <1121381985.17302.35.camel@duffman>

On Fri, 2005-07-08 at 22:13 -0700, Roland Dreier wrote:
> I just checked in a first version of a SCSI RDMA Protocol (SRP)
> initiator under infiniband/ulp/srp.

svn propdel svn:executable ib_srp.[ch]

please.

Also, I noticed a couple "if (0)" in ib_srp.c.  Do you plan to make a
debug level value or just delete these?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050714/4423c567/attachment.sig>

From ardavis at ichips.intel.com  Thu Jul 14 16:23:33 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Thu, 14 Jul 2005 16:23:33 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121351537.4423.1162.camel@hal.voltaire.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
	<1121351537.4423.1162.camel@hal.voltaire.com>
Message-ID: <42D6F3F5.5020806@ichips.intel.com>

Hal Rosenstock wrote:

>Hi Arlin,
>
>On Wed, 2005-07-13 at 12:57, Arlin Davis wrote:
>  
>
>>>One more question: Are you using IB_AT_ROUTE_FORCE_RESOLVE or not in the
>>>flags to the ib_at_route_by_ip call ? (What is the flag setting ?)
>>>      
>>>
>>no, tos and flags are both set to 0.
>>    
>>
>
>OK. I looked into this and it appears to me that the route is not copied
>back to userspace until the callback is made.
>
>-- Hal
>  
>
ok, I will always get the callback for now and ignore a return of 1. 
Is this the expected behavior because it does not match the documentation.

-arlin

>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>  
>



From arlin.r.davis at intel.com  Thu Jul 14 16:29:40 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Thu, 14 Jul 2005 16:29:40 -0700
Subject: [openib-general] [PATCH] uAT ib_at_callback_get_timed
Message-ID: <ORSMSX401FRaqbC8wSA00000054@orsmsx401.amr.corp.intel.com>


Hal,

I ran into a problem with ib_at_callback_get_timed(). Here is a fix.

Thanks

-arlin


Signed-off-by: Arlin Davis <ardavis at ichips.intel.com>

Index: at.c
===================================================================
--- at.c        (revision 2844)
+++ at.c        (working copy)
@@ -314,5 +314,8 @@
        if (!result)
                return -ETIMEDOUT;

+       if (result < 0)
+               return result;
+
        return ib_at_callback_get();
 }



From rolandd at cisco.com  Thu Jul 14 18:10:41 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 14 Jul 2005 18:10:41 -0700
Subject: [openib-general] [ANNOUNCE] Initial checkin of SRP initiator
In-Reply-To: <1121381985.17302.35.camel@duffman> (Tom Duffy's message of
	"Thu, 14 Jul 2005 15:59:45 -0700")
References: <52r7e8muj1.fsf@topspin.com> <1121381985.17302.35.camel@duffman>
Message-ID: <527jfsc1ri.fsf@topspin.com>

    Tom> svn propdel svn:executable ib_srp.[ch]
    Tom> please.

Weird:

    svn proplist https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/srp/ib_srp.{c,h}
    Properties on 'https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/srp/ib_srp.c':
      svn:keywords
    Properties on 'https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/srp/ib_srp.h':
      svn:keywords

    Tom> Also, I noticed a couple "if (0)" in ib_srp.c.  Do you plan
    Tom> to make a debug level value or just delete these?

Probably just delete them soon.

 - R.


From mamidala at cse.ohio-state.edu  Thu Jul 14 22:47:46 2005
From: mamidala at cse.ohio-state.edu (amith rajith mamidala)
Date: Fri, 15 Jul 2005 01:47:46 -0400 (EDT)
Subject: [openib-general] Error in opening IB cm device
In-Reply-To: <527jfsc1ri.fsf@topspin.com>
Message-ID: <Pine.GSO.4.40.0507150138500.3568-100000@kappa.cse.ohio-state.edu>


Hi,

We have installed the latest openib stack (Rev:2861) on the X86_64
platform. While running the pingpong tests I am encountering the
following error:

libucm: Error <-1:2> couldn't open IB cm device </dev/infiniband/ucm>
[1] Abort: Error getting HCA context

I am not sure why this happening ?

Thanks,
Amith



From mlleinin at hpcn.ca.sandia.gov  Thu Jul 14 23:36:17 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Thu, 14 Jul 2005 23:36:17 -0700
Subject: [openib-general] Error in opening IB cm device
In-Reply-To: <Pine.GSO.4.40.0507150138500.3568-100000@kappa.cse.ohio-state.edu>
References: <Pine.GSO.4.40.0507150138500.3568-100000@kappa.cse.ohio-state.edu>
Message-ID: <1121409378.20046.260.camel@localhost>

On Fri, 2005-07-15 at 01:47 -0400, amith rajith mamidala wrote:
> Hi,
> 
> We have installed the latest openib stack (Rev:2861) on the X86_64
> platform. While running the pingpong tests I am encountering the
> following error:
> 
> libucm: Error <-1:2> couldn't open IB cm device </dev/infiniband/ucm>
> [1] Abort: Error getting HCA context
> 
> I am not sure why this happening ?
> 

What does 'ls -l /dev/infiniband' say?  You should have:

[root at n64-81 ~]# ls -l /dev/infiniband/
total 0
crw-rw-rw-    1 root     root     231, 255 Jul 13 10:44 ucm
crw-rw-rw-    1 root     root     231, 192 Jul 13 10:45 uverbs0
crw-rw-rw-    1 root     root     231, 193 Jul 13 10:45 uverbs1

Try,

modprobe ib_uverbs
mknod -m 0666 /dev/infiniband/uverbs0 c 231 192
mknod -m 0666 /dev/infiniband/uverbs1 c 231 193


modprobe ib_ucm
mknod /dev/infiniband/ucm c 231 255

  - Matt




From halr at voltaire.com  Fri Jul 15 04:07:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 07:07:00 -0400
Subject: [openib-general] Re: [PATCH] uAT ib_at_callback_get_timed
In-Reply-To: <ORSMSX401FRaqbC8wSA00000054@orsmsx401.amr.corp.intel.com>
References: <ORSMSX401FRaqbC8wSA00000054@orsmsx401.amr.corp.intel.com>
Message-ID: <1121425620.4423.2139.camel@hal.voltaire.com>

On Thu, 2005-07-14 at 19:29, Arlin Davis wrote:
> I ran into a problem with ib_at_callback_get_timed(). Here is a fix.

Thanks. Applied.

-- Hal



From halr at voltaire.com  Fri Jul 15 04:14:55 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 07:14:55 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42D6F3F5.5020806@ichips.intel.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
	<1121351537.4423.1162.camel@hal.voltaire.com>
	<42D6F3F5.5020806@ichips.intel.com>
Message-ID: <1121425731.4423.2145.camel@hal.voltaire.com>

Hi Arlin,

On Thu, 2005-07-14 at 19:23, Arlin Davis wrote:
> >OK. I looked into this and it appears to me that the route is not copied
> >back to userspace until the callback is made.
> >
> >-- Hal
> >  
> >
> ok, I will always get the callback for now and ignore a return of 1. 
> Is this the expected behavior because it does not match the documentation.

I can look at changing the implementation but I'm curious about what
specifically in the documentation you are referring to.

-- Hal



From halr at voltaire.com  Fri Jul 15 05:20:50 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 08:20:50 -0400
Subject: [openib-general] matching an RMPP NACK to send or receive?
In-Reply-To: <42D6D2C6.8040903@ichips.intel.com>
References: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>
	<42D6D2C6.8040903@ichips.intel.com>
Message-ID: <1121430049.4423.2487.camel@hal.voltaire.com>

Hi Sean,

On Thu, 2005-07-14 at 17:01, Sean Hefty wrote:
> Eitan Zahavi wrote:
> > EZ: TID should consist of a client selected part and process ID. For kernels
> > I would select some part of the GUID too.
> > We need to minimize cases of same TID use. But in such case - abort both of
> > them.
> 
> I don't think that the receive handling code should make any assumptions 
> about the format of the TID.  

Agreed.

> Aborting both transactions seems a little 
> harsh, but not sure if there's another way.

I don't see another way either :-( All that could be done is to narrow
the window by checking on initiation of a send transaction that there is
not a matching TID in progress but this is not worth it IMO.

> For now, the code will only 
> abort a send, which should be the more common case.  I will need to add 
> aborting receives later and assume that actual TID collisions would be 
> extremely rare.

What status code will you use for this ?

> The RMPP status does help indicate whether a send or receive is being 
> aborted, but not in all cases.

What cases are not covered ?

-- Hal

> 
> - Sean
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From halr at voltaire.com  Fri Jul 15 06:02:18 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 09:02:18 -0400
Subject: [openib-general] Introducing IB MAD TCL API
In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C304B1@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED607C304B1@mtlex01.yok.mtl.com>
Message-ID: <1121432538.4423.2563.camel@hal.voltaire.com>

Hi Eitan,

On Thu, 2005-07-14 at 16:46, Eitan Zahavi wrote:
> Hi Hal,
> 
> Hal wrote:
> > Does this build ? 
> 
> > I get the following when I run autogen:
> > src/Makefile.am:46: AM_CFLAGS was already defined in condition TRUE,
> > which implies condition OSM_VENDOR_MTL_TRUE
> 
> EZ: It built for me. On many types of machines/systems.
> I guess the issue is really with the fact it did not build on a clean
> environment.

Do these build for anyone else besides you ?

> Can you please tar and gzip the source tree you are using such that I
> can double check.

Yes but there is nothing different from your tree in OpenIB.

> Also please cat /etc/redhat-release and gcc --version

Various:

Red Hat Linux release 9 (Shrike)
gcc (GCC) 3.2.2 20030222 (Red Hat Linux 3.2.2-5)

Fedora Core release 3 (Heidelberg)
gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)

> > Are these all ported and working in OpenIB ?
> EZ: They should - try only rely on osm vendor - and it should not have
> changed API. Did it?

It has not changed but the proof is in the pudding... Have you built and
run this with OpenIB ?

I don't see any OpenIB defines in the build (nor are the builds
following the OpenIB userspace conventions). 

I also see ibgd and other indications in the build.

> > Do the files have the correct licenses ?
> EZ: Yes. Did you see a wrong one?

Yes, there are some in ibdm. You should go through your tools and check
all them out yourself.

> > The trunk is for eligible for release. IMO, it seems a little early
> yet
> > for this. I would like to get them all built for OpenIB and play wit
> > them a little. 
> EZ: OK, but in order to integrate eventually to the trunk I will need
> to put them first on a branch - but probably in the right hierarchy -
> not a side branch - on a totally different place in the tree.

Do the tools not have the correct hierarchy already ?

How is the branch it is on different from any other branch ? You can
move it to another branch if that helps somehow.

-- Hal



From gshipman at lanl.gov  Fri Jul 15 07:36:34 2005
From: gshipman at lanl.gov (Galen Shipman)
Date: Fri, 15 Jul 2005 08:36:34 -0600
Subject: [openib-general] ulimit -l and ibv_reg_mr
Message-ID: <939e5142217fe38719fb8924dbe09f10@lanl.gov>

Hello all,

I am using ibv_reg_mr to register memory, if I attempt to register 
anything over what ulimit -l reports (32K) then ibv_reg_mr returns null 
and errno says Cannot allocate memory. I don't have root access so I 
can't do ulimit -l to increase my memlimit. What is the workaround for 
this issue? Modifying /etc/security/limits.conf? If so, I am using ssh 
to login remotely to the system is there still an issue with 
/etc/security/limits.conf and ssh as described in:  
http://www.mail-archive.com/openib-general at openib.org/msg06442.html

One last thing, what are the future plans regarding registering memory 
and memlimits?

Thanks,

Galen


  



From delaitt at cpc.wmin.ac.uk  Fri Jul 15 07:37:11 2005
From: delaitt at cpc.wmin.ac.uk (Thierry Delaitre)
Date: Fri, 15 Jul 2005 15:37:11 +0100 (BST)
Subject: [openib-general] Firmware needed for cougar MT23108 A1
Message-ID: <Pine.GSO.4.58.0507151534381.2014@seth.cpc.wmin.ac.uk>


Hi,

I'm trying to use the openib trunk with a cougar MT23108 A1. However, the
module complains that i have firmware 3.2.0 instead of 3.3.2. Where can i
find this firmware ? My Ib provider has got only 3.2 firmware.

Thanks,

Thierry.


From rolandd at cisco.com  Fri Jul 15 08:26:04 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 08:26:04 -0700
Subject: [openib-general] ulimit -l and ibv_reg_mr
In-Reply-To: <939e5142217fe38719fb8924dbe09f10@lanl.gov> (Galen Shipman's
	message of "Fri, 15 Jul 2005 08:36:34 -0600")
References: <939e5142217fe38719fb8924dbe09f10@lanl.gov>
Message-ID: <521x609jlf.fsf@topspin.com>

    Galen> Hello all, I am using ibv_reg_mr to register memory, if I
    Galen> attempt to register anything over what ulimit -l reports
    Galen> (32K) then ibv_reg_mr returns null and errno says Cannot
    Galen> allocate memory. I don't have root access so I can't do
    Galen> ulimit -l to increase my memlimit.

This is by design: the administrator has set a limit on the amount of
memory you can pin.  Registering memory for IB verbs requires pinning
the memory, and so you cannot register more memory than you are
allowed to pin.

    Galen> What is the workaround for this issue? Modifying
    Galen> /etc/security/limits.conf? If so, I am using ssh to login
    Galen> remotely to the system is there still an issue with
    Galen> /etc/security/limits.conf and ssh as described in:
    Galen> http://www.mail-archive.com/openib-general at openib.org/msg06442.html

If you follow the link to redhat bugzilla in that message you can find
a workaround in comment #3.  However, modifying
/etc/security/limits.conf and so on will require root access.

    Galen> One last thing, what are the future plans regarding
    Galen> registering memory and memlimits?

I don't have any plans to change the behavior, as I believe it is
correct right now.

 - R.


From rolandd at cisco.com  Fri Jul 15 08:28:31 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 08:28:31 -0700
Subject: [openib-general] Firmware needed for cougar MT23108 A1
In-Reply-To: <Pine.GSO.4.58.0507151534381.2014@seth.cpc.wmin.ac.uk> (Thierry
	Delaitre's message of "Fri, 15 Jul 2005 15:37:11 +0100 (BST)")
References: <Pine.GSO.4.58.0507151534381.2014@seth.cpc.wmin.ac.uk>
Message-ID: <52wtns84ww.fsf@topspin.com>

    Thierry> I'm trying to use the openib trunk with a cougar MT23108
    Thierry> A1. However, the module complains that i have firmware
    Thierry> 3.2.0 instead of 3.3.2. Where can i find this firmware ? 
    Thierry> My Ib provider has got only 3.2 firmware.

If you aren't experiencing any problems, then you may want to stick to
3.2.0 firmware.  The main new feature of 3.3.2 is support for MSI-X.
However, Shirley Ma reported having problems with 3.3.2 that went away
when she downgraded to 3.2.0.

If you really want to upgrade, then you can bug your HCA vendor for an
update or try downloading firmware from the Mellanox web site and
then preparing your own firmware images.

 - R.


From rangam at gmail.com  Fri Jul 15 08:39:16 2005
From: rangam at gmail.com (Srirangam Addepalli)
Date: Fri, 15 Jul 2005 10:39:16 -0500
Subject: [openib-general] VAPI_RETRY_EXC_ERR, vendor code=81
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6064BF8B1@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6064BF8B1@mtlex01.yok.mtl.com>
Message-ID: <8c02c3450507150839773f18c1@mail.gmail.com>

I was connecting the two systems with out a switch. Would this have anything 
to do with it,
i loook at the system error messages and i get the following
 ib0: no IPv6 routers present
ib1: no IPv6 routers present
 My eth subnet is 10.1.1.1 <http://10.1.1.1/>
 /sbin/ifconfig ib0 10.255.255.4 <http://10.255.255.4/> netmask
255.0.0.0<http://255.0.0.0/>and so forth for both the systems.
 Now i can ping from one system to the other using the ib addresses. How 
ever when i run the bandwidth script i get the error. I was under the 
assumption that this should work. Is this setup correct.
 Sicerly
 Rangam


On 7/14/05, Tziporet Koren <tziporet at mellanox.co.il> wrote: 
> 
> Its means that you have some connectivity problems thus got retry 
> exceeded.
> Check your cables or see if the switch dropped packets.
>  Tziporet
>  
> -----Original Message-----
> *From:* Srirangam Addepalli [mailto:rangam at gmail.com]
> *Sent:* Wednesday, July 13, 2005 9:52 PM
> *To:* openib-general at openib.org
> *Subject:* [openib-general] VAPI_RETRY_EXC_ERR, vendor code=81
> 
> Hello All,
>  I was trying to test bandwidth using the following command
>  /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 minigar c0-0 
> /usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
>  How ever i get the following error messages.
>  /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 c0-1 c0-0 
> /usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144
> Got completion with error, code=VAPI_RETRY_EXC_ERR, vendor code=81
> mpi_bandwidth: mpid/ch_vapi/viacheck.c:2109: viutil_spinandwaitcq: 
> Assertion `sc->status == VAPI_SUCCESS' failed. 
> bash: line 1: 6813 Aborted /usr/bin/env MPIRUN_HOST=c0-1 MPIRUN_PORT=34474 
> MPIRUN_RANK=1 MPIRUN_NPROCS=2 MPIRUN_ID=13416 MPIRUN_NODENAME=c0-0 
> /usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 262144 
> Timeout alarm signaled
> Cleaning up all processes ...done.
> Killed by signal 15.
>  why do i get this error. 
>  Rangam
>  
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050715/eaebccc7/attachment.html>

From suri at baymicrosystems.com  Fri Jul 15 09:03:55 2005
From: suri at baymicrosystems.com (Suresh Shelvapille)
Date: Fri, 15 Jul 2005 12:03:55 -0400
Subject: [openib-general] Generic device driver
In-Reply-To: <1121283452.4423.24.camel@hal.voltaire.com>
Message-ID: <200507151603.j6FG3wC8013553@ns1.baymicrosystems.com>

Hi:

I am using the Mellanox driver as a ref. for our own implementation of IB
device. I am only interested in MAD packets(for now!) and I have gone
through the mthca_init_one step by step. 

I see the mthca_create_agents() registering NULL for recv_handler(). 

What I am missing is, when a packet arrives at the PHY, how is the
recv_handler() being called? Just need the flow at this level. 

Thanks a lot.
Suri

-----Original Message-----
From: Hal Rosenstock [mailto:halr at voltaire.com] 
Sent: Wednesday, July 13, 2005 3:38 PM
To: Suresh Shelvapille
Cc: openib-general at openib.org
Subject: Re: [openib-general] Generic device driver

On Wed, 2005-07-13 at 15:19, Suresh Shelvapille wrote:
> Another newbee question. 
> 
> I only see a Mellanox HCA driver in the gen2/trunk, which is a PCI device.
> Where as, we have a proprietary implementation of the HCA, where the PHY
is
> accessed by the host via an FPGA (Host-bridge). 
> 
> 1. Is there a Generic Device Driver (I see it being mentioned in many
> places).

The generic device driver is a conccept. It is a set of required
(mandatory) APIs which the device driver needs to supply (at
registration time). Look at core/device.c.

> 2. If none, can I replace some files/functions within the Mellanox driver
> with the low level addresses etc, to achieve the same. 

Yes, you can use the Mellanox HCA driver structure (strip out the
implementation). You might want to change the names from mthca_xxxx to
fpga_name_xxx where fpga_name is some name used by your FPGA based
implementation.

Start with mthca_provider.c (and registration/deregistration) and build
up.

-- Hal



From rolandd at cisco.com  Fri Jul 15 09:15:50 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 09:15:50 -0700
Subject: [openib-general] Generic device driver
In-Reply-To: <200507151603.j6FG3wC8013553@ns1.baymicrosystems.com> (Suresh
	Shelvapille's message of "Fri, 15 Jul 2005 12:03:55 -0400")
References: <200507151603.j6FG3wC8013553@ns1.baymicrosystems.com>
Message-ID: <52oe9482q1.fsf@topspin.com>

    Suresh> What I am missing is, when a packet arrives at the PHY,
    Suresh> how is the recv_handler() being called? Just need the flow
    Suresh> at this level.

When a packet arrives the HCA executes a work request from the receive
queue of the appropriate queue pair.  A completion queue entry is
added to the completion queue associated with this receive queue.  If
the completion queue has had an event requested, then a completion
event will be generated and the driver needs to call the comp_handler
for the completion queue.

Chapter 10 of the IBA spec may be helpful for understanding the
semantics here.

In the case of MADs, comp_handler will be ib_mad_thread_completion_handler(),
which defers processing through a work queue and after a baroque path
eventually calls the consumer MAD agent's recv_handler.

In mthca, comp_handler is called because an event queue entry is
generated, which will usually cause an interrupt.  The interrupt
handler sees the event queue entry for the completion event and calls
mthca_cq_event().  However, there is no requirement for your device to
implement event queues or even to use interrupts.

 - R.


From mshefty at ichips.intel.com  Fri Jul 15 09:32:23 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 15 Jul 2005 09:32:23 -0700
Subject: [openib-general] matching an RMPP NACK to send or receive?
In-Reply-To: <1121430049.4423.2487.camel@hal.voltaire.com>
References: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>	
	<42D6D2C6.8040903@ichips.intel.com>
	<1121430049.4423.2487.camel@hal.voltaire.com>
Message-ID: <42D7E517.3030608@ichips.intel.com>

Hal Rosenstock wrote:
>>For now, the code will only 
>>abort a send, which should be the more common case.  I will need to add 
>>aborting receives later and assume that actual TID collisions would be 
>>extremely rare.
> 
> What status code will you use for this ?

I'm returning IB_WC_REM_ABORT_ERR and placing the actual RMPP status code in 
the vendor_err field.

>>The RMPP status does help indicate whether a send or receive is being 
>>aborted, but not in all cases.
> 
> What cases are not covered ?

I believe that the following status values are possible for either a send or 
receive:

resources exhausted
bad RMPP type
illegal status
unspecified and class specific

- Sean


From mamidala at cse.ohio-state.edu  Fri Jul 15 09:35:53 2005
From: mamidala at cse.ohio-state.edu (amith rajith mamidala)
Date: Fri, 15 Jul 2005 12:35:53 -0400 (EDT)
Subject: [openib-general] Error in opening IB cm device
In-Reply-To: <1121409378.20046.260.camel@localhost>
Message-ID: <Pine.GSO.4.40.0507151205500.5849-100000@kappa.cse.ohio-state.edu>

Hi,

I couldn't find /dev/infiniband in my installation. After adding this
manually and doing mknod as you suggested the error in opening the
/dev/infiniband/ucm disappeared. However, now the program
(ibv_rc_pingpong) is not able to create QP.


I might be missing some steps in the installation.

What I have done is as follows:

1. Link kernel code in place
~> rm -fr /usr/linux-2.6.11/drivers/infiniband
~> ln -s /usr/src/roland-uverbs/src/linux-kernel/infiniband
/usr/linux-2.6.11/drivers/infiniband

2. Build kernel components
~>cd /usr/linux-2.6.11
~>make menuconfig

You are presented with a menu. Scroll to "device drivers", select it,
select Infiniband support.
Select userspace support, Mellanox HCA support.

Exit saving kernel configuration.

~>make modules modules_install
This shall build and install relevant kernel modules.

Note how low level module (mthca) is loaded automatically upon reboot, so
if you change this,
reboot or unload it as I explain later.

3. Loading kernel modules.
~>udevstart 2>&1 > /dev/null
~>modprobe ib_mthca
~>modprobe ib_umad
~>modprobe ib_uverbs


However I was not able to add the udev rules as specified in the following
instruction. I could not locate the /etc/udev/udev.rules.

---------------------------------------------------------------------
3. Adding udev rules.
You need device files for kernel modules we just built.
We could add them later, manually, but udev is more convenient.

Add this to /etc/udev/udev.rules (if the file does not exist, udev may not
be installed, you'll
have to add devices manually, I'll skip this part here, lets assume its
OK).

KERNEL="umad*", NAME="infiniband/%k"
KERNEL="issm*", NAME="infiniband/%k"
KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
----------------------------------------------------------------------

Is this the cause for the problem ?


Thanks,
Amith




On Thu, 14 Jul 2005, Matt Leininger wrote:

> On Fri, 2005-07-15 at 01:47 -0400, amith rajith mamidala wrote:
> > Hi,
> >
> > We have installed the latest openib stack (Rev:2861) on the X86_64
> > platform. While running the pingpong tests I am encountering the
> > following error:
> >
> > libucm: Error <-1:2> couldn't open IB cm device </dev/infiniband/ucm>
> > [1] Abort: Error getting HCA context
> >
> > I am not sure why this happening ?
> >
>
> What does 'ls -l /dev/infiniband' say?  You should have:
>
> [root at n64-81 ~]# ls -l /dev/infiniband/
> total 0
> crw-rw-rw-    1 root     root     231, 255 Jul 13 10:44 ucm
> crw-rw-rw-    1 root     root     231, 192 Jul 13 10:45 uverbs0
> crw-rw-rw-    1 root     root     231, 193 Jul 13 10:45 uverbs1
>
> Try,
>
> modprobe ib_uverbs
> mknod -m 0666 /dev/infiniband/uverbs0 c 231 192
> mknod -m 0666 /dev/infiniband/uverbs1 c 231 193
>
>
> modprobe ib_ucm
> mknod /dev/infiniband/ucm c 231 255
>
>   - Matt
>
>




From rangam at gmail.com  Fri Jul 15 09:48:01 2005
From: rangam at gmail.com (Srirangam Addepalli)
Date: Fri, 15 Jul 2005 11:48:01 -0500
Subject: [openib-general] Topspin Errors
Message-ID: <8c02c345050715094830927a25@mail.gmail.com>

Hello,
 I have two three machines connected with out a switch, i can ping from all 
three to the others. How ever when i try a mpirun_ssh i get into errors. Are 
there any particular steps i need to look at if not using a switch.
 Thankyou
 Rangam
  Following is my test run
 /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 c0-0 c0-1 
/usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 26281
Got completion with error, code=VAPI_RETRY_EXC_ERR, vendor code=81
mpi_bandwidth: mpid/ch_vapi/viacheck.c:2109: viutil_spinandwaitcq: Assertion 
`sc->status == VAPI_SUCCESS' failed.
bash: line 1: 22751 Aborted /usr/bin/env
MPIRUN_HOST=compute-0-1.localMPIRUN_PORT=33538 MPIRUN_RANK=1
MPIRUN_NPROCS=2 MPIRUN_ID=22730
MPIRUN_NODENAME=c0-1 /usr/local/topspin/mpi/mpich/bin/mpi_bandwidth 1000 
26281
Timeout alarm signaled
Cleaning up all processes ...done.
Killed by signal 15.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050715/96661861/attachment.html>

From ardavis at ichips.intel.com  Fri Jul 15 10:04:03 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Fri, 15 Jul 2005 10:04:03 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121425731.4423.2145.camel@hal.voltaire.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>	
	<1121351537.4423.1162.camel@hal.voltaire.com>	
	<42D6F3F5.5020806@ichips.intel.com>
	<1121425731.4423.2145.camel@hal.voltaire.com>
Message-ID: <42D7EC83.8030406@ichips.intel.com>

Hal Rosenstock wrote:

>Hi Arlin,
>
>On Thu, 2005-07-14 at 19:23, Arlin Davis wrote:
>  
>
>>>OK. I looked into this and it appears to me that the route is not copied
>>>back to userspace until the callback is made.
>>>
>>>-- Hal
>>> 
>>>
>>>      
>>>
>>ok, I will always get the callback for now and ignore a return of 1. 
>>Is this the expected behavior because it does not match the documentation.
>>    
>>
>
>I can look at changing the implementation but I'm curious about what
>specifically in the documentation you are referring to.
>  
>

at.h

* The following asynchronous resolution function behavior is as follows:
 *    If the resolve operation can be fulfilled immediately, then the output
 *    structures are set and the number of filled structures is returned.
 *
 *    If the resolve operation cannot by fulfilled immediately and
 *    an ib_at_completion structure is not provided,
 *    then the function immediately returns -EWOULDBLOCK.
 *
 *     If ib_at_completion structure is provided and an asynchronous
 *    operation is started, the function immediately returns zero,
 *    and the request ID field (req_id) is set if the pointer is
 *    non NULL. This request ID may be used to cancel the operation,
 *    or to poll its status.

Your implementation is fine, as long as the correct return values are used
and the sync version works. I currently get returned a 1 but the ib_route
data is not filled in and the req_id == 0. I would expect a 0 returned 
and a
valid req_id if the ib_route cannot be processed.

for ret = ib_at_route_by_ip( async_cb provided ), I read this as: 

  if ret > 0  then ib_route output data is written, resolve complete
  if ret == 0 then req_id is written with valid id to cancel or poll
  if ret < 0 error

Am I reading this right?
 
-arlin

>-- Hal
>
>  
>



From rolandd at cisco.com  Fri Jul 15 10:33:39 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 10:33:39 -0700
Subject: [openib-general] Re: [PATCH] crash in ibv_destroy_ah
In-Reply-To: <20050712114846.GG9125@minantech.com> (Gleb Natapov's message
	of "Tue, 12 Jul 2005 14:48:46 +0300")
References: <20050712114846.GG9125@minantech.com>
Message-ID: <52eka07z4c.fsf@topspin.com>

Thanks, applied.


From rolandd at cisco.com  Fri Jul 15 10:36:30 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 10:36:30 -0700
Subject: [openib-general] Re: [PATCH] IPoIB: Eliminate NULL checks prior to
	calling kfree
In-Reply-To: <1121257492.4401.83.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "13 Jul 2005 08:24:53 -0400")
References: <1121257492.4401.83.camel@hal.voltaire.com>
Message-ID: <52acko7yzl.fsf@topspin.com>

Thanks, applied.


From rolandd at cisco.com  Fri Jul 15 10:38:08 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 10:38:08 -0700
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <1121365546.17302.3.camel@duffman> (Tom Duffy's message of
	"Thu, 14 Jul 2005 11:25:46 -0700")
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
	<1121365546.17302.3.camel@duffman>
Message-ID: <5264vc7ywv.fsf@topspin.com>

    Tom> Hrm, my arping on Fedora Core 4 now does real bad stuff:

Is it allocating a buffer as big as an ethernet address and then
trying to put a 20-byte IPoIB address in it?

 - R.


From bohra at cs.rutgers.edu  Fri Jul 15 10:53:16 2005
From: bohra at cs.rutgers.edu (Aniruddha Bohra)
Date: Fri, 15 Jul 2005 13:53:16 -0400
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <Pine.LNX.4.61.0507141226140.25603@jlentini-linux.nane.netapp.com>
References: <42D5599F.8020202@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>
	<42D56AB0.6070804@cs.rutgers.edu>
	<Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>
	<42D591FB.6010106@cs.rutgers.edu>
	<Pine.LNX.4.61.0507141226140.25603@jlentini-linux.nane.netapp.com>
Message-ID: <42D7F80C.4060706@cs.rutgers.edu>

James Lentini wrote:

>
> On Wed, 13 Jul 2005, Aniruddha Bohra wrote:
>
>>   Basically libibcm is not linked with dapltest and when the 
>> dat_provider
>> tries to load, and calls ib_cm_send_req, it fails.
>>
>>   So, just adding -libcm to the dapltest link commands make it work.
>> It would be great if it could be added to the svn repo.
>
>
> Thanks for pointing this out. I've just updated the makefile (revision 
> 2859).

Hello,
    The next problem -
When I start the dapltest, it binds to the address of eth0 and not of ib0.
 From the dapltest start log :

*****  DAPL  Characteristics  *****
Provider: IB1  Version 1.0  DAPL 1.2
Adapter:  by  Version 0.0
Supporting:
        64000 EPs with 64000 DTOs and 8 RDMA/RDs each
        64000 EVDs of up to 64000 entries  (default S/R size is 16/16)
        IOVs of up to 32 elements
        64000 LMRs (and 64000 RMRs) of up to 0x80000000 bytes
        Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes
        Maximum Private data size 92 bytes
        Local IP address 192.168.133.9
***** ***** ***** ***** ***** *****

When :

ib0       Link encap:UNSPEC  HWaddr 
00-00-00-14-FE-80-00-00-00-00-00-00-00-00-00-00 
          inet addr:10.10.10.24  Bcast:10.255.255.255  Mask:255.255.0.0
          inet6 addr: fe80::202:c901:81e:94d1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:23 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:128
          RX bytes:1764 (1.7 KiB)  TX bytes:1804 (1.7 KiB)

My dat.conf reads :
IB1 u1.2 nonthreadsafe default /usr/local/lib/libdapl.so mv_dapl.1.2 
"mthca0 1" " "

How can I ensure the binding to ib0, the 10.10.... interface?

Thanks
Aniruddha



From kingman at storagegear.com  Fri Jul 15 10:54:35 2005
From: kingman at storagegear.com (John Kingman)
Date: Fri, 15 Jul 2005 17:54:35 +0000 (UTC)
Subject: [openib-general] Re: Firmware needed for cougar MT23108 A1
References: <Pine.GSO.4.58.0507151534381.2014@seth.cpc.wmin.ac.uk>
Message-ID: <loom.20050715T194053-81@post.gmane.org>


Thierry> I'm trying to use the openib trunk with a cougar MT23108 A1. 
Thierry> However, the module complains that i have firmware 3.2.0 instead 
Thierry> of 3.3.2. Where can i find this firmware ? My Ib provider has got 
THierry> only 3.2 firmware.

You can find the firmware for the Mellanox 23108 on the Mellanox IB Gold
distributions.  Here are some recent correlations:

     IBGD     23108 firmware
    -------- ------------------------
     1.7.0    fw-23108-rel-3_3_2
     1.8.0    fw-23108-rel-3_3_3

IBGD is available via the Mellanox website at http://www.mellanox.com 

John



From tduffy at sun.com  Fri Jul 15 11:31:38 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 15 Jul 2005 11:31:38 -0700
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <5264vc7ywv.fsf@topspin.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
	<1121365546.17302.3.camel@duffman>  <5264vc7ywv.fsf@topspin.com>
Message-ID: <1121452298.15851.25.camel@duffman>

On Fri, 2005-07-15 at 10:38 -0700, Roland Dreier wrote:
>     Tom> Hrm, my arping on Fedora Core 4 now does real bad stuff:
> 
> Is it allocating a buffer as big as an ethernet address and then
> trying to put a 20-byte IPoIB address in it?

It seems to be happening at line 479 of arping.c:

	memset(he.sll_addr, -1, he.sll_halen);

where he.sll_addr is 8 big, he.sll_halen is 20

So, yeah.

BTW, FC4 compiles with -Wp,-D_FORTIFY_SOURCE=2 so it is now catching it.

Should struct sockaddr_ll be modified in include/linux/if_packet.h to
accommodate larger addresses?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050715/4a2d2f9d/attachment.sig>

From halr at voltaire.com  Fri Jul 15 11:47:30 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 14:47:30 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <1121452298.15851.25.camel@duffman>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
	<1121365546.17302.3.camel@duffman>  <5264vc7ywv.fsf@topspin.com>
	<1121452298.15851.25.camel@duffman>
Message-ID: <1121453250.4423.2574.camel@hal.voltaire.com>

On Fri, 2005-07-15 at 14:31, Tom Duffy wrote:
> On Fri, 2005-07-15 at 10:38 -0700, Roland Dreier wrote:
> >     Tom> Hrm, my arping on Fedora Core 4 now does real bad stuff:
> > 
> > Is it allocating a buffer as big as an ethernet address and then
> > trying to put a 20-byte IPoIB address in it?
> 
> It seems to be happening at line 479 of arping.c:
> 
> 	memset(he.sll_addr, -1, he.sll_halen);
> 
> where he.sll_addr is 8 big, he.sll_halen is 20
> 
> So, yeah.
> 
> BTW, FC4 compiles with -Wp,-D_FORTIFY_SOURCE=2 so it is now catching it.
> 
> Should struct sockaddr_ll be modified in include/linux/if_packet.h to
> accommodate larger addresses?

There will be other problems beyond that, right ? Won't we be back to
the sending of link level packets from user space again ? 

-- Hal



From tduffy at sun.com  Fri Jul 15 11:51:24 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 15 Jul 2005 11:51:24 -0700
Subject: [openib-general] [ANNOUNCE] Initial checkin of SRP initiator
In-Reply-To: <527jfsc1ri.fsf@topspin.com>
References: <52r7e8muj1.fsf@topspin.com> <1121381985.17302.35.camel@duffman>
	<527jfsc1ri.fsf@topspin.com>
Message-ID: <1121453484.15851.26.camel@duffman>

On Thu, 2005-07-14 at 18:10 -0700, Roland Dreier wrote:
>     Tom> svn propdel svn:executable ib_srp.[ch]
>     Tom> please.
> 
> Weird:
> 
>     svn proplist https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/srp/ib_srp.{c,h}
>     Properties on 'https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/srp/ib_srp.c':
>       svn:keywords
>     Properties on 'https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/srp/ib_srp.h':
>       svn:keywords

Huh, I wonder why they were executable in my tree.  Never mind.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050715/546a4247/attachment.sig>

From halr at voltaire.com  Fri Jul 15 11:53:22 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 14:53:22 -0400
Subject: [openib-general] matching an RMPP NACK to send or receive?
In-Reply-To: <42D7E517.3030608@ichips.intel.com>
References: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>
	<42D6D2C6.8040903@ichips.intel.com>
	<1121430049.4423.2487.camel@hal.voltaire.com>
	<42D7E517.3030608@ichips.intel.com>
Message-ID: <1121453601.4423.2591.camel@hal.voltaire.com>

On Fri, 2005-07-15 at 12:32, Sean Hefty wrote:
> Hal Rosenstock wrote:
> >>For now, the code will only 
> >>abort a send, which should be the more common case.  I will need to add 
> >>aborting receives later and assume that actual TID collisions would be 
> >>extremely rare.
> > 
> > What status code will you use for this ?
> 
> I'm returning IB_WC_REM_ABORT_ERR and placing the actual RMPP status code in 
> the vendor_err field.

What vendor errors are being defined ? I realize that is the only way to
do this now but aren't they really standard rather than vendor errors ?

> >>The RMPP status does help indicate whether a send or receive is being 
> >>aborted, but not in all cases.
> > 
> > What cases are not covered ?
> 
> I believe that the following status values are possible for either a send or 
> receive:
> 
> resources exhausted
> bad RMPP type
> illegal status
> unspecified and class specific

What's unspecified ? Is it reserved status in ABORT ?

There are some others too. Does the OpenIB RMPP not generate them ? Will
it pass them back up if it received an abort with these other statuses ?

-- Hal




From ardavis at ichips.intel.com  Fri Jul 15 12:02:58 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Fri, 15 Jul 2005 12:02:58 -0700
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <42D7F80C.4060706@cs.rutgers.edu>
References: <42D5599F.8020202@cs.rutgers.edu>	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>	<42D56AB0.6070804@cs.rutgers.edu>	<Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>	<42D591FB.6010106@cs.rutgers.edu>	<Pine.LNX.4.61.0507141226140.25603@jlentini-linux.nane.netapp.com>
	<42D7F80C.4060706@cs.rutgers.edu>
Message-ID: <42D80862.2060708@ichips.intel.com>

Aniruddha Bohra wrote:

> James Lentini wrote:
>

Hi Aniruddha ,

Sorry, uDAPL is still in development (3 phases) and you picked up right 
in the middle of IB_CM and IB_AT porting so things are a little tricky. 
I would strongly recommend sticking with phase 1 for now ( svn r2830 ).

There are basically three phases to this development:

   1.  (get svn r2830) First phase ran with a SOCKET_CM and ib_uverbs. 
Will scale out beyond 2 nodes. Does not require IPoIB.

   *2. Current Release:*
   (get svn r2831) Second phase with uCM and ib_uverbs. Path records are 
hand-rolled. IP addresses are not used so ignore
   any reporting for now. Local LID/GID is picked up from port 1 and the 
remote LID/GID is specified with enviroment variables.
   IP. You can use any remote address you want, it will simply pick up 
your DAPL_REMOTE_GID and DAPL_REMOTE_LID.
   This is limited to loopback or 2 node configurations. Does not 
require IPoIB.
    See openib/gen2/users/jlentini/userspace/dapl/openib/README for details.

    3. (in process, soon) Third and final phase will include uCM, uAT, 
and ib_uverbs. I have this working and expect to put
    together a  patch next week. Note:  uAT still in separate branch 
(/openib/gen2/branches/shaharf-ibat/src/userspace/libibat)
    This version will require IPoIB.

If you can wait a week then the uAT version will be ready and the IP 
addresses to GID mappings will all match.

-arlin

>
> My dat.conf reads :
> IB1 u1.2 nonthreadsafe default /usr/local/lib/libdapl.so mv_dapl.1.2 
> "mthca0 1" " "
>
> How can I ensure the binding to ib0, the 10.10.... interface?
>
> Thanks
> Aniruddha
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>



From mshefty at ichips.intel.com  Fri Jul 15 12:17:50 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 15 Jul 2005 12:17:50 -0700
Subject: [openib-general] matching an RMPP NACK to send or receive?
In-Reply-To: <1121453601.4423.2591.camel@hal.voltaire.com>
References: <506C3D7B14CDD411A52C00025558DED607C304B2@mtlex01.yok.mtl.com>	
	<42D6D2C6.8040903@ichips.intel.com>	
	<1121430049.4423.2487.camel@hal.voltaire.com>	
	<42D7E517.3030608@ichips.intel.com>
	<1121453601.4423.2591.camel@hal.voltaire.com>
Message-ID: <42D80BDE.6050907@ichips.intel.com>

Hal Rosenstock wrote:
>>I'm returning IB_WC_REM_ABORT_ERR and placing the actual RMPP status code in 
>>the vendor_err field.
> 
> What vendor errors are being defined ? I realize that is the only way to
> do this now but aren't they really standard rather than vendor errors ?

No vendor errors are being defined.  I'm simply returning the received RMPP 
Status in that field since it's not used in this case.  We can rename 
vendor_err to something else.

>>I believe that the following status values are possible for either a send or 
>>receive:
>>
>>resources exhausted
>>bad RMPP type
>>illegal status
>>unspecified and class specific
> 
> What's unspecified ? Is it reserved status in ABORT ?

Unspecified is status value 127 - "an error .. not covered by any other 
status code".  This is handled, but only aborts a send request if received.

> There are some others too. Does the OpenIB RMPP not generate them ? Will
> it pass them back up if it received an abort with these other statuses ?

I think that the other status codes occur only in response to a send OR to a 
receive.  So it's not that they're not generated, I just have a better idea 
on how to handle them.

Total time too long - assuming sent by receiver to sender.  This is handled.

Inconsistent last and payload length - sent by receiver to sender.  This is 
not transmitted currently, but should work fine if received.  (I need to add 
state information to transmit this.)

Inconsistent first and segment number - transmitted by receiver to sender. 
This is handled.

Bad RMPP type - transmitted either way.  This is checked for all messages, 
but will only abort a send request if received.

New window last too small - transmitted by sender to receiver.  This is handled.

Segment number too big - transmitted by sender to receiver.  This is handled.

Illegal status - transmitted either way.  This is checked for all messages, 
but will only abort a send request if received.

Unsupported version - transmitted by receiver to sender.  This is handled.

Too many retries - assuming sent by sender to receiver.  This is not 
handled.  (I need to add a new call from the MAD code into the RMPP layer to 
support this.)

- Sean


From halr at voltaire.com  Fri Jul 15 12:32:22 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 15:32:22 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42D7EC83.8030406@ichips.intel.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
	<1121351537.4423.1162.camel@hal.voltaire.com>
	<42D6F3F5.5020806@ichips.intel.com>
	<1121425731.4423.2145.camel@hal.voltaire.com>
	<42D7EC83.8030406@ichips.intel.com>
Message-ID: <1121455703.4423.2664.camel@hal.voltaire.com>

Hi Arlin,

On Fri, 2005-07-15 at 13:04, Arlin Davis wrote:
> at.h
> 
> * The following asynchronous resolution function behavior is as follows:
>  *    If the resolve operation can be fulfilled immediately, then the output
>  *    structures are set and the number of filled structures is returned.
>  *
>  *    If the resolve operation cannot by fulfilled immediately and
>  *    an ib_at_completion structure is not provided,
>  *    then the function immediately returns -EWOULDBLOCK.
>  *
>  *     If ib_at_completion structure is provided and an asynchronous
>  *    operation is started, the function immediately returns zero,
>  *    and the request ID field (req_id) is set if the pointer is
>  *    non NULL. This request ID may be used to cancel the operation,
>  *    or to poll its status.
> 
> Your implementation is fine, as long as the correct return values are used
> and the sync version works. I currently get returned a 1 but the ib_route
> data is not filled in and the req_id == 0. I would expect a 0 returned 
> and a
> valid req_id if the ib_route cannot be processed.
> 
> for ret = ib_at_route_by_ip( async_cb provided ), I read this as: 
> 
>   if ret > 0  then ib_route output data is written, resolve complete
>   if ret == 0 then req_id is written with valid id to cancel or poll
>   if ret < 0 error
> 
> Am I reading this right?

Yes, I copied that verbatim from kernel AT header into the user AT
library header. 

For user AT, I implemented (perhaps mistakenly) the copy when the
callback occurs. I will see what it takes to make the semantics similar.

-- Hal



From rolandd at cisco.com  Fri Jul 15 13:07:33 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 13:07:33 -0700
Subject: [openib-general] [RFC] First pass at InfiniBand SRP initiator
Message-ID: <52ackn7rzu.fsf@topspin.com>

Here's a first pass at an InfiniBand SCSI RDMA Protocol (SRP)
initiator.  This allows systems to talk to storage over an IB network--
IB SRP storage is available from a number of vendors.

This obviously isn't ready for merging yet, since it implements no
error handling, etc.  However, I'd appreciate any and all comments on
the code that exists, so that I can fix things up before going too far
off into the weeds.

One design note: all discovery of storage is assumed to be pushed off
into userspace, which then tells the kernel to connect to a given
target port by doing something like:

    echo id_ext=21000004cfe7a949,ioc_guid=0005ad00000015dd,dgid=fe800000000000000005ad00000015dd,pkey=ffff,service_id=0000000000000066 > /sys/class/infiniband_srp/srp-mthca1-1/add_target

Thanks,
  Roland

--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ linux-export/drivers/infiniband/ulp/srp/ib_srp.c	2005-07-15 13:02:00.278274501 -0700
@@ -0,0 +1,1344 @@
+/*
+ * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ib_srp.c 2860 2005-07-14 16:39:43Z roland $
+ */
+
+#include <linux/version.h>
+#include <linux/module.h>
+
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/err.h>
+#include <linux/idr.h>
+#include <linux/string.h>
+#include <linux/parser.h>
+
+#include <asm/atomic.h>
+
+#include <scsi/scsi.h>
+#include <scsi/scsi_device.h>
+#include <scsi/scsi_dbg.h>
+
+#include <ib_cache.h>
+
+#include "ib_srp.h"
+
+#define DRV_NAME	"ib_srp"
+#define PFX		DRV_NAME ": "
+#define DRV_VERSION	"0.01"
+#define DRV_RELDATE	"January 11, 2005"
+
+MODULE_AUTHOR("Roland Dreier");
+MODULE_DESCRIPTION("InfiniBand SCSI RDMA Protocol driver");
+MODULE_LICENSE("Dual BSD/GPL");
+
+static int topspin_workarounds = 1;
+
+module_param(topspin_workarounds, int, 0444);
+MODULE_PARM_DESC(topspin_workarounds,
+		 "Enable workarounds for Topspin/Cisco SRP target bugs if != 0");
+
+static const u8 topspin_oui[3] = { 0x00, 0x05, 0xad };
+
+static atomic_t srp_uid;
+
+static rwlock_t idr_lock;
+static DEFINE_IDR(target_idr);
+
+static void srp_add_one(struct ib_device *device);
+static void srp_remove_one(struct ib_device *device);
+
+static struct ib_client srp_client = {
+	.name   = "srp",
+	.add    = srp_add_one,
+	.remove = srp_remove_one
+};
+
+static inline struct srp_target_port *host_to_target(struct Scsi_Host *host)
+{
+	return (struct srp_target_port *) host->hostdata;
+}
+
+static const char *srp_target_info(struct Scsi_Host *host)
+{
+	return host_to_target(host)->target_name;
+}
+
+static struct srp_iu *srp_alloc_iu(struct srp_host *host, size_t size,
+				   unsigned int __nocast gfp_mask,
+				   enum dma_data_direction direction)
+{
+	struct srp_iu *iu;
+
+	iu = kmalloc(sizeof *iu, gfp_mask);
+	if (!iu)
+		return NULL;
+
+	iu->buf = kmalloc(size, gfp_mask);
+	if (!iu->buf) {
+		kfree(iu);
+		return NULL;
+	}
+
+	memset(iu->buf, 0, size);
+
+	iu->dma = dma_map_single(host->dev->dma_device, iu->buf, size, direction);
+	if (dma_mapping_error(iu->dma)) {
+		kfree(iu->buf);
+		kfree(iu);
+		return NULL;
+	}
+
+	iu->size      = size;
+	iu->direction = direction;
+
+	return iu;
+}
+
+static void srp_free_iu(struct srp_host *host, struct srp_iu *iu)
+{
+	if (!iu)
+		return;
+
+	dma_unmap_single(host->dev->dma_device, iu->dma, iu->size, iu->direction);
+	kfree(iu->buf);
+	kfree(iu);
+}
+
+static int srp_map_data(struct scsi_cmnd *scmnd, struct srp_target_port *target,
+			struct srp_iu *iu)
+{
+	struct srp_cmd *cmd = iu->buf;
+	int len;
+	u8 fmt;
+
+	if (!scmnd->request_buffer || scmnd->sc_data_direction == DMA_NONE)
+		return sizeof (struct srp_cmd);
+
+	if (scmnd->sc_data_direction != DMA_FROM_DEVICE &&
+	    scmnd->sc_data_direction != DMA_TO_DEVICE) {
+		printk(KERN_WARNING PFX "Unhandled data direction %d\n",
+		       scmnd->sc_data_direction);
+		return -EINVAL;
+	}
+
+	if (scmnd->use_sg) {
+		struct scatterlist *scat = scmnd->request_buffer;
+		int n;
+		int i;
+
+		n = dma_map_sg(target->srp_host->dev->dma_device,
+			       scat, scmnd->use_sg, scmnd->sc_data_direction);
+
+		if (n == 1) {
+			struct srp_direct_buf *buf = (void *) cmd->add_data;
+
+			fmt = SRP_DATA_DESC_DIRECT;
+
+			buf->va  = cpu_to_be64(sg_dma_address(scat));
+			buf->key = cpu_to_be32(target->srp_host->mr->rkey);
+			buf->len = cpu_to_be32(sg_dma_len(scat));
+
+			len = sizeof (struct srp_cmd) +
+				sizeof (struct srp_direct_buf);
+		} else {
+			struct srp_indirect_buf *buf = (void *) cmd->add_data;
+			u32 datalen = 0;
+
+			fmt = SRP_DATA_DESC_INDIRECT;
+
+			if (scmnd->sc_data_direction == DMA_TO_DEVICE)
+				cmd->data_out_desc_cnt = n;
+			else
+				cmd->data_in_desc_cnt = n;
+
+			buf->table_desc.va  = cpu_to_be64(iu->dma +
+							  sizeof *cmd +
+							  sizeof *buf);
+			buf->table_desc.key =
+				cpu_to_be32(target->srp_host->mr->rkey);
+			buf->table_desc.len =
+				cpu_to_be32(n * sizeof (struct srp_direct_buf));
+
+			for (i = 0; i < n; ++i) {
+				buf->desc_list[i].va  = cpu_to_be64(sg_dma_address(&scat[i]));
+				buf->desc_list[i].key =
+					cpu_to_be32(target->srp_host->mr->rkey);
+				buf->desc_list[i].len = cpu_to_be32(sg_dma_len(&scat[i]));
+
+				datalen += sg_dma_len(&scat[i]);
+			}
+
+			buf->len = cpu_to_be32(datalen);
+
+			len = sizeof (struct srp_cmd) +
+				sizeof (struct srp_indirect_buf) +
+				n * sizeof (struct srp_direct_buf);
+		}
+	} else {
+		struct srp_direct_buf *buf = (void *) cmd->add_data;
+		dma_addr_t dma;
+
+		dma = dma_map_single(target->srp_host->dev->dma_device,
+				     scmnd->request_buffer, scmnd->request_bufflen,
+				     scmnd->sc_data_direction);
+		if (dma_mapping_error(dma)) {
+			printk(KERN_WARNING PFX "unable to map %p/%d (dir %d)\n",
+			       scmnd->request_buffer, (int) scmnd->request_bufflen,
+			       scmnd->sc_data_direction);
+			return -EINVAL;
+		}
+
+		buf->va  = cpu_to_be64(dma);
+		buf->key = cpu_to_be32(target->srp_host->mr->rkey);
+		buf->len = cpu_to_be32(scmnd->request_bufflen);
+
+		fmt = SRP_DATA_DESC_DIRECT;
+
+		len = sizeof (struct srp_cmd) + sizeof (struct srp_direct_buf);
+	}
+
+	if (scmnd->sc_data_direction == DMA_TO_DEVICE)
+		cmd->buf_fmt = fmt << 4;
+	else
+		cmd->buf_fmt = fmt;
+
+
+	return len;
+}
+
+static void srp_unmap_data(struct scsi_cmnd *scmnd,
+			   struct srp_target_port *target,
+			   struct srp_cmd *cmd)
+{
+	if (!scmnd->request_buffer ||
+	    (scmnd->sc_data_direction != DMA_TO_DEVICE &&
+	     scmnd->sc_data_direction != DMA_FROM_DEVICE))
+	    return;
+
+	if (scmnd->use_sg)
+		dma_unmap_sg(target->srp_host->dev->dma_device,
+			     (struct scatterlist *) scmnd->request_buffer,
+			     scmnd->use_sg, scmnd->sc_data_direction);
+	else
+		dma_unmap_single(target->srp_host->dev->dma_device,
+				 be64_to_cpu(((struct srp_direct_buf *) cmd->add_data)->va),
+				 scmnd->request_bufflen,
+				 scmnd->sc_data_direction);
+}
+
+static void srp_process_rsp(struct srp_target_port *target, struct srp_rsp *rsp)
+{
+	struct scsi_cmnd *scmnd = (void *) (unsigned long) rsp->tag;
+	struct srp_iu *iu = (void *) scmnd->host_scribble;
+	unsigned long flags;
+	s32 delta;
+
+	delta = (s32) be32_to_cpu(rsp->req_lim_delta);
+
+	spin_lock_irqsave(&target->lock, flags);
+	target->req_lim += delta;
+	spin_unlock_irqrestore(&target->lock, flags);
+
+	scmnd->result = rsp->status;
+
+	if (rsp->flags & SRP_RSP_FLAG_SNSVALID) {
+		memcpy(scmnd->sense_buffer, rsp->data +
+		       be32_to_cpu(rsp->resp_data_len),
+		       min_t(int, be32_to_cpu(rsp->sense_data_len),
+			     SCSI_SENSE_BUFFERSIZE));
+	}
+
+	if (rsp->flags & (SRP_RSP_FLAG_DOOVER | SRP_RSP_FLAG_DOUNDER))
+		scmnd->resid = be32_to_cpu(rsp->data_out_res_cnt);
+	else if (rsp->flags & (SRP_RSP_FLAG_DIOVER | SRP_RSP_FLAG_DIUNDER))
+		scmnd->resid = be32_to_cpu(rsp->data_in_res_cnt);
+
+	if (!scmnd->host_scribble)
+		printk(KERN_ERR PFX "scsi command %p with NULL host_scribble!\n", scmnd);
+
+	srp_unmap_data(scmnd, target, iu->buf);
+	srp_free_iu(target->srp_host, iu);
+
+	scmnd->scsi_done(scmnd);
+}
+
+static void srp_handle_recv(struct srp_target_port *target, struct ib_wc *wc)
+{
+	struct srp_iu *iu;
+	u8 opcode;
+
+	iu = target->rx_ring[wc->wr_id & ~SRP_OP_RECV];
+
+	if (!iu->buf) {
+		printk(KERN_ERR PFX "iu with NULL buf; index %d (head %u, tail %u)\n",
+		       (int) wc->wr_id & ~SRP_OP_RECV, target->rx_head, target->rx_tail);
+		return;
+	} else
+		opcode = *(u8 *) iu->buf;
+
+	if (wc->status) {
+		printk(KERN_ERR PFX "failed recv status %d\n", wc->status);
+		goto out;
+	}
+
+	if (0) {
+		int i;
+
+		printk(KERN_ERR PFX "recv completion, opcode 0x%02x\n", opcode);
+
+		for (i = 0; i < wc->byte_len; ++i) {
+			if (i % 8 == 0)
+				printk(KERN_ERR "  [%02x] ", i);
+			printk(" %02x", ((u8 *) iu->buf)[i]);
+			if ((i + 1) % 8 == 0)
+				printk("\n");
+		}
+
+		if (wc->byte_len % 8)
+			printk("\n");
+	}
+
+	switch (opcode) {
+	case SRP_RSP:
+		srp_process_rsp(target, iu->buf);
+		break;
+
+	case SRP_T_LOGOUT:
+		/* XXX Handle target logout */
+		printk(KERN_WARNING PFX "Got target logout request\n");
+		break;
+
+	default:
+		printk(KERN_WARNING PFX "Unhandled SRP opcode 0x%02x\n", opcode);
+		break;
+	}
+
+out:
+	srp_free_iu(target->srp_host, iu);
+	++target->rx_tail;
+}
+
+static void srp_handle_send(struct srp_target_port *target, struct ib_wc *wc)
+{
+	if (wc->status) {
+		struct srp_iu *iu;
+
+		printk(KERN_ERR "failed send status %d\n", wc->status);
+
+		iu = target->tx_ring[wc->wr_id];
+
+		/* XXX fail associated SCSI command */
+
+		srp_free_iu(target->srp_host, iu);
+	}
+
+	++target->tx_tail;
+}
+
+static void srp_completion(struct ib_cq *cq, void *target_ptr)
+{
+	struct srp_target_port *target = target_ptr;
+	struct ib_wc wc;
+
+	ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
+	while (ib_poll_cq(cq, 1, &wc) > 0) {
+		if (wc.wr_id & SRP_OP_RECV)
+			srp_handle_recv(target, &wc);
+		else
+			srp_handle_send(target, &wc);
+	}
+}
+
+static void srp_qp_event(struct ib_event *event, void *context)
+{
+	printk(KERN_ERR PFX "QP event %d\n", event->event);
+}
+
+static int srp_post_recv(struct srp_target_port *target,
+			 unsigned int __nocast gfp_mask)
+{
+	struct srp_iu *iu;
+	struct ib_sge list;
+	struct ib_recv_wr wr, *bad_wr;
+	unsigned long flags;
+	unsigned int next;
+	int ret;
+
+	iu = srp_alloc_iu(target->srp_host, target->max_ti_iu_len,
+			  gfp_mask, DMA_FROM_DEVICE);
+	if (!iu)
+		return -ENOMEM;
+
+	list.addr   = iu->dma;
+	list.length = iu->size;
+	list.lkey   = target->srp_host->mr->lkey;
+
+	wr.next     = NULL;
+	wr.sg_list  = &list;
+	wr.num_sge  = 1;
+
+	spin_lock_irqsave(&target->lock, flags);
+
+	next = target->rx_head & (SRP_RQ_SIZE - 1);
+	wr.wr_id = next | SRP_OP_RECV;
+	target->rx_ring[next] = iu;
+
+	ret = ib_post_recv(target->qp, &wr, &bad_wr);
+	if (ret)
+		srp_free_iu(target->srp_host, iu);
+	else
+		++target->rx_head;
+
+	spin_unlock_irqrestore(&target->lock, flags);
+
+	return ret;
+}
+
+static int srp_post_send(struct srp_target_port *target,
+			 struct srp_iu *iu, int len)
+{
+	struct ib_sge list;
+	struct ib_send_wr wr, *bad_wr;
+	unsigned long flags;
+	int ret = 0;
+
+	spin_lock_irqsave(&target->lock, flags);
+	if (target->req_lim < 1) {
+		printk(KERN_ERR PFX "Target has req_lim %d\n", target->req_lim);
+		ret = -EAGAIN;
+	}
+
+	--target->req_lim;
+	spin_unlock_irqrestore(&target->lock, flags);
+
+	if (ret)
+		return ret;
+
+	list.addr   = iu->dma;
+	list.length = len;
+	list.lkey   = target->srp_host->mr->lkey;
+
+	wr.next       = NULL;
+	wr.wr_id      = target->tx_head & SRP_SQ_SIZE;
+	wr.sg_list    = &list;
+	wr.num_sge    = 1;
+	wr.opcode     = IB_WR_SEND;
+	wr.send_flags = IB_SEND_SIGNALED;
+
+	target->tx_ring[target->tx_head & SRP_SQ_SIZE] = iu;
+
+	ret = ib_post_send(target->qp, &wr, &bad_wr);
+
+	if (!ret)
+		++target->tx_head;
+
+	return ret;
+}
+
+static int srp_queuecommand(struct scsi_cmnd *scmnd,
+			    void (*done)(struct scsi_cmnd *))
+{
+	struct srp_target_port *target;
+	struct srp_iu *iu;
+	struct srp_cmd *cmd;
+	unsigned long flags;
+	int len;
+
+	read_lock_irqsave(&idr_lock, flags);
+	target = idr_find(&target_idr, scmnd->device->id);
+	read_unlock_irqrestore(&idr_lock, flags);
+
+	if (!target) {
+		printk(KERN_ERR PFX "queuecommand for unknown device id %d\n",
+		       scmnd->device->id);
+		scmnd->result = DID_ERROR << 16;
+		done(scmnd);
+		return 0;
+	}
+
+	if (0) {
+		printk(KERN_ERR PFX "command for %u: ", scmnd->device->id);
+		scsi_print_command(scmnd);
+	}
+
+	iu = srp_alloc_iu(target->srp_host, SRP_MAX_IU_LEN,
+			  GFP_ATOMIC, DMA_TO_DEVICE);
+	if (!iu) {
+		scmnd->result = DID_ERROR << 16;
+		printk(KERN_ERR PFX "couldn't allocate send IU\n");
+		done(scmnd);
+		return 0;
+	}
+
+	scmnd->scsi_done     = done;
+	scmnd->result        = 0;
+	scmnd->host_scribble = (void *) iu;
+
+	cmd = iu->buf;
+	memset(cmd, 0, sizeof *cmd);
+
+	cmd->opcode = SRP_CMD;
+	cmd->lun = cpu_to_be64((u64) scmnd->device->lun << 48);
+	memcpy(cmd->cdb, scmnd->cmnd, scmnd->cmd_len);
+	cmd->tag = (unsigned long) scmnd;
+
+	len = srp_map_data(scmnd, target, iu);
+	if (len < 0) {
+		printk(KERN_ERR PFX "Failed to map data\n");
+		goto err;
+	}
+
+	if (srp_post_recv(target, GFP_ATOMIC)) {
+		printk(KERN_ERR PFX "Recv failed\n");
+		goto err_unmap;
+	}
+
+	if (srp_post_send(target, iu, len)) {
+		printk(KERN_ERR PFX "Send failed\n");
+		goto err_unmap;
+	}
+
+	return 0;
+
+err_unmap:
+	srp_unmap_data(scmnd, target, cmd);
+
+err:
+	srp_free_iu(target->srp_host, iu);
+	return SCSI_MLQUEUE_HOST_BUSY;
+}
+
+static int srp_abort(struct scsi_cmnd *scmnd)
+{
+	printk(KERN_ERR PFX "srp_abort called\n");
+
+        scmnd->result = DID_ABORT << 16;
+	scmnd->scsi_done(scmnd);
+
+	return SUCCESS;
+}
+
+static int srp_reset(struct scsi_cmnd *scmnd)
+{
+	printk(KERN_ERR PFX "srp_reset called\n");
+
+        scmnd->result = DID_ABORT << 16;
+	scmnd->scsi_done(scmnd);
+
+	return SUCCESS;
+}
+
+static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
+{
+	struct srp_target_port *target = cm_id->context;
+	struct ib_qp_attr *qp_attr = NULL;
+	int attr_mask = 0;
+	int comp = 0;
+	int ret = 0;
+
+	switch (event->event) {
+	case IB_CM_REP_RECEIVED:
+		printk(KERN_DEBUG PFX "REP received\n");
+		comp = 1;
+
+		{
+			struct srp_login_rsp *rsp = event->private_data;
+
+			/* XXX check that opcode is SRP RSP */
+
+			target->max_ti_iu_len = be32_to_cpu(rsp->max_ti_iu_len);
+			target->req_lim       = be32_to_cpu(rsp->req_lim_delta);
+
+			target->scsi_host->can_queue = min(target->req_lim,
+							   target->scsi_host->can_queue);
+
+			printk(KERN_DEBUG "  req_lim_delta %d\n", be32_to_cpu(rsp->req_lim_delta));
+			printk(KERN_DEBUG "  max_it_iu_len %d\n", be32_to_cpu(rsp->max_it_iu_len));
+			printk(KERN_DEBUG "  max_ti_iu_len %d\n", be32_to_cpu(rsp->max_ti_iu_len));
+		}
+
+		qp_attr = kmalloc(sizeof *qp_attr, GFP_KERNEL);
+		if (!qp_attr) {
+			target->status = -ENOMEM;
+			break;
+		}
+
+		qp_attr->qp_state = IB_QPS_RTR;
+		target->status = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
+		if (target->status)
+			break;
+
+		qp_attr->rq_psn = 0; /* XXX */
+		attr_mask |= IB_QP_RQ_PSN;
+
+		target->status = ib_modify_qp(target->qp, qp_attr, attr_mask);
+		if (target->status)
+			break;
+
+		target->status = srp_post_recv(target, GFP_KERNEL);
+		if (target->status)
+			break;
+
+		qp_attr->qp_state = IB_QPS_RTS;
+		target->status = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
+		if (target->status)
+			break;
+
+		target->status = ib_modify_qp(target->qp, qp_attr, attr_mask);
+		if (target->status)
+			break;
+
+		target->status = ib_send_cm_rtu(cm_id, NULL, 0);
+		if (target->status)
+			break;
+
+		break;
+
+	case IB_CM_REJ_RECEIVED:
+		printk(KERN_DEBUG PFX "REJ received\n");
+		comp = 1;
+
+		if (event->param.rej_rcvd.reason == IB_CM_REJ_PORT_REDIRECT) {
+			/*
+			 * Additional Reject Info contains
+			 * ClassPortInfo, which has the RedirectGID
+			 * field at an offset of 8 bytes.
+			 */
+			memcpy(target->path.dgid.raw,
+			       event->param.rej_rcvd.ari + 8, 16);
+
+			target->status = SRP_PORT_REDIRECT;
+		} else if (topspin_workarounds &&
+			   !memcmp(&target->ioc_guid, topspin_oui, 3) &&
+			   event->param.rej_rcvd.reason == __constant_htons(25)) {
+			/*
+			 * Topspin/Cisco SRP gateways incorrectly send
+			 * reject reason code 25 when they mean 24
+			 * (port redirect).
+			 */
+			memcpy(target->path.dgid.raw,
+			       event->param.rej_rcvd.ari + 0, 16);
+
+			printk(KERN_DEBUG PFX "Topspin/Cisco redirect to target port GID %016llx%016llx\n",
+			       (unsigned long long) be64_to_cpu(target->path.dgid.global.subnet_prefix),
+			       (unsigned long long) be64_to_cpu(target->path.dgid.global.interface_id));
+
+			target->status = SRP_PORT_REDIRECT;
+		} else {
+			printk(KERN_WARNING "  REJ reason 0x%x\n",
+			       event->param.rej_rcvd.reason);
+			target->status = -ECONNRESET;
+			ret = 1;
+		}
+
+		break;
+
+	case IB_CM_MRA_RECEIVED:
+		printk(KERN_ERR PFX "MRA received\n");
+		break;
+
+	case IB_CM_DREP_RECEIVED:
+		printk(KERN_ERR PFX "DREP received\n");
+		break;
+
+	case IB_CM_TIMEWAIT_EXIT:
+		printk(KERN_ERR PFX "connection closed\n");
+
+		comp = 1;
+		ret  = 1;
+		target->status = 0;
+		break;
+
+	default:
+		printk(KERN_WARNING PFX "Unhandled CM event %d\n", event->event);
+		break;
+	}
+
+	if (comp)
+		complete(&target->done);
+
+	kfree(qp_attr);
+
+	return ret;
+}
+
+static struct ib_qp *srp_create_qp(struct srp_target_port *target,
+				   struct ib_qp_init_attr *init_attr)
+{
+	struct ib_qp *qp;
+	struct ib_qp_attr *attr;
+	int ret;
+
+	attr = kmalloc(sizeof *attr, GFP_KERNEL);
+	if (!attr)
+		return ERR_PTR(-ENOMEM);
+
+	ret = ib_find_cached_pkey(target->srp_host->dev,
+				  target->srp_host->port,
+				  be16_to_cpu(target->path.pkey),
+				  &attr->pkey_index);
+	if (ret) {
+		qp = ERR_PTR(ret);
+		goto out;
+	}
+
+	qp = ib_create_qp(target->srp_host->pd, init_attr);
+	if (IS_ERR(qp))
+		goto out;
+
+	attr->qp_state        = IB_QPS_INIT;
+	attr->qp_access_flags = (IB_ACCESS_REMOTE_READ |
+				    IB_ACCESS_REMOTE_WRITE);
+	attr->port_num        = target->srp_host->port;
+
+	ret = ib_modify_qp(qp, attr,
+			   IB_QP_STATE		|
+			   IB_QP_PKEY_INDEX	|
+			   IB_QP_ACCESS_FLAGS	|
+			   IB_QP_PORT);
+	if (ret) {
+		ib_destroy_qp(qp);
+		qp = ERR_PTR(ret);
+	}
+
+out:
+	kfree(attr);
+	return qp;
+}
+
+static void srp_path_rec_completion(int status,
+				    struct ib_sa_path_rec *pathrec,
+				    void *target_ptr)
+{
+	struct srp_target_port *target = target_ptr;
+	struct srp_host        *host   = target->srp_host;
+	struct ib_qp_init_attr *init_attr = NULL;
+
+	if (status) {
+		printk(KERN_ERR PFX "Got failed path rec status %d\n", status);
+		target->status = status;
+		goto out;
+	}
+
+	target->path = *pathrec;
+
+	/*
+	 * We may be getting a path for the second time because we
+	 * were redirected to a different port.  In that case, there's
+	 * no reason to create our CQ and QP again.
+	 */
+	if (target->cq) {
+		target->status = 0;
+		goto out;
+	}
+
+	init_attr = kmalloc(sizeof *init_attr, GFP_KERNEL);
+	if (!init_attr) {
+		target->status = -ENOMEM;
+		goto out;
+	}
+
+	memset(init_attr, 0, sizeof *init_attr);
+	target->cq = ib_create_cq(host->dev, srp_completion,
+				  NULL, target, SRP_CQ_SIZE);
+	if (IS_ERR(target->cq)) {
+		target->status = PTR_ERR(target->cq);
+		goto out_free;
+	}
+
+	ib_req_notify_cq(target->cq, IB_CQ_NEXT_COMP);
+
+	init_attr->event_handler       = srp_qp_event;
+	init_attr->cap.max_send_wr     = SRP_SQ_SIZE;
+	init_attr->cap.max_recv_wr     = SRP_RQ_SIZE;
+	init_attr->cap.max_recv_sge    = 1;
+	init_attr->cap.max_send_sge    = 1;
+	init_attr->sq_sig_type         = IB_SIGNAL_ALL_WR;
+	init_attr->qp_type             = IB_QPT_RC;
+	init_attr->send_cq             = target->cq;
+	init_attr->recv_cq             = target->cq;
+
+	target->qp = srp_create_qp(target, init_attr);
+	if (IS_ERR(target->qp)) {
+		target->status = PTR_ERR(target->qp);
+		ib_destroy_cq(target->cq);
+		goto out_free;
+	}
+
+	target->status = 0;
+
+out_free:
+	kfree(init_attr);
+
+out:
+	complete(&target->done);
+}
+
+static int srp_send_req(struct srp_target_port *target)
+{
+	struct {
+		struct ib_cm_req_param param;
+		struct srp_login_req   priv;
+	} *req = NULL;
+	int status;
+
+	req = kmalloc(sizeof *req, GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	memset(req, 0, sizeof *req);
+	req->param.primary_path 	      = &target->path;
+	req->param.alternate_path 	      = NULL;
+	req->param.service_id 		      = target->service_id;
+	req->param.qp_num 		      = target->qp->qp_num;
+	req->param.qp_type 		      = target->qp->qp_type;
+	req->param.starting_psn 	      = 0; /* XXX */
+	req->param.private_data 	      = &req->priv;
+	req->param.private_data_len 	      = sizeof req->priv;
+	req->param.responder_resources	      = 4;
+	req->param.remote_cm_response_timeout = 20;
+	req->param.flow_control 	      = 1;
+	req->param.local_cm_response_timeout  = 20;
+	req->param.retry_count 		      = 7;
+	req->param.rnr_retry_count 	      = 7;
+	req->param.max_cm_retries 	      = 15;
+
+	req->priv.opcode     	= SRP_LOGIN_REQ;
+	req->priv.tag        	= 0;
+	req->priv.req_it_iu_len = cpu_to_be32(SRP_MAX_IU_LEN);
+	req->priv.req_buf_fmt 	= cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
+					      SRP_BUF_FORMAT_INDIRECT);
+	memcpy(req->priv.initiator_port_id, target->srp_host->initiator_port_id, 16);
+	/*
+	 * Topspin/Cisco SRP targets will reject our login unless we
+	 * zero out the first 8 bytes of our initiator port ID.  The
+	 * second 8 bytes must be our local node GUID, but we always
+	 * use that anyway.
+	 */
+	if (topspin_workarounds && !memcmp(&target->ioc_guid, topspin_oui, 3)) {
+		printk(KERN_DEBUG PFX "Topspin/Cisco initiator port ID workaround "
+		       "activated for target GUID %016llx\n",
+		       (unsigned long long) be64_to_cpu(target->ioc_guid));
+		memset(req->priv.initiator_port_id, 0, 8);
+	}
+	memcpy(req->priv.target_port_id,     &target->id_ext, 8);
+	memcpy(req->priv.target_port_id + 8, &target->ioc_guid, 8);
+
+	status = ib_send_cm_req(target->cm_id, &req->param);
+	if (status) {
+		ib_destroy_qp(target->qp);
+		ib_destroy_cq(target->cq);
+	}
+
+	return status;
+}
+
+static void srp_release_target(struct srp_target_port *target)
+{
+	unsigned long flags;
+
+	/* XXX should send SRP_I_LOGOUT request */
+
+	init_completion(&target->done);
+	ib_send_cm_dreq(target->cm_id, NULL, 0);
+	wait_for_completion(&target->done);
+
+	ib_destroy_qp(target->qp);
+	ib_destroy_cq(target->cq);
+
+	write_lock_irqsave(&idr_lock, flags);
+	idr_remove(&target_idr, target->scsi_id);
+	write_unlock_irqrestore(&idr_lock, flags);
+}
+
+static struct scsi_host_template srp_template = {
+	.module 	       = THIS_MODULE,
+	.name   	       = DRV_NAME,
+	.info   	       = srp_target_info,
+	.queuecommand 	       = srp_queuecommand,
+        .eh_abort_handler      = srp_abort,
+        .eh_bus_reset_handler  = srp_reset,
+        .eh_host_reset_handler = srp_reset,
+	.can_queue 	       = SRP_SQ_SIZE,
+	.this_id 	       = -1,
+	.sg_tablesize 	       = SRP_MAX_INDIRECT,
+	.cmd_per_lun 	       = SRP_SQ_SIZE,
+	.use_clustering        = ENABLE_CLUSTERING
+};
+
+static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
+{
+	unsigned long flags;
+	int ret;
+
+	do {
+		if (!idr_pre_get(&target_idr, GFP_KERNEL))
+			return -ENOMEM;
+
+		write_lock_irqsave(&idr_lock, flags);
+		ret = idr_get_new(&target_idr, target, &target->scsi_id);
+		write_unlock_irqrestore(&idr_lock, flags);
+	} while (ret == -EAGAIN);
+
+	if (ret)
+		goto fail;
+
+	sprintf(target->target_name, "SRP.T10:%016llX",
+		 (unsigned long long) be64_to_cpu(target->id_ext));
+	target->scsi_host->unique_id = atomic_inc_return(&srp_uid);
+
+	if (scsi_add_host(target->scsi_host, host->dev->dma_device))
+		goto fail;
+
+	down(&host->target_mutex);
+	list_add_tail(&target->list, &host->target_list);
+	up(&host->target_mutex);
+
+	/* XXX: are we supposed to have a definition of SCAN_WILD_CARD ?? */
+	scsi_scan_target(&target->scsi_host->shost_gendev,
+			 0, target->scsi_id, ~0, 0);
+
+	return 0;
+
+fail:
+	write_lock_irqsave(&idr_lock, flags);
+	idr_remove(&target_idr, target->scsi_id);
+	write_unlock_irqrestore(&idr_lock, flags);
+
+	return ret;
+}
+
+static void srp_release_class_dev(struct class_device *class_dev)
+{
+	struct srp_host *host =
+		container_of(class_dev, struct srp_host, class_dev);
+
+	kfree(host);
+}
+
+static struct class srp_class = {
+	.name    = "infiniband_srp",
+	.release = srp_release_class_dev
+};
+
+/*
+ * Target ports are added by writing
+ *
+ *     id_ext=<SRP ID ext>,ioc_guid=<SRP IOC GUID>,dgid=<dest GID>,
+ *     pkey=<P_Key>,service_id=<service ID>
+ *
+ * to the add_target sysfs attribute.
+ */
+enum {
+	SRP_OPT_ERR		= 0,
+	SRP_OPT_ID_EXT		= 1 << 0,
+	SRP_OPT_IOC_GUID	= 1 << 1,
+	SRP_OPT_DGID		= 1 << 2,
+	SRP_OPT_PKEY		= 1 << 3,
+	SRP_OPT_SERVICE_ID	= 1 << 4,
+	SRP_OPT_ALL		= (SRP_OPT_ID_EXT	|
+				   SRP_OPT_IOC_GUID	|
+				   SRP_OPT_DGID		|
+				   SRP_OPT_PKEY		|
+				   SRP_OPT_SERVICE_ID),
+};
+
+static match_table_t srp_opt_tokens = {
+	{ SRP_OPT_ID_EXT,	"id_ext=%s" },
+	{ SRP_OPT_IOC_GUID,	"ioc_guid=%s" },
+	{ SRP_OPT_DGID,		"dgid=%s" },
+	{ SRP_OPT_PKEY,		"pkey=%x" },
+	{ SRP_OPT_SERVICE_ID,	"service_id=%s" },
+	{ SRP_OPT_ERR,		NULL }
+};
+
+static int srp_parse_options(const char *buf, struct srp_target_port *target)
+{
+	char *options;
+	char *p;
+	char dgid[3];
+	substring_t args[MAX_OPT_ARGS];
+	int opt_mask = 0;
+	int token;
+	int ret = -EINVAL;
+	int i;
+
+/* XXX Remove when 2.6.13 is released with kstrdup() */
+#if LINUX_VERSION_CODE <= KERNEL_VERSION(2,6,12)
+	{
+		int len = strlen(buf) + 1;
+		options = kmalloc(len, GFP_KERNEL);
+		if (!options)
+			return -ENOMEM;
+		memcpy(options, buf, len);
+	}
+#else
+	options = kstrdup(buf, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+#endif
+
+	while ((p = strsep(&options, ",")) != NULL) {
+		if (!*p)
+			continue;
+
+		token = match_token(p, srp_opt_tokens, args);
+		opt_mask |= token;
+
+		switch (token) {
+		case SRP_OPT_ID_EXT:
+			p = match_strdup(args);
+			target->id_ext = cpu_to_be64(simple_strtoull(p, NULL, 16));
+			kfree(p);
+			break;
+
+		case SRP_OPT_IOC_GUID:
+			p = match_strdup(args);
+			target->ioc_guid = cpu_to_be64(simple_strtoull(p, NULL, 16));
+			kfree(p);
+			break;
+
+		case SRP_OPT_DGID:
+			p = match_strdup(args);
+			if (strlen(p) != 32)
+				goto out;
+
+			for (i = 0; i < 16; ++i) {
+				strlcpy(dgid, p + i * 2, 3);
+				target->path.dgid.raw[i] = simple_strtoul(dgid, NULL, 16);
+			}
+			break;
+
+		case SRP_OPT_PKEY:
+			if (match_hex(args, &token))
+				goto out;
+			target->path.pkey = cpu_to_be16(token);
+			break;
+
+		case SRP_OPT_SERVICE_ID:
+			p = match_strdup(args);
+			target->service_id = cpu_to_be64(simple_strtoull(p, NULL, 16));
+			kfree(p);
+			break;
+
+		default:
+			goto out;
+		}
+	}
+
+	if (opt_mask == SRP_OPT_ALL)
+		ret = 0;
+
+out:
+	kfree(options);
+	return ret;
+}
+
+static ssize_t srp_create_target(struct class_device *class_dev,
+				 const char *buf, size_t count)
+{
+	struct srp_host *host =
+		container_of(class_dev, struct srp_host, class_dev);
+	struct Scsi_Host *target_host;
+	struct srp_target_port *target;
+	int ret;
+
+	target_host = scsi_host_alloc(&srp_template,
+				      sizeof (struct srp_target_port));
+	if (!target_host)
+		return -ENOMEM;
+
+	target = host_to_target(target_host);
+	memset(target, 0, sizeof *target);
+
+	ret = srp_parse_options(buf, target);
+	if (ret) {
+		scsi_host_put(target_host);
+		return ret;
+	}
+
+	spin_lock_init(&target->lock);
+	target->cm_id = ib_create_cm_id(srp_cm_handler, target);
+	if (IS_ERR(target->cm_id)) {
+		scsi_host_put(target_host);
+		return -ENOMEM;
+	}
+
+	ib_get_cached_gid(host->dev, host->port, 0, &target->path.sgid);
+
+	target->scsi_host  = target_host;
+	target->srp_host   = host;
+
+	printk(KERN_DEBUG PFX "new target: id_ext %016llx ioc_guid %016llx pkey %04x "
+	       "service_id %016llx dgid %04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x\n",
+	       (unsigned long long) be64_to_cpu(target->id_ext),
+	       (unsigned long long) be64_to_cpu(target->ioc_guid),
+	       be16_to_cpu(target->path.pkey),
+	       (unsigned long long) be64_to_cpu(target->service_id),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[0]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[2]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[4]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[6]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[8]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[10]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[12]),
+	       (int) be16_to_cpu(*(__be16 *) &target->path.dgid.raw[14]));
+
+retry_path:
+	target->path.numb_path = 1;
+
+	init_completion(&target->done);
+
+	target->path_query_id = ib_sa_path_rec_get(host->dev, host->port,
+						   &target->path,
+						   IB_SA_PATH_REC_DGID		|
+						   IB_SA_PATH_REC_SGID		|
+						   IB_SA_PATH_REC_NUMB_PATH	|
+						   IB_SA_PATH_REC_PKEY,
+						   1000, GFP_KERNEL,
+						   srp_path_rec_completion,
+						   target, &target->path_query);
+	if (target->path_query_id < 0) {
+		ret = target->path_query_id;
+		ib_destroy_cm_id(target->cm_id);
+		scsi_host_put(target_host);
+		goto out;
+	}
+
+	wait_for_completion(&target->done);
+
+	if (target->status < 0) {
+		printk(KERN_WARNING PFX "Path record query failed\n");
+		ret = target->status;
+		ib_destroy_cm_id(target->cm_id);
+		scsi_host_put(target_host);
+		goto out;
+	}
+
+	init_completion(&target->done);
+	ret = srp_send_req(target);
+	if (ret) {
+		scsi_host_put(target_host);
+		goto out;
+	}
+	wait_for_completion(&target->done);
+
+	/*
+	 * The CM event handling code will set status to
+	 * SRP_PORT_REDIRECT if we get a port redirect REJ back.
+	 */
+	if (target->status == SRP_PORT_REDIRECT)
+		goto retry_path;
+	else if (target->status < 0) {
+		printk(KERN_ERR PFX "Connection failed\n");
+		ret = target->status;
+		scsi_host_put(target_host);
+		goto out;
+	}
+
+	ret = srp_add_target(host, target);
+
+	if (ret)
+		scsi_host_put(target_host);
+
+out:
+	return ret ? ret : count;
+}
+
+static CLASS_DEVICE_ATTR(add_target, S_IWUSR, NULL, srp_create_target);
+
+static struct srp_host *srp_add_port(struct ib_device *device,
+				     u64 node_guid, u8 port)
+{
+	struct srp_host *host;
+
+	host = kmalloc(sizeof *host, GFP_KERNEL);
+	if (!host)
+		return NULL;
+
+	memset(host, 0, sizeof *host);
+
+	INIT_LIST_HEAD(&host->target_list);
+	init_MUTEX(&host->target_mutex);
+	host->dev  = device;
+	host->port = port;
+
+	host->initiator_port_id[7] = port;
+	memcpy(host->initiator_port_id + 8, &node_guid, 8);
+
+	host->pd   = ib_alloc_pd(device);
+	if (IS_ERR(host->pd))
+		goto err_free;
+
+	host->mr   = ib_get_dma_mr(host->pd,
+				   IB_ACCESS_LOCAL_WRITE |
+				   IB_ACCESS_REMOTE_READ |
+				   IB_ACCESS_REMOTE_WRITE);
+	if (IS_ERR(host->mr))
+		goto err_pd;
+
+	host->class_dev.class = &srp_class;
+	host->class_dev.dev   = device->dma_device;
+	snprintf(host->class_dev.class_id, BUS_ID_SIZE, "srp-%s-%d",
+		 device->name, port);
+
+	if (class_device_register(&host->class_dev))
+		goto err_mr;
+	if (class_device_create_file(&host->class_dev, &class_device_attr_add_target))
+		goto err_class;
+	/* XXX ibdev / port files as well */
+
+	/*
+	 * Take another reference so we can unregister and then free
+	 * IB resources afterwards.
+	 */
+	class_device_get(&host->class_dev);
+
+	return host;
+
+err_class:
+	class_device_unregister(&host->class_dev);
+
+err_mr:
+	ib_dereg_mr(host->mr);
+
+err_pd:
+	ib_dealloc_pd(host->pd);
+
+err_free:
+	kfree(host);
+
+	return NULL;
+}
+
+static void srp_add_one(struct ib_device *device)
+{
+	struct list_head *dev_list;
+	struct srp_host *host;
+	struct ib_device_attr *dev_attr;
+	int s, e, p;
+
+	dev_attr = kmalloc(sizeof *dev_attr, GFP_KERNEL);
+	if (!dev_attr)
+		return;
+
+	if (ib_query_device(device, dev_attr)) {
+		printk(KERN_WARNING PFX "Couldn't query node GUID for %s.\n",
+		       device->name);
+		goto out;
+	}
+
+	dev_list = kmalloc(sizeof *dev_list, GFP_KERNEL);
+	if (!dev_list)
+		goto out;
+
+	INIT_LIST_HEAD(dev_list);
+
+	if (device->node_type == IB_NODE_SWITCH) {
+		s = 0;
+		e = 0;
+	} else {
+		s = 1;
+		e = device->phys_port_cnt;
+	}
+
+	for (p = s; p <= e; ++p) {
+		host = srp_add_port(device, dev_attr->node_guid, p);
+		if (host)
+			list_add_tail(&host->list, dev_list);
+	}
+
+	ib_set_client_data(device, &srp_client, dev_list);
+
+out:
+	kfree(dev_attr);
+}
+
+static void srp_remove_one(struct ib_device *device)
+{
+	struct list_head *dev_list;
+	struct srp_host *host, *tmp_host;
+	struct srp_target_port *target, *tmp_target;
+
+	dev_list = ib_get_client_data(device, &srp_client);
+
+	list_for_each_entry_safe(host, tmp_host, dev_list, list) {
+		down(&host->target_mutex);
+
+		list_for_each_entry_safe(target, tmp_target,
+					 &host->target_list, list) {
+			scsi_remove_host(target->scsi_host);
+			srp_release_target(target);
+			scsi_host_put(target->scsi_host);
+		}
+
+		up(&host->target_mutex);
+
+		class_device_unregister(&host->class_dev);
+		ib_dereg_mr(host->mr);
+		ib_dealloc_pd(host->pd);
+		class_device_put(&host->class_dev);
+	}
+}
+
+static int __init srp_init_module(void)
+{
+	int ret;
+
+	atomic_set(&srp_uid, 0);
+	rwlock_init(&idr_lock);
+
+	ret = class_register(&srp_class);
+	if (ret) {
+		printk(KERN_ERR PFX "couldn't register class infiniband_srp\n");
+		return ret;
+	}
+
+	ret = ib_register_client(&srp_client);
+	if (ret) {
+		printk(KERN_ERR PFX "couldn't register IB client\n");
+		class_unregister(&srp_class);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void __exit srp_cleanup_module(void)
+{
+	ib_unregister_client(&srp_client);
+	class_unregister(&srp_class);
+}
+
+module_init(srp_init_module);
+module_exit(srp_cleanup_module);
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ linux-export/drivers/infiniband/ulp/srp/ib_srp.h	2005-07-15 13:02:00.288272334 -0700
@@ -0,0 +1,268 @@
+/*
+ * Copyright (c) 2005 Cisco Systems.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * $Id: ib_srp.h 2861 2005-07-14 17:05:01Z roland $
+ */
+
+#ifndef IB_SRP_H
+#define IB_SRP_H
+
+#include <linux/types.h>
+#include <linux/list.h>
+
+#include <asm/semaphore.h>
+
+#include <scsi/scsi_host.h>
+#include <scsi/scsi_cmnd.h>
+
+#include <ib_verbs.h>
+#include <ib_sa.h>
+#include <ib_cm.h>
+
+enum {
+	SRP_PORT_REDIRECT	= 1,
+
+	SRP_MAX_IU_LEN		= 256,
+
+	SRP_RQ_SHIFT    	= 6,
+	SRP_RQ_SIZE		= 1 << SRP_RQ_SHIFT,
+	SRP_SQ_SIZE		= SRP_RQ_SIZE - 1,
+	SRP_CQ_SIZE		= SRP_SQ_SIZE + SRP_RQ_SIZE
+};
+
+#define SRP_OP_RECV		(1 << 31)
+#define SRP_MAX_INDIRECT	((SRP_MAX_IU_LEN -			\
+				  sizeof (struct srp_cmd) -		\
+				  sizeof (struct srp_indirect_buf)) / 16)
+
+struct srp_host {
+	u8			initiator_port_id[16];
+	struct ib_device       *dev;
+	u8                      port;
+	struct ib_pd	       *pd;
+	struct ib_mr	       *mr;
+	struct class_device	class_dev;
+	struct list_head	target_list;
+	struct semaphore        target_mutex;
+	struct list_head	list;
+};
+
+struct srp_target_port {
+	spinlock_t		lock;
+
+	__be64			id_ext;
+	__be64			ioc_guid;
+	__be64			service_id;
+	struct srp_host	       *srp_host;
+	struct Scsi_Host       *scsi_host;
+	char			target_name[32];
+	unsigned int		scsi_id;
+
+	struct ib_sa_path_rec	path;
+	struct ib_sa_query     *path_query;
+	int			path_query_id;
+
+	struct ib_cm_id	       *cm_id;
+	struct ib_cq	       *cq;
+	struct ib_qp	       *qp;
+
+	int			max_ti_iu_len;
+	s32			req_lim;
+
+	unsigned		rx_head;
+	unsigned		rx_tail;
+	struct srp_iu	       *rx_ring[SRP_RQ_SIZE];
+
+	unsigned		tx_head;
+	unsigned		tx_tail;
+	struct srp_iu	       *tx_ring[SRP_SQ_SIZE + 1];
+
+	struct list_head	list;
+	struct completion	done;
+	int			status;
+};
+
+struct srp_iu {
+	dma_addr_t		dma;
+	void		       *buf;
+	size_t			size;
+	enum dma_data_direction	direction;
+};
+
+/*
+ * SRP protocol definitions
+ */
+
+enum {
+	SRP_LOGIN_REQ	= 0x00,
+	SRP_TSK_MGMT	= 0x01,
+	SRP_CMD		= 0x02,
+	SRP_I_LOGOUT	= 0x03,
+	SRP_LOGIN_RSP	= 0xc0,
+	SRP_RSP		= 0xc1,
+	SRP_LOGIN_REJ	= 0xc2,
+	SRP_T_LOGOUT	= 0x80,
+	SRP_CRED_REQ	= 0x81,
+	SRP_AER_REQ	= 0x82,
+	SRP_CRED_RSP	= 0x41,
+	SRP_AER_RSP	= 0x42
+};
+
+enum {
+	SRP_BUF_FORMAT_DIRECT   = 1 << 1,
+	SRP_BUF_FORMAT_INDIRECT = 1 << 2
+};
+
+enum {
+	SRP_NO_DATA_DESC       = 0,
+	SRP_DATA_DESC_DIRECT   = 1,
+	SRP_DATA_DESC_INDIRECT = 2
+};
+
+struct srp_direct_buf {
+	__be64	va;
+	__be32	key;
+	__be32  len;
+};
+
+/*
+ * We need the packed attribute because the SRP spec puts the list of
+ * descriptors at an offset of 20, which is not aligned to the size
+ * of struct srp_direct_buf.
+ */
+struct srp_indirect_buf {
+	struct srp_direct_buf	table_desc;
+	__be32			len;
+	struct srp_direct_buf	desc_list[0] __attribute__((packed));
+};
+
+enum {
+	SRP_MULTICHAN_SINGLE = 0,
+	SRP_MULTICHAN_MULTI  = 1
+};
+
+struct srp_login_req {
+	u8	opcode;
+	u8	reserved1[7];
+	u64	tag;
+	__be32	req_it_iu_len;
+	u8	reserved2[4];
+	__be16	req_buf_fmt;
+	u8	req_flags;
+	u8	reserved3[5];
+	u8	initiator_port_id[16];
+	u8	target_port_id[16];
+};
+
+struct srp_login_rsp {
+	u8	opcode;
+	u8	reserved1[3];
+	__be32	req_lim_delta;
+	u64	tag;
+	__be32	max_it_iu_len;
+	__be32	max_ti_iu_len;
+	__be16	buf_fmt;
+	u8	rsp_flags;
+	u8	reserved2[25];
+};
+
+struct srp_login_rej {
+	u8	opcode;
+	u8	reserved1[3];
+	__be32	reason;
+	u64	tag;
+	u8	reserved2[8];
+	__be16	buf_fmt;
+	u8	reserved3[6];
+};
+
+struct srp_i_logout {
+	u8	opcode;
+	u8	reserved[7];
+	u64	tag;
+};
+
+struct srp_t_logout {
+	u8	opcode;
+	u8	sol_not;
+	u8	reserved[2];
+	__be32	reason;
+	u64	tag;
+};
+
+/*
+ * We need the packed attribute because the SRP spec only aligns the
+ * 8-byte LUN field to 4 bytes.
+ */
+struct srp_cmd {
+	u8	opcode;
+	u8	sol_not;
+	u8	reserved1[3];
+	u8	buf_fmt;
+	u8	data_out_desc_cnt;
+	u8	data_in_desc_cnt;
+	u64	tag;
+	u8	reserved2[4];
+	__be64	lun __attribute__((packed));
+	u8	reserved3;
+	u8	task_attr;
+	u8	reserved4;
+	u8	add_cdb_len;
+	u8	cdb[16];
+	u8	add_data[0];
+};
+
+enum {
+	SRP_RSP_FLAG_RSPVALID = 1 << 0,
+	SRP_RSP_FLAG_SNSVALID = 1 << 1,
+	SRP_RSP_FLAG_DOOVER   = 1 << 2,
+	SRP_RSP_FLAG_DOUNDER  = 1 << 3,
+	SRP_RSP_FLAG_DIOVER   = 1 << 4,
+	SRP_RSP_FLAG_DIUNDER  = 1 << 5
+};
+
+struct srp_rsp {
+	u8	opcode;
+	u8	sol_not;
+	u8	reserved1[2];
+	__be32	req_lim_delta;
+	u64	tag;
+	u8	reserved2[2];
+	u8	flags;
+	u8	status;
+	__be32	data_out_res_cnt;
+	__be32	data_in_res_cnt;
+	__be32	sense_data_len;
+	__be32	resp_data_len;
+	u8	data[0];
+};
+
+#endif /* IB_SRP_H */


From ardavis at ichips.intel.com  Fri Jul 15 13:26:11 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Fri, 15 Jul 2005 13:26:11 -0700
Subject: [openib-general] uVerbs and multiple cq_event support? 
Message-ID: <42D81BE3.3020200@ichips.intel.com>

Hi Roland,

I have a question about uverbs cq_events. Currently, uverbs has one 
event (fd) handle that processes all CQ events. You mentioned earlier  
that you would possibly add support for more CQ events. Are you thinking 
one per CQ, or limiting to a small number like one per processor? If the 
thinking is NOT one per CQ then it may be somewhat restricitive to user 
applications that have multple connections with many CQs to process.

For instance, with uDAPL there are user wait objects (CQ_WAIT_OBJECT) 
that can be mapped directly to these CQ events. This enables the user 
thread to wait and wake up directly from the kernel CQ event processing 
which is very efficicent. However, this can only be done if verbs 
provides an event mechanism per CQ. Otherwise, I have to process the 
events in a separate thread and in turn process the user's event 
independent of the CQ event.

If the thinking is one per CQ then is fd's the way to go since we can 
have 1000's of CQs?

Comments?

Thanks,

-arlin


From rolandd at cisco.com  Fri Jul 15 13:35:39 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 13:35:39 -0700
Subject: [openib-general] Re: uVerbs and multiple cq_event support?
In-Reply-To: <42D81BE3.3020200@ichips.intel.com> (Arlin Davis's message of
	"Fri, 15 Jul 2005 13:26:11 -0700")
References: <42D81BE3.3020200@ichips.intel.com>
Message-ID: <5264vb7qp0.fsf@topspin.com>

    Arlin> Hi Roland, I have a question about uverbs
    Arlin> cq_events. Currently, uverbs has one event (fd) handle that
    Arlin> processes all CQ events. You mentioned earlier that you
    Arlin> would possibly add support for more CQ events. Are you
    Arlin> thinking one per CQ, or limiting to a small number like one
    Arlin> per processor? If the thinking is NOT one per CQ then it
    Arlin> may be somewhat restricitive to user applications that have
    Arlin> multple connections with many CQs to process.

My thinking was a small number, to match the 1.2 IBA spec's idea of
having a fixed number of CQ event handlers available and also to match
the capabilities of MSI-X interrupts.  However, the idea of being able
to dynamically create an arbitrary number of CQ event FDs is
interesting.  I need to think about how feasible this is.

    Arlin> If the thinking is one per CQ then is fd's the way to go
    Arlin> since we can have 1000's of CQs?

I don't know of a different way to implement this...

 - R.


From rolandd at cisco.com  Fri Jul 15 13:48:30 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 13:48:30 -0700
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <1121452298.15851.25.camel@duffman> (Tom Duffy's message of
	"Fri, 15 Jul 2005 11:31:38 -0700")
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
	<1121365546.17302.3.camel@duffman> <5264vc7ywv.fsf@topspin.com>
	<1121452298.15851.25.camel@duffman>
Message-ID: <521x5z7q3l.fsf@topspin.com>

    Tom> It seems to be happening at line 479 of arping.c:

    Tom> 	memset(he.sll_addr, -1, he.sll_halen);

    Tom> where he.sll_addr is 8 big, he.sll_halen is 20

    Tom> So, yeah.

    Tom> Should struct sockaddr_ll be modified in
    Tom> include/linux/if_packet.h to accommodate larger addresses?

I'm not sure what the right solution is yet.  Long term I guess
sockaddr_ll should grow.  I think that right now the address is being
cut off -- so arping.c can probably be fixed by replacing all uses of
foo.sll_halen with min(foo.sll_halen, sizeof foo.sll_addr).

 - R.


From tduffy at sun.com  Fri Jul 15 14:10:31 2005
From: tduffy at sun.com (Tom Duffy)
Date: Fri, 15 Jul 2005 14:10:31 -0700
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <521x5z7q3l.fsf@topspin.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
	<1121365546.17302.3.camel@duffman> <5264vc7ywv.fsf@topspin.com>
	<1121452298.15851.25.camel@duffman>  <521x5z7q3l.fsf@topspin.com>
Message-ID: <1121461831.15851.31.camel@duffman>

On Fri, 2005-07-15 at 13:48 -0700, Roland Dreier wrote:
> I'm not sure what the right solution is yet.  Long term I guess
> sockaddr_ll should grow.  I think that right now the address is being
> cut off -- so arping.c can probably be fixed by replacing all uses of
> foo.sll_halen with min(foo.sll_halen, sizeof foo.sll_addr).

Filed bug and submitted patch (for Fedora anyways):

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=163383

-tduffy

diff -urp iputils/arping.c iputils.tom/arping.c
--- iputils/arping.c	2005-07-15 13:51:15.533632784 -0700
+++ iputils.tom/arping.c	2005-07-15 13:50:47.967823000 -0700
@@ -59,6 +59,8 @@ int received = 0, brd_recv, req_recv;
 #define MS_TDIFF(tv1,tv2) ( ((tv1).tv_sec-(tv2).tv_sec)*1000 + \
 			   ((tv1).tv_usec-(tv2).tv_usec)/1000 )
 
+#define min(x,y) ((x)<(y) ? (x) : (y))
+
 void usage(void)
 {
 	fprintf(stderr,
@@ -476,7 +478,7 @@ main(int argc, char **argv)
 	}
 
 	he = me;
-	memset(he.sll_addr, -1, he.sll_halen);
+	memset(he.sll_addr, -1, min(he.sll_halen, sizeof he.sll_addr));
 
 	if (!quiet) {
 		printf("ARPING %s ", inet_ntoa(dst));



From delaitt at cpc.wmin.ac.uk  Fri Jul 15 14:45:43 2005
From: delaitt at cpc.wmin.ac.uk (Thierry Delaitre)
Date: Fri, 15 Jul 2005 22:45:43 +0100 (BST)
Subject: [openib-general] Firmware needed for cougar MT23108 A1
In-Reply-To: <52wtns84ww.fsf@topspin.com>
References: <Pine.GSO.4.58.0507151534381.2014@seth.cpc.wmin.ac.uk>
	<52wtns84ww.fsf@topspin.com>
Message-ID: <Pine.GSO.4.58.0507152239340.21876@seth.cpc.wmin.ac.uk>


On Fri, 15 Jul 2005, Roland Dreier wrote:

>     Thierry> I'm trying to use the openib trunk with a cougar MT23108
>     Thierry> A1. However, the module complains that i have firmware
>     Thierry> 3.2.0 instead of 3.3.2. Where can i find this firmware ?
>     Thierry> My Ib provider has got only 3.2 firmware.
>
> If you aren't experiencing any problems, then you may want to stick to
> 3.2.0 firmware.  The main new feature of 3.3.2 is support for MSI-X.
> However, Shirley Ma reported having problems with 3.3.2 that went away
> when she downgraded to 3.2.0.

3.2 is fine with me but the openib driver was complaining that i need to
upgrade to 3.3 and i believe the driver is not happy about this. I've not
used openib before as i'm using the topspin's ib kernel modules.

> If you really want to upgrade, then you can bug your HCA vendor for an
> update or try downloading firmware from the Mellanox web site and
> then preparing your own firmware images.

I've seen the firmware on the mellanox web site but the format of the
files do seem different than the topspin ones. I've seen on this mailing
list that some people have mentioned mellanox tools for converting images
but as i'm using topspin, it will be tricky to convert the file in the
right format, i believe.

Thierry.


From robert.j.woodruff at intel.com  Fri Jul 15 14:51:08 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Fri, 15 Jul 2005 14:51:08 -0700
Subject: [openib-general] Re: uVerbs and multiple cq_event support?
Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004F34A89@orsmsx408>


Roland>My thinking was a small number, to match the 1.2 IBA spec's idea
of
Roland >having a fixed number of CQ event handlers available and also to
match
Roland>the capabilities of MSI-X interrupts.  However, the idea of being
able
Roland>to dynamically create an arbitrary number of CQ event FDs is
Roland>interesting.  I need to think about how feasible this is.

    Arlin> If the thinking is one per CQ then is fd's the way to go
    Arlin> since we can have 1000's of CQs?

>I don't know of a different way to implement this...

>- R.

I am not sure that using an fd per CQ is the right way to go. I don't
think the kernel was really designed to have ten's of thousands
of fds for one devices. 

I think in gen1 we had the wait events done through ioctls, so that
we did not have to have multiple fds. Sean probably can explain this
method
in more detail.

woody

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general


From halr at voltaire.com  Fri Jul 15 14:48:14 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 15 Jul 2005 17:48:14 -0400
Subject: [openib-general] 2.6.12 Kernel OpenIB support
In-Reply-To: <521x5z7q3l.fsf@topspin.com>
References: <cd9967e2050710172345f01051@mail.gmail.com>
	<78d18e2050711083826cffc63@mail.gmail.com>
	<cd9967e20507122255444ca465@mail.gmail.com>
	<78d18e2050713072155462f61@mail.gmail.com>
	<1121265275.4401.321.camel@hal.voltaire.com>
	<78d18e2050713123235f668bc@mail.gmail.com>
	<1121365546.17302.3.camel@duffman> <5264vc7ywv.fsf@topspin.com>
	<1121452298.15851.25.camel@duffman>  <521x5z7q3l.fsf@topspin.com>
Message-ID: <1121464094.4423.2673.camel@hal.voltaire.com>

On Fri, 2005-07-15 at 16:48, Roland Dreier wrote:
> I'm not sure what the right solution is yet.  Long term I guess
> sockaddr_ll should grow. 

Is IB the first link that needs to expand this structure ? Isn't that
the tip of a larger set of kernel changes ?

-- Hal



From sean.hefty at intel.com  Fri Jul 15 15:22:34 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 15 Jul 2005 15:22:34 -0700
Subject: [openib-general] [PATCH] [RMPP] Add handling for ABORT/STOP
Message-ID: <ORSMSX401BBsQwY6YHk00000057@orsmsx401.amr.corp.intel.com>

The following patch provides the initial support for transmitting
and handling RMPP ABORT and STOP packets.  Most of the handling is
in place and was tested by jumping directly into the various error
paths.  These items are still open:

* Abort receive processing.  An aborted receive will currently
  just hang around until timing out.
* Handle status code 119 - Inconsistent last and payload length.
  The code currently ignores the payload length and relies solely
  on the last flag to indicate the end of a transfer.
* Handle status code 126 - Too many retries.  An aborted send will
  give up after a specified number of retries.  The receive will
  hang around on the remote node until timing out.

- Sean

Index: include/ib_mad.h
===================================================================
--- include/ib_mad.h	(revision 2867)
+++ include/ib_mad.h	(working copy)
@@ -90,6 +90,7 @@
 
 #define	IB_MGMT_RMPP_STATUS_SUCCESS		0
 #define	IB_MGMT_RMPP_STATUS_RESX		1
+#define	IB_MGMT_RMPP_STATUS_ABORT_MIN		118
 #define	IB_MGMT_RMPP_STATUS_T2L			118
 #define	IB_MGMT_RMPP_STATUS_BAD_LEN		119
 #define	IB_MGMT_RMPP_STATUS_BAD_SEG		120
@@ -100,6 +101,7 @@
 #define	IB_MGMT_RMPP_STATUS_UNV			125
 #define	IB_MGMT_RMPP_STATUS_TMR			126
 #define	IB_MGMT_RMPP_STATUS_UNSPEC		127
+#define	IB_MGMT_RMPP_STATUS_ABORT_MAX		127
 
 #define IB_QP0		0
 #define IB_QP1		__constant_htonl(1)
Index: core/mad_rmpp.c
===================================================================
--- core/mad_rmpp.c	(revision 2867)
+++ core/mad_rmpp.c	(working copy)
@@ -100,6 +100,121 @@ void ib_cancel_rmpp_recvs(struct ib_mad_
 	}
 }
 
+static int data_offset(u8 mgmt_class)
+{
+	if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM)
+		return offsetof(struct ib_sa_mad, data);
+	else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) &&
+		 (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END))
+		return offsetof(struct ib_vendor_mad, data);
+	else
+		return offsetof(struct ib_rmpp_mad, data);
+}
+
+static void format_ack(struct ib_rmpp_mad *ack,
+		       struct ib_rmpp_mad *data,
+		       struct mad_rmpp_recv *rmpp_recv)
+{
+	unsigned long flags;
+
+	memcpy(&ack->mad_hdr, &data->mad_hdr,
+	       data_offset(data->mad_hdr.mgmt_class));
+
+	ack->mad_hdr.method ^= IB_MGMT_METHOD_RESP;
+	ack->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_ACK;
+	ib_set_rmpp_flags(&ack->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
+
+	spin_lock_irqsave(&rmpp_recv->lock, flags);
+	rmpp_recv->last_ack = rmpp_recv->seg_num;
+	ack->rmpp_hdr.seg_num = cpu_to_be32(rmpp_recv->seg_num);
+	ack->rmpp_hdr.paylen_newwin = cpu_to_be32(rmpp_recv->newwin);
+	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
+}
+
+static void ack_recv(struct mad_rmpp_recv *rmpp_recv,
+		     struct ib_mad_recv_wc *recv_wc)
+{
+	struct ib_mad_send_buf *msg;
+	struct ib_send_wr *bad_send_wr;
+	int hdr_len, ret;
+
+	hdr_len = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr);
+	msg = ib_create_send_mad(&rmpp_recv->agent->agent, recv_wc->wc->src_qp,
+				 recv_wc->wc->pkey_index, rmpp_recv->ah, 1,
+				 hdr_len, sizeof(struct ib_rmpp_mad) - hdr_len,
+				 GFP_KERNEL);
+	if (!msg)
+		return;
+
+	format_ack((struct ib_rmpp_mad *) msg->mad,
+		   (struct ib_rmpp_mad *) recv_wc->recv_buf.mad, rmpp_recv);
+	ret = ib_post_send_mad(&rmpp_recv->agent->agent, &msg->send_wr,
+			       &bad_send_wr);
+	if (ret)
+		ib_free_send_mad(msg);
+}
+
+static int alloc_response_msg(struct ib_mad_agent *agent,
+			      struct ib_mad_recv_wc *recv_wc,
+			      struct ib_mad_send_buf **msg)
+{
+	struct ib_mad_send_buf *m;
+	struct ib_ah *ah;
+	int hdr_len;
+
+	ah = ib_create_ah_from_wc(agent->qp->pd, recv_wc->wc,
+				  recv_wc->recv_buf.grh, agent->port_num);
+	if (IS_ERR(ah))
+		return PTR_ERR(ah);
+
+	hdr_len = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr);
+	m = ib_create_send_mad(agent, recv_wc->wc->src_qp,
+			       recv_wc->wc->pkey_index, ah, 1, hdr_len,
+			       sizeof(struct ib_rmpp_mad) - hdr_len,
+			       GFP_KERNEL);
+	if (IS_ERR(m)) {
+		ib_destroy_ah(ah);
+		return PTR_ERR(m);
+	}
+	*msg = m;
+	return 0;
+}
+
+static void free_msg(struct ib_mad_send_buf *msg)
+{
+	ib_destroy_ah(msg->send_wr.wr.ud.ah);
+	ib_free_send_mad(msg);
+}
+
+static void nack_recv(struct ib_mad_agent_private *agent,
+		      struct ib_mad_recv_wc *recv_wc, u8 rmpp_status)
+{
+	struct ib_mad_send_buf *msg;
+	struct ib_rmpp_mad *rmpp_mad;
+	struct ib_send_wr *bad_send_wr;
+	int ret;
+
+	ret = alloc_response_msg(&agent->agent, recv_wc, &msg);
+	if (ret)
+		return;
+
+	rmpp_mad = (struct ib_rmpp_mad *) msg->mad;
+	memcpy(rmpp_mad, recv_wc->recv_buf.mad,
+	       data_offset(recv_wc->recv_buf.mad->mad_hdr.mgmt_class));
+
+	rmpp_mad->mad_hdr.method ^= IB_MGMT_METHOD_RESP;
+	rmpp_mad->rmpp_hdr.rmpp_version = IB_MGMT_RMPP_VERSION;
+	rmpp_mad->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_ABORT;
+	ib_set_rmpp_flags(&rmpp_mad->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
+	rmpp_mad->rmpp_hdr.rmpp_status = rmpp_status;
+	rmpp_mad->rmpp_hdr.seg_num = 0;
+	rmpp_mad->rmpp_hdr.paylen_newwin = 0;
+
+	ret = ib_post_send_mad(&agent->agent, &msg->send_wr, &bad_send_wr);
+	if (ret)
+		free_msg(msg);
+}
+
 static void recv_timeout_handler(void *data)
 {
 	struct mad_rmpp_recv *rmpp_recv = data;
@@ -115,8 +230,8 @@ static void recv_timeout_handler(void *d
 	list_del(&rmpp_recv->list);
 	spin_unlock_irqrestore(&rmpp_recv->agent->lock, flags);
 
-	/* TODO: send abort. */
 	rmpp_wc = rmpp_recv->rmpp_wc;
+	nack_recv(rmpp_recv->agent, rmpp_wc, IB_MGMT_RMPP_STATUS_T2L);
 	destroy_rmpp_recv(rmpp_recv);
 	ib_free_recv_mad(rmpp_wc);
 }
@@ -230,60 +345,6 @@ insert_rmpp_recv(struct ib_mad_agent_pri
 	return cur_rmpp_recv;
 }
 
-static int data_offset(u8 mgmt_class)
-{
-	if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM)
-		return offsetof(struct ib_sa_mad, data);
-	else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) &&
-		 (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END))
-		return offsetof(struct ib_vendor_mad, data);
-	else
-		return offsetof(struct ib_rmpp_mad, data);
-}
-
-static void format_ack(struct ib_rmpp_mad *ack,
-		       struct ib_rmpp_mad *data,
-		       struct mad_rmpp_recv *rmpp_recv)
-{
-	unsigned long flags;
-
-	memcpy(&ack->mad_hdr, &data->mad_hdr,
-	       data_offset(data->mad_hdr.mgmt_class));
-
-	ack->mad_hdr.method ^= IB_MGMT_METHOD_RESP;
-	ack->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_ACK;
-	ib_set_rmpp_flags(&ack->rmpp_hdr, IB_MGMT_RMPP_FLAG_ACTIVE);
-
-	spin_lock_irqsave(&rmpp_recv->lock, flags);
-	rmpp_recv->last_ack = rmpp_recv->seg_num;
-	ack->rmpp_hdr.seg_num = cpu_to_be32(rmpp_recv->seg_num);
-	ack->rmpp_hdr.paylen_newwin = cpu_to_be32(rmpp_recv->newwin);
-	spin_unlock_irqrestore(&rmpp_recv->lock, flags);
-}
-
-static void ack_recv(struct mad_rmpp_recv *rmpp_recv,
-		     struct ib_mad_recv_wc *recv_wc)
-{
-	struct ib_mad_send_buf *msg;
-	struct ib_send_wr *bad_send_wr;
-	int hdr_len, ret;
-
-	hdr_len = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr);
-	msg = ib_create_send_mad(&rmpp_recv->agent->agent, recv_wc->wc->src_qp,
-				 recv_wc->wc->pkey_index, rmpp_recv->ah, 1,
-				 hdr_len, sizeof(struct ib_rmpp_mad) - hdr_len,
-				 GFP_KERNEL);
-	if (!msg)
-		return;
-
-	format_ack((struct ib_rmpp_mad *) msg->mad,
-		   (struct ib_rmpp_mad *) recv_wc->recv_buf.mad, rmpp_recv);
-	ret = ib_post_send_mad(&rmpp_recv->agent->agent, &msg->send_wr,
-			       &bad_send_wr);
-	if (ret)
-		ib_free_send_mad(msg);
-}
-
 static inline int get_last_flag(struct ib_mad_recv_buf *seg)
 {
 	struct ib_rmpp_mad *rmpp_mad;
@@ -559,6 +620,34 @@ static int send_next_seg(struct ib_mad_s
 	return ib_send_mad(mad_send_wr);
 }
 
+static void abort_send(struct ib_mad_agent_private *agent, u64 tid,
+		       u8 rmpp_status)
+{
+	struct ib_mad_send_wr_private *mad_send_wr;
+	struct ib_mad_send_wc wc;
+	unsigned long flags;
+
+	spin_lock_irqsave(&agent->lock, flags);
+	mad_send_wr = ib_find_send_mad(agent, tid);
+	if (!mad_send_wr)
+		goto out;	/* Unmatched send */
+
+	if ((mad_send_wr->last_ack == mad_send_wr->total_seg) ||
+	    (!mad_send_wr->timeout) || (mad_send_wr->status != IB_WC_SUCCESS))
+		goto out;	/* Send is already done */
+
+	ib_mark_mad_done(mad_send_wr);
+	spin_unlock_irqrestore(&agent->lock, flags);
+
+	wc.status = IB_WC_REM_ABORT_ERR;
+	wc.vendor_err = rmpp_status;
+	wc.wr_id = mad_send_wr->wr_id;
+	ib_mad_complete_send_wr(mad_send_wr, &wc);
+	return;
+out:
+	spin_unlock_irqrestore(&agent->lock, flags);
+}
+
 static void process_rmpp_ack(struct ib_mad_agent_private *agent,
 			     struct ib_mad_recv_wc *mad_recv_wc)
 {
@@ -568,11 +657,21 @@ static void process_rmpp_ack(struct ib_m
 	int seg_num, newwin, ret;
 
 	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
-	if (rmpp_mad->rmpp_hdr.rmpp_status)
+	if (rmpp_mad->rmpp_hdr.rmpp_status) {
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_BAD_STATUS);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_BAD_STATUS);
 		return;
+	}
 
 	seg_num = be32_to_cpu(rmpp_mad->rmpp_hdr.seg_num);
 	newwin = be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin);
+	if (newwin < seg_num) {
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_W2S);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_W2S);
+		return;
+	}
 
 	spin_lock_irqsave(&agent->lock, flags);
 	mad_send_wr = ib_find_send_mad(agent, rmpp_mad->mad_hdr.tid);
@@ -583,8 +682,13 @@ static void process_rmpp_ack(struct ib_m
 	    (!mad_send_wr->timeout) || (mad_send_wr->status != IB_WC_SUCCESS))
 		goto out;	/* Send is already done */
 
-	if (seg_num > mad_send_wr->total_seg)
-		goto out;	/* Bad ACK */
+	if (seg_num > mad_send_wr->total_seg || seg_num > mad_send_wr->newwin) {
+		spin_unlock_irqrestore(&agent->lock, flags);
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_S2B);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_S2B);
+		return;
+	}
 
 	if (newwin < mad_send_wr->newwin || seg_num < mad_send_wr->last_ack)
 		goto out;	/* Old ACK */
@@ -628,6 +732,72 @@ out:
 	spin_unlock_irqrestore(&agent->lock, flags);
 }
 
+static struct ib_mad_recv_wc *
+process_rmpp_data(struct ib_mad_agent_private *agent,
+		  struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_rmpp_hdr *rmpp_hdr;
+	u8 rmpp_status;
+
+	rmpp_hdr = &((struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad)->rmpp_hdr;
+
+	if (rmpp_hdr->rmpp_status) {
+		rmpp_status = IB_MGMT_RMPP_STATUS_BAD_STATUS;
+		goto bad;
+	}
+
+	if (rmpp_hdr->seg_num == __constant_htonl(1)) {
+		if (!(ib_get_rmpp_flags(rmpp_hdr) & IB_MGMT_RMPP_FLAG_FIRST)) {
+			rmpp_status = IB_MGMT_RMPP_STATUS_BAD_SEG;
+			goto bad;
+		}
+		return start_rmpp(agent, mad_recv_wc);
+	} else {
+		if (ib_get_rmpp_flags(rmpp_hdr) & IB_MGMT_RMPP_FLAG_FIRST) {
+			rmpp_status = IB_MGMT_RMPP_STATUS_BAD_SEG;
+			goto bad;
+		}
+		return continue_rmpp(agent, mad_recv_wc);
+	}
+bad:
+	nack_recv(agent, mad_recv_wc, rmpp_status);
+	ib_free_recv_mad(mad_recv_wc);
+	return NULL;
+}
+
+static void process_rmpp_stop(struct ib_mad_agent_private *agent,
+			      struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+
+	if (rmpp_mad->rmpp_hdr.rmpp_status != IB_MGMT_RMPP_STATUS_RESX) {
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_BAD_STATUS);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_BAD_STATUS);
+	} else
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   rmpp_mad->rmpp_hdr.rmpp_status);
+}
+
+static void process_rmpp_abort(struct ib_mad_agent_private *agent,
+			       struct ib_mad_recv_wc *mad_recv_wc)
+{
+	struct ib_rmpp_mad *rmpp_mad;
+
+	rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+
+	if (rmpp_mad->rmpp_hdr.rmpp_status < IB_MGMT_RMPP_STATUS_ABORT_MIN ||
+	    rmpp_mad->rmpp_hdr.rmpp_status > IB_MGMT_RMPP_STATUS_ABORT_MAX) {
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_BAD_STATUS);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_BAD_STATUS);
+	} else
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   rmpp_mad->rmpp_hdr.rmpp_status);
+}
+
 struct ib_mad_recv_wc *
 ib_process_rmpp_recv_wc(struct ib_mad_agent_private *agent,
 			struct ib_mad_recv_wc *mad_recv_wc)
@@ -638,23 +808,29 @@ ib_process_rmpp_recv_wc(struct ib_mad_ag
 	if (!(rmpp_mad->rmpp_hdr.rmpp_rtime_flags & IB_MGMT_RMPP_FLAG_ACTIVE))
 		return mad_recv_wc;
 
-	if (rmpp_mad->rmpp_hdr.rmpp_version != IB_MGMT_RMPP_VERSION)
+	if (rmpp_mad->rmpp_hdr.rmpp_version != IB_MGMT_RMPP_VERSION) {
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_UNV);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_UNV);
 		goto out;
+	}
 
 	switch (rmpp_mad->rmpp_hdr.rmpp_type) {
 	case IB_MGMT_RMPP_TYPE_DATA:
-		if (rmpp_mad->rmpp_hdr.seg_num == __constant_htonl(1))
-			return start_rmpp(agent, mad_recv_wc);
-		else
-			return continue_rmpp(agent, mad_recv_wc);
+		return process_rmpp_data(agent, mad_recv_wc);
 	case IB_MGMT_RMPP_TYPE_ACK:
 		process_rmpp_ack(agent, mad_recv_wc);
 		break;
 	case IB_MGMT_RMPP_TYPE_STOP:
+		process_rmpp_stop(agent, mad_recv_wc);
+		break;
 	case IB_MGMT_RMPP_TYPE_ABORT:
-		/* TODO: process_rmpp_nack(agent, mad_recv_wc); */
+		process_rmpp_abort(agent, mad_recv_wc);
 		break;
 	default:
+		abort_send(agent, rmpp_mad->mad_hdr.tid,
+			   IB_MGMT_RMPP_STATUS_BADT);
+		nack_recv(agent, mad_recv_wc, IB_MGMT_RMPP_STATUS_BADT);
 		break;
 	}
 out:
@@ -714,7 +890,10 @@ int ib_process_rmpp_send_wc(struct ib_ma
 	if (rmpp_mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA) {
 		msg = (struct ib_mad_send_buf *) (unsigned long)
 		      mad_send_wc->wr_id;
-		ib_free_send_mad(msg);
+		if (rmpp_mad->rmpp_hdr.rmpp_type == IB_MGMT_RMPP_TYPE_ACK)
+			ib_free_send_mad(msg);
+		else
+			free_msg(msg);
 		return IB_RMPP_RESULT_INTERNAL;	 /* ACK, STOP, or ABORT */
 	}
 





From mshefty at ichips.intel.com  Fri Jul 15 15:36:11 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 15 Jul 2005 15:36:11 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <200507130248.08387.adobriyan@gmail.com>
References: <1121203934.14638.27.camel@duffman>	<20050712221725.GB14316@mellanox.co.il>
	<200507130248.08387.adobriyan@gmail.com>
Message-ID: <42D83A5B.5090808@ichips.intel.com>

Alexey Dobriyan wrote:
> ----------------------------------------------------------------------------
>    313          in_mad->mad_hdr.attr_id       = cpu_to_be16(0x12); /* PortCounters */
> ----------------------------------------------------------------------------
> drivers/infiniband/core/sysfs.c:313:32: warning: incorrect type in assignment (different base types)
> drivers/infiniband/core/sysfs.c:313:32:    expected unsigned short [unsigned] [usertype] attr_id
> drivers/infiniband/core/sysfs.c:313:32:    got restricted unsigned short [usertype] [force] <noident>
> ----------------------------------------------------------------------------
> Grepping for attr_id in drivers/infiniband/ shows that:
> 1) in_mad->attr_id is set to IB_SMP_ATTR_NODE_INFO (network order)
> 2) mad->mad_hdr.attr_id is compared with IB_SMP_ATTR_PORT_INFO (network order)
> 3) *->mad_hdr.attr_id is set to big-endian value
> 
> All this suggests that struct ib_mad_hdr::attr_id should be __be16 instead of
> u16. So, if attr_id is really something big-endian (infiniband people should
> know), convert it. If not (unlikely) all those cpu_to_be16() and htons() are
> bogus.

If anyone else has started to look at these, please let me know.  Otherwise, 
I will start trying to eliminate them.

- Sean



From rolandd at cisco.com  Fri Jul 15 15:47:10 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 15:47:10 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <42D83A5B.5090808@ichips.intel.com> (Sean Hefty's message of
	"Fri, 15 Jul 2005 15:36:11 -0700")
References: <1121203934.14638.27.camel@duffman>
	<20050712221725.GB14316@mellanox.co.il>
	<200507130248.08387.adobriyan@gmail.com>
	<42D83A5B.5090808@ichips.intel.com>
Message-ID: <52r7dz661d.fsf@topspin.com>

    Sean> If anyone else has started to look at these, please let me
    Sean> know.  Otherwise, I will start trying to eliminate them.

I have a couple of patches for this sort of stuff but nothing that useful.

 - R.


From rolandd at cisco.com  Fri Jul 15 15:48:16 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 15:48:16 -0700
Subject: [openib-general] Firmware needed for cougar MT23108 A1
In-Reply-To: <Pine.GSO.4.58.0507152239340.21876@seth.cpc.wmin.ac.uk> (Thierry
	Delaitre's message of "Fri, 15 Jul 2005 22:45:43 +0100 (BST)")
References: <Pine.GSO.4.58.0507151534381.2014@seth.cpc.wmin.ac.uk>
	<52wtns84ww.fsf@topspin.com>
	<Pine.GSO.4.58.0507152239340.21876@seth.cpc.wmin.ac.uk>
Message-ID: <52mzon65zj.fsf@topspin.com>

    Thierry> 3.2 is fine with me but the openib driver was complaining
    Thierry> that i need to upgrade to 3.3 and i believe the driver is
    Thierry> not happy about this. I've not used openib before as i'm
    Thierry> using the topspin's ib kernel modules.

The driver is just warning you that you don't have the latest
firmware.  If it works for you there's no need to upgrade.

 - R.


From rolandd at cisco.com  Fri Jul 15 15:52:21 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 15 Jul 2005 15:52:21 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <52r7dz661d.fsf@topspin.com> (Roland Dreier's message of "Fri,
	15 Jul 2005 15:47:10 -0700")
References: <1121203934.14638.27.camel@duffman>
	<20050712221725.GB14316@mellanox.co.il>
	<200507130248.08387.adobriyan@gmail.com>
	<42D83A5B.5090808@ichips.intel.com> <52r7dz661d.fsf@topspin.com>
Message-ID: <52irzb65sq.fsf@topspin.com>

For whatever it's worth, here's what I have accumulated so far:

--- infiniband/core/sysfs.c	(revision 2822)
+++ infiniband/core/sysfs.c	(working copy)
@@ -256,14 +256,14 @@ static ssize_t show_port_gid(struct ib_p
 		return ret;
 
 	return sprintf(buf, "%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x\n",
-		       be16_to_cpu(((u16 *) gid.raw)[0]),
-		       be16_to_cpu(((u16 *) gid.raw)[1]),
-		       be16_to_cpu(((u16 *) gid.raw)[2]),
-		       be16_to_cpu(((u16 *) gid.raw)[3]),
-		       be16_to_cpu(((u16 *) gid.raw)[4]),
-		       be16_to_cpu(((u16 *) gid.raw)[5]),
-		       be16_to_cpu(((u16 *) gid.raw)[6]),
-		       be16_to_cpu(((u16 *) gid.raw)[7]));
+		       be16_to_cpu(((__be16 *) gid.raw)[0]),
+		       be16_to_cpu(((__be16 *) gid.raw)[1]),
+		       be16_to_cpu(((__be16 *) gid.raw)[2]),
+		       be16_to_cpu(((__be16 *) gid.raw)[3]),
+		       be16_to_cpu(((__be16 *) gid.raw)[4]),
+		       be16_to_cpu(((__be16 *) gid.raw)[5]),
+		       be16_to_cpu(((__be16 *) gid.raw)[6]),
+		       be16_to_cpu(((__be16 *) gid.raw)[7]));
 }
 
 static ssize_t show_port_pkey(struct ib_port *p, struct port_attribute *attr,
@@ -335,11 +335,11 @@ static ssize_t show_pma_counter(struct i
 		break;
 	case 16:
 		ret = sprintf(buf, "%u\n",
-			      be16_to_cpup((u16 *)(out_mad->data + 40 + offset / 8)));
+			      be16_to_cpup((__be16 *)(out_mad->data + 40 + offset / 8)));
 		break;
 	case 32:
 		ret = sprintf(buf, "%u\n",
-			      be32_to_cpup((u32 *)(out_mad->data + 40 + offset / 8)));
+			      be32_to_cpup((__be32 *)(out_mad->data + 40 + offset / 8)));
 		break;
 	default:
 		ret = 0;
@@ -622,10 +622,10 @@ static ssize_t show_sys_image_guid(struc
 		return ret;
 
 	return sprintf(buf, "%04x:%04x:%04x:%04x\n",
-		       be16_to_cpu(((u16 *) &attr.sys_image_guid)[0]),
-		       be16_to_cpu(((u16 *) &attr.sys_image_guid)[1]),
-		       be16_to_cpu(((u16 *) &attr.sys_image_guid)[2]),
-		       be16_to_cpu(((u16 *) &attr.sys_image_guid)[3]));
+		       be16_to_cpu(((__be16 *) &attr.sys_image_guid)[0]),
+		       be16_to_cpu(((__be16 *) &attr.sys_image_guid)[1]),
+		       be16_to_cpu(((__be16 *) &attr.sys_image_guid)[2]),
+		       be16_to_cpu(((__be16 *) &attr.sys_image_guid)[3]));
 }
 
 static ssize_t show_node_guid(struct class_device *cdev, char *buf)
@@ -639,10 +639,10 @@ static ssize_t show_node_guid(struct cla
 		return ret;
 
 	return sprintf(buf, "%04x:%04x:%04x:%04x\n",
-		       be16_to_cpu(((u16 *) &attr.node_guid)[0]),
-		       be16_to_cpu(((u16 *) &attr.node_guid)[1]),
-		       be16_to_cpu(((u16 *) &attr.node_guid)[2]),
-		       be16_to_cpu(((u16 *) &attr.node_guid)[3]));
+		       be16_to_cpu(((__be16 *) &attr.node_guid)[0]),
+		       be16_to_cpu(((__be16 *) &attr.node_guid)[1]),
+		       be16_to_cpu(((__be16 *) &attr.node_guid)[2]),
+		       be16_to_cpu(((__be16 *) &attr.node_guid)[3]));
 }
 
 static CLASS_DEVICE_ATTR(node_type, S_IRUGO, show_node_type, NULL);
--- infiniband/core/verbs.c	(revision 2822)
+++ infiniband/core/verbs.c	(working copy)
@@ -5,6 +5,7 @@
  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Cisco Systems.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
--- infiniband/core/sa_query.c	(revision 2822)
+++ infiniband/core/sa_query.c	(working copy)
@@ -600,7 +600,7 @@ static void ib_sa_path_rec_release(struc
 int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
 		       struct ib_sa_path_rec *rec,
 		       ib_sa_comp_mask comp_mask,
-		       int timeout_ms, int gfp_mask,
+		       int timeout_ms, unsigned int __nocast gfp_mask,
 		       void (*callback)(int status,
 					struct ib_sa_path_rec *resp,
 					void *context),
@@ -702,7 +702,7 @@ static void ib_sa_service_rec_release(st
 int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
 			    struct ib_sa_service_rec *rec,
 			    ib_sa_comp_mask comp_mask,
-			    int timeout_ms, int gfp_mask,
+			    int timeout_ms, unsigned int __nocast gfp_mask,
 			    void (*callback)(int status,
 					     struct ib_sa_service_rec *resp,
 					     void *context),
@@ -785,7 +785,7 @@ int ib_sa_mcmember_rec_query(struct ib_d
 			     u8 method,
 			     struct ib_sa_mcmember_rec *rec,
 			     ib_sa_comp_mask comp_mask,
-			     int timeout_ms, int gfp_mask,
+			     int timeout_ms, unsigned int __nocast gfp_mask,
 			     void (*callback)(int status,
 					      struct ib_sa_mcmember_rec *resp,
 					      void *context),
--- infiniband/include/ib_cm.h	(revision 2822)
+++ infiniband/include/ib_cm.h	(working copy)
@@ -335,7 +335,7 @@ int ib_cm_listen(struct ib_cm_id *cm_id,
 struct ib_cm_req_param {
 	struct ib_sa_path_rec	*primary_path;
 	struct ib_sa_path_rec	*alternate_path;
-	u64			service_id;
+	__be64			service_id;
 	u32			qp_num;
 	enum ib_qp_type		qp_type;
 	u32			starting_psn;
--- infiniband/include/ib_mad.h	(revision 2822)
+++ infiniband/include/ib_mad.h	(working copy)
@@ -111,12 +111,12 @@ struct ib_mad_hdr {
 	u8	mgmt_class;
 	u8	class_version;
 	u8	method;
-	u16	status;
-	u16	class_specific;
-	u64	tid;
-	u16	attr_id;
+	__be16	status;
+	__be16	class_specific;
+	__be64	tid;
+	__be16	attr_id;
 	u16	resv;
-	u32	attr_mod;
+	__be32	attr_mod;
 };
 
 struct ib_rmpp_hdr {
@@ -124,8 +124,8 @@ struct ib_rmpp_hdr {
 	u8	rmpp_type;
 	u8	rmpp_rtime_flags;
 	u8	rmpp_status;
-	u32	seg_num;
-	u32	paylen_newwin;
+	__be32	seg_num;
+	__be32	paylen_newwin;
 };
 
 typedef u64 __bitwise ib_sa_comp_mask;
--- infiniband/include/ib_verbs.h	(revision 2822)
+++ infiniband/include/ib_verbs.h	(working copy)
@@ -5,6 +5,7 @@
  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Cisco Systems.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -50,8 +51,8 @@
 union ib_gid {
 	u8	raw[16];
 	struct {
-		u64	subnet_prefix;
-		u64	interface_id;
+		__be64	subnet_prefix;
+		__be64	interface_id;
 	} global;
 };
 
@@ -290,8 +291,8 @@ struct ib_global_route {
 };
 
 struct ib_grh {
-	u32		version_tclass_flow;
-	u16		paylen;
+	__be32		version_tclass_flow;
+	__be16		paylen;
 	u8		next_hdr;
 	u8		hop_limit;
 	union ib_gid	sgid;
@@ -694,7 +695,7 @@ struct ib_pd {
 struct ib_ah {
 	struct ib_device	*device;
 	struct ib_pd		*pd;
-	struct ib_uobject      *uobject;
+	struct ib_uobject	*uobject;
 };
 
 typedef void (*ib_comp_handler)(struct ib_cq *cq, void *cq_context);
--- infiniband/include/ib_sa.h	(revision 2822)
+++ infiniband/include/ib_sa.h	(working copy)
@@ -133,16 +133,16 @@ struct ib_sa_path_rec {
 	/* reserved */
 	union ib_gid dgid;
 	union ib_gid sgid;
-	u16          dlid;
-	u16          slid;
+	__be16       dlid;
+	__be16       slid;
 	int          raw_traffic;
 	/* reserved */
-	u32          flow_label;
+	__be32       flow_label;
 	u8           hop_limit;
 	u8           traffic_class;
 	int          reversible;
 	u8           numb_path;
-	u16          pkey;
+	__be16       pkey;
 	/* reserved */
 	u8           sl;
 	u8           mtu_selector;
@@ -176,18 +176,18 @@ struct ib_sa_path_rec {
 struct ib_sa_mcmember_rec {
 	union ib_gid mgid;
 	union ib_gid port_gid;
-	u32          qkey;
-	u16          mlid;
+	__be32       qkey;
+	__be16       mlid;
 	u8           mtu_selector;
 	u8           mtu;
 	u8           traffic_class;
-	u16          pkey;
+	__be16       pkey;
 	u8 	     rate_selector;
 	u8 	     rate;
 	u8 	     packet_life_time_selector;
 	u8 	     packet_life_time;
 	u8           sl;
-	u32          flow_label;
+	__be32       flow_label;
 	u8           hop_limit;
 	u8           scope;
 	u8           join_state;
@@ -238,7 +238,7 @@ struct ib_sa_mcmember_rec {
 struct ib_sa_service_rec {
 	u64		id;
 	union ib_gid	gid;		
-	u16 		pkey;
+	__be16 		pkey;
 	/* reserved */
 	u32		lease;
 	u8		key[16];
@@ -256,7 +256,7 @@ void ib_sa_cancel_query(int id, struct i
 int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
 		       struct ib_sa_path_rec *rec,
 		       ib_sa_comp_mask comp_mask,
-		       int timeout_ms, int gfp_mask,
+		       int timeout_ms, unsigned int __nocast gfp_mask,
 		       void (*callback)(int status,
 					struct ib_sa_path_rec *resp,
 					void *context),
@@ -267,7 +267,7 @@ int ib_sa_mcmember_rec_query(struct ib_d
 			     u8 method,
 			     struct ib_sa_mcmember_rec *rec,
 			     ib_sa_comp_mask comp_mask,
-			     int timeout_ms, int gfp_mask,
+			     int timeout_ms, unsigned int __nocast gfp_mask,
 			     void (*callback)(int status,
 					      struct ib_sa_mcmember_rec *resp,
 					      void *context),
@@ -278,7 +278,7 @@ int ib_sa_service_rec_query(struct ib_de
 			 u8 method,
 			 struct ib_sa_service_rec *rec,
 			 ib_sa_comp_mask comp_mask,
-			 int timeout_ms, int gfp_mask,
+			 int timeout_ms, unsigned int __nocast gfp_mask,
 			 void (*callback)(int status,
 					  struct ib_sa_service_rec *resp,
 					  void *context),
@@ -313,7 +313,7 @@ static inline int
 ib_sa_mcmember_rec_set(struct ib_device *device, u8 port_num,
 		       struct ib_sa_mcmember_rec *rec,
 		       ib_sa_comp_mask comp_mask,
-		       int timeout_ms, int gfp_mask,
+		       int timeout_ms, unsigned int __nocast gfp_mask,
 		       void (*callback)(int status,
 					struct ib_sa_mcmember_rec *resp,
 					void *context),
@@ -355,7 +355,7 @@ static inline int
 ib_sa_mcmember_rec_delete(struct ib_device *device, u8 port_num,
 			  struct ib_sa_mcmember_rec *rec,
 			  ib_sa_comp_mask comp_mask,
-			  int timeout_ms, int gfp_mask,
+			  int timeout_ms, unsigned int __nocast gfp_mask,
 			  void (*callback)(int status,
 					   struct ib_sa_mcmember_rec *resp,
 					   void *context),
--- infiniband/ulp/ipoib/ipoib_fs.c	(revision 2822)
+++ infiniband/ulp/ipoib/ipoib_fs.c	(working copy)
@@ -97,7 +97,7 @@ static int ipoib_mcg_seq_show(struct seq
 
 		for (n = 0, i = 0; i < sizeof mgid / 2; ++i) {
 			n += sprintf(gid_buf + n, "%x",
-				     be16_to_cpu(((u16 *)mgid.raw)[i]));
+				     be16_to_cpu(((__be16 *) mgid.raw)[i]));
 			if (i < sizeof mgid / 2 - 1)
 				gid_buf[n++] = ':';
 		}


From mshefty at ichips.intel.com  Fri Jul 15 16:26:16 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Fri, 15 Jul 2005 16:26:16 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <52irzb65sq.fsf@topspin.com>
References: <1121203934.14638.27.camel@duffman>	<20050712221725.GB14316@mellanox.co.il>	<200507130248.08387.adobriyan@gmail.com>	<42D83A5B.5090808@ichips.intel.com>
	<52r7dz661d.fsf@topspin.com> <52irzb65sq.fsf@topspin.com>
Message-ID: <42D84618.90807@ichips.intel.com>

Roland Dreier wrote:
> For whatever it's worth, here's what I have accumulated so far:

How would you like to proceed on this?  Do you want to complete your changes 
and commit them, or would you like for me to incorporate your patch into a 
larger set of changes?

- Sean


From halr at voltaire.com  Sat Jul 16 06:11:50 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 16 Jul 2005 09:11:50 -0400
Subject: [openib-general] [PATCH] kDAPL CM: Handle async destruction during
	address translation phase of connect
Message-ID: <1121519510.4419.35.camel@hal.voltaire.com>

Handle async destruction during address translation phase of connect.
Also, fix an error path in dapl_ib_connect. 
Minor changes to dapl_ib_cm_remote_addr in preparation for ATS support
for reverse translation (remote GID -> IP address)

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: dapl_openib_cm.c
===================================================================
--- dapl_openib_cm.c	(revision 2866)
+++ dapl_openib_cm.c	(working copy)
@@ -318,6 +318,11 @@ static void dapl_path_comp_handler(u64 r
 	int status;
 	enum dat_event_number event;
 
+	if (cm_ctx->destroy) {
+		kfree(cm_ctx);
+		return;
+	}
+
 	if (rec_num <= 0) {
 		printk(KERN_ERR "dapl_path_comp_handler: path resolution "
 		       "failed %d retry %d!!!\n", rec_num, cm_ctx->retries + 1);
@@ -347,6 +352,7 @@ static void dapl_path_comp_handler(u64 r
 
 	cm_ctx->param.primary_path = &cm_ctx->dapl_path;
 	cm_ctx->param.alternate_path = NULL;
+	cm_ctx->in_callback = 0;
 
 	status = ib_send_cm_req(cm_ctx->cm_id, &cm_ctx->param);
 	if (status) {
@@ -367,6 +373,11 @@ static void dapl_rt_comp_handler(u64 req
 	int status;
 	enum dat_event_number event;
 
+	if (cm_ctx->destroy) {
+		kfree(cm_ctx);
+		return;
+	}
+
 	if (rec_num <= 0) {
 		printk(KERN_ERR "dapl_rt_comp_handler: rec num %d retry %d\n",
 		       rec_num, cm_ctx->retries + 1);
@@ -484,6 +495,7 @@ int dapl_ib_connect(struct dapl_ep *ep,
 	cm_ctx->dapl_comp.fn = &dapl_rt_comp_handler;
 	cm_ctx->dapl_comp.context = cm_ctx;
 	cm_ctx->retries = 0;
+	cm_ctx->in_callback = 1;
 	status =
 	    ib_at_route_by_ip(((struct sockaddr_in *)remote_ia_address)->
 			      sin_addr.s_addr, 0, 0, 0, &cm_ctx->dapl_rt,
@@ -491,7 +503,7 @@ int dapl_ib_connect(struct dapl_ep *ep,
 	if (status < 0) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_at_route_by_ip failed "
 			     "with status %d\n", status);
-		dapl_destroy_cm_id(cm_ctx); /* ??? */
+		kfree(cm_ctx);
 		return -EAGAIN;
 	}
 
@@ -834,29 +846,28 @@ reject:
 	return status;
 }
 
-/*
- * ib_cm_get_remote_gid 
- */
-static int ib_cm_get_remote_gid(struct dapl_cm_ctx *cm_ctx, u8 *remote_gid)
+static int ib_cm_get_remote_gid(struct dapl_cm_ctx *cm_ctx,
+				union ib_gid *remote_gid)
 {
 	return -ENOSYS;	/* for now!!! */
 }
 
+/* 
+ * Get IP address corresponding to remote GID
+ */
 int dapl_ib_cm_remote_addr(struct dapl_ia *ia, struct dapl_cm_ctx *cm_ctx,
 			   struct sockaddr_in6 *remote_ia_address)
 {
-	int rc;
+	int status;
 	union ib_gid remote_gid;
 
-	rc = ib_cm_get_remote_gid(cm_ctx, (u8 *) &remote_gid);
-	if (rc < 0)
-		return rc;
+	status = ib_cm_get_remote_gid(cm_ctx, &remote_gid);
+	if (status < 0)
+		return status;
 
-	if (!memcmp(&ia->hca->gid, &remote_gid, 16)) {
+	if (!memcmp(&ia->hca->gid, &remote_gid, sizeof ia->hca->gid))
 		memcpy(remote_ia_address, &ia->hca->hca_address,
 		       sizeof *remote_ia_address);
-		return 0;
-	}
 
 	return 0;
 }





From halr at voltaire.com  Sat Jul 16 06:33:32 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 16 Jul 2005 09:33:32 -0400
Subject: [openib-general] [PATCH] dapl.h: Use real structure pointer rather
	than void pointer for CR in EP structure
Message-ID: <1121520812.4419.38.camel@hal.voltaire.com>

dapl.h: Use real structure pointer rather than void pointer for CR in EP
structure

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: dapl.h
===================================================================
--- dapl.h      (revision 2866)
+++ dapl.h      (working copy)
@@ -200,7 +200,7 @@ struct dapl_ep {
        struct sockaddr_in6 remote_ia_address;
 
        /* For passive connections we maintain a back pointer to the CR */
-       void *cr;
+       struct dapl_cr *cr;
 
        /* private data container */
        char private_data[DAPL_MAX_PRIVATE_DATA_SIZE];




From shakouba-reiko2 at dsnw.ne.jp  Fri Jul 15 14:09:03 2005
From: shakouba-reiko2 at dsnw.ne.jp (=?ISO-2022-JP?B?GyRCNGRLXE5vO1IbKEIg?=)
Date: Sat, 16 Jul 2005 06:09:03 +0900
Subject: [openib-general] =?iso-2022-jp?b?UmU6GyRCJDQwRkZiJE5GYk1GGyhC?=
	=?iso-2022-jp?b?KBskQjpGGyhCKQ==?=
Message-ID: <20050715.2109030437@shakouba-reiko2-dsnw.ne.jp>

先程、指名しました真美様より、再度あなた様への指名が入りましたので
メールにてご連絡致しました。
【指名者】真美
【プロフィール】
33歳・年収1000万・未婚
【タイトル】
私ではダメなんですか？
【メッセージ】
写真がダメだったんでしょうか？それとも援助金が少ないですか？
お金の事であれば希望に合わせます。100万までなら最初にお渡しできます
ので、一回だけの関係でもいいのですが、なんでダメなんですか？

↓こちらから返信できますのでご登録（無料）をして、お待ち下さい。↓
http://www3.7777888.com/p1/
直接、真美様からご連絡がきます。



From tziporet at mellanox.co.il  Sun Jul 17 03:59:07 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Sun, 17 Jul 2005 13:59:07 +0300
Subject: [openib-general] Firmware needed for cougar MT23108 A1
Message-ID: <506C3D7B14CDD411A52C00025558DED6064BF8C0@mtlex01.yok.mtl.com>

Mellanox just did a new FW release
You can see our web site: http://www.mellanox.com/products/firmware.html

Tziporet

-----Original Message-----
From: Roland Dreier [mailto:rolandd at cisco.com]
Sent: Saturday, July 16, 2005 1:48 AM
To: Thierry Delaitre
Cc: openib-general at openib.org
Subject: Re: [openib-general] Firmware needed for cougar MT23108 A1


    Thierry> 3.2 is fine with me but the openib driver was complaining
    Thierry> that i need to upgrade to 3.3 and i believe the driver is
    Thierry> not happy about this. I've not used openib before as i'm
    Thierry> using the topspin's ib kernel modules.

The driver is just warning you that you don't have the latest
firmware.  If it works for you there's no need to upgrade.

 - R.
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050717/b607474a/attachment.html>

From bohra at cs.rutgers.edu  Sun Jul 17 09:46:49 2005
From: bohra at cs.rutgers.edu (Aniruddha Bohra)
Date: Sun, 17 Jul 2005 12:46:49 -0400
Subject: [openib-general] uDAPL on OpenIB gen2
In-Reply-To: <42D80862.2060708@ichips.intel.com>
References: <42D5599F.8020202@cs.rutgers.edu>	<Pine.LNX.4.61.0507131424430.25603@jlentini-linux.nane.netapp.com>	<42D56AB0.6070804@cs.rutgers.edu>	<Pine.LNX.4.61.0507131547090.25603@jlentini-linux.nane.netapp.com>	<42D591FB.6010106@cs.rutgers.edu>	<Pine.LNX.4.61.0507141226140.25603@jlentini-linux.nane.netapp.com>
	<42D7F80C.4060706@cs.rutgers.edu>
	<42D80862.2060708@ichips.intel.com>
Message-ID: <42DA8B79.2090406@cs.rutgers.edu>



> Sorry, uDAPL is still in development (3 phases) and you picked up 
> right in the middle of IB_CM and IB_AT porting so things are a little 
> tricky. I would strongly recommend sticking with phase 1 for now ( svn 
> r2830 ).
>
> There are basically three phases to this development:
>
>   1.  (get svn r2830) First phase ran with a SOCKET_CM and ib_uverbs. 
> Will scale out beyond 2 nodes. Does not require IPoIB.
>
>   *2. Current Release:*
>   (get svn r2831) Second phase with uCM and ib_uverbs. Path records 
> are hand-rolled. IP addresses are not used so ignore
>   any reporting for now. Local LID/GID is picked up from port 1 and 
> the remote LID/GID is specified with enviroment variables.
>   IP. You can use any remote address you want, it will simply pick up 
> your DAPL_REMOTE_GID and DAPL_REMOTE_LID.
>   This is limited to loopback or 2 node configurations. Does not 
> require IPoIB.
>    See openib/gen2/users/jlentini/userspace/dapl/openib/README for 
> details.
>
>    3. (in process, soon) Third and final phase will include uCM, uAT, 
> and ib_uverbs. I have this working and expect to put
>    together a  patch next week. Note:  uAT still in separate branch 
> (/openib/gen2/branches/shaharf-ibat/src/userspace/libibat)
>    This version will require IPoIB.
>
> If you can wait a week then the uAT version will be ready and the IP 
> addresses to GID mappings will all match.
>
> -arlin

Hi Arlin,
    Since this is an important decision and I guess the support to r2830 
will vanish soon, I will wait
for a week. However, if you have even a development version that I can 
help test, I would be more
than happy to. In any case, as soon as you have something ready to test, 
I would like to do so.

Thanks
Aniruddha



From mst at mellanox.co.il  Sun Jul 17 10:27:19 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 17 Jul 2005 20:27:19 +0300
Subject: [openib-general] [PATCH] fork support
Message-ID: <20050717172719.GA29577@mellanox.co.il>

Hi!
Here's a patch to linux kernel to enable fork() support for uverbs.
I'm posting it here before forwarding to lkml. Please comment.

Tested on x86_64.

---

This patch adds PROT_DONTCOPY to mmap and mprotect, to set VM_DONTCOPY on vma.
This is needed for infiniband userspace verbs, where we need to protect against
  - the child process accessing the parent hardware page
  - the parent registered address (on which the driver did get_user_pages)
    getting remapped to another page by COW
One can imagine other uses, e.g. combined with mlock for real-time or security.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/include/asm-ppc64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ppc64/mman.h
+++ linux-2.6.12.2/include/asm-ppc64/mman.h
@@ -15,6 +15,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-cris/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-cris/mman.h
+++ linux-2.6.12.2/include/asm-cris/mman.h
@@ -10,6 +10,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-arm26/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-arm26/mman.h
+++ linux-2.6.12.2/include/asm-arm26/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-alpha/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-alpha/mman.h
+++ linux-2.6.12.2/include/asm-alpha/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-m68k/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-m68k/mman.h
+++ linux-2.6.12.2/include/asm-m68k/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-mips/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-mips/mman.h
+++ linux-2.6.12.2/include/asm-mips/mman.h
@@ -22,6 +22,7 @@
 #define PROT_SEM	0x10		/* page may be used for atomic ops */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 /*
  * Flags for mmap
Index: linux-2.6.12.2/include/asm-sparc64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sparc64/mman.h
+++ linux-2.6.12.2/include/asm-sparc64/mman.h
@@ -11,6 +11,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-v850/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-v850/mman.h
+++ linux-2.6.12.2/include/asm-v850/mman.h
@@ -7,6 +7,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-s390/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-s390/mman.h
+++ linux-2.6.12.2/include/asm-s390/mman.h
@@ -16,6 +16,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-parisc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-parisc/mman.h
+++ linux-2.6.12.2/include/asm-parisc/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-ppc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ppc/mman.h
+++ linux-2.6.12.2/include/asm-ppc/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-i386/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-i386/mman.h
+++ linux-2.6.12.2/include/asm-i386/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-sh/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sh/mman.h
+++ linux-2.6.12.2/include/asm-sh/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-x86_64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-x86_64/mman.h
+++ linux-2.6.12.2/include/asm-x86_64/mman.h
@@ -8,6 +8,7 @@
 #define PROT_SEM	0x8
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-ia64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ia64/mman.h
+++ linux-2.6.12.2/include/asm-ia64/mman.h
@@ -15,6 +15,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-sparc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sparc/mman.h
+++ linux-2.6.12.2/include/asm-sparc/mman.h
@@ -11,6 +11,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-m32r/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-m32r/mman.h
+++ linux-2.6.12.2/include/asm-m32r/mman.h
@@ -10,6 +10,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-frv/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-frv/mman.h
+++ linux-2.6.12.2/include/asm-frv/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/linux/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/linux/mman.h
+++ linux-2.6.12.2/include/linux/mman.h
@@ -47,9 +47,10 @@ static inline void vm_unacct_memory(long
 static inline unsigned long
 calc_vm_prot_bits(unsigned long prot)
 {
-	return _calc_vm_trans(prot, PROT_READ,  VM_READ ) |
-	       _calc_vm_trans(prot, PROT_WRITE, VM_WRITE) |
-	       _calc_vm_trans(prot, PROT_EXEC,  VM_EXEC );
+	return _calc_vm_trans(prot, PROT_READ,     VM_READ ) |
+	       _calc_vm_trans(prot, PROT_WRITE,    VM_WRITE) |
+	       _calc_vm_trans(prot, PROT_EXEC,     VM_EXEC ) |
+	       _calc_vm_trans(prot, PROT_DONTCOPY, VM_DONTCOPY );
 }
 
 /*
Index: linux-2.6.12.2/include/asm-h8300/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-h8300/mman.h
+++ linux-2.6.12.2/include/asm-h8300/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-arm/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-arm/mman.h
+++ linux-2.6.12.2/include/asm-arm/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/mm/mprotect.c
===================================================================
--- linux-2.6.12.2.orig/mm/mprotect.c
+++ linux-2.6.12.2/mm/mprotect.c
@@ -196,7 +196,7 @@ sys_mprotect(unsigned long start, size_t
 	end = start + len;
 	if (end <= start)
 		return -ENOMEM;
-	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM))
+	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM | PROT_DONTCOPY))
 		return -EINVAL;
 
 	reqprot = prot;
@@ -246,7 +246,7 @@ sys_mprotect(unsigned long start, size_t
 			goto out;
 		}
 
-		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC));
+		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC | VM_DONTCOPY));
 
 		if ((newflags & ~(newflags >> 4)) & 0xf) {
 			error = -EACCES;
Index: linux-2.6.12.2/mm/mmap.c
===================================================================
--- linux-2.6.12.2.orig/mm/mmap.c
+++ linux-2.6.12.2/mm/mmap.c
@@ -792,8 +792,8 @@ struct anon_vma *find_mergeable_anon_vma
 	 * Neither mlock nor madvise tries to remerge at present,
 	 * so leave their flags as obstructing a merge.
 	 */
-	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC);
-	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC);
+	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
+	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
 
 	if (near->anon_vma && vma->vm_end == near->vm_start &&
  			mpol_equal(vma_policy(vma), vma_policy(near)) &&
@@ -814,8 +814,8 @@ try_prev:
 	if (!near)
 		goto none;
 
-	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC);
-	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC);
+	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
+	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
 
 	if (near->anon_vma && near->vm_end == vma->vm_start &&
   			mpol_equal(vma_policy(near), vma_policy(vma)) &&

-- 
MST


From delusion at delusion.com  Sun Jul 17 16:18:08 2005
From: delusion at delusion.com (Deanan)
Date: Sun, 17 Jul 2005 16:18:08 -0700
Subject: [openib-general] mem register failing
In-Reply-To: <20050717190002.2EDCB2283DD@openib.ca.sandia.gov>
References: <20050717190002.2EDCB2283DD@openib.ca.sandia.gov>
Message-ID: <42DAE730.9050904@delusion.com>

Hi All,

Has anyone seen vapi_register_mr fail with "Not enough memory" at above 
~1MB? I'm getting this error in
the kernel:
Jul 18 04:11:24 e09 kernel:  MOSAL(1): ddr[9548]: 
usr/mellanox/src/vapi/kernel/mlxsys/mosal_iobuf.c:513: region not in 
process's address space. va=0x5028B0 size=0x20D000
Jul 18 04:11:24 e09 kernel:  VIPKL(1): 
usr/mellanox/src/vapi/kernel/vip/mmu.c[242]: make_iobuf: 
MOSAL_iobuf_register failed: va=0x5028B0size=0x20D000
Jul 18 04:11:24 e09 kernel:  VIPKL(1): [create_mr] MM_bld_hh_mr failed 
(-252:VAPI_ENOMEM)

Thanks,

Deanan






From mlleinin at hpcn.ca.sandia.gov  Sun Jul 17 23:21:53 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Sun, 17 Jul 2005 23:21:53 -0700
Subject: [openib-general] mem register failing
In-Reply-To: <42DAE730.9050904@delusion.com>
References: <20050717190002.2EDCB2283DD@openib.ca.sandia.gov>
	<42DAE730.9050904@delusion.com>
Message-ID: <1121667713.11021.383.camel@localhost>

Deanan,

   The stack you are using is not part of OpenIB.  I suggest you contact
Mellanox or your IB software vendor to help you through these support
issues.  

  - Matt


On Sun, 2005-07-17 at 16:18 -0700, Deanan wrote:
> Hi All,
> 
> Has anyone seen vapi_register_mr fail with "Not enough memory" at above 
> ~1MB? I'm getting this error in
> the kernel:
> Jul 18 04:11:24 e09 kernel:  MOSAL(1): ddr[9548]: 
> usr/mellanox/src/vapi/kernel/mlxsys/mosal_iobuf.c:513: region not in 
> process's address space. va=0x5028B0 size=0x20D000
> Jul 18 04:11:24 e09 kernel:  VIPKL(1): 
> usr/mellanox/src/vapi/kernel/vip/mmu.c[242]: make_iobuf: 
> MOSAL_iobuf_register failed: va=0x5028B0size=0x20D000
> Jul 18 04:11:24 e09 kernel:  VIPKL(1): [create_mr] MM_bld_hh_mr failed 
> (-252:VAPI_ENOMEM)
> 
> Thanks,
> 
> Deanan
> 
> 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 



From delusion at delusion.com  Mon Jul 18 00:06:45 2005
From: delusion at delusion.com (Deanan)
Date: Mon, 18 Jul 2005 00:06:45 -0700
Subject: [openib-general] mem register failing
In-Reply-To: <1121667713.11021.383.camel@localhost>
References: <20050717190002.2EDCB2283DD@openib.ca.sandia.gov>	
	<42DAE730.9050904@delusion.com>
	<1121667713.11021.383.camel@localhost>
Message-ID: <42DB5505.5060306@delusion.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/a33e8f70/attachment.html>

From mst at mellanox.co.il  Mon Jul 18 00:19:17 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 18 Jul 2005 10:19:17 +0300
Subject: [openib-general] wiki down
Message-ID: <20050718071917.GD29424@mellanox.co.il>

I get timeouts on http://openib.org/tiki


-- 
MST


From nacc at us.ibm.com  Mon Jul 18 01:27:40 2005
From: nacc at us.ibm.com (Nishanth Aravamudan)
Date: Mon, 18 Jul 2005 01:27:40 -0700
Subject: [openib-general] wiki down
In-Reply-To: <20050718071917.GD29424@mellanox.co.il>
References: <20050718071917.GD29424@mellanox.co.il>
Message-ID: <20050718082740.GA4205@us.ibm.com>

On 18.07.2005 [10:19:17 +0300], Michael S. Tsirkin wrote:
> I get timeouts on http://openib.org/tiki

Seems to work ok for me, right now...

Thanks,
Nish


From mst at mellanox.co.il  Mon Jul 18 02:42:35 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 18 Jul 2005 12:42:35 +0300
Subject: [openib-general] Re: wiki down
In-Reply-To: <20050718082740.GA4205@us.ibm.com>
References: <20050718071917.GD29424@mellanox.co.il>
	<20050718082740.GA4205@us.ibm.com>
Message-ID: <20050718094235.GH29424@mellanox.co.il>

Quoting r. Nishanth Aravamudan <nacc at us.ibm.com>:
> Subject: Re: wiki down
> 
> On 18.07.2005 [10:19:17 +0300], Michael S. Tsirkin wrote:
> > I get timeouts on http://openib.org/tiki
> 
> Seems to work ok for me, right now...

Never mind, the problem was located between the keyboard and the chair.

-- 
MST


From halr at voltaire.com  Mon Jul 18 05:08:39 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 18 Jul 2005 08:08:39 -0400
Subject: [openib-general] [PATCH] [kdapl CM] Implement reverse address
	translation (dapl_ib_cm_remote_addr)
Message-ID: <1121688501.4419.5820.camel@hal.voltaire.com>

[kdapl CM] Implement reverse address translation
(dapl_ib_cm_remote_addr)

Signed-off-by: Hal Rosenstock <halr at voltaire.com>


-- dapl_openib_cm.c.atkill	2005-07-16 08:54:39.000000000 -0400
+++ dapl_openib_cm.c	2005-07-17 20:53:30.000000000 -0400
@@ -220,6 +220,7 @@ static struct dapl_cm_ctx * dapl_get_cm_
 	cm_ctx->sp = comm_id->context;
 	comm_id->context = cm_ctx;
 	spin_lock_init(&cm_ctx->lock);
+	init_waitqueue_head(&cm_ctx->wait);
 	return cm_ctx;
 }
 
@@ -247,6 +248,9 @@ static int dapl_cm_passive_cb_handler(st
 
 	switch (event->event) {
 	case IB_CM_REQ_RECEIVED:
+		memcpy(&cm_ctx->dapl_path,
+		       ((struct ib_cm_req_event_param *)&event->param)->primary_path, 
+		       sizeof cm_ctx->dapl_path);
 		dapl_cr_callback(cm_ctx, DAT_CONNECTION_REQUEST_EVENT, 
 				 event->private_data, cm_ctx->sp);
 		break;
@@ -464,6 +468,7 @@ int dapl_ib_connect(struct dapl_ep *ep,
 	memset(cm_ctx, 0, sizeof *cm_ctx);
 
 	spin_lock_init(&cm_ctx->lock);
+	init_waitqueue_head(&cm_ctx->wait);
 	cm_ctx->ep = ep;
 	cm_ctx->cm_id = ib_create_cm_id(dapl_cm_active_cb_handler, cm_ctx);
 	if (IS_ERR(cm_ctx->cm_id)) {
@@ -846,28 +851,63 @@ reject:
 	return status;
 }
 
-static int ib_cm_get_remote_gid(struct dapl_cm_ctx *cm_ctx,
-				union ib_gid *remote_gid)
+static void dapl_ips_comp_handler(u64 req_id, void *context, int rec_num)
 {
-	return -ENOSYS;	/* for now!!! */
+	struct dapl_cm_ctx *cm_ctx = context;
+
+	atomic_set(&cm_ctx->wait_event, 0);
+	wake_up(&cm_ctx->wait);
 }
 
 /* 
- * Get IP address corresponding to remote GID
+ * dapl_ib_cm_remote_addr
+ *
+ * Obtain the IP address for a connection on an IA 
+ *
  */
 int dapl_ib_cm_remote_addr(struct dapl_ia *ia, struct dapl_cm_ctx *cm_ctx,
 			   struct sockaddr_in6 *remote_ia_address)
 {
+	union ib_gid *remote_gid;
+	u32 remote_ip;
+	struct sockaddr_in *ipv4_addr;
+	unsigned long flags;
 	int status;
-	union ib_gid remote_gid;
 
-	status = ib_cm_get_remote_gid(cm_ctx, &remote_gid);
-	if (status < 0)
-		return status;
+	spin_lock_irqsave(&cm_ctx->lock, flags);
+	if (atomic_read(&cm_ctx->wait_event) == 1) {
+		spin_unlock_irqrestore(&cm_ctx->lock, flags);
+		return -EAGAIN;
+	}
+	atomic_set(&cm_ctx->wait_event, 1);
+	spin_unlock_irqrestore(&cm_ctx->lock, flags);
 
-	if (!memcmp(&ia->hca->gid, &remote_gid, sizeof ia->hca->gid))
+	if (cm_ctx->sp)		/* passive side */
+		remote_gid = &cm_ctx->dapl_path.sgid;
+	else
+		remote_gid = &cm_ctx->dapl_path.dgid;
+	if (!memcmp(&ia->hca->gid, remote_gid, sizeof ia->hca->gid))
 		memcpy(remote_ia_address, &ia->hca->hca_address,
 		       sizeof *remote_ia_address);
 
+	/* Reverse ATS: lookup IP address by GID */
+	cm_ctx->dapl_comp.fn = &dapl_ips_comp_handler;
+	cm_ctx->dapl_comp.context = cm_ctx;
+	status = ib_at_ips_by_gid(remote_gid, &remote_ip, 1,
+				  &cm_ctx->dapl_comp);
+	if (status < 0) {
+		atomic_set(&cm_ctx->wait_event, 0);
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_at_ips_by_gid failed "
+			     "with status %d\n", status);
+		return -EAGAIN;
+	}
+
+	if (status != 1)
+		wait_event(cm_ctx->wait, !atomic_read(&cm_ctx->wait_event));
+
+	remote_ip = be32_to_cpu(remote_ip); /* or other way 'round ? */
+	ipv4_addr = (struct sockaddr_in *) remote_ia_address;
+	ipv4_addr->sin_family = AF_INET;
+	memcpy(&ipv4_addr->sin_addr.s_addr, &remote_ip, 4);
 	return 0;
 }




From rolandd at cisco.com  Mon Jul 18 05:15:00 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 18 Jul 2005 05:15:00 -0700
Subject: [openib-general] Re: [PATCH] fork support
In-Reply-To: <20050717172719.GA29577@mellanox.co.il> (Michael S. Tsirkin's
	message of "Sun, 17 Jul 2005 20:27:19 +0300")
References: <20050717172719.GA29577@mellanox.co.il>
Message-ID: <52d5pg5n0b.fsf@topspin.com>

Thanks for doing this.  I'm actually at the Kernel Summit right now,
so I'll see what lobbying I can do.  Overall looks good, although I
have a few comments:

 - This patch allows userspace to remove VM_DONTCOPY from vmas even if
   the kernel set it.  Right now, VM_DONTCOPY is not used except for a
   couple of DRM drivers, so I'm not sure if this matters or not.  If
   it is a problem, I guess the answer is to create a VM_USERDONTCOPY
   that works just like VM_DONTCOPY and have PROT_DONTCOPY affect that.

 - I know this isn't your fault, but is it time to try and move all
   the common PROT_XXX defines somewhere to avoid so much duplication?

 - R.


From dotanb at mellanox.co.il  Mon Jul 18 05:15:07 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Mon, 18 Jul 2005 15:15:07 +0300
Subject: [openib-general] calling to ibv_create_qp with big number in
	qp_init_attr.cap.max_ inline_data never return
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>

when i try to create a QP (in the user level) with the following parameters:

        struct ibv_qp_init_attr qp_init_attr = {
            .qp_type = IBV_QPT_RC,
            .recv_cq = rcq,
            .send_cq = scq,
            .sq_sig_all = 0,
            .cap.max_send_wr = 10,
            .cap.max_send_sge = 10,
            .cap.max_recv_wr = 10,
            .cap.max_recv_sge = 10,
	.cap.max_inline_data = 1075060724
        };

        qp = ibv_create_qp(pd, &qp_init_attr);
        if (qp == NULL) {
            perror("Error, failed to create QP");
            goto cleanup;
        }

the create_qp function never ends.

Dotan Barak
Software Verification Engineer
Mellanox Technologies LTD
mailto:dotanb at mellanox.co.il
Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
P.O. Box 86 Yokneam 20692 ISRAEL.
Home: +972-4-8289408  Cell: 052-4222383

[ May the fork be with you ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/80edd6cf/attachment.html>

From halr at voltaire.com  Mon Jul 18 05:11:54 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 18 Jul 2005 08:11:54 -0400
Subject: [openib-general] [PATCH] [TRIVIAL] [kdapl test]: Fix some
	formatting in transaction test
Message-ID: <1121688507.4419.5824.camel@hal.voltaire.com>

Fix some formatting in transaction test

Signed-off-by: Hal Rosenstock <halr at voltaire.com>


Index: dapl_transaction_test.c
===================================================================
-- dapl_transaction_test.c     (revision 2866)
+++ dapl_transaction_test.c     (working copy)
@@ -1924,7 +1924,7 @@ DT_Print_Transaction_Test (DT_Tdep_Print
                    test_ptr->remote_is_little_endian);
     DT_Tdep_PT_Printf (phead, "TransTest.base_port              : " F64x "\n",
                    test_ptr->base_port);
-    DT_Tdep_PT_Printf (phead, "TransTest.pz              : %p\n",
+    DT_Tdep_PT_Printf (phead, "TransTest.pz                     : %p\n",
                    test_ptr->pz);
     /* statistics */
     DT_Tdep_PT_Printf (phead, "TransTest.bytes_send             : %d\n",



From rolandd at cisco.com  Mon Jul 18 05:19:03 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 18 Jul 2005 05:19:03 -0700
Subject: [openib-general] calling to ibv_create_qp with big number in
	qp_init_attr.cap.max_ inline_data never return
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Mon, 18 Jul 2005 15:15:07 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>
Message-ID: <523bqc5mtk.fsf@topspin.com>

    Dotan> the create_qp function never ends.

Where does it hang?  Can you do strace on the process?  If it's stuck
sleeping, what does /proc/<pid>/wchan say?

 - R.


From biaxial at msn.com  Mon Jul 18 06:28:27 2005
From: biaxial at msn.com (Zachariah Tripp)
Date: Mon, 18 Jul 2005 12:28:27 -0100
Subject: [openib-general] Mortgage News Update.
Message-ID: <111.56e558d5.biaxial@msn.com>

Xmong. Npos alter. almonsted nocks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/08b788b7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: watershed.jpg
Type: image/jpeg
Size: 31221 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/08b788b7/attachment.jpg>

From mst at mellanox.co.il  Mon Jul 18 05:36:10 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 18 Jul 2005 15:36:10 +0300
Subject: [openib-general] Re: calling to ibv_create_qp with big number in
	qp_init_attr.cap.max_ inline_data never return
In-Reply-To: <523bqc5mtk.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>
	<523bqc5mtk.fsf@topspin.com>
Message-ID: <20050718123610.GA5758@mellanox.co.il>

Hi, Roland!
Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: calling to ibv_create_qp with big number in qp_init_attr.cap.max_ inline_data never return
> 
>     Dotan> the create_qp function never ends.
> 
> Where does it hang?  Can you do strace on the process?  If it's stuck
> sleeping, what does /proc/<pid>/wchan say?

Here:

        size = sizeof (struct mthca_next_seg) +
                qp->sq.max_gs * sizeof (struct mthca_data_seg);
        switch (qp->qpt) {
        case IBV_QPT_UD:
                if (mthca_is_memfree(pd->context))
                        size += sizeof (struct mthca_arbel_ud_seg);
                else
                        size += sizeof (struct mthca_tavor_ud_seg);
                break;
        default:
                /* bind seg is as big as atomic + raddr segs */
                size += sizeof (struct mthca_bind_seg);
        }

---->

        for (qp->sq.wqe_shift = 6; 1 << qp->sq.wqe_shift < size;
             qp->sq.wqe_shift++)
                ; /* nothing */


The problem here is that size is bigger than 0x40000000.
As a result 1 << qp->sq.wqe_shift gets to 0x80000000, which is negative,
so its less than size, and everything starts all over again.

Looking at the code, passing insanely huge values in qp params
will get all kind of overflows (e.g. size could get negative).

I think the best way is to check qp parameters for sanity in
mthca_create_qp.

-- 
MST


From dotanb at mellanox.co.il  Mon Jul 18 05:36:46 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Mon, 18 Jul 2005 15:36:46 +0300
Subject: [openib-general] RE: calling to ibv_create_qp with big number in
	qp_init_attr.cap. max_ inline_data never return
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A21A9@mtlex01.yok.mtl.com>

here is a test that reproduces the problem:


#include <stdlib.h>
#include <stdio.h>
#include <infiniband/verbs.h>

#define PORT_NUM 1                  /* IB port number to work with */
#define MR_SIZE (1024)              /* size of the MR */
#define CQ_SIZE 10                  /* size of the CQ */
#define QP_CAP_SG_WR 1 /* s/g and w/r of the QP */


/* structure of test parameters */
struct config_t {
    uint32_t                tcp_port;
    const char*             ip;
    const char               *dev_name;
    uint8_t                  ib_port;

    uint8_t                  is_daemon;

};

struct config_t config = {
    20000,          /* tcp_port */
    "0",            /* ip */
    "mthca0",       /* ib_dev */
    PORT_NUM,       /* ib_port */
};


int main(
    int argc,
    char *argv[])
{
    int test_result = 1;
    struct dlist *dev_list;
    struct ibv_device *ib_dev = NULL;
    struct ibv_context *ctx = NULL;
    struct ibv_pd *pd = NULL;
    struct ibv_cq *rcq = NULL;
    struct ibv_cq *scq = NULL;
    struct ibv_qp *qp = NULL;
    
    printf("Finding IB devices\n");
    /* get device names in the system */
    dev_list = ibv_get_devices();
    if (dev_list == NULL) {
        perror("Error, failed to get IB devices list");
        goto cleanup;
    }
    
    dlist_for_each_data(dev_list, ib_dev, struct ibv_device)
        if (!strcmp(ibv_get_device_name(ib_dev), config.dev_name)) {
            break;
        }

    if (ib_dev == NULL) {
        printf("Error, IB device %s wasn't found\n", config.dev_name);
        goto cleanup;
    }
    printf("Device %s was found\n", config.dev_name);
    
    ctx = ibv_open_device(ib_dev);
    if (ctx == NULL) {
        perror("Error, failed to open device");
        goto cleanup;
    }
    
    pd = ibv_alloc_pd(ctx);
    if (pd == NULL) {
        printf("Error, failed to allocate PD\n");
        goto cleanup;
    }
    printf("PD was allocated\n");

    rcq = ibv_create_cq(ctx, CQ_SIZE, NULL);
    if (rcq == NULL) {
        perror("Error, failed to create receive CQ");
        goto cleanup;
    }
    printf("Receive CQ was created with %u entries\n", rcq->cqe);

    scq = ibv_create_cq(ctx, CQ_SIZE, NULL);
    if (scq == NULL) {
        perror("Error, failed to create send CQ");
        goto cleanup;
    }
    printf("Send was created with %u entries\n", scq->cqe);

    {
        struct ibv_qp_init_attr qp_init_attr = {
            .qp_type = IBV_QPT_RC,
            .recv_cq = rcq,
            .send_cq = scq,
            .sq_sig_all = 0,
            .cap.max_send_wr = QP_CAP_SG_WR,
            .cap.max_send_sge = QP_CAP_SG_WR,
            .cap.max_recv_wr = QP_CAP_SG_WR,
            .cap.max_recv_sge = QP_CAP_SG_WR
        };

        qp_init_attr.cap.max_inline_data = 1075060724;
        
        printf("before calling create QP\n");
        qp = ibv_create_qp(pd, &qp_init_attr);
        printf("after calling create QP\n");
        if (qp == NULL) {
            perror("Error, failed to create QP");
            goto cleanup;
        }
    }
    printf("QP with number 0x%x was created\n", qp->qp_num);

    test_result = 0;
cleanup:
    return test_result;
}



Dotan

-----Original Message-----
From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
Sent: Monday, July 18, 2005 3:36 PM
To: Roland Dreier
Cc: Dotan Barak; openib-general at openib.org
Subject: Re: calling to ibv_create_qp with big number in
qp_init_attr.cap.max_ inline_data never return


Hi, Roland!
Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: calling to ibv_create_qp with big number in
qp_init_attr.cap.max_ inline_data never return
> 
>     Dotan> the create_qp function never ends.
> 
> Where does it hang?  Can you do strace on the process?  If it's stuck
> sleeping, what does /proc/<pid>/wchan say?

Here:

        size = sizeof (struct mthca_next_seg) +
                qp->sq.max_gs * sizeof (struct mthca_data_seg);
        switch (qp->qpt) {
        case IBV_QPT_UD:
                if (mthca_is_memfree(pd->context))
                        size += sizeof (struct mthca_arbel_ud_seg);
                else
                        size += sizeof (struct mthca_tavor_ud_seg);
                break;
        default:
                /* bind seg is as big as atomic + raddr segs */
                size += sizeof (struct mthca_bind_seg);
        }

---->

        for (qp->sq.wqe_shift = 6; 1 << qp->sq.wqe_shift < size;
             qp->sq.wqe_shift++)
                ; /* nothing */


The problem here is that size is bigger than 0x40000000.
As a result 1 << qp->sq.wqe_shift gets to 0x80000000, which is negative,
so its less than size, and everything starts all over again.

Looking at the code, passing insanely huge values in qp params
will get all kind of overflows (e.g. size could get negative).

I think the best way is to check qp parameters for sanity in
mthca_create_qp.

-- 
MST
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/0cf00277/attachment.html>

From mst at mellanox.co.il  Mon Jul 18 05:56:10 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 18 Jul 2005 15:56:10 +0300
Subject: [openib-general] Re: [PATCH] fork support
In-Reply-To: <52d5pg5n0b.fsf@topspin.com>
References: <20050717172719.GA29577@mellanox.co.il>
	<52d5pg5n0b.fsf@topspin.com>
Message-ID: <20050718125610.GB5758@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [PATCH] fork support
> 
> Thanks for doing this.  I'm actually at the Kernel Summit right now,
> so I'll see what lobbying I can do.

Thanks! Would it be easier, or harder, if I post this at lkml?

> Overall looks good, although I have a few comments:
> 
>  - This patch allows userspace to remove VM_DONTCOPY from vmas even if
>    the kernel set it.

Well, we'll need it to deregister stuff, wont we?

>    Right now, VM_DONTCOPY is not used except for a
>    couple of DRM drivers, so I'm not sure if this matters or not.

Right. I think its fine as it is - those drivers probably can switch
to using mprotect when its in.

>    If
>    it is a problem, I guess the answer is to create a VM_USERDONTCOPY
>    that works just like VM_DONTCOPY and have PROT_DONTCOPY affect that.


A somewhat bigger problem is that a legacy application that plays with
mprotect on registered buffers can accidentally remove VM_DONTCOPY just
by doing mprotect(PROT_WRITE|PROT_READ).
I toyed with an idea of yet another flag, say PROT_EXTENDED,
that says: dont ignore PROT_DONTCOPY.
So that

mprotect(PROT_WRITE|PROT_READ) doesnt touch VM_DONTCOPY
mprotect(PROT_EXTENDED) clears PROT_DONTCOPY
mprotect(PROT_EXTENDED|PROT_DONTCOPY) clears PROT_DONTCOPY

I went for simplicity. Do you think the above a good idea?

>  - I know this isn't your fault, but is it time to try and move all
>    the common PROT_XXX defines somewhere to avoid so much duplication?
> 
>  - R.
> 

Most of it is duplication, but some architectures define
PROT_ flags slightly differently. Easy to do ... let me know what do you
think.

-- 
MST


From halr at voltaire.com  Mon Jul 18 07:07:44 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 18 Jul 2005 10:07:44 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42D7EC83.8030406@ichips.intel.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
	<1121351537.4423.1162.camel@hal.voltaire.com>
	<42D6F3F5.5020806@ichips.intel.com>
	<1121425731.4423.2145.camel@hal.voltaire.com>
	<42D7EC83.8030406@ichips.intel.com>
Message-ID: <1121695663.4419.6024.camel@hal.voltaire.com>

Hi Arlin,

On Fri, 2005-07-15 at 13:04, Arlin Davis wrote:
> Your implementation is fine, as long as the correct return values are used
> and the sync version works. I currently get returned a 1 but the ib_route
> data is not filled in and the req_id == 0. I would expect a 0 returned 
> and a
> valid req_id if the ib_route cannot be processed.
> 
> for ret = ib_at_route_by_ip( async_cb provided ), I read this as: 
> 
>   if ret > 0  then ib_route output data is written, resolve complete
>   if ret == 0 then req_id is written with valid id to cancel or poll
>   if ret < 0 error
> 
> Am I reading this right?

Try this patch and let me know if this works for you.

Thanks.

-- Hal

Index: uat.c
===================================================================
--- uat.c	(revision 2849)
+++ uat.c	(working copy)
@@ -271,9 +271,15 @@ static ssize_t ib_uat_route_by_ip(struct
 			goto err4;
 		}
 	}
-	if (result == 1) 
+	if (result == 1) { 
+		/* Copy route back to userspace */
+		if (copy_to_user(ctx->user_ib_route, ctx->ib_route,
+				 sizeof(*ctx->user_ib_route))) {
+			result = -EFAULT;
+			goto err3;
+		}
 		ib_uat_route_callback(ctx->req_id, ctx, result);
-	else if (result < 0) {
+	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
 	}
@@ -363,9 +369,15 @@ static ssize_t ib_uat_paths_by_route(str
 			goto err4;
 		}
 	}
-	if (result == 1)
+	if (result == 1) {
+		/* Copy path records returned from SA to userspace */
+		if (copy_to_user(ctx->user_path_arr, ctx->path_arr,
+				 ctx->user_length)) {
+			result = -EFAULT;
+			goto err3;
+		}
 		ib_uat_path_callback(ctx->req_id, ctx, result);
-	else if (result < 0) {
+	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
 	}
@@ -453,9 +465,14 @@ static ssize_t ib_uat_ips_by_gid(struct 
 			goto err4;
 		}
 	}
-	if (result == 1)
+	if (result == 1) {
+		/* Copy IP addresses back to userspace */
+		if (copy_to_user(ctx->user_ips, ctx->ips, ctx->user_length)) {
+			result = -EFAULT;
+			goto err3;
+		}
 		ib_uat_ips_callback(ctx->req_id, ctx, result);
-	else if (result < 0) {
+	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
 	}







From dotanb at mellanox.co.il  Mon Jul 18 08:26:33 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Mon, 18 Jul 2005 18:26:33 +0300
Subject: [openib-general] ibv_query_device doesn't fill all the attributes
	with legal/updat ed values
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A21E6@mtlex01.yok.mtl.com>

i have an x86 machine with HCA: 23108.
i installed gen2 with svn revision: 2851.

i noticed that after executing ibv_query_device (user level), not all of the
attributes have a legal/updated values.

in HCA capabilities:

page_size_cap=0x0
hw_ver=0x0
atomic_cap=ATOMIC_NONE
max_mcast_grp=0x0
max_mcast_qp_attach=0x0
max_total_mcast_qp_attach=0x0
max_ah=0x0
max_fmr=0
device_cap_flags=0x00001c76
max_cq=0xff80
max_cqe=0xffff
max_pd=0x8000
max_qp_rd_atom=0x4
max_res_rd_atom=0x0
max_qp_init_rd_atom=0x4

in port properties:
max_mtu=invalid MTU
active_mtu=invalid MTU
max_msg_sz=0x4
max_vl_num=0x0
bad_pkey_cntr=0xda9f9fbc
subnet_timeout=0x0

(i compared those value with the output from vstat that i executed over the
VAPI driver and with the actual properties of the port).


maybe there are some more attributes that don't have a legal value ...


is it a bug?

Dotan Barak
Software Verification Engineer
Mellanox Technologies LTD
mailto:dotanb at mellanox.co.il
Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
P.O. Box 86 Yokneam 20692 ISRAEL.
Home: +972-4-8289408  Cell: 052-4222383

[ May the fork be with you ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/548ffc3c/attachment.html>

From robert.j.woodruff at intel.com  Mon Jul 18 08:42:33 2005
From: robert.j.woodruff at intel.com (Bob Woodruff)
Date: Mon, 18 Jul 2005 08:42:33 -0700
Subject: [openib-general] mem register failing
In-Reply-To: <42DB5505.5060306@delusion.com>
Message-ID: <ORSMSX408FRaqbC8wSA00000006@orsmsx408.amr.corp.intel.com>

>Jul 18 11:50:51 e02 kernel:  MOSAL(1): ddt[25303]: mosal_iobuf.c:869:
check_perm: 
>Found a hole in VM-space at va=0x641000 (given buffer spans from 0x502CB0
to >0x70FCAF)
>Jul 18 11:50:51 e02 kernel:  MOSAL(1): mosal_iobuf.c[239]: 
>MOSAL_iobuf_register_Rsmp_ab8db1c2: va=0x502CB0, size=0x20D000, requested 
>permissions=0x3- Given address doesn't match process address space
>Jul 18 11:50:51 e02 kernel:  VIPKL(1): mmu.c[228]: make_iobuf:
MOSAL_iobuf_register >failed: va=0x502CB0size=0x20D000
>Jul 18 11:50:51 e02 kernel:  VIPKL(1): [create_mr] MM_bld_hh_mr failed
>(-252:VAPI_ENOMEM)

These look like messages from the Mellanox VAPI driver,
not the MTHCA driver from openib.org. Sorry for the confusion.
I think that Mellanox is telling people that IBGD Gold "is" the
openib code, when in fact it is not and I wish they would stop
doing that as it simply adds to the confustion.
I believe IBGD Gold is based on some early code
donated to openib.org by Topspin. However, most, if not all of
that code was not used in the official openib.org trunk or
submitted to kernel.org.  VAPI was not used at all. 

I suggest you contact Mellanox directly for questions about these
errors.  

woody

_



From Sujal at Mellanox.com  Mon Jul 18 10:33:47 2005
From: Sujal at Mellanox.com (Sujal Das)
Date: Mon, 18 Jul 2005 10:33:47 -0700
Subject: [openib-general] OpenIB Workshop Linux Track: Proposed Agenda
Message-ID: <25AE7F432672D511B8DC00B0D0DF11DA0553A073@MTIEX01>

Hello all,

Here is a proposed agenda, list of speakers/presenters for the OpenIB
Workshop Linux Track.

Proposed speakers/presenters: please acknowledge participation and commit to
providing a brief abstract on your topic by August 1st. Let me know if the
date does not work.  Please see below for a sample of an abstract (extracted
from the last OpenIB Workshop).

Proposed content for tracks L1 - L10:
- Status of software - challenges and successes
- Future roadmap and plans for upcoming release candidates and their quality
assurance
- Kernel and/or OS distribution inclusion plans if any

Track L1:
	10-30-10.50 AM
	Core  
	Speaker(s): Hal Rosenstock - Voltaire, Roland Dreier - Cisco

Track L2
	10.50-11.10 AM
	iWARP Integration Issues Update
	Speaker: Clem Cole - Ammasso

Track L3
	11.10-11.30 AM
	kDAPL
	Speaker(s): Arkady Kanevsky, James Lentini - Network Appliance  

Track L4
	11.30.11.50 AM
	uDAPL
	Speakers: Arkady Kanevsky - Network appliance, A. Davis - Intel

Track L5
	11.50-12.10
	L5 - SDP
	Speaker: Libor

12.10-13.00: LUNCH

Track L6
	13.00-13.20
	MPI OSU
	Speaker(s): Dr. Panda - OSU, Gil Boch - Mellanox

Track L7
	13.20-14.00
	Open MPI 
	Speaker: [PLEASE ADVICE ON WHO THIS SHOULD BE]

Track L8
	14.00-14.20
	SRP Initiator
	Speaker(s): Roland - Cisco, Vu Pham - Mellanox

Track 9
	14.20-14.40
	iSER
	Speaker: Hal - Voltaire

Track 10
	14.40-15.00
	NFS RDMA Server and Client
	Speaker(s): Boris Shpolyansky, Vu Pham - Mellanox

15.00-15.30: BREAK

Track 11
	15.30-16.00
	Release and Test Processes
	Speaker: Hal - Voltaire, Amit Grig / Sujal Das - Mellanox

Track 12
	16.00 - 16.45
	Distribution Channels 
	Speaker: [NEED A SPEAKER FROM REDHAT AND NOVELL - PLEASE ASSIST]


Abstract Sample for Reference Only 
----------------------------------

(IMPORTANT: Please use "proposed content for tracks L1 - L10" mentioned
above to develop your abstract")

iSER is a new IETF extension to iSCSI allowing it to run over native RDMA
transport replacing the slow TCP transport. The session will describe the
new iSCSI Datamover layer, the iSER transport module and its relations with
iSCSI and InfiniBand, the standard mapping from iSCSI/iSER to FC and to
iSCSI/TCP. The session will also review plans for upcoming releases of iSER,
and discussions on progresses made in the areas of incorporating iSER into
the iSCSI-Linux project, the latest iSER RFC work in the IETF and iSER
integration with upcoming kernels and OS distributions.

 


	






L12 - Distribution Channels 
Speaker: [NEED A SPEAKER FROM REDHAT OR NOVELL - PLEASE ASSIST]
	
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050718/6ce2a5ea/attachment.html>

From gshipman at lanl.gov  Mon Jul 18 15:23:45 2005
From: gshipman at lanl.gov (Galen Shipman)
Date: Mon, 18 Jul 2005 16:23:45 -0600
Subject: [openib-general] IBV_WC_REM_INV_REQ_ERR and IBV_WC_WR_FLUSH_ERR on
	rdma write
Message-ID: <758ec31b04dab113b3b5829a353efaf6@lanl.gov>

Hello,

After posting an RDMA write I then poll the completion queue, at which 
time I get an IBV_WC_REM_INV_REQ_ERR error followed by a bunch of  
IBV_WC_WR_FLUSH_ERR errors.


The peer is seeing a bunch of  IBV_WC_WR_FLUSH_ERR errors on the 
polling of the completion queue.

I am not seeing this behavior when I use the same queue pair to SEND 
only the RDMA causes this.

I have verified the send descriptors local and remote address as well 
as the rkey.

Any ideas on why these errors could occur?

Thanks,

Galen



From rolandd at cisco.com  Mon Jul 18 17:09:26 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 18 Jul 2005 17:09:26 -0700
Subject: [openib-general] Re: calling to ibv_create_qp with big number in
 qp_init_attr.cap.max_ inline_data never return
In-Reply-To: <20050718123610.GA5758@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 18 Jul 2005 15:36:10 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>
	<523bqc5mtk.fsf@topspin.com> <20050718123610.GA5758@mellanox.co.il>
Message-ID: <52ll434pxl.fsf@topspin.com>

    Michael> I think the best way is to check qp parameters for sanity
    Michael> in mthca_create_qp.

Makes sense.

 - R.


From rolandd at cisco.com  Mon Jul 18 17:12:15 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 18 Jul 2005 17:12:15 -0700
Subject: [openib-general] Re: [PATCH] fork support
In-Reply-To: <20050718125610.GB5758@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 18 Jul 2005 15:56:10 +0300")
References: <20050717172719.GA29577@mellanox.co.il>
	<52d5pg5n0b.fsf@topspin.com> <20050718125610.GB5758@mellanox.co.il>
Message-ID: <52hder4psw.fsf@topspin.com>

    Michael> Thanks! Would it be easier, or harder, if I post this at lkml?

Probably easier...

    Roland> - This patch allows userspace to remove VM_DONTCOPY from
    Roland> vmas even if the kernel set it.

    Michael> Well, we'll need it to deregister stuff, wont we?

Right -- I wasn't very clear about the problem.  It's fine for
userspace to remove the VM_DONTCOPY if it was set by userspace.  But
the kernel may have some reason for setting VM_DONTCOPY on a vma, and
may not want userspace be able to remove the flag.

    Michael> I went for simplicity. Do you think the above a good
    Michael> idea?

Yes.

    Michael> Most of it is duplication, but some architectures define
    Michael> PROT_ flags slightly differently. Easy to do ... let me
    Michael> know what do you think.

Probably a future cleanup on top of this.

 - R.


From rolandd at cisco.com  Mon Jul 18 17:32:02 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 18 Jul 2005 17:32:02 -0700
Subject: [openib-general] IBV_WC_REM_INV_REQ_ERR and IBV_WC_WR_FLUSH_ERR
	on rdma write
In-Reply-To: <758ec31b04dab113b3b5829a353efaf6@lanl.gov> (Galen Shipman's
	message of "Mon, 18 Jul 2005 16:23:45 -0600")
References: <758ec31b04dab113b3b5829a353efaf6@lanl.gov>
Message-ID: <521x5v4ovx.fsf@topspin.com>

    Galen> After posting an RDMA write I then poll the completion
    Galen> queue, at which time I get an IBV_WC_REM_INV_REQ_ERR error
    Galen> followed by a bunch of IBV_WC_WR_FLUSH_ERR errors.

    Galen> The peer is seeing a bunch of IBV_WC_WR_FLUSH_ERR errors on
    Galen> the polling of the completion queue.

According to the IBA spec, a remote invalid request error is generated
when the responder detects an invalid message.  Possible causes are
"operation is not supported by this receive queue, insufficient
buffering to receive a new RDMA or Atomic Operation request...."

Have you set IBV_ACCESS_REMOTE_WRITE and/or IBV_ACCESS_REMOTE_READ in
your qp_attr.qp_access_flags?  How about max_dest_rd_atomic (number of
RDMAs outstanding as a responder) and max_rd_atomic (number of RDMAs
outstanding as an initiator)?

Flush errors will be generated for all remaining work requests after a
queue pair transitions to the error state.

 - R.


From gshipman at lanl.gov  Mon Jul 18 19:17:50 2005
From: gshipman at lanl.gov (Galen Mark Shipman)
Date: Mon, 18 Jul 2005 20:17:50 -0600 (MDT)
Subject: [openib-general] IBV_WC_REM_INV_REQ_ERR and 
	IBV_WC_WR_FLUSH_ERR on rdma write
In-Reply-To: <521x5v4ovx.fsf@topspin.com>
References: <758ec31b04dab113b3b5829a353efaf6@lanl.gov>
	<521x5v4ovx.fsf@topspin.com>
Message-ID: <23517.128.165.0.81.1121739470.squirrel@webmail.lanl.gov>


> Have you set IBV_ACCESS_REMOTE_WRITE and/or IBV_ACCESS_REMOTE_READ in
> your qp_attr.qp_access_flags?


That was it, I had set qp_attr.qp_access_flags = 0;

Thanks,

Galen



From mst at mellanox.co.il  Mon Jul 18 21:53:59 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 19 Jul 2005 07:53:59 +0300
Subject: [openib-general] Re: [PATCH] fork support
In-Reply-To: <52hder4psw.fsf@topspin.com>
References: <20050717172719.GA29577@mellanox.co.il>
	<52d5pg5n0b.fsf@topspin.com> <20050718125610.GB5758@mellanox.co.il>
	<52hder4psw.fsf@topspin.com>
Message-ID: <20050719045359.GA14370@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
>     Roland> - This patch allows userspace to remove VM_DONTCOPY from
>     Roland> vmas even if the kernel set it.
> 
>     Michael> Well, we'll need it to deregister stuff, wont we?
> 
> Right -- I wasn't very clear about the problem.  It's fine for
> userspace to remove the VM_DONTCOPY if it was set by userspace.  But
> the kernel may have some reason for setting VM_DONTCOPY on a vma, and
> may not want userspace be able to remove the flag.

You were clear enough. I expect this not to be a problem though.

>     Michael> Do you think the above a good idea?
> 
> Yes.

So, instead of the patch as I posted it, we want new flags to mprotect that
will let it touch only one flag at a time, leaving all the rest alone?
Something along the lines of:

#define PROT_SET 0x80000000
#define PROT_CLR 0x40000000

And now:
mprotect(..., PROT_SET | PROT_DONTCOPY)
to set
mprotect(..., PROT_CLR | PROT_DONTCOPY)
to clear

And PROT_DONTCOPY will be editable only when PROT_SET or PROT_CLR is set?

-- 
MST


From baito at daisuki.net  Tue Jul 19 02:31:51 2005
From: baito at daisuki.net (=?ISO-2022-JP?B?GyRCJCpGQCUiJWslUCUkJUg+cEpzGyhC?= NET)
Date: Tue, 19 Jul 2005 18:31:51 +0900
Subject: [openib-general] =?iso-2022-jp?b?GyRCJDMkTjJGJCo8ajdaJEsbKEI=?=
	=?iso-2022-jp?b?GyRCJSIlayVQJSQlSCQ3JF4kNyRnJCYbKEI=?=


突然のお知らせで大変申し訳ございません。

今現在の自分にあなたは満足してますか？？

お金もない、時間もない、運もない、そんな方のために
　
当社は期間限定の稼ぐ秘密を教えます！！

日本全国のパチンコ・パチスロ集客宣伝部員大募集してます。

エリアによって部員数の制限はあります！

情報料・登録料一切無料！あなたの負担は一切なし！！

詳しくはこちらから！！

↓↓
　　
http://aa112233.s149.xrea.com/

↑↑
お気軽にお問い合わせください！！

まったくギャンブルに興味のない方、昔は好きだった方

新たに始めてみてはどうですか？？

でも負ける、負けそう、お金がもったいない、その前に
これは簡単なお仕事です！！　

稼いで当然のアルバイトです！！あなたのヤル気次第！！

リスク一切無し！！そろそろ自分を変えてみてはどうですか？？

中途半端な気持ちは捨ててください！！お約束します！！

 詳しくはこちらから！！
↓↓
　　
http://aa112233.s149.xrea.com/

↑↑
お気軽にお問い合わせください！！




From baito at daisuki.net  Tue Jul 19 02:33:36 2005
From: baito at daisuki.net (=?ISO-2022-JP?B?GyRCJCpGQCUiJWslUCUkJUg+cEpzGyhC?= NET)
Date: Tue, 19 Jul 2005 18:33:36 +0900
Subject: [openib-general] =?iso-2022-jp?b?GyRCJDMkTjJGJCo8ajdaJEsbKEI=?=
	=?iso-2022-jp?b?GyRCJSIlayVQJSQlSCQ3JF4kNyRnJCYbKEI=?=


突然のお知らせで大変申し訳ございません。

今現在の自分にあなたは満足してますか？？

お金もない、時間もない、運もない、そんな方のために
　
当社は期間限定の稼ぐ秘密を教えます！！

日本全国のパチンコ・パチスロ集客宣伝部員大募集してます。

エリアによって部員数の制限はあります！

情報料・登録料一切無料！あなたの負担は一切なし！！

詳しくはこちらから！！

↓↓
　　
http://aa112233.s149.xrea.com/

↑↑
お気軽にお問い合わせください！！

まったくギャンブルに興味のない方、昔は好きだった方

新たに始めてみてはどうですか？？

でも負ける、負けそう、お金がもったいない、その前に
これは簡単なお仕事です！！　

稼いで当然のアルバイトです！！あなたのヤル気次第！！

リスク一切無し！！そろそろ自分を変えてみてはどうですか？？

中途半端な気持ちは捨ててください！！お約束します！！

 詳しくはこちらから！！
↓↓
　　
http://aa112233.s149.xrea.com/

↑↑
お気軽にお問い合わせください！！




From mlleinin at hpcn.ca.sandia.gov  Tue Jul 19 06:01:54 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Tue, 19 Jul 2005 06:01:54 -0700
Subject: [openib-general] OpenIB Workshop Linux Track: Proposed Agenda
In-Reply-To: <25AE7F432672D511B8DC00B0D0DF11DA0553A073@MTIEX01>
References: <25AE7F432672D511B8DC00B0D0DF11DA0553A073@MTIEX01>
Message-ID: <1121778114.11021.533.camel@localhost>

On Mon, 2005-07-18 at 10:33 -0700, Sujal Das wrote:

> 
> Track L7 
>         13.20-14.00 
>         Open MPI  
>         Speaker: [PLEASE ADVICE ON WHO THIS SHOULD BE]
> 
   I'd suggest Tim Woodall (LANL) and/or Jeff Squyres (UIndiana).

   - Matt




From rolandd at cisco.com  Tue Jul 19 06:53:33 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 19 Jul 2005 06:53:33 -0700
Subject: [openib-general] Re: [PATCH] fork support
In-Reply-To: <20050719045359.GA14370@mellanox.co.il> (Michael S. Tsirkin's
	message of "Tue, 19 Jul 2005 07:53:59 +0300")
References: <20050717172719.GA29577@mellanox.co.il>
	<52d5pg5n0b.fsf@topspin.com> <20050718125610.GB5758@mellanox.co.il>
	<52hder4psw.fsf@topspin.com> <20050719045359.GA14370@mellanox.co.il>
Message-ID: <52ll423ns2.fsf@topspin.com>

    Michael> So, instead of the patch as I posted it, we want new
    Michael> flags to mprotect that will let it touch only one flag at
    Michael> a time, leaving all the rest alone?  Something along the
    Michael> lines of:

    Michael> #define PROT_SET 0x80000000 #define PROT_CLR 0x40000000

    Michael> And now: mprotect(..., PROT_SET | PROT_DONTCOPY) to set
    Michael> mprotect(..., PROT_CLR | PROT_DONTCOPY) to clear
    Michael> And PROT_DONTCOPY will be editable only when PROT_SET or
    Michael> PROT_CLR is set?

Actually I thought about this some more.  I think the current
interface is fine (and probably specified by SUS or something like
that).  If someone is doing multiple mprotects, they should know
what's there.

 - R.


From dotanb at mellanox.co.il  Tue Jul 19 07:04:52 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Tue, 19 Jul 2005 17:04:52 +0300
Subject: [openib-general] where can i find functions that "convert"
	enumerated values to st ring?
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A234A@mtlex01.yok.mtl.com>

does the gen2 driver have functions that convert enumerated values to
strings?

for example, something like this:

static const char *atomic_cap_str(enum ibv_atomic_cap atom_cap)
{
        switch (atom_cap) {
        case IBV_ATOMIC_NONE:
                return "ATOMIC_NONE";
        case IBV_ATOMIC_HCA:
                return "ATOMIC_HCA";
        case IBV_ATOMIC_GLOB:
                return "ATOMIC_GLOB";
        default:
                return "Illegal atomic capability";
        }
}


Dotan Barak
Software Verification Engineer
Mellanox Technologies LTD
mailto:dotanb at mellanox.co.il
Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
P.O. Box 86 Yokneam 20692 ISRAEL.
Home: +972-4-8289408  Cell: 052-4222383

[ May the fork be with you ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050719/94b00aa0/attachment.html>

From rolandd at cisco.com  Tue Jul 19 07:08:07 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 19 Jul 2005 07:08:07 -0700
Subject: [openib-general] Re: where can i find functions that "convert"
 enumerated values to st ring?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A234A@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Tue, 19 Jul 2005 17:04:52 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A234A@mtlex01.yok.mtl.com>
Message-ID: <52zmsi28jc.fsf@topspin.com>

    Dotan> does the gen2 driver have functions that convert enumerated
    Dotan> values to strings?

No, I haven't written anything like that.

 - R.


From dotanb at mellanox.co.il  Tue Jul 19 07:11:04 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Tue, 19 Jul 2005 17:11:04 +0300
Subject: [openib-general] RE: where can i find functions that "convert"
	enumerated values t o string?
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A2350@mtlex01.yok.mtl.com>

do you plan to write it?

-----Original Message-----
From: Roland Dreier [mailto:rolandd at cisco.com]
Sent: Tuesday, July 19, 2005 5:08 PM
To: Dotan Barak
Cc: openib-general at openib.org
Subject: Re: where can i find functions that "convert" enumerated values
to st ring?


    Dotan> does the gen2 driver have functions that convert enumerated
    Dotan> values to strings?

No, I haven't written anything like that.

 - R.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050719/e957ab65/attachment.html>

From rolandd at cisco.com  Tue Jul 19 07:30:52 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 19 Jul 2005 07:30:52 -0700
Subject: [openib-general] Re: where can i find functions that "convert"
 enumerated values t o string?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A2350@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Tue, 19 Jul 2005 17:11:04 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A2350@mtlex01.yok.mtl.com>
Message-ID: <52u0iq27hf.fsf@topspin.com>

    Dotan> do you plan to write it?

Not any time soon -- it seems a low priority to me.

 - R.


From dotanb at mellanox.co.il  Tue Jul 19 08:04:06 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Tue, 19 Jul 2005 18:04:06 +0300
Subject: [openib-general] RE: where can i find functions that "convert"
	enumerated values t o string?
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A236D@mtlex01.yok.mtl.com>



> -----Original Message-----
> From: Roland Dreier [mailto:rolandd at cisco.com]
> Sent: Tuesday, July 19, 2005 5:31 PM
> To: Dotan Barak
> Cc: openib-general at openib.org
> Subject: Re: where can i find functions that "convert" 
> enumerated values
> t o string?
> 
> 
>     Dotan> do you plan to write it?
> 
> Not any time soon -- it seems a low priority to me.
> 
>  - R.
> 


If I'll send you functions that does the work, will you add them to the
driver?

Dotan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050719/752bddf4/attachment.html>

From damato at psc.edu  Tue Jul 19 09:01:57 2005
From: damato at psc.edu (Joe Damato)
Date: Tue, 19 Jul 2005 12:01:57 -0400
Subject: [openib-general] unable to run ibv_uc_pingpong and other example
	binaries
Message-ID: <42DD23F5.3060702@psc.edu>

Hello,
   
    First I would like to apologize for the long email.
   
    I am currently using the gen2 revision 2734 openib source and kernel 
2.6.11. I have compiled and installed libibverbs, libmthca and libibcm. 
I have also created the /dev/infiniband devices, and I have my card 
flashed to the latest firmware version.

[root at frodo examples]# pwd
/cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
[root at frodo examples]# ./ibv_uc_pingpong
libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
No IB devices found
[root at frodo examples]# ./ibv_rc_pingpong
libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
No IB devices found

but....

[root at frodo examples]# cd /dev/
[root at frodo dev]# ls infiniband/
issm0  issm1  ucm  umad0  umad1  uverbs0  uverbs1
[root at frodo dev]# ls -al infiniband/
total 0
drw-rw-rw-  2 root root      180 Jul 19 11:36 .
drwxr-xr-x  8 root root    13940 Jul 19 11:35 ..
crw-rw-rw-  1 root root 231,  64 Jul 19 11:35 issm0
crw-rw-rw-  1 root root 231,  65 Jul 19 11:35 issm1
crw-rw-rw-  1 root root 231, 255 Jul 19 11:36 ucm
crw-rw-rw-  1 root root 231,   0 Jul 19 11:35 umad0
crw-rw-rw-  1 root root 231,   1 Jul 19 11:35 umad1
crw-rw-rw-  1 root root 231, 192 Jul 19 11:36 uverbs0
crw-rw-rw-  1 root root 231, 193 Jul 19 11:36 uverbs1

also, I don't know if this helps but:

[root at frodo ~]# /usr/local/ib/bin/ibstat
CA 'mthca0'
        CA type: MT23108
        Number of ports: 2
        Firmware version: 3.3.2
        Hardware version: a1
        Node GUID: 0x0002c9010976c400
        System image GUID: 0x0002c9010976c403
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x00500a6a
                Port GUID: 0x0002c9010976c401
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 2
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00500a68
                Port GUID: 0x0002c9010976c402
[root at frodo ~]#

and

[root at frodo ~]# ps aux | grep opensm
root      2929  0.0  0.0  82368  1372 pts/0    Sl   11:35   0:00 
/usr/local/ib/bin/opensm

and

[root at frodo ~]# dmesg | grep mthca
ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
ib_mthca: Initializing  (0000:14:00.0)


Any suggestions on how to resolve this issue so I can test the IB set 
up? I currently have 8 other machines with IB, however when trying to 
run ibv_rc_pingpong they all report the same error  (each of the 8 
systems are running the same kernel and all cards are flashed to the 
latest firmware):

ost0# ./ibv_rc_pingpong
Couldn't get local LID

but when checking ibstat it shows that Port 2 is active

ost0# /usr/local/ib/bin/ibstat
CA 'mthca0'
        CA type: MT25208 (MT23108 compat mode)
        Number of ports: 2
        Firmware version: 4.6.2
        Hardware version: a0
        Node GUID: 0x0002c90200003f40
        System image GUID: 0x0002c90200003f43
        Port 1:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00510a68
                Port GUID: 0x0002c90200003f41
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 5
                LMC: 0
                SM lid: 1
                Capability mask: 0x00510a68
                Port GUID: 0x0002c90200003f42



any suggestions would be greatly appreciated.

Thanks,
Joe Damato


From mst at mellanox.co.il  Tue Jul 19 09:14:57 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 19 Jul 2005 19:14:57 +0300
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <42DD23F5.3060702@psc.edu>
References: <42DD23F5.3060702@psc.edu>
Message-ID: <20050719161457.GA15839@mellanox.co.il>

Quoting r. Joe Damato <damato at psc.edu>:
> Subject: unable to run ibv_uc_pingpong and other example binaries
> 
> Hello,
>   
>    First I would like to apologize for the long email.
>   
>    I am currently using the gen2 revision 2734 openib source and kernel 
> 2.6.11. I have compiled and installed libibverbs, libmthca and libibcm. 
> I have also created the /dev/infiniband devices, and I have my card 
> flashed to the latest firmware version.
> 
> [root at frodo examples]# pwd
> /cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
> [root at frodo examples]# ./ibv_uc_pingpong

You want to run .libs/ibv_uc_pingpong I think.
The one you are running is a shell script.

> libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found
> [root at frodo examples]# ./ibv_rc_pingpong
> libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found

Hmm. I didnt know libibverbs depends on libucm. 
Can you strace it?

> but....
> 
> [root at frodo examples]# cd /dev/
> [root at frodo dev]# ls infiniband/
> issm0  issm1  ucm  umad0  umad1  uverbs0  uverbs1
> [root at frodo dev]# ls -al infiniband/
> total 0
> drw-rw-rw-  2 root root      180 Jul 19 11:36 .
> drwxr-xr-x  8 root root    13940 Jul 19 11:35 ..
> crw-rw-rw-  1 root root 231,  64 Jul 19 11:35 issm0
> crw-rw-rw-  1 root root 231,  65 Jul 19 11:35 issm1
> crw-rw-rw-  1 root root 231, 255 Jul 19 11:36 ucm
> crw-rw-rw-  1 root root 231,   0 Jul 19 11:35 umad0
> crw-rw-rw-  1 root root 231,   1 Jul 19 11:35 umad1
> crw-rw-rw-  1 root root 231, 192 Jul 19 11:36 uverbs0
> crw-rw-rw-  1 root root 231, 193 Jul 19 11:36 uverbs1

BTW, you probably dont want umad and issm world-accessible, it can be
a security hole: these are management interfaces.

> [...]
> I currently have 8 other machines with IB, however when trying to 
> run ibv_rc_pingpong they all report the same error  (each of the 8 
> systems are running the same kernel and all cards are flashed to the 
> latest firmware):
> 
> ost0# ./ibv_rc_pingpong
> Couldn't get local LID
> 
> but when checking ibstat it shows that Port 2 is active

I think these tests assume that since local LID is 0 sm isnt running.
Try running opensm on one of the nodes.

-- 
MST


From damato at psc.edu  Tue Jul 19 09:28:06 2005
From: damato at psc.edu (Joe Damato)
Date: Tue, 19 Jul 2005 12:28:06 -0400
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <20050719161457.GA15839@mellanox.co.il>
References: <42DD23F5.3060702@psc.edu> <20050719161457.GA15839@mellanox.co.il>
Message-ID: <42DD2A16.3080800@psc.edu>

Michael S. Tsirkin wrote:

>Quoting r. Joe Damato <damato at psc.edu>:
>  
>
>>Subject: unable to run ibv_uc_pingpong and other example binaries
>>
>>Hello,
>>  
>>   First I would like to apologize for the long email.
>>  
>>   I am currently using the gen2 revision 2734 openib source and kernel 
>>2.6.11. I have compiled and installed libibverbs, libmthca and libibcm. 
>>I have also created the /dev/infiniband devices, and I have my card 
>>flashed to the latest firmware version.
>>
>>[root at frodo examples]# pwd
>>/cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
>>[root at frodo examples]# ./ibv_uc_pingpong
>>    
>>
>
>You want to run .libs/ibv_uc_pingpong I think.
>The one you are running is a shell script.
>
>  
>
OK, just tried it, same problem:
[root at frodo examples]# pwd
/cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
[root at frodo examples]# .libs/ibv_uc_pingpong
libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
No IB devices found

>>libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
>>libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
>>No IB devices found
>>[root at frodo examples]# ./ibv_rc_pingpong
>>libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
>>libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
>>No IB devices found
>>    
>>
>
>Hmm. I didnt know libibverbs depends on libucm. 
>Can you strace it?
>
>  
>
sure. strace:
[root at frodo examples]# strace .libs/ibv_uc_pingpong
execve(".libs/ibv_uc_pingpong", [".libs/ibv_uc_pingpong"], [/* 22 vars 
*/]) = 0
brk(0) = 0x504000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0x2aaaaaaab000
uname({sys="Linux", node="frodo.psc.edu", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/x86_64/libibverbs.so.1", O_RDONLY) = -1 ENOENT 
(No such file or directory)
open("/usr/local/lib/tls/libibverbs.so.1", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/x86_64/libibverbs.so.1", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/libibverbs.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\33\0\0"..., 
640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=91160, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0x2aaaaaaac000
mmap(NULL, 1066272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 
0) = 0x2aaaaaaad000
mprotect(0x2aaaaaab2000, 1045792, PROT_NONE) = 0
mmap(0x2aaaaabb1000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x2aaaaabb1000
close(3) = 0
open("/usr/local/lib/tls/x86_64/libsysfs.so.1", O_RDONLY) = -1 ENOENT 
(No such file or directory)
open("/usr/local/lib/tls/libsysfs.so.1", O_RDONLY) = -1 ENOENT (No such 
file or directory)
open("/usr/local/lib/x86_64/libsysfs.so.1", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/libsysfs.so.1", O_RDONLY) = -1 ENOENT (No such file 
or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=40696, ...}) = 0
mmap(NULL, 40696, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aaaaabb2000
close(3) = 0
open("/usr/lib64/libsysfs.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300/`\257"..., 
640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=52728, ...}) = 0
mmap(0x33af600000, 1097280, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x33af600000
mprotect(0x33af60c000, 1048128, PROT_NONE) = 0
mmap(0x33af70b000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb000) = 0x33af70b000
close(3) = 0
open("/usr/local/lib/tls/x86_64/libpthread.so.0", O_RDONLY) = -1 ENOENT 
(No such file or directory)
open("/usr/local/lib/tls/libpthread.so.0", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/x86_64/libpthread.so.0", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such 
file or directory)
open("/lib64/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0V\240\341"..., 
640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=115080, ...}) = 0
mmap(0x34e1a00000, 1131384, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e1a00000
mprotect(0x34e1a0f000, 1069944, PROT_NONE) = 0
mmap(0x34e1b0f000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x34e1b0f000
mmap(0x34e1b11000, 13176, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x34e1b11000
close(3) = 0
open("/usr/local/lib/tls/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/tls/libdl.so.2", O_RDONLY) = -1 ENOENT (No such 
file or directory)
open("/usr/local/lib/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such 
file or directory)
open("/usr/local/lib/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or 
directory)
open("/lib64/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\17 "..., 
640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=23184, ...}) = 0
mmap(0x34e1200000, 1061096, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e1200000
mprotect(0x34e1202000, 1052904, PROT_NONE) = 0
mmap(0x34e1302000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x34e1302000
close(3) = 0
open("/usr/local/lib/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/local/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file 
or directory)
open("/usr/local/lib/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such 
file or directory)
open("/usr/local/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or 
directory)
open("/lib64/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\305"..., 
640) = 640
lseek(3, 624, SEEK_SET) = 624
read(3, "\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\4\0\0\0"..., 32) 
= 32
fstat(3, {st_mode=S_IFREG|0755, st_size=1553232, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0x2aaaaabbc000
mmap(0x34e0f00000, 2321544, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34e0f00000
mprotect(0x34e102e000, 1084552, PROT_NONE) = 0
mmap(0x34e112d000, 24576, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12d000) = 0x34e112d000
mmap(0x34e1133000, 15496, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x34e1133000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0x2aaaaabbd000
arch_prctl(ARCH_SET_FS, 0x2aaaaabbd800) = 0
mprotect(0x34e112d000, 16384, PROT_READ) = 0
mprotect(0x34e1302000, 4096, PROT_READ) = 0
mprotect(0x34e1b0f000, 4096, PROT_READ) = 0
mprotect(0x34e0e19000, 4096, PROT_READ) = 0
munmap(0x2aaaaabb2000, 40696) = 0
set_tid_address(0x2aaaaabbd890) = 13505
rt_sigaction(SIGRTMIN, {0x34e1a05190, [], SA_RESTORER|SA_SIGINFO, 
0x34e1a0c380}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x34e1a05210, [], 
SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x34e1a0c380}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
_sysctl({{CTL_KERN, KERN_VERSION}, 2, 0x7ffffffff620, 35, (nil), 0}) = 0
brk(0) = 0x504000
brk(0x525000) = 0x525000
futex(0x34e13030a8, FUTEX_WAKE, 2147483647) = 0
getuid() = 0
geteuid() = 0
open("/usr/local/lib/infiniband", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
getdents64(3, /* 7 entries */, 4096) = 208
getdents64(3, /* 0 entries */, 4096) = 0
close(3) = 0
open("/usr/local/lib/infiniband/libibcm.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\f\0"..., 
640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=42601, ...}) = 0
mmap(NULL, 1058376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 
0) = 0x2aaaaabbe000
mprotect(0x2aaaaabc1000, 1046088, PROT_NONE) = 0
mmap(0x2aaaaacc0000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x2aaaaacc0000
close(3) = 0
open("/dev/infiniband/ucm", O_RDWR) = -1 ENXIO (No such device or address)
write(2, "libucm: Error <-1:6> couldn\'t op"..., 70libucm: Error <-1:6> 
couldn't open IB cm device </dev/infiniband/ucm>
) = 70
munmap(0x2aaaaabbe000, 1058376) = 0
open("/usr/local/lib/infiniband/mthca.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\24\0"..., 
640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=125471, ...}) = 0
mmap(NULL, 1070328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 
0) = 0x2aaaaabbe000
mprotect(0x2aaaaabc3000, 1049848, PROT_NONE) = 0
mmap(0x2aaaaacc3000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x2aaaaacc3000
close(3) = 0
open("/proc/mounts", O_RDONLY) = 3
futex(0x34e11346e0, FUTEX_WAKE, 2147483647) = 0
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0x2aaaaacc4000
read(3, "rootfs / rootfs rw 0 0\n/dev /dev"..., 1024) = 601
close(3) = 0
munmap(0x2aaaaacc4000, 4096) = 0
lstat("/sys/class/infiniband_verbs", 0x7ffffffff580) = -1 ENOENT (No 
such file or directory)
write(2, "libibverbs: Fatal: couldn\'t open"..., 65libibverbs: Fatal: 
couldn't open sysfs class 'infiniband_verbs'.
) = 65
write(2, "No IB devices found\n", 20No IB devices found
) = 20
exit_group(1) = ?

>>but....
>>
>>[root at frodo examples]# cd /dev/
>>[root at frodo dev]# ls infiniband/
>>issm0  issm1  ucm  umad0  umad1  uverbs0  uverbs1
>>[root at frodo dev]# ls -al infiniband/
>>total 0
>>drw-rw-rw-  2 root root      180 Jul 19 11:36 .
>>drwxr-xr-x  8 root root    13940 Jul 19 11:35 ..
>>crw-rw-rw-  1 root root 231,  64 Jul 19 11:35 issm0
>>crw-rw-rw-  1 root root 231,  65 Jul 19 11:35 issm1
>>crw-rw-rw-  1 root root 231, 255 Jul 19 11:36 ucm
>>crw-rw-rw-  1 root root 231,   0 Jul 19 11:35 umad0
>>crw-rw-rw-  1 root root 231,   1 Jul 19 11:35 umad1
>>crw-rw-rw-  1 root root 231, 192 Jul 19 11:36 uverbs0
>>crw-rw-rw-  1 root root 231, 193 Jul 19 11:36 uverbs1
>>    
>>
>
>BTW, you probably dont want umad and issm world-accessible, it can be
>a security hole: these are management interfaces.
>
>  
>
ok -- i'll fix that -- THANKS for the warning.

>>[...]
>>I currently have 8 other machines with IB, however when trying to 
>>run ibv_rc_pingpong they all report the same error  (each of the 8 
>>systems are running the same kernel and all cards are flashed to the 
>>latest firmware):
>>
>>ost0# ./ibv_rc_pingpong
>>Couldn't get local LID
>>
>>but when checking ibstat it shows that Port 2 is active
>>    
>>
>
>I think these tests assume that since local LID is 0 sm isnt running.
>Try running opensm on one of the nodes.
>
>  
>
opensm is running on frodo which is connected to the same IB switch as 
ost0 (all nodes are running on the same switch) so opensm is ALREADY 
running for this IB set up, but just for the heck of it:


[root at ost0 examples]$ /usr/local/ib/bin/opensm
-------------------------------------------------
OpenSM Rev:openib-1.0.0
Command Line Arguments:
Log File: /var/log/osm.log
-------------------------------------------------
using default guid 0x2c90200003f42

Suspended
[root at ost0 examples]$ bg
[1] /usr/local/ib/bin/opensm &
[root at ost0 examples]$ ./ibv_uc_pingpong
Couldn't get local LID


Any other suggestions?

Thanks -
Joe Damato




From mst at mellanox.co.il  Tue Jul 19 09:49:26 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 19 Jul 2005 19:49:26 +0300
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <42DD2A16.3080800@psc.edu>
References: <42DD23F5.3060702@psc.edu> <20050719161457.GA15839@mellanox.co.il>
	<42DD2A16.3080800@psc.edu>
Message-ID: <20050719164925.GB16116@mellanox.co.il>

Quoting r. Joe Damato <damato at psc.edu>:
> Subject: Re: unable to run ibv_uc_pingpong and other example binaries
> OK, just tried it, same problem:
> [root at frodo examples]# pwd
> /cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
> [root at frodo examples]# .libs/ibv_uc_pingpong
> libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>

OK, this one is because you installed libibcm into 
/usr/local/lib/infiniband which is a plugin directory instead if
/usr/local/lib, so it goes.

> open("/usr/local/lib/infiniband/libibcm.so", O_RDONLY) = 3

Simply remove /usr/local/lib/infiniband/libucm.so
and it will go away.

Longer term I think libucm config script must be fixed: libraries
go to /usr/local/lib by default. Hal?

Hal, I also think it may not be such a good idea for a library
to start opening files and print things out to standard error
when its loaded.
Rather, a library should have some sane init/cleanup functions.

> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.
> No IB devices found

hmm are uverbs loaded?

lsmod ?

ls /sys/class/infiniband_verbs/ ?

> opensm is running on frodo

do you call a fat slow box sam?

> [root at ost0 examples]$ ./ibv_uc_pingpong
> Couldn't get local LID

Yes but check the local lid in ibstat. Is it still 0?

-- 
MST


From mst at mellanox.co.il  Tue Jul 19 09:55:42 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 19 Jul 2005 19:55:42 +0300
Subject: [openib-general] [PATCH] fork support
Message-ID: <20050719165542.GB16028@mellanox.co.il>

Here's a patch to linux kernel to enable fork() support for
infiniband uverbs (userspace i/o initiator).
Please Cc me with comments.

---

This patch adds PROT_DONTCOPY to mmap and mprotect, to set VM_DONTCOPY on vma.
This is needed for infiniband userspace i/o, where we need to protect against
  - the child process accessing the parent hardware page
  - the parent registered address (on which the driver did get_user_pages)
    getting remapped to another page by COW
One can imagine other uses, e.g. combined with mlock for real-time or security.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/include/asm-ppc64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ppc64/mman.h
+++ linux-2.6.12.2/include/asm-ppc64/mman.h
@@ -15,6 +15,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-cris/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-cris/mman.h
+++ linux-2.6.12.2/include/asm-cris/mman.h
@@ -10,6 +10,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-arm26/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-arm26/mman.h
+++ linux-2.6.12.2/include/asm-arm26/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-alpha/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-alpha/mman.h
+++ linux-2.6.12.2/include/asm-alpha/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-m68k/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-m68k/mman.h
+++ linux-2.6.12.2/include/asm-m68k/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-mips/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-mips/mman.h
+++ linux-2.6.12.2/include/asm-mips/mman.h
@@ -22,6 +22,7 @@
 #define PROT_SEM	0x10		/* page may be used for atomic ops */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 /*
  * Flags for mmap
Index: linux-2.6.12.2/include/asm-sparc64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sparc64/mman.h
+++ linux-2.6.12.2/include/asm-sparc64/mman.h
@@ -11,6 +11,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-v850/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-v850/mman.h
+++ linux-2.6.12.2/include/asm-v850/mman.h
@@ -7,6 +7,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-s390/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-s390/mman.h
+++ linux-2.6.12.2/include/asm-s390/mman.h
@@ -16,6 +16,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-parisc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-parisc/mman.h
+++ linux-2.6.12.2/include/asm-parisc/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-ppc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ppc/mman.h
+++ linux-2.6.12.2/include/asm-ppc/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-i386/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-i386/mman.h
+++ linux-2.6.12.2/include/asm-i386/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-sh/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sh/mman.h
+++ linux-2.6.12.2/include/asm-sh/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-x86_64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-x86_64/mman.h
+++ linux-2.6.12.2/include/asm-x86_64/mman.h
@@ -8,6 +8,7 @@
 #define PROT_SEM	0x8
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-ia64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ia64/mman.h
+++ linux-2.6.12.2/include/asm-ia64/mman.h
@@ -15,6 +15,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-sparc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sparc/mman.h
+++ linux-2.6.12.2/include/asm-sparc/mman.h
@@ -11,6 +11,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-m32r/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-m32r/mman.h
+++ linux-2.6.12.2/include/asm-m32r/mman.h
@@ -10,6 +10,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-frv/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-frv/mman.h
+++ linux-2.6.12.2/include/asm-frv/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/linux/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/linux/mman.h
+++ linux-2.6.12.2/include/linux/mman.h
@@ -47,9 +47,10 @@ static inline void vm_unacct_memory(long
 static inline unsigned long
 calc_vm_prot_bits(unsigned long prot)
 {
-	return _calc_vm_trans(prot, PROT_READ,  VM_READ ) |
-	       _calc_vm_trans(prot, PROT_WRITE, VM_WRITE) |
-	       _calc_vm_trans(prot, PROT_EXEC,  VM_EXEC );
+	return _calc_vm_trans(prot, PROT_READ,     VM_READ ) |
+	       _calc_vm_trans(prot, PROT_WRITE,    VM_WRITE) |
+	       _calc_vm_trans(prot, PROT_EXEC,     VM_EXEC ) |
+	       _calc_vm_trans(prot, PROT_DONTCOPY, VM_DONTCOPY );
 }
 
 /*
Index: linux-2.6.12.2/include/asm-h8300/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-h8300/mman.h
+++ linux-2.6.12.2/include/asm-h8300/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-arm/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-arm/mman.h
+++ linux-2.6.12.2/include/asm-arm/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/mm/mprotect.c
===================================================================
--- linux-2.6.12.2.orig/mm/mprotect.c
+++ linux-2.6.12.2/mm/mprotect.c
@@ -196,7 +196,7 @@ sys_mprotect(unsigned long start, size_t
 	end = start + len;
 	if (end <= start)
 		return -ENOMEM;
-	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM))
+	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM | PROT_DONTCOPY))
 		return -EINVAL;
 
 	reqprot = prot;
@@ -246,7 +246,7 @@ sys_mprotect(unsigned long start, size_t
 			goto out;
 		}
 
-		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC));
+		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC | VM_DONTCOPY));
 
 		if ((newflags & ~(newflags >> 4)) & 0xf) {
 			error = -EACCES;
Index: linux-2.6.12.2/mm/mmap.c
===================================================================
--- linux-2.6.12.2.orig/mm/mmap.c
+++ linux-2.6.12.2/mm/mmap.c
@@ -792,8 +792,8 @@ struct anon_vma *find_mergeable_anon_vma
 	 * Neither mlock nor madvise tries to remerge at present,
 	 * so leave their flags as obstructing a merge.
 	 */
-	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC);
-	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC);
+	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
+	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
 
 	if (near->anon_vma && vma->vm_end == near->vm_start &&
  			mpol_equal(vma_policy(vma), vma_policy(near)) &&
@@ -814,8 +814,8 @@ try_prev:
 	if (!near)
 		goto none;
 
-	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC);
-	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC);
+	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
+	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
 
 	if (near->anon_vma && near->vm_end == vma->vm_start &&
   			mpol_equal(vma_policy(near), vma_policy(vma)) &&

-- 
MST


From halr at voltaire.com  Tue Jul 19 10:36:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 19 Jul 2005 13:36:00 -0400
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <20050719164925.GB16116@mellanox.co.il>
References: <42DD23F5.3060702@psc.edu>
	<20050719161457.GA15839@mellanox.co.il> <42DD2A16.3080800@psc.edu>
	<20050719164925.GB16116@mellanox.co.il>
Message-ID: <1121794559.4419.8723.camel@hal.voltaire.com>

On Tue, 2005-07-19 at 12:49, Michael S. Tsirkin wrote:
> Quoting r. Joe Damato <damato at psc.edu>:
> > Subject: Re: unable to run ibv_uc_pingpong and other example binaries
> > OK, just tried it, same problem:
> > [root at frodo examples]# pwd
> > /cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
> > [root at frodo examples]# .libs/ibv_uc_pingpong
> > libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
> 
> OK, this one is because you installed libibcm into 
> /usr/local/lib/infiniband which is a plugin directory instead if
> /usr/local/lib, so it goes.
> 
> > open("/usr/local/lib/infiniband/libibcm.so", O_RDONLY) = 3
> 
> Simply remove /usr/local/lib/infiniband/libucm.so
> and it will go away.

It appears to be invoking libucm just fine (the location is not the
issue). The issue is ucm module is not started.

> Longer term I think libucm config script must be fixed: libraries
> go to /usr/local/lib by default. Hal?

As far as I know, the current agreed upon default is
/usr/local/xxx/infiniband where xxx is lib and include.

> Hal, I also think it may not be such a good idea for a library
> to start opening files and print things out to standard error
> when its loaded.
> Rather, a library should have some sane init/cleanup functions.

Due to hot swap or other issues ? I think all the user space libraries
do this with perhaps the exception of libibverbs.

-- Hal



From rolandd at cisco.com  Tue Jul 19 10:47:04 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 19 Jul 2005 10:47:04 -0700
Subject: [openib-general] unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <42DD23F5.3060702@psc.edu> (Joe Damato's message of "Tue, 19
	Jul 2005 12:01:57 -0400")
References: <42DD23F5.3060702@psc.edu>
Message-ID: <52fyua1yef.fsf@topspin.com>

    Joe> [root at frodo examples]# ./ibv_rc_pingpong
    Joe> libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
    Joe> libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.

Why is ibv_rc_pingpong loading libucm?  I don't see where that could
be coming from.

Do you have sysfs mounted?  What do you see in /sys/class/infiniband_verbs?

 - R.


From rolandd at cisco.com  Tue Jul 19 10:52:40 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 19 Jul 2005 10:52:40 -0700
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <1121794559.4419.8723.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "19 Jul 2005 13:36:00 -0400")
References: <42DD23F5.3060702@psc.edu> <20050719161457.GA15839@mellanox.co.il>
	<42DD2A16.3080800@psc.edu> <20050719164925.GB16116@mellanox.co.il>
	<1121794559.4419.8723.camel@hal.voltaire.com>
Message-ID: <52br4y1y53.fsf@topspin.com>

    Hal> As far as I know, the current agreed upon default is
    Hal> /usr/local/xxx/infiniband where xxx is lib and include.

No, libraries should go in $PREFIX/lib.  Otherwise the default linker
path won't work.  Userspace drivers like mthca.so are a special case
because they're loaded using dlopen() instead of the linker.

$PREFIX/infiniband/include is a good idea because then code can do

    #include <infiniband/xxx.h>

which looks right.

 - R.


From halr at voltaire.com  Tue Jul 19 10:57:43 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 19 Jul 2005 13:57:43 -0400
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <52br4y1y53.fsf@topspin.com>
References: <42DD23F5.3060702@psc.edu>
	<20050719161457.GA15839@mellanox.co.il> <42DD2A16.3080800@psc.edu>
	<20050719164925.GB16116@mellanox.co.il>
	<1121794559.4419.8723.camel@hal.voltaire.com>
	<52br4y1y53.fsf@topspin.com>
Message-ID: <1121795863.4419.8831.camel@hal.voltaire.com>

On Tue, 2005-07-19 at 13:52, Roland Dreier wrote:
>     Hal> As far as I know, the current agreed upon default is
>     Hal> /usr/local/xxx/infiniband where xxx is lib and include.
> 
> No, libraries should go in $PREFIX/lib.  Otherwise the default linker
> path won't work.  

Is this a requirement ? What about setting the appropriate environment
variables or is it better not to do this ?

> Userspace drivers like mthca.so are a special case
> because they're loaded using dlopen() instead of the linker.
> 
> $PREFIX/infiniband/include is a good idea because then code can do
> 
>     #include <infiniband/xxx.h>
> 
> which looks right.

-- Hal



From mst at mellanox.co.il  Tue Jul 19 11:06:14 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 19 Jul 2005 21:06:14 +0300
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <1121794559.4419.8723.camel@hal.voltaire.com>
References: <42DD23F5.3060702@psc.edu> <20050719161457.GA15839@mellanox.co.il>
	<42DD2A16.3080800@psc.edu> <20050719164925.GB16116@mellanox.co.il>
	<1121794559.4419.8723.camel@hal.voltaire.com>
Message-ID: <20050719180614.GC16028@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: Re: unable to run ibv_uc_pingpong and other example binaries
> 
> On Tue, 2005-07-19 at 12:49, Michael S. Tsirkin wrote:
> > Quoting r. Joe Damato <damato at psc.edu>:
> > > Subject: Re: unable to run ibv_uc_pingpong and other example binaries
> > > OK, just tried it, same problem:
> > > [root at frodo examples]# pwd
> > > /cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
> > > [root at frodo examples]# .libs/ibv_uc_pingpong
> > > libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
> > 
> > OK, this one is because you installed libibcm into 
> > /usr/local/lib/infiniband which is a plugin directory instead if
> > /usr/local/lib, so it goes.
> > 
> > > open("/usr/local/lib/infiniband/libibcm.so", O_RDONLY) = 3
> > 
> > Simply remove /usr/local/lib/infiniband/libucm.so
> > and it will go away.
> 
> It appears to be invoking libucm just fine (the location is not the
> issue). The issue is ucm module is not started.
> 
> > Longer term I think libucm config script must be fixed: libraries
> > go to /usr/local/lib by default. Hal?
> 
> As far as I know, the current agreed upon default is
> /usr/local/xxx/infiniband where xxx is lib and include.

Hal, libibverbs looks for libraries under /usr/local/lib/infiniband
and assumes these are low-level driver plugins.
We *dont* want libibverbs to load libucm - its not a low-level driver plugin.

Look at libibverbs, libs are in /usr/local/lib, includes in
/usr/local/include/infiniband.
This is because with includes you can do

#include <infiniband/foo.h>

but this doesnt work with libraries.

> > Hal, I also think it may not be such a good idea for a library
> > to start opening files and print things out to standard error
> > when its loaded.
> > Rather, a library should have some sane init/cleanup functions.
> 
> Due to hot swap or other issues ?

Not necessarily. Errors during library loading are much harder
to isolate and debug.

> I think all the user space libraries
> do this with perhaps the exception of libibverbs.

What else is there :)? Management doesnt count, most applications
wont use it.

-- 
MST


From rolandd at cisco.com  Tue Jul 19 11:07:05 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 19 Jul 2005 11:07:05 -0700
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <1121795863.4419.8831.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "19 Jul 2005 13:57:43 -0400")
References: <42DD23F5.3060702@psc.edu> <20050719161457.GA15839@mellanox.co.il>
	<42DD2A16.3080800@psc.edu> <20050719164925.GB16116@mellanox.co.il>
	<1121794559.4419.8723.camel@hal.voltaire.com>
	<52br4y1y53.fsf@topspin.com>
	<1121795863.4419.8831.camel@hal.voltaire.com>
Message-ID: <527jfm1xh2.fsf@topspin.com>

    Roland> No, libraries should go in $PREFIX/lib.  Otherwise the
    Roland> default linker path won't work.

    Hal> Is this a requirement ? What about setting the appropriate
    Hal> environment variables or is it better not to do this ?

It's not a requirement, but what's the advantage of putting libraries
in $PREFIX/lib/infiniband?  Do we expect distros to do that, and then
put /usr/lib/infiniband in /etc/ld.so.conf?

If we decide to put libraries under infiniband, then I guess I have to
move userspace drivers to $PREFIX/lib/infiniband/drivers or something
like that.

 - R.


From damato at psc.edu  Tue Jul 19 11:10:28 2005
From: damato at psc.edu (Joe Damato)
Date: Tue, 19 Jul 2005 14:10:28 -0400
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <1121794559.4419.8723.camel@hal.voltaire.com>
References: <42DD23F5.3060702@psc.edu>	 <20050719161457.GA15839@mellanox.co.il>
	<42DD2A16.3080800@psc.edu>	 <20050719164925.GB16116@mellanox.co.il>
	<1121794559.4419.8723.camel@hal.voltaire.com>
Message-ID: <42DD4214.3020502@psc.edu>

Hal Rosenstock wrote:

>On Tue, 2005-07-19 at 12:49, Michael S. Tsirkin wrote:
>  
>
>>Quoting r. Joe Damato <damato at psc.edu>:
>>    
>>
>>>Subject: Re: unable to run ibv_uc_pingpong and other example binaries
>>>OK, just tried it, same problem:
>>>[root at frodo examples]# pwd
>>>/cluster/src/OPENIB_SRC/openib-2734/gen2/trunk/src/userspace/libibverbs/examples
>>>[root at frodo examples]# .libs/ibv_uc_pingpong
>>>libucm: Error <-1:6> couldn't open IB cm device </dev/infiniband/ucm>
>>>      
>>>
>>OK, this one is because you installed libibcm into 
>>/usr/local/lib/infiniband which is a plugin directory instead if
>>/usr/local/lib, so it goes.
>>
>>    
>>
>>>open("/usr/local/lib/infiniband/libibcm.so", O_RDONLY) = 3
>>>      
>>>
>>Simply remove /usr/local/lib/infiniband/libucm.so
>>and it will go away.
>>    
>>
>
>It appears to be invoking libucm just fine (the location is not the
>issue). The issue is ucm module is not started.
>
>  
>
ah yes, you are correct. I apologize for this -stupid- error on my part.

>>Longer term I think libucm config script must be fixed: libraries
>>go to /usr/local/lib by default. Hal?
>>    
>>
>
>As far as I know, the current agreed upon default is
>/usr/local/xxx/infiniband where xxx is lib and include.
>
>  
>
>>Hal, I also think it may not be such a good idea for a library
>>to start opening files and print things out to standard error
>>when its loaded.
>>Rather, a library should have some sane init/cleanup functions.
>>    
>>
>
>Due to hot swap or other issues ? I think all the user space libraries
>do this with perhaps the exception of libibverbs.
>
>-- Hal
>  
>



From ardavis at ichips.intel.com  Tue Jul 19 11:10:56 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Tue, 19 Jul 2005 11:10:56 -0700
Subject: [openib-general] uCM kernel oops
Message-ID: <42DD4230.9060702@ichips.intel.com>

Hi Libor,

I am running uCM and uAT with uDAPL and occasionally hit the following. 
Can you take a look?

Jul 19 11:10:17 iclust-19 kernel: UAT: Created struct
Jul 19 11:10:17 iclust-19 kernel: UCM: Created struct
Jul 19 11:10:17 iclust-19 kernel: UAT: Write. cmd <2> in <40> out <8> 
len <48>
Jul 19 11:10:17 iclust-19 kernel: UAT: Allocated context ID <4>
Jul 19 11:10:17 iclust-19 kernel: UAT: Write. cmd <7> in <8> out <32> 
len <16>
Jul 19 11:10:17 iclust-19 kernel: UAT: Destroyed context ID <4>
Jul 19 11:10:17 iclust-19 kernel: UCM: Write. cmd <0> in <8> out <4> len 
<16>
Jul 19 11:10:17 iclust-19 kernel: UCM: Allocated CM ID <2>
Jul 19 11:10:17 iclust-19 kernel: UAT: Write. cmd <0> in <40> out <8> 
len <48>
Jul 19 11:10:17 iclust-19 kernel: UAT: Allocated context ID <5>
Jul 19 11:10:17 iclust-19 kernel: UAT: Write. cmd <7> in <8> out <32> 
len <16>
Jul 19 11:10:17 iclust-19 kernel: UAT: Destroyed context ID <5>
Jul 19 11:10:17 iclust-19 kernel: UAT: Write. cmd <1> in <48> out <8> 
len <56>
Jul 19 11:10:17 iclust-19 kernel: UAT: Allocated context ID <6>
Jul 19 11:10:17 iclust-19 kernel: UAT: Write. cmd <7> in <8> out <32> 
len <16>
Jul 19 11:10:17 iclust-19 kernel: UAT: Destroyed context ID <6>
Jul 19 11:10:17 iclust-19 kernel: UCM: Write. cmd <5> in <64> out <0> 
len <72>
Jul 19 11:10:17 iclust-19 kernel: UCM: Event. CM ID <2> event <3>
Jul 19 11:10:17 iclust-19 kernel: UCM: Write. cmd <16> in <32> out <184> 
len <40>
Jul 19 11:10:17 iclust-19 kernel: UCM: Write. cmd <7> in <16> out <0> 
len <24>
Jul 19 11:10:18 iclust-19 kernel: UCM: Write. cmd <1> in <4> out <0> len 
<12>
Jul 19 11:10:18 iclust-19 kernel: UCM: Event. CM ID <2> event <7>
Jul 19 11:10:18 iclust-19 kernel: UCM: Destroyed CM ID <2>
Jul 19 11:10:18 iclust-19 kernel: Unable to handle kernel paging request 
at 0000000000100108 RIP:
Jul 19 11:10:18 iclust-19 kernel: 
<ffffffff880a10c8>{:ib_ucm:ib_ucm_ctx_put+120}
Jul 19 11:10:18 iclust-19 kernel: PGD 3f992067 PUD 0
Jul 19 11:10:18 iclust-19 kernel: Oops: 0002 [1] SMP
Jul 19 11:10:18 iclust-19 kernel: CPU 1
Jul 19 11:10:18 iclust-19 kernel: Modules linked in: ib_att ib_uat ib_at 
ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib ib_sa autofs4 ib_mthca ib_mad 
ib_core dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod
Jul 19 11:10:18 iclust-19 kernel: Pid: 2265, comm: ib_cm/1 Tainted: 
P      2.6.11
Jul 19 11:10:18 iclust-19 kernel: RIP: 0010:[<ffffffff880a10c8>] 
<ffffffff880a10c8>{:ib_ucm:ib_ucm_ctx_put+120}
Jul 19 11:10:18 iclust-19 kernel: RSP: 0018:ffff810035a75d18  EFLAGS: 
00010292
Jul 19 11:10:18 iclust-19 kernel: RAX: 0000000000100100 RBX: 
ffffffff880a48a0 RCX: ffff81003f902850
Jul 19 11:10:18 iclust-19 kernel: RDX: 0000000000200200 RSI: 
ffff81003590a8f8 RDI: ffff81003f902300
Jul 19 11:10:18 iclust-19 kernel: RBP: ffff81003f902800 R08: 
ffff81003590a8f8 R09: 0000000000000000
Jul 19 11:10:18 iclust-19 kernel: R10: 0000000000000000 R11: 
0000000000000000 R12: ffff81003f902840
Jul 19 11:10:18 iclust-19 kernel: R13: 0000000000000000 R14: 
ffff81003f902800 R15: ffff8100380fc400
Jul 19 11:10:18 iclust-19 kernel: FS:  0000000000000000(0000) 
GS:ffffffff8058c580(0000) knlGS:0000000000000000
Jul 19 11:10:18 iclust-19 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 
000000008005003b
Jul 19 11:10:18 iclust-19 kernel: CR2: 0000000000100108 CR3: 
0000000037016000 CR4: 00000000000006e0
Jul 19 11:10:18 iclust-19 kernel: Process ib_cm/1 (pid: 2265, threadinfo 
ffff810035a74000, task ffff810035850e90)
Jul 19 11:10:18 iclust-19 kernel: Stack: 0000000000000002 
ffff81003b9b7b80 ffff810038104b58 ffffffff880a183f
Jul 19 11:10:18 iclust-19 kernel:        ffff810035850e90 
ffff8100380fc400 ffff8100380fc400 ffff810038104ac0
Jul 19 11:10:18 iclust-19 kernel:        ffff8100380fc470 0000000000000292
Jul 19 11:10:18 iclust-19 kernel: Call 
Trace:<ffffffff880a183f>{:ib_ucm:ib_ucm_event_handler+1199}
Jul 19 11:10:18 iclust-19 kernel:        
<ffffffff8809b850>{:ib_cm:cm_work_handler+0} 
<ffffffff8809aada>{:ib_cm:cm_process_work+26}
Jul 19 11:10:19 iclust-19 kernel:        
<ffffffff8809ad52>{:ib_cm:cm_dreq_handler+418} 
<ffffffff8809c7f8>{:ib_cm:cm_work_handler+4008}
Jul 19 11:10:19 iclust-19 kernel:        
<ffffffff80133663>{__wake_up+67} 
<ffffffff8809b850>{:ib_cm:cm_work_handler+0}
Jul 19 11:10:19 iclust-19 kernel:        
<ffffffff80148510>{worker_thread+496} 
<ffffffff801335a0>{default_wake_function+0}
Jul 19 11:10:19 iclust-19 kernel:        
<ffffffff801335f0>{__wake_up_common+64} 
<ffffffff801335a0>{default_wake_function+0}
Jul 19 11:10:19 iclust-19 kernel:        
<ffffffff8014c840>{keventd_create_kthread+0} 
<ffffffff80148320>{worker_thread+0}
Jul 19 11:10:19 iclust-19 kernel:        
<ffffffff8014c840>{keventd_create_kthread+0} <ffffffff8014c7f9>{kthread+217}
Jul 19 11:10:19 iclust-19 kernel:        <ffffffff8010eec3>{child_rip+8} 
<ffffffff8014c840>{keventd_create_kthread+0}
Jul 19 11:10:19 iclust-19 kernel:        <ffffffff8014c720>{kthread+0} 
<ffffffff8010eebb>{child_rip+0}
Jul 19 11:10:19 iclust-19 kernel:
Jul 19 11:10:19 iclust-19 kernel:
Jul 19 11:10:19 iclust-19 kernel: Code: 48 89 50 08 48 89 02 48 c7 41 08 
00 02 20 00 48 c7 45 50 00
Jul 19 11:10:19 iclust-19 kernel: RIP 
<ffffffff880a10c8>{:ib_ucm:ib_ucm_ctx_put+120} RSP <ffff810035a75d18>
Jul 19 11:10:19 iclust-19 kernel: CR2: 0000000000100108

Thanks,

-arlin



From halr at voltaire.com  Tue Jul 19 11:23:03 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 19 Jul 2005 14:23:03 -0400
Subject: [openib-general] Re: unable to run ibv_uc_pingpong and other
	example binaries
In-Reply-To: <527jfm1xh2.fsf@topspin.com>
References: <42DD23F5.3060702@psc.edu>
	<20050719161457.GA15839@mellanox.co.il> <42DD2A16.3080800@psc.edu>
	<20050719164925.GB16116@mellanox.co.il>
	<1121794559.4419.8723.camel@hal.voltaire.com>
	<52br4y1y53.fsf@topspin.com>
	<1121795863.4419.8831.camel@hal.voltaire.com>
	<527jfm1xh2.fsf@topspin.com>
Message-ID: <1121797382.4419.8959.camel@hal.voltaire.com>

On Tue, 2005-07-19 at 14:07, Roland Dreier wrote:
> It's not a requirement, but what's the advantage of putting libraries
> in $PREFIX/lib/infiniband?  Do we expect distros to do that, and then
> put /usr/lib/infiniband in /etc/ld.so.conf?

I'll change ucm and any others I find like this.

Should management also be changed too ?

-- Hal



From ardavis at ichips.intel.com  Tue Jul 19 11:42:49 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Tue, 19 Jul 2005 11:42:49 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121695663.4419.6024.camel@hal.voltaire.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>	
	<1121351537.4423.1162.camel@hal.voltaire.com>	
	<42D6F3F5.5020806@ichips.intel.com>	
	<1121425731.4423.2145.camel@hal.voltaire.com>	
	<42D7EC83.8030406@ichips.intel.com>
	<1121695663.4419.6024.camel@hal.voltaire.com>
Message-ID: <42DD49A9.9060600@ichips.intel.com>

Hal Rosenstock wrote:

>Hi Arlin,
>
>On Fri, 2005-07-15 at 13:04, Arlin Davis wrote:
>  
>
>>Your implementation is fine, as long as the correct return values are used
>>and the sync version works. I currently get returned a 1 but the ib_route
>>data is not filled in and the req_id == 0. I would expect a 0 returned 
>>and a
>>valid req_id if the ib_route cannot be processed.
>>
>>for ret = ib_at_route_by_ip( async_cb provided ), I read this as: 
>>
>>  if ret > 0  then ib_route output data is written, resolve complete
>>  if ret == 0 then req_id is written with valid id to cancel or poll
>>  if ret < 0 error
>>
>>Am I reading this right?
>>    
>>
>
>Try this patch and let me know if this works for you.
>
>Thanks.
>
>-- Hal
>
>  
>
Hal,

Yes, it returns the route data (GID) when the return == 1.  However, I 
still get a callback event with a req_id ==0 which is somewhat 
unexpected. This seems unnecessary given that ib_at_route_by_ip() just 
returned the data. Is this the expected behavior?

Here is my output from the call and from the at_thread.

4116 Running as client
4116 Server Name: iclust-20-ib0
4116 Server Net Address: 192.168.0.20
dapl_ep_connect (0x515c40, {4294967232.4294967208.0.20}, 115D3, 5000000, 
0, (nil), 0, 0)
 connect: r_SID 71123, pdata (nil), plen 0
 connect: at_route ret=1, req_id 0, GID fe80000000000000 
0002c90200004071   <<<< ib_route info returned with call
dapl_ep_connect () returns 0x0
4116 Waiting for connect response

at_thread: get_callback
 rt_comp_handler: conn 0x519e30, req_id 0 rec_num 1                 
 rt_comp_handler: SRC GID subnet fe80000000000000 id 0002c9020000409d    
<<< get_callback event fires
 rt_comp_handler: DST GID subnet fe80000000000000 id 0002c90200004071
 at_thread: callback woke
 
-arlin

>
>
>
>
>
>  
>



From libor at topspin.com  Tue Jul 19 13:28:12 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 13:28:12 -0700
Subject: [openib-general] uVerbs and multiple cq_event support?
In-Reply-To: <42D81BE3.3020200@ichips.intel.com>;
	from ardavis@ichips.intel.com on Fri, Jul 15, 2005 at 01:26:11PM
	-0700
References: <42D81BE3.3020200@ichips.intel.com>
Message-ID: <20050719132812.A2231@topspin.com>

On Fri, Jul 15, 2005 at 01:26:11PM -0700, Arlin Davis wrote:
> Hi Roland,
> 
> I have a question about uverbs cq_events. Currently, uverbs has one 
> event (fd) handle that processes all CQ events. You mentioned earlier  
> that you would possibly add support for more CQ events. Are you thinking 
> one per CQ, or limiting to a small number like one per processor? If the 
> thinking is NOT one per CQ then it may be somewhat restricitive to user 
> applications that have multple connections with many CQs to process.
> 
> For instance, with uDAPL there are user wait objects (CQ_WAIT_OBJECT) 
> that can be mapped directly to these CQ events. This enables the user 
> thread to wait and wake up directly from the kernel CQ event processing 
> which is very efficicent. However, this can only be done if verbs 
> provides an event mechanism per CQ. Otherwise, I have to process the 
> events in a separate thread and in turn process the user's event 
> independent of the CQ event.
>
> If the thinking is one per CQ then is fd's the way to go since we can 
> have 1000's of CQs?

  An fd per CQ is definetly not the way to go for this reason. I don't
think that the wait object would need to map to an individual fd for
each CQ. The fd should become ready for read when there's a CQ event
pending and a read of the fd will tell you which CQ it was that had
the event. Only the wait object which represents that CQ needs to
notify the consumer, the rest can go back to waiting or continue waiting
depending how you implement the wait and locking around the fd. The
point being is that you do not need another thread.

  I'm just looking at the ABI, so I'm not sure which function
you should use to read the fd and get the info...

-Libor


From pauln at psc.edu  Tue Jul 19 14:32:52 2005
From: pauln at psc.edu (PAulN)
Date: Tue, 19 Jul 2005 17:32:52 -0400
Subject: [openib-general] ib_device_get_by_index();
	// porting from gen1 to gen2
Message-ID: <42DD7184.4090004@psc.edu>

Hi,
I have question regarding opening a device in gen2.  The code I'm porting
uses ib_device_get_by_index() which is not present in gen2.  Can someone
tell me the proper way to open a device in gen2?  Should I be using
the *client* calls from core/device.c?
Thanks,
Paul



From mshefty at ichips.intel.com  Tue Jul 19 14:39:45 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 19 Jul 2005 14:39:45 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <42D84618.90807@ichips.intel.com>
References: <1121203934.14638.27.camel@duffman>	<20050712221725.GB14316@mellanox.co.il>	<200507130248.08387.adobriyan@gmail.com>	<42D83A5B.5090808@ichips.intel.com>	<52r7dz661d.fsf@topspin.com>
	<52irzb65sq.fsf@topspin.com> <42D84618.90807@ichips.intel.com>
Message-ID: <42DD7321.1040506@ichips.intel.com>

Sean Hefty wrote:
> Roland Dreier wrote:
> 
>> For whatever it's worth, here's what I have accumulated so far:
> 
> How would you like to proceed on this?  Do you want to complete your 
> changes and commit them, or would you like for me to incorporate your 
> patch into a larger set of changes?

FYI - I decided to incorporate most of the changes into a larger change set. 
  I'm working on fixing anything that touches into the MAD code (SA, CM, 
SMI, etc.).  My plan is to defer changes to mthca for a subsequent patch, 
and decide on ipoib and spd later.

So far I think I've found one actual bug.

- Sean


From mst at mellanox.co.il  Tue Jul 19 15:04:34 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 01:04:34 +0300
Subject: [openib-general] Re: Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <42DD7321.1040506@ichips.intel.com>
References: <1121203934.14638.27.camel@duffman>
	<20050712221725.GB14316@mellanox.co.il>
	<200507130248.08387.adobriyan@gmail.com>
	<42D83A5B.5090808@ichips.intel.com>
	<52r7dz661d.fsf@topspin.com> <52irzb65sq.fsf@topspin.com>
	<42D84618.90807@ichips.intel.com>
	<42DD7321.1040506@ichips.intel.com>
Message-ID: <20050719220434.GA17097@mellanox.co.il>

Quoting r. Sean Hefty <mshefty at ichips.intel.com>:
> My plan is to defer changes to mthca for a subsequent 
> patch, and decide on ipoib and spd later.

spd -> sdp?
SDP has a known issue around zcopy iocb code: no dma api. This results
in lots of sparse warnings.
Sean, I'd suggest you dont touch that one since there's an outstanding patch
that rewrites all that code, anyway.


-- 
MST


From mshefty at ichips.intel.com  Tue Jul 19 15:06:54 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 19 Jul 2005 15:06:54 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <20050719220434.GA17097@mellanox.co.il>
References: <1121203934.14638.27.camel@duffman>
	<20050712221725.GB14316@mellanox.co.il>
	<200507130248.08387.adobriyan@gmail.com>
	<42D83A5B.5090808@ichips.intel.com>
	<52r7dz661d.fsf@topspin.com> <52irzb65sq.fsf@topspin.com>
	<42D84618.90807@ichips.intel.com>
	<42DD7321.1040506@ichips.intel.com>
	<20050719220434.GA17097@mellanox.co.il>
Message-ID: <42DD797E.2050104@ichips.intel.com>

Michael S. Tsirkin wrote:
> Quoting r. Sean Hefty <mshefty at ichips.intel.com>:
> 
>>My plan is to defer changes to mthca for a subsequent 
>>patch, and decide on ipoib and spd later.
> 
> 
> spd -> sdp?
> SDP has a known issue around zcopy iocb code: no dma api. This results
> in lots of sparse warnings.
> Sean, I'd suggest you dont touch that one since there's an outstanding patch
> that rewrites all that code, anyway.

Yeah - I meant SDP.  Thanks for the info.  I'll leave it alone (unless some 
other change causes a compile error or something similar).

- Sean


From ardavis at ichips.intel.com  Tue Jul 19 15:14:19 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Tue, 19 Jul 2005 15:14:19 -0700
Subject: [openib-general] uVerbs and multiple cq_event support?
In-Reply-To: <20050719132812.A2231@topspin.com>
References: <42D81BE3.3020200@ichips.intel.com>
	<20050719132812.A2231@topspin.com>
Message-ID: <42DD7B3B.8080704@ichips.intel.com>

Libor Michalek wrote:

>On Fri, Jul 15, 2005 at 01:26:11PM -0700, Arlin Davis wrote:
>  
>
>>Hi Roland,
>>
>>I have a question about uverbs cq_events. Currently, uverbs has one 
>>event (fd) handle that processes all CQ events. You mentioned earlier  
>>that you would possibly add support for more CQ events. Are you thinking 
>>one per CQ, or limiting to a small number like one per processor? If the 
>>thinking is NOT one per CQ then it may be somewhat restricitive to user 
>>applications that have multple connections with many CQs to process.
>>
>>For instance, with uDAPL there are user wait objects (CQ_WAIT_OBJECT) 
>>that can be mapped directly to these CQ events. This enables the user 
>>thread to wait and wake up directly from the kernel CQ event processing 
>>which is very efficicent. However, this can only be done if verbs 
>>provides an event mechanism per CQ. Otherwise, I have to process the 
>>events in a separate thread and in turn process the user's event 
>>independent of the CQ event.
>>
>>If the thinking is one per CQ then is fd's the way to go since we can 
>>have 1000's of CQs?
>>    
>>
>
>  An fd per CQ is definetly not the way to go for this reason. I don't
>think that the wait object would need to map to an individual fd for
>each CQ. The fd should become ready for read when there's a CQ event
>pending and a read of the fd will tell you which CQ it was that had
>the event. Only the wait object which represents that CQ needs to
>notify the consumer, the rest can go back to waiting or continue waiting
>depending how you implement the wait and locking around the fd. The
>point being is that you do not need another thread.
>  
>
This is actually how I implemented dapls_ib_wait_object_wait() in uDAPL 
(uCM and uAT version)
but I am not convinced it is any better then just using another thread. 
The method avoids another
thread but I now have a thundering herd all chasing the same event, 
sharing a mutex, and going
back to sleep if it was not yours. In addition, the user provided wait 
times have to be adjusted
for the next poll on the fd. Gets really ugly fast.

-arlin

>  I'm just looking at the ABI, so I'm not sure which function
>you should use to read the fd and get the info...
>  
>
>-Libor
>
>  
>



From libor at topspin.com  Tue Jul 19 16:03:45 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 16:03:45 -0700
Subject: [openib-general] [PATCH] SDP: use linux/list.h for binds
In-Reply-To: <1121366046.17302.9.camel@duffman>;
	from tduffy@sun.com on Thu, Jul 14, 2005 at 11:34:06AM -0700
References: <1121366046.17302.9.camel@duffman>
Message-ID: <20050719160345.B2231@topspin.com>

On Thu, Jul 14, 2005 at 11:34:06AM -0700, Tom Duffy wrote:
> This patch replaces the null terminated linked list for connections
> bound to a port and instead uses the list.h library from core Linux.
> 
> I have tested this on a few machines doing several ttcp's at once, but
> please inspect to verify correctness.

  Thanks. I've applied the patch since it's better then what was there.
However, the longer term solution needs to use a full hash table, the 
linear list is problematic when there are a lot of connections, and each
port_get() needs to traverse the entire list to check for collisions.

-Libor


From mst at mellanox.co.il  Tue Jul 19 16:09:23 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 02:09:23 +0300
Subject: [openib-general] Re: [PATCH] SDP: use linux/list.h for binds
In-Reply-To: <20050719160345.B2231@topspin.com>
References: <1121366046.17302.9.camel@duffman>
	<20050719160345.B2231@topspin.com>
Message-ID: <20050719230923.GD17097@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
> Subject: Re: [PATCH] SDP: use linux/list.h for binds
> 
> On Thu, Jul 14, 2005 at 11:34:06AM -0700, Tom Duffy wrote:
> > This patch replaces the null terminated linked list for connections
> > bound to a port and instead uses the list.h library from core Linux.
> > 
> > I have tested this on a few machines doing several ttcp's at once, but
> > please inspect to verify correctness.
> 
>   Thanks. I've applied the patch since it's better then what was there.
> However, the longer term solution needs to use a full hash table, the 
> linear list is problematic when there are a lot of connections, and each
> port_get() needs to traverse the entire list to check for collisions.
> 
> -Libor

Or a tree?

-- 
MST


From libor at topspin.com  Tue Jul 19 16:19:15 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 16:19:15 -0700
Subject: [openib-general] [PATCH] sdp_conn_put/sdp_conn_hold race
In-Reply-To: <20050706175949.GA8783@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Jul 06, 2005 at 08:59:49PM +0300
References: <20050706175949.GA8783@mellanox.co.il>
Message-ID: <20050719161915.C2231@topspin.com>

On Wed, Jul 06, 2005 at 08:59:49PM +0300, Michael S. Tsirkin wrote:
> The current sdp_conn_put/sdp_conn_hold implementation
> seems to be subject to the following race condition:
> 
> - thread A calls sdp_conn_put, atomic dec and test returns 0
> - thread B looks up the connection and calls sdp_conn_get,
>   incrementing the reference count back to 1
> - thread A now goes on to call sdp_conn_destroy, which will
>   destroy the connection
> - thread B is left with an invalid connection pointer 

  Thanks for bringing this up, I had noticed it, but never got back
around to actually fixing it. The solution description

> As a solution
> - sdp_conn_put is moved out of line. checking connection
>   reference count is done under the connection table spinlock
>   atomically, removing the connection from lookup table
>   if the reference drops to 0.
> 
> - a new call sdp_conn_put_light is provided for when we
>   know this isn't the last reference to the connection.
>   This is useful not only for performance reasons, but 
>   also for documentation/code clarity purposes: when
>   sdp_conn_put_light is used, this isn't the last reference,
>   when sdp_conn_put is used, this may be the last reference.

  Yes, this would be the right fix for the problem, I'll take a look
at the patch, but I only saw part 2/2 and not 1/2 on the list, can
you resend it? 

-Libor



From libor at topspin.com  Tue Jul 19 16:22:37 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 16:22:37 -0700
Subject: [openib-general] [PATCH] SDP: use linux/list.h for hca port list
In-Reply-To: <1121380212.17302.30.camel@duffman>;
	from tduffy@sun.com on Thu, Jul 14, 2005 at 03:30:12PM -0700
References: <1121380212.17302.30.camel@duffman>
Message-ID: <20050719162237.D2231@topspin.com>

On Thu, Jul 14, 2005 at 03:30:12PM -0700, Tom Duffy wrote:
> This patch changes the NULL terminated port list in the sdp hca object
> to a Linux list_head.  It also removes the sdev_hca *next member from
> the sdev_hca struct as this is never used, AFAICT.

  Thanks, applied. The unused member was left over from before devices
where registered dynamically, and instead were kept in a list.

-Libor


From mst at mellanox.co.il  Tue Jul 19 16:27:11 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 02:27:11 +0300
Subject: [openib-general] [PATCHv2] fix sdp aio race
Message-ID: <20050719232711.GF17097@mellanox.co.il>

Second attempt to solve race where AIO completed asynchronously
is bypassed by other completions.
Solution: set a bit in connection, and defer any cq polling
till work queue handles the AIO.
We have to pass the connection to iocb handlers for this to work.
I also got rid of in_atomic.

Libor, please comment.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_write.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_write.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_write.c
@@ -109,7 +109,7 @@ int sdp_event_write(struct sdp_sock *con
 		SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
 		SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
-		sdp_iocb_complete(iocb, 0);
+		sdp_iocb_complete(conn, iocb, 0);
 
 		break;
 	case SDP_DESC_TYPE_NONE:
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_rcvd.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_rcvd.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_rcvd.c
@@ -155,7 +155,7 @@ static int sdp_rcvd_send_sm(struct sdp_s
 
 			conn->src_sent--;
 
-			sdp_iocb_complete(iocb, 0);
+			sdp_iocb_complete(conn, iocb, 0);
 		}
 		/*
 		 * Cancel complete, clear the state.
@@ -212,7 +212,7 @@ static int sdp_rcvd_rdma_wr(struct sdp_s
 
 	conn->snk_sent--;
 
-	sdp_iocb_complete(iocb, 0);
+	sdp_iocb_complete(conn, iocb, 0);
 
 	return 0;
 }
@@ -273,7 +273,7 @@ static int sdp_rcvd_rdma_rd(struct sdp_s
 		SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
 		SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
-		sdp_iocb_complete(iocb, 0);
+		sdp_iocb_complete(conn, iocb, 0);
 	}
 	/*
 	 * If Source Cancel was in process, and there are no more outstanding
@@ -565,7 +565,7 @@ static int sdp_rcvd_snk_cancel_ack(struc
 	while ((iocb = sdp_iocb_q_get_head(&conn->r_snk))) {
 
 		conn->snk_sent--;
-		sdp_iocb_complete(iocb, 0);
+		sdp_iocb_complete(conn, iocb, 0);
 	}
 	/*
 	 * cancellation is complete. Cancel flag is cleared in RECV post.
@@ -687,7 +687,7 @@ static int sdp_rcvd_snk_avail(struct sdp
 				sdp_desc_q_put_head(&conn->send_queue,
 						    (struct sdpc_desc *)iocb);
 			else
-				sdp_iocb_complete(iocb, 0);
+				sdp_iocb_complete(conn, iocb, 0);
 		}
 		/*
 		 * If Source Cancel was in process, it should now 
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_proto.h
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_proto.h
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_proto.h
@@ -162,7 +162,8 @@ void sdp_iocb_q_put_tail(struct sdpc_ioc
 
 struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key);
 
-void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
+void sdp_iocb_q_cancel(struct sdp_sock *conn, struct sdpc_iocb_q *table,
+		       u32 mask, ssize_t comp);
 
 void sdp_iocb_q_remove(struct sdpc_iocb *iocb);
 
@@ -170,7 +171,8 @@ int sdp_iocb_register(struct sdpc_iocb *
 
 void sdp_iocb_release(struct sdpc_iocb *iocb);
 
-void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status);
+void sdp_iocb_complete(struct sdp_sock *conn, struct sdpc_iocb *iocb,
+		       ssize_t status);
 
 int sdp_iocb_lock(struct sdpc_iocb *iocb);
 
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_read.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_read.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_read.c
@@ -207,7 +207,7 @@ int sdp_event_read(struct sdp_sock *conn
 		SDP_CONN_STAT_READ_INC(conn, iocb->post);
 		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
 
-		sdp_iocb_complete(iocb, 0);
+		sdp_iocb_complete(conn, iocb, 0);
 
 		break;
 	case SDP_DESC_TYPE_NONE:
@@ -228,7 +228,7 @@ int sdp_event_read(struct sdp_sock *conn
 		SDP_CONN_STAT_READ_INC(conn, iocb->post);
 		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
 
-		sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0);
+		sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0);
 
 		break;
 	default:
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_send.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_send.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_send.c
@@ -859,7 +859,7 @@ static int sdp_send_data_iocb(struct sdp
 			SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
 			SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
 
-			sdp_iocb_complete(iocb, 0);
+			sdp_iocb_complete(conn, iocb, 0);
 		}
 
 		goto done;
@@ -1730,7 +1730,7 @@ static int sdp_inet_write_cancel(struct 
 			 * needs to be compelted.
 			 */
 			if (iocb->post > 0) {
-				sdp_iocb_complete(iocb, 0);
+				sdp_iocb_complete(conn, iocb, 0);
 				result = -EAGAIN;
 			} else {
 				sdp_iocb_destroy(iocb);
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_conn.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.c
@@ -665,7 +665,8 @@ static int sdp_desc_q_cancel_lookup_func
 	return ((element->type == SDP_DESC_TYPE_IOCB) ? 0 : -ERANGE);
 }
 
-static void sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
+static void sdp_desc_q_cancel_iocb(struct sdp_sock *conn,
+				   struct sdpc_desc_q *table, ssize_t error)
 {
 	struct sdpc_iocb *iocb;
 
@@ -675,24 +676,24 @@ static void sdp_desc_q_cancel_iocb(struc
 			 NULL))) {
 
 		sdp_iocb_q_remove(iocb);
-		sdp_iocb_complete(iocb, error);
+		sdp_iocb_complete(conn, iocb, error);
 	}
 }
 
 void sdp_iocb_q_cancel_all_read(struct sdp_sock *conn, ssize_t error)
 {
-	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
-	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(conn, &conn->r_pend, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(conn, &conn->r_snk, SDP_IOCB_F_ALL, error);
 
-	sdp_desc_q_cancel_iocb(&conn->r_src, error);
+	sdp_desc_q_cancel_iocb(conn, &conn->r_src, error);
 }
 
 void sdp_iocb_q_cancel_all_write(struct sdp_sock *conn, ssize_t error)
 {
-	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
+	sdp_iocb_q_cancel(conn, &conn->w_src, SDP_IOCB_F_ALL, error);
 
-	sdp_desc_q_cancel_iocb(&conn->send_queue, error);
-	sdp_desc_q_cancel_iocb(&conn->w_snk, error);
+	sdp_desc_q_cancel_iocb(conn, &conn->send_queue, error);
+	sdp_desc_q_cancel_iocb(conn, &conn->w_snk, error);
 }
 
 void sdp_iocb_q_cancel_all(struct sdp_sock *conn, ssize_t error)
@@ -927,7 +928,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
 	 * the function should only be called under the connection locks
 	 * spinlock to ensure the call is serialized to avoid races.
 	 */
-	for (;;) {
+	while(!(conn->lock.event & SDP_LOCK_F_DEFERRED)) {
 		/*
 		 * poll for a completion
 		 */
@@ -971,17 +972,16 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
  */
 void sdp_conn_internal_unlock(struct sdp_sock *conn)
 {
-	int calls = 0;
 	/*
 	 * poll CQs for events.
 	 */
-	if (conn->lock.event & SDP_LOCK_F_RECV_CQ)
-		calls += sdp_conn_cq_drain(conn->recv_cq, conn);
-
-	if (conn->lock.event & SDP_LOCK_F_SEND_CQ)
-		calls += sdp_conn_cq_drain(conn->send_cq, conn);
-
-	conn->lock.event = 0;
+	if ((conn->lock.event & SDP_LOCK_F_RECV_CQ) &&
+		!sdp_conn_cq_drain(conn->recv_cq, conn))
+		conn->lock.event &= ~SDP_LOCK_F_RECV_CQ;
+
+	if ((conn->lock.event & SDP_LOCK_F_SEND_CQ) &&
+		!sdp_conn_cq_drain(conn->send_cq, conn))
+		conn->lock.event &= ~SDP_LOCK_F_SEND_CQ;
 }
 
 /*
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_recv.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_recv.c
@@ -663,7 +663,7 @@ static int sdp_recv_buff_iocb_active(str
 	/*
 	 * callback to complete IOCB
 	 */
-	sdp_iocb_complete(iocb, 0);
+	sdp_iocb_complete(conn, iocb, 0);
 
 	return (buff->tail - buff->data);
 }
@@ -716,7 +716,7 @@ static int sdp_recv_buff_iocb_pending(st
 		/*
 		 * callback to complete IOCB
 		 */
-		sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0);
+		sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0);
 	}
 
 	return (buff->tail - buff->data);
@@ -875,7 +875,7 @@ static int sdp_inet_read_cancel(struct k
 				/*
 				 * callback to complete IOCB, or drop reference
 				 */
-				sdp_iocb_complete(iocb, 0);
+				sdp_iocb_complete(conn, iocb, 0);
 				result = -EAGAIN;
 			}
 			else {
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.h
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_conn.h
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.h
@@ -163,8 +163,9 @@ struct sdp_conn_lock {
 	wait_queue_head_t waitq;
 };
 
-#define SDP_LOCK_F_RECV_CQ 0x01 /* recv CQ event is pending */
-#define SDP_LOCK_F_SEND_CQ 0x02 /* send CQ event is pending */
+#define SDP_LOCK_F_RECV_CQ  0x01 /* recv CQ event is pending */
+#define SDP_LOCK_F_SEND_CQ  0x02 /* send CQ event is pending */
+#define SDP_LOCK_F_DEFERRED 0x04 /* work scheduled, pending completion */
 /*
  * SDP Connection structure.
  */
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_iocb.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.c
@@ -483,9 +483,8 @@ void sdp_iocb_release(struct sdpc_iocb *
 /*
  * do_iocb_complete - complete an IOCB for real in thread context
  */
-static void do_iocb_complete(void *arg)
+static void do_iocb_complete(struct sdpc_iocb *iocb)
 {
-	struct sdpc_iocb *iocb = (struct sdpc_iocb *)arg;
 	long value;
 	/*
 	 * release memory
@@ -508,21 +507,37 @@ static void do_iocb_complete(void *arg)
 	 * we ignore the value.
 	 */
 	(void)aio_complete(iocb->req, value, 0);
+
 	/*
 	 * delete IOCB
 	 */
 	sdp_iocb_destroy(iocb);
 }
 
+static void deferred_iocb_complete(void *arg)
+{
+	struct sdpc_iocb *iocb = arg;
+	struct sdp_sock *conn = iocb->conn;
+	do_iocb_complete(iocb);
+	if (conn) {
+		sdp_conn_lock(conn);
+		conn->lock.event &= SDP_LOCK_F_DEFERRED;
+		sdp_conn_unlock(conn);
+	}
+}
 /*
  * sdp_iocb_complete - complete an IOCB
  */
-void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status)
+void sdp_iocb_complete(struct sdp_sock *conn, struct sdpc_iocb *iocb,
+		       ssize_t status)
 {
 	iocb->status = status;
-	
-	if (in_atomic() || irqs_disabled()) {
-		INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb);
+
+	/* This is always called with connection locked,
+	 * so users == 0 means this was an irq lock. */
+	if (!conn->lock.users) {
+		iocb->conn = conn;
+		INIT_WORK(&iocb->completion, deferred_iocb_complete, iocb);
 		schedule_work(&iocb->completion);
 	} else
 		do_iocb_complete(iocb);
@@ -750,7 +765,8 @@ void sdp_iocb_q_put_head(struct sdpc_ioc
 /*
  * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
  */
-void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp)
+void sdp_iocb_q_cancel(struct sdp_sock *conn, struct sdpc_iocb_q *table,
+		       u32 mask, ssize_t comp)
 {
 	struct sdpc_iocb *iocb;
 	struct sdpc_iocb *next;
@@ -772,7 +788,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
 				    iocb->post, iocb->len);
 
 			sdp_iocb_q_remove(iocb);
-			sdp_iocb_complete(iocb, comp);
+			sdp_iocb_complete(conn, iocb, comp);
 		}
 
 		iocb = next;
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_event.c
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_event.c
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_event.c
@@ -169,8 +169,8 @@ void sdp_cq_event_handler(struct ib_cq *
 			/*
 			 * dispatch CQ completions.
 			 */
-			(void)sdp_conn_cq_drain(cq, conn);
-			event = 0;
+			if (!sdp_conn_cq_drain(cq, conn))
+				event = 0;
 		}
 	}
 	/*
Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.h
===================================================================
--- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_iocb.h
+++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.h
@@ -109,7 +109,8 @@ struct sdpc_iocb {
 	int           page_count;  /* number of physical pages. */
 	int           page_offset; /* offset into first page. */
 
-	struct work_struct completion; /* task for defered completion. */
+	struct work_struct completion; /* task for deferred completion. */
+	struct sdp_sock *conn; /* conn to unlock for deferred completion. */
 	/*
 	 * kernel iocb structure
 	 */

-- 
MST


From mst at mellanox.co.il  Tue Jul 19 16:28:18 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 02:28:18 +0300
Subject: [openib-general] Re: [PATCH] sdp_conn_put/sdp_conn_hold race
In-Reply-To: <20050719161915.C2231@topspin.com>
References: <20050706175949.GA8783@mellanox.co.il>
	<20050719161915.C2231@topspin.com>
Message-ID: <20050719232818.GG17097@mellanox.co.il>

Quoting r. Libor Michalek <libor at topspin.com>:
>   Yes, this would be the right fix for the problem, I'll take a look
> at the patch, but I only saw part 2/2 and not 1/2 on the list, can
> you resend it? 

OK. Coming.

-- 
MST


From mst at mellanox.co.il  Wed Jul  6 10:59:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 20:59:49 +0300
Subject: [openib-general] [PATCH] sdp_conn_put/sdp_conn_hold race
Message-ID: <20050706175949.GA8783@mellanox.co.il>

The current sdp_conn_put/sdp_conn_hold implementation
seems to be subject to the following race condition:

- thread A calls sdp_conn_put, atomic dec and test returns 0
- thread B looks up the connection and calls sdp_conn_get,
  incrementing the reference count back to 1
- thread A now goes on to call sdp_conn_destroy, which will
  destroy the connection
- thread B is left with an invalid connection pointer 

As a solution
- sdp_conn_put is moved out of line. checking connection
  reference count is done under the connection table spinlock
  atomically, removing the connection from lookup table
  if the reference drops to 0.

- a new call sdp_conn_put_light is provided for when we
  know this isn't the last reference to the connection.
  This is useful not only for performance reasons, but 
  also for documentation/code clarity purposes: when
  sdp_conn_put_light is used, this isn't the last reference,
  when sdp_conn_put is used, this may be the last reference.
  
Patches follow.
Please comment. 

-- 
MST


From mst at mellanox.co.il  Wed Jul  6 11:02:33 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 21:02:33 +0300
Subject: [openib-general] [PATCH 1/2] sdp_conn_put/sdp_conn_hold race
	condition
Message-ID: <20050706180233.GA8950@mellanox.co.il>

Protect sdp_conn_put by dev_root_s.sock_lock.
Add sdp_conn_put_light for when we know its not the last reference.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/sdp/sdp_conn.c
===================================================================
--- ulp/sdp/sdp_conn.c	(revision 2784)
+++ ulp/sdp/sdp_conn.c	(working copy)
@@ -595,12 +595,6 @@ static int sdp_conn_table_insert(struct 
 int sdp_conn_table_remove(struct sdp_sock *conn)
 {
 	int result = 0;
-	unsigned long flags;
-
-	/*
-	 * lock table
-	 */
-	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
 	/*
 	 * validate entry
 	 */
@@ -619,10 +613,6 @@ int sdp_conn_table_remove(struct sdp_soc
 	conn->hashent = SDP_DEV_SK_INVALID;
 
 done:
-	/*
-	 * unlock table
-	 */
-	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 	return result;
 }
 
@@ -706,13 +696,30 @@ void sdp_iocb_q_cancel_all(struct sdp_so
  */
 
 /*
- * sdp_conn_destruct - final destructor for connection.
+ * sdp_conn_put - reference counting and final destructor for connection.
  */
-void sdp_conn_destruct(struct sdp_sock *conn)
+void sdp_conn_put(struct sdp_sock *conn)
 {
+	unsigned long flags;
 	int dump = 0;
 	int result;
 
+
+	spin_lock_irqsave(&dev_root_s.sock_lock, flags);
+
+	if (!atomic_dec_and_test(&conn->refcnt)) {
+		spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
+		return;
+	}
+
+	result = sdp_conn_table_remove(conn);
+	if (result < 0)
+		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
+			     result, dev_root_s.sk_entry,
+			     dev_root_s.sk_size);
+
+	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
+
 	sdp_dbg_ctrl(conn, "DESTRUCT. <%08x:%04x> <%08x:%04x> <%u:%u>",
 		     conn->src_addr, conn->src_port,
 		     conn->dst_addr, conn->dst_port,
@@ -721,14 +728,6 @@ void sdp_conn_destruct(struct sdp_sock *
 	 * If the socket is bound, return the port
 	 */
 	(void)sdp_inet_port_put(conn);
-	/*
-	 * remove connection from table
-	 */
-	result = sdp_conn_table_remove(conn);
-	if (result < 0)
-		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
-			     result, dev_root_s.sk_entry,
-			     dev_root_s.sk_size);
 
 	sdp_conn_stat_dump(conn);
 	/*
Index: ulp/sdp/sdp_conn.h
===================================================================
--- ulp/sdp/sdp_conn.h	(revision 2784)
+++ ulp/sdp/sdp_conn.h	(working copy)
@@ -431,7 +431,6 @@ struct sdp_sock {
 extern void sdp_conn_internal_lock(struct sdp_sock *conn, unsigned long *flags);
 extern void sdp_conn_internal_unlock(struct sdp_sock *conn);
 extern void sdp_conn_relock(struct sdp_sock *conn);
-extern void sdp_conn_destruct(struct sdp_sock *conn);
 extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_sock *conn);
 
 #define SDP_CONN_LOCK_IRQ(conn, flags) \
@@ -479,12 +478,17 @@ static inline void sdp_conn_hold(struct 
 	atomic_inc(&conn->refcnt);
 }
 
-static inline void sdp_conn_put(struct sdp_sock *conn)
+/*
+ * safe to call if there's another reference on connection.
+ */
+static inline void sdp_conn_put_light(struct sdp_sock *conn)
 {
-	if (atomic_dec_and_test(&conn->refcnt))
-		sdp_conn_destruct(conn);
+	int noref = atomic_dec_and_test(&conn->refcnt);
+	BUG_ON(noref);
 }
 
+void sdp_conn_put(struct sdp_sock *conn);
+
 static inline void *hashent_arg(s32 hashent) 
 {
 	return (void *)(unsigned long)hashent;

-- 
MST


From mst at mellanox.co.il  Wed Jul  6 11:03:32 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 6 Jul 2005 21:03:32 +0300
Subject: [openib-general] [PATCH 2/2] sdp_conn_put/sdp_conn_hold race
	condition
Message-ID: <20050706180332.GA8965@mellanox.co.il>

Use sdp_conn_put_light in cases when we know there is another
reference on the connection.
Rule: sdp_conn_put should never be called on a locked connection:
since its safe to sdp_conn_unlock, this is not the last reference.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: ulp/sdp/sdp_actv.c
===================================================================
--- ulp/sdp/sdp_actv.c	(revision 2784)
+++ ulp/sdp/sdp_actv.c	(working copy)
@@ -325,7 +325,7 @@ error:
 
 	if (conn->state == SDP_CONN_ST_CLOSED) {
 		conn->cm_id = NULL;
-		sdp_conn_put(conn); /* CM reference */
+		sdp_conn_put_light(conn); /* CM reference */
 
 		return -EPROTO;
 	}
@@ -547,7 +547,7 @@ int sdp_cm_connect(struct sdp_sock *conn
 		/*
 		 * callback dosn't have this socket.
 		 */
-		sdp_conn_put(conn); /* address resolution reference */
+		sdp_conn_put_light(conn); /* address resolution reference */
 
 		return -EDESTADDRREQ;
 	}
Index: ulp/sdp/sdp_event.c
===================================================================
--- ulp/sdp/sdp_event.c	(revision 2784)
+++ ulp/sdp/sdp_event.c	(working copy)
@@ -187,7 +187,7 @@ static void sdp_cm_to_error(struct sdp_s
 {
 	sdp_conn_inet_error(conn, -ECONNRESET);
 	conn->cm_id = NULL;
-	sdp_conn_put(conn);	/* CM reference */
+	sdp_conn_put_light(conn);	/* CM reference */
 }
 /*
  * Connection establishment IB/CM callback functions
@@ -228,7 +228,7 @@ static int sdp_cm_idle(struct ib_cm_id *
 	}
 
 	conn->cm_id = NULL;
-	sdp_conn_put(conn); /* CM reference */
+	sdp_conn_put_light(conn); /* CM reference */
 	return -ENOENT;     /* ensure CM cleans-up identifier. */
 }
 

-- 
MST


From libor at topspin.com  Tue Jul 19 16:29:08 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 16:29:08 -0700
Subject: [openib-general] Re: [PATCH] SDP: use linux/list.h for binds
In-Reply-To: <20050719230923.GD17097@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Jul 20, 2005 at 02:09:23AM +0300
References: <1121366046.17302.9.camel@duffman>
	<20050719160345.B2231@topspin.com>
	<20050719230923.GD17097@mellanox.co.il>
Message-ID: <20050719162908.E2231@topspin.com>

On Wed, Jul 20, 2005 at 02:09:23AM +0300, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: [PATCH] SDP: use linux/list.h for binds
> > 
> > On Thu, Jul 14, 2005 at 11:34:06AM -0700, Tom Duffy wrote:
> > > This patch replaces the null terminated linked list for connections
> > > bound to a port and instead uses the list.h library from core Linux.
> > > 
> > > I have tested this on a few machines doing several ttcp's at once, but
> > > please inspect to verify correctness.
> > 
> >   Thanks. I've applied the patch since it's better then what was there.
> > However, the longer term solution needs to use a full hash table, the 
> > linear list is problematic when there are a lot of connections, and each
> > port_get() needs to traverse the entire list to check for collisions.
> 
> Or a tree?

  A tree would be fine as well, I was just thinking anything that has
a src_port lookup time of O(log n) instead of O(n). Also the tcp code
uses it's own hash table for it's port lookup. (net/ipv4/tcp_ipv4.c)
That seems like it might be overkill...

-Libor


From libor at topspin.com  Tue Jul 19 16:47:20 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 16:47:20 -0700
Subject: [openib-general] Re: [PATCH] ucm: Eliminate NULL checks prior to
	calling kfree
In-Reply-To: <1121257113.4401.60.camel@hal.voltaire.com>;
	from halr@voltaire.com on Wed, Jul 13, 2005 at 08:18:34AM -0400
References: <1121257113.4401.60.camel@hal.voltaire.com>
Message-ID: <20050719164720.F2231@topspin.com>

On Wed, Jul 13, 2005 at 08:18:34AM -0400, Hal Rosenstock wrote:
> ucm: Eliminate NULL checks prior to calling kfree
> 

  Thanks, I didn't realize it made the null check internally. Applied
and commited.

-Libor


From libor at topspin.com  Tue Jul 19 16:53:41 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 16:53:41 -0700
Subject: [openib-general] Re: [PATCHv2] libibcm: Fix some typos
In-Reply-To: <1121296463.4423.33.camel@hal.voltaire.com>;
	from halr@voltaire.com on Wed, Jul 13, 2005 at 07:14:24PM -0400
References: <1121296463.4423.33.camel@hal.voltaire.com>
Message-ID: <20050719165341.G2231@topspin.com>

On Wed, Jul 13, 2005 at 07:14:24PM -0400, Hal Rosenstock wrote:
> libibcm: Fix some typos

  Thanks, applied and commited.

-Libor


From libor at topspin.com  Tue Jul 19 17:14:53 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 17:14:53 -0700
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when debug
	level set
In-Reply-To: <1121298005.4423.56.camel@hal.voltaire.com>;
	from halr@voltaire.com on Wed, Jul 13, 2005 at 07:40:07PM -0400
References: <1121298005.4423.56.camel@hal.voltaire.com>
Message-ID: <20050719171453.H2231@topspin.com>

On Wed, Jul 13, 2005 at 07:40:07PM -0400, Hal Rosenstock wrote:
> ucm: Only print messages when debug level set

  I don't think you can assume that ucm_debug_level will always be 
initialized correctly, or is module_param_named clearing it? Otherwise
looks fine...

-Libor 

> Signed-off-by: Hal Rosenstock <halr at voltaire.com>
> 
> --- ucm.c.kfree	2005-07-13 08:10:00.000000000 -0400
> +++ ucm.c	2005-07-13 19:39:04.000000000 -0400
> @@ -49,6 +49,11 @@ MODULE_AUTHOR("Libor Michalek");
>  MODULE_DESCRIPTION("InfiniBand userspace Connection Manager access");
>  MODULE_LICENSE("Dual BSD/GPL");
>  
> +static int ucm_debug_level;
> +
> +module_param_named(debug_level, ucm_debug_level, int, 0644);
> +MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0");
> +


From libor at topspin.com  Tue Jul 19 17:30:41 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 17:30:41 -0700
Subject: [openib-general] Re:[ULP] how to choose appropriate ULPs for
	application
In-Reply-To: <6.2.0.14.2.20050713100906.0263f6f8@esmail.cup.hp.com>;
	from krause@cup.hp.com on Wed, Jul 13, 2005 at 10:17:18AM -0700
References: <BAY14-F68ED824B92D65272CBCE0A1DF0@phx.gbl>
	<Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.netapp.com>
	<Pine.LNX.4.61.0507130920130.25603@jlentini-linux.nane.neta
	pp.com> <6.2.0.14.2.20050713100906.0263f6f8@esmail.cup.hp.com>
Message-ID: <20050719173041.I2231@topspin.com>

On Wed, Jul 13, 2005 at 10:17:18AM -0700, Michael Krause wrote:
> At 06:39 AM 7/13/2005, James Lentini wrote:
> >On Tue, 12 Jul 2005, xg wang wrote:
> >
> >>     Frankly speaking, I can not distinguish the function of SDP and 
> >> DAPL. Since Lustre is a file system, it runs on kernel. So I think maybe 
> >> kDAPL is better.
> >
> >SDP stands for the Sockets Direct Protocol. The protocol is designed to 
> >support the Berkley Sockets API. This allows code already using the 
> >Sockets API to easily use InfiniBand by simply changing the socket type.
> 
> One clarification: SDP supports both synchronous and asynchronous 
> sockets.  The OpenGroup ICSC released the sockets extensions API a number 
> of months back that enables the full performance provided by SDP to be 
> tapped.  The SDP specifications can be found at the IBTA for IB and at the 
> RDMAC for RNIC web sites.

  I'd like to add that this SDP implementation does have asynchronous
sockets support in the form of linux 2.6 AIO, and there's even an
example of how to use AIO with sockets. Although I don't have time in
the immediate future it should be rather straight forward to implement
the OpenGroup ICSC sockets extensions API on top of linux AIO without
needing to make any kernel modifications. (with the possible exception
of the map/unmap calls) If anyones interested in doing this, I'd be
happy to discuss/help. 

  As you say the full performance only shows through when using
asynchronous sockets.


-Libor


From halr at voltaire.com  Tue Jul 19 17:34:17 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 19 Jul 2005 20:34:17 -0400
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when debug
	level set
In-Reply-To: <20050719171453.H2231@topspin.com>
References: <1121298005.4423.56.camel@hal.voltaire.com>
	<20050719171453.H2231@topspin.com>
Message-ID: <1121819656.4419.11173.camel@hal.voltaire.com>

On Tue, 2005-07-19 at 20:14, Libor Michalek wrote:
> On Wed, Jul 13, 2005 at 07:40:07PM -0400, Hal Rosenstock wrote:
> > ucm: Only print messages when debug level set
> 
>   I don't think you can assume that ucm_debug_level will always be 
> initialized correctly, or is module_param_named clearing it? Otherwise
> looks fine...

Same is true for IPoIB so it would need changing in both places if so.

-- Hal



From libor at topspin.com  Tue Jul 19 17:58:10 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 17:58:10 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>;
	from jlentini@netapp.com on Thu, Jul 14, 2005 at 02:04:52PM -0400
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
	<20050713151216.GB649@lst.de>
	<Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>
Message-ID: <20050719175810.J2231@topspin.com>

On Thu, Jul 14, 2005 at 02:04:52PM -0400, James Lentini wrote:
> On Wed, 13 Jul 2005, Christoph Hellwig wrote:
> > On Wed, Jul 13, 2005 at 10:18:41AM -0400, James Lentini wrote:
> >> In addition to the streamlined connection management interface, kDAPL
> >> also provides a unified event model. This is another feature that the
> >> verbs do not provide.
> >
> > It's probably the biggest mis-feature and would have to go aswell if
> > we mæssage what started as kdapl into the kernel rdma layer.
> 
> When the DAT Collaborative solicited requirements for their API, RDMA 
> consumers requested this feature. The idea is that a single 
> event source is a cleaner, simpler programming model.

  I'm not sure how it originally came to be, but it seems like a feature
that made sense in userspace and got moved to kernel space because of
the, in my opinion, misguided desire to have a unified kernel/userspace
API. Having written to the userspace DAPL, and kerel verbs/cm APIs, I would
not want to use the proposed event model in the kernel. Getting the locking
correct and good performance is difficult enough without your APIs hiding 
the source of your events.


-Libor



From libor at topspin.com  Tue Jul 19 18:05:05 2005
From: libor at topspin.com (Libor Michalek)
Date: Tue, 19 Jul 2005 18:05:05 -0700
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>;
	from sean.hefty@intel.com on Tue, Jul 12, 2005 at 03:00:59PM -0700
References: <1121203262.14638.16.camel@duffman>
	<ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
Message-ID: <20050719180505.K2231@topspin.com>

On Tue, Jul 12, 2005 at 03:00:59PM -0700, Sean Hefty wrote:
> >This would also be a good time to get NFSoRDMA ported to kDAPL as the
> >API is unlikely to change too much in the future.
> 
> I'm not as sure about the API.  Is it necessary to abstract away QPs,
> CQs, etc. from the user?  (The answer may depend on the intersection
> of the Infiniband and RNIC APIs.)  Is it possible to just extract out
> the kDAPL connection semantics and layer it directly over verbs?  Thus
> an app could talk directly to verbs and the kDAPL connection manager,
> rather than verbs and the kernel CM and SA client

  I think you hit the nail on the head. Ideally we would have a unified
verbs API, and then a seperate generic connection manager. The verb API
is already similar between the two transports, the real question is how
do ULPs establish connections in a transport neutral manner. In my mind
forget the other stuff, focus on the real issue.


-Libor


From caitlin.bestler at gmail.com  Tue Jul 19 22:26:12 2005
From: caitlin.bestler at gmail.com (Caitlin Bestler)
Date: Wed, 20 Jul 2005 07:26:12 +0200
Subject: [openib-general] kDAPL: ready for gen2/trunk/ inclusion?
In-Reply-To: <20050719175810.J2231@topspin.com>
References: <ORSMSX401FRaqbC8wSA00000046@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507131013220.25603@jlentini-linux.nane.netapp.com>
	<20050713151216.GB649@lst.de>
	<Pine.LNX.4.61.0507141340370.25603@jlentini-linux.nane.netapp.com>
	<20050719175810.J2231@topspin.com>
Message-ID: <469958e0050719222630b6e1f0@mail.gmail.com>

On 7/20/05, Libor Michalek <libor at topspin.com> wrote:
> On Thu, Jul 14, 2005 at 02:04:52PM -0400, James Lentini wrote:
> > On Wed, 13 Jul 2005, Christoph Hellwig wrote:
> > > On Wed, Jul 13, 2005 at 10:18:41AM -0400, James Lentini wrote:
> > >> In addition to the streamlined connection management interface, kDAPL
> > >> also provides a unified event model. This is another feature that the
> > >> verbs do not provide.
> > >
> > > It's probably the biggest mis-feature and would have to go aswell if
> > > we mæssage what started as kdapl into the kernel rdma layer.
> >
> > When the DAT Collaborative solicited requirements for their API, RDMA
> > consumers requested this feature. The idea is that a single
> > event source is a cleaner, simpler programming model.
> 
>   I'm not sure how it originally came to be, but it seems like a feature
> that made sense in userspace and got moved to kernel space because of
> the, in my opinion, misguided desire to have a unified kernel/userspace
> API. Having written to the userspace DAPL, and kerel verbs/cm APIs, I would
> not want to use the proposed event model in the kernel. Getting the locking
> correct and good performance is difficult enough without your APIs hiding
> the source of your events.
> 
> 

The fact that it is difficult is exactly why the consumer of kDAPL did
not want to deal with it. They wanted to be able to apply their application
specific logic to an infrastructure that was optimized for the platform.
It seems many of them felt that their application was complex enough
that they preferred not to have to become experts on the OS as well.

While some may consider development of kernel level OS neutral 
applications to be impractical, it is by far from unaminous. Embedded
developers universally prefer this model. It was also the primary mode
of development envisioned by the first DAT developers. The Sourceforge
reference implementation started as a KDAPL implmenetation. One of
the primary motivators in coming up with a new forum to define an RDMA
API was that the existing Kernel VIA interface was subject to proprietary
control. If developers had been confident that they could do everything
they needed in user mode they would have refined VIPL, not created DAT.

It is also important to note that these developers are only asking for
an OS neutral API -- not an OS neutral implementation. It is *their*
code that will deal with OS variants on other fronts, not the KDAPL
code. They only ask that kDAPL exist as a transport *and* OS neutral
API to enable them to concentrate on their application logic.

The evd callback does exactly that. It allows the developer to not
know whether the callback is in an ISR, bottom half or kernel thread.
Instead the application can control the degree, if any, that multiple
callbacks can occur from a single EVD (default is one). This allows
the typical kDAPL application to avoid locks altogether.

Adding a transport neutral CM interface is indeed a required part of
the "verb layer" transport neutral API. And that lower level API should
not deal with things like EVDs. And I still think that there are two
valid approaches to coming up with that lower level API: subset
kDAPL or expand the gen2 verbs.

But both of those will take time. Meanwhile kDAPL as already defined
is available today. It is suitable for almost all kernel RDMA development.
The fact that a more minimalist interface will be refined over the coming
quarters is not a reason that developers should not start using kDAPL
now. As good code engineering we should ensure that kDAPL is built
on top of the evolving verb layer transport. But in the meantime kDAPL
as is already provides transport neutral support. Any lower layer solution
will take time, especially for iWARP vendors.

The goal of encouraging RDMA development is not served by telling 
developers that they'll have to use some variant of kDAPL that is still
under discussion. Telling them that we will support kDAPL (with at
most a few exclusions) and then supplement it with transport neutral
verbs and CM interface for those that need more precise control will
encourage development.


From dotanb at mellanox.co.il  Tue Jul 19 22:57:58 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Wed, 20 Jul 2005 08:57:58 +0300
Subject: [openib-general] why does the value of the node_guid don't have the
	machine endian ess?
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A23C1@mtlex01.yok.mtl.com>

why do i need to change the endianess of node_guid when i use it from
query_hca?

doesn't the driver need to do it (only once)?


thanks
Dotan Barak
Software Verification Engineer
Mellanox Technologies LTD
mailto:dotanb at mellanox.co.il
Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
P.O. Box 86 Yokneam 20692 ISRAEL.
Home: +972-4-8289408  Cell: 052-4222383

[ May the fork be with you ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050720/ce8e2df6/attachment.html>

From mst at mellanox.co.il  Wed Jul 20 00:35:07 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 10:35:07 +0300
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when debug
	level set
In-Reply-To: <1121819656.4419.11173.camel@hal.voltaire.com>
References: <1121298005.4423.56.camel@hal.voltaire.com>
	<20050719171453.H2231@topspin.com>
	<1121819656.4419.11173.camel@hal.voltaire.com>
Message-ID: <20050720073507.GA25957@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: Re: [PATCHv3] ucm: Only print messages when debug level set
> 
> On Tue, 2005-07-19 at 20:14, Libor Michalek wrote:
> > On Wed, Jul 13, 2005 at 07:40:07PM -0400, Hal Rosenstock wrote:
> > > ucm: Only print messages when debug level set
> > 
> >   I don't think you can assume that ucm_debug_level will always be 
> > initialized correctly, or is module_param_named clearing it? Otherwise
> > looks fine...
> 
> Same is true for IPoIB so it would need changing in both places if so.
> 
> -- Hal

I always thought that in C global storage is zero-initialized.
What am I missing?

-- 
MST


From glebn at voltaire.com  Wed Jul 20 00:40:44 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 20 Jul 2005 10:40:44 +0300
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when
	debug level set
In-Reply-To: <20050720073507.GA25957@mellanox.co.il>
References: <1121298005.4423.56.camel@hal.voltaire.com>
	<20050719171453.H2231@topspin.com>
	<1121819656.4419.11173.camel@hal.voltaire.com>
	<20050720073507.GA25957@mellanox.co.il>
Message-ID: <20050720074044.GM28598@minantech.com>

On Wed, Jul 20, 2005 at 10:35:07AM +0300, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > Subject: Re: [PATCHv3] ucm: Only print messages when debug level set
> > 
> > On Tue, 2005-07-19 at 20:14, Libor Michalek wrote:
> > > On Wed, Jul 13, 2005 at 07:40:07PM -0400, Hal Rosenstock wrote:
> > > > ucm: Only print messages when debug level set
> > > 
> > >   I don't think you can assume that ucm_debug_level will always be 
> > > initialized correctly, or is module_param_named clearing it? Otherwise
> > > looks fine...
> > 
> > Same is true for IPoIB so it would need changing in both places if so.
> > 
> > -- Hal
> 
> I always thought that in C global storage is zero-initialized.
> What am I missing?
> 
Nothing, and if you initialise it explicitly you needlessly enlarge BSS.
There was kernel patches to remove zero initialisations while ago.

--
			Gleb.


From mst at mellanox.co.il  Wed Jul 20 01:39:56 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 11:39:56 +0300
Subject: [openib-general] Re: Error in opening IB cm device
In-Reply-To: <Pine.GSO.4.40.0507151205500.5849-100000@kappa.cse.ohio-state.edu>
References: <1121409378.20046.260.camel@localhost>
	<Pine.GSO.4.40.0507151205500.5849-100000@kappa.cse.ohio-state.edu>
Message-ID: <20050720083956.GB25957@mellanox.co.il>

Quoting r. amith rajith mamidala <mamidala at cse.ohio-state.edu>:
> Subject: Re: Error in opening IB cm device
> 
> Hi,
> 
> I couldn't find /dev/infiniband in my installation. After adding this
> manually and doing mknod as you suggested the error in opening the
> /dev/infiniband/ucm disappeared.

You shouldnt need /dev/infiniband/ucm for ibv_rc_pingpong to work
Delete /usr/local/lib/infiniband/libucm.so or move it to
/usr/local/lib/libucm.so

> However, now the program
> (ibv_rc_pingpong) is not able to create QP.

What kind of error do you see?

> [....]
> 
> However I was not able to add the udev rules as specified in the following
> instruction. I could not locate the /etc/udev/udev.rules.

Maybe you have /etc/udev/rules.d/?
Create a new file there, name it something like 60-infiniband.rules, and put

KERNEL="umad*", NAME="infiniband/%k"
KERNEL="issm*", NAME="infiniband/%k"
KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"

I have also put up a short page with installation instructions on the
Wiki. Check it out.

https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet


-- 
MST


From halr at voltaire.com  Wed Jul 20 03:42:35 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 20 Jul 2005 06:42:35 -0400
Subject: [openib-general] Re: Error in opening IB cm device
In-Reply-To: <20050720083956.GB25957@mellanox.co.il>
References: <1121409378.20046.260.camel@localhost>
	<Pine.GSO.4.40.0507151205500.5849-100000@kappa.cse.ohio-state.edu>
	<20050720083956.GB25957@mellanox.co.il>
Message-ID: <1121856154.4419.14792.camel@hal.voltaire.com>

On Wed, 2005-07-20 at 04:39, Michael S. Tsirkin wrote:
> Quoting r. amith rajith mamidala <mamidala at cse.ohio-state.edu>:
> > Subject: Re: Error in opening IB cm device
> > 
> > Hi,
> > 
> > I couldn't find /dev/infiniband in my installation. After adding this
> > manually and doing mknod as you suggested the error in opening the
> > /dev/infiniband/ucm disappeared.
> 
> You shouldnt need /dev/infiniband/ucm for ibv_rc_pingpong to work
> Delete /usr/local/lib/infiniband/libucm.so

This needs to be done manually.

> or move it to /usr/local/lib/libucm.so

There was a patch committed to cause the build to place the ucm library
in /usr/local/lib rather than /usr/local/lib/infiniband yesterday.

-- Hal



From halr at voltaire.com  Wed Jul 20 03:47:31 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 20 Jul 2005 06:47:31 -0400
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when debug
	level set
In-Reply-To: <20050720073507.GA25957@mellanox.co.il>
References: <1121298005.4423.56.camel@hal.voltaire.com>
	<20050719171453.H2231@topspin.com>
	<1121819656.4419.11173.camel@hal.voltaire.com>
	<20050720073507.GA25957@mellanox.co.il>
Message-ID: <1121856451.4419.14829.camel@hal.voltaire.com>

On Wed, 2005-07-20 at 03:35, Michael S. Tsirkin wrote:
> Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > Subject: Re: [PATCHv3] ucm: Only print messages when debug level set
> > 
> > On Tue, 2005-07-19 at 20:14, Libor Michalek wrote:
> > > On Wed, Jul 13, 2005 at 07:40:07PM -0400, Hal Rosenstock wrote:
> > > > ucm: Only print messages when debug level set
> > > 
> > >   I don't think you can assume that ucm_debug_level will always be 
> > > initialized correctly, or is module_param_named clearing it? Otherwise
> > > looks fine...
> > 
> > Same is true for IPoIB so it would need changing in both places if so.
> > 
> > -- Hal
> 
> I always thought that in C global storage is zero-initialized.
> What am I missing?

Right:
 
"A static object not explicitly initialized is initialized as if it (or
its members) were assigned the constant 0."

So this patch is OK to commit (and IPoIB is fine in terms of this).

-- Hal



From halr at voltaire.com  Wed Jul 20 04:30:54 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 20 Jul 2005 07:30:54 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42DD49A9.9060600@ichips.intel.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
	<1121351537.4423.1162.camel@hal.voltaire.com>
	<42D6F3F5.5020806@ichips.intel.com>
	<1121425731.4423.2145.camel@hal.voltaire.com>
	<42D7EC83.8030406@ichips.intel.com>
	<1121695663.4419.6024.camel@hal.voltaire.com>
	<42DD49A9.9060600@ichips.intel.com>
Message-ID: <1121859053.4419.15083.camel@hal.voltaire.com>

On Tue, 2005-07-19 at 14:42, Arlin Davis wrote:
> Yes, it returns the route data (GID) when the return == 1.  However, I 
> still get a callback event with a req_id ==0 which is somewhat 
> unexpected. This seems unnecessary given that ib_at_route_by_ip() just 
> returned the data. Is this the expected behavior?
> 
> Here is my output from the call and from the at_thread.
> 
> 4116 Running as client
> 4116 Server Name: iclust-20-ib0
> 4116 Server Net Address: 192.168.0.20
> dapl_ep_connect (0x515c40, {4294967232.4294967208.0.20}, 115D3, 5000000, 
> 0, (nil), 0, 0)
>  connect: r_SID 71123, pdata (nil), plen 0
>  connect: at_route ret=1, req_id 0, GID fe80000000000000 
> 0002c90200004071   <<<< ib_route info returned with call
> dapl_ep_connect () returns 0x0
> 4116 Waiting for connect response
> 
> at_thread: get_callback
>  rt_comp_handler: conn 0x519e30, req_id 0 rec_num 1                 
>  rt_comp_handler: SRC GID subnet fe80000000000000 id 0002c9020000409d    
> <<< get_callback event fires
>  rt_comp_handler: DST GID subnet fe80000000000000 id 0002c90200004071
>  at_thread: callback woke

Try this incremental patch:

Index: uat.c
===================================================================
--- uat.c	(revision 2884)
+++ uat.c	(working copy)
@@ -29,7 +29,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id:$
+ * $Id$
  */
 #include <linux/init.h>
 #include <linux/fs.h>
@@ -276,9 +276,10 @@ static ssize_t ib_uat_route_by_ip(struct
 		if (copy_to_user(ctx->user_ib_route, ctx->ib_route,
 				 sizeof(*ctx->user_ib_route))) {
 			result = -EFAULT;
+			kfree(ctx->ib_route);
+			ib_uat_ctx_put(ctx);
 			goto err3;
 		}
-		ib_uat_route_callback(ctx->req_id, ctx, result);
 	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
@@ -374,9 +375,10 @@ static ssize_t ib_uat_paths_by_route(str
 		if (copy_to_user(ctx->user_path_arr, ctx->path_arr,
 				 ctx->user_length)) {
 			result = -EFAULT;
+			kfree(ctx->path_arr);
+			ib_uat_ctx_put(ctx);
 			goto err3;
 		}
-		ib_uat_path_callback(ctx->req_id, ctx, result);
 	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
@@ -469,9 +471,10 @@ static ssize_t ib_uat_ips_by_gid(struct 
 		/* Copy IP addresses back to userspace */
 		if (copy_to_user(ctx->user_ips, ctx->ips, ctx->user_length)) {
 			result = -EFAULT;
+			kfree(ctx->ips);
+			ib_uat_ctx_put(ctx);
 			goto err3;
 		}
-		ib_uat_ips_callback(ctx->req_id, ctx, result);
 	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;





From halr at voltaire.com  Wed Jul 20 05:35:16 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 20 Jul 2005 08:35:16 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121859053.4419.15083.camel@hal.voltaire.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>
	<1121351537.4423.1162.camel@hal.voltaire.com>
	<42D6F3F5.5020806@ichips.intel.com>
	<1121425731.4423.2145.camel@hal.voltaire.com>
	<42D7EC83.8030406@ichips.intel.com>
	<1121695663.4419.6024.camel@hal.voltaire.com>
	<42DD49A9.9060600@ichips.intel.com>
	<1121859053.4419.15083.camel@hal.voltaire.com>
Message-ID: <1121862915.4419.15471.camel@hal.voltaire.com>

On Wed, 2005-07-20 at 07:30, Hal Rosenstock wrote:
> On Tue, 2005-07-19 at 14:42, Arlin Davis wrote:
> > Yes, it returns the route data (GID) when the return == 1.  However, I 
> > still get a callback event with a req_id ==0 which is somewhat 
> > unexpected. This seems unnecessary given that ib_at_route_by_ip() just 
> > returned the data. Is this the expected behavior?
> > 
> > Here is my output from the call and from the at_thread.
> > 
> > 4116 Running as client
> > 4116 Server Name: iclust-20-ib0
> > 4116 Server Net Address: 192.168.0.20
> > dapl_ep_connect (0x515c40, {4294967232.4294967208.0.20}, 115D3, 5000000, 
> > 0, (nil), 0, 0)
> >  connect: r_SID 71123, pdata (nil), plen 0
> >  connect: at_route ret=1, req_id 0, GID fe80000000000000 
> > 0002c90200004071   <<<< ib_route info returned with call
> > dapl_ep_connect () returns 0x0
> > 4116 Waiting for connect response
> > 
> > at_thread: get_callback
> >  rt_comp_handler: conn 0x519e30, req_id 0 rec_num 1                 
> >  rt_comp_handler: SRC GID subnet fe80000000000000 id 0002c9020000409d    
> > <<< get_callback event fires
> >  rt_comp_handler: DST GID subnet fe80000000000000 id 0002c90200004071
> >  at_thread: callback woke
> 
> Try this incremental patch:

This is v2 of this incremental patch:

Index: uat.c
===================================================================
--- uat.c	(revision 2884)
+++ uat.c	(working copy)
@@ -29,7 +29,7 @@
  * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  * SOFTWARE.
  *
- * $Id:$
+ * $Id$
  */
 #include <linux/init.h>
 #include <linux/fs.h>
@@ -276,9 +276,12 @@ static ssize_t ib_uat_route_by_ip(struct
 		if (copy_to_user(ctx->user_ib_route, ctx->ib_route,
 				 sizeof(*ctx->user_ib_route))) {
 			result = -EFAULT;
+			kfree(ctx->ib_route);
+			ib_uat_ctx_put(ctx);
 			goto err3;
 		}
-		ib_uat_route_callback(ctx->req_id, ctx, result);
+		kfree(ctx->ib_route);
+		ib_uat_ctx_put(ctx);
 	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
@@ -374,9 +377,12 @@ static ssize_t ib_uat_paths_by_route(str
 		if (copy_to_user(ctx->user_path_arr, ctx->path_arr,
 				 ctx->user_length)) {
 			result = -EFAULT;
+			kfree(ctx->path_arr);
+			ib_uat_ctx_put(ctx);
 			goto err3;
 		}
-		ib_uat_path_callback(ctx->req_id, ctx, result);
+		kfree(ctx->path_arr);
+		ib_uat_ctx_put(ctx);
 	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;
@@ -469,9 +475,12 @@ static ssize_t ib_uat_ips_by_gid(struct 
 		/* Copy IP addresses back to userspace */
 		if (copy_to_user(ctx->user_ips, ctx->ips, ctx->user_length)) {
 			result = -EFAULT;
+			kfree(ctx->ips);
+			ib_uat_ctx_put(ctx);
 			goto err3;
 		}
-		ib_uat_ips_callback(ctx->req_id, ctx, result);
+		kfree(ctx->ips);
+		ib_uat_ctx_put(ctx);
 	} else if (result < 0) {
 		ib_uat_ctx_put(ctx);
 		goto err3;





From halr at voltaire.com  Wed Jul 20 05:45:47 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 20 Jul 2005 08:45:47 -0400
Subject: [openib-general] [PATCH] ucm README: update library location
Message-ID: <1121863545.4419.15537.camel@hal.voltaire.com>

ucm library location is /usr/local/lib rather than
/usr/local/lib/infiniband

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: README
===================================================================
--- README      (revision 2878)
+++ README      (working copy)
@@ -8,7 +8,7 @@
 Typically the autogen and configure steps only need be done the first
 time unless configure.in or Makefile.am changes.
 
-Libraries are installed by default at /usr/local/lib/infiniband.
+Libraries are installed by default at /usr/local/lib.
 
 Device files
 




From glebn at voltaire.com  Wed Jul 20 06:56:33 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 20 Jul 2005 16:56:33 +0300
Subject: [openib-general] [PATH] problem dlopen libibverbs.so
Message-ID: <20050720135633.GN28598@minantech.com>

Hello,

 There is a problem with ibverbs library if it is not linked with an
application but dynamically loaded by the application in runtime by
dlopen(). It can't load userspace device-specific driver in later case.

Two programs are attached to demonstrate the problem. Compile first one
with:
$ gcc -libverbs linked.c -o linked
Compile second one with:
$ gcc dlopened.c -ldl -o dlopened

$ ./linked
Found 1 devices
$ ./dlopened
libibverbs: Warning: no userspace device-specific driver found for uverbs0
        driver search path: /usr/local/lib/infiniband
Found 0 devices

It appears that libmthca relies on back linking to find symbols from
libibverbs but back linking doesn't work in the later case.

I linked libmthca with libibverbs and this solved the problem.


Index: src/userspace/libmthca/Makefile.am
===================================================================
--- src/userspace/libmthca/Makefile.am	(revision 2885)
+++ src/userspace/libmthca/Makefile.am	(working copy)
@@ -13,7 +13,7 @@
 endif
 
 src_mthca_la_SOURCES = src/ah.c src/cq.c src/memfree.c src/mthca.c src/qp.c src/verbs.c
-src_mthca_la_LDFLAGS = -avoid-version -module \
+src_mthca_la_LDFLAGS = -avoid-version -module -libverbs\
     $(mthca_version_script)
 
 DEBIAN = debian/changelog debian/compat debian/control debian/copyright \
--
			Gleb.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linked.c
Type: text/x-csrc
Size: 237 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050720/0e61a844/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dlopened.c
Type: text/x-csrc
Size: 399 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050720/0e61a844/attachment-0001.c>

From mst at mellanox.co.il  Wed Jul 20 07:24:22 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 17:24:22 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720135633.GN28598@minantech.com>
References: <20050720135633.GN28598@minantech.com>
Message-ID: <20050720142422.GA27124@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> It appears that libmthca relies on back linking to find symbols from
> libibverbs but back linking doesn't work in the later case.
> 
> I linked libmthca with libibverbs and this solved the problem.

I wander, however, what happends if you load libibverbs from path
a, and mthca is linked with libibverbs from path b?
Any idea?

-- 
MST



From glebn at voltaire.com  Wed Jul 20 07:28:29 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 20 Jul 2005 17:28:29 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720142422.GA27124@mellanox.co.il>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
Message-ID: <20050720142829.GO28598@minantech.com>

On Wed, Jul 20, 2005 at 05:24:22PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > It appears that libmthca relies on back linking to find symbols from
> > libibverbs but back linking doesn't work in the later case.
> > 
> > I linked libmthca with libibverbs and this solved the problem.
> 
> I wander, however, what happends if you load libibverbs from path
> a, and mthca is linked with libibverbs from path b?
> Any idea?
> 
If two libibverbs are different it can fail or even crash I think.
This is the same as if libibverb tries to load libmthca compiled from
different svn revision.

--
			Gleb.


From mst at mellanox.co.il  Wed Jul 20 07:57:28 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 17:57:28 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720142829.GO28598@minantech.com>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
Message-ID: <20050720145728.GA27398@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> Subject: Re: [PATH] problem dlopen libibverbs.so
> 
> On Wed, Jul 20, 2005 at 05:24:22PM +0300, Michael S. Tsirkin wrote:
> > Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > It appears that libmthca relies on back linking to find symbols from
> > > libibverbs but back linking doesn't work in the later case.
> > > 
> > > I linked libmthca with libibverbs and this solved the problem.
> > 
> > I wander, however, what happends if you load libibverbs from path
> > a, and mthca is linked with libibverbs from path b?
> > Any idea?
>
> If two libibverbs are different it can fail or even crash I think.
> This is the same as if libibverb tries to load libmthca compiled from
> different svn revision.

May be a problem then: it might be quite hard to debug such crashes.

What about passing RTLD_GLOBAL to dlopen, instead?


-- 
MST


From glebn at voltaire.com  Wed Jul 20 08:02:04 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 20 Jul 2005 18:02:04 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720145728.GA27398@mellanox.co.il>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
Message-ID: <20050720150204.GP28598@minantech.com>

On Wed, Jul 20, 2005 at 05:57:28PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > Subject: Re: [PATH] problem dlopen libibverbs.so
> > 
> > On Wed, Jul 20, 2005 at 05:24:22PM +0300, Michael S. Tsirkin wrote:
> > > Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > It appears that libmthca relies on back linking to find symbols from
> > > > libibverbs but back linking doesn't work in the later case.
> > > > 
> > > > I linked libmthca with libibverbs and this solved the problem.
> > > 
> > > I wander, however, what happends if you load libibverbs from path
> > > a, and mthca is linked with libibverbs from path b?
> > > Any idea?
> >
> > If two libibverbs are different it can fail or even crash I think.
> > This is the same as if libibverb tries to load libmthca compiled from
> > different svn revision.
> 
> May be a problem then: it might be quite hard to debug such crashes.
> 
Like any other problem caused by mismatched libraries.

> What about passing RTLD_GLOBAL to dlopen, instead?
The strange thing is it doesn't work, but it should according to man
page.

--
			Gleb.


From mst at mellanox.co.il  Wed Jul 20 08:05:43 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 18:05:43 +0300
Subject: [openib-general] [ANNOUNCE] mstflint updated
Message-ID: <20050720150543.GD27398@mellanox.co.il>

Hi!
With mstflint, you can now use mthca device names, as follows:
./mstflint -d /sys/class/infiniband/mthca0/device/resource0 q

Thanks,
MST

-- 
MST


From mst at mellanox.co.il  Wed Jul 20 08:23:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 18:23:49 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720150204.GP28598@minantech.com>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
Message-ID: <20050720152349.GE27398@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> Subject: Re: [PATH] problem dlopen libibverbs.so
> 
> On Wed, Jul 20, 2005 at 05:57:28PM +0300, Michael S. Tsirkin wrote:
> > Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > Subject: Re: [PATH] problem dlopen libibverbs.so
> > > 
> > > On Wed, Jul 20, 2005 at 05:24:22PM +0300, Michael S. Tsirkin wrote:
> > > > Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > > It appears that libmthca relies on back linking to find symbols from
> > > > > libibverbs but back linking doesn't work in the later case.
> > > > > 
> > > > > I linked libmthca with libibverbs and this solved the problem.
> > > > 
> > > > I wander, however, what happends if you load libibverbs from path
> > > > a, and mthca is linked with libibverbs from path b?
> > > > Any idea?
> > >
> > > If two libibverbs are different it can fail or even crash I think.
> > > This is the same as if libibverb tries to load libmthca compiled from
> > > different svn revision.
> > 
> > May be a problem then: it might be quite hard to debug such crashes.
>
> Like any other problem caused by mismatched libraries.

Right, but I think we dont want another dependency if it can work
without it.

> > What about passing RTLD_GLOBAL to dlopen, instead?
>
> The strange thing is it doesn't work, but it should according to man
> page.

Better debug it then.


-- 
MST


From glebn at voltaire.com  Wed Jul 20 08:27:36 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 20 Jul 2005 18:27:36 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720152349.GE27398@mellanox.co.il>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
Message-ID: <20050720152736.GQ28598@minantech.com>

On Wed, Jul 20, 2005 at 06:23:49PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > Subject: Re: [PATH] problem dlopen libibverbs.so
> > 
> > On Wed, Jul 20, 2005 at 05:57:28PM +0300, Michael S. Tsirkin wrote:
> > > Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > Subject: Re: [PATH] problem dlopen libibverbs.so
> > > > 
> > > > On Wed, Jul 20, 2005 at 05:24:22PM +0300, Michael S. Tsirkin wrote:
> > > > > Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > > > It appears that libmthca relies on back linking to find symbols from
> > > > > > libibverbs but back linking doesn't work in the later case.
> > > > > > 
> > > > > > I linked libmthca with libibverbs and this solved the problem.
> > > > > 
> > > > > I wander, however, what happends if you load libibverbs from path
> > > > > a, and mthca is linked with libibverbs from path b?
> > > > > Any idea?
> > > >
> > > > If two libibverbs are different it can fail or even crash I think.
> > > > This is the same as if libibverb tries to load libmthca compiled from
> > > > different svn revision.
> > > 
> > > May be a problem then: it might be quite hard to debug such crashes.
> >
> > Like any other problem caused by mismatched libraries.
> 
> Right, but I think we dont want another dependency if it can work
> without it.
Agreed.

> 
> > > What about passing RTLD_GLOBAL to dlopen, instead?
> >
> > The strange thing is it doesn't work, but it should according to man
> > page.
> 
> Better debug it then.
> 
It seems that RTLD_GLOBAL doesn't work as expected because libibverbs
calls ldopen from constructor and it is to early for RTLD_GLOBAL to take
effect. If I ldopen mthca.so after dlopen ("libibverbs.so", RTLD_NOW | RTLD_GLOBAL) 
returns it actually succeeds. But alas it it to late.

--
			Gleb.


From mst at mellanox.co.il  Wed Jul 20 08:45:09 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 18:45:09 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720152736.GQ28598@minantech.com>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
Message-ID: <20050720154509.GG27398@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > What about passing RTLD_GLOBAL to dlopen, instead?
> > >
> > > The strange thing is it doesn't work, but it should according to man
> > > page.
> > 
> > Better debug it then.
> > 
> It seems that RTLD_GLOBAL doesn't work as expected because libibverbs
> calls ldopen from constructor and it is to early for RTLD_GLOBAL to take
> effect. If I ldopen mthca.so after dlopen ("libibverbs.so", RTLD_NOW | RTLD_GLOBAL) 
> returns it actually succeeds. But alas it it to late.
> 

I wander if its a better idea to scan the plugin directory on ibv_get_devices.
Seems to me it might be more debuggable for applications if they get
error messages when they call ibv_get_devices, and not at startup:
they can step in debugger to figure out what is printing these things.

Roland, would you take such a patch?

-- 
MST


From rolandd at cisco.com  Wed Jul 20 08:50:49 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 20 Jul 2005 08:50:49 -0700
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when
	debug level set
In-Reply-To: <20050719171453.H2231@topspin.com> (Libor Michalek's message of
	"Tue, 19 Jul 2005 17:14:53 -0700")
References: <1121298005.4423.56.camel@hal.voltaire.com>
	<20050719171453.H2231@topspin.com>
Message-ID: <52oe8xxyqu.fsf@topspin.com>

    Libor>   I don't think you can assume that ucm_debug_level will
    Libor> always be initialized correctly, or is module_param_named
    Libor> clearing it? Otherwise looks fine...

static int foo;

gets put in BSS and is zeroed by the loader.

 - R.


From rolandd at cisco.com  Wed Jul 20 08:52:30 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 20 Jul 2005 08:52:30 -0700
Subject: [openib-general] Re: why does the value of the node_guid don't have
 the machine endian ess?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A23C1@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Wed, 20 Jul 2005 08:57:58 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A23C1@mtlex01.yok.mtl.com>
Message-ID: <52k6jlxyo1.fsf@topspin.com>

    Dotan> why do i need to change the endianess of node_guid when i
    Dotan> use it from query_hca?

Which endianess do you want to get the node_guid in?

 - R.


From rolandd at cisco.com  Wed Jul 20 09:28:48 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 20 Jul 2005 09:28:48 -0700
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720154509.GG27398@mellanox.co.il> (Michael S. Tsirkin's
	message of "Wed, 20 Jul 2005 18:45:09 +0300")
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
Message-ID: <52d5pdwif3.fsf@topspin.com>

    Michael> I wander if its a better idea to scan the plugin
    Michael> directory on ibv_get_devices.  Seems to me it might be
    Michael> more debuggable for applications if they get error
    Michael> messages when they call ibv_get_devices, and not at
    Michael> startup: they can step in debugger to figure out what is
    Michael> printing these things.

    Michael> Roland, would you take such a patch?

I was thinking the same thing -- lazy initialization of libibverbs
when it is first use.  So yes, such a patch would be useful.

 - R.


From mst at mellanox.co.il  Wed Jul 20 10:19:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 20:19:12 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <52d5pdwif3.fsf@topspin.com>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com>
Message-ID: <20050720171912.GA28359@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [PATH] problem dlopen libibverbs.so
> 
>     Michael> I wander if its a better idea to scan the plugin
>     Michael> directory on ibv_get_devices.  Seems to me it might be
>     Michael> more debuggable for applications if they get error
>     Michael> messages when they call ibv_get_devices, and not at
>     Michael> startup: they can step in debugger to figure out what is
>     Michael> printing these things.
> 
>     Michael> Roland, would you take such a patch?
> 
> I was thinking the same thing -- lazy initialization of libibverbs
> when it is first use.  So yes, such a patch would be useful.
> 
>  - R.
> 

Actually, instead of keeping a global devie list, wouldnt it be a good idea
to add something like ibv_free_devices and let applications call this when
they are done going over the device list?

-- 
MST


From damato at psc.edu  Wed Jul 20 10:38:13 2005
From: damato at psc.edu (Joe Damato)
Date: Wed, 20 Jul 2005 13:38:13 -0400
Subject: [openib-general] fmr vs. mr?
Message-ID: <42DE8C05.2030908@psc.edu>

Hello -
         I am attempting to port some code from gen1 to gen2, has fmr 
been deprecated? What -is- fmr and how does it differ from a mr? If I am 
correct the type struct ib_mr has become struct ibv_mr...

        Any insight (or information on where I can find more info about 
this) is greatly appreciated!

Thanks-
Joe Damato


From gdror at mellanox.co.il  Wed Jul 20 10:53:45 2005
From: gdror at mellanox.co.il (Dror Goldenberg)
Date: Wed, 20 Jul 2005 20:53:45 +0300
Subject: [openib-general] fmr vs. mr?
Message-ID: <506C3D7B14CDD411A52C00025558DED60842E302@mtlex01.yok.mtl.com>



From: Joe Damato [mailto:damato at psc.edu] 
> 
>          I am attempting to port some code from gen1 to gen2, has fmr 
> been deprecated? 

fmr is in part of gen2 stack. Look in ib_verbs.h for struct ib_fmr.

-Dror
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050720/56d858f2/attachment.html>

From ardavis at ichips.intel.com  Wed Jul 20 11:16:39 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Wed, 20 Jul 2005 11:16:39 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121862915.4419.15471.camel@hal.voltaire.com>
References: <ORSMSX401FRaqbC8wSA0000004f@orsmsx401.amr.corp.intel.com>	
	<1121351537.4423.1162.camel@hal.voltaire.com>	
	<42D6F3F5.5020806@ichips.intel.com>	
	<1121425731.4423.2145.camel@hal.voltaire.com>	
	<42D7EC83.8030406@ichips.intel.com>	
	<1121695663.4419.6024.camel@hal.voltaire.com>	
	<42DD49A9.9060600@ichips.intel.com>	
	<1121859053.4419.15083.camel@hal.voltaire.com>
	<1121862915.4419.15471.camel@hal.voltaire.com>
Message-ID: <42DE9507.8090507@ichips.intel.com>

Hal Rosenstock wrote:

>>On Tue, 2005-07-19 at 14:42, Arlin Davis wrote:
>>    
>>
>>>Yes, it returns the route data (GID) when the return == 1.  However, I 
>>>still get a callback event with a req_id ==0 which is somewhat 
>>>unexpected. This seems unnecessary given that ib_at_route_by_ip() just 
>>>returned the data. Is this the expected behavior?
>>>
>>>
>>>      
>>>
>>Try this incremental patch:
>>    
>>

v2 patch works great. Thanks!

-arlin

>
>This is v2 of this incremental patch:
>
>
>
>  
>



From limichal at cisco.com  Wed Jul 20 11:17:23 2005
From: limichal at cisco.com (Libor Michalek)
Date: Wed, 20 Jul 2005 11:17:23 -0700
Subject: [openib-general] Re: [PATCHv3] ucm: Only print messages when debug
	level set
In-Reply-To: <1121856451.4419.14829.camel@hal.voltaire.com>;
	from halr@voltaire.com on Wed, Jul 20, 2005 at 06:47:31AM -0400
References: <1121298005.4423.56.camel@hal.voltaire.com>
	<20050719171453.H2231@topspin.com>
	<1121819656.4419.11173.camel@hal.voltaire.com>
	<20050720073507.GA25957@mellanox.co.il>
	<1121856451.4419.14829.camel@hal.voltaire.com>
Message-ID: <20050720111723.A30895@topspin.com>

On Wed, Jul 20, 2005 at 06:47:31AM -0400, Hal Rosenstock wrote:
> On Wed, 2005-07-20 at 03:35, Michael S. Tsirkin wrote:
> > Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > On Tue, 2005-07-19 at 20:14, Libor Michalek wrote:
> > > > On Wed, Jul 13, 2005 at 07:40:07PM -0400, Hal Rosenstock wrote:
> > > > > ucm: Only print messages when debug level set
> > > > 
> > > >   I don't think you can assume that ucm_debug_level will always be 
> > > > initialized correctly, or is module_param_named clearing it? Otherwise
> > > > looks fine...
> > > 
> > > Same is true for IPoIB so it would need changing in both places if so.
> > 
> > I always thought that in C global storage is zero-initialized.
> > What am I missing?
>  
> "A static object not explicitly initialized is initialized as if it (or
> its members) were assigned the constant 0."
> 
> So this patch is OK to commit (and IPoIB is fine in terms of this).

  Got it, applied and checked in.

-Libor


From rolandd at cisco.com  Wed Jul 20 11:17:48 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 20 Jul 2005 11:17:48 -0700
Subject: [openib-general] fmr vs. mr?
In-Reply-To: <42DE8C05.2030908@psc.edu> (Joe Damato's message of "Wed, 20
	Jul 2005 13:38:13 -0400")
References: <42DE8C05.2030908@psc.edu>
Message-ID: <528y01wddf.fsf@topspin.com>

>>>>> "Joe" == Joe Damato <damato at psc.edu> writes:

    Joe> I am attempting to port some code from gen1 to gen2, has fmr
    Joe> been deprecated? What -is- fmr and how does it differ from a
    Joe> mr? If I am correct the type struct ib_mr has become struct
    Joe> ibv_mr...

FMRs are "fast" memory regions, where the memory mapping is written
directly into the HCA's context, rather than through a firmware
command interface.  This means that FMRs are faster but are limited to
a small list of physical buffers of the same side.

Are you porting userspace code or kernel code?  FMRs are only
available to kernel space, since userspace cannot be trusted to write
addresses directly into the HCA's contet.

If you post the section of code you're stuck on, we can probably help
you port it.

 - R.


From ardavis at ichips.intel.com  Wed Jul 20 11:29:46 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Wed, 20 Jul 2005 11:29:46 -0700
Subject: [openib-general] Re: uVerbs and multiple cq_event support?
In-Reply-To: <5264vb7qp0.fsf@topspin.com>
References: <42D81BE3.3020200@ichips.intel.com> <5264vb7qp0.fsf@topspin.com>
Message-ID: <42DE981A.6020908@ichips.intel.com>

Roland Dreier wrote:

>    Arlin> Hi Roland, I have a question about uverbs
>    Arlin> cq_events. Currently, uverbs has one event (fd) handle that
>    Arlin> processes all CQ events. You mentioned earlier that you
>    Arlin> would possibly add support for more CQ events. Are you
>    Arlin> thinking one per CQ, or limiting to a small number like one
>    Arlin> per processor? If the thinking is NOT one per CQ then it
>    Arlin> may be somewhat restricitive to user applications that have
>    Arlin> multple connections with many CQs to process.
>
>My thinking was a small number, to match the 1.2 IBA spec's idea of
>having a fixed number of CQ event handlers available and also to match
>the capabilities of MSI-X interrupts.  However, the idea of being able
>to dynamically create an arbitrary number of CQ event FDs is
>interesting.  I need to think about how feasible this is.
>
>    Arlin> If the thinking is one per CQ then is fd's the way to go
>    Arlin> since we can have 1000's of CQs?
>
>I don't know of a different way to implement this...
>  
>
I guess this is no worse than sockets consuming a FD with every new 
connection.

> - R.
>
>  
>



From mst at mellanox.co.il  Wed Jul 20 11:44:18 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 21:44:18 +0300
Subject: [openib-general] Re: [PATCH] sdp_conn_put/sdp_conn_hold race
In-Reply-To: <20050706175949.GA8783@mellanox.co.il>
References: <20050706175949.GA8783@mellanox.co.il>
Message-ID: <20050720184418.GB28359@mellanox.co.il>

Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> The current sdp_conn_put/sdp_conn_hold implementation
> seems to be subject to the following race condition:

Libor, did you have the time to review this patch?


-- 
MST


From mst at mellanox.co.il  Wed Jul 20 11:45:34 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Wed, 20 Jul 2005 21:45:34 +0300
Subject: [openib-general] [PATCH] sdp trailing whitespace fix
Message-ID: <20050720184534.GC28359@mellanox.co.il>

Fix trailing whitespace in sdp_send.c.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_send.c
===================================================================
--- linux-2.6.12.2.orig/drivers/infiniband/ulp/sdp/sdp_send.c
+++ linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_send.c
@@ -2095,7 +2095,7 @@ skip: /* entry point for IOCB based tran
 		iocb->req  = req;
 		iocb->key  = req->ki_key;
 		iocb->addr = (unsigned long)msg->msg_iov->iov_base - copied;
-      
+
 		req->ki_cancel = sdp_inet_write_cancel;
 
 		result = sdp_iocb_lock(iocb);

-- 
MST


From glebn at voltaire.com  Wed Jul 20 12:05:35 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 20 Jul 2005 22:05:35 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720154509.GG27398@mellanox.co.il>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
Message-ID: <20050720190535.GA21291@minantech.com>

On Wed, Jul 20, 2005 at 06:45:09PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > > > > What about passing RTLD_GLOBAL to dlopen, instead?
> > > >
> > > > The strange thing is it doesn't work, but it should according to man
> > > > page.
> > > 
> > > Better debug it then.
> > > 
> > It seems that RTLD_GLOBAL doesn't work as expected because libibverbs
> > calls ldopen from constructor and it is to early for RTLD_GLOBAL to take
> > effect. If I ldopen mthca.so after dlopen ("libibverbs.so", RTLD_NOW | RTLD_GLOBAL) 
> > returns it actually succeeds. But alas it it to late.
> > 
> 
> I wander if its a better idea to scan the plugin directory on ibv_get_devices.
> Seems to me it might be more debuggable for applications if they get
> error messages when they call ibv_get_devices, and not at startup:
> they can step in debugger to figure out what is printing these things.
> 
I thought the same thing.

--
			Gleb.


From mst at mellanox.co.il  Wed Jul 20 15:18:30 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 01:18:30 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <52d5pdwif3.fsf@topspin.com>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com>
Message-ID: <20050720221830.GA5604@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [PATH] problem dlopen libibverbs.so
> 
>     Michael> I wander if its a better idea to scan the plugin
>     Michael> directory on ibv_get_devices.  Seems to me it might be
>     Michael> more debuggable for applications if they get error
>     Michael> messages when they call ibv_get_devices, and not at
>     Michael> startup: they can step in debugger to figure out what is
>     Michael> printing these things.
> 
>     Michael> Roland, would you take such a patch?
> 
> I was thinking the same thing -- lazy initialization of libibverbs
> when it is first use.  So yes, such a patch would be useful.
> 
>  - R.
> 

Question:

        dlist_for_each_data(driver_list, driver, struct ibv_driver) {
                dev = driver->init_func(verbs_dev);
                if (dev) {
                        dev->dev    = verbs_dev;
                        dev->ibdev  = ib_dev;
                        dev->driver = driver;

                        dlist_push(device_list, dev);

                        return;
                }
        }

Why is this done at library initialization?
Wouldnt device opening be a better place?
Am I missing something?

-- 
MST


From limichal at cisco.com  Wed Jul 20 14:07:50 2005
From: limichal at cisco.com (Libor Michalek)
Date: Wed, 20 Jul 2005 14:07:50 -0700
Subject: [openib-general] Re: [PATCH] sdp_conn_put/sdp_conn_hold race
In-Reply-To: <20050720184418.GB28359@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Jul 20, 2005 at 09:44:18PM +0300
References: <20050706175949.GA8783@mellanox.co.il>
	<20050720184418.GB28359@mellanox.co.il>
Message-ID: <20050720140750.B30895@topspin.com>

On Wed, Jul 20, 2005 at 09:44:18PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> > The current sdp_conn_put/sdp_conn_hold implementation
> > seems to be subject to the following race condition:
> 
> Libor, did you have the time to review this patch?

  Yes, looks good. The only question I had was about making
sdp_conn_put an inline, and calling to destruct only on the
last decrement. Basically split the sdp_conn_put code in
your patch, where sdp_conn_put is everything before the
unlock of sock_lock, and destruct is everything after. Was
there any reason you didn't do that? Is there a reasont to
split it like this...

  Looks like all the right places have _light().

-Libor


From mst at mellanox.co.il  Wed Jul 20 16:40:50 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 02:40:50 +0300
Subject: [openib-general] Re: [PATCH] sdp_conn_put/sdp_conn_hold race
In-Reply-To: <20050720140750.B30895@topspin.com>
References: <20050706175949.GA8783@mellanox.co.il>
	<20050720184418.GB28359@mellanox.co.il>
	<20050720140750.B30895@topspin.com>
Message-ID: <20050720234050.GB5692@mellanox.co.il>

Quoting r. Libor Michalek <limichal at cisco.com>:
> Subject: Re: [PATCH] sdp_conn_put/sdp_conn_hold race
> 
> On Wed, Jul 20, 2005 at 09:44:18PM +0300, Michael S. Tsirkin wrote:
> > Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> > > The current sdp_conn_put/sdp_conn_hold implementation
> > > seems to be subject to the following race condition:
> > 
> > Libor, did you have the time to review this patch?
> 
>   Yes, looks good. The only question I had was about making
> sdp_conn_put an inline, and calling to destruct only on the
> last decrement.

conn_put must first do
spin_lock_irqsave(&dev_root_s.sock_lock, flags);

so that would require making dev_root_s extern instead of static.

> Basically split the sdp_conn_put code in
> your patch, where sdp_conn_put is everything before the
> unlock of sock_lock, and destruct is everything after.

How do you mean? It seems unlock of sock_lock is the last operation
in sdp_conn_put. Must be careful not to reintroduce the race if its
changed.

>  Was
> there any reason you didn't do that? Is there a reasont to
> split it like this...

conn_put is done in places where we expect to have
the last reference. If we know its not, we can do _light.
So making it inline will increase code size
and wont save a function call.

>   Looks like all the right places have _light().
> 
> -Libor
> 

Right. How about we merge it, and then its easier to discuss
patches on top of it? BTW, my limited benchmarking showed
no effect on performance.

-- 
MST


From limichal at cisco.com  Wed Jul 20 15:07:15 2005
From: limichal at cisco.com (Libor Michalek)
Date: Wed, 20 Jul 2005 15:07:15 -0700
Subject: [openib-general] Re: [PATCH] sdp_conn_put/sdp_conn_hold race
In-Reply-To: <20050720234050.GB5692@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Jul 21, 2005 at 02:40:50AM +0300
References: <20050706175949.GA8783@mellanox.co.il>
	<20050720184418.GB28359@mellanox.co.il>
	<20050720140750.B30895@topspin.com>
	<20050720234050.GB5692@mellanox.co.il>
Message-ID: <20050720150715.C30895@topspin.com>

On Thu, Jul 21, 2005 at 02:40:50AM +0300, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <limichal at cisco.com>:
> > Subject: Re: [PATCH] sdp_conn_put/sdp_conn_hold race
> > 
> > On Wed, Jul 20, 2005 at 09:44:18PM +0300, Michael S. Tsirkin wrote:
> > > Quoting r. Michael S. Tsirkin <mst at mellanox.co.il>:
> > > > The current sdp_conn_put/sdp_conn_hold implementation
> > > > seems to be subject to the following race condition:
> > > 
> > > Libor, did you have the time to review this patch?
> > 
> >   Yes, looks good. The only question I had was about making
> > sdp_conn_put an inline, and calling to destruct only on the
> > last decrement.
> 
> conn_put must first do
> spin_lock_irqsave(&dev_root_s.sock_lock, flags);
> 
> so that would require making dev_root_s extern instead of static.

  You're right, I overlooked the spinlock declaration. 

> Right. How about we merge it, and then its easier to discuss
> patches on top of it? BTW, my limited benchmarking showed
> no effect on performance.

  Yes, performance looks the same to me as well. I checked it in
as is. Also, I was going to follow it up with this patch making
sdp_conn_table_remove a void, since all the error conditions
checked should never occur.

-Libor

Index: ulp/sdp/sdp_proto.h
===================================================================
--- ulp/sdp/sdp_proto.h	(revision 2873)
+++ ulp/sdp/sdp_proto.h	(working copy)
@@ -258,8 +258,6 @@
 			 off_t start_index,
 			 long *end_index);
 
-int sdp_conn_table_remove(struct sdp_sock *conn);
-
 struct sdp_sock *sdp_conn_table_lookup(s32 entry);
 
 struct sdp_sock *sdp_conn_alloc(int priority);
Index: ulp/sdp/sdp_conn.c
===================================================================
--- ulp/sdp/sdp_conn.c	(revision 2888)
+++ ulp/sdp/sdp_conn.c	(working copy)
@@ -572,28 +572,20 @@
 /*
  * sdp_conn_table_remove - remove a connection from the connection table
  */
-int sdp_conn_table_remove(struct sdp_sock *conn)
+static void sdp_conn_table_remove(struct sdp_sock *conn)
 {
-	int result = 0;
 	/*
 	 * validate entry
 	 */
-	if (SDP_DEV_SK_INVALID == conn->hashent)
-		goto done;
-
-	if (conn->hashent < 0 || conn != dev_root_s.sk_array[conn->hashent]) {
-		result = -ERANGE;
-		goto done;
-	}
+	BUG_ON(SDP_DEV_SK_INVALID == conn->hashent);
+	BUG_ON(conn->hashent < 0);
+	BUG_ON(conn != dev_root_s.sk_array[conn->hashent]);
 	/*
 	 * drop entry
 	 */
 	dev_root_s.sk_array[conn->hashent] = NULL;
 	dev_root_s.sk_entry--;
 	conn->hashent = SDP_DEV_SK_INVALID;
-
-done:
-	return result;
 }
 
 /*
@@ -691,11 +683,7 @@
 		return;
 	}
 
-	result = sdp_conn_table_remove(conn);
-	if (result < 0)
-		sdp_dbg_warn(conn, "Error <%d> removing connection <%u:%u>",
-			     result, dev_root_s.sk_entry,
-			     dev_root_s.sk_size);
+	sdp_conn_table_remove(conn);
 
 	spin_unlock_irqrestore(&dev_root_s.sock_lock, flags);
 


From libor at topspin.com  Wed Jul 20 16:46:06 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 20 Jul 2005 16:46:06 -0700
Subject: [openib-general] Re: userspace event reporting
In-Reply-To: <ORSMSX401IiGXix4A2D0000003c@orsmsx401.amr.corp.intel.com>;
	from sean.hefty@intel.com on Tue, Jul 12, 2005 at 11:53:06AM -0700
References: <ORSMSX401IiGXix4A2D0000003c@orsmsx401.amr.corp.intel.com>
Message-ID: <20050720164606.L2231@topspin.com>

On Tue, Jul 12, 2005 at 11:53:06AM -0700, Sean Hefty wrote:
> This fell out of the uCM connection ID discussion...
> 
> There's an issue reporting events to userspace clients for an object that
> a user may have destroyed.  The problem exists with user verbs, but is
> much more likely to be seen by a userspace CM client.  To avoid reporting
> events for a destroyed object, I think that something similar to the
> following could be used from userspace:
> 
> destroy() should set a state marking destruction and wait for a reference
> count to go to 0 before transitioning to the kernel.  The kernel code
> should destroy the associated kernel object and then discard any unclaimed
> events.
> 
> get_event() should check the object state and discard the event if the
> object is being destroyed.  It should increment a reference count before
> reporting any events.
> 
> put_event() should decrement the reference count and unblock destroy if
> the reference count goes to 0.

  Just a little background. I had left out the user defined context
variable because of the race condition, and I didn't try to solve
the race condition on purpose. The reason is that I can see a few
ways for the user to solve this themselves, entirely in userspace,
depending on the threading model being used. For example, the above 
can be done entirely in userspace.

  I could be convinced otherwise, but that was the thinking.

  
-Libor


From libor at topspin.com  Wed Jul 20 16:49:47 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 20 Jul 2005 16:49:47 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <ORSMSX401reBBsQwY6Y00000043@orsmsx401.amr.corp.intel.com>;
	from sean.hefty@intel.com on Tue, Jul 12, 2005 at 01:45:43PM -0700
References: <20050712203613.GC12896@mellanox.co.il>
	<ORSMSX401reBBsQwY6Y00000043@orsmsx401.amr.corp.intel.com>
Message-ID: <20050720164947.M2231@topspin.com>

On Tue, Jul 12, 2005 at 01:45:43PM -0700, Sean Hefty wrote:
> >> one way to handle this is to have destruction flush events first, then
> >> perform the actual destroy...
> >
> >Right, what I was trying to say in the other thread was: walk the event
> >list and kill events for the id that is destroyed.
> >
> >Does this make more sense?
> 
> I think this is needed, just not sure if it's sufficient. 

  In fact this is already done, we remove all pending events from the
event queue on destroy. As you explain it is also insufficient since
the user may already have called get_event() from another thread.

-Libor


From libor at topspin.com  Wed Jul 20 17:01:59 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 20 Jul 2005 17:01:59 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <002201c5876a$99b66dc0$6501a8c0@infiniconsys.com>;
	from ftillier@silverstorm.com on Tue, Jul 12, 2005 at 10:20:33PM
	-0700
References: <20050712221449.GA14316@mellanox.co.il>
	<002201c5876a$99b66dc0$6501a8c0@infiniconsys.com>
Message-ID: <20050720170159.N2231@topspin.com>

On Tue, Jul 12, 2005 at 10:20:33PM -0700, Fab Tillier wrote:
> > From: Michael S. Tsirkin [mailto:mst at mellanox.co.il]
> > Quoting r. Sean Hefty <sean.hefty at intel.com>:
> > >
> > > I believe that this is purely a userspace issue.  I can't see why
> > > using a mutex wouldn't work, but I believe that get_event() currently
> > > blocks waiting for an event.
> > >
> > > Note that get_event() may be reporting events associated with an
> > > object other than the one being destroyed.
> > 
> > Maybe, create a special kind of event after the object has been destroyed?
> 
> So I think anyway I look at it, the uCM ends up needing to shadow the
> CM IDs in user-land to maintain references.  If it does that, then Sean's
> earlier idea of performing reference counting and blocking destruction
> should work fine and maintain the synchronous behavior that exists today.

  Correct, this is the way it was done in the gen1 userspace CM. There
is a table, protected by a lock, that keeps a reference counted level
of indirection between the kernel cm_id and the user visible id.

  My question is do we really want this, since an application will likely
have yet another table containing the app specific connection structure.
How that table is locked and managed will differ based on the threading
model. I think it's an incorrect assumption that if we create this level
thread safety in the uCM then the app connection structure can be placed
in a uCM user supplied pointer (e.g. the kCM context variable) and avoid
this type of app level connection table.


-Libor


From libor at topspin.com  Wed Jul 20 17:11:31 2005
From: libor at topspin.com (Libor Michalek)
Date: Wed, 20 Jul 2005 17:11:31 -0700
Subject: [openib-general] Re: uCM init_qp_attr() [Re: uCM create connection
	ID]
In-Reply-To: <42C4A3F2.2020000@ichips.intel.com>;
	from ardavis@ichips.intel.com on Thu, Jun 30, 2005 at 07:01:22PM
	-0700
References: <42C1BA15.7060205@ichips.intel.com>
	<20050629111038.G26240@topspin.com>
	<42C2F991.60005@ichips.intel.com>
	<20050629181054.J26240@topspin.com>
	<42C4A3F2.2020000@ichips.intel.com>
Message-ID: <20050720171131.O2231@topspin.com>

On Thu, Jun 30, 2005 at 07:01:22PM -0700, Arlin Davis wrote:
> 
> The uDAPL code is now connecting properly but I am having difficulty 
> setting the QP states properly without the ib_cm_init_qp_attr() call. 
> Any chance of providing this call in uCM?

  Sean, To do this I'll need to add a kernel CM function to get the 
connections qp attributes from 'struct cm_id_private' inorder to pass
them to userspace. Any preference on what form the function should take?


-Libor


From ftillier at silverstorm.com  Wed Jul 20 17:17:09 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Wed, 20 Jul 2005 17:17:09 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050720170159.N2231@topspin.com>
Message-ID: <000301c58d89$8a630bb0$9c5aa8c0@infiniconsys.com>

> From: Libor Michalek [mailto:libor at topspin.com]
> Sent: Wednesday, July 20, 2005 5:02 PM
> 
>   My question is do we really want this, since an application will likely
> have yet another table containing the app specific connection structure.
> How that table is locked and managed will differ based on the threading
> model. I think it's an incorrect assumption that if we create this level
> thread safety in the uCM then the app connection structure can be placed
> in a uCM user supplied pointer (e.g. the kCM context variable) and avoid
> this type of app level connection table.

Ok, now that I think more about it, I think that having the ability to do an
async, per-CM ID get_event call would help tremendously.  The application knows
what CM IDs it has, and thus can initiate such a get_event request for each.
The app can then maintain its own reference count on its internal structures
(which is really what is needs to do), rather than rely on the CM providing the
reference counting and synchronization.

Is it even possible to make async get_event requests, from a coding perspective?
Would the resulting usage model work for clients?  If using an AIO read request,
could the file offset could be used to convey the CM ID being polled?

- Fab



From ftillier at silverstorm.com  Wed Jul 20 17:36:07 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Wed, 20 Jul 2005 17:36:07 -0700
Subject: [openib-general] Re: uCM init_qp_attr() [Re: uCM create
	connectionID]
In-Reply-To: <20050720171131.O2231@topspin.com>
Message-ID: <000501c58d8c$3050b2a0$9c5aa8c0@infiniconsys.com>

> From: Libor Michalek [mailto:libor at topspin.com]
> Sent: Wednesday, July 20, 2005 5:12 PM
> 
> On Thu, Jun 30, 2005 at 07:01:22PM -0700, Arlin Davis wrote:
> >
> > The uDAPL code is now connecting properly but I am having difficulty
> > setting the QP states properly without the ib_cm_init_qp_attr() call.
> > Any chance of providing this call in uCM?
> 
>   Sean, To do this I'll need to add a kernel CM function to get the
> connections qp attributes from 'struct cm_id_private' inorder to pass
> them to userspace. Any preference on what form the function should take?

Why can't you just expose the exiting kernel ib_cm_init_qp_attr call?  It seems
that call provides the access to cm_id_private.  As long as you buffer the
qp_attr structure properly, what are the issues?

- Fab



From limichal at cisco.com  Wed Jul 20 17:54:43 2005
From: limichal at cisco.com (Libor Michalek)
Date: Wed, 20 Jul 2005 17:54:43 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <000301c58d89$8a630bb0$9c5aa8c0@infiniconsys.com>;
	from ftillier@silverstorm.com on Wed, Jul 20, 2005 at 05:17:09PM
	-0700
References: <20050720170159.N2231@topspin.com>
	<000301c58d89$8a630bb0$9c5aa8c0@infiniconsys.com>
Message-ID: <20050720175443.A31437@topspin.com>

On Wed, Jul 20, 2005 at 05:17:09PM -0700, Fab Tillier wrote:
> > From: Libor Michalek [mailto:libor at topspin.com]
> > Sent: Wednesday, July 20, 2005 5:02 PM
> > 
> >   My question is do we really want this, since an application will likely
> > have yet another table containing the app specific connection structure.
> > How that table is locked and managed will differ based on the threading
> > model. I think it's an incorrect assumption that if we create this level
> > thread safety in the uCM then the app connection structure can be placed
> > in a uCM user supplied pointer (e.g. the kCM context variable) and avoid
> > this type of app level connection table.
> 
> Ok, now that I think more about it, I think that having the ability to do
> an async, per-CM ID get_event call would help tremendously.  The
> application knows what CM IDs it has, and thus can initiate such a
> get_event request for each. The app can then maintain its own reference
> count on its internal structures (which is really what is needs to do),
> rather than rely on the CM providing the reference counting and
> synchronization.

  Well, I think having a per-CM ID get_event is a useful thread based
programming model, but I don't think it's the one and only approach
for all applications. Again, it's a layer that could (should?) be 
implemented entirely in userspace on top of the current interface.

  Also, I agree with your last sentence. I don't think it necessarily
follows or depends on the previous two, it's true for all most all 
serious apps, not just if there is a per-CM ID get_event.

> Is it even possible to make async get_event requests, from a coding 
> perspective? Would the resulting usage model work for clients?  If
> using an AIO read request, could the file offset could be used to
> convey the CM ID being polled?

  Well, you can poll on the file descriptor, which means you can submit
an AIO poll request. You could submit an AIO read, and you would get
back the ABI data in your buffer, which contains the cm_id. However, I'm
not sure what you're getting at.


-Libor


From ftillier at silverstorm.com  Wed Jul 20 18:13:21 2005
From: ftillier at silverstorm.com (Fab Tillier)
Date: Wed, 20 Jul 2005 18:13:21 -0700
Subject: [openib-general] RE: userspace event reporting
In-Reply-To: <20050720175443.A31437@topspin.com>
Message-ID: <000601c58d91$6428b190$9c5aa8c0@infiniconsys.com>

> From: Libor Michalek [mailto:limichal at cisco.com]
> Sent: Wednesday, July 20, 2005 5:55 PM
> 
> On Wed, Jul 20, 2005 at 05:17:09PM -0700, Fab Tillier wrote:
> > > From: Libor Michalek [mailto:libor at topspin.com]
> > > Sent: Wednesday, July 20, 2005 5:02 PM
> > >
> > >   My question is do we really want this, since an application will likely
> > > have yet another table containing the app specific connection structure.
> > > How that table is locked and managed will differ based on the threading
> > > model. I think it's an incorrect assumption that if we create this level
> > > thread safety in the uCM then the app connection structure can be placed
> > > in a uCM user supplied pointer (e.g. the kCM context variable) and avoid
> > > this type of app level connection table.
> >
> > Ok, now that I think more about it, I think that having the ability to do
> > an async, per-CM ID get_event call would help tremendously.  The
> > application knows what CM IDs it has, and thus can initiate such a
> > get_event request for each. The app can then maintain its own reference
> > count on its internal structures (which is really what is needs to do),
> > rather than rely on the CM providing the reference counting and
> > synchronization.
> 
>   Well, I think having a per-CM ID get_event is a useful thread based
> programming model, but I don't think it's the one and only approach
> for all applications. Again, it's a layer that could (should?) be
> implemented entirely in userspace on top of the current interface.

I'm trying to think how you could implement the per-CM ID get event without
introducing an extra thread that would sit there polling for events, and then
perform the multiplexing.  Maybe having extra threads isn't a big deal, but
that's where the async model comes in handy - a single threaded app can get
exactly the behavior it wants using AIO.

>   Also, I agree with your last sentence. I don't think it necessarily
> follows or depends on the previous two, it's true for all most all
> serious apps, not just if there is a per-CM ID get_event.

Right - it's a matter of the CM enabling the app to do what it needs to do.

> > Is it even possible to make async get_event requests, from a coding
> > perspective? Would the resulting usage model work for clients?  If
> > using an AIO read request, could the file offset could be used to
> > convey the CM ID being polled?
> 
>   Well, you can poll on the file descriptor, which means you can submit
> an AIO poll request. You could submit an AIO read, and you would get
> back the ABI data in your buffer, which contains the cm_id. However, I'm
> not sure what you're getting at.

I don't know how AIO works in Linux.  The idea would be to allow a single
threaded client to issue a get_event AIO request for each CM ID that client has,
and then use the same thread to reap the AIO completions in whatever order they
come in.  This model allows the client to easily keep references on its internal
structures associated with each individual CM ID.

Does that make more sense?  Does Linux have a concept similar to the I/O
completion ports in Windows for gathering AIO requests?

- Fab



From glebn at voltaire.com  Wed Jul 20 23:21:39 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 21 Jul 2005 09:21:39 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720221830.GA5604@mellanox.co.il>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com> <20050720221830.GA5604@mellanox.co.il>
Message-ID: <20050721062139.GR28598@minantech.com>

On Thu, Jul 21, 2005 at 01:18:30AM +0300, Michael S. Tsirkin wrote:
> Quoting r. Roland Dreier <rolandd at cisco.com>:
> > Subject: Re: [PATH] problem dlopen libibverbs.so
> > 
> >     Michael> I wander if its a better idea to scan the plugin
> >     Michael> directory on ibv_get_devices.  Seems to me it might be
> >     Michael> more debuggable for applications if they get error
> >     Michael> messages when they call ibv_get_devices, and not at
> >     Michael> startup: they can step in debugger to figure out what is
> >     Michael> printing these things.
> > 
> >     Michael> Roland, would you take such a patch?
> > 
> > I was thinking the same thing -- lazy initialization of libibverbs
> > when it is first use.  So yes, such a patch would be useful.
> > 
> >  - R.
> > 
> 
> Question:
> 
>         dlist_for_each_data(driver_list, driver, struct ibv_driver) {
>                 dev = driver->init_func(verbs_dev);
>                 if (dev) {
>                         dev->dev    = verbs_dev;
>                         dev->ibdev  = ib_dev;
>                         dev->driver = driver;
> 
>                         dlist_push(device_list, dev);
> 
>                         return;
>                 }
>         }
> 
> Why is this done at library initialization?
> Wouldnt device opening be a better place?
> Am I missing something?
> 
If you are going to rewrite library initialisation and device discovery 
perhaps you should write support for hotpluggable devices too? Currently
libibdev scans sysfs only once during init. What if device was plugged
after application was launched?

What do you think?

--
			Gleb.


From mst at mellanox.co.il  Wed Jul 20 23:27:44 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 09:27:44 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050721062139.GR28598@minantech.com>
References: <20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com> <20050720221830.GA5604@mellanox.co.il>
	<20050721062139.GR28598@minantech.com>
Message-ID: <20050721062744.GJ27398@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> Subject: Re: [PATH] problem dlopen libibverbs.so
> 
> On Thu, Jul 21, 2005 at 01:18:30AM +0300, Michael S. Tsirkin wrote:
> > Quoting r. Roland Dreier <rolandd at cisco.com>:
> > > Subject: Re: [PATH] problem dlopen libibverbs.so
> > > 
> > >     Michael> I wander if its a better idea to scan the plugin
> > >     Michael> directory on ibv_get_devices.  Seems to me it might be
> > >     Michael> more debuggable for applications if they get error
> > >     Michael> messages when they call ibv_get_devices, and not at
> > >     Michael> startup: they can step in debugger to figure out what is
> > >     Michael> printing these things.
> > > 
> > >     Michael> Roland, would you take such a patch?
> > > 
> > > I was thinking the same thing -- lazy initialization of libibverbs
> > > when it is first use.  So yes, such a patch would be useful.
> > > 
> > >  - R.
> > > 
> > 
> > Question:
> > 
> >         dlist_for_each_data(driver_list, driver, struct ibv_driver) {
> >                 dev = driver->init_func(verbs_dev);
> >                 if (dev) {
> >                         dev->dev    = verbs_dev;
> >                         dev->ibdev  = ib_dev;
> >                         dev->driver = driver;
> > 
> >                         dlist_push(device_list, dev);
> > 
> >                         return;
> >                 }
> >         }
> > 
> > Why is this done at library initialization?
> > Wouldnt device opening be a better place?
> > Am I missing something?
>
> If you are going to rewrite library initialisation and device discovery 

Probably not yet :).

> perhaps you should write support for hotpluggable devices too? Currently
> libibdev scans sysfs only once during init. What if device was plugged
> after application was launched?
> 
> What do you think?

I have hotplug in the cards, but cant commit to any specific dates.

-- 
MST


From mst at mellanox.co.il  Thu Jul 21 00:51:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 10:51:54 +0300
Subject: [openib-general] [PATCH] sdp: replace mlock with get_user_pages
Message-ID: <20050721075154.GC12698@mellanox.co.il>

Hello, Libor!
Here's a patch updated against the latest bits.
I'd like it to get merged, if possible.

The existing code play with VMA's is the #1 source of sdp problems for me.
The patch below works with no problems for me, so far.

I am aware of the aio/synchronous ordering race we currently have, and I am
testing a patch for it, but please note the patch below doesnt make this problem
any worse, so I consider it an orthogonal issue.

There are also some additional optimisation ideas, like completing send iocbs
in IRQ context - again, what I have here is the simplest version,
and it performs at least as good as the mlock hack.

Please comment.

---

The following patch replaces the mlock hack with call
to get_user_pages. Since the application could have forked
while an iocb is outstanding, when an iocb is done
I do get_user_pages for a second time and copy data if
the physical address has changed.

Thus, changing ulimit is no longer required to get aio
working, processes are also allowed to fork and to call mlock/munlock
on the buffer.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_recv.c
===================================================================
--- linux-2.6.12.2.orig/drivers/infiniband/ulp/sdp/sdp_recv.c
+++ linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_recv.c
@@ -1347,6 +1347,8 @@ int sdp_inet_recv(struct kiocb  *req, st
 			iocb->addr = ((unsigned long)msg->msg_iov->iov_base -
 				      copied);
 
+			iocb->flags |= SDP_IOCB_F_RECV;
+
 			req->ki_cancel = sdp_inet_read_cancel;
 
 			result = sdp_iocb_lock(iocb);
Index: linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_iocb.c
===================================================================
--- linux-2.6.12.2.orig/drivers/infiniband/ulp/sdp/sdp_iocb.c
+++ linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_iocb.c
@@ -33,377 +33,182 @@
  * $Id: sdp_iocb.c 2762 2005-06-30 19:01:38Z libor $
  */
 
+#include <linux/pagemap.h>
 #include "sdp_main.h"
 
 static kmem_cache_t *sdp_iocb_cache = NULL;
 
-/*
- * memory locking functions
- */
-#include <linux/utsname.h>
-
-typedef int (*do_mlock_ptr_t)(unsigned long, size_t, int);
-static do_mlock_ptr_t mlock_ptr = NULL;
-
-/*
- * do_iocb_unlock - unlock the memory for an IOCB
- */
-static void do_iocb_unlock(struct sdpc_iocb *iocb)
-{
-	struct vm_area_struct *vma;
-
-	vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK));
-	if (!vma)
-		sdp_warn("No VMA for IOCB <%lx:%Zu> unlock",
-			 iocb->addr, iocb->size);
-
-	while (vma) {
-		sdp_dbg_data(NULL, 
-			     "unmark <%lx> <%p> <%08lx:%08lx> <%08lx> <%ld>",
-			     iocb->addr, vma, vma->vm_start, vma->vm_end,
-			     vma->vm_flags, (long)vma->vm_private_data);
-		
-		spin_lock(&iocb->mm->page_table_lock);
-		/*
-		 * if there are no more references to the vma
-		 */
-		vma->vm_private_data--;
- 
-		if (!vma->vm_private_data) {
-			/*
-			 * modify VM flags.
-			 */
-			vma->vm_flags &= ~(VM_DONTCOPY|VM_LOCKED);
-			/*
-			 * adjust locked page count
-			 */
-			vma->vm_mm->locked_vm -= ((vma->vm_end -
-						   vma->vm_start) >> 
-						  PAGE_SHIFT);
-		}
-
-		spin_unlock(&iocb->mm->page_table_lock);
-		/*
-		 * continue if the buffer continues onto the next vma
-		 */
-		if ((iocb->addr + iocb->size) > vma->vm_end)
-			vma = vma->vm_next;
-		else
-			vma = NULL;
-	}
+static void sdp_copy_one_page(struct page *from, struct page* to, 
+		      unsigned long iocb_addr, size_t iocb_size,
+		      unsigned long uaddr)
+{
+	size_t size_left = iocb_addr + iocb_size - uaddr;
+	size_t size = min(size_left,PAGE_SIZE);
+	unsigned long offset = uaddr % PAGE_SIZE;
+	unsigned long flags;
+
+	void* fptr;
+	void* tptr;
+
+	local_irq_save(flags);
+	fptr = kmap_atomic(from, KM_IRQ0);
+	tptr = kmap_atomic(to, KM_IRQ1);
+
+	memcpy(tptr + offset, fptr + offset, size);
+
+	kunmap_atomic(tptr, KM_IRQ1);
+	kunmap_atomic(fptr, KM_IRQ0);
+	local_irq_restore(flags);
+	set_page_dirty_lock(to);
 }
 
 /*
  * sdp_iocb_unlock - unlock the memory for an IOCB
+ * Copy if pages moved since.
+ * TODO: is this needed?
  */
 void sdp_iocb_unlock(struct sdpc_iocb *iocb)
 {
-	/*
-	 * check if IOCB is locked.
-	 */
+ 	int result;
+	struct page ** pages = NULL;
+	unsigned long uaddr;
+	int i;
+
 	if (!(iocb->flags & SDP_IOCB_F_LOCKED))
 		return;
-	/*
-	 * spin lock since this could be from interrupt context.
-	 */
-	down_write(&iocb->mm->mmap_sem);
-	
-	do_iocb_unlock(iocb);
-
-	up_write(&iocb->mm->mmap_sem);
 
+	/* For read, unlock and we are done */
+	if (!(iocb->flags & SDP_IOCB_F_RECV)) {
+		for (i = 0;i < iocb->page_count; ++i)
+			put_page(iocb->page_array[i]);
+		goto done;
+	}
+ 
+	/* For write, we must check the virtual pages did not get remapped */
+ 
+	/* As an optimisation (to avoid scanning the vma tree each time),
+	 * try to get all pages in one go. */
+	/* TODO: use cache for allocations? Allocate by chunks? */
+ 
+	pages = kmalloc((sizeof(struct page *) * iocb->page_count), GFP_KERNEL);
+	down_read(&iocb->mm->mmap_sem);
+	if (pages) {
+		result = get_user_pages(iocb->tsk, iocb->mm, iocb->addr,
+					iocb->page_count, 1, 0, pages, NULL);
+		if (result != iocb->page_count) {
+			kfree(pages);
+			pages = NULL;
+		}
+	}
+	for (i = 0, uaddr = iocb->addr; i < iocb->page_count;
+	     ++i, uaddr = (uaddr & PAGE_MASK) + PAGE_SIZE)
+	{
+		struct page* page;
+		set_page_dirty_lock(iocb->page_array[i]);
+
+		if (pages)
+			page = pages[i];
+		else {
+			result = get_user_pages(iocb->tsk, iocb->mm,
+						uaddr & PAGE_MASK,
+						1 , 1, 0, &page, NULL);
+			if (result != 1) {
+				page = NULL;
+			}
+		}
+		if (page && iocb->page_array[i] != page)
+			sdp_copy_one_page(iocb->page_array[i], page,
+					  iocb->addr, iocb->size, uaddr);
+		if (page)
+			put_page(page);
+		put_page(iocb->page_array[i]);
+ 	}
+	up_read(&iocb->mm->mmap_sem);
+	if (pages)
+		kfree(pages);
+ 
+done:
 	kfree(iocb->page_array);
-	kfree(iocb->addr_array);
+ 	kfree(iocb->addr_array);
 
 	iocb->page_array = NULL;
-	iocb->addr_array = NULL;
+ 	iocb->addr_array = NULL;
 	iocb->mm = NULL;
-	/*
-	 * mark IOCB unlocked.
-	 */
+	iocb->tsk = NULL;
 	iocb->flags &= ~SDP_IOCB_F_LOCKED;
 }
 
 /*
- * sdp_iocb_page_save - save page information for an IOCB
+ * sdp_iocb_lock - lock the memory for an IOCB
+ * We do not take a reference on the mm, AIO handles this for us.
  */
-static int sdp_iocb_page_save(struct sdpc_iocb *iocb)
+int sdp_iocb_lock(struct sdpc_iocb *iocb)
 {
-	unsigned int counter;
+	int result = -ENOMEM;
 	unsigned long addr;
 	size_t size;
-	int result = -ENOMEM;
-	struct page *page;
-	unsigned long pfn;
-	pgd_t *pgd;
-	pud_t *pud;
-	pmd_t *pmd;
-	pte_t *ptep;
-	pte_t  pte;
+	int i;
 
-	if (iocb->page_count <= 0 || iocb->size <= 0 || !iocb->addr)
-		return -EINVAL;
-	/*
+ 	/*
+	 * iocb->addr - buffer start address
+	 * iocb->size - buffer length
+	 * addr       - page aligned
+	 * size       - page multiple
+ 	 */
+ 	addr = iocb->addr & PAGE_MASK;
+	size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK));
+ 
+	iocb->page_offset = iocb->addr - addr;
+	
+ 	iocb->page_count = size >> PAGE_SHIFT;
+ 	/*
 	 * create array to hold page value which are later needed to register
 	 * the buffer with the HCA
-	 */
+ 	 */
+ 
+	/* TODO: use cache for allocations? Allocate by chunks? */
 	iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count),
 				   GFP_KERNEL);
 	if (!iocb->addr_array)
 		goto err_addr;
-
+ 
 	iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), 
 				   GFP_KERNEL);
 	if (!iocb->page_array)
 		goto err_page;
-	/*
-	 * iocb->addr - buffer start address
-	 * iocb->size - buffer length
-	 * addr       - page aligned
-	 * size       - page multiple
-	 */
-	addr = iocb->addr & PAGE_MASK;
-	size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK));
-
-	iocb->page_offset = iocb->addr - addr;
-	/*
-	 * Find pages used within the buffer which will then be registered
-	 * for RDMA
-	 */
-	spin_lock(&iocb->mm->page_table_lock);
-
-	for (counter = 0;
-	     size > 0;
-	     counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) {
-		pgd = pgd_offset_gate(iocb->mm, addr);
-		if (!pgd || pgd_none(*pgd))
-			break;
-
-		pud = pud_offset(pgd, addr);
-		if (!pud || pud_none(*pud))
-			break;
-		
-		pmd = pmd_offset(pud, addr);
-		if (!pmd || pmd_none(*pmd))
-			break;
-
-		ptep = pte_offset_map(pmd, addr);
-		if (!ptep) 
-			break;
-
-		pte = *ptep;
-		pte_unmap(ptep);
-
-		if (!pte_present(pte))
-			break;
-
-		pfn = pte_pfn(pte);
-		if (!pfn_valid(pfn))
-			break;
-        
-		page = pfn_to_page(pfn);
-
-		iocb->page_array[counter] = page;
-		iocb->addr_array[counter] = page_to_phys(page);
-	}
-
-	spin_unlock(&iocb->mm->page_table_lock);
-	
-	if (size > 0) {
-		result = -EFAULT;
-		goto err_find;
+ 
+	down_read(&current->mm->mmap_sem);
+ 
+        result = get_user_pages(current, current->mm,
+				iocb->addr, iocb->page_count,
+			      !!(iocb->flags & SDP_IOCB_F_RECV), 0,
+			      iocb->page_array, NULL);
+
+	up_read(&current->mm->mmap_sem);
+
+	if (result != iocb->page_count) {
+		sdp_dbg_err("unable to lock <%lx:%Zu> error <%d> <%d>",
+			    iocb->addr, iocb->size, result, iocb->page_count);
+		goto err_get;
 	}
-
-	return 0;
-err_find:
-	
+ 
+	iocb->flags |= SDP_IOCB_F_LOCKED;
+	iocb->mm     = current->mm;
+	iocb->tsk    = current;
+ 
+ 
+	for (i = 0; i< iocb->page_count; ++i) {
+		iocb->addr_array[i] = page_to_phys(iocb->page_array[i]);
+ 	}
+ 
+ 	return 0;
+ 
+err_get:
 	kfree(iocb->page_array);
-	iocb->page_array = NULL;
 err_page:
-
 	kfree(iocb->addr_array);
-	iocb->addr_array = NULL;
 err_addr:
-
-	return result;
-}
-
-/*
- * sdp_iocb_lock - lock the memory for an IOCB
- */
-int sdp_iocb_lock(struct sdpc_iocb *iocb)
-{
-	struct vm_area_struct *vma;
-	kernel_cap_t real_cap;
-	unsigned long limit;
-	int result = -ENOMEM;
-	unsigned long addr;
-	size_t        size;
-	
-	/*
-	 * mark IOCB as locked. We do not take a reference on the mm, AIO
-	 * handles this for us.
-	 */
-	iocb->flags |= SDP_IOCB_F_LOCKED;
-	iocb->mm     = current->mm;
-	/*
-	 * save and raise capabilities
-	 */
-	real_cap = cap_t(current->cap_effective);
-	cap_raise(current->cap_effective, CAP_IPC_LOCK);
-  
-	size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK));
-	addr = iocb->addr & PAGE_MASK;
-
-	iocb->page_count = size >> PAGE_SHIFT;
-
-	limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
-	limit >>= PAGE_SHIFT;
-	/*
-	 * lock the mm, if within the limit lock the address range.
-	 */
-	down_write(&iocb->mm->mmap_sem);
-
-	if (!((iocb->page_count + current->mm->locked_vm) > limit))
-		result = (*mlock_ptr)(addr, size, 1);
-	/*
-	 * process result
-	 */
-	if (result) {
-		sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>",
-			    iocb->addr, iocb->size, result, 
-			    iocb->page_count, iocb->mm->locked_vm, limit);
-		goto err_lock;
-	}
-	/*
-	 * look up the head of the vma queue, loop through the vmas, marking
-	 * them do not copy, reference counting, and saving them. 
-	 */
-	vma = find_vma(iocb->mm, addr);
-	if (!vma)
-		/*
-		 * sanity check.
-		 */
-		sdp_warn("No VMA for IOCB! <%lx:%Zu> lock",
-			 iocb->addr, iocb->size);
-
-	while (vma) {
-		spin_lock(&iocb->mm->page_table_lock);
-
-		if (!(VM_LOCKED & vma->vm_flags))
-			sdp_warn("Unlocked vma! <%08lx>", vma->vm_flags);
-
-		if (PAGE_SIZE < (unsigned long)vma->vm_private_data)
-			sdp_dbg_err("VMA: private daya in use! <%08lx>",
-				    (unsigned long)vma->vm_private_data);
-		
-		vma->vm_flags |= VM_DONTCOPY;
-		vma->vm_private_data++;
-
-		spin_unlock(&iocb->mm->page_table_lock);
-
-		sdp_dbg_data(NULL, 
-			     "mark <%lx> <0x%p> <%08lx:%08lx> <%08lx> <%ld>",
-			     iocb->addr, vma, vma->vm_start, vma->vm_end,
-			     vma->vm_flags, (long)vma->vm_private_data);
-
-		if ((addr + size) > vma->vm_end)
-			vma = vma->vm_next;
-		else
-			vma = NULL;
-	}
-
-	result = sdp_iocb_page_save(iocb);
-	if (result) {
-		sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", 
-			    result, iocb->addr, iocb->size);
-		goto err_save;
-	}
-
-	up_write(&iocb->mm->mmap_sem);
-	cap_t(current->cap_effective) = real_cap;
-
-	return 0;
-err_save:
-
-	do_iocb_unlock(iocb);
-err_lock:
-	/*
-	 * unlock the mm and restore capabilities.
-	 */
-	up_write(&iocb->mm->mmap_sem);
-	cap_t(current->cap_effective) = real_cap;
-
-	iocb->flags &= ~SDP_IOCB_F_LOCKED;
-	iocb->mm = NULL;
-
-	return result;
-}
-
-/*
- * IOCB memory locking init functions
- */
-struct kallsym_iter {
-	loff_t pos;
-	struct module *owner;
-	unsigned long value;
-	unsigned int nameoff; /* If iterating in core kernel symbols */
-	char type;
-	char name[128];
-};
-
-/*
- * sdp_mem_lock_init - initialize the userspace memory locking
- */
-static int sdp_mem_lock_init(void)
-{
-	struct file *kallsyms;
-	struct seq_file *seq;
-	struct kallsym_iter *iter;
-	loff_t pos = 0;
-	int ret = -EINVAL;
-	
-	sdp_dbg_init("Memory Locking initialization.");
-	
-	kallsyms = filp_open("/proc/kallsyms", O_RDONLY, 0);
-	if (!kallsyms) {
-		sdp_warn("Failed to open /proc/kallsyms");
-		goto done;
-	}
-
-	seq = (struct seq_file *)kallsyms->private_data;
-	if (!seq) {
-		sdp_warn("Failed to fetch sequential file.");
-		goto err_close;
-	}
-
-	for (iter = seq->op->start(seq, &pos);
-	     iter != NULL;
-	     iter = seq->op->next(seq, iter, &pos))
-		if (!strcmp(iter->name, "do_mlock"))
-			mlock_ptr = (do_mlock_ptr_t)iter->value;
-
-	if (!mlock_ptr)
-		sdp_warn("Failed to find lock pointer.");
-	else
-		ret = 0;
-
-err_close:
-	filp_close(kallsyms, NULL);
-done:
-	return ret;
-}
-
-/*
- * sdp_mem_lock_cleanup - cleanup the memory locking tables
- */
-static void sdp_mem_lock_cleanup(void)
-{
-	sdp_dbg_init("Memory Locking cleanup.");
-	/*
-	 * null out entries.
-	 */
-	mlock_ptr = NULL;
+ 	return result;
 }
 
 /*
@@ -802,28 +607,12 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q
 }
 
 /*
- * primary initialization/cleanup functions
- */
-
-/*
  * sdp_main_iocb_init - initialize the advertisment caches
  */
 int sdp_main_iocb_init(void)
 {
-	int result;
-
 	sdp_dbg_init("IOCB cache initialization.");
-	/*
-	 * initialize locking code.
-	 */
-	result =  sdp_mem_lock_init();
-	if (result < 0) {
-		sdp_warn("Error <%d> initializing memory locking.", result);
-		return result;
-	}
-	/*
-	 * initialize the caches only once.
-	 */
+
 	if (sdp_iocb_cache) {
 		sdp_warn("IOCB caches already initialized.");
 		return -EINVAL;
@@ -833,15 +622,10 @@ int sdp_main_iocb_init(void)
 					     sizeof(struct sdpc_iocb),
 					     0, SLAB_HWCACHE_ALIGN, NULL,
 					     NULL);
-	if (!sdp_iocb_cache) {
-		result = -ENOMEM;
-		goto error_iocb_c;
-	}
+	if (!sdp_iocb_cache)
+		return -ENOMEM;
 
 	return 0;
-error_iocb_c:
-	sdp_mem_lock_cleanup();
-	return result;
 }
 
 /*
@@ -858,8 +642,4 @@ void sdp_main_iocb_cleanup(void)
 	 * null out entries.
 	 */
 	sdp_iocb_cache = NULL;
-	/*
-	 * cleanup memory locking
-	 */
-	sdp_mem_lock_cleanup();
 }
Index: linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_iocb.h
===================================================================
--- linux-2.6.12.2.orig/drivers/infiniband/ulp/sdp/sdp_iocb.h
+++ linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_iocb.h
@@ -53,7 +53,8 @@
 #define SDP_IOCB_F_RDMA_R 0x00000010 /* IOCB is in RDMA read processing */
 #define SDP_IOCB_F_RDMA_W 0x00000020 /* IOCB is in RDMA write processing */
 #define SDP_IOCB_F_LOCKED 0x00000040 /* IOCB is locked in memory */
-#define SDP_IOCB_F_REG    0x00000080 /* IOCB is memory is registered */
+#define SDP_IOCB_F_REG    0x00000080 /* IOCB memory is registered */
+#define SDP_IOCB_F_RECV   0x00000100 /* IOCB is for a receive request */
 #define SDP_IOCB_F_ALL    0xFFFFFFFF /* IOCB all mask */
 /*
  * zcopy constants.
@@ -100,9 +101,10 @@ struct sdpc_iocb {
 	/*
 	 * page list. data for locking/registering userspace
 	 */
-	struct mm_struct *mm;      /* user mm struct */
-	unsigned long     addr;    /* user space address */
-	size_t            size;    /* total size of the user buffer */
+	struct mm_struct   *mm;      /* user mm struct */
+	struct task_struct *tsk;
+	unsigned long       addr;    /* user space address */
+	size_t              size;    /* total size of the user buffer */
 
 	struct page **page_array;  /* list of page structure pointers. */
 	u64          *addr_array;  /* list of physical page addresses. */

-- 
MST


From dotanb at mellanox.co.il  Thu Jul 21 01:58:29 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Thu, 21 Jul 2005 11:58:29 +0300
Subject: [openib-general] RE: why does the value of the node_guid don't have
	the machine en dian ess?
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A25B4@mtlex01.yok.mtl.com>

> -----Original Message-----
> From: Roland Dreier [mailto:rolandd at cisco.com]
> Subject: Re: why does the value of the node_guid don't have the machine
endianess? 
> 
>     Dotan> why do i need to change the endianess of node_guid when i
>     Dotan> use it from query_hca?
> 
> Which endianess do you want to get the node_guid in?

I expect the value to have the same endian-ness when printed with %llx and 
when read from sysfs.

swlab140:/home/dotanb/infiniband/newvstat/child>grep XXX vstat.c
printf("XXXXXXX %llx\n", device_attr.node_guid);

swlab140:/home/dotanb/infiniband/newvstat/child>./vstat | grep XXX
XXXXXXX 4018000002c90200

swlab140:/home/dotanb/infiniband/newvstat/child>cat
/sys/class/infiniband/mthca0/node_guid
0002:c902:0000:1840 

Why is that?

Dotan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050721/f4a04d99/attachment.html>

From halr at voltaire.com  Thu Jul 21 04:23:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 21 Jul 2005 07:23:10 -0400
Subject: [openib-general] [PATCH] [TRIVIAL] Fix some lidsdp typo
Message-ID: <1121944989.4403.38.camel@hal.voltaire.com>

[TRIVIAL] Fix some lidsdp typo

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Also, should config.h.in be removed ?

Index: src/port.c
===================================================================
--- src/port.c	(revision 2888)
+++ src/port.c	(working copy)
@@ -293,7 +293,7 @@ static int decide_family(
 /* --------------------------------------------------------------------- */
 
 /* ========================================================================= */
-/*..ioctl -- replacment ioctl call. */
+/*..ioctl -- replacement ioctl call. */
 int ioctl(
           int fd,
           int request,
@@ -342,7 +342,7 @@ int ioctl(
 } /* ioctl */
 
 /* ========================================================================= */
-/*..fcntl -- replacment fcntl call. */
+/*..fcntl -- replacement fcntl call. */
 int fcntl(
           int fd,
           int cmd,
@@ -391,7 +391,7 @@ int fcntl(
 
 
 /* ========================================================================= */
-/*..socket -- replacment socket call. */
+/*..socket -- replacement socket call. */
 int socket(
            int domain,
            int type,
@@ -456,7 +456,7 @@ int socket(
 } /* socket */
 
 /* ========================================================================= */
-/*..setsockopt -- replacment setsockopt call. */
+/*..setsockopt -- replacement setsockopt call. */
 int setsockopt(
                int fd,
                int level,
@@ -501,7 +501,7 @@ int setsockopt(
 } /* setsockopt */
 
 /* ========================================================================= */
-/*..connect -- replacment connect call. */
+/*..connect -- replacement connect call. */
 int connect(
             int fd,
             const struct sockaddr *serv_addr,
@@ -572,7 +572,7 @@ int connect(
 } /* connect */
 
 /* ========================================================================= */
-/*..listen -- replacment listen call. */
+/*..listen -- replacement listen call. */
 int listen(
            int fd,
            int backlog
@@ -629,7 +629,7 @@ int listen(
 #define SDP_UNBIND	259	/* Unbind socket */
 
 /* ========================================================================= */
-/*..bind -- replacment bind call. */
+/*..bind -- replacement bind call. */
 int bind(
          int fd,
          const struct sockaddr *my_addr,
@@ -765,7 +765,7 @@ done:
 } /* bind */
 
 /* ========================================================================= */
-/*..close -- replacment close call. */
+/*..close -- replacement close call. */
 int close(
           int fd
           )
@@ -805,7 +805,7 @@ int close(
 } /* close */
 
 /* ========================================================================= */
-/*..dup -- replacment socket call. */
+/*..dup -- replacement socket call. */
 int dup(
 	int fd
 	)
@@ -846,7 +846,7 @@ int dup(
 } /* dup */
 
 /* ========================================================================= */
-/*..getsockname -- replacment getsocknanme call. */
+/*..getsockname -- replacement getsocknanme call. */
 int getsockname(int fd,
 		struct sockaddr *name,
 		socklen_t *namelen)
@@ -884,7 +884,7 @@ int getsockname(int fd,
 } /* getsockname */
 
 /* ========================================================================= */
-/*..getpeername -- replacment getpeername call. */
+/*..getpeername -- replacement getpeername call. */
 int getpeername(int fd,
 		struct sockaddr *name,
 		socklen_t *namelen)
@@ -928,7 +928,7 @@ int getpeername(int fd,
 
 
 /* ========================================================================= */
-/*..accept -- replacment accept call. */
+/*..accept -- replacement accept call. */
 int accept(int fd,
 		struct sockaddr *addr,
 		socklen_t *addrlen)
@@ -1001,7 +1001,7 @@ int accept(int fd,
 
 
 /* ========================================================================= */
-/*..select -- replacment socket call. */
+/*..select -- replacement socket call. */
 int select(
        int n,  fd_set  *readfds,  fd_set  *writefds,
        fd_set *exceptfds, struct timeval *timeout






From mst at mellanox.co.il  Thu Jul 21 04:49:53 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 14:49:53 +0300
Subject: [openib-general] Re: [PATCH] [TRIVIAL] Fix some lidsdp typo
In-Reply-To: <1121944989.4403.38.camel@hal.voltaire.com>
References: <1121944989.4403.38.camel@hal.voltaire.com>
Message-ID: <20050721114953.GI12698@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> Subject: [PATCH] [TRIVIAL] Fix some lidsdp typo
> 
> [TRIVIAL] Fix some lidsdp typo
> 
> Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Thanks, applied.

> Also, should config.h.in be removed ?

Why should it?

-- 
MST


From halr at voltaire.com  Thu Jul 21 04:48:45 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 21 Jul 2005 07:48:45 -0400
Subject: [openib-general] Re: [PATCH] [TRIVIAL] Fix some lidsdp typo
In-Reply-To: <20050721114953.GI12698@mellanox.co.il>
References: <1121944989.4403.38.camel@hal.voltaire.com>
	<20050721114953.GI12698@mellanox.co.il>
Message-ID: <1121946525.4403.46.camel@hal.voltaire.com>

On Thu, 2005-07-21 at 07:49, Michael S. Tsirkin wrote:
> > Also, should config.h.in be removed ?
> 
> Why should it?

Does it get generated by the build process and is different for each
machine ?

-- Hal



From mst at mellanox.co.il  Thu Jul 21 05:00:38 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 15:00:38 +0300
Subject: [openib-general] Re: [PATCH] [TRIVIAL] Fix some lidsdp typo
In-Reply-To: <1121946525.4403.46.camel@hal.voltaire.com>
References: <1121944989.4403.38.camel@hal.voltaire.com>
	<20050721114953.GI12698@mellanox.co.il>
	<1121946525.4403.46.camel@hal.voltaire.com>
Message-ID: <20050721120038.GJ12698@mellanox.co.il>

Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > Also, should config.h.in be removed ?

I see what you mean. Removed.

-- 
MST


From mst at mellanox.co.il  Thu Jul 21 07:16:47 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 17:16:47 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <52d5pdwif3.fsf@topspin.com>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com>
Message-ID: <20050721141647.GN12698@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> I was thinking the same thing -- lazy initialization of libibverbs
> when it is first use.  So yes, such a patch would be useful.

OK. Here goes. Works fine for me. Gleb, can you please verify that it
helps in your setup?

---

Lazy initialization of libibverbs on ibv_get_devices.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: libibverbs/src/device.c
===================================================================
--- libibverbs/src/device.c	(revision 2894)
+++ libibverbs/src/device.c	(working copy)
@@ -46,8 +46,12 @@
 
 #include "ibverbs.h"
 
+static struct dlist *device_list;
+
 struct dlist *ibv_get_devices(void)
 {
+	if (!device_list)
+		device_list = ibverbs_init();
 	return device_list;
 }
 
Index: libibverbs/src/ibverbs.h
===================================================================
--- libibverbs/src/ibverbs.h	(revision 2894)
+++ libibverbs/src/ibverbs.h	(working copy)
@@ -50,11 +50,10 @@ struct ibv_driver {
 	ibv_driver_init_func init_func;
 };
 
-extern Dlist *device_list;
-
 extern int ibv_init_mem_map(void);
 extern int ibv_lock_range(void *base, size_t size);
 extern int ibv_unlock_range(void *base, size_t size);
+extern struct dlist * ibverbs_init(void);
 
 #define IBV_INIT_CMD(cmd, size, opcode)				\
 	do {							\
Index: libibverbs/src/init.c
===================================================================
--- libibverbs/src/init.c	(revision 2894)
+++ libibverbs/src/init.c	(working copy)
@@ -50,12 +50,10 @@
 #  define OPENIB_DRIVER_PATH_ENV "OPENIB_DRIVER_PATH"
 #endif
 
-Dlist *device_list;
-
 static char default_path[] = DRIVER_PATH;
 static const char *user_path;
 
-static Dlist *driver_list;
+static struct dlist *driver_list;
 
 static void load_driver(char *so_path)
 {
@@ -112,7 +110,7 @@ static void find_drivers(char *dir)
 		load_driver(so_glob.gl_pathv[i]);
 }
 
-static void init_drivers(struct sysfs_class_device *verbs_dev)
+static void init_drivers(struct sysfs_class_device *verbs_dev, struct dlist *device_list)
 {
 	struct sysfs_class_device *ib_dev; 
 	struct sysfs_attribute *attr;
@@ -187,13 +185,15 @@ static int check_abi_version(void)
 }
 
 
-static void INIT ibverbs_init(void)
+struct dlist *ibverbs_init(void)
 {
 	char *wr_path, *dir;
 	struct sysfs_class *cls;
-	Dlist *verbs_dev_list;
+	struct dlist *verbs_dev_list;
+	struct dlist *device_list;
 	struct sysfs_class_device *verbs_dev;
 
+
 	driver_list = dlist_new(sizeof (struct ibv_driver));
 	device_list = dlist_new(sizeof (struct ibv_device));
 	if (!driver_list || !device_list) {
@@ -202,7 +202,7 @@ static void INIT ibverbs_init(void)
 	}
 
 	if (ibv_init_mem_map())
-		return;
+		return NULL;
 
 	/*
 	 * Check if a driver is statically linked, and if so load it first.
@@ -227,18 +227,20 @@ static void INIT ibverbs_init(void)
 	cls = sysfs_open_class("infiniband_verbs");
 	if (!cls) {
 		fprintf(stderr, PFX "Fatal: couldn't open sysfs class 'infiniband_verbs'.\n");
-		return;
+		return NULL;
 	}
 
 	if (check_abi_version())
-		return;
+		return NULL;
 
 	verbs_dev_list = sysfs_get_class_devices(cls);
 	if (!verbs_dev_list) {
 		fprintf(stderr, PFX "Fatal: no infiniband class devices found.\n");
-		return;
+		return NULL;
 	}
 
 	dlist_for_each_data(verbs_dev_list, verbs_dev, struct sysfs_class_device)
-		init_drivers(verbs_dev);
+		init_drivers(verbs_dev, device_list);
+
+	return device_list;
 }

-- 
MST


From rolandd at cisco.com  Thu Jul 21 07:25:01 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 21 Jul 2005 07:25:01 -0700
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050720221830.GA5604@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 21 Jul 2005 01:18:30 +0300")
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il> <52d5pdwif3.fsf@topspin.com>
	<20050720221830.GA5604@mellanox.co.il>
Message-ID: <52d5pcuthe.fsf@topspin.com>

    Michael> Why is this done at library initialization?  Wouldnt
    Michael> device opening be a better place?  Am I missing
    Michael> something?

There's no good reason -- I just happened to put device discovery very
early.  It's probably a good idea to put off the scanning for drivers
and devices as late as possible.  It would also be nice to get rid of
the crappy dlist API that I lazily stole from libsysfs.

 - R.


From glebn at voltaire.com  Thu Jul 21 07:25:13 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 21 Jul 2005 17:25:13 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050721141647.GN12698@mellanox.co.il>
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com>
	<20050721141647.GN12698@mellanox.co.il>
Message-ID: <20050721142513.GT28598@minantech.com>

On Thu, Jul 21, 2005 at 05:16:47PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Roland Dreier <rolandd at cisco.com>:
> > I was thinking the same thing -- lazy initialization of libibverbs
> > when it is first use.  So yes, such a patch would be useful.
> 
> OK. Here goes. Works fine for me. Gleb, can you please verify that it
> helps in your setup?
Works for me too.

--
			Gleb.


From mst at mellanox.co.il  Thu Jul 21 07:42:52 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 21 Jul 2005 17:42:52 +0300
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <52d5pcuthe.fsf@topspin.com>
References: <20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il>
	<52d5pdwif3.fsf@topspin.com> <20050720221830.GA5604@mellanox.co.il>
	<52d5pcuthe.fsf@topspin.com>
Message-ID: <20050721144252.GR12698@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: [PATH] problem dlopen libibverbs.so
> 
>     Michael> Why is this done at library initialization?  Wouldnt
>     Michael> device opening be a better place?  Am I missing
>     Michael> something?
> 
> There's no good reason -- I just happened to put device discovery very
> early.  It's probably a good idea to put off the scanning for drivers
> and devices as late as possible.  It would also be nice to get rid of
> the crappy dlist API that I lazily stole from libsysfs.

Yes. For hotplug we will clearly need a way for the user to tell us
he isnt looking at the device list anymore.
Something like ibv_put_devices(dlist)?

So I expect we'll to do something like this in the framework of hotplug support,
but there's the kernel side of things to fix before this pure user-level stuff.

-- 
MST


From dotanb at mellanox.co.il  Thu Jul 21 08:22:40 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Thu, 21 Jul 2005 18:22:40 +0300
Subject: [openib-general] destroying an AH from user space causes
	segmentation fault
Message-ID: <506C3D7B14CDD411A52C00025558DED6084A266F@mtlex01.yok.mtl.com>

I'm using driver revision 2851 with HCA 23108.

Here is a back trace from a core dump of this seg fault:


Loaded symbols for /usr/local/lib/infiniband/mthca.so
#0  0x4001ef8b in mthca_free_av (ah=0x804db70) at ah.c:171

warning: Source file is more recent than executable.

171                     i = ((void *) ah->av - page->buf) / sizeof *ah->av;
(gdb) bt
#0  0x4001ef8b in mthca_free_av (ah=0x804db70) at ah.c:171
#1  0x40021d1d in mthca_destroy_ah (ah=0x804db70) at mthca.h:242
#2  0x4001bfdf in ibv_destroy_ah (ah=0x0) at verbs.c:194
#3  0x080491b7 in main (argc=1, argv=0xbf97a4b4) at sa_ping.c:361
(gdb) p ah
$1 = (struct mthca_ah *) 0x804db70
(gdb) p page
$2 = (struct mthca_ah_page *) 0x0



i attached a small test that reproduces this failure.

 <<destroy_av.c>> 
Dotan Barak
Software Verification Engineer
Mellanox Technologies LTD
mailto:dotanb at mellanox.co.il
Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
P.O. Box 86 Yokneam 20692 ISRAEL.
Home: +972-4-8289408  Cell: 052-4222383

[ May the fork be with you ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050721/9a539d48/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: destroy_av.c
Type: application/octet-stream
Size: 3119 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050721/9a539d48/attachment.obj>

From glebn at voltaire.com  Thu Jul 21 08:25:57 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Thu, 21 Jul 2005 18:25:57 +0300
Subject: [openib-general] destroying an AH from user space causes
	segmentation fault
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A266F@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6084A266F@mtlex01.yok.mtl.com>
Message-ID: <20050721152557.GA29109@minantech.com>

Should be fixed in the latest revision.

On Thu, Jul 21, 2005 at 06:22:40PM +0300, Dotan Barak wrote:
> I'm using driver revision 2851 with HCA 23108.
> 
> Here is a back trace from a core dump of this seg fault:
> 
> 
> Loaded symbols for /usr/local/lib/infiniband/mthca.so
> #0  0x4001ef8b in mthca_free_av (ah=0x804db70) at ah.c:171
> 
> warning: Source file is more recent than executable.
> 
> 171                     i = ((void *) ah->av - page->buf) / sizeof *ah->av;
> (gdb) bt
> #0  0x4001ef8b in mthca_free_av (ah=0x804db70) at ah.c:171
> #1  0x40021d1d in mthca_destroy_ah (ah=0x804db70) at mthca.h:242
> #2  0x4001bfdf in ibv_destroy_ah (ah=0x0) at verbs.c:194
> #3  0x080491b7 in main (argc=1, argv=0xbf97a4b4) at sa_ping.c:361
> (gdb) p ah
> $1 = (struct mthca_ah *) 0x804db70
> (gdb) p page
> $2 = (struct mthca_ah_page *) 0x0
> 
> 
> 
> i attached a small test that reproduces this failure.
> 
>  <<destroy_av.c>> 
> Dotan Barak
> Software Verification Engineer
> Mellanox Technologies LTD
> mailto:dotanb at mellanox.co.il
> Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
> P.O. Box 86 Yokneam 20692 ISRAEL.
> Home: +972-4-8289408  Cell: 052-4222383
> 
> [ May the fork be with you ]
> 


> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

--
			Gleb.


From limichal at cisco.com  Thu Jul 21 12:05:50 2005
From: limichal at cisco.com (Libor Michalek)
Date: Thu, 21 Jul 2005 12:05:50 -0700
Subject: [openib-general] Re: uCM kernel oops
In-Reply-To: <42DD4230.9060702@ichips.intel.com>;
	from ardavis@ichips.intel.com on Tue, Jul 19, 2005 at 11:10:56AM
	-0700
References: <42DD4230.9060702@ichips.intel.com>
Message-ID: <20050721120550.A31860@topspin.com>

On Tue, Jul 19, 2005 at 11:10:56AM -0700, Arlin Davis wrote:
> Hi Libor,
> 
> I am running uCM and uAT with uDAPL and occasionally hit the following. 
> Can you take a look?
>
> Jul 19 11:10:18 iclust-19 kernel: UCM: Write. cmd <1> in <4> out <0> len 
> <12>
> Jul 19 11:10:18 iclust-19 kernel: UCM: Event. CM ID <2> event <7>
> Jul 19 11:10:18 iclust-19 kernel: UCM: Destroyed CM ID <2>
> Jul 19 11:10:18 iclust-19 kernel: Unable to handle kernel paging request 
> <ffffffff880a10c8>{:ib_ucm:ib_ucm_ctx_put+120}
> Trace:<ffffffff880a183f>{:ib_ucm:ib_ucm_event_handler+1199}

  Looks like a race between the destroy command and a DREQ received
event. From looking at the code, it looks like it's possible for
two threads (cm event and userspace) to call ctx_put at the same
time and both try to perform the final zero referece object destroy.
Is this easily reproducible? Can you try the following patch?

-Libor

Index: infiniband/core/ucm.c
===================================================================
--- infiniband/core/ucm.c	(revision 2886)
+++ infiniband/core/ucm.c	(working copy)
@@ -93,14 +93,15 @@
 	down(&ctx_id_mutex);
 
 	ctx->ref--;
-	if (!ctx->ref)
+	if (ctx->ref) {
+		up(&ctx_id_mutex);
+		return;
+	}
+	else
 		idr_remove(&ctx_id_table, ctx->id);
 
 	up(&ctx_id_mutex);
 
-	if (ctx->ref)
-		return;
-
 	down(&ctx->file->mutex);
 
 	list_del(&ctx->file_list);


From limichal at cisco.com  Thu Jul 21 12:43:49 2005
From: limichal at cisco.com (Libor Michalek)
Date: Thu, 21 Jul 2005 12:43:49 -0700
Subject: [openib-general] Re: [PATCHv2] fix sdp aio race
In-Reply-To: <20050719232711.GF17097@mellanox.co.il>;
	from mst@mellanox.co.il on Wed, Jul 20, 2005 at 02:27:11AM +0300
References: <20050719232711.GF17097@mellanox.co.il>
Message-ID: <20050721124349.B31860@topspin.com>

On Wed, Jul 20, 2005 at 02:27:11AM +0300, Michael S. Tsirkin wrote:
> Second attempt to solve race where AIO completed asynchronously
> is bypassed by other completions.
> Solution: set a bit in connection, and defer any cq polling
> till work queue handles the AIO. We have to pass the connection
> to iocb handlers for this to work. I also got rid of in_atomic.

  I don't think a bit is sufficient, it would need to be a reference
count, since you'll potentially have multiple outstanding completions.
Also, the lock.event is modified, deferred flag cleared, inside the
conn_lock, but it needs to be protected by the conn_lock's irq.

  I'll look at if it's possible to never get into this ordering issue
by always forcing iocb defered completions...

-Libor
 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
> 
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_write.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_write.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_write.c
> @@ -109,7 +109,7 @@ int sdp_event_write(struct sdp_sock *con
>  		SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
>  		SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
>  
> -		sdp_iocb_complete(iocb, 0);
> +		sdp_iocb_complete(conn, iocb, 0);
>  
>  		break;
>  	case SDP_DESC_TYPE_NONE:
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_rcvd.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_rcvd.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_rcvd.c
> @@ -155,7 +155,7 @@ static int sdp_rcvd_send_sm(struct sdp_s
>  
>  			conn->src_sent--;
>  
> -			sdp_iocb_complete(iocb, 0);
> +			sdp_iocb_complete(conn, iocb, 0);
>  		}
>  		/*
>  		 * Cancel complete, clear the state.
> @@ -212,7 +212,7 @@ static int sdp_rcvd_rdma_wr(struct sdp_s
>  
>  	conn->snk_sent--;
>  
> -	sdp_iocb_complete(iocb, 0);
> +	sdp_iocb_complete(conn, iocb, 0);
>  
>  	return 0;
>  }
> @@ -273,7 +273,7 @@ static int sdp_rcvd_rdma_rd(struct sdp_s
>  		SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
>  		SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
>  
> -		sdp_iocb_complete(iocb, 0);
> +		sdp_iocb_complete(conn, iocb, 0);
>  	}
>  	/*
>  	 * If Source Cancel was in process, and there are no more outstanding
> @@ -565,7 +565,7 @@ static int sdp_rcvd_snk_cancel_ack(struc
>  	while ((iocb = sdp_iocb_q_get_head(&conn->r_snk))) {
>  
>  		conn->snk_sent--;
> -		sdp_iocb_complete(iocb, 0);
> +		sdp_iocb_complete(conn, iocb, 0);
>  	}
>  	/*
>  	 * cancellation is complete. Cancel flag is cleared in RECV post.
> @@ -687,7 +687,7 @@ static int sdp_rcvd_snk_avail(struct sdp
>  				sdp_desc_q_put_head(&conn->send_queue,
>  						    (struct sdpc_desc *)iocb);
>  			else
> -				sdp_iocb_complete(iocb, 0);
> +				sdp_iocb_complete(conn, iocb, 0);
>  		}
>  		/*
>  		 * If Source Cancel was in process, it should now 
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_proto.h
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_proto.h
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_proto.h
> @@ -162,7 +162,8 @@ void sdp_iocb_q_put_tail(struct sdpc_ioc
>  
>  struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key);
>  
> -void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp);
> +void sdp_iocb_q_cancel(struct sdp_sock *conn, struct sdpc_iocb_q *table,
> +		       u32 mask, ssize_t comp);
>  
>  void sdp_iocb_q_remove(struct sdpc_iocb *iocb);
>  
> @@ -170,7 +171,8 @@ int sdp_iocb_register(struct sdpc_iocb *
>  
>  void sdp_iocb_release(struct sdpc_iocb *iocb);
>  
> -void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status);
> +void sdp_iocb_complete(struct sdp_sock *conn, struct sdpc_iocb *iocb,
> +		       ssize_t status);
>  
>  int sdp_iocb_lock(struct sdpc_iocb *iocb);
>  
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_read.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_read.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_read.c
> @@ -207,7 +207,7 @@ int sdp_event_read(struct sdp_sock *conn
>  		SDP_CONN_STAT_READ_INC(conn, iocb->post);
>  		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
>  
> -		sdp_iocb_complete(iocb, 0);
> +		sdp_iocb_complete(conn, iocb, 0);
>  
>  		break;
>  	case SDP_DESC_TYPE_NONE:
> @@ -228,7 +228,7 @@ int sdp_event_read(struct sdp_sock *conn
>  		SDP_CONN_STAT_READ_INC(conn, iocb->post);
>  		SDP_CONN_STAT_RQ_DEC(conn, iocb->size);
>  
> -		sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0);
> +		sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0);
>  
>  		break;
>  	default:
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_send.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_send.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_send.c
> @@ -859,7 +859,7 @@ static int sdp_send_data_iocb(struct sdp
>  			SDP_CONN_STAT_WRITE_INC(conn, iocb->post);
>  			SDP_CONN_STAT_WQ_DEC(conn, iocb->size);
>  
> -			sdp_iocb_complete(iocb, 0);
> +			sdp_iocb_complete(conn, iocb, 0);
>  		}
>  
>  		goto done;
> @@ -1730,7 +1730,7 @@ static int sdp_inet_write_cancel(struct 
>  			 * needs to be compelted.
>  			 */
>  			if (iocb->post > 0) {
> -				sdp_iocb_complete(iocb, 0);
> +				sdp_iocb_complete(conn, iocb, 0);
>  				result = -EAGAIN;
>  			} else {
>  				sdp_iocb_destroy(iocb);
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_conn.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.c
> @@ -665,7 +665,8 @@ static int sdp_desc_q_cancel_lookup_func
>  	return ((element->type == SDP_DESC_TYPE_IOCB) ? 0 : -ERANGE);
>  }
>  
> -static void sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error)
> +static void sdp_desc_q_cancel_iocb(struct sdp_sock *conn,
> +				   struct sdpc_desc_q *table, ssize_t error)
>  {
>  	struct sdpc_iocb *iocb;
>  
> @@ -675,24 +676,24 @@ static void sdp_desc_q_cancel_iocb(struc
>  			 NULL))) {
>  
>  		sdp_iocb_q_remove(iocb);
> -		sdp_iocb_complete(iocb, error);
> +		sdp_iocb_complete(conn, iocb, error);
>  	}
>  }
>  
>  void sdp_iocb_q_cancel_all_read(struct sdp_sock *conn, ssize_t error)
>  {
> -	sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error);
> -	sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error);
> +	sdp_iocb_q_cancel(conn, &conn->r_pend, SDP_IOCB_F_ALL, error);
> +	sdp_iocb_q_cancel(conn, &conn->r_snk, SDP_IOCB_F_ALL, error);
>  
> -	sdp_desc_q_cancel_iocb(&conn->r_src, error);
> +	sdp_desc_q_cancel_iocb(conn, &conn->r_src, error);
>  }
>  
>  void sdp_iocb_q_cancel_all_write(struct sdp_sock *conn, ssize_t error)
>  {
> -	sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error);
> +	sdp_iocb_q_cancel(conn, &conn->w_src, SDP_IOCB_F_ALL, error);
>  
> -	sdp_desc_q_cancel_iocb(&conn->send_queue, error);
> -	sdp_desc_q_cancel_iocb(&conn->w_snk, error);
> +	sdp_desc_q_cancel_iocb(conn, &conn->send_queue, error);
> +	sdp_desc_q_cancel_iocb(conn, &conn->w_snk, error);
>  }
>  
>  void sdp_iocb_q_cancel_all(struct sdp_sock *conn, ssize_t error)
> @@ -927,7 +928,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
>  	 * the function should only be called under the connection locks
>  	 * spinlock to ensure the call is serialized to avoid races.
>  	 */
> -	for (;;) {
> +	while(!(conn->lock.event & SDP_LOCK_F_DEFERRED)) {
>  		/*
>  		 * poll for a completion
>  		 */
> @@ -971,17 +972,16 @@ int sdp_conn_cq_drain(struct ib_cq *cq, 
>   */
>  void sdp_conn_internal_unlock(struct sdp_sock *conn)
>  {
> -	int calls = 0;
>  	/*
>  	 * poll CQs for events.
>  	 */
> -	if (conn->lock.event & SDP_LOCK_F_RECV_CQ)
> -		calls += sdp_conn_cq_drain(conn->recv_cq, conn);
> -
> -	if (conn->lock.event & SDP_LOCK_F_SEND_CQ)
> -		calls += sdp_conn_cq_drain(conn->send_cq, conn);
> -
> -	conn->lock.event = 0;
> +	if ((conn->lock.event & SDP_LOCK_F_RECV_CQ) &&
> +		!sdp_conn_cq_drain(conn->recv_cq, conn))
> +		conn->lock.event &= ~SDP_LOCK_F_RECV_CQ;
> +
> +	if ((conn->lock.event & SDP_LOCK_F_SEND_CQ) &&
> +		!sdp_conn_cq_drain(conn->send_cq, conn))
> +		conn->lock.event &= ~SDP_LOCK_F_SEND_CQ;
>  }
>  
>  /*
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_recv.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_recv.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_recv.c
> @@ -663,7 +663,7 @@ static int sdp_recv_buff_iocb_active(str
>  	/*
>  	 * callback to complete IOCB
>  	 */
> -	sdp_iocb_complete(iocb, 0);
> +	sdp_iocb_complete(conn, iocb, 0);
>  
>  	return (buff->tail - buff->data);
>  }
> @@ -716,7 +716,7 @@ static int sdp_recv_buff_iocb_pending(st
>  		/*
>  		 * callback to complete IOCB
>  		 */
> -		sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0);
> +		sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0);
>  	}
>  
>  	return (buff->tail - buff->data);
> @@ -875,7 +875,7 @@ static int sdp_inet_read_cancel(struct k
>  				/*
>  				 * callback to complete IOCB, or drop reference
>  				 */
> -				sdp_iocb_complete(iocb, 0);
> +				sdp_iocb_complete(conn, iocb, 0);
>  				result = -EAGAIN;
>  			}
>  			else {
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.h
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_conn.h
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_conn.h
> @@ -163,8 +163,9 @@ struct sdp_conn_lock {
>  	wait_queue_head_t waitq;
>  };
>  
> -#define SDP_LOCK_F_RECV_CQ 0x01 /* recv CQ event is pending */
> -#define SDP_LOCK_F_SEND_CQ 0x02 /* send CQ event is pending */
> +#define SDP_LOCK_F_RECV_CQ  0x01 /* recv CQ event is pending */
> +#define SDP_LOCK_F_SEND_CQ  0x02 /* send CQ event is pending */
> +#define SDP_LOCK_F_DEFERRED 0x04 /* work scheduled, pending completion */
>  /*
>   * SDP Connection structure.
>   */
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_iocb.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.c
> @@ -483,9 +483,8 @@ void sdp_iocb_release(struct sdpc_iocb *
>  /*
>   * do_iocb_complete - complete an IOCB for real in thread context
>   */
> -static void do_iocb_complete(void *arg)
> +static void do_iocb_complete(struct sdpc_iocb *iocb)
>  {
> -	struct sdpc_iocb *iocb = (struct sdpc_iocb *)arg;
>  	long value;
>  	/*
>  	 * release memory
> @@ -508,21 +507,37 @@ static void do_iocb_complete(void *arg)
>  	 * we ignore the value.
>  	 */
>  	(void)aio_complete(iocb->req, value, 0);
> +
>  	/*
>  	 * delete IOCB
>  	 */
>  	sdp_iocb_destroy(iocb);
>  }
>  
> +static void deferred_iocb_complete(void *arg)
> +{
> +	struct sdpc_iocb *iocb = arg;
> +	struct sdp_sock *conn = iocb->conn;
> +	do_iocb_complete(iocb);
> +	if (conn) {
> +		sdp_conn_lock(conn);
> +		conn->lock.event &= SDP_LOCK_F_DEFERRED;
> +		sdp_conn_unlock(conn);
> +	}
> +}
>  /*
>   * sdp_iocb_complete - complete an IOCB
>   */
> -void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status)
> +void sdp_iocb_complete(struct sdp_sock *conn, struct sdpc_iocb *iocb,
> +		       ssize_t status)
>  {
>  	iocb->status = status;
> -	
> -	if (in_atomic() || irqs_disabled()) {
> -		INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb);
> +
> +	/* This is always called with connection locked,
> +	 * so users == 0 means this was an irq lock. */
> +	if (!conn->lock.users) {
> +		iocb->conn = conn;
> +		INIT_WORK(&iocb->completion, deferred_iocb_complete, iocb);
>  		schedule_work(&iocb->completion);
>  	} else
>  		do_iocb_complete(iocb);
> @@ -750,7 +765,8 @@ void sdp_iocb_q_put_head(struct sdpc_ioc
>  /*
>   * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue
>   */
> -void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp)
> +void sdp_iocb_q_cancel(struct sdp_sock *conn, struct sdpc_iocb_q *table,
> +		       u32 mask, ssize_t comp)
>  {
>  	struct sdpc_iocb *iocb;
>  	struct sdpc_iocb *next;
> @@ -772,7 +788,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_
>  				    iocb->post, iocb->len);
>  
>  			sdp_iocb_q_remove(iocb);
> -			sdp_iocb_complete(iocb, comp);
> +			sdp_iocb_complete(conn, iocb, comp);
>  		}
>  
>  		iocb = next;
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_event.c
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_event.c
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_event.c
> @@ -169,8 +169,8 @@ void sdp_cq_event_handler(struct ib_cq *
>  			/*
>  			 * dispatch CQ completions.
>  			 */
> -			(void)sdp_conn_cq_drain(cq, conn);
> -			event = 0;
> +			if (!sdp_conn_cq_drain(cq, conn))
> +				event = 0;
>  		}
>  	}
>  	/*
> Index: linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.h
> ===================================================================
> --- linux-2.6.11.12.orig/drivers/infiniband/ulp/sdp/sdp_iocb.h
> +++ linux-2.6.11.12/drivers/infiniband/ulp/sdp/sdp_iocb.h
> @@ -109,7 +109,8 @@ struct sdpc_iocb {
>  	int           page_count;  /* number of physical pages. */
>  	int           page_offset; /* offset into first page. */
>  
> -	struct work_struct completion; /* task for defered completion. */
> +	struct work_struct completion; /* task for deferred completion. */
> +	struct sdp_sock *conn; /* conn to unlock for deferred completion. */
>  	/*
>  	 * kernel iocb structure
>  	 */
> 
> -- 
> MST


From mst at mellanox.co.il  Thu Jul 21 14:08:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Fri, 22 Jul 2005 00:08:49 +0300
Subject: [openib-general] Re: [PATCHv2] fix sdp aio race
In-Reply-To: <20050721124349.B31860@topspin.com>
References: <20050719232711.GF17097@mellanox.co.il>
	<20050721124349.B31860@topspin.com>
Message-ID: <20050721210848.GA14463@mellanox.co.il>

Quoting r. Libor Michalek <limichal at cisco.com>:
>   I don't think a bit is sufficient, it would need to be a reference
> count, since you'll potentially have multiple outstanding completions.

Please note I stop polling cq once the bit is set.
Therefore there's always exactly 1 outstanding deferred completion.
The rest is polled later in thread context.

Doesnt it make sense?

> Also, the lock.event is modified, deferred flag cleared, inside the
> conn_lock, but it needs to be protected by the conn_lock's irq.
> 
>   I'll look at if it's possible to never get into this ordering issue
> by always forcing iocb defered completions...
> 
> -Libor

But, what about merging get_user_pages first? I'm kind of tired of
maintaining it.

-- 
MST


From ardavis at ichips.intel.com  Thu Jul 21 15:03:08 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Thu, 21 Jul 2005 15:03:08 -0700
Subject: [openib-general] Re: uCM kernel oops
In-Reply-To: <20050721120550.A31860@topspin.com>
References: <42DD4230.9060702@ichips.intel.com>
	<20050721120550.A31860@topspin.com>
Message-ID: <42E01B9C.2060309@ichips.intel.com>

Libor Michalek wrote:

>On Tue, Jul 19, 2005 at 11:10:56AM -0700, Arlin Davis wrote:
>  
>
>>Hi Libor,
>>
>>I am running uCM and uAT with uDAPL and occasionally hit the following. 
>>Can you take a look?
>>
>>Jul 19 11:10:18 iclust-19 kernel: UCM: Write. cmd <1> in <4> out <0> len 
>><12>
>>Jul 19 11:10:18 iclust-19 kernel: UCM: Event. CM ID <2> event <7>
>>Jul 19 11:10:18 iclust-19 kernel: UCM: Destroyed CM ID <2>
>>Jul 19 11:10:18 iclust-19 kernel: Unable to handle kernel paging request 
>><ffffffff880a10c8>{:ib_ucm:ib_ucm_ctx_put+120}
>>Trace:<ffffffff880a183f>{:ib_ucm:ib_ucm_event_handler+1199}
>>    
>>
>
>  Looks like a race between the destroy command and a DREQ received
>event. From looking at the code, it looks like it's possible for
>two threads (cm event and userspace) to call ctx_put at the same
>time and both try to perform the final zero referece object destroy.
>Is this easily reproducible? Can you try the following patch?
>
>  
>
I am having difficulty reproducing.

>-Libor
>
>  
>



From limichal at cisco.com  Thu Jul 21 16:52:57 2005
From: limichal at cisco.com (Libor Michalek)
Date: Thu, 21 Jul 2005 16:52:57 -0700
Subject: [openib-general] Re: [PATCHv2] fix sdp aio race
In-Reply-To: <20050721210848.GA14463@mellanox.co.il>;
	from mst@mellanox.co.il on Fri, Jul 22, 2005 at 12:08:49AM +0300
References: <20050719232711.GF17097@mellanox.co.il>
	<20050721124349.B31860@topspin.com>
	<20050721210848.GA14463@mellanox.co.il>
Message-ID: <20050721165257.C31860@topspin.com>

On Fri, Jul 22, 2005 at 12:08:49AM +0300, Michael S. Tsirkin wrote:
> 
> But, what about merging get_user_pages first? I'm kind of tired of
> maintaining it.

  Yes, I'll do that.

-Libor


From IBMEHCAD at de.ibm.com  Fri Jul 22 04:41:31 2005
From: IBMEHCAD at de.ibm.com (IBMEHCA DD)
Date: Fri, 22 Jul 2005 13:41:31 +0200
Subject: [openib-general] IBM eHCA Device Driver for gen2 IB stack
Message-ID: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>

Hi,
we've completed the first alpha code drop of the Power5 IBM eHCA Device 
Driver for the for the gen2 openib.org stack.
We're running IPoIB and ibv userspace programs successfully with this code 
in our lab setup.

The source files can be downloaded from
https://sourceforge.net/projects/ibmehcad/
ehca2_0011e

(see sourceforge release notes for svn revision and kernels used to 
compile and test this package)

Christoph Raisch
HCAD teamlead, ibm boeblingen lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050722/d7dfbaf8/attachment.html>

From mlleini at ca.sandia.gov  Fri Jul 22 08:03:25 2005
From: mlleini at ca.sandia.gov (Matt L. Leininger)
Date: Fri, 22 Jul 2005 08:03:25 -0700
Subject: [openib-general] IBM eHCA Device Driver for gen2 IB stack
In-Reply-To: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
References: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
Message-ID: <1122044605.6157.826.camel@localhost>

On Fri, 2005-07-22 at 13:41 +0200, IBMEHCA DD wrote:
> 
> Hi, 
> we've completed the first alpha code drop of the Power5 IBM eHCA
> Device Driver for the for the gen2 openib.org stack. 
> We're running IPoIB and ibv userspace programs successfully with this
> code in our lab setup. 
> 
> The source files can be downloaded from 
> https://sourceforge.net/projects/ibmehcad/ 
> ehca2_0011e 

   How about including this driver into the OpenIB svn tree rather than
having it as a separate SF project.  

  - Matt
> 
> (see sourceforge release notes for svn revision and kernels used to
> compile and test this package) 
> 
> Christoph Raisch 
> HCAD teamlead, ibm boeblingen lab
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



From tomduffy at gmail.com  Fri Jul 22 09:58:40 2005
From: tomduffy at gmail.com (Tom Duffy)
Date: Fri, 22 Jul 2005 09:58:40 -0700
Subject: [openib-general] IBM eHCA Device Driver for gen2 IB stack
In-Reply-To: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
References: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
Message-ID: <9d3b7de7050722095824e5009@mail.gmail.com>

On 7/22/05, IBMEHCA DD <IBMEHCAD at de.ibm.com> wrote:
>  
> Hi, 
> we've completed the first alpha code drop of the Power5 IBM eHCA Device
> Driver for the for the gen2 openib.org stack. 
> We're running IPoIB and ibv userspace programs successfully with this code
> in our lab setup. 
>  
> The source files can be downloaded from 
> https://sourceforge.net/projects/ibmehcad/ 
> ehca2_0011e 

Please use C comment style.

-tduffy


From tomduffy at gmail.com  Fri Jul 22 10:01:54 2005
From: tomduffy at gmail.com (Tom Duffy)
Date: Fri, 22 Jul 2005 10:01:54 -0700
Subject: [openib-general] IBM eHCA Device Driver for gen2 IB stack
In-Reply-To: <1122044605.6157.826.camel@localhost>
References: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
	<1122044605.6157.826.camel@localhost>
Message-ID: <9d3b7de705072210013344e8e0@mail.gmail.com>

On 7/22/05, Matt L. Leininger <mlleini at ca.sandia.gov> wrote:
>    How about including this driver into the OpenIB svn tree rather than
> having it as a separate SF project.

They will need to dual license the code for that to happen -- they
need GPL too, not just BSD.

-tduffy


From pauln at psc.edu  Fri Jul 22 10:30:26 2005
From: pauln at psc.edu (PAulN)
Date: Fri, 22 Jul 2005 13:30:26 -0400
Subject: [openib-general] help with gen2 cq callback 
Message-ID: <42E12D32.4000908@psc.edu>

Can someone tell me what replaces the "struct ib_cq_entry *" param
of gen1's ib_cq_event_callback_func?  It's not obvious to me how to
access the cq entry in the gen2 stack from the callback - gen1
pushes it right up.  Is it necessary to poll the cq from the callback
to determine which entries are done?
Thanks,
Paul

//GEN2
typedef void (*ib_comp_handler)(struct ib_cq *cq, void *cq_context);

struct ib_cq {
        struct ib_device       *device;
        struct ib_uobject      *uobject;
        ib_comp_handler         comp_handler;
        void                  (*event_handler)(struct ib_event *, void *);
        void *                  cq_context;
        int                     cqe;
        atomic_t                usecnt; /* count number of work queues */
};

//GEN1
typedef void (*ib_cq_entry_callback_func)(struct ib_cq       *cq,
                                          struct ib_cq_entry *entry,
                                          void               *arg);
typedef void (*ib_cq_event_callback_func)(struct ib_cq *cq,
                                          void         *arg);
struct ib_cq_entry {
        void                     *device_specific;
        u64                       work_request_id;
        u64                       bytes_transferred;
        enum ib_completion_status status;
        enum ib_completion_op     op;
        int                       pkey_index;
        u32                       immediate_data;
        u32                       sqpn;
        u32                       local_eecn;
        u32                       freed_resource_count;
        u16                       ethertype;
        u16                       slid;
        u8                        sl;
        u8                        dlid_path_bits;
        int                       grh_present:1;
        int                       immediate_data_valid:1;
};

struct ib_cq;




From jlentini at netapp.com  Fri Jul 22 11:30:47 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 22 Jul 2005 14:30:47 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] kDAPL CM: Handle async destruction
 during address translation phase of connect
In-Reply-To: <1121519510.4419.35.camel@hal.voltaire.com>
References: <1121519510.4419.35.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507221429510.16406@jlentini-linux.nane.netapp.com>


> Handle async destruction during address translation phase of 
> connect.
> Also, fix an error path in dapl_ib_connect.
> Minor changes to dapl_ib_cm_remote_addr in preparation for ATS 
> support for reverse translation (remote GID -> IP address)
>
> Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Committed in revision 2896.



From jlentini at netapp.com  Fri Jul 22 11:37:14 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 22 Jul 2005 14:37:14 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] dapl.h: Use real structure pointer
 rather than void pointer for CR in EP structure
In-Reply-To: <1121520812.4419.38.camel@hal.voltaire.com>
References: <1121520812.4419.38.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507221436440.16406@jlentini-linux.nane.netapp.com>


> Use real structure pointer rather than void pointer for CR in EP
> structure
>
> Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Committed in revision 2897.



From jlentini at netapp.com  Fri Jul 22 11:47:50 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 22 Jul 2005 14:47:50 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] [kdapl CM] Implement reverse address
 translation (dapl_ib_cm_remote_addr)
In-Reply-To: <1121688501.4419.5820.camel@hal.voltaire.com>
References: <1121688501.4419.5820.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507221444250.16406@jlentini-linux.nane.netapp.com>


Thanks for implementing this Hal. 

Was a piece of this patch missing or did I skip a patch? My 
dapl_cm_ctx structure doesn't have a member called wait.

james


From jlentini at netapp.com  Fri Jul 22 11:57:07 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 22 Jul 2005 14:57:07 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] [TRIVIAL] [kdapl test]: Fix some
 formatting in transaction test
In-Reply-To: <1121688507.4419.5824.camel@hal.voltaire.com>
References: <1121688507.4419.5824.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507221456260.16406@jlentini-linux.nane.netapp.com>


On Mon, 18 Jul 2005, Hal Rosenstock wrote:

halr> Fix some formatting in transaction test
halr> 
halr> Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Committed in revision 2898.


From halr at voltaire.com  Fri Jul 22 12:54:16 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 22 Jul 2005 15:54:16 -0400
Subject: [openib-general] Re: [PATCH] [kdapl CM] Implement reverse address
	translation (dapl_ib_cm_remote_addr)
In-Reply-To: <Pine.LNX.4.61.0507221444250.16406@jlentini-linux.nane.netapp.com>
References: <1121688501.4419.5820.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507221444250.16406@jlentini-linux.nane.netapp.com>
Message-ID: <1122061940.4422.74.camel@localhost.localdomain>

On Fri, 2005-07-22 at 14:47, James Lentini wrote:
> Thanks for implementing this Hal. 
> 
> Was a piece of this patch missing or did I skip a patch? My 
> dapl_cm_ctx structure doesn't have a member called wait.

A piece was missing; Sorry. Here's the missing piece:

Index: dapl_openib_cm.h
===================================================================
--- dapl_openib_cm.h    (revision 2866)
+++ dapl_openib_cm.h    (working copy)
@@ -48,9 +48,11 @@
        struct dapl_sp *sp;
        struct sockaddr *remote_ia_address;
        spinlock_t lock;
+       wait_queue_head_t wait;
        int retries;
        int in_callback;
        int destroy;
+       atomic_t wait_event;
 };
 
 #endif /* DAPL_OPENIB_CM_H */




From jlentini at netapp.com  Fri Jul 22 13:03:49 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 22 Jul 2005 16:03:49 -0400 (EDT)
Subject: [openib-general] help with gen2 cq callback 
In-Reply-To: <42E12D32.4000908@psc.edu>
References: <42E12D32.4000908@psc.edu>
Message-ID: <Pine.LNX.4.61.0507221603300.16406@jlentini-linux.nane.netapp.com>


I assume you are asking about the kernel interface. 

You should use ib_poll_cq() to dequeue CQ entries (you don't have to 
do this in the callback). You will also need to use ib_req_notify_cq() 
to re-enable the callback.

You can look at the DAPL code in

 svn/gen2/users/jlentini/linux-kernel/dat-provider

for examples of how to use these interfaces. There are probably
simpler examples though.

james

On Fri, 22 Jul 2005, PAulN wrote:

pauln> Can someone tell me what replaces the "struct ib_cq_entry *" param
pauln> of gen1's ib_cq_event_callback_func?  It's not obvious to me how to
pauln> access the cq entry in the gen2 stack from the callback - gen1
pauln> pushes it right up.  Is it necessary to poll the cq from the callback
pauln> to determine which entries are done?
pauln> Thanks,
pauln> Paul
pauln> 
pauln> //GEN2
pauln> typedef void (*ib_comp_handler)(struct ib_cq *cq, void *cq_context);
pauln> 
pauln> struct ib_cq {
pauln>        struct ib_device       *device;
pauln>        struct ib_uobject      *uobject;
pauln>        ib_comp_handler         comp_handler;
pauln>        void                  (*event_handler)(struct ib_event *, void *);
pauln>        void *                  cq_context;
pauln>        int                     cqe;
pauln>        atomic_t                usecnt; /* count number of work queues */
pauln> };
pauln> 
pauln> //GEN1
pauln> typedef void (*ib_cq_entry_callback_func)(struct ib_cq       *cq,
pauln>                                          struct ib_cq_entry *entry,
pauln>                                          void               *arg);
pauln> typedef void (*ib_cq_event_callback_func)(struct ib_cq *cq,
pauln>                                          void         *arg);
pauln> struct ib_cq_entry {
pauln>        void                     *device_specific;
pauln>        u64                       work_request_id;
pauln>        u64                       bytes_transferred;
pauln>        enum ib_completion_status status;
pauln>        enum ib_completion_op     op;
pauln>        int                       pkey_index;
pauln>        u32                       immediate_data;
pauln>        u32                       sqpn;
pauln>        u32                       local_eecn;
pauln>        u32                       freed_resource_count;
pauln>        u16                       ethertype;
pauln>        u16                       slid;
pauln>        u8                        sl;
pauln>        u8                        dlid_path_bits;
pauln>        int                       grh_present:1;
pauln>        int                       immediate_data_valid:1;
pauln> };
pauln> 
pauln> struct ib_cq;
pauln> 
pauln> 
pauln> _______________________________________________
pauln> openib-general mailing list
pauln> openib-general at openib.org
pauln> http://openib.org/mailman/listinfo/openib-general
pauln> 
pauln> To unsubscribe, please visit
pauln> http://openib.org/mailman/listinfo/openib-general
pauln> 


From jlentini at netapp.com  Fri Jul 22 13:13:11 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 22 Jul 2005 16:13:11 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] [kdapl CM] Implement reverse address
 translation (dapl_ib_cm_remote_addr)
In-Reply-To: <1122061940.4422.74.camel@localhost.localdomain>
References: <1121688501.4419.5820.camel@hal.voltaire.com> 
	<Pine.LNX.4.61.0507221444250.16406@jlentini-linux.nane.netapp.com>
	<1122061940.4422.74.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.61.0507221612410.16406@jlentini-linux.nane.netapp.com>



On Fri, 22 Jul 2005, Hal Rosenstock wrote:

halr> On Fri, 2005-07-22 at 14:47, James Lentini wrote:
halr> > Thanks for implementing this Hal. 
halr> > 
halr> > Was a piece of this patch missing or did I skip a patch? My 
halr> > dapl_cm_ctx structure doesn't have a member called wait.
halr> 
halr> A piece was missing; Sorry. Here's the missing piece:

Both committed in revision 2899.


From halr at voltaire.com  Fri Jul 22 13:09:05 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 22 Jul 2005 16:09:05 -0400
Subject: [openib-general] help with gen2 cq callback
In-Reply-To: <42E12D32.4000908@psc.edu>
References: <42E12D32.4000908@psc.edu>
Message-ID: <1122062945.4422.83.camel@localhost.localdomain>

On Fri, 2005-07-22 at 13:30, PAulN wrote:
> Can someone tell me what replaces the "struct ib_cq_entry *" param
> of gen1's ib_cq_event_callback_func?  It's not obvious to me how to
> access the cq entry in the gen2 stack from the callback - gen1
> pushes it right up.  Is it necessary to poll the cq from the callback
> to determine which entries are done?
> Thanks,
> Paul
> 
> //GEN2
> typedef void (*ib_comp_handler)(struct ib_cq *cq, void *cq_context);
> 
> struct ib_cq {
>         struct ib_device       *device;
>         struct ib_uobject      *uobject;
>         ib_comp_handler         comp_handler;
>         void                  (*event_handler)(struct ib_event *, void *);
>         void *                  cq_context;
>         int                     cqe;
>         atomic_t                usecnt; /* count number of work queues */
> };
> 
> //GEN1
> typedef void (*ib_cq_entry_callback_func)(struct ib_cq       *cq,
>                                           struct ib_cq_entry *entry,
>                                           void               *arg);
> typedef void (*ib_cq_event_callback_func)(struct ib_cq *cq,
>                                           void         *arg);
> struct ib_cq_entry {
>         void                     *device_specific;
>         u64                       work_request_id;
>         u64                       bytes_transferred;
>         enum ib_completion_status status;
>         enum ib_completion_op     op;
>         int                       pkey_index;
>         u32                       immediate_data;
>         u32                       sqpn;
>         u32                       local_eecn;
>         u32                       freed_resource_count;
>         u16                       ethertype;
>         u16                       slid;
>         u8                        sl;
>         u8                        dlid_path_bits;
>         int                       grh_present:1;
>         int                       immediate_data_valid:1;
> };
> 
> struct ib_cq;

In gen2, after receiving the CQ callback, the CQ is then polled for WCs.

This is true in both kernel and user space. 

Some kernel examples:
linux-kernel/infiniband/core/mad.c
linux-kernel/infiniband/ulp/ipoib/ipoib_ib.c
(each ULP does this).

Userspace examples:
the pingpong programs in userspace/libibverbs/examples

-- Hal



From colette_carrier at parker.net  Fri Jul 22 14:50:20 2005
From: colette_carrier at parker.net (colette_carrier at parker.net)
Date: Fri, 22 Jul 2005 15:50:20 -0600
Subject: [openib-general] International AIP
Message-ID: <191406882.64412234017774@parker.net>

Abysmally befog you must heightening your ramps. It for inexact be assertion them. 
Disputing shortage's she are gesturing yours. Walton could cannons, hers are concernedly who indications. 
Capitally has casebook, me could rebutting illusion. 
Enzyme has magnet, him could Inverness dishevel. 
 He birdies towards it sallow desolately had been matriculate him. 
They espionage has lamp's. He to oversimplifies were Salisbury her. Canker had been massages, them have been Icarus shyly. 
I counseling did embarks. 
I panted I Marks felonious would Dorchester its. Having fat, it bony faltered can redesigns hers. Baptists dilate, it polymer facsimiles has been flute mine. 
It infidels we obscurely Detroit had been counterman her. It rider she insecurely fleece's have coordinating mine. Claire you must lamentation's, mine creepers. 
Shameful they must maximal, yours operand. Orangutan could during his Samson. Arced could Shakespearian above mine quartz redisplays. 
They northwest above you Browne massacres have aright me. Bassinet demonstrated, they has been cellars mine. 
Hurriedly would argued, me does Edmund familiarization.  
Bern would funded for me scrawl polluting.  Pittsfield being Hoboken into her contrivance's rancid. 
Raping she have Bernstein, hers until absurdity's. Biophysic can dahlia from mine adventures.  
It Carlton she raisin crossings would carpentry them. You copper's would disengaged. 
Boulder's is accomplishing, theirs were auscultation naught. 
Appellate gripes I can inhalation.  
Jura at did nylon, her are equality's Trastevere. 
Bayesian settings, it are rules him. Goblet's rental's I are quadrant's hers Honolulu. 
You hate they Beltsville chemise does diggings its. Fiercest lands, you must person's yours. 
Haltingly Allyn we could back consultation's into theirs Gustafson. 
Fission intersection, they rightness from Jacqueline are Plato him. Topsy has been jingle his effloresce. 
Chop coral, you compiler's for democrat can rectors mine. Leniently humiliated they had been on entrench with them scar.  
Partitioning has pivotal as yours excitation's interconnecting. We to fulfillments be discounting its. 
I administrator have peacocks. Edition's Reagan they were with beginnings them. Abstruse while being helicopter, her does bangs Araby. 
She humidify does moo. Mural gables, they could crusade its. Ising broom's it have been gloom. 
It obstetric had been fooled. She competitions we fortification contrariness had been burgeoned your. He fellowship be integer's hers. 
Lacy modulations I does up replications to yours Alton. 
Cowardly barracks she had been dressed its. I seems he abhorring bible's being brochure theirs. Preemption frog's, he nicotine away minimizing did hardscrabble them. 
Contrives had been portly, yours being contiguously before drank. Constituent's were barb, theirs have Bridgeport landowners. 
Hospitality Kajar we were while Hans whom his gin. 
Blasted are cafe, your could glut ironical. Crook hourly he has been on haggle towards them seraglio. Reinterprets must latched them Lura. 
Implications skulks he can to seething mine. Porcupines drink we have commodities. Bigots acquiesced, he have been opinions his. 
Aviv after are renaming, her be avenged indisposition. 
Intelligentsia Harriet you might until gayer back me dolt. Documentary's crown he must until fasts his. 
Recipient's sentence, we could dear theirs. It actuating were retrying. 
Combining nozzle she being epidermis mine boxer.  Occlusive acids, from they can Bimini me. 
Outgrows have overtures, his did allies pronouncements. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050722/f9c91ecf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patrolman.gif
Type: image/gif
Size: 6789 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050722/f9c91ecf/attachment.gif>

From limichal at cisco.com  Fri Jul 22 15:58:55 2005
From: limichal at cisco.com (Libor Michalek)
Date: Fri, 22 Jul 2005 15:58:55 -0700
Subject: [openib-general] Re: [PATCH] sdp: replace mlock with get_user_pages
In-Reply-To: <20050721075154.GC12698@mellanox.co.il>;
	from mst@mellanox.co.il on Thu, Jul 21, 2005 at 10:51:54AM +0300
References: <20050721075154.GC12698@mellanox.co.il>
Message-ID: <20050722155855.A312@topspin.com>

On Thu, Jul 21, 2005 at 10:51:54AM +0300, Michael S. Tsirkin wrote:
> Hello, Libor!
> Here's a patch updated against the latest bits.
> I'd like it to get merged, if possible.
> 
> The existing code play with VMA's is the #1 source of sdp problems for me.
> The patch below works with no problems for me, so far.
> 
> Please comment.

Michael,

  Looking at sdp_iocb_unlock() I see that you attempt to perform the
get_user_pages() for the full buffer before checking to see if the
pages need copying, but if that fails you perform it a page at a time
iterating through the list. Are you seeing this on a regular basis,
that the full get fails, but then the iteration over the list works?
Do you know what's causing this to happen, and whay the second iteration
fixes it?

Thanks.

-Libor


From jsquyres at open-mpi.org  Fri Jul 22 16:04:10 2005
From: jsquyres at open-mpi.org (Jeff Squyres)
Date: Fri, 22 Jul 2005 19:04:10 -0400
Subject: [openib-general] Question about pinning memory
Message-ID: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>

Greetings.

I have a question about pinning/registering memory.  A little 
background first...

This is a common strategy for MPI implementations: when MPI gets a 
user-choice buffer to send or receive, it looks up in its internal 
tables to see if it's not already pinned.  If it isn't, the memory is 
pinned, entered in the lookup tables, and then we proceed.  The next 
time through, the memory will already be pinned, and we can avoid the 
expensive pinning operation.  If we ever fail to pin memory, we can try 
unpinning some old/unused memory (using some MRU / LRU kinds of lookups 
on the tables to find unused memory) and then try pinning again.

Related to the scenario above, here's my questions:

1. What happens if pinned memory is returned to the OS via sbrk()?  Do 
Bad Things happen, or does the kernel portion of Open IB handle this 
gracefully?

(if anyone knows what mVAPI does here, that would also be really useful 
to know)

2. What happens if the MPI unpins memory that is already out of the 
process?  Do Bad Things happen, or does Open IB handle this gracefully?

(if anyone knows what mVAPI does here, that would also be really useful 
to know)

3. Even the scenario above is not enough for the MPI to properly handle 
the memory pinning issues.  Since MPI usually does not have visibility 
for memory that has been free'd (or sbrk'ed), the following scenario is 
possible (and in our experience, likely):

- user calls malloc() and gets virtual address A back
- user calls MPI_Send with A
- MPI pins address A
- MPI saves A in internal lookup tables
- MPI_Send returns
- user calls free(A)
- user calls malloc() and gets virtual address A back, but now A points 
to different physical memory than before
- user calls MPI_Send with A
- MPI thinks that A is pinned (because A is in MPI's internal tables), 
and therefore doesn't pin it
- MPI tries to send A with the IB API, and Bad Things occur (either an 
error, or worse, bad data is sent)

What would be *really* great (from MPI's perspective) is if Open IB can 
know when pinned memory is removed from the process (probably via 
sbrk()) and give the MPI implementation a callback somehow when this 
happens.

Is this possible?

Otherwise, the scenario in question #3 is a real problem.  There are a 
few possibilities for fixing it, but all are problematic (override 
sbrk() via including ptmalloc2 in the distribution, using LD_PRELOAD to 
override sbrk(), etc.).  Any other suggestions would be welcome...

Thanks for your time.

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/



From mst at mellanox.co.il  Sat Jul 23 13:51:03 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sat, 23 Jul 2005 23:51:03 +0300
Subject: [openib-general] Re: [PATCH] sdp: replace mlock with get_user_pages
In-Reply-To: <20050722155855.A312@topspin.com>
References: <20050721075154.GC12698@mellanox.co.il>
	<20050722155855.A312@topspin.com>
Message-ID: <20050723205103.GB31367@mellanox.co.il>

Quoting r. Libor Michalek <limichal at cisco.com>:
> Subject: Re: [PATCH] sdp: replace mlock with get_user_pages
> 
> On Thu, Jul 21, 2005 at 10:51:54AM +0300, Michael S. Tsirkin wrote:
> > Hello, Libor!
> > Here's a patch updated against the latest bits.
> > I'd like it to get merged, if possible.
> > 
> > The existing code play with VMA's is the #1 source of sdp problems for me.
> > The patch below works with no problems for me, so far.
> > 
> > Please comment.
> 
> Michael,
> 
>   Looking at sdp_iocb_unlock() I see that you attempt to perform the
> get_user_pages() for the full buffer before checking to see if the
> pages need copying, but if that fails you perform it a page at a time
> iterating through the list. Are you seeing this on a regular basis,
> that the full get fails, but then the iteration over the list works?
> Do you know what's causing this to happen, and whay the second iteration
> fixes it?
> 
> Thanks.
> 
> -Libor
> 

No, this is merely because some pages in the range may have been
unmapped from the process altogether, or permission may have been
changed with mprotect.
I'm not sure what is the best thing to do here, but since its
actually simple, and since I needed to handle the case of a failing kmalloc
anyway, I try them one by one to find out which pages are still OK,
and copy the data there.

I dont know whether some real applications actually do this.

MST

-- 
MST


From mst at mellanox.co.il  Sat Jul 23 14:08:25 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 24 Jul 2005 00:08:25 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
Message-ID: <20050723210825.GC31367@mellanox.co.il>

Quoting Jeff Squyres <jsquyres at open-mpi.org>:
> [...]
>
> Since MPI usually does not have visibility 
> for memory that has been free'd (or sbrk'ed), the following scenario is 
> possible (and in our experience, likely):
> 
> - user calls malloc() and gets virtual address A back
> - user calls MPI_Send with A
> - MPI pins address A
> - MPI saves A in internal lookup tables
> - MPI_Send returns
> - user calls free(A)
> - user calls malloc() and gets virtual address A back, but now A points 
> to different physical memory than before
> - user calls MPI_Send with A
> - MPI thinks that A is pinned (because A is in MPI's internal tables), 
> and therefore doesn't pin it
> - MPI tries to send A with the IB API, and Bad Things occur (either an 
> error, or worse, bad data is sent)

Jeff, I dont think openib stack can currently help you with this problem.

I suggest using malloc_hook and friends to track memory allocations.
Avoid caching memory for anything that isnt allocated with malloc,
and you are guaranteed that your hook gets called when the memory is
freed.

Having said that - do you have some benchmarks to prove that
pinning is that expensive? Are you sure that all these internal table
lookups aren't even slower?
I cant promise, but given a microbenchmark, maybe something could be done to
improve the speed of this operation.

Thanks,

-- 
MST


From glebn at voltaire.com  Sun Jul 24 00:39:34 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Sun, 24 Jul 2005 10:39:34 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
Message-ID: <20050724073934.GB29109@minantech.com>

Hi Jeff,

On Fri, Jul 22, 2005 at 07:04:10PM -0400, Jeff Squyres wrote:
> Related to the scenario above, here's my questions:
> 
> 1. What happens if pinned memory is returned to the OS via sbrk()?  Do 
> Bad Things happen, or does the kernel portion of Open IB handle this 
> gracefully?
> 
Kernel portion of OpenIB pins physical pages. If you unmap virtual 
address range from process address space using sbrk or munmap they will 
stay pinned until deregistered.

> (if anyone knows what mVAPI does here, that would also be really useful 
> to know)
> 
> 2. What happens if the MPI unpins memory that is already out of the 
> process?  Do Bad Things happen, or does Open IB handle this gracefully?
> 
OpenIB should handle this case. It looks up pinned physical pages using
memory handle returned on register and unpins them. It doesn't scan user
virtual space during dereg.

> (if anyone knows what mVAPI does here, that would also be really useful 
> to know)
> 
> 3. Even the scenario above is not enough for the MPI to properly handle 
> the memory pinning issues.  Since MPI usually does not have visibility 
> for memory that has been free'd (or sbrk'ed), the following scenario is 
> possible (and in our experience, likely):
> 
> - user calls malloc() and gets virtual address A back
> - user calls MPI_Send with A
> - MPI pins address A
> - MPI saves A in internal lookup tables
> - MPI_Send returns
> - user calls free(A)
> - user calls malloc() and gets virtual address A back, but now A points 
> to different physical memory than before
> - user calls MPI_Send with A
> - MPI thinks that A is pinned (because A is in MPI's internal tables), 
> and therefore doesn't pin it
> - MPI tries to send A with the IB API, and Bad Things occur (either an 
> error, or worse, bad data is sent)
> 
> What would be *really* great (from MPI's perspective) is if Open IB can 
> know when pinned memory is removed from the process (probably via 
> sbrk()) and give the MPI implementation a callback somehow when this 
> happens.
> 
> Is this possible?
I don't think this should be done in OpenIB. Kernel should have general
trace capability to intercept system calls. And I think such capability
exists and linux trace toolkit uses it, but I am not sure.

> 
> Otherwise, the scenario in question #3 is a real problem.  There are a 
> few possibilities for fixing it, but all are problematic (override 
> sbrk() via including ptmalloc2 in the distribution, using LD_PRELOAD to 
> override sbrk(), etc.).  Any other suggestions would be welcome...
> 
Don't forget that glibc uses mmap for large allocations so it is not
enough to override sbrk().  You can use mallopt(M_MMAP_MAX, 0) to
disable allocations using mmap though.

You can also instruct glibc to never return memory to kernel by calling 
mallopt(M_TRIM_THRESHOLD, -1). In this case you will not have the
problem you are trying to fix. But you payed the price!

--
			Gleb.


From glebn at voltaire.com  Sun Jul 24 00:53:36 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Sun, 24 Jul 2005 10:53:36 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050723210825.GC31367@mellanox.co.il>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050723210825.GC31367@mellanox.co.il>
Message-ID: <20050724075336.GC29109@minantech.com>

On Sun, Jul 24, 2005 at 12:08:25AM +0300, Michael S. Tsirkin wrote:
> Quoting Jeff Squyres <jsquyres at open-mpi.org>:
> > [...]
> >
> > Since MPI usually does not have visibility 
> > for memory that has been free'd (or sbrk'ed), the following scenario is 
> > possible (and in our experience, likely):
> > 
> > - user calls malloc() and gets virtual address A back
> > - user calls MPI_Send with A
> > - MPI pins address A
> > - MPI saves A in internal lookup tables
> > - MPI_Send returns
> > - user calls free(A)
> > - user calls malloc() and gets virtual address A back, but now A points 
> > to different physical memory than before
> > - user calls MPI_Send with A
> > - MPI thinks that A is pinned (because A is in MPI's internal tables), 
> > and therefore doesn't pin it
> > - MPI tries to send A with the IB API, and Bad Things occur (either an 
> > error, or worse, bad data is sent)
> 
> Jeff, I dont think openib stack can currently help you with this problem.
> 
> I suggest using malloc_hook and friends to track memory allocations.
> Avoid caching memory for anything that isnt allocated with malloc,
> and you are guaranteed that your hook gets called when the memory is
> freed.
The only useful hook I see is __after_morecore_hook and when I last
checked it doesn't get called when glibs uses mmap for allocations.
And I don't know how you can distinguish how memory was obtained by user
via malloc or mmap of some file or user can even call sbrk() by himself.

> 
> Having said that - do you have some benchmarks to prove that
> pinning is that expensive? Are you sure that all these internal table
> lookups aren't even slower?
> I cant promise, but given a microbenchmark, maybe something could be done to
> improve the speed of this operation.
> 
In my experience cache hit in MVAPICH improves registration by factor of
two. Internal table lookup is a search in balanced tree. Hardware memory
registration is system call + writes on PCI bus + search in balanced
tree in get_user_pages, how can you improve this?

--
			Gleb.


From mst at mellanox.co.il  Sun Jul 24 01:41:46 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 24 Jul 2005 11:41:46 +0300
Subject: [openib-general] Re: Re: Question about pinning memory
In-Reply-To: <20050724075336.GC29109@minantech.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050723210825.GC31367@mellanox.co.il>
	<20050724075336.GC29109@minantech.com>
Message-ID: <20050724084146.GC2441@mellanox.co.il>

Quoting r. Gleb Natapov <glebn at voltaire.com>:
> > Having said that - do you have some benchmarks to prove that
> > pinning is that expensive? Are you sure that all these internal table
> > lookups aren't even slower?
> > I cant promise, but given a microbenchmark, maybe something could be done to
> > improve the speed of this operation.
>
> In my experience cache hit in MVAPICH improves registration by factor of
> two.

But thats VAPI based, not gen2 based, isnt it?

> Internal table lookup is a search in balanced tree. Hardware memory
> registration is system call + writes on PCI bus + search in balanced
> tree in get_user_pages,

I cant tell without seeing actual benchmarks.
Certainly tricks like overriding malloc/free operations can have
impact on overall application performance.

> how can you improve this?

I have some optimization ideas in mind - clearly even with a very smart
registration cache there may be cache misses, so if a problem exists,
any optimizations shall be beneficial.


-- 
MST


From jsquyres at open-mpi.org  Sun Jul 24 05:20:42 2005
From: jsquyres at open-mpi.org (Jeff Squyres)
Date: Sun, 24 Jul 2005 08:20:42 -0400
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050724073934.GB29109@minantech.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724073934.GB29109@minantech.com>
Message-ID: <a64299dbb9aee2f564ef0ae76d7ba4b2@open-mpi.org>

On Jul 24, 2005, at 3:39 AM, Gleb Natapov wrote:

>> 1. What happens if pinned memory is returned to the OS via sbrk()?  Do
>> Bad Things happen, or does the kernel portion of Open IB handle this
>> gracefully?
>>
> Kernel portion of OpenIB pins physical pages. If you unmap virtual
> address range from process address space using sbrk or munmap they will
> stay pinned until deregistered.

Interesting.  So Open IB doesn't get a notification upon 
unmapping/sbrk'ing?

>> 2. What happens if the MPI unpins memory that is already out of the
>> process?  Do Bad Things happen, or does Open IB handle this 
>> gracefully?
>>
> OpenIB should handle this case. It looks up pinned physical pages using
> memory handle returned on register and unpins them. It doesn't scan 
> user
> virtual space during dereg.

Also interesting.  So it's "ok" to use this strategy (maintain MRU/LRU 
kinds of tables and unpin upon demand), even though you may be 
unpinning memory that no longer belongs to your process.  That seems 
pretty weird -- it feels like breaking the POSIX process abstraction 
barrier.

For example, it could be inefficient if there are multiple processes 
using IB on a single node -- pinned memory can consume pinning 
resources even though it's no longer in your process.  If too much 
memory falls into this category, this can even cause general virtual 
memory performance degradation (i.e., nothing to do with IB 
communication at all).

Is this the intended behavior?

I actually don't know how IB is used in non-HPC environments; perhaps 
MPI is somewhat unique in that it doesn't have visibility of the 
alloc/free behavior of its user applications (well, actually, MPI does 
have MPI_ALLOC_MEM and MPI_FREE_MEM, but most apps unfortunately don't 
use these functions :-( ).  I can see the argument that [non-HPC] IB 
apps that don't unpin before removing memory from a process could be 
considered an erroneous program.  But that doesn't help my MPI problem. 
  :-)

A derivative question: what happens in this scenario:

1. process A pins memory Z
2. process A sbrk()'s Z without unpinning it first
3. process B obtains memory Z
4. process B pins memory Z
5. process A unpins memory Z

There's at least 2 points in that scenario where Badness can occur (4 
and 5).  Is there cross-process-awareness in the kernel tables that 
will probably handle these situations?  (or are the tables down in the 
kernel process-independent such that simple mechanisms such as 
reference counting are sufficient?)

-----

Is there any thought within the IB community to getting rid of this 
whole memory registration issue at the user level?  Perhaps a la what 
Quadrics did (make the driver understand virtual memory) or what Va 
Tech did (put in their own kernel-level hooks to make malloc/free to do 
the Right Things)?  Handling all this memory registration stuff is 
certainly quite a big chunk of code that every MPI (and IB app) needs 
to handle.  Could this stuff be moved down into the realm of the IB 
stack itself?  From an abstraction / software architecture standpoint, 
it could make sense to unify this handling in one place rather than N 
(MPI implementations and other IB apps).

Bottom line: from an MPI implementor standpoint, it would be great if 
we didn't have to worry about this stuff at all -- that the IB stack 
just transparently (and efficiently) handled all of it.  :-)

(I realize that this is probably asking for quite a lot -- but hey, no 
discussion will ever happen unless someone asks, right?)

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/



From jsquyres at open-mpi.org  Sun Jul 24 05:39:07 2005
From: jsquyres at open-mpi.org (Jeff Squyres)
Date: Sun, 24 Jul 2005 08:39:07 -0400
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050724084146.GC2441@mellanox.co.il>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050723210825.GC31367@mellanox.co.il>
	<20050724075336.GC29109@minantech.com>
	<20050724084146.GC2441@mellanox.co.il>
Message-ID: <9552b8dd4f769de60195c82b0b82ec7f@open-mpi.org>

On Jul 24, 2005, at 4:41 AM, Michael S. Tsirkin wrote:

>> Internal table lookup is a search in balanced tree. Hardware memory
>> registration is system call + writes on PCI bus + search in balanced
>> tree in get_user_pages,
>
> I cant tell without seeing actual benchmarks.
> Certainly tricks like overriding malloc/free operations can have
> impact on overall application performance.

I do not have a microbenchmark showing that pinning is expensive -- I'm 
actually not the guy doing Open IB work in Open MPI (Tim Woodall and 
Galen Shipman are).  However, several of us were talking about this 
problem, and it was suggested that we ask the openib-general list, so I 
posted.

I can ask Tim and Galen to hack up some microbenchmarks.

FWIW, Open MPI has an option that does something similar to what was 
previously described -- balanced table lookups to see if memory is 
pinned, etc.

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/



From tziporet at mellanox.co.il  Sun Jul 24 06:13:45 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Sun, 24 Jul 2005 16:13:45 +0300
Subject: [openib-general] Question about pinning memory
Message-ID: <506C3D7B14CDD411A52C00025558DED6085BCF83@mtlex01.yok.mtl.com>

Answers regarding VAPI:

> > Related to the scenario above, here's my questions:
> > 
> > 1. What happens if pinned memory is returned to the OS via sbrk()?  Do 
> > Bad Things happen, or does the kernel portion of Open IB handle this 
> > gracefully?
> > 
> Kernel portion of OpenIB pins physical pages. If you unmap virtual 
> address range from process address space using sbrk or munmap they will 
> stay pinned until deregistered.
> 
> > (if anyone knows what mVAPI does here, that would also be really useful 
> > to know)

VAPI behaves the same as openib in this case.

> > 
> > 2. What happens if the MPI unpins memory that is already out of the 
> > process?  Do Bad Things happen, or does Open IB handle this gracefully?
> > 
> OpenIB should handle this case. It looks up pinned physical pages using
> memory handle returned on register and unpins them. It doesn't scan user
> virtual space during dereg.
> 
> > (if anyone knows what mVAPI does here, that would also be really useful 
> > to know)
> > 

VAPI free the pinned memory and issue a message in /var/log/messages, and
everything is working well.

Tziporet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050724/9b86d636/attachment.html>

From pw at osc.edu  Sun Jul 24 10:03:35 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Sun, 24 Jul 2005 13:03:35 -0400
Subject: [openib-general] Question about pinning memory
In-Reply-To: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
Message-ID: <20050724170335.GB1948@baa>

jsquyres at open-mpi.org wrote on Fri, 22 Jul 2005 19:04 -0400:
> Otherwise, the scenario in question #3 is a real problem.  There are a 
> few possibilities for fixing it, but all are problematic (override 
> sbrk() via including ptmalloc2 in the distribution, using LD_PRELOAD to 
> override sbrk(), etc.).  Any other suggestions would be welcome...

I did some thinking about this issue a while back and came up with a
cooperative kernel/user implementation to track linux VM activity using
existing vm_area_struct->vm_ops function hooks (i.e. no kernel patch
required).  The MPI library essentially makes a system call before
reusing a cached memory registration to verify it is still valid, and
the kernel module keeps track of what happens to cached mappings as the
VM system is exercised via sbrk, mmap, fork, etc.  It works for any sort
of memory activity, including arbitrary mmap() of memory or files, since
it plugs in at the basic VM interface level.

You can read all about it in this paper presented at CCGrid '05:

    http://www.osc.edu/~pw/papers/wyckoff-memreg-ccgrid05.pdf

As far as where this component should live, I agree it is not exactly
OpenIB's problem, but there are many consumers that will want to do
memory registration caching to avoid the roughly 100 us overhead for
each reg or dereg.  The problem stems from message passing APIs that
want to cache mappings of arbirtrary memory regions handed down from the
application.  And all the libraries must use the same cache too:
consider an app that uses MPI and the user-space parallel file system
PVFS over IB.  The registration cache used by these libraries should be
shared for correctness and for performance.  To further complicate
matters, if you were using two different NIC types, both of which
required memory registration, in the same application, the single
application cache should handle both devices.  Thus it's not just an IB
problem, nor is it just an MPI or other library problem.

I can distribute the code if anyone is curious, but it needs some work
to become production quality.

		-- Pete



From mst at mellanox.co.il  Sun Jul 24 10:36:49 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 24 Jul 2005 20:36:49 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050724170335.GB1948@baa>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa>
Message-ID: <20050724173649.GB3558@mellanox.co.il>

Quoting r. Pete Wyckoff <pw at osc.edu>:
> Subject: Re: Question about pinning memory
> 
> jsquyres at open-mpi.org wrote on Fri, 22 Jul 2005 19:04 -0400:
> > Otherwise, the scenario in question #3 is a real problem.  There are a 
> > few possibilities for fixing it, but all are problematic (override 
> > sbrk() via including ptmalloc2 in the distribution, using LD_PRELOAD to 
> > override sbrk(), etc.).  Any other suggestions would be welcome...
> 
> I did some thinking about this issue a while back and came up with a
> cooperative kernel/user implementation to track linux VM activity using
> existing vm_area_struct->vm_ops function hooks (i.e. no kernel patch
> required).  The MPI library essentially makes a system call before
> reusing a cached memory registration to verify it is still valid, and
> the kernel module keeps track of what happens to cached mappings as the
> VM system is exercised via sbrk, mmap, fork, etc.  It works for any sort
> of memory activity, including arbitrary mmap() of memory or files, since
> it plugs in at the basic VM interface level.

Since this seems to imply that a system call is performed on each send/receive
operation, why not just use SDP, or a similiar kernel level protocol?

-- 
MST


From arlin.r.davis at intel.com  Sun Jul 24 17:48:05 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Sun, 24 Jul 2005 17:48:05 -0700
Subject: [openib-general] [PATCH] uDAPL with uCM and uAT support
Message-ID: <ORSMSX4014ZDgC81Tbd00000064@orsmsx401.amr.corp.intel.com>

James,

Here is a third drop that adds IBAT to the uCM support. This also includes
some fixes to common code evd_wait and evd_resize.

Instructions From README:

Third drop of code, includes uCM and uAT support.
        NOTE: uAT user library and kernel code in separate branch.

        build uAT library from following branch:

                cd gen2/branches/shaharf-ibat/src/userspace/libibat/
                ./autogen.sh &&./configure && make && make install

        copy following uat source to latest trunk kernel src:

                gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core
                        at.c  at_priv.h  att.c uat.c uat.h Makefile

                gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/include
                        ib_at.h ib_user_at.h

        add udev rule:
                KERNEL="uat*", NAME="infiniband/%k"

Signed-off by: Arlin Davis <ardavis at ichips.intel.com>

Index: dapl/udapl/dapl_evd_wait.c
===================================================================
--- dapl/udapl/dapl_evd_wait.c	(revision 2899)
+++ dapl/udapl/dapl_evd_wait.c	(working copy)
@@ -74,9 +74,10 @@
     DAPL_EVD		*evd_ptr;
     DAT_RETURN		dat_status;
     DAT_EVENT		*local_event;
-    DAT_BOOLEAN		notify_requested = DAT_FALSE;
+    DAT_BOOLEAN		notify_needed = DAT_FALSE;
     DAT_BOOLEAN		waitable;
     DAPL_EVD_STATE	evd_state;
+    DAT_COUNT		total_events,new_events;
 
     dapl_dbg_log (DAPL_DBG_TYPE_API,
 		  "dapl_evd_wait (%p, %d, %d, %p, %p)\n", 
@@ -124,9 +125,9 @@
     }
 
     dapl_dbg_log (DAPL_DBG_TYPE_EVD, 
-	          "dapl_evd_wait: EVD %p, CQ %p\n", 
-                  evd_ptr,
-		  (void *)evd_ptr->ib_cq_handle);
+	          "dapl_evd_wait: EVD %p, CQ %p, Timeout %d, Threshold %d\n", 
+              evd_ptr,(void *)evd_ptr->ib_cq_handle, time_out, threshold);
+  
 
     /*
      * Make sure there are no other waiters and the evd is active.
@@ -145,24 +146,12 @@
 					(DAT_COUNT) DAPL_EVD_STATE_OPEN,
 					(DAT_COUNT) DAPL_EVD_STATE_WAITED );
     dapl_os_unlock ( &evd_ptr->header.lock );
-
-    if ( evd_state != DAPL_EVD_STATE_OPEN )
+    if ( evd_state != DAPL_EVD_STATE_OPEN || !waitable)
     {
-	/* Bogus state, bail out */
 	dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
 	goto bail;
     }
 
-    if (!waitable)
-    {
-	/* This EVD is not waitable, reset the state and bail */
-	(void) dapl_os_atomic_assign ((DAPL_ATOMIC *)&evd_ptr->evd_state,
-					(DAT_COUNT) DAPL_EVD_STATE_WAITED,
-					evd_state);
-	dat_status = DAT_ERROR (DAT_INVALID_STATE, DAT_INVALID_STATE_EVD_UNWAITABLE);
-	goto bail;
-    }
-
     /*
      * We now own the EVD, even though we don't have the lock anymore,
      * because we're in the WAITED state.
@@ -182,37 +171,54 @@
 	 * return right away if the ib_cq_handle associate with these evd
 	 * equal to IB_INVALID_HANDLE
 	 */
-	dapls_evd_copy_cq(evd_ptr);
-
-	if (dapls_rbuf_count(&evd_ptr->pending_event_queue) >= threshold)
-	{
-	    break;
-	}
-
-	/*
-	 * Do not enable the completion notification if this evd is not 
-	 * a DTO_EVD or RMR_BIND_EVD
+	/* Logic to prevent missing completion between copy_cq (poll)
+	 * and completion_notify (re-arm)  
 	 */
-	if ( (!notify_requested) &&
-             ((evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
-              (evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG)) )
+	notify_needed = DAT_TRUE;
+	new_events = 0;
+	while (DAT_TRUE)
 	{
-	    dat_status = dapls_ib_completion_notify (
-		evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
-		evd_ptr,
-		(evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT) ?
-		     IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP );  
-
-	    DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
-	    /* FIXME report error */
-	    dapl_os_assert(dat_status == DAT_SUCCESS);
+		dapls_evd_copy_cq(evd_ptr); /* poll for new completions */
+		total_events = dapls_rbuf_count (&evd_ptr->pending_event_queue); 
+		new_events = total_events - new_events;
+		if (total_events >= threshold ||
+			(!new_events && notify_needed == DAT_FALSE))
+		{
+			break;
+		} 
+											
+		/*
+		 * Do not enable the completion notification if this evd is not 
+		 * a DTO_EVD or RMR_BIND_EVD
+		 */
+		if ( (evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
+			(evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG) )
+		{
+			dat_status = dapls_ib_completion_notify (
+					evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
+					evd_ptr,
+					(evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT)
?
+		     			IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP );  
+
+			DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
+			notify_needed = DAT_FALSE;
+			new_events = total_events;
+			
+			/* FIXME report error */
+			dapl_os_assert(dat_status == DAT_SUCCESS);
+		} 
+		else 
+		{
+			break;
+		}
 
-	    notify_requested = DAT_TRUE;
+	} /* while completions < threshold, and rearm needed */
 
-	    /* Try again.  */
-	    continue;
+	if (total_events >= threshold)
+	{
+		break;
 	}
-
+	
 
 	/*
 	 * Unused by poster; it has no way to tell how many
@@ -232,8 +238,6 @@
 #endif
 		dat_status = dapl_os_wait_object_wait (
 				&evd_ptr->wait_object, time_out );
-	
-	notify_requested = DAT_FALSE; /* We've used it up.  */
 
 	/* See if we were awakened by evd_set_unwaitable */
 	if ( !evd_ptr->evd_waitable )
@@ -243,13 +247,22 @@
 
 	if (dat_status != DAT_SUCCESS)
 	{
-	    /*
-	     * If the status is DAT_TIMEOUT, we'll break out of the
-	     * loop, *not* dequeue an event (because dat_status
-	     * != DAT_SUCCESS), set *nmore (as we should for timeout)
-	     * and return DAT_TIMEOUT.
-	     */
-	    break;
+		/*
+		 * If the status is DAT_TIMEOUT, we'll break out of the
+		 * loop, *not* dequeue an event (because dat_status
+		 * != DAT_SUCCESS), set *nmore (as we should for timeout)
+		 * and return DAT_TIMEOUT.
+		 */
+
+#if defined(DAPL_DBG)
+		dapls_evd_copy_cq(evd_ptr); /* poll */
+		dapl_dbg_log (DAPL_DBG_TYPE_EVD, 
+			"dapl_evd_wait: WAKEUP ERROR (0x%x): EVD %p, CQ %p, events? %d\n", 
+			dat_status,evd_ptr,(void *)evd_ptr->ib_cq_handle, 
+			dapls_rbuf_count(&evd_ptr->pending_event_queue) );
+#endif /* DAPL_DBG */
+
+		break;
 	}
     }
 	    
Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile	(revision 2899)
+++ dapl/udapl/Makefile	(working copy)
@@ -122,7 +122,8 @@
 #
 ifeq ($(VERBS),openib)
 PROVIDER = $(TOPDIR)/../openib
-CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
+CFLAGS   += -DOPENIB 
+#CFLAGS   += -DCQ_WAIT_OBJECT
 CFLAGS   += -I/usr/local/include/infiniband
 endif
 
@@ -232,7 +233,7 @@
 endif
 
 ifeq ($(VERBS),openib)
-LDFLAGS += -libverbs /usr/local/lib/infiniband/mthca.so -libcm
+LDFLAGS += -libverbs /usr/local/lib/infiniband/mthca.so -libcm -libat
 LDFLAGS += -rpath /usr/local/lib -L /usr/local/lib
 LDFLAGS += -rpath /usr/local/lib/infiniband -L /usr/local/lib/infiniband
 PROVIDER_SRCS  = dapl_ib_util.c dapl_ib_cq.c dapl_ib_qp.c 
Index: dapl/common/dapl_evd_resize.c
===================================================================
--- dapl/common/dapl_evd_resize.c	(revision 2899)
+++ dapl/common/dapl_evd_resize.c	(working copy)
@@ -67,72 +67,130 @@
 	IN	DAT_EVD_HANDLE	   evd_handle,
 	IN	DAT_COUNT	   evd_qlen )
 {
-    DAPL_IA		*ia_ptr;
-    DAPL_EVD		*evd_ptr;
-    DAT_COUNT   	pend_cnt;
-    DAT_RETURN		dat_status;
+    DAPL_IA          *ia_ptr;
+    DAPL_EVD         *evd_ptr;
+    DAT_EVENT        *event_ptr;
+    DAT_EVENT        *events;
+    DAT_EVENT        *orig_event;
+    DAPL_RING_BUFFER free_event_queue;
+    DAPL_RING_BUFFER pending_event_queue;
+    DAT_COUNT        pend_cnt;
+    DAT_COUNT        i;
 
     dapl_dbg_log (DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n",
 		  evd_handle, evd_qlen);
 
     if (DAPL_BAD_HANDLE (evd_handle, DAPL_MAGIC_EVD))
     {
-	dat_status = DAT_ERROR (DAT_INVALID_HANDLE,0);
-	goto bail;
+        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG1);
     }
 
     evd_ptr     = (DAPL_EVD *)evd_handle;
     ia_ptr      = evd_ptr->header.owner_ia;
 
-    if ( evd_qlen == evd_ptr->qlen )
+    if ((evd_qlen <= 0) || (evd_ptr->qlen > evd_qlen))
     {
-	 dat_status = DAT_SUCCESS;
-	 goto bail;
+        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
     }
 
     if ( evd_qlen > ia_ptr->hca_ptr->ia_attr.max_evd_qlen )
     {
-	dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
-	goto bail;
+	return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_TEVD);
     }
 
     dapl_os_lock(&evd_ptr->header.lock);
 
-    /* Don't try to resize if we are actively waiting */
     if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED)
     {
         dapl_os_unlock(&evd_ptr->header.lock);
-	dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
-	goto bail;
+        return DAT_ERROR (DAT_INVALID_STATE,0);
     }
 
     pend_cnt = dapls_rbuf_count(&evd_ptr->pending_event_queue);
     if (pend_cnt > evd_qlen) {
-	dapl_os_unlock(&evd_ptr->header.lock);
-	dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
-	goto bail;
+        dapl_os_unlock(&evd_ptr->header.lock);
+        return DAT_ERROR (DAT_INVALID_STATE,0);
+    }
+
+    if (DAT_SUCCESS != dapls_ib_cq_resize(evd_ptr->header.owner_ia,
+                                          evd_ptr,
+                                          &evd_qlen)) 
+    {
+        dapl_os_unlock(&evd_ptr->header.lock);
+        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
+    }
+
+    /* Allocate EVENTs */
+    events = (DAT_EVENT *) dapl_os_alloc (evd_qlen * sizeof (DAT_EVENT));
+    if (!events)
+    {
+        dapl_os_unlock(&evd_ptr->header.lock);
+        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
     }
+    event_ptr = events;
 
-    dat_status = dapls_ib_cq_resize(evd_ptr->header.owner_ia,
-				    evd_ptr,
-				    &evd_qlen);
-    if (dat_status != DAT_SUCCESS)
+    /* allocate free event queue */
+    if (DAT_SUCCESS != dapls_rbuf_alloc (&free_event_queue, evd_qlen))
     {
+        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
         dapl_os_unlock(&evd_ptr->header.lock);
-	goto bail;
+        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
     }
 
-    dat_status = dapls_evd_event_realloc (evd_ptr, evd_qlen);
-    if (dat_status != DAT_SUCCESS)
+    /* allocate pending event queue */
+    if (DAT_SUCCESS != dapls_rbuf_alloc (&pending_event_queue, evd_qlen))
     {
+        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
         dapl_os_unlock(&evd_ptr->header.lock);
-	goto bail;
+        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
     }
 
+    for (i = 0; i < pend_cnt; i++) 
+    {
+        orig_event = dapls_rbuf_remove(&evd_ptr->pending_event_queue);
+        if (orig_event == NULL) {
+            dapl_dbg_log (DAPL_DBG_TYPE_ERR, " Inconsistent event queue\n");
+            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
+            dapl_os_unlock(&evd_ptr->header.lock);
+	    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
+        }
+        memcpy(event_ptr, orig_event, sizeof(DAT_EVENT));
+        if (DAT_SUCCESS != dapls_rbuf_add(&pending_event_queue, event_ptr)) {
+            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
+            dapl_os_unlock(&evd_ptr->header.lock);
+	    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
+        }
+        event_ptr++;
+    }
+
+    for (i = pend_cnt; i < evd_qlen; i++)
+    {
+        if (DAT_SUCCESS != dapls_rbuf_add(&free_event_queue,
+                                          (void *) event_ptr)) {
+            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
+            dapl_os_unlock(&evd_ptr->header.lock);
+	    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
+        }
+        event_ptr++;
+    }
+
+    dapls_rbuf_destroy (&evd_ptr->free_event_queue);
+    dapls_rbuf_destroy (&evd_ptr->pending_event_queue);
+    if (evd_ptr->events)
+    {
+        dapl_os_free (evd_ptr->events, evd_ptr->qlen * sizeof (DAT_EVENT));
+    }
+    evd_ptr->free_event_queue    = free_event_queue;
+    evd_ptr->pending_event_queue = pending_event_queue;
+    evd_ptr->events              = events;
+    evd_ptr->qlen                = evd_qlen;
+
     dapl_os_unlock(&evd_ptr->header.lock);
 
- bail:
-    return dat_status;
+    dapl_dbg_log (DAPL_DBG_TYPE_RTN,
+			"dapl_evd_resize returns DAT_SUCCESS\n");
+
+    return DAT_SUCCESS;
 }
 
 /*
Index: dapl/openib/TODO
===================================================================
--- dapl/openib/TODO	(revision 2899)
+++ dapl/openib/TODO	(working copy)
@@ -1,7 +1,7 @@
 
 IB Verbs:
 - CQ resize?
-- query call to get current qp state 
+- query call to get current qp state, remote port number 
 - ibv_get_cq_event() needs timed event call and wakeup
 - query call to get device attributes
 - memory window support
@@ -9,8 +9,6 @@
 DAPL:
 - reinit EP needs a QP timewait completion notification
 - add cq_object wakeup, time based cq_object wait when verbs support arrives
-- update uDAPL code with real ATS support
-- etc, etc.
 
 Other:
 - Shared memory in udapl and kernel module to support?
Index: dapl/openib/dapl_ib_util.c
===================================================================
--- dapl/openib/dapl_ib_util.c	(revision 2899)
+++ dapl/openib/dapl_ib_util.c	(working copy)
@@ -111,27 +111,40 @@
 }
 
 
-/* just get IP address for hostname */
-int dapli_get_addr( char *addr, int addr_len)
+/* just get IP address, IPv4 only for now  */
+int dapli_get_hca_addr( struct dapl_hca *hca_ptr )
 {
-	struct sockaddr_in	*ipv4_addr = (struct sockaddr_in*)addr;
-	struct hostent		*h_ptr;
-	struct utsname		ourname;
-
-	if ( uname( &ourname ) < 0 ) 
-		return 1;
-
-	h_ptr = gethostbyname( ourname.nodename );
-	if ( h_ptr == NULL ) 
+	struct sockaddr_in	*ipv4_addr;
+	struct ib_at_completion	at_comp;	
+	struct dapl_at_record	at_rec;
+	int			status;
+	DAT_RETURN		dat_status;
+	
+	ipv4_addr = (struct sockaddr_in*)&hca_ptr->hca_address;
+	ipv4_addr->sin_family = AF_INET;
+	ipv4_addr->sin_addr.s_addr = 0;
+	
+	at_comp.fn = dapli_ip_comp_handler;
+	at_comp.context = &at_rec; 
+	at_rec.addr = &hca_ptr->hca_address;
+	at_rec.wait_object = &hca_ptr->ib_trans.wait_object;
+
+	/*  call with async_comp until the sync version works */
+	status = ib_at_ips_by_gid(&hca_ptr->ib_trans.gid, &ipv4_addr->sin_addr.s_addr, 1, 
+				  &at_comp, &at_rec.req_id);
+	
+	if (status < 0) 
 		return 1;
-
-	if ( h_ptr->h_addrtype == AF_INET ) {
-		ipv4_addr = (struct sockaddr_in*) addr;
-		ipv4_addr->sin_family = AF_INET;
-		dapl_os_memcpy( &ipv4_addr->sin_addr, h_ptr->h_addr_list[0], 4 );
-	} else 
+ 
+        if (status > 0)
+                dapli_ip_comp_handler(at_rec.req_id, (void*)ipv4_addr, status);
+	
+	/* wait for answer, 5 seconds max */
+	dat_status = dapl_os_wait_object_wait (&hca_ptr->ib_trans.wait_object,5000000);
+	 
+	if ((dat_status != DAT_SUCCESS ) || (!ipv4_addr->sin_addr.s_addr)) 
 		return 1;
-
+		
 	return 0;
 }
 
@@ -152,14 +165,17 @@
  */
 int32_t dapls_ib_init (void)
 {	
-	if (dapli_cm_thread_init())
-		return -1;
-	else
-		return 0;
+	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, " dapl_ib_init: \n" );
+	if (dapli_cm_thread_init() || dapli_at_thread_init()) 
+		return 1;
+
+	return 0;
 }
 
 int32_t dapls_ib_release (void)
 {
+	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, " dapl_ib_release: \n" );
+	dapli_at_thread_destroy();
 	dapli_cm_thread_destroy();
 	return 0;
 }
@@ -186,7 +202,6 @@
         IN   DAPL_HCA		*hca_ptr)
 {
 	struct dlist	*dev_list;
-	DAT_RETURN	dat_status = DAT_SUCCESS;
 
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
 		      " open_hca: %s - %p\n", hca_name, hca_ptr );
@@ -217,36 +232,46 @@
 			      ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
 		return DAT_INTERNAL_ERROR;
 	}
-  
+
 	/* set inline max with enviromment or default, get local lid and gid 0 */
 	hca_ptr->ib_trans.max_inline_send = 
 		dapl_os_get_env_val ( "DAPL_MAX_INLINE", INLINE_SEND_DEFAULT );
 
-	if ( dapli_get_lid(hca_ptr, hca_ptr->port_num,
-			   &hca_ptr->ib_trans.lid )) {
+	if (dapli_get_lid(hca_ptr, hca_ptr->port_num,
+			   &hca_ptr->ib_trans.lid)) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
 			      " open_hca: IB get LID failed for %s\n", 
 			      ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
-		return DAT_INTERNAL_ERROR;
+		goto bail;
 	}
 
-	if ( dapli_get_gid(hca_ptr, hca_ptr->port_num, 0,  
-			   &hca_ptr->ib_trans.gid )) {
+	if (dapli_get_gid(hca_ptr, hca_ptr->port_num, 0,  
+			   &hca_ptr->ib_trans.gid)) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
 			      " open_hca: IB get GID failed for %s\n", 
 			      ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
-		return DAT_INTERNAL_ERROR;
+		goto bail;
 	}
-
 	/* get the IP address of the device */
-	if ( dapli_get_addr((char*)&hca_ptr->hca_address, 
-			    sizeof(DAT_SOCK_ADDR6) )) {
+	if (dapli_get_hca_addr(hca_ptr)) {
 		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
 			      " open_hca: IB get ADDR failed for %s\n", 
 			      ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
-		return DAT_INTERNAL_ERROR;
+		goto bail;
+	}
+
+	/* one thread for each device open */
+	if (dapli_cq_thread_init(hca_ptr)) {
+		dapl_dbg_log (DAPL_DBG_TYPE_ERR, 
+			      " open_hca: cq_thread_init failed for %s\n", 
+			      ibv_get_device_name(hca_ptr->ib_trans.ib_dev) );
+		goto bail;
 	}
 
+	/* initialize cq_lock and wait object */
+	dapl_os_lock_init(&hca_ptr->ib_trans.cq_lock);
+	dapl_os_wait_object_init (&hca_ptr->ib_trans.wait_object);
+  
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
 		      " open_hca: %s, port %d, %s  %d.%d.%d.%d INLINE_MAX=%d\n", 
 		      ibv_get_device_name(hca_ptr->ib_trans.ib_dev), hca_ptr->port_num,
@@ -257,7 +282,19 @@
 		      ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff,
 		      hca_ptr->ib_trans.max_inline_send );
 
-	return dat_status;
+	dapl_dbg_log(DAPL_DBG_TYPE_CM,
+		     " open_hca: LID 0x%x GID subnet %016llx id %016llx\n",
+		     hca_ptr->ib_trans.lid,
+		     (unsigned long long)bswap_64(hca_ptr->ib_trans.gid.global.subnet_prefix),
+		     (unsigned long long)bswap_64(hca_ptr->ib_trans.gid.global.interface_id) );
+
+	return DAT_SUCCESS;
+
+bail:
+	ibv_close_device(hca_ptr->ib_hca_handle); 
+	hca_ptr->ib_hca_handle = IB_INVALID_HANDLE;
+	return DAT_INTERNAL_ERROR;
+
 }
 
 
@@ -282,10 +319,14 @@
 	dapl_dbg_log (DAPL_DBG_TYPE_UTIL," close_hca: %p\n",hca_ptr);
 
 	if (hca_ptr->ib_hca_handle != IB_INVALID_HANDLE) {
+		dapli_cq_thread_destroy(hca_ptr);
 		if (ibv_close_device(hca_ptr->ib_hca_handle)) 
 			return(dapl_convert_errno(errno,"ib_close_device"));
 		hca_ptr->ib_hca_handle = IB_INVALID_HANDLE;
 	}
+	
+	dapl_os_lock_destroy(&hca_ptr->ib_trans.cq_lock);
+
 	return (DAT_SUCCESS);
 }
   
@@ -448,35 +489,4 @@
     return DAT_SUCCESS;
 }
 
-#ifdef PROVIDER_SPECIFIC_ATTR
-
-/*
- * dapls_set_provider_specific_attr
- *
- * Input:
- *	attr_ptr		Pointer provider attributes
- *
- * Output:
- * 	none
- *
- * Returns:
- * 	void
- */
-DAT_NAMED_ATTR	ib_attrs[] = {
-    {
-    	"I_DAT_SEND_INLINE_THRESHOLD",
-    	"128"
-    },
-};
-
-#define SPEC_ATTR_SIZE( x )	(sizeof( x ) / sizeof( DAT_NAMED_ATTR))
-
-void dapls_set_provider_specific_attr(
-	IN DAT_PROVIDER_ATTR	*attr_ptr )
-{
-	attr_ptr->num_provider_specific_attr = SPEC_ATTR_SIZE( ib_attrs );
-	attr_ptr->provider_specific_attr     = ib_attrs;
-}
-
-#endif
 
Index: dapl/openib/dapl_ib_cm.c
===================================================================
--- dapl/openib/dapl_ib_cm.c	(revision 2899)
+++ dapl/openib/dapl_ib_cm.c	(working copy)
@@ -70,19 +70,8 @@
 static inline uint64_t cpu_to_be64(uint64_t x) { return x; }
 #endif
 
-#ifndef IB_AT
-
-#include <stdio.h>
-#include <unistd.h>
-#include <fcntl.h>
-#include <netinet/tcp.h>
-#include <sysfs/libsysfs.h>
-#include <signal.h>
-
-/* iclust-20 hard coded values, network order */
-#define REMOTE_GID	"fe80:0000:0000:0000:0002:c902:0000:4071"
-#define REMOTE_LID	"0002"
-
+static int			g_at_destroy;
+static DAPL_OS_THREAD		g_at_thread;
 static int			g_cm_destroy;
 static DAPL_OS_THREAD		g_cm_thread;
 static DAPL_OS_LOCK		g_cm_lock;
@@ -122,7 +111,7 @@
 	while (g_cm_destroy) {
 		struct timespec	sleep, remain;
 		sleep.tv_sec = 0;
-		sleep.tv_nsec = 200000000; /* 200 ms */
+		sleep.tv_nsec = 10000000; /* 10 ms */
 		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
 			     " cm_thread_destroy: waiting for cm_thread\n");
 		nanosleep (&sleep, &remain);
@@ -130,112 +119,70 @@
 	dapl_dbg_log(DAPL_DBG_TYPE_CM," cm_thread_destroy(%d) exit\n",getpid());
 }
 
-static int ib_at_route_by_ip(uint32_t dst_ip, uint32_t src_ip, int tos, uint16_t flags,
-			     struct ib_at_ib_route *ib_route,
-			     struct ib_at_completion *async_comp)
-{
-	struct dapl_cm_id *conn = (struct dapl_cm_id *)async_comp->context;
-	
-	dapl_dbg_log (
-		DAPL_DBG_TYPE_CM, 
-		" CM at_route_by_ip: conn %p cm_id %d src %d.%d.%d.%d -> dst %d.%d.%d.%d (%d)\n", 
-		conn,conn->cm_id,
-		src_ip >> 0 & 0xff, src_ip >> 8 & 0xff,
-		src_ip >> 16 & 0xff,src_ip >> 24 & 0xff,
-		dst_ip >> 0 & 0xff, dst_ip >> 8 & 0xff,
-		dst_ip >> 16 & 0xff,dst_ip >> 24 & 0xff, conn->service_id);
-
-	/* use req_id for loopback indication */
-	if (( src_ip == dst_ip ) || ( dst_ip == 0x0100007f ))
-		async_comp->req_id = 1;
-	else
-		async_comp->req_id = 0;
-		
-	return 1;			     
-}
-
-static int ib_at_paths_by_route(struct ib_at_ib_route *ib_route, uint32_t mpath_type,
-				struct ib_sa_path_rec *pr, int npath,
-				struct ib_at_completion *async_comp)
+int dapli_at_thread_init(void)
 {
-	struct dapl_cm_id *conn = (struct dapl_cm_id *)async_comp->context;
-	char *env, *token;
-	char  dgid[40];
-	uint16_t *p_gid = (uint16_t*)&ib_route->gid;
+	DAT_RETURN dat_status;
 
-	/* set local path record values and send to remote */
-	(void)dapl_os_memzero(pr, sizeof(*pr));
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," at_thread_init(%d)\n", getpid());
 
-	pr->slid = htons(conn->hca->ib_trans.lid);
-	pr->sgid.global.subnet_prefix = conn->hca->ib_trans.gid.global.subnet_prefix;
-	pr->sgid.global.interface_id  = conn->hca->ib_trans.gid.global.interface_id;
+	/* create thread to process AT async requests */
+	dat_status = dapl_os_thread_create(at_thread, NULL, &g_at_thread);
+	if (dat_status != DAT_SUCCESS)
+	{
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR, 
+			     " at_thread_init: failed to create thread\n");
+		return 1;
+	}
+	return 0;
+}
 
-	env = getenv("DAPL_REMOTE_LID");
-	if ( env == NULL )
-		env = REMOTE_LID;
-	ib_route->lid = strtol(env,NULL,0);
+void dapli_at_thread_destroy(void)
+{
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," at_thread_destroy(%d)\n", getpid());
 
-	env = getenv("DAPL_REMOTE_GID");
-	if ( env == NULL )
-		env = REMOTE_GID;
+	/* destroy cr_thread and lock */
+	g_at_destroy = 1;
+	pthread_kill( g_at_thread, SIGUSR1 );
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," at_thread_destroy(%d) SIGUSR1 sent\n",getpid());
+	while (g_at_destroy) {
+		struct timespec	sleep, remain;
+		sleep.tv_sec = 0;
+		sleep.tv_nsec = 10000000; /* 10 ms */
+		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+			     " at_thread_destroy: waiting for at_thread\n");
+		nanosleep (&sleep, &remain);
+	}
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," at_thread_destroy(%d) exit\n",getpid());
+}
 
-	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
-		     " ib_at_paths_by_route: remote LID %x GID %s\n",
-		     ib_route->lid,env);
+void dapli_ip_comp_handler(uint64_t req_id, void *context, int rec_num)
+{
+	struct dapl_at_record	*at_rec = context;
 
-	dapl_os_memcpy( dgid, env, 40 ); 
+	dapl_dbg_log(DAPL_DBG_TYPE_CM,
+		     " ip_comp_handler: ctxt %p, req_id %lld rec_num %d\n",
+		     context, req_id, rec_num);
 
-	/* get GID with token strings and delimiter */
-	token = strtok(dgid,":");
-	while (token) {
-		*p_gid = strtoul(token,NULL,16);
-		*p_gid = htons(*p_gid); /* convert each token to network order */
-		token = strtok(NULL,":");
-		p_gid++;
-	}
-       
-	/* set remote lid and gid, req_id is indication of loopback */
-	if ( !async_comp->req_id ) {
-		pr->dlid = htons(ib_route->lid);
-		pr->dgid.global.subnet_prefix = ib_route->gid.global.subnet_prefix;
-		pr->dgid.global.interface_id  = ib_route->gid.global.interface_id;
-	} else {
-		pr->dlid = pr->slid;
-		pr->dgid.global.subnet_prefix = pr->sgid.global.subnet_prefix;
-		pr->dgid.global.interface_id  = pr->sgid.global.interface_id;
-	}
-
-	pr->reversible = 0x1000000;
-	pr->pkey = 0xffff;
-	pr->mtu  = IBV_MTU_1024;
-	pr->mtu_selector  = 2;
-	pr->rate_selector = 2;
-	pr->rate          = 3;
-	pr->packet_life_time_selector = 2;
-	pr->packet_life_time          = 2;
+	if ((at_rec) && ( at_rec->req_id == req_id)) {
+		dapl_os_wait_object_wakeup(at_rec->wait_object);
+		return;
+	}
 	
-	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
-		     " ib_at_paths_by_route: SRC LID 0x%x GID subnet %016llx id %016llx\n",
-		     pr->slid,(unsigned long long)(pr->sgid.global.subnet_prefix),
-		     (unsigned long long)(pr->sgid.global.interface_id) );
-	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
-		     " ib_at_paths_by_route: DST LID 0x%x GID subnet %016llx id %016llx\n",
-		     pr->dlid,(unsigned long long)(pr->dgid.global.subnet_prefix),
-		     (unsigned long long)(pr->dgid.global.interface_id) );
-
-	dapli_path_comp_handler( async_comp->req_id, (void*)conn, 1);
-
-	return 0;
+	dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+		     " ip_comp_handler: at_rec->req_id %lld != req_id %lld\n",
+		     at_rec->req_id, req_id );
 }
 
-#endif /* ifndef IB_AT */
-
 static void dapli_path_comp_handler(uint64_t req_id, void *context, int rec_num)
 {
 	struct dapl_cm_id *conn = context;
 	int status;
 	ib_cm_events_t event;
 
+	dapl_dbg_log(DAPL_DBG_TYPE_CM,
+		     " path_comp_handler: ctxt %p, req_id %lld rec_num %d\n",
+		     context, req_id, rec_num);
+
 	if (rec_num <= 0) {
 		dapl_dbg_log(DAPL_DBG_TYPE_CM, 
 			     " path_comp_handler: resolution err %d retry %d\n",
@@ -249,7 +196,7 @@
 
 		status = ib_at_paths_by_route(&conn->dapl_rt, 0,
 					      &conn->dapl_path, 1,
-					      &conn->dapl_comp);
+					      &conn->dapl_comp, &conn->dapl_comp.req_id);
 		if (status) {
 			dapl_dbg_log(DAPL_DBG_TYPE_CM,
 				     " path_by_route: err %d id %lld\n",
@@ -287,6 +234,21 @@
 	int status;
 	ib_cm_events_t event;
 
+	dapl_dbg_log(DAPL_DBG_TYPE_CM,
+		     " rt_comp_handler: conn %p, req_id %lld rec_num %d\n",
+		     conn, req_id, rec_num);
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " rt_comp_handler: SRC GID subnet %016llx id %016llx\n",
+		     (unsigned long long)cpu_to_be64(conn->dapl_rt.sgid.global.subnet_prefix),
+		     (unsigned long long)cpu_to_be64(conn->dapl_rt.sgid.global.interface_id) );
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, 
+		     " rt_comp_handler: DST GID subnet %016llx id %016llx\n",
+		     (unsigned long long)cpu_to_be64(conn->dapl_rt.dgid.global.subnet_prefix),
+		     (unsigned long long)cpu_to_be64(conn->dapl_rt.dgid.global.interface_id) );
+
+
 	if (rec_num <= 0) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
 			     " dapl_rt_comp_handler: rec %d retry %d\n",
@@ -298,7 +260,8 @@
 		}
 
 		status = ib_at_route_by_ip(((struct sockaddr_in *)&conn->r_addr)->sin_addr.s_addr,
-					   0, 0, 0, &conn->dapl_rt, &conn->dapl_comp);
+					   0, 0, 0, &conn->dapl_rt, 
+					   &conn->dapl_comp,&conn->dapl_comp.req_id);
 		if (status < 0) {
 			dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_rt_comp_handler: "
 				    "ib_at_route_by_ip failed with status %d\n",
@@ -306,9 +269,16 @@
 			event = IB_CME_DESTINATION_UNREACHABLE;
 			goto bail;
 		}
-
 		if (status == 1)
 			dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, 1);
+
+		return;
+	}
+
+	if (!conn->dapl_rt.dgid.global.subnet_prefix || req_id != conn->dapl_comp.req_id) {
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			     " dapl_rt_comp_handler: ERROR: unexpected callback req_id=%d(%d)\n",
+			     req_id, conn->dapl_comp.req_id ); 
 		return;
 	}
 
@@ -316,7 +286,7 @@
 	conn->dapl_comp.context = conn;
 	conn->retries = 0;
 	status = ib_at_paths_by_route(&conn->dapl_rt, 0, &conn->dapl_path, 1,
-				      &conn->dapl_comp);
+				      &conn->dapl_comp, &conn->dapl_comp.req_id);
 	if (status) {
 		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
 			     "dapl_rt_comp_handler: ib_at_paths_by_route "
@@ -346,8 +316,6 @@
 		ib_cm_destroy_id(conn->cm_id);
 		if (conn->ep)
 			conn->ep->cm_handle = IB_INVALID_HANDLE;
-		if (conn->sp)
-			conn->sp->cm_srvc_handle = IB_INVALID_HANDLE;
 
 		/* take off the CM thread work queue and free */
 		dapl_os_lock( &g_cm_lock );
@@ -621,10 +589,8 @@
 }
 
 /* something to catch the signal */
-static void cm_handler(int signum)
+static void ib_sig_handler(int signum)
 {
-	dapl_dbg_log (DAPL_DBG_TYPE_CM," cm_thread(%d,0x%x): ENTER cm_handler %d\n",
-			getpid(),g_cm_thread,signum);
 	return;
 }
 
@@ -643,7 +609,7 @@
     	sigemptyset(&sigset);
 	sigaddset(&sigset, SIGUSR1);
     	pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
-	signal( SIGUSR1, cm_handler); 
+	signal( SIGUSR1, ib_sig_handler); 
 	
 	dapl_os_lock( &g_cm_lock );
 	while (!g_cm_destroy) {
@@ -667,7 +633,7 @@
 		dapl_dbg_log(DAPL_DBG_TYPE_CM,
 			" cm_thread: GET EVENT fd=%d n=%d\n",
 			ib_cm_get_fd(),ret);
-		if (ib_cm_event_get(&event)) { 
+		if (ib_cm_event_get_timed(0,&event)) { 
 			dapl_dbg_log(DAPL_DBG_TYPE_CM,
 				" cm_thread: ERR %s eventi_get on %d\n", 
 				strerror(errno), ib_cm_get_fd() );
@@ -732,6 +698,33 @@
 	g_cm_destroy = 0;
 }
 
+/* async AT processing thread */
+void at_thread(void *arg) 
+{
+	sigset_t sigset;
+
+	dapl_dbg_log (DAPL_DBG_TYPE_CM,
+		      " at_thread(%d,0x%x): ENTER: at_fd %d\n",
+		      getpid(), g_at_thread, ib_at_get_fd());
+
+    	sigemptyset(&sigset);
+	sigaddset(&sigset, SIGUSR1);
+    	pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
+	signal(SIGUSR1, ib_sig_handler); 
+	
+	while (!g_at_destroy) {
+		/* poll forever until callback or signal */
+		if (ib_at_callback_get_timed(-1) < 0) { 
+			dapl_dbg_log(DAPL_DBG_TYPE_CM,
+				" at_thread: SIG? ret=%s, destroy=%d\n", 
+				strerror(errno), g_at_destroy );
+		}
+		dapl_dbg_log(DAPL_DBG_TYPE_CM," at_thread: callback woke\n");
+	}
+	dapl_dbg_log(DAPL_DBG_TYPE_CM," at_thread(%d) EXIT \n", getpid());
+	g_at_destroy = 0;
+}
+
 /************************ DAPL provider entry points **********************/
 
 /*
@@ -826,33 +819,34 @@
 	conn->dapl_comp.context = conn;
 	conn->retries = 0;
 	dapl_os_memcpy(&conn->r_addr, r_addr, sizeof(DAT_SOCK_ADDR6));
+	
+	/* put on CM thread work queue */
+	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
+	dapl_os_lock( &g_cm_lock );
+	dapl_llist_add_tail(&g_cm_list, 
+			    (DAPL_LLIST_ENTRY*)&conn->entry, conn);
+	dapl_os_unlock(&g_cm_lock);
 
 	status = ib_at_route_by_ip(
 		((struct sockaddr_in *)&conn->r_addr)->sin_addr.s_addr, 
 		((struct sockaddr_in *)&conn->hca->hca_address)->sin_addr.s_addr, 
-		0, 0, &conn->dapl_rt, &conn->dapl_comp);
+		0, 0, &conn->dapl_rt, &conn->dapl_comp, &conn->dapl_comp.req_id);
+
+	dapl_dbg_log(DAPL_DBG_TYPE_CM, " connect: at_route ret=%d,%s req_id %d GID %016llx
%016llx\n", 
+		     status, strerror(errno), conn->dapl_comp.req_id,
+		     (unsigned long long)cpu_to_be64(conn->dapl_rt.dgid.global.subnet_prefix),
+		     (unsigned long long)cpu_to_be64(conn->dapl_rt.dgid.global.interface_id) );
 
 	if (status < 0) {
 		dat_status = dapl_convert_errno(errno,"ib_at_route_by_ip");
-		goto destroy;
+		dapli_destroy_cm_id(conn); 
+		return dat_status;
 	}
-	if (status == 1)
-		dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, 1);
 
-	
-	/* put on CM thread work queue */
-	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
-	dapl_os_lock( &g_cm_lock );
-	dapl_llist_add_tail(&g_cm_list, 
-			    (DAPL_LLIST_ENTRY*)&conn->entry, conn);
-	dapl_os_unlock(&g_cm_lock);
+	if (status > 0) 
+		dapli_rt_comp_handler(conn->dapl_comp.req_id, conn, status);
 
 	return DAT_SUCCESS;
-
-destroy:
-	dapli_destroy_cm_id(conn); 
-	return dat_status;
-
 }
 
 /*
@@ -992,6 +986,13 @@
 	conn->hca = ia_ptr->hca_ptr;
 	conn->service_id = ServiceID;
 
+	/* put on CM thread work queue */
+	dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
+	dapl_os_lock( &g_cm_lock );
+	dapl_llist_add_tail(&g_cm_list, 
+			(DAPL_LLIST_ENTRY*)&conn->entry, conn);
+	dapl_os_unlock(&g_cm_lock);
+
 	dapl_dbg_log(DAPL_DBG_TYPE_EP,
 		     " setup_listener(conn=%p cm_id=%d)\n",
 		     sp_ptr->cm_srvc_handle,conn->cm_id);
@@ -1003,19 +1004,13 @@
 			dat_status = DAT_CONN_QUAL_IN_USE;
 		else
 			dat_status = DAT_INSUFFICIENT_RESOURCES;
-	/* success */ 
-	} else  { 
-		/* put on CM thread work queue */
-		dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&conn->entry);
-		dapl_os_lock( &g_cm_lock );
-		dapl_llist_add_tail(&g_cm_list, 
-				(DAPL_LLIST_ENTRY*)&conn->entry, conn);
-		dapl_os_unlock(&g_cm_lock);
+
+        	dapli_destroy_cm_id(conn);
 		return dat_status;
 	}
 
-        dapli_destroy_cm_id(conn);
-	return dat_status;
+	/* success */ 
+	return DAT_SUCCESS;
 }
 
 
@@ -1047,9 +1042,11 @@
 			" remove_listener(ia_ptr %p sp_ptr %p cm_ptr %p)\n",
 			ia_ptr, sp_ptr, conn );
 	
-	if (sp_ptr->cm_srvc_handle != IB_INVALID_HANDLE) 
+	if (conn != IB_INVALID_HANDLE) { 
+		sp_ptr->cm_srvc_handle = NULL;
         	dapli_destroy_cm_id(conn);
-	
+	}	
+
 	return DAT_SUCCESS;
 }
 
Index: dapl/openib/dapl_ib_util.h
===================================================================
--- dapl/openib/dapl_ib_util.h	(revision 2899)
+++ dapl/openib/dapl_ib_util.h	(working copy)
@@ -53,6 +53,7 @@
 #include <byteswap.h>
 #include <infiniband/sa.h>
 #include <infiniband/cm.h>
+#include <infiniband/at.h>
 
 /* Typedefs to map common DAPL provider types to IB verbs */
 typedef	struct ibv_qp		*ib_qp_handle_t;
@@ -68,8 +69,8 @@
 
 #define IB_RC_RETRY_COUNT      7
 #define IB_RNR_RETRY_COUNT     7
-#define IB_CM_RESPONSE_TIMEOUT 20	/* 4 sec */
-#define IB_MAX_CM_RETRIES      4
+#define IB_CM_RESPONSE_TIMEOUT 18	/* 1 sec */
+#define IB_MAX_CM_RETRIES      7
 
 #define IB_REQ_MRA_TIMEOUT	27	/* a little over 9 minutes */
 #define IB_MAX_AT_RETRY		3
@@ -92,21 +93,12 @@
 	IB_CME_BROKEN
 } ib_cm_events_t;
 
-#ifndef IB_AT
-/* implement a quick hack to exchange GID/LID's until user IB_AT arrives */
-struct ib_at_ib_route {
-        union ibv_gid   gid;
-        uint16_t        lid;
+struct dapl_at_record {
+	uint64_t                req_id;
+	DAT_SOCK_ADDR6		*addr;
+	DAPL_OS_WAIT_OBJECT	*wait_object;
 };
 
-struct ib_at_completion {
-        void (*fn)(uint64_t req_id, void *context, int rec_num);
-        void *context;
-        uint64_t req_id;
-};
-
-#endif
-
 /* 
  * dapl_llist_entry in dapl.h but dapl.h depends on provider 
  * typedef's in this file first. move dapl_llist_entry out of dapl.h
@@ -122,6 +114,7 @@
 struct dapl_cm_id {
 	struct ib_llist_entry		entry;
 	DAPL_OS_LOCK			lock;
+	DAPL_OS_WAIT_OBJECT		wait_object;
 	int				retries;
 	int				destroy;
 	int				in_callback;
@@ -238,6 +231,10 @@
 { 
 	struct	ibv_device	*ib_dev;
 	ib_cq_handle_t		ib_cq_empty;
+	DAPL_OS_LOCK            cq_lock;
+	DAPL_OS_WAIT_OBJECT     wait_object;
+	int			cq_destroy;
+	DAPL_OS_THREAD		cq_thread;
 	int			max_inline_send;
 	uint16_t		lid;
 	union ibv_gid		gid;
@@ -257,11 +254,18 @@
 void cm_thread (void *arg);
 int dapli_cm_thread_init(void);
 void dapli_cm_thread_destroy(void);
+void at_thread (void *arg);
+int dapli_at_thread_init(void);
+void dapli_at_thread_destroy(void);
+void cq_thread (void *arg);
+int dapli_cq_thread_init(struct dapl_hca *hca_ptr);
+void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr);
 
-int dapli_get_lid(struct dapl_hca *hca_ptr, int port, uint16_t *lid );
+int dapli_get_lid(struct dapl_hca *hca_ptr, int port, uint16_t *lid);
 int dapli_get_gid(struct dapl_hca *hca_ptr, int port, int index, 
-		  union ibv_gid *gid );
-int dapli_get_addr(char *addr, int addr_len);
+		  union ibv_gid *gid);
+int dapli_get_hca_addr(struct dapl_hca *hca_ptr);
+void dapli_ip_comp_handler(uint64_t req_id, void *context, int rec_num);
 
 DAT_RETURN
 dapls_modify_qp_state ( IN ib_qp_handle_t	qp_handle,
Index: dapl/openib/README
===================================================================
--- dapl/openib/README	(revision 2899)
+++ dapl/openib/README	(working copy)
@@ -39,18 +39,33 @@
 
 	server:	dtest -s 
 	client:	dtest -h hostname
+
+Testing: dtest, dapltest - cl.sh regress.sh
 	
-setup/known issues:
-	
-	First drop with uCM (without IBAT), tested with simple dtest across 2 nodes. 
-	hand rolled path records require remote LID and GID set via enviroment:
+Setup:
 	
-	export DAPL_REMOTE_GID	"fe80:0000:0000:0000:0002:c902:0000:4071"
-	export DAPL_REMOTE_LID	"0002"
+	Third drop of code, includes uCM and uAT support.
+	NOTE: uAT user library and kernel code in separate branch.
 
-    Also, hard coded (RTR) for use with port 1 only.
-	   
-	no memory windows support in ibverbs, dat_create_rmr fails.
-	
+	build uAT library from following branch:
+
+		cd gen2/branches/shaharf-ibat/src/userspace/libibat/
+		./autogen.sh &&./configure && make && make install
+
+	copy following uat source to latest trunk kernel src:
 
+		gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core
+			at.c  at_priv.h  att.c uat.c uat.h Makefile
 
+		gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/include
+			ib_at.h ib_user_at.h
+
+	add udev rule:
+		KERNEL="uat*", NAME="infiniband/%k"
+
+	
+Known issues:
+	no memory windows support in ibverbs, dat_create_rmr fails.
+	some uCM scale up issues with an 8 thread dapltest in regress.sh
+	hard coded modify QP RTR to port 1, waiting for ib_cm_init_qp_attr call.
+	
Index: dapl/openib/dapl_ib_cq.c
===================================================================
--- dapl/openib/dapl_ib_cq.c	(revision 2899)
+++ dapl/openib/dapl_ib_cq.c	(working copy)
@@ -50,9 +50,96 @@
 #include "dapl_adapter_util.h"
 #include "dapl_lmr_util.h"
 #include "dapl_evd_util.h"
+#include "dapl_ring_buffer_util.h"
 #include <sys/poll.h>
+#include <signal.h>
 
+int dapli_cq_thread_init(struct dapl_hca *hca_ptr)
+{
+        DAT_RETURN dat_status;
+
+        dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cq_thread_init(%p)\n", hca_ptr);
+
+        /* create thread to process inbound connect request */
+        dat_status = dapl_os_thread_create( cq_thread, (void*)hca_ptr,
&hca_ptr->ib_trans.cq_thread);
+        if (dat_status != DAT_SUCCESS)
+        {
+                dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+                             " cq_thread_init: failed to create thread\n");
+                return 1;
+        }
+        return 0;
+}
+
+void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr)
+{
+        dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cq_thread_destroy(%p)\n", hca_ptr);
+
+        /* destroy cr_thread and lock */
+        hca_ptr->ib_trans.cq_destroy = 1;
+        pthread_kill(hca_ptr->ib_trans.cq_thread, SIGUSR1);
+        dapl_dbg_log(DAPL_DBG_TYPE_CM," cq_thread_destroy(%p) SIGUSR1 sent\n",hca_ptr);
+        while (hca_ptr->ib_trans.cq_destroy != 2) {
+                struct timespec sleep, remain;
+                sleep.tv_sec = 0;
+                sleep.tv_nsec = 10000000; /* 10 ms */
+                dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
+                             " cq_thread_destroy: waiting for cq_thread\n");
+                nanosleep (&sleep, &remain);
+        }
+        dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cq_thread_destroy(%d) exit\n",getpid());
+	return;
+}
+
+/* something to catch the signal */
+static void ib_cq_handler(int signum)
+{
+        return;
+}
+
+void cq_thread( void *arg )
+{
+	struct dapl_hca	*hca_ptr = arg;
+	struct dapl_evd	*evd_ptr;
+	struct ibv_cq	*ibv_cq = NULL;
+	sigset_t	sigset;
+	int		status = 0; 
+
+	dapl_dbg_log ( DAPL_DBG_TYPE_UTIL," cq_thread: ENTER hca %p\n",hca_ptr);
+  
+        sigemptyset(&sigset);
+        sigaddset(&sigset,SIGUSR1);
+        pthread_sigmask(SIG_UNBLOCK, &sigset, NULL);
+        signal(SIGUSR1, ib_cq_handler);
+
+	/* wait on DTO event, or signal to abort */
+	while (!hca_ptr->ib_trans.cq_destroy) {
+
+		struct pollfd cq_poll = {
+			.fd      = hca_ptr->ib_hca_handle->cq_fd[0],
+			.events  = POLLIN,
+			.revents = 0
+		};
 
+		status = poll(&cq_poll, 1, -1);
+		if ((status == 1) &&
+			(!ibv_get_cq_event(hca_ptr->ib_hca_handle, 0, &ibv_cq, (void*)&evd_ptr))) {
+	
+			if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD))
+				continue;
+
+			/* process DTO event via callback */
+			dapl_evd_dto_callback ( evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
+						evd_ptr->ib_cq_handle,
+						(void*)evd_ptr );
+		} else {
+
+		}
+	} 
+	hca_ptr->ib_trans.cq_destroy = 2;
+	dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cq_thread: EXIT: hca %p \n", hca_ptr);
+	return;
+}
 /*
  * Map all verbs DTO completion codes to the DAT equivelent.
  *
@@ -410,9 +497,9 @@
 		IN DAPL_EVD		*evd_ptr,
 		IN ib_wait_obj_handle_t	*p_cq_wait_obj_handle )
 {
-	dapl_dbg_log (	DAPL_DBG_TYPE_UTIL, 
+	dapl_dbg_log (	DAPL_DBG_TYPE_CM, 
 			" cq_object_create: (%p)=%p\n", 
-			p_cq_wait_obj_handle, *p_cq_wait_obj_handle);
+			p_cq_wait_obj_handle, evd_ptr );
 
 	/* set cq_wait object to evd_ptr */
 	*p_cq_wait_obj_handle = evd_ptr;
@@ -447,33 +534,86 @@
 {
 	DAPL_EVD		*evd_ptr = p_cq_wait_obj_handle;
 	ib_cq_handle_t		cq = evd_ptr->ib_cq_handle;
-	struct ibv_cq		*ibv_cq;
-	void			*ibv_ctx;
-	int			status = -ETIMEDOUT; 
+	struct ibv_cq		*ibv_cq = NULL;
+	void			*ibv_ctx = NULL;
+	int			status = 0; 
 
-	dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, 
+	dapl_dbg_log ( DAPL_DBG_TYPE_CM, 
 			" cq_object_wait: dev %p evd %p cq %p, time %d\n", 
 			cq->context, evd_ptr, cq, timeout );
 
-	/* Multiple EVD's sharing one event handle for now */
-	if (cq) {
-		struct pollfd cq_poll = { 
-        		.fd      = cq->context->cq_fd[0],
-			.events  = POLLIN
+	/* Multiple EVD's sharing one event handle for now until uverbs supports more */
+
+	/*
+	 *  This makes it very inefficient and tricky to manage multiple CQ per device open
+	 *  For example: 4 threads waiting on separate CQ events will all be woke when
+	 *  a CQ event fires. So the poll wakes up and the first thread to get to the
+	 *  the get_cq_event wins and the other 3 will block. The dapl_evd_wait code
+	 *  above will immediately do a poll_cq after returning from CQ wait and if
+	 *  nothing on the queue will call this wait again and go back to sleep. So
+	 *  as long as they all wake up, a mutex is held around the get_cq_event
+	 *  so no blocking occurs and they all return then everything should work.
+	 *  Of course, the timeout needs adjusted on the threads that go back to sleep.
+	 */
+	while (cq) {
+		struct pollfd cq_poll = {
+			.fd      = cq->context->cq_fd[0],
+			.events  = POLLIN,
+			.revents = 0
 		};
-		int	timeout_ms = -1;
+		int     timeout_ms = -1;
 
 		if (timeout != DAT_TIMEOUT_INFINITE)
 			timeout_ms = timeout/1000;
-		
+
+		/* check if another thread processed the event already, pending queue > 0 */
+		dapl_os_lock( &evd_ptr->header.owner_ia->hca_ptr->ib_trans.cq_lock );
+		if (dapls_rbuf_count(&evd_ptr->pending_event_queue)) {
+			dapl_os_unlock( &evd_ptr->header.owner_ia->hca_ptr->ib_trans.cq_lock );
+			break;	
+		}
+		dapl_os_unlock( &evd_ptr->header.owner_ia->hca_ptr->ib_trans.cq_lock );
+
+		dapl_dbg_log ( DAPL_DBG_TYPE_CM," cq_object_wait: polling\n");
 		status = poll(&cq_poll, 1, timeout_ms);
-		if (status == 1)
-			status = ibv_get_cq_event(cq->context, 
-						  0, &ibv_cq, &ibv_ctx);
-	}
-	dapl_dbg_log (DAPL_DBG_TYPE_UTIL, 
-		      " cq_object_wait: RET cq %p ibv_cq %p ibv_ctx %p %x\n",
-		      cq,ibv_cq,ibv_ctx,status);
+		dapl_dbg_log ( DAPL_DBG_TYPE_CM," cq_object_wait: poll returned
status=%d\n",status);
+
+		/*
+		 * If poll with timeout wakes then hold mutex around a poll with no timeout
+		 * so subsequent get_cq_events will be guaranteed not to block
+		 * If the event does not belong to this EVD then put it on proper EVD pending 
+		 * queue under the mutex.
+		 */
+		if (status == 1) {
+			dapl_os_lock( &evd_ptr->header.owner_ia->hca_ptr->ib_trans.cq_lock );
+			status = poll(&cq_poll, 1, 0);
+			if (status == 1) {
+				status = ibv_get_cq_event(cq->context,
+							  0, &ibv_cq, &ibv_ctx);
+
+				/* if event is not ours, put on proper evd pending queue */
+				/* force another wakeup */
+				if ((ibv_ctx != evd_ptr ) && 
+				    (!DAPL_BAD_HANDLE(ibv_ctx, DAPL_MAGIC_EVD))) {
+					dapl_dbg_log (DAPL_DBG_TYPE_CM,
+						      " cq_object_wait: ibv_ctx %p != evd %p\n",
+						      ibv_ctx, evd_ptr);
+					dapls_evd_copy_cq((struct evd_ptr*)ibv_ctx); 
+					dapl_os_unlock(
&evd_ptr->header.owner_ia->hca_ptr->ib_trans.cq_lock );
+					continue;
+				}	
+			}	
+			dapl_os_unlock( &evd_ptr->header.owner_ia->hca_ptr->ib_trans.cq_lock );
+			break;
+
+		} else if (status == 0) {
+			status = ETIMEDOUT;  
+			break;
+		}
+	}	
+	dapl_dbg_log (DAPL_DBG_TYPE_CM, 
+		      " cq_object_wait: RET evd %p cq %p ibv_cq %p ibv_ctx %p %s\n",
+		      evd_ptr, cq,ibv_cq,ibv_ctx,strerror(errno));
 	
 	return(dapl_convert_errno(status,"cq_wait_object_wait"));
 	





From halr at voltaire.com  Mon Jul 25 06:04:57 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Jul 2005 09:04:57 -0400
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
Message-ID: <1122296697.4407.9.camel@hal.voltaire.com>

Hi,

Here is a writeup on proposed IB endport liveness and resposiveness.
This "grew" out of discussions on ibping on this list some time ago.
Thanks in advance for any comments on this.

-- Hal

InfiniBand EndPort Liveness and Responsiveness

Motivation

  While there exist methods to obtain some degree of the liveness and
  responsiveness of an InfiniBand end port using existing mechanisms
  (e.g. SM Get attribute), these rely on a lack of protection as well
  as not allowing sufficient flexibility. In addition, less of the 
  end port stack may be exercised.

  IBA supports general service (GS) vendor classes. There are 2 ranges
  of these: the first range does not support RMPP whereas the second
  range does. OpenIB has a vendor OUI of 0x001405. A vendor class of 
  0x34 (in vendor class 2 range) would be utilized for this. The class
  version in the MAD header indicates the version of this. Currently,
  this is version 1.

Packet Format
  (Refer to IBA 1.2 p. 1006 for vendor class 2 MAD format)

  Bytes 40-255 are defined as vendor data as follows:

In MAD if not RMPP'd, or in first MAD if RMPP'd:
  Byte 40: Type (1 byte)
    1 - Echo request
    2 - Echo response
    3 - Timestamp request
    4 - Timestamp response
    5 - LID/GUID information request
    6 - LID/GUID information response
  
  Byte 41-43: Reserved

  Byte 44-45: Identifier
  Byte 46-47: Sequence Number

  Byte 48-255: Data for Type 

In subsequent (non first) RMPP MADs:
  Byte 40-255: Continued Data for Type


Echo

The data received in the echo request message must be returned
in the echo reply message.

The identifier and sequence number may be used by the echo sender to aid 
in matching the replies with the echo requests. For example, the identifier 
might be used like a port in TCP or UDP to identify a session, and the sequence
number might be incremented on each echo request sent. The echoer returns
these same values in the echo reply.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-255: Echo data

Identifier. 16 bits.
This field is used to help match echo requests to the associated reply.
It may be set to zero.

Sequence number. 16 bits.
This field is used to help match echo requests to the associated reply.
It may be zero.

Echo Data. Variable length.
Implementation specific data.


Timestamp

The data received (a timestamp) in the message is returned in the reply
together with an additional timestamp. The timestamp is 32 bits of
milliseconds since midnight UT.

The Originate Timestamp is the time the sender last touched the message
before sending it, the Receive Timestamp is the time the echoer first touched
it on receipt, and the Transmit Timestamp is the time the echoer last touched
the message on sending it.

If the time is not available in miliseconds or cannot be provided with respect
to midnight UT then any time can be inserted in a timestamp provided the high
order bit of the timestamp is also set to indicate this non-standard value.

The identifier and sequence number may be used by the echo sender to aid in
matching the replies with the requests. For example, the identifier might be
used like a port in TCP or UDP to identify a session, and the sequence number
might be incremented on each request sent. The destination returns these same
values in the reply.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-51: Originate timestamp
Byte 52-55: Receive timestamp
Byte 56-59: Transmit timestamp 

The preferred form for a timestamp value (the "standard value") is in units of
milliseconds since midnight Universal Time. However, it may be difficult to
provide this value with millisecond resolution. For example, many systems use
clocks that update only at line frequency, 50 or 60 times per second.
Therefore, some latitude is allowed in a "standard value":
(a) A "standard value" MUST be updated at least 15 times per second (i.e., at
most the six low-order bits of the value may be undefined).
(b) The accuracy of a "standard value" MUST approximate that of operator-set
CPU clocks, i.e., correct within a few minutes.


LID/GUID information

Obtain the LID and port GUID of the requested port.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-49: LID
Byte 50-57: Port GUID





From rolandd at cisco.com  Mon Jul 25 07:16:48 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 07:16:48 -0700
Subject: [openib-general] Question about pinning memory
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa>
Message-ID: <52k6jfotrj.fsf@topspin.com>

    Pete> I did some thinking about this issue a while back and came
    Pete> up with a cooperative kernel/user implementation to track
    Pete> linux VM activity using existing vm_area_struct->vm_ops
    Pete> function hooks (i.e. no kernel patch required).  The MPI
    Pete> library essentially makes a system call before reusing a
    Pete> cached memory registration to verify it is still valid, and
    Pete> the kernel module keeps track of what happens to cached
    Pete> mappings as the VM system is exercised via sbrk, mmap, fork,
    Pete> etc.  It works for any sort of memory activity, including
    Pete> arbitrary mmap() of memory or files, since it plugs in at
    Pete> the basic VM interface level.

This seems like the right approach to me.  I didn't like the patches
posted by Quadrics a while ago, since they added hooks to all of
low-level page table hot paths.  Working at the vma level seems much
saner.

How much of a performance hit is this system call?  It seems that you
need to do the call before every single work request is posted, which
could cause latency to go way up.

    Pete> I can distribute the code if anyone is curious, but it needs
    Pete> some work to become production quality.

Yes, please post the code soon.

Thanks,
  Roland


From rolandd at cisco.com  Mon Jul 25 07:31:46 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 07:31:46 -0700
Subject: [openib-general] Question about pinning memory
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724073934.GB29109@minantech.com>
	<a64299dbb9aee2f564ef0ae76d7ba4b2@open-mpi.org>
Message-ID: <5264uzot2l.fsf@topspin.com>

    Jeff> Interesting.  So Open IB doesn't get a notification upon
    Jeff> unmapping/sbrk'ing?

Right.  The kernel IB code works purely at the page level.

    Jeff> Also interesting.  So it's "ok" to use this strategy
    Jeff> (maintain MRU/LRU kinds of tables and unpin upon demand),
    Jeff> even though you may be unpinning memory that no longer
    Jeff> belongs to your process.  That seems pretty weird -- it
    Jeff> feels like breaking the POSIX process abstraction barrier.

In some sense, as long as the memory is pinned by a process, it still
belongs to the process.  There may no longer be a valid virtual
mapping for the pages but the original process still holds a reference
on them.

    Jeff> For example, it could be inefficient if there are multiple
    Jeff> processes using IB on a single node -- pinned memory can
    Jeff> consume pinning resources even though it's no longer in your
    Jeff> process.  If too much memory falls into this category, this
    Jeff> can even cause general virtual memory performance
    Jeff> degradation (i.e., nothing to do with IB communication at
    Jeff> all).

    Jeff> Is this the intended behavior?

I think so.  As long as there is a registered memory region that
covers a chunk of memory, that memory may be DMAed to by the IB
device.  So if a process frees memory but doesn't unregister it, the
system can't reuse it without risking memory corruption.

It is possible to use resource limits to limit the amount of memory
that a single process pins.

    Jeff> 1. process A pins memory Z 2. process A sbrk()'s Z without
    Jeff> unpinning it first 3. process B obtains memory Z 4. process
    Jeff> B pins memory Z 5. process A unpins memory Z

This can't happen.  Until process A unpins the memory, the pages have
an elevated reference count and will never be allocated for a
different process.

    Jeff> Is there any thought within the IB community to getting rid
    Jeff> of this whole memory registration issue at the user level?
    Jeff> Perhaps a la what Quadrics did (make the driver understand
    Jeff> virtual memory) or what Va Tech did (put in their own
    Jeff> kernel-level hooks to make malloc/free to do the Right
    Jeff> Things)?  Handling all this memory registration stuff is
    Jeff> certainly quite a big chunk of code that every MPI (and IB
    Jeff> app) needs to handle.  Could this stuff be moved down into
    Jeff> the realm of the IB stack itself?  From an abstraction /
    Jeff> software architecture standpoint, it could make sense to
    Jeff> unify this handling in one place rather than N (MPI
    Jeff> implementations and other IB apps).

There's definitely thinking about this, but the correct approach is by
no means obvious.  The Quadrics hooks seem too invasive to merge into
the kernel.  I'm not familiar with the VA Tech work.

Unfortunately the IB and iWARP specs are written in terms of
applications handling memory registration.  So we can't really expect
all RDMA hardware to have the hooks required to improve the memory
pinning interface.  As it becomes clearer what hardware capabilities
are common, and what the userspace requirements are, we should be able
to improve things.

 - R.


From glebn at voltaire.com  Mon Jul 25 07:37:59 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Mon, 25 Jul 2005 17:37:59 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <52k6jfotrj.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
Message-ID: <20050725143759.GJ19097@minantech.com>

On Mon, Jul 25, 2005 at 07:16:48AM -0700, Roland Dreier wrote:
>     Pete> I did some thinking about this issue a while back and came
>     Pete> up with a cooperative kernel/user implementation to track
>     Pete> linux VM activity using existing vm_area_struct->vm_ops
>     Pete> function hooks (i.e. no kernel patch required).  The MPI
>     Pete> library essentially makes a system call before reusing a
>     Pete> cached memory registration to verify it is still valid, and
>     Pete> the kernel module keeps track of what happens to cached
>     Pete> mappings as the VM system is exercised via sbrk, mmap, fork,
>     Pete> etc.  It works for any sort of memory activity, including
>     Pete> arbitrary mmap() of memory or files, since it plugs in at
>     Pete> the basic VM interface level.
> 
> This seems like the right approach to me.  I didn't like the patches
> posted by Quadrics a while ago, since they added hooks to all of
> low-level page table hot paths.  Working at the vma level seems much
> saner.
> 
> How much of a performance hit is this system call?  It seems that you
> need to do the call before every single work request is posted, which
> could cause latency to go way up.
I think the better approach would be to send signal to process whenever
vma list changes and let application verify whether cache entries
should be freed.

What do you think?

> 
>     Pete> I can distribute the code if anyone is curious, but it needs
>     Pete> some work to become production quality.
> 
> Yes, please post the code soon.
> 

--
			Gleb.


From jsquyres at open-mpi.org  Mon Jul 25 08:06:11 2005
From: jsquyres at open-mpi.org (Jeff Squyres)
Date: Mon, 25 Jul 2005 11:06:11 -0400
Subject: [openib-general] Question about pinning memory
In-Reply-To: <5264uzot2l.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724073934.GB29109@minantech.com>
	<a64299dbb9aee2f564ef0ae76d7ba4b2@open-mpi.org>
	<5264uzot2l.fsf@topspin.com>
Message-ID: <eb0d1f7f34d37c161350d639d00e860f@open-mpi.org>

On Jul 25, 2005, at 10:31 AM, Roland Dreier wrote:

> [...clarifications snipped...]
>     Jeff> Is there any thought within the IB community to getting rid
>     Jeff> of this whole memory registration issue at the user level?
>     Jeff> Perhaps a la what Quadrics did (make the driver understand
>     Jeff> virtual memory) or what Va Tech did (put in their own
>     Jeff> kernel-level hooks to make malloc/free to do the Right
>     Jeff> Things)?  Handling all this memory registration stuff is
>     Jeff> certainly quite a big chunk of code that every MPI (and IB
>     Jeff> app) needs to handle.  Could this stuff be moved down into
>     Jeff> the realm of the IB stack itself?  From an abstraction /
>     Jeff> software architecture standpoint, it could make sense to
>     Jeff> unify this handling in one place rather than N (MPI
>     Jeff> implementations and other IB apps).
>
> There's definitely thinking about this, but the correct approach is by
> no means obvious.  The Quadrics hooks seem too invasive to merge into
> the kernel.  I'm not familiar with the VA Tech work.

I agree that the kernel patches for Quadrics was somewhat painful (and 
kernel-dependent, which was sometimes a problem).

I don't know the specifics of the VA Tech work either, but my [crude] 
understanding was that they somehow made the memory [de-]registration 
totally opaque to user level (to include the MPI library).  I don't 
know if it was hidden in the IB stack or in the kernel itself.  Anyone 
here know more details on this?

> Unfortunately the IB and iWARP specs are written in terms of
> applications handling memory registration.  So we can't really expect
> all RDMA hardware to have the hooks required to improve the memory
> pinning interface.  As it becomes clearer what hardware capabilities
> are common, and what the userspace requirements are, we should be able
> to improve things.

I know nothing about how the kernel works, now how typical IB stacks 
work down in the kernel, so pardon me if this is a silly question: why 
do hardware capabilities have anything to do with this?  Isn't this a 
kernel-level and/or software API issue?

More specifically, the IB stacks already do checks to see if memory is 
registered before attempting send/receive operations.

...after thinking about this a little, I realize that for maximum 
performance, they may not, so I guess I don't know that this is true.  
But if it is, couldn't all memory [de-]registration be handled by the 
IB stack?  If it can, this would eliminate lookups and memory 
management up in the MPI level, which would potentially help decrease a 
little latency (i.e., the redundant lookups in the MPI layer could be 
removed).

I realize that there's a bunch of if's in there, but it would certainly 
be nice to not have to handle all this stuff.  Indeed, the MPI problem 
hasn't really been solved yet -- there have been a few proposals / 
techniques used in the past, but none are really attractive (all have 
benefits and drawbacks, and there are scenarios that break each of 
them):

- use mallopt() to not let memory be returned to the OS
- include a memory manager (e.g., ptmalloc2) in the MPI to override 
munmap/sbrk to catch when memory is returned to the OS
- use LD_PRELOAD to override munmap/sbrk

The method that Pete described in an earlier post could certainly be 
interesting, especially if it could be tweaked to not have to make a 
system call to check every memory reference (e.g., Gleb's suggestion of 
a signal / callback / whatever).  I would certainly take that over 
today's mechanisms (the 3 mentioned above), but would still like to 
push forward with the possibility of all the memory registration being 
handled at a level lower than MPI.

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/



From glebn at voltaire.com  Mon Jul 25 08:27:43 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Mon, 25 Jul 2005 18:27:43 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050724075336.GC29109@minantech.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050723210825.GC31367@mellanox.co.il>
	<20050724075336.GC29109@minantech.com>
Message-ID: <20050725152743.GA13014@minantech.com>

>> > Having said that - do you have some benchmarks to prove that
>> > pinning is that expensive? Are you sure that all these internal
>> > table
>> > lookups aren't even slower?
>> > I cant promise, but given a microbenchmark, maybe something could be
>> > done to
>> > improve the speed of this operation.
>>
>> In my experience cache hit in MVAPICH improves registration by factor
>> of
>> two.
>>
>But thats VAPI based, not gen2 based, isnt it?
>
What is the difference? gen2 do magic? 

>> Internal table lookup is a search in balanced tree. Hardware memory
>> registration is system call + writes on PCI bus + search in balanced
>> tree in get_user_pages,
>
>I cant tell without seeing actual benchmarks.
>Certainly tricks like overriding malloc/free operations can have
>impact on overall application performance.
There is no need to override malloc/free to do registration caching.
And here is the benchmark results for MVAPICH over gen2:
This is how much cycles takes to send buffer of 100000 bytes with cache
miss:      536388
cache hit: 9740

I used MPI_Isend for the test.

>
>> how can you improve this?
>
>I have some optimization ideas in mind - clearly even with a very smart
>registration cache there may be cache misses, so if a problem exists,
>any optimizations shall be beneficial.

It is always good to improve worst case.

--
			Gleb.


From pw at osc.edu  Mon Jul 25 08:37:29 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Mon, 25 Jul 2005 11:37:29 -0400
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050725143759.GJ19097@minantech.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
Message-ID: <20050725153729.GA19700@osc.edu>

glebn at voltaire.com wrote on Mon, 25 Jul 2005 17:37 +0300:
> On Mon, Jul 25, 2005 at 07:16:48AM -0700, Roland Dreier wrote:
> > How much of a performance hit is this system call?  It seems that you
> > need to do the call before every single work request is posted, which
> > could cause latency to go way up.
>
> I think the better approach would be to send signal to process whenever
> vma list changes and let application verify whether cache entries
> should be freed.

The system call is a nonblocking read() that returns 0 bytes when
there is no VM activity to report.

On my crusty old 1.5 GHz athlon, lmbench says:

    Simple syscall: 0.2337 microseconds
    Simple read: 0.3681 microseconds
    Signal handler overhead: 1.680 microseconds

I agree that the cost of kernel->user notification should scale with
VM activity and not message passing activity.  One way I tried was
to run a separate thread to block on the read() from the VM monitor
while the main process grabs an uncontended mutex around each cache
lookup instead of doing the read() itself.  This has a slight
scheduling race though.

I like your signal idea but am not fond of the signal mechanism,
especially as we (the library) can't trust the user not to break it.
Signaling a separate thread has the same race problem.  I'd prefer
to export a counter from the kernel that the user application can
see.  The kernel would atomically increment it when there is an
unread event, and the app can check the variable to decide if the
read() is necessary.

Give me some time to find the code and see if it still compiles
against recent 2.6.

		-- Pete



From glebn at voltaire.com  Mon Jul 25 08:43:01 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Mon, 25 Jul 2005 18:43:01 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <eb0d1f7f34d37c161350d639d00e860f@open-mpi.org>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724073934.GB29109@minantech.com>
	<a64299dbb9aee2f564ef0ae76d7ba4b2@open-mpi.org>
	<5264uzot2l.fsf@topspin.com>
	<eb0d1f7f34d37c161350d639d00e860f@open-mpi.org>
Message-ID: <20050725154301.GB13014@minantech.com>

On Mon, Jul 25, 2005 at 11:06:11AM -0400, Jeff Squyres wrote:
> On Jul 25, 2005, at 10:31 AM, Roland Dreier wrote:
> >Unfortunately the IB and iWARP specs are written in terms of
> >applications handling memory registration.  So we can't really expect
> >all RDMA hardware to have the hooks required to improve the memory
> >pinning interface.  As it becomes clearer what hardware capabilities
> >are common, and what the userspace requirements are, we should be able
> >to improve things.
> 
> I know nothing about how the kernel works, now how typical IB stacks 
> work down in the kernel, so pardon me if this is a silly question: why 
> do hardware capabilities have anything to do with this?  Isn't this a 
> kernel-level and/or software API issue?
> 
> More specifically, the IB stacks already do checks to see if memory is 
> registered before attempting send/receive operations.
> 
Application don't even enter kernel to send/receive data over IB after
registration. Hardware do all necessary checks.

> ...after thinking about this a little, I realize that for maximum 
> performance, they may not, so I guess I don't know that this is true.  
> But if it is, couldn't all memory [de-]registration be handled by the 
> IB stack?  If it can, this would eliminate lookups and memory 
> management up in the MPI level, which would potentially help decrease a 
> little latency (i.e., the redundant lookups in the MPI layer could be 
> removed).
How will it eliminate lookups? You still need to check cache before
registering memory. You have to obtain lkey/rkey somehow. If kernel
derigister memory automatically you will have cache not synced with 
hardware registration and it will cause troubles.

> 
> I realize that there's a bunch of if's in there, but it would certainly 
> be nice to not have to handle all this stuff.  Indeed, the MPI problem 
> hasn't really been solved yet -- there have been a few proposals / 
> techniques used in the past, but none are really attractive (all have 
> benefits and drawbacks, and there are scenarios that break each of 
> them):
> 
> - use mallopt() to not let memory be returned to the OS
> - include a memory manager (e.g., ptmalloc2) in the MPI to override 
> munmap/sbrk to catch when memory is returned to the OS
> - use LD_PRELOAD to override munmap/sbrk
And if some strange application calls directly to kernel non of them
will work.

> 
> The method that Pete described in an earlier post could certainly be 
> interesting, especially if it could be tweaked to not have to make a 
> system call to check every memory reference (e.g., Gleb's suggestion of 
> a signal / callback / whatever).  I would certainly take that over 
> today's mechanisms (the 3 mentioned above), but would still like to 
> push forward with the possibility of all the memory registration being 
> handled at a level lower than MPI.
> 
I think the Pete approach with callback and application maintaining
registration cache is the right one.

--
			Gleb.


From rolandd at cisco.com  Mon Jul 25 08:47:48 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 08:47:48 -0700
Subject: [openib-general] Question about pinning memory
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
Message-ID: <524qaiopjv.fsf@topspin.com>

    Gleb> I think the better approach would be to send signal to
    Gleb> process whenever vma list changes and let application verify
    Gleb> whether cache entries should be freed.

It's probably better to use a file descriptor for kernel upcalls to
userspace.  Then you don't have to go through the expensive signal
delivery path.  Also you're not limited to the info you can squeeze
into the siginfo.

In this case I'm not sure whether it's better to have things driven by
userspace or the kernel.  Having the kernel push new info up to
userspace seems more efficient, since the kernel knows exactly when
things have changed, but this seems to lead to synchronization
problems where userspace may post a work request using an obsolete
mapping before it gets the notification that the mapping changed.

 - R.


From glebn at voltaire.com  Mon Jul 25 08:55:38 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Mon, 25 Jul 2005 18:55:38 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <524qaiopjv.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
Message-ID: <20050725155538.GC13014@minantech.com>

On Mon, Jul 25, 2005 at 08:47:48AM -0700, Roland Dreier wrote:
>     Gleb> I think the better approach would be to send signal to
>     Gleb> process whenever vma list changes and let application verify
>     Gleb> whether cache entries should be freed.
> 
> It's probably better to use a file descriptor for kernel upcalls to
> userspace.  Then you don't have to go through the expensive signal
> delivery path.  Also you're not limited to the info you can squeeze
> into the siginfo.
Can you elaborate on kernel upcalls mechanism?

> 
> In this case I'm not sure whether it's better to have things driven by
> userspace or the kernel.  Having the kernel push new info up to
> userspace seems more efficient, since the kernel knows exactly when
> things have changed, but this seems to lead to synchronization
> problems where userspace may post a work request using an obsolete
> mapping before it gets the notification that the mapping changed.
>
This is responsibility of the user to make sure he is not unmaps or frees
memory that is currently in use.

--
			Gleb.


From glebn at voltaire.com  Mon Jul 25 08:58:46 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Mon, 25 Jul 2005 18:58:46 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050725153729.GA19700@osc.edu>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
Message-ID: <20050725155846.GD13014@minantech.com>

On Mon, Jul 25, 2005 at 11:37:29AM -0400, Pete Wyckoff wrote:
> I like your signal idea but am not fond of the signal mechanism,
> especially as we (the library) can't trust the user not to break it.
> Signaling a separate thread has the same race problem.  I'd prefer
> to export a counter from the kernel that the user application can
> see.  The kernel would atomically increment it when there is an
> unread event, and the app can check the variable to decide if the
> read() is necessary.
I also don't like signal much but... 
How userspace will read kernel counter without system call? Or do you
mean kernel will increment counter resided in userspace?

--
			Gleb.


From mshefty at ichips.intel.com  Mon Jul 25 09:38:08 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Mon, 25 Jul 2005 09:38:08 -0700
Subject: [openib-general] Re: uCM init_qp_attr() [Re: uCM
	create	connectionID]
In-Reply-To: <000501c58d8c$3050b2a0$9c5aa8c0@infiniconsys.com>
References: <000501c58d8c$3050b2a0$9c5aa8c0@infiniconsys.com>
Message-ID: <42E51570.3010102@ichips.intel.com>

Fab Tillier wrote:
>>From: Libor Michalek [mailto:libor at topspin.com]
>>>The uDAPL code is now connecting properly but I am having difficulty
>>>setting the QP states properly without the ib_cm_init_qp_attr() call.
>>>Any chance of providing this call in uCM?
>>
>>  Sean, To do this I'll need to add a kernel CM function to get the
>>connections qp attributes from 'struct cm_id_private' inorder to pass
>>them to userspace. Any preference on what form the function should take?
> 
> Why can't you just expose the exiting kernel ib_cm_init_qp_attr call?  It seems
> that call provides the access to cm_id_private.  As long as you buffer the
> qp_attr structure properly, what are the issues?

Can you just use the existing call?

- Sean


From rolandd at cisco.com  Mon Jul 25 09:47:59 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 09:47:59 -0700
Subject: [openib-general] Question about pinning memory
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com>
Message-ID: <523bq2n874.fsf@topspin.com>

    Gleb> Can you elaborate on kernel upcalls mechanism?

Not sure what you mean by the question.  I was just trying to say that
in general when the kernel has some information it wants to push to
userspace, it's probably best for userspace to get it via a read() on
a file descriptor rather than for the kernel to queue a signal.

    Gleb> This is responsibility of the user to make sure he is not
    Gleb> unmaps or frees memory that is currently in use.

Any registration caching scheme has to deal with a user using memory
X, freeing memory X, and then allocating memory Y at the same virtual
address as X.

 - R.


From Sujal at Mellanox.com  Mon Jul 25 09:51:11 2005
From: Sujal at Mellanox.com (Sujal Das)
Date: Mon, 25 Jul 2005 09:51:11 -0700
Subject: [openib-general] Datacenter Fabric Workshop Linux Track Agenda
	Update
Message-ID: <25AE7F432672D511B8DC00B0D0DF11DA0560B94F@MTIEX01>

Hello all:

 

Here is the updated agenda, list of speakers/presenters for the Workshop
Linux Track. 

 

Duration of the sessions has been changed to 30 minutes each.  Some names of
suggested speakers have been added or changed.

 

Please look below to see if your name appears below.  If so, you need to
confirm and send and abstract.  If you cannot, please inform me ASAP and
suggest an alternate if you can.

 

So far, I have received confirmation from five speakers and abstract from
one.  The due date for such confirmations and abstracts is August 1, 2005.
Please provide those at your earliest convenience.

 

Please see below for a sample of  an abstract (extracted from the last
OpenIB Workshop).

 

Proposed content for tracks L1 - L10:

- Status of software - challenges and successes

- Future roadmap and plans for upcoming release candidates and their quality
assurance

- Kernel and/or OS distribution inclusion plans if any

 

 Track L1:

            10-30-11:00 AM

            Core  

            Speaker(s): Hal Rosenstock - Voltaire, Roland Dreier - Cisco,
Sean Hefty - Intel

 

Track L2

            11:00-11.30 AM

            iWARP Integration Issues Update

            Speaker: Clem Cole - Ammasso

 

Track L3

            11.30-12.00 Noon

            kDAPL

            Speaker(s): Arkady Kanevsky, James Lentini - Network Appliance

 

Track L4

            12.00-12:30 PM

            uDAPL

            Speakers: Arkady Kanevsky - Network appliance, A. Davis - Intel

 

 

12.30-13.15: LUNCH

 

 

Track L5

            13.15-13.45

            L5 - SDP

            Speaker: Libor

 

 

Track L6

            13.45-14.15

            MPI OSU

            Speaker(s): Dr. Panda - OSU, Gil Boch or Dror Goldenberg -
Mellanox

 

Track L7

            14.15-14.45

            Open MPI 

            Speaker: Tim Woodall (LANL) and/or Jeff Squyres (UIndiana)

 

Track L8

            14.45-15.15

            SRP Initiator and Boot over IB

            Speaker(s): Roland - Cisco

            Abstract: SCSI RDMA Protocol (SRP) is an INCITS T10 standard for
storage over InfiniBand.  

            This session will cover the current state of the OpenIB SRP
initiator, and describe how it can be used to boot 

            diskless servers from IB block storage.  We will also survey the
diverse SRP targets that are available.

 

 

Track L9

            15.15-15.45

            iSER

            Speaker: Hal - Voltaire

 

15.45 - 16.00: BREAK

 

Track L10

            16.00-16.30

            NFS RDMA Server and Client

            Speaker(s): Boris Shpolyansky, Vu Pham - Mellanox

 

 

Track L11

            16.30-17.00

            Release and Test Processes

            Speaker: Hal - Voltaire, Amit Krig  or Sujal Das - Mellanox

 

Track 12

            17.00 - 17.30

            Distribution Channels 

            Speaker: [NEED A SPEAKER FROM REDHAT AND NOVELL - PLEASE ASSIST]

 

 

 Abstract Sample for Reference Only

 ----------------------------------

 

 (IMPORTANT: Please use "proposed content for tracks L1 - L10" 

 mentioned above to develop your abstract")

 

 iSER is a new IETF extension to iSCSI allowing it to run over native 

 RDMA transport replacing the slow TCP transport. The session will 

 describe the new iSCSI Datamover layer, the iSER transport module and 

 its relations with iSCSI and InfiniBand, the standard mapping from 

 iSCSI/iSER to FC and to iSCSI/TCP. The session will also review plans 

 for upcoming releases of iSER, and discussions on progresses made in 

 the areas of incorporating iSER into the iSCSI-Linux project, the 

 latest iSER RFC work in the IETF and iSER integration with upcoming kernels
and OS distributions.

 

 

Sujal Das

Mellanox Technologies Inc.

408 916 0007

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050725/5eda67ea/attachment.html>

From tduffy at sun.com  Mon Jul 25 09:58:34 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 25 Jul 2005 09:58:34 -0700
Subject: [openib-general] [PATCH] SDP: use linux/list.h for advt table
Message-ID: <1122310714.27947.8.camel@duffman>

This patch changes sdp_advt.[ch] to use linux lists.  I didn't change
the API, but it may be a good idea now that the lists are done a bit
differently.

Signed-off-by: Tom Duffy <tduffy at sun.com>

 sdp_advt.c |   51 +++++++++++++--------------------------------------
 sdp_advt.h |    7 ++++---
 2 files changed, 17 insertions(+), 41 deletions(-)

Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_advt.c
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_advt.c	(revision 2900)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_advt.c	(working copy)
@@ -33,6 +33,8 @@
  * $Id$
  */
 
+#include <linux/list.h>
+
 #include "sdp_main.h"
 
 static kmem_cache_t *sdp_advt_cache = NULL;
@@ -68,7 +70,6 @@ struct sdpc_advt *sdp_advt_create(void)
  */
 void sdp_advt_destroy(struct sdpc_advt *advt)
 {
-	BUG_ON(advt->next || advt->prev);
 	/*
 	 * return the object to its cache
 	 */
@@ -81,29 +82,16 @@ void sdp_advt_destroy(struct sdpc_advt *
 struct sdpc_advt *sdp_advt_q_get(struct sdpc_advt_q *table)
 {
 	struct sdpc_advt *advt;
-	struct sdpc_advt *next;
-	struct sdpc_advt *prev;
 
-	advt = table->head;
-	if (!advt)
+	if (list_empty(&table->head))
 		return NULL;
 
-	if (advt->next == advt && advt->prev == advt)
-		table->head = NULL;
-	else {
-		next = advt->next;
-		prev = advt->prev;
-		next->prev = prev;
-		prev->next = next;
+	advt = list_entry(table->head.next, struct sdpc_advt, list);
 
-		table->head = next;
-	}
+	list_del(&advt->list);
 
 	table->size--;
 
-	advt->next = NULL;
-	advt->prev = NULL;
-
 	return advt;
 }
 
@@ -112,7 +100,10 @@ struct sdpc_advt *sdp_advt_q_get(struct 
  */
 struct sdpc_advt *sdp_advt_q_look(struct sdpc_advt_q *table)
 {
-	return table->head;
+	if (list_empty(&table->head))
+		return NULL;
+	
+	return list_entry(table->head.next, struct sdpc_advt, list);
 }
 
 /*
@@ -120,25 +111,9 @@ struct sdpc_advt *sdp_advt_q_look(struct
  */
 void sdp_advt_q_put(struct sdpc_advt_q *table, struct sdpc_advt *advt)
 {
-	struct sdpc_advt *next;
-	struct sdpc_advt *prev;
-
 	BUG_ON(advt->table);
 
-	if (!table->head) {
-		advt->next = advt;
-		advt->prev = advt;
-		table->head = advt;
-	}
-	else {
-		next = table->head;
-		prev = next->prev;
-
-		prev->next = advt;
-		advt->prev = prev;
-		advt->next = next;
-		next->prev = advt;
-	}
+	list_add_tail(&advt->list, &table->head);
 
 	table->size++;
 }
@@ -148,7 +123,7 @@ void sdp_advt_q_put(struct sdpc_advt_q *
  */
 void sdp_advt_q_init(struct sdpc_advt_q *table)
 {
-	table->head = NULL;
+	INIT_LIST_HEAD(&table->head);
 	table->size = 0;
 }
 
@@ -157,11 +132,11 @@ void sdp_advt_q_init(struct sdpc_advt_q 
  */
 void sdp_advt_q_clear(struct sdpc_advt_q *table)
 {
-	struct sdpc_advt *advt;
+	struct sdpc_advt *advt, *tmp;
 	/*
 	 * drain the table of any objects
 	 */
-	while ((advt = sdp_advt_q_get(table)))
+	list_for_each_entry_safe(advt, tmp, &table->head, list)
 		sdp_advt_destroy(advt);
 }
 
Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_advt.h
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_advt.h	(revision 2900)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_advt.h	(working copy)
@@ -36,6 +36,8 @@
 #ifndef _SDP_ADVT_H
 #define _SDP_ADVT_H
 
+#include <linux/list.h>
+
 #include "sdp_queue.h"
 
 /*
@@ -47,8 +49,7 @@
  * SDP read/write advertisments
  */
 struct sdpc_advt {
-	struct sdpc_advt   *next;  /* next structure in table */
-	struct sdpc_advt   *prev;  /* previous structure in table */
+	struct list_head list;
 	u32                 type; /* element type. (for generic queue) */
 	struct sdpc_advt_q *table; /* table to which this object belongs */
 	void (*release)(struct sdpc_advt *advt); /* release the object */
@@ -67,7 +68,7 @@ struct sdpc_advt {
  * table for holding SDP advertisments.
  */
 struct sdpc_advt_q {
-	struct sdpc_advt *head; /* double linked list of advertisments */
+	struct list_head head; /* double linked list of advertisments */
 	s32 size;               /* current number of advertisments in table */
 };
 



From mst at mellanox.co.il  Mon Jul 25 10:14:54 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 25 Jul 2005 20:14:54 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <523bq2n874.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com> <523bq2n874.fsf@topspin.com>
Message-ID: <20050725171454.GB12206@mellanox.co.il>

Quoting r. Roland Dreier <rolandd at cisco.com>:
> Subject: Re: Question about pinning memory
> 
>     Gleb> Can you elaborate on kernel upcalls mechanism?
> 
> Not sure what you mean by the question.  I was just trying to say that
> in general when the kernel has some information it wants to push to
> userspace, it's probably best for userspace to get it via a read() on
> a file descriptor rather than for the kernel to queue a signal.

And given that you are performing a system call on each send/receive,
why not post the work request from kernel as well?
And since this is what SDP AIO does, why not just use it?

-- 
MST


From mst at mellanox.co.il  Mon Jul 25 10:19:28 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 25 Jul 2005 20:19:28 +0300
Subject: [openib-general] [PATCH repost] PROT_DONTCOPY: ifiniband uverbs
	fork support
In-Reply-To: <20050719165542.GB16028@mellanox.co.il>
References: <20050719165542.GB16028@mellanox.co.il>
Message-ID: <20050725171928.GC12206@mellanox.co.il>

Hi!
I posted this before but got no comments.
Here it is again, in case the reason was OLS.

---

This patch adds PROT_DONTCOPY to mmap and mprotect, to set VM_DONTCOPY on vma.
This is needed for infiniband userspace i/o, where we need to protect against
  - the child process accessing the parent hardware page
  - the parent registered address (on which the driver did get_user_pages)
    getting remapped to another page by COW
One can imagine other uses, e.g. combined with mlock for real-time or security.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/include/asm-ppc64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ppc64/mman.h
+++ linux-2.6.12.2/include/asm-ppc64/mman.h
@@ -15,6 +15,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-cris/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-cris/mman.h
+++ linux-2.6.12.2/include/asm-cris/mman.h
@@ -10,6 +10,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-arm26/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-arm26/mman.h
+++ linux-2.6.12.2/include/asm-arm26/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-alpha/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-alpha/mman.h
+++ linux-2.6.12.2/include/asm-alpha/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-m68k/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-m68k/mman.h
+++ linux-2.6.12.2/include/asm-m68k/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-mips/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-mips/mman.h
+++ linux-2.6.12.2/include/asm-mips/mman.h
@@ -22,6 +22,7 @@
 #define PROT_SEM	0x10		/* page may be used for atomic ops */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 /*
  * Flags for mmap
Index: linux-2.6.12.2/include/asm-sparc64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sparc64/mman.h
+++ linux-2.6.12.2/include/asm-sparc64/mman.h
@@ -11,6 +11,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-v850/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-v850/mman.h
+++ linux-2.6.12.2/include/asm-v850/mman.h
@@ -7,6 +7,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-s390/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-s390/mman.h
+++ linux-2.6.12.2/include/asm-s390/mman.h
@@ -16,6 +16,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-parisc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-parisc/mman.h
+++ linux-2.6.12.2/include/asm-parisc/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-ppc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ppc/mman.h
+++ linux-2.6.12.2/include/asm-ppc/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-i386/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-i386/mman.h
+++ linux-2.6.12.2/include/asm-i386/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-sh/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sh/mman.h
+++ linux-2.6.12.2/include/asm-sh/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-x86_64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-x86_64/mman.h
+++ linux-2.6.12.2/include/asm-x86_64/mman.h
@@ -8,6 +8,7 @@
 #define PROT_SEM	0x8
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-ia64/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-ia64/mman.h
+++ linux-2.6.12.2/include/asm-ia64/mman.h
@@ -15,6 +15,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-sparc/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-sparc/mman.h
+++ linux-2.6.12.2/include/asm-sparc/mman.h
@@ -11,6 +11,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-m32r/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-m32r/mman.h
+++ linux-2.6.12.2/include/asm-m32r/mman.h
@@ -10,6 +10,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-frv/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-frv/mman.h
+++ linux-2.6.12.2/include/asm-frv/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/linux/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/linux/mman.h
+++ linux-2.6.12.2/include/linux/mman.h
@@ -47,9 +47,10 @@ static inline void vm_unacct_memory(long
 static inline unsigned long
 calc_vm_prot_bits(unsigned long prot)
 {
-	return _calc_vm_trans(prot, PROT_READ,  VM_READ ) |
-	       _calc_vm_trans(prot, PROT_WRITE, VM_WRITE) |
-	       _calc_vm_trans(prot, PROT_EXEC,  VM_EXEC );
+	return _calc_vm_trans(prot, PROT_READ,     VM_READ ) |
+	       _calc_vm_trans(prot, PROT_WRITE,    VM_WRITE) |
+	       _calc_vm_trans(prot, PROT_EXEC,     VM_EXEC ) |
+	       _calc_vm_trans(prot, PROT_DONTCOPY, VM_DONTCOPY );
 }
 
 /*
Index: linux-2.6.12.2/include/asm-h8300/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-h8300/mman.h
+++ linux-2.6.12.2/include/asm-h8300/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/include/asm-arm/mman.h
===================================================================
--- linux-2.6.12.2.orig/include/asm-arm/mman.h
+++ linux-2.6.12.2/include/asm-arm/mman.h
@@ -8,6 +8,7 @@
 #define PROT_NONE	0x0		/* page can not be accessed */
 #define PROT_GROWSDOWN	0x01000000	/* mprotect flag: extend change to start of growsdown vma */
 #define PROT_GROWSUP	0x02000000	/* mprotect flag: extend change to end of growsup vma */
+#define PROT_DONTCOPY	0x04000000	/* dont copy to child on fork */
 
 #define MAP_SHARED	0x01		/* Share changes */
 #define MAP_PRIVATE	0x02		/* Changes are private */
Index: linux-2.6.12.2/mm/mprotect.c
===================================================================
--- linux-2.6.12.2.orig/mm/mprotect.c
+++ linux-2.6.12.2/mm/mprotect.c
@@ -196,7 +196,7 @@ sys_mprotect(unsigned long start, size_t
 	end = start + len;
 	if (end <= start)
 		return -ENOMEM;
-	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM))
+	if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM | PROT_DONTCOPY))
 		return -EINVAL;
 
 	reqprot = prot;
@@ -246,7 +246,7 @@ sys_mprotect(unsigned long start, size_t
 			goto out;
 		}
 
-		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC));
+		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC | VM_DONTCOPY));
 
 		if ((newflags & ~(newflags >> 4)) & 0xf) {
 			error = -EACCES;
Index: linux-2.6.12.2/mm/mmap.c
===================================================================
--- linux-2.6.12.2.orig/mm/mmap.c
+++ linux-2.6.12.2/mm/mmap.c
@@ -792,8 +792,8 @@ struct anon_vma *find_mergeable_anon_vma
 	 * Neither mlock nor madvise tries to remerge at present,
 	 * so leave their flags as obstructing a merge.
 	 */
-	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC);
-	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC);
+	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
+	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
 
 	if (near->anon_vma && vma->vm_end == near->vm_start &&
  			mpol_equal(vma_policy(vma), vma_policy(near)) &&
@@ -814,8 +814,8 @@ try_prev:
 	if (!near)
 		goto none;
 
-	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC);
-	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC);
+	vm_flags = vma->vm_flags & ~(VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
+	vm_flags |= near->vm_flags & (VM_READ|VM_WRITE|VM_EXEC|VM_DONTCOPY);
 
 	if (near->anon_vma && near->vm_end == vma->vm_start &&
   			mpol_equal(vma_policy(near), vma_policy(vma)) &&

-- 
MST


From mst at mellanox.co.il  Mon Jul 25 10:45:12 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 25 Jul 2005 20:45:12 +0300
Subject: [openib-general] [PATCH] io_remap_pfn_range
Message-ID: <20050725174512.GA12601@mellanox.co.il>

Use io_remap_pfn_range to remap IO pages (remap_pfn_range is for memory).

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-kernel/drivers/infiniband/hw/mthca/mthca_provider.c
===================================================================
--- linux-kernel/drivers/infiniband/hw/mthca/mthca_provider.c	(revision 2868)
+++ linux-kernel/drivers/infiniband/hw/mthca/mthca_provider.c	(working copy)
@@ -349,7 +349,7 @@ static int mthca_mmap_uar(struct ib_ucon
 
 	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 
-	if (remap_pfn_range(vma, vma->vm_start,
+	if (io_remap_pfn_range(vma, vma->vm_start,
 			    to_mucontext(context)->uar.pfn,
 			    PAGE_SIZE, vma->vm_page_prot))
 		return -EAGAIN;

-- 
MST


From jlentini at netapp.com  Mon Jul 25 11:00:02 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 25 Jul 2005 14:00:02 -0400 (EDT)
Subject: [openib-general] [ANNOUNCE] new kDAPL location
Message-ID: <Pine.LNX.4.61.0507251339460.5935@jlentini-linux.nane.netapp.com>


As discussed at the OpenIB BOF at OLS, I've add the kdapl sources to

gen2/trunk/src/linux-kernel/infiniband/ulp/kdapl

and the kdapltest sources to

gen2/utils/src/linux-kernel/kdapl/dapltest

I've made a few modifications to better align the kdapl directory and 
file structure with the rest of the code tree.

An updated README on how to build and use the code can be found at:

https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/kdapl/README

I'd like to thank Tom Duffy, Bernhard Fischer, Sean Hefty, Christoph 
Hellwig, Itamar Rabenstein, and Hal Rosenstock for their help with the 
porting effort.

james

James Lentini                email: jlentini at netapp.com
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300


From tduffy at sun.com  Mon Jul 25 11:18:20 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 25 Jul 2005 11:18:20 -0700
Subject: [openib-general] [PATCH] SDP: fix oops with port reuse
Message-ID: <1122315500.27947.25.camel@duffman>

This patch fixes an oops that I introduced in my conversion to use linux
lists for binds (committed in revision 2874).  If two sockets tried to
use the same port, after failing to get the port (again), it would
attempt a put and the second attempt would oops the machine.  This patch
fixes the problem that looked like this:

Unable to handle kernel NULL pointer dereference at 0000000000000008 RIP:
<ffffffff8830ca0e>{:ib_sdp:sdp_inet_port_put+62}
PGD 0
Oops: 0002 [1] PREEMPT SMP
CPU 0
Modules linked in: ib_sdp ib_cm ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core nfs lockd rfcomm l2cap bluetooth pcmcia yenta_socket rsrc_nonstatic pcmcia_core sunrpc ext3 jbd dm_mod video hotkey container button battery ac ohci_hcd tpm_atmel tpm hw_random shpchp ib_mthca ib_ipoib ib_sa ib_mad ib_core tg3 floppy mptspi xfs exportfs mptscsih mptbase sd_mod scsi_mod
Pid: 8914, comm: ttcp Not tainted 2.6.13-rc3openib
RIP: 0010:[<ffffffff8830ca0e>] <ffffffff8830ca0e>{:ib_sdp:sdp_inet_port_put+62}
RSP: 0018:ffff81006d79fdc8  EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff8100085f90b8 RCX: ffff8100085f9670
RDX: 0000000000000000 RSI: 0000000000000213 RDI: ffffffff8831d0b8
RBP: ffff8100085f90b8 R08: 0000000000000000 R09: ffff8100757f8e28
R10: ffff81006d79fd54 R11: 0000000000000000 R12: ffff8100085f9634
R13: ffff8100085f9640 R14: ffff81003f8db050 R15: 0000000000000000
FS:  00002aaaaaad7900(0000) GS:ffffffff80514800(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000000101000 CR4: 00000000000006e0
Process ttcp (pid: 8914, threadinfo ffff81006d79e000, task ffff81007366a030)
Stack: ffff8100085f90b8 ffffffff8830d78b 0000000000000213 ffffffff883091d0
       ffff810023c2b110 ffffffff802f2b29 ffff81003fff3100 ffff810023c2b110
       0000000000000000 0000000000000213
Call Trace:<ffffffff8830d78b>{:ib_sdp:sdp_conn_put+171} <ffffffff883091d0>{:ib_sdp:sdp_inet_release+816}
       <ffffffff802f2b29>{release_sock+25} <ffffffff802ef819>{sock_release+25}
       <ffffffff802f03b5>{sock_close+53} <ffffffff80189452>{__fput+178}
       <ffffffff80186508>{filp_close+104} <ffffffff80138a44>{put_files_struct+116}
       <ffffffff80139bf7>{do_exit+535} <ffffffff8013a76c>{do_group_exit+252}
       <ffffffff8010dda2>{system_call+126}

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 2904)
+++ drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -498,7 +498,8 @@ int sdp_inet_port_put(struct sdp_sock *c
 		return -EADDRNOTAVAIL;
 
 	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
-	list_del(&conn->bind_next);
+	if (conn->src_port)
+		list_del(&conn->bind_next);
 	conn->src_port = 0;
 	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
 



From itamar at mellanox.co.il  Mon Jul 25 12:08:59 2005
From: itamar at mellanox.co.il (Itamar Rabenstein)
Date: Mon, 25 Jul 2005 22:08:59 +0300
Subject: [openib-general] [ANNOUNCE] new kDAPL location
Message-ID: <91DB792C7985D411BEC300B40080D29CC35F0D@mtvex01.mtv.mtl.com>

from the readme:
...
Current Limitations

1. Only kdapltest -T T, -T Q, and -T S currently supported. 
   Other options are not.

Note that -T P is also supported.

 Itamar

> -----Original Message-----
> From: James Lentini [mailto:jlentini at netapp.com]
> Sent: Monday, July 25, 2005 9:00 PM
> To: openib-general at openib.org
> Subject: [openib-general] [ANNOUNCE] new kDAPL location
> 
> 
> 
> As discussed at the OpenIB BOF at OLS, I've add the kdapl sources to
> 
> gen2/trunk/src/linux-kernel/infiniband/ulp/kdapl
> 
> and the kdapltest sources to
> 
> gen2/utils/src/linux-kernel/kdapl/dapltest
> 
> I've made a few modifications to better align the kdapl directory and 
> file structure with the rest of the code tree.
> 
> An updated README on how to build and use the code can be found at:
> 
> https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/
ulp/kdapl/README

I'd like to thank Tom Duffy, Bernhard Fischer, Sean Hefty, Christoph 
Hellwig, Itamar Rabenstein, and Hal Rosenstock for their help with the 
porting effort.

james

James Lentini                email: jlentini at netapp.com
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general


From rolandd at cisco.com  Mon Jul 25 12:10:46 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 12:10:46 -0700
Subject: [openib-general] [PATCH] io_remap_pfn_range
In-Reply-To: <20050725174512.GA12601@mellanox.co.il> (Michael S. Tsirkin's
	message of "Mon, 25 Jul 2005 20:45:12 +0300")
References: <20050725174512.GA12601@mellanox.co.il>
Message-ID: <52br4qln0p.fsf@topspin.com>

Thanks, applied... our first piece of Xen support :)


From limichal at cisco.com  Mon Jul 25 12:20:15 2005
From: limichal at cisco.com (Libor Michalek)
Date: Mon, 25 Jul 2005 12:20:15 -0700
Subject: [openib-general] [PATCH] SDP: fix oops with port reuse
In-Reply-To: <1122315500.27947.25.camel@duffman>;
	from tduffy@sun.com on Mon, Jul 25, 2005 at 11:18:20AM -0700
References: <1122315500.27947.25.camel@duffman>
Message-ID: <20050725122015.A7871@topspin.com>

On Mon, Jul 25, 2005 at 11:18:20AM -0700, Tom Duffy wrote:
> This patch fixes an oops that I introduced in my conversion to use linux
> lists for binds (committed in revision 2874).  If two sockets tried to
> use the same port, after failing to get the port (again), it would
> attempt a put and the second attempt would oops the machine.  This patch
> fixes the problem that looked like this:

Tom,

  The original code at the begining of sdp_inet_port_put() was checking
to see if the connection was already in the bind list, and returning an
error if it was not in the list. The error check that's there now is
doing something different, and we return success if the connection
is not in the bind list and has a put() done on it.

  If we are regularly doing a put on a socket not in the bind list,
the function should be made void, and the check for list membership
not generate an error...

-Libor


> 
> Signed-off-by: Tom Duffy <tduffy at sun.com>
> 
> Index: drivers/infiniband/ulp/sdp/sdp_conn.c
> ===================================================================
> --- drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 2904)
> +++ drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
> @@ -498,7 +498,8 @@ int sdp_inet_port_put(struct sdp_sock *c
>  		return -EADDRNOTAVAIL;
>  
>  	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
> -	list_del(&conn->bind_next);
> +	if (conn->src_port)
> +		list_del(&conn->bind_next);
>  	conn->src_port = 0;
>  	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
>  
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


From limichal at cisco.com  Mon Jul 25 12:23:00 2005
From: limichal at cisco.com (Libor Michalek)
Date: Mon, 25 Jul 2005 12:23:00 -0700
Subject: [openib-general] Re: uCM kernel oops
In-Reply-To: <42E01B9C.2060309@ichips.intel.com>;
	from ardavis@ichips.intel.com on Thu, Jul 21, 2005 at 03:03:08PM
	-0700
References: <42DD4230.9060702@ichips.intel.com>
	<20050721120550.A31860@topspin.com>
	<42E01B9C.2060309@ichips.intel.com>
Message-ID: <20050725122300.B7871@topspin.com>

On Thu, Jul 21, 2005 at 03:03:08PM -0700, Arlin Davis wrote:
> Libor Michalek wrote:
> >On Tue, Jul 19, 2005 at 11:10:56AM -0700, Arlin Davis wrote:
> >
> >>Hi Libor,
> >>
> >>I am running uCM and uAT with uDAPL and occasionally hit the following. 
> >>Can you take a look?
> >
> >  Looks like a race between the destroy command and a DREQ received
> >event. From looking at the code, it looks like it's possible for
> >two threads (cm event and userspace) to call ctx_put at the same
> >time and both try to perform the final zero referece object destroy.
> >Is this easily reproducible? Can you try the following patch?
>
> I am having difficulty reproducing.

  Have you been able to reproduce, and see if this patch fixes the problem?
I'm going to go ahead and check in the patch since it's more correct then
what's there.

-Libor


From halr at voltaire.com  Mon Jul 25 12:19:30 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Jul 2005 15:19:30 -0400
Subject: [openib-general] kdapltest building
Message-ID: <1122319169.4421.7.camel@hal.voltaire.com>

Hi James,

With the latest changes, when building kdapltest as follows:

cd /path/to/svn/gen2/utils/src/linux-kernel/kdapl/dapltest/kdapl
make -f Makefile.user

gcc -g3  -Wall -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations -Werror -pipe -D__KDAPLTEST__ -D__LINUX__ -D__PENTIUM__ -I../../../../../../users/jlentini/userspace/dat/include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../mdep/linux/ -I./ -o /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../Obj/dapl_getopt.o -c /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../cmd/dapl_getopt.c
In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:36:22: dat/udat.h: No such file or directory
In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:56,
                 from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_bpool.h:51: parse error before "DAT_LMR_CONTEXT"

and other similar errors

-- Hal



From ardavis at ichips.intel.com  Mon Jul 25 12:26:49 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Mon, 25 Jul 2005 12:26:49 -0700
Subject: [openib-general] Re: uCM kernel oops
In-Reply-To: <20050725122300.B7871@topspin.com>
References: <42DD4230.9060702@ichips.intel.com>
	<20050721120550.A31860@topspin.com>
	<42E01B9C.2060309@ichips.intel.com>
	<20050725122300.B7871@topspin.com>
Message-ID: <42E53CF9.90803@ichips.intel.com>

Libor Michalek wrote:

>On Thu, Jul 21, 2005 at 03:03:08PM -0700, Arlin Davis wrote:
>  
>
>>Libor Michalek wrote:
>>    
>>
>>>On Tue, Jul 19, 2005 at 11:10:56AM -0700, Arlin Davis wrote:
>>>
>>>      
>>>
>>>>Hi Libor,
>>>>
>>>>I am running uCM and uAT with uDAPL and occasionally hit the following. 
>>>>Can you take a look?
>>>>        
>>>>
>>> Looks like a race between the destroy command and a DREQ received
>>>event. From looking at the code, it looks like it's possible for
>>>two threads (cm event and userspace) to call ctx_put at the same
>>>time and both try to perform the final zero referece object destroy.
>>>Is this easily reproducible? Can you try the following patch?
>>>      
>>>
>>I am having difficulty reproducing.
>>    
>>
>
>  
>
Sorry, I cannot reproduce.

>  Have you been able to reproduce, and see if this patch fixes the problem?
>I'm going to go ahead and check in the patch since it's more correct then
>what's there.
>
>-Libor
>
>  
>



From halr at voltaire.com  Mon Jul 25 12:23:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Jul 2005 15:23:25 -0400
Subject: [openib-general] [PATCH] [kdapl]: Update README for IBAT being part
	of IB core
Message-ID: <1122319404.4421.10.camel@hal.voltaire.com>

[kdapl]: Update README for IBAT being part of IB core

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Index: README
===================================================================
--- README	(revision 2907)
+++ README	(working copy)
@@ -10,26 +10,13 @@
 Building
 ----------------------------------------------------------------------
 
-1. Apply the kernel IBAT build patch
+1. Configure and build the kernel
 
-	> cd /path/to/linux/sources
-	> patch -p0 < /path/to/svn/gen2/users/jlentini/linux-kernel/patches/ibat_build_patch
-
-2. Setup the IBAT module 
-        
-	> cd /path/to/linux/sources/drivers/infiniband/include
-	> ln -s /path/to/svn/gen2/users/jlentini/linux-kernel/patches/ib_at.h ib_at.h
-	> cd /path/to/linux/sources/drivers/infiniband/core
-	> ln -s /path/to/svn/gen2/users/jlentini/linux-kernel/patches/at.c at.c
-
-
-3. Configure and build the kernel
-
    Follow the instructions for building the kernel components of OpenIB
    (https://openib.org/svn/gen2/trunk/README.kernel-build). Before building 
    your kernel, use the CONFIG_KDAPL[_XXX] options to setup kDAPL.
 
-4. Build kdapltest
+2. Build kdapltest
 
 	> cd /path/to/svn/gen2/utils/src/linux-kernel/kdapl/dapltest/kdapl
         > make





From halr at voltaire.com  Mon Jul 25 12:45:29 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Jul 2005 15:45:29 -0400
Subject: [openib-general] kdapltest building
In-Reply-To: <1122319169.4421.7.camel@hal.voltaire.com>
References: <1122319169.4421.7.camel@hal.voltaire.com>
Message-ID: <1122320728.4421.16.camel@hal.voltaire.com>

On Mon, 2005-07-25 at 15:19, Hal Rosenstock wrote:
> Hi James,
> 
> With the latest changes, when building kdapltest as follows:
> 
> cd /path/to/svn/gen2/utils/src/linux-kernel/kdapl/dapltest/kdapl
> make -f Makefile.user
> 
> gcc -g3  -Wall -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations -Werror -pipe -D__KDAPLTEST__ -D__LINUX__ -D__PENTIUM__ -I../../../../../../users/jlentini/userspace/dat/include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../mdep/linux/ -I./ -o /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../Obj/dapl_getopt.o -c /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../cmd/dapl_getopt.c
> In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
> /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:36:22: dat/udat.h: No such file or directory
> In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:56,
>                  from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
> /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_bpool.h:51: parse error before "DAT_LMR_CONTEXT"
> 
> and other similar errors

Makefile.user still has the following rule:
INCLUDE_DIRS   =  -I../../../../../../users/jlentini/userspace/dat/include/

-- Hal



From jlentini at netapp.com  Mon Jul 25 12:53:22 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 25 Jul 2005 15:53:22 -0400 (EDT)
Subject: [openib-general] Re: kdapltest building
In-Reply-To: <1122319169.4421.7.camel@hal.voltaire.com>
References: <1122319169.4421.7.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507251549260.5935@jlentini-linux.nane.netapp.com>


The makefile is setup to expect you have the entire OpenIB subversion 
tree checked out. The user level command (which is produced by 
Makefile.user) requires the user level DAT include file, udat.h, to 
build. 

The quick fix is to either checkout the entire OpenIB tree or edit 
Makefile.user to include a different directory with udat.h.

Long term, this dependency should be removed from kdaptlest.

james

On Mon, 25 Jul 2005, Hal Rosenstock wrote:

halr> Hi James,
halr> 
halr> With the latest changes, when building kdapltest as follows:
halr> 
halr> cd /path/to/svn/gen2/utils/src/linux-kernel/kdapl/dapltest/kdapl
halr> make -f Makefile.user
halr> 
halr> gcc -g3  -Wall -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations -Werror -pipe -D__KDAPLTEST__ -D__LINUX__ -D__PENTIUM__ -I../../../../../../users/jlentini/userspace/dat/include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../mdep/linux/ -I./ -o /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../Obj/dapl_getopt.o -c /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../cmd/dapl_getopt.c
halr> In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
halr> /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:36:22: dat/udat.h: No such file or directory
halr> In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:56,
halr>                  from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
halr> /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_bpool.h:51: parse error before "DAT_LMR_CONTEXT"
halr> 
halr> and other similar errors
halr> 
halr> -- Hal
halr> 


From halr at voltaire.com  Mon Jul 25 12:56:16 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Jul 2005 15:56:16 -0400
Subject: [openib-general] Re: kdapltest building
In-Reply-To: <Pine.LNX.4.61.0507251549260.5935@jlentini-linux.nane.netapp.com>
References: <1122319169.4421.7.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507251549260.5935@jlentini-linux.nane.netapp.com>
Message-ID: <1122321376.4421.19.camel@hal.voltaire.com>

On Mon, 2005-07-25 at 15:53, James Lentini wrote:
> The makefile is setup to expect you have the entire OpenIB subversion 
> tree checked out.

These are on separate branches.

> The user level command (which is produced by 
> Makefile.user) requires the user level DAT include file, udat.h, to 
> build. 
> 
> The quick fix is to either checkout the entire OpenIB tree or edit 
> Makefile.user to include a different directory with udat.h.

I worked around it.

-- Hal

> Long term, this dependency should be removed from kdaptlest.

> james
> 
> On Mon, 25 Jul 2005, Hal Rosenstock wrote:
> 
> halr> Hi James,
> halr> 
> halr> With the latest changes, when building kdapltest as follows:
> halr> 
> halr> cd /path/to/svn/gen2/utils/src/linux-kernel/kdapl/dapltest/kdapl
> halr> make -f Makefile.user
> halr> 
> halr> gcc -g3  -Wall -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations -Werror -pipe -D__KDAPLTEST__ -D__LINUX__ -D__PENTIUM__ -I../../../../../../users/jlentini/userspace/dat/include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../include/ -I/home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../mdep/linux/ -I./ -o /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../Obj/dapl_getopt.o -c /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/kdapl/../cmd/dapl_getopt.c
> halr> In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
> halr> /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:36:22: dat/udat.h: No such file or directory
> halr> In file included from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_proto.h:56,
> halr>                  from /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/cmd/dapl_getopt.c:28:
> halr> /home/hal/openib/utils/src/linux-kernel/kdapl/dapltest/include/dapl_bpool.h:51: parse error before "DAT_LMR_CONTEXT"
> halr> 
> halr> and other similar errors
> halr> 
> halr> -- Hal
> halr> 



From mst at mellanox.co.il  Mon Jul 25 15:32:00 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 26 Jul 2005 01:32:00 +0300
Subject: [openib-general] [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
Message-ID: <20050725223200.GA1545@mellanox.co.il>

Greg, Martin, does the following make sense?
If it does, should other architectures be updated as well?

---

Convert i386/pci to use io_remap_pfn_range instead of remap_pfn_range.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/arch/i386/pci/i386.c
===================================================================
--- linux-2.6.12.2.orig/arch/i386/pci/i386.c
+++ linux-2.6.12.2/arch/i386/pci/i386.c
@@ -295,9 +295,9 @@ int pci_mmap_page_range(struct pci_dev *
 	/* Write-combine setting is ignored, it is changed via the mtrr
 	 * interfaces on this platform.
 	 */
-	if (remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
-			     vma->vm_end - vma->vm_start,
-			     vma->vm_page_prot))
+	if (io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+			       vma->vm_end - vma->vm_start,
+			       vma->vm_page_prot))
 		return -EAGAIN;
 
 	return 0;


-- 
MST


From rolandd at cisco.com  Mon Jul 25 13:42:16 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 13:42:16 -0700
Subject: [openib-general] IBM eHCA Device Driver for gen2 IB stack
In-Reply-To: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
	(IBMEHCA DD's message of "Fri, 22 Jul 2005 13:41:31 +0200")
References: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
Message-ID: <52y87uk47r.fsf@topspin.com>

    IBMEHCA> Hi, we've completed the first alpha code drop of the
    IBMEHCA> Power5 IBM eHCA Device Driver for the for the gen2
    IBMEHCA> openib.org stack.  We're running IPoIB and ibv userspace
    IBMEHCA> programs successfully with this code in our lab setup.

    IBMEHCA> The source files can be downloaded from
    IBMEHCA> https://sourceforge.net/projects/ibmehcad/ ehca2_0011e

Thanks for posting this.  A few comments from a first read through.
These are in no particular order, with minor nitpicks mixed in with
more series problems.  Some more review is still needed before this
code should go upstream.

ehca (kernel driver):

 - Please use /* */ instead of // for comments

 - Can the debugging stuff be less ugly than EDEB_AV_EN() etc?
   Also the formatting of your goto labels is not very nice.  Instead
   of something like

      EHCA_REG_SMR_EXIT0:
	if (retcode == 0)

   I'd rather see

out:
	if (retcode == 0)

 - In ehca_classes.c, it would be better to avoid using vmalloc() to
   allocate your structures.

 - In ehca_common.h, all the stuff copied from from hvcall.h should be
   deleted.  Also, there are some functions that seem to use ugly
   return codes like H_Success for no good reason.

 - The definitions of p_to_u64() etc. seem unnecesary and needlessly
   obfuscated.  You can always do

	(unsigned long) ptr

   to cast a void * to u64, and

	(void *) (unsigned long) val

   to cast the other way.  I think all your uses of these functions
   would be clearer if you just open-coded the casts.

 - Similarly in ehca_qp.c,

	#define QP_ATTR_IS_SET(mask,attr)   (((mask)&(attr))!=0)

   is just needless obfuscation: just do the & directly where you need it.

 - In ehca_qp.c, is there a way we can take modqp_statetrans_table[]
   into common code and share with mthca_qp.c?

 - It seems many source files include ehca_classes_pSeries.h
   directly.  They should probably include ehca_classes.h.  Also
   ehca_classes_zSeries.h seems to be missing.

   Instead of defining P_SERIES on your command line, I think you
   should just include <linux/config.h> and test CONFIG_PPC_PSERIES
   and CONFIG_ARCH_S390 instead.

 - In ehca_main.c, ehca_hca_resources_show() violates the "one value
   per file" rule for sysfs.  Either put this in debugfs or split it
   up into separate attributes in a subdirectory.

 - What is ehca_register_pci() doing?  Why are you changing the main
   kernel's pci_dma_ops??  Shouldn't you be creating a device on your
   own virtual bus with it's own dma ops?

 - It seems ehca_nopage() accesses ehca_idr without taking the
   ehca_idr_sem, is this OK?

libehca:

 - It would probably be a good idea to use GNU autotools to build
   libehca.  In particular you want to make the destination directory
   configurable.

 - I don't think you should hard code -m64 in your CFLAGS or put
   /usr/local/lib64 anywhere -- I believe nearly everyone running
   ppc64 is using 32-bit userspace.  Have you tested 32-bit userspace
   on a 64-bit kernel?

 - I don't think the install should unconditionally delete
   /dev/infiniband and then create the device nodes -- this should
   really be handled by either udev or the distribution.

 - In ehca_reqs_core.c, get rid of all the

	#ifndef __KERNEL__
	#define ib_recv_wr ibv_recv_wr
	#define ib_send_wr ibv_send_wr
	//#define ib_ah ibv_ah
	#define ehca_av ehcau_av
	#define unlikely(x) x
	// ib_wr_opcode
	#define IB_WR_SEND IBV_WR_SEND
	#define IB_WR_SEND_WITH_IMM IBV_WR_SEND_WITH_IMM
	#define IB_WR_RDMA_WRITE IBV_WR_RDMA_WRITE
	#define IB_WR_RDMA_WRITE_WITH_IMM IBV_WR_RDMA_WRITE_WITH_IMM

   and so on.  It's not a good idea to try and use the same source for
   code that runs in the kernel and code that runs in userspace.
   Splitting your source also gets rid of stuff like

	#ifdef EHCA_USERDRIVER
				u8 *data = (u8 *) sge->addr;
	#else
				u8 *data = (u8 *) abs_to_virt(sge->addr);
	#endif
	
 - In ehca_uinit.c, is the LIBEHCA_FILE environment variable handling
   safe?  It seems that a user could trick a setuid process into doing
   something bad here.

 - In ehca_umain.c there seems to be some weird indentation:

	my_cq->ehca_cq_core.galpas.kernel.fw_handle = (u64)
	    mmap(NULL, 4096, PROT_WRITE, MAP_SHARED, context->cmd_fd,
		 ((u64) (my_cq->token) << 32) | 0x11000000);

 - Does it make sense to encapsulate to use of ppc64-specific assembly
   stuff like:

	__asm__ __volatile("dcbz 0,%0"::"r"(adr));
	__asm__ __volatile("dclst 0,%0"::"r"(adr));
	__asm__ __volatile__("sync":::"memory");	// serialize GAL register access

 - In ehca_utools.h, ehca_swapbl32 and ehca_swapbl16 seem kind of useless.
   Why not just use ntohl/ntohs directly?

Thanks,
  Roland


From tduffy at sun.com  Mon Jul 25 13:45:27 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 25 Jul 2005 13:45:27 -0700
Subject: [openib-general] [PATCH] SDP: fix oops with port reuse
In-Reply-To: <20050725122015.A7871@topspin.com>
References: <1122315500.27947.25.camel@duffman>
	<20050725122015.A7871@topspin.com>
Message-ID: <1122324327.27947.30.camel@duffman>

On Mon, 2005-07-25 at 12:20 -0700, Libor Michalek wrote:
>   The original code at the begining of sdp_inet_port_put() was checking
> to see if the connection was already in the bind list, and returning an
> error if it was not in the list. The error check that's there now is
> doing something different, and we return success if the connection
> is not in the bind list and has a put() done on it.
> 
>   If we are regularly doing a put on a socket not in the bind list,
> the function should be made void, and the check for list membership
> not generate an error...

Something like this?

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_inet.c
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_inet.c	(revision 2909)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_inet.c	(working copy)
@@ -1100,7 +1100,7 @@ static int sdp_inet_setopt(struct socket
 							SDP_MSG_HDR_SIZE));
 		break;
 	case SDP_UNBIND:
-		result = sdp_inet_port_put(conn);
+		sdp_inet_port_put(conn);
 		break;
 	default:
 		sdp_warn("SETSOCKOPT unimplemented option <%d:%d> conn <%d>.",
Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_proto.h
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(revision 2909)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(working copy)
@@ -287,7 +287,7 @@ struct sdp_sock *sdp_inet_listen_lookup(
 
 int sdp_inet_port_get(struct sdp_sock *conn, u16 port);
 
-int sdp_inet_port_put(struct sdp_sock *conn);
+void sdp_inet_port_put(struct sdp_sock *conn);
 
 void sdp_inet_port_inherit(struct sdp_sock *parent, struct sdp_sock *child);
 
Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 2909)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -490,19 +490,17 @@ done:
 /*
  * sdp_inet_port_put - unbind a socket from a port.
  */
-int sdp_inet_port_put(struct sdp_sock *conn)
+void sdp_inet_port_put(struct sdp_sock *conn)
 {
 	unsigned long flags;
-
-	if (list_empty(&dev_root_s.bind_list))
-		return -EADDRNOTAVAIL;
+	struct sdp_sock *i, *j;
 
 	spin_lock_irqsave(&dev_root_s.bind_lock, flags);
-	list_del(&conn->bind_next);
+	list_for_each_entry_safe(i, j, &dev_root_s.bind_list, bind_next)
+		if (conn == i)
+			list_del(&conn->bind_next);
 	conn->src_port = 0;
 	spin_unlock_irqrestore(&dev_root_s.bind_lock, flags);
-
-	return 0;
 }
 
 /*
@@ -694,7 +692,7 @@ void sdp_conn_put(struct sdp_sock *conn)
 	/*
 	 * If the socket is bound, return the port
 	 */
-	(void)sdp_inet_port_put(conn);
+	sdp_inet_port_put(conn);
 
 	sdp_conn_stat_dump(conn);
 	/*



From rolandd at cisco.com  Mon Jul 25 13:47:58 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 13:47:58 -0700
Subject: [openib-general] IBM eHCA Device Driver for gen2 IB stack
In-Reply-To: <52y87uk47r.fsf@topspin.com> (Roland Dreier's message of "Mon,
	25 Jul 2005 13:42:16 -0700")
References: <OF5A200DE0.CDF32BA0-ONC1257046.003B9E89-C1257046.00403954@de.ibm.com>
	<52y87uk47r.fsf@topspin.com>
Message-ID: <52u0iik3y9.fsf@topspin.com>

    Roland>  - What is ehca_register_pci() doing?  Why are you
    Roland> changing the main kernel's pci_dma_ops??  Shouldn't you be
    Roland> creating a device on your own virtual bus with it's own
    Roland> dma ops?

By the way, have you gotten anywhere on changing ppc64's dma_addr_t
from u32 to u64?  I spoke to some of the other pSeries guys at the
kernel summit last week, and they were already supportive of this
change for independent reasons.

 - R.


From pw at osc.edu  Mon Jul 25 14:01:36 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Mon, 25 Jul 2005 17:01:36 -0400
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050725155846.GD13014@minantech.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
Message-ID: <20050725210136.GB21836@osc.edu>

glebn at voltaire.com wrote on Mon, 25 Jul 2005 18:58 +0300:
> On Mon, Jul 25, 2005 at 11:37:29AM -0400, Pete Wyckoff wrote:
> > I like your signal idea but am not fond of the signal mechanism,
> > especially as we (the library) can't trust the user not to break it.
> > Signaling a separate thread has the same race problem.  I'd prefer
> > to export a counter from the kernel that the user application can
> > see.  The kernel would atomically increment it when there is an
> > unread event, and the app can check the variable to decide if the
> > read() is necessary.
>
> I also don't like signal much but... 
> How userspace will read kernel counter without system call? Or do you
> mean kernel will increment counter resided in userspace?

I'm using copy_to_user() to shadow the number of available VM events
in the queue to an int32 in the user application.  Had considered
mapping a shared page, but that seems like overkill for this tiny
bit of information.  Copy_to_user() is fairly fast and is only
invoked on (hopefully rare) VM activity.  Fortunately VM events
appear only to happen in the context of the user app so this is
possible.  And it drops the fastpath overhead before each MPI_Send()
to around 6 ns on a 2.4 GHz P4 when everything is in cache.

		-- Pete


From mst at mellanox.co.il  Mon Jul 25 14:11:46 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 26 Jul 2005 00:11:46 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050725210136.GB21836@osc.edu>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
	<20050725210136.GB21836@osc.edu>
Message-ID: <20050725211146.GA13882@mellanox.co.il>

Quoting r. Pete Wyckoff <pw at osc.edu>:
> I'm using copy_to_user() to shadow the number of available VM events
> in the queue to an int32 in the user application.

So, do you keep all VM events in this queue, forever?
Can a userspace application make this queue grow very large?

-- 
MST


From jlentini at netapp.com  Mon Jul 25 14:24:15 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 25 Jul 2005 17:24:15 -0400 (EDT)
Subject: [openib-general] [ANNOUNCE] new kDAPL location
In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35F0D@mtvex01.mtv.mtl.com>
References: <91DB792C7985D411BEC300B40080D29CC35F0D@mtvex01.mtv.mtl.com>
Message-ID: <Pine.LNX.4.61.0507251723280.12296@jlentini-linux.nane.netapp.com>


On Mon, 25 Jul 2005, Itamar Rabenstein wrote:

> from the readme:
> ...
> Current Limitations
>
> 1. Only kdapltest -T T, -T Q, and -T S currently supported.
>   Other options are not.
>
> Note that -T P is also supported.

Thanks for pointing that out. I've updated the README.


From tduffy at sun.com  Mon Jul 25 14:25:45 2005
From: tduffy at sun.com (Tom Duffy)
Date: Mon, 25 Jul 2005 14:25:45 -0700
Subject: [openib-general] [PATCH] SDP: priority is now unsigned int
Message-ID: <1122326745.420.7.camel@duffman>

sk_alloc() takes an "unsigned int __nocast priority" instead of an int.
This was changed in 2.6.13-rc3.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_proto.h
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(revision 2909)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_proto.h	(working copy)
@@ -260,7 +260,7 @@ int sdp_proc_dump_device(char *buffer,
 
 struct sdp_sock *sdp_conn_table_lookup(s32 entry);
 
-struct sdp_sock *sdp_conn_alloc(int priority);
+struct sdp_sock *sdp_conn_alloc(unsigned int priority);
 
 int sdp_conn_alloc_ib(struct sdp_sock *conn,
 		      struct ib_device *device, 
Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_conn.c
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(revision 2909)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -1132,7 +1130,7 @@ extern struct proto sdp_sk_proto;
 /*
  * sdp_conn_alloc - allocate a new socket, and init.
  */
-struct sdp_sock *sdp_conn_alloc(int priority)
+struct sdp_sock *sdp_conn_alloc(unsigned int priority)
 {
 	struct sdp_sock *conn;
 	struct sock *sk;
@@ -1140,7 +1138,7 @@ struct sdp_sock *sdp_conn_alloc(int prio
 
 	sk = sk_alloc(dev_root_s.proto, priority, &sdp_sk_proto, 1);
 	if (!sk) {
-		sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>",
+		sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%u>",
 			     dev_root_s.proto, priority);
 		return NULL;
 	}



From jlentini at netapp.com  Mon Jul 25 14:28:54 2005
From: jlentini at netapp.com (James Lentini)
Date: Mon, 25 Jul 2005 17:28:54 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] [kdapl]: Update README for IBAT being
	part of IB core
In-Reply-To: <1122319404.4421.10.camel@hal.voltaire.com>
References: <1122319404.4421.10.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507251728400.12296@jlentini-linux.nane.netapp.com>


Committed in revision 2911.


From gshipman at lanl.gov  Mon Jul 25 14:32:26 2005
From: gshipman at lanl.gov (Galen Shipman)
Date: Mon, 25 Jul 2005 15:32:26 -0600
Subject: [openib-general] Shared Receive Queue
Message-ID: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>

Hello,

I am working on some software using openib gen2 and I want to be able 
to optionally use the shared receive queue. Using ibv_create_srq, etc. 
I haven't found this API in the current trunk, what is the status of 
the shared receive queue and if known, what type of performance hit 
should I expect to take by using it?

Thanks,

Galen



From mst at mellanox.co.il  Mon Jul 25 14:34:27 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 26 Jul 2005 00:34:27 +0300
Subject: [openib-general] Re: Shared Receive Queue
In-Reply-To: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
References: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
Message-ID: <20050725213427.GB13882@mellanox.co.il>

Quoting r. Galen Shipman <gshipman at lanl.gov>:
> Subject: Shared Receive Queue
> 
> Hello,
> 
> I am working on some software using openib gen2 and I want to be able 
> to optionally use the shared receive queue. Using ibv_create_srq, etc. 
> I haven't found this API in the current trunk, what is the status of 
> the shared receive queue and if known, what type of performance hit 
> should I expect to take by using it?
> 
> Thanks,
> 
> Galen

AFAIK, shared receive queue isnt yet supported in gen2.

-- 
MST


From halr at voltaire.com  Mon Jul 25 14:47:25 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 25 Jul 2005 17:47:25 -0400
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
References: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
Message-ID: <1122328044.4421.100.camel@hal.voltaire.com>

On Mon, 2005-07-25 at 17:32, Galen Shipman wrote:
> Hello,
> 
> I am working on some software using openib gen2 and I want to be able 
> to optionally use the shared receive queue. Using ibv_create_srq, etc. 
> I haven't found this API in the current trunk, what is the status of 
> the shared receive queue and if known, what type of performance hit 
> should I expect to take by using it?

A proposed API was written up a while ago (mid June). Not sure how far
along the implementation is.

-- Hal



From rolandd at cisco.com  Mon Jul 25 15:05:32 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 15:05:32 -0700
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <1122328044.4421.100.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "25 Jul 2005 17:47:25 -0400")
References: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
	<1122328044.4421.100.camel@hal.voltaire.com>
Message-ID: <527jfek0cz.fsf@topspin.com>

    Hal> A proposed API was written up a while ago (mid June). Not
    Hal> sure how far along the implementation is.

I've not made much progress since then but I'll start working
seriously soon.

 - R.


From rolandd at cisco.com  Mon Jul 25 15:14:54 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 15:14:54 -0700
Subject: [openib-general] Re: [PATH] problem dlopen libibverbs.so
In-Reply-To: <20050721141647.GN12698@mellanox.co.il> (Michael S. Tsirkin's
	message of "Thu, 21 Jul 2005 17:16:47 +0300")
References: <20050720135633.GN28598@minantech.com>
	<20050720142422.GA27124@mellanox.co.il>
	<20050720142829.GO28598@minantech.com>
	<20050720145728.GA27398@mellanox.co.il>
	<20050720150204.GP28598@minantech.com>
	<20050720152349.GE27398@mellanox.co.il>
	<20050720152736.GQ28598@minantech.com>
	<20050720154509.GG27398@mellanox.co.il> <52d5pdwif3.fsf@topspin.com>
	<20050721141647.GN12698@mellanox.co.il>
Message-ID: <523bq2jzxd.fsf@topspin.com>

Thanks, I merged this.

 - R.


From rolandd at cisco.com  Mon Jul 25 15:16:16 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 15:16:16 -0700
Subject: [openib-general] Re: why does the value of the node_guid don't have
 the machine en dian ess?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A25B4@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Thu, 21 Jul 2005 11:58:29 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A25B4@mtlex01.yok.mtl.com>
Message-ID: <52y87uilan.fsf@topspin.com>

    Dotan> I expect the value to have the same endian-ness when
    Dotan> printed with %llx and when read from sysfs.

I don't think this is right.  sysfs is showing the GUID as a string.
On the other hand, I think it makes sense for the query verb to return
it in network order.

 - R.


From arlin.r.davis at intel.com  Mon Jul 25 15:20:29 2005
From: arlin.r.davis at intel.com (Arlin Davis)
Date: Mon, 25 Jul 2005 15:20:29 -0700
Subject: [openib-general] [PATCH] uDAPL dapl_os_wait_object_wait() 
Message-ID: <ORSMSX401EXUIEAOeIi0000006a@orsmsx401.amr.corp.intel.com>

James,

Here is a patch to fix dapl_os_wait_object_wait() returning 
EINVAL when passing nsec == 1000000000 to pthread_cond_timedwait(). 
Hit a rare case where _microsecs was exactly 1000000.

Thanks,

-arlin

Signed-off by: Arlin Davis <ardavis at ichips.intel.com>

Index: dapl/udapl/linux/dapl_osd.c
===================================================================
--- dapl/udapl/linux/dapl_osd.c (revision 2899)
+++ dapl/udapl/linux/dapl_osd.c (working copy)
@@ -242,16 +242,9 @@

        gettimeofday (&now, &tz);
        microsecs = now.tv_usec + (timeout_val % 1000000);
-       if (microsecs > 1000000)
-       {
-           now.tv_sec = now.tv_sec + timeout_val / 1000000 + 1;
-           now.tv_usec = microsecs - 1000000;
-       }
-       else
-       {
-           now.tv_sec = now.tv_sec + timeout_val / 1000000;
-           now.tv_usec = microsecs;
-       }
+
+       now.tv_sec = now.tv_sec + timeout_val/1000000 + (microsecs/1000000);
+       now.tv_usec = microsecs % 1000000;

        /* Convert timeval to timespec */
        future.tv_sec = now.tv_sec;




From rolandd at cisco.com  Mon Jul 25 15:23:19 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 15:23:19 -0700
Subject: [openib-general] calling to ibv_create_qp with big number in
	qp_init_attr.cap.max_ inline_data never return
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Mon, 18 Jul 2005 15:15:07 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A2198@mtlex01.yok.mtl.com>
Message-ID: <52mzoaikyw.fsf@topspin.com>

I just committed the change below to libmthca, which should fix this hang.

 - R.

--- verbs.c	(revision 2904)
+++ verbs.c	(working copy)
@@ -284,6 +284,14 @@ struct ibv_qp *mthca_create_qp(struct ib
 	struct mthca_qp       *qp;
 	int                    ret;
 
+	/* Sanity check QP size before proceeding */
+	if (attr->cap.max_send_wr     > 65536 ||
+	    attr->cap.max_recv_wr     > 65536 ||
+	    attr->cap.max_send_sge    > 64    ||
+	    attr->cap.max_recv_sge    > 64    ||
+	    attr->cap.max_inline_data > 1024)
+		return NULL;
+
 	qp = malloc(sizeof *qp);
 	if (!qp)
 		return NULL;


From rolandd at cisco.com  Mon Jul 25 15:25:04 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 15:25:04 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <42D84618.90807@ichips.intel.com> (Sean Hefty's message of
	"Fri, 15 Jul 2005 16:26:16 -0700")
References: <1121203934.14638.27.camel@duffman>
	<20050712221725.GB14316@mellanox.co.il>
	<200507130248.08387.adobriyan@gmail.com>
	<42D83A5B.5090808@ichips.intel.com> <52r7dz661d.fsf@topspin.com>
	<52irzb65sq.fsf@topspin.com> <42D84618.90807@ichips.intel.com>
Message-ID: <52iryyikvz.fsf@topspin.com>

I just checked in a big batch of cleanups that get rid of most of the
mthca warnings.  Do you want to check in your changes too?

 - R.


From rolandd at cisco.com  Mon Jul 25 15:17:19 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 15:17:19 -0700
Subject: [openib-general] Re: where can i find functions that "convert"
 enumerated values t o string?
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6084A236D@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Tue, 19 Jul 2005 18:04:06 +0300")
References: <506C3D7B14CDD411A52C00025558DED6084A236D@mtlex01.yok.mtl.com>
Message-ID: <52u0iiil8w.fsf@topspin.com>

    Dotan> If I'll send you functions that does the work, will you add
    Dotan> them to the driver?

If we can do this cleanly without bloating the source or compiled code
too much, I think it's OK.  What do you see applications using these
strings for?

 - R.


From mshefty at ichips.intel.com  Mon Jul 25 15:36:41 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Mon, 25 Jul 2005 15:36:41 -0700
Subject: [openib-general] Re: [PATCH 3/27] Add MAD helper functions
In-Reply-To: <52iryyikvz.fsf@topspin.com>
References: <1121203934.14638.27.camel@duffman>	<20050712221725.GB14316@mellanox.co.il>	<200507130248.08387.adobriyan@gmail.com>	<42D83A5B.5090808@ichips.intel.com>
	<52r7dz661d.fsf@topspin.com>	<52irzb65sq.fsf@topspin.com>
	<42D84618.90807@ichips.intel.com> <52iryyikvz.fsf@topspin.com>
Message-ID: <42E56979.1050302@ichips.intel.com>

Roland Dreier wrote:
> I just checked in a big batch of cleanups that get rid of most of the
> mthca warnings.  Do you want to check in your changes too?

I noticed that you committed those changes.  Thanks.  I haven't finished my 
changes yet, but will try to get to them tomorrow or Wednesday.  (I've been 
working on OpenIB presentation stuff, but I can make this a priority if 
needed.)  I have fixes for: cmpost, madeye, grmpp, mad, mad_rmpp, smi, and 
ud_header; and incomplete fixes for: cm, sa, umad, ucm, and user_mad.  I 
still need to test, but I think that this should cover the rest of the warnings.

- Sean


From pw at osc.edu  Mon Jul 25 16:26:24 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Mon, 25 Jul 2005 19:26:24 -0400
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050725211146.GA13882@mellanox.co.il>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
	<20050725210136.GB21836@osc.edu>
	<20050725211146.GA13882@mellanox.co.il>
Message-ID: <20050725232624.GA6812@baa>

mst at mellanox.co.il wrote on Tue, 26 Jul 2005 00:11 +0300:
> Quoting r. Pete Wyckoff <pw at osc.edu>:
> > I'm using copy_to_user() to shadow the number of available VM events
> > in the queue to an int32 in the user application.
> 
> So, do you keep all VM events in this queue, forever?
> Can a userspace application make this queue grow very large?

No, of course not.  Each event is currently 8 bytes:  hca handle and
memory handle, as supplied at registration time.  The size of the queue
is configurable, and only needs to be big enough to hold any forced
deregistrations that happen from VM activity like sbrk(<0) and munmap()
between MPI (or other library) calls.  (If you add the signaling
mechanism proposed by Gleb, perhaps with a separate thread, it might be
possible to drain the queue immediately as it approaches fullness.)

If there are more invalidations than will fit in the queue, the library
gets an error on the next read() from the module's character device.  At
that point the library should deregister all dynamic buffers, wipe its
registration cache, re-open the device (perhaps with a larger queue
depth) and continue with a now empty registration cache.  Actually it
must preserve and re-inform the kernel module about the registrations that
are still in use, but all this is knowable.

Hopefully this never happens, but abusive benchmarks might trigger an
overflow:

    buf = mmap(0, 1000 * 1024, ...)
    for (i=0; i<1000; i++)
	MPI_Send(buf + i * 1024, 1024, ...)
    munmap(buf, 1000 * 1024)

Each MPI_Send causes the library to register a separate 1 kB chunk.  The
munmap() will force the invalidation of all those cached registrations,
putting 1000 events in the queue at once.  We hope people don't write
apps like this, but if they do, at least the overall mechanism fails in
a way that doesn't generate any user-observable fault.  You just get the
same lousy throughput as if there were no registration cache at all.

		-- Pete


From rolandd at cisco.com  Mon Jul 25 16:36:53 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Mon, 25 Jul 2005 16:36:53 -0700
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050725232624.GA6812@baa> (Pete Wyckoff's message of "Mon, 25
	Jul 2005 19:26:24 -0400")
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
	<20050725210136.GB21836@osc.edu>
	<20050725211146.GA13882@mellanox.co.il> <20050725232624.GA6812@baa>
Message-ID: <52ackaihka.fsf@topspin.com>

    Pete> No, of course not.  Each event is currently 8 bytes: hca
    Pete> handle and memory handle, as supplied at registration time.
    Pete> The size of the queue is configurable, and only needs to be
    Pete> big enough to hold any forced deregistrations that happen
    Pete> from VM activity like sbrk(<0) and munmap() between MPI (or
    Pete> other library) calls.  (If you add the signaling mechanism
    Pete> proposed by Gleb, perhaps with a separate thread, it might
    Pete> be possible to drain the queue immediately as it approaches
    Pete> fullness.)

Hmm, thinking about this some more, it occurs to me that it might be
possible for the kernel and userspace to work together even more
efficiently.  Does the following make sense?

When memory is registered, userspace tells the kernel VM activity
monitor where its registration cache data structure for that
registration is.  If the kernel detects some VM activity that affects
that registration, then it can invalidate it directly in the userspace
cache data structure.

I guess there are two issues to resolve here:

 - Are there any races between the kernel marking something invalid
   and userspace marking it valid again?

 - How do we keep track of a process's registrations in the kernel so
   that we can efficiently find which (if any) registrations are
   affected by a VM operation?  (Maybe you've solved this already)

 - R.


From qfkuf at primary.net  Mon Jul 25 22:41:06 2005
From: qfkuf at primary.net (Earnest)
Date: Tue, 26 Jul 2005 00:41:06 -0500
Subject: [openib-general] abuilding but chromatography
Message-ID: <20050726045519.E61CA22834D@openib.ca.sandia.gov>

%URL
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050726/48018042/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: otetfzhc1.gif
Type: image/jpeg
Size: 1894 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050726/48018042/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: otetfzhc2.gif
Type: image/jpeg
Size: 1976 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050726/48018042/attachment-0001.jpe>

From glebn at voltaire.com  Tue Jul 26 00:05:23 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Tue, 26 Jul 2005 10:05:23 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <523bq2n874.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com> <523bq2n874.fsf@topspin.com>
Message-ID: <20050726070523.GF13014@minantech.com>

On Mon, Jul 25, 2005 at 09:47:59AM -0700, Roland Dreier wrote:
>     Gleb> Can you elaborate on kernel upcalls mechanism?
> 
> Not sure what you mean by the question.  I was just trying to say that
> in general when the kernel has some information it wants to push to
> userspace, it's probably best for userspace to get it via a read() on
> a file descriptor rather than for the kernel to queue a signal.
> 
This is what Pete did. We are back to on syscall on each registration.
And the worst case is three syscalls. First to check that mapping is valid,
second to deregister memory if it is not and third register new memory.

>     Gleb> This is responsibility of the user to make sure he is not
>     Gleb> unmaps or frees memory that is currently in use.
> 
> Any registration caching scheme has to deal with a user using memory
> X, freeing memory X, and then allocating memory Y at the same virtual
> address as X.
> 
Not exactly. When user frees memory by calling free() the memory is not
unmapped from process address space by libc. We want to cache the
registrations in this memory. We don't want to cache registration
across mmap/munmap/mmap (it is much harder to do if at all possible).
When libc unmaps memory by sbrk() or munmap the memory guaranteed to be
not used by correct program so it is safe to deregister it if we catch 
this event.

--
			Gleb.


From glebn at voltaire.com  Tue Jul 26 00:18:53 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Tue, 26 Jul 2005 10:18:53 +0300
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <52ackaihka.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
	<20050725210136.GB21836@osc.edu>
	<20050725211146.GA13882@mellanox.co.il>
	<20050725232624.GA6812@baa> <52ackaihka.fsf@topspin.com>
Message-ID: <20050726071853.GG13014@minantech.com>

On Mon, Jul 25, 2005 at 04:36:53PM -0700, Roland Dreier wrote:
>     Pete> No, of course not.  Each event is currently 8 bytes: hca
>     Pete> handle and memory handle, as supplied at registration time.
>     Pete> The size of the queue is configurable, and only needs to be
>     Pete> big enough to hold any forced deregistrations that happen
>     Pete> from VM activity like sbrk(<0) and munmap() between MPI (or
>     Pete> other library) calls.  (If you add the signaling mechanism
>     Pete> proposed by Gleb, perhaps with a separate thread, it might
>     Pete> be possible to drain the queue immediately as it approaches
>     Pete> fullness.)
> 
> Hmm, thinking about this some more, it occurs to me that it might be
> possible for the kernel and userspace to work together even more
> efficiently.  Does the following make sense?
> 
> When memory is registered, userspace tells the kernel VM activity
> monitor where its registration cache data structure for that
> registration is.  If the kernel detects some VM activity that affects
> that registration, then it can invalidate it directly in the userspace
> cache data structure.
It may set 'not valid' flag in struct ibv_mr. The question is how can you do it and
not penalise application that don't use memory registration.

> 
> I guess there are two issues to resolve here:
> 
>  - Are there any races between the kernel marking something invalid
>    and userspace marking it valid again?
As I said before we don't cache across mman/munmap so something that 
became invalid cannot became valid again.

--
			Gleb.


From hugh at veritas.com  Tue Jul 26 05:30:32 2005
From: hugh at veritas.com (Hugh Dickins)
Date: Tue, 26 Jul 2005 13:30:32 +0100 (BST)
Subject: [openib-general] Re: [PATCH repost] PROT_DONTCOPY: ifiniband uverbs
	fork support
In-Reply-To: <20050725171928.GC12206@mellanox.co.il>
References: <20050719165542.GB16028@mellanox.co.il>
	<20050725171928.GC12206@mellanox.co.il>
Message-ID: <Pine.LNX.4.61.0507261312460.16985@goblin.wat.veritas.com>

On Mon, 25 Jul 2005, Michael S. Tsirkin wrote:
> 
> This patch adds PROT_DONTCOPY to mmap and mprotect, to set VM_DONTCOPY on vma.
> This is needed for infiniband userspace i/o, where we need to protect against
>   - the child process accessing the parent hardware page
>   - the parent registered address (on which the driver did get_user_pages)
>     getting remapped to another page by COW
> One can imagine other uses, e.g. combined with mlock for real-time or security.

I don't much like it, but it does solve a real problem in an efficient way.

Partly I don't like it because of "PROT_DONTCOPY" itself: I'm queasy
about protection flags which are not protection flags, though I find
you're not the first to go down that road.

Is the patch tested?  I've not tried, but suspect the newflags shift
and mask won't work for it.  And I don't look forward to your adding
VM_MAYDONTCOPY - ugh!

> @@ -246,7 +246,7 @@ sys_mprotect(unsigned long start, size_t
>  			goto out;
>  		}
>  
> -		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC));
> +		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC | VM_DONTCOPY));
>  
>  		if ((newflags & ~(newflags >> 4)) & 0xf) {
>  			error = -EACCES;

I rather think it would all be more cleanly handled by dropping the mmap
and mprotect changes, adding an madvise instead.  Though you may object
that madvise is for optional behaviours, and this should be mandatory.

The other reason I dislike the patch is that the problem it fixes is
an old one, and I'd much rather have get_user_pages fix it for itself,
than ask the developer to do some additional magic to get around it.

But I've failed to work out a simple efficient alternative, which won't
burden the vast majority of get_user_pages usages which never hit the
issue.  So your way is probably appropriate, but I'd prefer madvise.

(Sorry, I won't be able to discuss further for a couple of days.)

Hugh


From mst at mellanox.co.il  Tue Jul 26 06:35:53 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 26 Jul 2005 16:35:53 +0300
Subject: [openib-general] Re: [PATCH repost] PROT_DONTCOPY: ifiniband uverbs
	fork support
In-Reply-To: <Pine.LNX.4.61.0507261312460.16985@goblin.wat.veritas.com>
References: <20050719165542.GB16028@mellanox.co.il>
	<20050725171928.GC12206@mellanox.co.il>
	<Pine.LNX.4.61.0507261312460.16985@goblin.wat.veritas.com>
Message-ID: <20050726133553.GA22276@mellanox.co.il>

Hi, Hugh!
Thanks for the comments.

Quoting Hugh Dickins <hugh at veritas.com>:
> Subject: Re: [PATCH repost] PROT_DONTCOPY: ifiniband uverbs fork support
> 
> On Mon, 25 Jul 2005, Michael S. Tsirkin wrote:
> > 
> > This patch adds PROT_DONTCOPY to mmap and mprotect, to set VM_DONTCOPY on vma.
> > This is needed for infiniband userspace i/o, where we need to protect against
> >   - the child process accessing the parent hardware page
> >   - the parent registered address (on which the driver did get_user_pages)
> >     getting remapped to another page by COW
> > One can imagine other uses, e.g. combined with mlock for real-time or security.
> 
> I don't much like it, but it does solve a real problem in an efficient way.
> 
> Partly I don't like it because of "PROT_DONTCOPY" itself: I'm queasy
> about protection flags which are not protection flags, though I find
> you're not the first to go down that road.

Yes. Compare with PROT_GROWSDOWN and such.

> Is the patch tested?  I've not tried, but suspect the newflags shift
> and mask won't work for it.

I tested this patch. I didnt test all thinkable configurations of
flags though - what do you mean by "newflags shift and mask"?

> And I don't look forward to your adding
> VM_MAYDONTCOPY - ugh!

We already have VM_DONTCOPY. Why would we need VM_MAYDONTCOPY and what
would it do?

>
>
> > @@ -246,7 +246,7 @@ sys_mprotect(unsigned long start, size_t
> >  			goto out;
> >  		}
> >  
> > -		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC));
> > +		newflags = vm_flags | (vma->vm_flags & ~(VM_READ | VM_WRITE | VM_EXEC | VM_DONTCOPY));
> >  
> >  		if ((newflags & ~(newflags >> 4)) & 0xf) {
> >  			error = -EACCES;
> 
> I rather think it would all be more cleanly handled by dropping the mmap
> and mprotect changes,

Well, mmap would be much better off if VM_DONTCOPY is set atomically, since
a process may fork after mmap is called but before madvise.

> adding an madvise instead.

I'm not opposed to this, on principle. But see below.

> Though you may object
> that madvise is for optional behaviours, and this should be mandatory.

What about a new system call?
Or a flag for mprotect that effectively turns it into a new system call?
Something like PROT_EXTENDED?

> The other reason I dislike the patch is that the problem it fixes is
> an old one, and I'd much rather have get_user_pages fix it for itself,

Please note that the problem this attempts to solve is not limited
to pages locked by get_user_pages: in an infiniband userspace initiator,
a hardware page is mapped into process memory and must not be inherited
by a child processes, otherwise hardware protection breaks.

> than ask the developer to do some additional magic to get around it.
>
> But I've failed to work out a simple efficient alternative, which won't
> burden the vast majority of get_user_pages usages which never hit the
> issue.

They dont hit it if they keep the mm semaphore, or if they only lock
pages for read.

> So your way is probably appropriate, but I'd prefer madvise.

The difficulty with changing get_user_pages, as I see it, is that
you wont be able to get away with a single DONTCOPY bit - you'll need
a full reference count for each page, no less.

> (Sorry, I won't be able to discuss further for a couple of days.)
> 
> Hugh
> 

Well, madvise currently cant break/merge VMAs, which is required
for VM_DONTCOPY. And it seems like making madvise do this opens
a whole cans of worms.

Hugh, so the patch is likely to be bigger in the madvise approach.
Considering this, and the fact that a full solution has to add
a flag to mmap, anyway, do you still think madvise is really the best way
to do it?


Regarding solving the problem automagically by get_user_pages:

What about a new VM_COPYONFORK flag, to trigger the old unix
behaviour of copying the vma on fork and a flag for get_user_pages that sets it?
Only users that dont keep the mm semaphore around
the get_user_pages/put_page operation would use this flag, others
would be unaffected. The flag will stay on until the VMA is destroyed.

MST


-- 
MST


From gshipman at lanl.gov  Tue Jul 26 07:34:49 2005
From: gshipman at lanl.gov (Galen M. Shipman)
Date: Tue, 26 Jul 2005 08:34:49 -0600
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <527jfek0cz.fsf@topspin.com>
References: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
	<1122328044.4421.100.camel@hal.voltaire.com>
	<527jfek0cz.fsf@topspin.com>
Message-ID: <A044E70E-88BE-4B1E-B66A-2E4611D7B39C@lanl.gov>

Hi Roland,

We have software which uses the mellanox shared receive queue and we  
are seeing a substantial performance penalty (on the order of 4-5  
usec on 0 byte messages). I don't know all the technical details but  
if you could keep this in mind when working on the shared receive  
queue perhaps the performance penalty could be much less in gen2.

Thanks,

Galen


On Jul 25, 2005, at 4:05 PM, Roland Dreier wrote:

>     Hal> A proposed API was written up a while ago (mid June). Not
>     Hal> sure how far along the implementation is.
>
> I've not made much progress since then but I'll start working
> seriously soon.
>
>  - R.
>



From rolandd at cisco.com  Tue Jul 26 07:38:10 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 26 Jul 2005 07:38:10 -0700
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <A044E70E-88BE-4B1E-B66A-2E4611D7B39C@lanl.gov> (Galen M.
	Shipman's message of "Tue, 26 Jul 2005 08:34:49 -0600")
References: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
	<1122328044.4421.100.camel@hal.voltaire.com>
	<527jfek0cz.fsf@topspin.com>
	<A044E70E-88BE-4B1E-B66A-2E4611D7B39C@lanl.gov>
Message-ID: <52r7dlhbu5.fsf@topspin.com>

    Galen> We have software which uses the mellanox shared receive
    Galen> queue and we are seeing a substantial performance penalty
    Galen> (on the order of 4-5 usec on 0 byte messages). I don't know
    Galen> all the technical details but if you could keep this in
    Galen> mind when working on the shared receive queue perhaps the
    Galen> performance penalty could be much less in gen2.

Is this performance penalty in the software?  It seems much more
likely that the issue is in the hardware implementation of SRQ.

 - R.


From tziporet at mellanox.co.il  Tue Jul 26 07:46:12 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Tue, 26 Jul 2005 17:46:12 +0300
Subject: [openib-general] Shared Receive Queue
Message-ID: <506C3D7B14CDD411A52C00025558DED6085BCFAA@mtlex01.yok.mtl.com>

which HW & FW are you using?

-----Original Message-----
From: Roland Dreier [mailto:rolandd at cisco.com]
Sent: Tuesday, July 26, 2005 5:38 PM
To: Galen M. Shipman
Cc: openib-general at openib.org
Subject: Re: [openib-general] Shared Receive Queue


    Galen> We have software which uses the mellanox shared receive
    Galen> queue and we are seeing a substantial performance penalty
    Galen> (on the order of 4-5 usec on 0 byte messages). I don't know
    Galen> all the technical details but if you could keep this in
    Galen> mind when working on the shared receive queue perhaps the
    Galen> performance penalty could be much less in gen2.

Is this performance penalty in the software?  It seems much more
likely that the issue is in the hardware implementation of SRQ.

 - R.
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050726/903f1cc5/attachment.html>

From ogerlitz at voltaire.com  Tue Jul 26 07:49:06 2005
From: ogerlitz at voltaire.com (Or Gerlitz)
Date: Tue, 26 Jul 2005 17:49:06 +0300
Subject: [openib-general] Shared Receive Queue
Message-ID: <D4F8F0B3820E754C887699BEF26A894094D24B@taurus.voltaire.com>

> We have software which uses the mellanox shared receive queue and we
> are seeing a substantial performance penalty (on the order of 4-5
> usec on 0 byte messages).

Are you comparing code that sends short messages either by rdma write
where the 
reciever does polling on memory (as done in some IB MPIs "short message
protocols")
to code that uses send/recv with SRQ --or-- code that does send/recv
where in one run 
the recv is to "non shared" queue (that is regular QP) and in the other
run to SRQ (QP attached to SRQ)?

Or.



From gshipman at lanl.gov  Tue Jul 26 07:56:39 2005
From: gshipman at lanl.gov (Galen M. Shipman)
Date: Tue, 26 Jul 2005 08:56:39 -0600
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <D4F8F0B3820E754C887699BEF26A894094D24B@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A894094D24B@taurus.voltaire.com>
Message-ID: <4AE4C7D7-03F1-4728-BB80-90FDFC37400E@lanl.gov>


I am using our own software which uses send receive in both cases. As  
I said the code paths are exactly the same except for posting of  
receive descriptors as well as initialization.

- Galen

On Jul 26, 2005, at 8:49 AM, Or Gerlitz wrote:

>> We have software which uses the mellanox shared receive queue and we
>> are seeing a substantial performance penalty (on the order of 4-5
>> usec on 0 byte messages).
>>
>
> Are you comparing code that sends short messages either by rdma write
> where the
> reciever does polling on memory (as done in some IB MPIs "short  
> message
> protocols")
> to code that uses send/recv with SRQ --or-- code that does send/recv
> where in one run
> the recv is to "non shared" queue (that is regular QP) and in the  
> other
> run to SRQ (QP attached to SRQ)?
>
> Or.
>
>



From ogerlitz at voltaire.com  Tue Jul 26 08:02:29 2005
From: ogerlitz at voltaire.com (Or Gerlitz)
Date: Tue, 26 Jul 2005 18:02:29 +0300 (IDT)
Subject: [openib-general] uVerbs and multiple cq_event support? 
Message-ID: <Pine.LNX.4.44.0507261802130.16120-100000@zuben>

Arlin> If the thinking is one per CQ then is fd's the way to 
Arlin> go since we can have 1000's of CQs?

First, in the IB user space apps (eg "IB MPI" and "IB DataBase IPC") 
i am aware to there's either --one-- or --two-- CQs attached to all 
the QPs. This design simplifies the app and saves interrupts, in other 
words if you attach different CQ to each QP, your app would probably scale 
quite bad since the CPU will be busy serving interrupts per IB connection.  

Libor> The fd should become ready for read when there's a CQ event pending 
Libor> and a read of the fd will tell you which CQ it was that had the event.

Second, for an app with > 1 CQ i can think of, eg some multi-threaded IB 
server opening a thread per connection, and each thread has its own IB 
resources, one can apply the logic as Libor suggested above.

So 1 fd per process seems enough.

Or.




From joecat at cmu.edu  Tue Jul 26 08:07:05 2005
From: joecat at cmu.edu (Spencer Whitman)
Date: Tue, 26 Jul 2005 11:07:05 -0400
Subject: [openib-general] Perftest problems
Message-ID: <42E65199.8050600@cmu.edu>

I'm trying to run either of the perf tests and i'm seg faulting on the 
client side =( .  For rdma_lat the client segfaults but the server seems 
to fail silently.  For rdma_bw the client segfaults and the server dies 
with the msg:
server read: Address family not supported by protocol
0/45: Couldn't read remote address

Any ideas or help?
Thanks,
=Spencer Whitman


From mst at mellanox.co.il  Tue Jul 26 08:10:31 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 26 Jul 2005 18:10:31 +0300
Subject: [openib-general] Re: Perftest problems
In-Reply-To: <42E65199.8050600@cmu.edu>
References: <42E65199.8050600@cmu.edu>
Message-ID: <20050726151031.GA27060@mellanox.co.il>

Quoting r. Spencer Whitman <joecat at cmu.edu>:
> Subject: Perftest problems
> 
> I'm trying to run either of the perf tests and i'm seg faulting on the 
> client side =( .  For rdma_lat the client segfaults but the server seems 
> to fail silently.  For rdma_bw the client segfaults and the server dies 
> with the msg:
> server read: Address family not supported by protocol
> 0/45: Couldn't read remote address
> 
> Any ideas or help?
> Thanks,
> =Spencer Whitman

gdb backtrace?

-- 
MST


From gshipman at lanl.gov  Tue Jul 26 09:02:55 2005
From: gshipman at lanl.gov (Galen M. Shipman)
Date: Tue, 26 Jul 2005 10:02:55 -0600
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <52iryxha6j.fsf@topspin.com>
References: <c7ed2fda4dfee8821066e969ed686d62@lanl.gov>
	<1122328044.4421.100.camel@hal.voltaire.com>
	<527jfek0cz.fsf@topspin.com>
	<A044E70E-88BE-4B1E-B66A-2E4611D7B39C@lanl.gov>
	<52r7dlhbu5.fsf@topspin.com>
	<ED864604-D460-4D46-B2DE-93C4FDCAB170@lanl.gov>
	<52iryxha6j.fsf@topspin.com>
Message-ID: <77F2201F-FB00-42BB-AEBC-C30CF7B7024A@lanl.gov>


> Right, I wasn't suggesting your software necessarily.  But you seem to
> think the difference is in the driver software, which doesn't seem
> likely to me.
>
>  - R.

Makes sense,  perhaps we will discuss this further with the mellanox  
people.

Thanks,

Galen




From jlentini at netapp.com  Tue Jul 26 09:57:35 2005
From: jlentini at netapp.com (James Lentini)
Date: Tue, 26 Jul 2005 12:57:35 -0400 (EDT)
Subject: [openib-general] Re: Error in opening IB cm device
In-Reply-To: <20050720083956.GB25957@mellanox.co.il>
References: <1121409378.20046.260.camel@localhost>
	<Pine.GSO.4.40.0507151205500.5849-100000@kappa.cse.ohio-state.edu>
	<20050720083956.GB25957@mellanox.co.il>
Message-ID: <Pine.LNX.4.61.0507261256240.12296@jlentini-linux.nane.netapp.com>



On Wed, 20 Jul 2005, Michael S. Tsirkin wrote:

mst> I have also put up a short page with installation instructions on the
mst> Wiki. Check it out.
mst> 
mst> https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet

The page looks good. Should ucm instructions be added to it?


From mst at mellanox.co.il  Tue Jul 26 10:21:14 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Tue, 26 Jul 2005 20:21:14 +0300
Subject: [openib-general] Re: Re: Error in opening IB cm device
In-Reply-To: <Pine.LNX.4.61.0507261256240.12296@jlentini-linux.nane.netapp.com>
References: <1121409378.20046.260.camel@localhost>
	<Pine.GSO.4.40.0507151205500.5849-100000@kappa.cse.ohio-state.edu>
	<20050720083956.GB25957@mellanox.co.il>
	<Pine.LNX.4.61.0507261256240.12296@jlentini-linux.nane.netapp.com>
Message-ID: <20050726172114.GB27446@mellanox.co.il>

Quoting r. James Lentini <jlentini at netapp.com>:
> Subject: Re: Re: Error in opening IB cm device
> 
> 
> 
> On Wed, 20 Jul 2005, Michael S. Tsirkin wrote:
> 
> mst> I have also put up a short page with installation instructions on the
> mst> Wiki. Check it out.
> mst> 
> mst> https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet
> 
> The page looks good. Should ucm instructions be added to it?
> 

I dont mind. Probably a good idea to put it in a separate paragraph,
though, like I do for management.

-- 
MST


From jlentini at netapp.com  Tue Jul 26 10:37:36 2005
From: jlentini at netapp.com (James Lentini)
Date: Tue, 26 Jul 2005 13:37:36 -0400 (EDT)
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
In-Reply-To: <1122296697.4407.9.camel@hal.voltaire.com>
References: <1122296697.4407.9.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com>


On Mon, 25 Jul 2005, Hal Rosenstock wrote:

> Hi,
>
> Here is a writeup on proposed IB endport liveness and resposiveness.
> This "grew" out of discussions on ibping on this list some time ago.
> Thanks in advance for any comments on this.
>
> -- Hal

Will this eventually be proposed for inclusion in the IBTA spec? I 
think these features would be useful for all IB implementations.


From mshefty at ichips.intel.com  Tue Jul 26 11:03:48 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Tue, 26 Jul 2005 11:03:48 -0700
Subject: [openib-general] Re: why does the value of the node_guid don't
	have the machine en dian ess?
In-Reply-To: <52y87uilan.fsf@topspin.com>
References: <506C3D7B14CDD411A52C00025558DED6084A25B4@mtlex01.yok.mtl.com>
	<52y87uilan.fsf@topspin.com>
Message-ID: <42E67B04.1070501@ichips.intel.com>

Roland Dreier wrote:
>     Dotan> I expect the value to have the same endian-ness when
>     Dotan> printed with %llx and when read from sysfs.
> 
> I don't think this is right.  sysfs is showing the GUID as a string.
> On the other hand, I think it makes sense for the query verb to return
> it in network order.

To clarify, node_guid in ib_device_attr is in network order, correct?  Do 
you know offhand which other fields of that structure are also in network 
order?  Is it just the two guid fields?

- Sean


From rolandd at cisco.com  Tue Jul 26 11:06:52 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 26 Jul 2005 11:06:52 -0700
Subject: [openib-general] Re: why does the value of the node_guid don't
	have the machine en dian ess?
In-Reply-To: <42E67B04.1070501@ichips.intel.com> (Sean Hefty's message of
	"Tue, 26 Jul 2005 11:03:48 -0700")
References: <506C3D7B14CDD411A52C00025558DED6084A25B4@mtlex01.yok.mtl.com>
	<52y87uilan.fsf@topspin.com> <42E67B04.1070501@ichips.intel.com>
Message-ID: <52sly1fnlv.fsf@topspin.com>

    Sean> To clarify, node_guid in ib_device_attr is in network order,
    Sean> correct?  Do you know offhand which other fields of that
    Sean> structure are also in network order?  Is it just the two
    Sean> guid fields?

I'm not positive but I believe that as you say, node_guid and
sys_image_guid are the only two fields returned in network order.

 - R.


From rishi.shah at soulcitypubs.com  Tue Jul 26 12:18:47 2005
From: rishi.shah at soulcitypubs.com (Soul City Publications)
Date: Tue, 26 Jul 2005 12:18:47 -0700
Subject: [openib-general] RAVE*SQ's Wedding Gift Guide!
Message-ID: <1572409438-1463747838-1122405556@soulcitypublications.b.tep1.com>

If you are reading this message your browswer does not support HTML.  Please visit http://soulcitypublications.c.topica.com/maadM3aabiXhBci5DeZe/ to read this message from RAVE*SQ.



====================================================================
Update Your Profile:
   http://soulcitypublications.f.topica.com/f/?a84NZf.ci5DeZ.b3Blbmli
Unsubscribe:
   http://soulcitypublications.f.topica.com/f/unsub.html/aafs57olsf4g91gfecd3h1q8_k8tp0mh9vxtc0koay9kz
Confirm Your Subscription:
   http://soulcitypublications.f.topica.com/f/?a84NZf.ci5DeZ.b3Blbmli.c
Report Unsolicited Email:
   http://topica.com/f/abuse.html?aafs57olsf4g91gfecd3h1q8_k8tp0mh9vxtc0koay9kz
Delivered by Topica:
   http://www.topica.com/?p=T3FOOTER
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050726/06d374b9/attachment.html>

From iod00d at hp.com  Tue Jul 26 13:29:21 2005
From: iod00d at hp.com (Grant Grundler)
Date: Tue, 26 Jul 2005 13:29:21 -0700
Subject: [openib-general] Datacenter Fabric Workshop Linux Track Agenda
	Update
In-Reply-To: <25AE7F432672D511B8DC00B0D0DF11DA0560B94F@MTIEX01>
References: <25AE7F432672D511B8DC00B0D0DF11DA0560B94F@MTIEX01>
Message-ID: <20050726202921.GE7530@esmail.cup.hp.com>

On Mon, Jul 25, 2005 at 09:51:11AM -0700, Sujal Das wrote:
> Here is the updated agenda, list of speakers/presenters for the Workshop
> Linux Track. 

Sujal,
I missed the previous announcement for this event.
Can you post a URL which explains the basic logistics (where/when)

thanks,
grant


From robert.j.woodruff at intel.com  Tue Jul 26 14:19:40 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Tue, 26 Jul 2005 14:19:40 -0700
Subject: [openib-general] [BUG]OpenSM double free or corruption
Message-ID: <1AC79F16F5C5284499BB9591B33D6F00050D9590@orsmsx408>

 Hi Hal,

I was setting up a new 32-node cluster with the openIB code 
here in Oregon. I ran into a problem where the mthca driver failed
to initialize due to the fact the the firmware was down rev. 
I origianlly did not notice that it had not initialized and started
opensm and got the following error from glibc, so there may be 
some bug in the error path in opensm if it cannot open the device or
something. 

[root at apu00 bin]# ./opensm &
[1] 14575
[root at apu00 bin]# -------------------------------------------------
OpenSM Rev:openib-1.0.0
Command Line Arguments:
 Log File: /var/log/osm.log
-------------------------------------------------
*** glibc detected *** double free or corruption (out):
0x0000007fbfffa980 ***

woody



From Sujal at Mellanox.com  Tue Jul 26 15:45:19 2005
From: Sujal at Mellanox.com (Sujal Das)
Date: Tue, 26 Jul 2005 15:45:19 -0700
Subject: [openib-general] Datacenter Fabric Workshop Linux Track Agend
	a Update
Message-ID: <25AE7F432672D511B8DC00B0D0DF11DA0560BC19@MTIEX01>

Hi Grant,

You can get more information at this site:
http://www.acteva.com/booking.cfm?bevaid=91491

Thanks

Sujal

-----Original Message-----
From: Grant Grundler [mailto:iod00d at hp.com] 
Sent: Tuesday, July 26, 2005 1:29 PM
To: Sujal Das
Cc: openib-general at openib.org
Subject: Re: [openib-general] Datacenter Fabric Workshop Linux Track Agenda
Update

On Mon, Jul 25, 2005 at 09:51:11AM -0700, Sujal Das wrote:
> Here is the updated agenda, list of speakers/presenters for the Workshop
> Linux Track. 

Sujal,
I missed the previous announcement for this event.
Can you post a URL which explains the basic logistics (where/when)

thanks,
grant
_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050726/dbff89a6/attachment.html>

From rolandd at cisco.com  Tue Jul 26 21:43:44 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Tue, 26 Jul 2005 21:43:44 -0700
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050726070523.GF13014@minantech.com> (Gleb Natapov's message
	of "Tue, 26 Jul 2005 10:05:23 +0300")
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com> <523bq2n874.fsf@topspin.com>
	<20050726070523.GF13014@minantech.com>
Message-ID: <52mzo8yi2n.fsf@cisco.com>

    Gleb> This is what Pete did. We are back to on syscall on each
    Gleb> registration.  And the worst case is three syscalls. First
    Gleb> to check that mapping is valid, second to deregister memory
    Gleb> if it is not and third register new memory.

I don't think this is an accurate description of Pete's work.  As I
understand it, the fast path (no VM operations have occurred since
registration was done) does not involve any system calls -- the MPI
library just checks the queue of VM events (which will likely be in
cache already), sees that it is empty, and proceeds with the IO.

I was just trying to offer a small refinement so the complex code to
handle the very rare case of queue overflow is not needed.

Of course if memory needs to be reregistered, then we hit the slow
patch (just like a cache miss for the registration cache).

    Gleb> Not exactly. When user frees memory by calling free() the
    Gleb> memory is not unmapped from process address space by
    Gleb> libc. We want to cache the registrations in this memory. We
    Gleb> don't want to cache registration across mmap/munmap/mmap (it
    Gleb> is much harder to do if at all possible).  When libc unmaps
    Gleb> memory by sbrk() or munmap the memory guaranteed to be not
    Gleb> used by correct program so it is safe to deregister it if we
    Gleb> catch this event.

The first statement in this paragraph is false.  It's easy to strace a
simple program that does something like

	x = malloc(1000000);
	free(x);

and watch glibc do an mmap(... MAP_ANONYMOUS ...) followed by
munmap().  Even for smaller allocations, glibc may use sbrk() to
shrink the heap in free().  You can read about M_TRIM_THRESHOLD and so
on in the mallopt() documentation.

This means that it is entirely possible for a correct program to have
different physical memory at the same virtual address, even if it does
nothing but malloc() and free().  Pete's work allows caching of all
registrations, including handling mmap() and everything else.

 - R.


From glebn at voltaire.com  Wed Jul 27 00:02:43 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 27 Jul 2005 10:02:43 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <52mzo8yi2n.fsf@cisco.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com> <523bq2n874.fsf@topspin.com>
	<20050726070523.GF13014@minantech.com> <52mzo8yi2n.fsf@cisco.com>
Message-ID: <20050727070243.GI13014@minantech.com>

On Tue, Jul 26, 2005 at 09:43:44PM -0700, Roland Dreier wrote:
>     Gleb> This is what Pete did. We are back to on syscall on each
>     Gleb> registration.  And the worst case is three syscalls. First
>     Gleb> to check that mapping is valid, second to deregister memory
>     Gleb> if it is not and third register new memory.
> 
> I don't think this is an accurate description of Pete's work.  As I
> understand it, the fast path (no VM operations have occurred since
> registration was done) does not involve any system calls -- the MPI
> library just checks the queue of VM events (which will likely be in
> cache already), sees that it is empty, and proceeds with the IO.
> 
Then we understand Pete differently. This is what he wrote in his first
email on the subject:
   The MPI library essentially makes a system call before
   reusing a cached memory registration to verify it is still valid.
How "the MPI library just checks the queue of VM events" without system
call? Do you mean kernel dumps events in some preallocated memory in
user process address space?

> I was just trying to offer a small refinement so the complex code to
> handle the very rare case of queue overflow is not needed.
> 
> Of course if memory needs to be reregistered, then we hit the slow
> patch (just like a cache miss for the registration cache).
> 
>     Gleb> Not exactly. When user frees memory by calling free() the
>     Gleb> memory is not unmapped from process address space by
>     Gleb> libc. We want to cache the registrations in this memory. We
>     Gleb> don't want to cache registration across mmap/munmap/mmap (it
>     Gleb> is much harder to do if at all possible).  When libc unmaps
>     Gleb> memory by sbrk() or munmap the memory guaranteed to be not
>     Gleb> used by correct program so it is safe to deregister it if we
>     Gleb> catch this event.
> 
> The first statement in this paragraph is false.  It's easy to strace a
> simple program that does something like
> 
> 	x = malloc(1000000);
> 	free(x);
> 
> and watch glibc do an mmap(... MAP_ANONYMOUS ...) followed by
> munmap().  Even for smaller allocations, glibc may use sbrk() to
> shrink the heap in free().  You can read about M_TRIM_THRESHOLD and so
> on in the mallopt() documentation.
> 
You are nitpicking. I know about mallopt() options, thank you. This is not
invalidates the paragraph though. We remove cached entry after libc does
munmap(). In fact that is what paragraph states if you read it till the
end.


> This means that it is entirely possible for a correct program to have
> different physical memory at the same virtual address, even if it does
> nothing but malloc() and free().  Pete's work allows caching of all
> registrations, including handling mmap() and everything else.
> 
Of cause it is possible. It seems you misunderstand me. What is not
possible is that this happens between malloc() and free() for pinned
memory. After user does free() on a chunk of pinned memory we do not
unpin memory till libc returns this memory to the kernel via
munmap()/sbrk() system calls. Again this is what I already wrote.

The solution that I what to see is this:
1) Program knows somehow (without system call) that vma list changes.
2) Program fetch new vma list (possibly by using system call).
3) Program unregister cached entries that are no longer in virtual address
space.

The 1 may be achieved by catching munmap()/sbrk() or by kernel sending
signal when vma list changes or something else. But NOT by read() on
some fd since this require system call on each cache hit or separate
thread to do blocking read(fd) and then you have synchronisation
problems.

If we catch munmap()/sbrk() step 2 is not required. This is even beter.

--
			Gleb.


From tziporet at mellanox.co.il  Wed Jul 27 00:28:03 2005
From: tziporet at mellanox.co.il (Tziporet Koren)
Date: Wed, 27 Jul 2005 10:28:03 +0300
Subject: [openib-general] Shared Receive Queue
Message-ID: <506C3D7B14CDD411A52C00025558DED6085BCFB4@mtlex01.yok.mtl.com>

The performance of SRQ was improved to be identical to regular QP in the
latest FW release for all devices.
Tziporet

-----Original Message-----
From: Galen M. Shipman [mailto:gshipman at lanl.gov]
Sent: Tuesday, July 26, 2005 5:57 PM
To: Or Gerlitz
Cc: openib-general at openib.org
Subject: Re: [openib-general] Shared Receive Queue



I am using our own software which uses send receive in both cases. As  
I said the code paths are exactly the same except for posting of  
receive descriptors as well as initialization.

- Galen

On Jul 26, 2005, at 8:49 AM, Or Gerlitz wrote:

>> We have software which uses the mellanox shared receive queue and we
>> are seeing a substantial performance penalty (on the order of 4-5
>> usec on 0 byte messages).
>>
>
> Are you comparing code that sends short messages either by rdma write
> where the
> reciever does polling on memory (as done in some IB MPIs "short  
> message
> protocols")
> to code that uses send/recv with SRQ --or-- code that does send/recv
> where in one run
> the recv is to "non shared" queue (that is regular QP) and in the  
> other
> run to SRQ (QP attached to SRQ)?
>
> Or.
>
>

_______________________________________________
openib-general mailing list
openib-general at openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050727/62243329/attachment.html>

From guyg at voltaire.com  Wed Jul 27 01:10:43 2005
From: guyg at voltaire.com (Guy German)
Date: Wed, 27 Jul 2005 11:10:43 +0300 (IDT)
Subject: [openib-general][PATCH][kdapl]: adding DAT_MEM_TYPE_IA support
Message-ID: <Pine.LNX.4.63.0507271104170.19809@beid.voltaire.com>

[kdapl]: this patch adds the option of using DAT_MEM_TYPE_IA in dat_lmr_kcreate

Signed-off-by: Guy German <guyg at voltaire.com>

Index: infiniband/ulp/kdapl/ib/dapl_openib_util.c
===================================================================
--- infiniband/ulp/kdapl/ib/dapl_openib_util.c	(revision 2914)
+++ infiniband/ulp/kdapl/ib/dapl_openib_util.c	(working copy)
@@ -234,8 +234,43 @@ int dapl_ib_mr_register(struct dapl_ia *
  	return -ENOSYS;
  }

+int dapl_ib_mr_register_ia(struct dapl_ia *ia, struct dapl_lmr *lmr,
+			   union dat_region_description phys_addr, u64 length,
+			   enum dat_mem_priv_flags privileges)
+{
+	int status;
+	int acl;
+	struct ib_mr *mr;
+	struct ib_phys_buf buf_list;
+	u64 iova = 0;
+	buf_list.addr = phys_addr.for_pa;
+	buf_list.size = length;
+ 
+	iova = buf_list.addr;
+	acl = dapl_ib_convert_mem_privileges(privileges);
+	acl |= IB_ACCESS_MW_BIND;
+	mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd,
+			    &buf_list, 1, acl, &iova); 
+	if (IS_ERR(mr)) {
+		status = PTR_ERR(mr);
+		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
+			"%s: ib_reg_phys_mr error code return = %d\n",
+			__FUNCTION__, status);
+		return status;
+	}
+	dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "%s: got handle mr=%p\n",
+	       __FUNCTION__, mr);
+
+	lmr->param.lmr_context = mr->lkey;
+	lmr->param.rmr_context = mr->rkey;
+	lmr->param.registered_size = length;
+	lmr->param.registered_address = phys_addr.for_pa;
+	lmr->mr = mr;
+	return 0;
+}
+
  int dapl_ib_mr_register_physical(struct dapl_ia *ia, struct dapl_lmr *lmr,
-				 void *phys_addr, u64 length,
+				 union dat_region_description phys_addr, u64 length,
  				 enum dat_mem_priv_flags privileges)
  {
  	int status;
@@ -246,7 +281,7 @@ int dapl_ib_mr_register_physical(struct
  	u64 iova = 0;
  	u64 *array;

-	array = (u64 *) phys_addr;
+	array = (u64 *)phys_addr.for_array; /* need to add for_u64_array to union */
  	buf_list = kmalloc(length * sizeof *buf_list, GFP_ATOMIC);
  	if (!buf_list)
  		return -ENOMEM;
@@ -265,8 +300,8 @@ int dapl_ib_mr_register_physical(struct
  	if (IS_ERR(mr)) {
  		status = PTR_ERR(mr);
  		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
-			     " ib_reg_phys_mr error code return = %d\n",
-			     status);
+			     "%s: ib_reg_phys_mr error code return = %d\n",
+			     __FUNCTION__, status);
  		return status;
  	}
  #if 0
@@ -274,8 +309,8 @@ int dapl_ib_mr_register_physical(struct
  	status = ib_query_mr(mr, &attr);
  	if (status < 0) {
  		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
-			     " ib_query_mr error code return from ib_query_mr = %d\n",
-			     status);
+			     "%s: ib_query_mr error code return from ib_query_mr = %d\n",
+			     __FUNCTION__, status);
  		ib_dereg_mr(mr);
  		return status;
  	}
@@ -283,10 +318,12 @@ int dapl_ib_mr_register_physical(struct

  	lmr->param.lmr_context = mr->lkey;
  	lmr->param.rmr_context = mr->rkey;
+	lmr->param.registered_size = length * PAGE_SIZE;
+	lmr->param.registered_address = array[0];
  	lmr->mr = mr;

  	dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
-		"dapl_ib_mr_register_physical(%p %d) got lkey 0x%x \n",
+		"%s: (%p %d) got lkey 0x%x \n", __FUNCTION__,
  		buf_list, length, lmr->param.lmr_context);
  	return 0;
  }
Index: infiniband/ulp/kdapl/ib/dapl_openib_util.h
===================================================================
--- infiniband/ulp/kdapl/ib/dapl_openib_util.h	(revision 2914)
+++ infiniband/ulp/kdapl/ib/dapl_openib_util.h	(working copy)
@@ -92,8 +92,13 @@ int dapl_ib_mr_register(struct dapl_ia *
  			void *virt_addr, u64 length,
  			enum dat_mem_priv_flags privileges);

+int dapl_ib_mr_register_ia(struct dapl_ia *ia, struct dapl_lmr *lmr,
+			   union dat_region_description phys_addr, u64 length,
+			   enum dat_mem_priv_flags privileges);
+
  int dapl_ib_mr_register_physical(struct dapl_ia *ia, struct dapl_lmr *lmr,
-				 void *phys_addr, u64 length,
+				 union dat_region_description phys_addr, 
+				 u64 length,
  				 enum dat_mem_priv_flags privileges);

  int dapl_ib_mr_deregister(struct dapl_lmr *lmr);
Index: infiniband/ulp/kdapl/ib/dapl_lmr.c
===================================================================
--- infiniband/ulp/kdapl/ib/dapl_lmr.c	(revision 2914)
+++ infiniband/ulp/kdapl/ib/dapl_lmr.c	(working copy)
@@ -125,20 +125,21 @@ error1:
  }

  static inline int dapl_lmr_create_physical(struct dapl_ia *ia,
-					   DAT_REGION_DESCRIPTION phys_addr,
-					   u64 page_count, struct dapl_pz *pz,
+					   union dat_region_description phys_addr,
+					   u64 page_count, 
+					   enum dat_mem_type mem_type,
+					   struct dapl_pz *pz,
  					   enum dat_mem_priv_flags privileges,
  					   struct dat_lmr **lmr,
-					   DAT_LMR_CONTEXT *lmr_context,
-					   DAT_RMR_CONTEXT *rmr_context,
+					   u32 *lmr_context,
+					   u32 *rmr_context,
  					   u64 *registered_length,
  					   u64 *registered_address)
  {
  	struct dapl_lmr *new_lmr;
-	u64 *array = phys_addr.for_array;
  	int status;

-	new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr,
+	new_lmr = dapl_lmr_alloc(ia, mem_type, phys_addr,
  				 page_count, (struct dat_pz *) pz, privileges);

  	if (NULL == new_lmr) {
@@ -146,8 +147,14 @@ static inline int dapl_lmr_create_physic
  		goto error1;
  	}

-	status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr.for_array,
-					      page_count, privileges);
+	if (DAT_MEM_TYPE_IA == mem_type) {
+		status = dapl_ib_mr_register_ia(ia, new_lmr, phys_addr,
+						page_count, privileges);
+	}
+	else {
+		status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr,
+						      page_count, privileges);
+	}

  	if (0 != status)
  		goto error2;
@@ -157,13 +164,13 @@ static inline int dapl_lmr_create_physic
  	if (lmr)
  		*lmr = (struct dat_lmr *)new_lmr;
  	if (lmr_context)
-		*lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context;
+		*lmr_context = (u32)new_lmr->param.lmr_context;
  	if (rmr_context)
-		*rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context;
+		*rmr_context = (u32)new_lmr->param.rmr_context;
  	if (registered_address)
-		*registered_address = array[0];
+		*registered_address = new_lmr->param.registered_address;
  	if (registered_length)
-		*registered_length = page_count * PAGE_SIZE;
+		*registered_length = new_lmr->param.registered_size;

  	return 0;

@@ -233,7 +240,7 @@ int dapl_lmr_kcreate(struct dat_ia *ia,
  	struct dapl_ia *dapl_ia;
  	struct dapl_pz *dapl_pz;
  	int status;
-
+
  	dapl_dbg_log(DAPL_DBG_TYPE_API,
  		     "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n",
  		     ia, mem_type);
@@ -250,10 +257,12 @@ int dapl_lmr_kcreate(struct dat_ia *ia,
  						 rmr_context, registered_length,
  						 registered_address);
  		break;
+	case DAT_MEM_TYPE_PLATFORM: // used as proprietary Tavor-FMR
  	case DAT_MEM_TYPE_PHYSICAL:
+	case DAT_MEM_TYPE_IA:
  		status = dapl_lmr_create_physical(dapl_ia, region_description,
-						  length, dapl_pz, privileges,
-						  lmr, lmr_context,
+						  length, mem_type, dapl_pz, 
+						  privileges, lmr, lmr_context,
  						  rmr_context,
  						  registered_length,
  						  registered_address);
@@ -275,8 +284,6 @@ int dapl_lmr_kcreate(struct dat_ia *ia,
  					     registered_address);
  		break;
  	}
-	case DAT_MEM_TYPE_PLATFORM:
-	case DAT_MEM_TYPE_IA:
  	case DAT_MEM_TYPE_BYPASS:
  		status = -ENOSYS;
  		break;
@@ -300,7 +307,8 @@ int dapl_lmr_free(struct dat_lmr *lmr)

  	switch (dapl_lmr->param.mem_type) {
  	case DAT_MEM_TYPE_PHYSICAL:
-	case DAT_MEM_TYPE_VIRTUAL:
+	case DAT_MEM_TYPE_PLATFORM:
+	case DAT_MEM_TYPE_IA:
  	case DAT_MEM_TYPE_LMR:
  	{
  		struct dapl_pz *pz;
@@ -316,8 +324,7 @@ int dapl_lmr_free(struct dat_lmr *lmr)
  		}
  		break;
  	}
-	case DAT_MEM_TYPE_PLATFORM:
-	case DAT_MEM_TYPE_IA:
+	case DAT_MEM_TYPE_VIRTUAL:
  	case DAT_MEM_TYPE_BYPASS:
  		status = -ENOSYS;
  		break;


From guyg at voltaire.com  Wed Jul 27 01:21:39 2005
From: guyg at voltaire.com (Guy German)
Date: Wed, 27 Jul 2005 11:21:39 +0300 (IDT)
Subject: [openib-general][PATCH][kdapltest]: adding memtype options to server
Message-ID: <Pine.LNX.4.63.0507271110520.19809@beid.voltaire.com>

[kdapltest]: this patch adds the option of using memory registration
types other then the default PHY on the server side.

Signed-off-by: Guy German <guyg at voltaire.com>

Index: dapltest/test/dapl_server.c
===================================================================
--- dapltest/test/dapl_server.c	(revision 2899)
+++ dapltest/test/dapl_server.c	(working copy)
@@ -232,7 +232,14 @@ DT_cs_Server (Params_t * params_ptr)
       * Create two buffers, large enough to hold ClientInfo and the largest
       * command we'll use.
       */
-    ps_ptr->bpool = DT_BpoolAlloc (NULL,
+ 
+    if (!(pt_ptr = DT_Alloc_Per_Test_Data (phead)))
+	    goto server_exit;
+    DT_MemListInit (pt_ptr);
+    memcpy ( (void *) (uintptr_t) &pt_ptr->Params,
+	    (const void *) params_ptr,
+	    sizeof (Params_t));
+    ps_ptr->bpool = DT_BpoolAlloc (pt_ptr,
  	    		       phead,
  			       ps_ptr->ia,
  			       ps_ptr->pz,
@@ -243,6 +250,7 @@ DT_cs_Server (Params_t * params_ptr)
  			       256, /* FIXME query for this value */
  			       FALSE,
  			       FALSE);
+    DT_Free_Per_Test_Data (pt_ptr);
      if (ps_ptr->bpool == 0)
      {
  	DT_Tdep_PT_Printf (phead,


From halr at voltaire.com  Wed Jul 27 04:33:59 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 07:33:59 -0400
Subject: [openib-general] Re: [BUG]OpenSM double free or corruption
In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00050D9590@orsmsx408>
References: <1AC79F16F5C5284499BB9591B33D6F00050D9590@orsmsx408>
Message-ID: <1122464039.4421.125.camel@hal.voltaire.com>

On Tue, 2005-07-26 at 17:19, Woodruff, Robert J wrote:
>  Hi Hal,
> 
> I was setting up a new 32-node cluster with the openIB code 
> here in Oregon. I ran into a problem where the mthca driver failed
> to initialize due to the fact the the firmware was down rev. 
> I origianlly did not notice that it had not initialized and started
> opensm and got the following error from glibc, so there may be 
> some bug in the error path in opensm if it cannot open the device or
> something. 
> 
> [root at apu00 bin]# ./opensm &
> [1] 14575
> [root at apu00 bin]# -------------------------------------------------
> OpenSM Rev:openib-1.0.0
> Command Line Arguments:
>  Log File: /var/log/osm.log
> -------------------------------------------------
> *** glibc detected *** double free or corruption (out):
> 0x0000007fbfffa980 ***

Do you have the log file as to where in the initialization this was
detected ? Do you know how to reproduce this ? I'll also do a code
inspection to see if I can find it.

-- Hal

> woody
> 



From halr at voltaire.com  Wed Jul 27 05:15:20 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 08:15:20 -0400
Subject: [openib-general] Re: [openib-commits] r2915 -
	gen2/users/jlentini/linux-kernel
Message-ID: <1122466520.4421.256.camel@hal.voltaire.com>

On Tue, 2005-07-26 at 17:20, jlentini at openib.org wrote:
> Author: jlentini
> Date: 2005-07-26 14:20:22 -0700 (Tue, 26 Jul 2005)
> New Revision: 2915
> 
> Modified:
>    gen2/users/jlentini/linux-kernel/README
> Log:
> Updated README with new code location
> Signed-off-by: James Lentini <jlentini at netapp.com>
> 
> 
> Modified: gen2/users/jlentini/linux-kernel/README
> ===================================================================
> --- gen2/users/jlentini/linux-kernel/README	2005-07-25 23:25:07 UTC (rev 2914)
> +++ gen2/users/jlentini/linux-kernel/README	2005-07-26 21:20:22 UTC (rev 2915)
> @@ -1,3 +1,7 @@
> +================================================================================
> + NOTE: The OpenIB kDAPL code is now located in the trunk at:
> + https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/kdap
                                                                      ^^
Should this be kdapl rather than kdap ?

> +================================================================================
>  
>  DAT Instructions for OpenIB





From halr at voltaire.com  Wed Jul 27 05:42:53 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 08:42:53 -0400
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
In-Reply-To: <Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com>
References: <1122296697.4407.9.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com>
Message-ID: <1122468122.4421.328.camel@hal.voltaire.com>

On Tue, 2005-07-26 at 13:37, James Lentini wrote:
> On Mon, 25 Jul 2005, Hal Rosenstock wrote:
> 
> > Hi,
> >
> > Here is a writeup on proposed IB endport liveness and resposiveness.
> > This "grew" out of discussions on ibping on this list some time ago.
> > Thanks in advance for any comments on this.
> >
> > -- Hal
> 
> Will this eventually be proposed for inclusion in the IBTA spec? I 
> think these features would be useful for all IB implementations.

Perhaps as optional additions. It depends on how this resonates here. So
far this has been the only comment :-(

-- Hal



From pw at osc.edu  Wed Jul 27 06:52:02 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Wed, 27 Jul 2005 09:52:02 -0400
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050727070243.GI13014@minantech.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com> <523bq2n874.fsf@topspin.com>
	<20050726070523.GF13014@minantech.com> <52mzo8yi2n.fsf@cisco.com>
	<20050727070243.GI13014@minantech.com>
Message-ID: <20050727135202.GA24336@osc.edu>

glebn at voltaire.com wrote on Wed, 27 Jul 2005 10:02 +0300:
> Then we understand Pete differently. This is what he wrote in his first
> email on the subject:
>    The MPI library essentially makes a system call before
>    reusing a cached memory registration to verify it is still valid.
> How "the MPI library just checks the queue of VM events" without system
> call? Do you mean kernel dumps events in some preallocated memory in
> user process address space?

Roland has it right.  In response to your comments I refined the
mechanism to avoid the system call:  the kernel writes an event
count to a registered userspace integer when something changes.

> Of cause it is possible. It seems you misunderstand me. What is not
> possible is that this happens between malloc() and free() for pinned
> memory. After user does free() on a chunk of pinned memory we do not
> unpin memory till libc returns this memory to the kernel via
> munmap()/sbrk() system calls. Again this is what I already wrote.
> 
> The solution that I what to see is this:
> 1) Program knows somehow (without system call) that vma list changes.
> 2) Program fetch new vma list (possibly by using system call).
> 3) Program unregister cached entries that are no longer in virtual address
> space.

This is what I did.  I'll just clean up the code and post it so you
can read it yourself.

		-- Pete


From jlentini at netapp.com  Wed Jul 27 06:54:56 2005
From: jlentini at netapp.com (James Lentini)
Date: Wed, 27 Jul 2005 09:54:56 -0400 (EDT)
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
In-Reply-To: <1122468122.4421.328.camel@hal.voltaire.com>
References: <1122296697.4407.9.camel@hal.voltaire.com> 
	<Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com>
	<1122468122.4421.328.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507270933240.12296@jlentini-linux.nane.netapp.com>



On Wed, 27 Jul 2005, Hal Rosenstock wrote:

> On Tue, 2005-07-26 at 13:37, James Lentini wrote:
>> On Mon, 25 Jul 2005, Hal Rosenstock wrote:
>>
>>> Hi,
>>>
>>> Here is a writeup on proposed IB endport liveness and resposiveness.
>>> This "grew" out of discussions on ibping on this list some time ago.
>>> Thanks in advance for any comments on this.
>>>
>>> -- Hal
>>
>> Will this eventually be proposed for inclusion in the IBTA spec? I
>> think these features would be useful for all IB implementations.
>
> Perhaps as optional additions. It depends on how this resonates here. So
> far this has been the only comment :-(

Being able to debug connectivity problems at the IB level strikes me 
as very useful. Perhaps some people expect to always use IPoIB on 
their IB network, and therefore plan to use IP tools. Even in these 
configurations, native IB protocols would be useful if IPoIB wasn't 
working.

It looks like you modeled the ECHO and TIMESTAMP formats after their 
ICMP counterparts. The TIMESTAMP type appears to have been dropped 
from ICMPv6. Do you know why?


From glebn at voltaire.com  Wed Jul 27 07:01:50 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Wed, 27 Jul 2005 17:01:50 +0300
Subject: [openib-general] Question about pinning memory
In-Reply-To: <20050727135202.GA24336@osc.edu>
References: <20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com> <524qaiopjv.fsf@topspin.com>
	<20050725155538.GC13014@minantech.com> <523bq2n874.fsf@topspin.com>
	<20050726070523.GF13014@minantech.com> <52mzo8yi2n.fsf@cisco.com>
	<20050727070243.GI13014@minantech.com>
	<20050727135202.GA24336@osc.edu>
Message-ID: <20050727140150.GO13014@minantech.com>

On Wed, Jul 27, 2005 at 09:52:02AM -0400, Pete Wyckoff wrote:
> glebn at voltaire.com wrote on Wed, 27 Jul 2005 10:02 +0300:
> > Then we understand Pete differently. This is what he wrote in his first
> > email on the subject:
> >    The MPI library essentially makes a system call before
> >    reusing a cached memory registration to verify it is still valid.
> > How "the MPI library just checks the queue of VM events" without system
> > call? Do you mean kernel dumps events in some preallocated memory in
> > user process address space?
> 
> Roland has it right.  In response to your comments I refined the
> mechanism to avoid the system call:  the kernel writes an event
> count to a registered userspace integer when something changes.
> 
Excellent.

> > Of cause it is possible. It seems you misunderstand me. What is not
> > possible is that this happens between malloc() and free() for pinned
> > memory. After user does free() on a chunk of pinned memory we do not
> > unpin memory till libc returns this memory to the kernel via
> > munmap()/sbrk() system calls. Again this is what I already wrote.
> > 
> > The solution that I what to see is this:
> > 1) Program knows somehow (without system call) that vma list changes.
> > 2) Program fetch new vma list (possibly by using system call).
> > 3) Program unregister cached entries that are no longer in virtual address
> > space.
> 
> This is what I did.  I'll just clean up the code and post it so you
> can read it yourself.
> 
I'll wait for the code then.

--
			Gleb.


From halr at voltaire.com  Wed Jul 27 07:21:37 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 10:21:37 -0400
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
In-Reply-To: <Pine.LNX.4.61.0507270933240.12296@jlentini-linux.nane.netapp.com>
References: <1122296697.4407.9.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com>
	<1122468122.4421.328.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507270933240.12296@jlentini-linux.nane.netapp.com>
Message-ID: <1122474097.4421.348.camel@hal.voltaire.com>

On Wed, 2005-07-27 at 09:54, James Lentini wrote:
> Being able to debug connectivity problems at the IB level strikes me 
> as very useful. Perhaps some people expect to always use IPoIB on 
> their IB network, and therefore plan to use IP tools. Even in these 
> configurations, native IB protocols would be useful if IPoIB wasn't 
> working.

Right; IPoIB relies on more working (multicasting, etc.) than this
proposal does.

> It looks like you modeled the ECHO and TIMESTAMP formats after their 
> ICMP counterparts. 

Yes.

> The TIMESTAMP type appears to have been dropped from ICMPv6. Do you know why?

No but it sounds like it might be relevant. The only thing I saw were
some comments related to security and replay attacks. Not sure if that
was the reason it was eliminated from ICMPv6.

-- Hal






From pw at osc.edu  Wed Jul 27 07:29:14 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Wed, 27 Jul 2005 10:29:14 -0400
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <52ackaihka.fsf@topspin.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
	<20050725210136.GB21836@osc.edu>
	<20050725211146.GA13882@mellanox.co.il>
	<20050725232624.GA6812@baa> <52ackaihka.fsf@topspin.com>
Message-ID: <20050727142914.GB24336@osc.edu>

rolandd at cisco.com wrote on Mon, 25 Jul 2005 16:36 -0700:
> Hmm, thinking about this some more, it occurs to me that it might be
> possible for the kernel and userspace to work together even more
> efficiently.  Does the following make sense?
> 
> When memory is registered, userspace tells the kernel VM activity
> monitor where its registration cache data structure for that
> registration is.  If the kernel detects some VM activity that affects
> that registration, then it can invalidate it directly in the userspace
> cache data structure.
> 
> I guess there are two issues to resolve here:
> 
>  - Are there any races between the kernel marking something invalid
>    and userspace marking it valid again?
> 
>  - How do we keep track of a process's registrations in the kernel so
>    that we can efficiently find which (if any) registrations are
>    affected by a VM operation?  (Maybe you've solved this already)

I think those are solvable problems and a reasonable approach.
There are a few reasons I didn't make this coupling more intimate
from the start:

    - Userspace may want actively to do something when a forced
      unregistration (due to munmap etc) happens, like allow
      registration of some other block that was not possible due to
      system limits on number or total size of registrations.
      Although I guess it could walk its cache when it goes to try
      to pin more and knows that the kernel has freed something.

    - Some find this sort of mixing unpalatable.  It constrains
      independent development of the userspace library and kernel
      module.  Backward compatibility might be a problem some day.

    - The VM event message tells userspace to make the call
      back into the device to perform the deregistration.  This is
      really just because of a constraint in the VAPI libraries and
      that I didn't want to have to change them.  Perhaps in OpenIB
      it will be more natural for the kernel to call the device to
      do the actual deregistration.  There is some messy error handling
      to deal with, though, and garbage collection in userspace too.
      Some amount of code duplication.

I expect the deregistration time to dominate this entire affair.
Avoiding the interaction with the library when forced invalidations
happen, as you suggest, has the potential to be faster, but the
complexity is a bit daunting.  I'd like to run some profiles
of apps that generate lots of VM activity and see where the time
goes.

		-- Pete


From halr at voltaire.com  Wed Jul 27 08:31:23 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 11:31:23 -0400
Subject: [openib-general] [PATCH] OpenIB scripts
Message-ID: <1122478283.4421.19.camel@hal.voltaire.com>

Add ucm to udev rules

Signed-off-by: Hal Rosenstock <halr at voltaire.com

Also, should these (https://openib.org/svn/trunk/contrib/mellanox/) be
moved to a gen2 location ?

Index: 90-ib.rules
===================================================================
--- 90-ib.rules (revision 2917)
+++ 90-ib.rules (working copy)
@@ -1,3 +1,4 @@
 KERNEL="umad*", NAME="infiniband/%k"
 KERNEL="issm*", NAME="infiniband/%k"
+KERNEL="ucm", NAME="infiniband/%k", MODE="0666"
 KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"




From robert.j.woodruff at intel.com  Wed Jul 27 08:48:38 2005
From: robert.j.woodruff at intel.com (Bob Woodruff)
Date: Wed, 27 Jul 2005 08:48:38 -0700
Subject: [openib-general] RE: [BUG]OpenSM double free or corruption
In-Reply-To: <1122464039.4421.125.camel@hal.voltaire.com>
Message-ID: <ORSMSX408FRaqbC8wSA00000009@orsmsx408.amr.corp.intel.com>

 
Hal wrote, 
>Do you have the log file as to where in the initialization this was
>detected ? Do you know how to reproduce this ? I'll also do a code
>inspection to see if I can find it.

>-- Hal

I don't have the log file, but it is easy to reproduce.
Load the stack on an HCA that has old firmware and the mthca
driver will not initialize and will report an error in dmesg.

Then simply start opensm and the error occurs. 

woody




From timur.tabi at ammasso.com  Wed Jul 27 09:30:39 2005
From: timur.tabi at ammasso.com (Timur Tabi)
Date: Wed, 27 Jul 2005 11:30:39 -0500
Subject: [openib-general] Question about pinning memory
In-Reply-To: <52mzo8yi2n.fsf@cisco.com>
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>	<20050724170335.GB1948@baa>
	<52k6jfotrj.fsf@topspin.com>	<20050725143759.GJ19097@minantech.com>
	<524qaiopjv.fsf@topspin.com>	<20050725155538.GC13014@minantech.com>
	<523bq2n874.fsf@topspin.com>	<20050726070523.GF13014@minantech.com>
	<52mzo8yi2n.fsf@cisco.com>
Message-ID: <42E7B6AF.8070801@ammasso.com>

Roland Dreier wrote:
> The first statement in this paragraph is false.  It's easy to strace a
> simple program that does something like
> 
> 	x = malloc(1000000);
> 	free(x);
> 
> and watch glibc do an mmap(... MAP_ANONYMOUS ...) followed by
> munmap().  Even for smaller allocations, glibc may use sbrk() to
> shrink the heap in free().  You can read about M_TRIM_THRESHOLD and so
> on in the mallopt() documentation.

I wouldn't rely on this at all.

libc is under no obligation to inform the kernel in any way when free() is called.  I just 
don't see any reliable way of having MPI or OpenIB or whatever get notified when an 
application "frees" a block of memory before it explicitely unregisters it.  There's 
nothing stopping an application from simply re-using a buffer for non-RDMA purposes 
whenever it wants.  I would rather err on the side of pinning too much memory.

I just don't see how this problem can be solved.  The application needs to specifically 
tell the library when it no longer wants a particular block of memory to be used for RDMA 
operations.  If it re-uses or deallocates such a block BEFORE telling the library, I 
consider that to be an application bug.  At that point, all we need to worry about is 
whether that buffer gets allocated to another process before deregistration, because that 
would be a security hole.

(Disclaimer: I just noticed this thread and I don't understanding yet everything that 
others have so far written.)

-- 
Timur Tabi
Staff Software Engineer
timur.tabi at ammasso.com

One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
      -- Ed Smylie, NASA engineer for Apollo 13


From rolandd at cisco.com  Wed Jul 27 10:04:54 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 10:04:54 -0700
Subject: [openib-general] Re: Question about pinning memory
In-Reply-To: <20050727142914.GB24336@osc.edu> (Pete Wyckoff's message of
	"Wed, 27 Jul 2005 10:29:14 -0400")
References: <9a700cdad0f028d8a6c9d19a50b2ca42@open-mpi.org>
	<20050724170335.GB1948@baa> <52k6jfotrj.fsf@topspin.com>
	<20050725143759.GJ19097@minantech.com>
	<20050725153729.GA19700@osc.edu>
	<20050725155846.GD13014@minantech.com>
	<20050725210136.GB21836@osc.edu>
	<20050725211146.GA13882@mellanox.co.il> <20050725232624.GA6812@baa>
	<52ackaihka.fsf@topspin.com> <20050727142914.GB24336@osc.edu>
Message-ID: <528xzsxjrd.fsf@cisco.com>

    Pete> I think those are solvable problems and a reasonable
    Pete> approach.  There are a few reasons I didn't make this
    Pete> coupling more intimate from the start:

All fair enough -- I was just scared of writing code to deal with the
queue overflow!

 - R.


From rolandd at cisco.com  Wed Jul 27 11:00:36 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 11:00:36 -0700
Subject: [openib-general] Proposed updates for 2.6.13
Message-ID: <52sly0w2m3.fsf@cisco.com>

Here is the set of patches I am planning on sending upstream this week
for inclusion in 2.6.13.  In addition to bug fixes, I am including
fasync support for uverbs -- it seems worth having this in 2.6.13, so
that code can rely on it any released kernel.

Please let me know if you think there are other changes we should send
upstream, or if you think some of the changes I'm including here
should be held for 2.6.14.

 - R.


Id: cf3efcf0bebe11027caa231666d2392640c63c32
tree 941fb82a37b295d23e172eb49e55477130a82dc5
parent 6b6a93c6876ea1c530d5d3f68e3678093a27fab0
author Roland Dreier <roland at eddore.topspincom.com> 1122348605 -0700
committer Roland Dreier <roland at eddore.topspincom.com> 1122348605 -0700

Fix handling of error CQ entries on mem-free HCAs: the doorbell count is never
valid so we shouldn't look at it.

Signed-off-by: Roland Dreier <rolandd at cisco.com>


======== diff against 6b6a93c6876ea1c530d5d3f68e3678093a27fab0 ========
diff --git a/drivers/infiniband/hw/mthca/mthca_cq.c b/drivers/infiniband/hw/mthca/mthca_cq.c
--- a/drivers/infiniband/hw/mthca/mthca_cq.c
+++ b/drivers/infiniband/hw/mthca/mthca_cq.c
@@ -373,8 +373,12 @@ static int handle_error_cqe(struct mthca
 	 * If we're at the end of the WQE chain, or we've used up our
 	 * doorbell count, free the CQE.  Otherwise just update it for
 	 * the next poll operation.
+	 *
+	 * This does not apply to mem-free HCAs: they don't use the
+	 * doorbell count field, and so we should always free the CQE.
 	 */
-	if (!(new_wqe & cpu_to_be32(0x3f)) || (!cqe->db_cnt && dbd))
+	if (mthca_is_memfree(dev) ||
+	    !(new_wqe & cpu_to_be32(0x3f)) || (!cqe->db_cnt && dbd))
 		return 0;
 
 	cqe->db_cnt   = cpu_to_be16(be16_to_cpu(cqe->db_cnt) - dbd);
======== end ========

Id: 53877608748c50917b400d8b2be71782179260d9
tree 668a14bdfabff09e73d3317ca43ddbfc042323f8
parent cf3efcf0bebe11027caa231666d2392640c63c32
author Roland Dreier <roland at eddore.topspincom.com> 1122349011 -0700
committer Roland Dreier <roland at eddore.topspincom.com> 1122349011 -0700

From: Gleb Natapov <glebn at voltaire.com>

Add support for O_ASYNC notifications on uverbs completion and asynchronous
event file descriptors.

Signed-off-by: Gleb Natapov <glebn at voltaire.com>
Signed-off-by: Roland Dreier <rolandd at cisco.com>


======== diff against cf3efcf0bebe11027caa231666d2392640c63c32 ========
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -61,6 +61,7 @@ struct ib_uverbs_event_file {
 	int					fd;
 	int					is_async;
 	wait_queue_head_t			poll_wait;
+	struct fasync_struct		       *async_queue;
 	struct list_head			event_list;
 };
 
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -257,11 +257,19 @@ static void ib_uverbs_event_release(stru
 	spin_unlock_irq(&file->lock);
 }
 
+static int ib_uverbs_event_fasync(int fd, struct file *filp, int on)
+{
+	struct ib_uverbs_event_file *file = filp->private_data;
+
+	return fasync_helper(fd, filp, on, &file->async_queue);
+}
+
 static int ib_uverbs_event_close(struct inode *inode, struct file *filp)
 {
 	struct ib_uverbs_event_file *file = filp->private_data;
 
 	ib_uverbs_event_release(file);
+	ib_uverbs_event_fasync(-1, filp, 0);
 	kref_put(&file->uverbs_file->ref, ib_uverbs_release_file);
 
 	return 0;
@@ -276,7 +284,8 @@ static struct file_operations uverbs_eve
 	 */
 	.read 	 = ib_uverbs_event_read,
 	.poll    = ib_uverbs_event_poll,
-	.release = ib_uverbs_event_close
+	.release = ib_uverbs_event_close,
+	.fasync  = ib_uverbs_event_fasync
 };
 
 void ib_uverbs_comp_handler(struct ib_cq *cq, void *cq_context)
@@ -296,6 +305,7 @@ void ib_uverbs_comp_handler(struct ib_cq
 	spin_unlock_irqrestore(&file->comp_file[0].lock, flags);
 
 	wake_up_interruptible(&file->comp_file[0].poll_wait);
+	kill_fasync(&file->comp_file[0].async_queue, SIGIO, POLL_IN);
 }
 
 static void ib_uverbs_async_handler(struct ib_uverbs_file *file,
@@ -316,6 +326,7 @@ static void ib_uverbs_async_handler(stru
 	spin_unlock_irqrestore(&file->async_file.lock, flags);
 
 	wake_up_interruptible(&file->async_file.poll_wait);
+	kill_fasync(&file->async_file.async_queue, SIGIO, POLL_IN);
 }
 
 void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr)
@@ -350,6 +361,7 @@ static int ib_uverbs_event_init(struct i
 	INIT_LIST_HEAD(&file->event_list);
 	init_waitqueue_head(&file->poll_wait);
 	file->uverbs_file = uverbs_file;
+	file->async_queue = NULL;
 
 	file->fd = get_unused_fd();
 	if (file->fd < 0)
======== end ========

Id: a42b5922ba01c304e81ecb94996b4f5b5def4c5f
tree 86c8e7b81466309f4d632587f477454754247016
parent 53877608748c50917b400d8b2be71782179260d9
author Roland Dreier <roland at eddore.topspincom.com> 1122349418 -0700
committer Roland Dreier <roland at eddore.topspincom.com> 1122349418 -0700

Fix handling of tx_head/tx_tail comparisons to handle wraparound.

Signed-off-by: Roland Dreier <rolandd at cisco.com>


======== diff against 53877608748c50917b400d8b2be71782179260d9 ========
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -81,7 +81,7 @@ void ipoib_free_ah(struct kref *kref)
 
 	unsigned long flags;
 
-	if (ah->last_send <= priv->tx_tail) {
+	if ((int) priv->tx_tail - (int) ah->last_send >= 0) {
 		ipoib_dbg(priv, "Freeing ah %p\n", ah->ah);
 		ib_destroy_ah(ah->ah);
 		kfree(ah);
@@ -355,7 +355,7 @@ static void __ipoib_reap_ah(struct net_d
 
 	spin_lock_irq(&priv->lock);
 	list_for_each_entry_safe(ah, tah, &priv->dead_ahs, list)
-		if (ah->last_send <= priv->tx_tail) {
+		if ((int) priv->tx_tail - (int) ah->last_send >= 0) {
 			list_del(&ah->list);
 			list_add_tail(&ah->list, &remove_list);
 		}
@@ -486,7 +486,7 @@ int ipoib_ib_dev_stop(struct net_device 
 			 * assume the HW is wedged and just free up
 			 * all our pending work requests.
 			 */
-			while (priv->tx_tail < priv->tx_head) {
+			while ((int) priv->tx_tail - (int) priv->tx_head < 0) {
 				tx_req = &priv->tx_ring[priv->tx_tail &
 							(IPOIB_TX_RING_SIZE - 1)];
 				dma_unmap_single(priv->ca->dma_device,
======== end ========

Id: 7b0c5a421d9fe71c6d72d1562bb9be8d34680cd5
tree 1f825143f41fd79eee4d246885d631aed840bf0e
parent a42b5922ba01c304e81ecb94996b4f5b5def4c5f
author Roland Dreier <roland at eddore.topspincom.com> 1122349504 -0700
committer Roland Dreier <roland at eddore.topspincom.com> 1122349504 -0700

From: Michael S. Tsirkin <mst at mellanox.co.il>

Use io_remap_pfn_range to remap IO pages (remap_pfn_range is for memory).

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
Signed-off-by: Roland Dreier <rolandd at cisco.com>


======== diff against a42b5922ba01c304e81ecb94996b4f5b5def4c5f ========
diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c
--- a/drivers/infiniband/hw/mthca/mthca_provider.c
+++ b/drivers/infiniband/hw/mthca/mthca_provider.c
@@ -349,9 +349,9 @@ static int mthca_mmap_uar(struct ib_ucon
 
 	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 
-	if (remap_pfn_range(vma, vma->vm_start,
-			    to_mucontext(context)->uar.pfn,
-			    PAGE_SIZE, vma->vm_page_prot))
+	if (io_remap_pfn_range(vma, vma->vm_start,
+			       to_mucontext(context)->uar.pfn,
+			       PAGE_SIZE, vma->vm_page_prot))
 		return -EAGAIN;
 
 	return 0;
======== end ========


From halr at voltaire.com  Wed Jul 27 11:45:19 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 14:45:19 -0400
Subject: [openib-general] Proposed updates for 2.6.13
In-Reply-To: <52sly0w2m3.fsf@cisco.com>
References: <52sly0w2m3.fsf@cisco.com>
Message-ID: <1122489918.4451.67.camel@hal.voltaire.com>

On Wed, 2005-07-27 at 14:00, Roland Dreier wrote:
> Here is the set of patches I am planning on sending upstream this week
> for inclusion in 2.6.13.  In addition to bug fixes, I am including
> fasync support for uverbs -- it seems worth having this in 2.6.13, so
> that code can rely on it any released kernel.
> 
> Please let me know if you think there are other changes we should send
> upstream, or if you think some of the changes I'm including here
> should be held for 2.6.14.

One other change to consider is updating ucm as follows:

r2886 | libor | 2005-07-20 14:27:14 -0400 (Wed, 20 Jul 2005) | 7 lines
Changed paths:
   M /gen2/trunk/src/linux-kernel/infiniband/core/ucm.c

From: Hal Rosenstock <halr at voltaire.com>

Only print messages when debug level set

Signed-off-by: Hal Rosenstock <halr at voltaire.com>
Signed-off-by: Libor Michalek <libor at topspin.com>

Things for 2.6.14:

r2868 | sean.hefty | 2005-07-15 19:34:32 -0400 (Fri, 15 Jul 2005) | 4
lines
Changed paths:
   M /gen2/trunk/src/linux-kernel/infiniband/core/mad_rmpp.c
   M /gen2/trunk/src/linux-kernel/infiniband/include/ib_mad.h

Add handling for ABORT / STOP RMPP MADs.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>

r2900 | halr | 2005-07-25 08:10:35 -0400 (Mon, 25 Jul 2005) | 5 lines
Changed paths:
   M /gen2/trunk/src/linux-kernel/infiniband/core/mad.c

Changed ib_mad_thread_completion_handler to conform to ib_comp_handler
declaration

Signed-off-by: Hal Rosenstock <halr at voltaire.com>


Also, various sparse warning changes


The only other changes have to do with copyrights. Should they be pushed
for 2.6.13 ?

-- Hal




From rolandd at cisco.com  Wed Jul 27 12:12:35 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 12:12:35 -0700
Subject: [openib-general] Proposed updates for 2.6.13
In-Reply-To: <1122489918.4451.67.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "27 Jul 2005 14:45:19 -0400")
References: <52sly0w2m3.fsf@cisco.com>
	<1122489918.4451.67.camel@hal.voltaire.com>
Message-ID: <52fyu0vza4.fsf@cisco.com>

    Hal> Only print messages when debug level set

I just saw Andrew send all the -mm changes up to Linus, so I'll queue
this one up as well.

    Hal> The only other changes have to do with copyrights. Should
    Hal> they be pushed for 2.6.13 ?

I don't think it's a good idea at this point in the release cycle.

 - R.


From steve_wooding at keysounds.co.uk  Wed Jul 27 12:27:27 2005
From: steve_wooding at keysounds.co.uk (Steve Wooding)
Date: Wed, 27 Jul 2005 20:27:27 +0100
Subject: [openib-general] Compile issue with 2.6.12 + realtime preemt patch
Message-ID: <42E7E01F.9040500@keysounds.co.uk>

Hi,

I would like to use the OpenIb drivers with a 2.6.12 kernel with the 
realtime-preemt patch maintained by Ingo Molnar. I get the following 
error in sdp_conn.c using svn 2909:

line 49: error: SPIN_LOCK_UNLOCKED undeclared in this function

Has anybody out there got this working or could give me clue how to fix it?

Cheers,

Steve.


From rolandd at cisco.com  Wed Jul 27 13:29:47 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 13:29:47 -0700
Subject: [openib-general] Compile issue with 2.6.12 + realtime preemt patch
In-Reply-To: <42E7E01F.9040500@keysounds.co.uk> (Steve Wooding's message of
	"Wed, 27 Jul 2005 20:27:27 +0100")
References: <42E7E01F.9040500@keysounds.co.uk>
Message-ID: <527jfcvvpg.fsf@cisco.com>

    Steve> Hi, I would like to use the OpenIb drivers with a 2.6.12
    Steve> kernel with the realtime-preemt patch maintained by Ingo
    Steve> Molnar. I get the following error in sdp_conn.c using svn
    Steve> 2909:

    Steve> line 49: error: SPIN_LOCK_UNLOCKED undeclared in this
    Steve> function

    Steve> Has anybody out there got this working or could give me
    Steve> clue how to fix it?

Change

    static spinlock_t psn_lock = SPIN_LOCK_UNLOCKED;

to

    static DEFINE_SPINLOCK(psn_lock);

Libor, we might as well make this change:


Initialize spinlock with DEFINE_SPINLOCK() instead of
SPIN_LOCK_UNLOCKED to be friendly to CONFIG_PREEMPT_RT patches.

Signed-off-by: Roland Dreier <rolandd at cisco.com>

--- infiniband/ulp/sdp/sdp_conn.c	(revision 2904)
+++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -46,7 +46,7 @@ static struct ib_client sdp_client = {
 	.remove = sdp_device_remove_one
 };
 
-static spinlock_t psn_lock = SPIN_LOCK_UNLOCKED;
+static DEFINE_SPINLOCK(psn_lock);
 static u32        psn_seed;
 
 /*


From rolandd at cisco.com  Wed Jul 27 13:53:57 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 13:53:57 -0700
Subject: [openib-general] Re: [PATCH 18/82] remove linux/version.h from
	drivers/infiniband/
In-Reply-To: <20050710193526.18.UYMiOz2756.2247.olh@nectarine.suse.de> (Olaf
	Hering's message of "Sun, 10 Jul 2005 19:35:26 +0000")
References: <20050710193526.18.UYMiOz2756.2247.olh@nectarine.suse.de>
Message-ID: <52y87sug0q.fsf@cisco.com>

Thanks, I've finally applied this to our repository, and will merge it
upstream for 2.6.14 (unless it shows up by some other route first).

 - R.


From steve_wooding at keysounds.co.uk  Wed Jul 27 14:03:48 2005
From: steve_wooding at keysounds.co.uk (Steve Wooding)
Date: Wed, 27 Jul 2005 22:03:48 +0100
Subject: [openib-general] Compile issue with 2.6.12 + realtime preemt patch
In-Reply-To: <527jfcvvpg.fsf@cisco.com>
References: <42E7E01F.9040500@keysounds.co.uk> <527jfcvvpg.fsf@cisco.com>
Message-ID: <42E7F6B4.3010705@keysounds.co.uk>

That's great Roland.

Thanks a lot.

Cheers,

Steve.

Roland Dreier wrote:

>    Steve> Hi, I would like to use the OpenIb drivers with a 2.6.12
>    Steve> kernel with the realtime-preemt patch maintained by Ingo
>    Steve> Molnar. I get the following error in sdp_conn.c using svn
>    Steve> 2909:
>
>    Steve> line 49: error: SPIN_LOCK_UNLOCKED undeclared in this
>    Steve> function
>
>    Steve> Has anybody out there got this working or could give me
>    Steve> clue how to fix it?
>
>Change
>
>    static spinlock_t psn_lock = SPIN_LOCK_UNLOCKED;
>
>to
>
>    static DEFINE_SPINLOCK(psn_lock);
>
>Libor, we might as well make this change:
>
>
>Initialize spinlock with DEFINE_SPINLOCK() instead of
>SPIN_LOCK_UNLOCKED to be friendly to CONFIG_PREEMPT_RT patches.
>
>Signed-off-by: Roland Dreier <rolandd at cisco.com>
>
>--- infiniband/ulp/sdp/sdp_conn.c	(revision 2904)
>+++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
>@@ -46,7 +46,7 @@ static struct ib_client sdp_client = {
> 	.remove = sdp_device_remove_one
> };
> 
>-static spinlock_t psn_lock = SPIN_LOCK_UNLOCKED;
>+static DEFINE_SPINLOCK(psn_lock);
> static u32        psn_seed;
> 
> /*
>
>  
>



From tduffy at sun.com  Wed Jul 27 14:13:18 2005
From: tduffy at sun.com (Tom Duffy)
Date: Wed, 27 Jul 2005 14:13:18 -0700
Subject: [openib-general] [PATCH] SDP: remove linux/version.h
Message-ID: <1122498798.6427.7.camel@duffman>

[ Libor, sorry for the dup's, my sendmail is fucked ]

No need for linux/version.h in SDP, either.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_main.h
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_main.h	(revision 2918)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/ulp/sdp/sdp_main.h	(working copy)
@@ -69,7 +69,6 @@
 /*
  * kernel includes
  */
-#include <linux/version.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/module.h>




From limichal at cisco.com  Wed Jul 27 14:30:45 2005
From: limichal at cisco.com (Libor Michalek)
Date: Wed, 27 Jul 2005 14:30:45 -0700
Subject: [openib-general] Re: [PATCH] SDP: remove linux/version.h
In-Reply-To: <1122498798.6427.7.camel@duffman>;
	from tduffy@sun.com on Wed, Jul 27, 2005 at 02:13:18PM -0700
References: <1122498798.6427.7.camel@duffman>
Message-ID: <20050727143045.A14399@topspin.com>

On Wed, Jul 27, 2005 at 02:13:18PM -0700, Tom Duffy wrote:
> [ Libor, sorry for the dup's, my sendmail is fucked ]
> 
> No need for linux/version.h in SDP, either.

  No problem, I've removed the include.

-Libor


From rolandd at cisco.com  Wed Jul 27 16:17:45 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 16:17:45 -0700
Subject: [openib-general] git tree on kernel.org
Message-ID: <52oe8nsusm.fsf@cisco.com>

I've created a git tree of patches to be sent upstream to Linus.  You
can pull it from

    http://www.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

or browse it with the web interface at

    http://www.kernel.org/git/?p=linux/kernel/git/roland/infiniband.git;a=summary

So far, this tree is just the head of Linus's tree plus the patches
listed below.  Once Linus pulls in the latest batch of IB patches from
Andrew, I'll add the ucm debug change from Hal and push this tree to Linus.

 - R.

commit 6d376756f2cf3478d5a4fdb8d18e958948366b9d
Author: Michael S. Tsirkin <mst at mellanox.co.il>
Date:   Wed Jul 27 14:42:45 2005 -0700

    [IB/mthca]: Use io_remap_pfn_range for PCI space
    
    Use io_remap_pfn_range to remap IO pages (remap_pfn_range is for memory).
    
    Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit 2181858bb814b51de8ec25b3ddd37cd06c53b0c9
Author: Roland Dreier <roland at eddore.topspincom.com>
Date:   Wed Jul 27 14:41:32 2005 -0700

    [IB/ipoib]: Fix unsigned comparisons to handle wraparound
    
    Fix handling of tx_head/tx_tail comparisons to handle wraparound.
    
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit abdf119b4dad015803819c3d046d20cfbd393e87
Author: Gleb Natapov <glebn at voltaire.com>
Date:   Wed Jul 27 14:40:00 2005 -0700

    [IB/uverbs]: Add O_ASYNC support
    
    Add support for O_ASYNC notifications on userspace verbs
    completion and asynchronous event file descriptors.
    
    Signed-off-by: Gleb Natapov <glebn at voltaire.com>
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit 42b1806d5cfc93bf8c3d7fa6e9e79e4ec860c678
Author: Roland Dreier <roland at eddore.topspincom.com>
Date:   Wed Jul 27 14:38:49 2005 -0700

    [IB/mthca]: Fix error CQ entry handling on mem-free HCAs
    
    Fix handling of error CQ entries on mem-free HCAs: the doorbell count
    is never valid so we shouldn't look at it.  This fixes problems exposed
    by new HCA firmware.
    
    Signed-off-by: Roland Dreier <rolandd at cisco.com>


From ardavis at ichips.intel.com  Wed Jul 27 17:39:46 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Wed, 27 Jul 2005 17:39:46 -0700
Subject: [openib-general] ib_uat kernel oops
Message-ID: <42E82952.2020302@ichips.intel.com>

Hi Hal,

I move from a 2.6.11 kernel to 2.6.12.3 and am now having some problems 
with IBAT. (svn 2919)

Running the simple example...

[ardavis at iclust-19 examples]$ ./uatt
uatt: main: uat test start
uatt: main: ib_at_route_by_ip: ret 0 errno 0 for request 1 id 1 1

[ardavis at iclust-19 examples]$
Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
iclust-19 kernel: Oops: 0002 [1] SMP

Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
iclust-19 kernel: CR2: 0000000000000000

Jul 27 17:36:41 iclust-19 kernel: Unable to handle kernel NULL pointer 
dereference at 0000000000000000 RIP:
Jul 27 17:36:41 iclust-19 kernel: 
<ffffffff88057310>{:ib_uat:ib_uat_callback+155}
Jul 27 17:36:41 iclust-19 kernel: PGD 34a77067 PUD 34a65067 PMD 0
Jul 27 17:36:41 iclust-19 kernel: Oops: 0002 [1] SMP
Jul 27 17:36:41 iclust-19 kernel: CPU 1
Jul 27 17:36:41 iclust-19 kernel: Modules linked in: ib_att ib_uat ib_at 
ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib  ib_sa ib_mthca ib_mad ib_core
Jul 27 17:36:41 iclust-19 kernel: Pid: 2248, comm: ib_at_wq/1 Not 
tainted 2.6.12.3
Jul 27 17:36:41 iclust-19 kernel: RIP: 0010:[<ffffffff88057310>] 
<ffffffff88057310>{:ib_uat:ib_uat_callback+155}
Jul 27 17:36:41 iclust-19 kernel: RSP: 0018:ffff8100362ade38  EFLAGS: 
00010282
Jul 27 17:36:41 iclust-19 kernel: RAX: ffff8100386384b0 RBX: 
ffff81003e8a5e00 RCX: 0000000000000000
Jul 27 17:36:41 iclust-19 kernel: RDX: ffff81003e8a5e10 RSI: 
0000000000000001 RDI: ffff81003e8a5e00
Jul 27 17:36:41 iclust-19 kernel: RBP: ffff810038638480 R08: 
0000000000000000 R09: 0000000000000000
Jul 27 17:36:41 iclust-19 kernel: R10: ffff81003e8a5e00 R11: 
0000000000000050 R12: 0000000000000001
Jul 27 17:36:41 iclust-19 kernel: R13: 0000000000000292 R14: 
ffff81002da5de50 R15: ffffffff880523ed
Jul 27 17:36:41 iclust-19 kernel: FS:  0000000000000000(0000) 
GS:ffffffff8058e580(0000) knlGS:0000000000000000
Jul 27 17:36:41 iclust-19 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 
000000008005003b
Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000 CR3: 
0000000034a3b000 CR4: 00000000000006e0
Jul 27 17:36:41 iclust-19 kernel: Process ib_at_wq/1 (pid: 2248, 
threadinfo ffff8100362ac000, task ffff81003ea6e f50)
Jul 27 17:36:41 iclust-19 kernel: Stack: ffff81002da5de50 
ffff81002da5de78 ffff810038501880 ffffffff88052408
Jul 27 17:36:41 iclust-19 kernel:        ffff81002da5de70 
ffffffff8014363e ffff8100385018c0 ffff810038501898
Jul 27 17:36:41 iclust-19 kernel:        ffff8100385018a8 ffffffffffffffff
Jul 27 17:36:41 iclust-19 kernel: Call 
Trace:<ffffffff88052408>{:ib_at:req_comp_work+27} 
<ffffffff8014363e>{work er_thread+501}
Jul 27 17:36:41 iclust-19 kernel:        
<ffffffff8013019d>{default_wake_function+0} <ffffffff8013019d>{default_ 
wake_function+0}
Jul 27 17:36:41 iclust-19 kernel:        
<ffffffff80147520>{keventd_create_kthread+0} <ffffffff80143449>{worker_ 
thread+0}
Jul 27 17:36:41 iclust-19 kernel:        
<ffffffff80147520>{keventd_create_kthread+0} <ffffffff801474f3>{kthread 
+204}
Jul 27 17:36:41 iclust-19 kernel:        <ffffffff8010f1a3>{child_rip+8} 
<ffffffff80147520>{keventd_create_kthre ad+0}
Jul 27 17:36:41 iclust-19 kernel:        <ffffffff80147427>{kthread+0} 
<ffffffff8010f19b>{child_rip+0}
Jul 27 17:36:41 iclust-19 kernel:
Jul 27 17:36:41 iclust-19 kernel:
Jul 27 17:36:41 iclust-19 kernel: Code: 48 89 11 48 8b 48 08 48 8d 53 20 
48 89 43 20 48 89 50 08 48
Jul 27 17:36:41 iclust-19 kernel: RIP 
<ffffffff88057310>{:ib_uat:ib_uat_callback+155} RSP <ffff8100362ade38>
Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000
.

any ideas?

Thanks,

-arlin




From halr at voltaire.com  Wed Jul 27 18:27:03 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 27 Jul 2005 21:27:03 -0400
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <42E82952.2020302@ichips.intel.com>
References: <42E82952.2020302@ichips.intel.com>
Message-ID: <1122514023.4451.70.camel@hal.voltaire.com>

Hi Arlin,

On Wed, 2005-07-27 at 20:39, Arlin Davis wrote:
> I move from a 2.6.11 kernel to 2.6.12.3 and am now having some problems 
> with IBAT. (svn 2919)
> 
> Running the simple example...
> 
> [ardavis at iclust-19 examples]$ ./uatt
> uatt: main: uat test start
> uatt: main: ib_at_route_by_ip: ret 0 errno 0 for request 1 id 1 1
> 
> [ardavis at iclust-19 examples]$
> Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
> iclust-19 kernel: Oops: 0002 [1] SMP
> 
> Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
> iclust-19 kernel: CR2: 0000000000000000
> 
> Jul 27 17:36:41 iclust-19 kernel: Unable to handle kernel NULL pointer 
> dereference at 0000000000000000 RIP:
> Jul 27 17:36:41 iclust-19 kernel: 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155}
> Jul 27 17:36:41 iclust-19 kernel: PGD 34a77067 PUD 34a65067 PMD 0
> Jul 27 17:36:41 iclust-19 kernel: Oops: 0002 [1] SMP
> Jul 27 17:36:41 iclust-19 kernel: CPU 1
> Jul 27 17:36:41 iclust-19 kernel: Modules linked in: ib_att ib_uat ib_at 
> ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib  ib_sa ib_mthca ib_mad ib_core
> Jul 27 17:36:41 iclust-19 kernel: Pid: 2248, comm: ib_at_wq/1 Not 
> tainted 2.6.12.3
> Jul 27 17:36:41 iclust-19 kernel: RIP: 0010:[<ffffffff88057310>] 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155}
> Jul 27 17:36:41 iclust-19 kernel: RSP: 0018:ffff8100362ade38  EFLAGS: 
> 00010282
> Jul 27 17:36:41 iclust-19 kernel: RAX: ffff8100386384b0 RBX: 
> ffff81003e8a5e00 RCX: 0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: RDX: ffff81003e8a5e10 RSI: 
> 0000000000000001 RDI: ffff81003e8a5e00
> Jul 27 17:36:41 iclust-19 kernel: RBP: ffff810038638480 R08: 
> 0000000000000000 R09: 0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: R10: ffff81003e8a5e00 R11: 
> 0000000000000050 R12: 0000000000000001
> Jul 27 17:36:41 iclust-19 kernel: R13: 0000000000000292 R14: 
> ffff81002da5de50 R15: ffffffff880523ed
> Jul 27 17:36:41 iclust-19 kernel: FS:  0000000000000000(0000) 
> GS:ffffffff8058e580(0000) knlGS:0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 
> 000000008005003b
> Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000 CR3: 
> 0000000034a3b000 CR4: 00000000000006e0
> Jul 27 17:36:41 iclust-19 kernel: Process ib_at_wq/1 (pid: 2248, 
> threadinfo ffff8100362ac000, task ffff81003ea6e f50)
> Jul 27 17:36:41 iclust-19 kernel: Stack: ffff81002da5de50 
> ffff81002da5de78 ffff810038501880 ffffffff88052408
> Jul 27 17:36:41 iclust-19 kernel:        ffff81002da5de70 
> ffffffff8014363e ffff8100385018c0 ffff810038501898
> Jul 27 17:36:41 iclust-19 kernel:        ffff8100385018a8 ffffffffffffffff
> Jul 27 17:36:41 iclust-19 kernel: Call 
> Trace:<ffffffff88052408>{:ib_at:req_comp_work+27} 
> <ffffffff8014363e>{work er_thread+501}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff8013019d>{default_wake_function+0} <ffffffff8013019d>{default_ 
> wake_function+0}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff80147520>{keventd_create_kthread+0} <ffffffff80143449>{worker_ 
> thread+0}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff80147520>{keventd_create_kthread+0} <ffffffff801474f3>{kthread 
> +204}
> Jul 27 17:36:41 iclust-19 kernel:        <ffffffff8010f1a3>{child_rip+8} 
> <ffffffff80147520>{keventd_create_kthre ad+0}
> Jul 27 17:36:41 iclust-19 kernel:        <ffffffff80147427>{kthread+0} 
> <ffffffff8010f19b>{child_rip+0}
> Jul 27 17:36:41 iclust-19 kernel:
> Jul 27 17:36:41 iclust-19 kernel:
> Jul 27 17:36:41 iclust-19 kernel: Code: 48 89 11 48 8b 48 08 48 8d 53 20 
> 48 89 43 20 48 89 50 08 48
> Jul 27 17:36:41 iclust-19 kernel: RIP 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155} RSP <ffff8100362ade38>
> Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000
> .
> 
> any ideas?

Which IBAT are you using ? It is the one in the trunk or the one on
shaharf-ibat branch ?

-- Hal



From gshipman at lanl.gov  Wed Jul 27 18:51:37 2005
From: gshipman at lanl.gov (Galen M. Shipman)
Date: Wed, 27 Jul 2005 19:51:37 -0600
Subject: [openib-general] Shared Receive Queue
In-Reply-To: <506C3D7B14CDD411A52C00025558DED6085BCFB2@mtlex01.yok.mtl.com>
References: <506C3D7B14CDD411A52C00025558DED6085BCFB2@mtlex01.yok.mtl.com>
Message-ID: <7886E0A6-F34C-47C4-8BDC-8D89524CE5E3@lanl.gov>

Tziporet,

This did appear to be a firmware issue. After the update I have a  
latency difference on the order of .2 uSec which seems reasonable,  
much better than a ~5. uSec penalty!

Thanks for the help,

Galen


On Jul 27, 2005, at 12:38 AM, Tziporet Koren wrote:

> Hi Galen,
> You are working with an old FW version that had issues with SRQ  
> performance.
> We did performance improvements and now the latency and BW with SRQ  
> are the same as with regular QP.
>
> The changes are available in the new FW release we did few weeks  
> ago: see http://www.mellanox.com/products/firmware.html
>
> Regarding your specific device: you are using MT25208 InfiniHost  
> III Ex, so you need to use FW 4.7.0
>
> In case you need help with FW burning please approach our FAE Todd  
> (cc here too).
>
> After you have new results please reply if the performance numbers  
> are OK now.
>
> Thanks,
> Tziporet
>
>  -----Original Message-----
> From: Galen M. Shipman [mailto:gshipman at lanl.gov]
> Sent: Tuesday, July 26, 2005 5:55 PM
> To: Tziporet Koren
> Subject: Re: [openib-general] Shared Receive Queue
>
> The following is from vstat..
>
>  hca_id=InfiniHost0
>         vendor_id=0x02C9
>         vendor_part_id=0x6278
>         hw_ver=0xA0
>         fw_ver=0x400050003
>         num_phys_ports=2
>                 port=1
>                 port_state=PORT_DOWN
>                 sm_lid=0x0000
>                 port_lid=0x0000
>                 port_lmc=0x00
>                 max_mtu=2048
>
>                 port=2
>                 port_state=PORT_ACTIVE
>                 sm_lid=0x0001
>                 port_lid=0x010a
>                 port_lmc=0x00
>                 max_mtu=2048
>
>
> On Jul 26, 2005, at 8:46 AM, Tziporet Koren wrote:
>
>> which HW & FW are you using?
>>
>> -----Original Message-----
>> From: Roland Dreier [mailto:rolandd at cisco.com]
>> Sent: Tuesday, July 26, 2005 5:38 PM
>> To: Galen M. Shipman
>> Cc: openib-general at openib.org
>> Subject: Re: [openib-general] Shared Receive Queue
>>
>>
>>     Galen> We have software which uses the mellanox shared receive
>>     Galen> queue and we are seeing a substantial performance penalty
>>     Galen> (on the order of 4-5 usec on 0 byte messages). I don't  
>> know
>>     Galen> all the technical details but if you could keep this in
>>     Galen> mind when working on the shared receive queue perhaps the
>>     Galen> performance penalty could be much less in gen2.
>>
>> Is this performance penalty in the software?  It seems much more
>> likely that the issue is in the hardware implementation of SRQ.
>>
>>  - R.
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/ 
>> openib-general
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050727/676391ed/attachment.html>

From tduffy at sun.com  Wed Jul 27 19:43:14 2005
From: tduffy at sun.com (Tom Duffy)
Date: Wed, 27 Jul 2005 19:43:14 -0700
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <1122514023.4451.70.camel@hal.voltaire.com>
References: <42E82952.2020302@ichips.intel.com>
	<1122514023.4451.70.camel@hal.voltaire.com>
Message-ID: <1122518594.17830.1.camel@duffman>

On Wed, 2005-07-27 at 21:27 -0400, Hal Rosenstock wrote:
> Which IBAT are you using ? It is the one in the trunk or the one on
> shaharf-ibat branch ?

uat.[ch] isn't in trunk yet, right?

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050727/a9e59f3f/attachment.sig>

From rolandd at cisco.com  Wed Jul 27 21:20:08 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 21:20:08 -0700
Subject: [openib-general] [git pull] Final InfiniBand updates for 2.6.13
Message-ID: <52hdefsgsn.fsf@cisco.com>

Linus, please pull from

    rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

This will update the following files:

 core/ucm.c                |   88 +++++++++++++++++++++-------------------------
 core/uverbs.h             |    1 
 core/uverbs_main.c        |   14 ++++++-
 hw/mthca/mthca_cq.c       |    6 ++-
 hw/mthca/mthca_provider.c |    6 +--
 ulp/ipoib/ipoib_ib.c      |    6 +--
 6 files changed, 66 insertions(+), 55 deletions(-)

through the following changes:

commit 79d81907594e1ec4d5171653dde7cb9e9cb87de2
Author: Hal Rosenstock <halr at voltaire.com>
Date:   Wed Jul 27 20:38:56 2005 -0700

    [IB/ucm]: Clean up userspace CM
    
    Only print debug messages when debug_level is set.
    Eliminate NULL checks prior to calling kfree.
    
    Signed-off-by: Hal Rosenstock <halr at voltaire.com>
    Signed-off-by: Libor Michalek <libor at topspin.com>

commit 6d376756f2cf3478d5a4fdb8d18e958948366b9d
Author: Michael S. Tsirkin <mst at mellanox.co.il>
Date:   Wed Jul 27 14:42:45 2005 -0700

    [IB/mthca]: Use io_remap_pfn_range for PCI space
    
    Use io_remap_pfn_range to remap IO pages (remap_pfn_range is for memory).
    
    Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit 2181858bb814b51de8ec25b3ddd37cd06c53b0c9
Author: Roland Dreier <roland at eddore.topspincom.com>
Date:   Wed Jul 27 14:41:32 2005 -0700

    [IB/ipoib]: Fix unsigned comparisons to handle wraparound
    
    Fix handling of tx_head/tx_tail comparisons to handle wraparound.
    
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit abdf119b4dad015803819c3d046d20cfbd393e87
Author: Gleb Natapov <glebn at voltaire.com>
Date:   Wed Jul 27 14:40:00 2005 -0700

    [IB/uverbs]: Add O_ASYNC support
    
    Add support for O_ASYNC notifications on userspace verbs
    completion and asynchronous event file descriptors.
    
    Signed-off-by: Gleb Natapov <glebn at voltaire.com>
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit 42b1806d5cfc93bf8c3d7fa6e9e79e4ec860c678
Author: Roland Dreier <roland at eddore.topspincom.com>
Date:   Wed Jul 27 14:38:49 2005 -0700

    [IB/mthca]: Fix error CQ entry handling on mem-free HCAs
    
    Fix handling of error CQ entries on mem-free HCAs: the doorbell count
    is never valid so we shouldn't look at it.  This fixes problems exposed
    by new HCA firmware.
    
    Signed-off-by: Roland Dreier <rolandd at cisco.com>


From gregkh at suse.de  Wed Jul 27 21:26:07 2005
From: gregkh at suse.de (Greg KH)
Date: Wed, 27 Jul 2005 21:26:07 -0700
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <20050725223200.GA1545@mellanox.co.il>
References: <20050725223200.GA1545@mellanox.co.il>
Message-ID: <20050728042607.GA12799@suse.de>

On Tue, Jul 26, 2005 at 01:32:00AM +0300, Michael S. Tsirkin wrote:
> Greg, Martin, does the following make sense?
> If it does, should other architectures be updated as well?
> 
> ---
> 
> Convert i386/pci to use io_remap_pfn_range instead of remap_pfn_range.
> 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
> 
> Index: linux-2.6.12.2/arch/i386/pci/i386.c
> ===================================================================
> --- linux-2.6.12.2.orig/arch/i386/pci/i386.c
> +++ linux-2.6.12.2/arch/i386/pci/i386.c
> @@ -295,9 +295,9 @@ int pci_mmap_page_range(struct pci_dev *
>  	/* Write-combine setting is ignored, it is changed via the mtrr
>  	 * interfaces on this platform.
>  	 */
> -	if (remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
> -			     vma->vm_end - vma->vm_start,
> -			     vma->vm_page_prot))
> +	if (io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
> +			       vma->vm_end - vma->vm_start,
> +			       vma->vm_page_prot))

Hm, you do realize that io_remap_pfn_range() is the same thing as
remap_pfn_range() on i386, right?

So, why would this patch change anything?

thanks,

greg k-h


From rolandd at cisco.com  Wed Jul 27 21:30:17 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Wed, 27 Jul 2005 21:30:17 -0700
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <20050728042607.GA12799@suse.de> (Greg KH's message of "Wed, 27
	Jul 2005 21:26:07 -0700")
References: <20050725223200.GA1545@mellanox.co.il>
	<20050728042607.GA12799@suse.de>
Message-ID: <52d5p3sgbq.fsf@cisco.com>

    Greg> Hm, you do realize that io_remap_pfn_range() is the same
    Greg> thing as remap_pfn_range() on i386, right?

    Greg> So, why would this patch change anything?

It's not the same thing under Xen.  I think this patch fixes userspace
access to PCI memory for XenLinux.

 - R.


From gregkh at suse.de  Wed Jul 27 21:50:05 2005
From: gregkh at suse.de (Greg KH)
Date: Wed, 27 Jul 2005 21:50:05 -0700
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <52d5p3sgbq.fsf@cisco.com>
References: <20050725223200.GA1545@mellanox.co.il>
	<20050728042607.GA12799@suse.de> <52d5p3sgbq.fsf@cisco.com>
Message-ID: <20050728045005.GA13070@suse.de>

On Wed, Jul 27, 2005 at 09:30:17PM -0700, Roland Dreier wrote:
>     Greg> Hm, you do realize that io_remap_pfn_range() is the same
>     Greg> thing as remap_pfn_range() on i386, right?
> 
>     Greg> So, why would this patch change anything?
> 
> It's not the same thing under Xen.  I think this patch fixes userspace
> access to PCI memory for XenLinux.

But Xen is a separate arch, and hence, will get different pci arch
specific functions, right?

In short, what is this patch trying to fix?  What is the problem anyone
is seeing with the existing code?

thanks,

greg k-h


From mst at mellanox.co.il  Wed Jul 27 23:48:59 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Thu, 28 Jul 2005 09:48:59 +0300
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <20050728042607.GA12799@suse.de>
References: <20050725223200.GA1545@mellanox.co.il>
	<20050728042607.GA12799@suse.de>
Message-ID: <20050728064859.GA11644@mellanox.co.il>

Quoting r. Greg KH <gregkh at suse.de>:
> Subject: Re: [PATCH] arch/xx/pci: remap_pfn_range -> io_remap_pfn_range
> 
> On Tue, Jul 26, 2005 at 01:32:00AM +0300, Michael S. Tsirkin wrote:
> > Greg, Martin, does the following make sense?
> > If it does, should other architectures be updated as well?
> > 
> Hm, you do realize that io_remap_pfn_range() is the same thing as
> remap_pfn_range() on i386, right?
> 
> So, why would this patch change anything?
> 
> thanks,
> 
> greg k-h
> 

It doesnt change any behaviour.
Its a style issue I'm trying to fix: even in arch specific code, isnt it better
to use a generic io_remap_pfn_range to map PCI memory?

-- 
MST


From gregkh at suse.de  Wed Jul 27 23:58:47 2005
From: gregkh at suse.de (Greg KH)
Date: Wed, 27 Jul 2005 23:58:47 -0700
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <20050728064859.GA11644@mellanox.co.il>
References: <20050725223200.GA1545@mellanox.co.il>
	<20050728042607.GA12799@suse.de>
	<20050728064859.GA11644@mellanox.co.il>
Message-ID: <20050728065847.GA18003@suse.de>

On Thu, Jul 28, 2005 at 09:48:59AM +0300, Michael S. Tsirkin wrote:
> Quoting r. Greg KH <gregkh at suse.de>:
> > Subject: Re: [PATCH] arch/xx/pci: remap_pfn_range -> io_remap_pfn_range
> > 
> > On Tue, Jul 26, 2005 at 01:32:00AM +0300, Michael S. Tsirkin wrote:
> > > Greg, Martin, does the following make sense?
> > > If it does, should other architectures be updated as well?
> > > 
> > Hm, you do realize that io_remap_pfn_range() is the same thing as
> > remap_pfn_range() on i386, right?
> > 
> > So, why would this patch change anything?
> > 
> > thanks,
> > 
> > greg k-h
> > 
> 
> It doesnt change any behaviour.
> Its a style issue I'm trying to fix: even in arch specific code, isnt it better
> to use a generic io_remap_pfn_range to map PCI memory?

As I said above, it's the same thing on i386.
But, if you are getting picky, you should fix all of the other arches
too, right?

Anyway, why change this, I don't really see the need.

thanks,

greg k-h


From guyg at voltaire.com  Thu Jul 28 00:14:15 2005
From: guyg at voltaire.com (Guy German)
Date: Thu, 28 Jul 2005 10:14:15 +0300 (IDT)
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
Message-ID: <Pine.LNX.4.63.0507281007040.20468@beid.voltaire.com>

[kdapl]: increments the module ref count on ia_open, thus making
sure the kdapl_ib module would not be rmmod-ed while in use

Signed-off-by: Guy German <guyg at voltaire.com>

Index: infiniband/ulp/kdapl/ib/dapl_ia.c
===================================================================
--- infiniband/ulp/kdapl/ib/dapl_ia.c	(revision 2914)
+++ infiniband/ulp/kdapl/ib/dapl_ia.c	(working copy)
@@ -649,6 +649,7 @@ int dapl_ia_open(const char *name, int a
  	status = 0;
  	*ia_ptr = (struct dat_ia *)ia;
  	*async_evd = (struct dat_evd *)evd;
+	try_module_get(THIS_MODULE);

  bail:
  	if (status != 0)
@@ -679,6 +680,7 @@ int dapl_ia_close(struct dat_ia *ia_ptr,
  	else
  		status = -EINVAL;

+	module_put(THIS_MODULE);
  	return status;
  }



From guyg at voltaire.com  Thu Jul 28 00:20:25 2005
From: guyg at voltaire.com (Guy German)
Date: Thu, 28 Jul 2005 10:20:25 +0300 (IDT)
Subject: [openib-general][PATCH][kdapltest]: inc/dec module ref count
Message-ID: <Pine.LNX.4.63.0507281014170.20468@beid.voltaire.com>

[kdapltest]: Prevents the oops occurred when removing the kdapltest module 
while kdapltest server is up.

Index: dapltest/kdapl/kdapl_module.c
===================================================================
--- dapltest/kdapl/kdapl_module.c	(revision 2899)
+++ dapltest/kdapl/kdapl_module.c	(working copy)
@@ -55,6 +55,8 @@ static int kdapltest_open(struct inode *
  {
  #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
  	MOD_INC_USE_COUNT;
+#else
+	try_module_get(THIS_MODULE);
  #endif
  	return 0;
  }
@@ -63,6 +65,8 @@ static int kdapltest_release(struct inod
  {
  #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
  	MOD_DEC_USE_COUNT;
+#else
+	module_put(THIS_MODULE);
  #endif
  	return 0;
  }


From kumiko_shibazaki at happy.moonlovezw.st  Thu Jul 28 03:22:28 2005
From: kumiko_shibazaki at happy.moonlovezw.st (kumiko_shibazaki at happy.moonlovezw.st)
Date: Thu, 28 Jul 2005 03:22:28 -0700 (PDT)
Subject: [openib-general] =?iso-2022-jp?b?GyRCQGhGfCRPTS1GcSQmOGYbKEI=?=
	=?iso-2022-jp?b?GyRCOkIkJCReJDckPxsoQm0oXyBfKW0=?=
Message-ID: 20050728190722.37296mail@mail.v--v-qpqpqp5789siftnet.lovey01.cc

-[PR]-$B"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\(B
$B!y!#!&!,!,!&!#!z!,!&!#!#!&!,!y!#!&!,!,!z!#!&!,!,!&!#!y!,!&!#!#!&!,!z(B

$B!!Bg(-?M(-$N(-Nx(-0&(-$r(-$7(-$^(-$;(-$s(-$+(-!zL5NA%a!<%k%"%I%l%9IU(B
$B!!(,(0(,(0(,(0(,(0(,(0(,(0(,(0(,(0(,(0(,(0(,(0!z0B?4%;%-%e%j%F%#!<(B
$B!!(B18$B:P0J>e8BDj%5%$%H$X(B
$B!!:#$9$02q0wEPO?!!!!"*(B $B!!(Bhttp://1251.cc/?a001

$B!y!#!&!,!,!&!#!z!,!&!#!#!&!,!y!#!&!,!,!z!#!&!,!,!&!#!y!,!&!#!#!&!,!z(B

-[PR]-$B"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\"#!\(B

$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"!9-9p"!Ck$+$i2K$7$F$k%W!<$N=w$N;R$HB(%(!{%A!*(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B!!!!!Z!!@dBP at -8r<g5A!!![0lDjCK=wHfN(!*(B
$B!!!!!!!!!!(B $B%a!<%kJV?.N(!"=P2q$$$N3NN($b>e>:!*!*!!(B  
$B!!!!!!40A4L5NA$GM7$Y$k!*3(J8;z!y<L%a!<%kBP1~!*0B?4$7$FM7$Y$k(B

$B!{!!$_$f$-!!!{(B
$B!!!!!!!!!!!!!!(Bhttp://1251.cc/?a001
26$B:P(BOL$B$r$d$C$F$^$9!#;d!V%(%C%AA0Ds!W$GCK at -$H=P2q$$;O$a$F$+$i:#F|$^$G!"(B
$B$[$\KhF|CK$N?M$H2q$C$F$^$9#w!*$J$s$+$9$C$+$j%O%^$C$F$7$^$C(B
$B$F!"BgBN2q$&$@$1$8$c$J$/$F%(%C%A$7$A$c$C$F$k!#:G9b!"0lF|$K;0?M$N?M$H%((B
$B%C%A$7$?$3$H$b$"$j$^$9!#(B

$B!Z40A4L5NA%(%C%AA0Ds![(B

$B!!!!!!!!!!"M!!(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"!9-9p"!CO0hJL8!:w$G$*6a$/$N$*Aj<j$,8+$D$+$k!*(B
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B!c!!:G?7%(%s%H%j!<>pJs!!!d(B
$B"#(B8$B!'(B35$B"#(B
$BCO!!!!!!!!0h!!!!Bg:eI\(B
$BL>!!!!!!!!A0!!!!%T!<%A(B
$BG/!!!!!!!!Np!!!!#2#1:P(B
$B at 1(B $B!!!!!!(B $B:B(B $B!!(B $B$*$R$D$8:B!#(B
$B7l(B   $B1U(B   $B7?!!!!(BA$B7?(B
$B<q!!!!!!!!L#!!!!$*<r$r0{$`;v!#(B
$B0l!!!!!!!!8@!!!!$N$s$S$j20$@$1$I0l at 87|L?4hD%$j$^$9!*%"%I8r494uK>$G$9!*(B

$B!!!!!!!!!!!!!XBg:e=P2q$$$O$3$A$i$+$i!Y(B

$B!!!!!!!!!!!!!!(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"!9-9p"!CO0hJL8!:w$G$*6a$/$N$*Aj<j$,8+$D$+$k!*(B
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B!c!!:G?7%(%s%H%j!<>pJs!!!d(B
$B"#(B9$B!'(B05$B"#(B
$BCO!!!!!!!!0h!!!!:k6L8)(B
$BL>!!!!!!!!A0!!!!$+$N$s(B
$BG/!!!!!!!!Np!!!!(B2$B#1:P(B
$B at 1(B $B!!!!!!(B $B:B(B $B!!(B $B$7$7:B(B
$B7l(B   $B1U(B   $B7?!!!!(BB$B7?(B
$B<q!!!!!!!!L#!!!!%I%i%$%V(B
$B0l!!!!!!!!8@!!!!3X at 8$@$+$i$$$D$G$b%R%^$@$h!J!1"`!1!(!KD>%"%I$8$c$J$$$H(B
$B!!!!!!!!!!!!!!!!%a!<%k$G$-$J$$$+$i59$7$/!*(B

$B!!!!!!!!!!!!!X:k6L=P2q$$$O$3$A$i$+$i!Y(B
$B!!!!!!!!!!!!!!(B
$B!!!!!!!!!!!!(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"!9-9p"!D>%"%I$P$i;5$-%5%$%H!*!*(B
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B!c!!:G?7%(%s%H%j!<>pJs!!!d(B
$B"#(B9$B!'(B15$B"#(B
$BCO!!!!!!!!0h!!!!L>8E20(B
$BL>!!!!!!!!A0!!!!$/$^$A$c$s(B
$BG/!!!!!!!!Np!!!!#2#5:P(B
$B at 1(B $B!!!!!!(B $B:B(B $B!!(B $B$U$?$4:B!#(B
$B7l(B   $B1U(B   $B7?!!!!(BAB$B7?(B
$B<q!!!!!!!!L#!!!!%+%i%*%1(B
$B0l!!!!!!!!8@!!!!:#F|$+$i$O$8$a$^$7$?(B(*$B!-(I*(B`*)(IN_/(B(((II(B(_ _ (II(B)(IV[<85H6^2<O=(B

$B!!!!!!!!!!!!!!!XL>8E20=P2q$$$O$3$A$i$+$i!Y(B
$B!!!!!!!!!!!!!!(B
$B!!!!!!!!!!!!!!!!(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B"!9-9p"!2q$($k$^$GJ]>c!*!*0B?4%5%$%H!*(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B!c!!:G?7%(%s%H%j!<>pJs!!!d(B
$B"#(B10$B!'(B15$B"#(B
$BCO!!!!!!!!0h!!!!El5~ET(B
$BL>!!!!!!!!A0!!!!Nh;R(B
$BG/!!!!!!!!Np!!!!#3#9:P(B
$B at 1(B $B!!!!!!(B $B:B(B $B!!(B $B$+$K:B(B
$B7l(B   $B1U(B   $B7?!!!!(BB$B7?(B
$B<q!!!!!!!!L#!!!!FI=q(B
$B0l!!!!!!!!8@!!!!IJ at n$K=;$s$G$k<gIX$G$9!#J?F|!"$*2q$$$G$-$kCK at -$r(B
$B!!!!!!!!!!!!!!!!C5$7$F$^$9!#>r7oEy65$($FD:$1$^$9$+!)4JC1$J%W%m%U$O(B
$B!!!!!!!!!!!!!!!!(B158$B!"(B45$B$G%9%j!<%5%$%:$O(B85$B!"(B60$B!"(B87(D)$B$G%9%?%$%k$OG/Np$N(B
$B!!!!!!!!!!!!!!!!$o$j$K<+?.$,$"$k$[$&$G$9!#59$7$/$*4j$$CW$7$^$9!#(B

$B!!!!!!!!!!!!!!!XEl5~=P2q$$$O$3$A$i$+$i!Y(B
$B!!(B
$B!!!!!!!!!!!!!!!!(Bhttp://1251.cc/?a001

$B(,(,#H$J7G<(HD(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(B

$B"((B $B#H$J9pGr7G<(HD$+$iH4?h(B

$B%+%-%3!'7{0l(B
$B%?%$%H%k!'=P2q$$7O$G$3$s$J;v$7$A$c$$$^$7$?!#(B

$B#P#C$N=P2q$$7O$GCN$j9g$C$??M:J!J#3#2:P!K$H(B6/25($BEZ(B)$B$K(B
$BM7$s$G$-$^$7$?!#(B

$B#P#C=P2q$$7O%5%$%H$G#2=54V%a!<%k$7$F2q$($^$7$?!#(B
$BBT$A9g$o$;$K$-$?$=$N?M:J$O6;$,$"$k3d$K$O!"OS$dB-$,:Y$/(B
$B2q$C$?$P$+$j$J$N$K>/$76=J3$7$F$7$^$$$^$7$?!#(B
$B%a!<%k$G!V?eB24[$K9T$-$?$$!*!W$H8@$o$l$F$$$?$N$G(B
$B9T$/;v$K$7$^$7$?!#(B
$B9T$-$N<V$NCf$GH`=w$,2?EY$bBN$K?($l$F$-$?$N$G!"(B
$B$3$l$O$+$J$j9%46?($J$N$+$J!)$H!&!&!&!&!#(B

$B!!(Bhttp://1251.cc/?a001

$B?eB24[$NCf$ONC$7$+$C$?$N$G$9$,!"<j$r7R$$$GJb$$$F$$$?$N$G(B
$BFs?M6&0U<1$7$F$+!"<j$K4@$r$+$$$F$$$^$7$?!#(B
$B!!(B
$BC6Fa$5$s$H$OIaCJ<j$r7R$$$GJb$/;v$J$I$J$$$=$&$G!"(B
$B5W!9$K<j$r7R$$$@$HH`=w$O4r$7$=$&$G$7$?!#(B
$B$7$P$i$/$9$k$H!"26$NOS$K6;$r2!$7IU$1$F$-$^$7$?!#(B
$B>/$7Gv0E$$>l=j$G%&%(%9%H$d9x!&!&!&$*?,$K<j$r2s$7$^$7$?$,(B
$B7y$,$kAG?6$j$bL5$7$G$9!#(B


$B?);v$r$7$F!"5"$j$N<V$NCf$G%[%F%k$KM6$C$F$_$^$7$?$,(B
$B!V:#F|2q$C$?$P$+$j$@$7!&!&!&!W$HCG$i$l$^$7$?!#(B
$B$G$b!"1X$K6a$$8x1`$N=j$K<V$r;_$a$F!V%-%9$7$F$b%$%$!)!W(B
$B$HJ9$/$H!"L\$rJD$8$F$/$l$^$7$?!#(B

$B!!(Bhttp://1251.cc/?a001

$B%A%e%C$H7Z$/!&!&!&(B
$B$=$7$F%G%#!<%W$J%-%9(B
$B26$N<j$O?M:J$N%&%(%9%H$+$i6;$K!&!&!&!#(B
$B<j$NCf$KBg$-$J6;$,<}$^$i$J$$$[$I$G?($j?4CO$,NI$/!"(B
$B2<$+$i>e$X$HYf$_$7$@$-$^$7$?!#(B
$B$=$N0l=V!&!&!&H`=w$O?0$rN%$7$^$7$?!#(B
$B>/$76C$$$?$h$&$G$7$?$,!"(B
$B:#EY$OH`=w$NJ}$+$i99$K7c$7$$%G%#!<%W%-%9!#(B
$B%7%c%D$N2<$+$i<j$rF~$l$FD>@\6;$r?($k$H!"(B
$BBg$-$JKD$i$_$K at h$,%D%s!*$H8G$/$J$C$F$$$^$9!#(B

$B!!(Bhttp://1251.cc/?a001

$B at e$rMm$^$;$J$,$iH`=w$N at m$C$?@h$r$D$^$_!"(B
$BYf$_$7$@$/$HH`=w$NB)$O$@$s$@$s$H7c$7$/$J$C$F$$$-$^$7$?!#(B
$B%7%c%D$r$a$/$j$=$N at m$C$?=j$r2<$+$igS$a$k$H(B
$BH`=w$OKM$r$&$D$m$JL\$G8+2<$m$7>/$7B)$r9S$/$7$^$7$?!#!"(B
$B$=$7$F!"B@$b$b$+$i<j$rF~$l$F%Q%s%F%#!<$N>e$+$i(B
$B3d$lL\$r$J$>$C$F$_$k$H!&!&!&CH$+$/G($l$F$-$F$$$^$9!#(B

$B!!(Bhttp://1251.cc/?a001

$B!V?M$K8+$i$l$A$c$&$h!&!&!&!W$HCQ$:$+$7$=$&$K(B
$BH`=w$,8@$C$F$-$^$7$?$,!"(B
$B26$K$O4X78$J$!!<!<!<!<$$!*(B
$BH`=w$,<+J,$+$i>/$7$:$D3+$$$F$/$l$?I($N4V$G(B
$B26$N<j$NF0$-$,7c$7$/$J$C$F$$$-$^$9!#(B
$B%Q%s%F%#!<$N2#$+$i;X$rF~$l$k$H!&!&!&$b$&%0%A%g%0%A%g$G$9!#(B
$B!!(B
$B!!(Bhttp://1251.cc/?a001

$B%L%k%C$H$9$s$J$j;X$rFsK\<u$1F~$l$F$/$l$?$N$G!"(B
$B%/%A%e%/%A%e$H2;$,$9$k$[$I$+$-2s$7$F$"$2$^$7$?!#(B
$BH`=w$O@<$r7|L?$K2fK}$7$F$$$^$7$?$,!&!&!&(B
$B?0$rN%$7Jz$-$D$$$F$-$F!V$"$C!"%$$C$A$c$&!A!*!W$H(B
$BBg$-$J@<$G6+$S$J$,$i2L$F$F$$$-$^$7$?!#(B

$B!!(Bhttp://1251.cc/?a001

$BEE<V$N;~4V$,$;$^$C$F$-$?$N$G!"26$OGzH/@#A0$NKV5/$7$?%A!{%3$N(B
$B$^$^<V$G1X$^$GAw$C$F$$$-$^$7$?!#2fK}$7$FGzH/$7$J$+$C$?$*$+$2!)$J$N$+!"(B
$B:#EY$O$b$C$H$f$C$/$j$7$h$&$M!*(B
$B$H!"2q$&LsB+$r$7$FJL$l$^$7$?!#(B

$B"'$3$3$N%5%$%H$O$*4+$a$G$9$h!*"'(B
$B!!(B
$B!!(Bhttp://1251.cc/?a001

$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B
$B!Z$4MxMQ0FFb![(B
$B!&%a!<%kAw$jJ|Bj(B!$B<u?.$bL5NA(B!
$B!&4|8B$,Mh$?$i<+F0E*$KB`2q!#K:$l$F$$$F$&$C$+$jDI2CNA6b!"$J$I$H$$$&;v$b(B
$B!!$J$/0B?40BA4%7%9%F%`$G$9!#(B
$B!!0B?4%5%$%H$O$3$A$i$+$i!!"M!!(Bhttp://1251.cc/?a001
$B(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(!(B



From halr at voltaire.com  Thu Jul 28 04:09:34 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 07:09:34 -0400
Subject: [openib-general][PATCH][kdapltest]: inc/dec module ref count
In-Reply-To: <Pine.LNX.4.63.0507281014170.20468@beid.voltaire.com>
References: <Pine.LNX.4.63.0507281014170.20468@beid.voltaire.com>
Message-ID: <1122548973.4451.561.camel@hal.voltaire.com>

Hi Guy,

On Thu, 2005-07-28 at 03:20, Guy German wrote:
> [kdapltest]: Prevents the oops occurred when removing the kdapltest module 
> while kdapltest server is up.
> 
> Index: dapltest/kdapl/kdapl_module.c
> ===================================================================
> --- dapltest/kdapl/kdapl_module.c	(revision 2899)
> +++ dapltest/kdapl/kdapl_module.c	(working copy)
> @@ -55,6 +55,8 @@ static int kdapltest_open(struct inode *
>   {
>   #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
>   	MOD_INC_USE_COUNT;
> +#else
> +	try_module_get(THIS_MODULE);
>   #endif
>   	return 0;
>   }
> @@ -63,6 +65,8 @@ static int kdapltest_release(struct inod
>   {
>   #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
>   	MOD_DEC_USE_COUNT;
> +#else
> +	module_put(THIS_MODULE);
>   #endif
>   	return 0;
>   }

This is in the right direction.

The OpenIB policy is to support the latest kernels only. So the #if
might be removed and just the 2.6 code remain in kdapl_module.c.

-- Hal



From guyg at voltaire.com  Thu Jul 28 04:22:17 2005
From: guyg at voltaire.com (Guy German)
Date: Thu, 28 Jul 2005 14:22:17 +0300 (IDT)
Subject: [openib-general][PATCH][kdapl]: check port status in dat_ia_open
Message-ID: <Pine.LNX.4.63.0507281403150.20557@beid.voltaire.com>

[kdapl]: this patch makes sure that the consumer has opened an 
Interface Adaptor's with active hca port. In this approach, the
consumer should try opening mthca0a and if failed - mthca0b.
There are also some spacing corrections at end.. :)


Signed-off-by: Guy German <guyg at voltaire.com>

Index: infiniband/ulp/kdapl/ib/dapl_ia.c
===================================================================
--- infiniband/ulp/kdapl/ib/dapl_ia.c	(revision 2914)
+++ infiniband/ulp/kdapl/ib/dapl_ia.c	(working copy)
@@ -580,6 +580,7 @@ int dapl_ia_open(const char *name, int a
  	struct dapl_hca *hca = NULL;
  	struct dapl_ia *ia = NULL;
  	struct dapl_evd *evd;
+	struct ib_port_attr port_attr;

  	dapl_dbg_log(DAPL_DBG_TYPE_API,
  		     "dapl_ia_open (%s, %d, %p, %p)\n",
@@ -587,7 +588,7 @@ int dapl_ia_open(const char *name, int a

  	status = dapl_provider_list_search(name, &provider);
  	if (0 != status) {
-		status = -EINVAL;
+		status = -ENODEV;
  		goto bail;
  	}

@@ -595,12 +596,24 @@ int dapl_ia_open(const char *name, int a
  	*ia_ptr = NULL;

  	hca = (struct dapl_hca *)provider->extension;
+	status = ib_query_port(hca->ib_hca_handle, hca->port_num, &port_attr);
+	if (status)
+		goto bail;
+	if (port_attr.state != IB_PORT_ACTIVE) {
+		dapl_dbg_log(DAPL_DBG_TYPE_WARN,
+				"%s: Port %d is not in ACTIVE state\n", 
+				__FUNCTION__, hca->port_num);
+
+		status = -EBUSY;
+		goto bail;
+	}
+
  	atomic_inc(&hca->handle_ref_count);

  	/* Allocate and initialize ia structure */
  	ia = dapl_ia_alloc(provider, hca);
  	if (!ia) {
-                atomic_dec(&hca->handle_ref_count);
+		atomic_dec(&hca->handle_ref_count);
  		status = -EINVAL;
  		goto bail;
  	}
@@ -637,18 +650,19 @@ int dapl_ia_open(const char *name, int a
  			goto bail;

  		atomic_inc(&evd->evd_ref_count);
- 		/* Register the handlers associated with the async EVD.  */
- 		status = dapl_ia_setup_callbacks(ia, evd);
- 		/* Assign the EVD so it gets cleaned up */
+		/* Register the handlers associated with the async EVD.  */
+		status = dapl_ia_setup_callbacks(ia, evd);
+		/* Assign the EVD so it gets cleaned up */
  		ia->cleanup_async_error_evd = TRUE;
  		ia->async_error_evd = evd;
-                if (status != 0)
-                        goto bail;
+		if (status != 0)
+			goto bail;
  	}

  	status = 0;
  	*ia_ptr = (struct dat_ia *)ia;
  	*async_evd = (struct dat_evd *)evd;


From guyg at voltaire.com  Thu Jul 28 04:27:22 2005
From: guyg at voltaire.com (Guy German)
Date: Thu, 28 Jul 2005 14:27:22 +0300
Subject: [openib-general][PATCH][kdapltest]: inc/dec module ref count
Message-ID: <D4F8F0B3820E754C887699BEF26A89405BB3BF@taurus.voltaire.com>

Right.
I was not sure this policy regards kdapltest as well.

Thanks,
Guy


Hal Rosenstock <mailto:halr at voltaire.com> wrote:
> Hi Guy,
> 
> On Thu, 2005-07-28 at 03:20, Guy German wrote:
>> [kdapltest]: Prevents the oops occurred when removing the kdapltest
>> module while kdapltest server is up. 
>> 
>> Index: dapltest/kdapl/kdapl_module.c
>> ===================================================================
>> --- dapltest/kdapl/kdapl_module.c	(revision 2899)
>> +++ dapltest/kdapl/kdapl_module.c	(working copy)
>> @@ -55,6 +55,8 @@ static int kdapltest_open(struct inode *   {
>>   #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
>>   	MOD_INC_USE_COUNT;
>> +#else
>> +	try_module_get(THIS_MODULE);
>>   #endif
>>   	return 0;
>>   }
>> @@ -63,6 +65,8 @@ static int kdapltest_release(struct inod   {
>>   #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)
>>   	MOD_DEC_USE_COUNT;
>> +#else
>> +	module_put(THIS_MODULE);
>>   #endif
>>   	return 0;
>>   }
> 
> This is in the right direction.
> 
> The OpenIB policy is to support the latest kernels only. So the #if
> might be removed and just the 2.6 code remain in kdapl_module.c.
> 
> -- Hal


From m+Ian.Pratt at cl.cam.ac.uk  Thu Jul 28 04:57:51 2005
From: m+Ian.Pratt at cl.cam.ac.uk (Ian Pratt)
Date: Thu, 28 Jul 2005 12:57:51 +0100
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
Message-ID: <A95E2296287EAD4EB592B5DEEFCE0E9D282808@liverpoolst.ad.cl.cam.ac.uk>

> >     Greg> Hm, you do realize that io_remap_pfn_range() is the same
> >     Greg> thing as remap_pfn_range() on i386, right?
> > 
> >     Greg> So, why would this patch change anything?
> > 
> > It's not the same thing under Xen.  I think this patch 
> fixes userspace 
> > access to PCI memory for XenLinux.
> 
> But Xen is a separate arch, and hence, will get different pci 
> arch specific functions, right?
> 
> In short, what is this patch trying to fix?  What is the 
> problem anyone is seeing with the existing code?

As I understand it, remap_pfn_range should be used for mapping pages
that are backed by memory, and io_remap_pfn_range should be used for
mapping MMIO regions.
There's a distinciton between the two for architectures like Sparc and
xen/x86. 

For example, drivers/char/mem.c uses io_remap_pfn_range for mmap'ing
/dev/mem

Ian


From halr at voltaire.com  Thu Jul 28 06:18:10 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 09:18:10 -0400
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <1122518594.17830.1.camel@duffman>
References: <42E82952.2020302@ichips.intel.com>
	<1122514023.4451.70.camel@hal.voltaire.com>
	<1122518594.17830.1.camel@duffman>
Message-ID: <1122556690.4451.715.camel@hal.voltaire.com>

On Wed, 2005-07-27 at 22:43, Tom Duffy wrote:
> On Wed, 2005-07-27 at 21:27 -0400, Hal Rosenstock wrote:
> > Which IBAT are you using ? It is the one in the trunk or the one on
> > shaharf-ibat branch ?
> 
> uat.[ch] isn't in trunk yet, right?

That was the case but is no longer. I moved the latest IBAT and user
IBAT to the trunk.

Also, the IBAT difference is not the issue. More later...

-- Hal



From halr at voltaire.com  Thu Jul 28 06:38:43 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 09:38:43 -0400
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <42E82952.2020302@ichips.intel.com>
References: <42E82952.2020302@ichips.intel.com>
Message-ID: <1122557923.4451.737.camel@hal.voltaire.com>

Hi Arlin,

On Wed, 2005-07-27 at 20:39, Arlin Davis wrote:
> Hi Hal,
> 
> I move from a 2.6.11 kernel to 2.6.12.3 and am now having some problems 
> with IBAT. (svn 2919)
> 
> Running the simple example...
> 
> [ardavis at iclust-19 examples]$ ./uatt
> uatt: main: uat test start
> uatt: main: ib_at_route_by_ip: ret 0 errno 0 for request 1 id 1 1
> 
> [ardavis at iclust-19 examples]$
> Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
> iclust-19 kernel: Oops: 0002 [1] SMP
> 
> Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
> iclust-19 kernel: CR2: 0000000000000000
> 
> Jul 27 17:36:41 iclust-19 kernel: Unable to handle kernel NULL pointer 
> dereference at 0000000000000000 RIP:
> Jul 27 17:36:41 iclust-19 kernel: 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155}
> Jul 27 17:36:41 iclust-19 kernel: PGD 34a77067 PUD 34a65067 PMD 0
> Jul 27 17:36:41 iclust-19 kernel: Oops: 0002 [1] SMP
> Jul 27 17:36:41 iclust-19 kernel: CPU 1
> Jul 27 17:36:41 iclust-19 kernel: Modules linked in: ib_att ib_uat ib_at 
> ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib  ib_sa ib_mthca ib_mad ib_core
> Jul 27 17:36:41 iclust-19 kernel: Pid: 2248, comm: ib_at_wq/1 Not 
> tainted 2.6.12.3
> Jul 27 17:36:41 iclust-19 kernel: RIP: 0010:[<ffffffff88057310>] 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155}
> Jul 27 17:36:41 iclust-19 kernel: RSP: 0018:ffff8100362ade38  EFLAGS: 
> 00010282
> Jul 27 17:36:41 iclust-19 kernel: RAX: ffff8100386384b0 RBX: 
> ffff81003e8a5e00 RCX: 0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: RDX: ffff81003e8a5e10 RSI: 
> 0000000000000001 RDI: ffff81003e8a5e00
> Jul 27 17:36:41 iclust-19 kernel: RBP: ffff810038638480 R08: 
> 0000000000000000 R09: 0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: R10: ffff81003e8a5e00 R11: 
> 0000000000000050 R12: 0000000000000001
> Jul 27 17:36:41 iclust-19 kernel: R13: 0000000000000292 R14: 
> ffff81002da5de50 R15: ffffffff880523ed
> Jul 27 17:36:41 iclust-19 kernel: FS:  0000000000000000(0000) 
> GS:ffffffff8058e580(0000) knlGS:0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 
> 000000008005003b
> Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000 CR3: 
> 0000000034a3b000 CR4: 00000000000006e0
> Jul 27 17:36:41 iclust-19 kernel: Process ib_at_wq/1 (pid: 2248, 
> threadinfo ffff8100362ac000, task ffff81003ea6e f50)
> Jul 27 17:36:41 iclust-19 kernel: Stack: ffff81002da5de50 
> ffff81002da5de78 ffff810038501880 ffffffff88052408
> Jul 27 17:36:41 iclust-19 kernel:        ffff81002da5de70 
> ffffffff8014363e ffff8100385018c0 ffff810038501898
> Jul 27 17:36:41 iclust-19 kernel:        ffff8100385018a8 ffffffffffffffff
> Jul 27 17:36:41 iclust-19 kernel: Call 
> Trace:<ffffffff88052408>{:ib_at:req_comp_work+27} 
> <ffffffff8014363e>{work er_thread+501}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff8013019d>{default_wake_function+0} <ffffffff8013019d>{default_ 
> wake_function+0}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff80147520>{keventd_create_kthread+0} <ffffffff80143449>{worker_ 
> thread+0}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff80147520>{keventd_create_kthread+0} <ffffffff801474f3>{kthread 
> +204}
> Jul 27 17:36:41 iclust-19 kernel:        <ffffffff8010f1a3>{child_rip+8} 
> <ffffffff80147520>{keventd_create_kthre ad+0}
> Jul 27 17:36:41 iclust-19 kernel:        <ffffffff80147427>{kthread+0} 
> <ffffffff8010f19b>{child_rip+0}
> Jul 27 17:36:41 iclust-19 kernel:
> Jul 27 17:36:41 iclust-19 kernel:
> Jul 27 17:36:41 iclust-19 kernel: Code: 48 89 11 48 8b 48 08 48 8d 53 20 
> 48 89 43 20 48 89 50 08 48
> Jul 27 17:36:41 iclust-19 kernel: RIP 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155} RSP <ffff8100362ade38>
> Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000
> .
> 
> any ideas?

The only thing I see is that if ib_uat_callback were invoked with a NULL
context this would fail as shown above.

Unfortunately, I don't see this. Is this reproducible ? The differences
in your setup v. mine are 2.6.12.3 v. 2.6.12.2, SMP, and perhaps
processor architecture.

Also, note that user AT is now on the trunk (and subsequent work will be
done here) so the shaharf-ibat branch no longer needs to be tracked.

-- Hal



From halr at voltaire.com  Thu Jul 28 06:48:00 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 09:48:00 -0400
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <42E82952.2020302@ichips.intel.com>
References: <42E82952.2020302@ichips.intel.com>
Message-ID: <1122558479.4451.739.camel@hal.voltaire.com>

On Wed, 2005-07-27 at 20:39, Arlin Davis wrote:
> Hi Hal,
> 
> I move from a 2.6.11 kernel to 2.6.12.3 and am now having some problems 
> with IBAT. (svn 2919)
> 
> Running the simple example...
> 
> [ardavis at iclust-19 examples]$ ./uatt
> uatt: main: uat test start
> uatt: main: ib_at_route_by_ip: ret 0 errno 0 for request 1 id 1 1
> 
> [ardavis at iclust-19 examples]$
> Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
> iclust-19 kernel: Oops: 0002 [1] SMP
> 
> Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
> iclust-19 kernel: CR2: 0000000000000000
> 
> Jul 27 17:36:41 iclust-19 kernel: Unable to handle kernel NULL pointer 
> dereference at 0000000000000000 RIP:
> Jul 27 17:36:41 iclust-19 kernel: 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155}
> Jul 27 17:36:41 iclust-19 kernel: PGD 34a77067 PUD 34a65067 PMD 0
> Jul 27 17:36:41 iclust-19 kernel: Oops: 0002 [1] SMP
> Jul 27 17:36:41 iclust-19 kernel: CPU 1
> Jul 27 17:36:41 iclust-19 kernel: Modules linked in: ib_att ib_uat ib_at 
> ib_ucm ib_cm ib_umad ib_uverbs ib_ipoib  ib_sa ib_mthca ib_mad ib_core
> Jul 27 17:36:41 iclust-19 kernel: Pid: 2248, comm: ib_at_wq/1 Not 
> tainted 2.6.12.3
> Jul 27 17:36:41 iclust-19 kernel: RIP: 0010:[<ffffffff88057310>] 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155}
> Jul 27 17:36:41 iclust-19 kernel: RSP: 0018:ffff8100362ade38  EFLAGS: 
> 00010282
> Jul 27 17:36:41 iclust-19 kernel: RAX: ffff8100386384b0 RBX: 
> ffff81003e8a5e00 RCX: 0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: RDX: ffff81003e8a5e10 RSI: 
> 0000000000000001 RDI: ffff81003e8a5e00
> Jul 27 17:36:41 iclust-19 kernel: RBP: ffff810038638480 R08: 
> 0000000000000000 R09: 0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: R10: ffff81003e8a5e00 R11: 
> 0000000000000050 R12: 0000000000000001
> Jul 27 17:36:41 iclust-19 kernel: R13: 0000000000000292 R14: 
> ffff81002da5de50 R15: ffffffff880523ed
> Jul 27 17:36:41 iclust-19 kernel: FS:  0000000000000000(0000) 
> GS:ffffffff8058e580(0000) knlGS:0000000000000000
> Jul 27 17:36:41 iclust-19 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 
> 000000008005003b
> Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000 CR3: 
> 0000000034a3b000 CR4: 00000000000006e0
> Jul 27 17:36:41 iclust-19 kernel: Process ib_at_wq/1 (pid: 2248, 
> threadinfo ffff8100362ac000, task ffff81003ea6e f50)
> Jul 27 17:36:41 iclust-19 kernel: Stack: ffff81002da5de50 
> ffff81002da5de78 ffff810038501880 ffffffff88052408
> Jul 27 17:36:41 iclust-19 kernel:        ffff81002da5de70 
> ffffffff8014363e ffff8100385018c0 ffff810038501898
> Jul 27 17:36:41 iclust-19 kernel:        ffff8100385018a8 ffffffffffffffff
> Jul 27 17:36:41 iclust-19 kernel: Call 
> Trace:<ffffffff88052408>{:ib_at:req_comp_work+27} 
> <ffffffff8014363e>{work er_thread+501}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff8013019d>{default_wake_function+0} <ffffffff8013019d>{default_ 
> wake_function+0}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff80147520>{keventd_create_kthread+0} <ffffffff80143449>{worker_ 
> thread+0}
> Jul 27 17:36:41 iclust-19 kernel:        
> <ffffffff80147520>{keventd_create_kthread+0} <ffffffff801474f3>{kthread 
> +204}
> Jul 27 17:36:41 iclust-19 kernel:        <ffffffff8010f1a3>{child_rip+8} 
> <ffffffff80147520>{keventd_create_kthre ad+0}
> Jul 27 17:36:41 iclust-19 kernel:        <ffffffff80147427>{kthread+0} 
> <ffffffff8010f19b>{child_rip+0}
> Jul 27 17:36:41 iclust-19 kernel:
> Jul 27 17:36:41 iclust-19 kernel:
> Jul 27 17:36:41 iclust-19 kernel: Code: 48 89 11 48 8b 48 08 48 8d 53 20 
> 48 89 43 20 48 89 50 08 48
> Jul 27 17:36:41 iclust-19 kernel: RIP 
> <ffffffff88057310>{:ib_uat:ib_uat_callback+155} RSP <ffff8100362ade38>
> Jul 27 17:36:41 iclust-19 kernel: CR2: 0000000000000000
> .
> 
> any ideas?

One more thing: were there any IBAT/user IBAT messages in the log before
the oops ? Thanks.

-- Hal



From rdunlap at xenotime.net  Thu Jul 28 07:18:50 2005
From: rdunlap at xenotime.net (Randy.Dunlap)
Date: Thu, 28 Jul 2005 07:18:50 -0700
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <A95E2296287EAD4EB592B5DEEFCE0E9D282808@liverpoolst.ad.cl.cam.ac.uk>
References: <A95E2296287EAD4EB592B5DEEFCE0E9D282808@liverpoolst.ad.cl.cam.ac.uk>
Message-ID: <20050728071850.6bed3966.rdunlap@xenotime.net>

On Thu, 28 Jul 2005 12:57:51 +0100 Ian Pratt wrote:

> > >     Greg> Hm, you do realize that io_remap_pfn_range() is the same
> > >     Greg> thing as remap_pfn_range() on i386, right?
> > > 
> > >     Greg> So, why would this patch change anything?
> > > 
> > > It's not the same thing under Xen.  I think this patch 
> > fixes userspace 
> > > access to PCI memory for XenLinux.
> > 
> > But Xen is a separate arch, and hence, will get different pci 
> > arch specific functions, right?
> > 
> > In short, what is this patch trying to fix?  What is the 
> > problem anyone is seeing with the existing code?
> 
> As I understand it, remap_pfn_range should be used for mapping pages
> that are backed by memory, and io_remap_pfn_range should be used for
> mapping MMIO regions.
> There's a distinciton between the two for architectures like Sparc and
> xen/x86. 
> 
> For example, drivers/char/mem.c uses io_remap_pfn_range for mmap'ing
> /dev/mem

That is my (limited) understanding also, but when I built
io_remap_pfn_range(), I didn't search all callers of
remap_pfn_range() to see which ones that I could fix (or break)
by changing them.  Mostly due to the possible breakage part.

---
~Randy


From dotanb at mellanox.co.il  Thu Jul 28 08:13:16 2005
From: dotanb at mellanox.co.il (Dotan Barak)
Date: Thu, 28 Jul 2005 18:13:16 +0300
Subject: [openib-general] create QP in user level which fails doesn't change
	the value of e rrno
Message-ID: <506C3D7B14CDD411A52C00025558DED60865DCF6@mtlex01.yok.mtl.com>

When i try to create a QP in user level, and this creation fails (for
example, try to use inline_data = 10000), the value of errno doesn't
indicate that there was any problem.

I think that the problem is that the creation of the QP fails from the user
level part of the driver.

should i "trust" the value of errno?


Dotan Barak
Software Verification Engineer
Mellanox Technologies LTD
mailto:dotanb at mellanox.co.il
Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245
P.O. Box 86 Yokneam 20692 ISRAEL.
Home: +972-4-8289408  Cell: 052-4222383

[ May the fork be with you ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/c8d61da7/attachment.html>

From m+Ian.Pratt at cl.cam.ac.uk  Thu Jul 28 07:40:34 2005
From: m+Ian.Pratt at cl.cam.ac.uk (Ian Pratt)
Date: Thu, 28 Jul 2005 15:40:34 +0100
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
Message-ID: <A95E2296287EAD4EB592B5DEEFCE0E9D282811@liverpoolst.ad.cl.cam.ac.uk>

 
> > For example, drivers/char/mem.c uses io_remap_pfn_range for 
> mmap'ing 
> > /dev/mem
> 
> That is my (limited) understanding also, but when I built 
> io_remap_pfn_range(), I didn't search all callers of
> remap_pfn_range() to see which ones that I could fix (or 
> break) by changing them.  Mostly due to the possible breakage part.

Yep, its not entirely trivial to determine the intended usage. In
particular, some of the sound drivers require both versions in the same
file. 

It's probably best to rely on Xen/Sparc users to find them on a case by
case basis and ask for them to be fixed (unless someone's feeling
brave...)

Ian 


From greg at kroah.com  Thu Jul 28 08:46:59 2005
From: greg at kroah.com (Greg KH)
Date: Thu, 28 Jul 2005 08:46:59 -0700
Subject: [openib-general] Re: [git pull] Final InfiniBand updates for 2.6.13
In-Reply-To: <52hdefsgsn.fsf@cisco.com>
References: <52hdefsgsn.fsf@cisco.com>
Message-ID: <20050728154659.GA23065@kroah.com>

On Wed, Jul 27, 2005 at 09:20:08PM -0700, Roland Dreier wrote:
> Linus, please pull from
> 
>     rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

Not to be a pain, but didn't we say that we would only do email patches
after -rc1 comes out at the last kernel summit instead of git pulls?

Just curious...

greg k-h


From rolandd at cisco.com  Thu Jul 28 09:02:32 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 09:02:32 -0700
Subject: [openib-general] Re: create QP in user level which fails doesn't
 change the value of e rrno
In-Reply-To: <506C3D7B14CDD411A52C00025558DED60865DCF6@mtlex01.yok.mtl.com>
	(Dotan Barak's message of "Thu, 28 Jul 2005 18:13:16 +0300")
References: <506C3D7B14CDD411A52C00025558DED60865DCF6@mtlex01.yok.mtl.com>
Message-ID: <52slxyrk9z.fsf@cisco.com>

    Dotan> should i "trust" the value of errno?

No, in general errno is only set by the system's C library.

 - R.


From jlentini at netapp.com  Thu Jul 28 09:10:35 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 12:10:35 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] uDAPL with uCM and uAT support
In-Reply-To: <ORSMSX4014ZDgC81Tbd00000064@orsmsx401.amr.corp.intel.com>
References: <ORSMSX4014ZDgC81Tbd00000064@orsmsx401.amr.corp.intel.com>
Message-ID: <Pine.LNX.4.61.0507281022470.12296@jlentini-linux.nane.netapp.com>


A couple of questions:

On Sun, 24 Jul 2005, Arlin Davis wrote:

> Index: dapl/udapl/dapl_evd_wait.c
> ===================================================================
> --- dapl/udapl/dapl_evd_wait.c	(revision 2899)
> +++ dapl/udapl/dapl_evd_wait.c	(working copy)
> @@ -145,24 +146,12 @@
> 					(DAT_COUNT) DAPL_EVD_STATE_OPEN,
> 					(DAT_COUNT) DAPL_EVD_STATE_WAITED );
>     dapl_os_unlock ( &evd_ptr->header.lock );
> -
> -    if ( evd_state != DAPL_EVD_STATE_OPEN )
> +    if ( evd_state != DAPL_EVD_STATE_OPEN || !waitable)
>     {
> -	/* Bogus state, bail out */
> 	dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
> 	goto bail;
>     }
>
> -    if (!waitable)
> -    {
> -	/* This EVD is not waitable, reset the state and bail */
> -	(void) dapl_os_atomic_assign ((DAPL_ATOMIC *)&evd_ptr->evd_state,
> -					(DAT_COUNT) DAPL_EVD_STATE_WAITED,
> -					evd_state);
> -	dat_status = DAT_ERROR (DAT_INVALID_STATE, DAT_INVALID_STATE_EVD_UNWAITABLE);
> -	goto bail;
> -    }
> -

Why don't we need to reset the state?

>     /*
>      * We now own the EVD, even though we don't have the lock anymore,
>      * because we're in the WAITED state.
> @@ -182,37 +171,54 @@
> 	 * return right away if the ib_cq_handle associate with these evd
> 	 * equal to IB_INVALID_HANDLE
> 	 */
> -	dapls_evd_copy_cq(evd_ptr);
> -
> -	if (dapls_rbuf_count(&evd_ptr->pending_event_queue) >= threshold)
> -	{
> -	    break;
> -	}
> -
> -	/*
> -	 * Do not enable the completion notification if this evd is not
> -	 * a DTO_EVD or RMR_BIND_EVD
> +	/* Logic to prevent missing completion between copy_cq (poll)
> +	 * and completion_notify (re-arm)
> 	 */
> -	if ( (!notify_requested) &&
> -             ((evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
> -              (evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG)) )
> +	notify_needed = DAT_TRUE;
> +	new_events = 0;
> +	while (DAT_TRUE)
> 	{
> -	    dat_status = dapls_ib_completion_notify (
> -		evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
> -		evd_ptr,
> -		(evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT) ?
> -		     IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP );
> -
> -	    DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
> -	    /* FIXME report error */
> -	    dapl_os_assert(dat_status == DAT_SUCCESS);
> +		dapls_evd_copy_cq(evd_ptr); /* poll for new completions */
> +		total_events = dapls_rbuf_count (&evd_ptr->pending_event_queue);
> +		new_events = total_events - new_events;
> +		if (total_events >= threshold ||
> +			(!new_events && notify_needed == DAT_FALSE))
> +		{
> +			break;
> +		}
> +
> +		/*
> +		 * Do not enable the completion notification if this evd is not
> +		 * a DTO_EVD or RMR_BIND_EVD
> +		 */
> +		if ( (evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
> +			(evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG) )
> +		{
> +			dat_status = dapls_ib_completion_notify (
> +					evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
> +					evd_ptr,
> +					(evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT)
> ?
> +		     			IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP );
> +
> +			DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
> +			notify_needed = DAT_FALSE;
> +			new_events = total_events;
> +
> +			/* FIXME report error */
> +			dapl_os_assert(dat_status == DAT_SUCCESS);
> +		}
> +		else
> +		{
> +			break;
> +		}
>
> -	    notify_requested = DAT_TRUE;
> +	} /* while completions < threshold, and rearm needed */
>
> -	    /* Try again.  */
> -	    continue;
> +	if (total_events >= threshold)
> +	{
> +		break;
> 	}
> -
> +

Can you explain the chages above?

This is my reading of how things used to work: the thread that called 
dapl_evd_wait polled the CQ until the number of events reaped was >= 
the threshold paramter. If a poll did not reap enough events, the 
thread turned notification on (in certain situations). I don't see why 
it need to do this though, since it went directly back to polling and 
the notification upcall essentially did nothing if there EVD was in 
the WAITED state.

Obviously you saw a bug in the old implementation. Is my description 
of how the code used to work wrong?

> Index: dapl/udapl/Makefile
> ===================================================================
> --- dapl/udapl/Makefile	(revision 2899)
> +++ dapl/udapl/Makefile	(working copy)
> @@ -122,7 +122,8 @@
> #
> ifeq ($(VERBS),openib)
> PROVIDER = $(TOPDIR)/../openib
> -CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
> +CFLAGS   += -DOPENIB
> +#CFLAGS   += -DCQ_WAIT_OBJECT

Do the OpenIB uverbs support a CQ_WAIT_OBJECT? If not, we should 
remove the line above. When I see it commented out, it makes me 
think that the functionality is supported but not implemented in 
uDAPL.

> Index: dapl/common/dapl_evd_resize.c
> ===================================================================
> --- dapl/common/dapl_evd_resize.c	(revision 2899)
> +++ dapl/common/dapl_evd_resize.c	(working copy)
> @@ -67,72 +67,130 @@
> 	IN	DAT_EVD_HANDLE	   evd_handle,
> 	IN	DAT_COUNT	   evd_qlen )
> {
> -    DAPL_IA		*ia_ptr;
> -    DAPL_EVD		*evd_ptr;
> -    DAT_COUNT   	pend_cnt;
> -    DAT_RETURN		dat_status;
> +    DAPL_IA          *ia_ptr;
> +    DAPL_EVD         *evd_ptr;
> +    DAT_EVENT        *event_ptr;
> +    DAT_EVENT        *events;
> +    DAT_EVENT        *orig_event;
> +    DAPL_RING_BUFFER free_event_queue;
> +    DAPL_RING_BUFFER pending_event_queue;
> +    DAT_COUNT        pend_cnt;
> +    DAT_COUNT        i;
>
>     dapl_dbg_log (DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n",
> 		  evd_handle, evd_qlen);
>
>     if (DAPL_BAD_HANDLE (evd_handle, DAPL_MAGIC_EVD))
>     {
> -	dat_status = DAT_ERROR (DAT_INVALID_HANDLE,0);
> -	goto bail;
> +        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG1);
>     }
>
>     evd_ptr     = (DAPL_EVD *)evd_handle;
>     ia_ptr      = evd_ptr->header.owner_ia;
>
> -    if ( evd_qlen == evd_ptr->qlen )
> +    if ((evd_qlen <= 0) || (evd_ptr->qlen > evd_qlen))
>     {
> -	 dat_status = DAT_SUCCESS;
> -	 goto bail;
> +        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
>     }

Your check to make sure that the new size is greater that 0 is a good 
one.

I don't think we should return an error when the consumer asks for a 
size less than the current size. A return of success seems more 
appropriate. We can remove the old behavoir of shrinking the EVD if 
you like. I'm sure that makes things simpler.

In general, I think we should try to retain the single return point 
convention (in other words, we should keep the "goto bail" stuff). I 
admit that we should have made better use of this. For example, there 
should have been a second "bail" label (bail_unlock?) just before the 
last call to dapl_os_unlock(). That would have saved us several lines 
of code in this function.

>
>     if ( evd_qlen > ia_ptr->hca_ptr->ia_attr.max_evd_qlen )
>     {
> -	dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
> -	goto bail;
> +	return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_TEVD);
>     }
>
>     dapl_os_lock(&evd_ptr->header.lock);
>
> -    /* Don't try to resize if we are actively waiting */
>     if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED)
>     {
>         dapl_os_unlock(&evd_ptr->header.lock);
> -	dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
> -	goto bail;
> +        return DAT_ERROR (DAT_INVALID_STATE,0);
>     }
>
>     pend_cnt = dapls_rbuf_count(&evd_ptr->pending_event_queue);
>     if (pend_cnt > evd_qlen) {
> -	dapl_os_unlock(&evd_ptr->header.lock);
> -	dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
> -	goto bail;
> +        dapl_os_unlock(&evd_ptr->header.lock);
> +        return DAT_ERROR (DAT_INVALID_STATE,0);
> +    }
> +
> +    if (DAT_SUCCESS != dapls_ib_cq_resize(evd_ptr->header.owner_ia,
> +                                          evd_ptr,
> +                                          &evd_qlen))
> +    {
> +        dapl_os_unlock(&evd_ptr->header.lock);
> +        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
> +    }
> +
> +    /* Allocate EVENTs */
> +    events = (DAT_EVENT *) dapl_os_alloc (evd_qlen * sizeof (DAT_EVENT));
> +    if (!events)
> +    {
> +        dapl_os_unlock(&evd_ptr->header.lock);
> +        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>     }
> +    event_ptr = events;
>
> -    dat_status = dapls_ib_cq_resize(evd_ptr->header.owner_ia,
> -				    evd_ptr,
> -				    &evd_qlen);
> -    if (dat_status != DAT_SUCCESS)
> +    /* allocate free event queue */
> +    if (DAT_SUCCESS != dapls_rbuf_alloc (&free_event_queue, evd_qlen))
>     {
> +        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>         dapl_os_unlock(&evd_ptr->header.lock);
> -	goto bail;
> +        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>     }
>
> -    dat_status = dapls_evd_event_realloc (evd_ptr, evd_qlen);
> -    if (dat_status != DAT_SUCCESS)
> +    /* allocate pending event queue */
> +    if (DAT_SUCCESS != dapls_rbuf_alloc (&pending_event_queue, evd_qlen))
>     {
> +        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>         dapl_os_unlock(&evd_ptr->header.lock);
> -	goto bail;
> +        return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>     }
>
> +    for (i = 0; i < pend_cnt; i++)
> +    {
> +        orig_event = dapls_rbuf_remove(&evd_ptr->pending_event_queue);
> +        if (orig_event == NULL) {
> +            dapl_dbg_log (DAPL_DBG_TYPE_ERR, " Inconsistent event queue\n");
> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
> +            dapl_os_unlock(&evd_ptr->header.lock);
> +	    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
> +        }
> +        memcpy(event_ptr, orig_event, sizeof(DAT_EVENT));
> +        if (DAT_SUCCESS != dapls_rbuf_add(&pending_event_queue, event_ptr)) {
> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
> +            dapl_os_unlock(&evd_ptr->header.lock);
> +	    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
> +        }
> +        event_ptr++;
> +    }
> +
> +    for (i = pend_cnt; i < evd_qlen; i++)
> +    {
> +        if (DAT_SUCCESS != dapls_rbuf_add(&free_event_queue,
> +                                          (void *) event_ptr)) {
> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
> +            dapl_os_unlock(&evd_ptr->header.lock);
> +	    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
> +        }
> +        event_ptr++;
> +    }
> +
> +    dapls_rbuf_destroy (&evd_ptr->free_event_queue);
> +    dapls_rbuf_destroy (&evd_ptr->pending_event_queue);
> +    if (evd_ptr->events)
> +    {
> +        dapl_os_free (evd_ptr->events, evd_ptr->qlen * sizeof (DAT_EVENT));
> +    }
> +    evd_ptr->free_event_queue    = free_event_queue;
> +    evd_ptr->pending_event_queue = pending_event_queue;
> +    evd_ptr->events              = events;
> +    evd_ptr->qlen                = evd_qlen;
> +


Can you hide this in dapls_ib_cq_resize()?


>     dapl_os_unlock(&evd_ptr->header.lock);
>
> - bail:
> -    return dat_status;
> +    dapl_dbg_log (DAPL_DBG_TYPE_RTN,
> +			"dapl_evd_resize returns DAT_SUCCESS\n");
> +
> +    return DAT_SUCCESS;
> }
>
> /*


From ardavis at ichips.intel.com  Thu Jul 28 09:17:13 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Thu, 28 Jul 2005 09:17:13 -0700
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <1122558479.4451.739.camel@hal.voltaire.com>
References: <42E82952.2020302@ichips.intel.com>
	<1122558479.4451.739.camel@hal.voltaire.com>
Message-ID: <42E90509.4030501@ichips.intel.com>

Hal Rosenstock wrote:

>On Wed, 2005-07-27 at 20:39, Arlin Davis wrote:
>  
>
>>Hi Hal,
>>
>>I move from a 2.6.11 kernel to 2.6.12.3 and am now having some problems 
>>with IBAT. (svn 2919)
>>
>>Running the simple example...
>>
>>[ardavis at iclust-19 examples]$ ./uatt
>>uatt: main: uat test start
>>uatt: main: ib_at_route_by_ip: ret 0 errno 0 for request 1 id 1 1
>>
>>[ardavis at iclust-19 examples]$
>>Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
>>iclust-19 kernel: Oops: 0002 [1] SMP
>>
>>Message from syslogd at iclust-19 at Wed Jul 27 17:36:41 2005 ...
>>iclust-19 kernel: CR2: 0000000000000000
>>
>>Jul 27 17:36:41 iclust-19 kernel: Unable to handle kernel NULL pointer 
>>dereference at 0000000000000000 RIP:
>>    
>>
>One more thing: were there any IBAT/user IBAT messages in the log before
>the oops ? Thanks.
>
>-- Hal
>
>  
>
No messages.
I was using the shaharf-ibat branch so I will pick up the latest from 
the trunk and retry.

-arlin



From gregkh at suse.de  Thu Jul 28 09:17:20 2005
From: gregkh at suse.de (Greg KH)
Date: Thu, 28 Jul 2005 09:17:20 -0700
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <A95E2296287EAD4EB592B5DEEFCE0E9D282808@liverpoolst.ad.cl.cam.ac.uk>
References: <A95E2296287EAD4EB592B5DEEFCE0E9D282808@liverpoolst.ad.cl.cam.ac.uk>
Message-ID: <20050728161720.GA23507@suse.de>

On Thu, Jul 28, 2005 at 12:57:51PM +0100, Ian Pratt wrote:
> > >     Greg> Hm, you do realize that io_remap_pfn_range() is the same
> > >     Greg> thing as remap_pfn_range() on i386, right?
> > > 
> > >     Greg> So, why would this patch change anything?
> > > 
> > > It's not the same thing under Xen.  I think this patch 
> > fixes userspace 
> > > access to PCI memory for XenLinux.
> > 
> > But Xen is a separate arch, and hence, will get different pci 
> > arch specific functions, right?
> > 
> > In short, what is this patch trying to fix?  What is the 
> > problem anyone is seeing with the existing code?
> 
> As I understand it, remap_pfn_range should be used for mapping pages
> that are backed by memory, and io_remap_pfn_range should be used for
> mapping MMIO regions.
> There's a distinciton between the two for architectures like Sparc and
> xen/x86. 

I agree, but not for i386, which is what the patch was trying to change.

If this is a fix for xen, fine, then say so in the changelog information
for the patch, as it is, no such information was given.

thanks,

greg k-h


From ardavis at ichips.intel.com  Thu Jul 28 10:32:44 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Thu, 28 Jul 2005 10:32:44 -0700
Subject: [openib-general] Re: [PATCH] uDAPL with uCM and uAT support
In-Reply-To: <Pine.LNX.4.61.0507281022470.12296@jlentini-linux.nane.netapp.com>
References: <ORSMSX4014ZDgC81Tbd00000064@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507281022470.12296@jlentini-linux.nane.netapp.com>
Message-ID: <42E916BC.6040305@ichips.intel.com>

James Lentini wrote:

>
> A couple of questions:
>
> On Sun, 24 Jul 2005, Arlin Davis wrote:
>
>> Index: dapl/udapl/dapl_evd_wait.c
>> ===================================================================
>> --- dapl/udapl/dapl_evd_wait.c    (revision 2899)
>> +++ dapl/udapl/dapl_evd_wait.c    (working copy)
>> @@ -145,24 +146,12 @@
>>                     (DAT_COUNT) DAPL_EVD_STATE_OPEN,
>>                     (DAT_COUNT) DAPL_EVD_STATE_WAITED );
>>     dapl_os_unlock ( &evd_ptr->header.lock );
>> -
>> -    if ( evd_state != DAPL_EVD_STATE_OPEN )
>> +    if ( evd_state != DAPL_EVD_STATE_OPEN || !waitable)
>>     {
>> -    /* Bogus state, bail out */
>>     dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
>>     goto bail;
>>     }
>>
>> -    if (!waitable)
>> -    {
>> -    /* This EVD is not waitable, reset the state and bail */
>> -    (void) dapl_os_atomic_assign ((DAPL_ATOMIC *)&evd_ptr->evd_state,
>> -                    (DAT_COUNT) DAPL_EVD_STATE_WAITED,
>> -                    evd_state);
>> -    dat_status = DAT_ERROR (DAT_INVALID_STATE, 
>> DAT_INVALID_STATE_EVD_UNWAITABLE);
>> -    goto bail;
>> -    }
>> -
>
>
> Why don't we need to reset the state?

good catch. Yes we need to reset back before bailing.

>
>>     /*
>>      * We now own the EVD, even though we don't have the lock anymore,
>>      * because we're in the WAITED state.
>> @@ -182,37 +171,54 @@
>>      * return right away if the ib_cq_handle associate with these evd
>>      * equal to IB_INVALID_HANDLE
>>      */
>> -    dapls_evd_copy_cq(evd_ptr);
>> -
>> -    if (dapls_rbuf_count(&evd_ptr->pending_event_queue) >= threshold)
>> -    {
>> -        break;
>> -    }
>> -
>> -    /*
>> -     * Do not enable the completion notification if this evd is not
>> -     * a DTO_EVD or RMR_BIND_EVD
>> +    /* Logic to prevent missing completion between copy_cq (poll)
>> +     * and completion_notify (re-arm)
>>      */
>> -    if ( (!notify_requested) &&
>> -             ((evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
>> -              (evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG)) )
>> +    notify_needed = DAT_TRUE;
>> +    new_events = 0;
>> +    while (DAT_TRUE)
>>     {
>> -        dat_status = dapls_ib_completion_notify (
>> -        evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
>> -        evd_ptr,
>> -        (evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT) ?
>> -             IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP );
>> -
>> -        DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
>> -        /* FIXME report error */
>> -        dapl_os_assert(dat_status == DAT_SUCCESS);
>> +        dapls_evd_copy_cq(evd_ptr); /* poll for new completions */
>> +        total_events = dapls_rbuf_count 
>> (&evd_ptr->pending_event_queue);
>> +        new_events = total_events - new_events;
>> +        if (total_events >= threshold ||
>> +            (!new_events && notify_needed == DAT_FALSE))
>> +        {
>> +            break;
>> +        }
>> +
>> +        /*
>> +         * Do not enable the completion notification if this evd is not
>> +         * a DTO_EVD or RMR_BIND_EVD
>> +         */
>> +        if ( (evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
>> +            (evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG) )
>> +        {
>> +            dat_status = dapls_ib_completion_notify (
>> +                    evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
>> +                    evd_ptr,
>> +                    (evd_ptr->completion_type == 
>> DAPL_EVD_STATE_SOLICITED_WAIT)
>> ?
>> +                         IB_NOTIFY_ON_SOLIC_COMP : 
>> IB_NOTIFY_ON_NEXT_COMP );
>> +
>> +            DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
>> +            notify_needed = DAT_FALSE;
>> +            new_events = total_events;
>> +
>> +            /* FIXME report error */
>> +            dapl_os_assert(dat_status == DAT_SUCCESS);
>> +        }
>> +        else
>> +        {
>> +            break;
>> +        }
>>
>> -        notify_requested = DAT_TRUE;
>> +    } /* while completions < threshold, and rearm needed */
>>
>> -        /* Try again.  */
>> -        continue;
>> +    if (total_events >= threshold)
>> +    {
>> +        break;
>>     }
>> -
>> +
>
>
> Can you explain the chages above?
>
> This is my reading of how things used to work: the thread that called 
> dapl_evd_wait polled the CQ until the number of events reaped was >= 
> the threshold paramter. If a poll did not reap enough events, the 
> thread turned notification on (in certain situations). I don't see why 
> it need to do this though, since it went directly back to polling and 
> the notification upcall essentially did nothing if there EVD was in 
> the WAITED state.
>
> Obviously you saw a bug in the old implementation. Is my description 
> of how the code used to work wrong?

You are correct, but there is a small window where you can miss the CQ 
event if you re-arm and wait but the CQ completion/event occured between 
the last poll and the rearm/wait.  This change adds another cq_poll 
after every re-arm to cover this senario and make sure notification is 
armed and new events are expected before going into the wait.

>
>> Index: dapl/udapl/Makefile
>> ===================================================================
>> --- dapl/udapl/Makefile    (revision 2899)
>> +++ dapl/udapl/Makefile    (working copy)
>> @@ -122,7 +122,8 @@
>> #
>> ifeq ($(VERBS),openib)
>> PROVIDER = $(TOPDIR)/../openib
>> -CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
>> +CFLAGS   += -DOPENIB
>> +#CFLAGS   += -DCQ_WAIT_OBJECT
>
>
> Do the OpenIB uverbs support a CQ_WAIT_OBJECT? If not, we should 
> remove the line above. When I see it commented out, it makes me think 
> that the functionality is supported but not implemented in uDAPL.

Yes, openIB supports direct wait objects via fd's. There are some issues 
with the openIB implementation that need to be worked out.

>
>> Index: dapl/common/dapl_evd_resize.c
>> ===================================================================
>> --- dapl/common/dapl_evd_resize.c    (revision 2899)
>> +++ dapl/common/dapl_evd_resize.c    (working copy)
>> @@ -67,72 +67,130 @@
>>     IN    DAT_EVD_HANDLE       evd_handle,
>>     IN    DAT_COUNT       evd_qlen )
>> {
>> -    DAPL_IA        *ia_ptr;
>> -    DAPL_EVD        *evd_ptr;
>> -    DAT_COUNT       pend_cnt;
>> -    DAT_RETURN        dat_status;
>> +    DAPL_IA          *ia_ptr;
>> +    DAPL_EVD         *evd_ptr;
>> +    DAT_EVENT        *event_ptr;
>> +    DAT_EVENT        *events;
>> +    DAT_EVENT        *orig_event;
>> +    DAPL_RING_BUFFER free_event_queue;
>> +    DAPL_RING_BUFFER pending_event_queue;
>> +    DAT_COUNT        pend_cnt;
>> +    DAT_COUNT        i;
>>
>>     dapl_dbg_log (DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n",
>>           evd_handle, evd_qlen);
>>
>>     if (DAPL_BAD_HANDLE (evd_handle, DAPL_MAGIC_EVD))
>>     {
>> -    dat_status = DAT_ERROR (DAT_INVALID_HANDLE,0);
>> -    goto bail;
>> +        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG1);
>>     }
>>
>>     evd_ptr     = (DAPL_EVD *)evd_handle;
>>     ia_ptr      = evd_ptr->header.owner_ia;
>>
>> -    if ( evd_qlen == evd_ptr->qlen )
>> +    if ((evd_qlen <= 0) || (evd_ptr->qlen > evd_qlen))
>>     {
>> -     dat_status = DAT_SUCCESS;
>> -     goto bail;
>> +        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
>>     }
>
>
> Your check to make sure that the new size is greater that 0 is a good 
> one.
>
> I don't think we should return an error when the consumer asks for a 
> size less than the current size. A return of success seems more 
> appropriate. We can remove the old behavoir of shrinking the EVD if 
> you like. I'm sure that makes things simpler.
>
agree. I see no need to shrink.

> In general, I think we should try to retain the single return point 
> convention (in other words, we should keep the "goto bail" stuff). I 
> admit that we should have made better use of this. For example, there 
> should have been a second "bail" label (bail_unlock?) just before the 
> last call to dapl_os_unlock(). That would have saved us several lines 
> of code in this function.
>
agree.

>>     if ( evd_qlen > ia_ptr->hca_ptr->ia_attr.max_evd_qlen )
>>     {
>> -    dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
>> -    goto bail;
>> +    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_TEVD);
>>     }
>>
>>     dapl_os_lock(&evd_ptr->header.lock);
>>
>> -    /* Don't try to resize if we are actively waiting */
>>     if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED)
>>     {
>>         dapl_os_unlock(&evd_ptr->header.lock);
>> -    dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
>> -    goto bail;
>> +        return DAT_ERROR (DAT_INVALID_STATE,0);
>>     }
>>
>>     pend_cnt = dapls_rbuf_count(&evd_ptr->pending_event_queue);
>>     if (pend_cnt > evd_qlen) {
>> -    dapl_os_unlock(&evd_ptr->header.lock);
>> -    dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
>> -    goto bail;
>> +        dapl_os_unlock(&evd_ptr->header.lock);
>> +        return DAT_ERROR (DAT_INVALID_STATE,0);
>> +    }
>> +
>> +    if (DAT_SUCCESS != dapls_ib_cq_resize(evd_ptr->header.owner_ia,
>> +                                          evd_ptr,
>> +                                          &evd_qlen))
>> +    {
>> +        dapl_os_unlock(&evd_ptr->header.lock);
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>> +    }
>> +
>> +    /* Allocate EVENTs */
>> +    events = (DAT_EVENT *) dapl_os_alloc (evd_qlen * sizeof 
>> (DAT_EVENT));
>> +    if (!events)
>> +    {
>> +        dapl_os_unlock(&evd_ptr->header.lock);
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>     }
>> +    event_ptr = events;
>>
>> -    dat_status = dapls_ib_cq_resize(evd_ptr->header.owner_ia,
>> -                    evd_ptr,
>> -                    &evd_qlen);
>> -    if (dat_status != DAT_SUCCESS)
>> +    /* allocate free event queue */
>> +    if (DAT_SUCCESS != dapls_rbuf_alloc (&free_event_queue, evd_qlen))
>>     {
>> +        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>         dapl_os_unlock(&evd_ptr->header.lock);
>> -    goto bail;
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>     }
>>
>> -    dat_status = dapls_evd_event_realloc (evd_ptr, evd_qlen);
>> -    if (dat_status != DAT_SUCCESS)
>> +    /* allocate pending event queue */
>> +    if (DAT_SUCCESS != dapls_rbuf_alloc (&pending_event_queue, 
>> evd_qlen))
>>     {
>> +        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>         dapl_os_unlock(&evd_ptr->header.lock);
>> -    goto bail;
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>     }
>>
>> +    for (i = 0; i < pend_cnt; i++)
>> +    {
>> +        orig_event = dapls_rbuf_remove(&evd_ptr->pending_event_queue);
>> +        if (orig_event == NULL) {
>> +            dapl_dbg_log (DAPL_DBG_TYPE_ERR, " Inconsistent event 
>> queue\n");
>> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>> +            dapl_os_unlock(&evd_ptr->header.lock);
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>> +        }
>> +        memcpy(event_ptr, orig_event, sizeof(DAT_EVENT));
>> +        if (DAT_SUCCESS != dapls_rbuf_add(&pending_event_queue, 
>> event_ptr)) {
>> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>> +            dapl_os_unlock(&evd_ptr->header.lock);
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>> +        }
>> +        event_ptr++;
>> +    }
>> +
>> +    for (i = pend_cnt; i < evd_qlen; i++)
>> +    {
>> +        if (DAT_SUCCESS != dapls_rbuf_add(&free_event_queue,
>> +                                          (void *) event_ptr)) {
>> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>> +            dapl_os_unlock(&evd_ptr->header.lock);
>> +        return DAT_ERROR 
>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>> +        }
>> +        event_ptr++;
>> +    }
>> +
>> +    dapls_rbuf_destroy (&evd_ptr->free_event_queue);
>> +    dapls_rbuf_destroy (&evd_ptr->pending_event_queue);
>> +    if (evd_ptr->events)
>> +    {
>> +        dapl_os_free (evd_ptr->events, evd_ptr->qlen * sizeof 
>> (DAT_EVENT));
>> +    }
>> +    evd_ptr->free_event_queue    = free_event_queue;
>> +    evd_ptr->pending_event_queue = pending_event_queue;
>> +    evd_ptr->events              = events;
>> +    evd_ptr->qlen                = evd_qlen;
>> +
>
>
>
> Can you hide this in dapls_ib_cq_resize()?
>
which part?

>
>>     dapl_os_unlock(&evd_ptr->header.lock);
>>
>> - bail:
>> -    return dat_status;
>> +    dapl_dbg_log (DAPL_DBG_TYPE_RTN,
>> +            "dapl_evd_resize returns DAT_SUCCESS\n");
>> +
>> +    return DAT_SUCCESS;
>> }
>>
>> /*
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>



From sean.hefty at intel.com  Thu Jul 28 10:35:40 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 10:35:40 -0700
Subject: [openib-general] [PATCH] fix sparse warnings
Message-ID: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com>

Patch to fix most of the remaining sparse warnings.

I did not try to fix any of the SDP related warnings, and some warnings will
still occur, but do not indicate any problems.

I'd like to get a couple of people to test this patch, plus any comments,
before committing it.  The majority of the changes are related to MADs and
the CM.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>



Index: include/ib_user_verbs.h
===================================================================
--- include/ib_user_verbs.h	(revision 2919)
+++ include/ib_user_verbs.h	(working copy)
@@ -143,8 +143,8 @@ struct ib_uverbs_query_device {
 
 struct ib_uverbs_query_device_resp {
 	__u64 fw_ver;
-	__u64 node_guid;
-	__u64 sys_image_guid;
+	__be64 node_guid;
+	__be64 sys_image_guid;
 	__u64 max_mr_size;
 	__u64 page_size_cap;
 	__u32 vendor_id;
Index: include/ib_cm.h
===================================================================
--- include/ib_cm.h	(revision 2919)
+++ include/ib_cm.h	(working copy)
@@ -115,7 +115,7 @@ struct ib_cm_req_event_param {
 	struct ib_sa_path_rec	*primary_path;
 	struct ib_sa_path_rec	*alternate_path;
 
-	u64			remote_ca_guid;
+	__be64			remote_ca_guid;
 	u32			remote_qkey;
 	u32			remote_qpn;
 	enum ib_qp_type		qp_type;
@@ -132,7 +132,7 @@ struct ib_cm_req_event_param {
 };
 
 struct ib_cm_rep_event_param {
-	u64			remote_ca_guid;
+	__be64			remote_ca_guid;
 	u32			remote_qkey;
 	u32			remote_qpn;
 	u32			starting_psn;
@@ -146,38 +146,38 @@ struct ib_cm_rep_event_param {
 };
 
 enum ib_cm_rej_reason {
-	IB_CM_REJ_NO_QP				= __constant_htons(1),
-	IB_CM_REJ_NO_EEC			= __constant_htons(2),
-	IB_CM_REJ_NO_RESOURCES			= __constant_htons(3),
-	IB_CM_REJ_TIMEOUT			= __constant_htons(4),
-	IB_CM_REJ_UNSUPPORTED			= __constant_htons(5),
-	IB_CM_REJ_INVALID_COMM_ID		= __constant_htons(6),
-	IB_CM_REJ_INVALID_COMM_INSTANCE		= __constant_htons(7),
-	IB_CM_REJ_INVALID_SERVICE_ID		= __constant_htons(8),
-	IB_CM_REJ_INVALID_TRANSPORT_TYPE	= __constant_htons(9),
-	IB_CM_REJ_STALE_CONN			= __constant_htons(10),
-	IB_CM_REJ_RDC_NOT_EXIST			= __constant_htons(11),
-	IB_CM_REJ_INVALID_GID			= __constant_htons(12),
-	IB_CM_REJ_INVALID_LID			= __constant_htons(13),
-	IB_CM_REJ_INVALID_SL			= __constant_htons(14),
-	IB_CM_REJ_INVALID_TRAFFIC_CLASS		= __constant_htons(15),
-	IB_CM_REJ_INVALID_HOP_LIMIT		= __constant_htons(16),
-	IB_CM_REJ_INVALID_PACKET_RATE		= __constant_htons(17),
-	IB_CM_REJ_INVALID_ALT_GID		= __constant_htons(18),
-	IB_CM_REJ_INVALID_ALT_LID		= __constant_htons(19),
-	IB_CM_REJ_INVALID_ALT_SL		= __constant_htons(20),
-	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= __constant_htons(21),
-	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= __constant_htons(22),
-	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= __constant_htons(23),
-	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(24),
-	IB_CM_REJ_INVALID_MTU			= __constant_htons(26),
-	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= __constant_htons(27),
-	IB_CM_REJ_CONSUMER_DEFINED		= __constant_htons(28),
-	IB_CM_REJ_INVALID_RNR_RETRY		= __constant_htons(29),
-	IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID	= __constant_htons(30),
-	IB_CM_REJ_INVALID_CLASS_VERSION		= __constant_htons(31),
-	IB_CM_REJ_INVALID_FLOW_LABEL		= __constant_htons(32),
-	IB_CM_REJ_INVALID_ALT_FLOW_LABEL	= __constant_htons(33)
+	IB_CM_REJ_NO_QP				= 1,
+	IB_CM_REJ_NO_EEC			= 2,
+	IB_CM_REJ_NO_RESOURCES			= 3,
+	IB_CM_REJ_TIMEOUT			= 4,
+	IB_CM_REJ_UNSUPPORTED			= 5,
+	IB_CM_REJ_INVALID_COMM_ID		= 6,
+	IB_CM_REJ_INVALID_COMM_INSTANCE		= 7,
+	IB_CM_REJ_INVALID_SERVICE_ID		= 8,
+	IB_CM_REJ_INVALID_TRANSPORT_TYPE	= 9,
+	IB_CM_REJ_STALE_CONN			= 10,
+	IB_CM_REJ_RDC_NOT_EXIST			= 11,
+	IB_CM_REJ_INVALID_GID			= 12,
+	IB_CM_REJ_INVALID_LID			= 13,
+	IB_CM_REJ_INVALID_SL			= 14,
+	IB_CM_REJ_INVALID_TRAFFIC_CLASS		= 15,
+	IB_CM_REJ_INVALID_HOP_LIMIT		= 16,
+	IB_CM_REJ_INVALID_PACKET_RATE		= 17,
+	IB_CM_REJ_INVALID_ALT_GID		= 18,
+	IB_CM_REJ_INVALID_ALT_LID		= 19,
+	IB_CM_REJ_INVALID_ALT_SL		= 20,
+	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= 21,
+	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= 22,
+	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= 23,
+	IB_CM_REJ_PORT_REDIRECT			= 24,
+	IB_CM_REJ_INVALID_MTU			= 26,
+	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= 27,
+	IB_CM_REJ_CONSUMER_DEFINED		= 28,
+	IB_CM_REJ_INVALID_RNR_RETRY		= 29,
+	IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID	= 30,
+	IB_CM_REJ_INVALID_CLASS_VERSION		= 31,
+	IB_CM_REJ_INVALID_FLOW_LABEL		= 32,
+	IB_CM_REJ_INVALID_ALT_FLOW_LABEL	= 33
 };
 
 struct ib_cm_rej_event_param {
@@ -221,8 +221,7 @@ struct ib_cm_sidr_req_event_param {
 	struct ib_cm_id		*listen_id;
 	struct ib_device	*device;
 	u8			port;
-
-	u16	pkey;
+	u16			pkey;
 };
 
 enum ib_cm_sidr_status {
@@ -284,12 +283,12 @@ typedef int (*ib_cm_handler)(struct ib_c
 struct ib_cm_id {
 	ib_cm_handler		cm_handler;
 	void			*context;
-	u64			service_id;
-	u64			service_mask;
+	__be64			service_id;
+	__be64			service_mask;
 	enum ib_cm_state	state;		/* internal CM/debug use */
 	enum ib_cm_lap_state	lap_state;	/* internal CM/debug use */
-	u32			local_id;
-	u32			remote_id;
+	__be32			local_id;
+	__be32			remote_id;
 };
 
 /**
@@ -329,13 +328,13 @@ void ib_destroy_cm_id(struct ib_cm_id *c
  *   IB_CM_ASSIGN_SERVICE_ID.
  */
 int ib_cm_listen(struct ib_cm_id *cm_id,
-		 u64 service_id,
-		 u64 service_mask);
+		 __be64 service_id,
+		 __be64 service_mask);
 
 struct ib_cm_req_param {
 	struct ib_sa_path_rec	*primary_path;
 	struct ib_sa_path_rec	*alternate_path;
-	u64			service_id;
+	__be64			service_id;
 	u32			qp_num;
 	enum ib_qp_type		qp_type;
 	u32			starting_psn;
@@ -527,7 +526,7 @@ int ib_send_cm_apr(struct ib_cm_id *cm_i
 
 struct ib_cm_sidr_req_param {
 	struct ib_sa_path_rec	*path;
-	u64			service_id;
+	__be64			service_id;
 	int			timeout_ms;
 	const void		*private_data;
 	u8			private_data_len;
Index: include/ib_user_mad.h
===================================================================
--- include/ib_user_mad.h	(revision 2919)
+++ include/ib_user_mad.h	(working copy)
@@ -70,8 +70,6 @@
  * @traffic_class - Traffic class in GRH
  * @gid - Remote GID in GRH
  * @flow_label - Flow label in GRH
- *
- * All multi-byte quantities are stored in network (big endian) byte order.
  */
 struct ib_user_mad_hdr {
 	__u32	id;
@@ -79,9 +77,9 @@ struct ib_user_mad_hdr {
 	__u32	timeout_ms;
 	__u32	retries;
 	__u32	length;
-	__u32	qpn;
-	__u32   qkey;
-	__u16	lid;
+	__be32	qpn;
+	__be32  qkey;
+	__be16	lid;
 	__u8	sl;
 	__u8	path_bits;
 	__u8	grh_present;
@@ -89,7 +87,7 @@ struct ib_user_mad_hdr {
 	__u8	hop_limit;
 	__u8	traffic_class;
 	__u8	gid[16];
-	__u32	flow_label;
+	__be32	flow_label;
 };
 
 /**
Index: include/ib_verbs.h
===================================================================
--- include/ib_verbs.h	(revision 2919)
+++ include/ib_verbs.h	(working copy)
@@ -50,8 +50,8 @@
 union ib_gid {
 	u8	raw[16];
 	struct {
-		u64	subnet_prefix;
-		u64	interface_id;
+		__be64	subnet_prefix;
+		__be64	interface_id;
 	} global;
 };
 
@@ -87,8 +87,8 @@ enum ib_atomic_cap {
 
 struct ib_device_attr {
 	u64			fw_ver;
-	u64			node_guid;
-	u64			sys_image_guid;
+	__be64			node_guid;
+	__be64			sys_image_guid;
 	u64			max_mr_size;
 	u64			page_size_cap;
 	u32			vendor_id;
@@ -290,8 +290,8 @@ struct ib_global_route {
 };
 
 struct ib_grh {
-	u32		version_tclass_flow;
-	u16		paylen;
+	__be32		version_tclass_flow;
+	__be16		paylen;
 	u8		next_hdr;
 	u8		hop_limit;
 	union ib_gid	sgid;
@@ -302,6 +302,8 @@ enum {
 	IB_MULTICAST_QPN = 0xffffff
 };
 
+#define IB_LID_PERMISSIVE	__constant_htons(0xFFFF)
+
 enum ib_ah_flags {
 	IB_AH_GRH	= 1
 };
Index: include/ib_mad.h
===================================================================
--- include/ib_mad.h	(revision 2919)
+++ include/ib_mad.h	(working copy)
@@ -113,12 +113,12 @@ struct ib_mad_hdr {
 	u8	mgmt_class;
 	u8	class_version;
 	u8	method;
-	u16	status;
-	u16	class_specific;
-	u64	tid;
-	u16	attr_id;
-	u16	resv;
-	u32	attr_mod;
+	__be16	status;
+	__be16	class_specific;
+	__be64	tid;
+	__be16	attr_id;
+	__be16	resv;
+	__be32	attr_mod;
 };
 
 struct ib_rmpp_hdr {
@@ -126,8 +126,8 @@ struct ib_rmpp_hdr {
 	u8	rmpp_type;
 	u8	rmpp_rtime_flags;
 	u8	rmpp_status;
-	u32	seg_num;
-	u32	paylen_newwin;
+	__be32	seg_num;
+	__be32	paylen_newwin;
 };
 
 typedef u64 __bitwise ib_sa_comp_mask;
@@ -141,9 +141,9 @@ typedef u64 __bitwise ib_sa_comp_mask;
  * the wire so we can't change the layout)
  */
 struct ib_sa_hdr {
-	u64			sm_key;
-	u16			attr_offset;
-	u16			reserved;
+	__be64			sm_key;
+	__be16			attr_offset;
+	__be16			reserved;
 	ib_sa_comp_mask		comp_mask;
 } __attribute__ ((packed));
 
Index: include/ib_sa.h
===================================================================
--- include/ib_sa.h	(revision 2919)
+++ include/ib_sa.h	(working copy)
@@ -133,16 +133,16 @@ struct ib_sa_path_rec {
 	/* reserved */
 	union ib_gid dgid;
 	union ib_gid sgid;
-	u16          dlid;
-	u16          slid;
+	__be16       dlid;
+	__be16       slid;
 	int          raw_traffic;
 	/* reserved */
-	u32          flow_label;
+	__be32       flow_label;
 	u8           hop_limit;
 	u8           traffic_class;
 	int          reversible;
 	u8           numb_path;
-	u16          pkey;
+	__be16       pkey;
 	/* reserved */
 	u8           sl;
 	u8           mtu_selector;
@@ -176,18 +176,18 @@ struct ib_sa_path_rec {
 struct ib_sa_mcmember_rec {
 	union ib_gid mgid;
 	union ib_gid port_gid;
-	u32          qkey;
-	u16          mlid;
+	__be32       qkey;
+	__be16       mlid;
 	u8           mtu_selector;
 	u8           mtu;
 	u8           traffic_class;
-	u16          pkey;
+	__be16       pkey;
 	u8 	     rate_selector;
 	u8 	     rate;
 	u8 	     packet_life_time_selector;
 	u8 	     packet_life_time;
 	u8           sl;
-	u32          flow_label;
+	__be32       flow_label;
 	u8           hop_limit;
 	u8           scope;
 	u8           join_state;
@@ -238,7 +238,7 @@ struct ib_sa_mcmember_rec {
 struct ib_sa_service_rec {
 	u64		id;
 	union ib_gid	gid;
-	u16 		pkey;
+	__be16 		pkey;
 	/* reserved */
 	u32		lease;
 	u8		key[16];
Index: include/ib_at.h
===================================================================
--- include/ib_at.h	(revision 2919)
+++ include/ib_at.h	(working copy)
@@ -56,7 +56,7 @@ enum ib_at_route_flags {
 
 struct ib_at_path_attr {
 	u16 qos_tag;
-	u16 pkey;
+	__be16 pkey;
 	u8 multi_path_type;
 };
 
Index: include/ib_smi.h
===================================================================
--- include/ib_smi.h	(revision 2919)
+++ include/ib_smi.h	(working copy)
@@ -41,8 +41,6 @@
 
 #include <ib_mad.h>
 
-#define IB_LID_PERMISSIVE			0xFFFF
-
 #define IB_SMP_DATA_SIZE			64
 #define IB_SMP_MAX_PATH_HOPS			64
 
@@ -51,16 +49,16 @@ struct ib_smp {
 	u8	mgmt_class;
 	u8	class_version;
 	u8	method;
-	u16	status;
+	__be16	status;
 	u8	hop_ptr;
 	u8	hop_cnt;
-	u64	tid;
-	u16	attr_id;
-	u16	resv;
-	u32	attr_mod;
-	u64	mkey;
-	u16	dr_slid;
-	u16	dr_dlid;
+	__be64	tid;
+	__be16	attr_id;
+	__be16	resv;
+	__be32	attr_mod;
+	__be64	mkey;
+	__be16	dr_slid;
+	__be16	dr_dlid;
 	u8	reserved[28];
 	u8	data[IB_SMP_DATA_SIZE];
 	u8	initial_path[IB_SMP_MAX_PATH_HOPS];
Index: include/ib_user_cm.h
===================================================================
--- include/ib_user_cm.h	(revision 2919)
+++ include/ib_user_cm.h	(working copy)
@@ -88,15 +88,15 @@ struct ib_ucm_attr_id {
 };
 
 struct ib_ucm_attr_id_resp {
-	__u64 service_id;
-	__u64 service_mask;
-	__u32 local_id;
-	__u32 remote_id;
+	__be64 service_id;
+	__be64 service_mask;
+	__be32 local_id;
+	__be32 remote_id;
 };
 
 struct ib_ucm_listen {
-	__u64 service_id;
-	__u64 service_mask;
+	__be64 service_id;
+	__be64 service_mask;
 	__u32 id;
 };
 
@@ -114,13 +114,13 @@ struct ib_ucm_private_data {
 struct ib_ucm_path_rec {
 	__u8  dgid[16];
 	__u8  sgid[16];
-	__u16 dlid;
-	__u16 slid;
+	__be16 dlid;
+	__be16 slid;
 	__u32 raw_traffic;
-	__u32 flow_label;
+	__be32 flow_label;
 	__u32 reversible;
 	__u32 mtu;
-	__u16 pkey;
+	__be16 pkey;
 	__u8  hop_limit;
 	__u8  traffic_class;
 	__u8  numb_path;
@@ -138,7 +138,7 @@ struct ib_ucm_req {
 	__u32 qpn;
 	__u32 qp_type;
 	__u32 psn;
-	__u64 sid;
+	__be64 sid;
 	__u64 data;
 	__u64 primary_path;
 	__u64 alternate_path;
@@ -200,7 +200,7 @@ struct ib_ucm_lap {
 struct ib_ucm_sidr_req {
 	__u32 id;
 	__u32 timeout;
-	__u64 sid;
+	__be64 sid;
 	__u64 data;
 	__u64 path;
 	__u16 pkey;
@@ -237,7 +237,7 @@ struct ib_ucm_req_event_resp {
 	/* port */
 	struct ib_ucm_path_rec primary_path;
 	struct ib_ucm_path_rec alternate_path;
-	__u64                  remote_ca_guid;
+	__be64                 remote_ca_guid;
 	__u32                  remote_qkey;
 	__u32                  remote_qpn;
 	__u32                  qp_type;
@@ -253,7 +253,7 @@ struct ib_ucm_req_event_resp {
 };
 
 struct ib_ucm_rep_event_resp {
-	__u64 remote_ca_guid;
+	__be64 remote_ca_guid;
 	__u32 remote_qkey;
 	__u32 remote_qpn;
 	__u32 starting_psn;
Index: core/mad_rmpp.c
===================================================================
--- core/mad_rmpp.c	(revision 2919)
+++ core/mad_rmpp.c	(working copy)
@@ -61,7 +61,7 @@ struct mad_rmpp_recv {
 	int seg_num;
 	int newwin;
 
-	u64 tid;
+	__be64 tid;
 	u32 src_qp;
 	u16 slid;
 	u8 mgmt_class;
@@ -620,7 +620,7 @@ static int send_next_seg(struct ib_mad_s
 	return ib_send_mad(mad_send_wr);
 }
 
-static void abort_send(struct ib_mad_agent_private *agent, u64 tid,
+static void abort_send(struct ib_mad_agent_private *agent, __be64 tid,
 		       u8 rmpp_status)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
Index: core/cm.c
===================================================================
--- core/cm.c	(revision 2919)
+++ core/cm.c	(working copy)
@@ -83,7 +83,7 @@ struct cm_port {
 struct cm_device {
 	struct list_head list;
 	struct ib_device *device;
-	u64 ca_guid;
+	__be64 ca_guid;
 	struct cm_port port[0];
 };
 
@@ -100,8 +100,8 @@ struct cm_work {
 	struct list_head list;
 	struct cm_port *port;
 	struct ib_mad_recv_wc *mad_recv_wc;	/* Received MADs */
-	u32 local_id;				/* Established / timewait */
-	u32 remote_id;
+	__be32 local_id;			/* Established / timewait */
+	__be32 remote_id;
 	struct ib_cm_event cm_event;
 	struct ib_sa_path_rec path[0];
 };
@@ -110,8 +110,8 @@ struct cm_timewait_info {
 	struct cm_work work;			/* Must be first. */
 	struct rb_node remote_qp_node;
 	struct rb_node remote_id_node;
-	u64 remote_ca_guid;
-	u32 remote_qpn;
+	__be64 remote_ca_guid;
+	__be32 remote_qpn;
 	u8 inserted_remote_qp;
 	u8 inserted_remote_id;
 };
@@ -132,11 +132,11 @@ struct cm_id_private {
 	struct cm_av alt_av;
 
 	void *private_data;
-	u64 tid;
-	u32 local_qpn;
-	u32 remote_qpn;
-	u32 sq_psn;
-	u32 rq_psn;
+	__be64 tid;
+	__be32 local_qpn;
+	__be32 remote_qpn;
+	__be32 sq_psn;
+	__be32 rq_psn;
 	int timeout_ms;
 	enum ib_mtu path_mtu;
 	u8 private_data_len;
@@ -253,7 +253,7 @@ static void cm_set_ah_attr(struct ib_ah_
 			   u16 dlid, u8 sl, u16 src_path_bits)
 {
 	memset(ah_attr, 0, sizeof ah_attr);
-	ah_attr->dlid = be16_to_cpu(dlid);
+	ah_attr->dlid = dlid;
 	ah_attr->sl = sl;
 	ah_attr->src_path_bits = src_path_bits;
 	ah_attr->port_num = port_num;
@@ -264,7 +264,7 @@ static void cm_init_av_for_response(stru
 {
 	av->port = port;
 	av->pkey_index = wc->pkey_index;
-	cm_set_ah_attr(&av->ah_attr, port->port_num, cpu_to_be16(wc->slid),
+	cm_set_ah_attr(&av->ah_attr, port->port_num, wc->slid,
 		       wc->sl, wc->dlid_path_bits);
 }
 
@@ -295,8 +295,9 @@ static int cm_init_av_by_path(struct ib_
 		return ret;
 
 	av->port = port;
-	cm_set_ah_attr(&av->ah_attr, av->port->port_num, path->dlid,
-		       path->sl, path->slid & 0x7F);
+	cm_set_ah_attr(&av->ah_attr, av->port->port_num,
+		       be16_to_cpu(path->dlid), path->sl,
+		       be16_to_cpu(path->slid) & 0x7F);
 	av->packet_life_time = path->packet_life_time;
 	return 0;
 }
@@ -309,26 +310,26 @@ static int cm_alloc_id(struct cm_id_priv
 	do {
 		spin_lock_irqsave(&cm.lock, flags);
 		ret = idr_get_new_above(&cm.local_id_table, cm_id_priv, 1,
-					(int *) &cm_id_priv->id.local_id);
+					(__force int *) &cm_id_priv->id.local_id);
 		spin_unlock_irqrestore(&cm.lock, flags);
 	} while( (ret == -EAGAIN) && idr_pre_get(&cm.local_id_table, GFP_KERNEL) );
 	return ret;
 }
 
-static void cm_free_id(u32 local_id)
+static void cm_free_id(__be32 local_id)
 {
 	unsigned long flags;
 
 	spin_lock_irqsave(&cm.lock, flags);
-	idr_remove(&cm.local_id_table, (int) local_id);
+	idr_remove(&cm.local_id_table, (__force int) local_id);
 	spin_unlock_irqrestore(&cm.lock, flags);
 }
 
-static struct cm_id_private * cm_get_id(u32 local_id, u32 remote_id)
+static struct cm_id_private * cm_get_id(__be32 local_id, __be32 remote_id)
 {
 	struct cm_id_private *cm_id_priv;
 
-	cm_id_priv = idr_find(&cm.local_id_table, (int) local_id);
+	cm_id_priv = idr_find(&cm.local_id_table, (__force int) local_id);
 	if (cm_id_priv) {
 		if (cm_id_priv->id.remote_id == remote_id)
 			atomic_inc(&cm_id_priv->refcount);
@@ -339,7 +340,7 @@ static struct cm_id_private * cm_get_id(
 	return cm_id_priv;
 }
 
-static struct cm_id_private * cm_acquire_id(u32 local_id, u32 remote_id)
+static struct cm_id_private * cm_acquire_id(__be32 local_id, __be32 remote_id)
 {
 	struct cm_id_private *cm_id_priv;
 	unsigned long flags;
@@ -356,8 +357,8 @@ static struct cm_id_private * cm_insert_
 	struct rb_node **link = &cm.listen_service_table.rb_node;
 	struct rb_node *parent = NULL;
 	struct cm_id_private *cur_cm_id_priv;
-	u64 service_id = cm_id_priv->id.service_id;
-	u64 service_mask = cm_id_priv->id.service_mask;
+	__be64 service_id = cm_id_priv->id.service_id;
+	__be64 service_mask = cm_id_priv->id.service_mask;
 
 	while (*link) {
 		parent = *link;
@@ -376,7 +377,7 @@ static struct cm_id_private * cm_insert_
 	return NULL;
 }
 
-static struct cm_id_private * cm_find_listen(u64 service_id)
+static struct cm_id_private * cm_find_listen(__be64 service_id)
 {
 	struct rb_node *node = cm.listen_service_table.rb_node;
 	struct cm_id_private *cm_id_priv;
@@ -400,8 +401,8 @@ static struct cm_timewait_info * cm_inse
 	struct rb_node **link = &cm.remote_id_table.rb_node;
 	struct rb_node *parent = NULL;
 	struct cm_timewait_info *cur_timewait_info;
-	u64 remote_ca_guid = timewait_info->remote_ca_guid;
-	u32 remote_id = timewait_info->work.remote_id;
+	__be64 remote_ca_guid = timewait_info->remote_ca_guid;
+	__be32 remote_id = timewait_info->work.remote_id;
 
 	while (*link) {
 		parent = *link;
@@ -424,8 +425,8 @@ static struct cm_timewait_info * cm_inse
 	return NULL;
 }
 
-static struct cm_timewait_info * cm_find_remote_id(u64 remote_ca_guid,
-						   u32 remote_id)
+static struct cm_timewait_info * cm_find_remote_id(__be64 remote_ca_guid,
+						   __be32 remote_id)
 {
 	struct rb_node *node = cm.remote_id_table.rb_node;
 	struct cm_timewait_info *timewait_info;
@@ -453,8 +454,8 @@ static struct cm_timewait_info * cm_inse
 	struct rb_node **link = &cm.remote_qp_table.rb_node;
 	struct rb_node *parent = NULL;
 	struct cm_timewait_info *cur_timewait_info;
-	u64 remote_ca_guid = timewait_info->remote_ca_guid;
-	u32 remote_qpn = timewait_info->remote_qpn;
+	__be64 remote_ca_guid = timewait_info->remote_ca_guid;
+	__be32 remote_qpn = timewait_info->remote_qpn;
 
 	while (*link) {
 		parent = *link;
@@ -484,7 +485,7 @@ static struct cm_id_private * cm_insert_
 	struct rb_node *parent = NULL;
 	struct cm_id_private *cur_cm_id_priv;
 	union ib_gid *port_gid = &cm_id_priv->av.dgid;
-	u32 remote_id = cm_id_priv->id.remote_id;
+	__be32 remote_id = cm_id_priv->id.remote_id;
 
 	while (*link) {
 		parent = *link;
@@ -598,7 +599,7 @@ static void cm_cleanup_timewait(struct c
 	spin_unlock_irqrestore(&cm.lock, flags);
 }
 
-static struct cm_timewait_info * cm_create_timewait_info(u32 local_id)
+static struct cm_timewait_info * cm_create_timewait_info(__be32 local_id)
 {
 	struct cm_timewait_info *timewait_info;
 
@@ -715,14 +716,15 @@ retest:
 EXPORT_SYMBOL(ib_destroy_cm_id);
 
 int ib_cm_listen(struct ib_cm_id *cm_id,
-		 u64 service_id,
-		 u64 service_mask)
+		 __be64 service_id,
+		 __be64 service_mask)
 {
 	struct cm_id_private *cm_id_priv, *cur_cm_id_priv;
 	unsigned long flags;
 	int ret = 0;
 
-	service_mask = service_mask ? service_mask : ~0ULL;
+	service_mask = service_mask ? service_mask :
+		       __constant_cpu_to_be64(~0ULL);
 	service_id &= service_mask;
 	if ((service_id & IB_SERVICE_ID_AGN_MASK) == IB_CM_ASSIGN_SERVICE_ID &&
 	    (service_id != IB_CM_ASSIGN_SERVICE_ID))
@@ -735,8 +737,8 @@ int ib_cm_listen(struct ib_cm_id *cm_id,
 
 	spin_lock_irqsave(&cm.lock, flags);
 	if (service_id == IB_CM_ASSIGN_SERVICE_ID) {
-		cm_id->service_id = __cpu_to_be64(cm.listen_service_id++);
-		cm_id->service_mask = ~0ULL;
+		cm_id->service_id = cpu_to_be64(cm.listen_service_id++);
+		cm_id->service_mask = __constant_cpu_to_be64(~0ULL);
 	} else {
 		cm_id->service_id = service_id;
 		cm_id->service_mask = service_mask;
@@ -752,18 +754,19 @@ int ib_cm_listen(struct ib_cm_id *cm_id,
 }
 EXPORT_SYMBOL(ib_cm_listen);
 
-static u64 cm_form_tid(struct cm_id_private *cm_id_priv,
-		       enum cm_msg_sequence msg_seq)
+static __be64 cm_form_tid(struct cm_id_private *cm_id_priv,
+			  enum cm_msg_sequence msg_seq)
 {
 	u64 hi_tid, low_tid;
 
 	hi_tid   = ((u64) cm_id_priv->av.port->mad_agent->hi_tid) << 32;
-	low_tid  = (u64) (cm_id_priv->id.local_id | (msg_seq << 30));
+	low_tid  = (u64) ((__force u32)cm_id_priv->id.local_id |
+			  (msg_seq << 30));
 	return cpu_to_be64(hi_tid | low_tid);
 }
 
 static void cm_format_mad_hdr(struct ib_mad_hdr *hdr,
-			      enum cm_msg_attr_id attr_id, u64 tid)
+			      __be16 attr_id, __be64 tid)
 {
 	hdr->base_version  = IB_MGMT_BASE_VERSION;
 	hdr->mgmt_class	   = IB_MGMT_CLASS_CM;
@@ -896,7 +899,7 @@ int ib_send_cm_req(struct ib_cm_id *cm_i
 			goto error1;
 	}
 	cm_id->service_id = param->service_id;
-	cm_id->service_mask = ~0ULL;
+	cm_id->service_mask = __constant_cpu_to_be64(~0ULL);
 	cm_id_priv->timeout_ms = cm_convert_to_ms(
 				    param->primary_path->packet_life_time) * 2 +
 				 cm_convert_to_ms(
@@ -963,7 +966,7 @@ static int cm_issue_rej(struct cm_port *
 	rej_msg->remote_comm_id = rcv_msg->local_comm_id;
 	rej_msg->local_comm_id = rcv_msg->remote_comm_id;
 	cm_rej_set_msg_rejected(rej_msg, msg_rejected);
-	rej_msg->reason = reason;
+	rej_msg->reason = cpu_to_be16(reason);
 
 	if (ari && ari_length) {
 		cm_rej_set_reject_info_len(rej_msg, ari_length);
@@ -977,8 +980,8 @@ static int cm_issue_rej(struct cm_port *
 	return ret;
 }
 
-static inline int cm_is_active_peer(u64 local_ca_guid, u64 remote_ca_guid,
-				    u32 local_qpn, u32 remote_qpn)
+static inline int cm_is_active_peer(__be64 local_ca_guid, __be64 remote_ca_guid,
+				    __be32 local_qpn, __be32 remote_qpn)
 {
 	return (be64_to_cpu(local_ca_guid) > be64_to_cpu(remote_ca_guid) ||
 		((local_ca_guid == remote_ca_guid) &&
@@ -1137,7 +1140,7 @@ static void cm_format_rej(struct cm_rej_
 		break;
 	}
 
-	rej_msg->reason = reason;
+	rej_msg->reason = cpu_to_be16(reason);
 	if (ari && ari_length) {
 		cm_rej_set_reject_info_len(rej_msg, ari_length);
 		memcpy(rej_msg->ari, ari, ari_length);
@@ -1276,7 +1279,7 @@ static int cm_req_handler(struct cm_work
 	cm_id_priv->id.cm_handler = listen_cm_id_priv->id.cm_handler;
 	cm_id_priv->id.context = listen_cm_id_priv->id.context;
 	cm_id_priv->id.service_id = req_msg->service_id;
-	cm_id_priv->id.service_mask = ~0ULL;
+	cm_id_priv->id.service_mask = __constant_cpu_to_be64(~0ULL);
 
 	cm_format_paths_from_req(req_msg, &work->path[0], &work->path[1]);
 	ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av);
@@ -1969,7 +1972,7 @@ static void cm_format_rej_event(struct c
 	param = &work->cm_event.param.rej_rcvd;
 	param->ari = rej_msg->ari;
 	param->ari_length = cm_rej_get_reject_info_len(rej_msg);
-	param->reason = rej_msg->reason;
+	param->reason = __be16_to_cpu(rej_msg->reason);
 	work->cm_event.private_data = &rej_msg->private_data;
 }
 
@@ -1978,20 +1981,20 @@ static struct cm_id_private * cm_acquire
 	struct cm_timewait_info *timewait_info;
 	struct cm_id_private *cm_id_priv;
 	unsigned long flags;
-	u32 remote_id;
+	__be32 remote_id;
 
 	remote_id = rej_msg->local_comm_id;
 
-	if (rej_msg->reason == IB_CM_REJ_TIMEOUT) {
+	if (__be16_to_cpu(rej_msg->reason) == IB_CM_REJ_TIMEOUT) {
 		spin_lock_irqsave(&cm.lock, flags);
-		timewait_info = cm_find_remote_id( *((u64 *) rej_msg->ari),
+		timewait_info = cm_find_remote_id( *((__be64 *) rej_msg->ari),
 						  remote_id);
 		if (!timewait_info) {
 			spin_unlock_irqrestore(&cm.lock, flags);
 			return NULL;
 		}
 		cm_id_priv = idr_find(&cm.local_id_table,
-				      (int) timewait_info->work.local_id);
+				      (__force int) timewait_info->work.local_id);
 		if (cm_id_priv) {
 			if (cm_id_priv->id.remote_id == remote_id)
 				atomic_inc(&cm_id_priv->refcount);
@@ -2032,7 +2035,7 @@ static int cm_rej_handler(struct cm_work
 		/* fall through */
 	case IB_CM_REQ_RCVD:
 	case IB_CM_MRA_REQ_SENT:
-		if (rej_msg->reason == IB_CM_REJ_STALE_CONN)
+		if (__be16_to_cpu(rej_msg->reason) == IB_CM_REJ_STALE_CONN)
 			cm_enter_timewait(cm_id_priv);
 		else
 			cm_reset_to_idle(cm_id_priv);
@@ -2553,7 +2556,7 @@ static void cm_format_sidr_req(struct cm
 	cm_format_mad_hdr(&sidr_req_msg->hdr, CM_SIDR_REQ_ATTR_ID,
 			  cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_SIDR));
 	sidr_req_msg->request_id = cm_id_priv->id.local_id;
-	sidr_req_msg->pkey = param->pkey;
+	sidr_req_msg->pkey = cpu_to_be16(param->pkey);
 	sidr_req_msg->service_id = param->service_id;
 
 	if (param->private_data && param->private_data_len)
@@ -2580,7 +2583,7 @@ int ib_send_cm_sidr_req(struct ib_cm_id 
 		goto out;
 
 	cm_id->service_id = param->service_id;
-	cm_id->service_mask = ~0ULL;
+	cm_id->service_mask = __constant_cpu_to_be64(~0ULL);
 	cm_id_priv->timeout_ms = param->timeout_ms;
 	cm_id_priv->max_cm_retries = param->max_cm_retries;
 	ret = cm_alloc_msg(cm_id_priv, &msg);
@@ -2621,7 +2624,7 @@ static void cm_format_sidr_req_event(str
 	sidr_req_msg = (struct cm_sidr_req_msg *)
 				work->mad_recv_wc->recv_buf.mad;
 	param = &work->cm_event.param.sidr_req_rcvd;
-	param->pkey = sidr_req_msg->pkey;
+	param->pkey = __be16_to_cpu(sidr_req_msg->pkey);
 	param->listen_id = listen_id;
 	param->device = work->port->mad_agent->device;
 	param->port = work->port->port_num;
@@ -2645,7 +2648,7 @@ static int cm_sidr_req_handler(struct cm
 	sidr_req_msg = (struct cm_sidr_req_msg *)
 				work->mad_recv_wc->recv_buf.mad;
 	wc = work->mad_recv_wc->wc;
-	cm_id_priv->av.dgid.global.subnet_prefix = wc->slid;
+	cm_id_priv->av.dgid.global.subnet_prefix = cpu_to_be64(wc->slid);
 	cm_id_priv->av.dgid.global.interface_id = 0;
 	cm_init_av_for_response(work->port, work->mad_recv_wc->wc,
 				&cm_id_priv->av);
@@ -2673,7 +2676,7 @@ static int cm_sidr_req_handler(struct cm
 	cm_id_priv->id.cm_handler = cur_cm_id_priv->id.cm_handler;
 	cm_id_priv->id.context = cur_cm_id_priv->id.context;
 	cm_id_priv->id.service_id = sidr_req_msg->service_id;
-	cm_id_priv->id.service_mask = ~0ULL;
+	cm_id_priv->id.service_mask = __constant_cpu_to_be64(~0ULL);
 
 	cm_format_sidr_req_event(work, &cur_cm_id_priv->id);
 	cm_process_work(cm_id_priv, work);
@@ -3175,10 +3178,10 @@ int ib_cm_init_qp_attr(struct ib_cm_id *
 }
 EXPORT_SYMBOL(ib_cm_init_qp_attr);
 
-static u64 cm_get_ca_guid(struct ib_device *device)
+static __be64 cm_get_ca_guid(struct ib_device *device)
 {
 	struct ib_device_attr *device_attr;
-	u64 guid;
+	__be64 guid;
 	int ret;
 
 	device_attr = kmalloc(sizeof *device_attr, GFP_KERNEL);
Index: core/user_mad.c
===================================================================
--- core/user_mad.c	(revision 2919)
+++ core/user_mad.c	(working copy)
@@ -271,7 +271,7 @@ static ssize_t ib_umad_write(struct file
 	struct ib_send_wr *bad_wr;
 	struct ib_rmpp_mad *rmpp_mad;
 	u8 method;
-	u64 *tid;
+	__be64 *tid;
 	int ret, length, hdr_len, data_len, rmpp_hdr_size;
 	int rmpp_active = 0;
 
@@ -316,7 +316,7 @@ static ssize_t ib_umad_write(struct file
 	if (packet->mad.hdr.grh_present) {
 		ah_attr.ah_flags = IB_AH_GRH;
 		memcpy(ah_attr.grh.dgid.raw, packet->mad.hdr.gid, 16);
-		ah_attr.grh.flow_label 	   = packet->mad.hdr.flow_label;
+		ah_attr.grh.flow_label 	   = be32_to_cpu(packet->mad.hdr.flow_label);
 		ah_attr.grh.hop_limit  	   = packet->mad.hdr.hop_limit;
 		ah_attr.grh.traffic_class  = packet->mad.hdr.traffic_class;
 	}
Index: core/mad.c
===================================================================
--- core/mad.c	(revision 2919)
+++ core/mad.c	(working copy)
@@ -693,7 +693,8 @@ static int handle_outgoing_dr_smp(struct
 		goto out;
 	}
 
-	build_smp_wc(send_wr->wr_id, smp->dr_slid, send_wr->wr.ud.pkey_index,
+	build_smp_wc(send_wr->wr_id, be16_to_cpu(smp->dr_slid),
+		     send_wr->wr.ud.pkey_index,
 		     send_wr->wr.ud.port_num, &mad_wc);
 
 	/* No GRH for DR SMP */
@@ -1554,7 +1555,7 @@ static int is_data_mad(struct ib_mad_age
 }
 
 struct ib_mad_send_wr_private*
-ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, u64 tid)
+ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, __be64 tid)
 {
 	struct ib_mad_send_wr_private *mad_send_wr;
 
@@ -1597,7 +1598,7 @@ static void ib_mad_complete_recv(struct 
 	struct ib_mad_send_wr_private *mad_send_wr;
 	struct ib_mad_send_wc mad_send_wc;
 	unsigned long flags;
-	u64 tid;
+	__be64 tid;
 
 	INIT_LIST_HEAD(&mad_recv_wc->rmpp_list);
 	list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list);
@@ -2165,7 +2166,8 @@ static void local_completions(void *data
 			 * Defined behavior is to complete response
 			 * before request
 			 */
-			build_smp_wc(local->wr_id, IB_LID_PERMISSIVE,
+			build_smp_wc(local->wr_id,
+				     be16_to_cpu(IB_LID_PERMISSIVE),
 				     0 /* pkey index */,
 				     recv_mad_agent->agent.port_num, &wc);
 
Index: core/at.c
===================================================================
--- core/at.c	(revision 2919)
+++ core/at.c	(working copy)
@@ -126,7 +126,7 @@ struct ib_at_src {
 	struct ib_device *dev;
 	struct net_device *netdev;
 	int port;
-	int pkey;
+	__be16 pkey;
 	union ib_gid gid;
 };
 
Index: core/mad_priv.h
===================================================================
--- core/mad_priv.h	(revision 2919)
+++ core/mad_priv.h	(working copy)
@@ -121,7 +121,7 @@ struct ib_mad_send_wr_private {
 	struct ib_send_wr send_wr;
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */
-	u64 tid;
+	__be64 tid;
 	unsigned long timeout;
 	int retries;
 	int retry;
@@ -144,7 +144,7 @@ struct ib_mad_local_private {
 	struct ib_send_wr send_wr;
 	struct ib_sge sg_list[IB_MAD_SEND_REQ_MAX_SG];
 	u64 wr_id;			/* client WR ID */
-	u64 tid;
+	__be64 tid;
 };
 
 struct ib_mad_mgmt_method_table {
@@ -210,7 +210,7 @@ extern kmem_cache_t *ib_mad_cache;
 int ib_send_mad(struct ib_mad_send_wr_private *mad_send_wr);
 
 struct ib_mad_send_wr_private *
-ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, u64 tid);
+ib_find_send_mad(struct ib_mad_agent_private *mad_agent_priv, __be64 tid);
 
 void ib_mad_complete_send_wr(struct ib_mad_send_wr_private *mad_send_wr,
 			     struct ib_mad_send_wc *mad_send_wc);
Index: core/ud_header.c
===================================================================
--- core/ud_header.c	(revision 2919)
+++ core/ud_header.c	(working copy)
@@ -195,6 +195,7 @@ void ib_ud_header_init(int     		    pay
 		       struct ib_ud_header *header)
 {
 	int header_len;
+	u16 packet_length;
 
 	memset(header, 0, sizeof *header);
 
@@ -209,7 +210,7 @@ void ib_ud_header_init(int     		    pay
 	header->lrh.link_version     = 0;
 	header->lrh.link_next_header =
 		grh_present ? IB_LNH_IBA_GLOBAL : IB_LNH_IBA_LOCAL;
-	header->lrh.packet_length    = (IB_LRH_BYTES     +
+	packet_length		     = (IB_LRH_BYTES     +
 					IB_BTH_BYTES     +
 					IB_DETH_BYTES    +
 					payload_bytes    +
@@ -218,8 +219,7 @@ void ib_ud_header_init(int     		    pay
 
 	header->grh_present          = grh_present;
 	if (grh_present) {
-		header->lrh.packet_length  += IB_GRH_BYTES / 4;
-
+		packet_length		   += IB_GRH_BYTES / 4;
 		header->grh.ip_version      = 6;
 		header->grh.payload_length  =
 			cpu_to_be16((IB_BTH_BYTES     +
@@ -230,7 +230,7 @@ void ib_ud_header_init(int     		    pay
 		header->grh.next_header     = 0x1b;
 	}
 
-	cpu_to_be16s(&header->lrh.packet_length);
+	header->lrh.packet_length = cpu_to_be16(packet_length);
 
 	if (header->immediate_present)
 		header->bth.opcode           = IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE;
Index: core/cm_msgs.h
===================================================================
--- core/cm_msgs.h	(revision 2919)
+++ core/cm_msgs.h	(working copy)
@@ -43,19 +43,17 @@
 
 #define IB_CM_CLASS_VERSION	2 /* IB specification 1.2 */
 
-enum cm_msg_attr_id {
-	CM_REQ_ATTR_ID	    = __constant_htons(0x0010),
-	CM_MRA_ATTR_ID	    = __constant_htons(0x0011),
-	CM_REJ_ATTR_ID	    = __constant_htons(0x0012),
-	CM_REP_ATTR_ID	    = __constant_htons(0x0013),
-	CM_RTU_ATTR_ID	    = __constant_htons(0x0014),
-	CM_DREQ_ATTR_ID	    = __constant_htons(0x0015),
-	CM_DREP_ATTR_ID	    = __constant_htons(0x0016),
-	CM_SIDR_REQ_ATTR_ID = __constant_htons(0x0017),
-	CM_SIDR_REP_ATTR_ID = __constant_htons(0x0018),
-	CM_LAP_ATTR_ID      = __constant_htons(0x0019),
-	CM_APR_ATTR_ID      = __constant_htons(0x001A)
-};
+#define CM_REQ_ATTR_ID	    __constant_htons(0x0010)
+#define CM_MRA_ATTR_ID	    __constant_htons(0x0011)
+#define CM_REJ_ATTR_ID	    __constant_htons(0x0012)
+#define CM_REP_ATTR_ID	    __constant_htons(0x0013)
+#define CM_RTU_ATTR_ID	    __constant_htons(0x0014)
+#define CM_DREQ_ATTR_ID	    __constant_htons(0x0015)
+#define CM_DREP_ATTR_ID	    __constant_htons(0x0016)
+#define CM_SIDR_REQ_ATTR_ID __constant_htons(0x0017)
+#define CM_SIDR_REP_ATTR_ID __constant_htons(0x0018)
+#define CM_LAP_ATTR_ID      __constant_htons(0x0019)
+#define CM_APR_ATTR_ID      __constant_htons(0x001A)
 
 enum cm_msg_sequence {
 	CM_MSG_SEQUENCE_REQ,
@@ -67,35 +65,35 @@ enum cm_msg_sequence {
 struct cm_req_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 rsvd4;
-	u64 service_id;
-	u64 local_ca_guid;
-	u32 rsvd24;
-	u32 local_qkey;
+	__be32 local_comm_id;
+	__be32 rsvd4;
+	__be64 service_id;
+	__be64 local_ca_guid;
+	__be32 rsvd24;
+	__be32 local_qkey;
 	/* local QPN:24, responder resources:8 */
-	u32 offset32;
+	__be32 offset32;
 	/* local EECN:24, initiator depth:8 */
-	u32 offset36;
+	__be32 offset36;
 	/*
 	 * remote EECN:24, remote CM response timeout:5,
 	 * transport service type:2, end-to-end flow control:1
 	 */
-	u32 offset40;
+	__be32 offset40;
 	/* starting PSN:24, local CM response timeout:5, retry count:3 */
-	u32 offset44;
-	u16 pkey;
+	__be32 offset44;
+	__be16 pkey;
 	/* path MTU:4, RDC exists:1, RNR retry count:3. */
 	u8 offset50;
 	/* max CM Retries:4, SRQ:1, rsvd:3 */
 	u8 offset51;
 
-	u16 primary_local_lid;
-	u16 primary_remote_lid;
+	__be16 primary_local_lid;
+	__be16 primary_remote_lid;
 	union ib_gid primary_local_gid;
 	union ib_gid primary_remote_gid;
 	/* flow label:20, rsvd:6, packet rate:6 */
-	u32 primary_offset88;
+	__be32 primary_offset88;
 	u8 primary_traffic_class;
 	u8 primary_hop_limit;
 	/* SL:4, subnet local:1, rsvd:3 */
@@ -103,12 +101,12 @@ struct cm_req_msg {
 	/* local ACK timeout:5, rsvd:3 */
 	u8 primary_offset95;
 
-	u16 alt_local_lid;
-	u16 alt_remote_lid;
+	__be16 alt_local_lid;
+	__be16 alt_remote_lid;
 	union ib_gid alt_local_gid;
 	union ib_gid alt_remote_gid;
 	/* flow label:20, rsvd:6, packet rate:6 */
-	u32 alt_offset132;
+	__be32 alt_offset132;
 	u8 alt_traffic_class;
 	u8 alt_hop_limit;
 	/* SL:4, subnet local:1, rsvd:3 */
@@ -120,12 +118,12 @@ struct cm_req_msg {
 
 } __attribute__ ((packed));
 
-static inline u32 cm_req_get_local_qpn(struct cm_req_msg *req_msg)
+static inline __be32 cm_req_get_local_qpn(struct cm_req_msg *req_msg)
 {
 	return cpu_to_be32(be32_to_cpu(req_msg->offset32) >> 8);
 }
 
-static inline void cm_req_set_local_qpn(struct cm_req_msg *req_msg, u32 qpn)
+static inline void cm_req_set_local_qpn(struct cm_req_msg *req_msg, __be32 qpn)
 {
 	req_msg->offset32 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
 					 (be32_to_cpu(req_msg->offset32) &
@@ -208,13 +206,13 @@ static inline void cm_req_set_flow_ctrl(
 					  0xFFFFFFFE));
 }
 
-static inline u32 cm_req_get_starting_psn(struct cm_req_msg *req_msg)
+static inline __be32 cm_req_get_starting_psn(struct cm_req_msg *req_msg)
 {
 	return cpu_to_be32(be32_to_cpu(req_msg->offset44) >> 8);
 }
 
 static inline void cm_req_set_starting_psn(struct cm_req_msg *req_msg,
-					   u32 starting_psn)
+					   __be32 starting_psn)
 {
 	req_msg->offset44 = cpu_to_be32((be32_to_cpu(starting_psn) << 8) |
 			    (be32_to_cpu(req_msg->offset44) & 0x000000FF));
@@ -288,13 +286,13 @@ static inline void cm_req_set_srq(struct
 				  ((srq & 0x1) << 3));
 }
 
-static inline u32 cm_req_get_primary_flow_label(struct cm_req_msg *req_msg)
+static inline __be32 cm_req_get_primary_flow_label(struct cm_req_msg *req_msg)
 {
-	return cpu_to_be32((be32_to_cpu(req_msg->primary_offset88) >> 12));
+	return cpu_to_be32(be32_to_cpu(req_msg->primary_offset88) >> 12);
 }
 
 static inline void cm_req_set_primary_flow_label(struct cm_req_msg *req_msg,
-						 u32 flow_label)
+						 __be32 flow_label)
 {
 	req_msg->primary_offset88 = cpu_to_be32(
 				    (be32_to_cpu(req_msg->primary_offset88) &
@@ -350,13 +348,13 @@ static inline void cm_req_set_primary_lo
 					  (local_ack_timeout << 3));
 }
 
-static inline u32 cm_req_get_alt_flow_label(struct cm_req_msg *req_msg)
+static inline __be32 cm_req_get_alt_flow_label(struct cm_req_msg *req_msg)
 {
-	return cpu_to_be32((be32_to_cpu(req_msg->alt_offset132) >> 12));
+	return cpu_to_be32(be32_to_cpu(req_msg->alt_offset132) >> 12);
 }
 
 static inline void cm_req_set_alt_flow_label(struct cm_req_msg *req_msg,
-					     u32 flow_label)
+					     __be32 flow_label)
 {
 	req_msg->alt_offset132 = cpu_to_be32(
 				 (be32_to_cpu(req_msg->alt_offset132) &
@@ -422,8 +420,8 @@ enum cm_msg_response {
  struct cm_mra_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 	/* message MRAed:2, rsvd:6 */
 	u8 offset8;
 	/* service timeout:5, rsvd:3 */
@@ -458,13 +456,13 @@ static inline void cm_mra_set_service_ti
 struct cm_rej_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 	/* message REJected:2, rsvd:6 */
 	u8 offset8;
 	/* reject info length:7, rsvd:1. */
 	u8 offset9;
-	u16 reason;
+	__be16 reason;
 	u8 ari[IB_CM_REJ_ARI_LENGTH];
 
 	u8 private_data[IB_CM_REJ_PRIVATE_DATA_SIZE];
@@ -495,45 +493,45 @@ static inline void cm_rej_set_reject_inf
 struct cm_rep_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
-	u32 local_qkey;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
+	__be32 local_qkey;
 	/* local QPN:24, rsvd:8 */
-	u32 offset12;
+	__be32 offset12;
 	/* local EECN:24, rsvd:8 */
-	u32 offset16;
+	__be32 offset16;
 	/* starting PSN:24 rsvd:8 */
-	u32 offset20;
+	__be32 offset20;
 	u8 resp_resources;
 	u8 initiator_depth;
 	/* target ACK delay:5, failover accepted:2, end-to-end flow control:1 */
 	u8 offset26;
 	/* RNR retry count:3, SRQ:1, rsvd:5 */
 	u8 offset27;
-	u64 local_ca_guid;
+	__be64 local_ca_guid;
 
 	u8 private_data[IB_CM_REP_PRIVATE_DATA_SIZE];
 
 } __attribute__ ((packed));
 
-static inline u32 cm_rep_get_local_qpn(struct cm_rep_msg *rep_msg)
+static inline __be32 cm_rep_get_local_qpn(struct cm_rep_msg *rep_msg)
 {
 	return cpu_to_be32(be32_to_cpu(rep_msg->offset12) >> 8);
 }
 
-static inline void cm_rep_set_local_qpn(struct cm_rep_msg *rep_msg, u32 qpn)
+static inline void cm_rep_set_local_qpn(struct cm_rep_msg *rep_msg, __be32 qpn)
 {
 	rep_msg->offset12 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
 			    (be32_to_cpu(rep_msg->offset12) & 0x000000FF));
 }
 
-static inline u32 cm_rep_get_starting_psn(struct cm_rep_msg *rep_msg)
+static inline __be32 cm_rep_get_starting_psn(struct cm_rep_msg *rep_msg)
 {
 	return cpu_to_be32(be32_to_cpu(rep_msg->offset20) >> 8);
 }
 
 static inline void cm_rep_set_starting_psn(struct cm_rep_msg *rep_msg,
-					   u32 starting_psn)
+					   __be32 starting_psn)
 {
 	rep_msg->offset20 = cpu_to_be32((be32_to_cpu(starting_psn) << 8) |
 			    (be32_to_cpu(rep_msg->offset20) & 0x000000FF));
@@ -600,8 +598,8 @@ static inline void cm_rep_set_srq(struct
 struct cm_rtu_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 
 	u8 private_data[IB_CM_RTU_PRIVATE_DATA_SIZE];
 
@@ -610,21 +608,21 @@ struct cm_rtu_msg {
 struct cm_dreq_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 	/* remote QPN/EECN:24, rsvd:8 */
-	u32 offset8;
+	__be32 offset8;
 
 	u8 private_data[IB_CM_DREQ_PRIVATE_DATA_SIZE];
 
 } __attribute__ ((packed));
 
-static inline u32 cm_dreq_get_remote_qpn(struct cm_dreq_msg *dreq_msg)
+static inline __be32 cm_dreq_get_remote_qpn(struct cm_dreq_msg *dreq_msg)
 {
 	return cpu_to_be32(be32_to_cpu(dreq_msg->offset8) >> 8);
 }
 
-static inline void cm_dreq_set_remote_qpn(struct cm_dreq_msg *dreq_msg, u32 qpn)
+static inline void cm_dreq_set_remote_qpn(struct cm_dreq_msg *dreq_msg, __be32 qpn)
 {
 	dreq_msg->offset8 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
 			    (be32_to_cpu(dreq_msg->offset8) & 0x000000FF));
@@ -633,8 +631,8 @@ static inline void cm_dreq_set_remote_qp
 struct cm_drep_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 
 	u8 private_data[IB_CM_DREP_PRIVATE_DATA_SIZE];
 
@@ -643,37 +641,37 @@ struct cm_drep_msg {
 struct cm_lap_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 
-	u32 rsvd8;
+	__be32 rsvd8;
 	/* remote QPN/EECN:24, remote CM response timeout:5, rsvd:3 */
-	u32 offset12;
-	u32 rsvd16;
+	__be32 offset12;
+	__be32 rsvd16;
 
-	u16 alt_local_lid;
-	u16 alt_remote_lid;
+	__be16 alt_local_lid;
+	__be16 alt_remote_lid;
 	union ib_gid alt_local_gid;
 	union ib_gid alt_remote_gid;
 	/* flow label:20, rsvd:4, traffic class:8 */
-	u32 offset56;
+	__be32 offset56;
 	u8 alt_hop_limit;
 	/* rsvd:2, packet rate:6 */
-	uint8_t offset61;
+	u8 offset61;
 	/* SL:4, subnet local:1, rsvd:3 */
-	uint8_t offset62;
+	u8 offset62;
 	/* local ACK timeout:5, rsvd:3 */
-	uint8_t offset63;
+	u8 offset63;
 
 	u8 private_data[IB_CM_LAP_PRIVATE_DATA_SIZE];
 } __attribute__  ((packed));
 
-static inline u32 cm_lap_get_remote_qpn(struct cm_lap_msg *lap_msg)
+static inline __be32 cm_lap_get_remote_qpn(struct cm_lap_msg *lap_msg)
 {
 	return cpu_to_be32(be32_to_cpu(lap_msg->offset12) >> 8);
 }
 
-static inline void cm_lap_set_remote_qpn(struct cm_lap_msg *lap_msg, u32 qpn)
+static inline void cm_lap_set_remote_qpn(struct cm_lap_msg *lap_msg, __be32 qpn)
 {
 	lap_msg->offset12 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
 					 (be32_to_cpu(lap_msg->offset12) &
@@ -693,17 +691,17 @@ static inline void cm_lap_set_remote_res
 					  0xFFFFFF07));
 }
 
-static inline u32 cm_lap_get_flow_label(struct cm_lap_msg *lap_msg)
+static inline __be32 cm_lap_get_flow_label(struct cm_lap_msg *lap_msg)
 {
-	return be32_to_cpu(lap_msg->offset56) >> 12;
+	return cpu_to_be32(be32_to_cpu(lap_msg->offset56) >> 12);
 }
 
 static inline void cm_lap_set_flow_label(struct cm_lap_msg *lap_msg,
-					 u32 flow_label)
+					 __be32 flow_label)
 {
-	lap_msg->offset56 = cpu_to_be32((flow_label << 12) |
-					 (be32_to_cpu(lap_msg->offset56) &
-					  0x00000FFF));
+	lap_msg->offset56 = cpu_to_be32(
+				 (be32_to_cpu(lap_msg->offset56) & 0x00000FFF) |
+				 (be32_to_cpu(flow_label) << 12));
 }
 
 static inline u8 cm_lap_get_traffic_class(struct cm_lap_msg *lap_msg)
@@ -766,8 +764,8 @@ static inline void cm_lap_set_local_ack_
 struct cm_apr_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 local_comm_id;
-	u32 remote_comm_id;
+	__be32 local_comm_id;
+	__be32 remote_comm_id;
 
 	u8 info_length;
 	u8 ap_status;
@@ -779,10 +777,10 @@ struct cm_apr_msg {
 struct cm_sidr_req_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 request_id;
-	u16 pkey;
-	u16 rsvd;
-	u64 service_id;
+	__be32 request_id;
+	__be16 pkey;
+	__be16 rsvd;
+	__be64 service_id;
 
 	u8 private_data[IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE];
 } __attribute__ ((packed));
@@ -790,26 +788,26 @@ struct cm_sidr_req_msg {
 struct cm_sidr_rep_msg {
 	struct ib_mad_hdr hdr;
 
-	u32 request_id;
+	__be32 request_id;
 	u8 status;
 	u8 info_length;
-	u16 rsvd;
+	__be16 rsvd;
 	/* QPN:24, rsvd:8 */
-	u32 offset8;
-	u64 service_id;
-	u32 qkey;
+	__be32 offset8;
+	__be64 service_id;
+	__be32 qkey;
 	u8 info[IB_CM_SIDR_REP_INFO_LENGTH];
 
 	u8 private_data[IB_CM_SIDR_REP_PRIVATE_DATA_SIZE];
 } __attribute__ ((packed));
 
-static inline u32 cm_sidr_rep_get_qpn(struct cm_sidr_rep_msg *sidr_rep_msg)
+static inline __be32 cm_sidr_rep_get_qpn(struct cm_sidr_rep_msg *sidr_rep_msg)
 {
 	return cpu_to_be32(be32_to_cpu(sidr_rep_msg->offset8) >> 8);
 }
 
 static inline void cm_sidr_rep_set_qpn(struct cm_sidr_rep_msg *sidr_rep_msg,
-				       u32 qpn)
+				       __be32 qpn)
 {
 	sidr_rep_msg->offset8 = cpu_to_be32((be32_to_cpu(qpn) << 8) |
 					(be32_to_cpu(sidr_rep_msg->offset8) &
Index: hw/mthca/mthca_mad.c
===================================================================
--- hw/mthca/mthca_mad.c	(revision 2919)
+++ hw/mthca/mthca_mad.c	(working copy)
@@ -194,7 +194,7 @@ int mthca_process_mad(struct ib_device *
 {
 	int err;
 	u8 status;
-	u16 slid = in_wc ? in_wc->slid : IB_LID_PERMISSIVE;
+	u16 slid = in_wc ? in_wc->slid : be16_to_cpu(IB_LID_PERMISSIVE);
 
 	/* Forward locally generated traps to the SM */
 	if (in_mad->mad_hdr.method == IB_MGMT_METHOD_TRAP &&
Index: hw/mthca/mthca_qp.c
===================================================================
--- hw/mthca/mthca_qp.c	(revision 2919)
+++ hw/mthca/mthca_qp.c	(working copy)
@@ -1469,8 +1469,8 @@ static int build_mlx_header(struct mthca
 		return err;
 	mlx->flags &= ~cpu_to_be32(MTHCA_NEXT_SOLICIT | 1);
 	mlx->flags |= cpu_to_be32((!sqp->qp.ibqp.qp_num ? MTHCA_MLX_VL15 : 0) |
-				  (sqp->ud_header.lrh.destination_lid == 0xffff ?
-				   MTHCA_MLX_SLR : 0) |
+				  (sqp->ud_header.lrh.destination_lid ==
+				   IB_LID_PERMISSIVE ? MTHCA_MLX_SLR : 0) |
 				  (sqp->ud_header.lrh.service_level << 8));
 	mlx->rlid = sqp->ud_header.lrh.destination_lid;
 	mlx->vcrc = 0;
@@ -1490,8 +1490,8 @@ static int build_mlx_header(struct mthca
 	}
 
 	sqp->ud_header.lrh.virtual_lane    = !sqp->qp.ibqp.qp_num ? 15 : 0;
-	if (sqp->ud_header.lrh.destination_lid == 0xffff)
-		sqp->ud_header.lrh.source_lid = 0xffff;
+	if (sqp->ud_header.lrh.destination_lid == IB_LID_PERMISSIVE)
+		sqp->ud_header.lrh.source_lid = IB_LID_PERMISSIVE;
 	sqp->ud_header.bth.solicited_event = !!(wr->send_flags & IB_SEND_SOLICITED);
 	if (!sqp->qp.ibqp.qp_num)
 		ib_get_cached_pkey(&dev->ib_dev, sqp->port,





From jlentini at netapp.com  Thu Jul 28 10:59:45 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 13:59:45 -0400 (EDT)
Subject: [openib-general][PATCH][kdapl]: adding DAT_MEM_TYPE_IA support
In-Reply-To: <Pine.LNX.4.63.0507271104170.19809@beid.voltaire.com>
References: <Pine.LNX.4.63.0507271104170.19809@beid.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507281338550.12296@jlentini-linux.nane.netapp.com>


Thanks for working on this Guy.

I received these errors when I tried to apply this patch:

patching file ib/dapl_openib_util.c
Hunk #1 FAILED at 234.
Hunk #2 FAILED at 281.
Hunk #3 FAILED at 300.
Hunk #4 FAILED at 309.
Hunk #5 FAILED at 318.
5 out of 5 hunks FAILED -- saving rejects to file 
ib/dapl_openib_util.c.rej
patching file ib/dapl_openib_util.h
Hunk #1 FAILED at 92.
1 out of 1 hunk FAILED -- saving rejects to file 
ib/dapl_openib_util.h.rej
patching file ib/dapl_lmr.c
Hunk #1 FAILED at 125.
Hunk #2 succeeded at 147 with fuzz 2.
Hunk #3 FAILED at 164.
Hunk #4 FAILED at 240.
Hunk #5 FAILED at 257.
Hunk #6 FAILED at 284.
Hunk #7 FAILED at 307.
Hunk #8 FAILED at 324.
7 out of 8 hunks FAILED -- saving rejects to file ib/dapl_lmr.c.rej

Due to the errors I had some trouble looking over your changes. Can 
you resend?

One high level comment. I would prefer to see the DAT_MEM_TYPE_IA 
support added in new function called dapl_lmr_create_ia rather than 
overloading the dapl_lmr_create_physical function. Adding a 
dapl_lmr_create_ia function would be more in keeping with the existing 
structure.

A few comments/questions below:

On Wed, 27 Jul 2005, Guy German wrote:

> [kdapl]: this patch adds the option of using DAT_MEM_TYPE_IA in 
> dat_lmr_kcreate
>
> Signed-off-by: Guy German <guyg at voltaire.com>
>
> Index: infiniband/ulp/kdapl/ib/dapl_openib_util.c
> ===================================================================
> --- infiniband/ulp/kdapl/ib/dapl_openib_util.c	(revision 2914)
> +++ infiniband/ulp/kdapl/ib/dapl_openib_util.c	(working copy)
> @@ -234,8 +234,43 @@ int dapl_ib_mr_register(struct dapl_ia *
> 	return -ENOSYS;
> }
>
> +int dapl_ib_mr_register_ia(struct dapl_ia *ia, struct dapl_lmr *lmr,
> +			   union dat_region_description phys_addr, u64 
> length,
> +			   enum dat_mem_priv_flags privileges)
> +{
> +	int status;
> +	int acl;
> +	struct ib_mr *mr;
> +	struct ib_phys_buf buf_list;
> +	u64 iova = 0;
> +	buf_list.addr = phys_addr.for_pa;
> +	buf_list.size = length;
> + +	iova = buf_list.addr;
> +	acl = dapl_ib_convert_mem_privileges(privileges);
> +	acl |= IB_ACCESS_MW_BIND;
> +	mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd,
> +			    &buf_list, 1, acl, &iova); +	if 
> (IS_ERR(mr)) {
> +		status = PTR_ERR(mr);
> +		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
> +			"%s: ib_reg_phys_mr error code return = %d\n",
> +			__FUNCTION__, status);
> +		return status;
> +	}
> +	dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "%s: got handle mr=%p\n",
> +	       __FUNCTION__, mr);
> +
> +	lmr->param.lmr_context = mr->lkey;
> +	lmr->param.rmr_context = mr->rkey;
> +	lmr->param.registered_size = length;
> +	lmr->param.registered_address = phys_addr.for_pa;
> +	lmr->mr = mr;
> +	return 0;
> +}
> +
> int dapl_ib_mr_register_physical(struct dapl_ia *ia, struct dapl_lmr *lmr,
> -				 void *phys_addr, u64 length,
> +				 union dat_region_description phys_addr, u64 
> length,
> 				 enum dat_mem_priv_flags privileges)
> {
> 	int status;
> @@ -246,7 +281,7 @@ int dapl_ib_mr_register_physical(struct
> 	u64 iova = 0;
> 	u64 *array;
>
> -	array = (u64 *) phys_addr;
> +	array = (u64 *)phys_addr.for_array; /* need to add for_u64_array to 
> union */

What does this comment mean?

> 	buf_list = kmalloc(length * sizeof *buf_list, GFP_ATOMIC);
> 	if (!buf_list)
> 		return -ENOMEM;
> @@ -265,8 +300,8 @@ int dapl_ib_mr_register_physical(struct
> 	if (IS_ERR(mr)) {
> 		status = PTR_ERR(mr);
> 		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
> -			     " ib_reg_phys_mr error code return = %d\n",
> -			     status);
> +			     "%s: ib_reg_phys_mr error code return = %d\n",
> +			     __FUNCTION__, status);
> 		return status;
> 	}
> #if 0
> @@ -274,8 +309,8 @@ int dapl_ib_mr_register_physical(struct
> 	status = ib_query_mr(mr, &attr);
> 	if (status < 0) {
> 		dapl_dbg_log(DAPL_DBG_TYPE_ERR,
> -			     " ib_query_mr error code return from ib_query_mr 
> = %d\n",
> -			     status);
> +			     "%s: ib_query_mr error code return from 
> ib_query_mr = %d\n",
> +			     __FUNCTION__, status);

We try to keep lines less than 80 characters. Is there a more concise 
way to convey the message above? In this case the original message 
appears to be poorly written.

> 		ib_dereg_mr(mr);
> 		return status;
> 	}
> @@ -283,10 +318,12 @@ int dapl_ib_mr_register_physical(struct
>
> 	lmr->param.lmr_context = mr->lkey;
> 	lmr->param.rmr_context = mr->rkey;
> +	lmr->param.registered_size = length * PAGE_SIZE;
> +	lmr->param.registered_address = array[0];
> 	lmr->mr = mr;
>
> 	dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
> -		"dapl_ib_mr_register_physical(%p %d) got lkey 0x%x \n",
> +		"%s: (%p %d) got lkey 0x%x \n", __FUNCTION__,
> 		buf_list, length, lmr->param.lmr_context);
> 	return 0;
> }
> Index: infiniband/ulp/kdapl/ib/dapl_openib_util.h
> ===================================================================
> --- infiniband/ulp/kdapl/ib/dapl_openib_util.h	(revision 2914)
> +++ infiniband/ulp/kdapl/ib/dapl_openib_util.h	(working copy)
> @@ -92,8 +92,13 @@ int dapl_ib_mr_register(struct dapl_ia *
> 			void *virt_addr, u64 length,
> 			enum dat_mem_priv_flags privileges);
>
> +int dapl_ib_mr_register_ia(struct dapl_ia *ia, struct dapl_lmr *lmr,
> +			   union dat_region_description phys_addr, u64 
> length,
> +			   enum dat_mem_priv_flags privileges);
> +
> int dapl_ib_mr_register_physical(struct dapl_ia *ia, struct dapl_lmr *lmr,
> -				 void *phys_addr, u64 length,
> +				 union dat_region_description phys_addr, + 
> u64 length,
> 				 enum dat_mem_priv_flags privileges);
>
> int dapl_ib_mr_deregister(struct dapl_lmr *lmr);
> Index: infiniband/ulp/kdapl/ib/dapl_lmr.c
> ===================================================================
> --- infiniband/ulp/kdapl/ib/dapl_lmr.c	(revision 2914)
> +++ infiniband/ulp/kdapl/ib/dapl_lmr.c	(working copy)
> @@ -125,20 +125,21 @@ error1:
> }
>
> static inline int dapl_lmr_create_physical(struct dapl_ia *ia,
> -					   DAT_REGION_DESCRIPTION phys_addr,
> -					   u64 page_count, struct dapl_pz 
> *pz,
> +					   union dat_region_description 
> phys_addr,
> +					   u64 page_count, + 
> enum dat_mem_type mem_type,
> +					   struct dapl_pz *pz,
> 					   enum dat_mem_priv_flags 
> privileges,
> 					   struct dat_lmr **lmr,
> -					   DAT_LMR_CONTEXT *lmr_context,
> -					   DAT_RMR_CONTEXT *rmr_context,
> +					   u32 *lmr_context,
> +					   u32 *rmr_context,

Why remove the DAT_LMR_CONTEXT and DAT_RMR_CONTEXT types? I would 
prefer to remove everyone of them at once.

> 					   u64 *registered_length,
> 					   u64 *registered_address)
> {
> 	struct dapl_lmr *new_lmr;
> -	u64 *array = phys_addr.for_array;
> 	int status;
>
> -	new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr,
> +	new_lmr = dapl_lmr_alloc(ia, mem_type, phys_addr,
> 				 page_count, (struct dat_pz *) pz, 
> privileges);
>
> 	if (NULL == new_lmr) {
> @@ -146,8 +147,14 @@ static inline int dapl_lmr_create_physic
> 		goto error1;
> 	}
>
> -	status = dapl_ib_mr_register_physical(ia, new_lmr, 
> phys_addr.for_array,
> -					      page_count, privileges);

Something is strange here. The call to dapl_ib_mr_register_physical is 
inside an else clause in the current code.

> +	if (DAT_MEM_TYPE_IA == mem_type) {
> +		status = dapl_ib_mr_register_ia(ia, new_lmr, phys_addr,
> +						page_count, privileges);
> +	}
> +	else {
> +		status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr,
> +						      page_count, 
> privileges);
> +	}
>
> 	if (0 != status)
> 		goto error2;
> @@ -157,13 +164,13 @@ static inline int dapl_lmr_create_physic
> 	if (lmr)
> 		*lmr = (struct dat_lmr *)new_lmr;
> 	if (lmr_context)
> -		*lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context;
> +		*lmr_context = (u32)new_lmr->param.lmr_context;
> 	if (rmr_context)
> -		*rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context;
> +		*rmr_context = (u32)new_lmr->param.rmr_context;
> 	if (registered_address)
> -		*registered_address = array[0];
> +		*registered_address = new_lmr->param.registered_address;
> 	if (registered_length)
> -		*registered_length = page_count * PAGE_SIZE;
> +		*registered_length = new_lmr->param.registered_size;
>
> 	return 0;
>
> @@ -233,7 +240,7 @@ int dapl_lmr_kcreate(struct dat_ia *ia,
> 	struct dapl_ia *dapl_ia;
> 	struct dapl_pz *dapl_pz;
> 	int status;
> -
> +
> 	dapl_dbg_log(DAPL_DBG_TYPE_API,
> 		     "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n",
> 		     ia, mem_type);
> @@ -250,10 +257,12 @@ int dapl_lmr_kcreate(struct dat_ia *ia,
> 						 rmr_context, 
> registered_length,
> 						 registered_address);
> 		break;
> +	case DAT_MEM_TYPE_PLATFORM: // used as proprietary Tavor-FMR
> 	case DAT_MEM_TYPE_PHYSICAL:
> +	case DAT_MEM_TYPE_IA:
> 		status = dapl_lmr_create_physical(dapl_ia, 
> region_description,
> -						  length, dapl_pz, 
> privileges,
> -						  lmr, lmr_context,
> +						  length, mem_type, dapl_pz, 
> +						  privileges, lmr, 
> lmr_context,
> 						  rmr_context,
> 						  registered_length,
> 						  registered_address);
> @@ -275,8 +284,6 @@ int dapl_lmr_kcreate(struct dat_ia *ia,
> 					     registered_address);
> 		break;
> 	}
> -	case DAT_MEM_TYPE_PLATFORM:
> -	case DAT_MEM_TYPE_IA:
> 	case DAT_MEM_TYPE_BYPASS:
> 		status = -ENOSYS;
> 		break;
> @@ -300,7 +307,8 @@ int dapl_lmr_free(struct dat_lmr *lmr)
>
> 	switch (dapl_lmr->param.mem_type) {
> 	case DAT_MEM_TYPE_PHYSICAL:
> -	case DAT_MEM_TYPE_VIRTUAL:
> +	case DAT_MEM_TYPE_PLATFORM:
> +	case DAT_MEM_TYPE_IA:
> 	case DAT_MEM_TYPE_LMR:
> 	{
> 		struct dapl_pz *pz;
> @@ -316,8 +324,7 @@ int dapl_lmr_free(struct dat_lmr *lmr)
> 		}
> 		break;
> 	}
> -	case DAT_MEM_TYPE_PLATFORM:
> -	case DAT_MEM_TYPE_IA:
> +	case DAT_MEM_TYPE_VIRTUAL:
> 	case DAT_MEM_TYPE_BYPASS:
> 		status = -ENOSYS;
> 		break;
>


From jlentini at netapp.com  Thu Jul 28 11:06:20 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 14:06:20 -0400 (EDT)
Subject: [openib-general][PATCH][kdapltest]: adding memtype options to
	server
In-Reply-To: <Pine.LNX.4.63.0507271110520.19809@beid.voltaire.com>
References: <Pine.LNX.4.63.0507271110520.19809@beid.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507281404370.12296@jlentini-linux.nane.netapp.com>


Hi Guy,

This patch has been garbled too. See below:

On Wed, 27 Jul 2005, Guy German wrote:

> [kdapltest]: this patch adds the option of using memory registration
> types other then the default PHY on the server side.
>
> Signed-off-by: Guy German <guyg at voltaire.com>
>
> Index: dapltest/test/dapl_server.c
> ===================================================================
> --- dapltest/test/dapl_server.c	(revision 2899)
> +++ dapltest/test/dapl_server.c	(working copy)
> @@ -232,7 +232,14 @@ DT_cs_Server (Params_t * params_ptr)
>      * Create two buffers, large enough to hold ClientInfo and the largest
>      * command we'll use.
>      */
> -    ps_ptr->bpool = DT_BpoolAlloc (NULL,

Something is missing here. If you remove the line above you can't 
replace it with the line below...

> + +    if (!(pt_ptr = DT_Alloc_Per_Test_Data (phead)))

...which has a double +

> +	    goto server_exit;
> +    DT_MemListInit (pt_ptr);
> +    memcpy ( (void *) (uintptr_t) &pt_ptr->Params,
> +	    (const void *) params_ptr,
> +	    sizeof (Params_t));
> +    ps_ptr->bpool = DT_BpoolAlloc (pt_ptr,
> 	    		       phead,
> 			       ps_ptr->ia,
> 			       ps_ptr->pz,
> @@ -243,6 +250,7 @@ DT_cs_Server (Params_t * params_ptr)
> 			       256, /* FIXME query for this value */
> 			       FALSE,
> 			       FALSE);
> +    DT_Free_Per_Test_Data (pt_ptr);
>     if (ps_ptr->bpool == 0)
>     {
> 	DT_Tdep_PT_Printf (phead,
>


From rolandd at cisco.com  Thu Jul 28 11:06:21 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 11:06:21 -0700
Subject: [openib-general] [PATCH] fix sparse warnings
In-Reply-To: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com> (Sean
	Hefty's message of "Thu, 28 Jul 2005 10:35:40 -0700")
References: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com>
Message-ID: <52mzo6pzz6.fsf@cisco.com>

    Sean> I'd like to get a couple of people to test this patch, plus
    Sean> any comments, before committing it.  The majority of the
    Sean> changes are related to MADs and the CM.

This seems to work OK in my very brief testing.  I think at least the
patch below (which includes a real bug fix to mthca_av.c) is needed
for mthca on top of your patch.  Also IPoIB needs some serious
cleanup, which I'll work on.

 - R.

--- infiniband/hw/mthca/mthca_av.c	(revision 2926)
+++ infiniband/hw/mthca/mthca_av.c	(working copy)
@@ -169,7 +169,7 @@ int mthca_read_ah(struct mthca_dev *dev,
 
 	header->lrh.service_level   = be32_to_cpu(ah->av->sl_tclass_flowlabel) >> 28;
 	header->lrh.destination_lid = ah->av->dlid;
-	header->lrh.source_lid      = ah->av->g_slid & 0x7f;
+	header->lrh.source_lid      = cpu_to_be16(ah->av->g_slid & 0x7f);
 	if (ah->av->g_slid & 0x80) {
 		header->grh_present = 1;
 		header->grh.traffic_class =
--- infiniband/hw/mthca/mthca_qp.c	(revision 2926)
+++ infiniband/hw/mthca/mthca_qp.c	(working copy)
@@ -1459,6 +1459,7 @@ static int build_mlx_header(struct mthca
 {
 	int header_size;
 	int err;
+	u16 pkey;
 
 	ib_ud_header_init(256, /* assume a MAD */
 			  sqp->ud_header.grh_present,
@@ -1495,13 +1496,11 @@ static int build_mlx_header(struct mthca
 	sqp->ud_header.bth.solicited_event = !!(wr->send_flags & IB_SEND_SOLICITED);
 	if (!sqp->qp.ibqp.qp_num)
 		ib_get_cached_pkey(&dev->ib_dev, sqp->port,
-				   sqp->pkey_index,
-				   &sqp->ud_header.bth.pkey);
+				   sqp->pkey_index, &pkey);
 	else
 		ib_get_cached_pkey(&dev->ib_dev, sqp->port,
-				   wr->wr.ud.pkey_index,
-				   &sqp->ud_header.bth.pkey);
-	cpu_to_be16s(&sqp->ud_header.bth.pkey);
+				   wr->wr.ud.pkey_index, &pkey);
+	sqp->ud_header.bth.pkey = cpu_to_be16(pkey);
 	sqp->ud_header.bth.destination_qpn = cpu_to_be32(wr->wr.ud.remote_qpn);
 	sqp->ud_header.bth.psn = cpu_to_be32((sqp->send_psn++) & ((1 << 24) - 1));
 	sqp->ud_header.deth.qkey = cpu_to_be32(wr->wr.ud.remote_qkey & 0x80000000 ?


From rolandd at cisco.com  Thu Jul 28 11:12:16 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 11:12:16 -0700
Subject: [openib-general] [PATCH] fix sparse warnings
In-Reply-To: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com> (Sean
	Hefty's message of "Thu, 28 Jul 2005 10:35:40 -0700")
References: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com>
Message-ID: <52iryupzpb.fsf@cisco.com>

By the way, is there any reason that ib_cm.h does

	IB_CM_REJ_PORT_REDIRECT			= 24,

and doesn't define reject reason 25?  It seems like it would be more
sensible to do

	IB_CM_REJ_PORT_CM_REDIRECT		= 24,
	IB_CM_REJ_PORT_REDIRECT			= 25,

(I use reason 25 in SRP code because the Topspin FC/SRP gateway has a
bug where it sends reason code 25 when it means 24)

 - R.


From rolandd at cisco.com  Thu Jul 28 11:17:53 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 11:17:53 -0700
Subject: [openib-general] [PATCH] fix sparse warnings
In-Reply-To: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com> (Sean
	Hefty's message of "Thu, 28 Jul 2005 10:35:40 -0700")
References: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com>
Message-ID: <52ek9ipzfy.fsf@cisco.com>

OK, I committed the trivial changes to IPoIB to make it mostly sparse
clean as well.  The remaining endianness warning is coming from the
fact that skb->protocol is supposed to be big-endian, but
<linux/skbuff.h> defines it to be __u16.

 - R.


From ardavis at ichips.intel.com  Thu Jul 28 11:18:47 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Thu, 28 Jul 2005 11:18:47 -0700
Subject: [openib-general] Re: ib_uat kernel oops
In-Reply-To: <42E90509.4030501@ichips.intel.com>
References: <42E82952.2020302@ichips.intel.com>	<1122558479.4451.739.camel@hal.voltaire.com>
	<42E90509.4030501@ichips.intel.com>
Message-ID: <42E92187.2050708@ichips.intel.com>

Arlin Davis wrote:

> Hal Rosenstock wrote:
>
>>>
>>>
>>> Jul 27 17:36:41 iclust-19 kernel: Unable to handle kernel NULL 
>>> pointer dereference at 0000000000000000 RIP:
>>>   
>>
>> One more thing: were there any IBAT/user IBAT messages in the log before
>> the oops ? Thanks.
>>
>> -- Hal
>>
>>
> No messages.
> I was using the shaharf-ibat branch so I will pick up the latest from 
> the trunk and retry.


Ok, everything is much better with the IBAT version from the trunk.

Thanks,

-arlin

>
> -arlin
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>



From mshefty at ichips.intel.com  Thu Jul 28 11:21:13 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 11:21:13 -0700
Subject: [openib-general] [PATCH] fix sparse warnings
In-Reply-To: <52mzo6pzz6.fsf@cisco.com>
References: <ORSMSX4011XvpFVjCRG0000006e@orsmsx401.amr.corp.intel.com>
	<52mzo6pzz6.fsf@cisco.com>
Message-ID: <42E92219.1050301@ichips.intel.com>

Roland Dreier wrote:
> This seems to work OK in my very brief testing.  I think at least the
> patch below (which includes a real bug fix to mthca_av.c) is needed
> for mthca on top of your patch.  Also IPoIB needs some serious
> cleanup, which I'll work on.

Thanks - I've integrated your patch and will commit shortly.  (I need to 
redo a portion of the patch surrounding at.c.)

- Sean


From damato at psc.edu  Thu Jul 28 11:26:46 2005
From: damato at psc.edu (Joe Damato)
Date: Thu, 28 Jul 2005 14:26:46 -0400
Subject: [openib-general] struct ib_cm_active_param ?
Message-ID: <42E92366.5070209@psc.edu>

Is struct ib_cm_active_param from gen1? If so, is there an equivalent 
struct in gen2? (Is it struct ib_cm_req_param ?)

The specific piece of code that is trying to be ported is: 

conn->ibc_connreq->cr_connparam = (struct ib_cm_active_param) {
                .qp                   = conn->ibc_qp,
                .req_private_data     = msg,
                .req_private_data_len = msg->ibm_nob,
                .responder_resources  = IBNAL_RESPONDER_RESOURCES,
                .initiator_depth      = IBNAL_RESPONDER_RESOURCES,
                .retry_count          = IBNAL_RETRY,
                .rnr_retry_count      = IBNAL_RNR_RETRY,
                .cm_response_timeout  = kibnal_tunables.kib_io_timeout,
                .max_cm_retries       = IBNAL_CM_RETRY,
                .flow_control         = IBNAL_FLOW_CONTROL,
        };


thanks.

-Joe Damato


From sean.hefty at intel.com  Thu Jul 28 11:23:53 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 11:23:53 -0700
Subject: [openib-general] [PATCH] fix sparse warnings
In-Reply-To: <52iryupzpb.fsf@cisco.com>
Message-ID: <ORSMSX4013SOlbpH71y0000006f@orsmsx401.amr.corp.intel.com>

>By the way, is there any reason that ib_cm.h does
>
>	IB_CM_REJ_PORT_REDIRECT			= 24,
>
>and doesn't define reject reason 25?  It seems like it would be more
>sensible to do
>
>	IB_CM_REJ_PORT_CM_REDIRECT		= 24,
>	IB_CM_REJ_PORT_REDIRECT			= 25,

I think that this is just an oversight on my part.  I'll fix as you suggested.
Thanks.

- Sean



From jlentini at netapp.com  Thu Jul 28 11:34:29 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 14:34:29 -0400 (EDT)
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
In-Reply-To: <1122474097.4421.348.camel@hal.voltaire.com>
References: <1122296697.4407.9.camel@hal.voltaire.com> 
	<Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com> 
	<1122468122.4421.328.camel@hal.voltaire.com> 
	<Pine.LNX.4.61.0507270933240.12296@jlentini-linux.nane.netapp.com>
	<1122474097.4421.348.camel@hal.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507281409200.12296@jlentini-linux.nane.netapp.com>



On Wed, 27 Jul 2005, Hal Rosenstock wrote:

> On Wed, 2005-07-27 at 09:54, James Lentini wrote:
>> Being able to debug connectivity problems at the IB level strikes me
>> as very useful. Perhaps some people expect to always use IPoIB on
>> their IB network, and therefore plan to use IP tools. Even in these
>> configurations, native IB protocols would be useful if IPoIB wasn't
>> working.
>
> Right; IPoIB relies on more working (multicasting, etc.) than this
> proposal does.
>
>> It looks like you modeled the ECHO and TIMESTAMP formats after their
>> ICMP counterparts.
>
> Yes.
>
>> The TIMESTAMP type appears to have been dropped from ICMPv6. Do you know why?
>
> No but it sounds like it might be relevant. The only thing I saw were
> some comments related to security and replay attacks. Not sure if that
> was the reason it was eliminated from ICMPv6.

I found several referencess like this one with Google:

  http://www.nessus.org/plugins/index.php?view=single&id=10114

  The remote host answers to an ICMP timestamp request. This
  allows an attacker to know the date which is set on your machine.

  This may help him to defeat all your time based authentication
  protocols.

  Solution : filter out the ICMP timestamp requests (13), and the
  outgoing ICMP timestamp replies (14).

Given that, I'd recommend removing it.


From mshefty at ichips.intel.com  Thu Jul 28 11:42:48 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 11:42:48 -0700
Subject: [openib-general] struct ib_cm_active_param ?
In-Reply-To: <42E92366.5070209@psc.edu>
References: <42E92366.5070209@psc.edu>
Message-ID: <42E92728.2030601@ichips.intel.com>

Joe Damato wrote:
> Is struct ib_cm_active_param from gen1? If so, is there an equivalent 
> struct in gen2? (Is it struct ib_cm_req_param ?)

You're correct.  ib_cm_req_param is the equivalent structure.

- Sean


From jlentini at netapp.com  Thu Jul 28 12:01:24 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 15:01:24 -0400 (EDT)
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
In-Reply-To: <Pine.LNX.4.63.0507281007040.20468@beid.voltaire.com>
References: <Pine.LNX.4.63.0507281007040.20468@beid.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507281447020.12296@jlentini-linux.nane.netapp.com>


Hmm...this is a tricky one. The ib_mthca module can be rmmod-ed at any 
time. That would break things in kdapl. Consider this scenario:

  consumer opens IA
  rmmod ib_mthca (IB support is pulled out from under us)
  rmmod kdapl_ib (user is trying to fix things)

With this patch, the kdapl_ib module's use count would be 1. The user 
would only be able to rmmod kdapl_ib if the kernel was compiled with 
CONFIG_MODULE_FORCE_UNLOAD.

>From my perspective, the current behavior seems better given the 
situation with ib_mthca. This seems acceptable if the ability to rmmod 
is restricted to knowledgeable sysadmins. What do you think?

One more comment below:

On Thu, 28 Jul 2005, Guy German wrote:

> [kdapl]: increments the module ref count on ia_open, thus making
> sure the kdapl_ib module would not be rmmod-ed while in use
>
> Signed-off-by: Guy German <guyg at voltaire.com>
>
> Index: infiniband/ulp/kdapl/ib/dapl_ia.c
> ===================================================================
> --- infiniband/ulp/kdapl/ib/dapl_ia.c	(revision 2914)
> +++ infiniband/ulp/kdapl/ib/dapl_ia.c	(working copy)
> @@ -649,6 +649,7 @@ int dapl_ia_open(const char *name, int a
> 	status = 0;
> 	*ia_ptr = (struct dat_ia *)ia;
> 	*async_evd = (struct dat_evd *)evd;
> +	try_module_get(THIS_MODULE);

We've been using the following convention to indicate that we don't 
care about the return value:

  (void) try_module_get(THIS_MODULE);

>
> bail:
> 	if (status != 0)
> @@ -679,6 +680,7 @@ int dapl_ia_close(struct dat_ia *ia_ptr,
> 	else
> 		status = -EINVAL;
>
> +	module_put(THIS_MODULE);
> 	return status;
> }
>


From sean.hefty at intel.com  Thu Jul 28 12:02:12 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 12:02:12 -0700
Subject: [openib-general] [PATCH] Add CM reject code 25 (Port and CM
	redirection)
Message-ID: <ORSMSX401FRaqbC8wSA00000070@orsmsx401.amr.corp.intel.com>

Patch to add CM reject code 25 as pointed out by Roland.

Roland, can you please verify that SRP changes look right?

Signed-off-by: Sean Hefty <sean.hefty at intel.com>


Index: src/userspace/libibcm/include/infiniband/cm.h
===================================================================
--- src/userspace/libibcm/include/infiniband/cm.h	(revision 2915)
+++ src/userspace/libibcm/include/infiniband/cm.h	(working copy)
@@ -137,7 +137,8 @@ enum ib_cm_rej_reason {
 	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= 21,
 	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= 22,
 	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= 23,
-	IB_CM_REJ_PORT_REDIRECT			= 24,
+	IB_CM_REJ_PORT_CM_REDIRECT		= 24,
+	IB_CM_REJ_PORT_REDIRECT			= 25,
 	IB_CM_REJ_INVALID_MTU			= 26,
 	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= 27,
 	IB_CM_REJ_CONSUMER_DEFINED		= 28,
Index: src/linux-kernel/infiniband/ulp/srp/ib_srp.c
===================================================================
--- src/linux-kernel/infiniband/ulp/srp/ib_srp.c	(revision 2926)
+++ src/linux-kernel/infiniband/ulp/srp/ib_srp.c	(working copy)
@@ -628,7 +628,7 @@ static int srp_cm_handler(struct ib_cm_i
 		printk(KERN_DEBUG PFX "REJ received\n");
 		comp = 1;
 
-		if (event->param.rej_rcvd.reason == IB_CM_REJ_PORT_REDIRECT) {
+		if (event->param.rej_rcvd.reason == IB_CM_REJ_PORT_CM_REDIRECT) {
 			/*
 			 * Additional Reject Info contains
 			 * ClassPortInfo, which has the RedirectGID
@@ -640,7 +640,7 @@ static int srp_cm_handler(struct ib_cm_i
 			target->status = SRP_PORT_REDIRECT;
 		} else if (topspin_workarounds &&
 			   !memcmp(&target->ioc_guid, topspin_oui, 3) &&
-			   event->param.rej_rcvd.reason == __constant_htons(25)) {
+			   event->param.rej_rcvd.reason == IB_CM_REJ_PORT_REDIRECT) {
 			/*
 			 * Topspin/Cisco SRP gateways incorrectly send
 			 * reject reason code 25 when they mean 24
Index: src/linux-kernel/infiniband/include/ib_cm.h
===================================================================
--- src/linux-kernel/infiniband/include/ib_cm.h	(revision 2928)
+++ src/linux-kernel/infiniband/include/ib_cm.h	(working copy)
@@ -169,7 +169,8 @@ enum ib_cm_rej_reason {
 	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= 21,
 	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= 22,
 	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= 23,
-	IB_CM_REJ_PORT_REDIRECT			= 24,
+	IB_CM_REJ_PORT_CM_REDIRECT		= 24,
+	IB_CM_REJ_PORT_REDIRECT			= 25,
 	IB_CM_REJ_INVALID_MTU			= 26,
 	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= 27,
 	IB_CM_REJ_CONSUMER_DEFINED		= 28,





From rolandd at cisco.com  Thu Jul 28 12:06:12 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 12:06:12 -0700
Subject: [openib-general] Re: [PATCH] Add CM reject code 25 (Port and CM
	redirection)
In-Reply-To: <ORSMSX401FRaqbC8wSA00000070@orsmsx401.amr.corp.intel.com> (Sean
	Hefty's message of "Thu, 28 Jul 2005 12:02:12 -0700")
References: <ORSMSX401FRaqbC8wSA00000070@orsmsx401.amr.corp.intel.com>
Message-ID: <52ack6px7f.fsf@cisco.com>

    Sean> Patch to add CM reject code 25 as pointed out by Roland.
    Sean> Roland, can you please verify that SRP changes look right?

Yes, it's exactly how I would change it.

 - R.


From halr at voltaire.com  Thu Jul 28 12:13:49 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 15:13:49 -0400
Subject: [openib-general] [PATCH] [IPoIB] Handle sending of unicast RARP
	responses
Message-ID: <1122578028.4541.50.camel@hal.voltaire.com>

[IPoIB] Handle sending of unicast RARP responses

Signed-off-by: Hal Rosenstock <halr at voltaire.com>

Any chance this can also make it to 2.6.13 ? Thanks.

-- Hal

Index: ipoib_main.c
===================================================================
--- ipoib_main.c	(revision 2929)
+++ ipoib_main.c	(working copy)
@@ -601,9 +601,10 @@ static int ipoib_start_xmit(struct sk_bu
 
 			ipoib_mcast_send(dev, (union ib_gid *) (phdr->hwaddr + 4), skb);
 		} else {
-			/* unicast GID -- should be ARP reply */
+			/* unicast GID -- should be ARP or RARP reply */
 
-			if (be16_to_cpup((__be16 *) skb->data) != ETH_P_ARP) {
+			if ((be16_to_cpup((__be16 *) skb->data) != ETH_P_ARP) &&
+			    (be16_to_cpup((__be16 *) skb->data) != ETH_P_RARP)) {
 				ipoib_warn(priv, "Unicast, no %s: type %04x, QPN %06x "
 					   IPOIB_GID_FMT "\n",
 					   skb->dst ? "neigh" : "dst",





From jlentini at netapp.com  Thu Jul 28 12:42:44 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 15:42:44 -0400 (EDT)
Subject: [openib-general] Re: [PATCH] uDAPL with uCM and uAT support
In-Reply-To: <42E916BC.6040305@ichips.intel.com>
References: <ORSMSX4014ZDgC81Tbd00000064@orsmsx401.amr.corp.intel.com>
	<Pine.LNX.4.61.0507281022470.12296@jlentini-linux.nane.netapp.com>
	<42E916BC.6040305@ichips.intel.com>
Message-ID: <Pine.LNX.4.61.0507281514120.12296@jlentini-linux.nane.netapp.com>



On Thu, 28 Jul 2005, Arlin Davis wrote:

> James Lentini wrote:
>
>> 
>> A couple of questions:
>> 
>> On Sun, 24 Jul 2005, Arlin Davis wrote:
>> 
>>>     /*
>>>      * We now own the EVD, even though we don't have the lock anymore,
>>>      * because we're in the WAITED state.
>>> @@ -182,37 +171,54 @@
>>>      * return right away if the ib_cq_handle associate with these evd
>>>      * equal to IB_INVALID_HANDLE
>>>      */
>>> -    dapls_evd_copy_cq(evd_ptr);
>>> -
>>> -    if (dapls_rbuf_count(&evd_ptr->pending_event_queue) >= threshold)
>>> -    {
>>> -        break;
>>> -    }
>>> -
>>> -    /*
>>> -     * Do not enable the completion notification if this evd is not
>>> -     * a DTO_EVD or RMR_BIND_EVD
>>> +    /* Logic to prevent missing completion between copy_cq (poll)
>>> +     * and completion_notify (re-arm)
>>>      */
>>> -    if ( (!notify_requested) &&
>>> -             ((evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
>>> -              (evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG)) )
>>> +    notify_needed = DAT_TRUE;
>>> +    new_events = 0;
>>> +    while (DAT_TRUE)
>>>     {
>>> -        dat_status = dapls_ib_completion_notify (
>>> -        evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
>>> -        evd_ptr,
>>> -        (evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT) ?
>>> -             IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP );
>>> -
>>> -        DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
>>> -        /* FIXME report error */
>>> -        dapl_os_assert(dat_status == DAT_SUCCESS);
>>> +        dapls_evd_copy_cq(evd_ptr); /* poll for new completions */
>>> +        total_events = dapls_rbuf_count (&evd_ptr->pending_event_queue);
>>> +        new_events = total_events - new_events;
>>> +        if (total_events >= threshold ||
>>> +            (!new_events && notify_needed == DAT_FALSE))
>>> +        {
>>> +            break;
>>> +        }
>>> +
>>> +        /*
>>> +         * Do not enable the completion notification if this evd is not
>>> +         * a DTO_EVD or RMR_BIND_EVD
>>> +         */
>>> +        if ( (evd_ptr->evd_flags & DAT_EVD_DTO_FLAG) ||
>>> +            (evd_ptr->evd_flags & DAT_EVD_RMR_BIND_FLAG) )
>>> +        {
>>> +            dat_status = dapls_ib_completion_notify (
>>> +                    evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle,
>>> +                    evd_ptr,
>>> +                    (evd_ptr->completion_type == 
>>> DAPL_EVD_STATE_SOLICITED_WAIT)
>>> ?
>>> +                         IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP 
>>> );
>>> +
>>> +            DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY);
>>> +            notify_needed = DAT_FALSE;
>>> +            new_events = total_events;
>>> +
>>> +            /* FIXME report error */
>>> +            dapl_os_assert(dat_status == DAT_SUCCESS);
>>> +        }
>>> +        else
>>> +        {
>>> +            break;
>>> +        }
>>> 
>>> -        notify_requested = DAT_TRUE;
>>> +    } /* while completions < threshold, and rearm needed */
>>> 
>>> -        /* Try again.  */
>>> -        continue;
>>> +    if (total_events >= threshold)
>>> +    {
>>> +        break;
>>>     }
>>> -
>>> +
>> 
>> 
>> Can you explain the chages above?
>> 
>> This is my reading of how things used to work: the thread that called 
>> dapl_evd_wait polled the CQ until the number of events reaped was >= the 
>> threshold paramter. If a poll did not reap enough events, the thread turned 
>> notification on (in certain situations). I don't see why it need to do this 
>> though, since it went directly back to polling and the notification upcall 
>> essentially did nothing if there EVD was in the WAITED state.
>> 
>> Obviously you saw a bug in the old implementation. Is my description of how 
>> the code used to work wrong?
>
> You are correct, but there is a small window where you can miss the CQ event 
> if you re-arm and wait but the CQ completion/event occured between the last 
> poll and the rearm/wait.  This change adds another cq_poll after every re-arm 
> to cover this senario and make sure notification is armed and new events are 
> expected before going into the wait.

Good catch.

>> 
>>> Index: dapl/udapl/Makefile
>>> ===================================================================
>>> --- dapl/udapl/Makefile    (revision 2899)
>>> +++ dapl/udapl/Makefile    (working copy)
>>> @@ -122,7 +122,8 @@
>>> #
>>> ifeq ($(VERBS),openib)
>>> PROVIDER = $(TOPDIR)/../openib
>>> -CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
>>> +CFLAGS   += -DOPENIB
>>> +#CFLAGS   += -DCQ_WAIT_OBJECT
>> 
>> 
>> Do the OpenIB uverbs support a CQ_WAIT_OBJECT? If not, we should remove the 
>> line above. When I see it commented out, it makes me think that the 
>> functionality is supported but not implemented in uDAPL.
>
> Yes, openIB supports direct wait objects via fd's. There are some issues with 
> the openIB implementation that need to be worked out.

Ok. Let's add a note that says when X is fixed uncomment this line.

>> 
>>> Index: dapl/common/dapl_evd_resize.c
>>> ===================================================================
>>> --- dapl/common/dapl_evd_resize.c    (revision 2899)
>>> +++ dapl/common/dapl_evd_resize.c    (working copy)
>>> @@ -67,72 +67,130 @@
>>>     IN    DAT_EVD_HANDLE       evd_handle,
>>>     IN    DAT_COUNT       evd_qlen )
>>> {
>>> -    DAPL_IA        *ia_ptr;
>>> -    DAPL_EVD        *evd_ptr;
>>> -    DAT_COUNT       pend_cnt;
>>> -    DAT_RETURN        dat_status;
>>> +    DAPL_IA          *ia_ptr;
>>> +    DAPL_EVD         *evd_ptr;
>>> +    DAT_EVENT        *event_ptr;
>>> +    DAT_EVENT        *events;
>>> +    DAT_EVENT        *orig_event;
>>> +    DAPL_RING_BUFFER free_event_queue;
>>> +    DAPL_RING_BUFFER pending_event_queue;
>>> +    DAT_COUNT        pend_cnt;
>>> +    DAT_COUNT        i;
>>> 
>>>     dapl_dbg_log (DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n",
>>>           evd_handle, evd_qlen);
>>> 
>>>     if (DAPL_BAD_HANDLE (evd_handle, DAPL_MAGIC_EVD))
>>>     {
>>> -    dat_status = DAT_ERROR (DAT_INVALID_HANDLE,0);
>>> -    goto bail;
>>> +        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG1);
>>>     }
>>> 
>>>     evd_ptr     = (DAPL_EVD *)evd_handle;
>>>     ia_ptr      = evd_ptr->header.owner_ia;
>>> 
>>> -    if ( evd_qlen == evd_ptr->qlen )
>>> +    if ((evd_qlen <= 0) || (evd_ptr->qlen > evd_qlen))
>>>     {
>>> -     dat_status = DAT_SUCCESS;
>>> -     goto bail;
>>> +        return DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
>>>     }
>> 
>> 
>> Your check to make sure that the new size is greater that 0 is a good one.
>> 
>> I don't think we should return an error when the consumer asks for a size 
>> less than the current size. A return of success seems more appropriate. We 
>> can remove the old behavoir of shrinking the EVD if you like. I'm sure that 
>> makes things simpler.
>> 
> agree. I see no need to shrink.
>
>> In general, I think we should try to retain the single return point 
>> convention (in other words, we should keep the "goto bail" stuff). I admit 
>> that we should have made better use of this. For example, there should have 
>> been a second "bail" label (bail_unlock?) just before the last call to 
>> dapl_os_unlock(). That would have saved us several lines of code in this 
>> function.
>> 
> agree.
>
>>>     if ( evd_qlen > ia_ptr->hca_ptr->ia_attr.max_evd_qlen )
>>>     {
>>> -    dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2);
>>> -    goto bail;
>>> +    return DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_TEVD);
>>>     }
>>> 
>>>     dapl_os_lock(&evd_ptr->header.lock);
>>> 
>>> -    /* Don't try to resize if we are actively waiting */
>>>     if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED)
>>>     {
>>>         dapl_os_unlock(&evd_ptr->header.lock);
>>> -    dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
>>> -    goto bail;
>>> +        return DAT_ERROR (DAT_INVALID_STATE,0);
>>>     }
>>> 
>>>     pend_cnt = dapls_rbuf_count(&evd_ptr->pending_event_queue);
>>>     if (pend_cnt > evd_qlen) {
>>> -    dapl_os_unlock(&evd_ptr->header.lock);
>>> -    dat_status = DAT_ERROR (DAT_INVALID_STATE,0);
>>> -    goto bail;
>>> +        dapl_os_unlock(&evd_ptr->header.lock);
>>> +        return DAT_ERROR (DAT_INVALID_STATE,0);
>>> +    }
>>> +
>>> +    if (DAT_SUCCESS != dapls_ib_cq_resize(evd_ptr->header.owner_ia,
>>> +                                          evd_ptr,
>>> +                                          &evd_qlen))
>>> +    {
>>> +        dapl_os_unlock(&evd_ptr->header.lock);
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>> +    }
>>> +
>>> +    /* Allocate EVENTs */
>>> +    events = (DAT_EVENT *) dapl_os_alloc (evd_qlen * sizeof (DAT_EVENT));
>>> +    if (!events)
>>> +    {
>>> +        dapl_os_unlock(&evd_ptr->header.lock);
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>>     }
>>> +    event_ptr = events;
>>> 
>>> -    dat_status = dapls_ib_cq_resize(evd_ptr->header.owner_ia,
>>> -                    evd_ptr,
>>> -                    &evd_qlen);
>>> -    if (dat_status != DAT_SUCCESS)
>>> +    /* allocate free event queue */
>>> +    if (DAT_SUCCESS != dapls_rbuf_alloc (&free_event_queue, evd_qlen))
>>>     {
>>> +        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>>         dapl_os_unlock(&evd_ptr->header.lock);
>>> -    goto bail;
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>>     }
>>> 
>>> -    dat_status = dapls_evd_event_realloc (evd_ptr, evd_qlen);
>>> -    if (dat_status != DAT_SUCCESS)
>>> +    /* allocate pending event queue */
>>> +    if (DAT_SUCCESS != dapls_rbuf_alloc (&pending_event_queue, evd_qlen))
>>>     {
>>> +        dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>>         dapl_os_unlock(&evd_ptr->header.lock);
>>> -    goto bail;
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>>     }
>>> 
>>> +    for (i = 0; i < pend_cnt; i++)
>>> +    {
>>> +        orig_event = dapls_rbuf_remove(&evd_ptr->pending_event_queue);
>>> +        if (orig_event == NULL) {
>>> +            dapl_dbg_log (DAPL_DBG_TYPE_ERR, " Inconsistent event 
>>> queue\n");
>>> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>> +            dapl_os_unlock(&evd_ptr->header.lock);
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>> +        }
>>> +        memcpy(event_ptr, orig_event, sizeof(DAT_EVENT));
>>> +        if (DAT_SUCCESS != dapls_rbuf_add(&pending_event_queue, 
>>> event_ptr)) {
>>> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>> +            dapl_os_unlock(&evd_ptr->header.lock);
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>> +        }
>>> +        event_ptr++;
>>> +    }
>>> +
>>> +    for (i = pend_cnt; i < evd_qlen; i++)
>>> +    {
>>> +        if (DAT_SUCCESS != dapls_rbuf_add(&free_event_queue,
>>> +                                          (void *) event_ptr)) {
>>> +            dapl_os_free(event_ptr, evd_qlen * sizeof (DAT_EVENT));
>>> +            dapl_os_unlock(&evd_ptr->header.lock);
>>> +        return DAT_ERROR 
>>> (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY);
>>> +        }
>>> +        event_ptr++;
>>> +    }
>>> +
>>> +    dapls_rbuf_destroy (&evd_ptr->free_event_queue);
>>> +    dapls_rbuf_destroy (&evd_ptr->pending_event_queue);
>>> +    if (evd_ptr->events)
>>> +    {
>>> +        dapl_os_free (evd_ptr->events, evd_ptr->qlen * sizeof 
>>> (DAT_EVENT));
>>> +    }
>>> +    evd_ptr->free_event_queue    = free_event_queue;
>>> +    evd_ptr->pending_event_queue = pending_event_queue;
>>> +    evd_ptr->events              = events;
>>> +    evd_ptr->qlen                = evd_qlen;
>>> +
>> 
>> 
>> 
>> Can you hide this in dapls_ib_cq_resize()?
>> 
> which part?

All the new stuff. :) Also some of this used to 
be done in dapls_evd_event_realloc(). I don't see you using that 
function anymore.

I assume that the need to restructure this function was based on 
OpenIB limitations. My preference would be to hide those in the 
dapls_ib_* code.

>
>> 
>>>     dapl_os_unlock(&evd_ptr->header.lock);
>>> 
>>> - bail:
>>> -    return dat_status;
>>> +    dapl_dbg_log (DAPL_DBG_TYPE_RTN,
>>> +            "dapl_evd_resize returns DAT_SUCCESS\n");
>>> +
>>> +    return DAT_SUCCESS;
>>> }
>>> 
>>> /*
>> 
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>> 
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>> 
>


From rolandd at cisco.com  Thu Jul 28 13:14:23 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 13:14:23 -0700
Subject: [openib-general] Re: [PATCH] [IPoIB] Handle sending of unicast RARP
	responses
In-Reply-To: <1122578028.4541.50.camel@hal.voltaire.com> (Hal Rosenstock's
	message of "28 Jul 2005 15:13:49 -0400")
References: <1122578028.4541.50.camel@hal.voltaire.com>
Message-ID: <5264uupu1s.fsf@cisco.com>

    Hal> [IPoIB] Handle sending of unicast RARP responses

Thanks, applied.

    Hal> Any chance this can also make it to 2.6.13 ? Thanks.

We'll see... I'll send this and another simple fix out today.

 - R.


From tduffy at sun.com  Thu Jul 28 11:50:55 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 28 Jul 2005 11:50:55 -0700
Subject: [openib-general] [PATCH] uat: make uat.c compile on 2.6.13-rc3
Message-ID: <1122576655.14803.6.camel@duffman>

This patch is similar to the one for ucm.  It updates the class code to
work with 2.6.13-rc3.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.13-rc3-openib/drivers/infiniband/core/uat.c
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/core/uat.c	(revision 2928)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/core/uat.c	(working copy)
@@ -806,8 +806,8 @@ static struct file_operations ib_uat_fop
 };
 
 
-static struct class_simple *ib_uat_class;
-static struct cdev	  ib_uat_cdev;
+static struct class *ib_uat_class;
+static struct cdev   ib_uat_cdev;
 
 static int __init ib_uat_init(void)
 {
@@ -827,17 +827,14 @@ static int __init ib_uat_init(void)
 		goto err_cdev;
 	}
 
-	ib_uat_class = class_simple_create(THIS_MODULE, "infiniband_uat");
+	ib_uat_class = class_create(THIS_MODULE, "infiniband_uat");
 	if (IS_ERR(ib_uat_class)) {
 		result = PTR_ERR(ib_uat_class);
 		uat_dbg("Error <%d> creating class\n", result);
 		goto err_class;
 	}
 
-	class_simple_device_add(ib_uat_class,
-				IB_UAT_DEV,
-				NULL,
-				"uat");
+	class_device_create(ib_uat_class, IB_UAT_DEV, NULL, "uat");
 	
 	idr_init(&ctx_id_table);
 	init_MUTEX(&ctx_id_mutex);
@@ -853,8 +850,8 @@ err_chr:
 
 static void __exit ib_uat_cleanup(void)
 {
-	class_simple_device_remove(IB_UAT_DEV);
-	class_simple_destroy(ib_uat_class);
+	class_device_destroy(ib_uat_class, IB_UAT_DEV);
+	class_destroy(ib_uat_class);
 	cdev_del(&ib_uat_cdev);
 	unregister_chrdev_region(IB_UAT_DEV, 1);
 }




From tduffy at sun.com  Thu Jul 28 13:23:16 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 28 Jul 2005 13:23:16 -0700
Subject: [openib-general][PATCH][kdapltest]: inc/dec module ref count
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405BB3BF@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A89405BB3BF@taurus.voltaire.com>
Message-ID: <1122582196.14803.9.camel@duffman>

On Thu, 2005-07-28 at 14:27 +0300, Guy German wrote:
> Right.
> I was not sure this policy regards kdapltest as well.

kdapltest can be an abomination since it never has to make it upstream.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/faca1e34/attachment.sig>

From rolandd at cisco.com  Thu Jul 28 13:31:45 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 13:31:45 -0700
Subject: [openib-general] [PATCH 1/2] [IB/cm]: Correct CM port redirect
	reject codes
In-Reply-To: <20057281331.dR47KhjBsU48JfGE@cisco.com>
Message-ID: <20057281331.7vqhiAJ1Yc0um2je@cisco.com>

Reject code 24 is port and CM redirection, not just port redirection.
Port redirection alone is code 25.

Therefore we should rename code 24 to IB_CM_REJ_PORT_CM_REDIRECT and
use IB_CM_REJ_PORT_REDIRECT for code 25.

Signed-off-by: Roland Dreier <rolandd at cisco.com>
---

 drivers/infiniband/include/ib_cm.h |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

4e38d36d88ead4e56f3155573976da84d5df18b3
diff --git a/drivers/infiniband/include/ib_cm.h b/drivers/infiniband/include/ib_cm.h
--- a/drivers/infiniband/include/ib_cm.h
+++ b/drivers/infiniband/include/ib_cm.h
@@ -169,7 +169,8 @@ enum ib_cm_rej_reason {
 	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= __constant_htons(21),
 	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= __constant_htons(22),
 	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= __constant_htons(23),
-	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(24),
+	IB_CM_REJ_PORT_CM_REDIRECT		= __constant_htons(24),
+	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(25),
 	IB_CM_REJ_INVALID_MTU			= __constant_htons(26),
 	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= __constant_htons(27),
 	IB_CM_REJ_CONSUMER_DEFINED		= __constant_htons(28),


From rolandd at cisco.com  Thu Jul 28 13:31:45 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 13:31:45 -0700
Subject: [openib-general] [PATCH 0/2] REALLY final InfiniBand updates for
	2.6.13
Message-ID: <20057281331.dR47KhjBsU48JfGE@cisco.com>

Here are two last-minute fixes for InfiniBand: one to avoid pain in
releasing 2.6.13 with an incorrect constant and then having to rename
the enum later, and one to fix RARP on IP-over-InfiniBand.

After Greg's scolding, I'm sending them as patches, but they're also
available in the git tree at

    rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

(pending mirror propagation)

The patches update the following files:

 include/ib_cm.h        |    3 ++-
 ulp/ipoib/ipoib_main.c |    5 +++--
 2 files changed, 5 insertions(+), 3 deletions(-)

through the following changes:

commit 0dca0f7bf82face7b700890318d5550fd542cabf
Author: Hal Rosenstock <halr at voltaire.com>
Date:   Thu Jul 28 13:17:26 2005 -0700

    [PATCH] [IPoIB] Handle sending of unicast RARP responses
    
    RARP replies are another valid case where IPoIB may need to send a
    unicast packet with no neighbour structure.
    
    Signed-off-by: Hal Rosenstock <halr at voltaire.com>
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

commit 4e38d36d88ead4e56f3155573976da84d5df18b3
Author: Roland Dreier <roland at eddore.topspincom.com>
Date:   Thu Jul 28 13:16:30 2005 -0700

    [PATCH] [IB/cm]: Correct CM port redirect reject codes
    
    Reject code 24 is port and CM redirection, not just port redirection.
    Port redirection alone is code 25.
    
    Therefore we should rename code 24 to IB_CM_REJ_PORT_CM_REDIRECT and
    use IB_CM_REJ_PORT_REDIRECT for code 25.
    
    Signed-off-by: Roland Dreier <rolandd at cisco.com>

Thanks,
  Roland


From rolandd at cisco.com  Thu Jul 28 13:31:45 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 13:31:45 -0700
Subject: [openib-general] [PATCH 2/2] [IPoIB] Handle sending of unicast RARP
	responses
In-Reply-To: <20057281331.7vqhiAJ1Yc0um2je@cisco.com>
Message-ID: <20057281331.6BOCwhBG35vXYVIR@cisco.com>

From: Hal Rosenstock <halr at voltaire.com>

RARP replies are another valid case where IPoIB may need to send a
unicast packet with no neighbour structure.

Signed-off-by: Hal Rosenstock <halr at voltaire.com>
Signed-off-by: Roland Dreier <rolandd at cisco.com>
---

 drivers/infiniband/ulp/ipoib/ipoib_main.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

0dca0f7bf82face7b700890318d5550fd542cabf
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -600,9 +600,10 @@ static int ipoib_start_xmit(struct sk_bu
 
 			ipoib_mcast_send(dev, (union ib_gid *) (phdr->hwaddr + 4), skb);
 		} else {
-			/* unicast GID -- should be ARP reply */
+			/* unicast GID -- should be ARP or RARP reply */
 
-			if (be16_to_cpup((u16 *) skb->data) != ETH_P_ARP) {
+			if ((be16_to_cpup((__be16 *) skb->data) != ETH_P_ARP) &&
+			    (be16_to_cpup((__be16 *) skb->data) != ETH_P_RARP)) {
 				ipoib_warn(priv, "Unicast, no %s: type %04x, QPN %06x "
 					   IPOIB_GID_FMT "\n",
 					   skb->dst ? "neigh" : "dst",


From ardavis at ichips.intel.com  Thu Jul 28 13:40:04 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Thu, 28 Jul 2005 13:40:04 -0700
Subject: [openib-general] [PATCH] uDAPL dapltest
Message-ID: <42E942A4.7070006@ichips.intel.com>

James,

Patch to fix build problem.

-arlin


Signed-off by: Arlin Davis <ardavis at ichips.intel.com>

Index: dapltest/test/dapl_bpool.c
===================================================================
--- dapltest/test/dapl_bpool.c  (revision 2930)
+++ dapltest/test/dapl_bpool.c  (working copy)
@@ -363,8 +363,8 @@
                     "BPOOL alloc_size     %x\n",
                     (int) bpool_ptr->alloc_size);
     DT_Tdep_PT_Printf (phead,
-                    "BPOOL pz_handle      %x\n",
-                    (int) bpool_ptr->pz_handle);
+                    "BPOOL pz_handle      %p\n",
+                     bpool_ptr->pz_handle);
     DT_Tdep_PT_Printf (phead,
                     "BPOOL num_segs       %x\n",
                     (int) bpool_ptr->num_segs);



From tduffy at sun.com  Thu Jul 28 13:48:15 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 28 Jul 2005 13:48:15 -0700
Subject: [openib-general] [PATCH] AT: cleanup some sparse warnings
Message-ID: <1122583695.14803.18.camel@duffman>

This patch cleans up a few sparse warnings in AT.

Signed-off-by: Tom Duffy <tduffy at sun.com>

Index: linux-2.6.13-rc3-openib/drivers/infiniband/core/at.c
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/core/at.c	(revision 2931)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/core/at.c	(working copy)
@@ -224,7 +224,7 @@ static void ib_dev_remove(struct ib_at_d
 	ib_dev->pend_op = 0;
 
 	dev_put(ib_dev->netdev);
-	ib_dev->netdev = 0;
+	ib_dev->netdev = NULL;
 	ib_dev->old_ip = 0;
 	ib_dev->ip = 0;
 	ib_dev->valid = 0;
@@ -325,7 +325,7 @@ static int ib_devs_changed(void)
 
 		for (i = 0; i < n; i++)
 			if (netdev == ibdevs[i]) {
-				ibdevs[i] = 0;	/* dev handled - not new */
+				ibdevs[i] = NULL; /* dev handled - not new */
 				break;
 			}
 
@@ -1165,7 +1165,7 @@ static int resolve_route(struct route_re
 	}
 
 	if (req->pend.type == IBAT_REQ_ATS)
-		return resolve_ats_route(req, 0);
+		return resolve_ats_route(req, NULL);
 
 	WARN("bad req %p type %d", req, req->pend.type);
 	return -1;
@@ -1401,7 +1401,7 @@ int ib_at_paths_by_route(struct ib_at_ib
 				IBAT_REQ_PATHREC, same_path_req);
 
 	if (req_start(&pending_reqs, &preq->pend, parent))
-		resolve_path(preq, 0);
+		resolve_path(preq, NULL);
 
 	return 0;			/* async req */
 }
@@ -1454,7 +1454,7 @@ int ib_at_ips_by_gid(union ib_gid *gid, 
 				IBAT_REQ_ATSARP, same_ats_ips_req);
 
 	if (req_start(&pending_reqs, &areq->pend, parent))
-		resolve_ats_ips(areq, 0);
+		resolve_ats_ips(areq, NULL);
 
 	return 0;			/* async req */
 }
@@ -1526,11 +1526,11 @@ static struct packet_type ib_at_arp_type
 };
 
 static struct notifier_block ib_at_netdev_notifier = {
-	notifier_call:	inetaddr_event,
+	.notifier_call = inetaddr_event,
 };
 
 static struct notifier_block ib_at_inetaddr_notifier = {
-	notifier_call:	inetaddr_event,
+	.notifier_call = inetaddr_event,
 };
 
 static int ib_at_init(void)
Index: linux-2.6.13-rc3-openib/drivers/infiniband/core/at_priv.h
===================================================================
--- linux-2.6.13-rc3-openib/drivers/infiniband/core/at_priv.h	(revision 2931)
+++ linux-2.6.13-rc3-openib/drivers/infiniband/core/at_priv.h	(working copy)
@@ -130,8 +130,8 @@ static const struct ib_field ats_rec_tab
 				 IB_SA_SERVICE_REC_SERVICE_GID)
 
 #define IB_ATS_SERVICE_NAME             "DAPL Address Translation Service"
-#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453)
-#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453)
+#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453ULL)
+#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453ULL)
 #define IB_ATS_OPENIB_MAGIC_KEY         cpu_to_be16(IB_OPENIB_OUI & 0xffff)
 
 //#define WARN(fmt, ...)	while (0) {}




From mshefty at ichips.intel.com  Thu Jul 28 13:59:48 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 13:59:48 -0700
Subject: [openib-general] [PATCH] AT: cleanup some sparse warnings
In-Reply-To: <1122583695.14803.18.camel@duffman>
References: <1122583695.14803.18.camel@duffman>
Message-ID: <42E94744.9020709@ichips.intel.com>

Tom Duffy wrote:
> This patch cleans up a few sparse warnings in AT.
>  
> Index: linux-2.6.13-rc3-openib/drivers/infiniband/core/at_priv.h
>  
>  #define IB_ATS_SERVICE_NAME             "DAPL Address Translation Service"
> -#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453)
> -#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453)
> +#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453ULL)
> +#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453ULL)
>  #define IB_ATS_OPENIB_MAGIC_KEY         cpu_to_be16(IB_OPENIB_OUI & 0xffff)

I usually use __constant_cpu_to_be* to byte swap constants.  (I think I did 
it to allow the use of the #define in case statements.)

- Sean


From mlleinin at hpcn.ca.sandia.gov  Thu Jul 28 13:59:42 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Thu, 28 Jul 2005 13:59:42 -0700
Subject: [openib-general] OpenIB main server power outage this weekend
Message-ID: <1122584382.4435.75.camel@localhost>

Hi folks,

   There is a scheduled power outage this Saturday (July 30th) for the
building that houses the main OpenIB server.  We expect the power to be
out from 7am-6pm PDT.  We'll be bringing down the server a few hours
before then and bring it back up as soon as the power is turned back on.
We've managed to schedule this outage on a Saturday to overlap with
everyone's weekend and minimize the impact to OpenIB work.

  Thanks and sorry for the inconvenience,

     - Matt




From Thomas.Duffy.99 at alumni.brown.edu  Thu Jul 28 14:02:08 2005
From: Thomas.Duffy.99 at alumni.brown.edu (Tom Duffy)
Date: Thu, 28 Jul 2005 14:02:08 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
	OpenRDMA community development discussion
In-Reply-To: <OFE3DDA7D4.D619FD04-ON8825704C.0059B3BC-8825704C.00622D7C@us.ibm.com>
References: <OFE3DDA7D4.D619FD04-ON8825704C.0059B3BC-8825704C.00622D7C@us.ibm.com>
Message-ID: <1122584529.14803.24.camel@duffman>

On Thu, 2005-07-28 at 10:52 -0700, Venkata Jagana wrote:       
> 5)  OpenIB enablement of iWARP announcement 
> 
>        Not much time left to discuss in detail the OpenIB announcement
> of iWARP enablement 
>        and what it means to OpenRDMA in terms of any additional
> conflict. 
> 
>        Venkat has clarified that based on previous email and conf call
> discussions between 
>        openRDMA and openIB groups that these two groups are
> complimentary to each other 
>        in terms of Upper layer protocols support but there is a
> conflict around modified Gen2 verbs 
>        usage versus RNIC-PI. 
> 
>         RNIC vendors seem to be quite strongly committed to supporting
> RNIC-PI. 
> 
>         There are different views expressed regarding the convergence
> of gen2 verbs and RNIC-PI. 
>         One view is that the convergence in the interfaces need to
> happen over a period of time so that 
>          there will be a single PI supporting both IB and iWARP in the
> linux mainline kernel. 
> 
>         The other view regarding the convergence is that there is no
> need to have this convergence happen 
>         since they can coexist in the kernel at the sametime with
> different APIs. 
> 
>          Only the time will tell what will happen but ultimately, it
> depends upon what the linux kernel maintainers 
>           have to say. 

At OLS (and in previous forums), the kernel maintainers have made it
*very* clear that there should only be one API.

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/5622e3b2/attachment.sig>

From tduffy at sun.com  Thu Jul 28 14:10:30 2005
From: tduffy at sun.com (Tom Duffy)
Date: Thu, 28 Jul 2005 14:10:30 -0700
Subject: [openib-general] [PATCH] AT: cleanup some sparse warnings
In-Reply-To: <42E94744.9020709@ichips.intel.com>
References: <1122583695.14803.18.camel@duffman>
	<42E94744.9020709@ichips.intel.com>
Message-ID: <1122585030.14803.30.camel@duffman>

On Thu, 2005-07-28 at 13:59 -0700, Sean Hefty wrote:
> Tom Duffy wrote:
> > This patch cleans up a few sparse warnings in AT.
> >  
> > Index: linux-2.6.13-rc3-openib/drivers/infiniband/core/at_priv.h
> >  
> >  #define IB_ATS_SERVICE_NAME             "DAPL Address Translation Service"
> > -#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453)
> > -#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453)
> > +#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453ULL)
> > +#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453ULL)
> >  #define IB_ATS_OPENIB_MAGIC_KEY         cpu_to_be16(IB_OPENIB_OUI & 0xffff)
> 
> I usually use __constant_cpu_to_be* to byte swap constants.  (I think I did 
> it to allow the use of the #define in case statements.)

I am not sure what you are getting at.  This patch corrects the error:

drivers/infiniband/core/at.c:102:12: warning: constant 0x10000ce100415453 is so big it is long

-tduffy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/042c149b/attachment.sig>

From jlentini at netapp.com  Thu Jul 28 14:14:50 2005
From: jlentini at netapp.com (James Lentini)
Date: Thu, 28 Jul 2005 17:14:50 -0400 (EDT)
Subject: [openib-general] IBAT error messages
Message-ID: <Pine.LNX.4.61.0507281712520.12296@jlentini-linux.nane.netapp.com>


Hal, do you see these error messages when building the IBAT code? I am 
using a vanilla Dell x86 machine.

drivers/infiniband/core/at.c: In function `build_ats_req':
drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type
drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type
drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type


From mshefty at ichips.intel.com  Thu Jul 28 14:27:52 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 14:27:52 -0700
Subject: [openib-general] [PATCH] AT: cleanup some sparse warnings
In-Reply-To: <1122585030.14803.30.camel@duffman>
References: <1122583695.14803.18.camel@duffman>	
	<42E94744.9020709@ichips.intel.com>
	<1122585030.14803.30.camel@duffman>
Message-ID: <42E94DD8.2020406@ichips.intel.com>

Tom Duffy wrote:
>>> #define IB_ATS_SERVICE_NAME             "DAPL Address Translation Service"
>>>-#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453)
>>>-#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453)
>>>+#define IB_ATS_SERVICE_ID               cpu_to_be64(0x10000ce100415453ULL)
>>>+#define IB_ATS_LAST_SERVICE_ID          cpu_to_be64(0x10000ce1ff415453ULL)
>>> #define IB_ATS_OPENIB_MAGIC_KEY         cpu_to_be16(IB_OPENIB_OUI & 0xffff)
> I am not sure what you are getting at.  This patch corrects the error:
> 
> drivers/infiniband/core/at.c:102:12: warning: constant 0x10000ce100415453 is so big it is long

The change is fine.  My comments were relative to the code itself, and not 
this specific change.  Sorry about the confusion.  I would write the first 
change as:

#define IB_ATS_SERVICE_ID  __constant_cpu_to_be64(0x10000ce100415453ULL)

(And so on for the others.)  I'm not sure which call is preferred, but if I 
remember correctly, I had compile issues trying to use #define labels in 
case statements that went to cpu_to_be*, rather than __constant_cpu_to_be*.

- Sean


From rolandd at cisco.com  Thu Jul 28 14:31:53 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Thu, 28 Jul 2005 14:31:53 -0700
Subject: [openib-general] [RFC] SRQ API and midlayer implementation
Message-ID: <52k6jaobw6.fsf@cisco.com>

Here are some additions to ib_verbs.h and verbs.c that add a
fleshed-out SRQ API and implement the midlayer stubs for SRQ
handling.  The API is slightly different than what I posted before (I
realized we need async event handlers for SRQs).

Any comments?  Does this look good to check in and use as a base for
implementing SRQ support in mthca?

 - R.


--- infiniband/include/ib_verbs.h	(revision 2929)
+++ infiniband/include/ib_verbs.h	(working copy)
@@ -5,6 +5,7 @@
  * Copyright (c) 2004 Topspin Corporation.  All rights reserved.
  * Copyright (c) 2004 Voltaire Corporation.  All rights reserved.
  * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Cisco Systems.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -243,9 +244,12 @@ struct ib_port_modify {
 
 enum ib_event_type {
 	IB_EVENT_CQ_ERR,
+	IB_EVENT_SRQ_ERR,
+	IB_EVENT_SRQ_LIMIT_REACHED,
 	IB_EVENT_QP_FATAL,
 	IB_EVENT_QP_REQ_ERR,
 	IB_EVENT_QP_ACCESS_ERR,
+	IB_EVENT_QP_LAST_WQE_REACHED,
 	IB_EVENT_COMM_EST,
 	IB_EVENT_SQ_DRAINED,
 	IB_EVENT_PATH_MIG,
@@ -263,6 +267,7 @@ struct ib_event {
 	union {
 		struct ib_cq	*cq;
 		struct ib_qp	*qp;
+		struct ib_srq	*srq;
 		u8		port_num;
 	} element;
 	enum ib_event_type	event;
@@ -385,6 +390,23 @@ enum ib_cq_notify {
 	IB_CQ_NEXT_COMP
 };
 
+enum ib_srq_attr_mask {
+	IB_SRQ_MAX_WR	= 1 << 0,
+	IB_SRQ_LIMIT	= 1 << 1,
+};
+
+struct ib_srq_attr {
+	u32	max_wr;
+	u32	max_sge;
+	u32	srq_limit;
+};
+
+struct ib_srq_init_attr {
+	void		      (*event_handler)(struct ib_event *, void *);
+	void		       *srq_context;
+	struct ib_srq_attr	attr;
+};
+
 struct ib_qp_cap {
 	u32	max_send_wr;
 	u32	max_recv_wr;
@@ -712,10 +734,11 @@ struct ib_cq {
 };
 
 struct ib_srq {
-	struct ib_device	*device;
-	struct ib_uobject	*uobject;
-	struct ib_pd		*pd;
-	void			*srq_context;
+	struct ib_device       *device;
+	struct ib_pd	       *pd;
+	struct ib_uobject      *uobject;
+	void		      (*event_handler)(struct ib_event *, void *);
+	void		       *srq_context;
 	atomic_t		usecnt;
 };
 
@@ -829,6 +852,18 @@ struct ib_device {
 	int                        (*query_ah)(struct ib_ah *ah,
 					       struct ib_ah_attr *ah_attr);
 	int                        (*destroy_ah)(struct ib_ah *ah);
+	struct ib_srq *            (*create_srq)(struct ib_pd *pd,
+						 struct ib_srq_init_attr *srq_init_attr,
+						 struct ib_udata *udata);
+	int                        (*modify_srq)(struct ib_srq *srq,
+						 struct ib_srq_attr *srq_attr,
+						 enum ib_srq_attr_mask srq_attr_mask);
+	int                        (*query_srq)(struct ib_srq *srq,
+						struct ib_srq_attr *srq_attr);
+	int                        (*destroy_srq)(struct ib_srq *srq);
+	int                        (*post_srq_recv)(struct ib_srq *srq,
+						    struct ib_recv_wr *recv_wr,
+						    struct ib_recv_wr **bad_recv_wr);
 	struct ib_qp *             (*create_qp)(struct ib_pd *pd,
 						struct ib_qp_init_attr *qp_init_attr,
 						struct ib_udata *udata);
@@ -1041,6 +1076,66 @@ int ib_query_ah(struct ib_ah *ah, struct
 int ib_destroy_ah(struct ib_ah *ah);
 
 /**
+ * ib_create_srq - Creates a SRQ associated with the specified protection
+ *   domain.
+ * @pd: The protection domain associated with the SRQ.
+ * @srq_init_attr: A list of initial attributes required to create the SRQ.
+ *
+ * srq_attr->max_wr and srq_attr->max_sge are read the determine the
+ * requested size of the SRQ, and set to the actual values allocated
+ * on return.  If ib_create_srq() succeeds, then max_wr and max_sge
+ * will always be at least as large as the requested values.
+ */
+struct ib_srq *ib_create_srq(struct ib_pd *pd,
+			     struct ib_srq_init_attr *srq_init_attr);
+
+/**
+ * ib_modify_srq - Modifies the attributes for the specified SRQ and then
+ *   transitions the SRQ to the given state.
+ * @srq: The SRQ to modify.
+ * @srq_attr: On input, specifies the SRQ attributes to modify.  On output,
+ *   the current values of selected SRQ attributes are returned.
+ * @srq_attr_mask: A bit-mask used to specify which attributes of the SRQ
+ *   are being modified.
+ *
+ * The mask may contain IB_SRQ_MAX_WR to resize the SRQ and/or
+ * IB_SRQ_LIMIT to set the SRQ's limit and request notification when
+ * the number of receives queued drops below the limit.
+ */
+int ib_modify_srq(struct ib_srq *srq,
+		  struct ib_srq_attr *srq_attr,
+		  enum ib_srq_attr_mask srq_attr_mask);
+
+/**
+ * ib_query_srq - Returns the attribute list and current values for the
+ *   specified SRQ.
+ * @srq: The SRQ to query.
+ * @srq_attr: The attributes of the specified SRQ.
+ */
+int ib_query_srq(struct ib_srq *srq,
+		 struct ib_srq_attr *srq_attr);
+
+/**
+ * ib_destroy_srq - Destroys the specified SRQ.
+ * @srq: The SRQ to destroy.
+ */
+int ib_destroy_srq(struct ib_srq *srq);
+
+/**
+ * ib_post_srq_recv - Posts a list of work requests to the specified SRQ.
+ * @srq: The SRQ to post the work request on.
+ * @recv_wr: A list of work requests to post on the receive queue.
+ * @bad_recv_wr: On an immediate failure, this parameter will reference
+ *   the work request that failed to be posted on the QP.
+ */
+static inline int ib_post_srq_recv(struct ib_srq *srq,
+				   struct ib_recv_wr *recv_wr,
+				   struct ib_recv_wr **bad_recv_wr)
+{
+	return srq->device->post_srq_recv(srq, recv_wr, bad_recv_wr);
+}
+
+/**
  * ib_create_qp - Creates a QP associated with the specified protection
  *   domain.
  * @pd: The protection domain associated with the QP.
--- infiniband/core/verbs.c	(revision 2929)
+++ infiniband/core/verbs.c	(working copy)
@@ -154,6 +154,66 @@ int ib_destroy_ah(struct ib_ah *ah)
 }
 EXPORT_SYMBOL(ib_destroy_ah);
 
+/* Shared receive queues */
+
+struct ib_srq *ib_create_srq(struct ib_pd *pd,
+			     struct ib_srq_init_attr *srq_init_attr)
+{
+	struct ib_srq *srq;
+
+	if (!pd->device->create_srq)
+		return ERR_PTR(-ENOSYS);
+
+	srq = pd->device->create_srq(pd, srq_init_attr, NULL);
+
+	if (!IS_ERR(srq)) {
+		srq->device    	   = pd->device;
+		srq->pd        	   = pd;
+		srq->uobject       = NULL;
+		srq->event_handler = srq_init_attr->event_handler;
+		srq->srq_context   = srq_init_attr->srq_context;
+		atomic_inc(&pd->usecnt);
+		atomic_set(&srq->usecnt, 0);
+	}
+
+	return srq;
+}
+EXPORT_SYMBOL(ib_create_srq);
+
+int ib_modify_srq(struct ib_srq *srq,
+		  struct ib_srq_attr *srq_attr,
+		  enum ib_srq_attr_mask srq_attr_mask)
+{
+	return srq->device->modify_srq(srq, srq_attr, srq_attr_mask);
+}
+EXPORT_SYMBOL(ib_modify_srq);
+
+int ib_query_srq(struct ib_srq *srq,
+		 struct ib_srq_attr *srq_attr)
+{
+	return srq->device->query_srq ?
+		srq->device->query_srq(srq, srq_attr) : -ENOSYS;
+}
+EXPORT_SYMBOL(ib_query_srq);
+
+int ib_destroy_srq(struct ib_srq *srq)
+{
+	struct ib_pd *pd;
+	int ret;
+
+	if (atomic_read(&srq->usecnt))
+		return -EBUSY;
+
+	pd = srq->pd;
+
+	ret = srq->device->destroy_srq(srq);
+	if (!ret)
+		atomic_dec(&pd->usecnt);
+
+	return ret;
+}
+EXPORT_SYMBOL(ib_destroy_srq);
+
 /* Queue pairs */
 
 struct ib_qp *ib_create_qp(struct ib_pd *pd,


From halr at voltaire.com  Thu Jul 28 14:29:48 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 17:29:48 -0400
Subject: [openib-general] Re: IBAT error messages
In-Reply-To: <Pine.LNX.4.61.0507281712520.12296@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0507281712520.12296@jlentini-linux.nane.netapp.com>
Message-ID: <1122586188.4541.223.camel@hal.voltaire.com>

On Thu, 2005-07-28 at 17:14, James Lentini wrote:
> Hal, do you see these error messages when building the IBAT code? I am 
> using a vanilla Dell x86 machine.
> 
> drivers/infiniband/core/at.c: In function `build_ats_req':
> drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type
> drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type
> drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type

I don't see these on my x86 machine but there is a patch from Tom
addressing this that will be incorporated shortly.

-- Hal



From mshefty at ichips.intel.com  Thu Jul 28 14:38:46 2005
From: mshefty at ichips.intel.com (Sean Hefty)
Date: Thu, 28 Jul 2005 14:38:46 -0700
Subject: [openib-general] [RFC] SRQ API and midlayer implementation
In-Reply-To: <52k6jaobw6.fsf@cisco.com>
References: <52k6jaobw6.fsf@cisco.com>
Message-ID: <42E95066.2090106@ichips.intel.com>

Roland Dreier wrote:
> Any comments?  Does this look good to check in and use as a base for
> implementing SRQ support in mthca?

Looks good to me.

- Sean


From halr at voltaire.com  Thu Jul 28 14:45:22 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 17:45:22 -0400
Subject: [openib-general] InfiniBand EndPort Liveness and Responsiveness
In-Reply-To: <Pine.LNX.4.61.0507281409200.12296@jlentini-linux.nane.netapp.com>
References: <1122296697.4407.9.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507261333220.12296@jlentini-linux.nane.netapp.com>
	<1122468122.4421.328.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507270933240.12296@jlentini-linux.nane.netapp.com>
	<1122474097.4421.348.camel@hal.voltaire.com>
	<Pine.LNX.4.61.0507281409200.12296@jlentini-linux.nane.netapp.com>
Message-ID: <1122587121.4541.262.camel@hal.voltaire.com>

On Thu, 2005-07-28 at 14:34, James Lentini wrote:
> I found several referencess like this one with Google:
> 
>   http://www.nessus.org/plugins/index.php?view=single&id=10114
> 
>   The remote host answers to an ICMP timestamp request. This
>   allows an attacker to know the date which is set on your machine.
> 
>   This may help him to defeat all your time based authentication
>   protocols.
> 
>   Solution : filter out the ICMP timestamp requests (13), and the
>   outgoing ICMP timestamp replies (14).
> 
> Given that, I'd recommend removing it.

It will be removed from the next version of the writeup. Thanks.

-- Hal



From guyg at voltaire.com  Thu Jul 28 15:04:44 2005
From: guyg at voltaire.com (Guy German)
Date: Fri, 29 Jul 2005 01:04:44 +0300
Subject: [openib-general][PATCH][kdapltest]: adding memtype options to
	server
Message-ID: <D4F8F0B3820E754C887699BEF26A8940107B54@taurus.voltaire.com>

Hi James,

>>Hi Guy,
>>This patch has been garbled too. See below:

I will resend all the garbled patches sunday morning

>> -    ps_ptr->bpool = DT_BpoolAlloc (NULL,
>>Something is missing here. If you remove the line above you can't 
>>replace it with the line below...
>> + +    if (!(pt_ptr = DT_Alloc_Per_Test_Data (phead)))
>>...which has a double +

Why not ?

I replaced the first line :

-    ps_ptr->bpool = DT_BpoolAlloc (NULL,

with :

+ if (!(pt_ptr = DT_Alloc_Per_Test_Data (phead)))
+    goto server_exit;
+    DT_MemListInit (pt_ptr);
+    memcpy ( (void *) (uintptr_t) &pt_ptr->Params,
+             (const void *) params_ptr,
+              sizeof (Params_t));
+    ps_ptr->bpool = DT_BpoolAlloc (pt_ptr,

What I basically did was allocating pt_ptr and 
copying the params to the ds before calling DT_BpoolAlloc.
This time I call it with pt_ptr instead of NULL (that leads
to default use of physical memory registration).

(The + + has probably to do with mailer/diff problems)


From halr at voltaire.com  Thu Jul 28 15:50:21 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 18:50:21 -0400
Subject: [openib-general] Re: [PATCH] AT: cleanup some sparse warnings
In-Reply-To: <1122583695.14803.18.camel@duffman>
References: <1122583695.14803.18.camel@duffman>
Message-ID: <1122591020.4605.10.camel@hal.voltaire.com>

On Thu, 2005-07-28 at 16:48, Tom Duffy wrote:
> This patch cleans up a few sparse warnings in AT.

Thanks. Applied.

-- Hal



From guyg at voltaire.com  Thu Jul 28 15:56:04 2005
From: guyg at voltaire.com (Guy German)
Date: Fri, 29 Jul 2005 01:56:04 +0300
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
Message-ID: <D4F8F0B3820E754C887699BEF26A8940107B58@taurus.voltaire.com>

Hi James,

That's an interesting point.
I don't know why the ib_mthca can be rmmod-ed at any 
time, without dependencies (I guess it's a trusted env.
issue).
I need to check what happens if you rmmod ib_mthca and 
then calling ia_close. It supposed to decrement 
the ref count even on failure from the hca.

Any way I think that in a production level kernel
you are not supposed to do rmmod AT ALL.

Thanks,
Guy.



-----Original Message-----
From: James Lentini [mailto:jlentini at netapp.com]
Sent: Thu 7/28/2005 10:01 PM
To: Guy German
Cc: openib-general
Subject: Re: [openib-general][PATCH][kdapl]: inc/dec module ref count
 

Hmm...this is a tricky one. The ib_mthca module can be rmmod-ed at any 
time. That would break things in kdapl. Consider this scenario:

  consumer opens IA
  rmmod ib_mthca (IB support is pulled out from under us)
  rmmod kdapl_ib (user is trying to fix things)

With this patch, the kdapl_ib module's use count would be 1. The user 
would only be able to rmmod kdapl_ib if the kernel was compiled with 
CONFIG_MODULE_FORCE_UNLOAD.

>From my perspective, the current behavior seems better given the 
situation with ib_mthca. This seems acceptable if the ability to rmmod 
is restricted to knowledgeable sysadmins. What do you think?

One more comment below:

On Thu, 28 Jul 2005, Guy German wrote:

> [kdapl]: increments the module ref count on ia_open, thus making
> sure the kdapl_ib module would not be rmmod-ed while in use
>
> Signed-off-by: Guy German <guyg at voltaire.com>
>
> Index: infiniband/ulp/kdapl/ib/dapl_ia.c
> ===================================================================
> --- infiniband/ulp/kdapl/ib/dapl_ia.c	(revision 2914)
> +++ infiniband/ulp/kdapl/ib/dapl_ia.c	(working copy)
> @@ -649,6 +649,7 @@ int dapl_ia_open(const char *name, int a
> 	status = 0;
> 	*ia_ptr = (struct dat_ia *)ia;
> 	*async_evd = (struct dat_evd *)evd;
> +	try_module_get(THIS_MODULE);

We've been using the following convention to indicate that we don't 
care about the return value:

  (void) try_module_get(THIS_MODULE);

>
> bail:
> 	if (status != 0)
> @@ -679,6 +680,7 @@ int dapl_ia_close(struct dat_ia *ia_ptr,
> 	else
> 		status = -EINVAL;
>
> +	module_put(THIS_MODULE);
> 	return status;
> }
>



From halr at voltaire.com  Thu Jul 28 15:54:44 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 28 Jul 2005 18:54:44 -0400
Subject: [openib-general] Re: IBAT error messages
In-Reply-To: <Pine.LNX.4.61.0507281712520.12296@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.61.0507281712520.12296@jlentini-linux.nane.netapp.com>
Message-ID: <1122591087.4605.14.camel@hal.voltaire.com>

On Thu, 2005-07-28 at 17:14, James Lentini wrote:
> Hal, do you see these error messages when building the IBAT code? I am 
> using a vanilla Dell x86 machine.
> 
> drivers/infiniband/core/at.c: In function `build_ats_req':
> drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type
> drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type
> drivers/infiniband/core/at.c:102: warning: integer constant is too large for "long" type

This should be fixed in r2932 and beyond thanks to Tom. Let me know if
this is still a problem.

-- Hal



From guyg at voltaire.com  Thu Jul 28 16:21:47 2005
From: guyg at voltaire.com (Guy German)
Date: Fri, 29 Jul 2005 02:21:47 +0300
Subject: [openib-general][PATCH][kdapl]: adding DAT_MEM_TYPE_IA support
Message-ID: <D4F8F0B3820E754C887699BEF26A8940107B5B@taurus.voltaire.com>

Hi James,

>Due to the errors I had some trouble looking over your changes. Can 
>you resend?

Will do.

>One high level comment. I would prefer to see the DAT_MEM_TYPE_IA 
>support added in new function called dapl_lmr_create_ia rather than 
>overloading the dapl_lmr_create_physical function. Adding a 
>dapl_lmr_create_ia function would be more in keeping with the existing 
>structure.

I think that we can save duplicated copy-pased-code if we stick to 
one function.

> +	array = (u64 *)phys_addr.for_array; /* need to add for_u64_array to 
> union */
>What does this comment mean?

I think the right way to do it is :
array = phys_addr.for_u64_array
(Givven the union consists of a new type u64* called "for_u64_array")

> -			     " ib_query_mr error code return from ib_query_mr 
> = %d\n",
> -			     status);
> +			     "%s: ib_query_mr error code return from 
> ib_query_mr = %d\n",
> +			     __FUNCTION__, status);
>We try to keep lines less than 80 characters. Is there a more concise 
>way to convey the message above? In this case the original message 
>appears to be poorly written.

I will try to fix it to 80 chars long.

> +					   u32 *lmr_context,
> +					   u32 *rmr_context,
>Why remove the DAT_LMR_CONTEXT and DAT_RMR_CONTEXT types? I would 
>prefer to remove everyone of them at once.

Ok, no problem. 
seemed like a good idea while I was there already .. :)

>Something is strange here. The call to dapl_ib_mr_register_physical is 
>inside an else clause in the current code.
> +	if (DAT_MEM_TYPE_IA == mem_type) {
> +		status = dapl_ib_mr_register_ia(ia, new_lmr, phys_addr,
> +						page_count, privileges);
> +	}
> +	else {
> +		status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr,
> +						      page_count, 
> privileges);
> +	}

This was done to allow DAT_MEM_TYPE_PLATFORM behave as PHYSICAL as well.
The reason is only temporary and propriatery. I don't mind changing
the else with an explicit "if"



From jagana at us.ibm.com  Thu Jul 28 18:21:15 2005
From: jagana at us.ibm.com (Venkata Jagana)
Date: Thu, 28 Jul 2005 18:21:15 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
	OpenRDMA community development discussion
In-Reply-To: <1122584529.14803.24.camel@duffman>
Message-ID: <OFA1D36135.40179B5E-ON8825704D.0000C540-8825704D.00077015@us.ibm.com>

> At OLS (and in previous forums), the kernel maintainers have made it
> *very* clear that there should only be one API.

It is good to know but can you be specific about who those maintainers 
are and under what context they mentioned this at OLS and what ever the 
previous forums that you are referring to (wish I had gone to OLS 
but couldn't).

Per networking summit report (http://lwn.net/Articles/144272/),
my understanding is that the maintainers are looking for RDMA code but
there is no mention about the API aspects.

If anyone attended any one of the summits (netconf or kernel) and
would be great if they can shed some light on this discussion.

Thanks
Venkat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/0cc2d70f/attachment.html>

From jkrauska at cisco.com  Thu Jul 28 18:39:00 2005
From: jkrauska at cisco.com (Joel Krauska)
Date: Thu, 28 Jul 2005 18:39:00 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
 OpenRDMA community development discussion
In-Reply-To: <OFA1D36135.40179B5E-ON8825704D.0000C540-8825704D.00077015@us.ibm.com>
References: <OFA1D36135.40179B5E-ON8825704D.0000C540-8825704D.00077015@us.ibm.com>
Message-ID: <42E988B4.9000209@cisco.com>

Venkata Jagana wrote:
> 
>  > At OLS (and in previous forums), the kernel maintainers have made it
>  > *very* clear that there should only be one API.
> 
> It is good to know but can you be specific about who those maintainers
> are and under what context they mentioned this at OLS and what ever the
> previous forums that you are referring to (wish I had gone to OLS
> but couldn't).

<my recollection of the discussion -- nothing more..>

I attended the OLS BOF on IB, and there was a bit of general scoffing 
that OpenRDMA plans to provide some of the same hooks that OpenIB 
already has...

(I don't know if this is true or what..)

The general statement I heard was that since OpenIB is already in the 
mainline, future similar functionality should play nice with OpenIB 
interfaces and not rewrite from scratch.

</my recollection of the discussion -- nothing more..>

They were also asking if any of the OpenRDMA folks were in the room, and
it seemed no one was..  (I don't consider myself "involved" in either 
project.)

You might drop Bob Woodruff a line and ask for any clarification.

http://www.linuxsymposium.org/2005/view_abstract.php?content_key=87

--joel


From boas1 at llnl.gov  Thu Jul 28 21:05:16 2005
From: boas1 at llnl.gov (Bill Boas)
Date: Thu, 28 Jul 2005 21:05:16 -0700
Subject: [openib-general] Aug 22 Workshop Announcement and Draft program
Message-ID: <6.1.2.0.2.20050728210057.02868e90@mail-lc.llnl.gov>

Please see this through www.openib.org. We look forward to as many as 
possible of you joining us there. If you see an error, have a question or 
suggestion please make to us through the Track leaders as posted in the 
program.

Please register as soon as you can. Also pass this information on freely to 
colleagues, friends and customers who may also be interested.

Thank you.

Bill.

Bill Boas                               bboas at llnl.gov
ICCD LLNL, B-453, R-2018                Wk: 925-422-4110
7000 East Ave, L-555                    Cell: 925-337-2224
Livermore, CA 94551                     Pgr: 877-203-2248 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Datacenter Fabric Workshop	Announcementv0.9.doc
Type: application/msword
Size: 30208 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/5aba8619/attachment.doc>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Datacenter Fabric Workshop Program Draft v0_5	AS.doc
Type: application/msword
Size: 105984 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050728/5aba8619/attachment-0001.doc>

From sean.hefty at intel.com  Fri Jul 29 01:18:55 2005
From: sean.hefty at intel.com (Sean Hefty)
Date: Fri, 29 Jul 2005 01:18:55 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
	OpenRDMA community development discussion
In-Reply-To: <42E988B4.9000209@cisco.com>
Message-ID: <ORSMSX401ZDgC81Tbdk00000072@orsmsx401.amr.corp.intel.com>

>> It is good to know but can you be specific about who those maintainers
>> are and under what context they mentioned this at OLS and what ever the
>> previous forums that you are referring to (wish I had gone to OLS
>> but couldn't).
>
>The general statement I heard was that since OpenIB is already in the
>mainline, future similar functionality should play nice with OpenIB
>interfaces and not rewrite from scratch.

During the IB BOF, I believe that it was Christoph who said something along the
lines of: an RNIC vendor that wants their driver included with Linux should
submit something that works with the OpenIB code currently in Linux with minimal
changes to the stack.  Once that's in, work can begin on slowly creating a
common RDMA API.

- Sean



From hch at lst.de  Fri Jul 29 05:01:49 2005
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 29 Jul 2005 14:01:49 +0200
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
	OpenRDMA community development discussion
In-Reply-To: <1122584529.14803.24.camel@duffman>
References: <OFE3DDA7D4.D619FD04-ON8825704C.0059B3BC-8825704C.00622D7C@us.ibm.com>
	<1122584529.14803.24.camel@duffman>
Message-ID: <20050729120149.GA30154@lst.de>

On Thu, Jul 28, 2005 at 02:02:08PM -0700, Tom Duffy wrote:
> At OLS (and in previous forums), the kernel maintainers have made it
> *very* clear that there should only be one API.

_and_ that this api is neither RNIC-PI or KDAPL.  In fact for anything
that doesn't look very similar to the current IB midlayer you'd need
very convincing arguments.



From hch at lst.de  Fri Jul 29 05:04:16 2005
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 29 Jul 2005 14:04:16 +0200
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
	OpenRDMA community development discussion
In-Reply-To: <ORSMSX401ZDgC81Tbdk00000072@orsmsx401.amr.corp.intel.com>
References: <42E988B4.9000209@cisco.com>
	<ORSMSX401ZDgC81Tbdk00000072@orsmsx401.amr.corp.intel.com>
Message-ID: <20050729120416.GB30154@lst.de>

On Fri, Jul 29, 2005 at 01:18:55AM -0700, Sean Hefty wrote:
> During the IB BOF, I believe that it was Christoph who said something along the
> lines of: an RNIC vendor that wants their driver included with Linux should
> submit something that works with the OpenIB code currently in Linux with minimal
> changes to the stack.  Once that's in, work can begin on slowly creating a
> common RDMA API.

Exactly.  Note that no one cares about yet another useless midlayer at
all, iWarp isn't interesting to us at all until you actually open up
your card drivers or rather start writing sane ones.  Roland writing the
new mthca driver of mellanox IB cards is what got the OpenIB ball
rolling.



From halr at voltaire.com  Fri Jul 29 07:17:58 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 29 Jul 2005 10:17:58 -0400
Subject: [openib-general] [RFCv2] InfiniBand EndPort Liveness and
	Responsiveness
Message-ID: <1122646269.4605.809.camel@hal.voltaire.com>

Hi,

Here is version 2 of a writeup on proposed IB endport liveness and
responsiveness. This "grew" out of discussions on ibping on this list
some time ago and is updated based on comments from James. Thanks again
in advance for any comments on this.

-- Hal

InfiniBand EndPort Liveness and Responsiveness

Motivation

  While there exist methods to obtain some degree of the liveness and
  responsiveness of an InfiniBand end port using existing mechanisms
  (e.g. SM Get attribute), these rely on a lack of protection as well
  as not allowing sufficient flexibility. In addition, less of the 
  end port stack may be exercised.

  IBA supports general service (GS) vendor classes. There are 2 ranges
  of these: the first range does not support RMPP whereas the second
  range does. OpenIB has a vendor OUI of 0x001405. A vendor class of 
  0x34 (in vendor class 2 range) would be utilized for this. The class
  version in the MAD header indicates the version of this. Currently,
  this is version 1.

Packet Format
  (Refer to IBA 1.2 p. 1006 for vendor class 2 MAD format)

  Bytes 40-255 are defined as vendor data as follows:

In MAD if not RMPP'd, or in first MAD if RMPP'd:
  Byte 40: Type (1 byte)
    1 - Echo request
    2 - Echo response
    3 - LID/GUID information request
    4 - LID/GUID information response
  
  Byte 41-43: Reserved

  Byte 44-45: Identifier
  Byte 46-47: Sequence Number

  Byte 48-255: Data for Type 

In subsequent (non first) RMPP MADs:
  Byte 40-255: Continued Data for Type


Echo

The data received in the echo request message must be returned
in the echo reply message.

The identifier and sequence number may be used by the echo sender to aid
in matching the replies with the echo requests. For example, the
identifier might be used like a port in TCP or UDP to identify a
session, and the sequence number might be incremented on each echo
request sent. The echoer returns these same values in the echo reply.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-255: Echo data

Identifier. 16 bits.
This field is used to help match echo requests to the associated reply.
It may be set to zero.

Sequence number. 16 bits.
This field is used to help match echo requests to the associated reply.
It may be zero.

Echo Data. Variable length.
Implementation specific data.


LID/GUID information

Obtain the LID and port GUID of the requested port.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-49: LID
Byte 50-57: Port GUID





From robert.j.woodruff at intel.com  Fri Jul 29 08:14:25 2005
From: robert.j.woodruff at intel.com (Woodruff, Robert J)
Date: Fri, 29 Jul 2005 08:14:25 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22)
	summary:OpenRDMA community development discussion
Message-ID: <1AC79F16F5C5284499BB9591B33D6F00051442E3@orsmsx408>

Venkat wrote, 
>If anyone attended any one of the summits (netconf or kernel) and 
>would be great if they can shed some light on this discussion. 

Roland attended the kernel summit and he was
also at the InfiniBand BOF where we discussed 
the possibility of modifying the
IB verbs to also support iWarp as probably the right way to
go. Not sure if this was discussed at the kernel summit or not,
but perhaps Roland can provide some insight on that question. 

woody


From jlentini at netapp.com  Fri Jul 29 09:27:52 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 29 Jul 2005 12:27:52 -0400 (EDT)
Subject: [openib-general][PATCH][kdapltest]: adding memtype options to
	server
In-Reply-To: <D4F8F0B3820E754C887699BEF26A8940107B54@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A8940107B54@taurus.voltaire.com>
Message-ID: <Pine.LNX.4.61.0507291225470.12296@jlentini-linux.nane.netapp.com>



On Fri, 29 Jul 2005, Guy German wrote:

> Hi James,
>
>>> Hi Guy,
>>> This patch has been garbled too. See below:
>
> I will resend all the garbled patches sunday morning
>
>>> -    ps_ptr->bpool = DT_BpoolAlloc (NULL,
>>> Something is missing here. If you remove the line above you can't
>>> replace it with the line below...
>>> + +    if (!(pt_ptr = DT_Alloc_Per_Test_Data (phead)))
>>> ...which has a double +
>
> Why not ?
>
> I replaced the first line :
>
> -    ps_ptr->bpool = DT_BpoolAlloc (NULL,
>
> with :
>
> + if (!(pt_ptr = DT_Alloc_Per_Test_Data (phead)))
> +    goto server_exit;
> +    DT_MemListInit (pt_ptr);
> +    memcpy ( (void *) (uintptr_t) &pt_ptr->Params,
> +             (const void *) params_ptr,
> +              sizeof (Params_t));
> +    ps_ptr->bpool = DT_BpoolAlloc (pt_ptr,
>
> What I basically did was allocating pt_ptr and
> copying the params to the ds before calling DT_BpoolAlloc.
> This time I call it with pt_ptr instead of NULL (that leads
> to default use of physical memory registration).
>
> (The + + has probably to do with mailer/diff problems)

I'll go through it again when you resend it.


From rolandd at cisco.com  Fri Jul 29 10:04:18 2005
From: rolandd at cisco.com (Roland Dreier)
Date: Fri, 29 Jul 2005 10:04:18 -0700
Subject: [PATCH] Don't open-code spinlock initializer (was: [openib-general]
	Compile issue with 2.6.12 + realtime preemt patch)
In-Reply-To: <527jfcvvpg.fsf@cisco.com> (Roland Dreier's message of "Wed, 27
	Jul 2005 13:29:47 -0700")
References: <42E7E01F.9040500@keysounds.co.uk> <527jfcvvpg.fsf@cisco.com>
Message-ID: <524qadmtm5.fsf_-_@cisco.com>

Libor, I think you missed this patch the first time around.


Initialize spinlock with DEFINE_SPINLOCK() instead of
SPIN_LOCK_UNLOCKED to be friendly to CONFIG_PREEMPT_RT patches.

Signed-off-by: Roland Dreier <rolandd at cisco.com>

--- infiniband/ulp/sdp/sdp_conn.c	(revision 2904)
+++ infiniband/ulp/sdp/sdp_conn.c	(working copy)
@@ -46,7 +46,7 @@ static struct ib_client sdp_client = {
 	.remove = sdp_device_remove_one
 };
 
-static spinlock_t psn_lock = SPIN_LOCK_UNLOCKED;
+static DEFINE_SPINLOCK(psn_lock);
 static u32        psn_seed;
 
 /*


From pw at osc.edu  Fri Jul 29 10:42:25 2005
From: pw at osc.edu (Pete Wyckoff)
Date: Fri, 29 Jul 2005 13:42:25 -0400
Subject: [openib-general] kernel VM monitor for memory registration caching
Message-ID: <20050729174225.GB2683@osc.edu>

Following up on the thread "Question about pinning memory" of last
week or so, I cleaned up the test implementation of the mechanism
I presented and put it on the web for download:

    http://www.osc.edu/~pw/dreg/

This is not meant for submission to openib or to the linux kernel.
It is just a prototype to show that the basic idea should work.

Here's the description from the web page:

    This kernel module and userspace library together implement a
    virtual memory (VM) monitoring system for the linux kernel.
    When communication libraries, such as MPI or PVFS2, support
    network devices that require memory registration, such as
    Infiniband, the libraries tend to register user memory
    dynamically and cache those registrations hoping to later reuse
    them.  Unfortunately, the user application can unmap the memory,
    through a call to free() or explicit munmap(), for example, and
    the communication library never finds out.  With this work, the
    kernel module notices the VM activity and notifies the userspace
    communication library of what happened.

I'll be happy to discuss the code with anyone who ends up wanting
to use it or improve upon it.

		-- Pete


From ardavis at ichips.intel.com  Fri Jul 29 11:24:10 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Fri, 29 Jul 2005 11:24:10 -0700
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <1121183117.4369.97.camel@hal.voltaire.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
	<1121183117.4369.97.camel@hal.voltaire.com>
Message-ID: <42EA744A.4070107@ichips.intel.com>

Hi Hal,

>Hi again Arlin,
>
>On Mon, 2005-07-11 at 18:37, Davis, Arlin R wrote:
>  
>
>>I was attempting to call ib_at_route_by_ip() in sync mode and get the
>>following oops:  
>>
>>You can change the att.c (line 193) example as follow to duplicate:
>>
>>  r = ib_at_route_by_ip(0x1400a8c0, 0, 0, 0,
>>                                      att_rt + i, NULL, NULL);
>>    
>>
>
>I just checked in a new version of uat.c and also the userspace library
>(libibat/src/at.c) which should fix this oops.
>
>Note that I did not unit test not supplying async completion routine (as
>you noticed) so there may be further issues here :-(
>  
>
Looks like there is no support for sync mode. Any idea when this will be 
added?

-arlin

>  
>
>>I will go back to async mode for now.
>>    
>>
>
>This is the safer option right now.
>
>-- Hal
>
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>  
>



From rep.nop at aon.at  Fri Jul 29 11:40:34 2005
From: rep.nop at aon.at (Bernhard Fischer)
Date: Fri, 29 Jul 2005 20:40:34 +0200
Subject: [openib-general] Re: [PATCH][dapl] cleanup dapl_cookie
In-Reply-To: <Pine.LNX.4.61.0506300955000.4300@jlentini-linux.nane.netapp.com>
References: <20050627173951.GB19279@aon.at>
	<Pine.LNX.4.61.0506281538250.4300@jlentini-linux.nane.netapp.com>
	<20050629215046.GB17011@aon.at>
	<Pine.LNX.4.61.0506300955000.4300@jlentini-linux.nane.netapp.com>
Message-ID: <20050729184034.GF15261@aon.at>

On Thu, Jun 30, 2005 at 10:04:36AM -0400, James Lentini wrote:
>
>
>On Wed, 29 Jun 2005, Bernhard Fischer wrote:
>
>>On Tue, Jun 28, 2005 at 03:39:59PM -0400, James Lentini wrote:
>>>
>>>Hi Bernhard,
>>>
>>>The changes look fine. Why the additional copyright? I need to be able
>>>to explain it to my legal department.
>>
>>My legaleeze states that whatever i do during work-time is contributed
>>to work and whatever is related to work done during leasure time has to
>>be attributed to /me _at_ _least_. As that snippet (which was a 
>>test-balloon
>>for that category) clearly was done in my spare time, i'm forced to
>>attribute it accordingly :-/
>>
>>Does that answer your question satisfactorily?
>
>Thanks Bernhard. That makes sense to me. My legal inquired about the 
>"all rights reserved" qualifier. All the copyrights I found in the 
>OpenIB tree (including NetApp's) use that language. I'll run this by 
>them.
>
As rev. 2934 i do not see this patch applied. To recap, it removed some
unneeded local variables (which my compiler wasn't smart enough to
eleminate on it's own -- gcc-4.0 and gcc-HEAD) and simplified some
conditionals and branches.

Back then, i only submitted the changes to dapl_cookie.c to see if such
kind of code simplifications would be accepted or not.

James, can you please elaborate why the patch was rejected?

Thank you,
Bernhard


From halr at voltaire.com  Fri Jul 29 11:42:02 2005
From: halr at voltaire.com (Hal Rosenstock)
Date: 29 Jul 2005 14:42:02 -0400
Subject: [openib-general] Re: ib_uat kernel panic without async callback
In-Reply-To: <42EA744A.4070107@ichips.intel.com>
References: <59278FC0C48A994BABABD069571E45680B63A442@orsmsx401.amr.corp.intel.com>
	<1121183117.4369.97.camel@hal.voltaire.com>
	<42EA744A.4070107@ichips.intel.com>
Message-ID: <1122662229.4605.914.camel@hal.voltaire.com>

On Fri, 2005-07-29 at 14:24, Arlin Davis wrote:
> >Note that I did not unit test not supplying async completion routine (as
> >you noticed) so there may be further issues here :-(
> >  
> >
> Looks like there is no support for sync mode. Any idea when this will be 
> added?

No idea yet. How important is this ? When would you need it ?

-- Hal



From jagana at us.ibm.com  Fri Jul 29 12:21:46 2005
From: jagana at us.ibm.com (Venkata Jagana)
Date: Fri, 29 Jul 2005 12:21:46 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:
	OpenRDMA community development discussion
In-Reply-To: <20050729120416.GB30154@lst.de>
Message-ID: <OF3BDB4043.C6349605-ON8725704D.0069A2A8-8825704D.006A5D5D@us.ibm.com>

Christoph Hellwig <hch at lst.de> wrote on 07/29/2005 05:04:16 AM:

> your card drivers or rather start writing sane ones.  Roland writing the
> new mthca driver of mellanox IB cards is what got the OpenIB ball
> rolling.
> 

Couldn't agree more on this. At a minimum, at least one RNIC vendor
must try to opensource their basic card driver based on the kAL layer
that UNM submitted already into OpenRDMA.

Thanks
Venkat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050729/6abb1281/attachment.html>

From jlentini at netapp.com  Fri Jul 29 12:59:03 2005
From: jlentini at netapp.com (James Lentini)
Date: Fri, 29 Jul 2005 15:59:03 -0400 (EDT)
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22)
	summary:OpenRDMA community development discussion
In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00051442E3@orsmsx408>
References: <1AC79F16F5C5284499BB9591B33D6F00051442E3@orsmsx408>
Message-ID: <Pine.LNX.4.61.0507291337490.12296@jlentini-linux.nane.netapp.com>



On Fri, 29 Jul 2005, Woodruff, Robert J wrote:

> Venkat wrote,
>> If anyone attended any one of the summits (netconf or kernel) and
>> would be great if they can shed some light on this discussion.
>
> Roland attended the kernel summit and he was
> also at the InfiniBand BOF where we discussed
> the possibility of modifying the
> IB verbs to also support iWarp as probably the right way to
> go. Not sure if this was discussed at the kernel summit or not,
> but perhaps Roland can provide some insight on that question.

The subject came up at the kernel summit when Jamal Hadi Salim 
reported on the presentations at Netconf, in particular Stephen 
Hemminger's, see http://vger.kernel.org/netconf2005.html).

It was noted that iWARP vendors are working with the OpenIB community 
on a common interface. The consensus was that this is the right 
direction.


From johnip at sgi.com  Fri Jul 29 14:46:03 2005
From: johnip at sgi.com (John Partridge)
Date: Fri, 29 Jul 2005 16:46:03 -0500
Subject: [openib-general] kdapl build error on ia64
Message-ID: <42EAA39B.2090501@sgi.com>

James,

I am trying to build the kdapl module on an IA64 (SGI Altix) machine using a
2.6.12-gentoo-r6 kernel and am getting the following build error :-

   LD      drivers/infiniband/core/built-in.o
   LD      drivers/infiniband/hw/mthca/built-in.o
   LD      drivers/infiniband/ulp/ipoib/built-in.o
   LD      drivers/infiniband/ulp/kdapl/built-in.o
   CC [M]  drivers/infiniband/ulp/kdapl/api.o
   LD [M]  drivers/infiniband/ulp/kdapl/kdapl.o
   LD      drivers/infiniband/ulp/kdapl/ib/built-in.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_openib_qp.o
In file included from drivers/infiniband/ulp/kdapl/ib/dapl.h:42,
                  from drivers/infiniband/ulp/kdapl/ib/dapl_openib_qp.c:38:
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74:73: pasting "ia64_cmpxchg1_" and ""acq"" does not give a valid preprocessing token
In file included from drivers/infiniband/ulp/kdapl/ib/dapl.h:42,
                  from drivers/infiniband/ulp/kdapl/ib/dapl_openib_qp.c:38:
drivers/infiniband/ulp/kdapl/ib/dapl_util.h: In function `dapl_os_atomic_assign':
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: `ia64_cmpxchg1_' undeclared (first use in this function)
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: (Each undeclared identifier is reported only once
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: for each function it appears in.)
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: syntax error before string constant
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74:73: pasting "ia64_cmpxchg2_" and ""acq"" does not give a valid preprocessing token
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: `ia64_cmpxchg2_' undeclared (first use in this function)
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: syntax error before string constant
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74:73: pasting "ia64_cmpxchg4_" and ""acq"" does not give a valid preprocessing token
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: `ia64_cmpxchg4_' undeclared (first use in this function)
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: syntax error before string constant
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74:73: pasting "ia64_cmpxchg8_" and ""acq"" does not give a valid preprocessing token
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: `ia64_cmpxchg8_' undeclared (first use in this function)
drivers/infiniband/ulp/kdapl/ib/dapl_util.h:74: error: syntax error before string constant
make[4]: *** [drivers/infiniband/ulp/kdapl/ib/dapl_openib_qp.o] Error 1
make[3]: *** [drivers/infiniband/ulp/kdapl/ib] Error 2
make[2]: *** [drivers/infiniband/ulp/kdapl] Error 2
make[1]: *** [drivers/infiniband] Error 2
make: *** [drivers] Error 2

I searched the archives and don't see a solution, but I think I've tracked it down
to a bug in ulp/kdapl/ib/dapl_util.h

root on mig133  > diff -ruN dapl_util.h dapl_util.h-johnip

=============== diff ===============================

	--- dapl_util.h 2005-07-29 16:36:17.514669886 -0500
	+++ dapl_util.h-johnip  2005-07-29 16:37:11.514578548 -0500
	@@ -71,7 +71,7 @@

	 #ifdef __ia64__

	-       current_value = ia64_cmpxchg("acq", v, match_value, new_value, 4);
	+       current_value = ia64_cmpxchg(acq, v, match_value, new_value, 4);

  	#elif defined (__PPC__)

============ diff end =======================================

With this fix the ia64 modules all build to completion with just a couple of warnings :-

/usr/src/linux
root on mig133  > make
   CHK     include/linux/version.h
make[1]: `arch/ia64/kernel/asm-offsets.s' is up to date.
   CHK     include/asm-ia64/offsets.h
   CHK     include/linux/compile.h
   CHK     usr/initramfs_list
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_openib_qp.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_openib_util.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_openib_cm.o
drivers/infiniband/ulp/kdapl/ib/dapl_openib_cm.c: In function `dapl_path_comp_handler':
drivers/infiniband/ulp/kdapl/ib/dapl_openib_cm.c:346: warning: long long int format, u64 arg (arg 3)
drivers/infiniband/ulp/kdapl/ib/dapl_openib_cm.c: In function `dapl_rt_comp_handler':
drivers/infiniband/ulp/kdapl/ib/dapl_openib_cm.c:416: warning: long long int format, u64 arg (arg 3)
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_cookie.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_cr.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_ep.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_evd.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_hca_util.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_ia.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_lmr.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_provider.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_pz.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_ring_buffer_util.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_rmr.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_sp.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_srq.o
   CC [M]  drivers/infiniband/ulp/kdapl/ib/dapl_util.o
   LD [M]  drivers/infiniband/ulp/kdapl/ib/kdapl_ib.o
   LD      drivers/infiniband/ulp/sdp/built-in.o
   LD      drivers/infiniband/ulp/srp/built-in.o
   GEN     .version
   CHK     include/linux/compile.h
   UPD     include/linux/compile.h
   CC      init/version.o
   LD      init/built-in.o
   LD      .tmp_vmlinux1


I have not yet tested kdapl, but will go ahead and try it and let you know how it works on IA64

John


-- 
John Partridge

Silicon Graphics Inc
Tel:  651-683-3428
Vnet: 233-3428
E-Mail: johnip at sgi.com


From shemminger at osdl.org  Fri Jul 29 15:52:56 2005
From: shemminger at osdl.org (Stephen Hemminger)
Date: Fri, 29 Jul 2005 15:52:56 -0700
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22)
	summary:OpenRDMA community development discussion
In-Reply-To: <Pine.LNX.4.61.0507291337490.12296@jlentini-linux.nane.netapp.com>
References: <1AC79F16F5C5284499BB9591B33D6F00051442E3@orsmsx408>
	<Pine.LNX.4.61.0507291337490.12296@jlentini-linux.nane.netapp.com>
Message-ID: <20050729155256.06b37931@localhost.localdomain>

On Fri, 29 Jul 2005 15:59:03 -0400 (EDT)
James Lentini <jlentini at netapp.com> wrote:

> 
> 
> On Fri, 29 Jul 2005, Woodruff, Robert J wrote:
> 
> > Venkat wrote,
> >> If anyone attended any one of the summits (netconf or kernel) and
> >> would be great if they can shed some light on this discussion.
> >
> > Roland attended the kernel summit and he was
> > also at the InfiniBand BOF where we discussed
> > the possibility of modifying the
> > IB verbs to also support iWarp as probably the right way to
> > go. Not sure if this was discussed at the kernel summit or not,
> > but perhaps Roland can provide some insight on that question.
> 
> The subject came up at the kernel summit when Jamal Hadi Salim 
> reported on the presentations at Netconf, in particular Stephen 
> Hemminger's, see http://vger.kernel.org/netconf2005.html).
> 
> It was noted that iWARP vendors are working with the OpenIB community 
> on a common interface. The consensus was that this is the right 
> direction.
>

The consensus was more of wait and see what comes up.  My discussion
at netconf was more of an informative session to get the participants to
know that activity is going on and some code may be coming. It was all
of 5 minutes at the end of a busy day, so it didn't count for much.

At the KS, Jamal's presentation caused some response. Which I don't
remember exactly, but Christoph may be able to fill in the details.


From ardavis at ichips.intel.com  Fri Jul 29 17:25:14 2005
From: ardavis at ichips.intel.com (Arlin Davis)
Date: Fri, 29 Jul 2005 17:25:14 -0700
Subject: [openib-general] [PATCH] uDAPL makefile
Message-ID: <42EAC8EA.80706@ichips.intel.com>

James,

no longer need to link with mthca.so

-arlin

Signed-off by: Arlin Davis <ardavis at ichips.intel.com>


Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile (revision 2919)
+++ dapl/udapl/Makefile (working copy)
@@ -122,7 +122,8 @@
 #
 ifeq ($(VERBS),openib)
 PROVIDER = $(TOPDIR)/../openib
-CFLAGS   += -DOPENIB -DCQ_WAIT_OBJECT
+CFLAGS   += -DOPENIB
+#CFLAGS   += -DCQ_WAIT_OBJECT
 CFLAGS   += -I/usr/local/include/infiniband
 endif

@@ -232,9 +233,8 @@
 endif

 ifeq ($(VERBS),openib)
-LDFLAGS += -libverbs /usr/local/lib/infiniband/mthca.so -libcm
+LDFLAGS += -libverbs -libcm -libat
 LDFLAGS += -rpath /usr/local/lib -L /usr/local/lib
-LDFLAGS += -rpath /usr/local/lib/infiniband -L /usr/local/lib/infiniband
 PROVIDER_SRCS  = dapl_ib_util.c dapl_ib_cq.c dapl_ib_qp.c
 PROVIDER_SRCS += dapl_ib_cm.c dapl_ib_mem.c
 endif



From mlleinin at hpcn.ca.sandia.gov  Sat Jul 30 17:37:48 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Sat, 30 Jul 2005 17:37:48 -0700
Subject: [openib-general] OpenIB server is back up
Message-ID: <1122770268.4431.298.camel@localhost>


  The OpenIB webpages, mail list, and svn are back up.


   - Matt



From mst at mellanox.co.il  Sun Jul 31 01:51:45 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Sun, 31 Jul 2005 11:51:45 +0300
Subject: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range ->
	io_remap_pfn_range
In-Reply-To: <20050728161720.GA23507@suse.de>
References: <A95E2296287EAD4EB592B5DEEFCE0E9D282808@liverpoolst.ad.cl.cam.ac.uk>
	<20050728161720.GA23507@suse.de>
Message-ID: <20050731085145.GE14384@mellanox.co.il>

Quoting r. Greg KH <gregkh at suse.de>:
> Subject: Re: [openib-general] Re: [PATCH] arch/xx/pci: remap_pfn_range -> io_remap_pfn_range
> If this is a fix for xen, fine, then say so in the changelog information
> for the patch, as it is, no such information was given.
> 
> thanks,
> 
> greg k-h
> 

Like this?

---

Convert i386/pci to use io_remap_pfn_range instead of remap_pfn_range.
This is good for Xen which reuses i386/pci/i386.c for domain 0 code.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/arch/i386/pci/i386.c
===================================================================
--- linux-2.6.12.2.orig/arch/i386/pci/i386.c
+++ linux-2.6.12.2/arch/i386/pci/i386.c
@@ -295,9 +295,9 @@ int pci_mmap_page_range(struct pci_dev *
 	/* Write-combine setting is ignored, it is changed via the mtrr
 	 * interfaces on this platform.
 	 */
-	if (remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
-			     vma->vm_end - vma->vm_start,
-			     vma->vm_page_prot))
+	if (io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+			       vma->vm_end - vma->vm_start,
+			       vma->vm_page_prot))
 		return -EAGAIN;
 
 	return 0;

-- 
MST


From glebn at voltaire.com  Sun Jul 31 03:31:57 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Sun, 31 Jul 2005 13:31:57 +0300
Subject: [openib-general] kernel VM monitor for memory registration caching
In-Reply-To: <20050729174225.GB2683@osc.edu>
References: <20050729174225.GB2683@osc.edu>
Message-ID: <20050731103157.GT13014@minantech.com>

Hello Pete,

On Fri, Jul 29, 2005 at 01:42:25PM -0400, Pete Wyckoff wrote:
> I'll be happy to discuss the code with anyone who ends up wanting
> to use it or improve upon it.

I glanced over the code and I have couple of questions/improvments.

 First of all, you have one user_delta per mm that user can poll from 
userspace. Is it possible to make user_delta to be part of dreg_region
instead of dreg_context and module will set it whenever
registration becomes invalid. Field 'invalid' will be added to buf_info
structure and pointer to it will be passed to kernel at registration
time.
 This way the userpace can look up cache and check if registration is
still valid. No need to rescan cache from userspace, we already scanned
it once from kernel after all. With your current approach userspace will
need to search for mr_handle in the cache and invalidate the entry that
holds it.


 You change vma_ops in vma to catch open/close events. What about
nopage() method in vma_ops? We have to forward it to original vma_ops?

Something like included patch (not even compiled).


--- dreg.c.org	2005-07-31 13:10:17.375403091 +0300
+++ dreg.c	2005-07-31 13:24:35.404872561 +0300
@@ -162,7 +162,10 @@
 
     pr_debug("%s: reg %p vma %p addr %lx\n", __func__, reg, vma, reg->addr);
     if (vma)
+    {
+	kfree (vma->vm_ops);
 	vma->vm_ops = reg->orig_ops;
+    }
     if (reg->addr)
 	mem_deregister(dc, reg);
     list_del(&reg->subordinate_list);
@@ -305,6 +308,7 @@
      * forget about it and do not build a new region for it.
      */
     if (list_empty(&temp_new_subordinate_list)) {
+	kfree (newvma->vm_ops);
 	newvma->vm_ops = orig_ops;
     } else {
 	reg = kmem_cache_alloc(dreg_region_cache, GFP_KERNEL);
@@ -510,7 +514,7 @@
       vma->vm_start, vma->vm_end, reg);
 
     reg->orig_ops = vma->vm_ops;
-    if (vma->vm_ops == &dreg_vm_ops) {
+    if (vma->vm_ops->close == dreg_vm_ops.close) {
 	/* chain off proper owner */
 	struct dreg_region *topreg;
 	pr_debug("%s: marked subordinate\n", __func__);
@@ -523,10 +527,22 @@
 	}
 	list_add(&reg->subordinate_list, &topreg->subordinate_list);
     } else {
+	struct vm_operations_struct *tmp_vm_ops;
 	/* non subordinate */
 	reg->vma = vma;
 	INIT_LIST_HEAD(&reg->subordinate_list);
-	vma->vm_ops = &dreg_vm_ops;  /* own this vma */
+
+	tmp_vm_ops = kmalloc (sizeof (struct vm_operations_struct), GFP_KERNEL);
+	memcpy (tmp_vm_ops, &dreg_vm_ops, sizeof (struct vm_operations_struct));
+	if (vma->vm_ops)
+	{
+		tmp_vm_ops->nopage = vma->vm_ops->nopage;
+#ifdef CONFIG_NUMA
+		tmp_vm_ops->set_policy = vma->vm_ops->set_policy;
+		tmp_vm_ops->get_policy = vma->vm_ops->get_policy;
+#endif
+	}
+	vma->vm_ops = tmp_vm_ops;  /* own this vma */
 	reg->orig_vm_start = vma->vm_start;
 	reg->orig_vm_end = vma->vm_end;
     }
--
			Gleb.


From mulix at mulix.org  Sun Jul 31 07:20:58 2005
From: mulix at mulix.org (Muli Ben-Yehuda)
Date: Sun, 31 Jul 2005 17:20:58 +0300
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
In-Reply-To: <Pine.LNX.4.63.0507281007040.20468@beid.voltaire.com>
References: <Pine.LNX.4.63.0507281007040.20468@beid.voltaire.com>
Message-ID: <20050731142058.GI20524@granada.merseine.nu>

On Thu, Jul 28, 2005 at 10:14:15AM +0300, Guy German wrote:

> [kdapl]: increments the module ref count on ia_open, thus making
> sure the kdapl_ib module would not be rmmod-ed while in use
> 
> Signed-off-by: Guy German <guyg at voltaire.com>
> 
> Index: infiniband/ulp/kdapl/ib/dapl_ia.c
> ===================================================================
> --- infiniband/ulp/kdapl/ib/dapl_ia.c	(revision 2914)
> +++ infiniband/ulp/kdapl/ib/dapl_ia.c	(working copy)
> @@ -649,6 +649,7 @@ int dapl_ia_open(const char *name, int a
>  	status = 0;
>  	*ia_ptr = (struct dat_ia *)ia;
>  	*async_evd = (struct dat_evd *)evd;
> +	try_module_get(THIS_MODULE);

This does not look right. First, try_module_get() may fail. Second,
you're calling it in the context of your module - what happens if
someone tried to remove it while dapl_ia_open() is running and before
the call? 

The right place to do module refcounting is in the caller, not in the
callee. In this case - whereever dapl_ia_open() can be called from.

>  bail:
>  	if (status != 0)
> @@ -679,6 +680,7 @@ int dapl_ia_close(struct dat_ia *ia_ptr,
>  	else
>  		status = -EINVAL;
> 
> +	module_put(THIS_MODULE);

Likewise.

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/



From mulix at mulix.org  Sun Jul 31 07:22:05 2005
From: mulix at mulix.org (Muli Ben-Yehuda)
Date: Sun, 31 Jul 2005 17:22:05 +0300
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
In-Reply-To: <Pine.LNX.4.61.0507281447020.12296@jlentini-linux.nane.netapp.com>
References: <Pine.LNX.4.63.0507281007040.20468@beid.voltaire.com>
	<Pine.LNX.4.61.0507281447020.12296@jlentini-linux.nane.netapp.com>
Message-ID: <20050731142205.GJ20524@granada.merseine.nu>

On Thu, Jul 28, 2005 at 03:01:24PM -0400, James Lentini wrote:

> We've been using the following convention to indicate that we don't 
> care about the return value:
> 
>  (void) try_module_get(THIS_MODULE);

This seems counter intuitive to me (the idea, not the syntax) -
try_module_get() can and will fail. If it doesn't matter that it
succeeds or fails, why call it at all?

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/



From guyg at voltaire.com  Sun Jul 31 08:08:11 2005
From: guyg at voltaire.com (Guy German)
Date: Sun, 31 Jul 2005 18:08:11 +0300
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
Message-ID: <D4F8F0B3820E754C887699BEF26A89405BB3CA@taurus.voltaire.com>

Hi Muli,

Muli Ben-Yehuda <mailto:mulix at mulix.org> wrote:
> On Thu, Jul 28, 2005 at 10:14:15AM +0300, Guy German wrote:
> 
>> [kdapl]: increments the module ref count on ia_open, thus making
>> sure the kdapl_ib module would not be rmmod-ed while in use
>> 
>> Signed-off-by: Guy German <guyg at voltaire.com>
>> 
>> Index: infiniband/ulp/kdapl/ib/dapl_ia.c
>> ===================================================================
>> --- infiniband/ulp/kdapl/ib/dapl_ia.c	(revision 2914)
>> +++ infiniband/ulp/kdapl/ib/dapl_ia.c	(working copy)
>> @@ -649,6 +649,7 @@ int dapl_ia_open(const char *name, int a 
>> status = 0; 
>>  	*ia_ptr = (struct dat_ia *)ia;
>>  	*async_evd = (struct dat_evd *)evd;
>> +	try_module_get(THIS_MODULE);
> 
> This does not look right. First, try_module_get() may fail. Second,
> you're calling it in the context of your module - what happens if
> someone tried to remove it while dapl_ia_open() is running and before
> the call? 

Wouldn't it be solved by moving the try_module_get call to the 
beginning of the dapl_ia_open function ? 

You are right - try_module_get() can fail when the module is not ready 
to be entered. should be something like: 
+ if (!try_module_get(THIS_MODULE))
+	return -EBUSY;

Guy.

> 
> The right place to do module refcounting is in the caller, not in the
> callee. In this case - whereever dapl_ia_open() can be called from.
> 
>>  bail:
>>  	if (status != 0)
>> @@ -679,6 +680,7 @@ int dapl_ia_close(struct dat_ia *ia_ptr,  	else
>>  		status = -EINVAL;
>> 
>> +	module_put(THIS_MODULE);
> 
> Likewise.
> 
> Cheers,
> Muli


From mulix at mulix.org  Sun Jul 31 08:13:30 2005
From: mulix at mulix.org (Muli Ben-Yehuda)
Date: Sun, 31 Jul 2005 18:13:30 +0300
Subject: [openib-general][PATCH][kdapl]: inc/dec module ref count
In-Reply-To: <D4F8F0B3820E754C887699BEF26A89405BB3CA@taurus.voltaire.com>
References: <D4F8F0B3820E754C887699BEF26A89405BB3CA@taurus.voltaire.com>
Message-ID: <20050731151330.GN20524@granada.merseine.nu>

On Sun, Jul 31, 2005 at 06:08:11PM +0300, Guy German wrote:
> Hi Muli,
> 
> Wouldn't it be solved by moving the try_module_get call to the 
> beginning of the dapl_ia_open function ? 

No. Even if it's theoretically the first line in the function, the
compiler can and will create a function prologue that will run before
you raise the reference count (same thing with decrementing the ref
count at module unload time and the function epilogue). You must do
module reference counting before executing even one instruction from
the module.

> You are right - try_module_get() can fail when the module is not ready 
> to be entered. should be something like: 
> + if (!try_module_get(THIS_MODULE))
> +	return -EBUSY;

Yes - but at the caller, not callee.

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/



From saito at venusnetwoerk.cx  Sun Jul 31 08:59:44 2005
From: saito at venusnetwoerk.cx (saito at venusnetwoerk.cx)
Date: Sun, 31 Jul 2005 08:59:44 -0700 (PDT)
Subject: [openib-general] =?iso-2022-jp?b?UmU6GyRCQGhGfCRON28kRxsoQg==?=
Message-ID: 20050801004346.87155mail@mail.ppp87521dood-system854.happyfan.system1.com

ご無沙汰しております。

女性紹介サークル『Venus Network』
代表　富田ゆかりでございます。

簡単に当サークルのご説明をさせて頂きますので
下記をご覧くださいませ。

当サークル男性会員様の利用明細でございます。


−−会員Ａ様のサークル利用明細−−

入会金 ：０円
年会費 ：０円
運営費 ：０円
女性紹介料 ：０円
************************

小計 ：０円
税 ：０円
合計 ：０円
************************

−−−−−−−−−−−−−−−−−

この明細をご覧いただき
おわかりになると思いますが、

当Venus Networkは
紹介料などの諸費用は一切無用の
完全無料の出会いサークルでございます。

そして、ご紹介するだけではなく
紹介後の女性とのお付き合いの仕方

いわゆる女心についての疑問や質問などに対しても全面的に
バックアップさせて頂きます。

その都度カスタマーセンターの者が
丁寧に対応致しますのでご安心くださいませ。

ご紹介につきましては日本全国に
多数の女性会員様がおりますので

まず貴方様の

・お住まいの地域
・理想の交際方法

以上２点をお聞きした上で

貴方様にあった女性をピックアップし
ご紹介させて頂きます。

お手数ですが下記の質問にお答えいただき
ご返信くださいませ。

■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■


◆Ｑ１、貴方様がお住まいの都道府県名

　　　ご記入下さい。『　　　　　』


◆Ｑ２、女性会員と理想の交際方法

１、真面目にお付き合い　　（真面目な交際・結婚前提）
２、高所得女性とお付き合い（事業資金などサポートして欲しい）
３、割り切ったお付き合い　（愛人・セフレ・人妻）
４、過激なお付き合い　　　（３Ｐ・乱交・ＳＭ）

　　　数字でご記入下さい。『　　』

■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■

貴方様からのご返信を頂きましたら好みの近い女性を
ご紹介させて頂きます。

それではご連絡お待ちしております。


Venus Network 代表　富田ゆかり




From mlleinin at hpcn.ca.sandia.gov  Sun Jul 31 14:57:01 2005
From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger)
Date: Sun, 31 Jul 2005 14:57:01 -0700
Subject: [openib-general] kernel VM monitor for memory registration caching
In-Reply-To: <20050731103157.GT13014@minantech.com>
References: <20050729174225.GB2683@osc.edu>
	<20050731103157.GT13014@minantech.com>
Message-ID: <1122847021.28769.25.camel@localhost>

FWIW here is a link to the quadrics vm patch that David Addison posted
on LKML a couple of months ago.  

http://lkml.org/lkml/2005/4/26/198

  - Matt




On Sun, 2005-07-31 at 13:31 +0300, Gleb Natapov wrote:
> Hello Pete,
> 
> On Fri, Jul 29, 2005 at 01:42:25PM -0400, Pete Wyckoff wrote:
> > I'll be happy to discuss the code with anyone who ends up wanting
> > to use it or improve upon it.
> 
> I glanced over the code and I have couple of questions/improvments.
> 
>  First of all, you have one user_delta per mm that user can poll from 
> userspace. Is it possible to make user_delta to be part of dreg_region
> instead of dreg_context and module will set it whenever
> registration becomes invalid. Field 'invalid' will be added to buf_info
> structure and pointer to it will be passed to kernel at registration
> time.
>  This way the userpace can look up cache and check if registration is
> still valid. No need to rescan cache from userspace, we already scanned
> it once from kernel after all. With your current approach userspace will
> need to search for mr_handle in the cache and invalidate the entry that
> holds it.
> 
> 
>  You change vma_ops in vma to catch open/close events. What about
> nopage() method in vma_ops? We have to forward it to original vma_ops?
> 
> Something like included patch (not even compiled).
> 
> 
> --- dreg.c.org	2005-07-31 13:10:17.375403091 +0300
> +++ dreg.c	2005-07-31 13:24:35.404872561 +0300
> @@ -162,7 +162,10 @@
>  
>      pr_debug("%s: reg %p vma %p addr %lx\n", __func__, reg, vma, reg->addr);
>      if (vma)
> +    {
> +	kfree (vma->vm_ops);
>  	vma->vm_ops = reg->orig_ops;
> +    }
>      if (reg->addr)
>  	mem_deregister(dc, reg);
>      list_del(&reg->subordinate_list);
> @@ -305,6 +308,7 @@
>       * forget about it and do not build a new region for it.
>       */
>      if (list_empty(&temp_new_subordinate_list)) {
> +	kfree (newvma->vm_ops);
>  	newvma->vm_ops = orig_ops;
>      } else {
>  	reg = kmem_cache_alloc(dreg_region_cache, GFP_KERNEL);
> @@ -510,7 +514,7 @@
>        vma->vm_start, vma->vm_end, reg);
>  
>      reg->orig_ops = vma->vm_ops;
> -    if (vma->vm_ops == &dreg_vm_ops) {
> +    if (vma->vm_ops->close == dreg_vm_ops.close) {
>  	/* chain off proper owner */
>  	struct dreg_region *topreg;
>  	pr_debug("%s: marked subordinate\n", __func__);
> @@ -523,10 +527,22 @@
>  	}
>  	list_add(&reg->subordinate_list, &topreg->subordinate_list);
>      } else {
> +	struct vm_operations_struct *tmp_vm_ops;
>  	/* non subordinate */
>  	reg->vma = vma;
>  	INIT_LIST_HEAD(&reg->subordinate_list);
> -	vma->vm_ops = &dreg_vm_ops;  /* own this vma */
> +
> +	tmp_vm_ops = kmalloc (sizeof (struct vm_operations_struct), GFP_KERNEL);
> +	memcpy (tmp_vm_ops, &dreg_vm_ops, sizeof (struct vm_operations_struct));
> +	if (vma->vm_ops)
> +	{
> +		tmp_vm_ops->nopage = vma->vm_ops->nopage;
> +#ifdef CONFIG_NUMA
> +		tmp_vm_ops->set_policy = vma->vm_ops->set_policy;
> +		tmp_vm_ops->get_policy = vma->vm_ops->get_policy;
> +#endif
> +	}
> +	vma->vm_ops = tmp_vm_ops;  /* own this vma */
>  	reg->orig_vm_start = vma->vm_start;
>  	reg->orig_vm_end = vma->vm_end;
>      }
> --
> 			Gleb.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 



From yaronh at voltaire.com  Sun Jul 31 20:56:42 2005
From: yaronh at voltaire.com (Yaron Haviv)
Date: Mon, 1 Aug 2005 06:56:42 +0300
Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22)
	summary:OpenRDMA community development discussion
Message-ID: <35EA21F54A45CB47B879F21A91F4862F6C59C9@taurus.voltaire.com>

> -----Original Message-----
> From: openib-general-bounces at openib.org [mailto:openib-general-
> bounces at openib.org] On Behalf Of Christoph Hellwig
> Sent: Friday, July 29, 2005 8:02 AM
> To: Tom Duffy
> Cc: Venkata Jagana; rdma-developers at lists.sourceforge.net; Christoph
> Hellwig; openib-general at openib.org
> Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22)
> summary:OpenRDMA community development discussion
> 
> On Thu, Jul 28, 2005 at 02:02:08PM -0700, Tom Duffy wrote:
> > At OLS (and in previous forums), the kernel maintainers have made it
> > *very* clear that there should only be one API.
> 
> _and_ that this api is neither RNIC-PI or KDAPL.  In fact for anything
> that doesn't look very similar to the current IB midlayer you'd need
> very convincing arguments.
> 

I assume it is not as simplistic as that 
iWarp CM model is quite different than IB, and iWarp doesn't have SA/SM
and a bunch of other IB specific things 

For example: 
The correct common abstraction is one where a user can issue a
connection by using a logical end-point address (such as an IP), and
doesn't have to deal with the IB or iWarp specific CM state machine or
SA/SM. 

If you look at DAPL you can break it to simple Verbs (e.g. send, ..)
where its just a simple overlay on to of the verbs (and may be
redundant) 
However there is a second part that implements a simple connection
establishment model (much like BSD) that can be mapped to both IB (CM,
SA, ..) or iWarp (TCP Syn/SynAck, ARP, etc'), this serves couple of main
purposes:
a. make it simple for ULP developer and put the complex part in a common
place   
b. define a common model for different HW

we can spend time and discuss theories and intentions, at the end of the
day an iWarp RNIC cannot just reside under IB-Verbs without major
changes to the overall infrastructure.
Several guys spent some time looking it over and came with an
abstraction that IS possible on top of IB & iWarp & foo, that is called
DAPL (or IT as another similar alternative)

It would probably be wise to try and merge that effort with IB-verbs
etc' 
(e.g. make the verbs portion of the API closer), and on the same time
preserve the effort that was done in kDAPL to overcome the differences
(e.g. in the CM, addressing portions)

Yaron

> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-
> general


From mst at mellanox.co.il  Sun Jul 31 23:35:29 2005
From: mst at mellanox.co.il (Michael S. Tsirkin)
Date: Mon, 1 Aug 2005 09:35:29 +0300
Subject: [openib-general] [PATCH] remove in_atomic
Message-ID: <20050801063529.GO14384@mellanox.co.il>

in_atomic isnt a reliable way to check that we are in an atomic context.
Just schedule work, always, since most cq polling is currently done under
a spinlock, anyway.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_iocb.c
===================================================================
--- linux-2.6.12.2.orig/drivers/infiniband/ulp/sdp/sdp_iocb.c
+++ linux-2.6.12.2/drivers/infiniband/ulp/sdp/sdp_iocb.c
@@ -325,12 +325,8 @@ static void do_iocb_complete(void *arg)
 void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status)
 {
 	iocb->status = status;
-	
-	if (in_atomic() || irqs_disabled()) {
-		INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb);
-		schedule_work(&iocb->completion);
-	} else
-		do_iocb_complete(iocb);
+	INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb);
+	schedule_work(&iocb->completion);
 }
 
 /*

-- 
MST


From glebn at voltaire.com  Sun Jul 31 23:55:03 2005
From: glebn at voltaire.com (Gleb Natapov)
Date: Mon, 1 Aug 2005 09:55:03 +0300
Subject: [openib-general] kernel VM monitor for memory registration caching
In-Reply-To: <1122847021.28769.25.camel@localhost>
References: <20050729174225.GB2683@osc.edu>
	<20050731103157.GT13014@minantech.com>
	<1122847021.28769.25.camel@localhost>
Message-ID: <20050801065503.GX13014@minantech.com>

On Sun, Jul 31, 2005 at 02:57:01PM -0700, Matt Leininger wrote:
> FWIW here is a link to the quadrics vm patch that David Addison posted
> on LKML a couple of months ago.  
> 
> http://lkml.org/lkml/2005/4/26/198
> 
I remember this thread. David's patch implements different approach to the
problem. Pete's patch do monitoring on VMA layer, David's patch do
monitoring on PTE level. Pete's patch help userspace detect pinned pages
that no longer accessible from userspace an thus can be unpinned, David's
patch tries to eliminate need in memory pinning an requires support from
driver I think.

>   - Matt
> 
> 
> 
> 
> On Sun, 2005-07-31 at 13:31 +0300, Gleb Natapov wrote:
> > Hello Pete,
> > 
> > On Fri, Jul 29, 2005 at 01:42:25PM -0400, Pete Wyckoff wrote:
> > > I'll be happy to discuss the code with anyone who ends up wanting
> > > to use it or improve upon it.
> > 
> > I glanced over the code and I have couple of questions/improvments.
> > 
> >  First of all, you have one user_delta per mm that user can poll from 
> > userspace. Is it possible to make user_delta to be part of dreg_region
> > instead of dreg_context and module will set it whenever
> > registration becomes invalid. Field 'invalid' will be added to buf_info
> > structure and pointer to it will be passed to kernel at registration
> > time.
> >  This way the userpace can look up cache and check if registration is
> > still valid. No need to rescan cache from userspace, we already scanned
> > it once from kernel after all. With your current approach userspace will
> > need to search for mr_handle in the cache and invalidate the entry that
> > holds it.
> > 
> > 
> >  You change vma_ops in vma to catch open/close events. What about
> > nopage() method in vma_ops? We have to forward it to original vma_ops?
> > 
> > Something like included patch (not even compiled).
> > 
> > 
> > --- dreg.c.org	2005-07-31 13:10:17.375403091 +0300
> > +++ dreg.c	2005-07-31 13:24:35.404872561 +0300
> > @@ -162,7 +162,10 @@
> >  
> >      pr_debug("%s: reg %p vma %p addr %lx\n", __func__, reg, vma, reg->addr);
> >      if (vma)
> > +    {
> > +	kfree (vma->vm_ops);
> >  	vma->vm_ops = reg->orig_ops;
> > +    }
> >      if (reg->addr)
> >  	mem_deregister(dc, reg);
> >      list_del(&reg->subordinate_list);
> > @@ -305,6 +308,7 @@
> >       * forget about it and do not build a new region for it.
> >       */
> >      if (list_empty(&temp_new_subordinate_list)) {
> > +	kfree (newvma->vm_ops);
> >  	newvma->vm_ops = orig_ops;
> >      } else {
> >  	reg = kmem_cache_alloc(dreg_region_cache, GFP_KERNEL);
> > @@ -510,7 +514,7 @@
> >        vma->vm_start, vma->vm_end, reg);
> >  
> >      reg->orig_ops = vma->vm_ops;
> > -    if (vma->vm_ops == &dreg_vm_ops) {
> > +    if (vma->vm_ops->close == dreg_vm_ops.close) {
> >  	/* chain off proper owner */
> >  	struct dreg_region *topreg;
> >  	pr_debug("%s: marked subordinate\n", __func__);
> > @@ -523,10 +527,22 @@
> >  	}
> >  	list_add(&reg->subordinate_list, &topreg->subordinate_list);
> >      } else {
> > +	struct vm_operations_struct *tmp_vm_ops;
> >  	/* non subordinate */
> >  	reg->vma = vma;
> >  	INIT_LIST_HEAD(&reg->subordinate_list);
> > -	vma->vm_ops = &dreg_vm_ops;  /* own this vma */
> > +
> > +	tmp_vm_ops = kmalloc (sizeof (struct vm_operations_struct), GFP_KERNEL);
> > +	memcpy (tmp_vm_ops, &dreg_vm_ops, sizeof (struct vm_operations_struct));
> > +	if (vma->vm_ops)
> > +	{
> > +		tmp_vm_ops->nopage = vma->vm_ops->nopage;
> > +#ifdef CONFIG_NUMA
> > +		tmp_vm_ops->set_policy = vma->vm_ops->set_policy;
> > +		tmp_vm_ops->get_policy = vma->vm_ops->get_policy;
> > +#endif
> > +	}
> > +	vma->vm_ops = tmp_vm_ops;  /* own this vma */
> >  	reg->orig_vm_start = vma->vm_start;
> >  	reg->orig_vm_end = vma->vm_end;
> >      }
> > --
> > 			Gleb.
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> > 
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> > 

--
			Gleb.